fl-aws
Moto
Our great sponsors
fl-aws | Moto | |
---|---|---|
1 | 32 | |
15 | 7,387 | |
- | 1.4% | |
0.0 | 9.8 | |
over 7 years ago | 3 days ago | |
Python | ||
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
fl-aws
-
Ask HN: Does anyone else find the AWS Lambda developer experience poor?
Living in my team's bubble i thought everyone runs or tries to run parallel environments: prod, staging, dev, but also an individual (person) or feature env. Why? Because there's no emulator or documentation that will teach you real behavior. Like others have said, AWS seems out of this world. Just like GCP and Azure i might add. Some things you don't expect and they mesmerize you how smart they are. Some you expect and you can't fathom how come you're the "only" one screaming. Random thought: this is how i ended up logging all I bumped into into "Fl-aws" https://github.com/andreineculau/fl-aws
Back to the point: reality is that many build their AWS environment (prod) manually, maybe they duplicate once (dev) also manually, maybe they use some automation for their "code" (lambda) but that's it. This implies it's practically impossible to run end-to-end tests. You can't do that in prod for obvious reasons and you can't do it in dev either - you have many devs queueing, maybe dev is not in sync with prod etc.
My team ran cloudformation end-to-end. We actually orchestrated and wrapped cloudformation (this is yet another topic for not using terraform etc) so that if smth couldn't be done in CFN, it would still be automated and reproducible. Long story short, in 30 minutes (it was this long because we had to wait for cloudfront etc) we had a new environment, ready to play with. A total sandbox. Every dev had their own and it was easy to deploy from a release artifact or a git branch to this environment. Similarly you could create a separate env for more elaborate changes to the architecture. And test in a live environment.
Finally to your question: how do you test end-to-end?
If we talk about lambdas because that's where the business logic lies in a "serverless" architecture, then the answer is by calling the system which will eventually call your lambda/s along the way. If your lambda ia sitting behind AWS gateway, then fire an http request. Is it triggered when objects land on S3? Then push some object to S3. How do you assert? Just the same - http response, S3 changes etc. Not to mention you can also check cloudwatch for specific log entries (though they are not instant).
With this type of a setup, which sounds complex, but it is not since it is 100% reproducible (also from project to project - I had several), adding this proxy-to-my-dev-machine lambda would mean I can make local changes and then fire unit AND end-to-end tests without any changes pushed to AWS, which is the main time/energy consumer imo.
PS: sorry for the wall of text. Like i said i recently realized that the development realities have huge discrepancies, so i tried to summarize my reality :)
Moto
-
OpenTF Announces Fork of Terraform
> OpenMoto
I dunno if you're trying to play on "hashimoto" but https://github.com/getmoto/moto#readme would be a prime name collision for any such "OpenMoto" name
But yes, please, to adopting Vault. I don't have a horse in the race about Consul but my suspicion is such an effort would only be worthwhile if trying to adopt Nomad, too, which I gravely doubt
-
Class Credentials does not exist
Unfortunately I do not believe AWS provides any "test" gateways. I do know there are mock AWS servers you can run on your own. The one I use is called Moto. It does not cover everything (unfortunately it's the most comprehensive out there AFAIK), but it's decent enough to test most standard calls via the sdk. I'm not sure if it covers authorization though...we tend to use security roles on tasks for authorization.
-
What is the development enviroment for AWS?
If using Python use Moto to mock AWS Services
-
Unit testing Athena ETL?
You can use a library such as moto https://github.com/getmoto/moto
- Looking for resources for building unit testing for boto3 code and mocking AWS services in pytest
-
Guide to AWS Serverless & Lambda Testing Best Practices — Part 1
The Pythonic motto library mocks AWS services, removing the need to deploy your application or pay for API calls against AWS services. Other programming languages have their motto implementation.
-
Mock AWS Services on Docker
Has anyone managed to configure moto (https://docs.getmoto.org/en/latest/) in a docker container in the similar way LocalStack does?
-
Unit Testing an Airflow Dag
As for mocking, you can take a look at the moto library for mocking the AWS SDK, or for more simple cases even just use a `unittest.Mock/MagicMock` object. If you're having trouble trying to use the mocks in your code, it's a good sign your code is too highly coupled and it'd pay to re-factor, for example using dependency injection, design patterns like adapter/facade etc. (but don't over-do it)
- Final FLiP Stack Weekly of 2022
-
Do unit tests make sense here?
To add on to the integration tests point, for mocking out your AWS resources you should check out moto if you don't want run your test against real AWS resources as they may cost you and is usually slower.
What are some alternatives?
aws-lambda-runtime-interface-emulator
LocalStack - 💻 A fully functional local AWS cloud stack. Develop and test your cloud & Serverless apps offline
aws-sdk-go - AWS SDK for the Go programming language.
aws-cdk-local - Thin wrapper script for using the AWS CDK CLI with LocalStack
VCR.py - Automatically mock your HTTP interactions to simplify and speed up testing
responses - A utility for mocking out the Python Requests library.
freezegun - Let your Python tests travel through time
Mocket - a socket mock framework - for all kinds of socket animals, web-clients included
betamax - A VCR imitation designed only for python-requests.
mock - The Python mock library
httpretty - Intercept HTTP requests at the Python socket level. Fakes the whole socket module
S3Mock - A simple mock implementation of the AWS S3 API startable as Docker image, TestContainer, JUnit 4 rule, JUnit Jupiter extension or TestNG listener