-
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
serverless-patterns
Serverless patterns. Learn more at the website: https://serverlessland.com/patterns.
-
serverless-application-model
The AWS Serverless Application Model (AWS SAM) transform is a AWS CloudFormation macro that transforms SAM templates into CloudFormation templates.
-
docker-lambda
Discontinued Docker images and test runners that replicate the live AWS Lambda environment
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
-
gateway
Drop-in replacement for Go net/http when running in AWS Lambda & API Gateway (by earthboundkid)
-
-
-
-
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
I use this companies services (seed.run as well). Frank and Jay are doing something truly special in this space.
Also, this guide they put together is definitive: https://serverless-stack.com/#guide
Had I found that years ago, I would have saved SO MANY headaches. I've implemented local debugging (hell, and doesn't actually replicate AWS). I've followed the CloudWatch logs, just like you, painful.
The SST local debugging with lambda is the best way forward. Deploying micro services on seed.run is also the way forward.
Suggestions:
1. If you are building APIs and using Lambda functions as targets from an API Gateway API, look into libraries like serverless-wsgi (Python) or wai-handler-hal (Haskell) that translate between API Gateway request/response payloads and some kind of ecosystem-native representation. Then as long as you're writing code where all state gets persisted outside of the request/response cycle, you can develop locally as if you were writing for a more normal deploy environment.
2. Look into the lambda runtime interface emulator ( https://github.com/aws/aws-lambda-runtime-interface-emulator... ). This lets you send invoke requests to a fake listener and locally test the lambda more easily. While the emulator is provided in the AWS container base images, you don't need to run it inside a container if you're deploying with zip files. (AWS-provided container images automatically enable the emulator if not running in a lambda runtime environment, and using docker for port remapping is nice. But not at all required.)
3. Get really good at capturing all requests to external services, and mocking them out for local testing. Whether this is with free monads, effect systems, gateway classes will depend on your language and library choices.
3) start-api - start the Lambda service emulator with a basic API Gateway emulator wrapped around it. This creates a local endpoint for each Lambda function that uses an API GW event source [https://docs.aws.amazon.com/serverless-application-model/lat...]
For testing action:
Using one of the above commands will invoke a Lambda function. The Lambda function will run locally and provide logs as well as stepping through the code in an IDE like AWS Cloud9 or VS Code. The Lambda function can also call out to service like DynamoDB, SQS, SNS, etc that reside in the cloud. Once the Lambda function is working as expected locally, it's time to deploy to a development environment and run E2E tests.
One other tool I would suggest it SAM Logs. SAM Logs can output logs for a specific Lambda function from CloudWatch to your terminal. This is a great way to debug async Lambda functions in the cloud.
I encourage you to visit https://serverlessland.com where we are constantly adding content to help developers with Serverless on AWS. I also have a series of SAM videos at https://s12d.com/sws. Additionally, we host Serverless Office Hours every Tuesday: twitch.tv/aws or youtube.com/serverlessland. During this time we answer any and all serverless related questions.
Having said all this. Our team is continuing to work towards making the development experience better. Threads like this are critical to our understanding of developer needs and we read them and take them to heart. If you would like to have a longer conversation please reach out to me at [email protected] or @edjgeek on Twitter.
This is a great thread and I understand some of the pain points that have been talked about. Serverless changes the model for how we develop and test code. There is a want to have everything local. But it is tough to build and maintain local emulators for all services. With that in mind, I encourage you to change the mindset of bringing the cloud to the developer to bringing the developer to the cloud.
When building serverless applications, the most tested and iterated upon part of the application is our code which usually resides in a Lambda function. Testing Lambda functions breaks down to to angles. 1) invocation: testing services invoking a Lambda function, and 2) action: what is the Lambda function doing. This is the only part of the application that should be tested locally through local emulation. The rest of the application is best tested in the cloud.
IMHO the best way to test a Lambda function locally is with AWS SAM: https://aws.amazon.com/serverless/sam/
For testing invocation:
A Lambda function can only be invoked through the AWS Lambda service. SAM has three ways to emulate the Lambda service: 1) invoke - locally invoke the Lambda function one time [https://docs.aws.amazon.com/serverless-application-model/lat...] This functionality is helpful if you want to mock invoking a Lambda function from a service like S3, SNS, SQS, etc. Simply add an event. To create the proper event structure, SAM provides a command called generate event. [https://docs.aws.amazon.com/serverless-application-model/lat...] 2) start-lambda - start an emulator of the Lambda service that can be reached via SDK or AWS CLI [https://docs.aws.amazon.com/serverless-application-model/lat...] 3) start-api - start the Lambda service emulator with a basic API Gateway emulator wrapped around it. This creates a local endpoint for each Lambda function that uses an API GW event source [https://docs.aws.amazon.com/serverless-application-model/lat...]
For testing action:
Using one of the above commands will invoke a Lambda function. The Lambda function will run locally and provide logs as well as stepping through the code in an IDE like AWS Cloud9 or VS Code. The Lambda function can also call out to service like DynamoDB, SQS, SNS, etc that reside in the cloud. Once the Lambda function is working as expected locally, it's time to deploy to a development environment and run E2E tests.
One other tool I would suggest it SAM Logs. SAM Logs can output logs for a specific Lambda function from CloudWatch to your terminal. This is a great way to debug async Lambda functions in the cloud.
I encourage you to visit https://serverlessland.com where we are constantly adding content to help developers with Serverless on AWS. I also have a series of SAM videos at https://s12d.com/sws. Additionally, we host Serverless Office Hours every Tuesday: twitch.tv/aws or youtube.com/serverlessland. During this time we answer any and all serverless related questions.
Having said all this. Our team is continuing to work towards making the development experience better. Threads like this are critical to our understanding of developer needs and we read them and take them to heart. If you would like to have a longer conversation please reach out to me at [email protected] or @edjgeek on Twitter.
If all you need is the ability to run a lambda function's code locally you might interested in docker-lambda[1]. I haven't really used localstack or SAM but a couple of years ago when we needed to run some lambda functions locally for development docker-lambda worked well enough.
[1] https://github.com/lambci/docker-lambda
Living in my team's bubble i thought everyone runs or tries to run parallel environments: prod, staging, dev, but also an individual (person) or feature env. Why? Because there's no emulator or documentation that will teach you real behavior. Like others have said, AWS seems out of this world. Just like GCP and Azure i might add. Some things you don't expect and they mesmerize you how smart they are. Some you expect and you can't fathom how come you're the "only" one screaming. Random thought: this is how i ended up logging all I bumped into into "Fl-aws" https://github.com/andreineculau/fl-aws
Back to the point: reality is that many build their AWS environment (prod) manually, maybe they duplicate once (dev) also manually, maybe they use some automation for their "code" (lambda) but that's it. This implies it's practically impossible to run end-to-end tests. You can't do that in prod for obvious reasons and you can't do it in dev either - you have many devs queueing, maybe dev is not in sync with prod etc.
My team ran cloudformation end-to-end. We actually orchestrated and wrapped cloudformation (this is yet another topic for not using terraform etc) so that if smth couldn't be done in CFN, it would still be automated and reproducible. Long story short, in 30 minutes (it was this long because we had to wait for cloudfront etc) we had a new environment, ready to play with. A total sandbox. Every dev had their own and it was easy to deploy from a release artifact or a git branch to this environment. Similarly you could create a separate env for more elaborate changes to the architecture. And test in a live environment.
Finally to your question: how do you test end-to-end?
If we talk about lambdas because that's where the business logic lies in a "serverless" architecture, then the answer is by calling the system which will eventually call your lambda/s along the way. If your lambda ia sitting behind AWS gateway, then fire an http request. Is it triggered when objects land on S3? Then push some object to S3. How do you assert? Just the same - http response, S3 changes etc. Not to mention you can also check cloudwatch for specific log entries (though they are not instant).
With this type of a setup, which sounds complex, but it is not since it is 100% reproducible (also from project to project - I had several), adding this proxy-to-my-dev-machine lambda would mean I can make local changes and then fire unit AND end-to-end tests without any changes pushed to AWS, which is the main time/energy consumer imo.
PS: sorry for the wall of text. Like i said i recently realized that the development realities have huge discrepancies, so i tried to summarize my reality :)
For developing API Gateway web servers on Lambda in Go, I use an adaptor, so it becomes a standard HTTP request, and I can just write a web service like any other. https://github.com/carlmjohnson/gateway For event based stuff… I have always avoided it for exactly this reason. Seems completely untestable and a pain to get working right.
Not a full solution, but when I was doing this I really got to love the awslogs utility:
https://github.com/jorgebastida/awslogs
It allows you to stream Cloudwatch logs from the command line, so you can grep them, save them to files, etc... (The web based Cloudwatch interface is terrible.)
Another suggestion is to try to modularize the core business logic in your lambda such that you separate the lambda-centric stuff from the rest of it. Obviously, though, if "the rest of it" is hitting other AWS services, you're going to hit the same testing roadblock.
Or you can try mocking, which may or may not provide much value for you. There's a python library for that, (moto), but it's not 100% up to date wrt AWS services/interfaces, last I had checked. Might be worth a try though.
https://github.com/spulec/moto
Not a full solution, but when I was doing this I really got to love the awslogs utility:
https://github.com/jorgebastida/awslogs
It allows you to stream Cloudwatch logs from the command line, so you can grep them, save them to files, etc... (The web based Cloudwatch interface is terrible.)
Another suggestion is to try to modularize the core business logic in your lambda such that you separate the lambda-centric stuff from the rest of it. Obviously, though, if "the rest of it" is hitting other AWS services, you're going to hit the same testing roadblock.
Or you can try mocking, which may or may not provide much value for you. There's a python library for that, (moto), but it's not 100% up to date wrt AWS services/interfaces, last I had checked. Might be worth a try though.
https://github.com/spulec/moto
I write Lambda functions in golang. In dev I run the function as local http server. In prod I map a custom domain to the lambda, routing is done internally to the function. API Gateway is only used for the top level route. This workflow is enabled by https://github.com/apex/gateway
In principle I try to avoid Amazon Web Services that lock me into the platform. So, for my dev stack I run a few other containers to give me the equivalent of the prod environment, RDS (MySQL or PostgreSQL), ES (OpenDistro), S3 (Minio), and SES
https://github.com/mozey/aws-local
Dev is easy because I can test everything locally, and the resulting code should be portable to any other cloud.
Obviously this approach might not be feasible if you choose to use AWS that do not have equivalent self-hosted options.
I can also recommend serverless.com framework for local development and running lambdas offline.
AWS provides a docker image for emulating DynamoDB which works great for local dev and will commonly be paired with lambdas.
Another option I have used recently for implementing node web services is https://github.com/apex/up which also has a nice local dev experience.
I'm reminded of a suggestion from someone else a few days go if you don't mind PHP
>https://bref.sh/ -- it allows running PHP apps on AWS Lambda effortlessly. With all the serverless craze, it's a very important piece of the ecosystem. After writing PHP apps for two decades, I am absolutely stunned how easy it is.
Related posts
-
Superpower REST API DX with Serverless ⚡ and DevOps Best Practices on AWS (🐍 Python Version)
-
Turbocharge your Lambda Functions with AWS Lambda Powertools for Python
-
Billing for SaaS with EMF and CloudWatch Metric Streams
-
Serverless Site Health Check Notification System
-
Advice for unattended vending machine startup with basic api, crud, and database needs