Our great sponsors
fl-aws | awslogs | |
---|---|---|
1 | 8 | |
15 | 4,750 | |
- | - | |
0.0 | 0.0 | |
over 7 years ago | 25 days ago | |
Python | ||
Apache License 2.0 | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
fl-aws
-
Ask HN: Does anyone else find the AWS Lambda developer experience poor?
Living in my team's bubble i thought everyone runs or tries to run parallel environments: prod, staging, dev, but also an individual (person) or feature env. Why? Because there's no emulator or documentation that will teach you real behavior. Like others have said, AWS seems out of this world. Just like GCP and Azure i might add. Some things you don't expect and they mesmerize you how smart they are. Some you expect and you can't fathom how come you're the "only" one screaming. Random thought: this is how i ended up logging all I bumped into into "Fl-aws" https://github.com/andreineculau/fl-aws
Back to the point: reality is that many build their AWS environment (prod) manually, maybe they duplicate once (dev) also manually, maybe they use some automation for their "code" (lambda) but that's it. This implies it's practically impossible to run end-to-end tests. You can't do that in prod for obvious reasons and you can't do it in dev either - you have many devs queueing, maybe dev is not in sync with prod etc.
My team ran cloudformation end-to-end. We actually orchestrated and wrapped cloudformation (this is yet another topic for not using terraform etc) so that if smth couldn't be done in CFN, it would still be automated and reproducible. Long story short, in 30 minutes (it was this long because we had to wait for cloudfront etc) we had a new environment, ready to play with. A total sandbox. Every dev had their own and it was easy to deploy from a release artifact or a git branch to this environment. Similarly you could create a separate env for more elaborate changes to the architecture. And test in a live environment.
Finally to your question: how do you test end-to-end?
If we talk about lambdas because that's where the business logic lies in a "serverless" architecture, then the answer is by calling the system which will eventually call your lambda/s along the way. If your lambda ia sitting behind AWS gateway, then fire an http request. Is it triggered when objects land on S3? Then push some object to S3. How do you assert? Just the same - http response, S3 changes etc. Not to mention you can also check cloudwatch for specific log entries (though they are not instant).
With this type of a setup, which sounds complex, but it is not since it is 100% reproducible (also from project to project - I had several), adding this proxy-to-my-dev-machine lambda would mean I can make local changes and then fire unit AND end-to-end tests without any changes pushed to AWS, which is the main time/energy consumer imo.
PS: sorry for the wall of text. Like i said i recently realized that the development realities have huge discrepancies, so i tried to summarize my reality :)
awslogs
-
Logging in Python Like a Pro
Using the official CLI (aws logs get-log-events) or https://github.com/jorgebastida/awslogs is pretty close to SSH-ing and grepping.
- Tail log groups with CW Logs Insights?
-
I use cw, which is OSS to tail AWS CloudWatch Logs
cw is a native executable targeting your OS, and not needed external dependencies such as pip and npm. Compared to awslogs which is famous helpful tool for CloudWatch Logs1, cw is written in golang and faster.
-
What are you using to analyze/visualize CloudFront logs?
Its a command line tool but some people I know also use awslogs
-
Ask HN: Does anyone else find the AWS Lambda developer experience poor?
Not a full solution, but when I was doing this I really got to love the awslogs utility:
https://github.com/jorgebastida/awslogs
It allows you to stream Cloudwatch logs from the command line, so you can grep them, save them to files, etc... (The web based Cloudwatch interface is terrible.)
Another suggestion is to try to modularize the core business logic in your lambda such that you separate the lambda-centric stuff from the rest of it. Obviously, though, if "the rest of it" is hitting other AWS services, you're going to hit the same testing roadblock.
Or you can try mocking, which may or may not provide much value for you. There's a python library for that, (moto), but it's not 100% up to date wrt AWS services/interfaces, last I had checked. Might be worth a try though.
https://github.com/spulec/moto
-
Datadog alternatives
Cloudwatch Logs is pretty meh visually, but awslogs can give you a pretty good `tail -f`-like experience, and Insights is pretty good. Cloudwatch Metric Filters give you a 'StatsD'-like experience, in that you can log out a certain message or code and then use its appearance as a metric.
-
Cloud watch logs from console always show tail. How to show head without having to click ‘show more’ over and over again?
Check out https://github.com/jorgebastida/awslogs , you can define a `--start`, and it also has a `--watch`, and can be piped the `grep` or whatever you want. It's a pretty flexible tool.
-
DevOps tools you should have on your belt
📖 awslogs - a simple command-line tool for querying groups, streams, and events from Amazon CloudWatch logs.
What are some alternatives?
aws-lambda-runtime-interface-emulator
Loguru - Python logging made (stupidly) simple
cw - The best way to tail AWS CloudWatch Logs from your terminal
GoAccess - GoAccess is a real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser.
serverless-offline - Emulate AWS λ and API Gateway locally when developing your Serverless project
aws-codebuild-docker-images - Official AWS CodeBuild repository for managed Docker images http://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref.html
faasd - A lightweight & portable faas engine
sls-dev-tools - Dev Tools for the Serverless World - Issues, PRs and ⭐️welcome!
saml2aws - CLI tool which enables you to login and retrieve AWS temporary credentials using a SAML IDP
Moto - A library that allows you to easily mock out tests based on AWS infrastructure.
terraforming - Export existing AWS resources to Terraform style (tf, tfstate) / No longer actively maintained