terraforming VS awslogs

Terraforming

I would also strongly recommend an iterative triage process: don't feel that you need to solve everything all once. It's a huge amount of progress if you can get to a stage where you understand the relationships between services, have enough monitoring to identify failures, and can iteratively move services to more granular IAM policies even if it's all still running in one big account. Simply using a tool like terraformer or terraforming to move the manual configuration into an IaC workflow is an accomplishment, especially since it helps you both make changes more confidently and identify where new changes are still being made. Depending on the politics, that last part might be important: you'll be unpopular if you take away everyone's rights but you'll probably have better luck first nudging people to make them using a sensible flow rather than by hand.

Haven't tried this but if I ever find myself in your situation I'll give it a spin https://github.com/dtan4/terraforming

There are a couple of tools to automate creating terraform configuration and prepopulate state from existing resources, like terraforming, terraformer or cf-terraforming. But it is still cumbersome and laborious and in my experience, it's usually way easier to just recreate everything within terraform from scratch wherever that's possible.

🧪 Terraforming export existing AWS resources to Terraform style (tf, tfstate).

Using the official CLI (aws logs get-log-events) or https://github.com/jorgebastida/awslogs is pretty close to SSH-ing and grepping.

cw is a native executable targeting your OS, and not needed external dependencies such as pip and npm. Compared to awslogs which is famous helpful tool for CloudWatch Logs1, cw is written in golang and faster.

Its a command line tool but some people I know also use awslogs

Not a full solution, but when I was doing this I really got to love the awslogs utility:

https://github.com/jorgebastida/awslogs

It allows you to stream Cloudwatch logs from the command line, so you can grep them, save them to files, etc... (The web based Cloudwatch interface is terrible.)

Another suggestion is to try to modularize the core business logic in your lambda such that you separate the lambda-centric stuff from the rest of it. Obviously, though, if "the rest of it" is hitting other AWS services, you're going to hit the same testing roadblock.

Or you can try mocking, which may or may not provide much value for you. There's a python library for that, (moto), but it's not 100% up to date wrt AWS services/interfaces, last I had checked. Might be worth a try though.

https://github.com/spulec/moto

Cloudwatch Logs is pretty meh visually, but awslogs can give you a pretty good `tail -f`-like experience, and Insights is pretty good. Cloudwatch Metric Filters give you a 'StatsD'-like experience, in that you can log out a certain message or code and then use its appearance as a metric.

Check out https://github.com/jorgebastida/awslogs , you can define a `--start`, and it also has a `--watch`, and can be piped the `grep` or whatever you want. It's a pretty flexible tool.

📖 awslogs - a simple command-line tool for querying groups, streams, and events from Amazon CloudWatch logs.