responses VS Moto

I use responses for this (for HTTP/HTTPS requests), to great affect. It's API is very nice. We have an API layer at work that's basically just a proxy to micro services, and responses is what we use to test it.

responses

Please format your code appropriately. But if you are using python requests you can use responses for mocking requests: https://github.com/getsentry/responses

You can use this https://github.com/getsentry/responses

The repsonses library is designed for mocking requests during tests https://github.com/getsentry/responses

I like using responses for mocking API calls in unit tests. It's easy to use and has less boilerplate than doing the mocks yourself. The bulk of your unit tests should be around any data transformation you do, and unit testing them should be fairly straightforward. I'd recommend writing integration tests as well, especially as IME the most common reason why data pipelines with API sources break is due to breaking API changes out of your control. I'd make them as high level as possible and make real API calls, preferably using credentials for a designated testing account. Assert the response from the API call conforms to the contract you expect. Additionally to integration test your code, you can test the main entrypoint to your script and make sure it spits out a file with the shape and contents you expect.

HTTP client library flexibility. Yes, requests pairs well with responses or requests-mock, and HTTPX has RESPX or pytest_httpx. Those testing helpers are an excellent match for the corresponding library, and should certainly be recommended. However, I don't always want to face the risk of rewriting all my tests if I replace the client library some day. And sometimes I am using an altogether different tool (even though both requests and HTTPX are quite awesome) such as urlopen, urllib3, httplib2, tornado, or aiohttp.

> OpenMoto

I dunno if you're trying to play on "hashimoto" but https://github.com/getmoto/moto#readme would be a prime name collision for any such "OpenMoto" name

But yes, please, to adopting Vault. I don't have a horse in the race about Consul but my suspicion is such an effort would only be worthwhile if trying to adopt Nomad, too, which I gravely doubt

Unfortunately I do not believe AWS provides any "test" gateways. I do know there are mock AWS servers you can run on your own. The one I use is called Moto. It does not cover everything (unfortunately it's the most comprehensive out there AFAIK), but it's decent enough to test most standard calls via the sdk. I'm not sure if it covers authorization though...we tend to use security roles on tasks for authorization.

If using Python use Moto to mock AWS Services

You can use a library such as moto https://github.com/getmoto/moto

The Pythonic motto library mocks AWS services, removing the need to deploy your application or pay for API calls against AWS services. Other programming languages have their motto implementation.

Has anyone managed to configure moto (https://docs.getmoto.org/en/latest/) in a docker container in the similar way LocalStack does?

As for mocking, you can take a look at the moto library for mocking the AWS SDK, or for more simple cases even just use a `unittest.Mock/MagicMock` object. If you're having trouble trying to use the mocks in your code, it's a good sign your code is too highly coupled and it'd pay to re-factor, for example using dependency injection, design patterns like adapter/facade etc. (but don't over-do it)

To add on to the integration tests point, for mocking out your AWS resources you should check out moto if you don't want run your test against real AWS resources as they may cost you and is usually slower.