venom VS embedded-postgres

You can also load fixtures in database directly, work with Kafka queues both as a producer (e.g. write an event to a Kafka queue, wait a few seconds and see that it was consumed by the service you test, and that some side effects can be observed) or as a consumer (e.g. make sure after an HTTP call, an event was correctly pushed to a queue), or even read a mailbox in IMAP to check that your service correctly send an email.

It's a bit rough on the edges sometimes, but I'd never go back on writing integration tests directly in my programming language. Declarative is the way to go.

[1]: https://github.com/ovh/venom

To write and run our integration tests, we'll use Venom. Venom is a tool created and made open-source by OVHcloud: https://github.com/ovh/venom

From my experience, generated tests are worthless for anything more serious than smoke tests. I prefer working with no tests than automated tests, I feel they give you a false sense of confidence.

The Step CI engine itself looks good though. It looks like a cleaner, but less powerful version of a tool (open source, build in-house) we used when I worked at OVHcloud, Venom: https://github.com/ovh/venom

Here's an example test file for the HTTP executor of Venom: https://github.com/ovh/venom/blob/master/tests/http.yml it's very close to Step CI format.

I'd still use Venom because it's way more powerful (you have DB executors for example, so after executing a POST request you can actually check in DB that you have what you expect) and I prefer focusing on actually writing integration tests instead of generating them.

Maybe this post sounds harsh (I feel it as I write it because I have strong feelings against test generation) but I think your approach is a good one for actually writing automated tests. Testing APIs declaratively like this has a great benefit: your tests work on an interface. You can migrate your API to a whole new stack and your tests remain the same. I did it multiple time at OVHcloud: one time migrating a huge API from a Go router to another (Gin->Echo), and another time migrating public APIs from a legacy, in-house Perl engine to a Go server.

I still struggle with GDB but my excuse is that I seldom use it.

When I was studying reverse engineering though, I came across a really cool kit (which I've yet to find an alternative for lldb, which would be nice given: rust)

I'd recommend checking it out, if for no other reason than it makes a lot of things really obvious (like watching what value lives in which register).

https://github.com/hugsy/gef

LLDB's closest alternative to this is called Venom, but it's not the same at all. https://github.com/ovh/venom

We incorporated [Venom](https://github.com/ovh/venom) into our workflow. It's great for initiating and managing a suite of yaml based tests. It didn't work out of the box for us due to the heavily asynchronous nature of our system, but after a few additions, it has helped my team greatly. We were often afraid to make large changes to critical pieces of the system since a full regression test could take a week or so to check everything. Now it takes an hour.

Little use if you’re not on the JVM but I’ve had great success with Embedded Postgres:

https://github.com/zonkyio/embedded-postgres

Each test just copies a template database so it’s ultra fast and avoids the need for complicated reset logic.

Outside of differences between assertion-based unit tests and property-based tests (both of which are worth doing), I don't think framework makes much difference. But your approach to testing definitely does.

I think every language having its own testing framework is good, even for things like functional tests which can often be externalised. Tests are an essential part of every project and should be well integrated with the rest of the codebase and the team creating it. Often, the tests are the only good place to go and see what an app actually _does_ and so they form an essential part of the documentation.

In my experience it's very rare that you can effectively create and maintain something like Cucumber tests owned by anyone but the team implementing the code so there's little benefit to translating from a text DSL like that. But the language used is definitely useful, so what I like to see is code in the implementation language that matches the Given/When/Then structure of those tests, but instead of reusable text steps you just have reusable functions which take parameters. This means you can easily refactor, and use the full functionality of your IDE to suggest and go to definitions etc. No matter what, you should treat your test code the same way you do everything else - abstractions matter, so functional tests at the top level should rarely just be about clicking on things and asserting other things, they should be in the language of the domain.

Functional tests are worth much more than unit tests. No only do they test the only things of actual business value, they are also more robust in the face of implementation refactorings and so require less rework (unless you're being overly specific with CSS selectors etc). Unit tests are often highly coupled to specific implementations and can be a poor investment, especially early in a project. I believe a good balance is functional and integration tests that explore the various paths through your app and prove everything's hooked up, coupled with property based unit tests for gnarly or repetitive logic that isn't worth endlessly iterating via the UI. All other unit tests are optional and at the discretion of the implementer.

You should be able to mock out every major articulation point in your code, but it's generally preferable if you can mock _real_ dependencies. That is, instead of mocking out a 'repository' abstraction that looks stuff up and returns canned data, have a real test database against which you look up real data (created by steps in your functional tests). This reduces risk and cognitive overhead (you're not having to encode too many assumptions in your test suite) and doesn't have to be as slow as people like to make out - Embedded Postgres is quite fast, for example:

https://github.com/zonkyio/embedded-postgres

Same with network services - it's not slow to chat to localhost and you'll find more issues testing proper round-trips. I have not found "assert that you called X" style testing with mocks useful - you care about outcomes, not implementation details.

Beyond all that, as long as you can make assertions that generate clear error messages, you're fine.

I use an embedded postgres testing library for the JVM that does something along those lines.

Well no actually it just unpacks the tar file in a temp dir and runs the full postgres, but it mostly feels like what you describe (minus the single file part) and starts surprisingly fast. That would totally work for a little proof of concept (https://github.com/zonkyio/embedded-postgres)