venom VS hitchstory

You can also load fixtures in database directly, work with Kafka queues both as a producer (e.g. write an event to a Kafka queue, wait a few seconds and see that it was consumed by the service you test, and that some side effects can be observed) or as a consumer (e.g. make sure after an HTTP call, an event was correctly pushed to a queue), or even read a mailbox in IMAP to check that your service correctly send an email.

It's a bit rough on the edges sometimes, but I'd never go back on writing integration tests directly in my programming language. Declarative is the way to go.

[1]: https://github.com/ovh/venom

To write and run our integration tests, we'll use Venom. Venom is a tool created and made open-source by OVHcloud: https://github.com/ovh/venom

From my experience, generated tests are worthless for anything more serious than smoke tests. I prefer working with no tests than automated tests, I feel they give you a false sense of confidence.

The Step CI engine itself looks good though. It looks like a cleaner, but less powerful version of a tool (open source, build in-house) we used when I worked at OVHcloud, Venom: https://github.com/ovh/venom

Here's an example test file for the HTTP executor of Venom: https://github.com/ovh/venom/blob/master/tests/http.yml it's very close to Step CI format.

I'd still use Venom because it's way more powerful (you have DB executors for example, so after executing a POST request you can actually check in DB that you have what you expect) and I prefer focusing on actually writing integration tests instead of generating them.

Maybe this post sounds harsh (I feel it as I write it because I have strong feelings against test generation) but I think your approach is a good one for actually writing automated tests. Testing APIs declaratively like this has a great benefit: your tests work on an interface. You can migrate your API to a whole new stack and your tests remain the same. I did it multiple time at OVHcloud: one time migrating a huge API from a Go router to another (Gin->Echo), and another time migrating public APIs from a legacy, in-house Perl engine to a Go server.

I still struggle with GDB but my excuse is that I seldom use it.

When I was studying reverse engineering though, I came across a really cool kit (which I've yet to find an alternative for lldb, which would be nice given: rust)

I'd recommend checking it out, if for no other reason than it makes a lot of things really obvious (like watching what value lives in which register).

https://github.com/hugsy/gef

LLDB's closest alternative to this is called Venom, but it's not the same at all. https://github.com/ovh/venom

We incorporated [Venom](https://github.com/ovh/venom) into our workflow. It's great for initiating and managing a suite of yaml based tests. It didn't work out of the box for us due to the heavily asynchronous nature of our system, but after a few additions, it has helped my team greatly. We were often afraid to make large changes to critical pieces of the system since a full regression test could take a week or so to check everything. Now it takes an hour.

I built something like this too:

https://github.com/hitchdev/hitchstory/blob/master/examples%...

I took the same approach to "docs are tests and tests are docs" with integration testing when I created this library: https://github.com/hitchdev/hitchstory

I realized at some point that a test and a how-to guide can and should actually be the same thing - not just for doctests, but for every kind of test.

It's not only 2x quicker to combine writing a test with writing docs, the test part and the docs part reinforce each other:

* Tests are more easily understandable when you attach written context intended for human consumption.

* Docs are better if they come attached to a guarantee that they're valid, not out of date and not missing crucial details.

* TDD is better if how-to docs are created as a side effect.

I have created a project for easily writing this type of test with YAML:

https://github.com/hitchdev/hitchstory

I dont think that this type of task is really appropriate for an LLM though. It is better to use hard abstractions for the truly deterministic stuff and for other stuff where you may need to do subtle trade offs (e.g. choosing a selector for the search bar) an LLM will generally do a bad job.

For those interested in the concept of having permanently up-to-date documentation with screenshots I built this testing framework based upon the idea that good documentation can be a autogenerated artefact of good tests:

https://github.com/hitchdev/hitchstory

I don't like gherkin. It's it has very awkward syntax, it's not type safe, it's very verbose, it has no ability to abstract scenarios and rather than being a source for generating the documentation it tries to be the documentation.

Nonetheless, there is a small number of projects where they either work around this or it doesn't matter as much. I find that most people that apply gherkin to their projects find it doesn't work - usually for one of the above reasons.

I built https://github.com/hitchdev/hitchstory as an alternative that has straightforward syntax (YAML), very strict type safety (StrictYAML), low verbosity, and is explicitly designed as a source for generating documentation rather than trying to be the documentation.

I built this because I had the same idea: https://github.com/hitchdev/hitchstory

If the specification can be tested and used to generate docs and can be rewritten based upon program output then the maintenance cost for producing docs like these plunges.

-c fsync=off -c synchronous_commit=off -c full_page_writes=off

I got the answer from Karen Jex at Djangocon 2023.

I used it to build some integration tests which exhibit best practices: https://github.com/hitchdev/hitchstory/tree/master/examples/...

I considered using tmpfs but I wanted to cache the entire database volume and couldnt figure out how to do that with podman.

This is incredible work.

To anyone curious, I highly recommend:

- https://hitchdev.com/hitchstory/approach/

- https://hitchdev.com/hitchstory/why-not/

From the overall RDD/BDD type home page:

- https://hitchdev.com/hitchstory/

The entire product site is a thing of richly informative beauty.

---

My only question was whether the generated 'docs' snippets would add value over just reading the story in your DASL. Any markdown site generator (such as the chosen Material for MKDocs) can just embed the ```yaml anyway. But then I realized what was generating e.g. …

- https://hitchdev.com/hitchstory/using/engine/rewrite-story/

… and how superior that is to typical docs, especially typical docstring or swagger factories.