bachelor-thesis VS cats

Yes, we looked into it. There is a chapter in my thesis[0] about RESTler and comparison with OpenAPI fuzzer. The main difference between those two fuzzer is that RESTler is a statefull fuzzer and OpenAPI fuzzer is a stateless fuzzer.

Thanks to being statefull, RESTler is able to analyze a dependencies between a requests. For example, it will not call and endpoint to get user details before calling endpoint to create a user. This should make it more efficient because it does not waste requests to calling endpoint that will return 404.

On the other hand, fuzzing is all about trying to supply unexpected input to the software. Therefore, OpenAPI fuzzer makes those requests, since it may cause some undefined behavior when we try to get user before creating one.

So while RESTler tries to check some invariant (for example if deleted user cannot be accessed) OpenAPI fuzzer tries to cause the service to crash by invalid input. RESTler is usually not able to crash the service by providing invalid input, because it needs to keep the fuzzing dictionary small (bigger dictionary would make the dependency finding slow). So I think, they complements each other nicely.

Another difference it in reporting error. RESTler considers only status code 500 as an error. However, OpenAPI specification states all possible status codes that we can get from an endpoint. OpenAPI fuzzer utilizes this and reports every time it receives and unexpected status code.

0: https://github.com/matusf/bachelor-thesis/releases/download/...

Out of this frustration I've built: https://github.com/Endava/cats. It's for APIs, but mostly addressing exactly this case: don't use strings for everything, if you choose to use it though, make sure you add patterns for checking if things are valid, make sure you think about all the corner cases and all the weird characters that can brake you app, and so on.

If the API has an OpenAPI spec available, you can use: https://github.com/Endava/cats

This is why we need better tools which will give benefits for the added complexity. If you need to create both the feature files AND the code, it's just complexity with little benefits. But frameworks like https://github.com/karatelabs/karate or https://github.com/Endava/cats are hiding this complexity and remove the code layer entirely. Which, in my view, this is where you need to be in 2023, particularly for API testing.

I've built a tool specifically to test if these kind of characters will reach API backends: https://github.com/Endava/cats. My idea was that APIs should explicitly reject or sanitise input containing such characters.

Cats by Endava