jsonschema VS Cerberus

Forms are hard. We use python jsonschema to write our form schemas and validation and use react json schema form for the front end. It's a long time in the making and we still have to write widgets and extensions to get everything we need. Good luck.

I had a requirement to process and validate large payloads of JSON concurrently for a web service, initially I implemented it using jsonschema and fastjsonschema but I found the whole JSON Schema Specification to be confusing at times and on top of that wanted better performance. Albeit there are ways to compile/cache the schema, I wanted to move away from the schema specification so I wrote a validation library inspired by the design of tiangolo/sqlmodel (type hints) to solve this problem easier.

This is a hard problem, because, if I understand you correctly, you issue is that you are using something like this: https://github.com/Julian/jsonschema, but want to make the error messages more specific. That means you will need to understand the package sufficiently to find out where it is encountering issues and then provide a more human readable error. Definitely doable, but the first piece, understanding the package enough to revise the messages is difficult.

I was expecting this to validate the configuration files are also valid for their use cases, not just valid JSON, TOML, etc.

If you're looking for that and Python is your jam, the library cerberus[0] is very good at it.

[0]: https://github.com/pyeve/cerberus

I wrote scrapy-test as a proof of concept for validating live pages for scrapy spiders if you're looking for some reference but if you're not using scrapy I'd recommend just adding validation tests using data validation tools like cerberus which is super underrated. I cover popular data validation techniques on this short blog I wrote if you want to learn more.

Other than that, have you looked into testing scrapers? Since scrapers are working with highly dynamic data writing good tests is quite a challange. For example, for parser monitoring using cerberus is a very cool tool which allows you to define loose requirements like "phone number should always be 9 numbers" etc.