msgspec VS pydantic

gjson [1] and a few other go packages offer a way to parse arbitrary JSON without requiring structs to hold them.

re: Python. I like PyRight/PyLance for Python typing, it seems to "just work" afaict. I also like msgspec for dataclass like behavior [2].

---

1: https://github.com/tidwall/gjson

2: https://jcristharif.com/msgspec/

Full support for validation and serialisation of attrs classes and msgspec Structs. Where previously only Pydantic models and types where supported, you can now mix and match any of these three libraries. In addition to this, adding support for another modelling library has been greatly simplified with the new plugin architecture

> Maybe it was very slow before

That is at least partly the case. I maintain msgspec[1], another Python JSON validation library. Pydantic V1 was ~100x slower at encoding/decoding/validating JSON than msgspec, which was more a testament to Pydantic's performance issues than msgspec's speed. Pydantic V2 is definitely faster than V1, but it's still ~10x slower than msgspec, and up to 2x slower than other pure-python implementations like mashumaro.

Recent benchmark here: https://gist.github.com/jcrist/d62f450594164d284fbea957fd48b...

[1]: https://github.com/jcrist/msgspec

While it's definitely much faster than pydantic V1 (which is a huge accomplishment!), it's still not exactly what I'd call "fast".

I maintain msgspec (https://github.com/jcrist/msgspec), a serialization/validation library which provides similar functionality to pydantic. Recent benchmarks of pydantic V2 against msgspec show msgspec is still 15-30x faster at JSON encoding, and 6-15x faster at JSON decoding/validating.

Benchmark (and conversation with Samuel) here: https://gist.github.com/jcrist/d62f450594164d284fbea957fd48b...

This is not to diminish the work of the pydantic team! For many users pydantic will be more than fast enough, and is definitely a more feature-filled tool. It's a good library, and people will be happy using it! But pydantic is not the only tool in this space, and rubbing some rust on it doesn't necessarily make it "fast".

https://github.com/jcrist/msgspec so I am using this instead of Pydantic.

Another thing: why pyserde rather than stuff like msgspec? https://github.com/jcrist/msgspec

Try msgspec | Maat | turbo for fast serialization and validation

Congratulations to the team, Pydantic is an amazing library.

If you find JSON serialization/deserialization a bottleneck, another interesting library (with much less features) for Python is msgspec: https://github.com/jcrist/msgspec

The solution? Leverage Pydantic, a Python library that enables runtime data validation using type annotations.

Pydantic author here. We have plans for an improvement to pydantic where JSON is parsed iteratively, which will make way for reading a file as we parse it. Details in https://github.com/pydantic/pydantic/issues/10032.

Our JSON parser, jiter (https://github.com/pydantic/jiter) already supports iterative parsing, so it's "just" a matter of solving the lifetimes in pydantic-core to validate as we parse.

This should make pydantic around 3x faster at parsing JSON and significantly reduce the memory overhead.

Across this five-post series, we’ve journeyed from Pydantic’s basics—type validation and nested models—to advanced integrations with FastAPI, SQLAlchemy, and scalable techniques. You’ve learned how to build declarative, type-safe models, handle complex APIs, and optimize performance. To deepen your knowledge, explore the Pydantic documentation, contribute to the open-source project, or experiment with real-world use cases. Check out our GitHub repo for code samples and a Pydantic cheat sheet. Thank you for joining us—happy coding!

To dive deeper, check out the Pydantic documentation for advanced features like nested models, custom validators, and settings management. Happy coding!

Pydantic is bundled with FastAPI and is excellent for modelling, validating, and serialising API responses.

Pydantic Settings is a python library that extends 🚀 Pydantic for dealing with settings management.

And here is where FastAPI ✨ shines: It leverages Pydantic powerful data validation to offer out of the box JSON Schema and OpenAPI specs via Swagger and Redoc.

If you carefully think about it, the system that extracts raw text from the PDF needs to both detect and render PDF form elements like checkboxes and radiobuttons in a way that LLMs can understand. In this example, we’ll use LLMWhisperer to extract PDF raw text representing checkboxes and radiobuttons. You can use LLMWhisperer completely free for processing up to 100 pages per day. As for structuring the output from LLMWhisperer, we’ll use GPT3.5-Turbo and we’ll use Langchain and Pydantic to help make our job easy.