orjson VS msgspec

orjson is the fastest JSON library available for python. It natively manages dataclass objects, datetime, numpy and UUID objects.

As we began venturing down that road, a few things emerged that would constitute significant changes to some of the core parts of Litestar, but there were two things in particular that started a chain reaction of changes by opening up further possibilities: The new DTOs and our switch from orjson to msgspec.

In version 1.45.0, we introduced msgspec as our serialization backend, replacing orjson. This had some immediate performance benefits, but that's not the main reason we made the switch.

Doesn't orjson do that already?

orjson

For most cases, you would want to go with python’s standard json library which removes dependencies on other libraries. On other hand you could try out ujsonwhich is simple replacement for python’s json library. If you want more speed and also want dataclass, datetime, numpy, and UUID instances and you are ready to deal with more complex code, then you can try your hands on orjson

Try https://github.com/ijl/orjson. Not exactly sure if that can help.

Full support for validation and serialisation of attrs classes and msgspec Structs. Where previously only Pydantic models and types where supported, you can now mix and match any of these three libraries. In addition to this, adding support for another modelling library has been greatly simplified with the new plugin architecture

> Maybe it was very slow before

That is at least partly the case. I maintain msgspec[1], another Python JSON validation library. Pydantic V1 was ~100x slower at encoding/decoding/validating JSON than msgspec, which was more a testament to Pydantic's performance issues than msgspec's speed. Pydantic V2 is definitely faster than V1, but it's still ~10x slower than msgspec, and up to 2x slower than other pure-python implementations like mashumaro.

Recent benchmark here: https://gist.github.com/jcrist/d62f450594164d284fbea957fd48b...

[1]: https://github.com/jcrist/msgspec

While it's definitely much faster than pydantic V1 (which is a huge accomplishment!), it's still not exactly what I'd call "fast".

I maintain msgspec (https://github.com/jcrist/msgspec), a serialization/validation library which provides similar functionality to pydantic. Recent benchmarks of pydantic V2 against msgspec show msgspec is still 15-30x faster at JSON encoding, and 6-15x faster at JSON decoding/validating.

Benchmark (and conversation with Samuel) here: https://gist.github.com/jcrist/d62f450594164d284fbea957fd48b...

This is not to diminish the work of the pydantic team! For many users pydantic will be more than fast enough, and is definitely a more feature-filled tool. It's a good library, and people will be happy using it! But pydantic is not the only tool in this space, and rubbing some rust on it doesn't necessarily make it "fast".

https://github.com/jcrist/msgspec so I am using this instead of Pydantic.

Another thing: why pyserde rather than stuff like msgspec? https://github.com/jcrist/msgspec

Try msgspec | Maat | turbo for fast serialization and validation

Congratulations to the team, Pydantic is an amazing library.

If you find JSON serialization/deserialization a bottleneck, another interesting library (with much less features) for Python is msgspec: https://github.com/jcrist/msgspec

This feature is yet to be released, but it will allow you to seamlessly use data modelled with for example Pydantic, SQLAlchemy, msgspec or dataclasses in your route handlers, without the need for an intermediary model; The conversion will be handled by the specific DTO "backend" implementation. This new paradigm also makes it trivial to add support for any such modelling library, by simply implementing an appropriate backend.