orjson VS msgspec

The library already has support for an HTTP client that allows bypassing Cloudflare - CurlImpersonateHttpClient. Since we have to work with JSON responses we could use parsel_crawler added in version 0.3.0, but I think this is excessive for such tasks, besides I like the high speed of orjson.. Therefore, we'll need to implement our crawler rather than using one of the ready-made ones.

(https://github.com/ijl/orjson). In this implementation, the same JSON

orjson – Fast, correct Python JSON library

orjson is the fastest JSON library available for python. It natively manages dataclass objects, datetime, numpy and UUID objects.

As we began venturing down that road, a few things emerged that would constitute significant changes to some of the core parts of Litestar, but there were two things in particular that started a chain reaction of changes by opening up further possibilities: The new DTOs and our switch from orjson to msgspec.

gjson [1] and a few other go packages offer a way to parse arbitrary JSON without requiring structs to hold them.

re: Python. I like PyRight/PyLance for Python typing, it seems to "just work" afaict. I also like msgspec for dataclass like behavior [2].

---

1: https://github.com/tidwall/gjson

2: https://jcristharif.com/msgspec/

Full support for validation and serialisation of attrs classes and msgspec Structs. Where previously only Pydantic models and types where supported, you can now mix and match any of these three libraries. In addition to this, adding support for another modelling library has been greatly simplified with the new plugin architecture

> Maybe it was very slow before

That is at least partly the case. I maintain msgspec[1], another Python JSON validation library. Pydantic V1 was ~100x slower at encoding/decoding/validating JSON than msgspec, which was more a testament to Pydantic's performance issues than msgspec's speed. Pydantic V2 is definitely faster than V1, but it's still ~10x slower than msgspec, and up to 2x slower than other pure-python implementations like mashumaro.

Recent benchmark here: https://gist.github.com/jcrist/d62f450594164d284fbea957fd48b...

[1]: https://github.com/jcrist/msgspec

While it's definitely much faster than pydantic V1 (which is a huge accomplishment!), it's still not exactly what I'd call "fast".

I maintain msgspec (https://github.com/jcrist/msgspec), a serialization/validation library which provides similar functionality to pydantic. Recent benchmarks of pydantic V2 against msgspec show msgspec is still 15-30x faster at JSON encoding, and 6-15x faster at JSON decoding/validating.

Benchmark (and conversation with Samuel) here: https://gist.github.com/jcrist/d62f450594164d284fbea957fd48b...

This is not to diminish the work of the pydantic team! For many users pydantic will be more than fast enough, and is definitely a more feature-filled tool. It's a good library, and people will be happy using it! But pydantic is not the only tool in this space, and rubbing some rust on it doesn't necessarily make it "fast".

https://github.com/jcrist/msgspec so I am using this instead of Pydantic.

Another thing: why pyserde rather than stuff like msgspec? https://github.com/jcrist/msgspec

Try msgspec | Maat | turbo for fast serialization and validation

Congratulations to the team, Pydantic is an amazing library.

If you find JSON serialization/deserialization a bottleneck, another interesting library (with much less features) for Python is msgspec: https://github.com/jcrist/msgspec