Loading Pydantic models from JSON without running out of memory

1. ZANJ

   1 1 9 8.7 Python

   saving complicated structures to disk without pickling

   

   https://github.com/mivanit/ZANJ

   You can have nested dataclasses, as well as specify custom serializers/loaders for things which aren't natively supported by json.

2. Stream

   getstream.io featured

   Stream - Scalable APIs for Chat, Feeds, Moderation, & Video. Stream helps developers build engaging apps that scale to millions with performant and flexible Chat, Feeds, Moderation, and Video APIs and SDKs powered by a global edge network and enterprise-grade infrastructure.

   

3. pydantic

   2 180 24,433 9.7 Python

   Data validation using Python type hints

   

   Pydantic author here. We have plans for an improvement to pydantic where JSON is parsed iteratively, which will make way for reading a file as we parse it. Details in https://github.com/pydantic/pydantic/issues/10032.

   Our JSON parser, jiter (https://github.com/pydantic/jiter) already supports iterative parsing, so it's "just" a matter of solving the lifetimes in pydantic-core to validate as we parse.

   This should make pydantic around 3x faster at parsing JSON and significantly reduce the memory overhead.

4. jiter

   3 5 429 8.0 Rust

   Fast iterable JSON parser.

   

   Pydantic author here. We have plans for an improvement to pydantic where JSON is parsed iteratively, which will make way for reading a file as we parse it. Details in https://github.com/pydantic/pydantic/issues/10032.

   Our JSON parser, jiter (https://github.com/pydantic/jiter) already supports iterative parsing, so it's "just" a matter of solving the lifetimes in pydantic-core to validate as we parse.

   This should make pydantic around 3x faster at parsing JSON and significantly reduce the memory overhead.

5. remark

   4 8 12,863 0.0 JavaScript

   A simple, in-browser, markdown-driven slideshow tool.

   

Suggest a related project