DiskCache VS pydantic

python-diskcache – disk-backed cache (Django-compatible). Faster than Redis and Memcached. Pure-Python

Based on your question about SQLite, it seems like you want to store the database inside of the program as opposed to on the internet. Furthermore, your data doesn't seem to be super relational to my knowledge. You might be better off using something like diskcache to store the data instead.

Give Grant some love and give grantjenks/python-diskcache a ⭐.

Do you know this project which covers most your needs ? http://www.grantjenks.com/docs/diskcache/

The solution? Leverage Pydantic, a Python library that enables runtime data validation using type annotations.

Pydantic author here. We have plans for an improvement to pydantic where JSON is parsed iteratively, which will make way for reading a file as we parse it. Details in https://github.com/pydantic/pydantic/issues/10032.

Our JSON parser, jiter (https://github.com/pydantic/jiter) already supports iterative parsing, so it's "just" a matter of solving the lifetimes in pydantic-core to validate as we parse.

This should make pydantic around 3x faster at parsing JSON and significantly reduce the memory overhead.

Across this five-post series, we’ve journeyed from Pydantic’s basics—type validation and nested models—to advanced integrations with FastAPI, SQLAlchemy, and scalable techniques. You’ve learned how to build declarative, type-safe models, handle complex APIs, and optimize performance. To deepen your knowledge, explore the Pydantic documentation, contribute to the open-source project, or experiment with real-world use cases. Check out our GitHub repo for code samples and a Pydantic cheat sheet. Thank you for joining us—happy coding!

To dive deeper, check out the Pydantic documentation for advanced features like nested models, custom validators, and settings management. Happy coding!

Pydantic is bundled with FastAPI and is excellent for modelling, validating, and serialising API responses.

Pydantic Settings is a python library that extends 🚀 Pydantic for dealing with settings management.

And here is where FastAPI ✨ shines: It leverages Pydantic powerful data validation to offer out of the box JSON Schema and OpenAPI specs via Swagger and Redoc.

If you carefully think about it, the system that extracts raw text from the PDF needs to both detect and render PDF form elements like checkboxes and radiobuttons in a way that LLMs can understand. In this example, we’ll use LLMWhisperer to extract PDF raw text representing checkboxes and radiobuttons. You can use LLMWhisperer completely free for processing up to 100 pages per day. As for structuring the output from LLMWhisperer, we’ll use GPT3.5-Turbo and we’ll use Langchain and Pydantic to help make our job easy.