Top 23 Python Serialization Projects
-
marshmallow
A lightweight library for converting complex objects to and from simple Python datatypes.
Project mention: The Pocket Guide To API Request Validation You Wish You Had Earlier | dev.to | 2022-01-06Marshmallow
-
Project mention: A simple Python package written using Rust and Pyo3 (XBRL XML to JSON/CSV parser) | reddit.com/r/Python | 2022-01-11
I recently discovered the amazing Pyo3 library which makes it relatively easy to write a Python package using a Rust library. This enables the development of Python packages that utilize the blazing speed of Rust with the usability of Python. Packages such as ORJSON use this approach to achieve state-of-the-art speeds through a Python package.
-
Scout APM
Less time debugging, more time building. Scout APM allows you to find and fix performance issues with no hassle. Now with error monitoring and external services monitoring, Scout is a developer's best friend when it comes to application development.
-
-
This major release of Flask is accompanied by major releases of Werkzeug, Jinja2, click, and itsdangerous which we'd also welcome and appreciate testing (their pre releases are installed with the Flask pre release).
-
Don't say "it's just about the quality of the implementing team".
-
-
-
SonarQube
Static code analysis for 29 languages.. Your projects are multi-language. So is SonarQube analysis. Find Bugs, Vulnerabilities, Security Hotspots, and Code Smells so you can release quality code every time. Get started analyzing your projects today for free.
-
I don't think JSON is really the problem - parsing 10MB of JSON is not so slow. For example, using Python's json.load takes about 800ms for a 47MB file on my system, using something like simdjson cuts that down to ~70ms.
-
Web pages are an interesting example of both structured and unstructured data. There are specific elements one could look at for certain information like the element or other semantic elements like
or. The problem though is that these elements are more like our "address" example earlier - they often contain more than just the strict data we are looking for. A title might have a prefix or suffix of the website's name. An article or section might have many other layers of,or any other elements to help form the site's structure. To top it off, the HTML structure can vary wildly from site to site. If you were wanting to extract data from multiple websites, it can get very hard very fast.That said, there are a number of ways to embed structured data into web pages. A web page could use Microdata, RDFa, JSON-LD or Open Graph to express structured data. More than that though, a web page can use multiple of these at the same time. Open Graph is commonly used as a method of defining details for a link preview while the others might express more complex data like product pricing or reviews.
Having standard formats like Microdata or JSON-LD are a good start but only represent the format of the data - we need a common vocabulary so we can understand the data those formats encode. One common vocabulary used is called Schema.org and provides over 700 types including types to describe people, places, products, recipes, reviews, vehicles, movies and medical devices. Using Schema.org for structured data on a website can help search engines provide richer experiences in the search results.
Summary
Structured data, through standardising expected properties and value formats, makes the sharing and processing of data easier. Web pages in particular benefit from encoding structured data in their mark-up where it can be used by search engines and other tools.
-
-
```
Cattrs has some problems with generics [1] [2]. Dacite and marshmallow-dataclasses don't support generics well either, with some issues around Union types.
They do work well for simple python types but what I'd like to see is guarantee that the serialisation operation is completely reversible and if not raise warning/exception.
-
Having run into these issues with Pydantic, we've been using Mashumaro[1], which, while not having all the bells and whistles of Pydantic, has served us pretty well.
-
-
-
Project mention: How I collect and use 50 sources of my personal data | news.ycombinator.com | 2021-02-06
Yep! In fact I've tried interoperating with Datasette (e.g. shared here https://news.ycombinator.com/item?id=25090643 )
One secret sauce is using 'automatic' caching of data in sqlite -- this allows both for faster access and having an additional interface for the data as a collateral https://github.com/karlicoss/cachew#readme
Still need to polish this a bit, but ultimately hoping to properly plug into Datasette, I'm impressed by its data exploration capabilities!
-
-
-
pySHACL is a pure Python module which allows for the validation of RDF graphs against Shapes Constraint Language (SHACL) graphs. It is built on top of rdflib.
-
-
dataconf
Simple dataclasses configuration management for Python with hocon/json/yaml/properties/env-vars/dict support.
Project mention: Show HN: Dataconf, Python dataclasses config (hocon/JSON/YAML/env-vars/dict) | news.ycombinator.com | 2021-11-07 -
lupin is a Python JSON object mapper
Python document object mapper (load python object from JSON and vice-versa)
-
Maat
Validation and transformation library powered by deductive ascending parser. Made to be extended for any kind of project.
https://github.com/Attumm/Maat/blob/main/tests/test_validati...
Since validation is written as dictionaries its possible to store the validations in caching db such as Redis.
And since its simple its easy to extend for anyone use case.
Benchmarks of pydantic has Maat around twice as Pydantic.
-
restframework-serializer-permissions
Drop in replacement for Django Restframework Serializers to add permission based field serialization.
Python Serialization related posts
- The Pocket Guide To API Request Validation You Wish You Had Earlier
- Project Althaia - looking for performance/accuracy feedback on my shallow fork of marshmallow
- Schema.org json-ld static site generator
- cattrs – an open source Python library for structuring and unstructuring data
- I use attrs instead of pydantic
- What is Structured Data?
- What is Structured Data for Advanced SEO?
Index
What are some of the best open-source Serialization projects in Python? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | marshmallow | 5,888 |
| 2 | orjson | 2,736 |
| 3 | Schematics | 2,465 |
| 4 | itsdangerous | 2,376 |
| 5 | strictyaml | 964 |
| 6 | serpy | 885 |
| 7 | KQ | 538 |
| 8 | pysimdjson | 506 |
| 9 | PyLD | 467 |
| 10 | colander | 413 |
| 11 | cattrs | 398 |
| 12 | mashumaro | 362 |
| 13 | Fast JSON schema for Python | 322 |
| 14 | jsons | 212 |
| 15 | cachew | 170 |
| 16 | hjson-py | 167 |
| 17 | Trafaret | 165 |
| 18 | pySHACL | 146 |
| 19 | desert | 104 |
| 20 | dataconf | 34 |
| 21 | lupin is a Python JSON object mapper | 19 |
| 22 | Maat | 18 |
| 23 | restframework-serializer-permissions | 7 |
Are you hiring? Post a new remote job listing for free.
