Processing JSON 2.5x faster than simdjson with msgspec

• ultrajson

  3 4,248 7.0 C

  Ultra fast JSON decoder and encoder written in C with Python bindings

  

ujson

• orjson

  17 5,540 8.3 Python

  Fast, correct Python JSON library supporting dataclasses, datetimes, and numpy

orjson

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• pysimdjson

  6 628 5.3 Python

  Python bindings for the simdjson project.

simdjson

• msgspec

  31 1,857 8.9 Python

  A fast serialization and validation library, with builtin support for JSON, MessagePack, YAML, and TOML

Msgspec achieves this performance by doing less work - it's only parsing the fields that are used for the query. Allocating objects in Python can be slow, by specifying the required fields for the query (though a type annotated schema), we reduce allocations to the bare minimum resulting in measurable speedups. There's a longer writeup in the example docs if you're interested.

• conda

  30 6,078 9.8 Python

  A system-level, binary package and environment manager running on all major operating systems and platforms.

  

I wrote this up as a demo of some potential optimization work in conda, but the same principle could be applied to other projects.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project