The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning. Learn more →
Top 7 Python Parquet Projects
-
petastorm
Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
amazon-s3-find-and-forget
Amazon S3 Find and Forget is a solution to handle data erasure requests from data lakes stored on Amazon S3, for example, pursuant to the European General Data Protection Regulation (GDPR)
-
dbd
dbd is a database prototyping tool that enables data analysts and engineers to quickly load and transform data in SQL databases.
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
Project mention: Parquet-WASM: Rust-based WebAssembly bindings to read and write Parquet data | news.ycombinator.com | 2024-04-22I'll let Kyle chime in but I tested it a few months ago with millions of polygons on an M2 16GB of RAM laptop and it worked very well.
There is a library by the same author called lonboard that provides the JS bits inside JupyterLab. https://github.com/developmentseed/lonboard
I think it is based on the Kepler.gl / Deck.gl data loaders that go straight to GPU from network.
Project mention: Launch HN: Grai (YC S22) – Open-Source Data Observability Platform | news.ycombinator.com | 2023-07-17Elastic v2 if one is interested in such things: https://github.com/grai-io/grai-core/blob/v0.1.33/LICENSE
Project mention: data-toolset: Upgrade from avro-tools and parquet-tools jars to a more user-friendly Python package. | /r/dataengineering | 2023-09-22
Python Parquet related posts
- FLaNK Stack for 4th of July
- pystore: NEW Data - star count:406.0
- pystore: NEW Data - star count:406.0
- pystore: NEW Data - star count:406.0
- pystore: NEW Data - star count:406.0
- pystore: NEW Data - star count:406.0
- pystore: NEW Data - star count:406.0
-
A note from our sponsor - WorkOS
workos.com | 25 Apr 2024
Index
What are some of the best open-source Parquet projects in Python? This list will help you:
Project | Stars | |
---|---|---|
1 | petastorm | 1,751 |
2 | pystore | 539 |
3 | lonboard | 385 |
4 | grai-core | 267 |
5 | amazon-s3-find-and-forget | 232 |
6 | dbd | 55 |
7 | data-toolset | 1 |
Sponsored