Top 3 Python Avro Projects

DataProfiler

61 1,357 6.3 Python

What's in your data? Extract schema, statistics and entities from datasets

Project mention: LongRoPE: Extending LLM Context Window Beyond 2M Tokens | news.ycombinator.com | 2024-02-22

It's been possible to skip tokenization for a long time, my team and I did it here - https://github.com/capitalone/DataProfiler
For what it's worth, we actually were working with LSTMs with nearly a billion params back in 2016-2017 area. Transformers made it far more effective to train and execute, but ultimately LSTMs are able to achieve similar results, though slow & require more training data.

clickhouse-sink-connector

1 168 9.8 Python

Replicate data from MySQL, Postgres and MongoDB to ClickHouse
InfluxDB

www.influxdata.com sponsored

Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
data-toolset

1 1 6.8 Python

Upgrade from avro-tools and parquet-tools jars to a more user-friendly Python package.

Project mention: data-toolset: Upgrade from avro-tools and parquet-tools jars to a more user-friendly Python package. | /r/dataengineering | 2023-09-22

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).