Top 23 Avro Open-Source Projects

• Apache Avro

  22 2,756 9.7 Java

  Apache Avro is a data serialization system.

  

Project mention: Open Table Formats Such as Apache Iceberg Are Inevitable for Analytical Data | news.ycombinator.com | 2024-01-18

Apache AVRO [1] is one but it has been largely replaced by Parquet [2] which is a hybrid row/columnar format

[1] https://avro.apache.org/

• rq

  10 2,254 3.2 Rust

  Record Query - A tool for doing record analysis and transformation (by dflemstr)

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• schema-registry

  7 2,136 10.0 Java

  Confluent Schema Registry for Kafka

  

Project mention: JR, quality Random Data from the Command line, part I | dev.to | 2023-05-07

So, is JR yet another faking library written in Go? Yes and no. JR indeed implements most of the APIs in fakerjs and Go fake it, but it's also able to stream data directly to stdout, Kafka, Redis and more (Elastic and MongoDB coming). JR can talk directly to Confluent Schema Registry, manage json-schema and Avro schemas, easily maintain coherence and referential integrity. If you need more than what is OOTB in JR, you can also easily pipe your data streams to other cli tools like kcat thanks to its flexibility.

• examples

  6 1,845 8.5 Shell

  Apache Kafka and Confluent Platform examples and demos (by confluentinc)

  

• DataProfiler

  61 1,362 6.3 Python

  What's in your data? Extract schema, statistics and entities from datasets

  

Project mention: LongRoPE: Extending LLM Context Window Beyond 2M Tokens | news.ycombinator.com | 2024-02-22

It's been possible to skip tokenization for a long time, my team and I did it here - https://github.com/capitalone/DataProfiler

For what it's worth, we actually were working with LSTMs with nearly a billion params back in 2016-2017 area. Transformers made it far more effective to train and execute, but ultimately LSTMs are able to achieve similar results, though slow & require more training data.

• avsc

  1 1,245 4.9 JavaScript

  Avro for JavaScript :zap:

• pmacct

  8 1,014 9.2 C

  pmacct is a small set of multi-purpose passive network monitoring tools [NetFlow IPFIX sFlow libpcap BGP BMP RPKI IGP Streaming Telemetry].

  

Project mention: NetFlow-equivalent analysis for mirrored traffic | /r/networking | 2023-07-12

If you want a tool that can ingest from a span port and generate netflow or IPFIX there is pmacct. This should work with your existing tooling that collects netflow data.

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• adam

  3 967 6.1 Scala

  ADAM is a genomics analysis platform with specialized file formats built using Apache Avro, Apache Spark, and Apache Parquet. Apache 2 licensed.

  

• kafkactl

  1 748 8.7 Go

  Command Line Tool for managing Apache Kafka

  

• Cinchoo ETL

  0 735 6.4 C#

  ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

• Avro4s

  2 716 5.9 Scala

  Avro schema generation and serialization / deserialization for Scala

Project mention: Scala 3 Macros: How to Read Annotations | /r/scala | 2023-06-26

for example https://github.com/sksamuel/avro4s - check AvroName and AvroNamespace

• vscode-data-preview

  2 522 2.7 TypeScript

  Data Preview 🈸 extension for importing 📤 viewing 🔎 slicing 🔪 dicing 🎲 charting 📊 & exporting 📥 large JSON array/config, YAML, Apache Arrow, Avro, Parquet & Excel data files

• SlimMessageBus

  1 433 7.1 C#

  Lightweight message bus interface for .NET (pub/sub and request-response) with transport plugins for popular message brokers.

• NoProto

  8 362 0.0 Rust

  Flexible, Fast & Compact Serialization with RPC

• compendium-client

  5 326 0.0 Haskell

  Mu (μ) is a purely functional framework for building micro services.

  

• mongo-kafka

  2 322 6.7 Java

  MongoDB Kafka Connector

  

Project mention: Difficulty configuring log4j when deploying code as plugin for an app | /r/CodingHelp | 2023-10-27

I am working on a custom Kafka-Mongo sink connector (specifically, a custom WriteModelStrategy to be used with the official Mongo sink connector here: https://github.com/mongodb/mongo-kafka ). My code is not a standalone, executable Java application but rather a JAR that augments the functionality of another Java application.

• kafka-connect-file-pulse

  4 305 8.9 Java

  🔗 A multipurpose Kafka Connect connector that makes it easy to parse, transform and stream any file, in any format, into Apache Kafka

  

Project mention: Kafka Connect Filepulse 2.13.0 is now available! This version includes support for SFTP and Alibaba OSS. It also contains many bug fixes and improvements. 🚀 | /r/apachekafka | 2023-09-15

• ABRiS

  2 221 5.9 Scala

  Avro SerDe for Apache Spark structured APIs.

  

• srclient

  1 221 2.4 Go

  Golang Client for Schema Registry

• rumble

  1 207 8.5 Java

  ⛈️ RumbleDB 1.21.0 "Hawthorn blossom" 🌳 for Apache Spark | Run queries on your large-scale, messy JSON-like data (JSON, text, CSV, Parquet, ROOT, AVRO, SVM...) | No install required (just a jar to download) | Declarative Machine Learning and more (by RumbleDB)

  

• avro4k

  2 180 6.0 Kotlin

  Avro support for kotlinx.serialization

  

Project mention: People who use Spring and Kotlin... | /r/Kotlin | 2023-09-26

I've just recently implemented Google Pub/Sub with Avro serialization using Avro4k library and it works fine https://github.com/avro-kotlin/avro4k

• clickhouse-sink-connector

  1 173 9.8 Python

  Replicate data from MySQL, Postgres and MongoDB to ClickHouse

  

• datagen

  7 133 6.1 TypeScript

  Generate authentic looking mock data based on a SQL, JSON or Avro schema and produce to Kafka in JSON or Avro format.

  

Project mention: What are your favorite tools or components in the Kafka ecosystem? | /r/apachekafka | 2023-05-31

For fake data, shameless plug for https://github.com/MaterializeInc/datagen/tree/main

• SaaSHub

  www.saashub.com sponsored

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Avro related posts

• People who use Spring and Kotlin...
  2 projects | /r/Kotlin | 26 Sep 2023
• Scala 3 Macros: How to Read Annotations
  1 project | /r/scala | 26 Jun 2023
• JR, quality Random Data from the Command line, part II
  3 projects | dev.to | 1 Jun 2023
• What are some good publicly available real-time data sources?
  6 projects | /r/dataengineering | 25 May 2023
• Haskell jobs at Standard Chartered, various locations and seniority
  1 project | /r/haskell | 17 Apr 2023
• Simulating Streaming Data for Fraud Detection with Datagen CLI
  3 projects | dev.to | 3 Apr 2023
• How train my SQL skills with real world data engineering problems ?
  2 projects | /r/dataengineering | 20 Mar 2023
• A note from our sponsor - WorkOS
  workos.com | 25 Apr 2024

  The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning. Learn more →

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com