Go Stream Processing

Open-source Go projects categorized as Stream Processing
Edit details
Topics: Go Golang ETL Kafka streaming-data
Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
Sponsored

Top 17 Go Stream Processing Projects

  • Benthos

    Fancy stream processing made operationally mundane

    Project mention: Ask HN: Who is hiring? (December 2023) | news.ycombinator.com | 2023-12-01

  • watermill

    Building event-driven applications the easy way in Go.

    Project mention: Microservices communication | /r/golang | 2023-12-09

    I’ve successfully worked on projects using an asynchronous event-driven way of connecting services. I really like the decoupling of business logic and the events triggering it. I highly recommend https://github.com/ThreeDotsLabs/watermill to be more flexible when it comes to choosing the actual technology driving the async patter. It might be NATS today but requirements might change and you need to change. Watermill prepares you for this.

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  • go-streams

    A lightweight stream processing library for Go

  • yomo

    🦖 Stateful Serverless Framework for building Geo-distributed Edge AI Infra

    Project mention: FireScroll - An unkillable multi-region KV database that scales reads to infinity | /r/golang | 2023-05-01

    I have used HTTP3 QUIC for really fast edge. This golang project: https://github.com/yomorun/yomo

  • peerdb

    Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage

    Project mention: Pgwire: a Rust library for PostgreSQL compatible application | news.ycombinator.com | 2024-03-20

    We at PeerDB (https://github.com/PeerDB-io/peerdb) were early adopters of Pgwire to implement our Postgres-compatible SQL Layer to do ETL. Very easy to work with. Saved us multiple months of effort to build it from scratch.

  • core

    Build platforms that flexibly mix SQL, batch, and stream processing paradigms (by gazette)

    Project mention: Lakehouse - event data | /r/dataengineering | 2023-07-19

    If you happen to be putting together a platform then look at Gazette. We use it to read/write streaming data into nice and orderly "fragment files" in cloud storage. It provides excellent durability guarantees and turns out to be an extremely efficient way to handle both reads and writes of streaming data. Readers get handed out pre-signed cloud storage URLs that are fetched by the client, and writes are batched into time and/or size bounded "fragments" that are durably replicated until being uploaded into cloud storage.

  • function-mesh

    The serverless framework purpose-built for event streaming applications.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  • gostream

    A Go port of the Java Streams API. Type-safe and functional Go Streams processing for Go 1.18+.

  • machine

    Machine is a workflow/pipeline library for processing data (by whitaker-io)

  • stream

    Go Stream, like Java 8 Stream. (by youthlin)

  • goio

    An implementation of IO, Stream, Fiber for golang inspired by cats and fs2

  • csvplus

    csvplus extends the standard Go encoding/csv package with fluent interface, lazy stream operations, indices and joins.

  • vaero

    Complete log pipeline solution in Python. Runs fast in Go.

  • stream

    stream is a concurrency pattern implementation using Go generics. a.k.a. my accidentally awesome library (by devnw)

  • styx

    Simple, high-performance event streaming broker

    Project mention: Iggy.rs – building message streaming in Rust | news.ycombinator.com | 2024-01-04

    Built something similar in Go with a friend a few years ago.

    https://github.com/thibauts/styx

  • turbine-go

    Turbine Library for Go

  • go-bench-stream

    🌊 Go Benchmarks for Stream Processing

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2024-03-20.

Go Stream Processing related posts

Index

What are some of the best open-source Stream Processing projects in Go? This list will help you:

Project Stars
1 Benthos 7,516
2 watermill 6,697
3 go-streams 1,746
4 yomo 1,607
5 peerdb 1,595
6 core 516
7 function-mesh 208
8 gostream 146
9 machine 142
10 stream 85
11 goio 78
12 csvplus 66
13 vaero 65
14 stream 63
15 styx 60
16 turbine-go 16
17 go-bench-stream 0

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com