Top 6 Go data-pipeline Projects

Rudderstack

83 3,926 9.8 Go

Privacy and Security focused Segment-alternative, in Golang and React

Project mention: Rudderstack Switches to Elastic License | news.ycombinator.com | 2023-09-08

memphis

52 3,149 9.9 Go

Memphis.dev is a highly scalable and effortless data streaming platform

Project mention: Memphis | /r/devopspro | 2023-05-11

InfluxDB

www.influxdata.com sponsored

Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
go-streams

9 1,753 6.6 Go

A lightweight stream processing library for Go
conduit

7 345 9.5 Go

Conduit streams data between data stores. Kafka Connect replacement. No JVM required. (by ConduitIO)

Project mention: Pulling CDC data from Postgres | /r/dataengineering | 2023-04-30

I'd like to mention Conduit + its Postgres connector. The Pg connector comes built-in, so all that is needed is a single Conduit binary to get started. It relies on WAL, but the connector creates the replication slot itself (if needed).

ordered-concurrently

3 37 1.3 Go

Ordered-concurrently a library for concurrent processing with ordered output in Go. Process work concurrently and returns output in a channel in the order of input. It is useful in concurrently processing items in a queue, and get output in the order provided by the queue.
pippin

3 14 6.7 Go

Go library to create and manage data pipelines on your machine

Project mention: Go concurrency simplified. Part 4: Post office as a data pipeline | dev.to | 2023-12-21

take a look at the concurrent code written by other devs out there: for example, feel free to check the internals of my library Pippin, but I bet there are many better projects out there to learn from - Google/Bing/DuckDuckGo/Kagi and ChatGPT can help to find the right one

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).