core
goio
Our great sponsors
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
core
-
Lakehouse - event data
If you happen to be putting together a platform then look at Gazette. We use it to read/write streaming data into nice and orderly "fragment files" in cloud storage. It provides excellent durability guarantees and turns out to be an extremely efficient way to handle both reads and writes of streaming data. Readers get handed out pre-signed cloud storage URLs that are fetched by the client, and writes are batched into time and/or size bounded "fragments" that are durably replicated until being uploaded into cloud storage.
-
Request for example Real Life programs in Go?
this is a real time data streaming tool built with Go - https://github.com/gazette/core
-
Hacker News top posts: Jul 16, 2022
Gazette – Build platforms that flexibly mix SQL, batch, and stream processing\ (0 comments)
- Gazette – Build platforms that flexibly mix SQL, batch, and stream processing
-
Issues we've encountered while building a Kafka based data processing pipeline
If you're in the Go ecosystem, Gazette [0] offers transactional integrations [1] with remote DB's for stateful processing pipelines, as well as local stores for embedded in-process state management.
It also natively stores data as files in cloud storage. Brokers are ephemeral, you don't need to migrate data between them, and you're not constrained by their disk size. Gazette defaults to exactly-once semantics, and has stronger replication guarantees (your R factor is your R factor, period -- no "in sync replicas").
Estuary Flow [2] is building on Gazette as an implementation detail to offer end-to-end integrations with external SaaS & DB's for building real-time dataflows, as a managed service.
[0]: https://github.com/gazette/core
goio
We haven't tracked posts mentioning goio yet.
Tracking mentions began in Dec 2020.
What are some alternatives?
Benthos - Fancy stream processing made operationally mundane
machine - Machine is a workflow/pipeline library for processing data
Commander - Build event-driven and event streaming applications with ease
go-streams - A lightweight stream processing library for Go
pg_cron - Run periodic jobs in PostgreSQL
yomo - 🦖 Stateful Serverless Framework for building Geo-distributed Edge AI Infra
go-formatter - A curated list of awesome Go frameworks, libraries and software
benthos - Fancy stream processing made operationally mundane [Moved to: https://github.com/benthosdev/benthos]
vala-www - Website of the Vala programming language
vectorsql - VectorSQL is a free analytics DBMS for IoT & Big Data, compatible with ClickHouse.