Top 6 Go Bigdata Projects

juicefs

12 8,066 8.9 Go

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

Project mention: Google Cloud Storage FUSE | news.ycombinator.com | 2023-05-02

See also: JuiceFS: https://juicefs.com/
Adds a DBMS or key-value store for metadata, making the filesystem much faster (POSIX, small overwrites don't have to replace a full object in the GCS/S3 backend).
Almost certainly a better solution if you want to turn your object storage into a mountable filesystem, with the (big) caveat that you can't access the files directly in the bucket (they are not stored transparently).
volcano

1 3,028 9.4 Go

A Cloud Native Batch System (Project under CNCF)

Project mention: Can we specify nodeSelector inline for a kubectl command | /r/kubernetes | 2022-08-19

Also, if you are creating bare pods, this sounds like batch scheduling and you should consider using Jobs instead, to have a pod controller. And then you could also consider the https://volcano.sh/ scheduler if it has a fitting scheduling plugin for your use case.
InfluxDB

www.influxdata.com
sponsored

Access the most powerful time series database as a service. Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression.
cds

0 926 3.5 Go

Data syncing in golang for ClickHouse. (by zeromicro)
sidekick

0 480 4.3 Go

High Performance HTTP Sidecar Load Balancer (by minio)
meteor

1 145 6.9 Go

Meteor is an easy-to-use, plugin-driven metadata collection framework to extract data from different sources and sink to any data catalog. (by odpf)

Project mention: Modern open-source data platform that empowers organizations to discover, transform, analyse and secure data faster and efficiently. | /r/dataengineering | 2022-06-21

Meteor is an easy-to-use, plugin-driven metadata collection framework to extract data from different sources and sink to any data catalog.
Meduce

1 4 10.0 Go

MapReduce library for data processing

Project mention: My MapReduce library | /r/golang | 2023-04-30

https://github.com/djordje200179/Meduce https://pkg.go.dev/github.com/djordje200179/meduce

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2023-05-02.