Top 10 Go Bigquery Projects

cloudquery

102 5,581 10.0 Go

The open source high performance ELT framework powered by Apache Arrow

Project mention: We might want to regularly keep track of how important each server is | news.ycombinator.com | 2024-02-06

Check out CloudQuery - https://github.com/cloudquery/cloudquery for an easy cloud asset inventory.

Rudderstack

83 3,926 9.8 Go

Privacy and Security focused Segment-alternative, in Golang and React

Project mention: Rudderstack Switches to Elastic License | news.ycombinator.com | 2023-09-08

InfluxDB

www.influxdata.com sponsored

Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
tbls

6 3,057 8.9 Go

tbls is a CI-Friendly tool for document a database, written in Go.

Project mention: FLaNK 25 December 2023 | dev.to | 2023-12-26

peerdb

5 1,595 9.9 Go

Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage

Project mention: Pgwire: a Rust library for PostgreSQL compatible application | news.ycombinator.com | 2024-03-20

We at PeerDB (https://github.com/PeerDB-io/peerdb) were early adopters of Pgwire to implement our Postgres-compatible SQL Layer to do ETL. Very easy to work with. Saved us multiple months of effort to build it from scratch.

scratchdata

5 1,027 9.4 Go

Scratch is a swiss army knife for big data.

Project mention: Debugging a Golang Bug with Non-Blocking Reads | news.ycombinator.com | 2024-03-12

Go team does acknowledge [1] it as a bug, so there is some point here
However, that said, I wonder if OP (duckdb) could have written their solution [2] differently. Shouldn't they be able to select from a Pipe as well as Error channel simultaneously? (similar to how they are doing it inside here [3]). If not, I would have create a go-routine that does blocking read on the Pipe and then pass it on to another channel to select on.
[1] https://github.com/golang/go/issues/66239
[2] https://github.com/scratchdata/scratchdata/blob/7c1a0fcd0e20...
[3] https://github.com/scratchdata/scratchdata/blob/7c1a0fcd0e20...

optimus

5 737 4.4 Go

Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management. (by raystack)

Project mention: Data Engineering Tools in Go | /r/dataengineering | 2023-05-18

You can check odpf github, they created some dataops tools using go, one of the example is optimus (https://github.com/odpf/optimus) which is a data pipeline orchestrator

bigquery-emulator

1 704 7.1 Go

BigQuery emulator server implemented in Go
WorkOS

workos.com sponsored

The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
transfer

7 527 9.4 Go

Database replication platform that leverages change data capture. Stream production data from databases to your data warehouse (Snowflake, BigQuery, Redshift) in real-time.

Project mention: Migrate mongodb Datawarehouse to snowflake | /r/snowflake | 2023-12-04

pgsink

5 76 0.0 Go

Logically replicate data out of Postgres into sinks (files, Google BigQuery, etc)
protobuf-bigquery-go

1 43 7.6 Go

Seamlessly save and load protocol buffers to and from BigQuery using Go.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).