failure VS cockroach

> It's not like there's likely hidden segfaults or memory vulns waiting there…

It's funny you mention that, because the now-deprecated 'failure' crate has such a memory-safety vulnerability: https://github.com/rust-lang-deprecated/failure/issues/336

Granted, consuming code is only vulnerable if they opt-in to implementing a provided trait method that most people should never ever need to implement. But I would still try to eliminate 'failure' from my dependency graph if possible (and I recently submitted PRs to two dependencies I was using to remove 'failure' from _their_ dependencies).

The Rust way. And then deprecate just before 1.0. Crying in failure crate

Cockroachdb created a go runtime patch which measures the Grunning time of a goroutine: https://github.com/cockroachdb/cockroach/pull/82356. It doesn't entirely solve the problem though.

Our entire backend is written in Go. We've built a platform that allows other companies to offer automatic data syncing to their customers' data warehouses. Go works great for building distributed systems like this (see K8s). We're not the only ones in the space building data intensive applications with Go. Pachyderm, Pinecone, Cockroach Labs and are all also doing it. We've been quite happy with how Go has worked for us.

I would recommend researching CockroachDB. It does what you're describing natively. The basis of it's design is to distribute data based on geography. It's inexpensive and very simple to install and manage. It's based on Postgres ( but doesn't support all Postgres features). We've been using it for about 1 year and it meets our needs for some of our workload. www.cockroachlabs.com

CockroachDB

You absolutely can write very high performance software in Go, that's kind of the point. You can efficiently interface with C libraries. You can create the sort of software everyone says should be done in Rust, like databases and web servers and system orchestration and games and every other goddamn thing that people will say isn't the right choice for Go.

PostgreSQL Parser separated from CockroachDB, a distributed DB.

Spencer Kimball (now CEO at CockroachDB) wrote an interesting article on this topic in 2021 where they created spencerkimball/stargazers based on a Python script. So I started thinking: could I create a data pipeline using Nifi and Kafka (two OSS tools often used with Druid) to get the API data into Druid - and then use SQL to do the analytics? The answer was yes! And I have documented the outcome below. Here’s my analytical pipeline for Github stars data using Nifi, Kafka and Druid.