walex VS cockroach

The WAL CDC approach: https://github.com/cpursley/walex?tab=readme-ov-file#publica...

CREATE PUBLICATION news_item FOR TABLE news WHERE (topic IS "AAPL");

What do you mean testing with processes?

I won't suggest these are the best written tests, but I test various processes, supervisors, etc like this:

https://github.com/cpursley/walex/blob/master/test/walex/sup...

You don’t send the entire WAL, just what you subscribe to - and you can even filter via SQL: https://github.com/cpursley/walex?tab=readme-ov-file#publica...

What we do is much if the business logic in Postgres. But then there’s all the other stuff like external integrations, etc.

We handled that by having an Even system built on the Postgres WAL that we use like a callback system.

I put together a little library in Elixir (that originally started out as forked Supabase realtime) for this:

https://github.com/cpursley/walex

Recently added the ability to configure WalEx to forward events to webhooks or EventRelay (so you don’t need to know Elixir).

If you're using Elixir, check out https://github.com/cpursley/walex

I've actually been thinking about turning this idea into a product where you can just point it at your postgres database and select the tables you want to listen to (with filters, like you describe). And have that forwarded to a webhook (with possibility of other protocols).

I'd love to hear folks thoughts on that (and if it would be something people would pay for).

Neat, this is the pattern I've been thinking about for a while now. Also glad to see this is Elixir based.

I've been using https://github.com/cpursley/walex (basically a fork of cainophile via a fork for subabase) to listen to Postgres changes in Elixir.

https://github.com/cpursley/walex

Which is based on WAL logs and doesnt have the same limitations.

Great writeup, just added this to the WalEx readme:

https://github.com/cpursley/walex

(WalEx is an Elixir lib for listening to the WAL and perform callback-like actions with the data)

Very cool.

I'm using Elixir to listen to change events via https://github.com/cpursley/walex (which I basically ripped off from Supabase).

CockroachDB is an open source distributed SQL database designed for scalability and resilience. While it offers SQL databases, CockroachDB is also compatible with PostgreSQL.

cockroachdb might be close: https://github.com/cockroachdb/cockroach

CockroachDB - SQL

For example, when you create a serverless postgres database with a platform like CockroachDB or Neon, you effectively get a connection string with a strong password. Anyone can connect to your database from anywhere so long as they have the right connection string. There are no security settings in these services to change this behavior.

> Yes you can on the android emulator. The biggest issue is compu arch in that case.

I can also download VirtualBox and run all Windows programs, that would mean that all Windows apps are Linux apps?

> Yes you can for the most part

You can't statically link glibc: https://github.com/cockroachdb/cockroach/issues/3392

glibc can break stuff: https://www.gamingonlinux.com/2022/08/valve-dev-understandab...

I had binaries break because the newer version if openssl was put under a slightly different name.

Also, worth noting, if you're already using PostgreSQL (or plan to) you might want to take a look at https://www.cockroachlabs.com/ they have a free tier too and CockroachDB has a PostgreSQL interface.

I know that some projects like cockroach use custom build tools like bazel. But we actually really like to use to be able to build our projects simply with the great go toolchain and don't really aim to dive deep into custom build solutions.

CockroachDB is designed to be globally distributed. It has to handle causality when resolving collisions. It has to account for having a write operation to arrive after another and still have time priority because it was sent out a few milliseconds earlier.

Cockroachdb created a go runtime patch which measures the Grunning time of a goroutine: https://github.com/cockroachdb/cockroach/pull/82356. It doesn't entirely solve the problem though.

Our entire backend is written in Go. We've built a platform that allows other companies to offer automatic data syncing to their customers' data warehouses. Go works great for building distributed systems like this (see K8s). We're not the only ones in the space building data intensive applications with Go. Pachyderm, Pinecone, Cockroach Labs and are all also doing it. We've been quite happy with how Go has worked for us.