cainophile
| cainophile | mssql-changefeed | |
|---|---|---|
| 2 | 4 | |
| 235 | 14 | |
| 1.7% | - | |
| 1.7 | 6.6 | |
| 3 months ago | 5 months ago | |
| Elixir | Go | |
| Apache License 2.0 | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
cainophile
-
How to Listen to Database Changes Using Postgres Triggers in Elixir
If you want to listen to database changes in Elixir you can also get really good stuff done by using Cainophile (https://github.com/cainophile/cainophile). Same mechanism. I don't know the details of Debezium so I can't say if you are leaving fantastic things on the table. But I've had good fun with Cainophile. For example I've used it in my videos on Electric SQL to react to changes in a Postgres database. It matches nicely with realtime-ish UI via LiveView. So meshes really good with the current Elixir stack.
-
Ask HN: Is there a way to subscribe to an SQL query for changes?
https://github.com/cainophile/cainophile
I happened to do a similar thing but I adapted cainophile into an Elixir “OffBroadway” producer:
mssql-changefeed
-
How to Listen to Database Changes Using Postgres Triggers in Elixir
We made mssql-changefeed for this purpose:
https://github.com/vippsas/mssql-changefeed
V1 requires to run a sweeper procedure in the background, but an upcoming v2 version does without the sweep loop. Unfortunately too fresh for readme to be updated, but relevant lines in tests to show it off:
https://github.com/vippsas/mssql-changefeed/blob/v1-lazy/go/...
https://github.com/vippsas/mssql-changefeed/blob/v1-lazy/go/...
https://github.com/vippsas/mssql-changefeed/blob/v1-lazy/go/...
Library itself:
https://github.com/vippsas/mssql-changefeed/blob/v1-lazy/mig...
-
Push-Based Outbox Pattern with Postgres Logical Replication
Not OP, but there is an approach here of using a dedicated loop worker to assign post-commit ID sequence. I.e. using the outbox pattern once, simply to assign a post-commit ID.
https://github.com/vippsas/mssql-changefeed/blob/main/MOTIVA...
I wish DBs had this more built in, it seems a critical feature of a DB these days and the commit log already have very similar sequence numbers internally...
-
The Next Generation of Materialize
We do something similar, but in 2), instead of using the outbox pattern, we make use (in several different settings) of integers that are guaranteed to increment in commit order, then each consumer can track where their cursor is on the feed of changes. This requires some more coordination but it means that publishers of changes don't need one outbox per consumer or similar.
Then you can have "processes" that query for new data in an input table, and update aggregates/derived tables from that simply by "select * ... where ChangeSequenceNumber > @MaxSequenceNumberFromPreviousExecution"...
The idea here implemented for Microsoft SQL for the OLTP case:
https://github.com/vippsas/mssql-changefeed
What are some alternatives?
noria - Fast web applications through dynamic, partially-stateful dataflow
marmot - A distributed SQLite replicator built on top of NATS
supavisor - A cloud-native, multi-tenant Postgres connection pooler.
timely-dataflow - A modular implementation of timely dataflow in Rust
pgcat - PostgreSQL pooler with sharding, load balancing and failover support.
PipelineDB - High-performance time-series aggregation for PostgreSQL
zeroeventhub
flow - 🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, with Estuary Flow. 🌊
rust-postgres - Native PostgreSQL driver for the Rust programming language
revori - A revision-oriented DBMS
postgrex_pubsub - A helper to turn postgres mutations into a pubsub channel