Que
realtime
Que | realtime | |
---|---|---|
10 | 54 | |
2,288 | 6,480 | |
0.3% | 1.0% | |
5.6 | 9.2 | |
11 days ago | 3 days ago | |
Ruby | Elixir | |
MIT License | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Que
-
Choose Postgres Queue Technology
> Can you define "low throughput"?
<1000 messages per minute
Not saying SKIP LOCKED can't work with that many. But you'll probably want to do something better.
FWIW, Que uses advisory locks [1]
[1] https://github.com/que-rb/que
-
Introducing tobox: a transactional outbox framework
Probably worth mentioning that aside from delayed_job there are at least two more modern alternatives backed by the DB: Que and good_job.
-
Sidekiq jobs in ActiveRecord transactions
Good article. Sidekiq is a good, well respected too. However if you are starting out I would recommend not using it, and instead choosing a DB based queue system. We have great success with que, but there are others like good_job.
-
SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a Postgres Queue
(not sure why this comment was dead, I vouched for it)
There are a lot of ways to implement a queue in an RDBMS and a lot of those ways are naive to locking behavior. That said, with PostgreSQL specifically, there are some techniques that result in an efficient queue without locking problems. The article doesn't really talk about their implementation so we can't know what they did, but one open source example is Que[1]. Que uses a combination of advisory locking rather than row-level locks and notification channels to great effect, as you can read in the README.
[1]: https://github.com/que-rb/que
-
Delayed Job vs. Sidekiq: Which Is Better?
https://github.com/que-rb/que
This one seems to be the most performant. By a lot too, from my understanding (haven't ran any benchmark myself, but the readme shows some good postgres knowledge)
-
Sidekiq VS Que - a user suggested alternative
2 projects | 3 Feb 2022
Que seems like a good alternative if one doesn't want to use Reids. However, given that most apps need Redis (and have it within their infrastructure) nowadays, I still think that Sidekiq is the better option in the generic case.
-
Devious SQL: Message Queuing Using Native PostgreSQL
Implementations that use advisory locks like https://github.com/que-rb/que are much more efficient (atleast when I last tested) and will easily reach 10k job/s on even very modest hardware.
There is a Go port of Que but you can also easily port it to any language you like. I have a currently non-OSS implementation in Rust that I might OSS someday when I have time to clean it up.
-
Postgres is a great pub/sub and job server
It’s also possible to use advisory locks to implement a job queue in Postgres. See e.g. Que[1]. Note there are a fair number of corner cases, so studying Que is wise if trying to implement something like this, as well as some (a bit older) elaboration[2].
We implemented a similar design to Que for a specific use case in our application that has a known low volume of jobs and for a variety of reasons benefits from this design over other solutions.
[1]: https://github.com/que-rb/que
-
Ruby Schedulers: Whenever vs Sidekiq Cron vs Sidekiq Scheduler
Do also take into consideration que-scheduler (disclaimer, am author). It is built on top of the robust que async job system.
realtime
-
A Technical Dive into PostgreSQL's replication mechanisms
You can LISTEN/NOTIFY. Or you can use logical replication and a custom subscriber.[1] Supabase uses the latter.[2]
[1]: https://www.postgresql.org/docs/current/logical-replication....
[2]: https://github.com/supabase/realtime
-
Supabase Studio: AI Assistant and User Impersonation
Supabase Realtime is great for building collaborative applications. You can receive database changes over websockets, store and synchronize data about user presence, and broadcast any data to clients via "channels".
-
Unpacking Elixir: Observability
We use :telemetry to collect usage data per tenant for Supabase Realtime.
We do this for rate limiting but it also makes it very easy for us to attach a listener (https://github.com/supabase/realtime/blob/main/lib/realtime/...) which ships these (per second) aggregates to BigQuery (via Logflare), which then the billing team can aggregate further to display and actually bill people with.
-
All the ways to capture changes in Postgres
Yo :D This is what Supabase Realtime does!
https://github.com/supabase/realtime
Spin up a Supabase database and then subscribe to changes with WebSockets.
You can play with it here once you have a db: https://realtime.supabase.com/inspector/new
-
Supabase Local Dev: migrations, branching, and observability
Every project is a Postgres database, wrapped in a suite of tools like Auth, Storage, Edge Functions, Realtime and Vectors, and encompassed by API middleware and logs.
- Sync client state globally over WebSockets in Realtime
-
Writing a chat application in Django 4.2 using async StreamingHttpResponse
Where can I learn more about this? I've been thinking of trying to integrate Supabase Realtime (https://github.com/supabase/realtime) into my Django app (without the rest of Supabase), but I'd also like to keep things even simpler if possible.
Also, what was the reason not to go with Gevent?
- Supabase Realtime – Broadcast, Presence, and Postgres Changes via WebSockets
-
How to Listen to Database Changes Using Postgres Triggers in Elixir
I believe #2 was the main driver for the supabase team to build their real-time component: https://github.com/supabase/realtime
Background/announcement: https://supabase.com/blog/supabase-realtime-multiplayer-gene...
-
How To Kill A Fly With A Shotgun
As a minor note, one of the linked articles talks about having used RethinkDB for its changefeeds and I made a mental note a bit back that if I ever want that supabase's realtime ( https://github.com/supabase/realtime ) provides something rather like that atop Postgres and I should try that before doing anything clever.
What are some alternatives?
Sidekiq - Simple, efficient background processing for Ruby
supabase - The open source Firebase alternative.
good_job - Multithreaded, Postgres-based, Active Job backend for Ruby on Rails.
debezium - Change data capture for a variety of databases. Please log issues at https://issues.redhat.com/browse/DBZ.
Delayed::Job - Database based asynchronous priority queue system -- Extracted from Shopify
blockscout - Blockchain explorer for Ethereum based network and a tool for inspecting and analyzing EVM based blockchains.
Resque - Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later.
Appwrite - Your backend, minus the hassle.
Karafka - Ruby and Rails efficient multithreaded Kafka processing framework
litestream - Streaming replication for SQLite.
Shoryuken - A super efficient Amazon SQS thread based message processor for Ruby
Hasura - Blazing fast, instant realtime GraphQL APIs on your DB with fine grained access control, also trigger webhooks on database events.