BeanstalkD
pg-boss
BeanstalkD | pg-boss | |
---|---|---|
14 | 12 | |
6,477 | 1,638 | |
0.3% | - | |
0.0 | 4.4 | |
3 days ago | 28 days ago | |
C | JavaScript | |
GNU General Public License v3.0 or later | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
BeanstalkD
-
Ruby 3.3
There's beanstalkd, it has a few Python libraries and it works out of the box with ActiveJob via Backburner.
https://beanstalkd.github.io/
-
A Developer's Journal: Simplifying the Twelve-Factor App
Messaging/Queueing Systems (Amazon SQS, RabbitMQ, Beanstalkd)
- Load Balancing
-
SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a Postgres Queue
Not when a queue is involved. IME trying to replicate something like beanstalkd (https://beanstalkd.github.io/) in postgres is asking for trouble for anything but trivial workloads.
If you're measuring throughput in jobs/s, use a real work queue.
-
Christmas giveaway: 10 copies of my book Domain-driven Design with Golang book, also AMA
Before Kafka was a standard, I created a go library for beanstalkd that act like an RPC.
-
PHP parallel processing idea
Then there are queue libraries like beanstalkd, RabbitMQ or built-in features like queues from Laravel. These will probably get you quicker to your goal then trying the process managing route.
- How to do distributed cronjobs with worker queues?
-
Write Your Own Task Queue
The only task queue I loved was beanstalkd -- it's beautifully written and highly performant. Starting it takes seconds and it's been running for a decade:
https://beanstalkd.github.io/
- Golang task queue
-
What are some popular background job processing frameworks in the Rust ecosystem?
It's not rust (it's C), but beanstalkd is a pretty incredible work queue that processes millions of jobs a day (10K+/s at peak) for my company. I know there are a few rust drivers available.
pg-boss
-
Choose Postgres Queue Technology
For running queues on Postgres with Node.js backend(s), I highly recommend https://github.com/timgit/pg-boss. I'm sure it has it scale limits. But if you're one of the 90% of the apps that never needs any kind of scale that a modern server can't easily handle then it's fantastic. You get transactional queueing of jobs, and it automatically handles syncing across multiple job processing servers using Postgres locks.
-
Build Your Own Personal Twitter Agent 🧠🐦⛓ with LangChain
Jobs use pg-boss, a postgres extension, to queue and run tasks under the hood.
-
SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a Postgres Queue
If you don't want to roll your own, look into https://github.com/timgit/pg-boss
- How/do you handle queue type workflows?
-
Which tool/library well adopted to use Postgres as a message broker?
I saw this https://github.com/timgit/pg-boss but it's more for jobs than for message with multiple consumers (having their own progress offset).
-
How to schedule tasks in a Node.js app 🕙
The best I've used till now. Has all kind of features and really great when you have a postgres dB in your stack. https://github.com/timgit/pg-boss
-
Cluster friendly task scheduler for NodeJS
Check out these; - https://github.com/mitranim/posterus - https://github.com/timgit/pg-boss - https://github.com/FirebaseExtended/firebase-queue - https://www.npmjs.com/package/rabbit-queue
-
You don't need distributed systems.
You can use the simplest option than implement a new service. Keep in mind that every running system can be a job scheduler, you can just use nodejs worker threads, Redis, or even your DB as a job scheduler, check PGBoss for example.
-
Launch HN: Convoy (YC W22) – Open-source cloud-native webhooks service
Both! For context, we're currently using https://github.com/timgit/pg-boss as a task queue on top of postgres and it works great. No need to complicate things with Redis. I believe it's quite straightforward to implement a task queue on top of postgres using the SKIP LOCKED functionality.
- Devious SQL: Message Queuing Using Native PostgreSQL
What are some alternatives?
RabbitMQ - Open source RabbitMQ: core server and tier 1 (built-in) plugins
worker - High performance Node.js/PostgreSQL job queue (also suitable for getting jobs generated by PostgreSQL triggers/functions out into a different work queue)
Apache Kafka - Mirror of Apache Kafka
celery - Distributed Task Queue (development branch)
Gearman
django-postgres-queue - A task queue for django
NATS - High-Performance server for NATS.io, the cloud and edge native messaging system.
Redis - Redis is an in-memory database that persists on disk. The data model is key-value, but many different kind of values are supported: Strings, Lists, Sets, Sorted Sets, Hashes, Streams, HyperLogLogs, Bitmaps.
nsq - A realtime distributed messaging platform
kue - Kue is a priority job queue backed by redis, built for node.js.