pg-boss
neoq
pg-boss | neoq | |
---|---|---|
12 | 5 | |
1,638 | 243 | |
- | - | |
4.4 | 8.3 | |
28 days ago | 11 days ago | |
JavaScript | Go | |
MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
pg-boss
-
Choose Postgres Queue Technology
For running queues on Postgres with Node.js backend(s), I highly recommend https://github.com/timgit/pg-boss. I'm sure it has it scale limits. But if you're one of the 90% of the apps that never needs any kind of scale that a modern server can't easily handle then it's fantastic. You get transactional queueing of jobs, and it automatically handles syncing across multiple job processing servers using Postgres locks.
-
Build Your Own Personal Twitter Agent 🧠🐦⛓ with LangChain
Jobs use pg-boss, a postgres extension, to queue and run tasks under the hood.
-
SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a Postgres Queue
If you don't want to roll your own, look into https://github.com/timgit/pg-boss
- How/do you handle queue type workflows?
-
Which tool/library well adopted to use Postgres as a message broker?
I saw this https://github.com/timgit/pg-boss but it's more for jobs than for message with multiple consumers (having their own progress offset).
-
How to schedule tasks in a Node.js app 🕙
The best I've used till now. Has all kind of features and really great when you have a postgres dB in your stack. https://github.com/timgit/pg-boss
-
Cluster friendly task scheduler for NodeJS
Check out these; - https://github.com/mitranim/posterus - https://github.com/timgit/pg-boss - https://github.com/FirebaseExtended/firebase-queue - https://www.npmjs.com/package/rabbit-queue
-
You don't need distributed systems.
You can use the simplest option than implement a new service. Keep in mind that every running system can be a job scheduler, you can just use nodejs worker threads, Redis, or even your DB as a job scheduler, check PGBoss for example.
-
Launch HN: Convoy (YC W22) – Open-source cloud-native webhooks service
Both! For context, we're currently using https://github.com/timgit/pg-boss as a task queue on top of postgres and it works great. No need to complicate things with Redis. I believe it's quite straightforward to implement a task queue on top of postgres using the SKIP LOCKED functionality.
- Devious SQL: Message Queuing Using Native PostgreSQL
neoq
- Show HN: Hatchet – Open-source distributed task queue
-
Choose Postgres Queue Technology
I just want to commend OP - if they’re here - for choosing an int64 for job IDs, and MD5 for hashing the payload in Neoq, the job library linked [0] from the article.
Especially given the emphasis on YAGNI, you don’t need a UUID primary key, and all of its problems they bring for B+trees (that thing RDBMS is built on), nor do you need the collision resistance of SHA256 - the odds of you creating a dupe job hash with MD5 are vanishingly small.
As to the actual topic, it’s fine IFF you carefully monitor for accumulating dead tuples, and adjust auto-vacuum for that table as necessary. While not something you’d run into at the start, at a modest scale you may start to see issues. May. You may also opt to switch to Redis or something else before that point anyway.
[0]: https://github.com/acaloiaro/neoq
-
Ask HN: Tell us about your project that's not done yet but you want feedback on
Neoq (https://github.com/acaloiaro/neoq) is a background job processor for Go.
Yes, another one. It began from my desire to have a robust Postgres-backed job processor. What I quickly realized was that the interface in front of the queue was what was really important. This allowed me to add both in-memory and Redis (provided by asynq) backends behind the same interface. Which allows dependent projects to switch between different backends in different settings/durable requirements. E.g. in-memory for testing/development, postgres when you're not running Google-scale jobs, and Redis for all the obvious use cases for a Redis-backed queue.
This allows me to swap out job queue backends without changing a line of job processor code.
I'm familiar with the theory that one shouldn't implement queues on Postgres, and to a large extent, I disagree with those theories. I'm confident you can point out a scenario in which one shouldn't, and I contend that those scenarios are the exception rather than the rule.
-
Examples of using task scheduler with Go?
I created a background processor called Neoq (https://github.com/acaloiaro/neoq) that is likely to interest you.
-
SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a Postgres Queue
This is exactly the thesis behind neoq: https://github.com/acaloiaro/neoq
What are some alternatives?
worker - High performance Node.js/PostgreSQL job queue (also suitable for getting jobs generated by PostgreSQL triggers/functions out into a different work queue)
starqueue
celery - Distributed Task Queue (development branch)
oban - 💎 Robust job processing in Elixir, backed by modern PostgreSQL and SQLite3
django-postgres-queue - A task queue for django
tembo - Monorepo for Tembo Operator, Tembo Stacks, and Tembo CLI
RabbitMQ - Open source RabbitMQ: core server and tier 1 (built-in) plugins
Asynq - Simple, reliable, and efficient distributed task queue in Go
Redis - Redis is an in-memory database that persists on disk. The data model is key-value, but many different kind of values are supported: Strings, Lists, Sets, Sorted Sets, Hashes, Streams, HyperLogLogs, Bitmaps.
pgtt - PostgreSQL extension to create, manage and use Oracle-style Global Temporary Tables and the others RDBMS
kue - Kue is a priority job queue backed by redis, built for node.js.
pgjobq - Atomic low latency job queues running on Postgres