neoq
starqueue
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
neoq
- Show HN: Hatchet – Open-source distributed task queue
-
Choose Postgres Queue Technology
I just want to commend OP - if they’re here - for choosing an int64 for job IDs, and MD5 for hashing the payload in Neoq, the job library linked [0] from the article.
Especially given the emphasis on YAGNI, you don’t need a UUID primary key, and all of its problems they bring for B+trees (that thing RDBMS is built on), nor do you need the collision resistance of SHA256 - the odds of you creating a dupe job hash with MD5 are vanishingly small.
As to the actual topic, it’s fine IFF you carefully monitor for accumulating dead tuples, and adjust auto-vacuum for that table as necessary. While not something you’d run into at the start, at a modest scale you may start to see issues. May. You may also opt to switch to Redis or something else before that point anyway.
[0]: https://github.com/acaloiaro/neoq
-
Ask HN: Tell us about your project that's not done yet but you want feedback on
Neoq (https://github.com/acaloiaro/neoq) is a background job processor for Go.
Yes, another one. It began from my desire to have a robust Postgres-backed job processor. What I quickly realized was that the interface in front of the queue was what was really important. This allowed me to add both in-memory and Redis (provided by asynq) backends behind the same interface. Which allows dependent projects to switch between different backends in different settings/durable requirements. E.g. in-memory for testing/development, postgres when you're not running Google-scale jobs, and Redis for all the obvious use cases for a Redis-backed queue.
This allows me to swap out job queue backends without changing a line of job processor code.
I'm familiar with the theory that one shouldn't implement queues on Postgres, and to a large extent, I disagree with those theories. I'm confident you can point out a scenario in which one shouldn't, and I contend that those scenarios are the exception rather than the rule.
-
Examples of using task scheduler with Go?
I created a background processor called Neoq (https://github.com/acaloiaro/neoq) that is likely to interest you.
-
SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a Postgres Queue
This is exactly the thesis behind neoq: https://github.com/acaloiaro/neoq
starqueue
-
Choose Postgres Queue Technology
MS SQL server, Postgres and MySQL all support SKIP LOCKED, which means they are all suitable for running queues.
I built a complete implementation in Python designed to work the same as SQS but be more simple:
https://github.com/starqueue/starqueue
Alternatively if you just want to quickly hack something into your application, here is a complete solution in Python with retries:
import psycopg2
-
SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a Postgres Queue
I wrote a message queue in Python called StarQueue.
It’s meant to be a simpler reimagining of Amazon SQS.
It has an HTTP API and behaves mostly like SQS.
I wrote it to support Postgres, Microsoft’s SQL server and so MySQL because they all support SKIP LOCKED.
At some point I turned it into a hosted service and only maintained the Postgres implementation though the MySQL and SQL server code is still in there.
It’s not an active project but the code is at https://github.com/starqueue/starqueue/
- Show Reddit: StarQueue - Postgres database backed message queue server for Python
-
Show HN: StarQueue database backed message queue server for Python
Hi folks,
This is a project I wrote and since I am doing nothing with it any more I thought I would publish the source code in case anyone finds it interesting.
https://github.com/starqueue/starqueue/
StarQueue is a message queue server written in Python.
It is designed to be a more simple copy of Amazon Simple Queue Server.
Clients access it via HTTP. The API is documented at https://github.com/starqueue/starqueue/tree/main/starqueueserver/website
The database is Postgres.
When I developed it initially, I included seamless support for Postgres, MySQL and Microsoft SQL server.
At some point in the development I gave up on all databases except Postgres, though I have left the MySQL and SQL server code in place.
I deployed StarQueue as an online service at one point (no longer online). This github repo is a copy of the source code for that service.
This project is not live and is archived, but I have posted it here in case anyone finds the source code interesting.
What are some alternatives?
oban - 💎 Robust job processing in Elixir, backed by modern PostgreSQL and SQLite3
pg-boss - Queueing jobs in Node.js using PostgreSQL like a boss
tembo - Monorepo for Tembo Operator, Tembo Stacks, and Tembo CLI
kubeblocks - KubeBlocks is an open-source control plane that runs and manages databases, message queues and other data infrastructure on K8s.
Asynq - Simple, reliable, and efficient distributed task queue in Go
BeanstalkD - Beanstalk is a simple, fast work queue.
pgtt - PostgreSQL extension to create, manage and use Oracle-style Global Temporary Tables and the others RDBMS
tqs - Tiny Queue Service (Server)
pgjobq - Atomic low latency job queues running on Postgres
worker - High performance Node.js/PostgreSQL job queue (also suitable for getting jobs generated by PostgreSQL triggers/functions out into a different work queue)
starlark-go - Starlark in Go: the Starlark configuration language, implemented in Go
arniesmtpbufferserver