litequeue
neoq
litequeue | neoq | |
---|---|---|
3 | 5 | |
138 | 244 | |
3.6% | - | |
7.4 | 8.3 | |
about 2 months ago | 15 days ago | |
Python | Go | |
MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
litequeue
-
Choose Postgres Queue Technology
To make sure you that the message you are trying to retrieve hasn't been locked already by another worker.
[0]: https://github.com/litements/litequeue/
[1]: https://github.com/litements/litequeue/blob/3fece7aa9e9a31e4...
-
SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a Postgres Queue
SQLite is missing some features like `SELECT FOR UPDATE`, but you can work around some issues with a few extra queries. I wrote litequeue[0] with this specific purpose. I haven't been able to use it a lot, so I don't have real-world numbers of how it scales, but the scaling limits depend on how fast you can insert into the database.
[0]: https://github.com/litements/litequeue
-
What's New in SQLite 3.35
The `RETURNING` is so awesome! I'm implementing a set of data structures on top of SQLite, one of them is a queue[0], and I had to do a transaction to lock a message and then return it, but this makes it easier.
There's one little issue I keep finding with SQLite, and it's that most virtual servers / VM images ship with version 3.22.0, and upgrading often means building from source.
In any case, SQLite is absolutely wonderful. My favorite way of building products is having a folder for all the DBs that I mount to docker-compose. This release makes it even better.
[0] https://github.com/litements/litequeue
neoq
- Show HN: Hatchet – Open-source distributed task queue
-
Choose Postgres Queue Technology
I just want to commend OP - if they’re here - for choosing an int64 for job IDs, and MD5 for hashing the payload in Neoq, the job library linked [0] from the article.
Especially given the emphasis on YAGNI, you don’t need a UUID primary key, and all of its problems they bring for B+trees (that thing RDBMS is built on), nor do you need the collision resistance of SHA256 - the odds of you creating a dupe job hash with MD5 are vanishingly small.
As to the actual topic, it’s fine IFF you carefully monitor for accumulating dead tuples, and adjust auto-vacuum for that table as necessary. While not something you’d run into at the start, at a modest scale you may start to see issues. May. You may also opt to switch to Redis or something else before that point anyway.
[0]: https://github.com/acaloiaro/neoq
-
Ask HN: Tell us about your project that's not done yet but you want feedback on
Neoq (https://github.com/acaloiaro/neoq) is a background job processor for Go.
Yes, another one. It began from my desire to have a robust Postgres-backed job processor. What I quickly realized was that the interface in front of the queue was what was really important. This allowed me to add both in-memory and Redis (provided by asynq) backends behind the same interface. Which allows dependent projects to switch between different backends in different settings/durable requirements. E.g. in-memory for testing/development, postgres when you're not running Google-scale jobs, and Redis for all the obvious use cases for a Redis-backed queue.
This allows me to swap out job queue backends without changing a line of job processor code.
I'm familiar with the theory that one shouldn't implement queues on Postgres, and to a large extent, I disagree with those theories. I'm confident you can point out a scenario in which one shouldn't, and I contend that those scenarios are the exception rather than the rule.
-
Examples of using task scheduler with Go?
I created a background processor called Neoq (https://github.com/acaloiaro/neoq) that is likely to interest you.
-
SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a Postgres Queue
This is exactly the thesis behind neoq: https://github.com/acaloiaro/neoq
What are some alternatives?
datasette-dateutil - dateutil functions for Datasette
starqueue
pgjobq - Atomic low latency job queues running on Postgres
oban - 💎 Robust job processing in Elixir, backed by modern PostgreSQL and SQLite3
Bedrock - Rock solid distributed database specializing in active/active automatic failover and WAN replication
tembo - Monorepo for Tembo Operator, Tembo Stacks, and Tembo CLI
sqlite_modern_cpp - The C++14 wrapper around sqlite library
Asynq - Simple, reliable, and efficient distributed task queue in Go
litestream - Streaming replication for SQLite.
pgtt - PostgreSQL extension to create, manage and use Oracle-style Global Temporary Tables and the others RDBMS