pg-boss
arniesmtpbufferserver | pg-boss | |
---|---|---|
6 | 12 | |
13 | 1,638 | |
- | - | |
2.4 | 4.4 | |
7 months ago | about 1 month ago | |
Python | JavaScript | |
MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
arniesmtpbufferserver
- Arnie – SMTP buffer server in – 100 lines of async Python
-
Choose Postgres Queue Technology
My guess is that many people are implementing queuing mechanisms just for sending email.
The Linux file system makes a perfectly good basis for a message queue since file moves are atomic.
You can see how this works in Arnie SMTP buffer server, a super simple queue just for emails, no database at all, just the file system.
https://github.com/bootrino/arniesmtpbufferserver
-
Things Unix can do atomically (2010)
A practical applications of atomic mv is building simple file based queuing mechanisms.
For example I wrote this SMTP buffer server which moves things to different directories as a simple form of message queue.
https://github.com/bootrino/arniesmtpbufferserver
Caveat I think this needs examination from the perspective of fsync - i.e. I suspect the code should be fsyncing at certain points but not sure.
I actually wrote (in Rust) a simple file based message queue using atomic mv. It instantly maxed out the SSD performance at about 30,000 messages/second.
-
Procrastinate: PostgreSQL-Based Task Queue for Python
Yeah I was using Celery for sending emails - nothing else.
And it was such a nightmare to configure and debug and such overkill for email buffering that in a fit of frustration I wrote the Arnie SMTP buffering server and ditched Celery.
https://github.com/bootrino/arniesmtpbufferserver
It's only 100 lines of code:
https://github.com/bootrino/arniesmtpbufferserver/blob/maste...
-
Show HN: Arnie SMTP buffer server in 100 lines of async Python
Here's the 100 lines of code:
https://github.com/bootrino/arniesmtpbufferserver/blob/master/arniesmtpbufferserver.py
Here's the github repo:
https://github.com/bootrino/arniesmtpbufferserver
It's MIT licensed.
Arnie is a server that has the single purpose of buffering outbound SMTP emails.
A typical web SAAS needs to send emails such as signup/signin/forgot password etc.
The web page code itself should not directly write this to an SMTP server. Instead they should be decoupled. There's a few reasons for this. One is, if there is an error in sending the email, then the whole thing simply falls over if that send was executed by the web page code - there's no chance to resend because the web request has completed. Also, execution of an SMTP request by a web page slows the response time down of that page, whilst the code goes through the process of connecting to the server and sending the email. So when you send SMTP email from your web application, the most performant and safest way to do it is to buffer them for sending. The buffering server will then queue them and send them and handle things like retries if the target SMTP server is down or throttled.
There's a few ways to solve this problem - you can set up a local email server and configure it for relaying. Or in the Python world people often use Celery. Complexity is the down side of using either Celery or an email server configured for relaying - both of these solutions have many more features than needed and can be complex to configure/run/troubleshoot.
Arnie is intended for small scale usage - for example a typical web server for a simple SAAS application. Large scale email traffic would require parallel sends to the SMTP server.
Arnie sequentially sends emails - it does not attempt to send email to the SMTP server in parallel. It probably could do fairly easily by spawning email send tasks, but SMTP parallelisation was not the goal in writing Arnie.
- Arnie - SMTP buffer server in ~ 100 lines of async Python
pg-boss
-
Choose Postgres Queue Technology
For running queues on Postgres with Node.js backend(s), I highly recommend https://github.com/timgit/pg-boss. I'm sure it has it scale limits. But if you're one of the 90% of the apps that never needs any kind of scale that a modern server can't easily handle then it's fantastic. You get transactional queueing of jobs, and it automatically handles syncing across multiple job processing servers using Postgres locks.
-
Build Your Own Personal Twitter Agent 🧠🐦⛓ with LangChain
Jobs use pg-boss, a postgres extension, to queue and run tasks under the hood.
-
SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a Postgres Queue
If you don't want to roll your own, look into https://github.com/timgit/pg-boss
- How/do you handle queue type workflows?
-
Which tool/library well adopted to use Postgres as a message broker?
I saw this https://github.com/timgit/pg-boss but it's more for jobs than for message with multiple consumers (having their own progress offset).
-
How to schedule tasks in a Node.js app 🕙
The best I've used till now. Has all kind of features and really great when you have a postgres dB in your stack. https://github.com/timgit/pg-boss
-
Cluster friendly task scheduler for NodeJS
Check out these; - https://github.com/mitranim/posterus - https://github.com/timgit/pg-boss - https://github.com/FirebaseExtended/firebase-queue - https://www.npmjs.com/package/rabbit-queue
-
You don't need distributed systems.
You can use the simplest option than implement a new service. Keep in mind that every running system can be a job scheduler, you can just use nodejs worker threads, Redis, or even your DB as a job scheduler, check PGBoss for example.
-
Launch HN: Convoy (YC W22) – Open-source cloud-native webhooks service
Both! For context, we're currently using https://github.com/timgit/pg-boss as a task queue on top of postgres and it works great. No need to complicate things with Redis. I believe it's quite straightforward to implement a task queue on top of postgres using the SKIP LOCKED functionality.
- Devious SQL: Message Queuing Using Native PostgreSQL
What are some alternatives?
starqueue
worker - High performance Node.js/PostgreSQL job queue (also suitable for getting jobs generated by PostgreSQL triggers/functions out into a different work queue)
kubeblocks - KubeBlocks is an open-source control plane that runs and manages databases, message queues and other data infrastructure on K8s.
celery - Distributed Task Queue (development branch)
pgjobq - Atomic low latency job queues running on Postgres
django-postgres-queue - A task queue for django
RabbitMQ - Open source RabbitMQ: core server and tier 1 (built-in) plugins
Redis - Redis is an in-memory database that persists on disk. The data model is key-value, but many different kind of values are supported: Strings, Lists, Sets, Sorted Sets, Hashes, Streams, HyperLogLogs, Bitmaps.
kue - Kue is a priority job queue backed by redis, built for node.js.
Chronicle Queue - Micro second messaging that stores everything to disk
Apache Kafka - Mirror of Apache Kafka
BeanstalkD - Beanstalk is a simple, fast work queue.