arniesmtpbufferserver VS windmill

My guess is that many people are implementing queuing mechanisms just for sending email.

The Linux file system makes a perfectly good basis for a message queue since file moves are atomic.

You can see how this works in Arnie SMTP buffer server, a super simple queue just for emails, no database at all, just the file system.

https://github.com/bootrino/arniesmtpbufferserver

A practical applications of atomic mv is building simple file based queuing mechanisms.

For example I wrote this SMTP buffer server which moves things to different directories as a simple form of message queue.

https://github.com/bootrino/arniesmtpbufferserver

Caveat I think this needs examination from the perspective of fsync - i.e. I suspect the code should be fsyncing at certain points but not sure.

I actually wrote (in Rust) a simple file based message queue using atomic mv. It instantly maxed out the SSD performance at about 30,000 messages/second.

Yeah I was using Celery for sending emails - nothing else.

And it was such a nightmare to configure and debug and such overkill for email buffering that in a fit of frustration I wrote the Arnie SMTP buffering server and ditched Celery.

https://github.com/bootrino/arniesmtpbufferserver

It's only 100 lines of code:

https://github.com/bootrino/arniesmtpbufferserver/blob/maste...

Here's the 100 lines of code:

https://github.com/bootrino/arniesmtpbufferserver/blob/master/arniesmtpbufferserver.py

Here's the github repo:

https://github.com/bootrino/arniesmtpbufferserver

It's MIT licensed.

Arnie is a server that has the single purpose of buffering outbound SMTP emails.

A typical web SAAS needs to send emails such as signup/signin/forgot password etc.

The web page code itself should not directly write this to an SMTP server. Instead they should be decoupled. There's a few reasons for this. One is, if there is an error in sending the email, then the whole thing simply falls over if that send was executed by the web page code - there's no chance to resend because the web request has completed. Also, execution of an SMTP request by a web page slows the response time down of that page, whilst the code goes through the process of connecting to the server and sending the email. So when you send SMTP email from your web application, the most performant and safest way to do it is to buffer them for sending. The buffering server will then queue them and send them and handle things like retries if the target SMTP server is down or throttled.

There's a few ways to solve this problem - you can set up a local email server and configure it for relaying. Or in the Python world people often use Celery. Complexity is the down side of using either Celery or an email server configured for relaying - both of these solutions have many more features than needed and can be complex to configure/run/troubleshoot.

Arnie is intended for small scale usage - for example a typical web server for a simple SAAS application. Large scale email traffic would require parallel sends to the SMTP server.

Arnie sequentially sends emails - it does not attempt to send email to the SMTP server in parallel. It probably could do fairly easily by spawning email send tasks, but SMTP parallelisation was not the goal in writing Arnie.

Data Handling: Utilizes Windmill for data pipelines, with a primary database powered by PostgreSQL. Auxiliary data storage is handled by MongoDB, with Redis for caching to optimize performance

Look very similar to the script builder portion of https://github.com/windmill-labs/windmill, but not open-source, not self-hostable, and without open-source integrations (https://hub.windmill.dev/)

disclaimer: I'm founder of ^

If you need a job queue on Postgres, https://windmill.dev provide an all-integrated developer platform with a Pg queue at its core that support jobs defined in python/typescript/sql

windmill.dev - Windmill is an open-source developer platform to quickly build production-grade multi-step automation and internal apps from minimal Python and Typescript scripts. As a free user, you can create and be a member of at most three non-premium workspaces.

For an alternative to airplane.dev, you can checkout Windmill.

https://github.com/windmill-labs/windmill

"Open-source developer infrastructure for internal tools (APIs, background jobs, workflows and UIs). Self-hostable alternative to Airplane, Pipedream, Superblocks and a simplified Temporal with autogenerated UIsm and custom UIs to trigger workflows and scripts as internal apps.

Scripts are turned into sharable UIs automatically, and can be composed together into flows or used into richer apps built with low-code. Supported script languages supported are: Python, TypeScript, Go, Bash, SQL, and GraphQL. "

If you search HN, you'll find the creator of Windmill comment on comparisons to airplane.dev:

https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

https://windmill.dev is a self-hostable OSS alternative to pipedream

(disclaimer: I'm founder)

I'm genuinely curious what server-side stuff on BC you are referring to. That may have been something added after our assessment. The way I'd generally approach something like that for any of the platforms would be using an external low/no code solution to process webhook data. But it would depend heavily on the use case. For a more developer friendly option I've been really impressed by windmill.dev. We use a mix of n8n and windmill for various needs.

Yes it goes in that direction, however note that you can already do this in a not too hard way.

Our openflow spec is both open-source and has a full openapi definition: https://github.com/windmill-labs/windmill/blob/main/openflow...

you can use that to generate client sdks in any languages and build your own dag with it. That's what one of our customer did building a reactflow to openflow library: https://github.com/Devessier/reactflow-to-windmill

It's not as good as the decorator way but we move fast and if you still have interest for it we could prioritize it (and ask for feedbacks :))