Sidekiq VS celery

I'm not sure I personally agree with this, and I'm not 100% sure the developer community at-large does either...

Let's take a few examples, which I've shared elsewhere in similar discussions:

- GitLab: Open Source or Open Core? Most would say Open Source, but (I assume) you would argue Open Core [0].

- Plausible: Open Source or Open Core? They say Open Source, but it's actually Open Core [1].

- Cal.com: Open Source or Open Core? They say Open Source, but once again, Open Core [2].

- Posthog: Open Source or Open Core? They say Open Source, still Open Core [3].

- Sidekiq: Open Source or Open Core? Once again: Open Core [4].

Yet, every dev I know would consider these projects Open Source. So there's a disconnect somewhere.

Under this mindset, very few open source startups are actually open source, yet everybody says they are?

I'm not trying to argue either way; I'm trying to point out a disconnect here.

[0]: https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/LICENS...

[1]: https://github.com/plausible/analytics/blob/2dd2f058d1dcae6f...

[2]: https://github.com/calcom/cal.com/blob/main/packages/feature...

[3]: https://github.com/PostHog/posthog/blob/master/ee/LICENSE

[4]: https://github.com/sidekiq/sidekiq/blob/main/COMM-LICENSE.tx...

Okay, back to our Rails app. In our app, the lifetime should be the whole Sidekiq process. Luckily, Sidekiq has internal documentation on how it runs. I won't copy the entire documentation here, just the part we are interested in:

Rollbar is a great error-tracking service. It alerts us on exceptions and errors, provides analysis tools and dashboard, so we can see, reproduce, and fix bugs quickly when something went wrong. This service has a possibility to log not only uncaught exceptions but any messages. By default, the messages are reported synchronously, but you can enable asynchronous reporting using Sidekiq, girl_friday, or Resque. Also, you can provide your own handler and a failover handler to be confident, that your error is tracked and delivered in the case of primary handler’s fail.

Hi there! I want to show off a little feature I made using hanami, htmx and a little bit of redis + sidekiq.

I'd say Sidekiq is the top competitor here.

There's something wrong at Redislabs, it took them over a year to get RESP3 rolled out into their hosted service, you'd expect a rollout of that to be a bit quicker when they're the owner of Redis.

It affected us when upgrading Sidekiq to version 7, which dropped support for older Redis, and their Envoy proxy setup didn't support HELLO and RESP3: https://github.com/sidekiq/sidekiq/issues/5594

That depends on how the `maxmemory-policy` is configured, and queue systems based on Redis will tell you not to allow eviction. https://github.com/sidekiq/sidekiq/wiki/Using-Redis#memory (it even logs a warnings if it detects your Redis is misconfigured IIRC).

Sidekiq https://sidekiq.org/: This one started as an open source project, once it got enough traction, the developer made a premium version of it, and makes money by selling licenses to businesses.

Sidekiq will drop in-progress jobs when a worker crashes. Sidekiq Pro can recover those jobs but with a large delay. Sidekiq is excellent overall but it’s not suitable for processing critical jobs with a low latency guarantee.

https://github.com/sidekiq/sidekiq/wiki/Reliability

> I'm not sure feature withholding has traditionally worked out well in the developer space.

I think it's worked out well for Sidekiq (https://sidekiq.org). I really like their model of layering valuable features between the OSS / Pro / Enterprise licenses.

There are some limitations when using Quorum Queues, this was only patched recently - https://github.com/celery/celery/issues/9149

Celery previously had an --autoreload flag that has since been removed. However, Django has automatic reloading built into its manage.py runserver command. The absence of automatic reloading in Celery workers creates a confusing development experience: updating Python code causes the Django server to reload with the current code, but any tasks that the server fires will run stale code in the Celery worker.

From what I've seen, there are sort of two paths. I'll provide a well known example from each.

1. lang specific distributed task library

For example, in Python, celery is a pretty popular task system. If you (the dev) are the one doing all the code and running the workflows, it might work well for you. You build the core code and functions, and it handles the processing and resource stuff with a little config.

* https://github.com/celery/celery

Or lower level:

* https://github.com/dask/dask

2. DAG Workflow systems

There are also whole systems for what you're describing. They've gotten especially popular in the ML ops and data engineering world. A common one is AirFlow:

* https://github.com/apache/airflow

https://github.com/celery/celery/issues/7674#issuecomment-12... .. https://news.ycombinator.com/item?id=40842365

https://docs.celeryq.dev/en/stable/getting-started/backends-...

Celery's flower utility doesn't work with SQS or GCP's {Cloud Tasks, Cloud Pub/Sub, Firebase Cloud Messaging FWIU} but does work with AMQP, which is a reliable messaging protocol.

RabbitMQ is backed by mnesia, an Erlang/OTP library for distributed Durable data storage. Mnesia: https://en.wikipedia.org/wiki/Mnesia

SQLite is written in C and has lots of tests because aerospace IIUC.

There are many extensions of SQLite; rqlite, cr-sqlite, postlite, electricsql, sqledge, and also WASM: sqlite-wasm, sqlite-wasm-http

celery/kombu > Transport brokers support / comparison table: https://github.com/celery/kombu?tab=readme-ov-file#transport...

Kombu has supported Apache Kafka since 2022, but celery doesn't yet support Kafka: https://github.com/celery/celery/issues/7674#issuecomment-12...

So this might be my understanding, but stuff like celery is more like an orchestrator that chunks up workloads (think Hadoop with multiple nodes).

In the Django world, you'd probably rely on Celery to do this for you. You're probably looking for something similar that works with Go. https://github.com/celery/celery