Sidekiq VS Apache Kafka

I'm not sure I personally agree with this, and I'm not 100% sure the developer community at-large does either...

Let's take a few examples, which I've shared elsewhere in similar discussions:

- GitLab: Open Source or Open Core? Most would say Open Source, but (I assume) you would argue Open Core [0].

- Plausible: Open Source or Open Core? They say Open Source, but it's actually Open Core [1].

- Cal.com: Open Source or Open Core? They say Open Source, but once again, Open Core [2].

- Posthog: Open Source or Open Core? They say Open Source, still Open Core [3].

- Sidekiq: Open Source or Open Core? Once again: Open Core [4].

Yet, every dev I know would consider these projects Open Source. So there's a disconnect somewhere.

Under this mindset, very few open source startups are actually open source, yet everybody says they are?

I'm not trying to argue either way; I'm trying to point out a disconnect here.

[0]: https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/LICENS...

[1]: https://github.com/plausible/analytics/blob/2dd2f058d1dcae6f...

[2]: https://github.com/calcom/cal.com/blob/main/packages/feature...

[3]: https://github.com/PostHog/posthog/blob/master/ee/LICENSE

[4]: https://github.com/sidekiq/sidekiq/blob/main/COMM-LICENSE.tx...

Okay, back to our Rails app. In our app, the lifetime should be the whole Sidekiq process. Luckily, Sidekiq has internal documentation on how it runs. I won't copy the entire documentation here, just the part we are interested in:

Rollbar is a great error-tracking service. It alerts us on exceptions and errors, provides analysis tools and dashboard, so we can see, reproduce, and fix bugs quickly when something went wrong. This service has a possibility to log not only uncaught exceptions but any messages. By default, the messages are reported synchronously, but you can enable asynchronous reporting using Sidekiq, girl_friday, or Resque. Also, you can provide your own handler and a failover handler to be confident, that your error is tracked and delivered in the case of primary handler’s fail.

Hi there! I want to show off a little feature I made using hanami, htmx and a little bit of redis + sidekiq.

I'd say Sidekiq is the top competitor here.

There's something wrong at Redislabs, it took them over a year to get RESP3 rolled out into their hosted service, you'd expect a rollout of that to be a bit quicker when they're the owner of Redis.

It affected us when upgrading Sidekiq to version 7, which dropped support for older Redis, and their Envoy proxy setup didn't support HELLO and RESP3: https://github.com/sidekiq/sidekiq/issues/5594

That depends on how the `maxmemory-policy` is configured, and queue systems based on Redis will tell you not to allow eviction. https://github.com/sidekiq/sidekiq/wiki/Using-Redis#memory (it even logs a warnings if it detects your Redis is misconfigured IIRC).

Sidekiq https://sidekiq.org/: This one started as an open source project, once it got enough traction, the developer made a premium version of it, and makes money by selling licenses to businesses.

Sidekiq will drop in-progress jobs when a worker crashes. Sidekiq Pro can recover those jobs but with a large delay. Sidekiq is excellent overall but it’s not suitable for processing critical jobs with a low latency guarantee.

https://github.com/sidekiq/sidekiq/wiki/Reliability

> I'm not sure feature withholding has traditionally worked out well in the developer space.

I think it's worked out well for Sidekiq (https://sidekiq.org). I really like their model of layering valuable features between the OSS / Pro / Enterprise licenses.

Apache Kafka — a distributed event streaming platform implementing a variant of the Raft consensus protocol (written in Java, integrated with Scala);

Spring does this during autoconfiguration. For example this and this. When the user adds a configuration then it gets to overwrite the default from the template. I am looking for something similar, perhaps simpler approach.

Source and documentation on GitHub

Code is still in trunk though. https://github.com/apache/kafka/tree/trunk/connect/file/src/main/java/org/apache/kafka/connect/file