bacalhau VS Hangfire

This is really interesting - we’ve tried really hard to solve some of these with Bacalhau[1] - a much simpler distributed compute platform. Would love your feedback!

[1] https://github.com/bacalhau-project/bacalhau

Disclosure: I confounded Bacalhau

It's an interesting question!

The problem is that the data is spread everywhere - no choice about that. So with that in mind, how do you query that data? Today, the idea is that you HAVE to put it into a central location. With tools like Bacalhau[1] and DuckDB [2], you no longer have to - a single query can be sharded amongst all your data - EFFECTIVELY giving you a lot of what you want from a data lake.

It's not a replacement, but if you can do a few of these items WITHOUT moving the data, you will be able to see really significant cost and time savings.

[1] https://github.com/bacalhau-project/bacalhau

[2] https://github.com/duckdb/duckdb

Very selfishly - distributed compute. Not decentralized, distributed.

Compute and data are being created and run everywhere, we need platforms that understand how to use it and get insights without (or before) moving it.

Our contribution: https://github.com/bacalhau-project/bacalhau (think Kubernetes but built for the distributed world).

Disclosure: I co-founded the Bacalhau Project

Thank you so much for the feedback. Yeah, we REALLY do want to figure out a better naming/reference scheme. Do you have anything you've seen you really like?

Disclosure: I work on Bacalhau[1]

https://github.com/bacalhau-project/bacalhau

Unpopular opinion incoming... What I see is yet another way that the backend JS world is finally achieving something .NET had years ago[0].

Node/Deno/Bun/etc. + npm sounds super straightforward at first glance (and it is at first). But I've thought for years that it's far easier to be productive as an organization on .NET in Visual Studio, since it's simpler to design, deliver, and maintain infrastructure.

[0] https://www.hangfire.io/

you can read about it here HangFire Documentation

I mostly work in .NET, and typically use Hangfire, but all languages has similar frameworks

Its hard to tell what tool would be a best fit without more information, but I would suggest looking at Hangfire for background job processing: https://www.hangfire.io/

You might want to look at https://www.hangfire.io/. Their docs explain how to set up queues: https://docs.hangfire.io/en/latest/background-processing/configuring-queues.html

In this article we looked at how to use Hangfire to schedule background jobs in ASP.NET according to our requirements. In a follow up article, I will talk about using Hangfire with a Redis storage. To learn more about Hangfire, you can visit the official website.

Easy to understand if you want to implement your own background service. If you want a more easy and complete tool you can use hangfire.

Try https://www.hangfire.io/

If, as you stated, you really need to use your own servers, that seems exactly like a job for Hangfire.