Kafunk
Hangfire
Kafunk | Hangfire | |
---|---|---|
1 | 62 | |
159 | 9,038 | |
- | 0.8% | |
1.7 | 9.4 | |
- | 16 days ago | |
F# | C# | |
- | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Kafunk
-
Walmart is migrating the remaining F# code into Java
Performance.
Generally speaking, F# was actually very fast, and had nice concurrency support, but there were times that wasn't the case.
For example, in 2016 I was part of the initiative to rewrite the ad feed. We had to read in several Kafka topics, do some joining on our end, and emit to a separate Kafka topic. This isn't terribly hard to write, but we were dealing on the order of about ~100gb of data being pushed into memory. This is hardly "big data" stuff, but it's enough to highlight some issues.
Specifically, the built F# persistent map structure was simply too slow to get the performance we wanted. I really like that structure, it's really handy and nice, but I ended up having to make heavy use of the ConcurrentDictionary that was built into .NET. This wasn't that hard or anything, but it made me a little sad that I had to move to a mutable store to get the performance I needed.
There was also the fact that the `async` monad, while generally very good and useful, had bizarre bottlenecks that were hard to measure. It was difficult to know when the async task was actually started, and when you tried to measure performance bottlenecks you were really only measuring the scheduler, not the actual performance. This isn't really F#'s fault, this is an issue with any kind of cooperative scheduling system, but occasionally to get the performance we needed we'd have to move to lower level threads instead of the pretty monadic stuff. Microsoft eventually released the Task monad which generally performed a bit better.
There were other things here and there; the Kafka client libraries for .NET simply aren't as good as the Java ones. Jet actually open-sourced their own (https://github.com/jet/kafunk) which did make it a bit more functional and nice, but it had performance issues as well, so a lot of us ended up using Confluent.
There were little annoyances specific to F# as well; there's no real concept of a monad transformer, so if you wanted to do something like, for example, combine an Option and an Async into generalized syntax, you'd have to write your own wrapper monad thing, which wasn't that hard but was sort of ad hoc.
The general rule of thumb was that the first draft of software, we would try and keep as functional and pretty. If that was too slow, we allowed mutation but only within a function. If that was too slow, we'd allow global mutation but only with thread-safe stuff.
Hangfire
- Hangfire – Background Processing in .NET and .NET Core Applications
-
Deno Cron
Unpopular opinion incoming... What I see is yet another way that the backend JS world is finally achieving something .NET had years ago[0].
Node/Deno/Bun/etc. + npm sounds super straightforward at first glance (and it is at first). But I've thought for years that it's far easier to be productive as an organization on .NET in Visual Studio, since it's simpler to design, deliver, and maintain infrastructure.
[0] https://www.hangfire.io/
-
Boosting Productivity with HangFire: Streamlining Background Job Processing
you can read about it here HangFire Documentation
-
How do you save a file at the end of the day within a function that is only called at certain times?
I mostly work in .NET, and typically use Hangfire, but all languages has similar frameworks
-
What can I use as a simple message bus with persistence in .NET?
Its hard to tell what tool would be a best fit without more information, but I would suggest looking at Hangfire for background job processing: https://www.hangfire.io/
-
Event Bus + Job APIs
You might want to look at https://www.hangfire.io/. Their docs explain how to set up queues: https://docs.hangfire.io/en/latest/background-processing/configuring-queues.html
-
Background Job Scheduling in .NET using Hangfire
In this article we looked at how to use Hangfire to schedule background jobs in ASP.NET according to our requirements. In a follow up article, I will talk about using Hangfire with a Redis storage. To learn more about Hangfire, you can visit the official website.
-
BackgroundService in .Net Core
Easy to understand if you want to implement your own background service. If you want a more easy and complete tool you can use hangfire.
-
Is there anything like this in C#?
Try https://www.hangfire.io/
-
Help in creating a new Service
If, as you stated, you really need to use your own servers, that seems exactly like a job for Hangfire.
What are some alternatives?
NetMQ - A 100% native C# implementation of ZeroMQ for .NET
QuartzNet - Quartz Enterprise Scheduler .NET
RawRabbit - A modern .NET framework for communication over RabbitMq
RabbitMQ.NET - RabbitMQ .NET client for .NET Standard 2.0+ and .NET 4.6.2+
MassTransit - Distributed Application Framework for .NET
Rebus - :bus: Simple and lean service bus implementation for .NET
Coravel - Near-zero config .NET library that makes advanced application features like Task Scheduling, Caching, Queuing, Event Broadcasting, and more a breeze!
NServiceBus - Build, version, and monitor better microservices with the most powerful service platform for .NET
Kafka Client
Confluent's .NET Client for Apache KafkaTM - Confluent's Apache Kafka .NET client
FluentScheduler - Automated job scheduler with fluent interface for the .NET platform.