Akka.net VS spark

akka.net actors. Actors all the way! https://getakka.net

If it's the former you may want to take a look at something like the actor model akka.net with persistent actors (https://getakka.net/articles/persistence/architecture.html). No need of an external message broker or mass transit (which is a wrapper over different message brokers). You could use sqllite for persisting the actors state to recover in case of a restart.

Spring and Hibernate are the goto libraries in Java land and I suspect that's the primary motivation for your colleague's recommendations. It's quite easy to bulldoze someone less experienced with your ideas so be careful of that. I'd avoid both. They aren't bad libraries at all but they have a 'legacy' feel and it will make your application less future proof. Would a distributed system be viable? If so then I'd recommend Akka, there'a .NET port of it that's well supported and maintained.

Is akka a good alternative?

However, I've found that sometimes, they are a little -too- functional. I'm a bit more preferential to Akka.Net's implementation of Option and Try, if only because they have good 'escape hatches' where you interrogate them in a more procedural manner.

This project is also what ultimately led to the creation of Akka.NET - I wrote an overview on how our application was built here: https://aaronstannard.com/markedup-akkadotnet/

You may also want to take a look at queues (e.g. RabbitMQ) or even something like Akka.NET or Microsoft Orleans.

From the title I immediately thought AKKA.NET or Orleans

Re: DataFrames, that's good to know. There is the DataFrame API which is part of the Microsoft.Data.Analysis NuGet package and that's the API that the issue is tracking and shown in the sample notebook I shared. That API has no dependencies on other systems. The DataFrame you're referring to is part of the .NET for Apache Spark library which has the dependency on Apache Spark which rqeuires some initial setup.

the data engineering ecosystem is new to me but my first impressions are that everything is catered toward JVM. The only somewhat promising option I've found for building a data pipeline in .NET is github.com/dotnet/spark.