tech.ml.dataset VS geni

Tablecloth by generateme is a friendly & expressive table-processing library built on top of tech.ml.dataset & dtype-next, Chris Nuernberger's high-performance data libraries.

For 1: This ns of tech.ml.dataset supports reading of multiple worksheets per file https://github.com/techascent/tech.ml.dataset/blob/master/src/tech/v3/libs/fastexcel.clj

After building tech.ml.dataset and charred I wanted to take the lessons learned there and apply them back into the base Clojure substrate of persistent maps, persistent vectors, and algorithmic primitives like group-by and frequencies.

Did you try tech.ml.dataset?

Checkout babashka for scritping and clj-python to use numpy from clojure, or https://github.com/techascent/tech.ml.dataset for pure clojure

Really high quality libraries for deep learning, dataset manipulation, and more

tech.ml.dataset has seen some major upgrades for discoverability - specifically the tech.ml.dataset main namespace has been revamped. If you use Cursive or Calva your intellisense will now work with the main namespaces.

It is based on a state-of-the art , high performance tabular dataset implementation, tech.ml.dataset and combines it with a innovative pipeline approach build with idiomatic and functional Clojure concepts in mind. Machine Learning models get pulled in as plugins from existing ecosystems, so are available from the start. Please find user guides and example code in GitHub at https://github.com/scicloj/scicloj.ml

sparkling is fine. there is also geni

I haven't tried Spark from Kotlin, but it's a nice experience working with it in Clojure, and I have yet to see a language more expressive than Clojure. :)

I think for the large scale stuff, wrappers like geni are pretty nice and built on top of established tech. There were several distributed computing platforms like onyx and storm that popped up in clojure as well that may be interesting to look at. clojure toolbox has a good index of libraries to examine.