Deeplearning4j VS Apache Spark

Have you considered migrating off of Python to just using JVM ML libraries then? I hear good things about Deeplearning4j, but there's quite a few.

I've gone to the linux workflow as directed in the docs and reconstructed the maven command line:

DL4J

Hey folks, just letting you know we see this thread and I appreciate you guys running these benchmarks. I'm not seeing any of your posts on our forums. I think I saw a notification from our examples but we do not actually monitor that. Please use: https://community.konduit.ai/ or at least the main repo dl4j issues: https://github.com/eclipse/deeplearning4j/issues and you'll get a lot more visibility. Thanks!

Also, the website is now redirected to: https://deeplearning4j.konduit.ai/

On top of this several popular projects have been built. This includes tensorflow-java and our project eclipse deeplearning4j: https://github.com/eclipse/deeplearning4j

Nd4j is actively developed. The latest commit was 6 hours ago. Nd4j is part of deeplearning4j which is now owned by eclipse (but the main contributors are from a company) https://github.com/eclipse/deeplearning4j/tree/master/nd4j

Recently I had to revisit the "JVM languages universe" again. Yes, language(s), plural! Java isn't the only language that uses the JVM. I previously used Scala, which is a JVM language, to use Apache Spark for Data Engineering workloads, but this is for another post 😉.

Consume data into third party software (then let Open Search or Apache Spark or Apache Pinot) for analysis/datascience, GIS systems (so you can put reports on a map) or any ticket management system

also, this knowledge applies to learning more about data engineering, as this field of software engineering relies heavily on the event-driven approach via tools like Spark, Flink, Kafka, etc.

Apache SeaTunnel is a data integration platform that offers the three pillars of data pipelines: sources, transforms, and sinks. It offers an abstract API over three possible engines: the Zeta engine from SeaTunnel or a wrapper around Apache Spark or Apache Flink. Be careful, as each engine comes with its own set of features.

Apache Spark - https://spark.apache.org/

A JVM based framework named "Spark", when https://spark.apache.org exists?

I recently saw a pull request that was merged to the Apache/Spark repository that apparently adds initial Python bindings for PySpark on K8s. I posted a comment to the PR asking a question about how to use spark-on-k8s in a Python Jupyter notebook, and was told to ask my question here.