Apache Spark VS missinglink

Recently I had to revisit the "JVM languages universe" again. Yes, language(s), plural! Java isn't the only language that uses the JVM. I previously used Scala, which is a JVM language, to use Apache Spark for Data Engineering workloads, but this is for another post 😉.

Consume data into third party software (then let Open Search or Apache Spark or Apache Pinot) for analysis/datascience, GIS systems (so you can put reports on a map) or any ticket management system

also, this knowledge applies to learning more about data engineering, as this field of software engineering relies heavily on the event-driven approach via tools like Spark, Flink, Kafka, etc.

Apache SeaTunnel is a data integration platform that offers the three pillars of data pipelines: sources, transforms, and sinks. It offers an abstract API over three possible engines: the Zeta engine from SeaTunnel or a wrapper around Apache Spark or Apache Flink. Be careful, as each engine comes with its own set of features.

Apache Spark - https://spark.apache.org/

A JVM based framework named "Spark", when https://spark.apache.org exists?

I recently saw a pull request that was merged to the Apache/Spark repository that apparently adds initial Python bindings for PySpark on K8s. I posted a comment to the PR asking a question about how to use spark-on-k8s in a Python Jupyter notebook, and was told to ask my question here.

You can also use the upstream https://github.com/spotify/missinglink if you're using Maven instead of sbt.

There's been a recent release of missing-link with support for Mult-Release JARs and running analysis on multiple projects concurrently. More info here (applies not only to Scala/sbt, so it's worth linking it here): https://old.reddit.com/r/scala/comments/lxmi4w/sbtmissinglink_032_has_been_released_multirelease/

There are still some issues with MissingLink, like that it can't handle Multi-Release JARs (but that's being fixed I write) or that it uses thread-unsafe caches and so it can't be executed in parallel (and so it can take a long time to finish on big projects with many modules). But taken all together I swear by it. It has already saved us from runtime failures after deployment many times (it's easy to get incompatible versions when working on big projects with big number of dependencies). Instead, we get a red build in CI -- just anybody should expect for goodness sake when using a strongly-typed language like Scala.