mdoc VS Apache Spark

Literally what scaladoc is, it comes with sbt. Although, it's better when enhanced with mdoc so that you get the standard microsite template like these. It would be nice to have an sbt serveDocs and if everyone would host their docs for external linking, but javadoc doesn't do that either.

The good news is that scaladoc is produced by default by sbt and published by default. So you can often pull it from the same repository your library jar came from, extract it with zip, and read the docs. But that's also totally unnecessary - javadoc.io allows you to put in your module info and serves the docs for you, so if there's an older version you can access the documentation this way. Rely on the type signatures, since they can't lie, whilst comments (including scaladoc comments) can. Honestly, library authors should be using mdoc and including examples on every public method, and that type of documentation is something you can almost always contribute to a project for a quick pr kudos.

I know it's not new but the "Snippet validation and results (mdoc)" features in mdoc are so cool. Really takes some of the tedium out of working with documentation since you can know that as you evolve your code the compiler will make sure you keep the docs in sync. Whole new level of Readme-Driven Development

Recently I had to revisit the "JVM languages universe" again. Yes, language(s), plural! Java isn't the only language that uses the JVM. I previously used Scala, which is a JVM language, to use Apache Spark for Data Engineering workloads, but this is for another post 😉.

Consume data into third party software (then let Open Search or Apache Spark or Apache Pinot) for analysis/datascience, GIS systems (so you can put reports on a map) or any ticket management system

also, this knowledge applies to learning more about data engineering, as this field of software engineering relies heavily on the event-driven approach via tools like Spark, Flink, Kafka, etc.

Apache SeaTunnel is a data integration platform that offers the three pillars of data pipelines: sources, transforms, and sinks. It offers an abstract API over three possible engines: the Zeta engine from SeaTunnel or a wrapper around Apache Spark or Apache Flink. Be careful, as each engine comes with its own set of features.

Apache Spark - https://spark.apache.org/

A JVM based framework named "Spark", when https://spark.apache.org exists?

I recently saw a pull request that was merged to the Apache/Spark repository that apparently adds initial Python bindings for PySpark on K8s. I posted a comment to the PR asking a question about how to use spark-on-k8s in a Python Jupyter notebook, and was told to ask my question here.