the-algorithm VS Apache Spark

> Wasn’t the tweet recommendation system “open sourced” as well? Does this guy know the difference between open source and “open source”?

What do you mean? There exists only one binding definition of open source

> https://opensource.org/osd

and either some product does satisfy it, or it doesn't. As far as I am aware

> https://github.com/twitter/the-algorithm

does satisfy the open source definition, so your sarcasm looks demagogical to me, but I am very willing to learn something new.

I'm not a Twitter user so if this is a dumb question I apologize.

This sounds like a pretty serious allegation. How do you know this is true? Is it in the source code?[1]

[1]: https://github.com/twitter/the-algorithm

There is already a bug report for this: https://github.com/twitter/the-algorithm/issues/1876

There is a way for Meta to fuck this up. The Algorithm is licensed under GPL, which is a copyleft license. That means any derivative works based on it must also be licensed and open sourced under GPL. If Meta doesn't do that, they may be on the hook.

And I believe the source for that was effectively opened up to the world: https://github.com/twitter/the-algorithm

Recently I had to revisit the "JVM languages universe" again. Yes, language(s), plural! Java isn't the only language that uses the JVM. I previously used Scala, which is a JVM language, to use Apache Spark for Data Engineering workloads, but this is for another post 😉.

Consume data into third party software (then let Open Search or Apache Spark or Apache Pinot) for analysis/datascience, GIS systems (so you can put reports on a map) or any ticket management system

also, this knowledge applies to learning more about data engineering, as this field of software engineering relies heavily on the event-driven approach via tools like Spark, Flink, Kafka, etc.

Apache SeaTunnel is a data integration platform that offers the three pillars of data pipelines: sources, transforms, and sinks. It offers an abstract API over three possible engines: the Zeta engine from SeaTunnel or a wrapper around Apache Spark or Apache Flink. Be careful, as each engine comes with its own set of features.

Apache Spark - https://spark.apache.org/

A JVM based framework named "Spark", when https://spark.apache.org exists?

I recently saw a pull request that was merged to the Apache/Spark repository that apparently adds initial Python bindings for PySpark on K8s. I posted a comment to the PR asking a question about how to use spark-on-k8s in a Python Jupyter notebook, and was told to ask my question here.