Apache Spark VS Trino

Apache Iceberg defines a table format that separates how data is stored from how data is queried. Any engine that implements the Iceberg integration — Spark, Flink, Trino, DuckDB, Snowflake, RisingWave — can read and/or write Iceberg data directly.

Apache Spark powers large-scale data analytics and machine learning, but as workloads grow exponentially, traditional static resource allocation leads to 30–50% resource waste due to idle Executors and suboptimal instance selection.

One of the key attributes of Apache License 2.0 is its flexible nature. Permitting use in both proprietary and open source environments, it has become the go-to choice for innovative projects ranging from the Apache HTTP Server to large-scale initiatives like Apache Spark and Hadoop. This flexibility is not solely legal; it is also philosophical. The license is designed to encourage transparency and maintain a healthy balance between freedom and accountability, ultimately making it easier for developers to adapt and contribute without restrictive legal barriers. Another modern twist discussed in the article is the concept of dual licensing. Dual licensing can offer an attractive method for additional commercial exploitation while still upholding open source principles. However, as the article cautions, dual licensing involves legal intricacy and demands rigor in managing Contributor License Agreements (CLAs), a challenge that the open source community navigates with ongoing debates. For developers looking to understand similar innovative approaches to licensing, further information can be explored at License Token.

[1] S. Russell and P. Norvig, Artificial Intelligence: A Modern Approach. Pearson, 2020. [2] F. Chollet, Deep Learning with Python. Manning Publications, 2018. [3] C. C. Aggarwal, Data Mining: The Textbook. Springer, 2015. [4] J. Dean and S. Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters," Communications of the ACM, vol. 51, no. 1, pp. 107-113, 2008. [5] Apache Software Foundation, "Apache Spark: Lightning-Fast Unified Analytics Engine," Available: https://spark.apache.org/. [6] Java Community Process, "Java Machine Learning Libraries and Frameworks," Available: https://www.oracle.com/java/.

Apache Spark isn’t just a framework for distributed data processing; it’s a rich ecosystem that includes libraries for machine learning, stream processing, and graph processing. A key aspect of Spark’s ecosystem is its reliance on community contributions. Developers from around the world collaborate on its GitHub repository, ensuring that Spark remains at the cutting edge of technology. The governance process, characterized by transparency and meritocracy, builds trust among contributors and sponsors alike. An essential component of Apache Spark’s model is its use of the Apache 2.0 license. This permissive license not only shields contributors with patent protection but also allows enterprises to integrate Spark into proprietary systems without legal hurdles. The license enables a free flow of innovation—companies can both use and contribute to Spark’s codebase, leading to enhancements that benefit the entire community. The funding mechanisms sustaining Apache Spark are as diverse as they are innovative. Corporate sponsorships play a significant role, with companies dedicating resources and finances to support ongoing development. Additionally, grant programs and community donations help maintain an ecosystem where improvements and new features are continuously shared with users worldwide. These sustainable funding practices ensure that Apache Spark can meet the demands of real-time analytics and high-volume data processing.

If you're designing an event-based pipeline, you can use a data streaming tool like Kafka to process data as it's collected by the pipeline. For a setup that already has data stored, you can use tools like Apache Spark to batch process and clean it before moving ahead with the pipeline.

PySpark is the Python API for Apache Spark, an open-source distributed computing system that enables fast, scalable data processing. PySpark allows Python developers to leverage the powerful capabilities of Spark for big data analytics, machine learning, and data engineering tasks without needing to delve into the complexities of Java or Scala.

Traditional databases — PostgreSQL, MySQL, etc. — store their data in proprietary formats. That format is optimized for that engine and can’t be directly accessed by anything else. Even if something like Trino can connect to Postgres, it’s still running queries through Postgres itself, not reading its storage directly. You’re just a client.

Trino: Trino (formerly known as PrestoSQL) is a high-performance distributed SQL query engine designed for data analysis. It offers efficient querying capabilities across multiple data sources, including various file formats, databases, and data lakes. These are some interesting background story between Trino and Presto: Presto was the original name of the project, and it was developed by Facebook. In December 2020, a significant portion of the Presto community decided to fork the project and renamed it Trino. Read more here: Trino Blog.

However, Iceberg defines the storage format, leaving the complexities of data ingestion and processing, especially for real-time streams, to separate systems. While query engines like Trino or Athena excel with static datasets, they aren't designed for continuous, low-latency ingestion and transformation of streaming data into Iceberg. This often forces engineers to integrate multiple complex tools, increasing operational overhead and fragility.

By the way, I wanted to continue to use the previous experiment with Flink SQL and Iceberg, but I found out Trino doesn't support Iceberg's DynamoDB catalog. Therefore, I had to create a new one.

We decided not to waste time reinventing the wheel and simply installed Trino on our servers. It’s a full featured SQL query engine that works on your data. Now our analysts can use it to work with data from AppMetr and execute queries at different levels of complexity.

DevRel for StarRocks. Trino doesn't have a great caching layer (https://github.com/trinodb/trino/pull/16375) and performance (https://github.com/trinodb/trino/issues/14237) and https://github.com/oap-project/Gluten-Trino. In benchmarks and community user testing, StarRocks has outperformed.

What if my SQL engine is Presto, Trino [1], or a similar query engine? If it's federating multiple source databases we peel the SQL back and get... SQL? Or you peel the SQL back and get... S3 + Mongo + Hadoop? Junior analysts would work at 1/10th the speed if they had to use those raw.

[1] https://trino.io/