Dask vs iceberg

Our great sponsors

WorkOS - The modern identity platform for B2B SaaS

InfluxDB - Power Real-Time Data Analytics at Scale

SaaSHub - Software Alternatives and Reviews

Our great sponsors

Dask		iceberg
	Project
32	Mentions	18
11,965	Stars	5,481
1.3%	Growth	3.5%
9.7	Activity	9.9
6 days ago	Latest Commit	5 days ago
Python	Language	Java
BSD 3-clause "New" or "Revised" License	License	Apache License 2.0

The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.

Dask

Posts with mentions or reviews of Dask. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2023-06-15.

The Distributed Tensor Algebra Compiler (2022)
4 projects | news.ycombinator.com | 15 Jun 2023
A peek into Location Data Science at Ola
6 projects | dev.to | 26 Sep 2022

Data scientists work on phenomenally large datasets, and Dask is a handy tool for exploration within the confines of a single cloud VM or their local PCs. Location data visualization is an essential part of deciding further algorithm development and roadmap for projects. This lays the foundation for data engineering and science to work at scale, with petabytes of data.
File format for large data with many columns
2 projects | /r/Python | 15 May 2022
What is the best way to save a csv.file in number only ? PC hangs when my file is more than 2GB
2 projects | /r/learnpython | 4 Apr 2022

Dask
Large Scale Hydrology: Geocomputational tools that you use
3 projects | /r/Hydrology | 13 Feb 2022

We're using a lot of Python. In addition to these, gridMET, Dask, HoloViz, and kerchunk.
msgspec - a fast & friendly JSON/MessagePack library
4 projects | /r/Python | 10 Feb 2022

I wrote this for speeding up the RPC messaging in dask, but figured it might be useful for others as well. The source is available on github here: https://github.com/jcrist/msgspec.
What does it mean to scale your python powered pipeline?
4 projects | dev.to | 3 Jan 2022

Dask: Distributed data frames, machine learning and more
Data pipelines with Luigi
4 projects | dev.to | 22 Dec 2021

To do that, we are efficiently using Dask, simply creating on-demand local (or remote) clusters on task run() method:
Is Numpy always more efficient than Pandas? And how much should we rely on Python anyway?
1 project | /r/datascience | 10 Dec 2021

Look into Dask, see: https://dask.org/
Ask HN: Is PySPark a Dead-End?
1 project | news.ycombinator.com | 5 Dec 2021

[1] https://dask.org/

iceberg

Posts with mentions or reviews of iceberg. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2023-07-06.

Iceberg won the table format war: But not in the way you thought it might
2 projects | /r/dataengineering | 6 Jul 2023
Lakehouse using AWS Athena on Iceberg Concerns
1 project | /r/dataengineering | 28 May 2023
apache/iceberg: Apache Iceberg
1 project | /r/devopsish | 13 Feb 2023
What are the main things I need to know to be hired as a Java developer?
4 projects | /r/java | 1 Feb 2023
Have you used Athena Iceberg for small(-ish) data?
1 project | /r/aws | 28 Jan 2023
Is Data Lakehouse a threat to Snowflake?
1 project | /r/dataengineering | 24 Jan 2023
Snowflake vs databricks cloud/labor cost
1 project | /r/snowflake | 6 Dec 2022

This is interesting, imo.
Setting the Table: Benchmarking Open Table Formats
2 projects | /r/dataengineering | 1 Dec 2022
Spark Dynamic Partition Overwrite Mode Replaces Existing Data
1 project | /r/apachespark | 20 Nov 2022

If you're using Iceberg as your table format, it had bugs with MERGE INTO with non-nullable columns up until September: https://github.com/apache/iceberg/pull/5679
How to migrate delta tables to iceberg?
1 project | /r/dataengineering | 5 Oct 2022

yeah, this as a capability is a WIP and discussion point in the iceberg community - https://github.com/apache/iceberg/pull/5331

What are some alternatives?

When comparing Dask and iceberg you can also consider the following projects:

Airflow - Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

kudu - Mirror of Apache Kudu

Numba - NumPy aware dynamic Python compiler using LLVM

hudi - Upserts, Deletes And Incremental Processing on Big Data.

Kedro - Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.

Apache Avro - Apache Avro is a data serialization system.

NetworkX - Network Analysis in Python

debezium - Change data capture for a variety of databases. Please log issues at https://issues.redhat.com/browse/DBZ.

Pandas - Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more

RocksDB - A library that provides an embeddable, persistent key-value store for fast storage.

Interactive Parallel Computing with IPython - IPython Parallel: Interactive Parallel Computing in Python

delta - An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs