CSV or Parquet File Format

1. delta

   1 74 8,144 9.9 Scala

   An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs (by delta-io)

   

   I prefer parquet (or delta for larger datasets. CSV for very small datasets, or the ones that will be later used/edited in Excel or Googke sheets.

2. Stream

   getstream.io featured

   Stream - Scalable APIs for Chat, Feeds, Moderation, & Video. Stream helps developers build engaging apps that scale to millions with performant and flexible Chat, Feeds, Moderation, and Video APIs and SDKs powered by a global edge network and enterprise-grade infrastructure.

   

3. Apache Arrow

   2 86 15,690 9.9 C++

   Apache Arrow is the universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics

   

   In fact I have asked Apache Github how to read select column of particular row group of a parquet file. https://github.com/apache/arrow/issues/35688

4. DuckDB

   3 70 30,765 10.0 C++

   DuckDB is an analytical in-process SQL database management system

   

   The Parquet-Go library is very complex, not yet success to use it. So I ask whether DuckDB can provide API https://github.com/duckdb/duckdb/issues/7776

Suggest a related project