Big Data Is Dead

• duckdb-wasm

  13 975 9.5 C++

  WebAssembly version of DuckDB

  

I witness the overengineering regarding "big" data tools and pipelines since many years... For a lot of use cases, data warehouses and data lakes are only in the gigabytes or single-digit terabytes range, thus their architecture could be much more simplified, e.g. running DuckDB on a decent EC2 instance.

In my experience, doing this will yield the query results faster than some other systems even starting the query execution (yes, I'm looking at you Athena)...

I even think that a lot of queries can be run from a browser nowadays, that's why I created https://sql-workbench.com/ with the help of DuckDB WASM (https://github.com/duckdb/duckdb-wasm) and perspective.js (https://github.com/finos/perspective).

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• sql-workbench

  5 15 2.6

  Public issue-tracking and feature suggestion for sql-workbench.com

I witness the overengineering regarding "big" data tools and pipelines since many years... For a lot of use cases, data warehouses and data lakes are only in the gigabytes or single-digit terabytes range, thus their architecture could be much more simplified, e.g. running DuckDB on a decent EC2 instance.

In my experience, doing this will yield the query results faster than some other systems even starting the query execution (yes, I'm looking at you Athena)...

I even think that a lot of queries can be run from a browser nowadays, that's why I created https://sql-workbench.com/ with the help of DuckDB WASM (https://github.com/duckdb/duckdb-wasm) and perspective.js (https://github.com/finos/perspective).

• polars

  146 27,185 10.0 Rust

  Dataframes powered by a multithreaded, vectorized query engine, written in Rust

  

Suggest a related project