LO VS duckdb-wasm

Absolutely. The memory management is manual like in C, but with addition of defers it is actually quite good to work with.

Here is the source for the allocator: https://github.com/glebbash/LO/blob/81f96bb09e3a0b1455b32497...

It also links to where I learned how to build the "proper" allocator from scratch.

While I was developing lole-lisp I happened to create a tool to generate C FFI bindings for Deno.

DuckDB has this feature:

duckdb> select 1,2,;

returns 1 and 2

https://shell.duckdb.org

DuckDB should give you everything with regards to DevEx that you seem to be needing.

You're not specifying what volume we're talking about here, but I'd be surprised if you can't go a long way with DuckDB. Ultimately the bottleneck for latency will be memory?

You can run it in a browser [0].

DuckDB's docs and blog are full of great content with regards to optimizing:

How to tune workloads: https://duckdb.org/docs/guides/performancehow_to_tune_worklo...

Performance guide: https://duckdb.org/docs/guides/performance/overview.html

Optimizers in DuckDB: https://duckdb.org/2024/11/14/optimizers.html

I'm sure there is more.

If I were in your shoes, even if you don't settle on DuckDB, it would be my go to benchmark due to ease of use and unoptimized speed.

Anything else out there like chDB (Clickhouse) or Apache Datafusion should be worth the effort.

For a detailed benchmark of DuckDB vs. Clickhouse for an orientation, see: https://bicortex.com/duckdb-vs-clickhouse-performance-compar...

0: https://github.com/duckdb/duckdb-wasm

I tried https://shell.duckdb.org/, but it was a very rough experience.

The "delete" button does not work. The "home" button inserts a whitespace. Pasting with "Ctrl+v" also does not work. Every keypress results in blinking, and there is a notable input lag.

When I tried a query

    duckdb> SELECT * FROM 'https://clickhouse-public-datasets.s3.amazonaws.com/github_events/partitioned_json/*.gz'

I witness the overengineering regarding "big" data tools and pipelines since many years... For a lot of use cases, data warehouses and data lakes are only in the gigabytes or single-digit terabytes range, thus their architecture could be much more simplified, e.g. running DuckDB on a decent EC2 instance.

In my experience, doing this will yield the query results faster than some other systems even starting the query execution (yes, I'm looking at you Athena)...

I even think that a lot of queries can be run from a browser nowadays, that's why I created https://sql-workbench.com/ with the help of DuckDB WASM (https://github.com/duckdb/duckdb-wasm) and perspective.js (https://github.com/finos/perspective).

i think duckdb-wasm is closer to 6MB over wire, but ~36MB once decompressed. (see net panel when loading https://shell.duckdb.org/)

the decompressed size should be okay since it's not the same as parsing and JITing 36MB of JS.