duckdb-wasm VS calculang

i think duckdb-wasm is closer to 6MB over wire, but ~36MB once decompressed. (see net panel when loading https://shell.duckdb.org/)

the decompressed size should be okay since it's not the same as parsing and JITing 36MB of JS.

Hey HN! We’ve built Pretzel, an open-source data exploration and visualization tool that runs fully in the browser and can handle large files (200 MB CSV on my 8gb MacBook air is snappy). It’s also reactive - so if, for example, you change a filter, all the data transform blocks after it re-evaluate automatically. You can try it here: https://pretzelai.github.io/ (static hosted webpage) or see a demo video here: https://www.youtube.com/watch?v=73wNEun_L7w

You can play with the demo CSV that’s pre-loaded (GitHub data of text-editor adjacent projects) or upload your own CSV/XLSX file. The tool runs fully in-browser—you can disconnect from the internet once the website loads—so feel free to use sensitive data if you like.

Here’s how it works: You upload a CSV file and then, explore your data as a series of successive data transforms and plots. For example, you might: (1) Remove some columns; (2) Apply some filters (remove nulls, remove outliers, restrict time range etc); (3) Do a pivot (i.e, a group-by but fancier); (4) Plot a chart; (5) Download the chart and the the transformed data. See screenshot: https://imgur.com/a/qO4yURI

In the UI, each transform step appears as a “Block”. You can always see the result of the full transform in a table on the right. The transform blocks are editable - for instance in the example above, you can go to step 2, change some filters and the reactivity will take care of re-computing all the cells that follow, including the charts.

We wanted Pretzel to run locally in the browser and be extremely performant on large files. So, we parse CSVs with the fastest CSV parser (uDSV: https://github.com/leeoniya/uDSV) and use DuckDB-Wasm (https://github.com/duckdb/duckdb-wasm) to do all the heavy lifting of processing the data. We also wanted to allow for chained data transformations where each new block operates on the result of the previous block. For this, we’re using PRQL (https://prql-lang.org/) since it maps 1-1 with chained data transform blocks - each block maps to a chunk of PRQL which when combined, describes the full data transform chain. (PRQL doesn’t support DuckDB’s Pivot statement though so we had to make some CTE based hacks).

There’s also an AI block: This is the only (optional) feature that requires an internet connection but we’re working on adding local model support via Ollama. For now, you can use your own OpenAI API key or use an AI server we provide (GPT4 proxy; it’s loaded with a few credits), specify a transform in plain english and get back the SQL for the transform which you can edit.

Our roadmap includes allowing API calls to create new columns; support for an SQL block with nice autocomplete features, and a Python block (using Pyodide to run Python in the browser) on the results of the data transforms, much like a jupyter notebook.

There’s two of us and we’ve only spent about a week coding this and fixing major bugs so there are still some bugs to iron out. We’d love for you to try this and to get your feedback!

Amazing, I was eagerly waiting for this one. Loading extensions in previous DuckDB-WASM releases didn't work seamlessly. Looks like now it's the case :D

ref: https://github.com/duckdb/duckdb-wasm/issues/1542#issuecomme...

Thanks!!

Btw, it's already happening:

Go to https://shell.duckdb.org, and type

I like Pandas, but you will love duckdb, which is solving this exact problem: https://duckdb.org/; https://shell.duckdb.org/

There's also https://shell.duckdb.org/ for playing around.

I know xls != csv, but would be cool to compare against https://github.com/duckdb/duckdb-wasm as well

I develop calculang, a language for calculations. It converts concise formulae into Javascript.

Since calculation formulae are usually inherently pure, a `--memo` flag to the compiler memoizes the lot.

Many of my models are written in this recursive style and only work because of memoization. Sometimes other tricks are needed e.g. careful ordering of the calls, so that the memo is available before the stack breaks.

My first calculang example is a bouncing ball: the position at t depends speed and the position at t-1. With loops we'll usually discover the same thing, but it's so much harder to follow.

First-class memoization to proliferate this style is a really neat thing.

[0] https://github.com/calculang/calculang

[1] bouncing ball code: https://github.com/declann/calculang-miscellaneous-models/bl...

[2] bouncing ball initial post: https://observablehq.com/@declann/calculang-bouncing-ball?co...

[3] if you can tolerate my slow-load WIP devtools and are on a Desktop, this is better, with buttons to navigate through the model development (only some of which are working now!): https://models-on-a-plane.pages.dev/stories/bounce/

My instinct about numbers (i.e. incl. statistics) is that they should be reproducible: on-demand, with results and workings easily subjected to interrogation.

In reality in many fields - my experience is in finance, but also in science - calculations are scattered across different languages and systems. This adds friction to any process about reproducing, understanding, analysing numbers.

This is some motivation for calculang, a language for calculations I develop. https://github.com/calculang/calculang

For the HN crew it's an under-development(!) functional language with properties to permit flexible designs that can scale. It's for numbers and if you share the model alongside your numbers people can check them according to the model and see the workings.

It goes well with visual number [dev/person]tools that I will release one of soon, and in the future watch for a browser extension for the workings behind numbers you are reading.

Important, it won't address the raw data part of the problem, but where numbers following from that are concerned, it might get closer to that instinct.

I'm developing calculang, a language for calculations: https://github.com/calculang/calculang

I've made some examples here: https://observablehq.com/collection/@declann/calculang

Some tooling work for visualizations and for showing the workings will be released in the next weeks.

There are no positive money flows; I've spent many years experimenting, developing, and now 1 year after a public release the twitter page where I make announcements has 24 followers: https://twitter.com/calculang

I'm a modelling consultant - I work with numbers, I think they should be simple, but they are disjointed across systems and entities and programming languages and spreadsheets. The friction accumulates everywhere: to get a simple result, to follow the workings, to do any analysis, to share one or the other.

I'm developing more tools to these ends, via my overarching project calculang, a language for calculations [1].

I also have a loan/repayment validator [2] but haven't added this QR code feature yet.

Bank letters e.g. "Interest rates are rising and now we want 100e more per month, every month" could use a QR code to an independent validator or to see the workings behind the 100 calculation.

Not using this in the real world now and there are security considerations to keep in mind, but reading state from a URL facilitates the usecase, QR codes that link physical numbers to their calculation and model.

Implementation of payroll calculator is an Observable notebook and thankfully it neatly supports all my strict requirements as a QR code demo.

[0] https://observablehq.com/@declann/payroll-playground-ireland...

+Feature tweet: https://twitter.com/calculang/status/1608183731533107206

[1] https://github.com/calculang/calculang

[2] https://observablehq.com/@declann/loan-validator-dev