interface-types VS prql

Some things that might greatly increase wasm usage and overall tooling:

1) Tools that run docker containers and serverless function services (like AWS lambda) to support providing a .wasm files instead

2) Garbage collection in the runtime to make GC languages easier to port to wasm

3) Dynamically typed languages (NodeJS, Python, Ruby) being able to compile to webassembly directly instead of porting the runtime to webassembly and then running the code through the runtime. This is a big ask though, basically needs to redesign the runtime completely

4) wasm-DOM bindings will enable other languages to do HTML rendering which will require new web frameworks for every language that wants to take over the space from JS. This will lead to (even more) fragmentation of the web ecosystem

5) A new wasm-first SDK (unrelated to the DOM) for building cross platform applications. I can see this taking off only if it is built-into the browsers and backed by some standards committee, so not very likely I think

6) Something like the Interface Types proposal ( https://github.com/WebAssembly/interface-types/blob/main/pro... ) becomes a thing allowing wasm programs to be consisted of modules written in several different languages and being able to call said modules with low or 0 runtime performance hit (and of course, no compilation to multiple CPU archs). So much of programming ecosystems are locked to specific languages (like data science with python) when there is little technical reason for it be like that.

AFAIK GC is irrelevant for "direct DOM access", you would rather want to hop into the following rabbit hole:

- reference types: https://github.com/WebAssembly/reference-types/blob/master/p...

- interface types (inactive): https://github.com/WebAssembly/interface-types/blob/main/pro...

- component model: https://github.com/WebAssembly/component-model

If this looks like a mess, that's because it is. Compared to that, the current solution to go through a Javascript shim doesn't look too bad IMHO.

I'm not really optimistic that particular aspect will get much improvement. Many people expected interface types to come save the day, but after a looong stagnation that proposal has been archived (for now) in favour of component types, which has much less potential for performance gains.

Really good questions. Unfortunately, most of the issues I found back then were fundamental ones. I've seen that Wasm has deprecated "Interface Types" and is now working on the "Component Model". But even then, as far as I understand that would only avoid the serialization and deserialization steps, and you would still need to copy complex types. It will be more performant, but I don't think it would be enough for Tremor either.

wasm doesn't know anything about the outside world on purpose. This allows it to be used in other domains. For direct access to the DOM et al, interface types are being developed. It's a non-trivial problem to interoperate with a dynamically typed GC'd language from any statically typed no-GC language that can compile to wasm.

You may want to look into WASM interface types, which is defining what amounts to am IDL for WASM and different languages have common calling conventions: https://hacks.mozilla.org/2019/08/webassembly-interface-type...

I don’t know if there’s a better intro article. I believe this is the current iteration of the proposal: https://github.com/WebAssembly/interface-types/blob/main/pro...

Yes, and if I'm not mistaken, this is the RFC

I see no reason why not. See the interface types proposal for a proposed solution.

Wanted to get thoughts from the Rust experts on this - the author of the Yew framework seems to think that Web Assembly Interface Types (https://github.com/WebAssembly/interface-types/blob/master/proposals/interface-types/Explainer.md) will allow Yew to eventually become faster than Vue, React, Angular, etc. Is there general consensus on this in the Rust community? The prospect of mixing Rust (for the performance critical pieces) with TS on the front end doesn't seem super appealing to me.

> I completely attribute this to SQL being difficult or "backwards" to parse. I mean backwards in the way that in SQL you start with what you want first (the SELECT) rather than what you have and widdling it down.

> The turning point for me was to just accept SQL for what it is.

Or just write PRQL and compile it to SQL

https://github.com/PRQL/prql

Hey HN! We’ve built Pretzel, an open-source data exploration and visualization tool that runs fully in the browser and can handle large files (200 MB CSV on my 8gb MacBook air is snappy). It’s also reactive - so if, for example, you change a filter, all the data transform blocks after it re-evaluate automatically. You can try it here: https://pretzelai.github.io/ (static hosted webpage) or see a demo video here: https://www.youtube.com/watch?v=73wNEun_L7w

You can play with the demo CSV that’s pre-loaded (GitHub data of text-editor adjacent projects) or upload your own CSV/XLSX file. The tool runs fully in-browser—you can disconnect from the internet once the website loads—so feel free to use sensitive data if you like.

Here’s how it works: You upload a CSV file and then, explore your data as a series of successive data transforms and plots. For example, you might: (1) Remove some columns; (2) Apply some filters (remove nulls, remove outliers, restrict time range etc); (3) Do a pivot (i.e, a group-by but fancier); (4) Plot a chart; (5) Download the chart and the the transformed data. See screenshot: https://imgur.com/a/qO4yURI

In the UI, each transform step appears as a “Block”. You can always see the result of the full transform in a table on the right. The transform blocks are editable - for instance in the example above, you can go to step 2, change some filters and the reactivity will take care of re-computing all the cells that follow, including the charts.

We wanted Pretzel to run locally in the browser and be extremely performant on large files. So, we parse CSVs with the fastest CSV parser (uDSV: https://github.com/leeoniya/uDSV) and use DuckDB-Wasm (https://github.com/duckdb/duckdb-wasm) to do all the heavy lifting of processing the data. We also wanted to allow for chained data transformations where each new block operates on the result of the previous block. For this, we’re using PRQL (https://prql-lang.org/) since it maps 1-1 with chained data transform blocks - each block maps to a chunk of PRQL which when combined, describes the full data transform chain. (PRQL doesn’t support DuckDB’s Pivot statement though so we had to make some CTE based hacks).

There’s also an AI block: This is the only (optional) feature that requires an internet connection but we’re working on adding local model support via Ollama. For now, you can use your own OpenAI API key or use an AI server we provide (GPT4 proxy; it’s loaded with a few credits), specify a transform in plain english and get back the SQL for the transform which you can edit.

Our roadmap includes allowing API calls to create new columns; support for an SQL block with nice autocomplete features, and a Python block (using Pyodide to run Python in the browser) on the results of the data transforms, much like a jupyter notebook.

There’s two of us and we’ve only spent about a week coding this and fixing major bugs so there are still some bugs to iron out. We’d love for you to try this and to get your feedback!

> Looks like PRQL doesn't have a Go library so I guess they just really wanted something in Go?

There's some C bindings and the example in the README shows integration with Go:

https://github.com/PRQL/prql/tree/main/prqlc/bindings/prqlc-...

Can someone tell me why PRQL is better? I went here: https://github.com/PRQL/prql

It looks nice, but what's the strengths compared to SQL?

PRQL [1] is a compile-to-SQL relational querying language that puts FROM first.

[1] https://prql-lang.org

https://prql-lang.org/ might be an answer for this. As a cross-database pipelined language, it would allow RAG to be intermixed with the query, and the syntax may(?) be more reliable to generate