db-benchmark VS pyodide

I strongly recommend data.table R. Tidyverse is an improvement on base R, no question. Data.table has less intuitive syntax and can be harder to learn, but is lightning fast and memory efficient. If you're working with more than 1M rows, you should be using data.table.

Here are some benchmarks: https://h2oai.github.io/db-benchmark/

Hi, good to hear that you guys care about testing. One thing apart from the Github issues that led me to believe it might not be super stable yet was the benchmark results on https://h2oai.github.io/db-benchmark/ which make it look like it couldn't handle the 50GB case due to a out of memory error. I see that the benchmark and the used versions are about a year old so maybe things changed a lot since then. Can you chime in regarding the current story of running bigger DBs like 1TB on a machine with just 32GB or so RAM? Especially regardung data mutations and DDL queries. Thanks!

Per these benchmarks it appears Polars is an order of magnitude more performant and it's lazy and Rust is just kinda sexy.

This is a much more comprehensive set of benchmarks: https://h2oai.github.io/db-benchmark/

Its not like pandas is faster than tidyverse either on all the bechmarks, and data.table is faster than both. https://h2oai.github.io/db-benchmark/

Language isn't really important here; what's important is tooling, and R definitely has the tooling. I would look at this benchmark reference for database-like operations, and you'll see that data.table (a very fast and memory-efficient R package) consistently ranks as one of the fastest tools out there that can also support a wide range of memory loads.

It seems to me that the Spark model is much more sensible in terms of performance. In Spark, individual tasks are finally compiled into optimized Java code. As I understand Dusk works, a separate Python process is run for each data subset. So because of this architecture, Dusk is unlikely to ever get Spark performance. By the way, both systems build and optimize the operation graph. This is confirmed by benchmarks: https://h2oai.github.io/db-benchmark/

pyodide/packages

I tried building this with a Service Worker first and it didn't work, because Pyodide needs XMLHttpRequest.

I opened an issue about that here: https://github.com/pyodide/pyodide/issues/2432

https://github.com/pyodide/pyodide/pull/1882

But some other low hanging fruit include unvendoring the special encodings for Asian languages (hopefully everyone uses utf8), the decimal library, and the xml library which are all quite large and only occasionally used.

This uses Pyodide [0] under the hood [1], which is CPython compiled to WebAssembly. In all my tests of it, loading takes a long time ~5 seconds. Coldbrew [2], another distribution of CPython on Wasm, is another option with similar load times.

And Brython [3] is a completely different option without long load time: a Python interpreter implemented in JavaScript.

If load time is important, Brython is pretty nice. If feature completeness is important, Pyodide and Coldbrew are probably best.

[0] https://github.com/pyscript/pyscript/blob/main/pyscriptjs/sr...

[1] https://github.com/pyodide/pyodide

[2] https://github.com/plasticityai/coldbrew

[3] https://github.com/brython-dev/brython

Python via Pyodide

Those interested in this should check out Pyodide[0]. It basically "just works" so long as the libraries you import are pure Python or are part of the core scientific stack (the Pyodide authors have done the work to manually port all the C code behind numpy, scipy, etc.).

What I really wish for is for ~all Python packages to work in the browser without manual porting of the underlying C/Rust/etc. being needed, since a lot of the interesting and useful libraries aren't pure Python, and manual porting is non-trivial.

I'm not sure what the best route to that future is, but I'm guessing it'd probably help if Python had a wasm runtime in its standard library[1], since then authors of libraries that use C/Rust/etc. might make cross-platform builds (perhaps by default).

Regarding this Pycon speech, it seems that it's related to this entry in the 3.11 changelog[2], which the speaker was heavily involved with:

> CPython now has experimental support for cross compiling to WebAssembly platform wasm32-emscripten. The effort is inspired by previous work like Pyodide. (Contributed by Christian Heimes and Ethan Smith in bpo-40280[3])

[0] https://github.com/pyodide/pyodide

[1] https://discuss.python.org/t/add-a-webassembly-wasm-runtime/...

[2] https://docs.python.org/3.11/whatsnew/3.11.html

[3] https://bugs.python.org/issue40280

Wow this is amazing. Now all I need to deploy static machine learning demos in documentation is https://github.com/pyodide/pyodide/issues/2198

yes you can do that : https://github.com/pyodide/pyodide