polars VS evcxr

polars might click better. Once you learn the expression API, you'll see that googling is much less needed as you can extrapolate that logic.

The new data tools I've seen are complex under the hood, but offer elegant user experiences, giving the best of both worlds.

You referenced a 500 line Python script being refactored with Rust and make me think of the Polars project: https://github.com/pola-rs/polars

Polars uses Rust to make DataFrame operations lightning fast. But you don't need to use Rust to use Polars. Just use the Polars Python API and you have an elegant way to scale on a single machine and perform analyses way faster.

I'm working on Dask and our end goal is the same. We want to provide users with syntax they're familiar with to scale their analyses locally & to clusters in the cloud. We also want to provide flexibility so users can provide highly custom analyses. Highly custom analyses are complex by nature, so these aren't "easy codebases" by any means, but Dask Futures / Dask Delayed makes the distributed cluster multiprocessing part a lot easier.

Anyways, I've just seen the data industry moving towards better & better tools. Delta Lake abstracting all the complications of maintaining all the complications of plain vanilla Parquet lakes is another example of the amazing tooling. Now the analyses and models... those seem to be getting more complicated.

I'd recommend checking out polars as an alternative to pandas - https://github.com/pola-rs/polars

It has a rather different api, and is significantly faster. Highly recommend it.

Have you looked at Polars? It's a new dataframe library that has an api that makes a lot more sense than pandas, and on top of that is much, much faster.

Polars is almost at 500k downloads, this is great to see the Rust ecosystem connecting with Python.

Are there any articles about how the authors of polars used pyo3 to expose a python API?

REPL... is something was at least tried; I don't know how good it is because I haven't tested it yet, but it sounds interesting and potentially helpful. https://github.com/google/evcxr

Tytytyty!!

I know this is not realistic, but the only thing rust is missing is faster compilation times. To be fair, it’s already super fast.

I was adding a repl[1] to create-react-app[2] (trying to achieve something like RoR’s console) and it took way too long for it to boot simply because the dependencies for something like a full scale web app take long to compile. Even with incremental compilation, that first boot time was too much.

[1] https://github.com/google/evcxr/blob/main/evcxr_repl

[2] https://github.com/Wulf/create-rust-app

There's also some ongoing work on Rust REPL, see https://github.com/google/evcxr - though it's still a bit of a hack. Might become even easier than Turbo Pascal itself, and comparable to home computer BASICs.

For a couple examples, evcxr, which provides a rust repl and jupyter kernel, uses rust-analyzer's crates to provide features such as completions in its repl.

I disagree. Rust is statically typed but we build good enough libraries to make working with matrices easy enough, the data scientists will rarely have to type them out themselves as they can be inferred. I was working with glam just recently and it's a very nice matrix library in which I didn't have to mess with the type system at all. There is also REPL to support rust https://github.com/google/evcxr that could be improved.

evcxr https://github.com/google/evcxr provides a Jupyter kernel, which is handy. But it's quite complex to install and set up if you don't already have Python installed.

You do need custom support to do this in Rust, but it's in the works - see https://github.com/google/evcxr for what seems to be the most current effort.

evcxr is the best one I've found. It's pretty good, but it is extremely slow I assume because it has to compile each line as you input it. Also I can't get it to load crates from my current project so you have to install them within the REPL.