evcxr VS polars

In theory you should be able to create Rust notebooks (Jupyter notebook) using evcxr so maybe some AI, data analysis, prototyping make sense if you aim for good performance in final application (protype in evcxr and use notebook as reference to implement final application in Rust for speed and safety).

You should check out evcxr

I think you might like this project: https://github.com/google/evcxr . It brings the REPL workflow to Rust, so having fast iteration should not be an issue.

While not Elixir good, the evcxr python notebook plugin gets you 50% of the way there.

https://depth-first.com/articles/2020/09/21/interactive-rust...

https://github.com/google/evcxr

I've started liking evcxr (https://github.com/google/evcxr) for REPL. It's a little slow compared to other REPLs, but still good enough to be usable after initial load.

Would such a project make it possible to have a faster rust repl? We can use evcxr, but it definitely doesn't feel first-class.

I'm a long-time python developer and develop on the side in Rust for about as long as you, I also found myself familiar with the available datastructures and algorithms, as well as some FP-inspired syntax ( i.e. iterators instead of for loops ) . evcxr is reminiscent of ipython, it can even be integrated with jupyter notebook. And indeed I was surprised to find myself in such a familiar world. It's definitely something that more python developers should be aware of.

e.g. https://github.com/pola-rs/polars/issues/8533

I have updated this issue https://github.com/pola-rs/polars/issues/8533, please kindly help to solve it. I have also sent similar issues to Pandas https://github.com/pandas-dev/pandas/issues/53249

could you open up an issue in github

https://github.com/jorgecarleitao/arrow2 https://github.com/apache/arrow-datafusion https://github.com/apache/arrow-ballista https://github.com/pola-rs/polars https://github.com/duckdb/duckdb

A new release of polars https://github.com/pola-rs/polars/releases/tag/rs-0.29.0 query engine/ DataFrame library.

I successfully ran four jobs with a billion rows yesterday while testing trillions of rows for more than a million files using Polars and Peaks on a step-by-step progressive basis. Previously, Polars failed on a single job, but after several bug fixes, it can now handle the workload. You can see https://github.com/pola-rs/polars/issues/7774

Over in polars we are using some of these tricks to greatly increase the parsing of ndjson. While no official benchmarks have been done, polars ndjson reader does seem to be faster than simdjson in many scenarios.

There are also several mentions of polars:

For data frames built on Apache Arrow and: https://github.com/pola-rs/polars/