data.table VS rust-dataframe

The data.table package may also make a huge difference in performance, and often simplifies the code as well https://github.com/Rdatatable/data.table

After the partial success with the development environment for R-yaml we tried another R package called data.table as part of the Open Source Development Course. Eventually we managed to run the tests of this too.

There's datatable. I haven't used it much, but the R version (data.table) is phenomenal.

data.table README.md

That's an old bug in data.table v1.12.2. It's been fixed for a while now. If you update your data.table version (e.g., install.packages("data.table") ) and retry then it should work fine.

Some suggestions: (1) https://github.com/Rdatatable/data.table Code based on the data.table will probably be fastest. There are a number of reasons for this. More here: https://cran.r-project.org/web/packages/data.table/vignettes/ and here: https://rdatatable.gitlab.io/data.table/library/data.table/html/datatable-optimize.html The GForce set of optimizations is well explained here: https://www.brodieg.com/2019/02/24/a-strategy-for-faster-group-statisitics/ (2) setDTthreads() is your friend in data.table (3) I have found (on Windows at least) Microsoft Open R use of parallel MKL faster than CRAN's latest release. See https://mran.microsoft.com/documents/rro/multithread Microsoft recommends using setMKLthreads() if it will help. (4) I think rfast ( https://github.com/RfastOfficial/Rfast ) is a library worth considering although I don't know if it will help you with brms and stan operations.

Take a look at the 22nd new feature of v1.14.3 on development here.

>Rust DataFrame implementation, built on Apache Arrow

https://github.com/nevi-me/rust-dataframe

A bit less mature/feature-complete than polars last time I looked. Does not seem to do anything with on-disk spillover from what I can see. But if you wanted to use Arrow to do that, nevi-me's crate may be a good place to start.