datatable VS DataProfiler

Aside from that, there is a Python translation of data.table (see documentation here), which might be worth looking into. However, it hasn't had any major updates in a while: the last release 2 years ago ...

There's datatable. I haven't used it much, but the R version (data.table) is phenomenal.

just use HDF5 or Parquet, or CSV + https://github.com/h2oai/datatable to speed up the file reading.

> For me I had with pandas the most issues using it's multiindex.

Yessss. I loathe indices, and have never been in a situation where I was better off with them than without them.

> Regarding fast you have something like Vaex on python sid

I've never used Vaex, but I've used datatable (https://github.com/h2oai/datatable) and polars (https://github.com/pola-rs/polars). Polars is my favorite API, but datatable was faster at reading data (Polars was faster in execution). I'll have to give Vaex a try at some point.

Yep. I think this is the downside to a package being entirely maintained by volunteers. In any case, Pandas is still the leading data wrangling package for Python. (I'm excited to see how datatable evolves.)

h2o's data.table clone is fine

https://github.com/h2oai/datatable

It's been possible to skip tokenization for a long time, my team and I did it here - https://github.com/capitalone/DataProfiler

For what it's worth, we actually were working with LSTMs with nearly a billion params back in 2016-2017 area. Transformers made it far more effective to train and execute, but ultimately LSTMs are able to achieve similar results, though slow & require more training data.

Great call out -- would you be willing to write up an issue for that on the repo? Thank you! https://github.com/capitalone/DataProfiler/issues/new/choose