dtplyr VS vaex

Can’t say I’ve used it, but isn’t that what dtplyr is supposed to provide?

https://dtplyr.tidyverse.org/

# I am trying to follow the example at https://dtplyr.tidyverse.org/

FYI the developer of tidytable has been developing dtplyr for the Tidyverse. You might like that too!

If you need speed, you should consider using dtplyr (or tidytable), or even dbplyr with duckdb.

https://dtplyr.tidyverse.org/ ?

If you want data.table speed, but using dplyr/tidy then dtplyr is a good package to have handy. Personally I love R, and choose R + NodeJS as my gotos for everything I do, and use Python only when I have to.

Also, that dataset is quite big, and the "base" Tidyverse will be excessively slow. You should supplement the "base" Tidyverse packages (i.e. dplyr and tidyr) with either dtplyr or dbplyr (+ duckDB). I'd suggest starting with dtplyr, which should handle 10M+ rows fine.

If you want, there's a "substitute" for dplyr called dtplyr (also part of the Tidyverse), which "translates" your dplyr/tidyr code into data.table behind the scenes, and allows you to make your modifications apply directly to the original dataset by default:

Install the dtplyr package and change your code to:

Get the best of both worlds (tidyverse + data.tables) with dtplyr, a data.table backend for dplyr.

Try vaex, vaex, using lazy evaluation and parallel calculations, you should be fine.

I'd recommend QuestDB. Worked with it multiple times for different algorithmic trading needs and it didn't disappoint. If you want to load data fast, I'd recommend this Python library.

How about vaex?

You might find vaex (https://github.com/vaexio/vaex) interesting if you work with HDF5.

My guess is that should be possible, feel free to hop onto https://github.com/vaexio/vaex/discussions !