dataiter VS dtplyr

Here's another alternative. I wrote Dataiter specifically as I too was frustrated with Pandas. In my experience if you design a new API from scratch (and don't try to reimplement the Pandas API as many projects have done!) and have some vision and consistent principles, it's well possible to get a good intuitive API as a result. Two relevant issues remain: You're limited by NumPy's datatypes and their problems, such as memory-hogging strings and a lack of a proper missing value (NA), and secondly, limited by the Python language, so compared to e.g. dplyr's non-standard evaluation, you'll need to use lambda functions, which are unfortunately clumsy and verbose.

https://github.com/otsaloma/dataiter

Here's a comparison of dplyr vs. Dataiter vs. Pandas, which should give quick overview of the similarieties and differences.

https://dataiter.readthedocs.io/en/latest/_static/comparison...

Agreed, dplyr is great.

I built my own data frame implementation on top of NumPy specifically trying to accomplish a better API, similar to dplyr. It's not exactly the same naming or operations, but should feel familiar and much simpler and consistent than Pandas. And no indexes or axes.

Having done this, a couple notes on what will unavoidably differ in Python

* It probably makes more sense in Python to use classes, so method chaining instead of function piping. I wish one could syntactically skip enclosing parantheses in Python though, method chains look a bit verbose.

* Python doesn't have R's "non-standard evaluation", so you end up needing lambda functions for arguments in method chains and group-wise aggregation etc. I'd be interested if someone has a better solution.

* NumPy (and Pandas) is still missing a proper missing value (NA). It's a big pain to try to work around that.

https://github.com/otsaloma/dataiter

Can’t say I’ve used it, but isn’t that what dtplyr is supposed to provide?

https://dtplyr.tidyverse.org/

# I am trying to follow the example at https://dtplyr.tidyverse.org/

FYI the developer of tidytable has been developing dtplyr for the Tidyverse. You might like that too!

If you need speed, you should consider using dtplyr (or tidytable), or even dbplyr with duckdb.

https://dtplyr.tidyverse.org/ ?

If you want data.table speed, but using dplyr/tidy then dtplyr is a good package to have handy. Personally I love R, and choose R + NodeJS as my gotos for everything I do, and use Python only when I have to.

Also, that dataset is quite big, and the "base" Tidyverse will be excessively slow. You should supplement the "base" Tidyverse packages (i.e. dplyr and tidyr) with either dtplyr or dbplyr (+ duckDB). I'd suggest starting with dtplyr, which should handle 10M+ rows fine.

If you want, there's a "substitute" for dplyr called dtplyr (also part of the Tidyverse), which "translates" your dplyr/tidyr code into data.table behind the scenes, and allows you to make your modifications apply directly to the original dataset by default:

Install the dtplyr package and change your code to:

Get the best of both worlds (tidyverse + data.tables) with dtplyr, a data.table backend for dplyr.