plydata VS Dask

from datar import f from datar.dplyr import mutate, filter, if_else from datar.tibble import tibble # or # from datar.all import f, mutate, filter, if_else, tibble df = tibble( x=range(4), y=['zero', 'one', 'two', 'three'] ) df >> mutate(z=f.x) """# output x y z 0 0 zero 0 1 1 one 1 2 2 two 2 3 3 three 3 """ df >> mutate(z=if_else(f.x>1, 1, 0)) """# output: x y z 0 0 zero 0 1 1 one 0 2 2 two 1 3 3 three 1 """ df >> filter(f.x>1) """# output: x y 0 2 two 1 3 three """ df >> mutate(z=if_else(f.x>1, 1, 0)) >> filter(f.z==1) """# output: x y z 0 2 two 1 1 3 three 1 """ Works with plotnine # example grabbed from https://github.com/has2k1/plydata import numpy from datar.base import sin, pi from plotnine import ggplot, aes, geom_line, theme_classic df = tibble(x=numpy.linspace(0, 2*pi, 500)) (df >> mutate(y=sin(f.x), sign=if_else(f.y>=0, "positive", "negative")) >> ggplot(aes(x='x', y='y')) + theme_classic() + geom_line(aes(color='sign'), size=1.2)) https://preview.redd.it/w0hs4m8fyf771.png?width=697&format=png&auto=webp&s=eadd7473a9e3393c2d58531c0b2b12f849c27e5e Easy to integrate with other libraries import klib from pipda import register_verb from datar.datasets import iris from datar.dplyr import pull dist_plot = register_verb(func=klib.dist_plot) iris >> pull(f.Sepal_Length) >> dist_plot() https://preview.redd.it/w8b8ouagyf771.png?width=892&format=png&auto=webp&s=3cc8f04e63be710f593b2b6128073f65cf7ffaa4 For more detailed and advanced usage, see https://pwwang.github.io/datar/

df = tibble( x=range(4), y=['zero', 'one', 'two', 'three'] ) df >> mutate(z=f.x) """# output x y z 0 0 zero 0 1 1 one 1 2 2 two 2 3 3 three 3 """ df >> mutate(z=if_else(f.x>1, 1, 0)) """# output: x y z 0 0 zero 0 1 1 one 0 2 2 two 1 3 3 three 1 """ df >> filter(f.x>1) """# output: x y 0 2 two 1 3 three """ df >> mutate(z=if_else(f.x>1, 1, 0)) >> filter(f.z==1) """# output: x y z 0 2 two 1 1 3 three 1 """ ``` Works with plotnine ```python example grabbed from https://github.com/has2k1/plydata import numpy from datar.base import sin, pi from plotnine import ggplot, aes, geom_line, theme_classic df = tibble(x=numpy.linspace(0, 2*pi, 500)) (df >> mutate(y=sin(f.x), sign=if_else(f.y>=0, "positive", "negative")) >> ggplot(aes(x='x', y='y')) + theme_classic() + geom_line(aes(color='sign'), size=1.2)) ```  Easy to integrate with other libraries ```python import klib from pipda import register_verb from datar.datasets import iris from datar.dplyr import pull dist_plot = register_verb(func=klib.dist_plot) iris >> pull(f.Sepal_Length) >> dist_plot() ```  For more detailed and advanced usage, see https://pwwang.github.io/datar/

Data scientists work on phenomenally large datasets, and Dask is a handy tool for exploration within the confines of a single cloud VM or their local PCs. Location data visualization is an essential part of deciding further algorithm development and roadmap for projects. This lays the foundation for data engineering and science to work at scale, with petabytes of data.

Dask

We're using a lot of Python. In addition to these, gridMET, Dask, HoloViz, and kerchunk.

I wrote this for speeding up the RPC messaging in dask, but figured it might be useful for others as well. The source is available on github here: https://github.com/jcrist/msgspec.

Dask: Distributed data frames, machine learning and more

To do that, we are efficiently using Dask, simply creating on-demand local (or remote) clusters on task run() method:

Look into Dask, see: https://dask.org/

[1] https://dask.org/