xarray VS fugue

If this idea sounds inderesting, you might want to look at xarray for a more established project: https://github.com/pydata/xarray

Interesting - I'm curious whether you feel that Xarray covers these use cases already?

https://xarray.dev/

Especially as I've said before that Hyperspy shares so many features in common with Xarray that Hyperspy should just use Xarray under the hood.

https://github.com/hyperspy/hyperspy/discussions/3405

Sadly, several python projects do not use semantic versioning, for example xarray [0] and dask. Numpy can make backward incompatible changes after a warning for two releases[1]. In general, the python packaging docs do not really read as an endorsement of semantic versioning [2]:

> A majority of Python projects use a scheme that resembles semantic versioning. However, most projects, especially larger ones, do not strictly adhere to semantic versioning, since many changes are technically breaking changes but affect only a small fraction of users...

[0] https://github.com/pydata/xarray/issues/6176

PyTorch and JAX are used heavily in climate science on the ML side. For more general analytics, not so much. Many of our users like to use Xarray as a high-level API. There has been some work to integrate Xarray with PyTorch (https://github.com/pydata/xarray/issues/3232) but we're not there yet.

The Python Array API standard should help align these different back-ends: https://data-apis.org/array-api/latest/

Does polars have N-D labelled arrays, and if so can it perform computations on them quickly? I've been thinking of moving from pandas to xarray [0], but might consider poplars too if it has some of that functionality.

[0] https://xarray.dev/

https://xarray.dev

Pydata's xarray

Please integrate it with Fugue.

https://github.com/fugue-project/fugue

Fugue is an interesting library in this space , though I haven’t tried it

https://github.com/fugue-project/fugue

A unified interface for distributed computing. Fugue executes SQL, Python, and Pandas code on Spark, Dask and Ray without any rewrites.

fugue — distributed computing done easy

This looks similar to fugue, am I right? How do they compare?

I am looking forward to how Substrait could help removing this friction. It aims to provide a standardised intermediate query language (lower level than SQL) to connect frontend user interfaces like SQL or data frame libraries with backend analytical computing engines. It is linked to the Arrow ecosystem. Something like Ibis or Fugue could become the front and DuckDB the backend engine.

There's dask-sql, but I think it is being abandoned for fugue-project. I'm actually excited for this project as it is trying to provide a backend agnostic solution, which would seem like a difficult, lofty goal. I wish them luck.