n5 VS tensorstore

Provides a uniform API for reading and writing multiple array formats, including zarr and N5.

For readers' context: zarr is a self-describing n-dimensional array hierarchy format specification which can sit over more or less any key-value store. If you've ever used HDF5, it's basically that, but array chunks are exploded over the file system/ cloud store, and all the metadata is JSON. It's gaining traction in the biological imaging and geo/meteorological data communities, among other places. Work on the v3 specification is in progress, which aims to abstract away a generic protocol, as well as fold in the community behind N5, an almost-identical format used by a small but vocal number of bio-imaging labs.

shhh...don't tell them about Tensorflow, or tf.Tensor, or theano.tensor, or paddle.to_tensor or torch.Tensor or torch.Tensor or mindspore.Tensor or tensorstore or ... But in all seriousness your pitchforks might be a little late to the party here. Rightly or wrongly (large) segments of the machine learning community have adopted terminology to describe things (for your sake, I hope you have not seen what they did with the term 'deconvolution'). This is not a new phenomena either and has been happening for at least a decade and almost certainly longer. However, there are other libraries that use the array terminology np.array, mx.nd.array, chainer, jax.numpy.array so it is definitely not unanimous. Whether you think this is an egregious mis-representation of the nomenclature or not is irrelevant now as it has a pretty established use within the community. Language is an evolving untamed beast - don't get angry, get building!

There are tons of other ways to store vector data, one was just recently released - https://github.com/google/tensorstore

Continue reading | Github | Google Full Blog

Today we are introducing TensorStore, an open-source C++ and Python software library designed for storage and manipulation of n-dimensional data that: