Activeloop Hub VS dwc

u/Remote_Cancel_7977 we just launched 100+ computer vision datasets via Activeloop Hub yesterday on r/ML (#1 post for the day!). Note: we do not intend to compete with HuggingFace (we're building the database for AI). Accessing computer vision datasets via Hub is much faster than via HuggingFace though, according to some third-party benchmarks. :)

u/gopietz good question. htype="class_label" will work, but querying doesn't support multi-dimensional labels yet. Would you mind opening an issue requesting that feature?

We've recently added a Huggingface integration that allows ingestion of HuggingFace datasets.

Hub, our open-source package, lets you stream datasets while training to PyTorch/TensorFlow. Check out how we achieved 95% GPU utilization while training on ImageNet at 50% less cost. We're building the Database for AI, with everything it should contain. If there's an adjacent feature that would make it more useful for your workflow, do let us know!

Our early users love the tool and I hope you'll love it too. We have many more features other than visualization on the roadmap (the current feature list includes querying, version control UI, and integrates through our open-source package Hub (dataset format for AI) with TensorFlow, PyTorch, Sagemaker, other tools on the roadmap.

Please take a look at our open-source dataset format https://github.com/activeloopai/hub and a tutorial on htypes https://docs.activeloop.ai/how-hub-works/visualization-and-htype

The platform allows to: - Inspect the data with all its bounding boxes, masks, etc, and have important stats such as distribution of the labels (adding more stuff in the future to fight bias and improve data quality). - Query datasets to create new, highly specific ones - Version control datasets (while visualizing the changes). I'm confident that if you've ever worked on iteratively improving your models, dataset versioning is probably something you've done. - Stream computer vision datasets while training in PyTorch/Tensorflow via Hub, our open source package (we might add an even more straightforward way to the UI). - For larger organizations access management is important, and we do take care of that.

The visualization interfaces with our open-source dataset format for AI, enabling workflows such as querying/filtering to create datasets/inspect subsamples, tracking changes to the data with data version control visualization (e.g. cross-referencing if the transformations applied had intended effects), and will have integrations with other tools (e.g. experiment tracking, labelling) very soon.

Yes, we're not entirely relevant for your use case, especially if the data is not that big/complex, and benefits that you'd get from switching to Hub format are not as pronounced in case of text as they are in case of computer vision datasets (actually, we still have a couple of diehard NLP community members, but they have ridiculously big text datasets). I presume your university system doesn't use unstructured data like videos/images/audio, either, so our product wouldn't be very helpful in that regard. I do wish you tons of luck and patience though (>10ˆ6?! good Lord...)

Hub.

As a community we are trying to release more benchmark datasets for different kinds of training such as the herbarium specimens described in this paper: https://doi.org/10.3897/BDJ.7.e31817 It’s not an especially large dataset (only 1,800 specimens) but collecting, curating and annotating this often ends up being a multi-person process. There are a few places you can deposit research datasets or ML models like Zenodo (https://zenodo.org/) and get a DOI. In our sector Darwin Core is one of the key data standards for describing data and when we need to extend the standard we try and use existing ones (such as those on schema.org or

With regards to longevity, when we're planning our infrastructure and how we're actually going to store our digital data we have to think in the long, long term (100+ years), much as we have to when considering how to store the physical specimens. Currently we manage our own data centre which stores all our collections and image data but we’re exploring cloud options currently. In terms of how we store the actual data, we try to map to well known standards and ontologies (such as Darwin Core - https://dwc.tdwg.org/) to ensure our data is interoperable with others and can be managed using community standards. On the Data Portal specifically, we use a versioning system to make sure that data is available long term, even if it’s been changed since it was originally made public (this happens regularly as taxonomists love to reclassify specimens!). This is particularly important when users cite our data using DOIs which should be persistent and always available.