[P] H5Records : Store large datasets in one single files with index access

This page summarizes the projects mentioned and recommended in the original post on /r/MachineLearning
Pytorch dialogue hdf5-library Machine Learning hdf5

InfluxDB
Our great sponsors
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • WorkOS - The modern identity platform for B2B SaaS
  • SaaSHub - Software Alternatives and Reviews
Our great sponsors

  • H5Record

    Large dataset storage format for Pytorch

    Maybe you can checkout the video branch where sequence of images is supported. You can checkout the test/test_img_seq.py example to use it.

  • indexed-file

    Simple class aimed to provide a simple indexed file scheme.

    Similarly, I've made what I called indexed-file which essentially can act as file-like object and that you can access to entries via their indices. As for the format of the data (JSON, pickle) and the compression, it's left to the user.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  • DialoGPT

    Large-scale pretraining for dialogue

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts