Working with data files too large for RAM

• h5py

  5 2,003 8.8 Python

  HDF5 for Python -- The h5py package is a Pythonic interface to the HDF5 binary data format.

  

There's some good answers here, but another option I haven't seen suggested: Convert your txt file to HDF5 (Regardless if you follow my approach here, you should really consider converting your data to anything but a txt file). There's a nice library for working with it in python called h5py. The HDF format is designed specifically with working with very large sets of data (it even has compression options), often scientific in nature, but it's not a database. As far as how this fixes the specific issue you you've described, you can utilize numpy slicing to load one chunk your data at a time. Here's a stackoverflow answer which discusses a solution.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project