Python – Create large ZIP archives without memory inflation

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

Our great sponsors
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • WorkOS - The modern identity platform for B2B SaaS
  • SaaSHub - Software Alternatives and Reviews
  • ZipZap

    zip file I/O library for iOS, macOS and tvOS

  • Streaming zip file is going to have issues when compression ratio is not 0 for some implementations (such as ZipFileStream from Java / Android). I fixed this a few years ago for ZipZap: https://github.com/pixelglow/ZipZap/pull/155

    Particularly, the local header (per file) is written before the file content, and you cannot get accurate compressed size prior to actual compression. Central file header is at the end of the file, so there is no problem to get accurate compressed size, but some zip reader implementations (such as ZipFileStream) cannot use central file header (because it is at the end), and will use the wrong compressed size.

  • zipstream

    Simple python library for streaming zip files which are created dynamically, without using any temporary files

  • How does it compares to https://github.com/kbbdy/zipstream?

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
  • zipfly

    Python Zip Stream

  • The code is perplexing. Why do you need to say int('0x1', 16) and int('0x2', 16)? Why not just write 0x1 and 0x2? Or just plain 1 and 2?

    I'm also perplexed by the goal as this seems to just call zipfile.write under the hood, which already does what this tries to accomplish?

    [0] https://github.com/BuzonIO/zipfly/blob/master/zipfly/zipfly....

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts