Pigz: A parallel implementation of gzip for multi-core machines

• pigz

  9 2,549 3.4 C

  A parallel implementation of gzip for modern multi-processor, multi-core machines.

The bit I found most interesting was actually:

https://github.com/madler/pigz/blob/master/try.h

https://github.com/madler/pigz/blob/master/try.c

which implements try/catch for C99.

• htslib

  7 772 9.0 C

  C library for high-throughput sequencing data formats

  

There is another nice multi-core gzip based library called BGZF[1]. It is commonly used in bioinformatics. BGZF has the added advantage that it is block compressed with built in indexing method to permit seeking in compressed files.

[1] https://github.com/samtools/htslib

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• mpifileutils

  4 160 5.1 C

  File utilities designed for scalability and performance.

  

If you ever run into the limitations of a single machine, dbz2 is also a fun little app for this sort of thing. You can run it across multiple machines and it'll automatically balance the workload across them.

https://github.com/hpc/mpifileutils/blob/master/man/dbz2.1

• PostgreSQL

  405 14,734 10.0 C

  Mirror of the official PostgreSQL GIT repository. Note that this is just a *mirror* - we don't work with pull requests on github. To contribute, please see https://wiki.postgresql.org/wiki/Submitting_a_Patch

  

Postgres uses a similar custom implementation of try/catch [1].

[1] https://github.com/postgres/postgres/blob/master/src/include...

Suggest a related project