mpifileutils VS htslib

If you ever run into the limitations of a single machine, dbz2 is also a fun little app for this sort of thing. You can run it across multiple machines and it'll automatically balance the workload across them.

https://github.com/hpc/mpifileutils/blob/master/man/dbz2.1

If you want something that scales horizontally, dcmp from https://github.com/hpc/mpifileutils is an option.

There is another nice multi-core gzip based library called BGZF[1]. It is commonly used in bioinformatics. BGZF has the added advantage that it is block compressed with built in indexing method to permit seeking in compressed files.

[1] https://github.com/samtools/htslib

In contrast, processing can start immediately and only transfer what is necessary if tooling can read bytes of data directly from Amazon S3. Tools based on htslib can do this, so you can run something like:

Another idea is add in reading cloud data natively e.g. htslib which samtools relies on can read s3 directly. https://github.com/samtools/htslib

Greetings! I'm a bioinformatics software dev in San Francisco and I'm looking for others interested in working on a high-performance genetic data analysis project. The project is in C, using https://github.com/samtools/htslib.

I'm working on a bioinformatics-related binding called ruby-htslib. htslib makes heavy use of bit fields throughout the library, so supporting bit fields is inevitable.