htslib
mpifileutils
| htslib | mpifileutils | |
|---|---|---|
| 7 | 4 | |
| 775 | 160 | |
| 1.5% | 0.6% | |
| 8.9 | 5.1 | |
| 5 days ago | 25 days ago | |
| C | C | |
| GNU General Public License v3.0 or later | BSD 3-clause "New" or "Revised" License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
htslib
- Gentoo -Os vs -O3 application startup time?
- Does anyone know of a repository for actual genetic data?
-
Pigz: A parallel implementation of gzip for multi-core machines
There is another nice multi-core gzip based library called BGZF[1]. It is commonly used in bioinformatics. BGZF has the added advantage that it is block compressed with built in indexing method to permit seeking in compressed files.
[1] https://github.com/samtools/htslib
-
Tips for scalable workflows on AWS
In contrast, processing can start immediately and only transfer what is necessary if tooling can read bytes of data directly from Amazon S3. Tools based on htslib can do this, so you can run something like:
-
Software Development Project
Another idea is add in reading cloud data natively e.g. htslib which samtools relies on can read s3 directly. https://github.com/samtools/htslib
-
Interested in Bioinformatics / C pair-programming opportunity / learning / portfolio project.
Greetings! I'm a bioinformatics software dev in San Francisco and I'm looking for others interested in working on a high-performance genetic data analysis project. The project is in C, using https://github.com/samtools/htslib.
-
ffi-bitfield
I'm working on a bioinformatics-related binding called ruby-htslib. htslib makes heavy use of bit fields throughout the library, so supporting bit fields is inevitable.
mpifileutils
-
Pigz: A parallel implementation of gzip for multi-core machines
If you ever run into the limitations of a single machine, dbz2 is also a fun little app for this sort of thing. You can run it across multiple machines and it'll automatically balance the workload across them.
https://github.com/hpc/mpifileutils/blob/master/man/dbz2.1
- MpiFileUtils: File utilities designed for scalability and performance
-
Go Find Duplicates: blazingly-fast simple-to-use tool to find duplicate files
If you want something that scales horizontally, dcmp from https://github.com/hpc/mpifileutils is an option.
- You can list a directory containing 8M files, but not with ls
What are some alternatives?
genozip - A modern compressor for genomic files (FASTQ, SAM/BAM/CRAM, VCF, FASTA, GFF/GTF/GVF, 23andMe...), up to 5x better than gzip and faster too
fclones - Efficient Duplicate File Finder
seqtk - Toolkit for processing sequences in FASTA/Q formats
rmlint - Extremely fast tool to remove duplicates and other lint from your filesystem
bwa-mem2 - The next version of bwa-mem
pigz - A parallel implementation of gzip for modern multi-processor, multi-core machines.
cyvcf2 - cython + htslib == fast VCF and BCF processing
duphard - A simple utility to detect duplicate files and replace them with hard links.
aws-genomics-workflows - Genomics Workflows on AWS
coreutils - Enhancements to the GNU coreutils (especiall head)
libdna - ♥ Essential Functions for DNA Manipulation
jdupes - A powerful duplicate file finder and an enhanced fork of 'fdupes'.