seqtk VS htslib

seqtk: A lightweight and versatile tool for processing FASTQ and FASTA files. https://github.com/lh3/seqtk

Perhaps seqtk could be helpful https://github.com/lh3/seqtk

You could also check out tools specialized for FASTA processing like https://github.com/shenwei356/seqkit and https://github.com/lh3/seqtk

If I were to analyze I would use a random fastq sampler like Seqtk and bring all your samples to a lowest read depth of your 27 libraries although I wouldn't analyze a library with less than 2mil reads. 5 mil is fine for differential, you can obviously get more reads and probably received more information but increasing read depth may plateau.

It's a little toolkit written by one of the Illuminati of the Bioinformatics world: seqtk on GH

I recommend installing seqtk, which makes this easy. Of course sed/awk/perl are theoretically entirely sufficient but why make life more difficult than necessary?

Most SRA files are grouped by projects. On a basic level for something common like RNA-seq you will have replicates of the control and treatment/diseased samples. Each file (i.e. sample) contains raw sequencing reads, usually millions per sample. You could randomly subsample the sequencing reads very easily using many tools (common choice is https://github.com/lh3/seqtk). There is no way you are going to assemble an animal genome with MB file sizes (for example the human genome itself is already over 3GB in size). You should probably look for bacterial or viral DNA samples and subset those to an appropriate size.

There is another nice multi-core gzip based library called BGZF[1]. It is commonly used in bioinformatics. BGZF has the added advantage that it is block compressed with built in indexing method to permit seeking in compressed files.

[1] https://github.com/samtools/htslib

In contrast, processing can start immediately and only transfer what is necessary if tooling can read bytes of data directly from Amazon S3. Tools based on htslib can do this, so you can run something like:

Another idea is add in reading cloud data natively e.g. htslib which samtools relies on can read s3 directly. https://github.com/samtools/htslib

Greetings! I'm a bioinformatics software dev in San Francisco and I'm looking for others interested in working on a high-performance genetic data analysis project. The project is in C, using https://github.com/samtools/htslib.

I'm working on a bioinformatics-related binding called ruby-htslib. htslib makes heavy use of bit fields throughout the library, so supporting bit fields is inevitable.