adam VS Biopython

FYI: ADAM seems to do that

We presented using Parquet formats for bioinformatics 2012/13-ish at the Bioinformatics Open Source Conference (BOSC) and got laughed out of the place.

While using Apache Spark for bioinformatics [0] never really took off, I still think Parquet formats for bioinformatics [1] is a good idea, especially with DuckDB, Apache Arrow, etc. supporting Parquet out of the box.

0 - https://github.com/bigdatagenomics/adam

1 - https://github.com/bigdatagenomics/bdg-formats

We're here, still plugging along.

ADAM is a genomics analysis platform with specialized file formats built using Apache Avro, Apache Spark, and Apache Parquet. Apache 2 licensed.

https://github.com/bigdatagenomics/adam

Usearch will do all-against-all comparisons, cluster sequences, and produce alignments for each cluster. You can set the clustering threshold (proportion of residues identical). The alignments are in fasta format, which is pretty standard. If all you want is basic similarity it might be easiest to just write something that calculates normalized Hamming distances (typically called p-distances in the molecular evolution literature) between pairs of sequences. I suspect the biopython fasta reader (you can install biopython from https://biopython.org/) will be good enough.

Yes, the computing tool for biological computation.

Beside that their core lectures to non-computer scientists are public (survey), workshops by software carpentry move around the globe. Maybe your intent to seed hands-on knowledge is in similar tune before heading for biopython, bioperl, bioawk. It doesn't hurt to tap into resources initially written for non-labrats either, e.g. about regular expressions by programming historian.

Well, what language do you want to learn? What is your background so far? Assuming it is more on the side of biology, software carpentry's Python may eventually lead to biopython? Though there equally is a chance for AWK (Hack the planet's text! and bioawk...

I think you mind find something of a community around BioPython, which might be helpful. Just looking at the capabilities will probably be instructive as well.

Python is the main programming language I use nowadays. In particular numpy and pandas are of course extremely useful. I also use biopython package - a collection of software tools for biological computation written in Python by an international group of researchers and developers.