Our great sponsors
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
So just for some context, I'm coming at this from an angle of a developer, not really of a bioinformatician. First of all, linting in my opinion, is really lacking. Sure the nf-core toolset lints, but that's more so for checking that your code-base follows nf-core guidlelines. It does not flag bugs or inconsistencies with your workflows, or processes. There's really no language support (syntax, language rules, etc.) you would expect to find with other DSL's or programming languages in general. Furthermore, while the script section error messages within a process are decently informative, most errors don't really help. An example is like the one in this github issue.
Hi everyone. So I've been using nextflow for about a month or so, having developed a few pipelines and I've found the debugging experience absolutely abysmal. Although nextflow has great observability with tower, and great community support with nf-core, the uninformative error messages is souring the experience for me. There are soooo many pipeline frameworks out there, but I'm wondering if anyone has come across one similar to nextflow in offering observability, a strong community behind it, multiple executors (container image based preferably) and an awesome debugging experience? I would favor a python based approach, but not sure snakemake is the one I'm looking for.
I wrote a package for my needs called GenoPype that is basically what you're describing. https://github.com/jolespin/genopype I don't claim that it's better software than Snakemake or Nextflow, I just developed something that works for exactly what I need it to do in creating log files, intermediate directories, checkpoints, i/o checks, acceptable return codes, etc.
It really depends on your use cases, I've seen a lot of those tools that lock you into a certain syntax, framework or weird language (for instance Groovy). If you'd like to use core python or Jupyter notebooks I'd recommend Ploomber, the community support is really strong, there's an emphasis on observability and you can deploy it on any executor like Slurm, AWS Batch or Airflow. In addition, there's a free managed compute (cloud edition) where you can run certain bioinformatics flows like Alphafold or Cripresso2