Profiling workflows with the Amazon Genomics CLI

• gatk4-data-processing

  1 135 0.0 wdl

  Workflows for processing high-throughput sequencing data for variant discovery with GATK4 and related tools

  

To run a workflow with AGC you use commands like the following - in this case I’m running the GATK data processing workflow that is included as an example when you install AGC:

• amazon-genomics-cli

  1 146 4.4 Go
  

This makes it difficult for steps we want to do later that involve the job-id. The JSON formatter for agc is not currently used by the agc logs workflow command. I created an issue to address this to help the team prioritize it. I could probably solve this in the short term with some regular expressions, but I’d rather not. There is another way to consistently get machine parseable output.

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• awscurl

  3 707 6.8 Python

  curl-like access to AWS resources with AWS Signature Version 4 request signing.

Under the hood, agc uses the GA4GH Workflow Execution Schema (WES) to communicate with AWS resources to run workflows. Each AGC context deploys a GA4GH WES endpoint which you can send requests to directly. The endpoint requires AWS SigV4 signed requests, so the easiest way to talk to it is with a tool like awscurl, which does the request signing for you.

• amazon-genomics-cli-demos

  1 3 0.0 Jupyter Notebook

  Demos of what can be done with the Amazon Genomics CLI

  

Importantly, the method above can be used for any workflow that AGC can run so you don’t have to remember engine specific details of how to get such profiling information. This opens up the possibility of comparing the performance a workflow across workflow engines, which is something I’ll explore further in a future blog post. In the meantime, I recommend generating some plots of your own! All the Python code I used above is available in Github.

• jq

  306 25,063 0.0 C

  Discontinued Command-line JSON processor [Moved to: https://github.com/jqlang/jq] (by stedolan)

  

Note the WES endpoint Url needs to have “ga4gh/wes/v1” appended to it. We can extract this value using jq.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project