[D] Should I go with Prefect, Argo or Flyte for Model Training and ML workflow orchestration?

• Airflow

  169 34,627 10.0 Python

  Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

  

If components/ecosystem are important factors in your evaluation, you could also consider Apache Airflow. It has been around for longer, and has a very large set of components/plugins, both official and 3rd party.

• dagster

  46 10,274 10.0 Python

  An orchestration platform for the development, production, and observation of data assets.

  

You could also consider Dagster, which aims to improve Apache Airflow's shortcomings. Also, take a look at MyMLOps, where you can get a quick overview of open-source orchestration tools.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• prefect-deployment-patterns

  1 93 0.0 Python

  Code examples showing flow deployment to various types of infrastructure

Have you used infrastructure blocks in Prefect? You could easily build a block for Sagemaker deploying infrastructure for the flow running with GPUs, then run other flow in a local process, yet another one as Kubernetes job, Docker container, ECS task, AWS batch, etc. Super easy to set up, even from the UI or from CI/CD. There are a bunch of templates and examples here: https://github.com/anna-geller/prefect-deployment-patterns

Suggest a related project