[D] Should I go with Prefect, Argo or Flyte for Model Training and ML workflow orchestration?

This page summarizes the projects mentioned and recommended in the original post on reddit.com/r/MachineLearning

Our great sponsors
  • Sonar - Write Clean Python Code. Always.
  • InfluxDB - Build time-series-based applications quickly and at scale.
  • SaaSHub - Software Alternatives and Reviews
  • Airflow

    Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

    If components/ecosystem are important factors in your evaluation, you could also consider Apache Airflow. It has been around for longer, and has a very large set of components/plugins, both official and 3rd party.

  • dagster

    An orchestration platform for the development, production, and observation of data assets.

    You could also consider Dagster, which aims to improve Apache Airflow's shortcomings. Also, take a look at MyMLOps, where you can get a quick overview of open-source orchestration tools.

  • Sonar

    Write Clean Python Code. Always.. Sonar helps you commit clean code every time. With over 225 unique rules to find Python bugs, code smells & vulnerabilities, Sonar finds the issues while you focus on the work.

  • prefect-deployment-patterns

    Code examples showing flow deployment to various types of infrastructure

    Have you used infrastructure blocks in Prefect? You could easily build a block for Sagemaker deploying infrastructure for the flow running with GPUs, then run other flow in a local process, yet another one as Kubernetes job, Docker container, ECS task, AWS batch, etc. Super easy to set up, even from the UI or from CI/CD. There are a bunch of templates and examples here: https://github.com/anna-geller/prefect-deployment-patterns

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts