[D] Should I go with Prefect, Argo or Flyte for Model Training and ML workflow orchestration?

• Airflow

  73 28,905 10.0 Python

  Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

  

  If components/ecosystem are important factors in your evaluation, you could also consider Apache Airflow. It has been around for longer, and has a very large set of components/plugins, both official and 3rd party.

• dagster

  20 6,364 10.0 Python

  An orchestration platform for the development, production, and observation of data assets.

  

  You could also consider Dagster, which aims to improve Apache Airflow's shortcomings. Also, take a look at MyMLOps, where you can get a quick overview of open-source orchestration tools.

• Sonar

  www.sonarsource.com

  sponsored

  Write Clean Python Code. Always.. Sonar helps you commit clean code every time. With over 225 unique rules to find Python bugs, code smells & vulnerabilities, Sonar finds the issues while you focus on the work.

  

• prefect-deployment-patterns

  1 48 10.0 Python

  Code examples showing flow deployment to various types of infrastructure

  Have you used infrastructure blocks in Prefect? You could easily build a block for Sagemaker deploying infrastructure for the flow running with GPUs, then run other flow in a local process, yet another one as Kubernetes job, Docker container, ECS task, AWS batch, etc. Super easy to set up, even from the UI or from CI/CD. There are a bunch of templates and examples here: https://github.com/anna-geller/prefect-deployment-patterns

Suggest a related project