[D] Should I go with Prefect, Argo or Flyte for Model Training and ML workflow orchestration?

This page summarizes the projects mentioned and recommended in the original post on /r/MachineLearning

InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured
  • Airflow

    Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

  • If components/ecosystem are important factors in your evaluation, you could also consider Apache Airflow. It has been around for longer, and has a very large set of components/plugins, both official and 3rd party.

  • dagster

    An orchestration platform for the development, production, and observation of data assets.

  • You could also consider Dagster, which aims to improve Apache Airflow's shortcomings. Also, take a look at MyMLOps, where you can get a quick overview of open-source orchestration tools.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
  • prefect-deployment-patterns

    Code examples showing flow deployment to various types of infrastructure

  • Have you used infrastructure blocks in Prefect? You could easily build a block for Sagemaker deploying infrastructure for the flow running with GPUs, then run other flow in a local process, yet another one as Kubernetes job, Docker container, ECS task, AWS batch, etc. Super easy to set up, even from the UI or from CI/CD. There are a bunch of templates and examples here: https://github.com/anna-geller/prefect-deployment-patterns

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • New to data orchestration? Start here.

    2 projects | dev.to | 2 Jun 2021
  • AI Strategy Guide: How to Scale AI Across Your Business

    4 projects | dev.to | 11 May 2024
  • StackStorm – IFTTT for Ops

    7 projects | news.ycombinator.com | 5 Nov 2023
  • A High-Performance, Java-Based Orchestration Platform

    1 project | /r/java | 11 Oct 2023
  • Kestra is an open-source data orchestration platform for complex workflows

    1 project | news.ycombinator.com | 4 Oct 2023