Why ML should be written as pipelines from the get-go

• zenml

  1 1,402 10.0 Python

  Discontinued ZenML 🙏: MLOps framework to create reproducible ML pipelines for production machine learning. [Moved to: https://github.com/zenml-io/zenml] (by maiot-io)

  

ZenML is an exercise in finding the right layer of abstraction for ML. Here, we treat pipelines as first-class citizens. This means that data scientists are exposed to pipelines directly in the framework, but not in the same manner as the data pipelines from the ETL space (Prefect, Airflow et al.). Pipelines are treated as experiments — meaning they can be compared and analyzed directly. Only when it is time to flip over to productionalization, can they be converted to classical data pipelines.

• cortex

  33 7,989 2.2 Go

  Production infrastructure for machine learning at scale

  

Technologies: Flask/FastAPI, Kubernetes, Docker, Cortex, Seldon

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• Airflow

  169 34,485 10.0 Python

  Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

  

ZenML is an exercise in finding the right layer of abstraction for ML. Here, we treat pipelines as first-class citizens. This means that data scientists are exposed to pipelines directly in the framework, but not in the same manner as the data pipelines from the ETL space (Prefect, Airflow et al.). Pipelines are treated as experiments — meaning they can be compared and analyzed directly. Only when it is time to flip over to productionalization, can they be converted to classical data pipelines.

Suggest a related project