Top 18 kubeflow Open-Source Projects

• kubeflow

  3 13,759 8.3 TypeScript

  Machine Learning Toolkit for Kubernetes

  

• pipelines

  2 3,457 9.8 Python

  Machine Learning Pipelines for Kubeflow

  

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• kserve

  3 3,111 9.4 Python

  Standardized Serverless ML Inference Platform on Kubernetes

  

• elyra

  2 1,780 6.3 Python

  Elyra extends JupyterLab with an AI centric approach.

  

• training-operator

  2 1,477 8.9 Go

  Distributed ML Training and Fine-Tuning on Kubernetes

  

Project mention: [Discussion] Guidance on training ML models on Kubernetes | /r/MachineLearning | 2023-05-24

You could use https://github.com/kubeflow/training-operator directly.

• couler

  1 891 5.2 Python

  Unified Interface for Constructing and Managing Workflows on different workflow engines, such as Argo Workflows, Tekton Pipelines, and Apache Airflow.

  

Project mention: (Not) to Write a Pipeline | news.ycombinator.com | 2023-06-27

author seems to be describing the kind of patterns you might make with https://argoproj.github.io/argo-workflows/ . or see for example https://github.com/couler-proj/couler , which is an sdk for describing tasks that may be submitted to different workflow engines on the backend.

it's a little confusing to me that the author seems to object to "pipelines" and then equate them with messaging-queues. for me at least, "pipeline" vs "workflow-engine" vs "scheduler" are all basically synonyms in this context. those things may or may not be implemented with a message-queue for persistence, but the persistence layer itself is usually below the level of abstraction that $current_problem is really concerned with. like the author says, eventually you have to track state/timestamps/logs, but you get that from the beginning if you start with a workflow engine.

i agree with author that message-queues should not be a knee-jerk response to most problems because the LoE for edge-cases/observability/monitoring is huge. (maybe reach for a queue only if you may actually overwhelm whatever the "scheduler" can handle.) but don't build the scheduler from scratch either.. use argowf, kubeflow, or a more opinionated framework like airflow, mlflow, databricks, aws lamda or step-functions. all/any of these should have config or api that's robust enough to express rate-limit/retry stuff. almost any of these choices has better observability out-of-the-box than you can easily get from a queue. but most importantly.. they provide idioms for handling failure that data-science folks and junior devs can work with. the right way to structure code is just much more clear and things like structuring messages/events, subclassing workers, repeating/retrying tasks, is just harder to mess up.

• arena

  1 710 8.3 Go

  A CLI for Kubeflow. (by kubeflow)

  

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• data-on-eks

  2 517 9.3 HCL

  DoEKS is a tool to build, deploy and scale Data & ML Platforms on Amazon EKS

  

• mpi-operator

  1 403 7.3 Go

  Kubernetes Operator for MPI-based applications (distributed training, HPC, etc.)

  

• deployKF

  2 275 8.6 Shell

  deployKF builds machine learning platforms on Kubernetes. We combine the best of Kubeflow, Airflow†, and MLflow† into a complete platform.

  

Project mention: We are excited to announce the release of deployKF! It's an open-source project that makes it actually easy to deploy and maintain Kubeflow (and more) on Kubernetes. | /r/Kubeflow | 2023-08-12

• hongbomiao.com

  1 202 10.0 HCL

  🦋 A personal research and development (R&D) lab that facilitates the sharing of knowledge.

• Kubernetes-Guide

  3 188 5.8 Go

  Kubernetes Guide. Learn all about Kubernetes monitoring, networking, and containers. Whether you're running Kubernetes Locally or in the Cloud ( Azure, AWS, and GCP).

• awesome-kubeflow

  1 182 6.3

  A curated list of awesome projects and resources related to Kubeflow (a CNCF incubating project)

• kfp-tekton

  0 163 8.4 TypeScript

  Kubeflow Pipelines on Tekton

  

• community

  1 152 7.7 Jsonnet

  Information about the Kubeflow community including proposals and governance information. (by kubeflow)

  

Project mention: How can we read variables from file and use them in promql? | /r/grafana | 2023-05-29

However I am not able to figure it out, how can I feed the string xyz_stack_1 to grafana. I have setup docker-compose.yaml file to start up all the containers. The configuration is done through prometheus.yaml, grafana.ini, dashboards.yaml and datasources.yaml

• Fast-Kubeflow

  7 72 4.0 Python

  This repo covers Kubeflow Environment with LABs: Kubeflow GUI, Jupyter Notebooks on pods, Kubeflow Pipelines, Experiments, KALE, KATIB (AutoML: Hyperparameter Tuning), KFServe (Model Serving), Training Operators (Distributed Training), Projects, etc.

• soopervisor

  3 43 5.0 Python

  ☁️ Export Ploomber pipelines to Kubernetes (Argo), Airflow, AWS Batch, SLURM, and Kubeflow.

  

• kubeflow-bootstrap

  1 20 9.3 Shell

  🪐 1-click Kubeflow using ArgoCD

  

Project mention: Show HN: Kubeflow's Missing Helm Chart | news.ycombinator.com | 2024-03-19

Kubeflow is an ML platform like Sagemaker or Databricks that you can self-host in a Kubernetes cluster.

Installing/deploying it is as complicated as it sounds, but we've put together an infrastructure project that lets you '1-click' install it even in tiny environments.

The GH repo (also linked in blog) allows you to start Kubeflow in a codespace or small device using a docker container -- this is both good for trying it out and developing it into your own internal ML platform.

https://github.com/treebeardtech/kubeflow-helm

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

kubeflow related posts

• We are excited to announce the release of deployKF! It's an open-source project that makes it actually easy to deploy and maintain Kubeflow (and more) on Kubernetes.

  1 project | /r/Kubeflow | 12 Aug 2023

• awslabs/data-on-eks: DoEKS is a tool to build, deploy and scale Data Platforms on Amazon EKS

  1 project | /r/devopsish | 6 Apr 2023

• Fast-Kubeflow: Kubeflow Tutorial, Sample Usage Scenarios (Howto: Hands-on LAB)

  3 projects | /r/mlops | 4 Jan 2023

• Fast-Kubeflow: Kubeflow Tutorial, Sample Usage Scenarios (Howto: Hands-on LAB)

  3 projects | /r/dataengineering | 4 Jan 2023

• Fast-Kubeflow: Kubeflow Tutorial, Sample Usage Scenarios (Howto: Hands-on LAB)

  1 project | /r/programming | 4 Jan 2023

• Fast-Kubeflow: Kubeflow Tutorial, Sample Usage Scenarios (Howto: Hands-on LAB)

  3 projects | /r/deeplearning | 4 Jan 2023

• Fast-Kubeflow: Kubeflow Tutorial, Sample Usage Scenarios (Howto: Hands-on LAB)

  3 projects | /r/kubernetes | 28 Dec 2022
• A note from our sponsor - SaaSHub
  www.saashub.com | 21 May 2024

  SaaSHub helps you find the best software and product alternatives Learn more →

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com