[D] Productionalizing machine learning pipelines for small teams

• flyte

  31 4,761 9.8 Go

  Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.

  

The most important thing is versioning and reproducibility, https://github.com/flyteorg/flyte is an option for data pipelines but is quite complicated. As long as the path from raw data to input to the model is traceable any solution is fine.

• polyaxon

  9 3,479 8.7 Python

  MLOps Tools For Managing & Orchestrating The Machine Learning LifeCycle

  

For running experiments, http://polyaxon.com/ is a really good free open-source package that has lots of nice integrations so you can quickly run experiments in k8s but it might be overkill in some cases.

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• ploomber

  121 3,374 7.4 Python

  The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️

  

I wrote a detailed survey on this. However, I'm biased since I have a project of my own: Ploomber.

Suggest a related project