Our great sponsors
-
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
As of early March 2022 this is the new CI pipeline that we use here at ZenML and the feedback from my colleagues -- fellow engineers -- has been very positive overall. I am sure there will be tweaks, changes and refactorings in the future, but for now, this feels Zen.
Caching is a powerful way to speed up repeating processes. We run 'poetry install' in one such process that is necessary for each aspect of our CI pipeline. We didn't want to commit the 'poetry.lock' file to ensure we would keep ZenML compatible with the newest versions of packages that we are integrating with and test regardless of the state on the developer's machine. On average the Poetry installation would take between 10-15 minutes for each cell on the OS-Python Version matrix.
We ran into one such case at ZenML. One of our integrations is Kubeflow Pipelines. This integration needs to spin up a cluster of pods using k3d in order to deploy Kubeflow Pipelines. Then all of our other integration tests are run on this cluster. This whole process takes about 1 hour to run and so it is not something we want running for each push on each PR. Instead, we want to have some control over when it is appropriate.
Related posts
- Deployment automation for ML projects of all shapes and sizes
- A tutorial on how to handle prediction uncertainty in production systems, by using Bayesian inference and probabilistic programs
- [P] [D] How are you approaching prediction uncertainty in ML systems?
- Bodywork: MLOps tool for deploying ML projects to Kubernetes
- Tool for mapping executable Python modules to Kubernetes deployments