MLServer VS Lean and Mean Docker containers

You've already mentioned Seldon Core which is well worth looking at but if you're just after the raw multi-model serving aspect rather than a fully-fledged deployment framework you should maybe take a look at the individual inference servers: Triton Inference Server and MLServer both support multi-model serving for a wide variety of frameworks (and custom python models). MLServer might be a better option as it has an MLFlow runtime but only you will be able to decide that. There also might be other inference servers that do MMS that I'm not aware of.

Ever trained a new model and just wanted to use it through an API straight away? Sometimes you don't want to bother writing Flask code or containerizing your model and running it in Docker. If that sounds like you, you definitely want to check out MLServer. It's a python based inference server that recently went GA and what's really neat about it is that it's a highly-performant server designed for production environments too. That means that, by serving models locally, you are running in the exact same environment as they will be in when they get to production.

Seldon | Multiple positions | London/Cambridge UK | Onsite/Remote | Full time | seldon.io

At Seldon we are building industry leading solutions for deploying, monitoring, and explaining machine learning models. We are an open-core company with several successful open source projects like:

* https://github.com/SeldonIO/seldon-core

* https://github.com/SeldonIO/mlserver

* https://github.com/SeldonIO/alibi

* https://github.com/SeldonIO/alibi-detect

* https://github.com/SeldonIO/tempo

We are hiring for a range of positions, including software engineers(go, k8s), ml engineers (python, go), frontend engineers (js), UX designer, and product managers. All open positions can be found at https://www.seldon.io/careers/

SlimToolkit empowers developers to create better, smaller, and more secure containers without sacrificing their existing workflows. Explore the project on GitHub or visit the official website to learn more.

And if you want to make the container quickly secure without bloats, maybe give this a try https://github.com/slimtoolkit/slim

Slim.ai presents the data in a more user friendly way than many of the other tools in this post. On top of its open source SlimToolkit for identifying the contents of an image, Slim.ai uses Trivy for vulnerability scanning.

What about https://github.com/slimtoolkit/slim?

A last practice that I do not use at all and which may interest you is to use slim toolkit to keep only the useful elements in your final image.

Anyone tried using https://github.com/docker-slim/docker-slim To minify an image?..

Distroless images are better left for people with serious need for lightweight images and good Linux knowledge because they require lot of planning with the build so that they stay light and work. If you need lighter images but docker isn't your main tool and you can't afford to take hours and hours of practicing different build strategies you can check docker-slim (https://dockersl.im/). With this tool you can easily size down the images.