awesome-mlops VS kserve

I'm in the same boat. Data scientist shifting towards ML engineering-MLOps. The guide seems quite complete. I am also doing the ML DevOps engineer, which has end-to-end projects and has been helpful so far. I also feel that most ML engineers will be Mlops too, as most companies will not distinguish between the two, so I try to focus on this part. Here is a quite comprehensive list of resources: https://github.com/visenger/awesome-mlops

Good Reference: https://github.com/visenger/awesome-mlops (The Link above has so many Guides, It's insane) https://madewithml.com/

I recently did a deep-dive into MLOps for a client, and I've found that https://ml-ops.org/ offers a great overview. Some topics are a bit too "zoomed out", but they still touch on most important aspects of building an end-to-end product. I found it a great starting point when doing research, and picking and choosing some key points from each section + some google helped a lot. Give it a look, you'll probably find some useful things in there.

MLOps

She made mentions to ML-Ops and MLFlow including Vertex AI the GCP implementation. I will post the video as soon as it is available. In the meantime, you can enjoy any other talk from Nerea Luis

From your post, you seem to be trained for data science for physics modeling, so I'd recommend to get started with https://ml-ops.org/ and for the data engineering part, I found this https://github.com/andkret/Cookbook open source cookbook to be invaluable.

Inference servers essentially turn a model running on CPU and/or GPU hardware into a microservice.

Many of them support the kserve API standard[0] that supports everything from model loading/unloading to (of course) inference requests across models, versions, frameworks, etc.

So in the case of Triton[1] you can have any number of different TensorFlow/torch/tensorrt/onnx/etc models, versions, and variants. You can have one or more Triton instances running on hardware with access to local GPUs (for this example). Then you can put standard REST and or grpc load balancers (or whatever you want) in front of them, hit them via another API, whatever.

Now all your applications need to do to perform inference is do an HTTP POST (or use a client[2]) for model input, Triton runs it on a GPU (or CPU if you want), and you get back whatever the model output is.

Not a sales pitch for Triton but it (like some others) can also do things like dynamic batching with QoS parameters, automated model profiling and performance optimization[3], really granular control over resources, response caching, python middleware for application/biz logic, accelerated media processing with Nvidia DALI, all kinds of stuff.

[0] - https://github.com/kserve/kserve

[1] - https://github.com/triton-inference-server/server

[2] - https://github.com/triton-inference-server/client

[3] - https://github.com/triton-inference-server/model_analyzer

Kubeflow has multiple components: central dashboard, Kubeflow Notebooks to manage Jupyter notebooks, Kubeflow Pipelines for building and deploying portable, scalable machine learning (ML) workflows based on Docker containers, KF Serving for model serving (apparently superseded by KServe), Katib for hyperparameter tuning and model search, and training operators such as TFJob for training TF models on Kubernetes.

If you can run on Kubernetes then KFServing is an open source solution that allows for GPU inference and is built upon Knative to allow scale to zero for GPU based inference. From release 0.5 it also has capabilities for multi-model serving as a alpha feature to allow multiple models to share the same server (and via NVIDIA Triton the same GPU).