MLflow VS onnxruntime

In search for a tool that needs comparable configuration I looked into mlflow and found this. https://github.com/mlflow/mlflow/blob/master/mlflow/environment_variables.py There they define a class _EnvironmentVariable and create many objects out of it, for any variable they need. The get method of this class is in principle a decorated os.getenv. Maybe that is something I can take as orientation.

I have been using DVC and MLflow since then DVC had only data tracking and MLflow only model tracking. I can say both are awesome now and maybe the only factor I would like to mention is that IMO, MLflow is a bit harder to learn while DVC is just a git practically.

Dumb question, how does this lib compare to other libs like MLFlow, https://mlflow.org/?

While Transformers already includes integration with MLflow, users still have to provide their own MLflow server, either locally or on a Cloud provider. And that can be a bit of a pain.

I have an old labmate who uses a similar setup with MLFlow and can endorse it.

MLflow - an open-source platform for managing your ML lifecycle. What’s great is that they also support popular Python libraries like TensorFlow, PyTorch, scikit-learn, and R.

even for people who don't have an ML background there's now a lot of very fully-featured model deployment environments that allow self-hosting (kubeflow has a good self-hosting option, as do mlflow and metaflow), handle most of the complicated stuff involved in just deploying an individual model, and work pretty well off the shelf.

Here, we’ll implement the experimentation workflow using DagsHub, Google Colab, MLflow, and data version control (DVC). We’ll focus on how to do this without diving deep into the technicalities of building or designing a workbench from scratch. Going that route might increase the complexity involved, especially if you are in the early stages of understanding ML workflows, just working on a small project, or trying to implement a proof of concept.

MLflow

I am seeing growing demand for MLflow (https://mlflow.org/) and I am seeing a lot of people looking at Databricks as commercial offering for MLflow. Alternatively, some popele are implementing something like Managing your Machine Learning lifecycle with MLflow. Therefore, I think this was on my wish list last year, but I really hope AWS announce a Managed MLFlow Service. I know version 2.X is too new but at least 1.X would be great start.

https://news.ycombinator.com/item?id=35696031 ... TIL about wonnx: https://github.com/webonnx/wonnx#in-the-browser-using-webgpu...

microsoft/onnxruntime: https://github.com/microsoft/onnxruntime

Apache/arrow has language-portable Tensors for cpp: https://arrow.apache.org/docs/cpp/api/tensor.html and rust: https://docs.rs/arrow/latest/arrow/tensor/struct.Tensor.html and Python: https://arrow.apache.org/docs/python/api/tables.html#tensors https://arrow.apache.org/docs/python/generated/pyarrow.Tenso...

Fwiw it looks like the llama.cpp Tensor is from ggml, for which there are CUDA and OpenCL implementations (but not yet ROCm, or a WebGPU shim for use with emscripten transpilation to WASM): https://github.com/ggerganov/llama.cpp/blob/master/ggml.h

Are the recommendable ways to cast e.g. arrow Tensors to pytorch/tensorflow?

FWIU, Rust has a better compilation to WASM; and that's probably faster than already-compiled-to-JS/ES TensorFlow + WebGPU.

What's a fair benchmark?

Before continue, ensure that the ONNX runtime installed on your operating system, because the library that integrated to the Rust package may not work correctly. To install it, you can download the archive for your operating system from here, extract and copy contents of "lib" subfolder to the system libraries path of your operating system.

I can't seem to figure if the PR for the WebGPU backend for onnxruntime is supposed to land in a 1.14 release, a 1.15 release, has already landed, isn't yet scheduled to land, etc? https://github.com/microsoft/onnxruntime/pull/14579

https://github.com/microsoft/onnxruntime/releases I don't see it in any releases yet?

https://github.com/microsoft/onnxruntime/milestone/4 I don't see it in the upcoming milestone.

I don't see any examples or docs that go with it

https://github.com/microsoft/onnxruntime/wiki/Upcoming-Relea... This seems to be out of date

https://github.com/microsoft/onnxruntime/tree/rel-1.15.0 I do see the js/webgpu work merged into here so I guess it'll be released in 1.15.0

https://onnxruntime.ai/docs/reference/releases-servicing.htm...

> Official releases of ONNX Runtime are managed by the core ONNX Runtime team. A new release is published approximately every quarter, and the upcoming roadmap can be found here.

ONNX Runtime v1.14.0 was Feb 10th

Contribute to Microsoft's ONNX runtime, it's helping accelerate non-Nvidia hardware for all sorts of ML goodness: https://onnxruntime.ai/

Gonna respond here and correct both comments

>Some context for those who aren't in the loop: ONNX Runtime (https://onnxruntime.ai/) is a standardization format for AI models.

It's just an IR, one of many - every framework has its own.

>Nowadays, it's extremely easy to export models in the ONNX format, especially language models with tools like Hugging Face transformers which have special workflows for it.

Meh it's poorly supported by both PyTorch and TF. Why support Microsoft's IR when you have your own.

>probably the most performant ML runtime at this point.

Not even by a long-shot - first party compilers are generally faster because of smoother interop but even amongst third-party you have TRT and TVM. TBH I have no idea what anyone uses ONNX for these days (legacy?).

Some context for those who aren't in the loop: ONNX Runtime (https://onnxruntime.ai/) is a standarization format for AI models. Nowadays, it's extremely easy to export models in the ONNX format, especially language models with tools like Hugging Face transformers which have special workflows for it.

ONNX support in the browser was lacking and limited to CPU, but with a WebGPU backend it may now finally be feasible to run models in the browser, which opens up interesting oppertunities.

PyInstaller seemed like the most maintained and developed tool to freeze python script into an executable, so I went with it. As expected, the freezed interface with the model was gigabytes large, so I had to figure out how to squeeze this. Fortunately, Onnx worked wonders and packaged the model into an inference only state, so I could throw away the Pytorch and Torchtext dependencies when freezing with Pyinstaller.Now the size of the executable with the model was 43MB instead of 4GB.

[3] https://github.com/microsoft/onnxruntime/issues/11695#issuec...

In this blog post, we focus on invoking the OCR service. However, if you want to learn how to deploy your machine learning model with Cape, you can check this example, where we deploy an image classification model with the ONNX runtime.