Multi-model serving options

• jina

  126 20,978 8.7 Python

  ☁️ Build multimodal AI applications with cloud-native stack

  

  Jina let’s you serve all of your models through the same Gateway while deploying them as individual microservices. You can also tie your models together in a pipeline if needed. Also some nice ML focussed features such as dynamic batching.

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• server

  27 8,123 9.4 Python

  The Triton Inference Server provides an optimized cloud and edge inferencing solution. (by triton-inference-server)

  

  You've already mentioned Seldon Core which is well worth looking at but if you're just after the raw multi-model serving aspect rather than a fully-fledged deployment framework you should maybe take a look at the individual inference servers: Triton Inference Server and MLServer both support multi-model serving for a wide variety of frameworks (and custom python models). MLServer might be a better option as it has an MLFlow runtime but only you will be able to decide that. There also might be other inference servers that do MMS that I'm not aware of.

• MLServer

  4 692 9.5 Python

  An inference server for your machine learning models, including support for multiple frameworks, multi-model serving and more

  

  You've already mentioned Seldon Core which is well worth looking at but if you're just after the raw multi-model serving aspect rather than a fully-fledged deployment framework you should maybe take a look at the individual inference servers: Triton Inference Server and MLServer both support multi-model serving for a wide variety of frameworks (and custom python models). MLServer might be a better option as it has an MLFlow runtime but only you will be able to decide that. There also might be other inference servers that do MMS that I'm not aware of.

Suggest a related project