BentoML VS haystack

Link to GitHub -->

For 2), I am aware of a few options. Triton inference server is an obvious one as is the ‘transformer-deploy’ version from LDS. My only reservation here is that they require the model compilation or are architecture specific. I am aware of others like Bento, Ray serving and TorchServe. Ideally I would have something that allows any (PyTorch model) to be used without the extra compilation effort (or at least optionally) and has some convenience things like ease of use, easy to deploy, easy to host multiple models and can perform some dynamic batching. Anyway, I am really interested to hear people's experience here as I know there are now quite a few options! Any help is appreciated! Disclaimer - I have no affiliation or are connected in any way with the libraries or companies listed here. These are just the ones I know of. Thanks in advance.

In this category I’m a big fan of https://github.com/bentoml/BentoML

What I like about it is their idiomatic developer experience. It reminds me of other Pythonic frameworks like Flask and Django in a good way.

I have no affiliation with them whatsoever, just an admirer.

Github Page: https://github.com/bentoml/BentoML

Elastic License 2: https://github.com/bentoml/bentoctl/blob/v0.3.1/LICENSE.md which also applies to their Yatai kubernetes thing, but strangely not (yet?) to the similarly named repo which is Apache-2: https://github.com/bentoml/BentoML/blob/main/LICENSE

Using a general-purpose framework such as FastAPI involves writing a lot of boilerplate code just to get your API endpoint up and running. If deploying a model for a demo is the only thing you are interested in and you do not mind losing some flexibility, you might want to use a specialized serving framework instead. One example is BentoML, which will allow you to get an optimized serving endpoint for your model up and running much faster and with less overhead than a generic web framework. Framework-specific serving solutions such as Tensorflow Serving and TorchServe typically offer optimized performance but can only be used to serve models trained using Tensorflow or PyTorch, respectively.

BentoML - BentoML is a flexible, high-performance framework for serving, managing, and deploying machine learning models.

Also mlflow is not that optimized because it doesnt microbatch like torchserve/tfserving/bentoml. https://github.com/bentoml/BentoML/tree/master/benchmark

People like to be on the AI bandwagon, but to have good AI models, you need good LLM (large language models). Welcome to Haystack, it's an end-to-end LLM framework that allows you to build applications powered by LLMs, Transformer models, vector search and more. The latest version is a rewrite of the Haystack framework, and includes a new package, powerful pipelines, customisable components, prompt templating, and more. Read all about them in the Haystack release notes.

View on GitHub

4. Haystack by Deepset | Github | tutorial

Haystack for production. We cannot afford breaking changes in our production apps. Its stable, documentation is excellent and did I mention its' STABLE!??

I recently conducted some experiments with Llama2 and Haystack (https://github.com/deepset-ai/haystack), the NLP/LLM framework.

The notebook can be helpful for those trying to load Llama2 on Colab.

1) Installed Transformers from the main branch (and other libraries)

there is an alternative that is production-grade - deepset haystack https://haystack.deepset.ai/

p.s. i am contributor so there could be bias

You could use text embedding in vector store using Haystack depending on the level of context that is needed, the hardware resources, and your comfort with software development

banks is a side project by my colleague Massi. We work on the open source NLP framework Haystack together (https://github.com/deepset-ai/haystack) and prompt templating is obviously a 'hot' topic and it's something to design these types of projects around. Massi had this concept in mind to use Jinja (or any other templating language for that matter) as the basis, and clearly decided to take it for a spin. I personally am in love with this style of prompt templating and think it makes the whole process insanely flexible and easy to read.