langserve VS uvicorn

LangServe is a subproject of LangChain, which can help developers deploy LangChain's Runnable and Chain through REST API. At the same time, it also provides a client for calling Runnable deployed on the server, including multiple versions such as Python and TypeScript, and provides Playground by default for online trial after deployment.

LangServe: This enables developers to deploy and serve up LangChain applications as a REST API.

You can also take a look at LangServe (https://github.com/langchain-ai/langserve) where we try to do this in a more easy way

FastAPI & Uvicorn

This tells Heroku to run uvicorn, which is a web server implementation in Python.

FastAPI uses Uvicorn, an ASGI (Asynchronous Server Gateway Interface) web server implementation for Python.

Now, let’s run our FastAPI application using Uvicorn: uvicorn main:app --reload

Uvicorn + Starlette 8k requests/s

I switched to Hypercorn because Uvicorn currently supports HTTP/1.1 and WebSockets as mentioned at https://www.uvicorn.org

This will set up Chroma and run it as a server with uvicorn, making port 8000 accessible outside the net docker network. The command also mounts a persistent docker volume for Chroma's database, found at chroma/chroma from your project's root.

There's an open discussion under the Uvicorn repository and we prepared a repository for Reproduction GitHub Repo

FastAPI is a popular Python Web framework that developers use to create RESTful APIs. It is based on Pydantic and Python-type hints that assist in the serialization, deserialization, and validation of data. In this tutorial, we will use FastAPI to create a simple "Hello World" application. We test and run the application locally. FastAPI requires a ASGI server to run the application production such as Uvicorn.

- [3] https://github.com/encode/uvicorn/issues/527