mosec VS gunicorn

Mosec - A high-performance serving framework for ML models, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine. Simple and faster alternative to NVIDIA Triton.

- Yes C++ would be better, but you can try mosec. It has a Python interface and helps you handle all the difficult things about Python multiprocessing. The web service part is implemented in Rust thus it's fast enough for machine learning services.

Contribute to open source projects: Modelz is built on top of envd, mosec, modelz-llm and many other open source projects. If you're interested in contributing to these projects, you can check out their GitHub repositories and start contributing.

You could first create the image with the framework you like (e.g. bentoml or https://github.com/mosecorg/mosec for light weight).

If you're interested in machine learning model serving, can check mosec: https://github.com/mosecorg/mosec

During the development of a machine learning serving project Mosec, I used a lot of multiprocessing to make it more efficient. I want to share some experiences and some researches related to Python multiprocessing.

That's a good example. I have met the same situation before. I have created a discussion in GitHub to track the DAG progress.

I'm hoping so – gunicorn has a long-open pull request that would fix `--reuse-port`, which currently does nothing

https://github.com/benoitc/gunicorn/pull/2938

However, I'm curious about the best way to deploy, specifically with regard to WSGI. I've tried using the raw eventlet WSGI server (`eventlet.wsgi.server(eventlet.listen(("", 8000)), application)`). I then start it with `python manage.py runserver`. This has worked okay, but I'm unsure about how scalable it is. It seems like the standard stack is Django + Gunicorn + NGINX. Based on `python-socketio` documentation, this should be possible. I tried django + eventlet + gunicorn, but it seems like gunicorn a) [doesn't play nice with eventlet](https://github.com/benoitc/gunicorn/pull/2581) and b) only supports one worker. Gevent + Gunicorn doesn't have this bug, but still only supports one worker. Also, I'm not sure how actively maintained gevent is. So I'm not sure how scalable either Gunicorn + eventlet or Gunicorn + geventlet is as a WSGI server. So I'm not sure if Gunicorn is my best bet, or if it's too limited.

For building your API, I recommend using the Flask library. It is very beginner-friendly, and you will be able to build a simple API in a matter of minutes! Keep in mind that, for a more serious project, you should definitely use something like gunicorn to run you API as a production server.

Interesting looks like it might actually be a python bug. Somehow just changing from sys.exit(0) -> os._exit(0) apparently fixes it.

https://github.com/benoitc/gunicorn/pull/2820

The cool thing is that you can easily migrate your WSGI- application such as Flask, Django, or Gunicorn to AWS.

Yeah, that's kind of the impression I was getting. I stumbled across a github issue for gunicorn along these lines.

Taking a glimpse at gunicorn's code it looks like they pretty much all do the same: 2. seems to be creating a wsgi app using django's internals, and 3. uses 2.