Server Alternatives

server reviews and mentions

• Scuda – Virtual GPU over IP
  4 projects | news.ycombinator.com | 9 Oct 2024

  This is very interesting but many of the motivations listed are far better served with alternate approaches.

  For "remote" model training there is NCCL + Deepspeed/FSDP/etc. For remote inferencing there are solutions like Triton Inference Server[0] that can do very high-performance hosting of any model for inference. For LLMs specifically there are nearly countless implementations.

  That said, the ability to use this for testing is interesting but I wonder about GPU contention and as others have noted the performance of such a solution will be terrible even with relatively high speed interconnect (100/400gb ethernet, etc).

  NCCL has been optimized to support DMA directly between network interfaces and GPUs which is of course considerably faster than solutions like this. Triton can also make use of shared memory, mmap, NCCL, MPI, etc which is one of the many tricks it uses for very performant inference - even across multiple chassis over another network layer.

  [0] - https://github.com/triton-inference-server/server

• Everything you need to know about Python 3.13 – JIT and GIL went up the hill
  2 projects | news.ycombinator.com | 28 Sep 2024

  As always, it depends a lot on what you're doing, and a lot of people are using Python for AI.

  One of the drawbacks of multi-processing versus multi-threading is that you cannot share memory (easily, cheaply) between processes. During model training, and even during inference, this becomes a problem.

  For example, imagine a high volume, low latency, synchronous computer vision inference service. If you're handling each request in a different process, then you're going to have to jump through a bunch of hoops to make this performant. For example, you'll need to use shared memory to move data around, because images are large, and sockets are slow. Another issue is that each process will need a different copy of the model in GPU memory, which is a problem in a world where GPU memory is at a premium. You could of course have a single process for the GPU processing part of your model, and then automatically batch inputs into this process, etc. etc. (and people do) but all this is just to work around the lack of proper threading support in Python.

  By the way, if anyone is struggling with these challenges today, I recommend taking a peek at nvidia's [triton](https://github.com/triton-inference-server/server) inference server, which handles a lot of these details for you. It supports things like zero-copy sharing of tensors between parts of your model running in different processes/threads and does auto-batching between requests as well. Especially auto-batching gave us big throughput increase with a minor latency penalty!

• Best LLM Inference Engines and Servers to Deploy LLMs in Production
  6 projects | dev.to | 5 Jun 2024
• FLaNK Weekly 08 Jan 2024
  41 projects | dev.to | 8 Jan 2024
• Is there any open source app to load a model and expose API like OpenAI?
  5 projects | /r/LocalLLaMA | 9 Dec 2023
• "A matching Triton is not available"
  1 project | /r/StableDiffusion | 15 Oct 2023
• best way to serve llama V2 (llama.cpp VS triton VS HF text generation inference)
  3 projects | /r/LocalLLaMA | 25 Sep 2023

  I am wondering what is the best / most cost-efficient way to serve llama V2. - llama.cpp (is it production ready or just for playing around?) ? - Triton inference server ? - HF text generation inference ?

• Triton Inference Server - Backend
  2 projects | /r/learnmachinelearning | 13 Jun 2023
• Single RTX 3080 or two RTX 3060s for deep learning inference?
  1 project | /r/computervision | 12 Apr 2023

  For inference of CNNs, memory should really not be an issue. If it is a software engineering problem, not a hardware issue. FP16 or Int8 for weights is fine and weight size won’t increase due to the high resolution. And during inference memory used for hidden layer tensors can be reused as soon as the last consumer layer has been processed. You likely using something that is designed for training for inference and that blows up the memory requirement, or if you are using TensorRT or something like that, you need to be careful to avoid that every tasks loads their own copy of the library code into the GPU. Maybe look at https://github.com/triton-inference-server/server

• Machine Learning Inference Server in Rust?
  4 projects | /r/rust | 21 Mar 2023

  I am looking for something like [Triton Inference Server](https://github.com/triton-inference-server/server) or [TFX Serving](https://www.tensorflow.org/tfx/guide/serving), but in Rust. I came across [Orkon](https://github.com/vertexclique/orkhon) which seems to be dormant and a bunch of examples off of the [Awesome-Rust-MachineLearning](https://github.com/vaaaaanquish/Awesome-Rust-MachineLearning)

• A note from our sponsor - SaaSHub
  www.saashub.com | 12 Jan 2025

  SaaSHub helps you find the best software and product alternatives Learn more →