A 10 year old Xeon is all you need (for 26B-A4B MTP Drafters without GPU)

1. llama.cpp

   1 1,032 115,929 10.0 C++

   LLM inference in C/C++

   

   llama.cpp includes a benchmarking tool called llama-bench https://github.com/ggml-org/llama.cpp/blob/master/tools/llam...

   ik_llama includes llama-sweep-bench https://github.com/ikawrakow/ik_llama.cpp/blob/main/examples...

   When comparing hardware, the output of these tools is very helpful to let others put it into context. The post says the output is "reading speed" but knowing the prefill and token generation speeds would be a lot more helpful.

2. SaaSHub

   www.saashub.com featured

   SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

   

3. ik_llama.cpp

   2 10 658 9.7 C++

   Discontinued llama.cpp fork with additional SOTA quants and improved performance [GET https://api.github.com/repos/ikawrakow/ik_llama.cpp: 404 - Not Found // See: https://docs.github.com/rest]

   

   llama.cpp includes a benchmarking tool called llama-bench https://github.com/ggml-org/llama.cpp/blob/master/tools/llam...

   ik_llama includes llama-sweep-bench https://github.com/ikawrakow/ik_llama.cpp/blob/main/examples...

   When comparing hardware, the output of these tools is very helpful to let others put it into context. The post says the output is "reading speed" but knowing the prefill and token generation speeds would be a lot more helpful.

Suggest a related project