Qwen3.8-Flash-Next: A New Architecture, Towards Ultimate Cost-Efficiency

1. eider

   1 8 26 9.6 Rust

   experiment in performant nvfp4 inference on DGX Spark

   

   I have nvfp4 quant fitting fine in 128GB on DGX Spark, but with paging (from nVME) of the n-gram table. Resident ~80GiB for weights & context.

   On branch of https://github.com/rdaum/eider (for DGX Spark). ~12 tok/sec decode without speculative decoding (will come later)

   Still actively working on this. Prefill currently sucks. Will merge to main by end of day.

2. AppSignal

   www.appsignal.com sponsored

   AppSignal knows why the f*#k it crashed. Stop vibe-debugging. Every exception, every backtrace, grouped so you see patterns, not noise.

   

3. llama.cpp

   2 1,130 130,611 10.0 C++

   LLM inference in C/C++

   

   The largest unsloth published gguf also fits and runs just fine on a cpu-only machine with 128GB RAM, using llama-server PR 27742

   https://github.com/ggml-org/llama.cpp/pull/27742

4. llama.cpp

   3 6 1 9.9

   LLM inference in C/C++ (by alainnothere)

   

   For the impatient, I merged llama.cpp tentative branches to get it running here https://github.com/alainnothere/llama.cpp/tree/disk-cache-ev..., thing runs at 23.54 token/sec and my setup runs at high 30 the 3.8 dense 27B.

   and this is the pelican from the iq4_xs model

5. llama.cpp

   4 3 243 0.0 C++

   LLM inference in C/C++ (by unslothai)

   

6. dgx-spark-qwen38

   5 1 222 9.8 Python

   Fastest measured Qwen3.8-27B config for DGX Spark (GB10): SGLang + NVFP4 + DFlash2, 72 tok/s greedy median, 1M context, deterministic boots. Flash-Next 176B on one box. One command, everything pinned, and a local cockpit to drive it.

   

   Data point for the "fits in 128GB?" subthread: on a single DGX Spark (GB10, 128GB unified) I'm getting 34 to 42 tok/s decode with the NVFP4 checkpoint on SGLang, with the MTP head active, full native 262K context, vision and tool calls working, and prefix caching (a 30K-token conversation re-serves in 0.5s instead of 18.4s cold).

   The 51B n-gram table is the whole trick: it never needs to be resident. Three small patches mmap it from NVMe and the Linux page cache does the rest; decode only touches ~2.5KB of table per token, so measured overhead is under 3 percent. Credit to hashd1ve for the SGLang patches and blazux for proving the approach on vLLM.

   I packaged it as a one-command, sha256-pinned install (also serves the 27B as a switchable target): https://github.com/hasso5703/dgx-spark-qwen38 and wrote up the numbers and failure modes here: https://forums.developer.nvidia.com/t/qwen3-8-flash-next/381...

7. SaaSHub

   www.saashub.com sponsored

   SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

   

Suggest a related project