Rapid-MLX VS prompt-api

💡 💡 Make the fallback cheap to operate. The whole point of using Nano on the supported path is reduced cost. If your fallback is GPT-5.5 at $5/M tokens, you've moved the bill, not deleted it. Two patterns work well: (1) route the fallback to a smaller hosted model (Haiku, Gemini Flash, Mistral Small) that matches Nano's "short summarization" sweet spot; (2) for Mac users specifically, run Rapid-MLX as your /api/llm endpoint — Apple Silicon owners get on-device performance via your server's Mac, not theirs. Same thesis as our DeepClaude guide: the harness is one product, the model is another, and you can swap them.

> Large-context requests auto-route to a cloud LLM (GPT-5, Claude, etc.) when local prefill would be slow. Routing based on new tokens after cache hit. --cloud-model openai/gpt-5 --cloud-threshold 20000

https://github.com/raullenchai/Rapid-MLX

I've verified this end-to-end with structured output (output_type=BaseModel), streaming, multi-turn conversations, and multi-tool workflows. Test suite here.

pip install git+https://github.com/raullenchai/vllm-mlx.git

The technical name for this is the Prompt API (official spec, Chrome docs). It's been in Chrome's bowels since version 138 — initially behind a flag, now also available as an Origin Trial for production sites. The big news is that a critical mass of developers just figured out it's there, and the demos hitting HN every week (Decaf rewriting comments, Subtitle Insights translating YouTube live, the side-panel UI above) are no longer "look what's possible" — they're "I shipped it last weekend."

I guess this is the Dunning-Kruger effect everyone talks about!

To understand just enough to regurgitate what happened, but miss why it happened... and then assume someone who's pointing at the much more relevant why is just plain wrong.

Because the why requires actually understanding of things like developer mindshare rather than regurgitating search results.

-

The hint I'll leave if you're willing to consider maybe you don't know everything ever... look at who's feedback is being promoted when Chrome wants to do obviously unpopular things on the web: https://github.com/webmachinelearning/prompt-api/blob/main/R...

https://github.com/mozilla/standards-positions/issues/1213

And model for yourself what happens if the developer interest exceeds vendor refusal in magnitude, so Google just ships the thing, without a feature flag, to a massive percentage of the web-going world.

https://github.com/webmachinelearning/prompt-api

https://developer.chrome.com/docs/ai/built-in :

> Standardization effort: We're working to standardize all of these APIs for cross-browser compatibility.

> The Language Detector API and Translator API have been adopted by the W3C WebML Working Group. We've asked Mozilla and WebKit for their standards positions.

> The Summarizer API, Writer API, and Rewriter API have also been adopted by the W3C WebML Working Group. We've asked asked Mozilla and WebKit for their standards positions.

webmachinelearning/webmcp:

I don't know if Vivaldi supports the new Prompt API [0] that Grammit uses to run the local LLM.

As far as I know, the only browsers supporting it currently are Chrome [1] and Edge [2].

[0] https://github.com/webmachinelearning/prompt-api

[1] https://developer.chrome.com/docs/extensions/ai/prompt-api

[2] https://learn.microsoft.com/en-us/microsoft-edge/web-platfor...

There is a Prompt API in development that's available in both Chrome and Edge to give pages to a local LLM. Chrome extensions have access to it and I believe websites can request access as part of an origin trial.

The model is fully managed by the browser. It's currently the Gemini Nano model on Chrome, and they are testing a version of the Gemma 3n model in beta channels. Edge uses phi-4-mini.

You can learn more here: https://github.com/webmachinelearning/prompt-api

I wish they would copy this one https://github.com/webmachinelearning/prompt-api and include a few options for small self-hosted LLMs using WebGPU or some built-in accelerated AI for Firefox.