openai-partial-stream
fetch-event-source
| openai-partial-stream | fetch-event-source | |
|---|---|---|
| 2 | 5 | |
| 118 | 2,413 | |
| 0.0% | 0.0% | |
| 6.7 | 0.0 | |
| about 1 year ago | over 1 year ago | |
| HTML | TypeScript | |
| MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
openai-partial-stream
- Parse JSON Stream Fast for Realtime UI in LLM AI App
-
How streaming LLM APIs work
When you ask to return JSON data using streaming, you will notice that the response is incomplete and unparseable by JSON libraries, resulting in malformed errors. You will have to wait for the entire stream to complete.
To solve this problem I tried to define a spec and built a lib for it:
- https://github.com/st3w4r/openai-partial-stream/tree/main
- https://github.com/st3w4r/openai-partial-stream/blob/main/sp...
fetch-event-source
-
We Chose Long Polling over WebSockets
Not sure about the default `EventSource` object in JavaScript, but the Microsoft polyfill that I use (https://github.com/Azure/fetch-event-source) supports `POST` and there's an option `openWhenHidden` which controls how it reacts to users tabbing away.
-
How streaming LLM APIs work
EventSource is really really limited. However, you can instead use Fetch via something https://github.com/Azure/fetch-event-source to consume SSEs.
-
Datastar v0.12.0 (Htmx+Alpine Alternative)
* `text/event-stream` relies on a modified version of [Azure's fetch-event-source](https://github.com/Azure/fetch-event-source). They handle errors and chunking better than I was. New version has been tested out to 4 billion fragments rendered on a single page with no errors.
-
How I put ChatGPT into a WYSIWYG editor
The EventSource Web API for handling SSEs (built into most modern browsers) unfortunately supports only GET requests, which was quite limiting when a POST request with larger body JSON data was required. As an alternative, you can use the Fetch API or a ready library like Microsoft’s Fetch Event Source.
-
GraphQL over SSE (Server-Sent Events)
Aforementioned limitations are taken care with a specialised SSE client (inspired by the awesome @microsoft/fetch-event-source) and two separate connection modes: the HTTP/1 safe "single connection mode" that uses a single SSE connection for receiving events with separate HTTP requests dictating the behaviour, and the HTTP/2+ "distinct connections mode" that uses distinct SSE connections for each GraphQL operation, accommodating the parameters in the request itself.
What are some alternatives?
jiter - Fast iterable JSON parser.
better-sse - ⬆ Dead simple, dependency-less, spec-compliant server-sent events implementation written in TypeScript
Web-of-Thoughts - Web of thoughts is a web app that uses the OpenAI API to generate mindmaps based on a given prompt. These mindmaps will first generate initial thoughts, link those thoughts, critique those thoughts and finally try to resolve these thoughts into a final answer.
graphql-sse - Zero-dependency, HTTP/1 safe, simple, GraphQL over Server-Sent Events Protocol server and client.
LLMEvaluation - A comprehensive guide to LLM evaluation methods designed to assist in identifying the most suitable evaluation techniques for various use cases, promote the adoption of best practices in LLM assessment, and critically assess the effectiveness of these evaluation methods.