fetch-event-source VS marked

* `text/event-stream` relies on a modified version of [Azure's fetch-event-source](https://github.com/Azure/fetch-event-source). They handle errors and chunking better than I was. New version has been tested out to 4 billion fragments rendered on a single page with no errors.

The EventSource Web API for handling SSEs (built into most modern browsers) unfortunately supports only GET requests, which was quite limiting when a POST request with larger body JSON data was required. As an alternative, you can use the Fetch API or a ready library like Microsoft’s Fetch Event Source.

Aforementioned limitations are taken care with a specialised SSE client (inspired by the awesome @microsoft/fetch-event-source) and two separate connection modes: the HTTP/1 safe "single connection mode" that uses a single SSE connection for receiving events with separate HTTP requests dictating the behaviour, and the HTTP/2+ "distinct connections mode" that uses distinct SSE connections for each GraphQL operation, accommodating the parameters in the request itself.

Next, install gray-matter to extract metadata from the front matter of markdown files, and marked to convert the markdown files to HTML:

📑 Marked Markdown parser. Use it to create your own markdown editor.

Vrite SDK provides a few built-in input and output transformers. These are functions, with standardized signatures to process the content from and into Vrite. In this case, gfmInputTransformer is essentially a GitHub Flavored Markdown parser, using Marked.js under the hood.

Another contestant in this realm is Bright[1]. It runs entirely on the server and doesn't increase bundle size as seen here[2]. Regarding parsing speed tree-sitter is without a doubt performant since it is written in Rust, but I don't have any problems "parsing on every keystroke" with a setup containing Marked[3], highlight.js[4] and a sanitizer. I did however experience performance issues with other Markdown parser libraries than Marked.

[1]: https://bright.codehike.org/

[2]: https://aihelperbot.com/test-suite

[3]: https://github.com/markedjs/marked

[4]: https://highlightjs.org/

The project uses markedJS to convert markdown into HTML, this is their GitHub page.

To handle pasting block Markdown content like this, I had to tap into ProseMirror and implement a custom mechanism (though somewhat based on TipTap’s paste rules), detecting starting and ending points of the blocks and parsing them with Marked.js.

I am using marked for markdown parsing together with marked-highlighting to handle syntax highlighting and everything is working as it should.

I'm trying to render user input using SvelteMarkdown (that uses marked).

GPT API returns data in markdown format. You can parse it using a Markdown library and string manipulation. On Electron app I developed https://jhappsproducts.gumroad.com/l/gpteverywhere, I used https://github.com/markedjs/marked and a code syntax highlighting package to display code blocks. And used JavaScript string manipulation to detect when code blocks start and end so I could add COPY/SAVE buttons to the blocks. I hope this helps, and happy coding! :)

Again, with streaming enabled, you’ll now receive new tokens as soon as they’re available. Given that OpenAI’s API uses Markdown in its response format, a full message will need to be put together from the incoming tokens and parsed to HTML, as accepted by the replaceContent function. For this purpose, I’ve used the Marked.js parser.