ntast VS mdast

No more procrastination. No more meta-analysis, or origin story, or imposter-syndrome-driven apologies for not building the whole thing on ntast (long story). It's link time!

The real solution is semantic routing. You want to be able to define routing rules based on something like mdast (https://github.com/syntax-tree/mdast) . This is certainly on my itinerary and I've built a few hacked versions. This would allow for things like terminal rendering and quite a bit more.

I ended up using mdast to split each section into chunks between 1000 and 2000 characters. I made exceptions for tables, code blocks, blockquotes, and lists.

The actual splitting of the node into those 3 new nodes will depend on the type of AST node (see docs). If you look at other remark plugins, you might find examples or utils for doing that more easily.

contentlayer uses remark to parse the markdown in an mdast. We can now use remark plugins to modify the mdast. Then rehype comes into play and converts the mdast into a hast. rehype plugins can now modify the hast. Finally the hast is converted into react components.

remark parses and process the Markdown text, and converts it into an abstract syntax tree (AST). This tree is composed of unist nodes, which follow the Markdown Abstract Syntax Tree (mdast) structure.

Because we will be inspecting a mdast, which is a markdown abstract syntax tree built upon unist, we can take advantage of the many existing unist utilities to inspect our tree's nodes.