Incremental Parsing in Go

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

Our great sponsors
  • InfluxDB - Build time-series-based applications quickly and at scale.
  • SonarLint - Clean code begins in your IDE with SonarLint
  • SaaSHub - Software Alternatives and Reviews
  • ntfy

    Send push notifications to your phone or desktop using PUT/POST

    Interesting read. Thank you for sharing. I always found parsers fascinating and mystical. It seems like these parser functions (which i think are analogous to what Rob Pike calls state functions) are a common way to do parsing, though i know very little about it. I especially found the combinators intriguing, though I don't care much for the functional programming syntax in a language like Go.

    Anyway, thanks for sharing.

    Tangentially, I wrote a little mini parser [0] of my own for my side project. It is inspired by Rob Pike's talk on parsers [1]. It doesn't use state functions, but instead just uses the call stack to keep track of where we are.

    [0] https://github.com/binwiederhier/ntfy/blob/main/server/actio...

    [1] https://www.youtube.com/watch?v=HxaD_trXwRE and https://go.dev/src/text/template/parse/lex.go

  • aretext

    Minimalist text editor with vim-compatible key bindings.

    Look at the current Makefile:

    https://github.com/aretext/aretext/blob/main/Makefile

    Build is literally a `go build ...` and install is `go install`. Adding any other language to the mix would make this a polyglot project and not be "equally easy to set up". The other question is, do both parsers exist? In this write-up they point to tree-sitter as a possibility which is a JS program that produces C code. This would be viable, but here's the author's take:

    > I considered integrating tree-sitter, an incremental parsing library with parsers for many existing languages. However, running JavaScript to generate parsers and linking to a C library would have greatly complicated the build process. Today, aretext can be built on almost any platform using a single go install command. I’ve had users install aretext on ARM laptops, FreeBSD servers, Chromebooks, and Android phones. To maintain portability, I wanted a pure Go implementation.

    So this wasn't some casual decision, but something they at least considered long enough to describe here.

    And the parsing library itself is only around 1200 lines total (comments, blanks, and code). The parsers for each language add a lot more, of course, but should be roughly equivalent given the same library and interface. I imagine that if this project really takes off and performance becomes a real problem they can do the rewrite at that point. Right now, the code works, seems to work fast enough for its author and primary users, and it's trivial to install on any platform supported by Go. So yes, it would have been a premature optimization to complicate the build process, probably reduce the number of supported platforms (or greatly increase the effort to support the same number of platforms), just to have a slightly faster parser.

  • InfluxDB

    Build time-series-based applications quickly and at scale.. InfluxDB is the Time Series Platform where developers build real-time applications for analytics, IoT and cloud-native services. Easy to start, it is available in the cloud or on-premises.

  • scc

    Sloc, Cloc and Code: scc is a very fast accurate code counter with complexity calculations and COCOMO estimates written in pure Go

    I've seen some real world example where Go was as fast or faster than Rust for CPU / io intensive task.

    Go is a fast language even with a GC.

    https://github.com/boyter/scc/#performance

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts