microjit-bench VS prism

As to the benchmark set, please give https://github.com/Shopify/yjit-bench a try in the future :) It has some interesting real-ish benchmarks and a warmup harness designed for JIT.

IRB type completion comes as a result of a chain of events which starts from the incredible work done by Kevin Newton (et al) to write a new canonical Ruby parser called Prism in C99 with no dependencies [1].

With Prism, you can then create tool suites like syntax_tree [2], which then leads Prettier formatters [3], a new Ruby LSP [4], which unlocks a new Ruby LSP VS Code extension [5], not to mention a laundry list of other gems like Rubocop and of course Ruby itself that will benefit from a faster and more maintainable Ruby parser.

It's a beautiful illustration of the power of questioning conventions, going back to first principles to uncover better solutions to previously solved problems, whose new solutions create new capabilities which unlocks the ability to solve new problems.

[1]: https://github.com/ruby/prism

Kevin Newton talked about Prism, a new Ruby parser. He discussed the challenges that come with parsing Ruby. He shared what's next, and what we can and should expect from our Ruby tooling in the future. He ended with an impassioned discussion about building a contributor community around a single tool.

It’s disappointing there doesn’t seem to be an easily available blog post or announcement (maybe I just didn’t find it), but the design doc lists a few motivations: https://github.com/ruby/prism/blob/main/docs/design.md

It looks like there was also a podcast interview last year that touches on the origins of the project: https://topenddevs.com/podcasts/ruby-rogues/episodes/the-new...

Reading between the lines it looks to me like this is motivated by ripper (the old parser) not being a great fit for tooling around ruby like IDE LSP integrations and such. Ripper isn’t fault tolerant (if the script has a a syntax error you don’t get a partial tree, just an exception); being implemented in ruby enough itself that it kind of depends on ruby which isn’t always convenient for integration (IDEs like vscode make plugins in JS easy, prism comes with node bindings), and maybe being enough of a crufty old code base that maintaining it and fixing those design issues was deemed impracticable.

Also worth noting if it wasn’t clear I’m pretty sure this parser is not being used or intended to be used for a ruby runtime to actually execute scripts, and that’s not what ripper was for either. This is for tooling that operates on ruby files for other purposes: syntax highlighting, linting, stuff like that.

I thought yarp was where ruby parsing was heading