polars_resources VS prism

Paul Reece gave an introduction to the Polars library. This started with an introduction to the Series and Data Frame data structures. Paul then walked through data cleaning operations available in Polars. He shared a repository of resources and code samples at the end of his session.

I am most excited about the parser change, previously discussed here:

https://news.ycombinator.com/item?id=36310130 - Rewriting the Ruby parser (2023-06-13, 176 comments)

I remember being taught to use yacc in our compiler course because "writing it by hand is too hard". But looks like Ruby joins the growing list of languages that have hand-written parsers, apparently working with generated parsers turned out to be even harder in the long run.

That said, replacing a ~16k line parse.y[1] with a 22k line prism.c[2] is a pretty bold move.

[1] https://github.com/ruby/ruby/blob/master/parse.y

[2] https://github.com/ruby/prism/blob/main/src/prism.c

IRB type completion comes as a result of a chain of events which starts from the incredible work done by Kevin Newton (et al) to write a new canonical Ruby parser called Prism in C99 with no dependencies [1].

With Prism, you can then create tool suites like syntax_tree [2], which then leads Prettier formatters [3], a new Ruby LSP [4], which unlocks a new Ruby LSP VS Code extension [5], not to mention a laundry list of other gems like Rubocop and of course Ruby itself that will benefit from a faster and more maintainable Ruby parser.

It's a beautiful illustration of the power of questioning conventions, going back to first principles to uncover better solutions to previously solved problems, whose new solutions create new capabilities which unlocks the ability to solve new problems.

[1]: https://github.com/ruby/prism

Kevin Newton talked about Prism, a new Ruby parser. He discussed the challenges that come with parsing Ruby. He shared what's next, and what we can and should expect from our Ruby tooling in the future. He ended with an impassioned discussion about building a contributor community around a single tool.

It’s disappointing there doesn’t seem to be an easily available blog post or announcement (maybe I just didn’t find it), but the design doc lists a few motivations: https://github.com/ruby/prism/blob/main/docs/design.md

It looks like there was also a podcast interview last year that touches on the origins of the project: https://topenddevs.com/podcasts/ruby-rogues/episodes/the-new...

Reading between the lines it looks to me like this is motivated by ripper (the old parser) not being a great fit for tooling around ruby like IDE LSP integrations and such. Ripper isn’t fault tolerant (if the script has a a syntax error you don’t get a partial tree, just an exception); being implemented in ruby enough itself that it kind of depends on ruby which isn’t always convenient for integration (IDEs like vscode make plugins in JS easy, prism comes with node bindings), and maybe being enough of a crufty old code base that maintaining it and fixing those design issues was deemed impracticable.

Also worth noting if it wasn’t clear I’m pretty sure this parser is not being used or intended to be used for a ruby runtime to actually execute scripts, and that’s not what ripper was for either. This is for tooling that operates on ruby files for other purposes: syntax highlighting, linting, stuff like that.

I thought yarp was where ruby parsing was heading