dotfiles VS tree-sitter

find-invalid-utf8: walks a directory tree and prints invalid UTF-8 in files using nice hex escapes with coloring. This is useful for honing on in where invalid UTF-8 occur. You have a good bet of finding some by checking out any moderately sized repository of code. The Linux kernel used to have some. The Mozilla repo does. The CPython repo does too. This is why it's important for CLI tools to deal with invalid UTF-8 gracefully in some way.

Yeah it's great! I used it to implement a little utility to convert a subset of SMS/MMS messages from an XML backup to a more readable plain text version: https://github.com/BurntSushi/dotfiles/blob/0b075d79a6ff8812a1f48a37b9858938b3eadc58/bin/rust/searchsms/main.rs

My dotfiles: https://github.com/BurntSushi/dotfiles

Here are some selected scripts folks might find interesting.

Here's my backup script that I use to encrypt my data at rest before shipping it off to s3. Runs every night and is idempotent. I use s3 lifecycle rules to keep data around for 6 months after it's deleted. That way, if my script goofs, I can recover: https://github.com/BurntSushi/dotfiles/blob/2f58eedf3b7f7dae...

I have so many machines running Archlinux that I wrote my own little helper for installing Arch that configures the machine in the way I expect: https://github.com/BurntSushi/dotfiles/blob/2f58eedf3b7f7dae...

A tiny little script to recover the git commit message you spent 10 minutes writing, but "lost" because something caused the actual commit to fail (like a gpg error): https://github.com/BurntSushi/dotfiles/blob/2f58eedf3b7f7dae...

A script that produces a GitHub permalink from just a file path and some optional file numbers. Pass --clip to put it on your clipboard: https://github.com/BurntSushi/dotfiles/blob/2f58eedf3b7f7dae... --- I use it with this vimscript function to quickly generate permalinks from my editor: https://github.com/BurntSushi/dotfiles/blob/2f58eedf3b7f7dae...

A wrapper around 'gh' (previously: 'hub') that lets you run 'hub-rollup pr-number' and it will automatically rebase that PR into your current branch. This is useful for creating one big "rollup" branch of a bunch of PRs. It is idempotent. https://github.com/BurntSushi/dotfiles/blob/2f58eedf3b7f7dae...

Scale a video without having to memorize ffmpeg's crazy CLI syntax: https://github.com/BurntSushi/dotfiles/blob/2f58eedf3b7f7dae...

Under X11, copy something to your clipboard using the best tool available: https://github.com/BurntSushi/dotfiles/blob/2f58eedf3b7f7dae...

Otherwise, the most common "breakage" I get is when I forget to update in a while. Used to be a mostly non-issue until package signing became a thing. Now I get lots of signing errors when I update. When that happens, I run this script and it usually fixes things: https://github.com/BurntSushi/dotfiles/blob/2f58eedf3b7f7dae7f0a7cea1a641459e25e5d07/bin/pacman-fix-keys

Sadly, at work, I can't use my own bespoke setup any more. I'm effectively forced to use GNOME, which has the same braindead support for multiple monitors that KDE has. These days, I just gave up on multiple monitors and work on my laptop in my sunroom at home. Back when I was in the office and using multiple monitors, it pretty much sucked, but I did write a little script that lets me at least move focus between monitors using my keyboard while respecting the window stacking order. I had intended to expand it with more stuff, but then COVID hit, my sunroom became my work environment and multiple monitors became a luxury I didn't care about any more. For work, anyway.

The zsh builtin with a custom TIMEFMT: https://github.com/BurntSushi/dotfiles/blob/965383e6eeb0bad4...

That's what I used to do, but I switched to Josh's strategy a couple years ago.[1] It doesn't "blow up" git-status. If some new piece of software creates a new directory with a bunch of random stuff in it, git-status will just show you the directory since it is is untracked, and not everything in it.

[1]: https://github.com/BurntSushi/dotfiles/blob/965383e6eeb0bad4...

But I did that enough that I've scripted most of it: https://github.com/BurntSushi/dotfiles/blob/caed7921e48d112cc8932b33b81013fcbbcb2e08/bin/arch-install

For docs and license, see: https://github.com/BurntSushi/dotfiles/tree/master/.doc

This is why I kept mine private for so many years too. A bit ago, I cleaned house and published them: https://github.com/BurntSushi/dotfiles

From the tree-sitter side, I am tracking https://github.com/tree-sitter/tree-sitter/issues/1942

We rely on the Rust bindings from the main tree-sitter repository on the C library. Loading and matching the queries on the source file is done by this library, and so out of scope for Topiary.

Is tree-sitter really slower than TextMate grammars? Some benchmarks indicate that this isn't really the case [1]. On the other hand, breaking parse trees is a real issue, because the error-recovery in tree-sitter is pretty rudimentary [2][3], but as you said, it's not an issue for Shiki.

Several TextMate grammars suffer from inaccuracy bugs, and issues of maintainability. Perhaps the biggest hindrance in the adoption of tree-sitter, is that the most popular editor, VSCode, still doesn't support it.

[1]: https://github.com/microsoft/vscode/pull/161479

[2]: https://github.com/tree-sitter/tree-sitter/issues/1870

[3]: https://github.com/tree-sitter/tree-sitter/issues/224

Tree-sitter is going to be a lot slower to load: https://github.com/tree-sitter/tree-sitter/issues/1942.

It's ok for things like editor but using it in a SSG for example is hard when just loading the syntaxes will take multiple times the time it takes to render the whole thing with a regex based highlighter.

u/lewis6991 I think the biggest performance gain was made by tree-sitter itself: https://github.com/tree-sitter/tree-sitter/pull/2085

I asked on the repo's discussions but responses are limited and not explanatory (I'm not shaming anyone here, discussions aren't a place for detailed how-tos and documentations anyway).

The following discussion here. TS query cannot be incremental, that is why I regard it as design fault.

Hi, so I've recently decided to give Neovim yet another try, this time using some predefined plugins with kickstart.nvim, for syntax it uses tree-sitter.

Another thing that comes to my mind is Tree-sitter. In my understanding it generates syntax trees that can not only be used for syntax highlighting but also for building advanced text manipulation in plug-ins. Maybe it can save you from implementing the parsing in the first place. There is already a Latex-Grammar and a rust create to use treesitter.

What do you think about treesitter? https://github.com/tree-sitter/tree-sitter

The idea is to sync changes in the text to a tree structure, then have all the structure manipulation functions built on top of it. See the gif here for a visual representation: https://github.com/nvim-treesitter/playground