tree-sitter-cpp VS tree-sitter-bash

Grammar page (https://github.com/tree-sitter/tree-sitter-cpp) reference two documents at the very end:

- Hyperlinked C++ BNF Grammar (https://alx71hub.github.io/hcb/)

- EBNF Syntax: C++ (ISO/IEC 14882:1998(E)) https://www.externsoft.ch/download/cpp-iso.html

The second doc has a year in the title, so it's ancient af.

(use-package treesit :ensure nil :custom ;; Some stuff taken from here: https://robbmann.io/posts/emacs-treesit-auto/ (treesit-extra-load-path '("/usr/lib64/")) (treesit-language-source-alist '((bash . ("https://github.com/tree-sitter/tree-sitter-bash")) (c . ("https://github.com/tree-sitter/tree-sitter-c")) (c++ . ("https://github.com/tree-sitter/tree-sitter-cpp")) (csharp . ("https://github.com/tree-sitter/tree-sitter-c-sharp")) (css . ("https://github.com/tree-sitter/tree-sitter-css")) (elixir ("https://github.com/elixir-lang/tree-sitter-elixir")) (html . ("https://github.com/tree-sitter/tree-sitter-html")) (java . ("https://github.com/tree-sitter/tree-sitter-java")) (javascript . ("https://github.com/tree-sitter/tree-sitter-javascript")) (json . ("https://github.com/tree-sitter/tree-sitter-json")) (lua . ("https://github.com/Azganoth/tree-sitter-lua")) (makefile . ("https://github.com/alemuller/tree-sitter-make")) (org . ("https://github.com/milisims/tree-sitter-org")) (python . ("https://github.com/tree-sitter/tree-sitter-python")) (tsx . ("https://github.com/tree-sitter/tree-sitter-typescript" "master" "tsx/src")) (typescript . ("https://github.com/tree-sitter/tree-sitter-typescript" "master" "typescript/src")) (ruby . ("https://github.com/tree-sitter/tree-sitter-ruby")) (rust . ("https://github.com/tree-sitter/tree-sitter-rust")) (sql . ("https://github.com/m-novikov/tree-sitter-sql")) (toml . ("https://github.com/tree-sitter/tree-sitter-toml")) (yaml . ("https://github.com/ikatyang/tree-sitter-yaml")))) (major-mode-remap-alist '((c-mode . c-ts-mode) (c++-mode . c++-ts-mode) (csharp-mode . csharp-ts-mode) (css-mode . css-ts-mode) (html-mode . html-ts-mode) (java-mode . java-ts-mode) (js-mode . js-ts-mode) (json-mode . json-ts-mode) (makefile-mode . makefile-ts-mode) ;; (org-mode . org-ts-mode) ;; not mature yet (python-mode . python-ts-mode) (typescript-mode . typescript-ts-mode) (ruby-mode . ruby-ts-mode) (rust-mode . rust-ts-mode) (toml-mode . toml-ts-mode) (yaml-mode . yaml-ts-mode))) (treesit-auto-fallback-alist '((toml-ts-mode . conf-toml-mode) (typescript-ts-mode . nil) (tsx-ts-mode . nil))) (treesit-font-lock-settings t) (treesit-simple-indent t) (treesit-defun-type-regexp t)) (use-package treesit-auto :demand t :config (setq treesit-auto-install t) (global-treesit-auto-mode))

The feature isn't working well yet on C and C++. If I recall correctly it's based on Tree-Sitter[1] parsing, and there are still too many bugs in corresponding grammars - tree-sitter-c[2] and tree-sitter-cpp[3]. Hopefully, it will be greatly improved in the future as the share of the existing and newly written code in C and C++ is quite significant.

[1] https://tree-sitter.github.io/tree-sitter/

[2] https://github.com/tree-sitter/tree-sitter-c/issues

[3] https://github.com/tree-sitter/tree-sitter-cpp/issues

There's treesitter-cpp parser that works to some extent already. However, for complex language like c++, I think it would be better to use lsp's semantic highlighting instead

C++ is currently unsupported, but there's a good tree-sitter implementation I can add: https://github.com/tree-sitter/tree-sitter-cpp

(1) Zulip Chat - https://zulip.com/ - seems to be reasonably popular, but more people should know about it

I’ve been using it for over 5 years now [1], and it’s as good as ever. It’s way faster than any other chat app I’ve used. It has a good UI and conversation model. It has a simple and functional API that lets me curl threads and write blog posts based on them.

(only problem is that I Ctrl-+ in my browser to make the font bigger – I think it’s too dense for most people)

(2) re2c regex to state machine compiler - https://re2c.org

A gem from the 90’s, which people have done a great job maintaining and improving (getting Go and Rust target support in the last few years). I started using it in 2016, and used for a new program a few months ago. I came to the conclusion that it should have been built into C, because C has shitty string processing – and Ken Thompson both invented C AND brought regular languages to computing !!

In comparison, treesitter lexers are very low level, fiddly, and error prone. I recently saw dozens of ad hoc fixes to the tree-sitter-bash lexer, which is unsurprising if you look at the structure of the code (manually crawling through backslashes and braces in C).

https://github.com/tree-sitter/tree-sitter-bash/blob/master/...

These fixes are definitely appreciated, but I think it indicates a problem with the model itself.

(based on https://lobste.rs/s/endspx/software_you_are_thankful_for#c_y...)

[1] https://www.oilshell.org/blog/2018/04/26.html

(use-package treesit :ensure nil :custom ;; Some stuff taken from here: https://robbmann.io/posts/emacs-treesit-auto/ (treesit-extra-load-path '("/usr/lib64/")) (treesit-language-source-alist '((bash . ("https://github.com/tree-sitter/tree-sitter-bash")) (c . ("https://github.com/tree-sitter/tree-sitter-c")) (c++ . ("https://github.com/tree-sitter/tree-sitter-cpp")) (csharp . ("https://github.com/tree-sitter/tree-sitter-c-sharp")) (css . ("https://github.com/tree-sitter/tree-sitter-css")) (elixir ("https://github.com/elixir-lang/tree-sitter-elixir")) (html . ("https://github.com/tree-sitter/tree-sitter-html")) (java . ("https://github.com/tree-sitter/tree-sitter-java")) (javascript . ("https://github.com/tree-sitter/tree-sitter-javascript")) (json . ("https://github.com/tree-sitter/tree-sitter-json")) (lua . ("https://github.com/Azganoth/tree-sitter-lua")) (makefile . ("https://github.com/alemuller/tree-sitter-make")) (org . ("https://github.com/milisims/tree-sitter-org")) (python . ("https://github.com/tree-sitter/tree-sitter-python")) (tsx . ("https://github.com/tree-sitter/tree-sitter-typescript" "master" "tsx/src")) (typescript . ("https://github.com/tree-sitter/tree-sitter-typescript" "master" "typescript/src")) (ruby . ("https://github.com/tree-sitter/tree-sitter-ruby")) (rust . ("https://github.com/tree-sitter/tree-sitter-rust")) (sql . ("https://github.com/m-novikov/tree-sitter-sql")) (toml . ("https://github.com/tree-sitter/tree-sitter-toml")) (yaml . ("https://github.com/ikatyang/tree-sitter-yaml")))) (major-mode-remap-alist '((c-mode . c-ts-mode) (c++-mode . c++-ts-mode) (csharp-mode . csharp-ts-mode) (css-mode . css-ts-mode) (html-mode . html-ts-mode) (java-mode . java-ts-mode) (js-mode . js-ts-mode) (json-mode . json-ts-mode) (makefile-mode . makefile-ts-mode) ;; (org-mode . org-ts-mode) ;; not mature yet (python-mode . python-ts-mode) (typescript-mode . typescript-ts-mode) (ruby-mode . ruby-ts-mode) (rust-mode . rust-ts-mode) (toml-mode . toml-ts-mode) (yaml-mode . yaml-ts-mode))) (treesit-auto-fallback-alist '((toml-ts-mode . conf-toml-mode) (typescript-ts-mode . nil) (tsx-ts-mode . nil))) (treesit-font-lock-settings t) (treesit-simple-indent t) (treesit-defun-type-regexp t)) (use-package treesit-auto :demand t :config (setq treesit-auto-install t) (global-treesit-auto-mode))

I ran into this issue as well a few weeks ago, there's already a few issues for it: https://github.com/tree-sitter/tree-sitter-bash/issues/151

All 3 of these are linked to under "Related Tools".

Those projects use a pretty basic tokenizer

https://github.com/luckymarmot/Paw-cURLImporter/blob/master/...

Insomnia and Postman use commander.js for their bash parsing which looks like it just splits on space:

https://github.com/tj/commander.js/blob/02a124c7d58dbae2ef11...

and only support a few of curl's options. Curlconverter uses tree-sitter's Bash grammar https://github.com/tree-sitter/tree-sitter-bash/ and tries to at least correctly discard all of curl's options and any options that curl used to have.