refex VS pre-commit

As someone who worked on a similar tool (https://github.com/ssbr/refex/tree/main/refex/fix/fixers, I did a bunch of the work to prep this for open-sourcing, though I think all my contributions are hidden behind the "Google-internal" anonymization), having auto-applied or auto-appliable fixers like this is super useful.

They can be auto-applied by post-commit (e.g. a generic `git fixcommit` style command that runs all the relevant lint tools and fixes them in the working copy, letting you review before push), or applied during code review (automatic comments with a "click here to apply fix" interface), both of which are nice.

Plus the same underlying tooling can be used to write more complex one-off fixes that may be used for migrations or cleanups.

You might like Refex, which automates AST transformations like this:

There's lots of confusion about what semgrep does here, which is kind of unfortunate. I haven't touched it much, but I have built a very similar tool (I'm one of the contributors to refex[1], which is a very similar project).

The starting point of semantic grep is very useful. When you have a big codebase, you often want to detect antipatterns, or not even antipatterns, but just uses of a thing, say you're renaming a method and want to track down the callers.

Being able to act on the AST, instead of hoping you searched up all of the variants of whitespace and line breaks and, depending on the specific example, different uses of argument passing, is really useful.

But often when you're semantically grepping, your goal is to replace something with something else (this is what refex was initially built for: to aide in large scale changes in python, as a sort of equivalent to the C++ tools that Google uses).

But then you want to shift left even further: once you have a pattern that you want to replace once, you can just enforce that a linter yell at you when anyone does it again. So it's very natural to develop a linter-style thing on top of one of these[2].

This is, as I understand it sort of the same thing that happens in C++: clang-tidy and clang-format are written on top of AST libraries that can be used for ad-hoc analysis and transformations, but you can also just plug them into a linter.

The thing is, for most organizations, enforcing code style and best practices is more valuable than apply a refactoring to 10M lines of code, because most organizations don't have 10M lines of code to refactor. That doesn't mean that these tools aren't also useful for ad-hoc transforms and exploratory analysis. They absolutely are!

[1]: https://github.com/ssbr/refex

[2]: https://github.com/ssbr/refex/tree/main/refex/fix

Today we are going to look at how to setup Black (a python code formatter) and pre-commit (a package for handling git hooks in python) to automatically format you code on commit.

# See https://pre-commit.com for more information # See https://pre-commit.com/hooks.html for more hooks repos: - repo: https://github.com/pre-commit/pre-commit-hooks rev: v3.2.0 hooks: - id: trailing-whitespace - id: end-of-file-fixer - id: check-yaml - id: check-toml - id: check-added-large-files - repo: local hooks: - id: tox lint name: tox-validation entry: pdm run tox -e test,lint language: system files: ^src\/.+py$|pyproject.toml|^tests\/.+py$ types_or: [python, toml] pass_filenames: false - id: tox docs name: tox-docs language: system entry: pdm run tox -e docs types_or: [python, rst, toml] files: ^src\/.+py$|pyproject.toml|^docs\/ pass_filenames: false - repo: https://github.com/pdm-project/pdm rev: 2.10.4 # a PDM release exposing the hook hooks: - id: pdm-lock-check - repo: https://github.com/jumanjihouse/pre-commit-hooks rev: 3.0.0 hooks: - id: markdownlint

Pre-commit hooks act as the first line of defense in maintaining code quality, seamlessly integrating with linters and code formatters. They automatically execute these tools each time a developer tries to commit code to the repository, ensuring the code adheres to the project's standards. If the hooks detect issues, the commit is paused until the issues are resolved, guaranteeing that only code meeting quality standards makes it into the repository.

Pre-commit Hooks: Pre-commit is a tool that can be set up to enforce coding rules and standards before you commit your changes to your code repository. This ensures that you can't even check in (commit) code that doesn't meet your standards. This allows a code reviewer to focus on the architecture of a change while not wasting time with trivial style nitpicks.

Ah, fair enough!

On my team we use pre-commit[0] a lot. I guess I would define the history to be something like "has this commit ever been run through our pre-commit hooks?". If you rewrite history, you'll (usually) produce commits that have not been through pre-commit (and they've therefore dodged a lot of static checks that might catch code that wasn't working, at that point in time). That gives some manner of objectivity to the "history", although it does depend on each user having their pre-commit hooks activated in their local workspace.

[0]: https://pre-commit.com/

Pre-commit is a framework for managing and maintaining multi-language pre-commit hooks. It supports hooks for various programming languages. Using this framework, you only have to specify a list of hooks you want to run before every commit, and pre-commit handles the installation and execution of those hooks despite your project’s primary language.

You can learn more here: pre-commit.com

Note this includes support for pre-commit.

As a nice next step, you could also add mypy to check your type hints are consistent, and automate running all this via pre-commit hooks set up with… pre-commit.