papermill VS pre-commit

Pandas docs > Comparison with spreadsheets: https://pandas.pydata.org/docs/getting_started/comparison/co...

Pandas docs > I/O > Excel files: https://pandas.pydata.org/docs/user_guide/io.html#excel-file...

nteract/papermill: https://github.com/nteract/papermill :

> papermill is a tool for parameterizing, executing, and analyzing Jupyter Notebooks. [...]

> This opens up new opportunities for how notebooks can be used. For example:

> - Perhaps you have a financial report that you wish to run with different values on the first or last day of a month or at the beginning or end of the year, using parameters makes this task easier.

"The World Excel Championship is being broadcast on ESPN" (2022) https://news.ycombinator.com/item?id=32420925 :

> Computational notebook speedrun ideas:

There is Papermill ... https://github.com/nteract/papermill

https://github.com/jupyter/enhancement-proposals/pull/103#is...

Papermill is one tool for running Jupyter notebooks as reports; with the date in the filename. https://papermill.readthedocs.io/en/latest/

You may be interested in papermill to address the parametrized analysis problem [1]. I think (but I'm not positive) this is what the data team at a previous job used to automate running notebooks for all sorts nightly reports.

[1] https://papermill.readthedocs.io/en/latest/#

I'm using Papermill to operationalize Notebooks (https://github.com/nteract/papermill), it e.g. also has airflow support. I'm really happy with papermill for automatic notebook execution, in my field it's nice that we can go very quickly from analysis to operations -- while having super transparent "logging" in the executed notebooks.

papermill bcpandas fastapi

But you can execute your notebook using Jupyter-run or papermill.

As a first option, we will use Papermill, which has a Python API that allows us to run different notebooks using some functions:

Papermill can also target cloud storage outputs for hosting rendered notebooks, execute notebooks from custom Python code, and even be used within distributed data pipelines like Dagster (see Dagstermill). For more information, see the papermill documentation.

Papermill Source Code

Today we are going to look at how to setup Black (a python code formatter) and pre-commit (a package for handling git hooks in python) to automatically format you code on commit.

# See https://pre-commit.com for more information # See https://pre-commit.com/hooks.html for more hooks repos: - repo: https://github.com/pre-commit/pre-commit-hooks rev: v3.2.0 hooks: - id: trailing-whitespace - id: end-of-file-fixer - id: check-yaml - id: check-toml - id: check-added-large-files - repo: local hooks: - id: tox lint name: tox-validation entry: pdm run tox -e test,lint language: system files: ^src\/.+py$|pyproject.toml|^tests\/.+py$ types_or: [python, toml] pass_filenames: false - id: tox docs name: tox-docs language: system entry: pdm run tox -e docs types_or: [python, rst, toml] files: ^src\/.+py$|pyproject.toml|^docs\/ pass_filenames: false - repo: https://github.com/pdm-project/pdm rev: 2.10.4 # a PDM release exposing the hook hooks: - id: pdm-lock-check - repo: https://github.com/jumanjihouse/pre-commit-hooks rev: 3.0.0 hooks: - id: markdownlint

Pre-commit hooks act as the first line of defense in maintaining code quality, seamlessly integrating with linters and code formatters. They automatically execute these tools each time a developer tries to commit code to the repository, ensuring the code adheres to the project's standards. If the hooks detect issues, the commit is paused until the issues are resolved, guaranteeing that only code meeting quality standards makes it into the repository.

Pre-commit Hooks: Pre-commit is a tool that can be set up to enforce coding rules and standards before you commit your changes to your code repository. This ensures that you can't even check in (commit) code that doesn't meet your standards. This allows a code reviewer to focus on the architecture of a change while not wasting time with trivial style nitpicks.

Ah, fair enough!

On my team we use pre-commit[0] a lot. I guess I would define the history to be something like "has this commit ever been run through our pre-commit hooks?". If you rewrite history, you'll (usually) produce commits that have not been through pre-commit (and they've therefore dodged a lot of static checks that might catch code that wasn't working, at that point in time). That gives some manner of objectivity to the "history", although it does depend on each user having their pre-commit hooks activated in their local workspace.

[0]: https://pre-commit.com/

Pre-commit is a framework for managing and maintaining multi-language pre-commit hooks. It supports hooks for various programming languages. Using this framework, you only have to specify a list of hooks you want to run before every commit, and pre-commit handles the installation and execution of those hooks despite your project’s primary language.

You can learn more here: pre-commit.com

Note this includes support for pre-commit.

As a nice next step, you could also add mypy to check your type hints are consistent, and automate running all this via pre-commit hooks set up with… pre-commit.