-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
-
ploomber-engine
A toolbox 🧰 for Jupyter notebooks 📙: testing, experiment tracking, debugging, profiling, and more!
Papermill is great but has quite some limitations because it spins up a new process to run the notebook:
- You cannot extract live variables (needed for testing)
- Cannot use pdb for debugging
- Cannot profile memory usage
You can do all of that with ploomber-engine (https://github.com/ploomber/ploomber-engine).
-
I looked Papermill back in a day, but found it easier call nbclient and nbconvert directly
https://github.com/tradingstrategy-ai/trade-executor/blob/ma...
-
marimo
A reactive notebook for Python — run reproducible experiments, execute as a script, deploy as an app, and version with git.
Have you considered using marimo notebooks?
https://github.com/marimo-team/marimo
marimo notebooks are stored as pure Python (executable as scripts, versionable with git), and they largely eliminate the hidden state problem that affects Jupyter notebooks -- delete a variable and it's automatically removed from program memory, run a cell and all other cells that use its variables are marked as stale.
marimo notebooks are also readily parametrized with CLI arguments, so you can do: python notebook.py -- -foo 1 -bar 2 ...
Disclosure: I'm a marimo developer.
-
Are we talking about the same https://github.com/tweag/jupyenv ?
It encapsulates the kernel, which encapsulates pretty much everything for the notebook, right? I haven't worked with Slurm or OpenPBS, but I think if you let nix build the images that your tasks are running in then I think you're covered for pretty much everything except things that only exist at runtime like database connections. Not a perfect black box, but close.
