pgcontents VS jupytext

First, the format. The ipynb format does not play nicely with git since it stores the cell's source code and output in the same file. But Jupyter has built-in mechanisms to allow other formats to look like notebooks. For example, here's a library that allows you to store notebooks on a postgres database (I know this isn't practical, but it's a great example). To give more practical advice, jupytext allows you to open .py files as notebooks. So you can develop interactively but in the backend, you're storing .py files.

First, yes, this is a common question. IPython does not try to deal with that, it's just the executing engine.

Notebooks, do not have to be stored in ipynb form, I would suggest to look at https://github.com/mwouts/jupytext, and notebook UI is inherently not design for multi-file and application developpement. So training humans will always be necessary.

Technically Jupyter Notebook does not even care that notebooks are files, you could save then using say postgres (https://github.com/quantopian/pgcontents) , and even sync content between notebooks.

I'm not too well informed anymore on this particular topic, but there are other folks at https://www.quansight.com/ that might be more aware, you can also ask on discourse.jupyter.org, I'm pretty sure you can find threads on those issues.

I think on the Jupyter side we could do a better job curating and exposing many tools to help with that, but there are just so many hours in the day...

I also recommend I don't like notebook from Joel Grus, https://www.youtube.com/watch?v=7jiPeIFXb6U it's a really funny talk, a lot of the points are IMHO invalid as Joel is misinformed on how things can be configured, but still a great watch.

Jupytext (https://github.com/mwouts/jupytext) has been designed exactly for this

jupytext is your friend.

So if you run your own jupyter server, -jupy+text can be a great workflow : it takes your notebook synchronized with other formats (python file, makdown, ...), so you can edit your py/md file with neovim, and refresh the browser to execute the notebook.

Automatically convert ipynb files to py when saving them on JupyterLab

I don't use Jupyter so I don't know for sure, but on a quick glance you might want to look at https://github.com/mwouts/jupytext to see if that could help at all.

The format is only partially invented, it follows Jupytext [0], but adds support for cell metadata. There is no obvious way to get that in fenced codeblocks, especially with the ability to spread it over multiple lines so it plays well with version control.

One more consideration is that it's not "Markdown with code blocks interspersed", one might as well use plaintext or AsciiDoc.

Of course there are tradeoffs.. I wish I had more time to work on it.

[0]: https://github.com/gzuidhof/starboard-notebook/blob/master/d...

[1]: https://github.com/mwouts/jupytext

Using jupytext (allows you to open .md files as notebooks) + jupyter gives you pretty much the same experience. The main issue is that the cell's output will be discarded. To fix it, you can use ploomber to generate an output HTML, so the workflow goes like this:

First, the format. The ipynb format does not play nicely with git since it stores the cell's source code and output in the same file. But Jupyter has built-in mechanisms to allow other formats to look like notebooks. For example, here's a library that allows you to store notebooks on a postgres database (I know this isn't practical, but it's a great example). To give more practical advice, jupytext allows you to open .py files as notebooks. So you can develop interactively but in the backend, you're storing .py files.