pgcontents VS jupyter_console

First, the format. The ipynb format does not play nicely with git since it stores the cell's source code and output in the same file. But Jupyter has built-in mechanisms to allow other formats to look like notebooks. For example, here's a library that allows you to store notebooks on a postgres database (I know this isn't practical, but it's a great example). To give more practical advice, jupytext allows you to open .py files as notebooks. So you can develop interactively but in the backend, you're storing .py files.

First, yes, this is a common question. IPython does not try to deal with that, it's just the executing engine.

Notebooks, do not have to be stored in ipynb form, I would suggest to look at https://github.com/mwouts/jupytext, and notebook UI is inherently not design for multi-file and application developpement. So training humans will always be necessary.

Technically Jupyter Notebook does not even care that notebooks are files, you could save then using say postgres (https://github.com/quantopian/pgcontents) , and even sync content between notebooks.

I'm not too well informed anymore on this particular topic, but there are other folks at https://www.quansight.com/ that might be more aware, you can also ask on discourse.jupyter.org, I'm pretty sure you can find threads on those issues.

I think on the Jupyter side we could do a better job curating and exposing many tools to help with that, but there are just so many hours in the day...

I also recommend I don't like notebook from Joel Grus, https://www.youtube.com/watch?v=7jiPeIFXb6U it's a really funny talk, a lot of the points are IMHO invalid as Joel is misinformed on how things can be configured, but still a great watch.

Sounds like you'd be interested in jupyter-console, which lets you do something like ilua for any language that has a Jupyter kernel. Well, in theory; the one language kernel I use that would really benefit from it, F# (which has a pretty shit command-line repl) doesn't work correctly. It's flawless with the OCaml kernel, but there isn't much point to running that one on the terminal considering utop is insanely good already.

You likely want to use something like https://github.com/jupyter/jupyter_console, or ipykernel directly to have a persistent python process. One issue is that shells are text based, so you have to do a lot of serialisation/deserialisation.

But honestly at that point I would just look into https://xon.sh/ that blends Python and Shell together. IPython and Xonsh devs are friends, so if you need anything from one into the other it's likely doable.