pgcontents VS mercury

First, the format. The ipynb format does not play nicely with git since it stores the cell's source code and output in the same file. But Jupyter has built-in mechanisms to allow other formats to look like notebooks. For example, here's a library that allows you to store notebooks on a postgres database (I know this isn't practical, but it's a great example). To give more practical advice, jupytext allows you to open .py files as notebooks. So you can develop interactively but in the backend, you're storing .py files.

First, yes, this is a common question. IPython does not try to deal with that, it's just the executing engine.

Notebooks, do not have to be stored in ipynb form, I would suggest to look at https://github.com/mwouts/jupytext, and notebook UI is inherently not design for multi-file and application developpement. So training humans will always be necessary.

Technically Jupyter Notebook does not even care that notebooks are files, you could save then using say postgres (https://github.com/quantopian/pgcontents) , and even sync content between notebooks.

I'm not too well informed anymore on this particular topic, but there are other folks at https://www.quansight.com/ that might be more aware, you can also ask on discourse.jupyter.org, I'm pretty sure you can find threads on those issues.

I think on the Jupyter side we could do a better job curating and exposing many tools to help with that, but there are just so many hours in the day...

I also recommend I don't like notebook from Joel Grus, https://www.youtube.com/watch?v=7jiPeIFXb6U it's a really funny talk, a lot of the points are IMHO invalid as Joel is misinformed on how things can be configured, but still a great watch.

I'm build dashboards in Jupyter Lab. My plotting libraries are Altair, matplotlib, seaborn, Plotly - all work well in notebook.

My favorite is Altair. It provides interactivity for charts, so you can move/zoom your plots and have tooltips. It is much lighter than Plotly after saving the notebook to ipynb file. Altair charts looks much better than in matplotlib. One drawback, that exporting to PDF doesn't work. To serve notebook as dashboard with code hidden, I use Mercury framework, you can check example https://runmercury.com/tutorials/vega-altair-dashboard/

disclaimer: I'm author of Mercury framework https://github.com/mljar/mercury

Web App is using two open-source packages that I've created:

- MLJAR AutoML - Python package for AutoML on tabular data https://github.com/mljar/mljar-supervised

- Mercury - framework for converting Jupyter Notebooks into Web App https://github.com/mljar/mercury

You can run Web App locally. What is more, you can adjust notebook's code for your needs. For example, you can set different validation strategies or evalutaion metrics or longer training times. The notebooks in the repo are good starting point for you to develop more advanced apps.

The GitHub repository https://github.com/mljar/mercury

We are working on open-source web framework Mercury that converts Python notebooks to Web Apps.

It is very opinionated:

- it has no callbacks - we automatically re-execute cells below updated widget

- it has no layout widgets, all input widgets are always in the left sidebar

Thanks to above decisions you don't need to change notebook's code to have web app and fit to the framework.

The simplicity of the framework is very important to us. We also care about deployment simplicity. That's why we created a shared hosting service called Mercury Cloud. You can deploy notebook by uploading a file.

The GitHub repository https://github.com/mljar/mercury

Documentation https://RunMercury.com/docs/

Mercury Cloud https://cloud.runmercury.com

Analysis was done in Jupyter Notebook with Python 3.10, Pandas, Matplotlib, wordcloud and Mercury framework.

Analysis was done in Jupyter Notebook with Python 3.10 kernel, Pandas, Matplotlib, wordcloud and Mercury framework to share notebook as a web application with widgets and code hidden. Gif created in Canva.