Our great sponsors
-
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
ipython
Official repository for IPython itself. Other repos in the IPython organization contain things like the website, documentation builds, etc.
-
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
-
-
-
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
You likely want to use something like https://github.com/jupyter/jupyter_console, or ipykernel directly to have a persistent python process. One issue is that shells are text based, so you have to do a lot of serialisation/deserialisation.
But honestly at that point I would just look into https://xon.sh/ that blends Python and Shell together. IPython and Xonsh devs are friends, so if you need anything from one into the other it's likely doable.
1) This is painful. There are tools to help, but the most effective means I've found are having a policy to only commit notebooks in a reset, clean state (enforced with githook).
2) I don't understand. I've written full testing frameworks for applications as notebooks as a means of having code documentation that enforced/tested the non-programmatic statements in the document. Using tools like papermill (https://papermill.readthedocs.io/en/latest/), you can easily write a unit test as a notebook with a whole host of documentation around what it's doing, execute, and inspect the result (failed execution vs. final state of the notebook vs. whatever you want)
3) Projects like ipynb (https://ipynb.readthedocs.io/en/stable/) allow you to import notebooks as if they were python modules. Some projects have different opinions of what that means to match different use cases. Papermill allows you have an interface with a notebook that is more like a system call than importing a module. I've personally used papermill and ipynb and found both enjoyable for different flavors of blending applications and notebooks.
First, yes, this is a common question. IPython does not try to deal with that, it's just the executing engine.
Notebooks, do not have to be stored in ipynb form, I would suggest to look at https://github.com/mwouts/jupytext, and notebook UI is inherently not design for multi-file and application developpement. So training humans will always be necessary.
Technically Jupyter Notebook does not even care that notebooks are files, you could save then using say postgres (https://github.com/quantopian/pgcontents) , and even sync content between notebooks.
I'm not too well informed anymore on this particular topic, but there are other folks at https://www.quansight.com/ that might be more aware, you can also ask on discourse.jupyter.org, I'm pretty sure you can find threads on those issues.
I think on the Jupyter side we could do a better job curating and exposing many tools to help with that, but there are just so many hours in the day...
I also recommend I don't like notebook from Joel Grus, https://www.youtube.com/watch?v=7jiPeIFXb6U it's a really funny talk, a lot of the points are IMHO invalid as Joel is misinformed on how things can be configured, but still a great watch.
Thanks, fun fact, IPython is one of the first project as started to contribute to when doing my PhD and one of my first "serious" python programs.
See my first PR https://github.com/ipython/ipython/pull/776.
GitHub lost some of the original (non-rebased) commits, but I had semicolon at the ends of the lines.
And yes I stayed because it was "Fun". Hope to see more contributions !
AST in a more general concept see https://en.wikipedia.org/wiki/Abstract_syntax_tree, it basically turn your text into a tree where f(a) + f(b) is `call(plus, call(f, a), call(f,b))`. Using https://github.com/alexmojaki/stack_data we can say "the error occurred in `a`, while trying to call `f`, while calling `plus`, get the range in the original text and make them yellow.
You likely want to use something like https://github.com/jupyter/jupyter_console, or ipykernel directly to have a persistent python process. One issue is that shells are text based, so you have to do a lot of serialisation/deserialisation.
But honestly at that point I would just look into https://xon.sh/ that blends Python and Shell together. IPython and Xonsh devs are friends, so if you need anything from one into the other it's likely doable.
Yeah, mostly I lack time to catch up with Jonathan Slenders works, and have stronger backward compatibility requirements. b=But ptpython and pyipython are both great.
I should also look into Rich and Textual
https://bpython-interpreter.org/ is also another alternative python shell, and of course https://xon.sh
Yes, bpython is good. I have plans to make the documentation better (https://github.com/jupyter/papyri) but so far I only have a few hours per week I can spend on IPython. Jedi from david halter should also get some love for better completion.
First, yes, this is a common question. IPython does not try to deal with that, it's just the executing engine.
Notebooks, do not have to be stored in ipynb form, I would suggest to look at https://github.com/mwouts/jupytext, and notebook UI is inherently not design for multi-file and application developpement. So training humans will always be necessary.
Technically Jupyter Notebook does not even care that notebooks are files, you could save then using say postgres (https://github.com/quantopian/pgcontents) , and even sync content between notebooks.
I'm not too well informed anymore on this particular topic, but there are other folks at https://www.quansight.com/ that might be more aware, you can also ask on discourse.jupyter.org, I'm pretty sure you can find threads on those issues.
I think on the Jupyter side we could do a better job curating and exposing many tools to help with that, but there are just so many hours in the day...
I also recommend I don't like notebook from Joel Grus, https://www.youtube.com/watch?v=7jiPeIFXb6U it's a really funny talk, a lot of the points are IMHO invalid as Joel is misinformed on how things can be configured, but still a great watch.
This is a daily pain we've experienced while working in the industry! Our projects would usually allocate a few weeks to refactor notebooks before deployment! So we started working on an open-source framework to help us produce maintainable work from Jupyter. It allows easy git collaboration and eases deployment. We got a dozen companies using it in production already. https://github.com/ploomber/ploomber
I believe this is the issue:
https://github.com/psf/black/issues/1143
Not entirely clear what the reasons are for adding the Black dependency to IPython....
I'm working on a solution that helps with transforming notebooks into web applications (with GUI). You just need to define YAML config (similar to R Markdown) and the framework will generate web app with interactive widgets. After change in widgets, user clicks Run button and the whole notebook is executed, converted to HTML and displayed to the user.
The framework is called Mercury and is open-source https://github.com/mljar/mercury