sagemaker-run-notebook VS jupytext

Notebooks are more of a tech/design debt. Job was originally created in notebooks and never refactored into a more maintainable solution. Do you have a recommendation on services to use with the papermill library? I saw the Sagemaker Convenience Package seems to rely on this a bit, but seems a little out of place for Sagemaker.

I am used to working on jupyter notebooks and have some experience with python scripting for extracting data. I created a notebook that extracts comments, runs sentiment analysis, and dumps data to S3- however, I am struggling with the CRON aspect of things. I want to automate running of the script every day at a set time. I am using the sagemaker-run-notebook package to do this, specifically using the GUI Jupyterlab extension. However, I am not sure what the parameters need to be. I tried a few combinations, and this is my best guess at the values. I am getting a JSON error, which I am not sure I understand. Can someone with experience working with on the sagemaker-run-notebook extension please help me out?

Jupytext (https://github.com/mwouts/jupytext) has been designed exactly for this

jupytext is your friend.

So if you run your own jupyter server, -jupy+text can be a great workflow : it takes your notebook synchronized with other formats (python file, makdown, ...), so you can edit your py/md file with neovim, and refresh the browser to execute the notebook.

Automatically convert ipynb files to py when saving them on JupyterLab

I don't use Jupyter so I don't know for sure, but on a quick glance you might want to look at https://github.com/mwouts/jupytext to see if that could help at all.

The format is only partially invented, it follows Jupytext [0], but adds support for cell metadata. There is no obvious way to get that in fenced codeblocks, especially with the ability to spread it over multiple lines so it plays well with version control.

One more consideration is that it's not "Markdown with code blocks interspersed", one might as well use plaintext or AsciiDoc.

Of course there are tradeoffs.. I wish I had more time to work on it.

[0]: https://github.com/gzuidhof/starboard-notebook/blob/master/d...

[1]: https://github.com/mwouts/jupytext

Using jupytext (allows you to open .md files as notebooks) + jupyter gives you pretty much the same experience. The main issue is that the cell's output will be discarded. To fix it, you can use ploomber to generate an output HTML, so the workflow goes like this:

First, the format. The ipynb format does not play nicely with git since it stores the cell's source code and output in the same file. But Jupyter has built-in mechanisms to allow other formats to look like notebooks. For example, here's a library that allows you to store notebooks on a postgres database (I know this isn't practical, but it's a great example). To give more practical advice, jupytext allows you to open .py files as notebooks. So you can develop interactively but in the backend, you're storing .py files.