tunel VS docker-stacks

This weekend I've started to make tiny progress on my library "tunel," which means as it sounds - an elegant derivation of "tunneL' so you can imagine we are doing lots of tricks with ssh! It currently offers (and this will be extended) developer interfaces for HPC! -> https://github.com/vsoch/tunel. So far I have a few apps ready to go for standard slurm and/or singularity via sockets, and I should be adding an HTCondor launcher next. After that I plan to develop "app templates" so anyone can easily make an app that will easily run via HPC. If anyone knows of an existing tool for some visualization / monitoring or other that they would like me to investigate as an app, please let me know!

See https://github.com/jupyter/docker-stacks

I have an IPython notebook running in a docker container in Google Container Engine, the container is based on this image jupyter/all-spark-notebook

Yes sir :) love the jupyter docker stacks: https://github.com/jupyter/docker-stacks

I use an image that allows me to run spark on jupyter notebooks and use files from my local machine to test code. Here is a good one https://github.com/jupyter/docker-stacks/tree/main/pyspark-notebook

Today I will tell you a story behind making Jupyter Docker Stacks able to build aarch64 images without many (almost any) compromises. If you want to read how it is implemented now, you can skip to the last section of this post.

FROM ubuntu:20.04 LABEL maintainer="PhysicsReplicatorAI " ARG NB_USER="coqui_user" \ NB_UID="1000" \ NB_GID="100" USER root ENV DEBIAN_FRONTEND=noninteractive \ TERM=xterm \ TZ=America/New_York # -------------------------------------------------------------------------------- # Update/Install the basics # -------------------------------------------------------------------------------- RUN apt-get update -y && \ apt-get install -y --no-install-recommends \ sudo locales apt-utils g++ libsndfile1-dev && \ apt-get -y clean autoclean && \ apt-get -y autoremove --purge && \ echo "en_US.UTF-8 UTF-8" > /etc/locale.gen && \ locale-gen # -------------------------------------------------------------------------------- # Configure environment # -------------------------------------------------------------------------------- ENV SHELL=/bin/bash \ NB_USER="${NB_USER}" \ NB_UID=${NB_UID} \ NB_GID=${NB_GID} \ LC_ALL=en_US.UTF-8 \ LANG=en_US.UTF-8 \ LANGUAGE=en_US.UTF-8 \ PATH="/home/${NB_USER}/.local/bin:${PATH}" \ HOME="/home/${NB_USER}" # -------------------------------------------------------------------------------- # Create coqui_user # User creation method/pattern obtained from: # https://github.com/jupyter/docker-stacks/blob/master/base-notebook/Dockerfile # -------------------------------------------------------------------------------- COPY fix-permissions /usr/local/bin/fix-permissions RUN chmod a+rx /usr/local/bin/fix-permissions && \ sed -i 's/^#force_color_prompt=yes/force_color_prompt=yes/' /etc/skel/.bashrc && \ echo "auth requisite pam_deny.so" >> /etc/pam.d/su && \ sed -i.bak -e 's/^%admin/#%admin/' /etc/sudoers && \ sed -i.bak -e 's/^%sudo/#%sudo/' /etc/sudoers && \ useradd -l -m -s /bin/bash -N -u "${NB_UID}" "${NB_USER}" && \ chmod g+w /etc/passwd && \ fix-permissions "${HOME}" && \ cd "${HOME}" && \ apt-get install -y --no-install-recommends python3-dev python3-pip && \ apt-get -y autoclean && \ apt-get -y autoremove --purge && \ apt-get -y purge $(dpkg --get-selections | grep deinstall | sed s/deinstall//g) && \ rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* # -------------------------------------------------------------------------------- # Install TTS # -------------------------------------------------------------------------------- USER ${NB_UID} RUN fix-permissions "${HOME}" && \ python3 -m pip install --no-cache-dir --upgrade TTS && \ rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* EXPOSE 5002 ENTRYPOINT ["tts-server"]

Pretty much what title says. I want to build an image on top of the scy-py image with my preferred libraries such as NLTK, spaCy and stanza. Is there a way to do that?

The disadvantages are that everytime I rebuild the entire docker image I have to pray that all the dependencies that I did not list did not change, because if they did, the build fails with a cryptic error message. Also if the list of pip installs gets too long, pip is no longer able to resolve all the dependencies and then the docker image does not build either. So actually not having a requirements.txt might be a disadvantage. Further, I do not like the fact that I have both Miniconda and "regular Python" in one file. This happened, because this Dockerfile evolved from one of the Jupyter Notebook Dockerfiles. My development environment is Jupyter and I wanted my production environment as close as possible to my development environment.