Deploying a New Hub

Adding a Hub

Introduction

This document assumes that you have followed the walkthrough at least once before. You already have:

  • Logged in with an account that has sudo permissions
  • Docker installed
  • Proxy configured, secured with HTTPS

This walk-through will guide you through the bare-minimal steps to set up a new hub.

Preliminaries

First, we need to decide on the following:

  • Name of new hub
  • An available port for it to be hosted on

We will run the following commands to find out what is already in use:

docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

You will see the names and ports formatted printed out. Here is an example of what that might look like:

NAMES                STATUS              PORTS
math8660-user-troy   Up 32 minutes       8888/tcp
math-user-michael    Up 3 days           8888/tcp
math8660             Up 3 days           0.0.0.0:8002->8000/tcp
math8660-db          Up 3 days           5432/tcp
math                 Up 3 days           0.0.0.0:8000->8000/tcp
math-db              Up 3 days           5432/tcp

What we are seeing is a list of processes being run by Docker and their respective ports. The ones that read 8888/tcp are on the local docker-network (these are the single-user notebook servers). It is fine for these numbers to conflict since they are not ports open on the main server. The databases on 5432 are similarly of no concern to us.

If the output of this command is blank (perhaps this is your first time), then nothing is running, and so you can stick with the default name and port number in our deployment.

We can also see that three hubs are live on this server, listening for connections on local ports 8000 and 8001 in addition to the two user containers.

We simply need to choose any port other than 8000 or 8002 (e.g. 8001, 8003, 8451, 8762, etc.).

We also see that the two names in use are math (default in our deployment) and math8660, so we would have to choose a different name. In our deployment, this name is ALSO where your hub will live (example.com/math8660 and example.com/math are live in the above output).

The proxy configurations must direct traffic to these “locations” (/math and /math8660) for it to be publicly accessible. See the Proxy section of the Walkthrough.

Before proceeding, let us get the necessary files.

Clone the repository from GitHub or copy an existing hub’s directory and rename the folder, or upload the files in any way you want.

Rename it to something memorable (perhaps the name of your hub). Here we choose “deploy” as our folder name.

git clone https://github.com/mathematicalmichael/jupyterhub-deploy-docker.git
mv jupyterhub-deploy-docker/ deploy/
cd deploy

The last line brings us into the deploy directory, where the instructions that follow pick up. They assume that you have cloned the repo.

Quick Install

I have added a convenience script that allows for automatic creation of a hub. It creates one default hub-admin user and prints the password as output when it completes the build process and the hub is live. It will also print out the entry that needs to be added to /etc/nginx/sites-enabled/hub.conf in order to add your hub.

Instructions

If you have no existing hub named math and port 8000 is available, then nothing needs to be edited. (If you need to reassign HUB_NAME and PORT_NUM, edit the first lines of .env). Be aware of re-using old names from previous deployments1. Once you are ready, run:

This may take 10-15 minutes in total but will handle everything for you.

./setup.sh

This script presumes this is your first run. It creates several files that are only required initially, and re-running it will delete any changes you made to userlist unless you comment out the first line (which creates this file).

A dialogue will come across the screen tell you the password to get into the hub and what to add to the proxy configuration. Once you log in, you can add users through the Admin panel.

On the CU Denver campus, (or from behind any private network), you will likely encounter a warning in your browser that you must Add an Exception to handle.

hub-error Chrome, Firefox, Edge, etc. all have variations on the following dialogue windows that you must go through before seeing your hub:

hub-error-confirm

However, to manage shared volumes for groups/teams, you must change userlist accordingly.

Group Memberships

The setup.sh script additionally creates one group (team) named shared and associated volume, mounting it to hub-admin’s working directory. All shared (group) directories are prepended with shared-, which is what jupyterhub_config.py is expecting when it reads userlist to determine group memberships.

If you open up userlist and add more users who belong to the shared group, they will also see this folder and be able to read/write to it. You can add any phrase (groupname) to a user’s line, but be sure to create the volume and set its permissions before refreshing the hub (if you do not do this, Docker will create the volume automatically, but users will only be given read permissions by default).

To create volumes for new groups and set the permissions, run the following command (which parses information to find where Docker has linked the new volume, and passes the directory to chmod:

export SHARED_VOLUME_NAME=newGroupNameHere
docker volume create shared-$SHARED_VOLUME_NAME
sudo chmod 777 $(docker inspect shared-$SHARED_VOLUME_NAME | grep "Mountpoint" | awk '{print $2}' | sed 's/"//g' | sed 's/,//g')

Be sure to change the above to reflect any group names you (or your students) choose, and run it for every new group name.

An example userlist might look like (group orders do not matter):

hub-admin shared admin 
mpilosov admin shared team-1
halljord team-1
tbutler admin team-1

Finding out Passwords

After adding users/groups to userlist, you will have to restart hub for changes to take effect. This can be accomplished through the Control Panel by clicking “Shut Down Hub” and refreshing the page).

Log in as hub-admin and visit /hub/login_list to see all users and passwords from userlist.

Additionally, if you edit the last line of .env and uncomment the last lines of show_login.sh, you can see individual user passwords from the command-line by running ./show_login.sh from the root project directory (here, /deploy).

Sharing Passwords

To share passwords across hubs, simply be sure that secrets/oauth.env is the same. Copy it over from any existing hubs you desire.

Restarting the Hub

You will need to run the following from the hub’s root directory for these changes (as well as those made to userlist), to take effect:

docker-compose down
docker-compose build
docker-compose up -d

These commands stop and destroy the containers running the hub and database, remove the existing image, re-build the Hub’s image and re-launch it (with no data loss, since volumes and networks are preserved).




Manual Install

If you want to perform the steps carried out in setup.sh yourself, you can follow these directions (or simply run the commands in that file one-by-one). This script is designed to save you time and effort.

Configure Settings

Now that we are in the deploy directory (or whatever you named your hub):

mkdir secrets
make secrets/postgres.env
make secrets/oauth.env
make userlist

If you copied an existing hub, be aware that your secret files will be shared between the two (same login credentials).

At the very least, remove postgres.env and re-make the secret key there. If you want passwords to be shared for users across hubs, keep oauth.env the same among the hubs. Passwords are set during the build process.

You will see some dialogue regarding how to add users to the hub initially. We only need to be concerned with adding one administrative user at this time since we can add users later through the Hub’s interface. Run the following command and edit the userlist according to the prompt you just saw, which will show you how to format this file.

vim userlist

Now, run vim .env to change the name of your hub and the port to avoid conflicts with running Docker processes.

The noteboook image that gets built by default is quite sizable.

You may consider changing this in the .env file ($DOCKER_NOTEBOOK_IMAGE) if you’re just testing it out and want to make sure it works (perhaps to jupyter/minimal-notebook or another pre-built stack ). But, the one we ship is feature-rich. Consider it a “show off what this can do” example.

The notebook image that gets build starts off with $DOCKER_NOTEBOOK_IMAGE and adds in some features. To change the image, edit singleuser/Dockerfile and delete as much as you would like.

At any point you can re-build this image with make notebook_image (which we will run in a moment in the next step).

Build

We are now ready to build our hub! The last line also runs it as a background process.

make build
make notebook_image
docker-compose up -d

And that is all. Sit back and wait. It will take a while.

You might see a bunch of red messages like these fly by during the build process.

chgrp: changing group of '/opt/conda/var/cache/fontconfig/0c78243b-3123-48a4-91b4-49cb45a27aaf-le64.cache-7': Operation not permitted
chgrp: changing group of '/opt/conda/var/cache/fontconfig/2fd305a6-4303-4a09-8894-d1594f7ec636-le64.cache-7': Operation not permitted
.
.
.

As far as I can tell, this is not a problem and occurs somewhere outside of the instructions I have added on top of the pre-built images supplied by Jupyter.

If successful, you should see green done output at the very end.

When you want to destroy the images that get built (make build runs docker-compose build with some other options to configure it properly), and the associated containers created from them, you can run:

docker-compose down

to clean up. Since data is external to the containers that host our application, you won’t lose any configurations at all.

Direct Traffic

From the walkthrough, you should have already configured the nginx proxy. Now all that is left is to tell our proxy the “location” of our hub so that it can become publicly accessible.

sudo vim /etc/nginx/sites-enabled/hub.conf

You need to look for the server object (if secured, on port 443) that matches the domain name that this hub will exist on, and add an entry inside of this server configuration. It will look like this (for the default hub that ships with the deployment, specified in .env):

location /math {
    proxy_pass http://127.0.0.1:8000;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    # websocket headers
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
}

Now, when example.com/math is hit, our proxy (nginx) will direct traffic to localhost (127.0.0.1). Specifically, to port 8000. Make sure that the entry you add matches the .env file. Here, we show the default.

Acquire Credentials

To get your password, vim Makefile and uncomment the following line (around 5): include secrets/oauth.env (Alternatively, you can just run source secrets/oauth.env and skip editing the file).

Once that is done, you can run make login to find out the password for whatever user is specified in .env.

Hopefully the user in .env matches at least one of the admin users you put in userlist, otherwise you would be querying passwords for non-existent users. If not, edit .env and userlist and run docker-compose down; docker-compose up -d to re-build the images.

Test

You can now log in by visiting example.com/math and ensure that everything is working as expected out of the box.

snapshot-of-login

You can add users through Control Panel > Admin! ![snapshot-of-admin][hub-admin]

Now that you have your password, log in to the hub. If you navigate to /hub/login_list you will see passwords for everyone in userlist. Since the connection is not yet secure, do not do this yet.

You can bring up or take down your hub with docker-compose up -d and docker-compose down, but must run these in the same directory as the docker-compose.yml file.




Management

Adding Packages

Assuming you are ssh’ed into the server and inside the directory that has your hub, simply edit singleuser/Dockerfile and use the Docker Reference on the next page to run commands.

Managing Users/Work

When logged in to the hub, you can access Control Panel > Admin to add users and access their servers.

To disable the feature of Admins being able to log in as the students, set c.JupyterHub.admin_access = False in jupyterhub_config.py.

Group Sharing

Sharing files between groups is a feature I personally coded into the jupyterhub_config.py file. Towards the top of the script, the file userlist is opened (in the container… so if you update it, you can run docker restart hub-name since this configuration file only runs at hub startup). Better yet, do this from the Admin menu directly.

Each line in userlist has the name of the student, followed by their group names, separated by spaces.

Instead of restarting through the docker command, you can “Power Down” the Hub from the Control Panel in JupyterHub. This has the effect of propagating changes to lists of users. Docker takes care of re-starting the hub automatically.

If userlist is given the right permissions with chown 777 userlist, and mounted inside the professor’s directory, it may be possible to manage group memberships without ever logging into the math-hub server (except to update the single-user notebook image), editing the userlist right from the web-interface, provided the group volume permissions have previously been set correctly.

jupyterhub_config.py will check the user’s groups and mount all appropriate volumes. A volume is created for the group if one does not exist.

By default, when volumes are created, the permissions are not set in a way where users can write files.

To fix this, you would find the shared group volume in question, use docker inspect to find out where it is, and change the permissions with chmod. Here is an example:

pilosovm@math-hub:~$ docker volume ls | grep "shared-*" | awk '{print $2}'
ro_shared_volume
rw_shared_volume
shared-broncos
shared-test-group
pilosovm@math-hub:~$
pilosovm@math-hub:~$ docker volume inspect shared-broncos | grep "Mountpoint"
        "Mountpoint": "/var/lib/docker/volumes/shared-broncos/_data",
pilosovm@math-hub:~$ chmod 777 /var/lib/docker/volumes/shared-broncos/_data


In this manner, you can create new volumes (e.g. docker volume create shared-groupname) and set the permissions as desired with chmod, modifying userlist to add groupname to the appropriate users.

Rinse, repeat.

The architecture implemented here mounts volumes based on the words that follow the username in userlist and prepends them with shared. If you want to avoid naming conflicts, use group names such as math8660-group1, math8660-group2, etc., or something unique chosen by the group members). It is up to you to avoid naming conflicts across multiple hubs when sharing group volumes.

More customization is possible. For example, you can have multiple hubs set up on the same server (e.g. at example.com/hub1/ and example.com/hub2/, etc.) but create one shared-admin volume that every administrator has access to. To accomplish this, (or to add any new group volume):

docker volume create shared-admin
docker volume inspect shared-admin | grep "Mountpoint"

then use chmod 777 PATH where PATH matches the path in the output of the last command. Now this volume is visible to anyone for whom it is mounted.

However, in jupyterhub_config.py, we have this:

33                 for i in range(1,len(parts)):
34                     group_id = parts.pop()
35                     if group_id != 'admin': # no need for an admin group.
36                         group_map[user_name].append(group_id)

We purposefully disable this from happening by default. To enable it, simply change the logic here. Erase line 35, un-indent line 36, and the shared-admin volume will be mounted for all admins of all hubs. Don’t forget to restart your hub for the changes to take effect.


  1. The command docker-compose down does not remove volumes created during the build process. If using a name of a hub that once existed, remove the old volumes with docker volume rm (use docker volume ls to list existing volumes and look for -data endings). ^