Use Custom Software in Jobs Using Docker

Linux containers are a way to build a self-contained environment that includes software, libraries, and other tools. This guide shows how to submit jobs that use Docker containers.

Table of Contents

Start Here

To run a job using a Docker container, you will need access to a Docker container image that has been built and placed onto the DockerHub website or another Docker registry service like [https://quay.io/] or a GitLab registry.

If you have an existing Docker container with your software installed inside, go straight to Use a Docker Container in a Job and Tips for HTCondor integration

If you do not have an existing Docker container you can:

Use a Docker Container in a Job

Jobs that run inside a Docker container will be almost exactly the same as normal HTCondor jobs. The main change is indicating which Docker container to use and an optional “container universe” option:

# HTC Submit File

# Provide HTCondor with the name of the Docker container
container_image = docker://user/repo:tag
universe = container

executable = myExecutable.sh
transfer_input_files = other_job_files

log = job.log
error = job.err
output = job.out

request_cpus = 1
request_memory = 4GB
request_disk = 2GB

queue

In the above, change the address of the Docker container image as needed based on the container you are using.

💻 Non Docker Hub Registries

If your container is hosted on a Docker registry other than DockerHub, just add the registry prefix before the container repository name. For example, for a container hosted in the Red Hat repository quay.io, the option should be:

container_image = docker://quay.io/user/repo:tag

Tips for HTCondor integration

When your job starts, HTCondor will pull the indicated image from DockerHub, and use it to run your job. You do not need to run any Docker commands yourself.

Other pieces of the job (your executable and input files) should be just like a non-Docker job submission.

We recommend using container tags. Importantly, whenever you make a significant change to your container, you will want to use a new tag name to ensure that your jobs are getting an updated version of the container, and not an ‘old’ version that has been cached by DockerHub or CHTC.

Find an existing Docker container image

The easiest way to get a Docker container image for running a job is to use a public or pre-existing image on DockerHub or another registry. You can find images by getting an account on DockerHub and searching for the software you want to use.

Sample images:

An image supported by a group will be continuously updated and the versions will be indicated by “tags”. We recommend choosing a specific tag (or tags) of the container to use in CHTC.

Build your own Docker container image

You can also build your own Docker container image and upload it to DockerHub. See this link to our guide on building containers or the Docker documentation for more information.

Testing Docker containers

If you want to test your jobs or container, you have two options:

  • We have a guide on exploring and testing Docker containers on your own computer here:
  • You can test a container interactively in CHTC by using a normal Docker job submit file and using the interactive flag with condor_submit: 
      [alice@ap]$ condor_submit -i docker.sub

    This should start a session inside the indicated Docker container and connect you to it using ssh. Type exit to end the interactive job. Note: Files generated during your interactive job with Docker will not be transfered back to the submit node. If you have a directory on staging, you can transfer the files there instead; if you have questions about this, please contact a facilitator.

HTC guides