Running Python Jobs

Quickstart: Python

Option A - Own Container (recommended)

Build a container with Python & packages installed inside. Start by reviewing the container build instructions for either Apptainer or Docker. Then, when following the build instructions, use the example recipes for either a .def or Dockerfile, respectively.

  1. How to build your own container: Apptainer / Docker
  2. Tips and tricks, container recipes: Example container recipes for Python
  3. Use your container in your HTC jobs: Apptainer / Docker

Option B - Existing Container

Use an existing container with a base installation of Python:

  1. Choose an existing container. See OSG Base Containers or DockerHub Python Containers.
  2. Use the container in your HTC jobs: Apptainer / Docker

More information

All CHTC machines have a base installation of Python 3. The exact versions and packages installed, however, can vary from machine to machine. You should be able to include simple python commands in your calculations, i.e., python3 simple-script.py.

If you need a specific version of Python 3 or would like to install your own packages, we recommend that you use a container as described above.

The example recipes provided above for building your own container are intended for python packages that can be installed using python3 -m pip install. Additional software can be installed when building your own container.

For packages that need to be installed with conda install, see the guide on Conda.

Executable

When using a container, you can use a python .py script as the submit file executable, provided that the first line (the “shebang”) in the .py file is

#!/usr/bin/env python3

with the rest of the file containing the commands that you want to run using Python.

Alternatively, you can use a bash .sh script as the submit file executable, and in that file you can use the python3 command:

#!/bin/bash

python3 my-script.py

In this case, remember to include your .py file in the transfer_input_files line of your submit file.

HTC guides