Using Python Virtual Environments with Slurm

High Performance Computing ?? Comments 10/set/2018 seg

Using specific packages within a shared resources environment can sometimes be a hassle, some users may need different package versions, or even packages that conflict with each other. Relinquishing the package installation location to local user directories makes this process easier to manage.

In this post we'll see how to use virtual environments to manage your Python packages, this is useful not only to separate your Python packages from the system ones but also to be able to separate packages for each project, and end up making these environments portable and replicable across platforms.

As a plus we'll see how to create a Slurm batch file to submit a job to a Slurm queue

Overview


  • virtualenv - command used to create our virtual environments.
  • $HOME/.virtualenvs/ - the chosen directory to contain our virtual environments.
  • $HOME/.virtualenvs/AdS/ - the directory where this specific virtual environment will be placed in.
  • sympy - in this example we will be installing the sympy Python package.
  • source bin/activate - to enter the virtual environment we have to source its bin/activate file.
  • deactivate - to exit the virtual environment we have to issue the deactivate command.
  • AdS_sympy.sbatch - Slurm jobs use a batch file with specific headers.

Requirements

To be able to replicate this guide will need:

Virtual Environment

First let's create our virtual environment

t0rrant@bastion01 $ mkdir $HOME/.virtualenvs
t0rrant@bastion01 $ virtualenv $HOME/.virtualenvs/AdS

Now let's get into the virtual environment

t0rrant@bastion01 $ source $HOME/.virtualenvs/AdS/bin/activate
(AdS) t0rrant@bastion01 $

We can now install the sympy package

(AdS) t0rrant@bastion01 $ pip install sympy

Now let's take a look at our test file:

(AdS) t0rrant@bastion01 $ cat AdS_sympy_test.py
#!/usr/bin/env python
from sympy import test

test("_basic")

We only need to be within the virtual environment to actually use the sympy package (i.e: execute our test file) or to install new packages. The virtual environment will be used by the Slurm job as we will see.

So, let us exit the virtual environment:

(AdS) t0rrant@bastion01 $ deactivate
t0rrant@bastion01 $

Slurm Batch File

A quick look into the batch file, we must activate the virtual environment before running our code (deactivating it here is not mandatory as the process will end):

t0rrant@bastion01 $ cd AdS_sympy_test
t0rrant@bastion01 $ cat AdS_sympy.sbatch
#!/bin/bash
#SBATCH --job-name=AdS_sympy_test
#SBATCH --output=/home/t0rrant/AdS_sympy_test/AdS_sympy_test-%j.out
#SBATCH --error=/home/t0rrant/AdS_sympy_test/AdS_sympy_test-%j.err
#SBATCH --mail-user=<your e-mail should go here>
#SBATCH --mail-type=ALL
#SBATCH --time=1:00:00
#SBATCH --mem=1G

RUNPATH=/home/t0rrant/AdS_sympy_test/
cd $RUNPATH
source $HOME/.virtualenvs/AdS/bin/activate
python AdS_sympy_test.py

Slurm options:

--job-name This will determine the job name which appears in commands like sacct, smap or sview.
--output Determines the location of the file to which the stdout should be redirected to.
--error Determines the location of the file to which the stderr should be redirected to.
--mail-user The e-mail address to which job events should be sent to.
--mail-type The types of events that should be sent, here we use all of them. For more info see [the docs].
--time The amount of time we want allocated for this job.
--mem The amount of memory we want allocated for this job.

Looks good, let's submit it to the queue:

t0rrant@bastion01 $ sbatch AdS_sympy.sbatch
Submitted batch job 27047

Now check for errors and output:

t0rrant@bastion01 $ cat /home/t0rrant/AdS_sympy_test/AdS_sympy_test-27047.err

No errors! =)

t0rrant@bastion01 $ cat /home/t0rrant/AdS_sympy_test/AdS_sympy_test-27047.out
============================= test process starts ==============================
executable:         /home/t0rrant/.virtualenvs/AdS/bin/python  (2.7.13-final-0) [CPython]
architecture:       64-bit
cache:              yes
ground types:       python
numpy:              None
random seed:        7576360
hash randomization: on (PYTHONHASHSEED=1519633959)

sympy/core/tests/test_basic.py[16] ................                         [OK]

================== tests finished: 16 passed, in 0.09 seconds ==================
t0rrant@bastion01 $

As we can see it used the python executable from our virtual environment, used the sympy package also from that virtual environment and successfully ran the basic tests from the sympy package.


Manuel Torrinha is an information systems engineer, with more than 10 years of experience in managing GNU/Linux environments. Has an MSc in Information Systems and Computer Engineering. Work interests include High Performance Computing, Data Analysis, and IT management and Administration. Knows diverse programming, scripting and markup languages. Speaks Portuguese and English.

Related content