Using specific packages within a shared resources environment can sometimes be a hassle, some users may need different package versions, or even packages that conflict with each other. Relinquishing the package installation location to local user directories makes this process easier to manage.
In this post we'll see how to use virtual environments to manage your Python packages, this is useful not only to separate your Python packages from the system ones but also to be able to separate packages for each project, and end up making these environments portable and replicable across platforms.
As a plus we'll see how to create a Slurm batch file to submit a job to a Slurm queue
- virtualenv - command used to create our virtual environments.
- $HOME/.virtualenvs/ - the chosen directory to contain our virtual environments.
- $HOME/.virtualenvs/AdS/ - the directory where this specific virtual environment will be placed in.
- sympy - in this example we will be installing the sympy Python package.
- source bin/activate - to enter the virtual environment we have to source its bin/activate file.
- deactivate - to exit the virtual environment we have to issue the deactivate command.
- AdS_sympy.sbatch - Slurm jobs use a batch file with specific headers.
First let's create our virtual environment
t0rrant@bastion01 $ mkdir $HOME/.virtualenvs t0rrant@bastion01 $ virtualenv $HOME/.virtualenvs/AdS
Now let's get into the virtual environment
t0rrant@bastion01 $ source $HOME/.virtualenvs/AdS/bin/activate (AdS) t0rrant@bastion01 $
We can now install the sympy package
(AdS) t0rrant@bastion01 $ pip install sympy
Now let's take a look at our test file:
(AdS) t0rrant@bastion01 $ cat AdS_sympy_test.py #!/usr/bin/env python from sympy import test test("_basic")
We only need to be within the virtual environment to actually use the sympy package (i.e: execute our test file) or to install new packages. The virtual environment will be used by the Slurm job as we will see.
So, let us exit the virtual environment:
(AdS) t0rrant@bastion01 $ deactivate t0rrant@bastion01 $
Slurm Batch File
A quick look into the batch file, we must activate the virtual environment before running our code (deactivating it here is not mandatory as the process will end):
t0rrant@bastion01 $ cd AdS_sympy_test t0rrant@bastion01 $ cat AdS_sympy.sbatch #!/bin/bash #SBATCH --job-name=AdS_sympy_test #SBATCH --output=/home/t0rrant/AdS_sympy_test/AdS_sympy_test-%j.out #SBATCH --error=/home/t0rrant/AdS_sympy_test/AdS_sympy_test-%j.err #SBATCH --mail-user=<your e-mail should go here> #SBATCH --mail-type=ALL #SBATCH --time=1:00:00 #SBATCH --mem=1G RUNPATH=/home/t0rrant/AdS_sympy_test/ cd $RUNPATH source $HOME/.virtualenvs/AdS/bin/activate python AdS_sympy_test.py
|--job-name||This will determine the job name which appears in commands like sacct, smap or sview.|
|--output||Determines the location of the file to which the stdout should be redirected to.|
|--error||Determines the location of the file to which the stderr should be redirected to.|
|--mail-user||The e-mail address to which job events should be sent to.|
|--mail-type||The types of events that should be sent, here we use all of them. For more info see [the docs].|
|--time||The amount of time we want allocated for this job.|
|--mem||The amount of memory we want allocated for this job.|
Looks good, let's submit it to the queue:
t0rrant@bastion01 $ sbatch AdS_sympy.sbatch Submitted batch job 27047
Now check for errors and output:
t0rrant@bastion01 $ cat /home/t0rrant/AdS_sympy_test/AdS_sympy_test-27047.err
No errors! =)
t0rrant@bastion01 $ cat /home/t0rrant/AdS_sympy_test/AdS_sympy_test-27047.out ============================= test process starts ============================== executable: /home/t0rrant/.virtualenvs/AdS/bin/python (2.7.13-final-0) [CPython] architecture: 64-bit cache: yes ground types: python numpy: None random seed: 7576360 hash randomization: on (PYTHONHASHSEED=1519633959) sympy/core/tests/test_basic.py ................ [OK] ================== tests finished: 16 passed, in 0.09 seconds ================== t0rrant@bastion01 $
As we can see it used the python executable from our virtual environment, used the sympy package also from that virtual environment and successfully ran the basic tests from the sympy package.