Automatic mulit-threading with python numpy

This came up while running python codes on the cluster. It turns out that numpy vector operations are automatically parallelized if numpy is linked against certain libraries, e.g. openBLAS or MKL, during compilation. Those linear algebra libraries will automatically use the max number of available cores (or if your processor is HT, 2x number of physical cores) for matrix operations. While that might seem convenient, it actually made a lot of people unhappy because of the overhead involved with multithreading lots of tiny matrix operations. Fortunately, there is a way to control the max number of threads used, and some devs are working on a way of dynamic control via numpy.

I created the following basic test script. It generates two random matrices, then multiplies them together. The random number generation is a serial operation, but the dot product is parallelized by default.

import os
#must set these before loading numpy:
os.environ["OMP_NUM_THREADS"] = '8' # export OMP_NUM_THREADS=4
os.environ["OPENBLAS_NUM_THREADS"] = '8' # export OPENBLAS_NUM_THREADS=4
os.environ["MKL_NUM_THREADS"] = '8' # export MKL_NUM_THREADS=6
#os.environ["VECLIB_MAXIMUM_THREADS"] = '4' # export VECLIB_MAXIMUM_THREADS=4
#os.environ["NUMEXPR_NUM_THREADS"] = '4' # export NUMEXPR_NUM_THREADS=6

import numpy as np
import time

#np.__config__.show() #looks like I have MKL and blas
np.show_config()

start_time=time.time()
#test script:
a = np.random.randn(5000, 50000)
b = np.random.randn(50000, 5000)
ran_time=time.time()-start_time
print("time to complete random matrix generation was %s seconds" % ran_time)
np.dot(a, b) #this line should be multi-threaded
print("time to complete dot was %s seconds" % (time.time() - start_time - ran_time))

The lines under import os set environment variables. The one(s) you need to set depend on what your numpy is linked against, as shown by np.show_config(). Note that those must be set before importing numpy.

I ran some experiments on one of the compute nodes (dual e5-2690v2) using slurm execution. Software was anaconda 5.2, so anyone with a recent anaconda should have similar behavior. My np.show_config() returned information about MKL and openBLAS, so I think those are the relevant variables to set.

Test 1: slurm cpus-per-task not set, ntasks=1, no thread limiting variables set.
Results: No multi-threading because slurm defaults to one cpu per task.

Test 2: slurm cpus-per-task=10, ntasks=1, no thread limiting variable set.
Results: dot used 10 threads (10.4s)

Test 3: slurm cpus-per-task=20, ntasks=1, no thread limiting variable set.
Results: dot used 20 threads (5.4s)

Test 4: slurm cpus-per-task=4, ntasks=1, no thread limiting variable set.
Results: dot used 4 threads (24.8s)

Test 5: slurm cpus-per-task=10, ntasks=1, ntasks-per-socket=1, OMP_NUM_THREADS=4
Results: dot used 4 threads (24.8s)

Test 6: slurm cpus-per-task=10, ntasks=1, ntasks-per-socket=1, OPENBLAS_NUM_THREADS=4
Results: dot used 10 threads (10.4s)

Test 7: slurm cpus-per-task=10, ntasks=1, ntasks-per-socket=1, MKL_NUM_THREADS=4
Results: dot used 4 threads (24.9s)

Test 8: slurm cpus-per-task=10, ntasks=1, ntasks-per-socket=1, VECLIB_MAXIMUM_THREADS=4
Results: dot used 10 threads (10.4s)

Test 9: slurm cpus-per-task=10, ntasks=1, ntasks-per-socket=1, NUMEXPR_NUM_THREADS=4
Results: dot used 10 threads (10.4s)

Test 10: slurm cpus-per-task=10, ntasks=1, ntasks-per-socket=1, OMP_NUM_THREADS=8, OPENBLAS_NUM_THREADS=8, MKL_NUM_THREADS=8
Results: dot used 8 threads (12.5s)

As you can see above, setting either MKL or OMP_NUM_THREADS will limit the number of threads, though apparently openBLAS is not being used, at least for dot. Also, limiting the number of cpus available will also limit the number of threads.

For this code, which has to run on 100's of different cases that can be run simultaneously, it looks like giving one full socket per case (10 cores) is optimal. The environment variables don't need to be set because the default behavior is to use all available cores (limited by slurm). That's assuming np.dot is a good indicator, which it might not be because her code is far more complicated.

Anyways, I hope someone finds this useful.

Using the Cluster for Astrophysics-y stuff

I spend months building this awesome cluster, and the first person to fully use it is my partner, haha. Since they put up with the mess for so long, I think this is fair.

I don't have the physics background to understand their research, but they're processing a ton of astronomical data, and the algorithms require are very cpu intensive. Their scripts are in python, each case needs 8 cores, and they have a few hundred cases. Time to queue the cluster...

I installed anaconda on the headnode in /opt so it'd be available system wide. I created an environment module for this anaconda version, then  created the virtual environment(s) wanted. Then instead of installing it on each slave node, I just tar'd, copied the whole anaconda directory to /opt on each node, and untar'd. I also copied the environment modulefile. If I ever make massive changes, it will be faster/easier to re-clone all of the drives again, but for single program installs, it's faster to scp/ssh into each node and make modifications. I'll look into something like pssh in the future to make this sort of thing easier. Anyways, now that anaconda was installed, and the virtual environments available on all nodes, it was time to get slurm working.

Turns out it's super simple to call a python script from an sbatch script. You only need three lines after the #SBATCH setting lines:

module load python/anaconda-5.2
source activate myenv
python .py

However, when I tried running this for multiple cases, it would only assign one job per node, even though I had ntasks=1 and cpus-per-task=8. Theoretically, it should assign two jobs per node since each node has 20 cores (cores=cpus to slurm if hyperthreading is off). I had set SelectType and SelectParameters correctly in the slurm.conf, but it turns out that you have to add the OverSubscribe=YES parameter to the partition definition in slurm.conf, or slurm defaults to 1 job per node. This allowed for scheduling two of these jobs per node. I updated the slurm instructions in the software guide part 3 to cover this.

I couldn't find the answer to this online, but it turns out that it's perfectly fine to activate the same anaconda virtual environment more than once per node as long as you do it in separate terminal sessions (separate jobs for slurm) and are not making modifications to the environment during the runs. This made life a lot easier because it meant that we didn't have to try to track which environments were being used at the time of job launching.