Multiprocessing on a Cluster

Multiprocessing is your go-to parallelisation method on a cluster. A cluster is a collection of networked machines (often times referred to as ”nodes”), which are usually similar in hardware architecture. A cluster typically gives you access to hundreds, if not thousands of CPU cores. As clusters involve many machines, it can often be very difficult to network each of the processes together and facilitate communication. Fortunately, there is a package, ClusterManagers.jl, which removes most of the hard work.

In this course, we will focus on using SLURM¹. Other scheduling systems may have support inside of ClusterManagers.jl, and it is likely trivial to switch to a different cluster.

As a starting point, let’s introduce a bash script, which is used to request resources from the scheduler:

#!/bin/bash

#SBATCH --ntasks=32
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=2048
#SBATCH --time=00:10:00
#SBATCH -o test_job_%j.out

julia --project run_code.jl

This will request resources for $32$ CPUs and around $64$ GB of memory for a maximum time of $10$ minutes. It also starts the process with running the run_code.jl julia file. We can take a look inside to see the process for setting up the nodes:

using Pkg
using Distributed
using ClusterManagers
using BSON

println("Setting up SLURM!")
num_tasks = parse(Int, ENV["SLURM_NTASKS"])
cpus_per_task = parse(Int, ENV["SLURM_CPUS_PER_TASK"])
ENV["JULIA_NUM_THREADS"] = cpus_per_task
current_project = dirname(Pkg.project().path)
addprocs(SlurmManager(num_tasks);
    exeflags=[
        "--project=$current_project",
        "--threads=$cpus_per_task"
    ]
)
println("Workers: $(length(workers()))")

@everywhere include("allfunctions.jl")

parameters = get_parameters()

results = get_results(parameters)

BSON.@save "results.bson" results

At first, we have to perform a bit of prep work, by adding a separate process for each CPU and calling the correct cluster manager - SlurmManager. This also sets the number of threads Julia can use, such that each process has the right amount of threads. We have a separate file which defines the functions we need in our experiment. The get_results function internally calls functions like pmap which make use of multiprocessing.

Note: We make use of the BSON.jl library to serialise our results and save them in a file. BSON is similar to JSON but stores the data structure in binary, instead of in plain-text. This allows a compressed storage of information, at the cost of losing human readability. One does not need to implement their own serialisation and de-serialisation methods, but instead make use of BSON.jl through the @load and @save macros.

Footnotes