Exercises

Question 1

Take the following workload that scales linearly:

function example_inner_loop(k)
    s = zero(Float64)
    for _ in 1:k
        s += rand(Float64)
    end
    s
end
function example_workload(n, k=100)
    results = Vector{Float64}(undef, n)
    for i in 1:n
        results[i] = example_inner_loop(k)
    end
    return results
end

(i). What does the parameter k control?

(ii). Implement a multithreaded version of example_workload.

(iii). Implement a parallel version of example_workload using multiprocessing.

(iv). Design an experiment to compare the two implementations of example_workload, and the original, serial, implementation, as you vary n.

(v). Use the results of the previous experiment to estimate the latency relationship of both Threads.@threads and Distributed.pmap to the variables n and k.

Question 2

If you have an embarrassingly parallel problem, which relies on non-thread safe code, which parallel programming paradigm is most suitable?

Answer: Multiprocessing - as each process will have its own isolated memory copy which can ensure parallel processing without race conditions.

Question 3

Look at the following struct:

struct RunningStats{T}
    min::T
    max::T
    mean::T
    num_samples::Int
end

(i). Define a function which reduces two samples of type T into a single RunningStats.

(ii). Extend the previous function to define reductions between single values of type T, and RunningStats, in any order, as well as between two RunningStats.

(iii). Test this custom reduction using the reduce function and the Statistics.jl standard library on a vector of random floats. Calculate the statistics of this vector and compare them to the reduced statistics output.

(iv). Write a multiprocessing parallel implementation of the reduction calculated in the previous part, using a Monte-Carlo process of your choice which produces a single value.

Question 4

Take the following setup:

using Random
using LinearAlgebra
using Statistics

const rng = Random.Xoshiro(1234)
function generate_seeds(rng, n)
    seeds = zeros(Int, n)
    for i in 1:n
        seeds[i] = rand(rng, Int)
    end
    return seeds
end
const n = 32
const seeds = generate_seeds(rng, n)

function long_calculation(seed; l=10)
    c_rng = Random.Xoshiro(seed)

    k = 12
    s = 2^k
    matrix_size = (s,s)

    matrix = rand(c_rng, Float64, matrix_size) .* 2 .- 1
    cache = similar(matrix)

    for j = 1:l
        mul!(cache, matrix, matrix)
        matrix .= cache ./ s .- matrix
    end

    return Statistics.mean(cache)
end

const results = map(long_calculation, seeds)

(i). Use multiprocessing to parallelise this calculation and make sure the number of BLAS threads is set to $1$.

(ii). Implement memoised caching on the long calculation, which uses the disk as a cache.

(iii). Use the previous parts to create a script which can recover from failure, without losing too much progress.