Multiprocessing

In the last chapter, we talked about scheduling tasks across different threads in a program, all of which have access to the same set of shared memory. Multiprocessing takes a different approach, and isolates each worker in their own process. Isolation means that each worker cannot access another’s memory. Any sharing of information has to be done via communication between each process. This communication can happen over sockets if each process lives on the same machine. If the processes exist across multiple nodes, this communication is facilitated via network communication over ports.

In multiprocessing, one starts $n$ processes independently, all of which have their own isolated memory space for storing a copy of their code as well as space for the heap and the stack. One must implement some framework or library to facilitate communication of these different processes. The way in which the processes are connected is called the topology. There are different models that can be used to structure the processes, but one of the most common is the master-worker configuration, which has one central process (the master) which coordinates the worker processes.

Multiprocessing is the commonly found parallel programming paradigm in languages like MATLAB and Python. In MATLAB, using parfor instead of for will distribute the calculation of the inner loop across multiple processes. While this is a very easy way to parallelise code, if one does not have a parallel pool (i.e. a group of MATLAB processes) running, it can take several minutes to start executing the code. Multiprocessing can have a huge latency overhead, as each worker has to start an entirely new process. This involves loading an entire copy of the language runtime, and orchestrating the communication channels between each process so that tasks can be appropriately scheduled. In the case of MATLAB, you are loading up an entire copy of the MATLAB runtime (excluding the GUI) for each worker requested.

Fortunately, Python’s runtime is much smaller, and so the overhead is actually very small when using multiprocessing. Python¹ must use multiprocessing due to the built-in Global Interpreter Lock (GIL), which interferes with threads that try to run at the same time, effectively allowing only concurrency but not parallelism inside a single process.

In Julia, the standard way to use multiprocessing is via the Distributed.jl package, included in the base library.

We can summarise the main traits of multiprocessing below:

• A multiprocessing paradigm uses several copies of the runtime to perform tasks in parallel. Each copy lives in a separate process, unable to access the memory space of another process.
• Any communication between processes must occur via sockets or over the network, and is usually very high latency. This includes sharing the results of calculations.
• Each process can have access to multiple threads allowing the combination of multiprocessing and multithreading paradigms.
• All the processes used need not exist on the same machine, but instead, can be spread across several machines. We often refer to a group of machines networked together as a cluster, which is usually where multiprocessing is used the most.

Footnotes