JM

High Performance Computing in Julia

Course Info

  1. 1. Introduction

    1. 1.1. Setup

  2. 2. Foundations

    1. 2.1. Overview
    2. 2.2. von Neumann Architecture
    3. 2.3. Data Types
    4. 2.4. Operating Systems
    5. 2.5. Compilers and Machine Code
    6. 2.6. Hardware
    7. 2.7. Summary

  3. 3. Julia

    1. 3.1. Overview

  4. 4. Measuring Performance

    1. 4.1. Overview
    2. 4.2. Timing
    3. 4.3. Benchmarking
    4. 4.4. Profiling
    5. 4.5. Memory Allocations
    6. 4.6. Computational Complexity

  5. 5. Optimisation

    1. 5.1. Overview
    2. 5.2. Caching and Memory Locality
    3. 5.3. Stack vs Heap Memory
    4. 5.4. Reducing Allocations
    5. 5.5. Memoisation
    6. 5.6. Vectorisation vs Loops
    7. 5.7. Views
    8. 5.8. Speed of Operations
    9. 5.9. Type Stability
    10. 5.10. Performance Annotations
    11. 5.11. Conclusion

  6. 6. Parallel Programming

    1. 6.1. Overview
    2. 6.2. Dependency Graphs
    3. 6.3. Theoretical Expectations
    4. 6.4. Maps and Reductions
    5. 6.5. Thread Safety and Race Conditions
    6. 6.6. Producer and Consumer
    7. 6.7. When to Parallelise?

  7. 7. Multithreading

    1. 7.1. Overview
    2. 7.2. Monte Carlo Pi Estimation
    3. 7.3. Naive Parallel Implementation
    4. 7.4. False Sharing
    5. 7.5. Chunking
    6. 7.6. Task-based Parallelism
    7. 7.7. Packages
    8. 7.8. Summary
    9. 7.9. Exercises

  8. 8. Multiprocessing

    1. 8.1. Overview
    2. 8.2. Multiprocessing in Julia
    3. 8.3. Multiprocessing on a Cluster
    4. 8.4. Message Passing Interface
    5. 8.5. Summary
    6. 8.6. Exercises

  9. 9. GPU Programming

    1. 9.1. Introduction to GPU Programming
    2. 9.2. Modern GPU Hardware
    3. 9.3. GPU Vendor Overview
    4. 9.4. CUDA.jl
    5. 9.5. AMDGPU.jl
    6. 9.6. Portable GPU Kernels
    7. 9.7. CUDA Libraries
    8. 9.8. Benchmarking & Profiling
    9. 9.9. Tips
    10. 9.10. Case Study - Monte-Carlo Simulations
    11. 9.11. Exercises

Table of Contents

Message Passing

Arguably the most common approach to multiprocessing is by using a message passing approach. MPI (Message Passing Interface) is a standardised specification for facilitating message-passing for parallel computing in distributed systems. As this is only a specification, there exist several implementations such as OpenMPI and MPICH. Some of these implementations are specialised for the network hardware used on a specific cluster.

The MPI approach has the developer write a program which will execute simultaneously across each process. The MPI library exposes routines which allow each process to be able to identify itself and send data to and from each process. Additionally, the MPI library provides a barrier function which allows all the processes to be able to synchronise their execution.

In Julia, one can use the MPI.jl package to follow this programming model which may be effective for your use case. However, Distributed.jl remains the standard for multiprocessing, as it has a much less verbose programming model, making it easier to get started.

Back Next