JM

High Performance Computing in Julia

Course Info

  1. 1. Introduction

    1. 1.1. Setup

  2. 2. Foundations

    1. 2.1. Overview
    2. 2.2. von Neumann Architecture
    3. 2.3. Data Types
    4. 2.4. Operating Systems
    5. 2.5. Compilers and Machine Code
    6. 2.6. Hardware
    7. 2.7. Summary

  3. 3. Julia

    1. 3.1. Overview

  4. 4. Measuring Performance

    1. 4.1. Overview
    2. 4.2. Timing
    3. 4.3. Benchmarking
    4. 4.4. Profiling
    5. 4.5. Memory Allocations
    6. 4.6. Computational Complexity

  5. 5. Optimisation

    1. 5.1. Overview
    2. 5.2. Caching and Memory Locality
    3. 5.3. Stack vs Heap Memory
    4. 5.4. Reducing Allocations
    5. 5.5. Memoisation
    6. 5.6. Vectorisation vs Loops
    7. 5.7. Views
    8. 5.8. Speed of Operations
    9. 5.9. Type Stability
    10. 5.10. Performance Annotations
    11. 5.11. Conclusion

  6. 6. Parallel Programming

    1. 6.1. Overview
    2. 6.2. Dependency Graphs
    3. 6.3. Theoretical Expectations
    4. 6.4. Maps and Reductions
    5. 6.5. Thread Safety and Race Conditions
    6. 6.6. Producer and Consumer
    7. 6.7. When to Parallelise?

  7. 7. Multithreading

    1. 7.1. Overview
    2. 7.2. Monte Carlo Pi Estimation
    3. 7.3. Naive Parallel Implementation
    4. 7.4. False Sharing
    5. 7.5. Chunking
    6. 7.6. Task-based Parallelism
    7. 7.7. Packages
    8. 7.8. Summary
    9. 7.9. Exercises

  8. 8. Multiprocessing

    1. 8.1. Overview
    2. 8.2. Multiprocessing in Julia
    3. 8.3. Multiprocessing on a Cluster
    4. 8.4. Message Passing Interface
    5. 8.5. Summary
    6. 8.6. Exercises

  9. 9. GPU Programming

    1. 9.1. Introduction to GPU Programming
    2. 9.2. Modern GPU Hardware
    3. 9.3. GPU Vendor Overview
    4. 9.4. CUDA.jl
    5. 9.5. AMDGPU.jl
    6. 9.6. Portable GPU Kernels
    7. 9.7. CUDA Libraries
    8. 9.8. Benchmarking & Profiling
    9. 9.9. Tips
    10. 9.10. Case Study - Monte-Carlo Simulations
    11. 9.11. Exercises

Table of Contents

GPU Vendor Overview

Each manufacturer has differing architectures and programming models to support their GPUs. The most popular of these is NVIDIA and the CUDA programming model, which is NVIDIA’s proprietary platform for programming and using their GPUs. AMD has ROCm, which is a software stack for programming on their GPUs. Intel has also been developing oneAPI, which they intend to be useful across different hardware applications.

The main difference between CUDA and the other software stacks is that CUDA is proprietary, unlike the open source ROCm and oneAPI. There is also the OpenCL project, and its more modern successor SYCL (which oneAPI is built on), which aim to provide an API for programming across many devices, including GPUs from different manufacturers and even CPUs. Apple’s GPUs are programmed with their own Metal API.

Julia has a back-end package for each of these vendors - CUDA.jl for NVIDIA, AMDGPU.jl for AMD, oneAPI.jl for Intel and Metal.jl for Apple - and they deliberately share a common interface. On top of these sits KernelAbstractions.jl, which lets you write a single kernel that can be compiled for any of these back-ends (and for the CPU), at the cost of giving up some vendor-specific features.

CUDA remains an important part of this course because many modern HPC clusters provide NVIDIA GPUs and CUDA.jl exposes a mature set of vendor libraries. We also cover AMDGPU.jl for systems running ROCm. The common array-programming examples work with either device-array type, and our custom kernels use KernelAbstractions.jl so that one source implementation targets both back-ends.

Vendor-specific libraries and profiling tools still differ, so code that depends on them should remain in the corresponding CUDA or AMDGPU section.

Back Next