JM

High Performance Computing in Julia

Course Info

  1. 1. Introduction

    1. 1.1. Setup

  2. 2. Foundations

    1. 2.1. Overview
    2. 2.2. von Neumann Architecture
    3. 2.3. Data Types
    4. 2.4. Operating Systems
    5. 2.5. Compilers and Machine Code
    6. 2.6. Hardware
    7. 2.7. Summary

  3. 3. Julia

    1. 3.1. Overview

  4. 4. Measuring Performance

    1. 4.1. Overview
    2. 4.2. Timing
    3. 4.3. Benchmarking
    4. 4.4. Profiling
    5. 4.5. Memory Allocations
    6. 4.6. Computational Complexity

  5. 5. Optimisation

    1. 5.1. Overview
    2. 5.2. Caching and Memory Locality
    3. 5.3. Stack vs Heap Memory
    4. 5.4. Reducing Allocations
    5. 5.5. Memoisation
    6. 5.6. Vectorisation vs Loops
    7. 5.7. Views
    8. 5.8. Speed of Operations
    9. 5.9. Type Stability
    10. 5.10. Performance Annotations
    11. 5.11. Conclusion

  6. 6. Parallel Programming

    1. 6.1. Overview
    2. 6.2. Dependency Graphs
    3. 6.3. Theoretical Expectations
    4. 6.4. Maps and Reductions
    5. 6.5. Thread Safety and Race Conditions
    6. 6.6. Producer and Consumer
    7. 6.7. When to Parallelise?

  7. 7. Multithreading

    1. 7.1. Overview
    2. 7.2. Monte Carlo Pi Estimation
    3. 7.3. Naive Parallel Implementation
    4. 7.4. False Sharing
    5. 7.5. Chunking
    6. 7.6. Task-based Parallelism
    7. 7.7. Packages
    8. 7.8. Summary
    9. 7.9. Exercises

  8. 8. GPU Programming

    1. 8.1. Introduction to GPU Programming
    2. 8.2. Modern GPU Hardware
    3. 8.3. GPU Vendor Overview
    4. 8.4. High Level Introduction to CUDA
    5. 8.5. CUDA Kernels
    6. 8.6. CUDA Libraries
    7. 8.7. Benchmarking & Profiling
    8. 8.8. Tips
    9. 8.9. Case Study - Monte-Carlo Simulations
    10. 8.10. Exercises

Table of Contents

GPU Vendor Overview

Each manufacturer has differing architectures and programming models to support their GPUs. The most popular of these is NVIDIA and the CUDA programming model, which is NVIDIA’s proprietary platform for programming and using their GPUs. AMD has ROCm, which is a software stack for programming on their GPUs. Intel has also been developing oneAPI, which they intend to be useful across different hardware applications.

The main difference between CUDA and the other software stacks is that CUDA is proprietary, unlike the open source ROCm and oneAPI. There is also the OpenCL project which aims to provide an API for programming across many devices, including GPUs from different manufacturers and even CPUs.

For this book, we are choosing CUDA as our vehicle to explore GPU programming, as it is the most fully-featured and supported of the software stacks. Many modern HPC clusters will have NVIDIA GPUs available, and they have the vast majority of the market share of the desktop GPU market.

Skills from learning to use CUDA, should be applicable to other software stacks, but know that there is a learning curve when transitioning from one platform to another.

Back Next