JM

High Performance Computing in Julia

Course Info

  1. 1. Introduction

    1. 1.1. Setup

  2. 2. Foundations

    1. 2.1. Overview
    2. 2.2. von Neumann Architecture
    3. 2.3. Data Types
    4. 2.4. Operating Systems
    5. 2.5. Compilers and Machine Code
    6. 2.6. Hardware
    7. 2.7. Summary

  3. 3. Julia

    1. 3.1. Overview

  4. 4. Measuring Performance

    1. 4.1. Overview
    2. 4.2. Timing
    3. 4.3. Benchmarking
    4. 4.4. Profiling
    5. 4.5. Memory Allocations
    6. 4.6. Computational Complexity

  5. 5. Optimisation

    1. 5.1. Overview
    2. 5.2. Caching and Memory Locality
    3. 5.3. Stack vs Heap Memory
    4. 5.4. Reducing Allocations
    5. 5.5. Memoisation
    6. 5.6. Vectorisation vs Loops
    7. 5.7. Views
    8. 5.8. Speed of Operations
    9. 5.9. Type Stability
    10. 5.10. Performance Annotations
    11. 5.11. Conclusion

  6. 6. Parallel Programming

    1. 6.1. Overview
    2. 6.2. Dependency Graphs
    3. 6.3. Theoretical Expectations
    4. 6.4. Maps and Reductions
    5. 6.5. Thread Safety and Race Conditions
    6. 6.6. Producer and Consumer
    7. 6.7. When to Parallelise?

  7. 7. Multithreading

    1. 7.1. Overview
    2. 7.2. Monte Carlo Pi Estimation
    3. 7.3. Naive Parallel Implementation
    4. 7.4. False Sharing
    5. 7.5. Chunking
    6. 7.6. Task-based Parallelism
    7. 7.7. Packages
    8. 7.8. Summary
    9. 7.9. Exercises

  8. 8. Multiprocessing

    1. 8.1. Overview
    2. 8.2. Multiprocessing in Julia
    3. 8.3. Multiprocessing on a Cluster
    4. 8.4. Message Passing Interface
    5. 8.5. Summary
    6. 8.6. Exercises

  9. 9. GPU Programming

    1. 9.1. Introduction to GPU Programming
    2. 9.2. Modern GPU Hardware
    3. 9.3. GPU Vendor Overview
    4. 9.4. CUDA.jl
    5. 9.5. AMDGPU.jl
    6. 9.6. Portable GPU Kernels
    7. 9.7. CUDA Libraries
    8. 9.8. Benchmarking & Profiling
    9. 9.9. Tips
    10. 9.10. Case Study - Monte-Carlo Simulations
    11. 9.11. Exercises

Table of Contents

Exercises

  1. In the Case Study we found that the hand-written mc_random_walk_kernel! kernel was slower than the fused broadcast version, mc_random_walk_fused!. Rewrite the kernel so that it matches (or beats) the fused version, benchmarking both after synchronize(get_backend(y_gpu)) and checking that the two still produce statistically equivalent results. Hints: pre-generate all normal deviates in a device array and have each kernel invocation consume them; it is also worth sweeping KernelAbstractions workgroup sizes rather than assuming that 256256 is optimal.

  2. Take some of your existing code and port it to the GPU using the inbuilt array operations, then write a custom KernelAbstractions.@kernel for the more complicated operations. Verify correctness by comparing against the CPU result with isapprox, and benchmark both versions after synchronize(get_backend(array)) to check that moving to the GPU was actually worthwhile once the host-to-device copies are counted.

Back