JM

High Performance Computing in Julia

Course Info

  1. 1. Introduction

    1. 1.1. Setup

  2. 2. Foundations

    1. 2.1. Overview
    2. 2.2. von Neumann Architecture
    3. 2.3. Data Types
    4. 2.4. Operating Systems
    5. 2.5. Compilers and Machine Code
    6. 2.6. Hardware
    7. 2.7. Summary

  3. 3. Julia

    1. 3.1. Overview

  4. 4. Measuring Performance

    1. 4.1. Overview
    2. 4.2. Timing
    3. 4.3. Benchmarking
    4. 4.4. Profiling
    5. 4.5. Memory Allocations
    6. 4.6. Computational Complexity

  5. 5. Optimisation

    1. 5.1. Overview
    2. 5.2. Caching and Memory Locality
    3. 5.3. Stack vs Heap Memory
    4. 5.4. Reducing Allocations
    5. 5.5. Memoisation
    6. 5.6. Vectorisation vs Loops
    7. 5.7. Views
    8. 5.8. Speed of Operations
    9. 5.9. Type Stability
    10. 5.10. Performance Annotations
    11. 5.11. Conclusion

  6. 6. Parallel Programming

    1. 6.1. Overview
    2. 6.2. Dependency Graphs
    3. 6.3. Theoretical Expectations
    4. 6.4. Maps and Reductions
    5. 6.5. Thread Safety and Race Conditions
    6. 6.6. Producer and Consumer
    7. 6.7. When to Parallelise?

  7. 7. Multithreading

    1. 7.1. Overview
    2. 7.2. Monte Carlo Pi Estimation
    3. 7.3. Naive Parallel Implementation
    4. 7.4. False Sharing
    5. 7.5. Chunking
    6. 7.6. Task-based Parallelism
    7. 7.7. Packages
    8. 7.8. Summary
    9. 7.9. Exercises

  8. 8. GPU Programming

    1. 8.1. Introduction to GPU Programming
    2. 8.2. Modern GPU Hardware
    3. 8.3. GPU Vendor Overview
    4. 8.4. High Level Introduction to CUDA
    5. 8.5. CUDA Kernels
    6. 8.6. CUDA Libraries
    7. 8.7. Benchmarking & Profiling
    8. 8.8. Tips
    9. 8.9. Case Study - Monte-Carlo Simulations
    10. 8.10. Exercises

Table of Contents

Conclusion

Optimisation is a very broad topic, and we are only able to cover the introduction to this topic here. Many of the tips discussed in this chapter are applicable to other languages. Many tips specific to Julia are given in the “Performance Tips” section of the Julia manual, which are certainly worth reading, but the main points are included in this chapter.

We have opted not to talk about specific optimisations by using the correct algorithm for a task, as this is often too specific to a certain problem. Instead, it should be your responsibility as the developer to choose the appropriate algorithm for the job. You can then use the techniques in this chapter to make your implementation of that algorithm as performant as possible, using a single core.

When writing high performance code, it is imperative that you first optimise the existing code as described in this chapter, before moving onto optimising via parallelism. The next few chapters will cover the basics of parallelism over the main paradigms available to programmers today. It is important to remember that the main source of speed up comes from optimising the serial version of your code, and these benefits can often be compounded by using the right parallel paradigm to further speed up the execution of your code.

Key Takeaways

The main optimisation strategies we’ve covered include:

  1. Memory Management: Understanding stack vs heap allocation and minimizing heap allocations
  2. Cache Efficiency: Writing cache-friendly code by respecting memory layout and locality
  3. Reducing Allocations: Preallocating arrays instead of growing them dynamically
  4. Type Stability: Ensuring the compiler can infer types at compile time
  5. Vectorisation: Using broadcasting and SIMD operations effectively
  6. Performance Annotations: Using @inbounds, @fastmath, and other compiler hints when appropriate

Remember that premature optimisation should be avoided - always profile your code first to identify actual bottlenecks before applying these techniques. The tools we covered in theMeasuring Performance chapter are essential for this process.

Back Next