Benchmarking & Profiling

Benchmarking GPU code can be a bit tricky. Launching a kernel simply sends the program to the GPU and does not wait for it to finish executing. The hidden setup selects CUDA or AMDGPU, so the following examples run on either device:

a = to_gpu(rand(Float32, 256, 256)); b = similar(a);
display(@benchmark begin $b .= $a .* $a end)

BenchmarkTools.Trial: 10000 samples with 1 evaluation per sample.
 Range (min … max):  13.400 μs …   7.261 ms  ┊ GC (min … max): 0.00% … 97.71%
 Time  (median):     14.970 μs               ┊ GC (median):    0.00%
 Time  (mean ± σ):   18.348 μs ± 121.340 μs  ┊ GC (mean ± σ):  3.87% ±  0.98%

     ▁▅▆█▆▇▃▂                                                   
  ▁▃▆█████████▆▅▄▃▃▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ ▂
  13.4 μs         Histogram: frequency by time         24.9 μs <

 Memory estimate: 3.62 KiB, allocs estimate: 81.

This only measures the time taken to launch the kernel, not the time taken to execute it. Synchronize the array’s KernelAbstractions backend before stopping the timer:

display(@benchmark begin
	$b .= $a .* $a
	gpu_synchronize($b)
end)

BenchmarkTools.Trial: 10000 samples with 1 evaluation per sample.
 Range (min … max):  48.870 μs …  4.782 ms  ┊ GC (min … max): 0.00% … 92.54%
 Time  (median):     90.650 μs              ┊ GC (median):    0.00%
 Time  (mean ± σ):   90.395 μs ± 61.435 μs  ┊ GC (mean ± σ):  0.49% ±  0.93%

  ▃  ▁  ▃▇▂      ▇█▄▄▂▄▃▁                                     ▂
  ██▇███████▆▇▇▇▇█████████▇▇▄▅▅▄▁▆▆▄▅▅▁▁▄▄▄▄▃▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▆ █
  48.9 μs      Histogram: log(frequency) by time       202 μs <

 Memory estimate: 3.62 KiB, allocs estimate: 81.

Which shows us that the kernel actually took a lot longer to run. CUDA.jl also provides CUDA.@elapsed and CUDA.@time; AMDGPU.jl has analogous vendor-specific profiling tools.

Remember to synchronize the selected backend when benchmarking GPU code, or the measurement only includes kernel-launch overhead.

Application Profiling

When you have a larger application, with multiple calls to custom CUDA kernels, it is sometimes better to profile an entire application. We can use the NVIDIA Nsight system to profile an application. The CUDA.jl documentation¹ suggests installing NSight Systems directly from the NVIDIA website².

Example: Ensure that your NSight Systems installation is working correctly by typing the following into the terminal:

nsys --version

On Windows, you will need to run the terminal as an Administrator and also ensure that the folder with nsys.exe is in your system PATH environment variable.

NSight Systems can be used to view the bottlenecks in your system and diagnose whether kernels are being launched too frequently and hurting performance.

Footnotes