Benchmarking

Video

If the changes in measurement are only small fluctuations, we can average over a number of repeats to get a better idea of how long the code takes to run. This is where the idea of benchmarking comes in. Benchmarking can be thought of as a more systematic approach to timing a piece of code. In Julia, benchmarking can be handled by a package called BenchmarkTools.jl, which will be used frequently throughout this course. Just like the @time macro from before, this package provides a few more macros which are used to systematically time a piece of code. Let’s apply the @benchmark macro to our rand function:

import BenchmarkTools: @benchmark
display(@benchmark rand(1000,1000))

BenchmarkTools.Trial: 1656 samples with 1 evaluation per sample.
 Range (min … max):  1.286 ms … 8.274 ms  ┊ GC (min … max):  0.00% … 38.64%
 Time  (median):     2.806 ms             ┊ GC (median):     0.00%
 Time  (mean ± σ):   3.016 ms ± 1.777 ms  ┊ GC (mean ± σ):  12.09% ± 21.40%

  █                                                          
  ██▄▂▂▂▂▂▁▂▂▁▂▁▂▁▁▁▁▁▁▁▁▃▄▃▃▃▃▂▂▂▁▂▁▂▁▁▁▂▁▁▂▁▁▂▄▄▄▅▄▄▃▃▃▂▂ ▂
  1.29 ms        Histogram: frequency by time       5.83 ms <

 Memory estimate: 7.63 MiB, allocs estimate: 3.

Note: If you run @benchmark in the REPL, it will return a benchmark object which will be printed to the console, without the need to use the display function.

This outputs a much more comprehensive breakdown of the performance of the rand function. The first line reports how many samples were collected and how many evaluations of the code went into each sample. You are then given the range (the fastest and the slowest sample), the median and the mean ± standard deviation of the sample times, each paired with the percentage of that time which was spent in garbage collection. Underneath is a histogram of the samples, and the final line gives the total memory allocated per evaluation along with the number of allocations. One can see that this benchmark yielded a huge range of results. Notice that this benchmark also talks about memory and allocation data which is usually highly correlated with the execution speed of a function, which is discussed in more detail later in the chapter.

Why are some executions of this function so much faster than what we have seen before? To answer this, we must remember that Julia is a “Just-in-Time” compiled language. This means that when Julia executes a piece of code for the first time, it has to compile it into something the computer can process. Most of the in-built timing methods in Julia also include this compilation time. This is one of the main reasons for using the external package: it warms your code up before it starts collecting samples, so the timings it reports do not include the compilation time.

If you only care about the minimum time taken, you might prefer to use the @btime macro over the @benchmark since it gives a much more succinct output:

import BenchmarkTools: @btime
@btime rand(1000,1000);

1.289 ms (3 allocations: 7.63 MiB)

Note: This macro reports the minimum time taken over a set of evaluations (the value it returns is the result of the expression itself, so it can be wrapped around code you still want the answer from). This can give misleading results for code that may be intermittently slowed down, such as code with heavy amounts of allocations. Specifically, for code with a high number of allocations, it is often better to look at the histogram of results from a full @benchmark. If you are embedding the function you have benchmarked with @btime inside a loop, calling it N times, you may expect the result to take N times longer than the number given you by @btime, but this is often an underestimate. This is also key when comparing two implementations, as it is often important to compare the averages and not just the best case scenario.

All timings are usually given with an SI prefix, such as n, μ or m to indicate nano, micro or milli respectively. Remember, these stand for 10⁻⁹, 10⁻⁶ and 10⁻³ respectively.

If you want to be able to time a function and store the result in a variable, you can use the @belapsed macro to simply return the timing, aggregated (via the minimum) over many samples. If you need more precise control over how the benchmark is run, you should consult the documentation for BenchmarkTools.jl.

One very important factor when benchmarking using the macros from BenchmarkTools.jl, is to ensure that you are properly interpolating variables. There are usually allocations when using variables to call the functions, as these are often global variables, which are heap allocated. Let’s take a look at an example:

my_arr = rand(1024);
display(@benchmark sum(my_arr))

BenchmarkTools.Trial: 10000 samples with 950 evaluations per sample.
 Range (min … max):   95.779 ns … 452.968 ns  ┊ GC (min … max): 0.00% … 69.36%
 Time  (median):      97.884 ns               ┊ GC (median):    0.00%
 Time  (mean ± σ):   100.780 ns ±  11.886 ns  ┊ GC (mean ± σ):  0.18% ±  1.69%

  ▅▇██▆▄▂ ▁▂▃▄▅▄▃▃▂▁▂▂▂▂▂▁▁                                     ▂
  █████████████████████████▇█▇▇▇▇▇▇▆▇▇▆▅▆▆▅▁▅▆▆▃▅▄▄▅▄▄▅▄▅▅▄▃▆▅▄ █
  95.8 ns       Histogram: log(frequency) by time        137 ns <

 Memory estimate: 16 bytes, allocs estimate: 1.

Here, we have an allocation of just 16 bytes, but it does exist. We don’t expect this sum to allocate at all. The reason for this, is using the global variable to pass into our function. Instead of using this global variable, we should interpolate the contents inside the benchmark:

display(@benchmark sum($my_arr))

BenchmarkTools.Trial: 10000 samples with 975 evaluations per sample.
 Range (min … max):  70.205 ns … 194.687 ns  ┊ GC (min … max): 0.00% … 0.00%
 Time  (median):     71.897 ns               ┊ GC (median):    0.00%
 Time  (mean ± σ):   73.656 ns ±   6.056 ns  ┊ GC (mean ± σ):  0.00% ± 0.00%

  ▄▆█▇▇▄▂▃     ▃▅▂▃▂▁                                          ▂
  ████████▇▆▇▇█████████▇▇▇▆▆▆▇▆▆▆▆▅▆▅▅▄▄▅▄▄▅▃▄▅▂▄▃▃▄▃▄▄▃▄▂▄▅▂▃ █
  70.2 ns       Histogram: log(frequency) by time       101 ns <

 Memory estimate: 0 bytes, allocs estimate: 0.

Notice that this removed the mysterious allocation, while also seemingly improving the performance. It is critical that when passing in arguments to a function to be benchmarked via BenchmarkTools.jl, you must interpolate the values being passed into the function. Similarly, if you want to avoid creating a temporary variable, but do not want to benchmark its creation, you can interpolate:

display(@benchmark sum($(rand(1024))))

BenchmarkTools.Trial: 10000 samples with 974 evaluations per sample.
 Range (min … max):  69.795 ns … 184.302 ns  ┊ GC (min … max): 0.00% … 0.00%
 Time  (median):     71.324 ns               ┊ GC (median):    0.00%
 Time  (mean ± σ):   73.547 ns ±   8.971 ns  ┊ GC (mean ± σ):  0.00% ± 0.00%

  ██▆▄▂ ▂▅▄▃▂▁                                                 ▂
  █████████████▇▇▇█▇▇▇▇▆▅▇▇▆▆▆▅▅▆▅▄▃▅▅▄▃▄▄█▇▄▃▁▅▄▃▄▅▃▁▃▃▁▁▃▁▅▅ █
  69.8 ns       Histogram: log(frequency) by time       125 ns <

 Memory estimate: 0 bytes, allocs estimate: 0.

Interpolation is key to accurate benchmarking using the macros. There are many options to expand on this capability, such as setup arguments, which may be of interest to low level benchmarking.