benchtests/README

   1 Using the glibc microbenchmark suite
   2 ====================================
   3
   4 The glibc microbenchmark suite automatically generates code for specified
   5 functions, builds and calls them repeatedly for given inputs to give some
   6 basic performance properties of the function.
   7
   8 Running the benchmark:
   9 =====================
  10
  11 The benchmark needs python 2.7 or later in addition to the
  12 dependencies required to build the GNU C Library.  One may run the
  13 benchmark by invoking make as follows:
  14
  15   $ make bench
  16
  17 This runs each function for 10 seconds and appends its output to
  18 benchtests/bench.out.  To ensure that the tests are rebuilt, one could run:
  19
  20   $ make bench-clean
  21
  22 The duration of each test can be configured setting the BENCH_DURATION variable
  23 in the call to make.  One should run `make bench-clean' before changing
  24 BENCH_DURATION.
  25
  26   $ make BENCH_DURATION=1 bench
  27
  28 The benchmark suite does function call measurements using architecture-specific
  29 high precision timing instructions whenever available.  When such support is
  30 not available, it uses clock_gettime (CLOCK_MONOTONIC).  One can force the
  31 benchmark to use clock_gettime by invoking make as follows:
  32
  33   $ make USE_CLOCK_GETTIME=1 bench
  34
  35 Again, one must run `make bench-clean' before changing the measurement method.
  36
  37 On x86 processors, RDTSCP instruction provides more precise timing data
  38 than RDTSC instruction.  All x86 processors since 2010 support RDTSCP
  39 instruction.  One can force the benchmark to use RDTSCP by invoking make
  40 as follows:
  41
  42   $ make USE_RDTSCP=1 bench
  43
  44 One must run `make bench-clean' before changing the measurement method.
  45
  46 Running benchmarks on another target:
  47 ====================================
  48
  49 If the target where you want to run benchmarks is not capable of building the
  50 code or you're cross-building, you could build and execute the benchmark in
  51 separate steps.  On the build system run:
  52
  53   $ make bench-build
  54
  55 and then copy the source and build directories to the target and run the
  56 benchmarks from the build directory as usual:
  57
  58   $ make bench
  59
  60 make sure the copy preserves timestamps by using either rsync or scp -p
  61 otherwise the above command may try to build the benchmark again.  Benchmarks
  62 that require generated code to be executed during the build are skipped when
  63 cross-building.
  64
  65 Building benchmarks as static executables:
  66 =========================================
  67
  68 To build benchmarks as static executables, on the build system, run:
  69
  70   $ make STATIC-BENCHTESTS=yes bench-build
  71
  72 You can copy benchmark executables to another machine and run them
  73 without copying the source nor build directories.
  74
  75 Running subsets of benchmarks:
  76 ==============================
  77
  78 To run only a subset of benchmarks, one may invoke make as follows
  79
  80   $ make bench BENCHSET="bench-pthread bench-math malloc-thread"
  81
  82 where BENCHSET may be a space-separated list of the following values:
  83
  84     bench-math
  85     bench-pthread
  86     bench-string
  87     hash-benchset
  88     malloc-thread
  89     math-benchset
  90     stdio-common-benchset
  91     stdlib-benchset
  92     string-benchset
  93     wcsmbs-benchset
  94
  95 Adding a function to benchtests:
  96 ===============================
  97
  98 If the name of the function is `foo', then the following procedure should allow
  99 one to add `foo' to the bench tests:
 100
 101 - Append the function name to the bench variable in the Makefile.
 102
 103 - Make a file called `foo-inputs` to provide the definition and input for the
 104   function.  The file should have some directives telling the parser script
 105   about the function and then one input per line.  Directives are lines that
 106   have a special meaning for the parser and they begin with two hashes '##'.
 107   The following directives are recognized:
 108
 109   - args: This should be assigned a colon separated list of types of the input
 110     arguments.  This directive may be skipped if the function does not take any
 111     inputs.  One may identify output arguments by nesting them in <>.  The
 112     generator will create variables to get outputs from the calling function.
 113   - ret: This should be assigned the type that the function returns.  This
 114     directive may be skipped if the function does not return a value.
 115   - includes: This should be assigned a comma-separated list of headers that
 116     need to be included to provide declarations for the function and types it
 117     may need (specifically, this includes using "#include <header>").
 118   - include-sources: This should be assigned a comma-separated list of source
 119     files that need to be included to provide definitions of global variables
 120     and functions (specifically, this includes using "#include "source").
 121     See pthread_once-inputs and pthreads_once-source.c for an example of how
 122     to use this to benchmark a function that needs state across several calls.
 123   - init: Name of an initializer function to call to initialize the benchtest.
 124   - name: See following section for instructions on how to use this directive.
 125
 126   Lines beginning with a single hash '#' are treated as comments.  See
 127   pow-inputs for an example of an input file.
 128
 129 Multiple execution units per function:
 130 =====================================
 131
 132 Some functions have distinct performance characteristics for different input
 133 domains and it may be necessary to measure those separately.  For example, some
 134 math functions perform computations at different levels of precision (64-bit vs
 135 240-bit vs 768-bit) and mixing them does not give a very useful picture of the
 136 performance of these functions.  One could separate inputs for these domains in
 137 the same file by using the `name' directive that looks something like this:
 138
 139   ##name: 240bits
 140
 141 All inputs after the ##name: 240bits directive and until the next `name'
 142 directive (or the end of file) are part of the "240bits" benchmark and
 143 will be output separately in benchtests/bench.out.  See the pow-inputs file
 144 for an example of what such a partitioned input file would look like.
 145
 146 It is also possible to measure latency and reciprocal throughput of a
 147 (partial) trace extracted from a real workload.  In this case the whole trace
 148 is iterated over multiple times rather than repeating every input multiple
 149 times.  This can be done via:
 150
 151   ##name: workload-<name>
 152
 153 where <name> is simply used to distinguish between different traces in the
 154 same file.  To create such a trace, you can simply extract using printf()
 155 values uses for a specific application, or generate random values in some
 156 interval.  See the expf-inputs file for an example of this workload mechanism.
 157
 158 Benchmark Sets:
 159 ==============
 160
 161 In addition to standard benchmarking of functions, one may also generate
 162 custom outputs for a set of functions.  This is currently used by string
 163 function benchmarks where the aim is to compare performance between
 164 implementations at various alignments and for various sizes.
 165
 166 To add a benchset for `foo':
 167
 168 - Add `foo' to the benchset variable.
 169 - Write your bench-foo.c that prints out the measurements to stdout.
 170 - On execution, a bench-foo.out is created in $(objpfx) with the contents of
 171   stdout.
 172
 173 Reading String Benchmark Results:
 174 ================================
 175
 176 Some of the string benchmark results are now in JSON to make it easier to read
 177 in scripts.  Use the benchtests/compare_strings.py script to show the results
 178 in a tabular format, generate graphs and more. Run
 179
 180     benchtests/scripts/compare_strings.py -h
 181
 182 for usage information.