llvm-project
MBR - MLIR Benchmark Runner
MBR is a tool to run benchmarks. It measures compilation and running times of benchmark programs. It uses MLIR's python bindings for MLIR benchmarks.
Installation
To build and enable MLIR benchmarks, pass
while building MLIR. If you make some changes to the
files itself, build
again with
.
Writing benchmarks
As mentioned in the intro, this tool measures compilation and running times. An MBR benchmark is a python function that returns two callables, a compiler and a runner. Here's an outline of a benchmark; we explain its working after the example code.
def benchmark_something(): # Preliminary setup def compiler(): # Compiles a program and creates an "executable object" that can be # called to invoke the compiled program. ...
def runner(executable_object): # Sets up arguments for executable_object and calls it. The # executable_object is returned by the compiler. # Returns an integer representing running time in nanoseconds. ...
return compiler, runner
The benchmark function's name must be prefixed by
and benchmarks
must be in the python files prefixed by
for them to be
discoverable. The file and function prefixes are configurable using the
configuration file
relative to this README's directory.
A benchmark returns two functions, a
and a
. The
returns a callable which is accepted as an argument by the runner function.
So the two functions work like this
: configures and returns a callable.compiler
: takes that callable in as input, sets up its arguments, and calls it. Returns an int representing running time in nanoseconds.runner
The
callable is optional if there is no compilation step, for
example, for benchmarks involving numpy. In that case, the benchmarks look
like this.
def benchmark_something(): # Preliminary setup def runner(): # Run the program and return the running time in nanoseconds. ...
return None, runner
In this case, the runner does not take any input as there is no compiled object to invoke.
Running benchmarks
MLIR benchmarks can be run like this
PYTHONPATH=<path_to_python_mlir_core> <other_env_vars> python <llvm-build-path>/bin/mlir-mbr --machine <machine_identifier> --revision <revision_string> --result-stdout <path_to_start_search_for_benchmarks>
For a description of command line arguments, run
python mlir/utils/mbr/mbr/main.py -h
And to learn more about the other arguments, check out the LNT's documentation page here.
If you want to run only specific benchmarks, you can use the positional argument
appropriately.
- If you want to run benchmarks in a specific directory or a file, set
to that.top_level_path - If you want to run a specific benchmark function, set the
to the file containing that benchmark function, followed by atop_level_path
, and then the benchmark function name. For example,::
.mlir/benchmark/python/benchmark_sparse.py::benchmark_sparse_mlir_multiplication
Configuration
Various aspects about the framework can be configured using the configuration
file in the
relative to the directory of this README.