Last article In this article, we will further understand the common methods of google benchmark.

Index of this article

• Passing parameters to test cases
• Simplify the generation of multiple similar test cases
• Using parameter generator

Passing parameters to test cases

Previously, all our test cases accepted only one benchmark:: State & type parameter. What if we need to pass additional parameters to the test case?

For example, if we need to implement a queue, there are two implementations available, ring buffer and linked list. Now we will test the performance of the two schemes in different situations:

// Necessary data structure
#include "ring.h"
#include "linked_ring.h"

// ring buffer test
static void bench_array_ring_insert_int_10(benchmark::State& state)
{
    auto ring = ArrayRing<int>(10);
    for (auto _: state) {
        for (int i = 1; i <= 10; ++i) {
            ring.insert(i);
        }
        state.PauseTiming(); // Pause time
        ring.clear();
        state.ResumeTiming(); // Recovery Timing
    }
}
BENCHMARK(bench_array_ring_insert_int_10);

// Testing of linked list
static void bench_linked_queue_insert_int_10(benchmark::State &state)
{
    auto ring = LinkedRing<int>{};
    for (auto _:state) {
        for (int i = 0; i < 10; ++i) {
            ring.insert(i);
        }
        state.PauseTiming();
        ring.clear();
        state.ResumeTiming();
    }
}
BENCHMARK(bench_linked_queue_insert_int_10);

// There are also tests for deletion, and tests for string, which are highly repetitive code and are not listed here.

Obviously, the above tests are no different except for the type of test and the amount of data inserted. If you can control the input parameters, you can write a lot less repetitive code.

Writing duplicate code is a waste of time and often means you're doing something stupid, as google engineers have noticed for a long time. Although the test case can only accept a benchmark:: State & type parameter, we can pass the parameter to the state object and get it in the test case:

static void bench_array_ring_insert_int(benchmark::State& state)
{
    auto length = state.range(0);
    auto ring = ArrayRing<int>(length);
    for (auto _: state) {
        for (int i = 1; i <= length; ++i) {
            ring.insert(i);
        }
        state.PauseTiming();
        ring.clear();
        state.ResumeTiming();
    }
}
BENCHMARK(bench_array_ring_insert_int)->Arg(10);

The example above shows how to pass and get parameters:

1. Arg Method of Objects Generated by BENCHMARK Macro for Passing Parameters
2. The parameters passed in will be stored in the state object, which is acquired by range method. The parameter 0 at the time of invocation is the requirement of the parameters, and corresponds to the first parameter.

The Arg method can only pass one parameter at a time. What if it wants to pass more than one parameter at a time? It's also simple:

static void bench_array_ring_insert_int(benchmark::State& state)
{
    auto ring = ArrayRing<int>(state.range(0));
    for (auto _: state) {
        for (int i = 1; i <= state.range(1); ++i) {
            ring.insert(i);
        }
        state.PauseTiming();
        ring.clear();
        state.ResumeTiming();
    }
}
BENCHMARK(bench_array_ring_insert_int)->Args({10, 10});

The above example is meaningless, just to show how to pass multiple parameters. The Args method accepts a vector object, so we can simplify the code by using the bracket initializer provided by c++11. The parameters are still obtained through the state.range method. 1 corresponds to the second parameter passed in.

It's worth noting that parameter passing can only accept integers. If you want to use other types of additional parameters, you need to think about something else.

Simplify the generation of multiple similar test cases

The ultimate goal of passing parameters to test cases is to generate multiple test cases without writing duplicate code. After knowing how to pass parameters, you may write as follows:

static void bench_array_ring_insert_int(benchmark::State& state)
{
    auto length = state.range(0);
    auto ring = ArrayRing<int>(length);
    for (auto _: state) {
        for (int i = 1; i <= length; ++i) {
            ring.insert(i);
        }
        state.PauseTiming();
        ring.clear();
        state.ResumeTiming();
    }
}
// Next we generate test cases with 10, 100, 1000 test inserts
BENCHMARK(bench_array_ring_insert_int)->Arg(10);
BENCHMARK(bench_array_ring_insert_int)->Arg(100);
BENCHMARK(bench_array_ring_insert_int)->Arg(1000);

Here we generate three instances that produce the following results:

It seems to work well, doesn't it?

Yes, the results are correct, but remember what we said earlier - don't write duplicate code! Yes, above we wrote the use case generation manually, and there was avoidable duplication.

Fortunately, Arg and Args will register the parameters used by our test cases to generate new test cases with use case names/parameters, and return a pointer to the BENCHMARK macro-generated object. In other words, if we want to generate multiple tests with only different parameters, we only need to call Arg and Args in a chain:

BENCHMARK(bench_array_ring_insert_int)->Arg(10)->Arg(100)->Arg(1000);

The result is the same as above.

But this is not the optimal solution. We still call the Arg method repeatedly, and if we need more use cases, we have to do more work.

google benchmark also has a solution: we can use Range method to automatically generate parameters within a certain range.

First look at the prototype of Range:

BENCHMAEK(func)->Range(int64_t start, int64_t limit);

start denotes the starting value of the parameter range, limit denotes the ending value of the range, and Range acts on a closed interval.

But if we rewrite the code like this, we will get a wrong test result:

BENCHMARK(bench_array_ring_insert_int)->Range(10, 1000);

Why? That's because Range defaults to a power of a base except start and limit, and base defaults to 8, so we'll see 64 and 512, which are the sum of squares and cubes of 8, respectively.

It's easy to change this behavior by simply resetting the base, using the Range Multiplier method:

BENCHMARK(bench_array_ring_insert_int)->RangeMultiplier(10)->Range(10, 1000);

Now the results are back to normal.

Ranges can be used to deal with multiple parameters:

BENCHMARK(func)->RangeMultiplier(10)->Ranges({{10, 1000}, {128， 256}});

The first scope specifies the scope of the first incoming parameter of the test case, while the second scope specifies the possible value of the second incoming parameter (note that this is not the scope).

Equivalent to the following code:

BENCHMARK(func)->Args({10, 128})
               ->Args({100, 128})
               ->Args({1000, 128})
               ->Args({10, 256})
               ->Args({100, 256})
               ->Args({1000, 256})

In fact, a Cartesian product is made by using the range of the first parameter generated to the parameters specified later.

Using parameter generator

What if I want to customize more complex parameters that are irregular? At this point, you need to implement a custom parameter generator.

The signature of a parameter generator is as follows:

void CustomArguments(benchmark::internal::Benchmark* b);

We calculate the parameters in the generator and then call the Arg or Args method of the benchmark::internal::Benchmark object to pass in the parameters as in the previous two sections.

Then we use the Apply method to apply the generator to the test case.

BENCHMARK(func)->Apply(CustomArguments);

In fact, the principle of this process is not complicated, I will give a simple explanation:

1. The BENCHMARK macro produces a benchmark::internal::Benchmark object and then returns its pointer.
2. Passing parameters to the benchmark::internal::Benchmark object requires methods such as Arg and Args.
3. Apply methods apply functions in parameters to themselves
4. We use benchmark::internal::Benchmark object pointer b Args and other methods to pass parameters in the generator, when b actually points to our test case.

So far, it's clear how the generator works, and of course, the conclusion from the above is that we can let Apply do more.

Here's how Apply works:

// This time we generated 100, 200,..., 1000 test cases, and these parameters could not be generated with range.
static void custom_args(benchmark::internal::Benchmark* b)
{
    for (int i = 100; i <= 1000; i += 100) {
        b->Arg(i);
    }
}

BENCHMARK(bench_array_ring_insert_int)->RangeMultiplier(10)->Apply(custom_args);

Test results of custom parameters:

So far, all the methods of passing parameters to test cases have been introduced.

In the next article, I will show you how to write test cases as templates. Passing parameters can only solve part of the duplicate code. For test cases of different types to be tested with similar methods, using templates will greatly reduce our unnecessary work.