Mistakes in Micro Benchmarks

so on my last post i showed a bunch of small micro benchmark, and aside from the actual results, i wasn’t really sure what was going on there. luckily, i know a few perf experts, so i was able to lean on them.

in particular, the changes that were recommended were:

• don’t make just a single tiny operation, it is easy to get too much jitter in the setup for the call if the op is too cheap.
• pay attention to potential data issues, the compiler/jit can decide to put something on a register, in which case you are benching the cpu directly, which won’t be the case in the real world.

i also learned how to get the actual assembly being run, which is great. all in all, we get the following benchmark code:

[benchmarktask(platform: benchmarkplatform.x86,
            jitversion: benchmarkjitversion.ryujit)]
[benchmarktask(platform: benchmarkplatform.x86,
            jitversion: benchmarkjitversion.legacyjit)]
[benchmarktask(platform: benchmarkplatform.x64,
                jitversion: benchmarkjitversion.legacyjit)]
[benchmarktask(platform: benchmarkplatform.x64,
                jitversion: benchmarkjitversion.ryujit)]
public unsafe class tocastornottocast
{
    byte* p1, p2, p3, p4;
    fooheader* h1, h2,h3,h4;
    public tocastornottocast()
    {
        p1 = (byte*)marshal.allochglobal(1024);
        p2 = (byte*)marshal.allochglobal(1024);
        p3 = (byte*)marshal.allochglobal(1024);
        p4 = (byte*)marshal.allochglobal(1024);
        h1 = (fooheader*)p1;
        h2 = (fooheader*)p2;
        h3 = (fooheader*)p3;
        h4 = (fooheader*)p4;
    }

    [benchmark]
    [operationsperinvoke(4)]
    public void nocast()
    {
        h1->pagenumber++;
        h2->pagenumber++;
        h3->pagenumber++;
        h4->pagenumber++;
    }

    [benchmark]
    [operationsperinvoke(4)]
    public void cast()
    {
        ((fooheader*)p1)->pagenumber++;
        ((fooheader*)p2)->pagenumber++;
        ((fooheader*)p3)->pagenumber++;
        ((fooheader*)p4)->pagenumber++;
    }
}

and the following results:

          method | platform |       jit |   avrtime |    stddev |             op/s |
---------------- |--------- |---------- |---------- |---------- |----------------- |
            cast |      x64 | legacyjit | 0.2135 ns | 0.0113 ns | 4,683,511,436.74 |
          nocast |      x64 | legacyjit | 0.2116 ns | 0.0017 ns | 4,725,696,633.67 |
            cast |      x64 |    ryujit | 0.2177 ns | 0.0038 ns | 4,593,221,104.97 |
          nocast |      x64 |    ryujit | 0.2097 ns | 0.0006 ns | 4,769,090,600.54 |

---------------- |--------- |---------- |---------- |---------- |----------------- |
            cast |      x86 | legacyjit | 0.7465 ns | 0.1743 ns | 1,339,630,922.79 |
          nocast |      x86 | legacyjit | 0.7474 ns | 0.1320 ns | 1,337,986,425.19 |
            cast |      x86 |    ryujit | 0.7481 ns | 0.3014 ns | 1,336,808,932.91 |
          nocast |      x86 |    ryujit | 0.7426 ns | 0.0039 ns | 1,346,537,728.81 |

interestingly enough, the nocast approach is faster in pretty much all setups.

here is the assembly code for legacyjit in x64:

for ryujit, the code is identical for the cast code, and the only difference in the no casting code is that the mov edx, ecx is mov rdx,rcx in ryujit.

as an aside, x64 assembly code is much easier to read than x86 assembly code.

in short, casting or not casting has a very minor performance difference, but not casting allows us to save a pointer reference in the object, which means it will be somewhat smaller, and if we are going to have a lot of them, then that can be a pretty nice space saving.