mirror of
https://github.com/carbon-language/carbon-lang.git
synced 2026-09-24 22:02:23 +01:00
Chandler CarruthandRichard Smith
3e7b8ade82
It was pointed out in review that the approach to building random inputs
for the benchmark would produce significantly varying input lengths and
cause unavoidable noise of as much as 3% between runs.
While initially, these were only used for very coarse measurements, this
is a definite problem, and so this PR restructures things along the
lines suggested. Rather than generating random lengths or random
selections according to distributions, instead the lengths and coverage
of sets are done deterministically. Then the sequences are randomized in
order to prevent getting stuck in a silent local minima or maxima.
The technique works to ensure that the randomness is done on every run
of the benchmark, not just every run of the program, so that we don't
have noise hidden from run-to-run. That means fresh memory allocation
and some wasted time computed freshly shuffled inputs, but makes sure
the unavoidable noise from things like ASLR show up (as much as
possible) even with simple repetitions of the benchmark runs.
As part of this, this PR moves towards building an entirely custom
distribution of identifier lengths based on the direct measurements of
the LLVM codebase. These measurements are provided by a script that will
be in a subsequent PR, but the exact distribution doesn't matter as much
as our ability to control it and build on it in a fully deterministic
way.
To help with analyzing all of these, we also start tracking the bytes
processed in addition to the tokens processed as rates. The naming is
changed to be similar, and this produces the following nice output from
the benchmark now:
```
---------------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations bytes_per_second tokens_per_second
---------------------------------------------------------------------------------------------------------------
BM_ValidKeywords 3365571 ns 3365577 ns 206 177.449M/s 29.7126M/s
BM_ValidIdentifiers<1, 64, false> 12646151 ns 12645834 ns 52 116.466M/s 7.90774M/s
BM_ValidIdentifiers<1, 1, true> 4388390 ns 4388141 ns 161 65.1988M/s 22.7887M/s
BM_ValidIdentifiers<3, 5, true> 16286551 ns 16286572 ns 43 35.1334M/s 6.14003M/s
BM_ValidIdentifiers<3, 16, true> 15798770 ns 15797567 ns 44 69.4229M/s 6.33009M/s
BM_ValidIdentifiers<12, 64, true> 15670499 ns 15670257 ns 38 243.421M/s 6.38152M/s
BM_ValidMix/10/40 7415940 ns 7415564 ns 94 134.257M/s 13.4852M/s
BM_ValidMix/25/30 7349454 ns 7349402 ns 95 121.762M/s 13.6065M/s
BM_ValidMix/50/20 6691223 ns 6690969 ns 105 100.3M/s 14.9455M/s
BM_ValidMix/75/10 5131517 ns 5131263 ns 135 86.4556M/s 19.4884M/s
```
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Toolchain
A design is currently maintained in Google Drive. It'll be migrated to markdown once we are confident in its stability.