This adds a benchmark that tries to synthesize mixtures of symbols,
keywords, and identifiers. It tweaks the distribution of identifier
lengths based on some empirical measurements of, for example LLVM's
codebase.
Also establish a framework for skewing the symbol distribution, although
that one is based entirely on intuition and not measurements. It should
be adjusted as we have measurements.
The ratios between symbols, keywords, and identifiers is also
unmeasured, but several different ratios are covered.
Neither literals nor grouping symbols are included yet, as both present
some additional challenges in forming them, and this seemed like
a plausible increment in expanding the benchmark coverage.
Currently this is focused on benchmarking the identifier and token
lexing paths, but should expand in the future to cover other parts of
the lexer.
In order to effectively benchmark tokens, this adds support to the
`token_kind` library to produce a list of all the tokens in Carbon that
the benchmark can use to create random inputs.
For identifiers, the benchmark has support for benchmarking different
distributions of identifier sizes so it is easy to zoom into the
performance specifically of short or long identifiers.
This benchmarking is motivated by profiling overall toolchain
performance and noticing that an unreasonable amount of time is spent in
the lexer. In turn, the identifier lexing was surprisingly hot. I have
performance improvements in the works following this, but wanted to
separately introduce the benchmarking framework as the review focus will
be completely different.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>