Currently this is focused on benchmarking the identifier and token
lexing paths, but should expand in the future to cover other parts of
the lexer.
In order to effectively benchmark tokens, this adds support to the
`token_kind` library to produce a list of all the tokens in Carbon that
the benchmark can use to create random inputs.
For identifiers, the benchmark has support for benchmarking different
distributions of identifier sizes so it is easy to zoom into the
performance specifically of short or long identifiers.
This benchmarking is motivated by profiling overall toolchain
performance and noticing that an unreasonable amount of time is spent in
the lexer. In turn, the identifier lexing was surprisingly hot. I have
performance improvements in the works following this, but wanted to
separately introduce the benchmarking framework as the review focus will
be completely different.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
