Files
carbon-lang/toolchain/lex
Chandler Carruth 95a1cc8cba Tweak lexer to improve generated code. (#3296)
This is a collection of tweaks and they are ones I'm less confident in
FWIW. I set out to make the generated code cleaner and reduce loading
pointers through pointers in a bunch of cases. It also works to reduce
the working-set-size, and reduce the set of mutated values on each
iteration. But I didn't benchmark at each step and it's not clear that
incremental benchmarks will even be meaningful, so its hard to say which
tweaks were load bearing and which weren't.

For example, with this, the core lexer dispatch loop doesn't mutate
anything in memory -- the position is passed in register and incremented
in register throughout. The pointer to the source text, the size, and
the pointer to the lexer are also passed in registers.

By making the tokenized buffer a direct member of the lexer, all of its
members can be accessed at a constant offset from the lexer pointer
itself. The cost of this is a move at the end of lexing, but especially
as the buffer size gets large this seems like a trivial cost compared to
the previous double-indirection.

This uses a (significantly) cheaper representation for the line index --
the `Line` type is optimized for dense *storage*, and is not a great
type for using in a tight loop like the lexer. Once using a good index
and once the line table is readily available from the above, we can
simply index the line table rather than store (and update) a separate
pointer.

Last but not least, this removes the column as suggested in a previous
review. Now it is computed from the position and the line information.

My macro benchmark looks like it improves in the 2% - 5% range, but its
getting into the noise sadly (or maybe this is good?).

On a quiet AMD server with 20 runs before/after I get reasonably
compelling across-the-board improvements on all the microbenchmarks,
including 5% on `RandomSource` which is key:

```
BM_ValidKeywords                               3.44ms ± 1%  3.11ms ± 1%   -9.52%  (p=0.000 n=19+20)
BM_ValidIdentifiers<1, 64, false>              11.2ms ± 1%  10.8ms ± 2%   -3.23%  (p=0.000 n=19+20)
BM_ValidIdentifiers<1, 1, true>                3.96ms ± 1%  3.72ms ± 1%   -6.10%  (p=0.000 n=18+20)
BM_ValidIdentifiers<3, 5, true>                13.5ms ± 1%  13.2ms ± 3%   -2.45%  (p=0.000 n=19+20)
BM_ValidIdentifiers<3, 16, true>               13.5ms ± 2%  13.1ms ± 2%   -2.81%  (p=0.000 n=19+20)
BM_ValidIdentifiers<12, 64, true>              15.6ms ± 1%  15.2ms ± 3%   -2.41%  (p=0.000 n=19+20)
BM_HorizontalWhitespace/1                      13.6ms ± 2%  13.3ms ± 3%   -2.34%  (p=0.000 n=19+20)
BM_HorizontalWhitespace/4                      13.9ms ± 2%  13.6ms ± 3%   -2.06%  (p=0.000 n=19+20)
BM_HorizontalWhitespace/16                     14.5ms ± 2%  14.3ms ± 2%   -1.17%  (p=0.002 n=19+20)
BM_HorizontalWhitespace/64                     18.5ms ± 3%  18.1ms ± 2%   -2.17%  (p=0.000 n=19+19)
BM_HorizontalWhitespace/128                    24.7ms ± 1%  24.4ms ± 2%   -1.29%  (p=0.000 n=18+20)
BM_RandomSource                                8.38ms ± 2%  7.92ms ± 2%   -5.50%  (p=0.000 n=19+20)
BM_GroupingSymbols/1/0/0                       6.63ms ± 2%  6.30ms ± 2%   -4.97%  (p=0.000 n=19+20)
BM_GroupingSymbols/2/0/0                       5.65ms ± 2%  5.27ms ± 2%   -6.75%  (p=0.000 n=19+20)
BM_GroupingSymbols/3/0/0                       4.51ms ± 1%  4.12ms ± 1%   -8.62%  (p=0.000 n=19+18)
BM_GroupingSymbols/4/0/0                       4.23ms ± 1%  3.82ms ± 1%   -9.65%  (p=0.000 n=19+18)
BM_GroupingSymbols/8/0/0                       3.50ms ± 1%  3.08ms ± 1%  -12.21%  (p=0.000 n=19+19)
BM_GroupingSymbols/16/0/0                      3.38ms ± 1%  2.97ms ± 1%  -11.98%  (p=0.000 n=19+19)
BM_GroupingSymbols/32/0/0                      4.10ms ± 1%  3.69ms ± 1%  -10.02%  (p=0.000 n=17+19)
BM_GroupingSymbols/0/1/0                       5.85ms ± 2%  5.70ms ± 1%   -2.54%  (p=0.000 n=19+20)
BM_GroupingSymbols/0/2/0                       4.60ms ± 2%  4.46ms ± 1%   -3.12%  (p=0.000 n=19+19)
BM_GroupingSymbols/0/3/0                       3.32ms ± 2%  3.18ms ± 1%   -4.21%  (p=0.000 n=18+20)
BM_GroupingSymbols/0/4/0                       2.94ms ± 2%  2.82ms ± 1%   -4.22%  (p=0.000 n=19+20)
BM_GroupingSymbols/0/8/0                       1.95ms ± 1%  1.83ms ± 1%   -6.47%  (p=0.000 n=17+19)
BM_GroupingSymbols/0/16/0                      1.40ms ± 1%  1.27ms ± 1%   -9.12%  (p=0.000 n=19+20)
BM_GroupingSymbols/0/32/0                      1.19ms ± 1%  1.05ms ± 1%  -11.50%  (p=0.000 n=19+20)
BM_GroupingSymbols/0/0/1                       5.88ms ± 2%  5.71ms ± 1%   -2.87%  (p=0.000 n=19+20)
BM_GroupingSymbols/0/0/2                       4.62ms ± 1%  4.45ms ± 2%   -3.58%  (p=0.000 n=18+20)
BM_GroupingSymbols/0/0/3                       3.32ms ± 1%  3.17ms ± 1%   -4.72%  (p=0.000 n=16+17)
BM_GroupingSymbols/0/0/4                       2.95ms ± 1%  2.81ms ± 2%   -4.50%  (p=0.000 n=16+19)
BM_GroupingSymbols/0/0/8                       1.96ms ± 1%  1.82ms ± 2%   -6.80%  (p=0.000 n=19+20)
BM_GroupingSymbols/0/0/16                      1.40ms ± 1%  1.27ms ± 1%   -8.96%  (p=0.000 n=19+20)
BM_GroupingSymbols/0/0/32                      1.19ms ± 1%  1.05ms ± 1%  -11.64%  (p=0.000 n=19+20)
BM_GroupingSymbols/32/1/0                      4.04ms ± 1%  3.60ms ± 1%  -10.81%  (p=0.000 n=17+20)
BM_GroupingSymbols/32/2/0                      3.93ms ± 1%  3.51ms ± 1%  -10.55%  (p=0.000 n=18+20)
BM_GroupingSymbols/32/3/0                      3.84ms ± 1%  3.44ms ± 1%  -10.38%  (p=0.000 n=18+19)
BM_GroupingSymbols/32/4/0                      3.76ms ± 2%  3.38ms ± 1%  -10.21%  (p=0.000 n=19+20)
BM_GroupingSymbols/32/8/0                      3.51ms ± 1%  3.14ms ± 1%  -10.61%  (p=0.000 n=19+19)
BM_GroupingSymbols/32/16/0                     3.11ms ± 1%  2.77ms ± 1%  -10.95%  (p=0.000 n=19+20)
BM_GroupingSymbols/32/32/0                     2.54ms ± 2%  2.25ms ± 1%  -11.77%  (p=0.000 n=19+16)
BM_GroupingSymbols/32/32/1                     2.50ms ± 1%  2.22ms ± 2%  -11.19%  (p=0.000 n=19+20)
BM_GroupingSymbols/32/32/2                     2.48ms ± 1%  2.21ms ± 1%  -11.15%  (p=0.000 n=19+20)
BM_GroupingSymbols/32/32/3                     2.46ms ± 1%  2.18ms ± 1%  -11.11%  (p=0.000 n=18+20)
BM_GroupingSymbols/32/32/4                     2.43ms ± 1%  2.16ms ± 1%  -11.04%  (p=0.000 n=19+20)
BM_GroupingSymbols/32/32/8                     2.35ms ± 1%  2.09ms ± 1%  -11.11%  (p=0.000 n=19+20)
BM_GroupingSymbols/32/32/16                    2.20ms ± 1%  1.96ms ± 1%  -11.30%  (p=0.000 n=19+18)
BM_GroupingSymbols/32/32/32                    1.98ms ± 1%  1.75ms ± 1%  -11.54%  (p=0.000 n=19+19)
BM_BlankLines/1                                14.6ms ± 1%  14.2ms ± 3%   -2.53%  (p=0.000 n=19+19)
BM_BlankLines/4                                18.3ms ± 1%  17.4ms ± 2%   -4.68%  (p=0.000 n=19+20)
BM_BlankLines/16                               34.9ms ± 2%  31.6ms ± 3%   -9.42%  (p=0.000 n=19+20)
BM_BlankLines/64                                102ms ± 2%    91ms ± 3%  -10.75%  (p=0.000 n=19+20)
BM_BlankLines/128                               190ms ± 2%   169ms ± 3%  -10.82%  (p=0.000 n=19+20)
BM_CommentLines/1/0/0                          16.2ms ± 1%  15.8ms ± 2%   -2.29%  (p=0.000 n=19+19)
BM_CommentLines/4/0/0                          21.0ms ± 1%  19.9ms ± 2%   -5.17%  (p=0.000 n=19+19)
BM_CommentLines/128/0/0                         237ms ± 1%   211ms ± 2%  -10.93%  (p=0.000 n=19+20)
BM_CommentLines/1/30/0                         17.0ms ± 1%  16.8ms ± 3%   -1.36%  (p=0.001 n=17+20)
BM_CommentLines/4/30/0                         27.2ms ± 2%  26.0ms ± 2%   -4.24%  (p=0.000 n=19+20)
BM_CommentLines/128/30/0                        255ms ± 2%   223ms ± 2%  -12.58%  (p=0.000 n=19+20)
BM_CommentLines/1/70/0                         19.7ms ± 1%  19.2ms ± 1%   -2.35%  (p=0.000 n=18+18)
BM_CommentLines/4/70/0                         29.0ms ± 2%  27.6ms ± 1%   -4.81%  (p=0.000 n=18+17)
BM_CommentLines/128/70/0                        273ms ± 7%   249ms ± 3%   -9.05%  (p=0.000 n=20+20)
BM_CommentLines/1/0/2                          16.4ms ± 2%  16.0ms ± 1%   -2.49%  (p=0.000 n=19+15)
BM_CommentLines/4/0/2                          22.0ms ± 2%  20.7ms ± 1%   -5.84%  (p=0.000 n=19+17)
BM_CommentLines/128/0/2                         256ms ± 2%   218ms ± 1%  -14.88%  (p=0.000 n=19+17)
BM_CommentLines/1/30/2                         17.4ms ± 1%  17.0ms ± 2%   -2.26%  (p=0.000 n=19+19)
BM_CommentLines/4/30/2                         28.5ms ± 1%  26.9ms ± 2%   -5.67%  (p=0.000 n=19+18)
BM_CommentLines/128/30/2                        288ms ± 5%   246ms ± 3%  -14.70%  (p=0.000 n=19+20)
BM_CommentLines/1/70/2                         19.9ms ± 1%  19.4ms ± 1%   -2.56%  (p=0.000 n=19+18)
BM_CommentLines/4/70/2                         30.0ms ± 2%  28.5ms ± 2%   -5.20%  (p=0.000 n=19+19)
BM_CommentLines/128/70/2                        300ms ± 3%   265ms ± 3%  -11.69%  (p=0.000 n=18+20)
BM_CommentLines/1/0/8                          16.7ms ± 2%  16.3ms ± 2%   -2.22%  (p=0.000 n=19+19)
BM_CommentLines/4/0/8                          26.4ms ± 1%  25.2ms ± 1%   -4.42%  (p=0.000 n=19+18)
BM_CommentLines/128/0/8                         285ms ± 2%   248ms ± 1%  -13.16%  (p=0.000 n=19+18)
BM_CommentLines/1/30/8                         17.9ms ± 2%  17.4ms ± 1%   -2.99%  (p=0.000 n=19+17)
BM_CommentLines/4/30/8                         32.0ms ± 1%  30.5ms ± 2%   -4.73%  (p=0.000 n=19+20)
BM_CommentLines/128/30/8                        320ms ± 2%   280ms ± 3%  -12.52%  (p=0.000 n=18+20)
BM_CommentLines/1/70/8                         20.7ms ± 2%  20.0ms ± 2%   -3.22%  (p=0.000 n=19+19)
BM_CommentLines/4/70/8                         33.5ms ± 2%  31.8ms ± 1%   -5.23%  (p=0.000 n=19+19)
BM_CommentLines/128/70/8                        338ms ± 3%   298ms ± 2%  -11.64%  (p=0.000 n=19+20)
```
2023-10-13 21:30:24 +00:00
..
2023-09-01 02:39:04 +00:00
2023-09-15 21:28:15 +00:00
2023-09-01 02:39:04 +00:00
2023-09-01 02:39:04 +00:00
2023-09-01 02:39:04 +00:00
2023-09-01 02:39:04 +00:00