mirror of https://github.com/carbon-language/carbon-lang.git synced 2026-10-04 08:51:05 +01:00

Files

T

Chandler CarruthandRichard Smith b2db2ca7cf Optimize the outer lexer loop. (#3140 )

Previously, the code would try each form of lexing and let that
sub-lexing routine reject the code. This was very branch heavy and also
hard to optimize -- lots of hard to inline function calls, etc.

However, it's really nice to keep the different categories of lexing
broken out into their own functions rather than flattening this into
a huge state machine.

So this creates a miniature explicit state machine by building a table
of function pointers that wrap the methods on the lexer. The main lexing
loop simply indexes this table with the first byte of the source code,
and calls the dispatch function pointer returned.

The result is that the main lex loop is *incredibly* tight code, and the
only time spent appears to be stalls waiting on memory for the next
byte. =]

As part of this, optimize symbol lexing specifically by recognizing all
the symbols that are exactly one character -- IE, we don't even need to
look at the *next* character, there is no max-munch or anything else.
For these, we pre-detect the exact token kind and hand that into the
symbol lexing routine to avoid re-computing it. The symbols in this
category are really frequent symbols in practice like `,` and `;`, so
this seems likely worthwhile in practice.

The one-byte-dispatch should also be reasonably extendable in the
future. For example, I suspect this is the likely hot-path for non-ASCII
lexing, where we see the UTF-8 marker at the start of a token and most
(if not all) of the token is non-ASCII. We can use this table to
dispatch immediately to an optimized routine dedicated to UTF-8
processing, without any slowdown for other inputs.

The benchmark results are best for keyword lexing because that is the
fastest thing to lex -- it goes form 25 mt/s to 30 mt/s. Other
improvements are less dramatic, but I think this is still worthwhile
because it gives a really strong basis for both expanded functionality
without performance hit and further optimizations.

---------

Co-authored-by: Richard Smith <richard@metafoo.co.uk>

2023-08-23 21:56:00 +00:00

base

Add lint fixes to the buildifier setup. (#3109 )

2023-08-22 21:01:42 +00:00

codegen

Add lint fixes to the buildifier setup. (#3109 )

2023-08-22 21:01:42 +00:00

diagnostics

Add lint fixes to the buildifier setup. (#3109 )

2023-08-22 21:01:42 +00:00

driver

Add lint fixes to the buildifier setup. (#3109 )