Commit Graph
56 Commits
Author SHA1 Message Date
Chandler Carruth 6f5934a505 Unblock more lexer inlining. (#3274)
The big change is to make the lexer helpers have internal linkage,
making all of them easy to inline into single call sites.

Looking at the profile showed several other cases of unfortunate
out-of-line functions. Two were due to the code size produced for checks
-- those are switched to `DCHECK`s to remove that code from optimized
builds. The loss of coverage seems minor.

A last one was closing open groups. This was a surprising routine to be
hot, but it the paths to discover "nothing to do here" were intertwined
into the code. This PR extracts this common trace into a separate
function that delegates to the looping recovery path. This lets the hot
path inline easily.

At this point, for a large lexing benchmark I'm using, 50% of the time
is in the identifier hash table at this point. The remaining
improvements are to actually make some of the hot routines like symbol
lexing and comment lexing faster.
2023-10-10 21:30:07 +00:00
Chandler Carruth 03c3b86758 Switch lexer to fully table-driven design. (#3273)
This uses the musttail dispatched table approach to drive the entire
lexing. The result is that there is no main lexer loop at all in a
traditional sense, now everything is driven through tail recursive
dispatch on the next byte of the source text.

This should be easy to extend still -- the design pattern is to add
lexer methods for handling specific cases, and then add a dispatch
function to dispatch to them from the table. For example, we can add a
method that handles decoding UTF-8 outside of the ASCII subset and set
the table entries used by non-ASCII initial bytes to dispatch to it.

The performance is already surprisingly good, benchmarks show a modest
improvement across the board. That's despite there still being some
*serious* performance issues that I'll fix in a separate patch. There
are also opportunities to leverage this structure more heavily as needed
by putting more specialized dispatch targets in for specific bytes.

A follow-up PR will re-organize the functions here, as almost all of the
methods on the `Lexer` should become private, but I wanted to keep that
a separate change since it will probably render the diff even more hard
to read than it already is.
2023-10-07 03:35:24 +00:00
Chandler CarruthandRichard Smith d552545c6d Move dispatch routines to be static member functions. (#3265)
These routines have a regular signature and are used to build a table of
function pointers for fast dispatch. However, the previous approach
relied on lambdas to build these functions which resulted in very hard
to read functions in the profile and backtrace.

In preparation for expanding dispatch to handle (many) more cases in the
lexer and also enabling more aggressive inlining into the dispatch
routines, I wanted to tidy up how they appear.

This PR alone shouldn't have any interesting functional effect, it's
just re-organizing the code.

Co-authored-by: Richard Smith <richard@metafoo.co.uk>
2023-10-05 00:03:56 +00:00
Chandler Carruth a46ca6bf7a Add a start-of-file token and parse node. (#3263)
This removes a (very) hot branch in the lexer where we need to special
case when a token is the first token and can't look at its previous
token. It also seems like a generally nice change to the structure of
both the token buffer and parse tree as there are now bracketing
elements for both ends and we should be able to avoid similar branching
in the future.

Mostly mechanical updates to the lexer and parser code to handle this,
but also needed to special case the location information in the
autoupdate code. And then the usual large body of auto-updated tests.

No benchmark data for this change alone as in isolation and in the
current lexer structure it doesn't make a big difference. But this
branch was particularly difficult to handle when trying to update the
whitespace skipping code to be faster, and so I think it is worth
systematically avoiding the special case here.
2023-10-04 23:36:35 +00:00
Jon Ross-Perkins 2ecab78297 Support multi-file lex printing and testing. (#3214)
Lex now prints its yaml as:
```
- filename: name
  tokens: [ ... ]
```

New support in file_test allows the `filename` marker at the top to
define the default file number for later lines, meaning multi-file
output from lexing is now associated with the appropriate file. Similar
support will probably also apply to lowering, semir, and other places
that print a filename once for the full dump.

This hammers a bit at how line number replacements work in file_test,
allowing stacking them so that lex errors and stdout can both be
line-associated properly. I've tried to make the autoupdate more
frequently work in one pass, now also taking into account the file index
when doing line replacements.

There are still some issues with EndOfFile that it may be good to
discuss: because CHECK lines are appended to the end of the file now,
and the EndOfFile token points at the last line including comments, new
lex tests now take two runs to autoupdate (because without CHECK lines,
the EndOfFile points at a content line, which content is then inserted
after). Note that removing CHECK lines from the test is not a solution:
autoupdate also started inserting blank lines, which breaks this for a
similar reason. One solution here might be to not have EndOfFile
associate with a line or column, which has been a bit of an issue
regardless.

Also fixes a small issue with toolchain's autoupdate script.
2023-09-13 16:14:09 +00:00
Jon Ross-PerkinsandChandler Carruth ec182fb00d Rename lexer dir to lex (#3179)
Continuing with #3070. Just a dir and file rename (only prefix change is
lexer_file_test). Everything in the lex dir should be marked as a move.

Note, I think this closes #3070. There may still be further cleanup
later, but the organizational changes suggested there are being
completed.

---------

Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
2023-09-01 02:39:04 +00:00