Commit Graph
11 Commits
Author SHA1 Message Date
Chandler CarruthandRichard Smith cf42cc8498 Make TokenKind API inlinable. (#3142)
Previously, accessing any property of a token's kind required both
a function call to an out-of-line method and loading the relevant data
out of a table. These method calls were often roughly as expensive as
the load out of the table -- they would push and pop registers, and so
would touch a stack cache line not unlike the table cache line.

We can inline them in a way that carefully leaves the table definitions
in the single TU so we don't get tons of copies of data that have to be
merged by the linker. But the access to the table can be in an inline
function that allows the call overhead to evaporate and these to turn
into just table loads in the callers as well.

While in theory we could rely on forms of LTO to acheive this, it
doesn't seem worth relying on that. This is an easy win with very little
cost in reality.

Several of the lexer benchmarks improve by 1% or 2% from removing the
function call overhead. It's not a huge win, but it's nice.

Even more nice is that the profile is cleaned up significantly, clearly
focusing on the core functions where time is spent in lexing and
parsing.

This also pulls my big-picture benchmark of 10mloc file down to under
4.3s to lex and parse excitingly. Basically, we're above 2.3mloc/s
lexing and parsing. Not *quite* the 10mloc/s that I'm hoping for, but
still progress.

---------

Co-authored-by: Richard Smith <richard@metafoo.co.uk>
2023-08-25 23:07:23 +00:00
Chandler CarruthandRichard Smith b2db2ca7cf Optimize the outer lexer loop. (#3140)
Previously, the code would try each form of lexing and let that
sub-lexing routine reject the code. This was very branch heavy and also
hard to optimize -- lots of hard to inline function calls, etc.

However, it's really nice to keep the different categories of lexing
broken out into their own functions rather than flattening this into
a huge state machine.

So this creates a miniature explicit state machine by building a table
of function pointers that wrap the methods on the lexer. The main lexing
loop simply indexes this table with the first byte of the source code,
and calls the dispatch function pointer returned.

The result is that the main lex loop is *incredibly* tight code, and the
only time spent appears to be stalls waiting on memory for the next
byte. =]

As part of this, optimize symbol lexing specifically by recognizing all
the symbols that are exactly one character -- IE, we don't even need to
look at the *next* character, there is no max-munch or anything else.
For these, we pre-detect the exact token kind and hand that into the
symbol lexing routine to avoid re-computing it. The symbols in this
category are really frequent symbols in practice like `,` and `;`, so
this seems likely worthwhile in practice.

The one-byte-dispatch should also be reasonably extendable in the
future. For example, I suspect this is the likely hot-path for non-ASCII
lexing, where we see the UTF-8 marker at the start of a token and most
(if not all) of the token is non-ASCII. We can use this table to
dispatch immediately to an optimized routine dedicated to UTF-8
processing, without any slowdown for other inputs.

The benchmark results are best for keyword lexing because that is the
fastest thing to lex -- it goes form 25 mt/s to 30 mt/s. Other
improvements are less dramatic, but I think this is still worthwhile
because it gives a really strong basis for both expanded functionality
without performance hit and further optimizations.

---------

Co-authored-by: Richard Smith <richard@metafoo.co.uk>
2023-08-23 21:56:00 +00:00
Jon Ross-PerkinsandRichard Smith 09242eeebd Run clang-tidy over toolchain code. (#3105)
Amongst other changes, this removes the implicit StringRef cast in
formatter code due to google-explicit-constructor. Only one line of code
relied on the implicit cast, and style says "Do not define implicit
conversions."
(https://google.github.io/styleguide/cppguide.html#Implicit_Conversions).

---------

Co-authored-by: Richard Smith <richard@metafoo.co.uk>
2023-08-16 15:21:58 +00:00
Richard Smith 06ce3b0161 Parsing, semantic analysis, and lowering for and, or, not. (#2897)
Lowering for `and` and `or` is not yet complete because `Branch` lowering isn't done yet.
2023-06-14 13:03:13 -07:00
Jon Ross-PerkinsandChandler Carruth 78ac6cb7d1 Switch TokenKind to EnumBase (#2509)
This shouldn't have any behavior change, it's just using #2504

Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
2023-01-05 14:11:30 -08:00
Jon Ross-Perkins d42d864e82 Make TokenKind's API closer to toolchain's typical API setup. (#2456)
This is somewhat based on the name vs Name difference, but I figured I'd split it out and just sweep up the API on the whole while looking at a different approach to #2453
2022-12-08 15:31:04 -08:00
Jon Meow af694b97cb Prefix most macro names with CARBON_ (#1232)
I'm doing this to avoid macro name conflicts, following https://google.github.io/styleguide/cppguide.html#Preprocessor_Macros: "If you do export a macro from a header, it must have a globally unique name. To achieve this, it must be named with a prefix consisting of your project's namespace name (but upper case)."

Commands run:

```
sed -i 's/\(DCHECK\|CHECK\|FATAL\|MAKE_UNIQUE_NAME\|MAKE_UNIQUE_NAME_IMPL\|RAW_EXITING_STREAM\|RETURN_IF_ERROR\|RETURN_IF_ERROR_IMPL\|ASSIGN_OR_RETURN\|ASSIGN_OR_RETURN_IMPL\|DIAGNOSTIC_KIND\|RETURN_IF_STACK_LIMITED\)(/CARBON_\1(/g' $(git ls-files *.cpp *.h *.lpp *.ypp *.def ':!third_party')
sed -i 's/#undef DIAGNOSTIC_KIND/#undef CARBON_DIAGNOSTIC_KIND/' toolchain/diagnostics/diagnostic_registry.def
```

Note this isn't *quite* everything, but it's intended to be a large pass at everything:

```
╚╡git grep '#define ' *.cpp *.h *.lpp *.ypp *.def ':!third_party' | grep -v '#define CARBON' | grep -v _H_
explorer/syntax/lexer.lpp:  #define YY_USER_ACTION                                             \
explorer/syntax/lexer.lpp:  #define SIMPLE_TOKEN(name) \
explorer/syntax/lexer.lpp:  #define ARG_TOKEN(name, arg) \
explorer/syntax/parse_and_lex_context.h:#define YY_DECL                                                         \
migrate_cpp/cpp_refactoring/var_decl.cpp:#define ABSTRACT_TYPE(Class, Base)
migrate_cpp/cpp_refactoring/var_decl.cpp:#define TYPE(Class, Base)     \
```

We may in particular want to do a pass to clean up #ifdef guards and make them be CARBON_ rooted.
2022-05-06 15:30:25 -07:00
Jon MeowandChandler Carruth a562f872e7 Switch from assert to CHECK (#975)
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
2021-12-08 08:38:27 -08:00
Jon MeowandChandler Carruth 652cd8c636 Style updates, mostly _ naming (#970)
There are some declaration order changes, and a few test classes switched from `struct` to `class`. However, this PR is mostly adopting `_` naming of private member variables due to the shift in naming style. None of what's here should have behavior impacts, it should just be style.

Note, there are a lot of things that *look* like they could be accessor-named, but I'm not doing that in this change. Happy to do it separately if you want me to do another PR focused on it.

Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
2021-12-07 09:46:44 -08:00
Richard SmithandChandler Carruth a83c22288f [toolchain] Implement lexing and parsing support for #543. (#693)
Lex [iuf][1-9][0-9]* as a new kind of "sized type literal" token. When
parsing that token, form a literal expression.

Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
2021-08-02 15:37:43 -07:00
Chandler Carruth 8f8ab23a77 Move the toolchain into a top-level directory. (#567)
This should clean up our top level directory and the build patterns.

No non-mechanical edits here. Just injecting `toolchain/` and
`TOOLCHAIN_` and then running formatting tools.
2021-06-08 03:01:37 -07:00