Commit Graph
10 Commits
Author SHA1 Message Date
Chandler CarruthandRichard Smith b2db2ca7cf Optimize the outer lexer loop. (#3140)
Previously, the code would try each form of lexing and let that
sub-lexing routine reject the code. This was very branch heavy and also
hard to optimize -- lots of hard to inline function calls, etc.

However, it's really nice to keep the different categories of lexing
broken out into their own functions rather than flattening this into
a huge state machine.

So this creates a miniature explicit state machine by building a table
of function pointers that wrap the methods on the lexer. The main lexing
loop simply indexes this table with the first byte of the source code,
and calls the dispatch function pointer returned.

The result is that the main lex loop is *incredibly* tight code, and the
only time spent appears to be stalls waiting on memory for the next
byte. =]

As part of this, optimize symbol lexing specifically by recognizing all
the symbols that are exactly one character -- IE, we don't even need to
look at the *next* character, there is no max-munch or anything else.
For these, we pre-detect the exact token kind and hand that into the
symbol lexing routine to avoid re-computing it. The symbols in this
category are really frequent symbols in practice like `,` and `;`, so
this seems likely worthwhile in practice.

The one-byte-dispatch should also be reasonably extendable in the
future. For example, I suspect this is the likely hot-path for non-ASCII
lexing, where we see the UTF-8 marker at the start of a token and most
(if not all) of the token is non-ASCII. We can use this table to
dispatch immediately to an optimized routine dedicated to UTF-8
processing, without any slowdown for other inputs.

The benchmark results are best for keyword lexing because that is the
fastest thing to lex -- it goes form 25 mt/s to 30 mt/s. Other
improvements are less dramatic, but I think this is still worthwhile
because it gives a really strong basis for both expanded functionality
without performance hit and further optimizations.

---------

Co-authored-by: Richard Smith <richard@metafoo.co.uk>
2023-08-23 21:56:00 +00:00
Jon Ross-PerkinsandRichard Smith 09242eeebd Run clang-tidy over toolchain code. (#3105)
Amongst other changes, this removes the implicit StringRef cast in
formatter code due to google-explicit-constructor. Only one line of code
relied on the implicit cast, and style says "Do not define implicit
conversions."
(https://google.github.io/styleguide/cppguide.html#Implicit_Conversions).

---------

Co-authored-by: Richard Smith <richard@metafoo.co.uk>
2023-08-16 15:21:58 +00:00
Richard Smith 06ce3b0161 Parsing, semantic analysis, and lowering for and, or, not. (#2897)
Lowering for `and` and `or` is not yet complete because `Branch` lowering isn't done yet.
2023-06-14 13:03:13 -07:00
Jon Ross-PerkinsandChandler Carruth 78ac6cb7d1 Switch TokenKind to EnumBase (#2509)
This shouldn't have any behavior change, it's just using #2504

Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
2023-01-05 14:11:30 -08:00
Jon Ross-Perkins d42d864e82 Make TokenKind's API closer to toolchain's typical API setup. (#2456)
This is somewhat based on the name vs Name difference, but I figured I'd split it out and just sweep up the API on the whole while looking at a different approach to #2453
2022-12-08 15:31:04 -08:00
Jon Meow af694b97cb Prefix most macro names with CARBON_ (#1232)
I'm doing this to avoid macro name conflicts, following https://google.github.io/styleguide/cppguide.html#Preprocessor_Macros: "If you do export a macro from a header, it must have a globally unique name. To achieve this, it must be named with a prefix consisting of your project's namespace name (but upper case)."

Commands run:

```
sed -i 's/\(DCHECK\|CHECK\|FATAL\|MAKE_UNIQUE_NAME\|MAKE_UNIQUE_NAME_IMPL\|RAW_EXITING_STREAM\|RETURN_IF_ERROR\|RETURN_IF_ERROR_IMPL\|ASSIGN_OR_RETURN\|ASSIGN_OR_RETURN_IMPL\|DIAGNOSTIC_KIND\|RETURN_IF_STACK_LIMITED\)(/CARBON_\1(/g' $(git ls-files *.cpp *.h *.lpp *.ypp *.def ':!third_party')
sed -i 's/#undef DIAGNOSTIC_KIND/#undef CARBON_DIAGNOSTIC_KIND/' toolchain/diagnostics/diagnostic_registry.def
```

Note this isn't *quite* everything, but it's intended to be a large pass at everything:

```
╚╡git grep '#define ' *.cpp *.h *.lpp *.ypp *.def ':!third_party' | grep -v '#define CARBON' | grep -v _H_
explorer/syntax/lexer.lpp:  #define YY_USER_ACTION                                             \
explorer/syntax/lexer.lpp:  #define SIMPLE_TOKEN(name) \
explorer/syntax/lexer.lpp:  #define ARG_TOKEN(name, arg) \
explorer/syntax/parse_and_lex_context.h:#define YY_DECL                                                         \
migrate_cpp/cpp_refactoring/var_decl.cpp:#define ABSTRACT_TYPE(Class, Base)
migrate_cpp/cpp_refactoring/var_decl.cpp:#define TYPE(Class, Base)     \
```

We may in particular want to do a pass to clean up #ifdef guards and make them be CARBON_ rooted.
2022-05-06 15:30:25 -07:00
Jon MeowandChandler Carruth a562f872e7 Switch from assert to CHECK (#975)
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
2021-12-08 08:38:27 -08:00
Jon MeowandChandler Carruth 652cd8c636 Style updates, mostly _ naming (#970)
There are some declaration order changes, and a few test classes switched from `struct` to `class`. However, this PR is mostly adopting `_` naming of private member variables due to the shift in naming style. None of what's here should have behavior impacts, it should just be style.

Note, there are a lot of things that *look* like they could be accessor-named, but I'm not doing that in this change. Happy to do it separately if you want me to do another PR focused on it.

Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
2021-12-07 09:46:44 -08:00
Richard SmithandChandler Carruth a83c22288f [toolchain] Implement lexing and parsing support for #543. (#693)
Lex [iuf][1-9][0-9]* as a new kind of "sized type literal" token. When
parsing that token, form a literal expression.

Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
2021-08-02 15:37:43 -07:00
Chandler Carruth 8f8ab23a77 Move the toolchain into a top-level directory. (#567)
This should clean up our top level directory and the build patterns.

No non-mechanical edits here. Just injecting `toolchain/` and
`TOOLCHAIN_` and then running formatting tools.
2021-06-08 03:01:37 -07:00