Can specify which tokens are allowed generally, and any additional
tokens that only occur when the parse node has an error.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
A `let` declaration is represented by a `bind_name` node in SemIR:
```carbon-semir
%b: i32 = bind_name "b", %a
```
Because `Check` encounters the pattern before it sees the value, we
first create the `bind_name` node with an unset value and don't add it
to the block. Then, once we've seen and converted the initializer, we
update the `bind_name` to have the value and add it to the current
block, after the initializer code.
This removes a (very) hot branch in the lexer where we need to special
case when a token is the first token and can't look at its previous
token. It also seems like a generally nice change to the structure of
both the token buffer and parse tree as there are now bracketing
elements for both ends and we should be able to avoid similar branching
in the future.
Mostly mechanical updates to the lexer and parser code to handle this,
but also needed to special case the location information in the
autoupdate code. And then the usual large body of auto-updated tests.
No benchmark data for this change alone as in isolation and in the
current lexer structure it doesn't make a big difference. But this
branch was particularly difficult to handle when trying to update the
whitespace skipping code to be faster, and so I think it is worth
systematically avoiding the special case here.
This makes the difference between errors and lower-level diagnostics
visible to users, and aligns the toolchain's behavior with the
expectations in `driver_fuzzer.cpp`.
Rationale: this convention avoids forcing closely-related code to be far
apart in the namespace hierarchy, and vice versa. By the same token, it
makes the namespace hierarchy more consistent with the directory
hierarchy.
Previous code could fail silently, only caught with yaml output
mismatches. This uses ErrorOr to be explicit about the error detection;
recovery isn't supported because we should only print valid yaml (in
tests, at least).
Also moves the test helpers from //toolchain/base to
//toolchain/testing.
Switches tree_test to use the matchers so that, on mismatch, gtest
prints a path to the mismatch (the straight value means it was just
printing "not equal").
Addresses
https://github.com/carbon-language/carbon-lang/pull/3217#discussion_r1323586836
Specifically this should improve linking by producing one large binary
instead of one-per-directory. The inclusion of the driver hits the size
issue. Separating out things which have more llvm deps has been
discussed, but I'm not doing that here because I think the semantics
layer will need to depend on clang for interop, and we'd lose a lot of
the benefits that way. Also, having just one place to look seems
simpler.
Includes supporting changes to file_test infrastructure, the most
significant of which is probably passing tests via file instead of a
really large args thing, using a custom rule to do that. That's because
dealing with the layered filegroups that allow the toolchain setup is
more complicated, and this approach scales well.
Combined test time is ~9s, so not sharding right now.
I wasn't sure if people would prefer having the autoupdate script under
testing, so I left it alone for now.
This updates SourceBuffer to diagnostics. Some additional edits to
diagnostics were necessary due to issues moving arguments around, which
seems to stem from a compile error with clang 14 (fixed in later
versions).
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Rearranges driver logic into CompilationUnits in order to associate
artifacts from the various stages of compilation.
Note, I'm not totally sure what the right thing to do is for
lower/codegen, so I'm just doing a rote change there for now that
mirrors prior phases (this is all the code supports anyways, so is
probably right for now regardless).
SourceBuffer error output is moved local for consistency with other
steps, and so that it's less ambiguous whether the error should be
expected to already include a filename.
Continuing with #3070. Just a dir and file rename (only prefix change is
lexer_file_test). Everything in the lex dir should be marked as a move.
Note, I think this closes#3070. There may still be further cleanup
later, but the organizational changes suggested there are being
completed.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Continuing with #3070. Just a dir and file rename (mostly removing
prefixes, although for parse_tree_fuzzer and parse_tree_file_test I'm
dropping "tree" instead of "parse"). Everything in the parse dir should
be marked as a move.
josh11bandRichard Smith
Chandler Carruth
Geoff Romer
Jon Ross-Perkins