Change syntax for package declaration to put the `impl` keyword at the
start and remove the `api` keyword.
To support this, rearrange processing of package, library, and import
declarations to use the general modifier handling support in declaration
parsing rather than special-case logic.
There is an ambiguity in `impl package.Foo as Bar`, which we resolve by
treating `package` as an introducer after a modifier only if it's not
followed by `.`.
This provides `export import` logic in lex, parse, and check; `export
name` logic is only in lex and parse, not check.
I think with `export name` I'm going to need to modify import_ref and
some consolidation logic, whereas `export import` seems feasible to keep
as primarily import logic. Given the implementations were looking like
they'd diverge more substantially, I thought it'd be helpful to cut the
PR here.
When we expect a semicolon, we should be able to provide more standard
recovery.
Note this affects a test of `base`, but it looks like a partial
improvement (the prior line was moving too far). Still, it looks like
recovery isn't handling `{}` quite right. I'm not trying to address that
here though -- leaving a TODO.
The expected parse tree size is validated for valid parse trees, which
is possible even if lex encounters an error. In that case, we were
missing recovery tokens, resulting in a crash. AddToken should be the
only place we call `token_infos_.push_back`.
This is achieving a similar goal as #3849, using placeholders instead of
an ambiguous start node to clarify structure and incrementally simplify
checking. The benefit isn't quite as big here because both paths are
structs, and so checking is more consistent than paren exprs versus
tuples. But I think this removes the only other multi-purpose parse
node.
This uses StructLiteral/StructTypeLiteral naming, reflecting equivalent
SemIR naming. Note, I would lean towards renaming StructLiteral to
StructValueLiteral, but I think consistency in naming takes precedence.
Any renames of StructLiteral might be better in a separate PR.
StructFieldType/StructFieldValue -> StructTypeField/StructField is
trying to making the reading more consistent with
StructTypeLiteral/StructLiteral. SemIR has StructTypeField but not a
value equivalent.
I've been thinking about this since we decided to add placeholders in
the parse tree. This allows a clearer division of work in check
handling, where we were doing work for ExprOpenParen that's only
necessary for tuples (splitting/renaming handle_paren.cpp accordingly).
The purpose of the newline is to make it clearer where a given
diagnostic begins and ends, particularly as the first message of a
diagnostic may not be the error.
This is a trivial code change, but ripples edits through test files.
For now, a builtin function is defined by specifying a string literal
initializer in a function declaration:
```carbon
fn MyBuiltin(a: i32) -> i32 = "builtin.name";
```
End-to-end support is included for a sample `"int.add"` builtin
performing integer addition, covering constant evaluation and code
generation.
The implementation here needs substantial refactoring before we'll be
ready to start adding more builtins. That refactoring work will be
coming next. This change is aiming to checkpoint some incremental
progress.
On the parsing side, we treat `a.(b)` as a member access whose second
operand is a `ParenExpr` rather than a `MemberName`. A new node category
is added for the union of `MemberName` and `ParenExpr` to support this.
Checking is mostly reusing the same pieces we already have for simple
member access. Compound member access is in most ways a simplified form
of simple member access because it doesn't need to do any lookup.
This handles toolchain failures per-file. The intent is to allow placing
both "success" and "fail" tests in the same file, using splits. However,
this PR only adds support and updates existing tests to continue
passing.
This doesn't add full support. I'm separating it out to make the effects
of the modifier changes clearer for review. I'm restructuring a little
with the expectation that we'll have some more categories of modifier
keywords in the future (similar to `extern`, these may not be in a "set"
such as access), and thus easily scaling up to a few more would be
useful.
This undoes parts of #3515 in order to allow PushGlobalInit to be called
when the initializer is called, instead of at the end of the binding
pattern. The current approach is fragile because supported patterns will
become more complex. We also will likely want similar support in `let`,
which puts the initializer first, so this offers a consistent approach
for both.
[Looking
back](https://discord.com/channels/655572317891461132/655578254970716160/1184237511766179840),
this is more or less the second option in that message, but using the
PeekNextIs to avoid vagueness about what's being popped first.
Note I'm putting in PeekNextIs for what I'm hoping will be a pretty
narrow use-case. I could've added depth arguments to the Peek functions,
but that would've rippled through a number of APIs and it's not clear to
me that this has generic utility. I mean, right now it could just be
PeekNextIsVariableInitializer, since it's only optional in that case.
Adds `BindAlias` with a hybrid of `BindName` and `NameRef` semantics. I
think it's slightly closer to `BindName` because it introduces a name,
so I'm going more in that direction. This also matches the need for
`bind_name_id` with imports on enclosing scopes.
Note, only things that look like a name reference are being allowed on
the RHS of `alias`. This includes builtins that look like name
references, such as `bool`, but not ones that turn into values
underneath, such as `false`.
Use two different nodes for "<type> followed by `as`" and "<type>
omitted before `as`, use `self`", so it is easier to determine which
case. Later the second case will push the type id for `self` onto the
node stack, making the two paths more similar.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Also stop supporting `var` with initializer inside `for`.
Resolves TODO in `handle_variable.cpp`
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Previously would fail with:
```
CHECK failure at toolchain/parse/node_kind.cpp:74: Lex::TokenKind::Error == expected_token_kind || token_kind == expected_token_kind: Created parse node with NodeKind NamedConstraintDecl and has_error 1 for lexical token kind Constraint, but expected token kind Semi
```
Issue found by fuzzing.
These are intended to allow the structure of a parse tree node to be
described more precisely in code, to support these use cases:
- Automated checking that the parse tree conforms to the expected
structure. (Added to `Tree::Verify`.)
- Easier reading and understanding of the structure of the parse tree by
toolchain developers. (See `parse/typed_nodes.h`.)
- Easier navigation of the parse tree, for example for tooling uses and
for use when forming diagnostics.
On this last point, an object representing the file may be inspecting
using `Tree::ExtractFile`, as in:
```
auto file = tree->ExtractFile();
for (AnyDeclId decl_id : file.decls) {
// `decl_id` is convertible to a `NodeId`.
if (std::optional<FunctionDecl> fn_decl =
tree->ExtractAs<FunctionDecl>(decl_id)) {
// fn_decl->params is a `TuplePatternId` (which extends `NodeId`)
// that is guaranteed to reference a `TuplePattern`.
std::optional<TuplePattern> params = tree->Extract(fn_decl->params);
// `params` has a value unless there was an error in that node.
} else if (auto class_def = tree->ExtractAs<ClassDefinition>(decl_id)) {
// ...
}
}
```
The `Extract...` functions collect the child nodes into the typed parse
node's fields (internally using a `Tree::SiblingIterator`) for easy
access. However, this is not as fast as directly observing the tree
structure using the postorder strategy being used by the check stage.
These functions rely on using struct reflection on the typed parse node
definitions from `parse/typed_nodes.h` to get the expected structure of
child nodes and then populate them.
Note that validating these in `Tree::Verify` adds significant cost to
it, and is currently included in the parsing stage. Without this change,
a 10 mloc test case of lex & parse takes 4.129 s ± 0.041 s. With this
change, it takes 5.768 s ± 0.036 s.
This builds upon and completes #3393.
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Move handling of mismatched brackets out of the main lexing loop into a
separate pass that is only run if there are mismatched brackets This is
done in preparation for using both lookahead and lookbehind to work out
how to match brackets, and to get this code far away from the hot lexing
loop.
Fix bracket insertion location to be immediately after the token that
we're inserting the bracket after, rather than potentially at the end of
a comment. When there are open brackets at the end of the file, say that
there are open brackets, not that there's a closing bracket without a
matching opening bracket.
Note #3486 rewrites the macro behavior, and is already approved: so this
PR is only for the changed enforcement during error. Also, #3493 already
changed several things to allow any token while this PR was awaiting
review, but this still changes enforcement for `For` and `If`.
This was brought up on
[#toolchain](https://discord.com/channels/655572317891461132/655578254970716160/1182066616456970251),
and I think this any-on-error approach gets at least some support. We
could try setting it to the introducer, but it's quite possible we want
it to be something like the token which led to the parse error, rather
than a static token. That leads to a conclusion that, most typically,
we'll expect arbitrary tokens when error conditions may lead to tokens
which aren't the expected token.
A couple related, recent `CARBON_IF_ERROR` crash fixes can be found in
#3404 and #3424. Something like #3404 would've been needed regardless
because `namespace` didn't have `CARBON_IF_ERROR` before, although I
might've missed the underlying issue with declarations because only
`namespace` had a relevant test (that is, if #3404 had added
`CARBON_ANY_TOKEN_ON_ERROR`, I wouldn't have had a crash in #3462).
#3424 would've been avoided with this change because there was a
`CARBON_IF_ERROR`, and it was just too restrictive.
This leaves a single state for each in the expr loop. I was trying to
think through ways to have per-token states, but they felt sort of
bulky.
Note this is more verbose: but I think the long-term is going to be that
when we start wanting to add handlers, we're going to need to switch to
different names based on the token found. As a consequence, the parse
state logic will end up diverging a little, and we'll just want to align
towards boilerplate handlers.
Short-term, this opens up a path for saying that each parse node
corresponds to precisely one token in success states, and separates out
what were becoming big handler functions in check.
StructFieldUnknown was used previously for invalid parses. But we have
added other, more common ways of talking about those; so this is
removing the special-case.
This remains structurally valid although we don't use this operator or
have a design for it, this at least fixes a fuzzer-found crash.
---------
Co-authored-by: josh11b <josh11b@users.noreply.github.com>
This is also doing the parse node split, allowing lower reliance in
formatter on the tokenized buffer (something that I may be touching more
due to import handling).
(Split out of #3410)
Consistently use `ParenExpr` solely for parenthesized single
expressions, and use more syntax-oriented terminology for states and
nodes that might represent either a `ParenExpr` or a tuple literal.
In theory because none are allowed. This is to improve consistency in
handle_decl_name_scope's modifier handling, removing the namespace
special-case.
I noticed there's a crash bug on `impl <declaration>` which I'll address
separately.
This builds on #3461.
This is supporting a direction that all parse nodes should correspond to
a single token, allowing for reduced tokenized buffer access during
checking (it's still necessary for diagnostics, and some literals).
One of the justifications for a unified parse node was implementation
LOC: note this is slightly smaller, using macros to reduce some
duplication. While this does add more switching in HandleDeclScopeLoop,
that's offset by less explicit switching in the check handlers. Also, I
think the duplication in HandleDeclScopeLoop can be reduced by shifting
the flow there, which I'll do in a separate PR.
Richard Smith
Jon Ross-Perkins
josh11bandJosh L
CJ Johnson
czapiga
kshokhin
Geoff Romer
