Includes support for the `impls`, `=`, and `==` requirement operators to
the right of a `where`, but `and` to allow multiple requirements is
still a TODO.
---------
Co-authored-by: Josh L <josh11b@users.noreply.github.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
These appear in `where` clauses, as in:
```
U:! InterfaceB where .C = Vector(.D)
V:! type where Vector(.Self) impls Sortable
```
`.Self` can additionally appear in type expressions in a binding pattern
such as `T:! InterfaceA(.Self)`.
---------
Co-authored-by: Josh L <josh11b@users.noreply.github.com>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Instead of tracking the token associated with a parse node in the `.def`
file macro, track it on the typed node instead. List the token as a
field inside the node structure to show the order of the token relative
to the other components of the grammar production, and to allow the
token index to be accessed when the node is extracted.
Remove the corresponding information from the `.def` file, leaving
behind just a list of parse node kinds in the majority of cases.
This also removes the checking of the token kind associated with a parse
node in the case where the parse node has errors. Previously we had a
flag on the node kind to indicate whether we should check this, but per
[discord
discussion](https://discord.com/channels/655572317891461132/655578254970716160/1246214418979881052),
we have decided to remove this.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Instead of tracking the bracketing and child count information in the
kind macro in `node_kind.def`, provide it to `NodeKind::Define` in
`typed_nodes.h`. If a node is both bracketed and has a fixed child
count, track both facts and check them both in tree verification, since
it's easy to do so now.
The overall goal here is to reduce `node_kind.def` down to a simple list
of names. I have a slightly different approach in mind for the token
kinds.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Carbon Infra Bot <carbon-external-infra@google.com>
Parse the name of a declaration as a sequence of `NameQualifier`s --
which have a name, possibly parameters, and a trailing period --
followed by a name and possibly parameters. This prepares us for parsing
declarations of members of generic classes and similar cases, but
actually supporting such member redeclarations is left to a future
change.
We previously required functions to have parameters, but no longer do,
following the direction of #3848. Cases like namespaces that can't
actually have parameters are now diagnosed in check instead of in parse.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Change syntax for package declaration to put the `impl` keyword at the
start and remove the `api` keyword.
To support this, rearrange processing of package, library, and import
declarations to use the general modifier handling support in declaration
parsing rather than special-case logic.
There is an ambiguity in `impl package.Foo as Bar`, which we resolve by
treating `package` as an introducer after a modifier only if it's not
followed by `.`.
This provides `export import` logic in lex, parse, and check; `export
name` logic is only in lex and parse, not check.
I think with `export name` I'm going to need to modify import_ref and
some consolidation logic, whereas `export import` seems feasible to keep
as primarily import logic. Given the implementations were looking like
they'd diverge more substantially, I thought it'd be helpful to cut the
PR here.
This is achieving a similar goal as #3849, using placeholders instead of
an ambiguous start node to clarify structure and incrementally simplify
checking. The benefit isn't quite as big here because both paths are
structs, and so checking is more consistent than paren exprs versus
tuples. But I think this removes the only other multi-purpose parse
node.
This uses StructLiteral/StructTypeLiteral naming, reflecting equivalent
SemIR naming. Note, I would lean towards renaming StructLiteral to
StructValueLiteral, but I think consistency in naming takes precedence.
Any renames of StructLiteral might be better in a separate PR.
StructFieldType/StructFieldValue -> StructTypeField/StructField is
trying to making the reading more consistent with
StructTypeLiteral/StructLiteral. SemIR has StructTypeField but not a
value equivalent.
I've been thinking about this since we decided to add placeholders in
the parse tree. This allows a clearer division of work in check
handling, where we were doing work for ExprOpenParen that's only
necessary for tuples (splitting/renaming handle_paren.cpp accordingly).
For now, a builtin function is defined by specifying a string literal
initializer in a function declaration:
```carbon
fn MyBuiltin(a: i32) -> i32 = "builtin.name";
```
End-to-end support is included for a sample `"int.add"` builtin
performing integer addition, covering constant evaluation and code
generation.
The implementation here needs substantial refactoring before we'll be
ready to start adding more builtins. That refactoring work will be
coming next. This change is aiming to checkpoint some incremental
progress.
This doesn't add full support. I'm separating it out to make the effects
of the modifier changes clearer for review. I'm restructuring a little
with the expectation that we'll have some more categories of modifier
keywords in the future (similar to `extern`, these may not be in a "set"
such as access), and thus easily scaling up to a few more would be
useful.
This undoes parts of #3515 in order to allow PushGlobalInit to be called
when the initializer is called, instead of at the end of the binding
pattern. The current approach is fragile because supported patterns will
become more complex. We also will likely want similar support in `let`,
which puts the initializer first, so this offers a consistent approach
for both.
[Looking
back](https://discord.com/channels/655572317891461132/655578254970716160/1184237511766179840),
this is more or less the second option in that message, but using the
PeekNextIs to avoid vagueness about what's being popped first.
Note I'm putting in PeekNextIs for what I'm hoping will be a pretty
narrow use-case. I could've added depth arguments to the Peek functions,
but that would've rippled through a number of APIs and it's not clear to
me that this has generic utility. I mean, right now it could just be
PeekNextIsVariableInitializer, since it's only optional in that case.
Adds `BindAlias` with a hybrid of `BindName` and `NameRef` semantics. I
think it's slightly closer to `BindName` because it introduces a name,
so I'm going more in that direction. This also matches the need for
`bind_name_id` with imports on enclosing scopes.
Note, only things that look like a name reference are being allowed on
the RHS of `alias`. This includes builtins that look like name
references, such as `bool`, but not ones that turn into values
underneath, such as `false`.
Use two different nodes for "<type> followed by `as`" and "<type>
omitted before `as`, use `self`", so it is easier to determine which
case. Later the second case will push the type id for `self` onto the
node stack, making the two paths more similar.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Also stop supporting `var` with initializer inside `for`.
Resolves TODO in `handle_variable.cpp`
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This change adds a `BindSymbolicName` instruction for generic bindings,
paralleling the existing `BindName`. A mechanism is also added to allow
both kinds of binding to be accessed uniformly, for convenience in the
case where the two different kinds of binding are treated the same.
Generic bindings of type `type` are allowed to be used as types,
although no operations are provided for such types. For now lowering
treats these types as empty structs, which seems like a reasonable
lowering for non-monomorphized unconstrained types.
Previously would fail with:
```
CHECK failure at toolchain/parse/node_kind.cpp:74: Lex::TokenKind::Error == expected_token_kind || token_kind == expected_token_kind: Created parse node with NodeKind NamedConstraintDecl and has_error 1 for lexical token kind Constraint, but expected token kind Semi
```
Issue found by fuzzing.
These are intended to allow the structure of a parse tree node to be
described more precisely in code, to support these use cases:
- Automated checking that the parse tree conforms to the expected
structure. (Added to `Tree::Verify`.)
- Easier reading and understanding of the structure of the parse tree by
toolchain developers. (See `parse/typed_nodes.h`.)
- Easier navigation of the parse tree, for example for tooling uses and
for use when forming diagnostics.
On this last point, an object representing the file may be inspecting
using `Tree::ExtractFile`, as in:
```
auto file = tree->ExtractFile();
for (AnyDeclId decl_id : file.decls) {
// `decl_id` is convertible to a `NodeId`.
if (std::optional<FunctionDecl> fn_decl =
tree->ExtractAs<FunctionDecl>(decl_id)) {
// fn_decl->params is a `TuplePatternId` (which extends `NodeId`)
// that is guaranteed to reference a `TuplePattern`.
std::optional<TuplePattern> params = tree->Extract(fn_decl->params);
// `params` has a value unless there was an error in that node.
} else if (auto class_def = tree->ExtractAs<ClassDefinition>(decl_id)) {
// ...
}
}
```
The `Extract...` functions collect the child nodes into the typed parse
node's fields (internally using a `Tree::SiblingIterator`) for easy
access. However, this is not as fast as directly observing the tree
structure using the postorder strategy being used by the check stage.
These functions rely on using struct reflection on the typed parse node
definitions from `parse/typed_nodes.h` to get the expected structure of
child nodes and then populate them.
Note that validating these in `Tree::Verify` adds significant cost to
it, and is currently included in the parsing stage. Without this change,
a 10 mloc test case of lex & parse takes 4.129 s ± 0.041 s. With this
change, it takes 5.768 s ± 0.036 s.
This builds upon and completes #3393.
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
This builds on the series of changes to NodeKinds, aiming to simplify
the NodeKind implementation a little, also making it clearer that
there's a single associated token for each parse node (or, for
placeholders/invalid parses, not validated).
Note that prior to the relevant changes, there were nodes with multiple
tokens. This change is also locking in the approach of one token per
parse node, by refactoring macros to stop supporting multiple.
Note #3486 rewrites the macro behavior, and is already approved: so this
PR is only for the changed enforcement during error. Also, #3493 already
changed several things to allow any token while this PR was awaiting
review, but this still changes enforcement for `For` and `If`.
This was brought up on
[#toolchain](https://discord.com/channels/655572317891461132/655578254970716160/1182066616456970251),
and I think this any-on-error approach gets at least some support. We
could try setting it to the introducer, but it's quite possible we want
it to be something like the token which led to the parse error, rather
than a static token. That leads to a conclusion that, most typically,
we'll expect arbitrary tokens when error conditions may lead to tokens
which aren't the expected token.
A couple related, recent `CARBON_IF_ERROR` crash fixes can be found in
#3404 and #3424. Something like #3404 would've been needed regardless
because `namespace` didn't have `CARBON_IF_ERROR` before, although I
might've missed the underlying issue with declarations because only
`namespace` had a relevant test (that is, if #3404 had added
`CARBON_ANY_TOKEN_ON_ERROR`, I wouldn't have had a crash in #3462).
#3424 would've been avoided with this change because there was a
`CARBON_IF_ERROR`, and it was just too restrictive.
This leaves a single state for each in the expr loop. I was trying to
think through ways to have per-token states, but they felt sort of
bulky.
Note this is more verbose: but I think the long-term is going to be that
when we start wanting to add handlers, we're going to need to switch to
different names based on the token found. As a consequence, the parse
state logic will end up diverging a little, and we'll just want to align
towards boilerplate handlers.
Short-term, this opens up a path for saying that each parse node
corresponds to precisely one token in success states, and separates out
what were becoming big handler functions in check.
StructFieldUnknown was used previously for invalid parses. But we have
added other, more common ways of talking about those; so this is
removing the special-case.
This remains structurally valid although we don't use this operator or
have a design for it, this at least fixes a fuzzer-found crash.
---------
Co-authored-by: josh11b <josh11b@users.noreply.github.com>
This is also doing the parse node split, allowing lower reliance in
formatter on the tokenized buffer (something that I may be touching more
due to import handling).
(Split out of #3410)
Consistently use `ParenExpr` solely for parenthesized single
expressions, and use more syntax-oriented terminology for states and
nodes that might represent either a `ParenExpr` or a tuple literal.
In theory because none are allowed. This is to improve consistency in
handle_decl_name_scope's modifier handling, removing the namespace
special-case.
I noticed there's a crash bug on `impl <declaration>` which I'll address
separately.
This builds on #3461.
This is supporting a direction that all parse nodes should correspond to
a single token, allowing for reduced tokenized buffer access during
checking (it's still necessary for diagnostics, and some literals).
One of the justifications for a unified parse node was implementation
LOC: note this is slightly smaller, using macros to reduce some
duplication. While this does add more switching in HandleDeclScopeLoop,
that's offset by less explicit switching in the check handlers. Also, I
think the duplication in HandleDeclScopeLoop can be reduced by shifting
the flow there, which I'll do in a separate PR.
Brymer Meneses
CJ Johnson
czapiga
kshokhin
Geoff Romer
