Adds an algorithm to compute where to insert brackets to repair
bracketing mismatches during lexing. This takes indentation, as well as
a number of other cues, into account to predict where the brackets
should have gone. Detects when there is ambiguity between solutions and
makes no suggestion in that case. Reduces the problem by splitting on
properly bracketed top-level constructs, then uses a beam search to find
good candidate solutions quickly.
This includes both a fuzzer and an eval tool that can be used to
determine how well the algorithm fares against a given corpus of valid
Carbon code, by damaging it in various ways and seeing whether the
algorithm can correctly fix it. On all the eval modes, this algorithm
can correctly infer the positions for over 80% of lost brackets (and can
correctly restore 95+% of brackets in some modes), with low rates of
incorrect suggestions.
See added documentation for full details.
Assisted-by: Gemini via Antigravity, Claude via Claude Code
It's hard to track 2-space indent levels across large vertical gaps, and
it can be hard to suppress the instinct to read the dump as if it were
preorder. This change introduces a new dump mode that addresses both
problems by using box-drawing characters to explicitly represent the
parent-child edges of the tree. This new mode is the default, but the
old behavior remains available with
`--parse-dump-format=yaml-postorder`.
This is in addition to finding a `where` on the RHS of another `where`.
Since a generic binding introduces `.Self`, any `where` expression that
isn't part of a facet type modifying the binding itself would introduce
an ambiguous `.Self`.
Add virtual parse nodes for let, var, and form bindings, which goes
before the type. This allows us to track if `where` appears in the
binding's type. We only need to look for an invalid `where` if any
appeared in the type. We combine these three nodes together into a
single node kind, which requires us to remove the name from it as a
child. We move it up to the Pattern node again, and rename the
PatternStart nodes to PatternTypeStart as they are now located in the
middle of the Pattern nodes, just before the type.
And we only need to thaw `.Self` in generic bindings. Non-generic
bindings can only have `.Self` through a `where` expression, since the
name is not provided otherwise to non-generic bindings. And `where`
expressions thaw their `.Self` independently. So the binding only needs
to thaw a `.Self` that it introduced, which is only for generic
bindings.
Implement the toolchain side of proposal #7254, removing the `:!`
binding
syntax for generic and template parameters in favor of the keywords
`generic`,
`template`, and `runtime` plus contextual defaults for phase.
For valid programs this is semantics-preserving: each binding resolves
to the
same phase, and produces the same SemIR, as it did under `:!`/`:`. The
parser
derives a binding's phase from its syntactic context plus any explicit
phase
keyword; new diagnostics and error recovery for misused keywords are
described
below.
Implementation details for each component:
- Lexer: remove the `:!` (`ColonExclaim`) token, move its virtual
parse-node
budget onto `:`, and add the `generic` and `runtime` keywords.
- Parser: thread a `BindingContext` (`ExplicitParam`, `DeducedParam`, or
`CompileTimeEntityParam`) from declaration introducers down through
parameter
lists to each binding pattern, using a one-token lookahead to
distinguish a
name-qualifier parameter list from a declaration's own final list.
Parameters
of a compile-time entity (`class`, `interface`, `constraint`, `choice`,
`alias`, `export`, `namespace`) and deduced `[]` parameters default to
checked
generic; explicit function parameters and local bindings default to
runtime.
`HandleBindingPattern` resolves the phase from that context plus the
keyword: a
`generic` keyword needs no node of its own (the phase is carried by the
binding's node kind), while a `runtime` keyword is preserved as a
`RuntimeBindingName` node so `check` can name it in a diagnostic. A
phase
keyword that is merely redundant with the contextual default is
diagnosed
here, without invalidating the parse tree.
- Check: a phase keyword that is invalid for its context (for example
`runtime`
on a checked-generic parameter) is diagnosed here, and recovers by
building an
error binding that still introduces the name so that later uses of it do
not
produce cascading errors.
The removed `:!` syntax is now rejected as an ordinary parse error.
The `form`/`:?`/`->?` ("extended types") portion of proposal #7254 is
left for a
separate change.
Assisted-by: Claude Code
When parsing a pattern, if we encounter something that isn't pattern
syntax, try parsing as an expression instead. We only need one-token
lookahead to distinguish pattern syntax from expression syntax.
Track a precedence group through pattern parsing so that we can allow
different kinds of expressions in a top-level pattern (such as the
operand of `let`) and in a nested pattern (such as a subpattern of a
tuple pattern or within grouping parens). For example, we do not allow
`case if ...`, and for now I've chosen to also not allow logical or
relational operators at the top level of a pattern, so `case 1 + 1` is
OK, but `case 1 == 1` and `case true and false` require parentheses.
This decision should be ratified or revisited by a design proposal.
Very basic check support is also provided, only sufficient to form an
`ExprPattern` instruction and nothing beyond that. For now, all pattern
matching against an `ExprPattern` fails with a TODO error. To support
that, I've switched from calling `BeginSubpattern` in the parent handler
of a pattern and `EndSubpatternAs*` in the pattern handler itself to
calling both functions in parent handlers, with `EndSubpattern`
converting an expression into an expression pattern where needed.
Depends on #6976.
Assisted-by: Gemini via Google Antigravity
This change ensures that a function signature always starts with an
`IdentifierNameMaybeBeforeSignature` node (renamed from
`IdentifierNameBeforeParams`), even in the case of function declarations
like `fn F -> T` that have no parameter list. As a consequence, this
ensures that we push new entries onto `pattern_block_stack` and
`full_pattern_stack` when we start processing the function signature.
Every test that used `addr` before #6283 should be using `ref` after
this PR. In most cases that was done in #6283, but this PR transitions a
few that I missed in that first pass. In addition, #6283 cloned the old
`addr` tests from `foo.carbon` to `foo_addr.carbon` in order to maintain
test coverage during the transition; this PR removes those cloned tests.
`fn destroy` is being removed per decision on #6124. It seems like the
relevant decision will result in no more keyword-based function names,
so this is removing all related support.
This proposal renames the syntax used to mark an overriding definition
of a virtual method from `impl fn` to `override fn` to avoid ambiguity:
besides indicating an overriding virtual function, it can be parsed as
an "impl" declaration when the construct following "impl" begins with a
lambda introduced by "fn".
Closes#5711
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Richard Smith
Geoff Romer
Dana Jansens
Chandler Carruth
Elliott KaltandRichard Smith
