Replaces several quadratic-time steps in the algorithm with linear or n
log n implementations. Worst-case runtime before hitting the complexity
bailout drops from 25ms to about 12ms, and typical runtime is
single-digit ms on my development machine.
Assisted-by: Claude Code
Makes following changes to Carbon::Format()
- TokenKind::Period (i.e. `.`) should never have a space before or after
it.
- TokenKind::CloseSquareParen (i.e. `]`) should be treated as packed
content (no space preceeding it)
- Only exception I can think of is `impl forall [...]`
- Remove preceeding space from `[` and `(` if previous token was an
identifier (or identifier-ish token)
- Remove seperator following `++` / `--` unary operators.
- Explicit gaps in source code should be retained, up to 2 new lines.
Multiple test files were added to test formatting.
I imagine eventually this will need to be updated to read parse tree to
gather more context but this atleast lets us get a decent-ish format for
many of our current sample files (e.g. sieve.carbon)
Assisted-With: Gemini / Antigravity
---------
Co-authored-by: David Blaikie <dblaikie@gmail.com>
Adds a virtual node to `DefaultValuePattern` to end the
`ExprRegionForPattern` before checking the expression
for the default value.
When checking the default value expression, the context
was still configured to interpret expressions as patterns,
which caused some corruption of state with tuple-pattern
subpatterns.
Corrects an assertion failure I found while working on
feedback from #7665.
We previously passed in the start of file token, classified as
BracketTokenKind::Other, which allowed the bracket fixer to consider
corrections where it inserted tokens (such as a `{`) *before* the
start-of-file token.
Fixes#7672.
Adds an algorithm to compute where to insert brackets to repair
bracketing mismatches during lexing. This takes indentation, as well as
a number of other cues, into account to predict where the brackets
should have gone. Detects when there is ambiguity between solutions and
makes no suggestion in that case. Reduces the problem by splitting on
properly bracketed top-level constructs, then uses a beam search to find
good candidate solutions quickly.
This includes both a fuzzer and an eval tool that can be used to
determine how well the algorithm fares against a given corpus of valid
Carbon code, by damaging it in various ways and seeing whether the
algorithm can correctly fix it. On all the eval modes, this algorithm
can correctly infer the positions for over 80% of lost brackets (and can
correctly restore 95+% of brackets in some modes), with low rates of
incorrect suggestions.
See added documentation for full details.
Assisted-by: Gemini via Antigravity, Claude via Claude Code
- Enhance toolchain/parse/node_kind.def with category x-macros
(CARBON_PARSE_NODE_KIND_DECLARATION, CARBON_PARSE_NODE_KIND_EXPRESSION,
CARBON_PARSE_NODE_KIND_PATTERN, CARBON_PARSE_NODE_KIND_STATEMENT) that
default to CARBON_PARSE_NODE_KIND, keeping node_kind.def as the single
source of truth without manual expansion divergence.
- Decompose the monolithic toolchain/parse/extract.cpp translation unit
into separate compilation units for declarations, expressions, patterns,
and statements by expanding respective category x-macros.
- Forward-declare Tree and TokenizedBuffer in parse and lex dump headers
to reduce header inclusion depth.
Assisted-by: Antigravity with Gemini
- Consolidate token kind test assertions into helper functions using
EXPECT_THAT(spelling.str(), MatchesRegex(...)) under a single test case
rather than generating ~130 separate TEST classes and redundant matcher
instantiations.
- Slashes individual translation unit compilation time from 59.1s to ~3s
while keeping EXPECT_THAT for clear diagnostic error output and full
token coverage.
Assisted-by: Antigravity with Gemini
Addresses part of issue raised in #7159 by implementing float.negate
builtin for FloatLiteralValues
The following code now compiles:
```
let a: f64 = -1.0;
```
To achieve this I switch the mantissa from being unsigned to signed.
Lexed literals within source file will still always be unsigned, however
it is now possible to create negative FloatLiteralValue constants. Main
non-local changes this causes is:
- All llvm::APInt parsing / printing calls flipped isSigned param
- Zero extension replaced with sign extension
- getActiveBits() replaced with getSignificantBits() for min bit width
calculation
This is in addition to finding a `where` on the RHS of another `where`.
Since a generic binding introduces `.Self`, any `where` expression that
isn't part of a facet type modifying the binding itself would introduce
an ambiguous `.Self`.
Add virtual parse nodes for let, var, and form bindings, which goes
before the type. This allows us to track if `where` appears in the
binding's type. We only need to look for an invalid `where` if any
appeared in the type. We combine these three nodes together into a
single node kind, which requires us to remove the name from it as a
child. We move it up to the Pattern node again, and rename the
PatternStart nodes to PatternTypeStart as they are now located in the
middle of the Pattern nodes, just before the type.
And we only need to thaw `.Self` in generic bindings. Non-generic
bindings can only have `.Self` through a `where` expression, since the
name is not provided otherwise to non-generic bindings. And `where`
expressions thaw their `.Self` independently. So the binding only needs
to thaw a `.Self` that it introduced, which is only for generic
bindings.
The `//@include-in-dumps` and `//@dump-sem-ir-begin`/`-end` tooling
directives were consumed for their side effects without a comment
record, so the tokens and comments together no longer reconstructed the
source: tooling that re-emits a file from them, such as `carbon format`,
silently dropped the directive lines. Now each recognized directive line
is also recorded as an ordinary full-line comment alongside its side
effect.
Adjacent full-line comments coalesce into one comment record only within
a category, determined by the byte after the `//` introducer: ordinary
comments (whitespace, or the end of the line or file), `//@...`
directives, and invalid introducers. A transition between categories
starts a new record, so a directive next to a comment block is its own
comment, while all the invalid spellings lump together to keep the
diagnostic noise at one per run.
The category boundary also fixes a lost directive: the invalid-comment
bulk skip compared only the `//` prefix, so `//!x` directly above
`//@dump-sem-ir-begin` absorbed the directive line and its side effect
was never recorded. Invalid comment runs now skip line by line (they
start from a diagnosed error, so they are not hot) and stop at a
whitespace or `@` introducer; the SIMD bulk skip handles only ordinary
comment blocks, whose prefix comparison already includes the whitespace
byte.
Nothing outside the lexer and the formatter reads comment records, and
lex dumps do not include comments, so no other behavior changes.
Assisted-by: Claude Code
Implement the toolchain side of proposal #7254, removing the `:!`
binding
syntax for generic and template parameters in favor of the keywords
`generic`,
`template`, and `runtime` plus contextual defaults for phase.
For valid programs this is semantics-preserving: each binding resolves
to the
same phase, and produces the same SemIR, as it did under `:!`/`:`. The
parser
derives a binding's phase from its syntactic context plus any explicit
phase
keyword; new diagnostics and error recovery for misused keywords are
described
below.
Implementation details for each component:
- Lexer: remove the `:!` (`ColonExclaim`) token, move its virtual
parse-node
budget onto `:`, and add the `generic` and `runtime` keywords.
- Parser: thread a `BindingContext` (`ExplicitParam`, `DeducedParam`, or
`CompileTimeEntityParam`) from declaration introducers down through
parameter
lists to each binding pattern, using a one-token lookahead to
distinguish a
name-qualifier parameter list from a declaration's own final list.
Parameters
of a compile-time entity (`class`, `interface`, `constraint`, `choice`,
`alias`, `export`, `namespace`) and deduced `[]` parameters default to
checked
generic; explicit function parameters and local bindings default to
runtime.
`HandleBindingPattern` resolves the phase from that context plus the
keyword: a
`generic` keyword needs no node of its own (the phase is carried by the
binding's node kind), while a `runtime` keyword is preserved as a
`RuntimeBindingName` node so `check` can name it in a diagnostic. A
phase
keyword that is merely redundant with the contextual default is
diagnosed
here, without invalidating the parse tree.
- Check: a phase keyword that is invalid for its context (for example
`runtime`
on a checked-generic parameter) is diagnosed here, and recovers by
building an
error binding that still introduces the name so that later uses of it do
not
produce cascading errors.
The removed `:!` syntax is now rejected as an ordinary parse error.
The `form`/`:?`/`->?` ("extended types") portion of proposal #7254 is
left for a
separate change.
Assisted-by: Claude Code
`match_first` is a declaration followed by a curly-brace block of
declarations.
The check phase will ensure only impl decls (or defns) are inside the
block.
`ErrorRecoveryBuffer::Apply` flagged each inserted token via the
pre-insertion index of the token it was inserted before. Once any
earlier insertion had been applied, that index no longer matches the
token's position in the merged list: with two closers inserted at the
same point (`{((}`), the second recovery token was left unflagged, and
with insertions at two separate points (`{(} {(}`), a real token was
flagged in the second one's place.
`IsRecoveryToken` now reports exactly the inserted tokens. This will
matter for `carbon format`: a recovery token's text exists in no source
byte range, so a minimal-edit model must know not to anchor edits on it.
Assisted-by: Claude Code
Carbon currently requires a comment to be the only non-whitespace on its
line. A `//` comment that follows other content on a line, called a
_trailing comment_, is a lexer error. This proposal removes that
restriction, allowing a comment to follow other content on a line.
Everything else about comments is unchanged: a comment still begins with
`//`, still requires whitespace after the `//`, and still runs to the
end of the line. Carbon continues to provide only line comments; no
block or intra-line comments are added.
Three observations motivate the change. First, trailing comments are
well suited to short _annotations_ attached to a specific entity or
value on a line. Second, the lexer design now makes it trivial to lex
trailing comments, and in fact requires extra logic and potentially cost
to reject them. Third, C++ code routinely uses trailing comments, so
allowing them lets Carbon carry the layout of migrated code over
directly, rather than reworking each comment to read well in a different
structure.
Implementation notes (beyond the proposal's design):
Keeping trailing comments cheap to lex required a few supporting
changes, all of which keep the cost off the lexer's hot path:
- The lexer already dispatches `//` to comment lexing wherever it
appears, so classifying a comment as trailing is a single O(1) check of
whether the `//` is the line's first non-whitespace (`start + indent`).
The hot comment path is otherwise unchanged.
- That check relies on each line's recorded indentation being its real
leading whitespace. Multi-line string literals previously recorded the
column where the literal opened for the lines they span; they now record
the true (closing-delimiter) indentation instead.
- Parser error recovery (`SkipPastLikelyEnd`) had relied on that
opening-column indentation to keep tokens following a multi-line string
literal attached to the same construct. It now reconstructs that
relationship directly by consulting the line on which the literal
opened, including when other tokens follow the closing delimiter (such
as `''' + "more"`). This is on the cold recovery path.
- `CommentData` records the trailing bit in the high bit of its length
field, keeping it at 8 bytes.
Assisted-by: Claude Code
`IsRawIdentifier` checked `token_text.starts_with("r#")` and then read
`token_text[2]`, but `starts_with` only guarantees a length of two. An
`r` identifier immediately followed by `#` at the end of the source --
so the token text is exactly `r#` -- made the `token_text[2]` read run
off the end. Guard on the length first.
Found by fuzzing. The read is reached only via `GetTokenText`, so the
parser fuzzer, which does not request token text, never hit it.
Assisted-by: Claude Code
Implements proposal #7016: `self` moves from the deduced implicit list
(`fn F[self: Self]()`) to the front of the explicit list. Its type may
be written explicitly (`fn F(self: Self)`) or omitted, in which case it
defaults to `Self` (`fn F(self)`, `fn F(ref self)`); `self` in the
implicit list is rejected.
Throughout checking, `self` is modeled as the first explicit parameter.
Because a method is just a function whose first parameter is `self`, it
can also be called as an ordinary function with the receiver passed
explicitly (`Type.M(obj, ...)`), not only as `obj.M(...)`. A new
`SemIR::CallArgParamPatterns` helper chooses the parameters matched
against the explicit arguments, excluding a leading `self` only when it
is supplied as a method-call receiver; arity checking, conversion, and
generic deduction use it. The resulting SemIR and lowering are
unchanged: `self` is still `call_param0`, and witnesses, thunks, and
vtables are unaffected.
An omitted `self` type is parsed as a `SelfBindingPattern` node with no
type expression; checking synthesizes the `Self` type so it behaves
exactly like `self: Self`. However, the exact spelling used must match
between a forward declaration and a definition, following #3763's rules
around declaration matching.
Generated functions, thunks, and C++ interop import/export build `self`
as the first explicit parameter, and the `self`-type override (e.g.
Derived->Base for a virtual override) applies to the explicit `self`.
Placement is validated by new diagnostics: `SelfInImplicitParamList`,
`SelfNotFirstParam`, and `SelfOutsideParamList`. The benchmark source
generator and the documentation adopt the `(self)` shorthand; the
prelude, the examples, and the test data are migrated in the following
commits.
Assisted-by: Claude Code with Claude Opus 4.7
---------
Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com>
If a keyword or a sized type literal (eg, `f2`) is used in a context
where we are confident that we are expecting an identifier -- either
before a `:` in a binding pattern or after a `.` in a member access or
designator -- then recover as if a raw identifier was used.
This appears to be a particular stumbling block for coding agents, so
seems worth paying special attention to.
Add a mechanism to the tokenized buffer to track additional tokens
synthesized for error recovery so that we can keep the lexed token
sequence immutable and still satisfy the invariants throughout the rest
of the toolchain for recovery tokens. Thanks to chandlerc for suggesting
this approach!
This adds the `static` token to the lexer and parses it as a modifier.
In check, `FullPatternStack::Kind::FieldDecl` is now used for both
static and non-static vars. Static vars get treated basically the same
as `NameBindingDecl`s.
Global initialization is used for static var initializers. To make the
necessary stack information available to `pattern_match.cpp`, the
`full_pattern_stack` and `decl_introducer_state_stack` are now popped
later in `handle_let_and_var.cpp`.
In lowering, each class's body is checked for `VarStorage` insts and
lowered the same as global vars.
The benchmarks themselves aren't really specific to `driver`.
Keeping the source generation near to the primary use case of
benchmarking also seems like a more discoverable location.
I feel a little bad doing this reorganization right after I gave a talk
with links to a bunch of this code, but seems good to reorganize a bit
before doing some work to extend things now that we have full standard
library support for C++ benchmarking and other improvements.
Assisted-by: Antigravity with Gemini
As part of this, move functions that seem reasonable to make out-of-line
to a separate `_impl.h` header file that is only included where the
explicit instantiation _definition_ is provided.
By using explicit instantiation we can make these templates behave more
like non-template classes in terms of supporting out-of-line definitions
that don't need to be compiled by every translation unit. The set of
eventual instantiations here is fundamentally known, and there tend to
be headers that define a canonical "leaf" type where it makes sense to
trigger the explicit instantiation.
Where we already had a `.cpp` file to put the explicit instantiation
definition, use it. But in some places we didn't have such a `.cpp` file
so this PR adds those.
This also requires that we have precise constraints on APIs that _can't_
be instantiated for specific argument types, as now we don't do this
lazily.
Combined, this appears to reduce the sum of object file sizes in the
`check` directory by almost 40% (122mb -> 74mb) in my measurement.
My actual goal was to improve compile times, but so far I don't have a
great methodology for measuring these... But the object file size
reduction seems to confirm this is a net win and likely represents a
non-trivial improvement in compile time.
Assisted-by: Antigravity with Gemini
Because this is an `__attribute__((used))` method in a templated base
class it forces a _huge_ amount of template instantiation in every
translation unit.
Often this was just printing the members of the type, which is still
useful in some cases (such as test output), but adds no value in the
debugger.
A more successful pattern for dumping has been namespace level
functions, and particularly static ones that more transparently don't
expand the non-debugger API surface. Add the few missing functions there
that cover `Printable` types with more interesting contents.
For several of these, it just gives us a "dump the whole thing" function
as a compliment to "dump this entity in the thing". These probably
aren't especially high value, but moving them here they become cheap, so
I've left them in.
For a couple, this expands the rich dumping support of SemIR constructs,
which should be substantially more useful than the previous `Dump`
behavior.
This reduces `check` cumulative object file size by another 14%.
Assisted-by: Antigravity with Gemini
By default clang-format interprets function-like macro invocations as
function calls. E.g. the argument of `CARBON_KIND(llvm::ListSeparator*
sep)` is interpreted as an expression, meaning the `*` is an infix
binary operator, so it inserts a space before the `*`. This change
teaches clang-format that `CARBON_KIND(x)` and
`CARBON_ASSIGN_OR_RETURN(x)` rewrite to `x`, which is close enough to
the truth to enable it to format them correctly. See the [clang-format
docs](https://clang.llvm.org/docs/ClangFormatStyleOptions.html#macros)
for details.
I was looking for uses of APInt that care about the bit width we're
using, just searching for uses of "64", since #6908 started applying the
minimum with of 64 bits more explicitly.
- numeric_literal.cpp: piping through the sign bit request, allowing
`exponent` to assume it's already 64-bit (putting the CHECK in to just
expose the logic, keeping it outside the `if` because the `if` is an
edge case and I was thinking to avoid edge case inconsistencies slipping
by)
- inst_fingerprinter.cpp: reducing logic to copy words
Assisted-by: Google Antigravity with Gemini
This implements the leads decision made in #6821, proposal #6910. The
proposal is pending, but I figured it's relatively safe to just do given
the decision.
Assisted-by: Google Antigravity with Gemini
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
When lexing a hash-prefixed character literal, the lexer assumed that
the hash level of escape sequences inside the literal was zero, which
allowed unclosed escape sequences inside the literal which crashed the
compiler.
Closes#6799
Add support for compile-time functions. `eval fn` is analogous to C++
`constexpr`, and is evaluated at compile time when it has compile-time
arguments. `musteval fn` is analogous to C++ `consteval`, and requires
that its arguments be available at compile time and is always evaluated
at compile time. For now we require the modifier to match across
redeclarations of the function. The specific modifier syntax here is a
placeholder and not yet part of an approved design.
Limitations: Only very basic support for evaluation is provided. So far
there's no support for mutable state or `if` expressions, but otherwise
control flow and passing and returning values should work. Carbon
evaluation recursion is modeled by C++ recursion for now, so you can
overflow the toolchain stack easily. Functions that use in-place
initialization will generally not work yet, as they are modeled as
passing a non-compile-time-constant reference to a temporary to the
call.
Add missing categorization of `name_binding_decl` as `NotExpr` to match
other similar declaration instructions like `FunctionDecl`, so that we
can uniformly skip over them when they occur within function bodies.
Assisted-by: Gemini 3 Pro and Flash via Antigravity
Mainly because "sorting_diagnostic_consumer" is legacy, since
`SortingDiagnosticConsumer` became `SortingConsumer`. Also better
reflecting contents of these files.
Where I'm not renaming, I'm less positive about dropping "diagnostics"
from "file_diagnostics" and "null_diagnostics" (which contain both a
consumer and emitter, and "null.h" seems like poor naming), so not doing
that here. Also "diagnostic.h" contains `struct Diagnostic`, so is a
decent fit.
Assisted-by: Google Antigravity with Gemini 3 Flash
OwningArrayRef is being removed upstream, per
https://github.com/llvm/llvm-project/pull/169126. This replaces uses
with `SmallVector`.
I've also made a separate commit which does init changes; these aren't
strictly necessary, but I added to make it a little more idiomatic in
spots.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
This adds the necessary parser infrastructure to recognize and parse
lambda expressions in Carbon.
Key changes:
- Added and Parse Node Kinds.
- Updated to use to accommodate the growing number of node kinds.
- Implemented parser states and handlers for lambda syntax ( or ).
- Added structure to .
- Added diagnostics for missing lambda bodies.
- Added a stub in phase to defer semantic analysis using .
- Added parser tests for lambdas.
Every test that used `addr` before #6283 should be using `ref` after
this PR. In most cases that was done in #6283, but this PR transitions a
few that I missed in that first pass. In addition, #6283 cloned the old
`addr` tests from `foo.carbon` to `foo_addr.carbon` in order to maintain
test coverage during the transition; this PR removes those cloned tests.
This fixes various violations of C++'s One Definition Rule, where we
accidentally gave the same static data member multiple definitions in
different translation units. Clang happens to emit such definitions with
weak linkage, which allows us to get away with this without link errors,
but it's still formally incorrect.
Also switch keyword order around for a handful of instances of
`constexpr inline`, per agreement in open discussion.
This happens to reduce the size of a `-c dbg` toolchain binary by 7.2
MiB, presumably by making more of our symbols and especially debug info
discardable.
This turns `Cpp` into a keyword, and makes it map to `NameId::Cpp` and
`PackageNameId::Cpp`.
Per discussion with zygoloid, the keyword versus identifier question is
deliberately kept open by #4846. This PR switches to a keyword because
mapping to a specific `PackageNameId` works best with a special `NameId`
not backed by an `IdentifierId`. We could in theory make it work using
`IdentifierId` or a runtime-tracked `PackageNameId` for `Cpp` (e.g.
stored on `SemIR::File`), but this approach is consistent with `Core`
and so seemed like a good starting point.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
They don't get stored anywhere yet, but this type checks the
declarations and diagnoses errors in their form, such as not placing a
facet type after `impls` or a type before it.
`fn destroy` is being removed per decision on #6124. It seems like the
relevant decision will result in no more keyword-based function names,
so this is removing all related support.
This proposal renames the syntax used to mark an overriding definition
of a virtual method from `impl fn` to `override fn` to avoid ambiguity:
besides indicating an overriding virtual function, it can be parsed as
an "impl" declaration when the construct following "impl" begins with a
lambda introduced by "fn".
Closes#5711
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Adds a unit test, and some smaller edits:
- Remove the `=` when defining names, in order to change `}` placement
by clang-format on uses.
- context:
https://github.com/carbon-language/carbon-lang/pull/6053#discussion_r2343423178
- I believe with `EnumBase` that keeping the `=` had been a deliberate
choice, so this PR is intended to confirm that removing it is okay.
- Delete `EnumMaskBase::name`
- context:
https://github.com/carbon-language/carbon-lang/pull/6053#discussion_r2344233707
- We can't just do nothing because `EnumBase::name` uses indexing that's
incompatible with `EnumMaskBase`.
- Some small comment cleanups.
- Tests don't need to be in the `Carbon` namespace anymore, macros work
fine in other namespaces, but it's still the right namespace.
- Documentation on `EnumBase::name` seems to be referring to a prior
structure, wherein we had a macro defining the function instead of the
`Names` array.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Echoing what was added in #5608, updating existing uses. Unfortunately
there's divergent behavior for operators versus constructors, so keeping
the nolint on those.
Following the direction of #5913, add support for parsing an `unsafe as`
operator. For now, we allow one additional conversion using `unsafe as`
beyond the conversions supported by `as`: we permit pointer conversions
that remove qualifiers, such as `const T*` -> `T*`.
Add a content keyword to file_test, `[[@0xAB]]`, that expands to the
code unit 0xAB, and use that instead of putting raw malformed code units
in test files.
Instead of printing the raw input bytes in snippets in diagnostics,
replace non-printable characters with <AB> in the output, being careful
to still compute the location of the caret and underscore properly.
Richard Smith
DavidLoftusandDavid Blaikie
Lucile Rose Nihlen
Özgür T. Önsoy
Chandler Carruth
Dana Jansens
Nicholas Bishop
Geoff Romer
Jon Ross-Perkins
Ilya
Burak Emir
Elliott KaltandRichard Smith