Carbon currently requires a comment to be the only non-whitespace on its
line. A `//` comment that follows other content on a line, called a
_trailing comment_, is a lexer error. This proposal removes that
restriction, allowing a comment to follow other content on a line.
Everything else about comments is unchanged: a comment still begins with
`//`, still requires whitespace after the `//`, and still runs to the
end of the line. Carbon continues to provide only line comments; no
block or intra-line comments are added.
Three observations motivate the change. First, trailing comments are
well suited to short _annotations_ attached to a specific entity or
value on a line. Second, the lexer design now makes it trivial to lex
trailing comments, and in fact requires extra logic and potentially cost
to reject them. Third, C++ code routinely uses trailing comments, so
allowing them lets Carbon carry the layout of migrated code over
directly, rather than reworking each comment to read well in a different
structure.
Implementation notes (beyond the proposal's design):
Keeping trailing comments cheap to lex required a few supporting
changes, all of which keep the cost off the lexer's hot path:
- The lexer already dispatches `//` to comment lexing wherever it
appears, so classifying a comment as trailing is a single O(1) check of
whether the `//` is the line's first non-whitespace (`start + indent`).
The hot comment path is otherwise unchanged.
- That check relies on each line's recorded indentation being its real
leading whitespace. Multi-line string literals previously recorded the
column where the literal opened for the lines they span; they now record
the true (closing-delimiter) indentation instead.
- Parser error recovery (`SkipPastLikelyEnd`) had relied on that
opening-column indentation to keep tokens following a multi-line string
literal attached to the same construct. It now reconstructs that
relationship directly by consulting the line on which the literal
opened, including when other tokens follow the closing delimiter (such
as `''' + "more"`). This is on the cold recovery path.
- `CommentData` records the trailing bit in the high bit of its length
field, keeping it at 8 bytes.
Assisted-by: Claude Code
A lambda introducer (`fn` in expression context) with no parameters and
no body -- for example `(fn)`, where `)` immediately follows `fn` --
left `HandleLambdaAfterParams` calling `ReturnErrorOnState()` without
ever emitting a `Lambda` node. The orphaned `LambdaIntroducer` leaf was
then left where an expression was required, so `Parse` produced a tree
that failed its own verification and aborted via `CARBON_FATAL`.
Recover the way `HandleLambdaBody` already does for a missing body after
a return type: emit a placeholder `InvalidParse` body and finish a
complete `Lambda` node, so the lambda stays a valid expression and the
surrounding construct (here a `ParenExpr`) extracts cleanly.
Found by fuzzing.
Assisted-by: Claude Code
Instead of traversing the entire pattern block looking for
`VarPattern`s, we keep track of them on creation, and then build
`VarStorage`s directly from that list.
This also gets rid of the global `var_storage_map`, and instead keep
that information narrowly scoped to each full-pattern, and consume it in
a single linear traversal instead of with random-access lookups. To
enable that, this fixes a parse bug where nested `var` patterns were
getting diagnosed but not marked as errors.
Malformed `base` declarations with an omitted colon need two different
recovery paths. For `extend base`, the consumed `extend` modifier
requires the parse tree to retain its `BaseColon` and base expression
children, so this synthesizes an errored `BaseColon` and continues
parsing the expression.
Other malformed forms, such as `base calss X {}`, now use the standard
declaration-error recovery: emit `ExpectedAfterBase`, skip past the
likely declaration end, and form an errored `BaseDecl` without inventing
a colon or cascading diagnostics.
The regression covers `extend base Foo;`, bare `base;`, and the reviewer
counterexample `base calss X {}`.
Tests:
- `prek run --files toolchain/parse/handle_base.cpp
toolchain/parse/testdata/class/fail_base.carbon`
- `./scripts/run_bazelisk.py test -c dbg //toolchain/parse/...`
- `./scripts/run_bazelisk.py test -c dbg //toolchain/testing:file_test`
AI assistance: OpenAI Codex helped inspect the parser recovery path,
implement the change, and run verification. The operator reviewed and
authorized the contribution.
Assisted-by: OpenAI Codex
Implements proposal #7016: `self` moves from the deduced implicit list
(`fn F[self: Self]()`) to the front of the explicit list. Its type may
be written explicitly (`fn F(self: Self)`) or omitted, in which case it
defaults to `Self` (`fn F(self)`, `fn F(ref self)`); `self` in the
implicit list is rejected.
Throughout checking, `self` is modeled as the first explicit parameter.
Because a method is just a function whose first parameter is `self`, it
can also be called as an ordinary function with the receiver passed
explicitly (`Type.M(obj, ...)`), not only as `obj.M(...)`. A new
`SemIR::CallArgParamPatterns` helper chooses the parameters matched
against the explicit arguments, excluding a leading `self` only when it
is supplied as a method-call receiver; arity checking, conversion, and
generic deduction use it. The resulting SemIR and lowering are
unchanged: `self` is still `call_param0`, and witnesses, thunks, and
vtables are unaffected.
An omitted `self` type is parsed as a `SelfBindingPattern` node with no
type expression; checking synthesizes the `Self` type so it behaves
exactly like `self: Self`. However, the exact spelling used must match
between a forward declaration and a definition, following #3763's rules
around declaration matching.
Generated functions, thunks, and C++ interop import/export build `self`
as the first explicit parameter, and the `self`-type override (e.g.
Derived->Base for a virtual override) applies to the explicit `self`.
Placement is validated by new diagnostics: `SelfInImplicitParamList`,
`SelfNotFirstParam`, and `SelfOutsideParamList`. The benchmark source
generator and the documentation adopt the `(self)` shorthand; the
prelude, the examples, and the test data are migrated in the following
commits.
Assisted-by: Claude Code with Claude Opus 4.7
---------
Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com>
If a keyword or a sized type literal (eg, `f2`) is used in a context
where we are confident that we are expecting an identifier -- either
before a `:` in a binding pattern or after a `.` in a member access or
designator -- then recover as if a raw identifier was used.
This appears to be a particular stumbling block for coding agents, so
seems worth paying special attention to.
Add a mechanism to the tokenized buffer to track additional tokens
synthesized for error recovery so that we can keep the lexed token
sequence immutable and still satisfy the invariants throughout the rest
of the toolchain for recovery tokens. Thanks to chandlerc for suggesting
this approach!
This adds the `static` token to the lexer and parses it as a modifier.
In check, `FullPatternStack::Kind::FieldDecl` is now used for both
static and non-static vars. Static vars get treated basically the same
as `NameBindingDecl`s.
Global initialization is used for static var initializers. To make the
necessary stack information available to `pattern_match.cpp`, the
`full_pattern_stack` and `decl_introducer_state_stack` are now popped
later in `handle_let_and_var.cpp`.
In lowering, each class's body is checked for `VarStorage` insts and
lowered the same as global vars.
Class vars are still restricted to simple `name: type` bindings, not
full patterns. This is now handled in the check phase instead of during
parsing.
This is in preparation for supporting `static var`.
For #6830, add support for inline C++ fragments as a declaration rather
than as a packaging directive. For now, this uses `inline Cpp
<string-literal>;` as syntax. The prior `import Cpp inline
<string-literal>;` is left alone for the time being. We can decide
separately whether to remove that.
`inline Cpp` requires that there was at least one `import Cpp`. It's not
clear to me if that's the right design long-term, but it seems
reasonable for now.
Assisted-by: Gemini via Google Antigravity
When parsing a pattern, if we encounter something that isn't pattern
syntax, try parsing as an expression instead. We only need one-token
lookahead to distinguish pattern syntax from expression syntax.
Track a precedence group through pattern parsing so that we can allow
different kinds of expressions in a top-level pattern (such as the
operand of `let`) and in a nested pattern (such as a subpattern of a
tuple pattern or within grouping parens). For example, we do not allow
`case if ...`, and for now I've chosen to also not allow logical or
relational operators at the top level of a pattern, so `case 1 + 1` is
OK, but `case 1 == 1` and `case true and false` require parentheses.
This decision should be ratified or revisited by a design proposal.
Very basic check support is also provided, only sufficient to form an
`ExprPattern` instruction and nothing beyond that. For now, all pattern
matching against an `ExprPattern` fails with a TODO error. To support
that, I've switched from calling `BeginSubpattern` in the parent handler
of a pattern and `EndSubpatternAs*` in the pattern handler itself to
calling both functions in parent handlers, with `EndSubpattern`
converting an expression into an expression pattern where needed.
Depends on #6976.
Assisted-by: Gemini via Google Antigravity
Do not treat it as a 1-tuple pattern as we used to. The design indicates
that `(pattern)` is invalid, but this appears to be an oversight, and
grouping parens appear to be the intended interpretation.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
Use the same node kind for the body of `case` and `default` handlers. We
don't need to distinguish these in check, so don't create extra node
kinds for them.
In order to make the nodes properly delimited, make the label (`case
...` or `default`) nodes be children of the `=>` node rather than
siblings. This allows us to use the node kind of the `=>` as the
bracketing node for the complete handler, rather than having two
different bracketing node kinds, one for each kind of label.
Fixes a compiler crash that occurs when a malformed lambda is provided
as an operand to an operator that strictly expects an expression
Changes:
- Emits an `InvalidParse` dummy node at the current position to act as a
placeholder for the missing body
- Changed state transitions so that `LambdaIntroducer` gets properly
wrapped into a `Lambda` node
Closes#6823
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
Co-authored-by: Carbon Infra Bot <carbon-external-infra@google.com>
This change ensures that a function signature always starts with an
`IdentifierNameMaybeBeforeSignature` node (renamed from
`IdentifierNameBeforeParams`), even in the case of function declarations
like `fn F -> T` that have no parameter list. As a consequence, this
ensures that we push new entries onto `pattern_block_stack` and
`full_pattern_stack` when we start processing the function signature.
Implementation of unused pattern bindings #2022, continued.
Whereas previous PR #6460 took care of parsing, and PR #6479 prepared
the stage by using _ in some test cases, this PR has the the actual
implementation, using a simple dataflow analysis.
---------
Co-authored-by: Burak Emir <bqe@google.com>
Co-authored-by: jonmeow <jperkins@google.com>
Add support for compile-time functions. `eval fn` is analogous to C++
`constexpr`, and is evaluated at compile time when it has compile-time
arguments. `musteval fn` is analogous to C++ `consteval`, and requires
that its arguments be available at compile time and is always evaluated
at compile time. For now we require the modifier to match across
redeclarations of the function. The specific modifier syntax here is a
placeholder and not yet part of an approved design.
Limitations: Only very basic support for evaluation is provided. So far
there's no support for mutable state or `if` expressions, but otherwise
control flow and passing and returning values should work. Carbon
evaluation recursion is modeled by C++ recursion for now, so you can
overflow the toolchain stack easily. Functions that use in-place
initialization will generally not work yet, as they are modeled as
passing a non-compile-time-constant reference to a temporary to the
call.
Add missing categorization of `name_binding_decl` as `NotExpr` to match
other similar declaration instructions like `FunctionDecl`, so that we
can uniformly skip over them when they occur within function bodies.
Assisted-by: Gemini 3 Pro and Flash via Antigravity
This adds the necessary parser infrastructure to recognize and parse
lambda expressions in Carbon.
Key changes:
- Added and Parse Node Kinds.
- Updated to use to accommodate the growing number of node kinds.
- Implemented parser states and handlers for lambda syntax ( or ).
- Added structure to .
- Added diagnostics for missing lambda bodies.
- Added a stub in phase to defer semantic analysis using .
- Added parser tests for lambdas.
Every test that used `addr` before #6283 should be using `ref` after
this PR. In most cases that was done in #6283, but this PR transitions a
few that I missed in that first pass. In addition, #6283 cloned the old
`addr` tests from `foo.carbon` to `foo_addr.carbon` in order to maintain
test coverage during the transition; this PR removes those cloned tests.
This turns `Cpp` into a keyword, and makes it map to `NameId::Cpp` and
`PackageNameId::Cpp`.
Per discussion with zygoloid, the keyword versus identifier question is
deliberately kept open by #4846. This PR switches to a keyword because
mapping to a specific `PackageNameId` works best with a special `NameId`
not backed by an `IdentifierId`. We could in theory make it work using
`IdentifierId` or a runtime-tracked `PackageNameId` for `Cpp` (e.g.
stored on `SemIR::File`), but this approach is consistent with `Core`
and so seemed like a good starting point.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
They don't get stored anywhere yet, but this type checks the
declarations and diagnoses errors in their form, such as not placing a
facet type after `impls` or a type before it.
Type check named constraint decls and definitions. We don't correctly
error if you put a `fn` inside them. There is no support for `require`
or `alias` yet, so there's nothing useful you can do with them yet.
We have attempted to share code between `interface` and `constraint` as
they are quite similar. First by splitting out some of
handle_interface.cpp to a separate file. Second by sharing some code
paths when you want a facet type from either one, as they both turn into
a facet type.
`fn destroy` is being removed per decision on #6124. It seems like the
relevant decision will result in no more keyword-based function names,
so this is removing all related support.
This proposal renames the syntax used to mark an overriding definition
of a virtual method from `impl fn` to `override fn` to avoid ambiguity:
besides indicating an overriding virtual function, it can be parsed as
an "impl" declaration when the construct following "impl" begins with a
lambda introduced by "fn".
Closes#5711
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Decouples associated constants from being special cased in let handlers.
Enforces associated constant grammar restrictions in parsing instead of
checking.
Closes#5411
Following the direction of #5913, add support for parsing an `unsafe as`
operator. For now, we allow one additional conversion using `unsafe as`
beyond the conversions supported by `as`: we permit pointer conversions
that remove qualifiers, such as `const T*` -> `T*`.
Add a `Core.String` class to the prelude representing a string view, and
rename the `String` keyword to `str` and make it evaluate to
`Core.String`.
`Core.String` is represented as a pair of a pointer to a character
(actually, to the first character of a string, but we don't have a way
of modeling that yet) and a size (which should be pointer-width, but is
currently always a `u64` as we don't have a `usize` equivalent yet).
`Core.String` values are generated directly by the toolchain for string
literal expressions.
This follows the direction established at the recent summit, but the
design implemented here has not been through the proposal process yet.
We add a virtual node (`CompileTimeBindingPatternStart`) as the first
child of `CompileTimeBindingPattern` which holds the identifier
underneath it, so that it is checked just before the type expression of
the `CompileTimeBindingPattern`. When we reach this virtual node during
check, we add `.Self` as a name in the current scope, and when we reach
`CompileTimeBindingPattern` we remove it from scope, which ensures it's
present during only the checking of the type expression for the compile
time pattern.
At the moment the `.Self` has a different type (it's a `TypeType`) than
other `.Self` in the facet type (which are a single `FacetType`), but
the intention is to immediately substitute it out of the facet type
entirely, replacing it with a reference to the compile time binding (a
`BindSymbolicName`) itself. A TODO has been added for this.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
toolchain/check/testdata/builtins/char/basics.carbon and
toolchain/lower/testdata/builtins/char.carbon are probably the most
interesting tests here. The parse tests is required because this adds a
new node kind, and we need coverage of it; but the attached info is
minor. There's a fair amount of test churn here because I'm adding the
Core.Char and Core.CharLiteral types as new singletons.
My intent here is that `CharId` is always a unicode code point, even
when the type is a `Char` and thus must be a single UTF-8 code unit
(single byte). This mainly means the stored value of a `CharValue` can
be printed internally without knowing the type.
---------
Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This adds support for importing C++ code directly from source rather
than via a `#include`.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Adds minimal defaults for codegen so that those tests don't need to
specify the full command line.
Moves driver/testdata/compile to check/testdata/basics/raw_sem_ir, which
seems like it better reflects the focus. Removes the textual IR test
because we have plenty of those now.
Moves a multiline token diagnostic test to parse because (a) diagnostics
doesn't have other tests and (b) this is really testing the way that
parse structures the location.
Drops `--include-diagnostic-kind` from some driver tests that aren't
testing a diagnostic, so the flag felt a bit like noise (even before
this change, there are a few tests that don't specify it because they're
not intended to test an emitted diagnostic, just high-level diagnostic
behaviors).
Adds some comments to reflect my understanding of why a test exists,
when I was pausing to think about it.
Modifies the stdin test to do more dumps; which happens to expose we
currently misbehave.
According to approved syntax at
https://github.com/carbon-language/carbon-lang/blob/trunk/proposals/p3848.md#syntax-defined,
`fn F[]` without explicit parameters should be valid. This makes it
work, then adds some validation to prevent `class C[]` in check.
Note that for `fn`, positional parameters are a TODO -- but this allows
me to test validation in `fn destroy[]` which is rejected, not just a
TODO.
Chandler Carruth
Geoff Romer
mstr-six
Richard Smith
Nicholas Bishop
Özgür
Elliott KaltandRichard Smith
