Clarify that the functions that search for a witness in a facet type no
longer take inputs from the query directly, but now take them from an
identified facet type constructed from the query. That means the self
type is already canonicalized.
We used to have to batch these at the end of pattern traversal in order
to avoid accidentally adding them to a block that was speculatively
pushed for an expression within a pattern, but that's no longer a
concern with the more precise handling of those speculative blocks in
#7445.
In #7436 we stopped substituting `.Self` when collecting witnesses out
of a facet type. While this was correct, it did not capture all the
cases that need to avoid substituting `.Self`. And it poisoned the
`IdentifiedFacetType` cache by not replacing `.Self` but storing the
result in the cache. This led to incoherent behaviour, where the result
of an impl lookup would change depending on which ones had been done
previously.
Now we use a flag to track for each `.Self` if we're currently
type-checking inside the scope where it was introduced in a facet type.
While inside that scope, identify should not replace the `.Self`. Any
use of it should remain as-is since we don't yet know what value will
replace it. We call this state "frozen" since it should not be modified
by identify. This requires a substitution step when we leave the scope
that introduced the `.Self`, to remove the flag. The flag is set in the
`EntityName` of the `SymbolicBinding`, and is part of the canonical
value, since `.Self` can become part of types, which are constants, and
the flag needs to follow it for correct behaviour.
We also have to ensure the flag is the same when doing comparison with
constants from inside a facet type and constants from outside. For
instance in `(Z where .Z1 = ()) where .Z2 = .Z1`, when we arrive at the
second `.Z1` its `.Self` will be frozen, while the `.Z1 = ()` contains a
non-frozen `.Self`. So we add the frozen flag to the first when storing
it in `where_stack` in order to compare the constant values of the two
`.Z1`.
The `WhereExpr` requirement inst kinds now have an `InstConstantKind` of
`AlwaysUnique` instead of `Never`. This allows us to add them to the
usual InstBlocks, and in an `eval fn` body they have a constant value,
so eval does not fail when trying to call that function. We have to be
careful to not consider `AlwaysUnique` as being actually concrete
though, since their constant value erases `.Self`-dependence. This
allows us to stop special casing them when thawing the requirements
block in a `WhereExpr`, and we can just thaw each `InstId` in the block
in a straightforward manner.
We add the new flag to the instruction's fingerprint and name in
formatted semir.
This adds SemIR structs and implements building `observe` lists, as well
as naming, formatting, and importing `observe` declarations.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
This allows C++ to call Carbon functions with generic type parameters,
with some conditions. Example:
```carbon
interface I {
fn Doit(self);
}
class A {
impl as I { fn Doit(unused self) {} }
}
class B {
impl as I { fn Doit(unused self) {} }
}
fn F[T:! I](t: T) {
t.Doit();
}
inline Cpp '''
void G() {
Carbon::A a;
Carbon::B b;
Carbon::F(a);
Carbon::F(b);
}
''';
```
The initial support is limited; only explicit parameters are handled
currently.
`CarbonExternalASTSource::GetOrExportFunctionToCpp` now generates a
`clang::FunctionTemplateDecl` for generic Carbon functions. If C++ code
attempts to call that templated function,
`CarbonExternalASTSource::LoadExternalSpecializations` will be called
with the template argument types of that call site. Then we can generate
a specialized thunk for those argument types for C++ to call.
This prevents an issue when trying to check in changes top
bazel-generated template files,
which aren't actually valid C++, making clangd_tidy issue errors and
block the commit.
This follows the pattern of our main compile benchmarks, but instead of
running the compile through a library API, it does so by running a
separate process. This lets us see the performance that we would expect
from `make` or another build system invoking the compiler, as opposed to
a minimal view from the library API.
Assisted-by: Claude Code
In some cases the pattern block can depend on the initializer, so it
must be sequenced after it. See #7469 for a more detailed explanation of
why this is necessary.
Each file dump now starts with a `; ---` comment and ends with a blank
line. This makes it easier to visually scan the dump for a file of
interest. The comment format is somewhat arbitrary; I chose `---` to
align with the `--- filename.carbon` separator in SemIR dumps, but
without the filename, because that appears on each of the next two lines
already.
In `SourceGenTest.IdentifierByteSumStableAcrossSeeds`, the `first`
variable was storing `llvm::StringRef`s pointing to memory allocated by
a temporary `SourceGen` instance. This memory was freed at the end of
the loop iteration, leaving `first` with dangling references that were
accessed in subsequent iterations.
Fix this by storing `std::string` copies of the identifiers in `first`
to own the memory, and use `llvm::equal` to compare them.
Assisted-by: Antigravity with Gemini
This performs `.Self` substitution in a single step, for the whole facet
type, instead of doing it individually for each constraint visited in
the top-level facet type. Then we don't need to track state to avoid
subst in constraints that come from other named constraints.
The semir changes are because we now generate a whole other FacetType
from the substitution.
Add a result for Subst() to return when you want to recurse into the
instructions operands but not the type_id. This comes up when recursing
and looking for facet types written in an instruction, but not
referenced indirectly through a type_id.
We can't skip adding the instruction to the worklist entirely, since we
need to pop it back off to rebuild the containing instruction later. So
we just mark it with a skip flag, and don't call Subst() on it.
The suggestion for a change to Subst was made in
https://github.com/carbon-language/carbon-lang/pull/7367#discussion_r3423446839.
`GetIdentifiersImpl` unconditionally sliced the 64-entry
`IdentifierLengthCounts` table even for uniform distributions, which the
API documents as having no `max_length` limit. Requesting a uniform
distribution with `max_length > 64` therefore tripped an out-of-bounds
slice assertion. Only compute the table slice on the non-uniform path,
where `max_length <= 64` is already enforced.
Add `IdentifierByteSumStableAcrossSeeds`, which exercises this path (a
uniform request up to length 200) and checks the core invariant that the
total identifier byte count is independent of the random seed across a
spread of parameters.
Assisted-by: Claude Code
---------
Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com>
`ErrorRecoveryBuffer::Apply` flagged each inserted token via the
pre-insertion index of the token it was inserted before. Once any
earlier insertion had been applied, that index no longer matches the
token's position in the merged list: with two closers inserted at the
same point (`{((}`), the second recovery token was left unflagged, and
with insertions at two separate points (`{(} {(}`), a real token was
flagged in the second one's place.
`IsRecoveryToken` now reports exactly the inserted tokens. This will
matter for `carbon format`: a recovery token's text exists in no source
byte range, so a minimal-edit model must know not to anchor edits on it.
Assisted-by: Claude Code
Carbon currently requires a comment to be the only non-whitespace on its
line. A `//` comment that follows other content on a line, called a
_trailing comment_, is a lexer error. This proposal removes that
restriction, allowing a comment to follow other content on a line.
Everything else about comments is unchanged: a comment still begins with
`//`, still requires whitespace after the `//`, and still runs to the
end of the line. Carbon continues to provide only line comments; no
block or intra-line comments are added.
Three observations motivate the change. First, trailing comments are
well suited to short _annotations_ attached to a specific entity or
value on a line. Second, the lexer design now makes it trivial to lex
trailing comments, and in fact requires extra logic and potentially cost
to reject them. Third, C++ code routinely uses trailing comments, so
allowing them lets Carbon carry the layout of migrated code over
directly, rather than reworking each comment to read well in a different
structure.
Implementation notes (beyond the proposal's design):
Keeping trailing comments cheap to lex required a few supporting
changes, all of which keep the cost off the lexer's hot path:
- The lexer already dispatches `//` to comment lexing wherever it
appears, so classifying a comment as trailing is a single O(1) check of
whether the `//` is the line's first non-whitespace (`start + indent`).
The hot comment path is otherwise unchanged.
- That check relies on each line's recorded indentation being its real
leading whitespace. Multi-line string literals previously recorded the
column where the literal opened for the lines they span; they now record
the true (closing-delimiter) indentation instead.
- Parser error recovery (`SkipPastLikelyEnd`) had relied on that
opening-column indentation to keep tokens following a multi-line string
literal attached to the same construct. It now reconstructs that
relationship directly by consulting the line on which the literal
opened, including when other tokens follow the closing delimiter (such
as `''' + "more"`). This is on the cold recovery path.
- `CommentData` records the trailing bit in the high bit of its length
field, keeping it at 8 bytes.
Assisted-by: Claude Code
Instead of maintaining a stack of pending subpatterns which might or
might not contain expressions, we mark non-nesting regions during
pattern handling that might contain an expression. The implementation
remains largely the same; the difference is that callers are expected to
end a pending expression region as soon as possible, rather than wait
for the end of the subpattern. This makes it possible to emit
non-pattern insts during pattern handling, without the risk that they
will get caught in a pending expression region further up the stack.
When we find a named constraint during identity, we recurse into it. The
specific args of the named constraint may contain references to `.Self`
which can then make `.Self` appear inside the named constraint, which
was making us replace `.Self` at multiple levels and incorrectly. The
first specific argument replaces `Self` in the named constraint, and we
pass in the self-type of the identify operation. This may contain
`.Self` and we should _not_ be replacing the `.Self` references with the
self-type that they are contained within. This led to infinite cycles.
In the meantime, we have made the toolchain reject any ambiguous `.Self`
from being constructed. So we know there is only one value of `.Self`
around in a facet type.
So now we replace `.Self` only in the top level facet type during
identity. That means replacing `.Self` in the specifics of the named
constraints that we recurse into. But we do _not_ replace `.Self`
anymore inside those named constraints. This resolves the infinite loop.
At the same time, when we are identifying an `impls` constraint from
earlier in the same facet type, like `C impls Z(.Self)` we are
identifying with a self-type of `C`. We want the output to use `C` as
the self-type since we should get back an identified facet type that
says `C impls Z(.Self)`. But we do _not_ want to replace the `.Self`
there since we're inside a facet type and the `.Self` does not refer to
`C`. So we parameterize `TryToIdentifyFacetType` to not replace `.Self`
when identifying an `impls` constraint from the `where_stack()`. This
resolves a large number of `fail_todo_` tests.
Impl lookup identifies the types of facets in the query, replacing
`.Self` in each type with the facet. This allows references using the
facet from outside the facet type to match similar structures inside the
facet type.
However replacing `.Self` with the facet can re-evaluate symbolic impl
lookups inside the facet type. These can perform deduction in generic
`final impl`s, which can attempt to convert the facet. Convert does an
impl lookup with that facet in the query, which causes us to form an
infinite recursion cycle.
Add tests that caused such a cycle, to demonstrate we no longer crash.
Some of these tests fail impl lookups using concrete values in the query
that match values from a `final impl`, with TODOs to address them.
This proposal removes the `:!` syntax for generics and templates in
favor of keywords (`generic`, `template`, `runtime`) and contextual
defaults for phase. It also replaces `:?` with `fwd` and introduces
`exttype` for extended types.
Assisted-by: Antigravity with Gemini, and Claude
---------
Co-authored-by: Geoff Romer <gromer@google.com>
A lambda introducer (`fn` in expression context) with no parameters and
no body -- for example `(fn)`, where `)` immediately follows `fn` --
left `HandleLambdaAfterParams` calling `ReturnErrorOnState()` without
ever emitting a `Lambda` node. The orphaned `LambdaIntroducer` leaf was
then left where an expression was required, so `Parse` produced a tree
that failed its own verification and aborted via `CARBON_FATAL`.
Recover the way `HandleLambdaBody` already does for a missing body after
a return type: emit a placeholder `InvalidParse` body and finish a
complete `Lambda` node, so the lambda stays a valid expression and the
surrounding construct (here a `ParenExpr`) extracts cleanly.
Found by fuzzing.
Assisted-by: Claude Code
One API fix: BumpPtrAllocator no longer tracks the amount of memory it's
handed out separately from the amount of memory it has allocated from
the system.
Assisted-by: Gemini via Antigravity
`IsRawIdentifier` checked `token_text.starts_with("r#")` and then read
`token_text[2]`, but `starts_with` only guarantees a length of two. An
`r` identifier immediately followed by `#` at the end of the source --
so the token text is exactly `r#` -- made the `token_text[2]` read run
off the end. Guard on the length first.
Found by fuzzing. The read is reached only via `GetTokenText`, so the
parser fuzzer, which does not request token text, never hit it.
Assisted-by: Claude Code
Our development instructions now recommend installing a number of
binaries through `cargo`. Updating these binaries has to be done by
hand. So we can provide a script that users can use to update them
easily and regularly.
Adds `toolchain/benchmarking/prelude_benchmark.cpp`, which measures the
time to compile the Core prelude and reports the most interesting SemIR
memory statistics as benchmark counters.
The prelude is compiled by checking a file that imports it: the implicit
prelude import causes the check phase to lex, parse, and check the full
set of prelude files, so this is a direct measure of prelude compilation
cost. Four input variations exercise increasing amounts of the prelude:
an empty file, a minimal single-type use, an operator-heavy file that
hits many impls, and a compact file that pulls in a wide swath of the
prelude.
Memory usage is queried directly: `Driver::set_mem_usage` takes a
`MemUsage` that a compile merges each file's usage into; the benchmark
passes one, compiles, and sums the entries by label. A compilation unit
collects into its own `MemUsage` whenever usage is dumped or a sink is
provided (decided in `SetMultiUnitCache`); after a file is done it dumps
that `MemUsage` per-file as before and, if a sink was provided, merges
into it via a new `MemUsage::Add(const MemUsage&)` overload. `MemUsage`
also exposes its entries via a public `Entry` type and an `entries()`
accessor.
Also extends `scripts/bench_runner.py` to (1) treat Mem-prefixed
counters as cost metrics (smaller is better) and (2) tolerate metrics
that aren't reported by every benchmark in a binary.
Assisted-by: Claude Code
---------
Co-authored-by: Christopher Di Bella <cjdb.ns@gmail.com>
Instead of traversing the entire pattern block looking for
`VarPattern`s, we keep track of them on creation, and then build
`VarStorage`s directly from that list.
This also gets rid of the global `var_storage_map`, and instead keep
that information narrowly scoped to each full-pattern, and consume it in
a single linear traversal instead of with random-access lookups. To
enable that, this fixes a parse bug where nested `var` patterns were
getting diagnosed but not marked as errors.
Rumdl already appears to have _significantly_ fewer bugs than prettier,
and a solid LSP for editor integration.
The tool is: https://github.com/rvben/rumdl/
I've separated out the change across three commits for easier review.
The configuration tries to match the existing formatting, the changes to
the all the files are to correct issues found by the new tool.
Assisted-by: Antigravity with Gemini
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Support multi-file compilation, and in particular imports of files from
the prelude, in `carbon language_server`.
In order to properly interface with `CompileDriver`, also switch over to
building a proper VFS from the documents we're given.
Assisted-by: Gemini via Antigravity
In #7378 we forbid writing `where` on the RHS of another `where`
expression. However it's still possible to inject a `where` expression
through eval, using an `alias` or an `eval fn`. We address this by
looking in the constant value of constraints of a `WhereExpr` (which
sees the outcome of eval) and searching for a nested `where` there. We
can determine a facet type was written with a nested `where` by seeing
that it has some non-extend constraint.
---------
Co-authored-by: David Blaikie <dblaikie@gmail.com>
If the deduction fails while forming the parameter type, ensure that we
print an actual diagnostic saying what went wrong.
And always ensure that an invalid array bounds error points at a
location. The `inst_id` given to `EvalConstantInst` always has a
location, but the `bounds_id` instruction inside it may be canonical
when it's coming from inside a larger type. So when it is, fall back to
using the location of the whole array inst.
These files aren't exactly written in Carbon, but rather in some
meta-language with file splits and semantically meaningful comments, and
in any case getting red squiggles for expected errors is distracting and
largely unhelpful.
(We *could* teach the LSP to run the test and produce errors if the test
doesn't match its expectations, but it's not clear that that would be
helpful in practice either.)
This disallows building a facet type that contains another facet type
with non-extend constraints in it. Which in turn prevents the
possibility of introducing a different `.Self` into a facet type.
Eval can still insert a facet type with non-extend constraints, as we
only prevent it for `where` being written into the facet type. There is
a TODO in handle_where.cpp for this and some tests in
toolchain/check/testdata/facet/nested_facet_types_from_eval.carbon
In `PerformActionHelper`, use the unqualified type for lookup.
In `PerformInstanceBinding`, propagate qualifiers to the unbound element
type's class type when doing the `ConvertToValueOrRefOfType` conversion,
and to the element type when forming the `ClassElementAccess` instr
(except for `partial`, which is only used if the member being accessed
is `base`).
In handle_operator.cpp, prevent assignment to a reference to a const
type.
Move the existing derived->base conversion earlier in
`PerformBuiltinConversion`, into the block that handles qualifier
conversions. This allows, for example, converting from `partial Derived`
to `partial Base` -- see the tests in
`toolchain/check/testdata/class/inheritance/derived_to_base.carbon`.
The function to set the visible declarations with a given name
overwrites any existing declarations imported from an AST file, so we
need to avoid calling that for declaration contexts whose names are
managed by Clang to avoid clobbering names imported from modules.
Assisted-by: Gemini via Antigravity
Dana Jansens
Geoff Romer
Nicholas Bishop
Özgür T. ÖnsoyandDana Jansens
Lucile Rose Nihlen
Chandler Carruth
Richard Smith
dependabot[bot]