Commit Graph
26 Commits
Author SHA1 Message Date
Chandler Carruth ff0d3ff107 Add another C++ keyword to avoid: enum (#8069)
I've seen at least one failure on a PR because of randomly producing
this as an identifier.
2026-10-31 22:55:52 +00:00
Chandler CarruthandRichard Smith 39e69f0b4f Address a subtle instability in source_gen and shard its tests (#8052)
Previously, it was possible for rounding to result in small percentage
constructs to be dropped entirely, breaking invariants. For example,
when generating >100 entities, a 2% share should reliably be >=2.

Some parts of source gen already propagated remainders to avoid exactly
this issue, but we weren't propagating them everywhere, notably across
depths and different definitions in plans.

We fix this by introducing a helper for this more robust form of percent
scaling that carries remainders across the slices. It also seeds this
rounding from Phi across [0,100) to keep the edge cases uniformly
distributed without causing instability from the source generator's
seed. The seeded remainder doesn't impact providing the invariant, so
the fix should be durable.

Once this is done, we can shard the tests without the separate processes
allowing the above failure mode to occur.

Assisted-by: Antigravity with Gemini

---------

Co-authored-by: Richard Smith <richard@metafoo.co.uk>
2026-10-30 05:04:57 +00:00
Chandler CarruthandGeoff Romer 33863ed07c Enhance benchmark source to have more call expressions (#8029)
This adds free function declarations to the top of the source and
adjusts many of the synthesized expressions in the benchmark source to
emit calls to them. This gets the distribution of expressions in our
generated definitions to be significantly more realistic compared to
existing C++ function bodies here and in LLVM.

This still doesn't get us to start calling more methods, have class
types in the signatures of the calls, or have more interesting call
graphs, but seems like a reasonable first step and still an improvement
in the overall fidelity of our benchmarks.

Illustrative generated Carbon:

```carbon
fn Abcde(x: i32, y: i64*) -> (bool, i64);
fn Fghij(p: i32*) -> i32;
// ...
fn MyClass.Method(self, p: i32) -> i32 {
  var acc64: i64 = 0;
  var acc: i32 = acc64 as i32;
  acc = acc + self.Checksum() + p + 1;
  let (abc: bool, def: i64) = Abcde(0, &acc64);
  acc = (acc + (if abc then 1 else 0) + (def as i32)) + Fghij(&acc) * 2;
  return 0;
}
```

Assisted-by: Claude Code and Antigravity with Gemini

---------

Co-authored-by: Geoff Romer <gromer@google.com>
2026-10-27 06:34:44 +00:00
Chandler CarruthandGeoff Romer 600fd55ba7 Add a realistic header compile benchmark (#8023)
Add `BM_CompileApiFileRealisticHeader`, which models a typical C++
header. Each class defines 2 getters, 1 predicate, 2 forwarders, and 2
function definitions inline, and declares 2 methods along with the
private methods that the forwarders call. The dense declarations
benchmarks keep modeling a declaration-heavy API file.

The counts come from classifying, with libclang, the member functions
defined in class definitions in the hand-written headers that the
toolchain includes: 1,612 in the Carbon toolchain's headers, 10,406 in
LLVM's, and 13,366 in Clang's. Classified the same way, the generated
C++ defines 69% of its member functions inline in the class definition,
against 61-70% in the real headers. 78% of its inline definitions are a
single statement, against 72-82%, and 21% return the result of a member
call, against 19-21%. Only 2% have control flow, against 7-13%, because
most inline function definitions have only the three statements that
every plan-based definition has.

Assisted-by: Claude Code and Antigravity with Gemini

---------

Co-authored-by: Geoff Romer <gromer@google.com>
2026-10-26 20:20:06 +00:00
Chandler CarruthandGeoff Romer 789f582eb1 Extend plan-based definition synthesis with class-typed locals and destructuring (#8016)
This re-lands #8001 (reverted in #8009 after accidentally merging with
earlier unmerged PRs in the stack).

Build on the plan-based emission to make function definitions cover a
more representative breadth of the language: class-typed local
variables, and local variables declared with destructuring. This
requires some complexity to initialize and manage the class type
variables, but ultimately should make all of our compile time benchmark
data more representative by covering a more realistic breadth of the
language surface.

Claude's rough sampling of Carbon, Clang, and LLVM source suggests class
types range from 30% to 80% of the locals, and structured bindings range
from 0.2% to 2%. When synthesizing, we target approximately 50% of
declarations as class types and 2% as destructurings as we anticipate
destructuring to be of growing popularity and so want to ensure we're
measuring any overhead there.

Assisted-by: Claude Code and Antigravity with Gemini

Co-authored-by: Geoff Romer <gromer@google.com>
2026-10-23 15:58:42 +00:00
Chandler CarruthandDana Jansens c50e21695f Allow detecting benchmark dry runs and reduce benchmark test costs (#8013)
We use the dry-run mode for running benchmarks in a test. This is
important, but doesn't need to use large batches of files. Doing so
would cause the compilation benchmark to become excessively slow,
especially on GitHub runners.

Sadly, Google Benchmark doesn't expose its dry-run mode, but we already
have our own `main` function that injects custom logic for our
benchmarks. Extend that to detect the dry run flags and environment
variables so that we can customize the benchmark behavior in that mode.
Then we use that detection to reduce the batch size.

Also cleans up the counters computation to both be a bit better
organized (I think), and to avoid keeping the tokenized buffers we build
to count tokens around during the timed running of the benchmark.

Also cleans up a bug in some clamping code spotted while fixing this.

Assisted-by: Antigravity with Gemini

---------

Co-authored-by: Dana Jansens <danakj@orodu.net>
2026-10-22 18:00:21 +00:00
Chandler Carruth 5aa7f35a1f Add another C++ keyword we need to avoid (#8015)
The enhancements to benchmarking means we can (when _very_ unlucky) hit
this now.
2026-10-22 13:22:12 +00:00
Chandler Carruth 57ba07a978 Switch inline function definition synthesis to use plans (#8000)
Now that we have a robust plan-based layer for generating function
definitions, we can use that when generating inline function definitions
rather than the very ad-hoc previous inline function definition
synthesis. This also requires switching some of the more ad-hoc
heuristics for governing the size of inline function definitions to more
robust histograms representing the desired distributions.

Some rough analysis of existing C++ code in Clang, Carbon, and LLVM
suggests that multi-statement inline function definitions and
out-of-line
definitions in implementation files have compatible distributions of
shape and statement kinds when their sizes overlap, and so we don't end
up needing to overly customize that part of the plan generation.

Getters, predicates, and forwarding functions continue to be separate
constructs from multi-statement function definitions. I looked at trying
to unify them, but that adds significantly more complexity. For now, the
plan-based synthesis is only used for definitions where we have >1
statements (happens to be >=3 because of the structure of these
definitions).

Assisted-by: Claude Code and Antigravity with Gemini
2026-10-22 07:25:01 +00:00
Chandler Carruth ebc1557345 Switch the field types to a separate type pool (#7999)
Fields have constraints on what types can be used distinct from other
uses. Previously, they shared a pool with inline function return types
when function definitions are included, the more general type pool used
for function declarations otherwise. However, fields have a unique
constraint: they cannot have the type currently being defined.

Sharing a type pool for both function returns and fields either
requires us to not use the last class in the pool (including in any
function returns), or it requires us to have a large number of function
returns inside the last class as every reference to that class can only
come from function returns inside the class. So we end up either with
an artificial constraint on return types, or being unable to have
a small number of functions per class. The latter is a pattern we want
to be able to use though, as it comes up frequently in implementation
files.

The other problem is that as we allow inline function definitions of
increasing complexity (similar to the out-of-line definitions from
plans in #7998), we end up needing different mechanisms for ensuring
return types (when class types) are spelled out a seed-independent
number of times compared to fields. Today, both inline return types
and field types are spelled out twice when class types. But inline
return types will need a more complex way of ensuring seed invariant
references once they use plan-based definitions as well.

This PR splits the shared pool into distinct pools, and does so
regardless of whether there are definitions being emitted. This in turn
allows us to simplify some things. We can omit the last class from the
pool entirely, and switch the pool to follow `declared_types_weight` in
fields. Previously it had deviated sharply from that weighting.

Assisted-by: Claude Code and Antigravity with Gemini
2026-10-22 02:07:50 +00:00
Chandler Carruthandjosh11b 9319bb0369 Add more complex function definition generation and an impl-file benchmark (#7998)
Previously, we only had the dense API declarations benchmark. This adds
a benchmark that aims to emulate what a realistic implementation file in
Carbon would look like, based on what `.cpp` files look like in Carbon,
Clang, and LLVM. It's still a little artificial in order to provide
stability, but it should give us coverage of compile time performance of
statements and function definitions at more realistic scales.

In order to generate more interesting and longer function definitions,
this introduces a plan-based approach to guide how each feature of a
definition is produced. Initially, it identifies these as roughly:

- local variables
- updates to variables
- `if`, `else`, `while` blocks
- "guard" statements -- conditions with a return, break, or continue
- read of a parameter

The plan also manages the nesting depth of these constructs.

Because the plan establishes how many of these constructs exist in a
function definition, and at what nesting depth, the emission can
randomize the order and arrangement without changing the size of the
function. This follows the pattern in all our benchmark source
generation of getting byte-identical stability while avoiding
predictable structures. Technically, I think the current approach goes a
bit too far because function definitions I suspect are somewhat
predictable in their structure in practice. But a conservative benchmark
I think is a good starting point. =]

The target distributions for definition sizes, maximum depths, and
statement mixes come from a very rough Claude-powered analysis of 10,697
function definitions in 533 Carbon toolchain, LLVM, and Clang
implementation files. These were analyzed and classified by Claude using
a one-off Python script and libclang. I haven't reviewed any of the
analysis carefully, the resulting distributions look plausible enough as
a starting point. Comparing real code against generated C++ from the
plans, each number per-definition average:

| statements | definitions | max depth   | ifs         | loops       |
|------------|-------------|-------------|-------------|-------------|
| 7-9        | 10.9 / 11%  | 1.25 / 1.23 | 1.74 / 1.86 | 0.23 / 0.15 |
| 14-19      | 7.3 / 7%    | 1.89 / 1.88 | 3.53 / 3.71 | 0.61 / 0.42 |
| 28-39      | 3.3 / 3%    | 2.76 / 2.75 | 7.75 / 8.46 | 1.26 / 1.27 |
| 60+        | 1.5 / 2%    | 4.03 / 3.80 | 25.0 / 27.1 | 3.72 / 4.04 |

To maintain stability, every definition has an accumulator, a statement
reading its parameters, and a `return`, so real definitions of one to
three statements become three.

Note that updates are currently also used instead of calls to other
functions -- a later PR will switch those to calls so that we form more
plausible call graphs. In general, subsequent PRs will refine and expand
the plan to generate increasingly more realistic function definitions.

There are probably lots of improvements that could be made here, both to
generate higher quality definitions and to improve the generation logic
to be more efficient, etc. But already this has allowed _much_ more
extensive benchmarking of Carbon's compile-time performance so it seems
likely a reasonable increment.

Assisted-by: Claude Code and Antigravity with Gemini

---------

Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com>
2026-10-21 23:02:26 +00:00
Chandler Carruth 70c696e782 Revert "Extend plan-based definition synthesis with class-typed locals and destructuring" (#8009)
Reverts carbon-language/carbon-lang#8001

This depended on 3 unmerged PRs.
2026-10-21 18:24:43 +00:00
Chandler CarruthandGeoff Romer c3877cf701 Extend plan-based definition synthesis with class-typed locals and destructuring (#8001)
Build on the plan-based emission to make function definitions cover a
more representative breadth of the language: class-typed local
variables, and local variables declared with destructuring. This
requires some complexity to initialize and manage the class type
variables, but ultimately should make all of our compile time benchmark
data more representative by covering a more realistic breadth of the
language surface.

Claude's rough sampling of Carbon, Clang, and LLVM source suggests class
types range from 30% to 80% of the locals, and structured bindings range
from 0.2% to 2%. When synthesizing, we target approximately 50% of
declarations as class types and 2% as destructurings as we anticipate
destructuring to be of growing popularity and so want to ensure we're
measuring any overhead there.

Assisted-by: Claude Code and Antigravity with Gemini

---------

Co-authored-by: Geoff Romer <gromer@google.com>
2026-10-21 17:43:23 +00:00
Chandler Carruth 3f5c187557 Generate inline function definitions with bodies (#7918)
Add four kinds of inline definition to generated classes, each with its
own count in `ClassParams`. Three are among the most common kinds of
inline definition in the Carbon, LLVM, and Clang headers: methods that
return a field, test a field, or forward their parameters to a private
method. The fourth computes over its parameters, and stands in for
longer bodies. Here is one class, without the comment before each
declaration:

    class ADGjE {
      fn Zotm() -> bool;

      fn fx(y: ADGjE) -> ADGjE {
        var acc: i32 = 0;
        acc = acc + y.Checksum();
        var aIx: i32 = 1;
        var aTc: i32 = aIx + 1;
        aIx = aTc;
        return ADGjE.Make();
      }

      fn GJ2m(self, N: c125y) -> ADGjE;

      fn Mg(self) -> i64 { return self.zGrc; }

      fn UE(self) -> bool { return self.LBYM == 0; }

      fn sU(self, z: i64, J: i32) -> ADGjE {
        return self.sUImpl(z, J);
      }

      private fn FYct(self) -> ADGjE;

      private fn sUImpl(self, z: i64, J: i32) -> ADGjE;

      fn Make() -> ADGjE {
return {.tag = 0, .pV0t = 0, .zxcQ = c125y.Make(), .zGrc = 0, .LBYM =
0};
      }

      fn Checksum(self) -> i32 {
        return self.tag + 1;
      }

      private var tag: i32;
      private var pV0t: i32;
      private var zxcQ: c125y;
      private var zGrc: i64;
      private var LBYM: i32;
    }

A computation reads each parameter with a consumer template for its
type, which calls `Checksum` for a class type, and returns a literal or
the class's `Make`. Getters and predicates read fields of fixed types,
because Carbon can't copy a class value that doesn't implement
`Core.Copy`.

The dense declarations benchmarks now have a computation, a getter, and
a forwarder in each class, so their results aren't comparable with
earlier runs.

A file still has the same size for every seed. Type uses and names are
split into pools whose entries are spelled the same number of times, and
getter and predicate field types cycle through the eligible types. Each
computation gets one parameter beyond its random count so that each
class has parameters that can use its own type. The line estimates now
count the slot that `self` takes when a method's parameters wrap, and
round to the nearest whole class.

Assisted-by: Claude Code
2026-10-14 20:52:05 +00:00
Chandler Carruth c578085840 Add determinism checks for generated source size and fix a crash (#7917)
Benchmarks are only comparable if the source generator produces the same
number of lines and bytes for a given configuration with any seed. Test
this across many seeds and class shapes, and check it in the compile
benchmark for the files it generates. Also check that generation emits
every name in its name pools, which the byte count depends on.

With field-heavy classes, generation could crash depending on the
shuffle. A class becomes a valid type only after its field types are
chosen, so references to the last class can only go on its own return
and parameter types. When the shuffled pool left more references to it
than that, `GetValidTypeName` found no valid type and failed its check.
Cap the references to each class at its number of functions and methods,
each of which has a return type that can use it, and replace the removed
references with fixed types so the byte count stays seed-independent.

Assisted-by: Claude Code
2026-10-09 16:58:56 +00:00
Chandler Carruth 46b750bf90 Compile the generated C++ in source_gen_test (#7905)
`source_gen_test` checked the line count of the generated C++ but never
compiled it. Compile it with `ClangRunner`, the same way the C++ compile
benchmarks do, which resolves the TODO.

Also require both languages to compile with no diagnostic output at all,
rather than only without errors. A warning in generated code would add
diagnostic emission to what the compile benchmarks measure.

Assisted-by: Claude Code
2026-10-06 17:14:54 +00:00
Chandler Carruth 632a81fcb6 Keep benchmark values passed to DoNotOptimize in registers (#7870)
Clang implements the `"+r,m"` constraint in `benchmark::DoNotOptimize`
by always choosing memory, so each call stores the value to the stack
and loads it back. Most of our benchmarks call it on a loop counter or
another value carried to the next iteration, which puts that store and
reload on the loop's critical path.

On an Apple M1, the cost of that round trip depends on which register
the compiler uses to address the stack slot, and unrelated code changes
move it. Changing only that register from `x29` to `sp`, with the same
address, made `BM_SetLookupHitPtr<Set<int>>` 41.5% to 49.1% slower.

Add `Carbon::Testing::DoNotOptimize` in
`//testing/base:benchmark_helpers`, and switch every benchmark to it. It
only accepts types it can keep in registers: integers, enums, and
pointers go into a `"+r"` constraint with a `"memory"` clobber, and
containers pass their `data()` pointer to that form, so the compiler
must assume the call reads and writes their contents. Any other type is
a compile error. The container form doesn't block optimizing based on a
container's size; passing the container's address does.

Benchmarks that pass a loop counter or carried value no longer measure a
store and reload per iteration, so their numbers aren't comparable with
earlier runs. For example, the hashing latency benchmarks no longer
include one in each hash's latency.

Assisted-by: Claude Code
2026-10-02 00:15:21 +00:00
Chandler Carruth 0bdb59df45 Try to reduce timeouts of compile_benchmark_test (#7836)
The '...Exec...' version of this benchmark runs subprocesses in a tight
benchmarking loop which seems like the likely culprit for timeouts we're
seeing on GitHub. Reduce testing to only a single one of those
benchmarks to hopefully reduce the frequency.
2026-09-25 20:11:25 +00:00
Chandler Carruth d037848a96 Replace hashtable ForEach callback with range-based iteration (#7806)
Replaces the callback-based `ForEach` methods on `RawHashtable`, `Map`,
and
`Set` with a range object supporting range-for loops, structured
bindings, and
the standard range concepts.

- Adds `.entries()` on `Map`, `Set`, and `RawHashtable`, returning a
range that
  models `std::ranges::forward_range` and `std::ranges::common_range`.
  Obtaining one is an explicit call rather than `begin()`/`end()` on the
container, as scanning a whole table is costly and shouldn't be hidden.
- Iterating a `Map` yields a `std::pair` of key and value references,
which
fits in two registers and is returned without being materialized in
memory.
- `Map::Range` and `Set::Range` are aliases of the raw hashtable's range
rather
than wrappers around it. The raw iterator produces the user-facing
reference
itself -- a `KeyT&` for a set, a pair of references for a map -- picked
by
`StorageEntry`, which is already specialized on whether there is a value
  type. That leaves one iterator to reason about instead of three.
- Deletes the rvalue `.entries()` overloads on the owning containers, as
a
  range built from a temporary table would dangle. Views don't own their
  storage, so the operation remains available on them.
- In release builds, the walk over the groups is a single induction
variable: a
  negative byte offset counting up to zero, anchored at the ends of the
metadata and entry arrays. Both arrays are then reached by indexed
addressing
off a base that stays put, and the entry pointer is formed only once a
group
  with a present entry has been found.
- In debug builds, the range hashes the table's metadata when it is
built and
re-checks that hash when it is destroyed, catching mutation of the table
while a range is live. It also picks a random starting group and a
random odd
group stride, which varies the traversal order between ranges while
still
visiting every group exactly once. That entropy is drawn when the range
is
built rather than in `begin()`, so `begin()` stays a pure function of
the
  range and the multi-pass guarantee holds.
- Removes `ForEachEntry` and all of its callers.

Measured against the iteration benchmark added in its own commit, a
traversal is at or ahead of what the callback compiled to across nearly
the
whole size range. The largest tables spend 3-5% fewer cycles, small
`Set`s as
much as 24% fewer, and instruction counts stay within about 1%. What
remains
behind is a handful of mid-sized `Map`s by up to 1%, and `Set` at 65536,
which
sits at exactly half its load factor, by 2%.

Both revisions were built with `-c opt --copt=-march=x86-64-v3` and
compared
with:

```
./scripts/bench_runner.py --exp_benchmark=... --base_benchmark=... \
    --benchmark_args=--benchmark_perf_counters=INSTRUCTIONS,CYCLES \
    --benchmark_args='--benchmark_filter=(Set|Map)Iterate<(Set|Map)<' \
    --extra_metrics_filter='(INSTRUCTIONS|CYCLES)'
```

Trimmed below to the primary integer configurations and to the two
counters;
the pointer- and string-keyed configurations follow the same pattern.

```
 Benchmark                             ┃           CYCLES            ┃        INSTRUCTIONS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 BM_MapIterate<Map<int, int>>/1....... │ 👍  -6.032%      p=1.14e-05 │      ??          p=0.752
                             baseline: │     12.06      ±   1.520%   │     64         ±   3.125%
                           experiment: │     11.33      ±   2.765%   │     65.5       ±   3.817%
                                       │                             │
 BM_MapIterate<Map<int, int>>/2....... │      ??          p=0.155    │      ??          p=0.343
                             baseline: │      7.587     ±   1.285%   │     41         ±   0.000%
                           experiment: │      7.652     ±   0.865%   │     41         ±   2.439%
                                       │                             │
 BM_MapIterate<Map<int, int>>/3....... │      ??          p=0.343    │ 👍  -1.020%      p=0.0039
                             baseline: │      6.663     ±   4.260%   │     32.67      ±   2.041%
                           experiment: │      6.368     ±  12.224%   │     32.33      ±   2.062%
                                       │                             │
 BM_MapIterate<Map<int, int>>/4....... │      ??          p=0.343    │ 👍  -1.786%      p=0.0297
                             baseline: │      6.091     ±  15.470%   │     28         ±   3.571%
                           experiment: │      5.957     ±   8.932%   │     27.5       ±   3.636%
                                       │                             │
 BM_MapIterate<Map<int, int>>/8....... │      ??          p=0.323    │ 👍  -1.220%      p=0.000148
                             baseline: │      4.845     ±   0.800%   │     20.5       ±   0.000%
                           experiment: │      4.814     ±   3.585%   │     20.25      ±   0.000%
                                       │                             │
 BM_MapIterate<Map<int, int>>/16...... │ 👍  -2.195%      p=0.00908  │ 👍   0.769%      p=6.58e-06
                             baseline: │      4.312     ±   0.187%   │     16.25      ±   0.000%
                           experiment: │      4.218     ±   2.368%   │     16.13      ±   0.000%
                                       │                             │
 BM_MapIterate<Map<int, int>>/32...... │      ??          p=0.236    │ 👍   0.442%      p=9.53e-06
                             baseline: │      4.051     ±   1.084%   │     14.13      ±   0.000%
                           experiment: │      4.063     ±   0.737%   │     14.06      ±   0.000%
                                       │                             │
 BM_MapIterate<Map<int, int>>/64...... │      ??          p=0.693    │ 👎   0.227%      p=4.52e-06
                             baseline: │      4.021     ±   0.239%   │     13.75      ±   0.000%
                           experiment: │      4.019     ±   0.417%   │     13.78      ±   0.000%
                                       │                             │
 BM_MapIterate<Map<int, int>>/256..... │ 👍   0.360%      p=0.00119  │ 👎   0.754%      p=1.37e-05
                             baseline: │      3.996     ±   0.173%   │     13.47      ±   0.000%
                           experiment: │      3.982     ±   0.272%   │     13.57      ±   0.000%
                                       │                             │
 BM_MapIterate<Map<int, int>>/4096.... │ 👍   0.581%      p=1.96e-05 │ 👎   0.923%      p=1.96e-05
                             baseline: │      4.005     ±   0.816%   │     13.38      ±   0.000%
                           experiment: │      3.981     ±   0.192%   │     13.5       ±   0.000%
                                       │                             │
 BM_MapIterate<Map<int, int>>/65536... │ 👍  -4.957%      p=1.14e-05 │ 👎   0.934%      p=1.14e-05
                             baseline: │      5.307     ±   0.501%   │     13.38      ±   0.000%
                           experiment: │      5.044     ±   1.746%   │     13.5       ±   0.000%
                                       │                             │
 BM_MapIterate<Map<int, int>>/1048576. │ 👍  -3.947%      p=9.09e-05 │ 👎   0.935%      p=3.3e-05
                             baseline: │      6.074     ±   0.807%   │     13.38      ±   0.000%
                           experiment: │      5.834     ±   2.159%   │     13.5       ±   0.000%
                                       │                             │
 BM_MapIterate<Map<int, int>>/16777216 │      ??          p=0.155    │ 👎   0.935%      p=2.11e-05
                             baseline: │      5.082     ±   3.650%   │     13.38      ±   0.000%
                           experiment: │      5.012     ±   1.316%   │     13.5       ±   0.000%
                                       │                             │
 BM_MapIterate<Map<int, int>>/56...... │ 👎   0.825%      p=0.0268   │ 👍   0.270%      p=1.14e-05
                             baseline: │      3.918     ±   0.501%   │     13.21      ±   0.000%
                           experiment: │      3.951     ±   0.342%   │     13.18      ±   0.000%
                                       │                             │
 BM_MapIterate<Map<int, int>>/224..... │ 👎   0.788%      p=0.000504 │ 👎   0.346%      p=1.64e-05
                             baseline: │      3.895     ±   0.111%   │     12.89      ±   0.000%
                           experiment: │      3.926     ±   0.285%   │     12.94      ±   0.000%
                                       │                             │
 BM_MapIterate<Map<int, int>>/3584.... │ 👎   1.028%      p=0.000148 │ 👎   0.545%      p=1.14e-05
                             baseline: │      3.913     ±   0.427%   │     12.79      ±   0.000%
                           experiment: │      3.954     ±   0.325%   │     12.86      ±   0.000%
                                       │                             │
 BM_MapIterate<Map<int, int>>/57344... │      ??          p=0.236    │ 👎   0.558%      p=2.55e-06
                             baseline: │      4.574     ±   0.721%   │     12.79      ±   0.000%
                           experiment: │      4.51      ±   3.709%   │     12.86      ±   0.000%
                                       │                             │
 BM_MapIterate<Map<int, int>>/917504.. │ 👍  -3.826%      p=6.58e-06 │ 👎   0.559%      p=2.33e-05
                             baseline: │      5.221     ±   0.507%   │     12.79      ±   0.000%
                           experiment: │      5.021     ±   0.556%   │     12.86      ±   0.000%
                                       │                             │
 BM_MapIterate<Map<int, int>>/14680064 │ 👍  -3.839%      p=1.37e-05 │ 👎   0.559%      p=3.31e-05
                             baseline: │      5.129     ±   1.194%   │     12.79      ±   0.000%
                           experiment: │      4.932     ±   1.475%   │     12.86      ±   0.000%
                                       │                             │

 Benchmark                        ┃           CYCLES            ┃        INSTRUCTIONS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 BM_SetIterate<Set<int>>/1....... │ 👍  -3.104%      p=0.000583 │      ??          p=0.206
                        baseline: │     11.2       ±   6.323%   │     60         ±   3.333%
                      experiment: │     10.85      ±   5.820%   │     61         ±   3.279%
                                  │                             │
 BM_SetIterate<Set<int>>/2....... │ 👍  -7.037%      p=0.0362   │      ??          p=0.155
                        baseline: │      7.086     ±  16.857%   │     37         ±   0.000%
                      experiment: │      6.587     ±   0.479%   │     36         ±   4.167%
                                  │                             │
 BM_SetIterate<Set<int>>/3....... │ 👎   1.400%      p=2.34e-05 │ 👍  -1.163%      p=0.00136
                        baseline: │      5.363     ±   0.463%   │     28.67      ±   2.326%
                      experiment: │      5.438     ±  32.763%   │     28.33      ±   1.176%
                                  │                             │
 BM_SetIterate<Set<int>>/4....... │      ??          p=0.968    │      ??          p=0.286
                        baseline: │      4.642     ±  32.751%   │     23.5       ±   2.128%
                      experiment: │      4.658     ±  38.416%   │     23.63      ±   3.704%
                                  │                             │
 BM_SetIterate<Set<int>>/8....... │ 👍 -23.823%      p=3.74e-06 │ 👍  -1.515%      p=5.52e-05
                        baseline: │      4.701     ±   6.589%   │     16.5       ±   0.000%
                      experiment: │      3.581     ±   7.790%   │     16.25      ±   0.000%
                                  │                             │
 BM_SetIterate<Set<int>>/16...... │ 👍  -4.502%      p=1.37e-05 │ 👍  -1.020%      p=3.31e-05
                        baseline: │      3.124     ±   0.585%   │     12.25      ±   0.000%
                      experiment: │      2.983     ±   0.625%   │     12.13      ±   0.000%
                                  │                             │
 BM_SetIterate<Set<int>>/32...... │ 👍  -4.032%      p=5.46e-06 │ 👍   0.617%      p=1.96e-05
                        baseline: │      2.957     ±   0.260%   │     10.13      ±   0.000%
                      experiment: │      2.838     ±   0.434%   │     10.06      ±   0.000%
                                  │                             │
 BM_SetIterate<Set<int>>/64...... │ 👍  -5.054%      p=4.52e-06 │ 👎   0.321%      p=1.37e-05
                        baseline: │      2.937     ±   0.301%   │      9.75      ±   0.000%
                      experiment: │      2.788     ±   1.143%   │      9.781     ±   0.000%
                                  │                             │
 BM_SetIterate<Set<int>>/256..... │ 👍  -5.325%      p=1.14e-05 │ 👎   1.073%      p=6.58e-06
                        baseline: │      2.916     ±   0.220%   │      9.469     ±   0.000%
                      experiment: │      2.761     ±   0.142%   │      9.57      ±   0.000%
                                  │                             │
 BM_SetIterate<Set<int>>/4096.... │ 👍  -4.865%      p=4.52e-06 │ 👎   1.317%      p=2.34e-05
                        baseline: │      2.921     ±   0.194%   │      9.381     ±   0.000%
                      experiment: │      2.779     ±   0.224%   │      9.504     ±   0.000%
                                  │                             │
 BM_SetIterate<Set<int>>/65536... │ 👎   1.961%      p=3.93e-05 │ 👎   1.332%      p=1.49e-05
                        baseline: │      4.015     ±   0.482%   │      9.375     ±   0.000%
                      experiment: │      4.094     ±   0.613%   │      9.5       ±   0.000%
                                  │                             │
 BM_SetIterate<Set<int>>/1048576. │ 👍  -4.843%      p=1.14e-05 │ 👎   1.333%      p=5.38e-06
                        baseline: │      5.239     ±   0.144%   │      9.375     ±   0.000%
                      experiment: │      4.986     ±   0.139%   │      9.5       ±   0.000%
                                  │                             │
 BM_SetIterate<Set<int>>/16777216 │ 👍   0.840%      p=0.0362   │ 👎   1.333%      p=2.52e-06
                        baseline: │      3.719     ±   1.420%   │      9.375     ±   0.000%
                      experiment: │      3.688     ±   1.308%   │      9.5       ±   0.000%
                                  │                             │
 BM_SetIterate<Set<int>>/56...... │ 👍  -2.857%      p=9.53e-06 │ 👍   0.388%      p=3.31e-05
                        baseline: │      2.942     ±   0.439%   │      9.214     ±   0.000%
                      experiment: │      2.858     ±   0.619%   │      9.179     ±   0.000%
                                  │                             │
 BM_SetIterate<Set<int>>/224..... │ 👍  -2.161%      p=2.34e-05 │ 👎   0.502%      p=4.52e-06
                        baseline: │      2.888     ±   0.347%   │      8.893     ±   0.000%
                      experiment: │      2.826     ±   0.450%   │      8.938     ±   0.000%
                                  │                             │
 BM_SetIterate<Set<int>>/3584.... │ 👍  -1.750%      p=6.58e-06 │ 👎   0.793%      p=2.34e-05
                        baseline: │      2.89      ±   0.261%   │      8.792     ±   0.000%
                      experiment: │      2.84      ±   0.411%   │      8.862     ±   0.000%
                                  │                             │
 BM_SetIterate<Set<int>>/57344... │      ??          p=0.502    │ 👎   0.812%      p=2.78e-05
                        baseline: │      3.684     ±   4.246%   │      8.786     ±   0.000%
                      experiment: │      3.644     ±   4.431%   │      8.857     ±   0.000%
                                  │                             │
 BM_SetIterate<Set<int>>/917504.. │ 👍  -2.629%      p=0.000148 │ 👎   0.813%      p=3.08e-06
                        baseline: │      4.372     ±   0.693%   │      8.786     ±   0.000%
                      experiment: │      4.257     ±   0.210%   │      8.857     ±   0.000%
                                  │                             │
 BM_SetIterate<Set<int>>/14680064 │ 👍  -2.927%      p=0.0219   │ 👎   0.813%      p=3.03e-06
                        baseline: │      4.154     ±   3.286%   │      8.786     ±   0.000%
                      experiment: │      4.032     ±   3.198%   │      8.857     ±   0.000%
                                  │                             │
```

Assisted-by: Antigravity with Opus
2026-09-18 20:19:46 +00:00
Lucile Rose Nihlen 625f2ca629 precompile and cache Carbon prelude (#7432)
Refactors the link driver to automatically compile and cache the carbon
prelude for use in linking.

Implements a `carbon_library` rule for compiling the Core library
dependencies in the examples.
2026-07-16 18:02:14 +00:00
Chandler Carruth 8aa14eab46 Introduce subprocess executing compile benchmarks (#7456)
This follows the pattern of our main compile benchmarks, but instead of
running the compile through a library API, it does so by running a
separate process. This lets us see the performance that we would expect
from `make` or another build system invoking the compiler, as opposed to
a minimal view from the library API.

Assisted-by: Claude Code
2026-07-07 20:17:39 +00:00
Chandler Carruth e1cf833c45 Fix ASan heap-use-after-free in SourceGenTest (#7465)
In `SourceGenTest.IdentifierByteSumStableAcrossSeeds`, the `first`
variable was storing `llvm::StringRef`s pointing to memory allocated by
a temporary `SourceGen` instance. This memory was freed at the end of
the loop iteration, leaving `first` with dangling references that were
accessed in subsequent iterations.

Fix this by storing `std::string` copies of the identifiers in `first`
to own the memory, and use `llvm::equal` to compare them.

Assisted-by: Antigravity with Gemini
2026-07-07 15:04:40 +00:00
Chandler Carruthandjosh11b 383cfbb023 Fix uniform identifier generation for lengths over 64 (#7459)
`GetIdentifiersImpl` unconditionally sliced the 64-entry
`IdentifierLengthCounts` table even for uniform distributions, which the
API documents as having no `max_length` limit. Requesting a uniform
distribution with `max_length > 64` therefore tripped an out-of-bounds
slice assertion. Only compute the table slice on the non-uniform path,
where `max_length <= 64` is already enforced.

Add `IdentifierByteSumStableAcrossSeeds`, which exercises this path (a
uniform request up to length 200) and checks the core invariant that the
total identifier byte count is independent of the random seed across a
spread of parameters.

Assisted-by: Claude Code

---------

Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com>
2026-07-07 06:52:57 +00:00
Chandler CarruthandChristopher Di Bella 361c832713 Add a prelude compilation benchmark (#7368)
Adds `toolchain/benchmarking/prelude_benchmark.cpp`, which measures the
time to compile the Core prelude and reports the most interesting SemIR
memory statistics as benchmark counters.

The prelude is compiled by checking a file that imports it: the implicit
prelude import causes the check phase to lex, parse, and check the full
set of prelude files, so this is a direct measure of prelude compilation
cost. Four input variations exercise increasing amounts of the prelude:
an empty file, a minimal single-type use, an operator-heavy file that
hits many impls, and a compact file that pulls in a wide swath of the
prelude.

Memory usage is queried directly: `Driver::set_mem_usage` takes a
`MemUsage` that a compile merges each file's usage into; the benchmark
passes one, compiles, and sums the entries by label. A compilation unit
collects into its own `MemUsage` whenever usage is dumped or a sink is
provided (decided in `SetMultiUnitCache`); after a file is done it dumps
that `MemUsage` per-file as before and, if a sink was provided, merges
into it via a new `MemUsage::Add(const MemUsage&)` overload. `MemUsage`
also exposes its entries via a public `Entry` type and an `entries()`
accessor.

Also extends `scripts/bench_runner.py` to (1) treat Mem-prefixed
counters as cost metrics (smaller is better) and (2) tolerate metrics
that aren't reported by every benchmark in a binary.

Assisted-by: Claude Code

---------

Co-authored-by: Christopher Di Bella <cjdb.ns@gmail.com>
2026-06-29 05:39:44 +00:00
Chandler Carruthandjosh11b 7871237c15 Move self to the explicit () parameter list (proposal #7016) (#7272)
Implements proposal #7016: `self` moves from the deduced implicit list
(`fn F[self: Self]()`) to the front of the explicit list. Its type may
be written explicitly (`fn F(self: Self)`) or omitted, in which case it
defaults to `Self` (`fn F(self)`, `fn F(ref self)`); `self` in the
implicit list is rejected.

Throughout checking, `self` is modeled as the first explicit parameter.
Because a method is just a function whose first parameter is `self`, it
can also be called as an ordinary function with the receiver passed
explicitly (`Type.M(obj, ...)`), not only as `obj.M(...)`. A new
`SemIR::CallArgParamPatterns` helper chooses the parameters matched
against the explicit arguments, excluding a leading `self` only when it
is supplied as a method-call receiver; arity checking, conversion, and
generic deduction use it. The resulting SemIR and lowering are
unchanged: `self` is still `call_param0`, and witnesses, thunks, and
vtables are unaffected.

An omitted `self` type is parsed as a `SelfBindingPattern` node with no
type expression; checking synthesizes the `Self` type so it behaves
exactly like `self: Self`. However, the exact spelling used must match
between a forward declaration and a definition, following #3763's rules
around declaration matching.

Generated functions, thunks, and C++ interop import/export build `self`
as the first explicit parameter, and the `self`-type override (e.g.
Derived->Base for a virtual override) applies to the explicit `self`.
Placement is validated by new diagnostics: `SelfInImplicitParamList`,
`SelfNotFirstParam`, and `SelfOutsideParamList`. The benchmark source
generator and the documentation adopt the `(self)` shorthand; the
prelude, the examples, and the test data are migrated in the following
commits.

Assisted-by: Claude Code with Claude Opus 4.7

---------

Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com>
2026-06-10 15:30:43 +00:00
Chandler Carruth 22d09f7f25 Add C++ compile benchmarking (#7220)
We had source generation support for some time, but needed to get all
the runtimes set up correctly so that standard library headers are
available. Now that this is in place, we can benchmark both languages.

Also fixes a bug in the C++ source generation causing compile failures.

Assisted-by: Antigravity with Gemini
2026-05-18 23:31:14 +00:00
Chandler Carruth e041afd98d Centralize benchmarking infrastructure and the toolchain-wide benchmarks (#7212)
The benchmarks themselves aren't really specific to `driver`.

Keeping the source generation near to the primary use case of
benchmarking also seems like a more discoverable location.

I feel a little bad doing this reorganization right after I gave a talk
with links to a bunch of this code, but seems good to reorganize a bit
before doing some work to extend things now that we have full standard
library support for C++ benchmarking and other improvements.

Assisted-by: Antigravity with Gemini
2026-05-15 20:17:23 +00:00