Previously, we only had the dense API declarations benchmark. This adds
a benchmark that aims to emulate what a realistic implementation file in
Carbon would look like, based on what `.cpp` files look like in Carbon,
Clang, and LLVM. It's still a little artificial in order to provide
stability, but it should give us coverage of compile time performance of
statements and function definitions at more realistic scales.
In order to generate more interesting and longer function definitions,
this introduces a plan-based approach to guide how each feature of a
definition is produced. Initially, it identifies these as roughly:
- local variables
- updates to variables
- `if`, `else`, `while` blocks
- "guard" statements -- conditions with a return, break, or continue
- read of a parameter
The plan also manages the nesting depth of these constructs.
Because the plan establishes how many of these constructs exist in a
function definition, and at what nesting depth, the emission can
randomize the order and arrangement without changing the size of the
function. This follows the pattern in all our benchmark source
generation of getting byte-identical stability while avoiding
predictable structures. Technically, I think the current approach goes a
bit too far because function definitions I suspect are somewhat
predictable in their structure in practice. But a conservative benchmark
I think is a good starting point. =]
The target distributions for definition sizes, maximum depths, and
statement mixes come from a very rough Claude-powered analysis of 10,697
function definitions in 533 Carbon toolchain, LLVM, and Clang
implementation files. These were analyzed and classified by Claude using
a one-off Python script and libclang. I haven't reviewed any of the
analysis carefully, the resulting distributions look plausible enough as
a starting point. Comparing real code against generated C++ from the
plans, each number per-definition average:
| statements | definitions | max depth | ifs | loops |
|------------|-------------|-------------|-------------|-------------|
| 7-9 | 10.9 / 11% | 1.25 / 1.23 | 1.74 / 1.86 | 0.23 / 0.15 |
| 14-19 | 7.3 / 7% | 1.89 / 1.88 | 3.53 / 3.71 | 0.61 / 0.42 |
| 28-39 | 3.3 / 3% | 2.76 / 2.75 | 7.75 / 8.46 | 1.26 / 1.27 |
| 60+ | 1.5 / 2% | 4.03 / 3.80 | 25.0 / 27.1 | 3.72 / 4.04 |
To maintain stability, every definition has an accumulator, a statement
reading its parameters, and a `return`, so real definitions of one to
three statements become three.
Note that updates are currently also used instead of calls to other
functions -- a later PR will switch those to calls so that we form more
plausible call graphs. In general, subsequent PRs will refine and expand
the plan to generate increasingly more realistic function definitions.
There are probably lots of improvements that could be made here, both to
generate higher quality definitions and to improve the generation logic
to be more efficient, etc. But already this has allowed _much_ more
extensive benchmarking of Carbon's compile-time performance so it seems
likely a reasonable increment.
Assisted-by: Claude Code and Antigravity with Gemini
---------
Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com>
Build on the plan-based emission to make function definitions cover a
more representative breadth of the language: class-typed local
variables, and local variables declared with destructuring. This
requires some complexity to initialize and manage the class type
variables, but ultimately should make all of our compile time benchmark
data more representative by covering a more realistic breadth of the
language surface.
Claude's rough sampling of Carbon, Clang, and LLVM source suggests class
types range from 30% to 80% of the locals, and structured bindings range
from 0.2% to 2%. When synthesizing, we target approximately 50% of
declarations as class types and 2% as destructurings as we anticipate
destructuring to be of growing popularity and so want to ensure we're
measuring any overhead there.
Assisted-by: Claude Code and Antigravity with Gemini
---------
Co-authored-by: Geoff Romer <gromer@google.com>
Add four kinds of inline definition to generated classes, each with its
own count in `ClassParams`. Three are among the most common kinds of
inline definition in the Carbon, LLVM, and Clang headers: methods that
return a field, test a field, or forward their parameters to a private
method. The fourth computes over its parameters, and stands in for
longer bodies. Here is one class, without the comment before each
declaration:
class ADGjE {
fn Zotm() -> bool;
fn fx(y: ADGjE) -> ADGjE {
var acc: i32 = 0;
acc = acc + y.Checksum();
var aIx: i32 = 1;
var aTc: i32 = aIx + 1;
aIx = aTc;
return ADGjE.Make();
}
fn GJ2m(self, N: c125y) -> ADGjE;
fn Mg(self) -> i64 { return self.zGrc; }
fn UE(self) -> bool { return self.LBYM == 0; }
fn sU(self, z: i64, J: i32) -> ADGjE {
return self.sUImpl(z, J);
}
private fn FYct(self) -> ADGjE;
private fn sUImpl(self, z: i64, J: i32) -> ADGjE;
fn Make() -> ADGjE {
return {.tag = 0, .pV0t = 0, .zxcQ = c125y.Make(), .zGrc = 0, .LBYM =
0};
}
fn Checksum(self) -> i32 {
return self.tag + 1;
}
private var tag: i32;
private var pV0t: i32;
private var zxcQ: c125y;
private var zGrc: i64;
private var LBYM: i32;
}
A computation reads each parameter with a consumer template for its
type, which calls `Checksum` for a class type, and returns a literal or
the class's `Make`. Getters and predicates read fields of fixed types,
because Carbon can't copy a class value that doesn't implement
`Core.Copy`.
The dense declarations benchmarks now have a computation, a getter, and
a forwarder in each class, so their results aren't comparable with
earlier runs.
A file still has the same size for every seed. Type uses and names are
split into pools whose entries are spelled the same number of times, and
getter and predicate field types cycle through the eligible types. Each
computation gets one parameter beyond its random count so that each
class has parameters that can use its own type. The line estimates now
count the slot that `self` takes when a method's parameters wrap, and
round to the nearest whole class.
Assisted-by: Claude Code
Benchmarks are only comparable if the source generator produces the same
number of lines and bytes for a given configuration with any seed. Test
this across many seeds and class shapes, and check it in the compile
benchmark for the files it generates. Also check that generation emits
every name in its name pools, which the byte count depends on.
With field-heavy classes, generation could crash depending on the
shuffle. A class becomes a valid type only after its field types are
chosen, so references to the last class can only go on its own return
and parameter types. When the shuffled pool left more references to it
than that, `GetValidTypeName` found no valid type and failed its check.
Cap the references to each class at its number of functions and methods,
each of which has a return type that can use it, and replace the removed
references with fixed types so the byte count stays seed-independent.
Assisted-by: Claude Code
`source_gen_test` checked the line count of the generated C++ but never
compiled it. Compile it with `ClangRunner`, the same way the C++ compile
benchmarks do, which resolves the TODO.
Also require both languages to compile with no diagnostic output at all,
rather than only without errors. A warning in generated code would add
diagnostic emission to what the compile benchmarks measure.
Assisted-by: Claude Code
Refactors the link driver to automatically compile and cache the carbon
prelude for use in linking.
Implements a `carbon_library` rule for compiling the Core library
dependencies in the examples.
In `SourceGenTest.IdentifierByteSumStableAcrossSeeds`, the `first`
variable was storing `llvm::StringRef`s pointing to memory allocated by
a temporary `SourceGen` instance. This memory was freed at the end of
the loop iteration, leaving `first` with dangling references that were
accessed in subsequent iterations.
Fix this by storing `std::string` copies of the identifiers in `first`
to own the memory, and use `llvm::equal` to compare them.
Assisted-by: Antigravity with Gemini
`GetIdentifiersImpl` unconditionally sliced the 64-entry
`IdentifierLengthCounts` table even for uniform distributions, which the
API documents as having no `max_length` limit. Requesting a uniform
distribution with `max_length > 64` therefore tripped an out-of-bounds
slice assertion. Only compute the table slice on the non-uniform path,
where `max_length <= 64` is already enforced.
Add `IdentifierByteSumStableAcrossSeeds`, which exercises this path (a
uniform request up to length 200) and checks the core invariant that the
total identifier byte count is independent of the random seed across a
spread of parameters.
Assisted-by: Claude Code
---------
Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com>
The benchmarks themselves aren't really specific to `driver`.
Keeping the source generation near to the primary use case of
benchmarking also seems like a more discoverable location.
I feel a little bad doing this reorganization right after I gave a talk
with links to a bunch of this code, but seems good to reorganize a bit
before doing some work to extend things now that we have full standard
library support for C++ benchmarking and other improvements.
Assisted-by: Antigravity with Gemini
Chandler Carruthandjosh11b
Lucile Rose Nihlen