Previously, we only had the dense API declarations benchmark. This adds
a benchmark that aims to emulate what a realistic implementation file in
Carbon would look like, based on what `.cpp` files look like in Carbon,
Clang, and LLVM. It's still a little artificial in order to provide
stability, but it should give us coverage of compile time performance of
statements and function definitions at more realistic scales.
In order to generate more interesting and longer function definitions,
this introduces a plan-based approach to guide how each feature of a
definition is produced. Initially, it identifies these as roughly:
- local variables
- updates to variables
- `if`, `else`, `while` blocks
- "guard" statements -- conditions with a return, break, or continue
- read of a parameter
The plan also manages the nesting depth of these constructs.
Because the plan establishes how many of these constructs exist in a
function definition, and at what nesting depth, the emission can
randomize the order and arrangement without changing the size of the
function. This follows the pattern in all our benchmark source
generation of getting byte-identical stability while avoiding
predictable structures. Technically, I think the current approach goes a
bit too far because function definitions I suspect are somewhat
predictable in their structure in practice. But a conservative benchmark
I think is a good starting point. =]
The target distributions for definition sizes, maximum depths, and
statement mixes come from a very rough Claude-powered analysis of 10,697
function definitions in 533 Carbon toolchain, LLVM, and Clang
implementation files. These were analyzed and classified by Claude using
a one-off Python script and libclang. I haven't reviewed any of the
analysis carefully, the resulting distributions look plausible enough as
a starting point. Comparing real code against generated C++ from the
plans, each number per-definition average:
| statements | definitions | max depth | ifs | loops |
|------------|-------------|-------------|-------------|-------------|
| 7-9 | 10.9 / 11% | 1.25 / 1.23 | 1.74 / 1.86 | 0.23 / 0.15 |
| 14-19 | 7.3 / 7% | 1.89 / 1.88 | 3.53 / 3.71 | 0.61 / 0.42 |
| 28-39 | 3.3 / 3% | 2.76 / 2.75 | 7.75 / 8.46 | 1.26 / 1.27 |
| 60+ | 1.5 / 2% | 4.03 / 3.80 | 25.0 / 27.1 | 3.72 / 4.04 |
To maintain stability, every definition has an accumulator, a statement
reading its parameters, and a `return`, so real definitions of one to
three statements become three.
Note that updates are currently also used instead of calls to other
functions -- a later PR will switch those to calls so that we form more
plausible call graphs. In general, subsequent PRs will refine and expand
the plan to generate increasingly more realistic function definitions.
There are probably lots of improvements that could be made here, both to
generate higher quality definitions and to improve the generation logic
to be more efficient, etc. But already this has allowed _much_ more
extensive benchmarking of Carbon's compile-time performance so it seems
likely a reasonable increment.
Assisted-by: Claude Code and Antigravity with Gemini
---------
Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com>
Build on the plan-based emission to make function definitions cover a
more representative breadth of the language: class-typed local
variables, and local variables declared with destructuring. This
requires some complexity to initialize and manage the class type
variables, but ultimately should make all of our compile time benchmark
data more representative by covering a more realistic breadth of the
language surface.
Claude's rough sampling of Carbon, Clang, and LLVM source suggests class
types range from 30% to 80% of the locals, and structured bindings range
from 0.2% to 2%. When synthesizing, we target approximately 50% of
declarations as class types and 2% as destructurings as we anticipate
destructuring to be of growing popularity and so want to ensure we're
measuring any overhead there.
Assisted-by: Claude Code and Antigravity with Gemini
---------
Co-authored-by: Geoff Romer <gromer@google.com>
Add four kinds of inline definition to generated classes, each with its
own count in `ClassParams`. Three are among the most common kinds of
inline definition in the Carbon, LLVM, and Clang headers: methods that
return a field, test a field, or forward their parameters to a private
method. The fourth computes over its parameters, and stands in for
longer bodies. Here is one class, without the comment before each
declaration:
class ADGjE {
fn Zotm() -> bool;
fn fx(y: ADGjE) -> ADGjE {
var acc: i32 = 0;
acc = acc + y.Checksum();
var aIx: i32 = 1;
var aTc: i32 = aIx + 1;
aIx = aTc;
return ADGjE.Make();
}
fn GJ2m(self, N: c125y) -> ADGjE;
fn Mg(self) -> i64 { return self.zGrc; }
fn UE(self) -> bool { return self.LBYM == 0; }
fn sU(self, z: i64, J: i32) -> ADGjE {
return self.sUImpl(z, J);
}
private fn FYct(self) -> ADGjE;
private fn sUImpl(self, z: i64, J: i32) -> ADGjE;
fn Make() -> ADGjE {
return {.tag = 0, .pV0t = 0, .zxcQ = c125y.Make(), .zGrc = 0, .LBYM =
0};
}
fn Checksum(self) -> i32 {
return self.tag + 1;
}
private var tag: i32;
private var pV0t: i32;
private var zxcQ: c125y;
private var zGrc: i64;
private var LBYM: i32;
}
A computation reads each parameter with a consumer template for its
type, which calls `Checksum` for a class type, and returns a literal or
the class's `Make`. Getters and predicates read fields of fixed types,
because Carbon can't copy a class value that doesn't implement
`Core.Copy`.
The dense declarations benchmarks now have a computation, a getter, and
a forwarder in each class, so their results aren't comparable with
earlier runs.
A file still has the same size for every seed. Type uses and names are
split into pools whose entries are spelled the same number of times, and
getter and predicate field types cycle through the eligible types. Each
computation gets one parameter beyond its random count so that each
class has parameters that can use its own type. The line estimates now
count the slot that `self` takes when a method's parameters wrap, and
round to the nearest whole class.
Assisted-by: Claude Code
Benchmarks are only comparable if the source generator produces the same
number of lines and bytes for a given configuration with any seed. Test
this across many seeds and class shapes, and check it in the compile
benchmark for the files it generates. Also check that generation emits
every name in its name pools, which the byte count depends on.
With field-heavy classes, generation could crash depending on the
shuffle. A class becomes a valid type only after its field types are
chosen, so references to the last class can only go on its own return
and parameter types. When the shuffled pool left more references to it
than that, `GetValidTypeName` found no valid type and failed its check.
Cap the references to each class at its number of functions and methods,
each of which has a return type that can use it, and replace the removed
references with fixed types so the byte count stays seed-independent.
Assisted-by: Claude Code
Clang implements the `"+r,m"` constraint in `benchmark::DoNotOptimize`
by always choosing memory, so each call stores the value to the stack
and loads it back. Most of our benchmarks call it on a loop counter or
another value carried to the next iteration, which puts that store and
reload on the loop's critical path.
On an Apple M1, the cost of that round trip depends on which register
the compiler uses to address the stack slot, and unrelated code changes
move it. Changing only that register from `x29` to `sp`, with the same
address, made `BM_SetLookupHitPtr<Set<int>>` 41.5% to 49.1% slower.
Add `Carbon::Testing::DoNotOptimize` in
`//testing/base:benchmark_helpers`, and switch every benchmark to it. It
only accepts types it can keep in registers: integers, enums, and
pointers go into a `"+r"` constraint with a `"memory"` clobber, and
containers pass their `data()` pointer to that form, so the compiler
must assume the call reads and writes their contents. Any other type is
a compile error. The container form doesn't block optimizing based on a
container's size; passing the container's address does.
Benchmarks that pass a loop counter or carried value no longer measure a
store and reload per iteration, so their numbers aren't comparable with
earlier runs. For example, the hashing latency benchmarks no longer
include one in each hash's latency.
Assisted-by: Claude Code
Refactors the link driver to automatically compile and cache the carbon
prelude for use in linking.
Implements a `carbon_library` rule for compiling the Core library
dependencies in the examples.
This follows the pattern of our main compile benchmarks, but instead of
running the compile through a library API, it does so by running a
separate process. This lets us see the performance that we would expect
from `make` or another build system invoking the compiler, as opposed to
a minimal view from the library API.
Assisted-by: Claude Code
We had source generation support for some time, but needed to get all
the runtimes set up correctly so that standard library headers are
available. Now that this is in place, we can benchmark both languages.
Also fixes a bug in the C++ source generation causing compile failures.
Assisted-by: Antigravity with Gemini
The benchmarks themselves aren't really specific to `driver`.
Keeping the source generation near to the primary use case of
benchmarking also seems like a more discoverable location.
I feel a little bad doing this reorganization right after I gave a talk
with links to a bunch of this code, but seems good to reorganize a bit
before doing some work to extend things now that we have full standard
library support for C++ benchmarking and other improvements.
Assisted-by: Antigravity with Gemini
Chandler Carruthandjosh11b
Lucile Rose Nihlen