This adds free function declarations to the top of the source and
adjusts many of the synthesized expressions in the benchmark source to
emit calls to them. This gets the distribution of expressions in our
generated definitions to be significantly more realistic compared to
existing C++ function bodies here and in LLVM.
This still doesn't get us to start calling more methods, have class
types in the signatures of the calls, or have more interesting call
graphs, but seems like a reasonable first step and still an improvement
in the overall fidelity of our benchmarks.
Illustrative generated Carbon:
```carbon
fn Abcde(x: i32, y: i64*) -> (bool, i64);
fn Fghij(p: i32*) -> i32;
// ...
fn MyClass.Method(self, p: i32) -> i32 {
var acc64: i64 = 0;
var acc: i32 = acc64 as i32;
acc = acc + self.Checksum() + p + 1;
let (abc: bool, def: i64) = Abcde(0, &acc64);
acc = (acc + (if abc then 1 else 0) + (def as i32)) + Fghij(&acc) * 2;
return 0;
}
```
Assisted-by: Claude Code and Antigravity with Gemini
---------
Co-authored-by: Geoff Romer <gromer@google.com>
Add `BM_CompileApiFileRealisticHeader`, which models a typical C++
header. Each class defines 2 getters, 1 predicate, 2 forwarders, and 2
function definitions inline, and declares 2 methods along with the
private methods that the forwarders call. The dense declarations
benchmarks keep modeling a declaration-heavy API file.
The counts come from classifying, with libclang, the member functions
defined in class definitions in the hand-written headers that the
toolchain includes: 1,612 in the Carbon toolchain's headers, 10,406 in
LLVM's, and 13,366 in Clang's. Classified the same way, the generated
C++ defines 69% of its member functions inline in the class definition,
against 61-70% in the real headers. 78% of its inline definitions are a
single statement, against 72-82%, and 21% return the result of a member
call, against 19-21%. Only 2% have control flow, against 7-13%, because
most inline function definitions have only the three statements that
every plan-based definition has.
Assisted-by: Claude Code and Antigravity with Gemini
---------
Co-authored-by: Geoff Romer <gromer@google.com>
We use the dry-run mode for running benchmarks in a test. This is
important, but doesn't need to use large batches of files. Doing so
would cause the compilation benchmark to become excessively slow,
especially on GitHub runners.
Sadly, Google Benchmark doesn't expose its dry-run mode, but we already
have our own `main` function that injects custom logic for our
benchmarks. Extend that to detect the dry run flags and environment
variables so that we can customize the benchmark behavior in that mode.
Then we use that detection to reduce the batch size.
Also cleans up the counters computation to both be a bit better
organized (I think), and to avoid keeping the tokenized buffers we build
to count tokens around during the timed running of the benchmark.
Also cleans up a bug in some clamping code spotted while fixing this.
Assisted-by: Antigravity with Gemini
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
Now that we have a robust plan-based layer for generating function
definitions, we can use that when generating inline function definitions
rather than the very ad-hoc previous inline function definition
synthesis. This also requires switching some of the more ad-hoc
heuristics for governing the size of inline function definitions to more
robust histograms representing the desired distributions.
Some rough analysis of existing C++ code in Clang, Carbon, and LLVM
suggests that multi-statement inline function definitions and
out-of-line
definitions in implementation files have compatible distributions of
shape and statement kinds when their sizes overlap, and so we don't end
up needing to overly customize that part of the plan generation.
Getters, predicates, and forwarding functions continue to be separate
constructs from multi-statement function definitions. I looked at trying
to unify them, but that adds significantly more complexity. For now, the
plan-based synthesis is only used for definitions where we have >1
statements (happens to be >=3 because of the structure of these
definitions).
Assisted-by: Claude Code and Antigravity with Gemini
Previously, we only had the dense API declarations benchmark. This adds
a benchmark that aims to emulate what a realistic implementation file in
Carbon would look like, based on what `.cpp` files look like in Carbon,
Clang, and LLVM. It's still a little artificial in order to provide
stability, but it should give us coverage of compile time performance of
statements and function definitions at more realistic scales.
In order to generate more interesting and longer function definitions,
this introduces a plan-based approach to guide how each feature of a
definition is produced. Initially, it identifies these as roughly:
- local variables
- updates to variables
- `if`, `else`, `while` blocks
- "guard" statements -- conditions with a return, break, or continue
- read of a parameter
The plan also manages the nesting depth of these constructs.
Because the plan establishes how many of these constructs exist in a
function definition, and at what nesting depth, the emission can
randomize the order and arrangement without changing the size of the
function. This follows the pattern in all our benchmark source
generation of getting byte-identical stability while avoiding
predictable structures. Technically, I think the current approach goes a
bit too far because function definitions I suspect are somewhat
predictable in their structure in practice. But a conservative benchmark
I think is a good starting point. =]
The target distributions for definition sizes, maximum depths, and
statement mixes come from a very rough Claude-powered analysis of 10,697
function definitions in 533 Carbon toolchain, LLVM, and Clang
implementation files. These were analyzed and classified by Claude using
a one-off Python script and libclang. I haven't reviewed any of the
analysis carefully, the resulting distributions look plausible enough as
a starting point. Comparing real code against generated C++ from the
plans, each number per-definition average:
| statements | definitions | max depth | ifs | loops |
|------------|-------------|-------------|-------------|-------------|
| 7-9 | 10.9 / 11% | 1.25 / 1.23 | 1.74 / 1.86 | 0.23 / 0.15 |
| 14-19 | 7.3 / 7% | 1.89 / 1.88 | 3.53 / 3.71 | 0.61 / 0.42 |
| 28-39 | 3.3 / 3% | 2.76 / 2.75 | 7.75 / 8.46 | 1.26 / 1.27 |
| 60+ | 1.5 / 2% | 4.03 / 3.80 | 25.0 / 27.1 | 3.72 / 4.04 |
To maintain stability, every definition has an accumulator, a statement
reading its parameters, and a `return`, so real definitions of one to
three statements become three.
Note that updates are currently also used instead of calls to other
functions -- a later PR will switch those to calls so that we form more
plausible call graphs. In general, subsequent PRs will refine and expand
the plan to generate increasingly more realistic function definitions.
There are probably lots of improvements that could be made here, both to
generate higher quality definitions and to improve the generation logic
to be more efficient, etc. But already this has allowed _much_ more
extensive benchmarking of Carbon's compile-time performance so it seems
likely a reasonable increment.
Assisted-by: Claude Code and Antigravity with Gemini
---------
Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com>
Build on the plan-based emission to make function definitions cover a
more representative breadth of the language: class-typed local
variables, and local variables declared with destructuring. This
requires some complexity to initialize and manage the class type
variables, but ultimately should make all of our compile time benchmark
data more representative by covering a more realistic breadth of the
language surface.
Claude's rough sampling of Carbon, Clang, and LLVM source suggests class
types range from 30% to 80% of the locals, and structured bindings range
from 0.2% to 2%. When synthesizing, we target approximately 50% of
declarations as class types and 2% as destructurings as we anticipate
destructuring to be of growing popularity and so want to ensure we're
measuring any overhead there.
Assisted-by: Claude Code and Antigravity with Gemini
---------
Co-authored-by: Geoff Romer <gromer@google.com>
Add four kinds of inline definition to generated classes, each with its
own count in `ClassParams`. Three are among the most common kinds of
inline definition in the Carbon, LLVM, and Clang headers: methods that
return a field, test a field, or forward their parameters to a private
method. The fourth computes over its parameters, and stands in for
longer bodies. Here is one class, without the comment before each
declaration:
class ADGjE {
fn Zotm() -> bool;
fn fx(y: ADGjE) -> ADGjE {
var acc: i32 = 0;
acc = acc + y.Checksum();
var aIx: i32 = 1;
var aTc: i32 = aIx + 1;
aIx = aTc;
return ADGjE.Make();
}
fn GJ2m(self, N: c125y) -> ADGjE;
fn Mg(self) -> i64 { return self.zGrc; }
fn UE(self) -> bool { return self.LBYM == 0; }
fn sU(self, z: i64, J: i32) -> ADGjE {
return self.sUImpl(z, J);
}
private fn FYct(self) -> ADGjE;
private fn sUImpl(self, z: i64, J: i32) -> ADGjE;
fn Make() -> ADGjE {
return {.tag = 0, .pV0t = 0, .zxcQ = c125y.Make(), .zGrc = 0, .LBYM =
0};
}
fn Checksum(self) -> i32 {
return self.tag + 1;
}
private var tag: i32;
private var pV0t: i32;
private var zxcQ: c125y;
private var zGrc: i64;
private var LBYM: i32;
}
A computation reads each parameter with a consumer template for its
type, which calls `Checksum` for a class type, and returns a literal or
the class's `Make`. Getters and predicates read fields of fixed types,
because Carbon can't copy a class value that doesn't implement
`Core.Copy`.
The dense declarations benchmarks now have a computation, a getter, and
a forwarder in each class, so their results aren't comparable with
earlier runs.
A file still has the same size for every seed. Type uses and names are
split into pools whose entries are spelled the same number of times, and
getter and predicate field types cycle through the eligible types. Each
computation gets one parameter beyond its random count so that each
class has parameters that can use its own type. The line estimates now
count the slot that `self` takes when a method's parameters wrap, and
round to the nearest whole class.
Assisted-by: Claude Code
Benchmarks are only comparable if the source generator produces the same
number of lines and bytes for a given configuration with any seed. Test
this across many seeds and class shapes, and check it in the compile
benchmark for the files it generates. Also check that generation emits
every name in its name pools, which the byte count depends on.
With field-heavy classes, generation could crash depending on the
shuffle. A class becomes a valid type only after its field types are
chosen, so references to the last class can only go on its own return
and parameter types. When the shuffled pool left more references to it
than that, `GetValidTypeName` found no valid type and failed its check.
Cap the references to each class at its number of functions and methods,
each of which has a return type that can use it, and replace the removed
references with fixed types so the byte count stays seed-independent.
Assisted-by: Claude Code
Clang implements the `"+r,m"` constraint in `benchmark::DoNotOptimize`
by always choosing memory, so each call stores the value to the stack
and loads it back. Most of our benchmarks call it on a loop counter or
another value carried to the next iteration, which puts that store and
reload on the loop's critical path.
On an Apple M1, the cost of that round trip depends on which register
the compiler uses to address the stack slot, and unrelated code changes
move it. Changing only that register from `x29` to `sp`, with the same
address, made `BM_SetLookupHitPtr<Set<int>>` 41.5% to 49.1% slower.
Add `Carbon::Testing::DoNotOptimize` in
`//testing/base:benchmark_helpers`, and switch every benchmark to it. It
only accepts types it can keep in registers: integers, enums, and
pointers go into a `"+r"` constraint with a `"memory"` clobber, and
containers pass their `data()` pointer to that form, so the compiler
must assume the call reads and writes their contents. Any other type is
a compile error. The container form doesn't block optimizing based on a
container's size; passing the container's address does.
Benchmarks that pass a loop counter or carried value no longer measure a
store and reload per iteration, so their numbers aren't comparable with
earlier runs. For example, the hashing latency benchmarks no longer
include one in each hash's latency.
Assisted-by: Claude Code
Refactors the link driver to automatically compile and cache the carbon
prelude for use in linking.
Implements a `carbon_library` rule for compiling the Core library
dependencies in the examples.
This follows the pattern of our main compile benchmarks, but instead of
running the compile through a library API, it does so by running a
separate process. This lets us see the performance that we would expect
from `make` or another build system invoking the compiler, as opposed to
a minimal view from the library API.
Assisted-by: Claude Code
We had source generation support for some time, but needed to get all
the runtimes set up correctly so that standard library headers are
available. Now that this is in place, we can benchmark both languages.
Also fixes a bug in the C++ source generation causing compile failures.
Assisted-by: Antigravity with Gemini
The benchmarks themselves aren't really specific to `driver`.
Keeping the source generation near to the primary use case of
benchmarking also seems like a more discoverable location.
I feel a little bad doing this reorganization right after I gave a talk
with links to a bunch of this code, but seems good to reorganize a bit
before doing some work to extend things now that we have full standard
library support for C++ benchmarking and other improvements.
Assisted-by: Antigravity with Gemini
Chandler CarruthandGeoff Romer
Lucile Rose Nihlen