mirror of
https://github.com/carbon-language/carbon-lang.git
synced 2026-11-18 04:10:29 +00:00
Previously, we only had the dense API declarations benchmark. This adds a benchmark that aims to emulate what a realistic implementation file in Carbon would look like, based on what `.cpp` files look like in Carbon, Clang, and LLVM. It's still a little artificial in order to provide stability, but it should give us coverage of compile time performance of statements and function definitions at more realistic scales. In order to generate more interesting and longer function definitions, this introduces a plan-based approach to guide how each feature of a definition is produced. Initially, it identifies these as roughly: - local variables - updates to variables - `if`, `else`, `while` blocks - "guard" statements -- conditions with a return, break, or continue - read of a parameter The plan also manages the nesting depth of these constructs. Because the plan establishes how many of these constructs exist in a function definition, and at what nesting depth, the emission can randomize the order and arrangement without changing the size of the function. This follows the pattern in all our benchmark source generation of getting byte-identical stability while avoiding predictable structures. Technically, I think the current approach goes a bit too far because function definitions I suspect are somewhat predictable in their structure in practice. But a conservative benchmark I think is a good starting point. =] The target distributions for definition sizes, maximum depths, and statement mixes come from a very rough Claude-powered analysis of 10,697 function definitions in 533 Carbon toolchain, LLVM, and Clang implementation files. These were analyzed and classified by Claude using a one-off Python script and libclang. I haven't reviewed any of the analysis carefully, the resulting distributions look plausible enough as a starting point. Comparing real code against generated C++ from the plans, each number per-definition average: | statements | definitions | max depth | ifs | loops | |------------|-------------|-------------|-------------|-------------| | 7-9 | 10.9 / 11% | 1.25 / 1.23 | 1.74 / 1.86 | 0.23 / 0.15 | | 14-19 | 7.3 / 7% | 1.89 / 1.88 | 3.53 / 3.71 | 0.61 / 0.42 | | 28-39 | 3.3 / 3% | 2.76 / 2.75 | 7.75 / 8.46 | 1.26 / 1.27 | | 60+ | 1.5 / 2% | 4.03 / 3.80 | 25.0 / 27.1 | 3.72 / 4.04 | To maintain stability, every definition has an accumulator, a statement reading its parameters, and a `return`, so real definitions of one to three statements become three. Note that updates are currently also used instead of calls to other functions -- a later PR will switch those to calls so that we form more plausible call graphs. In general, subsequent PRs will refine and expand the plan to generate increasingly more realistic function definitions. There are probably lots of improvements that could be made here, both to generate higher quality definitions and to improve the generation logic to be more efficient, etc. But already this has allowed _much_ more extensive benchmarking of Carbon's compile-time performance so it seems likely a reasonable increment. Assisted-by: Claude Code and Antigravity with Gemini --------- Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com>