mirror of
https://github.com/carbon-language/carbon-lang.git
synced 2026-09-24 21:40:12 +01:00
55dcf665845296b9791ad9655c30de660c7ed1a4
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Chandler CarruthandChristopher Di Bella
|
361c832713 |
Add a prelude compilation benchmark (#7368)
Adds `toolchain/benchmarking/prelude_benchmark.cpp`, which measures the time to compile the Core prelude and reports the most interesting SemIR memory statistics as benchmark counters. The prelude is compiled by checking a file that imports it: the implicit prelude import causes the check phase to lex, parse, and check the full set of prelude files, so this is a direct measure of prelude compilation cost. Four input variations exercise increasing amounts of the prelude: an empty file, a minimal single-type use, an operator-heavy file that hits many impls, and a compact file that pulls in a wide swath of the prelude. Memory usage is queried directly: `Driver::set_mem_usage` takes a `MemUsage` that a compile merges each file's usage into; the benchmark passes one, compiles, and sums the entries by label. A compilation unit collects into its own `MemUsage` whenever usage is dumped or a sink is provided (decided in `SetMultiUnitCache`); after a file is done it dumps that `MemUsage` per-file as before and, if a sink was provided, merges into it via a new `MemUsage::Add(const MemUsage&)` overload. `MemUsage` also exposes its entries via a public `Entry` type and an `entries()` accessor. Also extends `scripts/bench_runner.py` to (1) treat Mem-prefixed counters as cost metrics (smaller is better) and (2) tolerate metrics that aren't reported by every benchmark in a binary. Assisted-by: Claude Code --------- Co-authored-by: Christopher Di Bella <cjdb.ns@gmail.com> |
||
 Dana Jansens
|
233a58fcb1 |
Fix ty errors for missing generic parameters (#7326)
Mostly these errors were around `dict` missing arguments, and they are almost always `[str, Any]`. But a real thorn here was `xml.etree.ElementTree.Element`. `ty` insists that this is a generic type, and indeed it appears to be one, or becoming one, in some python version. But it is not generic in python 3.12. So we are stuck in an unsolvable land where: - `ty` gives an error unless you write `[str]` on the type, because it thinks it is generic. - python3.12 gives an error if you do write `[str]` on the type, because it thinks it is not generic. Forcing `ty` to target exactly python 3.12 does not help. So I have just used a linter-ignore comment on that line. |
||
|
Chandler Carruth
|
b344f3af12 |
Switch a few stragglers to uv and update python to 3.12 (#7296)
Not sure how these got missed when moving other things to `uv`, but this should clean them up. The bump to Python 3.12 is so that we can use `@override` with the simple import from `typing`. This is needed by the newest versions of `ty` to do type checking. Added the relevant `@override` annotations. Assisted-by: Antigravity with Gemini |
||
|
Chandler Carruth
|
3ef128ac91 |
Switch to Astral Python tools: ruff and ty (#7243)
This replaces black, flake8, and mypy with the more modern and efficient tools `ruff` and `ty` from Astral. Assisted-by: Antigravity with Gemini |
||
 Jon Ross-Perkins
|
4b076291c9 |
pre-commit autoupdate (#6845)
Assisted-by: Google Antigravity with Gemini |
||
|
Chandler Carruth
|
1a97021875 |
Enhance benchmark runner args and defaults (#5797)
There are some very useful default arguments, so teach the runner script to directly provide them. They can be easily overridden if needed. As part of this, change the default run count to 10 which much more often produces statistically significant error bars and results. Also, tweak the processing of the results to provide a stable order based on the source order, even when randomized interleaving is enabled. The randomized interleaving improves the statistical strength of the benchmarks significantly, but displaying the results in the source order is much more understandable. This should give roughly the best of both worlds. |
||
|
Chandler CarruthandDana Jansens
|
9a4a9a9730 |
Introduce a benchmark running script (#5706)
This script runs benchmarks written using Google Benchmark repeatedly,
and collects the results from JSON to render them nicely and provide
statistical information across the runs.
Because this runs the binaries repeatedly, this can help account for
run-to-run variations that are pervasive in many of Carbon's benchmarks,
such as ASLR and other process-specific differences.
It's most basic mode runs a benchmark multiple times and shows both
median and confidence intervals.
It also supports two comparison modes:
1) Regular expressions can be provided that describe collections of
related benchmarks where one is the "main" benchmark and the others
are comparable. For example, Carbon's data structure vs. data
structures from LLVM or Abseil. These will be rendered with the main
benchmark first, followed by a comparison relative to a "baseline" of
each comparable benchmark.
2) A baseline benchmark binary, and potentially different command line
flags, can be provided to run two benchmark binaries and compute
a comparison for each benchmark within them.
Across all of these, the script works to present the best text UI it can
in the console. I may have gotten a bit obsessed with rendering the
benchmark results in a way that is really pretty. There are lots of
fancy color coding and progress bars, etc., when run in in the terminal.
For the basic mode without any comparisons, the results look like:
```
Computing statistically significant deltas only wherethe P-value < 𝛂 of 0.05
Metric key:
BenchmarkName... <median> ± <% at 95th conf>
Benchmark ┃ CPU Time ┃ bytes_per_second
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━
BM_LatencyHash<RandValues<uint8_t>, CarbonHashBench>. │ 3.051 ns ± 2.721% │ 327.8 M ± 2.765%
BM_LatencyHash<RandValues<uint8_t>, AbseilHashBench>. │ 3.395 ns ± 4.377% │ 294.6 M ± 4.572%
BM_LatencyHash<RandValues<uint8_t>, LLVMHashBench>... │ 6.125 ns ± 2.662% │ 163.3 M ± 2.726%
BM_LatencyHash<RandValues<uint16_t>, CarbonHashBench> │ 3.105 ns ± 3.947% │ 644.1 M ± 4.109%
BM_LatencyHash<RandValues<uint16_t>, AbseilHashBench> │ 3.433 ns ± 4.308% │ 582.6 M ± 4.502%
BM_LatencyHash<RandValues<uint16_t>, LLVMHashBench>.. │ 6.127 ns ± 2.540% │ 326.5 M ± 2.587%
BM_LatencyHash<RandValues<uint32_t>, CarbonHashBench> │ 3.082 ns ± 2.846% │ 1.298 G ± 2.923%
BM_LatencyHash<RandValues<uint32_t>, AbseilHashBench> │ 3.401 ns ± 3.611% │ 1.176 G ± 3.739%
BM_LatencyHash<RandValues<uint32_t>, LLVMHashBench>.. │ 6.209 ns ± 4.064% │ 644.3 M ± 4.236%
BM_LatencyHash<RandValues<uint64_t>, CarbonHashBench> │ 3.122 ns ± 2.871% │ 2.563 G ± 2.956%
BM_LatencyHash<RandValues<uint64_t>, AbseilHashBench> │ 3.426 ns ± 2.811% │ 2.335 G ± 2.892%
BM_LatencyHash<RandValues<uint64_t>, LLVMHashBench>.. │ 6.497 ns ± 3.081% │ 1.231 G ± 3.179%
```
For the first comparison mode on one of Carbon's benchmarks, the results
look like:
```
Computing statistically significant deltas only wherethe P-value < 𝛂 of 0.05
Metric key:
BenchmarkName... <median> ± <% at 95th conf>
vs Comparable: 👍 <delta> p=<U-test P-value>
<median> ± <% at 95th conf>
Benchmark ┃ CPU Time ┃ bytes_per_second
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━
BM_LatencyHash<RandValues<uint8_t>, CarbonHashBench>. │ 3.037 ns ± 1.781% │ 329.2 M ± 1.813%
vs Abseil: │ 👍 -8.200% p=0.000183 │ 👍 8.933% p=0.000183
│ 3.309 ns ± 2.064% │ 302.2 M ± 2.022%
vs LLVM: │ 👍 -49.401% p=0.000183 │ 👍 97.632% p=0.000183
│ 6.003 ns ± 1.502% │ 166.6 M ± 1.480%
│ │
BM_LatencyHash<RandValues<uint16_t>, CarbonHashBench> │ 3.026 ns ± 1.816% │ 661 M ± 1.784%
vs Abseil: │ 👍 -8.599% p=0.000183 │ 👍 9.408% p=0.000183
│ 3.311 ns ± 1.873% │ 604.1 M ± 1.839%
vs LLVM: │ 👍 -49.829% p=0.000183 │ 👍 99.319% p=0.000183
│ 6.031 ns ± 2.806% │ 331.6 M ± 2.730%
│ │
BM_LatencyHash<RandValues<uint32_t>, CarbonHashBench> │ 3.017 ns ± 2.696% │ 1.326 G ± 2.625%
vs Abseil: │ 👍 -9.754% p=0.000183 │ 👍 10.808% p=0.000183
│ 3.344 ns ± 1.537% │ 1.196 G ± 1.514%
vs LLVM: │ 👍 -49.857% p=0.000183 │ 👍 99.427% p=0.000183
│ 6.018 ns ± 3.269% │ 664.7 M ± 3.167%
│ │
BM_LatencyHash<RandValues<uint64_t>, CarbonHashBench> │ 3.025 ns ± 3.395% │ 2.644 G ± 3.284%
vs Abseil: │ 👍 -9.812% p=0.000183 │ 👍 10.879% p=0.000183
│ 3.354 ns ± 2.640% │ 2.385 G ± 2.572%
vs LLVM: │ 👍 0.476x p=0.000183 │ 👍 2.101x p=0.000183
│ 6.357 ns ± 2.477% │ 1.258 G ± 2.418%
│ │
```
For the second mode, in this case comparing a baseline build with `-Oz`
vs an experiment with `-Os`, the results look like:
```
Computing statistically significant deltas only wherethe P-value < 𝛂 of 0.05
Metric key:
BenchmarkName... 👍 <delta> p=<U-test P-value>
baseline: <median> ± <% at 95th conf>
experiment: <median> ± <% at 95th conf>
Benchmark ┃ CPU Time ┃ bytes_per_second
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━
BM_LatencyHash<RandValues<std::pair<uint32_t, uint32_t>>, CarbonHashBench> │ 👍 -35.870% p=0.000557 │ 👍 55.930% p=0.000557
baseline: │ 5.704 ns ± 1.877% │ 1.403 G ± 1.911%
experiment: │ 3.658 ns ± 4.209% │ 2.187 G ± 4.039%
│ │
BM_LatencyHash<RandValues<std::pair<uint32_t, uint64_t>>, CarbonHashBench> │ 👍 -19.475% p=0.00119 │ 👍 24.186% p=0.00119
baseline: │ 4.974 ns ± 3.029% │ 3.217 G ± 3.124%
experiment: │ 4.005 ns ± 4.297% │ 3.995 G ± 4.120%
│ │
BM_LatencyHash<RandValues<std::pair<uint32_t, int*>>, CarbonHashBench>.... │ 👍 -11.740% p=0.00153 │ 👍 13.302% p=0.00153
baseline: │ 4.634 ns ± 3.433% │ 3.453 G ± 3.555%
experiment: │ 4.09 ns ± 2.999% │ 3.912 G ± 2.911%
│ │
```
The script itself uses a new tool for managing dependencies called `uv`:
https://docs.astral.sh/uv/ This tool allows for the script to contain an
inline set of dependencies that will be installed and cached for
subsequent runs. This seemed particularly important as dependencies like
SciPy and NumPy can be particularly difficult to manager or keep
installed in other ways, but are essential to this scripts statistical
analysis. So far, the `uv` system has been working remarkably well for
me and been a relatively pleasant experience on the whole.
I have included as much of the Python dependencies as have good type
information into the MyPy configuration to get good type checking in
pre-commit however.
Last but not least, this has been a pet project of mine for a quite a
while and so may be a bit rough around the edges as I added and tweaked
functionality based on specific benchmarks I was looking at. It feels
like its gotten useful enough to contribute somewhere, but totally open
to any refactoring or improvements needed. I tried to take a few passes
over it to organize and document the code before sending it, but I'm
sure there are still some things that could use improvement.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
|