mirror of
https://github.com/carbon-language/carbon-lang.git
synced 2026-10-05 14:31:03 +01:00
Chandler CarruthandDana Jansens
9a4a9a9730
This script runs benchmarks written using Google Benchmark repeatedly,
and collects the results from JSON to render them nicely and provide
statistical information across the runs.
Because this runs the binaries repeatedly, this can help account for
run-to-run variations that are pervasive in many of Carbon's benchmarks,
such as ASLR and other process-specific differences.
It's most basic mode runs a benchmark multiple times and shows both
median and confidence intervals.
It also supports two comparison modes:
1) Regular expressions can be provided that describe collections of
related benchmarks where one is the "main" benchmark and the others
are comparable. For example, Carbon's data structure vs. data
structures from LLVM or Abseil. These will be rendered with the main
benchmark first, followed by a comparison relative to a "baseline" of
each comparable benchmark.
2) A baseline benchmark binary, and potentially different command line
flags, can be provided to run two benchmark binaries and compute
a comparison for each benchmark within them.
Across all of these, the script works to present the best text UI it can
in the console. I may have gotten a bit obsessed with rendering the
benchmark results in a way that is really pretty. There are lots of
fancy color coding and progress bars, etc., when run in in the terminal.
For the basic mode without any comparisons, the results look like:
```
Computing statistically significant deltas only wherethe P-value < 𝛂 of 0.05
Metric key:
BenchmarkName... <median> ± <% at 95th conf>
Benchmark ┃ CPU Time ┃ bytes_per_second
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━
BM_LatencyHash<RandValues<uint8_t>, CarbonHashBench>. │ 3.051 ns ± 2.721% │ 327.8 M ± 2.765%
BM_LatencyHash<RandValues<uint8_t>, AbseilHashBench>. │ 3.395 ns ± 4.377% │ 294.6 M ± 4.572%
BM_LatencyHash<RandValues<uint8_t>, LLVMHashBench>... │ 6.125 ns ± 2.662% │ 163.3 M ± 2.726%
BM_LatencyHash<RandValues<uint16_t>, CarbonHashBench> │ 3.105 ns ± 3.947% │ 644.1 M ± 4.109%
BM_LatencyHash<RandValues<uint16_t>, AbseilHashBench> │ 3.433 ns ± 4.308% │ 582.6 M ± 4.502%
BM_LatencyHash<RandValues<uint16_t>, LLVMHashBench>.. │ 6.127 ns ± 2.540% │ 326.5 M ± 2.587%
BM_LatencyHash<RandValues<uint32_t>, CarbonHashBench> │ 3.082 ns ± 2.846% │ 1.298 G ± 2.923%
BM_LatencyHash<RandValues<uint32_t>, AbseilHashBench> │ 3.401 ns ± 3.611% │ 1.176 G ± 3.739%
BM_LatencyHash<RandValues<uint32_t>, LLVMHashBench>.. │ 6.209 ns ± 4.064% │ 644.3 M ± 4.236%
BM_LatencyHash<RandValues<uint64_t>, CarbonHashBench> │ 3.122 ns ± 2.871% │ 2.563 G ± 2.956%
BM_LatencyHash<RandValues<uint64_t>, AbseilHashBench> │ 3.426 ns ± 2.811% │ 2.335 G ± 2.892%
BM_LatencyHash<RandValues<uint64_t>, LLVMHashBench>.. │ 6.497 ns ± 3.081% │ 1.231 G ± 3.179%
```
For the first comparison mode on one of Carbon's benchmarks, the results
look like:
```
Computing statistically significant deltas only wherethe P-value < 𝛂 of 0.05
Metric key:
BenchmarkName... <median> ± <% at 95th conf>
vs Comparable: 👍 <delta> p=<U-test P-value>
<median> ± <% at 95th conf>
Benchmark ┃ CPU Time ┃ bytes_per_second
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━
BM_LatencyHash<RandValues<uint8_t>, CarbonHashBench>. │ 3.037 ns ± 1.781% │ 329.2 M ± 1.813%
vs Abseil: │ 👍 -8.200% p=0.000183 │ 👍 8.933% p=0.000183
│ 3.309 ns ± 2.064% │ 302.2 M ± 2.022%
vs LLVM: │ 👍 -49.401% p=0.000183 │ 👍 97.632% p=0.000183
│ 6.003 ns ± 1.502% │ 166.6 M ± 1.480%
│ │
BM_LatencyHash<RandValues<uint16_t>, CarbonHashBench> │ 3.026 ns ± 1.816% │ 661 M ± 1.784%
vs Abseil: │ 👍 -8.599% p=0.000183 │ 👍 9.408% p=0.000183
│ 3.311 ns ± 1.873% │ 604.1 M ± 1.839%
vs LLVM: │ 👍 -49.829% p=0.000183 │ 👍 99.319% p=0.000183
│ 6.031 ns ± 2.806% │ 331.6 M ± 2.730%
│ │
BM_LatencyHash<RandValues<uint32_t>, CarbonHashBench> │ 3.017 ns ± 2.696% │ 1.326 G ± 2.625%
vs Abseil: │ 👍 -9.754% p=0.000183 │ 👍 10.808% p=0.000183
│ 3.344 ns ± 1.537% │ 1.196 G ± 1.514%
vs LLVM: │ 👍 -49.857% p=0.000183 │ 👍 99.427% p=0.000183
│ 6.018 ns ± 3.269% │ 664.7 M ± 3.167%
│ │
BM_LatencyHash<RandValues<uint64_t>, CarbonHashBench> │ 3.025 ns ± 3.395% │ 2.644 G ± 3.284%
vs Abseil: │ 👍 -9.812% p=0.000183 │ 👍 10.879% p=0.000183
│ 3.354 ns ± 2.640% │ 2.385 G ± 2.572%
vs LLVM: │ 👍 0.476x p=0.000183 │ 👍 2.101x p=0.000183
│ 6.357 ns ± 2.477% │ 1.258 G ± 2.418%
│ │
```
For the second mode, in this case comparing a baseline build with `-Oz`
vs an experiment with `-Os`, the results look like:
```
Computing statistically significant deltas only wherethe P-value < 𝛂 of 0.05
Metric key:
BenchmarkName... 👍 <delta> p=<U-test P-value>
baseline: <median> ± <% at 95th conf>
experiment: <median> ± <% at 95th conf>
Benchmark ┃ CPU Time ┃ bytes_per_second
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━
BM_LatencyHash<RandValues<std::pair<uint32_t, uint32_t>>, CarbonHashBench> │ 👍 -35.870% p=0.000557 │ 👍 55.930% p=0.000557
baseline: │ 5.704 ns ± 1.877% │ 1.403 G ± 1.911%
experiment: │ 3.658 ns ± 4.209% │ 2.187 G ± 4.039%
│ │
BM_LatencyHash<RandValues<std::pair<uint32_t, uint64_t>>, CarbonHashBench> │ 👍 -19.475% p=0.00119 │ 👍 24.186% p=0.00119
baseline: │ 4.974 ns ± 3.029% │ 3.217 G ± 3.124%
experiment: │ 4.005 ns ± 4.297% │ 3.995 G ± 4.120%
│ │
BM_LatencyHash<RandValues<std::pair<uint32_t, int*>>, CarbonHashBench>.... │ 👍 -11.740% p=0.00153 │ 👍 13.302% p=0.00153
baseline: │ 4.634 ns ± 3.433% │ 3.453 G ± 3.555%
experiment: │ 4.09 ns ± 2.999% │ 3.912 G ± 2.911%
│ │
```
The script itself uses a new tool for managing dependencies called `uv`:
https://docs.astral.sh/uv/ This tool allows for the script to contain an
inline set of dependencies that will be installed and cached for
subsequent runs. This seemed particularly important as dependencies like
SciPy and NumPy can be particularly difficult to manager or keep
installed in other ways, but are essential to this scripts statistical
analysis. So far, the `uv` system has been working remarkably well for
me and been a relatively pleasant experience on the whole.
I have included as much of the Python dependencies as have good type
information into the MyPy configuration to get good type checking in
pre-commit however.
Last but not least, this has been a pet project of mine for a quite a
while and so may be a bit rough around the edges as I added and tweaked
functionality based on specific benchmarks I was looking at. It feels
like its gotten useful enough to contribute somewhere, but totally open
to any refactoring or improvements needed. I tried to take a few passes
over it to organize and document the code before sending it, but I'm
sure there are still some things that could use improvement.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
257 lines
8.0 KiB
YAML
257 lines
8.0 KiB
YAML
# Part of the Carbon Language project, under the Apache License v2.0 with LLVM
|
|
# Exceptions. See /LICENSE for license information.
|
|
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
|
|
|
|
# Update versions with:
|
|
# pre-commit autoupdate --freeze && pre-commit run -a
|
|
# See https://pre-commit.com for more information
|
|
# See https://pre-commit.com/hooks.html for more hooks
|
|
|
|
default_language_version:
|
|
python: python3 # Defaults to python2, so override it.
|
|
|
|
repos:
|
|
- repo: https://github.com/pre-commit/pre-commit-hooks
|
|
rev: cef0300fd0fc4d2a87a85fa2093c6b283ea36f4b # frozen: v5.0.0
|
|
hooks:
|
|
- id: check-added-large-files
|
|
- id: check-case-conflict
|
|
- id: check-executables-have-shebangs
|
|
- id: check-merge-conflict
|
|
- id: check-symlinks
|
|
exclude: ^bazel-execroot$
|
|
- id: check-yaml
|
|
- id: detect-private-key
|
|
- id: end-of-file-fixer
|
|
exclude: '^(.*/fuzzer_corpus/.*|.*\.svg)$'
|
|
- id: mixed-line-ending
|
|
args: ['--fix=lf']
|
|
exclude: '^(.*/fuzzer_corpus/.*|.*\.svg)$'
|
|
- id: trailing-whitespace
|
|
exclude: '^(.*/fuzzer_corpus/.*|.*/testdata/.*\.golden|.*\.svg)$'
|
|
- repo: https://github.com/google/pre-commit-tool-hooks
|
|
rev: efaea7c61c774c0b1a9805fd999e754a2d19dbd1 # frozen: v1.2.5
|
|
hooks:
|
|
- id: check-google-doc-style
|
|
- id: markdown-toc
|
|
- repo: local
|
|
hooks:
|
|
- id: fix-cc-deps
|
|
name: Fix missing C++ deps
|
|
entry: scripts/fix_cc_deps.py
|
|
language: python
|
|
files: ^.*/(BUILD|[^/]+\.(h|cpp))$
|
|
pass_filenames: false
|
|
|
|
# Formatters should be run late so that they can re-format any prior changes.
|
|
- repo: https://github.com/psf/black
|
|
rev: 8a737e727ac5ab2f1d4cf5876720ed276dc8dc4b # frozen: 25.1.0
|
|
hooks:
|
|
- id: black
|
|
- repo: local
|
|
hooks:
|
|
- id: prettier
|
|
name: prettier
|
|
language: node
|
|
additional_dependencies: ['prettier@3.3.3']
|
|
types_or: [html, javascript, json, markdown, yaml]
|
|
entry: npx prettier@3.3.3 --write --log-level=warn
|
|
- repo: local
|
|
hooks:
|
|
- id: buildifier
|
|
name: Bazel buildifier
|
|
entry: scripts/run_buildifier.py
|
|
# Beyond just formatting, explicitly fix lint warnings.
|
|
args: ['--lint=fix', '--warnings=all', '-r', '.']
|
|
language: python
|
|
files: |
|
|
(?x)^(
|
|
.*BUILD.*|
|
|
.*MODULE.bazel.*|
|
|
.*WORKSPACE.*|
|
|
.*\.bzl
|
|
)$
|
|
- id: check-bazel-mod-deps
|
|
# Check this after buildifier because buildifier may modify inputs, and
|
|
# MODULE.bazel.lock includes line/column details.
|
|
name: Check bazel mod deps
|
|
entry: scripts/bazel_mod_deps.py
|
|
language: python
|
|
files: |
|
|
(?x)^(
|
|
.*MODULE.bazel.*|
|
|
.*WORKSPACE.*
|
|
)$
|
|
- id: clang-format
|
|
name: clang-format
|
|
entry: clang-format
|
|
types_or: [c++, def]
|
|
language: python
|
|
args: ['-i']
|
|
additional_dependencies: ['clang-format==20.1.4']
|
|
|
|
- repo: local
|
|
hooks:
|
|
- id: check-header-guards
|
|
# This should run after clang-format, which may reformat a guard.
|
|
name: Check header guards
|
|
entry: scripts/check_header_guards.py
|
|
language: python
|
|
files: ^.*\.h$
|
|
- id: check-sha-filenames
|
|
# This may rename files, so it's deliberately between formatters and
|
|
# linters.
|
|
name: Check fuzzer SHA filenames
|
|
entry: scripts/check_sha_filenames.py
|
|
language: python
|
|
files: ^.*/fuzzer_corpus/.*$
|
|
- id: check-toolchain-diagnostics
|
|
name: Check toolchain diagnostics
|
|
entry: toolchain/diagnostics/check_diagnostics.py
|
|
language: python
|
|
files: |
|
|
(?x)^(
|
|
toolchain/.*\.cpp|
|
|
toolchain/.*\.h|
|
|
toolchain/diagnostics/check_diagnostics\.py|
|
|
toolchain/diagnostics/diagnostic_kind\.def
|
|
)$
|
|
pass_filenames: false
|
|
|
|
# Run linters last, as formatters and other checks may fix issues.
|
|
- repo: local
|
|
hooks:
|
|
- id: forbid-llvm-googletest
|
|
name: Checks for deps on LLVM's version of GoogleTest
|
|
entry: scripts/forbid_llvm_googletest.py
|
|
language: python
|
|
files: ^.*/BUILD$
|
|
pass_filenames: false
|
|
- repo: https://github.com/PyCQA/flake8
|
|
rev: 4b5e89b4b108a6c1a000c591d334a99a80d34c7b # frozen: 7.2.0
|
|
hooks:
|
|
- id: flake8
|
|
- repo: https://github.com/pre-commit/mirrors-mypy
|
|
rev: 'f40886d54c729f533f864ed6ce584e920feb0af7' # frozen: v1.15.0
|
|
hooks:
|
|
- id: mypy
|
|
# Use setup.cfg to match the command line.
|
|
args:
|
|
- --config-file=setup.cfg
|
|
# This should match the requirements added in the WORKSPACE pip_install.
|
|
additional_dependencies:
|
|
- gql >= 2.0.0, < 3.0.0
|
|
- PyGitHub
|
|
- rich
|
|
# Exclusions are:
|
|
# - p#### scripts because they're not tested or maintained.
|
|
# - lit.cfg.py because it has multiple copies, breaking mypy.
|
|
# - Unit tests because they sometimes violate typing, such as by
|
|
# assigning a mock to a function.
|
|
exclude: |
|
|
(?x)^(
|
|
proposals/(?!scripts/).*|
|
|
.*/lit\.cfg\.py|
|
|
.*_test\.py
|
|
)$
|
|
- repo: https://github.com/codespell-project/codespell
|
|
rev: 63c8f8312b7559622c0d82815639671ae42132ac # frozen: v2.4.1
|
|
hooks:
|
|
- id: codespell
|
|
args: ['-I', '.codespell_ignore', '--uri-ignore-words-list', '*']
|
|
# Test data may contain intentional misspellings, as well as short,
|
|
# meaningless identifiers that codespell incorrectly identifies as
|
|
# typos but that we would want to detect in other contexts.
|
|
exclude: |
|
|
(?x)^(
|
|
.*/testdata/.*|
|
|
.*/fuzzer_corpus/.*
|
|
)$
|
|
- repo: https://github.com/google/pre-commit-tool-hooks
|
|
rev: efaea7c61c774c0b1a9805fd999e754a2d19dbd1 # frozen: v1.2.5
|
|
hooks:
|
|
- id: check-copyright
|
|
args:
|
|
- --copyright
|
|
- |+
|
|
Part of the Carbon Language project, under the Apache License v2.0 with LLVM
|
|
Exceptions. See /LICENSE for license information.
|
|
SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
|
|
- --custom_format
|
|
- '\.(carbon|c|json|scss|ypp)(\.tmpl)?$'
|
|
- ''
|
|
- '// '
|
|
- ''
|
|
- --custom_format
|
|
- '\.(js|ts|mjs)$'
|
|
- '/*'
|
|
- ' * '
|
|
- ' */'
|
|
- --custom_format
|
|
- '\.(l|lpp|y)$'
|
|
- '/*'
|
|
- ''
|
|
- '*/'
|
|
- --custom_format
|
|
- '\.(plist)$'
|
|
- '<!--'
|
|
- ''
|
|
- '\-->'
|
|
- --custom_format
|
|
- '\.vim$'
|
|
- ''
|
|
- '" '
|
|
- ''
|
|
- --custom_format
|
|
- '\.scm$'
|
|
- ''
|
|
- '; '
|
|
- ''
|
|
- --custom_format
|
|
- '\.lua$'
|
|
- ''
|
|
- '-- '
|
|
- ''
|
|
exclude: |
|
|
(?x)^(
|
|
.bazelversion|
|
|
.github/pull_request_template.md|
|
|
.python-version|
|
|
compile_flags.txt|
|
|
github_tools/requirements.txt|
|
|
third_party/.*|
|
|
utils/vscode/esbuild.js|
|
|
website/.ruby-version|
|
|
website/Gemfile.lock|
|
|
.*\.def|
|
|
.*\.png|
|
|
.*\.svg|
|
|
.*/fuzzer_corpus/.*|
|
|
.*/testdata/.*\.golden
|
|
)$
|
|
- id: check-links
|
|
- repo: local
|
|
hooks:
|
|
- id: check-build-graph
|
|
name: Check build graph
|
|
entry: scripts/check_build_graph.py
|
|
language: python
|
|
files: |
|
|
(?x)^(
|
|
.*BUILD.*|
|
|
.*MODULE.bazel.*|
|
|
.*WORKSPACE.*|
|
|
.*\.bzl
|
|
)$
|
|
|
|
# This excludes third-party code, and patches to third-party code.
|
|
exclude: |
|
|
(?x)^(
|
|
MODULE.bazel.lock|
|
|
bazel/bazel_clang_tidy/.*\.patch|
|
|
bazel/google_benchmark/.*\.patch|
|
|
bazel/libpfm/.*\.patch|
|
|
bazel/llvm_project/.*\.patch|
|
|
third_party/examples/.*/carbon/.*|
|
|
)$
|