Make it easier to see test performance (#5363)

Trying to make it easier to see possible bottlenecks.

Disabling hyperthreading seems like a significant reduction in
contention (15% improvement for me). Going down by half again reduces a
contention a little further, but not significantly from what I see. My
thought is that just flipping the flag is going to work best for people
cross-system versus a "divide by four", but welcome to other opinions
there. Note, I'm not digging into the source of the contention here,
just observing it.

Current default on my system (equivalent to `--threads=128`):

```
Running tests with 128 thread(s)
...
Ran 1272 tests in 3955 ms wall time, 397615 ms across threads
```

Disabling hyperthreads (equivalent to `--threads=64`):

```
Running tests with 64 thread(s)
...
Ran 1272 tests in 3520 ms wall time, 161957 ms across threads
```

`--threads=32`:
```
Running tests with 32 thread(s)
...
Ran 1272 tests in 3329 ms wall time, 69327 ms across threads
```

And for `./autoupdate_testdata.py --threads=64 --print_slowest_tests=5`:

```
Running tests with 64 thread(s)
...
Ran 1272 tests in 3417 ms wall time, 157946 ms across threads
  Slowest tests:
  - toolchain/lower/testdata/function/generic/call_recursive_basic.carbon: 1508 ms, 1484 ms in Run
  - toolchain/lower/testdata/builtins/print_read.carbon: 1506 ms, 1506 ms in Run
  - toolchain/lower/testdata/array/field.carbon: 1488 ms, 1487 ms in Run
  - toolchain/lower/testdata/builtins/int.carbon: 1482 ms, 1475 ms in Run
  - toolchain/lower/testdata/function/definition/params_one.carbon: 1472 ms, 1471 ms in Run
```

In test:

```
==================== Test output for //toolchain/testing:file_test:
Running tests with 64 thread(s)
...
Ran 1272 tests in 2968 ms wall time, 177732 ms across threads
  Slowest tests:
  - toolchain/lower/testdata/builtins/int.carbon: 1544 ms, 1533 ms in Run
  - toolchain/lower/testdata/array/function_param.carbon: 1539 ms, 1537 ms in Run
  - toolchain/lower/testdata/basics/zero.carbon: 1537 ms, 1536 ms in Run
  - toolchain/lower/testdata/function/call/params_one.carbon: 1535 ms, 1534 ms in Run
  - toolchain/lower/testdata/function/definition/params_zero.carbon: 1531 ms, 1531 ms in Run
[==========] Running 1272 tests from 1 test suite.
[----------] Global test environment set-up.
```
This commit is contained in:
Jon Ross-Perkins
2025-04-25 22:06:18 +00:00
committed by GitHub
parent d617cca530
commit 7181a37997
4 changed files with 84 additions and 5 deletions
+2
View File
@@ -164,11 +164,13 @@ auto RunTestFile(const FileTestBase& test_base, bool dump_output,
llvm::raw_svector_ostream output_stream(test_file.actual_stdout);
llvm::raw_svector_ostream error_stream(test_file.actual_stderr);
Timer timer;
ErrorOr<FileTestBase::RunResult> run_result =
dump_output ? test_base.Run(test_args_ref, fs, input_stream, llvm::outs(),
llvm::errs())
: test_base.Run(test_args_ref, fs, input_stream,
output_stream, error_stream);
test_file.run_elapsed_ms = timer.elapsed_ms();
// Ensure stdout/stderr are always fetched, even when discarded on error.
if (test_file.capture_console_output) {