I've migrated the toolchain autoupdate scripts here, I just need a
little more time to do the explorer side (which I need to check
performance, that may require multithreading as we do in the current
script). However, this felt substantial enough to share and it means I
can handle autoupdate in more of the toolchain, including preparatory
work for autoupdate on multi-file tests.
Once explorer is done I'll remove the old script.
This is a simplification of the construction, although somewhat limiting
(it means that the caller can't register the same file multiple times,
though I stopped doing that anyways since it was causing confusion).
What this more importantly _allows_ is logic on the FileTestBase child
itself that's not test-specific -- in particular, autoupdate
functionality which wouldn't use RUN_ALL_TESTS.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
I'm integrating absl flag support with a few thoughts here...
1. It simplifies the handling of files in file_test.
- Removes the need for a separate subset target.
3. Looking forward, I'm planning to add a flag to allow for autoupdate
of golden files.
4. In explorer, there's been confusion about having file_test run tests
"twice" so hopefully it's clearer when it's a separate target with a
different flag on the target.
This is sufficient to convert remaining toolchain tests to file_test. %s
and %t are currently used. While #2978 might use %T with lit, I think
that's resolving a lit-specific issue that's not necessary when
TEST_TMPDIR is readily available for any necessary operations (i.e.,
setting the working directory). With ARGS, I think it's feasible to
switch to file_test without %T.
Also adds documentation to file_test_base.h, which was starting to feel
like a significant gap.
This is not yet handled by autoupdate, but I'm eyeing that next.
Replacing direct raw_string_ostream uses. I figure the wrapper should be used more consistently.
There are still remaining raw_string_ostream uses that weren't compatible -- I'm continuing to look at those, but felt it was cleaner to have this on its own.
In explorer, we already support parsing a string_view, so use that. In toolchain, we need to build support, probably using vfs, so that's a todo.
bazel test //explorer:file_test --runs_per_test=5
- branch: Stats over 250 runs: max = 18.3s, min = 5.2s, avg = 11.2s, dev = 2.9s
- trunk: Stats over 250 runs: max = 22.3s, min = 5.9s, avg = 12.1s, dev = 2.8s
Not a dramatic improvement, but maybe more effective long-term, and this'd been requested on #2876
I'm looking at this as I start thinking about handling `import`. Syntax is based on llvm's `split-file` tool.
The `std::vector` -> `llvm::SmallVector` switch is minor, I'm doing it here because I had to touch everything anyways and I think for tests I'll lean slightly more towards the toolchain's way of doing things versus explorer's.
I believe this broke when I was changing path handling.
Bad:
```
bazel test //testing/file_test:file_test_base_test.subset --test_arg=/usr/local/...elided.../execroot/carbon/bazel-out/k8-fastbuild/bin/testing/file_test/file_test_base_test.runfiles/carbon/testing/file_test/example.carbon
```
Good:
```
bazel test //testing/file_test:file_test_base_test.subset --test_arg=testing/file_test/example.carbon
```
The intent of this change is that instead of paths looking like `explorer/testdata/foo/bar.carbon` (repo-relative), they're now just `bar.carbon` (local). The consequence is that paths should be a bit more durable in various environments, and just paths should be shorter and easier to read.
The explorer's prelude is an exception to this since it comes from data, rather than being the test target. Due to the change in approaches, it needs the regex again.
Uses #2829
This is really part of #2811, but is extracted out to allow a little review in parallelism because #2811 expects #2813. Getting this in will allow migration of toolchain tests, whereas #2811 is focused on explorer tests. For explorer test timing information, see #2811.
The syntax being used for matching deliberately mirrors the `FileCheck` setup, partly for compatibility if something changes, partly so there's nothing new to learn, partly so that we don't need to build more test updating.
Individual tests look like:
```
[ RUN ] ParseAndExecuteTestFile.explorer/parse_and_execute/testdata/assert/convert.carbon
To test this file alone, run:
bazel test //explorer/parse_and_execute:file_test.subset --test_arg=explorer/parse_and_execute/testdata/assert/convert.carbon
[ OK ] ParseAndExecuteTestFile.explorer/parse_and_execute/testdata/assert/convert.carbon (202 ms)
```
The printed command line is intended to assist developers in debugging a single test, particularly when sharding the main test. The use of a single `.subset` target means the total number of targets is constant even as the number of test files increases, which may be important for some `bazel` execution environments. I plan to make similar changes to the `glob_sh_run` implementation so that we have consistent setups, i.e. that we no longer create target-per-file scaling risks.
This uses `native_test` to share the test binary, avoiding re-linking if files are individually run.
Investigation did reveal a mistake where STDOUT/STDERR wasn't prefixed on empty output lines; this PR fixes that mistake, so that output is fully covered.
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Jon Ross-Perkins
