Commit Graph
32 Commits
Author SHA1 Message Date
Jon Ross-Perkins fbe917b949 Create a UnifiedDiffMatcher to make golden test failures easier to understand (#6897)
Right now I think everyone has the habit of doing an autoupdate then
using source control for a diff. This is offering an option of better
diff output from the test.

For example:

```
TEST: toolchain/driver/testdata/fail_flush_errors.carbon !
Ran 1 tests in 81 ms wall time, 8 ms across threads
testing/file_test/file_test_base.cpp:264: Failure
Value of: SplitOutput(test_file.actual_stderr)
Expected: matches elements with union diff
  Actual: { "fail_flush_errors.carbon:22:3: error: name `undeclared1` not found [NameNotFound]", "  undeclared1;", "  ^~~~~~~~~~~", "", "fail_flush_errors.carbon:31:3: error: `Core.String` implicitly referenced here, but package `Core` not found [CoreNotFound]", "  \"undec\\x6Cared2\";", "  ^~~~~~~~~~~~~~~~", "", "fail_flush_errors.carbon:35:3: error: name `undeclared2` not found [NameNotFound]", "  undeclared2;", "  ^~~~~~~~~~~", "", "fail_flush_errors.carbon:43:3: error: name `undeclared3` not found [NameNotFound]", "  undeclared3;", "  ^~~~~~~~~~~", "", "" }, union diff (- expected, + actual):
=== diff in expected elements 0 to 2:
+ fail_flush_errors.carbon:22:3: error: name `undeclared1` not found [NameNotFound]
    undeclared1;
    ^~~~~~~~~~~

=== diff in expected elements 4 to 9:
    "undec\x6Cared2";
    ^~~~~~~~~~~~~~~~

+ fail_flush_errors.carbon:35:3: error: name `undeclared2` not found [NameNotFound]
    undeclared2;
    ^~~~~~~~~~~

=== diff end

Stack trace:
  0x55e476d29efd: Carbon::Testing::FileTestCase::TestBody()
  0x55e476dbd1f2: testing::internal::HandleExceptionsInMethodIfSupported<>()
  0x55e476dbcf57: testing::Test::Run()
  0x55e476dbf0bf: testing::TestInfo::Run()
... Google Test internal frames ...


To test this file alone, run:
  bazel test //toolchain/testing:file_test --test_arg=--file_tests=toolchain/driver/testdata/fail_flush_errors.carbon

testing/file_test/file_test_base.cpp:277: Failure
Failed
Autoupdate would make changes to the file content. Run:
bazel run //toolchain/testing:file_test -- --autoupdate --file_tests=toolchain/driver/testdata/fail_flush_errors.carbon
Stack trace:
  0x55e476d2a5f0: Carbon::Testing::FileTestCase::TestBody()
  0x55e476dbd1f2: testing::internal::HandleExceptionsInMethodIfSupported<>()
  0x55e476dbcf57: testing::Test::Run()
  0x55e476dbf0bf: testing::TestInfo::Run()
... Google Test internal frames ...

[  FAILED  ] ToolchainFileTest.toolchain/driver/testdata/fail_flush_errors.carbon, where GetParam() = toolchain/driver/testdata/fail_flush_errors.carbon (93 ms)
```

Assisted-by: Google Antigravity with Gemini
2026-03-13 21:39:21 +00:00
Chandler Carruth 93c7c9ad96 Consolidate on @rules_cc and update it to the latest version (#6580)
Also consolidate on using `//bazel/cc_rules:defs.bzl` where appropriate.

Also update a couple of Bazel modules deps of `@rules_cc` to the latest
versions.
2026-01-13 01:06:45 +00:00
Geoff Romer 2e59a0d520 Handle jj diff-style conflict markers in autoupdate (#6559) 2026-01-07 17:17:40 +00:00
Jon Ross-Perkins 49212feac6 Allow repeated includes, particularly for min_prelude (#5741)
Make min_prelude parts include what they use, and remove the inclusions
which were for indirect uses from the main min_prelude files.
2025-06-26 21:59:55 +00:00
Jon Ross-Perkins fcf445b517 Print captured stdout/stderr on test crashes (#5740)
Related to #5733, but as a general fix, this will hopefully make it a
little easier to debug test/autoupdate crashes.
2025-06-26 20:28:21 +00:00
Jon Ross-Perkins 51b4abec20 Allow args, includes, and splits in file_test includes (#5610)
This builds a mechanism for the toolchain to construct more complex
min_prelude files, which in turn should allow the toolchain to stop
special-casing the min_prelude directory. For example, instead of:

```
      args.insert(args.end(), {"--custom-core",
                               "--exclude-dump-file-prefix=include_files/"});
```

This should allow (in a `min_prelude` file):

```
// EXTRA-ARGS: --custom-core --exclude-dump-file-prefix=include_files/
```

Then when that min_prelude is included, it'd be used.

But also, this should allow sharing between min_prelude files with use
of `INCLUDE-FILE`, which as we make progressively more complex
min_preludes might become useful.
2025-06-05 16:23:15 +00:00
Jon Ross-PerkinsandChandler Carruth b1004012c3 Add linkstamp support to get the target name (#5451)
This removes hardcoding of the test target name from file_test.

---------

Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
2025-05-12 19:01:01 +00:00
Richard SmithandJon Ross-Perkins 66caff2c26 Make the file_test binary work without custom environment variables. (#5442)
Instead of crashing when run outside of `bazel`, make the toolchain's
`file_test` binary work properly when no test-specific environment
variables are set. This makes it a lot easier to run `file_test` under a
debugger.

There are two main changes here:

- Don't crash if `$TEST_TMPDIR` is unset. Instead, fall back to LLVM's
temporary directory (typically `$TMPDIR`). We already did this in some
places in tests. We now do it in more places.
- Don't fall back to a target label of `<target>` in the reproduction
commands if `$TEST_TARGET` is unset, because this causes all the tests
to fail because their output doesn't match the expected output due to a
differing bazel run command. Instead explicitly specify the target from
the `FileTestBase`-derived class.

Infrastructure for this has been added generally, but only rolled out to
the toolchain `file_test` binary for now.

---------

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
2025-05-07 22:28:13 +00:00
Dana Jansens 9a6c74f0cd Introduce FindIfOrNull() FindIfOrNone() and Contains() (#5322)
`FindIfOrNull` returns a pointer to the element in the range if it's
found, and nullptr otherwise. `FindIfOrNone` returns a copy of the
element in the range if it's found, and `T::None` (for a range of
elements of type `T`) otherwise. `Contains` returns a bool indicating
whether the element in the range is found.

These functions replace `llvm::find()` and `llvm::find_if()` when you
want a single answer back instead of an iterator. This avoids the need
to check against `end()`, allowing the return condition to be tested as
a standard bool.

We replace uses of `find()` and `find_if()` that did not require an
iterator with these new helpers.

Note that the return type of `FindIfOrNull` is a pointer since we can
not write `optional<T&>`, which must be tested for null. If the null
check is omitted, UB occurs and the resulting code may end up with an
incorrect pointer (https://crbug.com/40153300) into the range (or
elsewhere), rather than a null dereference. And this would be very
confusing to debug. Hopefully debug builds and sanitizers keep this from
being an issue we sink a bunch of time into debugging.
2025-04-18 14:17:48 +00:00
Jon Ross-Perkins 8c3fa80691 Add cc rule wrappers for cc_env (#5277)
Rules executed by bazel don't necessarily have the right environment to
find the symbolizer, which was the intent of `cc_env` setting
`LLVM_SYMBOLIZER_PATH`. So far, this has kind of been a case-by-case
fix, but every so often I'm trying to debug a crash in a test that
doesn't provide it. Rather continuing down this route, instead add
drop-in wrappers for cc rules so that it's hard to forget.

Note `bazel/cc_rules` is intended to mirror `bazel/carbon_rules` and
`bazel/cc_toolchains`, rather than `@rules_cc`.

AFAICT there isn't a great way to add this as a default for the `bazel
run` environment. It's not typically going to be set on its own,
forwarding `$PATH` would be too broad, and the [action
`env_sets`](https://bazel.build/docs/cc-toolchain-config-reference#using-action-config)
I think are not quite what we need (I think those don't include output
execution, only compilation).
2025-04-11 19:58:54 +00:00
Jon Ross-Perkins 868efb7c86 Make INCLUDE-FILE less sensitive to path changes (#5159)
It's helpful for stability to not have the path in the repo reflected in
test files, something I'm separately running into. So to reduce this,
align INCLUDE-FILE with other split behavior:

- Use the filename (with a "include_files/" subdir to disambiguate),
rather than the full path.
- Note if we eventually want to support splits in these, the same
approach could be extended.
- Only provide as an arg if the user requests files as args.

Factoring AddFile back because it's hard to share; I'm also advocating
to remove the prelude manifest, which would mean the remaining call
could be removed.
2025-03-21 16:03:25 +00:00
Dana Jansens d58b523a5e Add INCLUDE-FILE: and --custom-core for file tests to specify a minimal prelude library (#5080)
The INCLUDE-FILE option is only used in the toolchain tests for now. If
specified in a file test, the given file path is added to the test's
arguments. For toolchain tests this makes the file's package available
to the test. The `--custom-core` command line flag is added to the
driver, which avoids adding the production `Core` package to the command
line. Together, these allow a test to provide their own minimal `Core`
package.

For example, this would replace `Core` with the package and prelude in
`facet_types.carbon`.
```
// INCLUDE-FILE: toolchain/testing/min_prelude/facet_types.carbon
// EXTRA-ARGS: --custom-core
```

To support this:
* //testing knows how to parse INCLUDE-FILE out of the header of a test
file.
* //testing adds the file to the virtual file system, and includes it in
the test's arguments.
* //toolchain/driver grows the --custom-core command line flag to avoid
loading the production `Core` package.

Tests that were creating their own minimal prelude to define BitAnd on
types are now pointed to
toolchain/testing/min_prelude/facet_types.carbon as the prelude. They no
longer need to `import Core` in each test as a result.

Such tests are no longer `no_prelude`, but instead have their own
prelude. So they are moved to a `min_prelude` subdirectory.

Closes #5076
2025-03-10 19:44:50 +00:00
Jon Ross-Perkins 536bfd9cbf Switch test manifests to embedded C++ (#5036)
Inconsistent execution environments make using a path-as-define
difficult, so switch to an embedded file.

Also fixes the lldb launch so that passing tests run cleanly, and adds
TEST_TARGET to gdb (but without testing there). I'm dropping `sourceMap`
because it's not handled quite correctly (also not great to be trying to
pass source mappings in two different ways), and `env` didn't seem to be
working as intended either; maybe specifying `initCommands` causes other
things to not be evaluated. But the straight `initCommands` looks like
it's working. I used lldb to validate execution of these changes.

```
Running initCommands:
(lldb) command script import external/+llvm_project+llvm-project/llvm/utils/lldbDataFormatters.py
(lldb) settings set target.source-map "." "/usr/local/google/home/jperkins/dev/carbon-lang"
(lldb) settings set target.source-map "/proc/self/cwd" "/usr/local/google/home/jperkins/dev/carbon-lang"
(lldb) env TEST_TARGET=//toolchain/testing:file_test
(lldb) env TEST_TMPDIR=/tmp
Running tests with 128 thread(s)
.
Done!
Note: Google Test filter = ToolchainFileTest.toolchain/check/testdata/const/collapse.carbon
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from ToolchainFileTest
[ RUN      ] ToolchainFileTest.toolchain/check/testdata/const/collapse.carbon
[       OK ] ToolchainFileTest.toolchain/check/testdata/const/collapse.carbon (0 ms)
[----------] 1 test from ToolchainFileTest (0 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (1 ms total)
[  PASSED  ] 1 test.
Process 3869310 exited with status = 0 (0x00000000) 
```
2025-02-27 23:31:13 +00:00
Jon Ross-Perkins 46752eeed6 Change manifest passing to drop the flag outside explorer (#5025) 2025-02-27 00:01:09 +00:00
Jon Ross-PerkinsandGeoff Romer 961f20e859 Make FileTest run tests async by default (#4991)
This takes the mechanism currently used for autoupdate and expands it to
the regular tests (deliberately trying to unify logic for
test/autoupdate/dump to deliver consistent behavior). I'm seeing about a
85% reduction in test time, though results will vary based on test
system.

This does some small edits to test output to make it fit better with the
new flow. Note I'm stopping printing of the "here's how to run" on every
test by default, since it's autoupdated into file content by default.
However, it's still there for test failures.

---------

Co-authored-by: Geoff Romer <gromer@google.com>
2025-02-24 23:20:08 +00:00
Jon Ross-Perkins 3d6f14fca5 Refactor FileTestBase to split logic between multiple files (#4968)
I'm refactoring logic to try to make these files more manageable,
particularly as I'm looking at ways to use more threads. I'm renaming
FileTestBase::TestContext to TestFile and FileTestBase::TestFile to
TestFile::Split to try to be more consistent in how we talk about test
files and file splits in general. This change is not expected to change
any behavior, it's just refactoring.

This PR has two commits:

- Copying files for viewing deltas in GH
- Moving logic

The delta of the two commits is the main PR. The second commit can be
viewed on its own to see the delta versus file_test_base.* files where
the logic currently rests (there's still a reordering in run_test.cpp as
part of making one function static).
2025-02-18 17:10:23 +00:00
Jon Ross-PerkinsandRichard Smith 38d25cf622 Share ReadFile in tests (#4967)
Small cleanup for code sharing. Note, `*ReadFile` will trigger a
CHECK-failure on error; the relevant implementations would previously
have failed silently (empty string).

---------

Co-authored-by: Richard Smith <richard@metafoo.co.uk>
2025-02-15 00:17:01 +00:00
Jon Ross-Perkins aec951d7c3 Fix autoupdate handling of carriage return (#4840)
Switching from RE2 to StrReplaceAll because it seems a fair fit for what
actually needs to be done here. Also pick up \t for visibility reasons.

This came up because clangd's LSP-related APIs print carriage returns.
2025-01-24 16:41:57 +00:00
Jon Ross-PerkinsandGeoff Romer 4c4c4a4d2c Add RawStringOstream for slightly simpler streaming to strings (#4817)
This adds a RawStringOstream. Versus TestRawOstream, which is
consolidated over to RawStringOstream, it uses a string for storage
instead of a vector, mainly to support move-to-string semantics. Versus
llvm::raw_string_ostream, it owns the string and supports pwrite (which
is needed for driver and its fd_ostream compatibility requirement).

This converts most uses of llvm::raw_string_ostream, leaving behind a
few in InstNamer that explicitly cannot own the string, such as:

```
     llvm::raw_string_ostream(name)
          << "_" << tree.tokens().GetColumnNumber(token);
```

I have this as its own library so that it can use CHECK.

Yes this doesn't save much code, but it's code we repeatedly write.

---------

Co-authored-by: Geoff Romer <gromer@google.com>
2025-01-18 01:11:44 +00:00
Chandler CarruthandJon Ross-Perkins d3a5b0eee7 Add utilities for managing a toolchain install, and install and use LLD. (#3993)
The install directory contains the BUILD logic for creating an
installable tree of data files and executables for the toolchain, and
a library to facilitate toolchain code accessing the paths to their data
within this installation.

Then adds an installation of LLD in a synthetic LLVM installation, and
teaches the Clang runner to configure this and use it for linking
instead of the system linker.

Currently, the install paths only really manage access to the LLVM
binaries installed and used by the Clang runner for linking, but
eventually other data files like the prelude and runtime libraries will
be fleshed out as well. There are TODOs for moving more things over here
such as the prelude.

One interesting aspect of this is where to put helpers like parts of
LLVM in our install. This PR suggests nesting those files under
`lib/carbon`. While using a `lib` subdirectory isn't a perfect fit for
the FHS (Filesystem Hierarchy Standard), having a single location where
private data is collected is significantly superior to spreading them
across the system. This also matches similar patterns used by Clang
itself and several other language toolchains and standard libraries.

The install directory also provides a natural place for us to build out
packaging rules to create installable packages in various formats, but
that remains future work.

---------

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
2024-06-01 03:19:52 +00:00
Jon Ross-Perkins a3b1c433be Remove legacy repo_name settings (#3772)
I'd kept these in to separate the bazel module update from the BUILD
file changes, then forgot about it. I think all of these can be cleanly
removed now. I think it's something we should clean up for consistency
with the bazel central repository names; I think it's best to reduce
that divergence.

llvm_zlib and llvm_zstd remain because of how llvm depends on the
particular names.
2024-03-13 22:58:56 +00:00
Richard SmithandJon Ross-Perkins bf8697113a Move llvm::Initialize* calls to main. (#3449)
Per their documentation, the `llvm::Initialize*` functions are only
supposed to be called by the main program, not by a library like
toolchain/codegen. Fixes a hang due to a data race in multithreaded
autoupdate.

Add a utility class `Carbon::InitLLVM` to do the common LLVM
initialization shared by all Carbon tools, optionally including
initializing the LLVM targets. Because the LLVM targets add a lot of
binary size, only initialize them for binaries that opt in by depending
on a new target `//common:all_llvm_targets`.

Also fix `//explorer:file_test` and `//explorer:file_test.trace` to
share a binary rather than linking an identical binary twice.

---------

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
2023-12-07 01:45:47 +00:00
Jonathan B. CoeandChandler Carruth 8c28a0494e Add size="small" to test targets where advised (#3326)
Running `bazel test //...` reported:

```
Test execution time outside of range for MODERATE tests.
Consider setting timeout="short" or size="small".
```

This change adds size="small" to avoid such warnings being reported.

---------

Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
2023-10-24 06:52:00 +00:00
Jon Ross-Perkins 7a1d54f8e3 Merge filesystem logic into FileTest and update test files. (#3183)
two_files.carbon has odd STDOUT placement, but will pursue that
separately.
2023-09-05 18:40:00 +00:00
Jon Ross-Perkins 605763d62d Add lint fixes to the buildifier setup. (#3109)
The main motivation for this is to get python loads in using the
`native-py` lint fix. However, enabling that made me wonder, maybe we
should fix in general?

`native-cc` is delayed, but not wholly cancelled (and `native-py`
picking up might indicate `native-cc` won't be too far behind). There's
also some automated fixes for `.append` and dict sorting -- this felt
okay to me, maybe not something to eagerly add but probably not worth
stopping buildifier from fixing (I've noticed the warnings in the past
and had been ignoring them).

Running everything does mean that load orders are sorted automatically
now, which I think is a positive. Most generally, I think these fixes
aren't _harmful_, and having them done automatically seems beneficial:
my biggest concern about `native-py` and `native-cc` was actually that
regressions wouldn't be caught, but this addresses that issue
automatically.
2023-08-22 21:01:42 +00:00
Jon Ross-Perkins b5167b2d69 Implement autoupdate for file_test. (#3043)
I've migrated the toolchain autoupdate scripts here, I just need a
little more time to do the explorer side (which I need to check
performance, that may require multithreading as we do in the current
script). However, this felt substantial enough to share and it means I
can handle autoupdate in more of the toolchain, including preparatory
work for autoupdate on multi-file tests.

Once explorer is done I'll remove the old script.
2023-08-02 21:32:21 +00:00
Jon Ross-Perkins 3f24aa28b7 Introduce flag handling into file_test. (#3030)
I'm integrating absl flag support with a few thoughts here...

1. It simplifies the handling of files in file_test.
    - Removes the need for a separate subset target.
3. Looking forward, I'm planning to add a flag to allow for autoupdate
of golden files.
4. In explorer, there's been confusion about having file_test run tests
"twice" so hopefully it's clearer when it's a separate target with a
different flag on the target.
2023-07-27 23:36:21 +00:00
Jon Ross-Perkins d18c1347d7 Migrate compatible uses to TestRawOstream. (#2891)
Replacing direct raw_string_ostream uses. I figure the wrapper should be used more consistently.

There are still remaining raw_string_ostream uses that weren't compatible -- I'm continuing to look at those, but felt it was cleaner to have this on its own.
2023-06-14 09:31:56 -07:00
Jon Ross-Perkins 6586179c8d Add support for splitting a test file. (#2876)
I'm looking at this as I start thinking about handling `import`. Syntax is based on llvm's `split-file` tool.

The `std::vector` -> `llvm::SmallVector` switch is minor, I'm doing it here because I had to touch everything anyways and I think for tests I'll lean slightly more towards the toolchain's way of doing things versus explorer's.
2023-06-07 08:56:20 -07:00
Jon Ross-Perkins 6b7a522b3f Provide local paths for file tests. (#2830)
The intent of this change is that instead of paths looking like `explorer/testdata/foo/bar.carbon` (repo-relative), they're now just `bar.carbon` (local). The consequence is that paths should be a bit more durable in various environments, and just paths should be shorter and easier to read.

The explorer's prelude is an exception to this since it comes from data, rather than being the test target. Due to the change in approaches, it needs the regex again.

Uses #2829
2023-05-18 10:20:51 -07:00
Jon Ross-Perkins 4247a39acd Add testonly to file_test (#2826)
I don't know why gtest doesn't have testonly set in public, but that's the root of this.
2023-05-15 16:31:57 -07:00
Jon Ross-PerkinsandChandler Carruth 941e60ade6 Add framework for replacing lit with cc_test (#2814)
This is really part of #2811, but is extracted out to allow a little review in parallelism because #2811 expects #2813. Getting this in will allow migration of toolchain tests, whereas #2811 is focused on explorer tests. For explorer test timing information, see #2811.

The syntax being used for matching deliberately mirrors the `FileCheck` setup, partly for compatibility if something changes, partly so there's nothing new to learn, partly so that we don't need to build more test updating.

Individual tests look like:

```
[ RUN      ] ParseAndExecuteTestFile.explorer/parse_and_execute/testdata/assert/convert.carbon

To test this file alone, run:
  bazel test //explorer/parse_and_execute:file_test.subset --test_arg=explorer/parse_and_execute/testdata/assert/convert.carbon

[       OK ] ParseAndExecuteTestFile.explorer/parse_and_execute/testdata/assert/convert.carbon (202 ms)
```

The printed command line is intended to assist developers in debugging a single test, particularly when sharding the main test. The use of a single `.subset` target means the total number of targets is constant even as the number of test files increases, which may be important for some `bazel` execution environments. I plan to make similar changes to the `glob_sh_run` implementation so that we have consistent setups, i.e. that we no longer create target-per-file scaling risks.

This uses `native_test` to share the test binary, avoiding re-linking if files are individually run.

Investigation did reveal a mistake where STDOUT/STDERR wasn't prefixed on empty output lines; this PR fixes that mistake, so that output is fully covered.

Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
2023-05-12 10:05:08 -07:00