diff --git a/.agents/skills/toolchain_development/SKILL.md b/.agents/skills/toolchain_development/SKILL.md index c0a65ece5596..fa763816f6d2 100644 --- a/.agents/skills/toolchain_development/SKILL.md +++ b/.agents/skills/toolchain_development/SKILL.md @@ -52,9 +52,13 @@ SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception ### Updating test data Carbon tests often use `file_test` (for example, -`//toolchain/testing/file_test`). If you change compiler behavior, you likely -need to update expected test outputs. **Do not manually edit thousands of lines -of expected output.** Use the script: +`//toolchain/testing/file_test`). For detailed guidelines on authoring tests, +including file splits, naming conventions (`fail_`, `todo_`), and generating +minimal output with SemIR dumps, please refer to the **Toolchain tests** skill. + +If you change compiler behavior, you likely need to update expected test +outputs. **Do not manually edit thousands of lines of expected output.** Use the +script: ```bash ./toolchain/autoupdate_testdata.py diff --git a/.agents/skills/toolchain_tests/SKILL.md b/.agents/skills/toolchain_tests/SKILL.md new file mode 100644 index 000000000000..569e97714b61 --- /dev/null +++ b/.agents/skills/toolchain_tests/SKILL.md @@ -0,0 +1,136 @@ +--- +name: Toolchain tests +description: + Instructions for authoring, structuring, and running toolchain tests using + the file_test infrastructure. +--- + +# Toolchain tests + + + +## Introduction + +This skill provides guidelines and patterns for creating and updating tests for +the Carbon toolchain, especially file tests in `toolchain/*/testdata/` (for +example, `toolchain/check/testdata/`). + +Toolchain tests evaluate Carbon source files through Lexing, Parsing, Checking, +and optionally Lowering. Output (for example SemIR dumps, Clang errors) is +captured and validated using inline CHECK records. + +## Structure and Authoring + +### File Layout and Headers + +Test files must start with the standard Carbon license, followed by +configuration comments. Separate sections with blank comment lines (`//`). + +```carbon +// Part of the Carbon Language project, under the Apache License v2.0 with LLVM +// Exceptions. See /LICENSE for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +// INCLUDE-FILE: toolchain/testing/testdata/min_prelude/... +// +// AUTOUPDATE +``` + +- `// AUTOUPDATE` is mandatory for files using CHECK markers. +- `// TIP:` lines are automatically generated by the autoupdater. You do not + need to hand-write them. It is harmless to add them, but the script will + handle it. + +### Minimized Preludes + +When writing tests entirely unrelated to the Core package, specify a minimal +prelude file using `// INCLUDE-FILE`. Usually, include +`toolchain/testing/testdata/min_prelude/` scripts, such as `int.carbon` or +`primitives.carbon`. This significantly speeds up execution and minimizes STDOUT +noise. + +### Split Tests and `[[@TEST_NAME]]` + +A single physical file can test multiple scenarios using split constraints: + +```carbon +// --- passing_case.carbon +library "[[@TEST_NAME]]"; + +// ... + +// --- fail_bad_case.carbon +library "[[@TEST_NAME]]"; +// ... +``` + +- Use `library "[[@TEST_NAME]]";` in each split when necessary to prevent name + conflicts or redefining the default library. +- Exactly `[[@TEST_NAME]]` (including the brackets) should be used. The test + infrastructure automatically replaces it with the split's filename minus + `todo_` and `fail_` prefixes. +- **Do not put code that is expected to pass and code that is expected to fail + into the same split.** Validation relies on non-failing splits producing + absolutely no errors and failing splits producing the correct compiler + errors independently. + +### File Prefixing: `fail_` and `todo_` + +Expected failures must be differentiated from unexpected failures (and from +bugs). Include prefixes to name individual split files or the main test: + +- `fail_...`: The test should and does produce compiler errors. +- `todo_fail_...`: The test should produce errors but currently does not. +- `fail_todo_...`: The test does produce errors or crashes, but it shouldn't + (or produces the wrong errors or otherwise misbehaves with errors). +- `todo_...`: The test has some incorrect behavior, but doesn't produce errors + currently, and shouldn't. + +**Main File Naming**: The main test file (and any split-files) must have a +`fail_` prefix if they have an associated error. **Exception**: The main file +may omit `fail_` if it contains a least one split that has a `fail_` prefix. + +Both the `fail_` and `todo_` prefixes are stripped from filename properties like +`[[@TEST_NAME]]`. + +### Test Code Comments + +- **No agent thinking:** Do not include comments describing your reasoning or + "train of thought" (for example, "Wait, but...") inside the test files. Any + comments left in tests should be concise and describe what the test _itself_ + is validating for human readers. + +## SemIR Dumps and Minimizing Output + +Limit STDOUT checks to the logic under test. Always use `//@dump-sem-ir-begin` +and `//@dump-sem-ir-end` around the specific declarations/blocks where SemIR +output is desired. Only use these markers and **not** +`--dump-sem-ir-ranges=if-present` or similar extra args—new tests use +`//@dump-sem-ir...` to naturally filter output to the highlighted segments based +on the default behavior. + +```carbon +//@dump-sem-ir-begin +fn F(x:? form(ref i32)); +//@dump-sem-ir-end +``` + +## Creating/Updating the Output + +AI tools should **never** hand-write or manually touch `// CHECK:STDOUT:` or +`// CHECK:STDERR:` comments. + +Write your Carbon test code, headers, and `// AUTOUPDATE` then run the test +updater: + +```bash +./toolchain/autoupdate_testdata.py toolchain/PATH/TO/YOUR/TEST.carbon +``` + +Review the updated test outputs (for example, by way of `git diff`). Ensure +logic paths are correctly tested rather than producing massive boilerplate +blocks.