Files
carbon-lang/testing/file_test
Chandler CarruthandJon Ross-Perkins d3a5b0eee7 Add utilities for managing a toolchain install, and install and use LLD. (#3993)
The install directory contains the BUILD logic for creating an
installable tree of data files and executables for the toolchain, and
a library to facilitate toolchain code accessing the paths to their data
within this installation.

Then adds an installation of LLD in a synthetic LLVM installation, and
teaches the Clang runner to configure this and use it for linking
instead of the system linker.

Currently, the install paths only really manage access to the LLVM
binaries installed and used by the Clang runner for linking, but
eventually other data files like the prelude and runtime libraries will
be fleshed out as well. There are TODOs for moving more things over here
such as the prelude.

One interesting aspect of this is where to put helpers like parts of
LLVM in our install. This PR suggests nesting those files under
`lib/carbon`. While using a `lib` subdirectory isn't a perfect fit for
the FHS (Filesystem Hierarchy Standard), having a single location where
private data is collected is significantly superior to spreading them
across the system. This also matches similar patterns used by Clang
itself and several other language toolchains and standard libraries.

The install directory also provides a natural place for us to build out
packaging rules to create installable packages in various formats, but
that remains future work.

---------

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
2024-06-01 03:19:52 +00:00
..

file_test

BUILD

A typical BUILD target will look like:

load("rules.bzl", "file_test")

file_test(
    name = "my_file_test",
    srcs = ["my_file_test.cpp"],
    tests = glob(["testdata/**"]),
    deps = [
        ":my_lib",
        "//testing/file_test:file_test_base",
        "@googletest//:gtest",
        "@llvm-project//llvm:Support",
    ],
)

Implementation

A typical implementation will look like:

#include "my_library.h"

#include "testing/file_test/file_test_base.h"

namespace Carbon::Testing {
namespace {

class MyFileTest : public FileTestBase {
 public:
  using FileTestBase::FileTestBase;

  // Called as part of individual test executions.
  auto Run(const llvm::SmallVector<llvm::StringRef>& test_args,
           const llvm::SmallVector<TestFile>& test_files,
           llvm::raw_pwrite_stream& stdout, llvm::raw_pwrite_stream& stderr)
      -> ErrorOr<RunResult> override {
    return MyFunctionality(test_args, stdout, stderr);
  }

  // Provides arguments which are used in tests that don't provide ARGS.
  auto GetDefaultArgs() -> llvm::SmallVector<std::string> override {
    return {"default_args", "%s"};
  }
};

}  // namespace

// Registers for the framework to construct the tests.
CARBON_FILE_TEST_FACTORY(MyFileTest);

}  // namespace Carbon::Testing

Filename fail_ prefixes

When a run fails, information about what pieces failed are returned on RunResult. This affects whether a fail_ prefix on the file is required, including in combination with split-file tests (using the // --- <filename> comment marker).

The main test file and any split-files must have a fail_ prefix if and only if they have an associated error. An exception is that the main test file may omit fail_ when it contains split-files that have a fail_ prefix.

Comment markers

Settings in files are provided in comments, similar to FileCheck syntax. bazel run :file_test -- --autoupdate automatically constructs compatible CHECK:STDOUT: and CHECK:STDERR: lines.

Supported comment markers are:

  • // AUTOUDPATE
    // NOAUTOUPDATE
    

    Controls whether the checks in the file will be autoupdated if --autoupdate is passed. Exactly one of these two markers must be present. If the file uses splits, AUTOUPDATE must currently be before any splits.

    When autoupdating, CHECKs will be inserted starting below AUTOUPDATE. When a CHECK has line information, autoupdate will try to insert the CHECK immediately next to the line it's associated with, with stderr CHECKs preceding the line and stdout CHECKs following the line. When that happens, any subsequent CHECK lines without line information, or that refer to lines appearing earlier, will immediately follow. As an exception, if no STDOUT check line refers to any line in the test, all STDOUT check lines are placed at the end of the file instead of immediately after AUTOUPDATE.

  • // ARGS: <arguments>

    Provides a space-separated list of arguments, which will be passed to RunWithFiles as test_args. These are intended for use by the command as arguments.

    Supported replacements within arguments are:

    • %s

      Replaced with the list of files. Currently only allowed as a standalone argument, not a substring.

    • %t

      Replaced with ${TEST_TMPDIR}/temp_file.

    ARGS can be specified at most once. If not provided, the FileTestBase child is responsible for providing default arguments.

  • // SET-CHECK-SUBSET

    By default, all lines of output must have a CHECK match. Adding this as a option sets it so that non-matching lines are ignored. All provided CHECK:STDOUT: and CHECK:STDERR: lines must still have a match in output.

    SET-CHECK-SUBSET can be specified at most once.

  • // --- <filename>

    By default, all file content is provided to the test as a single file in test_files. Using this marker allows the file to be split into multiple files which will all be passed to test_files.

    Files are not created on disk; it's expected the child will create an InMemoryFilesystem if needed.

  • // CHECK:STDOUT: <output line>
    // CHECK:STDERR: <output line>
    

    These provide a match for output from the command. See SET-CHECK-SUBSET for how to change from full to subset matching of output.

    Output line matchers may contain [[@LINE+offset] and {{regex}} syntaxes, similar to FileCheck.