Files
carbon-lang/scripts/target_determinator.py
T
2751f02258 Introduce git-based target selection. (#3106)
This adds access to the `target-determinator` tool that computes the
possibly impacted set of targets between the current checkout and a
specific `git` commit. The tool is downloaded (and cached) with a Python
script that wraps it and allows us to easily run it in our CI
environment.

And finally, switches the CI for pull requests and the merge queue to
use this script and run a minimal set of impacted tests rather than all
of them. In order to make this as simple as possible, this also merges
the previously separate build and test steps of our CI.

Note that the CI testing post-submit (push events) continues to use a
blanket target pattern so that we have a good backstop in case something
outside of what is tracked here changes. The important paths, especially
the ones that might be in an interactive critical path, are the PR and
merge queue.

This has been tested on the `action-test` branch to try to make sure it
works. Some example runs for folks to inspect:

- `push` event with blanket test:
https://github.com/carbon-language/carbon-lang/actions/runs/6070060958/job/16465422569
- A "large" PR that impacts a bunch of toolchain tests:
- ubuntu-opt:
https://github.com/carbon-language/carbon-lang/actions/runs/6070063050/job/16465428107?pr=3188
- macos-opt:
https://github.com/carbon-language/carbon-lang/actions/runs/6070063050/job/16465428420?pr=3188
- The merge queue for this PR:
- ubuntu-opt:
https://github.com/carbon-language/carbon-lang/actions/runs/6070202665/job/16465825387
- macos-opt:
https://github.com/carbon-language/carbon-lang/actions/runs/6070202665/job/16465825740
- A "small" PR that impacts one test:
- ubuntu-opt:
https://github.com/carbon-language/carbon-lang/actions/runs/6070659668/job/16467147641?pr=3189
- macos-opt:
https://github.com/carbon-language/carbon-lang/actions/runs/6070659668/job/16467147933?pr=3189
- The merge queue for the small PR:
- ubuntu-opt:
https://github.com/carbon-language/carbon-lang/actions/runs/6070754584/job/16467427268
- macos-opt:
https://github.com/carbon-language/carbon-lang/actions/runs/6070754584/job/16467427736

I have observed one failure while testing with this PR, but I've not yet
been able to reproduce it. Every other failure (including empty lists,
etc.) I've tried to address. However, we may have to keep an eye on
actions after this to make sure there isn't some scenario I've not been
able to work through in a test branch.

---------

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Geoff Romer <gromer@google.com>
2023-09-08 00:18:50 +00:00

103 lines
2.8 KiB
Python
Executable File

#!/usr/bin/env python3
"""Computes the potentially differing rules from some git commit.
Wraps the "target-determinator" Go program here:
https://github.com/bazel-contrib/target-determinator
The purpose is to compute the potentially impacted set of targets from some
provided Git commit to the current checkout.
This script will ensure a cached version of the latest release is available, and
then forward a limited set of flags to it. This script also filters the
resulting targets using `bazel query` to make it the most relevant list for
continuous integration.
"""
__copyright__ = """
Part of the Carbon Language project, under the Apache License v2.0 with LLVM
Exceptions. See /LICENSE for license information.
SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
"""
import subprocess
import argparse
import tempfile
import sys
from pathlib import Path
import scripts_utils
def log(s: str) -> None:
print(s, file=sys.stderr)
def filter_targets(bazel: Path, targets: str) -> str:
with tempfile.NamedTemporaryFile(mode="w+") as tmp:
query = (
f"let t = set({targets}) in "
"kind(rule, $t) except attr(tags, manual, $t)\n"
)
query_lines = query.splitlines()
if len(query_lines) <= 10:
query_snippet = "\n".join(query_lines)
else:
query_snippet = "\n".join(
query_lines[:5] + ["..."] + query_lines[-5:]
)
log(f"Bazel query snippet:\n```\n{query_snippet}\n```")
tmp.write(query)
try:
p = subprocess.run(
[str(bazel), "query", f"--query_file={tmp.name}"],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
check=True,
encoding="utf-8",
)
return p.stdout
except subprocess.CalledProcessError as err:
log(err.stderr)
raise
def main() -> None:
parser = argparse.ArgumentParser(__doc__)
parser.add_argument(
"baseline", nargs=1, help="Git commit of the diff baseline."
)
parser.add_argument(
"args",
nargs="*",
help="Remaining args to forward to the underlying tool.",
)
parsed_args = parser.parse_args()
scripts_utils.chdir_repo_root()
bazel = Path(scripts_utils.locate_bazel())
target_determinator = scripts_utils.get_target_determinator()
p = subprocess.run(
[
target_determinator,
f"--bazel={bazel}",
parsed_args.baseline[0],
]
+ parsed_args.args,
check=True,
stdout=subprocess.PIPE,
encoding="utf-8",
)
targets = p.stdout
if targets.strip() != "":
targets = filter_targets(bazel, targets)
log(f"Found {len(targets.splitlines())} impacted targets!")
print(targets.rstrip())
if __name__ == "__main__":
main()