The `--dump-cpp-ast` file tests strip references to Clang builtins so
that the expected output is target-independent. The filter anchored a
`__`-prefixed builtin identifier on a preceding space or quote, which
matches the x86-64 `__va_list_tag` spelling but not the AArch64
`std::__va_list`, where `__` is preceded by the `::` namespace
qualifier. That left a single `RecordType 'std::__va_list'` line
unfiltered on AArch64, producing a spurious autoupdate diff for
`thunk_ast.carbon`.
Anchor the match on a preceding `:` as well so namespace-qualified
builtins are also filtered.
Carbon's test workflow covered Linux on x86-64 and macOS on AArch64, but
had no Linux AArch64 coverage, so AArch64-specific issues that don't
reproduce on macOS could land unnoticed. Add an `ubuntu-22.04-arm`
runner to the matrix.
The release used for Linux does not publish the monolithic
`LLVM-*-Linux-ARM64` package, only a `clang+llvm-*-aarch64-linux-gnu`
community build with a smaller tool set, so the Ubuntu setup now selects
the tarball by `runner.arch`. The prune step uses `rm -f` since the two
packages do not ship an identical set of tools to remove.
Assisted-by: Claude Code with Claude Opus 4.7
This removes the need to install any specific version of Python or
figure out how to configure it by instead asking users to install `uv`
and letting it manage Python. Among other advantages, `uv` is designed
to be fast enough to embed directly into our scripts.
We were already using this in `bench_runner.py` so that the script could
import non standard library dependencies. Moving to it for the rest of
our Python unifies the approach and will also enable dependencies
whenever needed.
I've left `github_tools` alone as it has special handling with its own
Bazel setup.
I've updated the contributing tools to explain the approach here.
This also switches to a more Bazel-based install layout, skipping the
FHS-based synthetic layout. The FHS-based layout is still reconstructed
explicitly when building an installable tar-ball.
The biggest change is to configure the just-built install as a Bazel
toolchain, including allowing it to build its own runtime libraries as
native Bazel libraries. This removes the need for a monolithic runtimes
build, all of that code logic is removed.
This should also pave the way to using the just-built toolchain for
doing a full 3-stage bootstrap. Building the 2nd stage is included here
as it was a particularly effective way to test that the Bazel
integration was fully working. Adding a 3rd-stage check for stability is
future work, but should be pretty easy.
There is a down-side: this uses the busybox to do the runtimes
compilation, which means they will be re-built after ~any change to
Carbon. However, the integration with Bazel should largely pay for this,
and we can continue to factor the tests away from depending on built
runtimes in most cases.
Now that we're building and testing the runtimes more directly, this
surfaced a problem with the layout of runtimes on macOS that is fixed
here. All of the Darwin OSes use a custom layout for their resource
directory compared to other targets. We now model this in both the C++
built runtimes and the Bazel built runtimes.
Assisted-by: Gemini via Antigravity
In addition to the general updates, this switches to a required python
3.10 for pre-commit (3.9 is losing support from black).
Note endpoints for build actions are expanding significantly: see
https://app.stepsecurity.io/github/carbon-language/carbon-lang/actions/runs/22779388360?tab=recommendations&jobId=66080970460
for example, I think just the sources are being increased as a
side-effect of updates (and possibly also things not performing as well
as they should have before).
Similarly allowing sudo in pre-commit because it was actually causing
errors in part of build setup, which used sudo to remove files.
Assisted-by: Google Antigravity with Gemini
The Ubuntu 24 builders have a newer GLIBC than is present on the
compiler-explorer machines:
https://github.com/compiler-explorer/compiler-explorer/issues/7636#issuecomment-2880962252.
This results in the following error when running Carbon nightly:
```
/opt/compiler-explorer/carbon-trunk/bin/carbon: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by /opt/compiler-explorer/carbon-trunk/bin/carbon)
```
To resolve this, we need to build Carbon in a sysroot with a compatible
glibc version, and the most straightforward way to do that is to bump
our builders back down to Ubuntu 22.
To do that, we can't use apt.llvm.org again, since Ubuntu 22 is no
longer supported there. So we revert back to pulling a Linux X64 tarball
from the LLVM GitHub Releases page. Instead of getting an
ubuntu-specific tarball (which does not exist), we grab the generic
Linux one, which seems to work fine.
Note that the binaries in the LLVM release package appear to depend on
glibc version 2.34, as determined by `objdump -T bin/clang|grep GLIBC_|
sed 's/.*GLIBC_\([.0-9]*\).*/\1/g' | sort -Vu`, so these binaries should
hopefully be okay to package with the Carbon toolchain for the
compiler-explorer machines as well.
The version of clangd/clang-tidy on developer machines has slowly
diverged from the one on the CI builders, which is causing a slowly
increasing amount of pain as clang-tidy CI runs fail (incorrectly) over
things that a newer clangd/clang-tidy was perfectly fine with locally.
This bumps the Clang version used in the ubuntu builders to 19, which is
the most recent in Debian stable.
We use https://apt.llvm.org instead of LLVM's GitHub releases
(https://github.com/llvm/llvm-project/releases) as the former more
reliably has packages for newer Clang/LLVM versions on x64. The
community-build releases binaries on LLVM's GitHub have stopped
including Ubuntu packages that match the GitHub x64 Ubuntu workers for
some time (for at least the 18 and 19 releases).
By moving to apt.llvm.org packages we only download and install the
headers and libraries needed for development, rather than every output
of building llvm, which is much faster and saves lots of disk space. We
also remove the system installations of other versions of clang/llvm so
we should end up using negative disk space. We can no longer easily
cache the installation but apt.llvm.org is a reliable end point.
We bump the ubuntu image version for the github workers to 24.04, as
apt.llvm.org has stopped building images for 22.10 in 2022 at its end of
life.
The `pre_commit` workflow disabled sudo unlike the other workflows that
install Clang/LLVM, including the `clang-tidy` workflow (which is also
run on `pull_request`). We bring it into alignment with the other
workflows so that we can install the llvm packages. And we lock its
ubuntu image to 24.04 so that it can be moved in lockstep with the other
workflows that depend on Clang/LLVM.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Trying to improve resilience against failures such as:
```
INFO: Repository rules_jvm_external+ instantiated at:
<builtin>: in <toplevel>
Repository rule http_archive defined at:
/home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/bazel_tools/tools/build_defs/repo/http.bzl:392:31: in <toplevel>
ERROR: /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/bazel_tools/tools/build_defs/repo/http.bzl:137:45: An error occurred during the fetch of repository 'rules_jvm_external+':
Traceback (most recent call last):
File "/home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/bazel_tools/tools/build_defs/repo/http.bzl", line 137, column 45, in _http_archive_impl
download_info = ctx.download_and_extract(
```
https://github.com/carbon-language/carbon-lang/actions/runs/14719625211/job/41311145495?pr=5379
It looks like GitHub currently has a high rate of these, which it
shouldn't, but also maybe we can do a little more to weather these
service issues.
To show flag behavior:
```
╚╡./scripts/run_bazel.py --attempts=5 --retry-all-errors :foo
Command ':foo' not found. Try 'bazel help'.
Retrying exit code 2 because it may be transient...
Command ':foo' not found. Try 'bazel help'.
Retrying exit code 2 because it may be transient...
Command ':foo' not found. Try 'bazel help'.
Retrying exit code 2 because it may be transient...
Command ':foo' not found. Try 'bazel help'.
Retrying exit code 2 because it may be transient...
Command ':foo' not found. Try 'bazel help'.
╚╡./scripts/run_bazel.py --attempts=5 --retry-all-errors query //... | wc -l
INFO: Invocation ID: ffdf9480-0245-442d-885f-ee91a3d86b68
Loading: 0 packages loaded
367
╚╡./scripts/run_bazel.py --attempts=5 :foo
Command ':foo' not found. Try 'bazel help'.
```
On the last run, [test (ubuntu-22.04,
opt)](https://github.com/carbon-language/carbon-lang/actions/runs/14737410033/job/41366888473?pr=5386)
has an example of this working:
```
INFO: Invocation ID: 3f276297-6dc5-4007-a333-dcadd4db55f4
no actions running
no actions running
no actions running
no actions running
no actions running
no actions running
no actions running
no actions running
no actions running
WARNING: Download from https://github.com/bazelbuild/bazel-skylib/releases/download/1.7.1/bazel-skylib-1.7.1.tar.gz failed: class java.io.IOException GET returned 618 jwt:jwt-not-provided
INFO: Repository bazel_skylib+ instantiated at:
<builtin>: in <toplevel>
Repository rule http_archive defined at:
/home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/bazel_tools/tools/build_defs/repo/http.bzl:392:31: in <toplevel>
ERROR: /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/bazel_tools/tools/build_defs/repo/http.bzl:137:45: An error occurred during the fetch of repository 'bazel_skylib+':
Traceback (most recent call last):
File "/home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/bazel_tools/tools/build_defs/repo/http.bzl", line 137, column 45, in _http_archive_impl
download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://github.com/bazelbuild/bazel-skylib/releases/download/1.7.1/bazel-skylib-1.7.1.tar.gz] to /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/bazel_skylib+/temp62784[869](https://github.com/carbon-language/carbon-lang/actions/runs/14737410033/job/41366888473?pr=5386#step:5:887)09382546624/bazel-skylib-1.7.1.tar.gz: GET returned 618 jwt:jwt-not-provided
no actions running
no actions running
ERROR: Error loading '@@rules_python+//python/extensions:python.bzl' for module extensions, requested by /home/runner/work/carbon-lang/carbon-lang/MODULE.bazel:147:23: at /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/rules_python+/python/extensions/python.bzl:48:6: at /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/rules_python+/python/private/python.bzl:17:6: at /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/bazel_features+/features.bzl:3:6: Encountered error while reading extension file 'globals.bzl': no such package '@@bazel_features++version_extension+bazel_features_globals//': no such package '@@bazel_skylib+//lib': java.io.IOException: Error downloading [https://github.com/bazelbuild/bazel-skylib/releases/download/1.7.1/bazel-skylib-1.7.1.tar.gz] to /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/bazel_skylib+/temp6278486909382546624/bazel-skylib-1.7.1.tar.gz: GET re
ERROR: Error loading '@@rules_cc+//cc:extensions.bzl' for module extensions, requested by https://bcr.bazel.build/modules/rules_cc/0.1.1/MODULE.bazel:12:29: at /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/rules_cc+/cc/extensions.bzl:16:6: at /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/bazel_features+/features.bzl:3:6: Encountered error while reading extension file 'globals.bzl': no such package '@@bazel_features++version_extension+bazel_features_globals//': no such package '@@bazel_skylib+//lib': java.io.IOException: Error downloading [https://github.com/bazelbuild/bazel-skylib/releases/download/1.7.1/bazel-skylib-1.7.1.tar.gz] to /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/bazel_skylib+/temp6278486909382546624/bazel-skylib-1.7.1.tar.gz: GET returned 618 jwt:jwt-not-provided: at /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/rules_cc+/cc/extensions.bzl:16:6: at /h
ERROR: Error loading '@@rules_python+//python/extensions:python.bzl' for module extensions, requested by /home/runner/work/carbon-lang/carbon-lang/MODULE.bazel:147:23: at /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/rules_python+/python/extensions/python.bzl:48:6: at /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/rules_python+/python/private/python.bzl:17:6: at /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/bazel_features+/features.bzl:3:6: Encountered error while reading extension file 'globals.bzl': no such package '@@bazel_features++version_extension+bazel_features_globals//': no such package '@@bazel_skylib+//lib': java.io.IOException: Error downloading [https://github.com/bazelbuild/bazel-skylib/releases/download/1.7.1/bazel-skylib-1.7.1.tar.gz] to /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/bazel_skylib+/temp6278486909382546624/bazel-skylib-1.7.1.tar.gz: GET re
INFO: Invocation ID: e2062912-715f-47ec-a0bc-9d1b4fee9e5d
no actions running
no actions running
no actions running
no actions running
no actions running
no actions running
<root> (carbon@_)
Retrying a failure because it may be transient...
INFO: Invocation ID: 7d4ee250-882f-480d-9c8d-2d92e67462fc
Loading: 0 packages loaded
367
```
This is trying to reduce how much we run tidy over due to poor
performance of tidy.
It's opening the door for issues where a header file is modified in a
way that introduces tidy issues in a different target. However, that's
not the typical tidy issue we see.
There may also be a risk where a source file indirectly becomes a source
file for a target in a way that `same_pkg_direct_rdeps` doesn't return,
but I'm not sure that's applicable for how we use targets.
An example of this locating a diagnostic error can be found in the "Add
tidy issue" commit's run (which I cancelled before it finished running,
but note the error):
https://github.com/carbon-language/carbon-lang/actions/runs/13932649182/job/38993348130?pr=5144
Trying to improve robustness against failures such as
[here](https://github.com/carbon-language/carbon-lang/actions/runs/11845529962/job/33011203628):
```
WARNING: Download from https://ftp.gnu.org/gnu/m4/m4-1.4.18.tar.xz failed: class java.net.ConnectException Connection refused
```
Which is coming from the target-determinator invocation:
```
subprocess.CalledProcessError: Command '['/home/runner/.cache/carbon-lang-scripts/target-determinator', '--bazel=/usr/local/bin/bazelisk', 'a720921dfb99c21f08832663c91a1f9e48a1bcc4']' returned non-zero exit status 1.
```
My thought is that the `bazel cquery` should trigger equivalent
downloads (equally though, I'd thought the `bazel mod deps` would do
that, so I don't want to give the impression of confidence).
It's harder to inject this into the target-determinator command line,
since we'd need something that retries by default.
I expect the change here will try rerunning a couple times, then fail if
it's a permanent failure.
I've seen similar failures, and believe this is flaky. Here's the
specific example that caused me to try a workaround:
https://github.com/carbon-language/carbon-lang/actions/runs/11560505812/job/32177497296
I don't see a difference in the runner information between failing and
passing runs, which might've indicated a canary. That's why I'm doing
this as a trivial retry.
Because clang-tidy is slow (and I'm not sure we can make it really
fast), trying to run it slightly less. Also, I noticed we can shave a
few minutes by disabling apt removal without losing too much free space.
Note that since this removes the old clang-tidy, I'll need to change the
branch protections before merging.
This should free up quite a bit of space and even make our builds faster
by avoiding duplicating every output. The syntax for this flag is
counter intuitive, so I've left a comment explaining it.
There is still a lot more we should try to do here, but this gets us
a decent start and should be enough for some of the developers working
on Carbon to use for simple cases, or tools like Compiler Explorer.
This first factors most of the complex setup for our CI workflow into
local actions that we can reuse in other workflows. It's actually
a fairly nice cleanup I think even without the specific use case of
a nightly release process. These use "composite" actions which is the
cleanest model for factoring out a subsequence of steps from a workflow.
I've tried to keep the inputs fairly minimal and easy to spot.
Next, it adds a cron-scheduled action that builds and creates a nightly
release. It also supports explicitly running the workflow, which can be
useful if the nightly release is broken in some way: a fix can be landed
and then a force run used.
Tested the `tests` workflow using our `action-test` branch:
https://github.com/carbon-language/carbon-lang/actions/runs/9545546278
Also tested the nightly release action by hacking in a `push` trigger
and using *draft* release. If you have admin rights, you can see an
example draft nightly build here:
https://github.com/carbon-language/carbon-lang/releases/tag/untagged-6a85cdce7b3a784a2e6b
Note that the GitHub releases are easily deleted, and even the tags
created in the repository can be cleaned up as needed so it should be
fairly harmless to fix this forward if needed.
Dana Jansens
Chandler Carruth
Jon Ross-Perkins