Note that this will require changing the branch protections to use new
names for all of the checks and be somewhat disruptive. There aren't any
really good ways I could find of fixing this. Some options that I
explored:
- Have a single `pre-merge` workflow file that contains all of the other
workflows, splitting as much of the logic as we can into re-usable
files. This would basically merge testing, `pre-commit`, and
`clangd-tidy` checking into a single workflow file. However, it would
also delay the pre-commit suggestions action to only run once _all_ of
these finish, rather than as soon as pre-commit finishes.
- Serialize `pre-commit` and the rest of `pre-merge` to get the effect
of the above option but without the downside. Instead, the downside
would be serializing some of our actions.
- Have a single `pre-merge` workflow that triggers whenever any of the
other workflows completes, and have it check whether all the others have
completed. It will fail until it reaches that point. This requires
passing in GitHub keys to the workflow so that it can check the status
of other checks, and documentation online seems to indicate it is
sometimes flaky, I assume because of racing triggers of events or
check-status not being guaranteed consistent in the queries.
- Have a single `pre-merge` workflow that polls, waiting for all the
other workflows to finish using some Python logic. This requires
building and maintaining code to poll GitHub, keys to authorize that
polling, and handling all of the failure modes of a polling operation --
timeouts, network issues, etc.
Maybe there are others, but not sure what they look like. Suggestions
welcome here.
I'm hesitant to either delay the pre-commit suggestions or serialize
pre-commit execution. And the complexity or flakiness of the other two
options seem worse than having to re-work the branch protections each
time the naming here changes. But interested if folks think a different
direction would be better.
Assisted-by: Antigravity with Gemini
This also switches to a more Bazel-based install layout, skipping the
FHS-based synthetic layout. The FHS-based layout is still reconstructed
explicitly when building an installable tar-ball.
The biggest change is to configure the just-built install as a Bazel
toolchain, including allowing it to build its own runtime libraries as
native Bazel libraries. This removes the need for a monolithic runtimes
build, all of that code logic is removed.
This should also pave the way to using the just-built toolchain for
doing a full 3-stage bootstrap. Building the 2nd stage is included here
as it was a particularly effective way to test that the Bazel
integration was fully working. Adding a 3rd-stage check for stability is
future work, but should be pretty easy.
There is a down-side: this uses the busybox to do the runtimes
compilation, which means they will be re-built after ~any change to
Carbon. However, the integration with Bazel should largely pay for this,
and we can continue to factor the tests away from depending on built
runtimes in most cases.
Now that we're building and testing the runtimes more directly, this
surfaced a problem with the layout of runtimes on macOS that is fixed
here. All of the Darwin OSes use a custom layout for their resource
directory compared to other targets. We now model this in both the C++
built runtimes and the Bazel built runtimes.
Assisted-by: Gemini via Antigravity
In addition to the general updates, this switches to a required python
3.10 for pre-commit (3.9 is losing support from black).
Note endpoints for build actions are expanding significantly: see
https://app.stepsecurity.io/github/carbon-language/carbon-lang/actions/runs/22779388360?tab=recommendations&jobId=66080970460
for example, I think just the sources are being increased as a
side-effect of updates (and possibly also things not performing as well
as they should have before).
Similarly allowing sudo in pre-commit because it was actually causing
errors in part of build setup, which used sudo to remove files.
Assisted-by: Google Antigravity with Gemini
The Ubuntu 24 builders have a newer GLIBC than is present on the
compiler-explorer machines:
https://github.com/compiler-explorer/compiler-explorer/issues/7636#issuecomment-2880962252.
This results in the following error when running Carbon nightly:
```
/opt/compiler-explorer/carbon-trunk/bin/carbon: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by /opt/compiler-explorer/carbon-trunk/bin/carbon)
```
To resolve this, we need to build Carbon in a sysroot with a compatible
glibc version, and the most straightforward way to do that is to bump
our builders back down to Ubuntu 22.
To do that, we can't use apt.llvm.org again, since Ubuntu 22 is no
longer supported there. So we revert back to pulling a Linux X64 tarball
from the LLVM GitHub Releases page. Instead of getting an
ubuntu-specific tarball (which does not exist), we grab the generic
Linux one, which seems to work fine.
Note that the binaries in the LLVM release package appear to depend on
glibc version 2.34, as determined by `objdump -T bin/clang|grep GLIBC_|
sed 's/.*GLIBC_\([.0-9]*\).*/\1/g' | sort -Vu`, so these binaries should
hopefully be okay to package with the Carbon toolchain for the
compiler-explorer machines as well.
The version of clangd/clang-tidy on developer machines has slowly
diverged from the one on the CI builders, which is causing a slowly
increasing amount of pain as clang-tidy CI runs fail (incorrectly) over
things that a newer clangd/clang-tidy was perfectly fine with locally.
This bumps the Clang version used in the ubuntu builders to 19, which is
the most recent in Debian stable.
We use https://apt.llvm.org instead of LLVM's GitHub releases
(https://github.com/llvm/llvm-project/releases) as the former more
reliably has packages for newer Clang/LLVM versions on x64. The
community-build releases binaries on LLVM's GitHub have stopped
including Ubuntu packages that match the GitHub x64 Ubuntu workers for
some time (for at least the 18 and 19 releases).
By moving to apt.llvm.org packages we only download and install the
headers and libraries needed for development, rather than every output
of building llvm, which is much faster and saves lots of disk space. We
also remove the system installations of other versions of clang/llvm so
we should end up using negative disk space. We can no longer easily
cache the installation but apt.llvm.org is a reliable end point.
We bump the ubuntu image version for the github workers to 24.04, as
apt.llvm.org has stopped building images for 22.10 in 2022 at its end of
life.
The `pre_commit` workflow disabled sudo unlike the other workflows that
install Clang/LLVM, including the `clang-tidy` workflow (which is also
run on `pull_request`). We bring it into alignment with the other
workflows so that we can install the llvm packages. And we lock its
ubuntu image to 24.04 so that it can be moved in lockstep with the other
workflows that depend on Clang/LLVM.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Because clang-tidy is slow (and I'm not sure we can make it really
fast), trying to run it slightly less. Also, I noticed we can shave a
few minutes by disabling apt removal without losing too much free space.
Note that since this removes the old clang-tidy, I'll need to change the
branch protections before merging.
Also a small pass on workflow names.
Note, I'm a little concerned that the test/nightly release/pre-commit
endpoints may be fragile. At the same time, it's also where it may be
most useful, to prevent network access by arbitrary test code. I think
this is imperfect, but maybe we can try it out and see if it's much of
an issue.
Note, the discord wiki action is currently broken, this should fix it.
There is still a lot more we should try to do here, but this gets us
a decent start and should be enough for some of the developers working
on Carbon to use for simple cases, or tools like Compiler Explorer.
This first factors most of the complex setup for our CI workflow into
local actions that we can reuse in other workflows. It's actually
a fairly nice cleanup I think even without the specific use case of
a nightly release process. These use "composite" actions which is the
cleanest model for factoring out a subsequence of steps from a workflow.
I've tried to keep the inputs fairly minimal and easy to spot.
Next, it adds a cron-scheduled action that builds and creates a nightly
release. It also supports explicitly running the workflow, which can be
useful if the nightly release is broken in some way: a fix can be landed
and then a force run used.
Tested the `tests` workflow using our `action-test` branch:
https://github.com/carbon-language/carbon-lang/actions/runs/9545546278
Also tested the nightly release action by hacking in a `push` trigger
and using *draft* release. If you have admin rights, you can see an
example draft nightly build here:
https://github.com/carbon-language/carbon-lang/releases/tag/untagged-6a85cdce7b3a784a2e6b
Note that the GitHub releases are easily deleted, and even the tags
created in the repository can be cleaned up as needed so it should be
fairly harmless to fix this forward if needed.
This adds a defined Carbon version to the Bazel build and codebase that
can be used both to implement features like version checks and to report
a meaningful version on the command line. This replaces a hard-coded
string and a TODO in the driver.
As part of this, it adds support for defining the version in Bazel, and
special build flags for overriding relevant parts such as the
pre-release marker used. The exact structure and meaning of our version
string, including the pre-release parts, is implemented here in line
with the draft proposal:
https://docs.google.com/document/d/11S5VAPe5Pm_BZPlajWrqDDVr9qc7-7tS2VshqO0wWkk/edit?resourcekey=0-2YFC9Uvl4puuDnWlr2MmYw
This also introduces a workspace status command to the repository to
extract the git commit SHA and other information when building, and the
logic to stamp that into binaries as part of the version string when
useful. The technique used leverages weak symbols with whole archive
linking to allow a link-time override of unstamped data with stamped
data in the leaf executable. This makes building with `--stamp` a
reasonable default, especially for development builds. The CI system is
explicitly opted out of this as there it has no benefit.
Last but not least, all of these are wired into the install rules so
that we build installable packages with the version number in a
conventional place in the directory and filename.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This is coming out of advice from
https://securityscorecards.dev/viewer/?uri=github.com/carbon-language/carbon-lang
Updates action versions and pins by checksum. This is something we
already do in other places, and doing it with actions just seems
consistent.
Trying to add token permissions at the same time.
step-security/harden-runner is being set up to monitor network traffic
for actions, with the idea that we'd set it to block unexpected traffic
in the future.
Note, https://app.stepsecurity.io/secureworkflow generates the pinned
checksums and adds the harden-runner. In some cases it added token
permissions, but we have a couple it doesn't recognize (`gh` executions,
other custom commands, jlumbroso/free-disk-space, reviewdog/action-setup
as examples) so I'm trying to guess my best.
Because these are workflows, testing is limited.
This should also unblock our switch to C++20 and other improvements.
There are three core parts of the change --
1) Updating our infrastructure to fetch and find Clang-16.
2) Updating our documentation to reflect this and help folks with any
system issues they encounter.
The infrastructure change is unfortunately tricky. We can't get Clang 16
easily on GitHub's runner images, and in the past we've had persistent
problems with flakiness when our actions download this much during their
runs. Due to the flakiness, we've previously removed all downloading of
dependencies outside of Bazel itself, and added retry loops around Bazel
specifically to overcome flaky downloads.
This change tries to address these problems by populating the Clang and
LLVM toolchain in a place that we can then cache using the built-in
GitHub action caching infrastructure. This seems like by far the least
likely to flake way of downloading extra things into our runs. And since
these are relatively slow moving dependencies, we should populate this
cache very, very rarely.
For Linux, this downloads the binary release artifact from GitHub,
prunes out large parts of it that we don't need, and then caches this as
a local toolchain. This proves both small and fast.
For macOS, this uses a trick to cache the destination of Homebrew
installs. It unfortunately caches the *entire* Homebrew installation
though, and so it also goes to some lengths to prune and minimize how
much is installed from Homebrew. The result is "only" a 2gb cache image.
Because of the size and slower download and filesystem, the macOS runs
see a 1 - 2 minute slowdown.
We might extend the Linux infrastructure here usefully if we want to
test multiple LLVM versions. We might also extend the macOS version to
get a cheaper way to prune parts of the system and free up disk space,
or to cache other Homebrew installed tools if needed.
Last but not least, this brought to the forefront an issue with our C++
toolchain integration which relied on a specific CMake build option
being set in the LLVM toolchain install. This option isn't used in the
official release artifacts. Instead, switch to a more robust approach to
linking libc++abi statically that shouldn't have these problems.
Beyond the infrastructure changes, this also updates the documentation
to reflect requiring Clang 16 or newer, and adds some extra tips for
folks that are missing this.
The documentation is also updated to address a problem with getting the
right libc++abi files installed to support the more robust linking
strategy. This may reduce the problems we've seen in the past around
libc++abi and linking on other Linux distros as well.
---------
Co-authored-by: josh11b <josh11b@users.noreply.github.com>
This captures the exit code of Bazel and checks for success or permanent
errors on each attempt. It also sleeps a small amount between attempts.
We should be able to increase the retries and sleeps as needed to
minimize flakiness here, and Bazel should even persist incremental
progress efficiently. Hopefully this helps reduce the failure rate of
our CI.
It also changes how we build on a `push` to use a single Bazel clause to
hold this logic.
Managed to get one of the download failures when testing this, and the
retry logic worked but there was a bug in the success logic.
Otherwise seems to work:
- Synthetic failure:
https://github.com/carbon-language/carbon-lang/actions/runs/6872431707/job/18690894069
- Success:
https://github.com/carbon-language/carbon-lang/actions/runs/6872461061
---------
Co-authored-by: josh11b <josh11b@users.noreply.github.com>
We're seeing issues building on the 20230911.1.0 release. I think there
may be a misconfiguration; using `/usr/lib/llvm-15/bin/clang++`, I'm
getting errors such as:
```
external/com_google_absl/absl/base/config.h:56:10: fatal error: 'cstddef' file not found
#include <cstddef>
^~~~~~~~~
```
e.g.:
https://github.com/carbon-language/carbon-lang/actions/runs/6166275499/job/16735504697
(the passing ubuntu run is on a 20230903.1.0 image using llvm-14)
(note if this PR fails testing, it needs to be merged before it would
take effect)
Reported on https://github.com/actions/runner-images/issues/8253
The expensive local actions will be separately gated to not overwhelm
the machine, and currently highly asynchronous actions are a dominant
part of our builds due to downloading cached artifacts. Without a high
concurrency, these are downloaded roughly 2-at-a-time currently.
I've verified that on a small machine without a good cache this doesn't
seem to generate huge amounts of work and local build and test actions
are successfully gated on the local flags.
There is already a Bazel issue tracking this limitation:
https://github.com/bazelbuild/bazel/issues/6394
At present, it looks like tools consume 3GB and fastbuild consumes 20GB
versus 23GB available. We've apparently been on the edge of this, and
are just now getting pushed over.
https://github.com/carbon-language/carbon-lang/actions/runs/5625169220/job/15243450729
shows what this looks like (review actions are run from trunk versions,
not this branch).
Trying to fix merge queue support for PRs that don't need tests, #2971
didn't work because it lacks the matrix configuration. This takes a
different approach from that, instead filtering each step based on
affected paths. This way we're also less likely to see skew in behavior.
We keep seeing fragility installing software, with both brew (e.g., the recent python issues) and apt.llvm.org (currently flaky).
At this point, images for both ubuntu and macos have versions of llvm-14 that seem to successfully compile:
https://github.com/carbon-language/carbon-lang/actions/runs/3474670746/jobs/5808098087
Although we may want to figure out a way to resume running llvm-15 so that we can see compatibility issues, this seems preferable for baseline testing in order to reduce maintenance churn.
In addition to the above changes, this also configures cancellation more precisely, and stops installing bazel/bazelisk (it should already be preinstalled).
We've been having issues with asan builds on linux. This change should fix all of that. A build run can be found at:
https://github.com/carbon-language/carbon-lang/actions/runs/3093378863/jobs/5005683059
(currently in progress, but I'm expecting it to succeed at this point)
It may be that the issues with asan builds were actually related to caching. That is, maybe the brew build command didn't change enough between v14 and v15 that the cache hits were still an issue. We did notice this with 15.0.0 versus 15.0.1 include paths (that is, bazel wasn't happy using the cached results of a 15.0.0 build due to the skew in include paths). In order to address this, I've added CACHE_VERSION to the remote_cache setup. I've also set up corresponding buckets in Cloud.
However, I'm also switching Linux to llvm-15 and apt. I'd originally been looking at this because the issues were linux-specific, and we've previously had linux-specific issues with Homebrew. Although it may have been the cache all along, I would prefer to keep this setup (if nothing else, it made the caching issues more obvious, even though we were still confused by the include path manifestation).
This was noticed because macos stopped installing the brew llvm, and they don't have clang-format. However, our tests don't actually need clang-format (and even if they did, we should probably match the pre-commit version), so it seems superfluous to check for.
Keeping brew because there's an advantage to consistency on tool versions, just running it on all platforms instead of linux-only.
- brew instead of pyenv to centralize package management
- can't recall why we used pyenv before, may not have been the best choice.
- pyenv has been a burden in updating versions, a single `brew upgrade` works better
- python3.9 because it's the latest and greatest, feels weird going to old versions if we're recommending installs.
- bazel 5.0.0 pre-release due to https://github.com/bazelbuild/bazel/commit/b9fc66d327debcfbdb2964afdba35a1cc8919b81
- earlier versions want `python` to be on the path, which pyenv did but brew does not
For most people this will mean: `rm -rf ~/.pyenv && brew uninstall pyenv && brew install python@3.9 && pip3 install -U pip && pip3 install pre-commit`, plus removing any `pyenv` invocations from the shell `rc` file.
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
The built-in GitHub action cache is too broken for us to use currently.
Pull request runs will evict trunk runs from the cache easily, which
will remove a useful baseline and make builds generally uncached and
extremely slow.
Instead, move to using Bazel's remote caching functionality with a GCP
storage bucket. This gives us nearly limitless storage and is actually
vastly simpler than the prior arrangement. It should also allow cache
sharing between different build configurations and other benefits.
This cache isn't usable by anyone else sadly, but it should at least
provide a better scaling technique for our CI.
This does require us to move from `pull_request` workflow to
a `pull_request_target` workflow and run the CI within the domain of the
project. This is only really safe for us to do while the project remains
private. When we go public we'll want to make the cache publicly
readable and change this code so that when not running on a branch in
the main repository we use the cache in a read-only mode and only write
back updates for branch runs. This should still be effective but
requires a bit more complexity here and in the remote cache including
setting up public access. For now, going with the simple if slightly
less secure model.
Chandler Carruth
Jon Ross-Perkins
Dana Jansens
Richard Smith
Adrien Leravat
SADIK KUZU
Jon Meow