Generally seems to be working as intended: clang-tidy has variance from
a few minutes to an hour; clang-tidy hovers around 10 minutes. In this
case, the long tail of slow execution is more visible, partly because
tests will often take close to 10 minutes, if not more.
Branch enforcement should already be switched.
Currently it's called `assign_reviewers` as it was copied from the
workflow that does said task. But this workflow is setting labels, so
call it `set_labels`.
The Ubuntu 24 builders have a newer GLIBC than is present on the
compiler-explorer machines:
https://github.com/compiler-explorer/compiler-explorer/issues/7636#issuecomment-2880962252.
This results in the following error when running Carbon nightly:
```
/opt/compiler-explorer/carbon-trunk/bin/carbon: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by /opt/compiler-explorer/carbon-trunk/bin/carbon)
```
To resolve this, we need to build Carbon in a sysroot with a compatible
glibc version, and the most straightforward way to do that is to bump
our builders back down to Ubuntu 22.
To do that, we can't use apt.llvm.org again, since Ubuntu 22 is no
longer supported there. So we revert back to pulling a Linux X64 tarball
from the LLVM GitHub Releases page. Instead of getting an
ubuntu-specific tarball (which does not exist), we grab the generic
Linux one, which seems to work fine.
Note that the binaries in the LLVM release package appear to depend on
glibc version 2.34, as determined by `objdump -T bin/clang|grep GLIBC_|
sed 's/.*GLIBC_\([.0-9]*\).*/\1/g' | sort -Vu`, so these binaries should
hopefully be okay to package with the Carbon toolchain for the
compiler-explorer machines as well.
The version of clangd/clang-tidy on developer machines has slowly
diverged from the one on the CI builders, which is causing a slowly
increasing amount of pain as clang-tidy CI runs fail (incorrectly) over
things that a newer clangd/clang-tidy was perfectly fine with locally.
This bumps the Clang version used in the ubuntu builders to 19, which is
the most recent in Debian stable.
We use https://apt.llvm.org instead of LLVM's GitHub releases
(https://github.com/llvm/llvm-project/releases) as the former more
reliably has packages for newer Clang/LLVM versions on x64. The
community-build releases binaries on LLVM's GitHub have stopped
including Ubuntu packages that match the GitHub x64 Ubuntu workers for
some time (for at least the 18 and 19 releases).
By moving to apt.llvm.org packages we only download and install the
headers and libraries needed for development, rather than every output
of building llvm, which is much faster and saves lots of disk space. We
also remove the system installations of other versions of clang/llvm so
we should end up using negative disk space. We can no longer easily
cache the installation but apt.llvm.org is a reliable end point.
We bump the ubuntu image version for the github workers to 24.04, as
apt.llvm.org has stopped building images for 22.10 in 2022 at its end of
life.
The `pre_commit` workflow disabled sudo unlike the other workflows that
install Clang/LLVM, including the `clang-tidy` workflow (which is also
run on `pull_request`). We bring it into alignment with the other
workflows so that we can install the llvm packages. And we lock its
ubuntu image to 24.04 so that it can be moved in lockstep with the other
workflows that depend on Clang/LLVM.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Trying to improve resilience against failures such as:
```
INFO: Repository rules_jvm_external+ instantiated at:
<builtin>: in <toplevel>
Repository rule http_archive defined at:
/home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/bazel_tools/tools/build_defs/repo/http.bzl:392:31: in <toplevel>
ERROR: /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/bazel_tools/tools/build_defs/repo/http.bzl:137:45: An error occurred during the fetch of repository 'rules_jvm_external+':
Traceback (most recent call last):
File "/home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/bazel_tools/tools/build_defs/repo/http.bzl", line 137, column 45, in _http_archive_impl
download_info = ctx.download_and_extract(
```
https://github.com/carbon-language/carbon-lang/actions/runs/14719625211/job/41311145495?pr=5379
It looks like GitHub currently has a high rate of these, which it
shouldn't, but also maybe we can do a little more to weather these
service issues.
To show flag behavior:
```
╚╡./scripts/run_bazel.py --attempts=5 --retry-all-errors :foo
Command ':foo' not found. Try 'bazel help'.
Retrying exit code 2 because it may be transient...
Command ':foo' not found. Try 'bazel help'.
Retrying exit code 2 because it may be transient...
Command ':foo' not found. Try 'bazel help'.
Retrying exit code 2 because it may be transient...
Command ':foo' not found. Try 'bazel help'.
Retrying exit code 2 because it may be transient...
Command ':foo' not found. Try 'bazel help'.
╚╡./scripts/run_bazel.py --attempts=5 --retry-all-errors query //... | wc -l
INFO: Invocation ID: ffdf9480-0245-442d-885f-ee91a3d86b68
Loading: 0 packages loaded
367
╚╡./scripts/run_bazel.py --attempts=5 :foo
Command ':foo' not found. Try 'bazel help'.
```
On the last run, [test (ubuntu-22.04,
opt)](https://github.com/carbon-language/carbon-lang/actions/runs/14737410033/job/41366888473?pr=5386)
has an example of this working:
```
INFO: Invocation ID: 3f276297-6dc5-4007-a333-dcadd4db55f4
no actions running
no actions running
no actions running
no actions running
no actions running
no actions running
no actions running
no actions running
no actions running
WARNING: Download from https://github.com/bazelbuild/bazel-skylib/releases/download/1.7.1/bazel-skylib-1.7.1.tar.gz failed: class java.io.IOException GET returned 618 jwt:jwt-not-provided
INFO: Repository bazel_skylib+ instantiated at:
<builtin>: in <toplevel>
Repository rule http_archive defined at:
/home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/bazel_tools/tools/build_defs/repo/http.bzl:392:31: in <toplevel>
ERROR: /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/bazel_tools/tools/build_defs/repo/http.bzl:137:45: An error occurred during the fetch of repository 'bazel_skylib+':
Traceback (most recent call last):
File "/home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/bazel_tools/tools/build_defs/repo/http.bzl", line 137, column 45, in _http_archive_impl
download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://github.com/bazelbuild/bazel-skylib/releases/download/1.7.1/bazel-skylib-1.7.1.tar.gz] to /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/bazel_skylib+/temp62784[869](https://github.com/carbon-language/carbon-lang/actions/runs/14737410033/job/41366888473?pr=5386#step:5:887)09382546624/bazel-skylib-1.7.1.tar.gz: GET returned 618 jwt:jwt-not-provided
no actions running
no actions running
ERROR: Error loading '@@rules_python+//python/extensions:python.bzl' for module extensions, requested by /home/runner/work/carbon-lang/carbon-lang/MODULE.bazel:147:23: at /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/rules_python+/python/extensions/python.bzl:48:6: at /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/rules_python+/python/private/python.bzl:17:6: at /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/bazel_features+/features.bzl:3:6: Encountered error while reading extension file 'globals.bzl': no such package '@@bazel_features++version_extension+bazel_features_globals//': no such package '@@bazel_skylib+//lib': java.io.IOException: Error downloading [https://github.com/bazelbuild/bazel-skylib/releases/download/1.7.1/bazel-skylib-1.7.1.tar.gz] to /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/bazel_skylib+/temp6278486909382546624/bazel-skylib-1.7.1.tar.gz: GET re
ERROR: Error loading '@@rules_cc+//cc:extensions.bzl' for module extensions, requested by https://bcr.bazel.build/modules/rules_cc/0.1.1/MODULE.bazel:12:29: at /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/rules_cc+/cc/extensions.bzl:16:6: at /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/bazel_features+/features.bzl:3:6: Encountered error while reading extension file 'globals.bzl': no such package '@@bazel_features++version_extension+bazel_features_globals//': no such package '@@bazel_skylib+//lib': java.io.IOException: Error downloading [https://github.com/bazelbuild/bazel-skylib/releases/download/1.7.1/bazel-skylib-1.7.1.tar.gz] to /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/bazel_skylib+/temp6278486909382546624/bazel-skylib-1.7.1.tar.gz: GET returned 618 jwt:jwt-not-provided: at /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/rules_cc+/cc/extensions.bzl:16:6: at /h
ERROR: Error loading '@@rules_python+//python/extensions:python.bzl' for module extensions, requested by /home/runner/work/carbon-lang/carbon-lang/MODULE.bazel:147:23: at /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/rules_python+/python/extensions/python.bzl:48:6: at /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/rules_python+/python/private/python.bzl:17:6: at /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/bazel_features+/features.bzl:3:6: Encountered error while reading extension file 'globals.bzl': no such package '@@bazel_features++version_extension+bazel_features_globals//': no such package '@@bazel_skylib+//lib': java.io.IOException: Error downloading [https://github.com/bazelbuild/bazel-skylib/releases/download/1.7.1/bazel-skylib-1.7.1.tar.gz] to /home/runner/.cache/bazel/_bazel_runner/8f839eaeb716f9d034eabdfa7ebecdb0/external/bazel_skylib+/temp6278486909382546624/bazel-skylib-1.7.1.tar.gz: GET re
INFO: Invocation ID: e2062912-715f-47ec-a0bc-9d1b4fee9e5d
no actions running
no actions running
no actions running
no actions running
no actions running
no actions running
<root> (carbon@_)
Retrying a failure because it may be transient...
INFO: Invocation ID: 7d4ee250-882f-480d-9c8d-2d92e67462fc
Loading: 0 packages loaded
367
```
I'm doing this separately from removing the explorer/ and installers/
directories so that it's easier to review the side-effects.
Note for the main README, I didn't think it was worth keeping a mention
of "used to have a prototype interpreter, now archived" versus focusing
on the existing toolchain (for "Currently, we have fleshed out").
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This is trying to reduce how much we run tidy over due to poor
performance of tidy.
It's opening the door for issues where a header file is modified in a
way that introduces tidy issues in a different target. However, that's
not the typical tidy issue we see.
There may also be a risk where a source file indirectly becomes a source
file for a target in a way that `same_pkg_direct_rdeps` doesn't return,
but I'm not sure that's applicable for how we use targets.
An example of this locating a diagnostic error can be found in the "Add
tidy issue" commit's run (which I cancelled before it finished running,
but note the error):
https://github.com/carbon-language/carbon-lang/actions/runs/13932649182/job/38993348130?pr=5144
We were discussing this, `long term` only applies to issues and not PRs
(per past discussion, we don't really expect PRs should be inactive for
months). Renaming to `long term issue` to be more specific and hopefully
reduce confusion.
Without this, the merge queue blocks on results but they never get
triggered and show up. This will need to merge before we re-enable
`clang-tidy` enforcement on the trunk branch.
Trying to improve robustness against failures such as
[here](https://github.com/carbon-language/carbon-lang/actions/runs/11845529962/job/33011203628):
```
WARNING: Download from https://ftp.gnu.org/gnu/m4/m4-1.4.18.tar.xz failed: class java.net.ConnectException Connection refused
```
Which is coming from the target-determinator invocation:
```
subprocess.CalledProcessError: Command '['/home/runner/.cache/carbon-lang-scripts/target-determinator', '--bazel=/usr/local/bin/bazelisk', 'a720921dfb99c21f08832663c91a1f9e48a1bcc4']' returned non-zero exit status 1.
```
My thought is that the `bazel cquery` should trigger equivalent
downloads (equally though, I'd thought the `bazel mod deps` would do
that, so I don't want to give the impression of confidence).
It's harder to inject this into the target-determinator command line,
since we'd need something that retries by default.
Came up due to [libc++ install
issues](https://discord.com/channels/655572317891461132/655577725347561492/1302023663155023905)
We've discussed clang version verification, and adding that as long as
I'm in here. The more significant bit is the libc++ check, which if it's
not installed should fail like:
```
(tons of output)
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
/usr/lib/llvm-16/lib/clang/16/include
/usr/local/include
/usr/include/x86_64-linux-gnu
/usr/include
End of search list.
/usr/local/google/home/jperkins/.cache/bazel/_bazel_jperkins/85deb7d9d96f7e0e80b42618a55969d7/external/_main~clang_toolchain_extension~bazel_cc_toolchain/_temp:6:2: error: "No libc++ install found!"
#error "No libc++ install found!"
^
1 error generated.
ERROR: Analysis of target '//toolchain:toolchain' failed; build aborted: Analysis failed
INFO: Elapsed time: 0.265s, Critical Path: 0.08s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully
```
pre-commit runs bazel, and GitHub runners have an old clang by default
(caught by the new check), so I'm installing here for a consistent
version.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Removes the separate language server binary; I'm not sure we need to
provide it. Instead, `carbon language-server` is added as a subcommand.
Moves //language_server to //toolchain/language_server. Splits into a
trivial language_server.h, and a substantive server.h. I wasn't sure
about a better name, but wanted the split similar to check/check.h,
lex/lex.h, etc. At the same time, the class is probably going to be a
little big so not a good fit to through into just a cpp file.
This fixes some style issues with the language server class, but
generally I'm trying to not address things here in order to keep it
simpler.
I expect the change here will try rerunning a couple times, then fail if
it's a permanent failure.
I've seen similar failures, and believe this is flaky. Here's the
specific example that caused me to try a workaround:
https://github.com/carbon-language/carbon-lang/actions/runs/11560505812/job/32177497296
I don't see a difference in the runner information between failing and
passing runs, which might've indicated a canary. That's why I'm doing
this as a trivial retry.
Because clang-tidy is slow (and I'm not sure we can make it really
fast), trying to run it slightly less. Also, I noticed we can shave a
few minutes by disabling apt removal without losing too much free space.
Note that since this removes the old clang-tidy, I'll need to change the
branch protections before merging.
This should free up quite a bit of space and even make our builds faster
by avoiding duplicating every output. The syntax for this flag is
counter intuitive, so I've left a comment explaining it.
Demo site: https://jonmeow.carbon-lang.dev/
I'm trying to keep work under the `/website` subdirectory so that the
misc files don't interfere with unrelated views of the repository. The
`prebuild.py` script does some work to move things around and add
frontmatter, helping the jekyll generation.
I'm using the "just-the-docs" theme because I think it's a decent match
for what we want, and getting jekyll up and running with it wasn't too
difficult. Note #1526 proposed using Docusaurus; I started out there,
but was having trouble getting it working with newer versions. The
plugins in particular I got stuck trying to make work, which sent me
looking for options that we could have working with less customization.
I do lean towards jekyll though, because it's what GH uses so hopefully
we can get a more consistent experience.
Having a website has been approved for a while under #1492, but hasn't
been a priority. I'm mainly doing this because I want to just be able to
point people to carbon-lang.dev and have easy links that way.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Also a small pass on workflow names.
Note, I'm a little concerned that the test/nightly release/pre-commit
endpoints may be fragile. At the same time, it's also where it may be
most useful, to prevent network access by arbitrary test code. I think
this is imperfect, but maybe we can try it out and see if it's much of
an issue.
Note, the discord wiki action is currently broken, this should fix it.
This configures the nightly build to generate release notes and provides
a template for organizing them. The organization is done through
labeling of PRs and categorizing them based on those labels. I've tried
to provide a rough categorization that seems reasonable for folks.
While doing this I looked at our PR labeling and found a few bugs that
were preventing many labels form being attached. I've fixed those and
significantly expanded the coverage of file-based labeling. I've also
added code to do author-based labeling for automated PRs so those can be
separated out from human PRs.
There was a file that got into the testing filegroups but not the actual
built tarball. The result was that the installation failed to locate
itself correctly.
We also didn't have any testing to catch this. I'd like to add a bit
more end-to-end testing long-term, but for now just add a Python test
that validates the same set of files are in both.
Sadly, building and testing the compressed release is really slow and
not likely worthwhile outside of the actual release, but it would be
good to catch this stuff earlier. I tried switching from `bz2` to `gz`,
which made very little file size difference (so we should do it
anyways), and it is still too slow to do routinely. Instead, I've added
a Starlark macro that builds both the `pkg_tar` and the `py_test` to
validate it for both uncompressed `tar` and compressed `tar.gz`. This
lets us continuously test the `tar` version, and just test the `tar.gz`
in the nightly release run.
Updates the nightly release workflow to both test, run this specific
test, and use `gz`.
Last but not least, removes the `pkg_zip` as I don't think we have a use
case for this yet and some TODOs that should have been removed when the
version got added.
There is still a lot more we should try to do here, but this gets us
a decent start and should be enough for some of the developers working
on Carbon to use for simple cases, or tools like Compiler Explorer.
This first factors most of the complex setup for our CI workflow into
local actions that we can reuse in other workflows. It's actually
a fairly nice cleanup I think even without the specific use case of
a nightly release process. These use "composite" actions which is the
cleanest model for factoring out a subsequence of steps from a workflow.
I've tried to keep the inputs fairly minimal and easy to spot.
Next, it adds a cron-scheduled action that builds and creates a nightly
release. It also supports explicitly running the workflow, which can be
useful if the nightly release is broken in some way: a fix can be landed
and then a force run used.
Tested the `tests` workflow using our `action-test` branch:
https://github.com/carbon-language/carbon-lang/actions/runs/9545546278
Also tested the nightly release action by hacking in a `push` trigger
and using *draft* release. If you have admin rights, you can see an
example draft nightly build here:
https://github.com/carbon-language/carbon-lang/releases/tag/untagged-6a85cdce7b3a784a2e6b
Note that the GitHub releases are easily deleted, and even the tags
created in the repository can be cleaned up as needed so it should be
fairly harmless to fix this forward if needed.
This adds a defined Carbon version to the Bazel build and codebase that
can be used both to implement features like version checks and to report
a meaningful version on the command line. This replaces a hard-coded
string and a TODO in the driver.
As part of this, it adds support for defining the version in Bazel, and
special build flags for overriding relevant parts such as the
pre-release marker used. The exact structure and meaning of our version
string, including the pre-release parts, is implemented here in line
with the draft proposal:
https://docs.google.com/document/d/11S5VAPe5Pm_BZPlajWrqDDVr9qc7-7tS2VshqO0wWkk/edit?resourcekey=0-2YFC9Uvl4puuDnWlr2MmYw
This also introduces a workspace status command to the repository to
extract the git commit SHA and other information when building, and the
logic to stamp that into binaries as part of the version string when
useful. The technique used leverages weak symbols with whole archive
linking to allow a link-time override of unstamped data with stamped
data in the leaf executable. This makes building with `--stamp` a
reasonable default, especially for development builds. The CI system is
explicitly opted out of this as there it has no benefit.
Last but not least, all of these are wired into the install rules so
that we build installable packages with the version number in a
conventional place in the directory and filename.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This is coming out of advice from
https://securityscorecards.dev/viewer/?uri=github.com/carbon-language/carbon-lang
Updates action versions and pins by checksum. This is something we
already do in other places, and doing it with actions just seems
consistent.
Trying to add token permissions at the same time.
step-security/harden-runner is being set up to monitor network traffic
for actions, with the idea that we'd set it to block unexpected traffic
in the future.
Note, https://app.stepsecurity.io/secureworkflow generates the pinned
checksums and adds the harden-runner. In some cases it added token
permissions, but we have a couple it doesn't recognize (`gh` executions,
other custom commands, jlumbroso/free-disk-space, reviewdog/action-setup
as examples) so I'm trying to guess my best.
Because these are workflows, testing is limited.
Due to my workflow, I prefer to run `pre-commit` locally when I forget
to after an amend. While reviewdog can be helpful in the UI, it's mostly
making me mark comments as resolved as part of re-pushing. Rather than
continuing with this, maybe opting out is best?
Jon Ross-Perkins
Chandler Carruth
