I accidentally put in one I'd considered, instead of the one I'd decided was probably the best fit, and the names are so similar I didn't notice.
NOTE: This still isn't totally working, but I think it will when we go public.
This was noticed because macos stopped installing the brew llvm, and they don't have clang-format. However, our tests don't actually need clang-format (and even if they did, we should probably match the pre-commit version), so it seems superfluous to check for.
Keeping brew because there's an advantage to consistency on tool versions, just running it on all platforms instead of linux-only.
Moves common script logic into utils.py (not a great name, but couldn't come up with better). This is in particular to make the buildifier.py script really trivial, allowing that pre-commit to be easily added. However, scripts have also been diverging on how we find bazel, so I'm trying to unify that.
The advantage of reimplementing buildifier's pre-commit is that (a) we can now run buildifier server-side, and (b) we can stop advising installing it manually. Then the only Linux-specific package manager is Cargo, which is only used for watchman, which is optional -- so stop highlighting Linux-specific package managers in the tool instructions.
- brew instead of pyenv to centralize package management
- can't recall why we used pyenv before, may not have been the best choice.
- pyenv has been a burden in updating versions, a single `brew upgrade` works better
- python3.9 because it's the latest and greatest, feels weird going to old versions if we're recommending installs.
- bazel 5.0.0 pre-release due to https://github.com/bazelbuild/bazel/commit/b9fc66d327debcfbdb2964afdba35a1cc8919b81
- earlier versions want `python` to be on the path, which pyenv did but brew does not
For most people this will mean: `rm -rf ~/.pyenv && brew uninstall pyenv && brew install python@3.9 && pip3 install -U pip && pip3 install pre-commit`, plus removing any `pyenv` invocations from the shell `rc` file.
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
- Automatically mark issues/PRs `inactive` after 90 days without comments
- Archive PRs (only) after 14 more days
- Adding "long term" for issues which can't be resolved in a few months, but shouldn't be marked `inactive`.
The built-in GitHub action cache is too broken for us to use currently.
Pull request runs will evict trunk runs from the cache easily, which
will remove a useful baseline and make builds generally uncached and
extremely slow.
Instead, move to using Bazel's remote caching functionality with a GCP
storage bucket. This gives us nearly limitless storage and is actually
vastly simpler than the prior arrangement. It should also allow cache
sharing between different build configurations and other benefits.
This cache isn't usable by anyone else sadly, but it should at least
provide a better scaling technique for our CI.
This does require us to move from `pull_request` workflow to
a `pull_request_target` workflow and run the CI within the domain of the
project. This is only really safe for us to do while the project remains
private. When we go public we'll want to make the cache publicly
readable and change this code so that when not running on a branch in
the main repository we use the cache in a read-only mode and only write
back updates for branch runs. This should still be effective but
requires a bit more complexity here and in the remote cache including
setting up public access. For now, going with the simple if slightly
less secure model.
This was necessary when I was testing this on my own PR branch, but
without this rebuilds on trunk won't save a fresh cache for folks to use
on *their* pull commit branches. We just need a fresh key here.
Also add one more flag to our `.bazelrc` to improve the overall build
efficiency.
While the first build with this will be very slow (likely a couple of
hours), the caching layer we are using now should effectively cache this
for subsequent builds so that only the changed files have to be
recompiled. Updates to Clang and LLVM themselves will still result in
a slow rebuild, but normal development should remain fast.
The Bazel disk cache provides a stable, durable, and supported
persistent cache of the intermediate steps of Bazel builds. This is like
`ccache` but covering more steps and more reliable.
It is based on the same fundamental infrastructure that enables
a *remote* build cache, and at some point we may want to enable that so
that our CI can pre-seed a cache for users. But for now, this just
enables local support for this.
For local builds this will ensure that intermediate artifacts are
re-used when changing branches, syncing, across Bazel server restarts or
even upgrades. It should dramatically reduce the need to re-build large
and slow components like all of Clang and LLVM. Largely, those should
only be rebuilt when we update our pinned revision of upstream and then
cached persistently on a given machine. Enabling this by default
requires hard-coding a path, and so I've also added built-in support for
importing a user config file if present so that folks can customize this
(and any other Bazel features) as desired.
This also wires the disk cache up for the GitHub cache in our CI. By
using the designated Bazel disk cache for this, we get a *much* simpler
and *much* more robust solution. So much so that I've re-arranged the
key structure so we can more aggressively re-use old caches. The only
downside is that nothing ever gets removed automatically from the cache.
I've added a layer that will trim any unused file roughly once a day.
For CI, the benefits of this are huge. None of the previous hacks are
needed to locate and preserve the cache. The design of the cache is
deeply durable and so we don't need to aggressively invalidate it. Last
but not least, it focuses the cache on the raw artifacts, which compress
exceptionally well. With the other changes to shrink the build outputs
and this one combined, I expect we will be able to enable all of the
build targets across all four config/os combinations without any
significant pressure on our space quota for GitHub action caches.
Now that LLVM 12 has been released we no longer have any need to
bootstrap LLVM to get the desired featureset. LLVM 12 is available
widely, including in Homebrew across multiple platforms and in the
GitHub action runners.
Sadly, the Linux distribution builds of LLVM-12 are largely broken and
not as useful for us. The Homebrew Linux install was also broken
originally, but I've worked extensively with the Homebrew folks to get
the Linux install into a really good shape. It should now work reliably.
There are two primary bugs in Linux LLVM packages that need to be fixed
before we can just use them:
- https://bugs.llvm.org/show_bug.cgi?id=43604
- https://bugs.llvm.org/show_bug.cgi?id=46321
Once those are addressed and point releases with the fixes widely
available we can further simplify things.
Even with the need to use Homebrew installs, using the released LLVM has
the extra advantage of making it easy to properly support Darwin ARM and
I've added that configuration so that I can test things there.
Last but not least, this will significantly shrink our build outputs
which should allow building much more in continuous integration on
GitHub actions without exceeding the action cache size limits. I've even
added several tweaks and adjustments to the compile and build flags to
improve the build performance and reduce the build output size.
Once this is landed and stable, we can consider adding the refactoring
tooling back to our CI.
One of the biggest downsides of this path is that our CI has to download
and install the LLVM toolchain from Homebrew on each run. This is pretty
slow (takes a couple of minutes). But it is a fixed overhead -- it won't
get worse over time. Eventually, we can either look at a much fancier
action configuration to avoid this or hopefully the Debian packages will
get updated and we can move back to those.
The bootstrapping has served us long enough at this point. We can
resurrect it if we ever find a compelling reason for breaking off of the
latest LLVM release as our host toolchain.
Co-authored-by: Jon Meow <46229924+jonmeow@users.noreply.github.com>
This picks up a newer version of LLVM and the LLVM Bazel integration.
The big change here is that we can configure the LLVM targets that are
built, which allows us to dramatically reduce the build costs by
focusing on a couple of CPUs for the time being.
There are a few API updates needed as well.
This also rotates the cache version so we start with a clean Bazel cache
from here. Otherwise we'd potentially pay the cost of carrying around
stale bits of LLVM endlessly.
This is both a bit tricky and really easy with Bazel. The easy part,
especially compared to other build systems is that we can do this
regardless of the state of the repository -- Bazel will hermetically
check that everything is up-to-date, allowing the cache to be a bit
stale but still totally functional.
The easy part is that we can just ask Bazel to use an output base that
we cache and restore. This is super nice and even avoids most of the
Bazel installation bits.
The tricky part is that we need this to reconnect correctly to the
installed tree, so we need to exclude a crucial symlink that will then
get re-created at the right moment.
This gets really tricky due to LLVM and Clang (and this we would
struggle with no matter what build system we used). Building LLVM
creates a *ton* of object code. Just a huge amount. As a consequence,
we'd run into GitHub's limit for action cache size (5gb) really quickly
with 4 configurations.
So we can do another bit of tricky business and exclude the downloaded
`external` tree. This gets rebuilt easily, and there's no real need to
download it with the cached state -- its downloaded either way.
There are two follow-ups that I'd like to make here. One is to prod the
Bazel team to make things like persisting your output base a bit easier
to do reliably. Even better to make it easier to do *partial*
persisting.
Second follow-up is to work to make our usage of LLVM *much* less
wasteful. There are a bunch of steps here from changing how we use
sanitizers to how LLVM is built. Those will be follow-ups though.
I tried what I could, but I don't think this action will work while Carbon is private.
"Error: Resource not accessible by integration"
https://github.com/carbon-language/carbon-lang/runs/2441129284?check_suite_focus=true
I'd seen this mentioned on some actions:
https://github.com/marketplace/actions/github-project-automation#personal-access-token-secretsgithub_token
However, I thought it might've been a side-effect of the APIs they were using (I did have this running on a local repo), but I think fork PRs are where the problem lies...
This probably isn't exactly right, but it at least gives *some* low
latency way to update when all the dependencies have been addressed.
Right now, I think only the cron run will unblock dependent PRs.
This supports tracking PR-to-issue and issue-to-issue dependencies in
addition to PR-to-PR, and this seems likely to be increasingly important
as we have decisions being made via issues.
No since spending the GitHub action minutes (or waiting to merge) on
Bazel when only changing markdown or other files that aren't part of the
build and test.
This is primarily using the configuration matrix facilities of GitHub
actions to consolidate the overall configuration. Beyond avoiding
duplication, this also allows the default and release builds to run in
parallel. The checkout time is duplicated between these, but the rest of
the time is nicely parallelized. This has the most dramatic effect on
macOS builds. Overall, this should reduce the latency on testing from
22-28 minutes to 15-20 minutes from what I've seen which seems
worthwhile.
This does reduce the detail provided in the names of the different
configurations. However, the Bazel build mode is preserved. That seems
like the most critical pieces of information.
Much of this started with me just trying to learn more about GitHub
actions, but once understanding how the job matrix worked, it seemed
worthwhile to send out as an actual change.
- Shift some things around to adjust to being in bazel.
- Add separate build/serve/publish scripts for use by bazel (not set up for direct execution, but bazel requires +x).
- Fix some tests I noticed not running unittest.main as a result of the switch.
- Move md files into filegroups for build reuse.
- Disable automatic site publishing (now `bazel run //website/jekyll:publish`)
Site publishing seems like it'd be too much trouble to keep automated... The C++ toolchain essentially needs to be set up due to the repo config, along with syncing the LLVM submodules, etc. That seems a bit annoying to do on each run of the publish-docs action, and not something I really want to maintain. If we get a CI, we can focus on it more there, but this feels like it'd just be a one-off to maintain as a github workflow.
The jekyll build keeps top-level without expanding them. So the site works, but the publish copy failed because some symlinks to files that aren't visible through the site (like CONTRIBUTING.md, versus the site's CONTRIBUTING.html) didn't work.
One implication here is that both proposals/README.md and the website should use basically the same format for their proposal lists. Another subtle change is that the website sidebar was previously sorted by *filename*, and is now sorted by *title*, which seems better because it's something that readers can see.
Enumerating why files change:
- gen_sidebar.py is the centerpiece, with a couple helpful functions for re-use elsewhere
- Delete the old sidebar html includes
- Modify the Makefile to handle gen_sidebar.py reasonably well (let's be honest... I'm not great at Makefiles)
- Move proposal listing out to its own file (proposals.py) for re-use
- Add a few PYTHONPATH things + `__init__.py` files to get modules importing correctly (may be a better way at this, I've hit a wall though).
Added here:
- .pre-commit-config.yaml
- docs/project/contribution_tools.md
- src/scripts/pre-commit-toc.js
- CONTRIBUTING.md
- proposals/* (just to switch TOC formats)
The rest consists of auto-generated fixes.
The switch from doctoc to markdown-toc is because doctoc wants to put a TOC in every file, whereas markdown-toc only does it if you have an insertion point.
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
* Add github action to publish docs.
This change add a placeholder action which:
(A) Authenticates with gcloud using the appropriate GCP AS.
(B) Copies the docs/ folder as-is into
www.carbon-lang.dev/test-doc-root/
SADIK KUZU
Jon Ross-Perkins
Jon Meow
Matthew Riley
Dave Abrahams
Bryce Adelstein Lelbach aka wash
Eric