This updates to bazel 8.0.0, also updating bazel mod deps and tools to
make that function. The release is a couple weeks old, and we haven't
updated in a while, and it's a major release. Note it includes some
incompatible flag flips that this is trying to update with respect to.
I'll try generally enabling incompatible support separately.
The most visible bazel behavior change here will be the change from `~`
to `+` in repo path names. (If you're curious,
https://github.com/bazelbuild/bazel/issues/23127 indicates this fixes a
Windows performance issue)
Note that this is building on top of both the action env update in #4728
(which got me started down this path) and the proto removal in #4731
(which would add significant work to this update). Only the commit
starting at "Work towards bazel 8.0.0" is specifically part of this PR.
This restores the symlinks for the installation, but teaches the busybox
info search to look for a relative path to the busybox binary itself
before walking through symlinks. This let's it find the tree structure
when directly invoking `prefix_root/bin/carbon` or similar, either
inside of a Bazel rule or from the command line, and mirrors how we
expect the installed tree to look. This works even when Bazel resolves
the symlink target fully, and potentially to something nonsensical like
a CAS file.
In order to make a convenient Bazel target that can be used with `bazel
run //toolchain`, this adds an override to explicitly set the desired
argv[0] to use when selecting a mode for the busybox and a busybox
binary. Currently, the workaround uses an environment variable because
that required the least amount of plumbing, and seems a useful override
mechanism generally, but I'm open to other approaches.
This should allow a few things to work a bit more nicely:
- It should handle sibling symlinks like `clang++` to `clang` or
`ld.lld` to `lld`, where that symlink in turn points at the busybox.
We want to use *initial* `argv[0]` value to select the mode there.
- It avoids bouncing through Python (or other subprocesses) when
invoking the `carbon` binary in Bazel rules, which will be nice for
building the example code and benchmarking.
It does come at a cost of removing one feature: the initial symlink
can't be some unrelated alias like `my_carbon_symlink` -- we expect the
*first* argv[0] name to have the meaningful filename for selecting
a busybox mode.
It also trades the complexity of the Python script for some complexity
in the busybox search in order to look for a relative `carbon-busybox`
binary. On the whole, I think that tradeoff is worthwhile, but it isn't
free.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
The only documentation I can find for `.bazelversion` is in
https://github.com/bazelbuild/bazelisk/blob/master/README.md. However,
`bazel` will error out if it doesn't match the `.bazelversion`. For
example:
```
╚╡/usr/bin/bazel build :all
ERROR: The project you're trying to build requires Bazel 7.3.0 (specified in [elided]/carbon-lang/.bazelversion), but it wasn't found in /usr/bin.
```
Switching to this because the `MODULE.bazel` requires tighter version
pinning in order to avoid churn, and this should help catch mistakes
early.
In `MODULE.bazel`, demote mention of the version issue because it should
only really occur now when the version is being deliberately changed, so
the connection should be more apparent.
As described in symlink_helpers.bzl, copied here for visibility:
Symlinking busybox things needs special logic.
This is because Bazel doesn't cache the actual symlink, resulting in
essentially resolved symlinks being produced in place of the expected
tool. As a consequence, we can't rely on the symlink name when dealing
with busybox entries.
An example repro of this using a local build cache is:
bazel build //toolchain
bazel clean
bazel build //toolchain
We could in theory get reasonable behavior with
`ctx.actions.declare_symlink`, but that's disallowed in our `.bazelrc`
for cross-environment compatibility.
The particular approach here uses the Python script as a launching pad
so that the busybox still receives an appropriate location in argv[0],
allowing it to find other files in the lib directory. Arguments are
inserted to get equivalent behavior as if symlink resolution had
occurred.
The underlying bug is noted at:
https://github.com/bazelbuild/bazel/issues/23620
Came up due to [libc++ install
issues](https://discord.com/channels/655572317891461132/655577725347561492/1302023663155023905)
We've discussed clang version verification, and adding that as long as
I'm in here. The more significant bit is the libc++ check, which if it's
not installed should fail like:
```
(tons of output)
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
/usr/lib/llvm-16/lib/clang/16/include
/usr/local/include
/usr/include/x86_64-linux-gnu
/usr/include
End of search list.
/usr/local/google/home/jperkins/.cache/bazel/_bazel_jperkins/85deb7d9d96f7e0e80b42618a55969d7/external/_main~clang_toolchain_extension~bazel_cc_toolchain/_temp:6:2: error: "No libc++ install found!"
#error "No libc++ install found!"
^
1 error generated.
ERROR: Analysis of target '//toolchain:toolchain' failed; build aborted: Analysis failed
INFO: Elapsed time: 0.265s, Critical Path: 0.08s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully
```
pre-commit runs bazel, and GitHub runners have an old clang by default
(caught by the new check), so I'm installing here for a consistent
version.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
This should substantially reduce the total build size of `-c dbg`
builds, and especially improve cache hits during incremental
development. I'm seeing over 50% reduction total on Linux in just the
raw size of a complete debug build. Even on macOS where we can't use
split DWARF there are substantial reductions.
Note that LLDB and GDB want slightly different flags to have the best
experience with split debug information, and so the build and
documentation have been updated to enable LLDB's flags by default but
provide clear instructions for switching to GDB's flags, and they are
structured so that this can be done persistently for an individual
developer.
I was considering making a "/bazel" alias for this, but I'm really on
the fence about whether that's a good choice. However, I do think this
is good to have to simplify installs.
Fixes#3071 and #3896
Also makes an alias so that this is easier to find. Verified that
running the alias still finds prelude files.
Note the actual target is printed when building (although a symlink is
also created, using that symlink confuses file-finding).
```
╚╡bazel build :carbon
...
Target //toolchain/install:prefix_root/bin/carbon up-to-date:
bazel-bin/toolchain/install/prefix_root/bin/carbon
```
On macOS, Homebrew recommends `pipx` to install Python binaries, or
installing `pre-commit` directly, since it is available.
Here is the message in full.
```console
❯ pip3 install pre-commit
error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try brew install
xyz, where xyz is the package you are trying to
install.
If you wish to install a non-brew-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip.
If you wish to install a non-brew packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.
```
~~Also, unless there is a compelling reason to install a particular 3.x
version of Python, it may be better to let Homebrew use whatever version
people may already have installed (or let it automatically install the
version they've tested).~~
Here are the details of the pre-commit Homebrew package:
```
❯ brew info pre-commit
==> pre-commit: stable 3.6.2 (bottled), HEAD
Framework for managing multi-language pre-commit hooks
https://pre-commit.com/
/opt/homebrew/Cellar/pre-commit/3.6.2 (679 files, 9.9MB) *
Poured from bottle using the formulae.brew.sh API on 2024-03-08 at 22:22:11
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/p/pre-commit.rb
License: MIT
==> Dependencies
Required: libyaml ✔, python@3.12 ✔
```
Installing directly with Homebrew is a supported option in the
pre-commit documentation. https://pre-commit.com/#install
This should also unblock our switch to C++20 and other improvements.
There are three core parts of the change --
1) Updating our infrastructure to fetch and find Clang-16.
2) Updating our documentation to reflect this and help folks with any
system issues they encounter.
The infrastructure change is unfortunately tricky. We can't get Clang 16
easily on GitHub's runner images, and in the past we've had persistent
problems with flakiness when our actions download this much during their
runs. Due to the flakiness, we've previously removed all downloading of
dependencies outside of Bazel itself, and added retry loops around Bazel
specifically to overcome flaky downloads.
This change tries to address these problems by populating the Clang and
LLVM toolchain in a place that we can then cache using the built-in
GitHub action caching infrastructure. This seems like by far the least
likely to flake way of downloading extra things into our runs. And since
these are relatively slow moving dependencies, we should populate this
cache very, very rarely.
For Linux, this downloads the binary release artifact from GitHub,
prunes out large parts of it that we don't need, and then caches this as
a local toolchain. This proves both small and fast.
For macOS, this uses a trick to cache the destination of Homebrew
installs. It unfortunately caches the *entire* Homebrew installation
though, and so it also goes to some lengths to prune and minimize how
much is installed from Homebrew. The result is "only" a 2gb cache image.
Because of the size and slower download and filesystem, the macOS runs
see a 1 - 2 minute slowdown.
We might extend the Linux infrastructure here usefully if we want to
test multiple LLVM versions. We might also extend the macOS version to
get a cheaper way to prune parts of the system and free up disk space,
or to cache other Homebrew installed tools if needed.
Last but not least, this brought to the forefront an issue with our C++
toolchain integration which relied on a specific CMake build option
being set in the LLVM toolchain install. This option isn't used in the
official release artifacts. Instead, switch to a more robust approach to
linking libc++abi statically that shouldn't have these problems.
Beyond the infrastructure changes, this also updates the documentation
to reflect requiring Clang 16 or newer, and adds some extra tips for
folks that are missing this.
The documentation is also updated to address a problem with getting the
right libc++abi files installed to support the more robust linking
strategy. This may reduce the problems we've seen in the past around
libc++abi and linking on other Linux distros as well.
---------
Co-authored-by: josh11b <josh11b@users.noreply.github.com>
Proposal to focus implementation effort for the next 1-2 years on the
Carbon
toolchain instead of the explorer. This will impact the explorer in a
few ways:
- We will keep the explorer's code in place, building, and passing its
basic
tests. It can remain a good baseline for exploring Carbon's language
features.
- We won't prioritize expanding the explorer's coverage of Carbon
features or
other improvements -- it is good enough for what we need until the
toolchain
catches up.
- We will stop actively fuzzing and expanding test coverage for the
explorer.
- Eventually, when we want to resume work on the explorer, we'll
evaluate the
best platform to build on -- the current explorer codebase or on top of
the
toolchain's semantic IR.
Also tries to update the core readme and contributing docs to reflect
this.
This Pull Request added support for Bazel installation and setup in the
contribution_tools.md file un the docs directory. It added a link to the
releases page of the bazel repository for debian/ubuntu users to
download the binary and execute the commands provided in the
documentation.
Closes#3439
Adds some notes about the required build flags for lldb to successfully
find the symbols on macOS debug builds. Also adds a recommended debugger
configuration for interactive debugging in VSCode on macOS.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
You need bazel version 6.0.0 (with --host_per_file_copt) to build
carbon. This adds versioning info and adds that Linux, Mac and Windows
can use bazelisk.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Ubuntu 22.04 base image and examples using github, a copy instruction and volumes.
I was using this to run examples on a windows machine since it is more comfortable than switching OS or using the WSL subsystem directly.
If this is something that other people would like to have I can add other versions of ubuntu and other linux distributions.
Key rewrite points:
- Making more use of apt now, since we no longer need `brew` for llvm on Linux.
- Gets ahead of issues with brew's llvm 15 on Linux.
- Clear list of commands for a typical setup.
- Less verbose text about various tool options (I think this was just too much).
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
The toolchain embeds the assumption that `clang` and `llvm-ar` are
adjacent, which may not be true on all host platforms. Moreover,
in #1842 and #1843, we test if Homebrow LLVM is in `PATH` by checking
that `llvm-ar` is adjacent to `clang` and if it isn't, we `fail()` the
build, even if `llvm-ar` *is* in `PATH`.
Instead, actually check `PATH` with `repository_ctx.which`. This however
necessitates the assumption that `llvm-ar` and other LLVM binutils
are adjacent, and subsequently that `clang` and `ld.lld` are adjacent.
It appears that we don't seem to always be using these tools, but we
should avoid embedding wrong assumptions regardless.
Update docs to reflect this change.
> The only issue that I currently have with this one is that the `clangd` extension is having some hiccups, not being able to find includes like `<gmock/gmock.h>` and `<benchmark/benchmark.h>` and thus showing extra errors. However, this only affects the VSCode extension and bazel seems to work fine.
This might be an issue with the compilation database: you might try ./scripts/create_compdb.py to help.
Fixes#1527
On Debian, the zlib1g-dev package is required to build and should be given as a requirement.
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Older versions of GDB (before version 10.1) don't work with the
DWARF v5 debugging format the LLVM uses. The error message
which GDB shows when this happens is a bit misleading though.
Provide documentation about how to deal with that error message.
This proposal establishes a plan for moving away from the embedded copy
of LLVM and instead downloading it with Bazel.
The goal is that after this lands, we will do a history-rewrite to
cleanup the repository. There are instructions on how folks can move any
in-flight work over to the newly tidied repo.
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: josh11b <josh11b@users.noreply.github.com>
Moves common script logic into utils.py (not a great name, but couldn't come up with better). This is in particular to make the buildifier.py script really trivial, allowing that pre-commit to be easily added. However, scripts have also been diverging on how we find bazel, so I'm trying to unify that.
The advantage of reimplementing buildifier's pre-commit is that (a) we can now run buildifier server-side, and (b) we can stop advising installing it manually. Then the only Linux-specific package manager is Cargo, which is only used for watchman, which is optional -- so stop highlighting Linux-specific package managers in the tool instructions.
- brew instead of pyenv to centralize package management
- can't recall why we used pyenv before, may not have been the best choice.
- pyenv has been a burden in updating versions, a single `brew upgrade` works better
- python3.9 because it's the latest and greatest, feels weird going to old versions if we're recommending installs.
- bazel 5.0.0 pre-release due to https://github.com/bazelbuild/bazel/commit/b9fc66d327debcfbdb2964afdba35a1cc8919b81
- earlier versions want `python` to be on the path, which pyenv did but brew does not
For most people this will mean: `rm -rf ~/.pyenv && brew uninstall pyenv && brew install python@3.9 && pip3 install -U pip && pip3 install pre-commit`, plus removing any `pyenv` invocations from the shell `rc` file.
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Now that LLVM 12 has been released we no longer have any need to
bootstrap LLVM to get the desired featureset. LLVM 12 is available
widely, including in Homebrew across multiple platforms and in the
GitHub action runners.
Sadly, the Linux distribution builds of LLVM-12 are largely broken and
not as useful for us. The Homebrew Linux install was also broken
originally, but I've worked extensively with the Homebrew folks to get
the Linux install into a really good shape. It should now work reliably.
There are two primary bugs in Linux LLVM packages that need to be fixed
before we can just use them:
- https://bugs.llvm.org/show_bug.cgi?id=43604
- https://bugs.llvm.org/show_bug.cgi?id=46321
Once those are addressed and point releases with the fixes widely
available we can further simplify things.
Even with the need to use Homebrew installs, using the released LLVM has
the extra advantage of making it easy to properly support Darwin ARM and
I've added that configuration so that I can test things there.
Last but not least, this will significantly shrink our build outputs
which should allow building much more in continuous integration on
GitHub actions without exceeding the action cache size limits. I've even
added several tweaks and adjustments to the compile and build flags to
improve the build performance and reduce the build output size.
Once this is landed and stable, we can consider adding the refactoring
tooling back to our CI.
One of the biggest downsides of this path is that our CI has to download
and install the LLVM toolchain from Homebrew on each run. This is pretty
slow (takes a couple of minutes). But it is a fixed overhead -- it won't
get worse over time. Eventually, we can either look at a much fancier
action configuration to avoid this or hopefully the Debian packages will
get updated and we can move back to those.
The bootstrapping has served us long enough at this point. We can
resurrect it if we ever find a compelling reason for breaking off of the
latest LLVM release as our host toolchain.
Co-authored-by: Jon Meow <46229924+jonmeow@users.noreply.github.com>
This matches the version on Ubuntu LTS and other OSes. The only problem
I found with it in our testing is that Bazel confusingly sets the locale
to use `LANG=en_US` by default which breaks UTF-8 support. We may need
to shift this on Windows, but this seems like a reasonable first step.
Co-authored-by: Jon Meow <46229924+jonmeow@users.noreply.github.com>
Jon Ross-Perkins
Chandler CarruthandJon Ross-Perkins
Dana JansensandJon Ross-Perkins
Brymer Meneses
Nathan Youngman
Richard Smith
Sree Dananjay S
Lucile Rose NihlenandChandler Carruth
Samiur KhanandJon Ross-Perkins
Avi levy
Nico Z
cabmeurer
3405691582
Vitalii
yale
SADIK KUZU
Joshua Peterson
Parisa KhaleghiandJon Ross-Perkins
Wolfgang E SanyerandChandler Carruth
mconst
Jon Meow