This has been working really well for me, is incredibly faster than the
other approach, and some commits continue to hit bugs in the old system
where files that aren't even going to be run through `clangd-tidy` end
up tripping up the execution. Hopefully all of that is resolved with the
new version.
This also switches to a more Bazel-based install layout, skipping the
FHS-based synthetic layout. The FHS-based layout is still reconstructed
explicitly when building an installable tar-ball.
The biggest change is to configure the just-built install as a Bazel
toolchain, including allowing it to build its own runtime libraries as
native Bazel libraries. This removes the need for a monolithic runtimes
build, all of that code logic is removed.
This should also pave the way to using the just-built toolchain for
doing a full 3-stage bootstrap. Building the 2nd stage is included here
as it was a particularly effective way to test that the Bazel
integration was fully working. Adding a 3rd-stage check for stability is
future work, but should be pretty easy.
There is a down-side: this uses the busybox to do the runtimes
compilation, which means they will be re-built after ~any change to
Carbon. However, the integration with Bazel should largely pay for this,
and we can continue to factor the tests away from depending on built
runtimes in most cases.
Now that we're building and testing the runtimes more directly, this
surfaced a problem with the layout of runtimes on macOS that is fixed
here. All of the Darwin OSes use a custom layout for their resource
directory compared to other targets. We now model this in both the C++
built runtimes and the Bazel built runtimes.
Assisted-by: Gemini via Antigravity
Noticed this in bazel central registry, I'm interested in trying it out.
It's using a faster approach, but leaving the other around for the
moment in case it doesn't work out well.
Assisted-by: Google Antigravity with Gemini
Roll LLVM to `6811a83c81500ee373adfc0d9978ff9625a4cf1c`.
This includes https://github.com/llvm/llvm-project/pull/183831 which
moved the functionality of `finish()` on `DiagnosticConsumer`s into the
destructors, and removed the `finish()` method. So, our callers to
`finish()` are migrated to cause the destructor to run at that time
instead.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
By using git_override, we get some validation from the sha, while
removing the sha256 on the .tar.gz which has been brittle lately. Note
the difference between downloading via sha is this still locally
validates content.
Versus something like #6844, this doesn't update the llvm version, just
how we get it.
Assisted-by: Google Antigravity with Gemini
This isn't as interesting as others, as it only involves compile
options.
It also adds a missing flag of `-fno-lto` as these objects can't be
LTO-ed.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This is a mostly routine update, with some edits for a benchmark API
change.
I'm not updating LLVM here, since that could conflict with other ongoing
work.
I wasn't sure I'd be able to really test this code path, but then
I remembered that Bazel has a whole platform for running Bazel from
within an integration test, and it turns out to work brilliantly. It
even lets us point the child Bazel invocations to the just-built
toolchain.
This should both give us confidence that we don't accidentally hit
a Bazel incompatibility with the example project, and it should ensure
that if something about the installed toolchain would stop being
compatible with building via Bazel we'll catch it early.
The tests are integration tests and so a bit slow: 15s or so. But
`//examples/...` is already pretty expensive and no other testing
patterns are impacted.
This brings some fixes:
- The handling of `zlib` and `zstd` are much cleaner
- Three of our patches are no longer needed
This also includes the fixes from #6562
It also moves us from `zlib` to `zlib-ng` which is a much better basis
for what we want, and likely makes our toolchain faster when generating
debug info at least.
It fixes another API change in terms of which headers provide the
`createInvocation` we use.
Lastly, it cleans up the deps test to correctly recognize the wrappers
for `zlib-ng` and `zstd`, as well as improving the documentation for why
we allow dependencies on them.
Also consolidate on using `//bazel/cc_rules:defs.bzl` where appropriate.
Also update a couple of Bazel modules deps of `@rules_cc` to the latest
versions.
This builds on the previous work to flesh out more on-demand runtimes
building. It adds building of the `libc++.a` archive runtime.
A number of changes are required for this to work:
- The runtimes build infrastructure needs to support building sources
from multiple parts of LLVM rather than a single part. We do this by
lifting the root of the runtimes source paths up a level to a common
runtimes tree, and installing the runtimes sources below this
directory.
- Both libc++ and libc++abi runtimes sources need to be installed, and
we even need to install some interesting parts of llvm-libc that are
used in the build of libc++.
- We need to generate the site configuration header file for libc++ from
the CMake template. This includes both setting up a set of
platform-independent defines and introducing some basic Bazel support
for processing the CMake template itself.
Doing all of this also exposed some missing features and limitations of
the runtimes building infrastructure that are addressed here.
One note is that all of this just adds libc++ to the explicit
`build-runtimes` command for testing. It doesn't yet trigger
automatically building these prior to linking, or configuring any of the
other subcommands to automatically use these runtimes. All of that will
come in follow-up PRs.
Also, this makes the `clang_runtimes_test` ... _very_ slow in our
default build configuration. Compiling libc++, even with many threads on
a large Linux server requires up to 50 seconds. I'm open to any
suggestions on how to handle this, including disabling the test in
non-optimized builds. I have some ideas to speed this up, but
fundamentally building libc++ is... not cheap.
I did look at some of the existing Bazel tools to process the CMake
template, but they all seemed significantly more complex than what we
need and didn't have broad adoption. Given that, it seemed slightly
better to just roll our own given the simple format.
Two of the new LLVM patch are currently under review upstream and so
hopefully temporary:
- https://github.com/llvm/llvm-project/pull/169155
- https://github.com/llvm/llvm-project/pull/169292
This is the first real step towards building libc++ itself, and fleshes
out both the core runtimes management logic and the archive-based
runtimes logic for a quite simple runtime.
Nothing here causes us to _use_ libunwind, and in fact this doesn't
include even the "on-demand" aspect of building `libunwind`. Instead,
this just wires it up to the explicit `build-runtimes` subcommand for
simple testing. The full integration along side the target directory is
future work.
This removes the need for a patch and improves on the quality of the
rules significantly. A follow-up PR will use this to apply a number of
fixes to how we build the runtimes.
This is the first step to having Clang's runtime libraries fully
available for the Carbon toolchain. This PR focuses on the lowest level
runtimes, the CRT files and the builtins library.
The goal is to intercept Clang runs where it needs these
target-dependent pieces to be available, and build them on demand using
our Clang-running infrastructure. This avoids most of the subprocess
overhead, but there is still some due to missing features in Clang.
This requires exporting the sources for these runtimes from the Bazel
build, and installing them in our target-independent resource directory.
We then build a simplified "build" of these sources within the
`ClangRunner` itself to produce the specific artifacts and layout
expected by Clang.
It also required fixing our use of Clang on macOS to have a default
system root in order to successfully compile or link.
It also required cleaning up how the `ClangRunner` used target
information more generally -- instead of taking the target as
a constructor parameter, it manages its target internally and relies on
the Clang target-specifying command line flags.
I looked at whether we could split this into another layer separate from
the `ClangRunner`, but that proved frustratingly difficult to manage.
While we support building these on-demand as part of a detected link,
that doesn't seem feasible as we don't have the necessary separation
between compilation runs of Clang and link runs of Clang. However,
I have tried to factor the internals to provide as clear of separation
as I could across these.
I have also created a stand-alone subcommand to directly build the
runtimes which allows for easy testing. It also supports building them
into a specific directory, and that directory can in turn be passed to
a Clang invocation. This is designed to work both at the API level with
`ClangRunner` and at the subcommand level.
Currently, the only part of the commandline that is detected and
forwarded to the runtimes build is the target. Eventually, the plan is
to expand this so that we can build a maximally tailored set of runtimes
for a given compilation.
The other big TODO here is to actually implement caching storage of
these runtimes so they aren't built on every execution. Right now, this
uses a somewhat hack-y build of a temporary directory, but this isn't
expected to be suitable long-term. Building these runtimes on *every*
link makes those commands take approximately 15 seconds with an ASan
build like our default development build, and just over 2 seconds in an
optimized build. Because of this, I've kept all of this disabled by
default for now. The goal is that once caching and some other
improvements land, we can enable this by default.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
- Update bazel to 8.3.1, just to stay reasonably up to date.
- Bazel warned about the platforms version, so I generally updated
packages that have central registry versions.
- Note there's a newer re2 in the central registry, but I got a download
error with it.
- `--experimental_guard_against_concurrent_changes` is deprecated; I
wasn't sure it's worth explicitly setting
`--guard_against_concurrent_changes=full`, but figured it may be
consistent (it's not clear to me -- see
https://github.com/bazelbuild/bazel/pull/25874).
- Updates incompatible flags.
- `rules_flex` is no longer used, so enable its flag.
- Fixes `sh_test` deps for
`--incompatible_disable_autoloads_in_main_repo`
- Broadens the exception for `rules_cc` and `bazel_tools` due to changes
to runfiles deps; trying to avoid minutiae that shouldn't affect the
decision.
Fixes a compile failure with the new version, essentially:
```
external/+llvm_project+llvm-project/llvm/include/llvm/Support/FormatVariadicDetails.h:157:1: error: implicit instantiation of undefined template 'llvm::support::detail::missing_format_adapter<clang::LookupResultKind>'
```
rules_cc@0.1.0 was yanked from
[BCR](https://registry.bazel.build/modules/rules_cc) due to prematurely
removing cc_proto_library, this inconsistently causes the following
build error:
> ERROR: Error computing the main repository mapping: Yanked version
detected in your resolved dependency graph: rules_cc@0.1.0, for the
reason: rules_cc 0.1.0 is yanked due to incompatible change (prematurely
removing cc_proto_library from defs.bzl), please upgrade to 0.1.1.
Yanked versions may contain serious vulnerabilities and should not be
used. To fix this, use a bazel_dep on a newer version of this module. To
continue using this version, allow it using the --allow_yanked_versions
flag or the BZLMOD_ALLOW_YANKED_VERSIONS env variable.
This PR updates to 0.1.1 as recomended in warning and
[bazelbuild/rules_cc#268](https://github.com/bazelbuild/rules_cc/issues/268#issuecomment-2651269117).
- The actual reason I started this: minor lowering updates in the golden
LLVM IR
- Process.inc changed enough to need a patch context update.
- https://github.com/llvm/llvm-project/pull/123126 added `proto_library`
uses without a `load`, which is broken in bazel 8
- Just commenting these out because we don't use them. I'll follow up
separately about a possible fix, but continuing to use `WORKSPACE` is a
bigger issue LLVM probably should address.
- Note this update is also triggering removal of `migrate_cpp`, in #4887
The WORKSPACE file is deprecated; support is already off by default, and
it'll be removed in the next major bazel release. Our main dependency is
tree-sitter, and I'm trying to address that here.
We're currently using https://github.com/elliottt/rules_tree_sitter, but
that hasn't been updated in a couple years, meaning it lacks
MODULE.bazel support. In the registry, there's
https://registry.bazel.build/modules/tree-sitter-bazel, but this is only
the *parser* libraries of tree-sitter, not the *generator*. I'm using it
for that much, at least.
For the *generator*, which transforms grammar.js to parser.c/h, I'm just
requiring a non-hermetic invocation (i.e., people who want to work on it
will need to install tree-sitter; see the README.md updates). I tried
running it manually, but parser.c is about 600 KB; pre-commit rejects
files that large and I don't think an exception makes sense to override
for this (it'd probably also grow substantially if the grammar were
updated to cover more syntax). In order to make the non-hermetic call
not break "bazel build //..." for most developers, I'm marking most
targets in the package as manual.
Note, I did look long and hard at using `aspect_rules_js`/`rules_nodejs`
to invoke npm. This took a lot of time, and I have a commit that's
mostly working, except I hit a point where it uses `declare_symlink`
which we disallow for compatibility reasons (commit "Lots of work for
figuring out rule_js uses declare_symlink" on the PR). As a consequence,
I think we can't use the primary supported ways to have hermetic npm
calls.
Also, `treesitter` -> `tree_sitter` because it's generally called
`tree-sitter`, two words. We even had a `treesitter/src/tree_sitter`
directory so it's a bit inconsistent.
As far as bugs here, the parser library breaks bazel queries, e.g. the
error:
```
ERROR: Evaluation of query "somepath(//..., @llvm-project//third-party/unittest:gtest)" failed: preloading transitive closure failed: no such package '@@[unknown repo 'platforms' requested from @@tree-sitter-bazel+]//': The repository '@@[unknown repo 'platforms' requested from @@tree-sitter-bazel+]' could not be resolved: No repository visible as '@platforms' from repository '@@tree-sitter-bazel+'
```
I'm just excluding tree_sitter from queries where I can to work around
the error.
Just noticed the versions were old while I was updating tcmalloc in
#4784. I've tested and it doesn't seem to introduce issues; the compile
commands may actually work a little better.
This updates to bazel 8.0.0, also updating bazel mod deps and tools to
make that function. The release is a couple weeks old, and we haven't
updated in a while, and it's a major release. Note it includes some
incompatible flag flips that this is trying to update with respect to.
I'll try generally enabling incompatible support separately.
The most visible bazel behavior change here will be the change from `~`
to `+` in repo path names. (If you're curious,
https://github.com/bazelbuild/bazel/issues/23127 indicates this fixes a
Windows performance issue)
Note that this is building on top of both the action env update in #4728
(which got me started down this path) and the proto removal in #4731
(which would add significant work to this update). Only the commit
starting at "Work towards bazel 8.0.0" is specifically part of this PR.
As part of migrating to the latest bazel configurations in #4729, I'm
running into proto toolchain issues. For example:
"Error: <target @@protobuf+//:cc_toolchain> (rule
'proto_lang_toolchain') doesn't contain declared provider
'ProtoLangToolchainInfo'"
Although we may eventually want more use of proto, right now the only
use is for the explorer fuzzer. The explorer codebase is essentially
frozen, so continuing to run it isn't gaining us much (in fact, we've
already disabled autofuzzing for it).
So, rather than trying to fix the proto setup, this change:
1. Deletes `explorer/fuzzing`
2. Removes proto portions of `testing/fuzzing`, which were only in-use
by the explorer
3. Removes some ancillary proto support, which would otherwise break
from the bazel changes and would be difficult to validate as "still
working"
This change is partly isolated in order to make it easier to revive bits
of (3).
The only documentation I can find for `.bazelversion` is in
https://github.com/bazelbuild/bazelisk/blob/master/README.md. However,
`bazel` will error out if it doesn't match the `.bazelversion`. For
example:
```
╚╡/usr/bin/bazel build :all
ERROR: The project you're trying to build requires Bazel 7.3.0 (specified in [elided]/carbon-lang/.bazelversion), but it wasn't found in /usr/bin.
```
Switching to this because the `MODULE.bazel` requires tighter version
pinning in order to avoid churn, and this should help catch mistakes
early.
In `MODULE.bazel`, demote mention of the version issue because it should
only really occur now when the version is being deliberately changed, so
the connection should be more apparent.
The .tar.gz link is currently broken (I think wget used to work, now it
doesn't); not sure if that's deliberate since it's a download page. I'm
hoping the new location remains more reliable.
Note I tried using SourceForge's git directly. That works locally, but
on the action runners it seems to be blocked:
```
fatal: unable to access 'https://git.code.sf.net/p/perfmon2/libpfm4/': Failed to connect to git.code.sf.net port 443 after 5 ms: Connection refused
```
Chandler Carruth
Jon Ross-Perkins
Dana JansensandChandler Carruth
zadig
josh11bandJosh L
DavidLoftus
Boaz Brickner