Without this, basic `bazel test //...` style wildcards would build a
bunch of extra configurations because of gaps excluding things. With
this, the action count of a normal build should be much more reasonable.
The switch from `target_compatible_with` to tagging is based on looking
at what ends up being most idiomatic and easiest -- trying to articulate
the complex and convoluted compatible with restrictions that would avoid
extraneous build configurations was really painful and this seems much
simpler and easier to deploy in a systematic way.
While here, also change the name of a rule that confused me to no end
while debugging this -- the rule that installs a `.bzl` file that
happens to be spelled `carbon_runtimes` is very different from all of
the other "installed carbon runtimes" kind of things in the tree. Adding
the file extension helps make that (much) more obvious.
Note that this is essentially a re-do of #7088 but now without any
dependencies that can mess up the merge.
Assisted-by: Antigravity with Gemini
This shouldn't change any functionality, but simplifies (significantly)
the logic in the installed toolchain, and also provides a better
conceptual balance between these.
I've tried to minimize the changes beyond a pure refactoring, but it was
a bit tricky to get everything working so some things have been mixed
in...
Assisted-by: Antigravity with Gemini
This reverts commit 4babfdbf22.
This was a stacked PR that was merged by accident, losing the commit and
description of the base change. Reverting and will re-land
independently.
Without this, basic `bazel test //...` style wildcards would build
a bunch of extra configurations because of gaps excluding things. With
this, the action count of a normal build should be much more reasonable.
The switch from `target_compatible_with` to tagging is based on looking
at what ends up being most idiomatic and easiest -- trying to articulate
the complex and convoluted compatible with restrictions that would avoid
extraneous build configurations was really painful and this seems much
simpler and easier to deploy in a systematic way.
While here, also change the name of a rule that confused me to no end
while debugging this -- the rule that installs a `.bzl` file that
happens to be spelled `carbon_runtimes` is very different from all of
the other "installed carbon runtimes" kind of things in the tree. Adding
the file extension helps make that (much) more obvious.
Assisted-by: Antigravity with Gemini
This takes the bootstrap support that was added and makes it available
under convenient user-facing flags for while we're doing development.
For example, to build a bootstrap compiler and use it to build and run
the tests under `//common/...` you can now use:
```
bazel test --//:bootstrap_stage=1 --//:bootstrap_exec_config=true //common/...
```
This will use the stage1 bootstrap compiler, and it will build that
compiler in the exec config (so it is optimized and the above even works
when cross-building with Bazel).
Assisted-by: Antigravity with Gemini
Noticed this when testing the Carbon toolchain with a more complex
environment, don't have any way to observe this at the moment in Bazel
though.
Assisted-by: Antigravity with Gemini
This worked correctly in the system Clang toolchain, but was not
configured correctly in the Carbon toolchains. The test is designed to
let us cover all of these.
Assisted-by: Antigravity with Gemini
The runtimes and bootstrap Bazel logic was previously built around
defining custom Bazel platforms constrained with `constraint_settings`.
The use of platforms added significant complexity, including the need to
"save" and "restore" the original platform, and other complexity
stemming from changing the platform as a whole.
This PR switches to use the simpler tool of build settings, and
`target_settings` on the toolchain rather than platform compatibility.
This remove the entire need to save and restore the platform, and also
generally simplifies things.
This PR also fixes some bugs in the bootstrap that were hidden by the
use of platforms, such as the need to carefully manage the different
inputs to the runtimes build so that generated inputs pick up the
correct exec configuration -- the exec transition happened to do this
"automatically", but it seems better to handle explicitly. And it cleans
up an extraneous copy of `carbon_runtimes.bzl` that snuck in somehow.
Assisted-by: Antigravity with Gemini
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
Fixes#7031
Also switches the previous symlinks test to be a more full integration
test. While a bit slow, it does seem worthwhile to have something that
tests things end-to-end, both with the prebuilt runtimes and the
on-demand runtimes. This test is already reasonably well separated from
the rest of the toolchain so incremental development shouldn't be
negatively impacted. And since we turned off ASan by default, it isn't
completely infeasibly expensive.
Assisted-by: Antigravity with Gemini
This also switches to a more Bazel-based install layout, skipping the
FHS-based synthetic layout. The FHS-based layout is still reconstructed
explicitly when building an installable tar-ball.
The biggest change is to configure the just-built install as a Bazel
toolchain, including allowing it to build its own runtime libraries as
native Bazel libraries. This removes the need for a monolithic runtimes
build, all of that code logic is removed.
This should also pave the way to using the just-built toolchain for
doing a full 3-stage bootstrap. Building the 2nd stage is included here
as it was a particularly effective way to test that the Bazel
integration was fully working. Adding a 3rd-stage check for stability is
future work, but should be pretty easy.
There is a down-side: this uses the busybox to do the runtimes
compilation, which means they will be re-built after ~any change to
Carbon. However, the integration with Bazel should largely pay for this,
and we can continue to factor the tests away from depending on built
runtimes in most cases.
Now that we're building and testing the runtimes more directly, this
surfaced a problem with the layout of runtimes on macOS that is fixed
here. All of the Darwin OSes use a custom layout for their resource
directory compared to other targets. We now model this in both the C++
built runtimes and the Bazel built runtimes.
Assisted-by: Gemini via Antigravity
directory
This cleans up the `//toolchain/install/BUILD` file and the tree
generally to be more focused on arranging the actual installation rather
than preparing inputs to that installation.
I picke `//toolchain/runtimes` so we can put other runtimes preparation
logic there, but open to any other suggested organization.
There are other runtimes things that would in theory make sense to move
such as the `prebuilt_runtimes` logic, but a subsequent PR will delete
those and so I'm leaving them where they are for now.
This shifts the Bazel toolchain configuration of our installation to
build all of the Clang runtimes Carbon uses on-demand natively in Bazel.
We export the information about how to build into a generated Starlark
file, and emit BUILD files and Starlark logic into the installation to
orchestrate the build.
This requires some complex management of Bazel toolchains -- we need to
first set-up a "runtimes toolchain" that doesn't have runtimes of its
own, but can be used to _build_ runtimes. Then we build the runtimes
using that toolchain, and assemble them into the standard layout for a
Carbon runtimes tree. Finally we configure the _actual_ toolchain with
this built tree.
Currently, this is only setup for the installed toolchain, but I plan to
factor this runtimes build into one that can be used directly as well to
break up the monolithic runtimes build step into Bazel-integrated build
of the runtimes. This will also serve as the foundation for adding
bootstrapping support directly to our Bazel build.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
This moves the most complex of the logic fully into Starlark: both the
many different platform sources list, and the overriding of generic
files with architecture specific files.
This also fixes significant bugs in the AArch64 build where we were
skipping numerous files: all of the outlined atomics and `emupac.cpp`.
This PR forcibly disables `emupac.cpp` as fixing that will require a
more significant change.
We previously set `LLVM_SYMBOLIZER_PATH` to a bogus path ending
`.../binllvm-symbolizer`. Because this var was set, LLVM's symbolizer
lookup would also skip looking in `$PATH`, so this was causing
symbolization to never happen unless `LLVM_SYMBOLIZER_PATH` was
explicitly set in the environment.
The goal here is to be able to construct a build of the runtimes
directly in Bazel, or by emitting `BUILD` files, or by emitting into C++
code and using that on-demand. For that, we want a single source of
truth, and that source in Starlark.
This should also make the information more generally useful, and so I'm
moving as much as I can into the LLVM Bazel build. Apologies as that
makes the diffs extra annoying.
I do plan on upstreaming the Bazel parts of this, but would like to get
everything working in Carbon and stabilized first.
While here, I've also made a change suggested for the future in the
initial review by lifting the C++ template out of a string literal in
the `.bzl` file, and into an actual separate C++ file.
This only moves libc++, libc++abi, and libunwind. I want to get those
three working end-to-end before I work on the builtins or `crtbegin` and
`crtend`, as those have a bunch of additional complexity.
This also only uses the info in the C++ on-demand build. It seemed like
a reasonable increment to start code review, and my plan is to work on
other build strategies in a follow-up PR. If that doesn't work, let me
know and I'll come back once I have at least a second use of the info
here.
When the response file contains the subcommand itself, or when there are
`-Xcarbon` flags within the response file that we need to re-organize,
we need to hoist the expansion into the busybox itself.
I've left the response file expansion in the `ClangRunner` so that
library users can still use them, including in the VFS of the runner.
It's also useful to handle `-Xcarbon`-style flags even when using
subcommands rather than a symlink to the busybox: build systems often
have a facility to append flags, but appending doesn't let us inject
flags easily into the `carbon` driver itself. So this PR moves the
`-Xcarbon` reorganization to happen in all cases, and to insert them
before the first subcommand or positional parameter. When teaching Bazel
to link by running `carbon link ...` commands, this lets us do things
like `bazel build --linkopt=-Xcarbon=-v` to enable verbose logging.
I've not added a test here as we don't really have much testing of the
busybox. I can move the current symlinks test to be more of an
integration test of the busybox logic if desired, but would be a
somewhat larger change and maybe worth separating out. This will end up
tested in the Bazel example in a subsequent PR that starts using it in
the installed crosstool configuration.
This removes support for strange symlink structures _within_ an
install-shaped tree, but AFAIK, that is not one of the (frustratingly
many) cases where we need them. Avoiding this significantly shortens and
reduces repetition in the commandline formed by the busybox, and also
appears to work better when running the busybox from inside a Bazel
checkout.
The motivation here is to fix issues that arose when more heavily using
the installed toolchain with the example Bazel project. As more of that
functionality lands, this should also be tested there.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This let's you point Bazel at an installed toolchain or download one of
our release archives. When you do, it will configure itself as a C++
Bazel toolchain. This toolchain works reasonably well, but doesn't cache
the C++ runtimes, and so linking is inefficient. The next step will be
to pivot the runtimes from the implicitly on-demand (which can't cache
when using a sandboxed build system like Bazel) to _explicit_ on-demand
runtimes directly with Bazel support.
I've included an example Bazel project that uses this and provides a
bunch of documentation and an example script that should let folks try
this out easily.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
Co-authored-by: David Blaikie <dblaikie@gmail.com>
The `install_paths` library depends on the `llvm_tools` library, which
depends on all of LLVM in order to allow _invoking_ the LLVM tools in
addition to listing and manipulating them. The `install_paths` also
depends on the Clang version number which for some reason depends
transitively on a large fraction of LLVM. That should probably be fixed,
but we don't actually need it anyways, we can just prune our dependency.
Because the digest builder is built in the _exec_ configuration, this
was pulling in most of LLVM and Clang to build in the exec configuration
as well, adding about 4000 actions or a roughly 30% overhead to complete
rebuilds. The time impact is likely closer to 2x because many of the
slowest actions are here.
Hopefully this makes our bots take much less time when rebuilding.
Previously, we used the FHS "prefix" concept as the basis of the
install, but this makes it hard to integrate an installed toolchain with
Bazel (or similar) build system where it wants the "root" of the
toolchain to have some specific files (`MODULES.bazel` or
`BUILD.bazel`), and cannot reference anything outside that directory
tree.
An easy solution is to make the `lib/carbon` directory the root of the
install and never walking up from it. Then we simply have a `bin/carbon`
symlink to the busybox that is useful for getting the command into the
PATH, but isn't used for anything else. The FHS-constrained install
paths surround a root we fully control the layout and files within.
While initially motivated by trying to make a single toolchain structure
that works both for installation and for Bazel, it actually makes the
paths we end up using in the toolchain much simpler. We no longer have
awkward `.../lib/carbon/../../lib/carbon/...` sequences in the toolchain
which is cleaner and even a (trivial) efficiency gain.
As I was doing this I noticed several out-of-date comments that I tried
to fix, and I tried to improve some code reuse rather than re-computing
paths.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
This enables on-demand building of runtimes by default, and enables
their header files for all of the Clang invocations. This also switches
the default flags to use the LLVM-provided runtimes (compiler-rt,
libunwind, and libcxx).
This also switches even `llvm_symlinks_test` to use the Bazel prebuilt
runtimes, which requires having a way to pass a Carbon flag even when
invoking the busybox as `clang` or `clang++`. This uses the pattern that
has worked for other Clang wrappers of spelling flags:
`-X<tool-name>=--flag=value`
Last but not least, this updates the Carbon Bazel rules to use our
installed and the Bazel prebuilt runtimes. With that, we make the C++
interop hello-world be enabled by default as this should pass reliably
on both Linux and macOS now.
When building in Bazel actions, notably building runtimes, using
absolute paths makes the results non-hermetic and generally less
cache-friendly.
This restructures the code to only form an absolute path as part of the
`bazel run` change of working directory. It also tries to make the API
for doing this a bit more clear by taking the `exe_path` and
transforming it internally.
To support this, this PR also generalizes the `RemovingDir` to support
relative paths. While these can be tricky -- the working directory needs
to not change while they exist -- that isn't a reason to fully exclude
them and they're useful for implementing relative-path runtimes, etc.
Previously this was kept in `//toolchain/install` so it would be near to
the code that actually defines the installation layout. However, that
creates somewhat unfortunate dependency cycles between
`//toolchain/install` and other directories. Exacerbating this, a
subsequent PR is likely to add dependencies on it from
`//toolchain/base` itself that suggests that is the correct layering.
This PR just moves the code mechanically with as few other edits as
possible.
This builds on the previous work to flesh out more on-demand runtimes
building. It adds building of the `libc++.a` archive runtime.
A number of changes are required for this to work:
- The runtimes build infrastructure needs to support building sources
from multiple parts of LLVM rather than a single part. We do this by
lifting the root of the runtimes source paths up a level to a common
runtimes tree, and installing the runtimes sources below this
directory.
- Both libc++ and libc++abi runtimes sources need to be installed, and
we even need to install some interesting parts of llvm-libc that are
used in the build of libc++.
- We need to generate the site configuration header file for libc++ from
the CMake template. This includes both setting up a set of
platform-independent defines and introducing some basic Bazel support
for processing the CMake template itself.
Doing all of this also exposed some missing features and limitations of
the runtimes building infrastructure that are addressed here.
One note is that all of this just adds libc++ to the explicit
`build-runtimes` command for testing. It doesn't yet trigger
automatically building these prior to linking, or configuring any of the
other subcommands to automatically use these runtimes. All of that will
come in follow-up PRs.
Also, this makes the `clang_runtimes_test` ... _very_ slow in our
default build configuration. Compiling libc++, even with many threads on
a large Linux server requires up to 50 seconds. I'm open to any
suggestions on how to handle this, including disabling the test in
non-optimized builds. I have some ideas to speed this up, but
fundamentally building libc++ is... not cheap.
I did look at some of the existing Bazel tools to process the CMake
template, but they all seemed significantly more complex than what we
need and didn't have broad adoption. Given that, it seemed slightly
better to just roll our own given the simple format.
Two of the new LLVM patch are currently under review upstream and so
hopefully temporary:
- https://github.com/llvm/llvm-project/pull/169155
- https://github.com/llvm/llvm-project/pull/169292
This is the first real step towards building libc++ itself, and fleshes
out both the core runtimes management logic and the archive-based
runtimes logic for a quite simple runtime.
Nothing here causes us to _use_ libunwind, and in fact this doesn't
include even the "on-demand" aspect of building `libunwind`. Instead,
this just wires it up to the explicit `build-runtimes` subcommand for
simple testing. The full integration along side the target directory is
future work.
This layer allows runtimes to be built on-demand but cached in a
consistent and re-usable location on the system. It handles careful
filesystem operations to ensure consistency even in the face of multiple
versions and build configurations.
This addresses a number of TODOs from the initial runtimes building
on-demand, and sets the stage to scale up to more runtimes.
This doesn't switch on-demand runtimes to be on by default, I wanted to
wait and make that change as a separate step.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
This is the first step to having Clang's runtime libraries fully
available for the Carbon toolchain. This PR focuses on the lowest level
runtimes, the CRT files and the builtins library.
The goal is to intercept Clang runs where it needs these
target-dependent pieces to be available, and build them on demand using
our Clang-running infrastructure. This avoids most of the subprocess
overhead, but there is still some due to missing features in Clang.
This requires exporting the sources for these runtimes from the Bazel
build, and installing them in our target-independent resource directory.
We then build a simplified "build" of these sources within the
`ClangRunner` itself to produce the specific artifacts and layout
expected by Clang.
It also required fixing our use of Clang on macOS to have a default
system root in order to successfully compile or link.
It also required cleaning up how the `ClangRunner` used target
information more generally -- instead of taking the target as
a constructor parameter, it manages its target internally and relies on
the Clang target-specifying command line flags.
I looked at whether we could split this into another layer separate from
the `ClangRunner`, but that proved frustratingly difficult to manage.
While we support building these on-demand as part of a detected link,
that doesn't seem feasible as we don't have the necessary separation
between compilation runs of Clang and link runs of Clang. However,
I have tried to factor the internals to provide as clear of separation
as I could across these.
I have also created a stand-alone subcommand to directly build the
runtimes which allows for easy testing. It also supports building them
into a specific directory, and that directory can in turn be passed to
a Clang invocation. This is designed to work both at the API level with
`ClangRunner` and at the subcommand level.
Currently, the only part of the commandline that is detected and
forwarded to the runtimes build is the target. Eventually, the plan is
to expand this so that we can build a maximally tailored set of runtimes
for a given compilation.
The other big TODO here is to actually implement caching storage of
these runtimes so they aren't built on every execution. Right now, this
uses a somewhat hack-y build of a temporary directory, but this isn't
expected to be suitable long-term. Building these runtimes on *every*
link makes those commands take approximately 15 seconds with an ASan
build like our default development build, and just over 2 seconds in an
optimized build. Because of this, I've kept all of this disabled by
default for now. The goal is that once caching and some other
improvements land, we can enable this by default.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This removes a bunch of manual filesystem helpers and complexity that
are directly provided by the new library.
It also moves all of the install paths detection to use
`std::filesystem::path` instead of the LLVM path library. The goal is to
consolidate all our logic onto a single stack, and the standard one
seems the best for that purpose. This does give up some of the
optimizations of this code to avoid memory allocation, but in practice
that likely isn't a critical issue. And with the new filesystem library
we can likely do more to avoid that by using directory-object-relative
filesystem access. However, that will have to wait for moving more parts
of the toolchain over to use this set of filesystem abstractions. There
is a related TODO left in the manifest handling code.
Stop using the clang tooling library to build an ASTUnit; that library
is set up to process clang frontend arguments, assuming that something
has already built frontend arguments from the compiler arguments. It is
also too encapsulated and doesn't let us inspect and modify the compiler
invocation before it's executed.
Instead, build the AST unit directly in two phases:
* FIrst, take a list of clang driver arguments and convert them into a
list of compiler arguments, using `clang::createInvocation`. Internally,
this uses the clang driver to build a frontend invocation, including
building system-specific include paths as needed.
* Then, directly build an ASTUnit from this compiler invocation.
I've factored this so that we can split out the `createInvocation` step,
with the intention that we may want to move it out of check and into the
carbon driver with the rest of the driver-level argument handling, and
we may want to customize some of the clang options before we invoke the
clang frontend with that set of options.
In order to make the invocation reusable, it no longer depends on the
name of the carbon file importing the C++ code. In place of synthesizing
a header file name as `<foo.carbon>.generated.cpp_imports.h`, we now
insert line marker directives into the generated header so that errors
in that header cause Clang to point a diagnostic back at the Carbon
source file itself. This results in a minor improvement in the
diagnostic output: we no longer refer to a nonexistent generated file.
But the snippet still contains text that doesn't match the source code,
so it remains imperfect.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
When finding an executable, this validates that the returned binary is a
symlink back to the same thing as /proc/self/exe, also using that as a
fallback for different things.
Looking back at #3912, we started using `findProgramByName` in order to
avoid path canonicalization done by `GetMainExecutable`. That created
issues as in #5096, wherein an `argv[0]` that's not explicit enough
(`llvm-symbolizer` instead of the full path, done in [LLVM's
Signals.cpp](https://github.com/llvm/llvm-project/blob/4f60f45130c6bd96c79e468fe9927a29af760f56/llvm/lib/Support/Signals.cpp#L198))
leads to incorrect results (finding an `llvm-symbolizer` in `$PATH`).
One option to fix this would be to patch LLVM to provide an absolute
path for `llvm-symbolizer`. However, I'll suggest that passing a
filename in `argv[0]` is not terribly uncommon, and could be a migration
limitation if we force it. The failure mode is also opaque; for example:
```
$ /bin/sh -c "exec -a llvm-symbolizer ./bazel-bin/toolchain/carbon"
error: expected carbon-busybox symlink at `/usr/lib/llvm-19/bin/llvm-symbolizer`
```
Combined with the `setenv` of `LLVM_SYMBOLIZER_PATH` in
`busybox_main.cpp`, this is intended to fix#5096.
Found this is very useful for debugging since otherwise it's hard to
tell what happened in the process.
Based on similar logic in `scripts/target_determinator.py`.
Rules executed by bazel don't necessarily have the right environment to
find the symbolizer, which was the intent of `cc_env` setting
`LLVM_SYMBOLIZER_PATH`. So far, this has kind of been a case-by-case
fix, but every so often I'm trying to debug a crash in a test that
doesn't provide it. Rather continuing down this route, instead add
drop-in wrappers for cc rules so that it's hard to forget.
Note `bazel/cc_rules` is intended to mirror `bazel/carbon_rules` and
`bazel/cc_toolchains`, rather than `@rules_cc`.
AFAICT there isn't a great way to add this as a default for the `bazel
run` environment. It's not typically going to be set on its own,
forwarding `$PATH` would be too broad, and the [action
`env_sets`](https://bazel.build/docs/cc-toolchain-config-reference#using-action-config)
I think are not quite what we need (I think those don't include output
execution, only compilation).
We only had the single `clang` symlink, but in case its useful to use
the toolchain with some other build system that expects `clang++`, or
even `clang-cl` or `clang-cpp`, fill in the rest of the symlinks.
The different `clang` flavors don't really need anything to support in
the subcommand as there is already an excellent way to get the exact
behavior of these names using Clang's `--driver-mode` flag, so these
just use that. That makes this change really *only* about busybox
behavior.
We don't really have a dedicated test path for things that are only
exposed via the symlinks, so I've added a simple Python integration test
we can use for that. I can backfill some testing of other symlinks if
useful (the `ld.lld` one might be worthwhile), although there is minimal
interesting logic to cover there.
This adds support for most of the remaining LLVM command line tools
using a generic, generated wrapper. The subcommand interface for these
is (much) less interesting than our other subcommands, but it gives us
a uniform and consistent layer.
Note that I structured these as nested sub-sub-commands below an `llvm`
subcommand because of an expectation that we will want to add more, and
ones that don't use this generic layer. Some concrete future work:
- Add the `opt` and `llc` tools as subcommands for easier debugging and
experimentation with LLVM IR output from Carbon's toolchain.
- Potentially sink `lld` below the `llvm` layer given that it has
significantly less user visibility than commands like `clang`.
Unfortunately, the current driver subcommand APIs make nested
subcommands awkward. I've added a somewhat rough hack here to let the
LLVM tools go in, but there are some TODOs that I want to address in
a follow-up that works to adjust the structure of this code to be more
conducive to nesting like this.
Depends on #5048 -- only the last commit should be reviewed here.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
This removes the separately built and installed LLD binary. The symlinks
used by Clang when directly invoking LLD now point back to the main
`carbon-busybox` binary and dispatch through the newly added subcommand.
With this change we're down to shipping a single busybox binary in the
toolchain, removing duplicate installed copies of LLD and all its LLVM
dependencies. =]
The LLD subcommand works a bit differently from the `clang` subcommand
because the CLI for LLD is specific to which platform flavor of linker
is being invoked.
As part of this, I've extracted some of the common functionality in the
Clang runner into a base class that can be re-used. I expect to use this
again in a follow-up change to add subcommands to run other LLVM tools.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Sadly, the Clang driver unconditionally injects the `-disable-free` flag
to CC1 invocations. =/ This ends up with us leaking memory when invoking
Clang programmatically, for example in unit tests.
I've fixed this by post-processing the flags. The alternatives I see all
involve duplicating even more code from within Clang's internals into
our runner, and we already have a lot of that. I have left a TODO to try
and follow up with upstream about fixing this in a more sustainable way,
but wanted to get the testing in place anyways.
I've also threaded a flag through the various layers so that when we're
on the command line we can actually skip this and get the same compile
time benefits that Clang itself gets from disabling freeing all of the
internal data structures.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This digs the Clang builtin headers out of the Clang package and
reconfigures them to install into our installation prefix, under the
LLVM installation subtree.
---------
Co-authored-by: Ilya Biryukov <ibiryukov@google.com>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
The language server needs stdin, and for tests we should be passing it
around. My intent is to pass in a faux stdin to Driver for language
server tests.
As long as I'm adding a new parameter, I was looking at also changing
the way streams are passed in to Driver for style (pointers since
they're held past construction lifetime). Since these are all stored in
DriverEnv, I thought it might be a net improvement to use the struct
directly, getting more explicit parameter names and also removing the
need for `SetFuzzing`.
I'm trying here to avoid functional changes, but there are a couple
additional fixes like removing an obsolete `find_insensitive` and
refactoring how `ValidateOptions` handles errors (because it reduces the
number of spots that operate on error_stream).
Both of these I noticed from testing #4785, but they occur at head.
```
(elided)/execroot/_main/common/raw_hashtable.h:532:40: error: do not use nested 'std::max' calls, use an initializer list instead [modernize-min-max-use-initializer-list,-warnings-as-errors]
532 | static constexpr ssize_t Alignment = std::max<ssize_t>(
| ^
533 | {alignof(MetadataGroup), alignof(StorageEntry<KeyT, ValueT>)});
| ~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| static_cast<long>(alignof(MetadataGroup)) static_cast<long>(alignof(StorageEntry<KeyT, ValueT>))
(elided)/execroot/_main/toolchain/install/busybox_info_test.cpp:259:8: error: unused local variable 'usr_prefix' of type 'std::filesystem::path' [bugprone-unused-local-non-trivial-variable,-warnings-as-errors]
259 | auto usr_prefix = MakeInstallTree(dir_ / "usr");
| ^
(elided)/execroot/_main/toolchain/install/busybox_info_test.cpp:260:8: error: unused local variable 'usr_local_prefix' of type 'std::filesystem::path' [bugprone-unused-local-non-trivial-variable,-warnings-as-errors]
260 | auto usr_local_prefix = MakeInstallTree(dir_ / "usr/local");
| ^
```
The std::max diagnostic seems a little confused, but the initializer
list seems like it can be dropped without any loss. The unused locals
diagnostic is correct.
Neither of these seem like they should be newer than my last clang-tidy
pass, maybe I just missed them in other sweeps.
This restores the symlinks for the installation, but teaches the busybox
info search to look for a relative path to the busybox binary itself
before walking through symlinks. This let's it find the tree structure
when directly invoking `prefix_root/bin/carbon` or similar, either
inside of a Bazel rule or from the command line, and mirrors how we
expect the installed tree to look. This works even when Bazel resolves
the symlink target fully, and potentially to something nonsensical like
a CAS file.
In order to make a convenient Bazel target that can be used with `bazel
run //toolchain`, this adds an override to explicitly set the desired
argv[0] to use when selecting a mode for the busybox and a busybox
binary. Currently, the workaround uses an environment variable because
that required the least amount of plumbing, and seems a useful override
mechanism generally, but I'm open to other approaches.
This should allow a few things to work a bit more nicely:
- It should handle sibling symlinks like `clang++` to `clang` or
`ld.lld` to `lld`, where that symlink in turn points at the busybox.
We want to use *initial* `argv[0]` value to select the mode there.
- It avoids bouncing through Python (or other subprocesses) when
invoking the `carbon` binary in Bazel rules, which will be nice for
building the example code and benchmarking.
It does come at a cost of removing one feature: the initial symlink
can't be some unrelated alias like `my_carbon_symlink` -- we expect the
*first* argv[0] name to have the meaningful filename for selecting
a busybox mode.
It also trades the complexity of the Python script for some complexity
in the busybox search in order to look for a relative `carbon-busybox`
binary. On the whole, I think that tradeoff is worthwhile, but it isn't
free.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Pursuant to discussion regarding #4699, turn on
`misc-non-private-member-variables-in-classes` using the
`IgnoreClassesWithAllMemberVariablesBeingPublic` flag (the check treats
structs as classes, so we need this for structs with all-public
members). Updates the style guide notes to match, which should be pretty
minor due to the scoping of test fixtures.
Also fixes some underscore uses in test files on the way. Basically this
is keeping the style for [class data member
naming](https://google.github.io/styleguide/cppguide.html#Variable_Names)
even while making them public.
As discussed in #4530 . This required switching to using
`llvm::IntrusiveRefCntPtr` in a number of places.
---------
Co-authored-by: Josh L <josh11b@users.noreply.github.com>
Chandler Carruth
Richard Smith
Boaz Brickner
Thomas Köppe
josh11bandJosh L