Copy MultiplexExternalSemaSource.h and MultiplexExternalSemaSource.cpp
from https://github.com/llvm/llvm-project/pull/204458 into
third_party/llvm, and apply a few minor changes to allow them to compile
and pass precommit checks.
This allows
`0011-Add-empty-constructor-and-GetSources-method-to-Multi.patch` to be
removed, which brings Carbon closer to being able to compile on an
unmodified LLVM toolchain.
In generate_ast.cpp, an `CarbonExternalASTSource` is installed that has
a `Check::Context` pointer. During lowering, this `ExternalASTSource` is
still installed, and using it can cause a crash if the now-invalid
pointer is dereferenced.
Fix by adding a new `ReadOnlyASTSource` in sem_ir, and using that during
lowering.
`CarbonExternalASTSource` now inherits from `ReadOnlyASTSource` to avoid
some code duplication.
In generate_ast.cpp, we now always install a multiplex source, even if
there's only one child source. Clang internally keeps pointers to the
top-level `ExternalASTSource` installed via `setExternalSource`, and
those pointers aren't updated if `setExternalSource` is called again. By
using `MultiplexExternalSemaSource`, we can keep the top-level
`ExternalASTSource` pointer the same, and only update its children.
Using `MultiplexExternalSemaSource` this way requires a new constructor
and a method to modify its child sources; added a new LLVM patch adding
those.
Originally landed in #7335, reverted in #7353 due to ASAN errors.
Changes since original:
* Use LLVM RTTI to make `Lower::Context::Finalize` less brittle.
Add LLVM RTTI to `ReadOnlyASTSource` (and `CarbonExternalASTSource`).
Change Finalize so that instead of just deleting the last multiplex
child source, it erases any multiplex child sources that match
`ReadOnlyASTSource`; this includes `CarbonExternalASTSource` since it's
a subclass.
* Fix ASAN error by updating the `MultiplexExternalSemaSource` earlier
in lowering. It is sometimes accessed during PrepareToLower, so update
it in `Context::GetFileContext` rather than `Context::Finalize`.
Fixes https://github.com/carbon-language/carbon-lang/issues/7142
In generate_ast.cpp, an `CarbonExternalASTSource` is installed that has
a `Check::Context` pointer. During lowering, this `ExternalASTSource` is
still installed, and using it can cause a crash if the now-invalid
pointer is dereferenced.
Fix by adding a new `ReadOnlyASTSource` in sem_ir, and using that during
lowering.
`CarbonExternalASTSource` now inherits from `ReadOnlyASTSource` to avoid
some code duplication.
In generate_ast.cpp, we now always install a multiplex source, even if
there's only one child source. Clang internally keeps pointers to the
top-level `ExternalASTSource` installed via `setExternalSource`, and
those pointers aren't updated if `setExternalSource` is called again. By
using `MultiplexExternalSemaSource`, we can keep the top-level
`ExternalASTSource` pointer the same, and only update its children.
Using `MultiplexExternalSemaSource` this way requires a new constructor
and a method to modify its child sources; added a new LLVM patch adding
those.
https://github.com/carbon-language/carbon-lang/issues/7142
When building Carbon on Fedora, clang reports the
resource directory as `/usr/bin/../lib/clang/22`.
This fails to string match against `/usr/lib/clang/22`
and so bazel reports an error.
This PR canonicalizes the path returned by clang
so that it will string match successfully.
Not sure how these got missed when moving other things to `uv`, but this
should clean them up.
The bump to Python 3.12 is so that we can use `@override` with the
simple import from `typing`. This is needed by the newest versions of
`ty` to do type checking. Added the relevant `@override` annotations.
Assisted-by: Antigravity with Gemini
This removes the need to install any specific version of Python or
figure out how to configure it by instead asking users to install `uv`
and letting it manage Python. Among other advantages, `uv` is designed
to be fast enough to embed directly into our scripts.
We were already using this in `bench_runner.py` so that the script could
import non standard library dependencies. Moving to it for the rest of
our Python unifies the approach and will also enable dependencies
whenever needed.
I've left `github_tools` alone as it has special handling with its own
Bazel setup.
I've updated the contributing tools to explain the approach here.
Use `Label` to mark labels that are local to this module. Remove
workspace root when forming manifest. Add explicit import for name that
is not available implicitly in an imported module.
Assisted-by: Gemini via Antigravity
Without this, basic `bazel test //...` style wildcards would build a
bunch of extra configurations because of gaps excluding things. With
this, the action count of a normal build should be much more reasonable.
The switch from `target_compatible_with` to tagging is based on looking
at what ends up being most idiomatic and easiest -- trying to articulate
the complex and convoluted compatible with restrictions that would avoid
extraneous build configurations was really painful and this seems much
simpler and easier to deploy in a systematic way.
While here, also change the name of a rule that confused me to no end
while debugging this -- the rule that installs a `.bzl` file that
happens to be spelled `carbon_runtimes` is very different from all of
the other "installed carbon runtimes" kind of things in the tree. Adding
the file extension helps make that (much) more obvious.
Note that this is essentially a re-do of #7088 but now without any
dependencies that can mess up the merge.
Assisted-by: Antigravity with Gemini
This shouldn't change any functionality, but simplifies (significantly)
the logic in the installed toolchain, and also provides a better
conceptual balance between these.
I've tried to minimize the changes beyond a pure refactoring, but it was
a bit tricky to get everything working so some things have been mixed
in...
Assisted-by: Antigravity with Gemini
This reverts commit 4babfdbf22.
This was a stacked PR that was merged by accident, losing the commit and
description of the base change. Reverting and will re-land
independently.
Without this, basic `bazel test //...` style wildcards would build
a bunch of extra configurations because of gaps excluding things. With
this, the action count of a normal build should be much more reasonable.
The switch from `target_compatible_with` to tagging is based on looking
at what ends up being most idiomatic and easiest -- trying to articulate
the complex and convoluted compatible with restrictions that would avoid
extraneous build configurations was really painful and this seems much
simpler and easier to deploy in a systematic way.
While here, also change the name of a rule that confused me to no end
while debugging this -- the rule that installs a `.bzl` file that
happens to be spelled `carbon_runtimes` is very different from all of
the other "installed carbon runtimes" kind of things in the tree. Adding
the file extension helps make that (much) more obvious.
Assisted-by: Antigravity with Gemini
Rather than defining our own action groups, work to re-use the
`rules_cc` ones, as they are (much) more comprehensive. Also, completely
eliminate the `codegen` action group as it was not well used. For
example, `-march` flags and `-O` flags change the preprocessor macros
defined. There isn't a really great "codegen" heuristic, so just pass
those flags to all compiles which is simpler anyways.
I'm tempted to do the same with preprocessor actions, but maybe it makes
sense to have that one stay separate.
Assisted-by: Antigravity with Gemini
This takes the bootstrap support that was added and makes it available
under convenient user-facing flags for while we're doing development.
For example, to build a bootstrap compiler and use it to build and run
the tests under `//common/...` you can now use:
```
bazel test --//:bootstrap_stage=1 --//:bootstrap_exec_config=true //common/...
```
This will use the stage1 bootstrap compiler, and it will build that
compiler in the exec config (so it is optimized and the above even works
when cross-building with Bazel).
Assisted-by: Antigravity with Gemini
Noticed this when testing the Carbon toolchain with a more complex
environment, don't have any way to observe this at the moment in Bazel
though.
Assisted-by: Antigravity with Gemini
This worked correctly in the system Clang toolchain, but was not
configured correctly in the Carbon toolchains. The test is designed to
let us cover all of these.
Assisted-by: Antigravity with Gemini
The runtimes and bootstrap Bazel logic was previously built around
defining custom Bazel platforms constrained with `constraint_settings`.
The use of platforms added significant complexity, including the need to
"save" and "restore" the original platform, and other complexity
stemming from changing the platform as a whole.
This PR switches to use the simpler tool of build settings, and
`target_settings` on the toolchain rather than platform compatibility.
This remove the entire need to save and restore the platform, and also
generally simplifies things.
This PR also fixes some bugs in the bootstrap that were hidden by the
use of platforms, such as the need to carefully manage the different
inputs to the runtimes build so that generated inputs pick up the
correct exec configuration -- the exec transition happened to do this
"automatically", but it seems better to handle explicitly. And it cleans
up an extraneous copy of `carbon_runtimes.bzl` that snuck in somehow.
Assisted-by: Antigravity with Gemini
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
Also increases the default optimization to `-Og` which is likely to give
faster turn-around time which is what we want to optimize for here. This
should also _substantially_ shrink binary sizes, etc.
ASan is still available via `--config=asan`, and is added to the CI
infrastructure. However, my current thought is to only run it after push
rather than in PRs and in the merge queue.
This also switches to a more Bazel-based install layout, skipping the
FHS-based synthetic layout. The FHS-based layout is still reconstructed
explicitly when building an installable tar-ball.
The biggest change is to configure the just-built install as a Bazel
toolchain, including allowing it to build its own runtime libraries as
native Bazel libraries. This removes the need for a monolithic runtimes
build, all of that code logic is removed.
This should also pave the way to using the just-built toolchain for
doing a full 3-stage bootstrap. Building the 2nd stage is included here
as it was a particularly effective way to test that the Bazel
integration was fully working. Adding a 3rd-stage check for stability is
future work, but should be pretty easy.
There is a down-side: this uses the busybox to do the runtimes
compilation, which means they will be re-built after ~any change to
Carbon. However, the integration with Bazel should largely pay for this,
and we can continue to factor the tests away from depending on built
runtimes in most cases.
Now that we're building and testing the runtimes more directly, this
surfaced a problem with the layout of runtimes on macOS that is fixed
here. All of the Darwin OSes use a custom layout for their resource
directory compared to other targets. We now model this in both the C++
built runtimes and the Bazel built runtimes.
Assisted-by: Gemini via Antigravity
Fixes#6895
Note this is just a short-term fix to avoid confusion, as the compile
structure needs to change on the whole.
Assisted-by: Google Antigravity with Gemini
This shifts the Bazel toolchain configuration of our installation to
build all of the Clang runtimes Carbon uses on-demand natively in Bazel.
We export the information about how to build into a generated Starlark
file, and emit BUILD files and Starlark logic into the installation to
orchestrate the build.
This requires some complex management of Bazel toolchains -- we need to
first set-up a "runtimes toolchain" that doesn't have runtimes of its
own, but can be used to _build_ runtimes. Then we build the runtimes
using that toolchain, and assemble them into the standard layout for a
Carbon runtimes tree. Finally we configure the _actual_ toolchain with
this built tree.
Currently, this is only setup for the installed toolchain, but I plan to
factor this runtimes build into one that can be used directly as well to
break up the monolithic runtimes build step into Bazel-integrated build
of the runtimes. This will also serve as the foundation for adding
bootstrapping support directly to our Bazel build.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
This moves the most complex of the logic fully into Starlark: both the
many different platform sources list, and the overriding of generic
files with architecture specific files.
This also fixes significant bugs in the AArch64 build where we were
skipping numerous files: all of the outlined atomics and `emupac.cpp`.
This PR forcibly disables `emupac.cpp` as fixing that will require a
more significant change.
This isn't as interesting as others, as it only involves compile
options.
It also adds a missing flag of `-fno-lto` as these objects can't be
LTO-ed.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
The goal here is to be able to construct a build of the runtimes
directly in Bazel, or by emitting `BUILD` files, or by emitting into C++
code and using that on-demand. For that, we want a single source of
truth, and that source in Starlark.
This should also make the information more generally useful, and so I'm
moving as much as I can into the LLVM Bazel build. Apologies as that
makes the diffs extra annoying.
I do plan on upstreaming the Bazel parts of this, but would like to get
everything working in Carbon and stabilized first.
While here, I've also made a change suggested for the future in the
initial review by lifting the C++ template out of a string literal in
the `.bzl` file, and into an actual separate C++ file.
This only moves libc++, libc++abi, and libunwind. I want to get those
three working end-to-end before I work on the builtins or `crtbegin` and
`crtend`, as those have a bunch of additional complexity.
This also only uses the info in the C++ on-demand build. It seemed like
a reasonable increment to start code review, and my plan is to work on
other build strategies in a follow-up PR. If that doesn't work, let me
know and I'll come back once I have at least a second use of the info
here.
This let's you point Bazel at an installed toolchain or download one of
our release archives. When you do, it will configure itself as a C++
Bazel toolchain. This toolchain works reasonably well, but doesn't cache
the C++ runtimes, and so linking is inefficient. The next step will be
to pivot the runtimes from the implicitly on-demand (which can't cache
when using a sandboxed build system like Bazel) to _explicit_ on-demand
runtimes directly with Bazel support.
I've included an example Bazel project that uses this and provides a
bunch of documentation and an example script that should let folks try
this out easily.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
Co-authored-by: David Blaikie <dblaikie@gmail.com>
This requires re-working our config features to be usable in
feature-level `requires` clauses in addition to `with_feature_set` by
always including all of the features, but controlling whether the
features are enabled or disabled based on the target.
This is a little more verbose in the config features, but lets us use
them more widely and is a bit more principled.
This lets us use a single undconditional feature for linking with flag
sets that are enabled based on the underlying OS. While here, tidy up
the feature names a bit.
The diff here may look really bad without aggressive whitespace
ignoring, but none of the contents of the two flag sets changed --
they've just be indented more and placed into a single list.
Now the CPU flags feature can be unconditionally added as part of the
optimization features and another of the conditions in the main
configuration goes away.
The failure to pass these to links was probably harmless, but it's
better to include it there as well.
This removes another chunk of platform-specific feature construction and
simplifies the code further.
Also removes a now-stale comment about adding more platform-specific
features.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
This PR merges the OS-specific Clang flags into the main Clang flags
features using feature-based constraints instead of separate features
conditionally added. Similarly for libc++. This also move flags to more
correctly live in the Clang flag set vs. the libc++ flag set as some of
these flags were specific to using libc++.
To make this change, the libc++ feature needs to be computed rather than
being fixed, as we need to add search paths based on the installed
location of LLVM and Clang.
All of this only works when the OS-config flags work. The earlier PR
adding these had a bug -- _none_ of the OS features would ever be
enabled. This didn't result in a problem as the initial use was only to
_disable_ flags on the wrong OS. Now that we're enabling flags, we have
to get it right by marking all of these as `enabled`.
This leaves behind project-specific features such as the system header
management of our dependencies and the fancy cache management string.
No expected changes here, but yet another slightly different order of
flags.
This introduces the first pieces of a cleaner way to configure toolchain
components on target dimensions: dedicated features for those target
dimensions.
With that, we extract a `libcxx_feature` that can always be present but
disables its flags on unsupported targets.
With `-stdlib` in its own feature, move `-std=c++20` to not require
a variable but directly live in the flags.
This should enable us to extract the largest remaining feature into its
own file cleanly by removing dynamic configuration of it, along with
libcxx.
Further refactoring of target-specific logic will follow in its
footsteps.
This brings some fixes:
- The handling of `zlib` and `zstd` are much cleaner
- Three of our patches are no longer needed
This also includes the fixes from #6562
It also moves us from `zlib` to `zlib-ng` which is a much better basis
for what we want, and likely makes our toolchain faster when generating
debug info at least.
It fixes another API change in terms of which headers provide the
`createInvocation` we use.
Lastly, it cleans up the deps test to correctly recognize the wrappers
for `zlib-ng` and `zstd`, as well as improving the documentation for why
we allow dependencies on them.
This moves the simplest parts of the toolchain config into separate
files. These parts are either unparameterized or trivially parameterized
and so easily extracted from the main file.
I tried to minimize the interesting edits here, but wasn't _completely_
successful I'm afraid. I'll try to describe them.
First, all of the interesting content of the new files is copied and
re-indented, no interesting edits were done.
The main file sees some more significant edits in order to realize this
refactoring:
- Extract the feature array building to a helper method.
- Collapse some extraneous features as there was no where to extract
them.
- Restructure how the array itself is built to support building it using
array fragments from the various files.
The only interesting semantic change I'm aware of here is that this
somewhat changes the order of command line flags in compiles and links.
The previous order was "fine", but not especially logical. I've tried to
more logically have features that should "override" or are "more
specific" come later here. However, that results in a slightly different
ordering. None of the current features had any flags that overlap, so
this should have no behavior change other than the changed flag order.
This is only the first step, however. There remain complex features in
the main configuration that I want to move out. However, to make those
moves simple requires some significant changes to how these remaining
features work and so I wanted to break them out. I've tried to leave
TODOs that can help as breadcrumbs on the parts of this refactoring that
aren't yet complete.
The comments also are mostly what we already had. I'm happy to try and
add some, but not sure how much I can cover as there is a _lot_ of code
here that I'm just moving around. Please let me know if there are
particularly places that would benefit from comments.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
These checks include a full check that a red-black tree satisfies its
invariants on every erase. This leads to
`llvm::DWARFDebugAranges::construct` becoming quadratic in the number of
debug symbols in the binary, which means that in `-c dbg`, symbolization
of backtraces is astronomically slow, and in practice never completes.
(I left it for over 12 hours and it did not finish.)
Reduce the libc++ hardening mode from *debug* to *extensive* to turn off
the checks that have unbounded performance impact.
Nicholas Bishop
David Blaikie
Lucile Rose Nihlen
Chandler Carruth
Richard Smith
Jon Ross-Perkins
Justin Horvitz
