Use `Label` to mark labels that are local to this module. Remove
workspace root when forming manifest. Add explicit import for name that
is not available implicitly in an imported module.
Assisted-by: Gemini via Antigravity
Without this, basic `bazel test //...` style wildcards would build a
bunch of extra configurations because of gaps excluding things. With
this, the action count of a normal build should be much more reasonable.
The switch from `target_compatible_with` to tagging is based on looking
at what ends up being most idiomatic and easiest -- trying to articulate
the complex and convoluted compatible with restrictions that would avoid
extraneous build configurations was really painful and this seems much
simpler and easier to deploy in a systematic way.
While here, also change the name of a rule that confused me to no end
while debugging this -- the rule that installs a `.bzl` file that
happens to be spelled `carbon_runtimes` is very different from all of
the other "installed carbon runtimes" kind of things in the tree. Adding
the file extension helps make that (much) more obvious.
Note that this is essentially a re-do of #7088 but now without any
dependencies that can mess up the merge.
Assisted-by: Antigravity with Gemini
This shouldn't change any functionality, but simplifies (significantly)
the logic in the installed toolchain, and also provides a better
conceptual balance between these.
I've tried to minimize the changes beyond a pure refactoring, but it was
a bit tricky to get everything working so some things have been mixed
in...
Assisted-by: Antigravity with Gemini
This reverts commit 4babfdbf22.
This was a stacked PR that was merged by accident, losing the commit and
description of the base change. Reverting and will re-land
independently.
Without this, basic `bazel test //...` style wildcards would build
a bunch of extra configurations because of gaps excluding things. With
this, the action count of a normal build should be much more reasonable.
The switch from `target_compatible_with` to tagging is based on looking
at what ends up being most idiomatic and easiest -- trying to articulate
the complex and convoluted compatible with restrictions that would avoid
extraneous build configurations was really painful and this seems much
simpler and easier to deploy in a systematic way.
While here, also change the name of a rule that confused me to no end
while debugging this -- the rule that installs a `.bzl` file that
happens to be spelled `carbon_runtimes` is very different from all of
the other "installed carbon runtimes" kind of things in the tree. Adding
the file extension helps make that (much) more obvious.
Assisted-by: Antigravity with Gemini
Rather than defining our own action groups, work to re-use the
`rules_cc` ones, as they are (much) more comprehensive. Also, completely
eliminate the `codegen` action group as it was not well used. For
example, `-march` flags and `-O` flags change the preprocessor macros
defined. There isn't a really great "codegen" heuristic, so just pass
those flags to all compiles which is simpler anyways.
I'm tempted to do the same with preprocessor actions, but maybe it makes
sense to have that one stay separate.
Assisted-by: Antigravity with Gemini
This takes the bootstrap support that was added and makes it available
under convenient user-facing flags for while we're doing development.
For example, to build a bootstrap compiler and use it to build and run
the tests under `//common/...` you can now use:
```
bazel test --//:bootstrap_stage=1 --//:bootstrap_exec_config=true //common/...
```
This will use the stage1 bootstrap compiler, and it will build that
compiler in the exec config (so it is optimized and the above even works
when cross-building with Bazel).
Assisted-by: Antigravity with Gemini
Noticed this when testing the Carbon toolchain with a more complex
environment, don't have any way to observe this at the moment in Bazel
though.
Assisted-by: Antigravity with Gemini
This worked correctly in the system Clang toolchain, but was not
configured correctly in the Carbon toolchains. The test is designed to
let us cover all of these.
Assisted-by: Antigravity with Gemini
The runtimes and bootstrap Bazel logic was previously built around
defining custom Bazel platforms constrained with `constraint_settings`.
The use of platforms added significant complexity, including the need to
"save" and "restore" the original platform, and other complexity
stemming from changing the platform as a whole.
This PR switches to use the simpler tool of build settings, and
`target_settings` on the toolchain rather than platform compatibility.
This remove the entire need to save and restore the platform, and also
generally simplifies things.
This PR also fixes some bugs in the bootstrap that were hidden by the
use of platforms, such as the need to carefully manage the different
inputs to the runtimes build so that generated inputs pick up the
correct exec configuration -- the exec transition happened to do this
"automatically", but it seems better to handle explicitly. And it cleans
up an extraneous copy of `carbon_runtimes.bzl` that snuck in somehow.
Assisted-by: Antigravity with Gemini
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
Also increases the default optimization to `-Og` which is likely to give
faster turn-around time which is what we want to optimize for here. This
should also _substantially_ shrink binary sizes, etc.
ASan is still available via `--config=asan`, and is added to the CI
infrastructure. However, my current thought is to only run it after push
rather than in PRs and in the merge queue.
This also switches to a more Bazel-based install layout, skipping the
FHS-based synthetic layout. The FHS-based layout is still reconstructed
explicitly when building an installable tar-ball.
The biggest change is to configure the just-built install as a Bazel
toolchain, including allowing it to build its own runtime libraries as
native Bazel libraries. This removes the need for a monolithic runtimes
build, all of that code logic is removed.
This should also pave the way to using the just-built toolchain for
doing a full 3-stage bootstrap. Building the 2nd stage is included here
as it was a particularly effective way to test that the Bazel
integration was fully working. Adding a 3rd-stage check for stability is
future work, but should be pretty easy.
There is a down-side: this uses the busybox to do the runtimes
compilation, which means they will be re-built after ~any change to
Carbon. However, the integration with Bazel should largely pay for this,
and we can continue to factor the tests away from depending on built
runtimes in most cases.
Now that we're building and testing the runtimes more directly, this
surfaced a problem with the layout of runtimes on macOS that is fixed
here. All of the Darwin OSes use a custom layout for their resource
directory compared to other targets. We now model this in both the C++
built runtimes and the Bazel built runtimes.
Assisted-by: Gemini via Antigravity
This let's you point Bazel at an installed toolchain or download one of
our release archives. When you do, it will configure itself as a C++
Bazel toolchain. This toolchain works reasonably well, but doesn't cache
the C++ runtimes, and so linking is inefficient. The next step will be
to pivot the runtimes from the implicitly on-demand (which can't cache
when using a sandboxed build system like Bazel) to _explicit_ on-demand
runtimes directly with Bazel support.
I've included an example Bazel project that uses this and provides a
bunch of documentation and an example script that should let folks try
this out easily.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
Co-authored-by: David Blaikie <dblaikie@gmail.com>
This requires re-working our config features to be usable in
feature-level `requires` clauses in addition to `with_feature_set` by
always including all of the features, but controlling whether the
features are enabled or disabled based on the target.
This is a little more verbose in the config features, but lets us use
them more widely and is a bit more principled.
This lets us use a single undconditional feature for linking with flag
sets that are enabled based on the underlying OS. While here, tidy up
the feature names a bit.
The diff here may look really bad without aggressive whitespace
ignoring, but none of the contents of the two flag sets changed --
they've just be indented more and placed into a single list.
Now the CPU flags feature can be unconditionally added as part of the
optimization features and another of the conditions in the main
configuration goes away.
The failure to pass these to links was probably harmless, but it's
better to include it there as well.
This removes another chunk of platform-specific feature construction and
simplifies the code further.
Also removes a now-stale comment about adding more platform-specific
features.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
This PR merges the OS-specific Clang flags into the main Clang flags
features using feature-based constraints instead of separate features
conditionally added. Similarly for libc++. This also move flags to more
correctly live in the Clang flag set vs. the libc++ flag set as some of
these flags were specific to using libc++.
To make this change, the libc++ feature needs to be computed rather than
being fixed, as we need to add search paths based on the installed
location of LLVM and Clang.
All of this only works when the OS-config flags work. The earlier PR
adding these had a bug -- _none_ of the OS features would ever be
enabled. This didn't result in a problem as the initial use was only to
_disable_ flags on the wrong OS. Now that we're enabling flags, we have
to get it right by marking all of these as `enabled`.
This leaves behind project-specific features such as the system header
management of our dependencies and the fancy cache management string.
No expected changes here, but yet another slightly different order of
flags.
This introduces the first pieces of a cleaner way to configure toolchain
components on target dimensions: dedicated features for those target
dimensions.
With that, we extract a `libcxx_feature` that can always be present but
disables its flags on unsupported targets.
With `-stdlib` in its own feature, move `-std=c++20` to not require
a variable but directly live in the flags.
This should enable us to extract the largest remaining feature into its
own file cleanly by removing dynamic configuration of it, along with
libcxx.
Further refactoring of target-specific logic will follow in its
footsteps.
This brings some fixes:
- The handling of `zlib` and `zstd` are much cleaner
- Three of our patches are no longer needed
This also includes the fixes from #6562
It also moves us from `zlib` to `zlib-ng` which is a much better basis
for what we want, and likely makes our toolchain faster when generating
debug info at least.
It fixes another API change in terms of which headers provide the
`createInvocation` we use.
Lastly, it cleans up the deps test to correctly recognize the wrappers
for `zlib-ng` and `zstd`, as well as improving the documentation for why
we allow dependencies on them.
This moves the simplest parts of the toolchain config into separate
files. These parts are either unparameterized or trivially parameterized
and so easily extracted from the main file.
I tried to minimize the interesting edits here, but wasn't _completely_
successful I'm afraid. I'll try to describe them.
First, all of the interesting content of the new files is copied and
re-indented, no interesting edits were done.
The main file sees some more significant edits in order to realize this
refactoring:
- Extract the feature array building to a helper method.
- Collapse some extraneous features as there was no where to extract
them.
- Restructure how the array itself is built to support building it using
array fragments from the various files.
The only interesting semantic change I'm aware of here is that this
somewhat changes the order of command line flags in compiles and links.
The previous order was "fine", but not especially logical. I've tried to
more logically have features that should "override" or are "more
specific" come later here. However, that results in a slightly different
ordering. None of the current features had any flags that overlap, so
this should have no behavior change other than the changed flag order.
This is only the first step, however. There remain complex features in
the main configuration that I want to move out. However, to make those
moves simple requires some significant changes to how these remaining
features work and so I wanted to break them out. I've tried to leave
TODOs that can help as breadcrumbs on the parts of this refactoring that
aren't yet complete.
The comments also are mostly what we already had. I'm happy to try and
add some, but not sure how much I can cover as there is a _lot_ of code
here that I'm just moving around. Please let me know if there are
particularly places that would benefit from comments.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
These checks include a full check that a red-black tree satisfies its
invariants on every erase. This leads to
`llvm::DWARFDebugAranges::construct` becoming quadratic in the number of
debug symbols in the binary, which means that in `-c dbg`, symbolization
of backtraces is astronomically slow, and in practice never completes.
(I left it for over 12 hours and it did not finish.)
Reduce the libc++ hardening mode from *debug* to *extensive* to turn off
the checks that have unbounded performance impact.
Also consolidate on using `//bazel/cc_rules:defs.bzl` where appropriate.
Also update a couple of Bazel modules deps of `@rules_cc` to the latest
versions.
Sadly, the formatter for starlark doesn't fully canonicalize the
formatting -- new lines and trailing `,`s can influence this formatting.
I've tried to pick a canonical format for these:
- Collapse as many balanced delimited sequences into a single line
without exceeding 80-columns.
- Collapse as many single comma-separated elements in a delimited region
into single lines with multiple opening constructs and single lines with
multiple closing constructs, reducing indentation and lines that consist
of only an opening delimited construct.
Generally, my goal with these heuristics was to minimize the number of
lines and indentation without creating irregularities, formatting
incompatible with `buildifier`, or egregiously long lines.
I've also tried to lexicographically sort named parameters where there
isn't any important ordering and currently there was a mixture just so
that we have a canonical ordering.
I've removed some redundant parentheses around arrays.
And lastly, I've reformatted some quite long lines to follow a pattern
that fits easily in 80-columns.
This shouldn't result in any behavior changes, just trying to tidy
things up here before making some more significant edits to refactor
this into composable logic instead of a single monolith.
If others have suggestions for different formatting, I'm happy to
change. I don't have any strong feelings about the formatting here, I
just wanted it to be consistent.
This is based on #5664 because it's fixing an issue which `DEBUG` would
catch. That's also why I'm switching to `DEBUG` from `EXTENSIVE`; I
think we should be okay with the performance cost in `file_test`, which
is probably our main concern.
Note digging into this also got me to notice that the flags weren't
actually enabled; this is fixing the define names.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
This avoids reallocating the backing buffer in ValueStore so that
references into the ValueStore are never invalidated when adding new
values. This works especially well since we never delete values from a
ValueStore.
The strategy used is to allocate chunks of a fixed size, and inserting
into each chunk until it is full before allocating the next. The
ValueStore starts with an initial allocated chunk in all cases, so that
there is only a single indirection for adding and accessing values from
this chunk. After it's full, additional chunks are allocated in a
vector, so two indirections are required to add or access values in
these chunks.
This obviates the need for
https://github.com/carbon-language/carbon-lang/pull/5529 as we no longer
need to worry about holding pointers into a ValueStore.
We introduce a Flatten operation for ranges. It flattens a "range over
ranges over Ts" down to a "range over Ts". This allows us to make an
range over the values in the ValueStore from a range over the chunks in
the ValueStore. See
https://doc.rust-lang.org/stable/std/iter/trait.Iterator.html#method.flatten
for inspiration for this name choice. Flatten is used in one other case
where we were writing two levels of for loops to do the same thing.
The `array_ref()` accessor is changed to `values()` and its now a range
(typed as a `ValueStoreRange`) over all values as references (like
ArrayRef was, but without random access).
As pointers to a ValueStore can no longer be invalidated, we remove the
ASAN poisoning feature and support from ValueStore.
This may cause a regression in our compile benchmark of up to 5%, though
that is close to or within the noise of the benchmark. We can look at
ways to optimize things further in the future. Perhaps by tuning the
chunk size further, or by making later chunks larger than earlier
chunks, or other strategies.
This reverts commit 1889ee3904.
We have identified that this is causing ODR violations, because the
`fuzzer` feature is being added `cc_fuzz_test` targets, and thus any
includes they make, but not to the rest of the build. Any include that
is seen from both places has ODR violations if it branches on
FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION.
We need to apply fuzzer globally when building fuzz targets somehow, or
not set different defines in fuzzer.
The `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` flag is a standard flag
proposed by LibFuzzer that is meant to inform compiled code that it is
being built for fuzzing, as described here:
https://llvm.org/docs/LibFuzzer.html#fuzzer-friendly-build-mode
We add the flag to our `fuzzing` feature/config, and enable DCHECKs when
under fuzzing so that we can catch bugs that currently are caught on the
other side of DCHECK, even if they don't cause ASAN to trap a read/write
beyond the capacity of a value store.
With this enabled, entities that live in value stores are poisoned
whenever any action is taken that might invalidate pointers and
references to those options -- in particular, adding another item to
that value store, or attempting to load any entity from an import IR.
Subsequent uses of those pointers or references then trigger an ASan
failure.
This detects latent bugs where the pointer or reference to the entity
would become stale if we got unlucky about when the value store
reallocates, even in cases where the reallocation didn't actually
happen.
This is not enabled by default: it finds a lot of latent bugs, so our
tests don't pass with this option. This PR also includes fixes for a few
of those bugs.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Co-authored-by: Carbon Infra Bot <carbon-external-infra@google.com>
#5445 updates to bazel 8.2.1, this does more updates (including to
buildifier, which does autofixes like the `sh_test` loads in the other
PR).
Note I'm using the latest available clang-format wheel. That's not
really something I expect people to have installed, but should mostly be
consistent. I'm specifically skipping clang-format 18 because it had
some broad regressions, and 19 got really confused by a `requires` on a
trailing return. Using the latest seemed probably okay since most people
won't see the difference. Do note that trailing returns in macros,
https://github.com/llvm/llvm-project/issues/47664, seems to be cropping
up again as an issue.
Ran into this trying to debug #5428
Before:
```
lex.cpp:793:21: runtime error: null pointer passed as argument 1, which is declared to never be null
string.h:90:51: note: nonnull attribute specified here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior lex.cpp:793:21
```
After:
```
lex.cpp:793:21: runtime error: null pointer passed as argument 1, which is declared to never be null
string.h:90:51: note: nonnull attribute specified here
#0 0x5572db22a680 in Carbon::Lex::Lexer::MakeLines(llvm::StringRef) /proc/self/cwd/toolchain/lex/lex.cpp:793:14
#1 0x5572db227ee4 in Lex /proc/self/cwd/toolchain/lex/lex.cpp:738:3
(etc)
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior lex.cpp:793:21
```
Interestingly, even though this is labelled as UB, it's using the
ASAN_SYMBOLIZER_PATH (specifically not LLVM_SYMBOLIZER_PATH). But
canonically UBSAN_SYMBOLIZER_PATH may also be used per
https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/ubsan/ubsan_flags.cpp#L53,
so I'm adding it to the list out of an excess of caution.
Also doing some small related cleanup:
- Removing `ASAN_SYMBOLIZER_PATH` from `--test_env` because it should
now be getting overridden by these settings (also was a little
inconsistent in that `LLVM_SYMBOLIZER_PATH` was not included).
- Improving the environment construction and documentation.
Uses bazel 8 flags since there's the update in #4729. bazelisk was used
to generate the list of flags (see bazelrc comment). The overall
approach is trying to do our best to follow
https://bazel.build/release/backward-compatibility, in particular since
`--incompatible_strict_action_env` had come up (essentially just
adopting as much as we can now).
The default visibility changes in `carbon_rules/BUILD` and
`cc_toolchains/BUILD` files are for
`--incompatible_config_setting_private_default_visibility`. That's
enough for fastbuild, but we use tcmalloc in opt, and it has an issue.
In `clang_toolchain.BUILD` it's for
`--incompatible_check_visibility_for_toolchains`, even though
`rules_shell` then breaks on it. `manifest/defs.bzl` changes are for
`--incompatible_disable_target_default_provider_fields` which
`rules_pkg` breaks on. Even though these flags are off, I'm keeping the
changes since we should eventually enable the flags.
I'm still looking at the tree sitter rules due to the WORKSPACE issue,
but I think that needs more substantial work.
As part of migrating to the latest bazel configurations in #4729, I'm
running into proto toolchain issues. For example:
"Error: <target @@protobuf+//:cc_toolchain> (rule
'proto_lang_toolchain') doesn't contain declared provider
'ProtoLangToolchainInfo'"
Although we may eventually want more use of proto, right now the only
use is for the explorer fuzzer. The explorer codebase is essentially
frozen, so continuing to run it isn't gaining us much (in fact, we've
already disabled autofuzzing for it).
So, rather than trying to fix the proto setup, this change:
1. Deletes `explorer/fuzzing`
2. Removes proto portions of `testing/fuzzing`, which were only in-use
by the explorer
3. Removes some ancillary proto support, which would otherwise break
from the bazel changes and would be difficult to validate as "still
working"
This change is partly isolated in order to make it easier to revive bits
of (3).
Also make `-gsimple-template-names` lldb-only due to it tripping up gdb
in some cases (I came across it breaking SmallVector pretty printing
where gdb wouldn't associate a simplified type named declaration with a
simplified type named definition in another translation unit - seems gdb
can associate a type decl/def when it sees both (if you step into both
translation units or otherwise trigger gdb loading/parsing them) but it
doesn't seem able to /search/ for the type definition). Filed
https://sourceware.org/bugzilla/show_bug.cgi?id=32421 for this.
A couple of other things I'd like to do, but don't know how:
* It might be nice to allow opting into or out of gdb_index (with the
default being 'on' for gdb_flags, but you could opt out). But doesn't
seem super important.
* We should turn off fission by default, it seems - bazel has trouble
making the .dwo files available at the same path as is in the binary
especially on partial rebuilds. (not sure if we can do that, I guess
we can make fission a no-op/doesn't add any flags, even if we can't
change the fission default in bazel itself)
* can we have gdb_flags imply/disable lldb_flags? (so you can use
--features=gdb_flags without always having to add
--features=-lldb_flags)
In llvm 17 the _LIBCPP_ENABLE_ASSERTIONS flag was split into two:
- _LIBCPP_ENABLE_HARDENED_MODE for fast checks
- _LIBCPP_ENABLE_DEBUG_MODE for expensive checks
We kept HARDENED_MODE enabled in debug, but we can also turn it on for
opt builds.
In llvm 18, the _LIBCPP_ENABLE_HARDENED_MODE was further split into 4
settings, NONE, FAST, EXTENSIVE, DEBUG. As seen in the recent blog post
https://security.googleblog.com/2024/11/retrofitting-spatial-safety-to-hundreds.html
the FAST hardening mode is indeed very fast and has minimal impact,
while helping to catch a lot of bugs.
So we enable the FAST checks in opt builds, and EXTENSIVE checks in
debug builds.
The EXTENSIVE checks are the same that Chrome enables in every build
configuration, so we could consider enabling it in opt builds as well:
https://source.chromium.org/chromium/chromium/src/+/main:build/config/compiler/BUILD.gn;l=1127;drc=a8260dee097dde71ca4464c0c8d897a80c353db2
Came up due to [libc++ install
issues](https://discord.com/channels/655572317891461132/655577725347561492/1302023663155023905)
We've discussed clang version verification, and adding that as long as
I'm in here. The more significant bit is the libc++ check, which if it's
not installed should fail like:
```
(tons of output)
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
/usr/lib/llvm-16/lib/clang/16/include
/usr/local/include
/usr/include/x86_64-linux-gnu
/usr/include
End of search list.
/usr/local/google/home/jperkins/.cache/bazel/_bazel_jperkins/85deb7d9d96f7e0e80b42618a55969d7/external/_main~clang_toolchain_extension~bazel_cc_toolchain/_temp:6:2: error: "No libc++ install found!"
#error "No libc++ install found!"
^
1 error generated.
ERROR: Analysis of target '//toolchain:toolchain' failed; build aborted: Analysis failed
INFO: Elapsed time: 0.265s, Critical Path: 0.08s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully
```
pre-commit runs bazel, and GitHub runners have an old clang by default
(caught by the new check), so I'm installing here for a consistent
version.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Richard Smith
Chandler Carruth
Jon Ross-PerkinsandDana Jansens
josh11bandJosh L
