This removes the need for a patch and improves on the quality of the
rules significantly. A follow-up PR will use this to apply a number of
fixes to how we build the runtimes.
This is the first step to having Clang's runtime libraries fully
available for the Carbon toolchain. This PR focuses on the lowest level
runtimes, the CRT files and the builtins library.
The goal is to intercept Clang runs where it needs these
target-dependent pieces to be available, and build them on demand using
our Clang-running infrastructure. This avoids most of the subprocess
overhead, but there is still some due to missing features in Clang.
This requires exporting the sources for these runtimes from the Bazel
build, and installing them in our target-independent resource directory.
We then build a simplified "build" of these sources within the
`ClangRunner` itself to produce the specific artifacts and layout
expected by Clang.
It also required fixing our use of Clang on macOS to have a default
system root in order to successfully compile or link.
It also required cleaning up how the `ClangRunner` used target
information more generally -- instead of taking the target as
a constructor parameter, it manages its target internally and relies on
the Clang target-specifying command line flags.
I looked at whether we could split this into another layer separate from
the `ClangRunner`, but that proved frustratingly difficult to manage.
While we support building these on-demand as part of a detected link,
that doesn't seem feasible as we don't have the necessary separation
between compilation runs of Clang and link runs of Clang. However,
I have tried to factor the internals to provide as clear of separation
as I could across these.
I have also created a stand-alone subcommand to directly build the
runtimes which allows for easy testing. It also supports building them
into a specific directory, and that directory can in turn be passed to
a Clang invocation. This is designed to work both at the API level with
`ClangRunner` and at the subcommand level.
Currently, the only part of the commandline that is detected and
forwarded to the runtimes build is the target. Eventually, the plan is
to expand this so that we can build a maximally tailored set of runtimes
for a given compilation.
The other big TODO here is to actually implement caching storage of
these runtimes so they aren't built on every execution. Right now, this
uses a somewhat hack-y build of a temporary directory, but this isn't
expected to be suitable long-term. Building these runtimes on *every*
link makes those commands take approximately 15 seconds with an ASan
build like our default development build, and just over 2 seconds in an
optimized build. Because of this, I've kept all of this disabled by
default for now. The goal is that once caching and some other
improvements land, we can enable this by default.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Stop using the clang tooling library to build an ASTUnit; that library
is set up to process clang frontend arguments, assuming that something
has already built frontend arguments from the compiler arguments. It is
also too encapsulated and doesn't let us inspect and modify the compiler
invocation before it's executed.
Instead, build the AST unit directly in two phases:
* FIrst, take a list of clang driver arguments and convert them into a
list of compiler arguments, using `clang::createInvocation`. Internally,
this uses the clang driver to build a frontend invocation, including
building system-specific include paths as needed.
* Then, directly build an ASTUnit from this compiler invocation.
I've factored this so that we can split out the `createInvocation` step,
with the intention that we may want to move it out of check and into the
carbon driver with the rest of the driver-level argument handling, and
we may want to customize some of the clang options before we invoke the
clang frontend with that set of options.
In order to make the invocation reusable, it no longer depends on the
name of the carbon file importing the C++ code. In place of synthesizing
a header file name as `<foo.carbon>.generated.cpp_imports.h`, we now
insert line marker directives into the generated header so that errors
in that header cause Clang to point a diagnostic back at the Carbon
source file itself. This results in a minor improvement in the
diagnostic output: we no longer refer to a nonexistent generated file.
But the snippet still contains text that doesn't match the source code,
so it remains imperfect.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
This is based on #5664 because it's fixing an issue which `DEBUG` would
catch. That's also why I'm switching to `DEBUG` from `EXTENSIVE`; I
think we should be okay with the performance cost in `file_test`, which
is probably our main concern.
Note digging into this also got me to notice that the flags weren't
actually enabled; this is fixing the define names.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
This avoids reallocating the backing buffer in ValueStore so that
references into the ValueStore are never invalidated when adding new
values. This works especially well since we never delete values from a
ValueStore.
The strategy used is to allocate chunks of a fixed size, and inserting
into each chunk until it is full before allocating the next. The
ValueStore starts with an initial allocated chunk in all cases, so that
there is only a single indirection for adding and accessing values from
this chunk. After it's full, additional chunks are allocated in a
vector, so two indirections are required to add or access values in
these chunks.
This obviates the need for
https://github.com/carbon-language/carbon-lang/pull/5529 as we no longer
need to worry about holding pointers into a ValueStore.
We introduce a Flatten operation for ranges. It flattens a "range over
ranges over Ts" down to a "range over Ts". This allows us to make an
range over the values in the ValueStore from a range over the chunks in
the ValueStore. See
https://doc.rust-lang.org/stable/std/iter/trait.Iterator.html#method.flatten
for inspiration for this name choice. Flatten is used in one other case
where we were writing two levels of for loops to do the same thing.
The `array_ref()` accessor is changed to `values()` and its now a range
(typed as a `ValueStoreRange`) over all values as references (like
ArrayRef was, but without random access).
As pointers to a ValueStore can no longer be invalidated, we remove the
ASAN poisoning feature and support from ValueStore.
This may cause a regression in our compile benchmark of up to 5%, though
that is close to or within the noise of the benchmark. We can look at
ways to optimize things further in the future. Perhaps by tuning the
chunk size further, or by making later chunks larger than earlier
chunks, or other strategies.
This reverts commit 1889ee3904.
We have identified that this is causing ODR violations, because the
`fuzzer` feature is being added `cc_fuzz_test` targets, and thus any
includes they make, but not to the rest of the build. Any include that
is seen from both places has ODR violations if it branches on
FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION.
We need to apply fuzzer globally when building fuzz targets somehow, or
not set different defines in fuzzer.
The `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` flag is a standard flag
proposed by LibFuzzer that is meant to inform compiled code that it is
being built for fuzzing, as described here:
https://llvm.org/docs/LibFuzzer.html#fuzzer-friendly-build-mode
We add the flag to our `fuzzing` feature/config, and enable DCHECKs when
under fuzzing so that we can catch bugs that currently are caught on the
other side of DCHECK, even if they don't cause ASAN to trap a read/write
beyond the capacity of a value store.
With this enabled, entities that live in value stores are poisoned
whenever any action is taken that might invalidate pointers and
references to those options -- in particular, adding another item to
that value store, or attempting to load any entity from an import IR.
Subsequent uses of those pointers or references then trigger an ASan
failure.
This detects latent bugs where the pointer or reference to the entity
would become stale if we got unlucky about when the value store
reallocates, even in cases where the reallocation didn't actually
happen.
This is not enabled by default: it finds a lot of latent bugs, so our
tests don't pass with this option. This PR also includes fixes for a few
of those bugs.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Co-authored-by: Carbon Infra Bot <carbon-external-infra@google.com>
#5445 updates to bazel 8.2.1, this does more updates (including to
buildifier, which does autofixes like the `sh_test` loads in the other
PR).
Note I'm using the latest available clang-format wheel. That's not
really something I expect people to have installed, but should mostly be
consistent. I'm specifically skipping clang-format 18 because it had
some broad regressions, and 19 got really confused by a `requires` on a
trailing return. Using the latest seemed probably okay since most people
won't see the difference. Do note that trailing returns in macros,
https://github.com/llvm/llvm-project/issues/47664, seems to be cropping
up again as an issue.
- Updates incompatible flags.
- `rules_flex` is no longer used, so enable its flag.
- Fixes `sh_test` deps for
`--incompatible_disable_autoloads_in_main_repo`
- Broadens the exception for `rules_cc` and `bazel_tools` due to changes
to runfiles deps; trying to avoid minutiae that shouldn't affect the
decision.
Ran into this trying to debug #5428
Before:
```
lex.cpp:793:21: runtime error: null pointer passed as argument 1, which is declared to never be null
string.h:90:51: note: nonnull attribute specified here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior lex.cpp:793:21
```
After:
```
lex.cpp:793:21: runtime error: null pointer passed as argument 1, which is declared to never be null
string.h:90:51: note: nonnull attribute specified here
#0 0x5572db22a680 in Carbon::Lex::Lexer::MakeLines(llvm::StringRef) /proc/self/cwd/toolchain/lex/lex.cpp:793:14
#1 0x5572db227ee4 in Lex /proc/self/cwd/toolchain/lex/lex.cpp:738:3
(etc)
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior lex.cpp:793:21
```
Interestingly, even though this is labelled as UB, it's using the
ASAN_SYMBOLIZER_PATH (specifically not LLVM_SYMBOLIZER_PATH). But
canonically UBSAN_SYMBOLIZER_PATH may also be used per
https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/ubsan/ubsan_flags.cpp#L53,
so I'm adding it to the list out of an excess of caution.
Also doing some small related cleanup:
- Removing `ASAN_SYMBOLIZER_PATH` from `--test_env` because it should
now be getting overridden by these settings (also was a little
inconsistent in that `LLVM_SYMBOLIZER_PATH` was not included).
- Improving the environment construction and documentation.
- Add pkg rules to the search, specifically so that we can pull the full
release tar instead of the busybox binary.
- Shift the tree_sitter exclusion to the script, so that it's easier to
use the script.
- tcmalloc_if_linux_opt isn't really interesting (it's referenced by
.bazelrc).
- Misc tiny changes to the query, just to simplify it.
Rules executed by bazel don't necessarily have the right environment to
find the symbolizer, which was the intent of `cc_env` setting
`LLVM_SYMBOLIZER_PATH`. So far, this has kind of been a case-by-case
fix, but every so often I'm trying to debug a crash in a test that
doesn't provide it. Rather continuing down this route, instead add
drop-in wrappers for cc rules so that it's hard to forget.
Note `bazel/cc_rules` is intended to mirror `bazel/carbon_rules` and
`bazel/cc_toolchains`, rather than `@rules_cc`.
AFAICT there isn't a great way to add this as a default for the `bazel
run` environment. It's not typically going to be set on its own,
forwarding `$PATH` would be too broad, and the [action
`env_sets`](https://bazel.build/docs/cc-toolchain-config-reference#using-action-config)
I think are not quite what we need (I think those don't include output
execution, only compilation).
I'm doing this separately from removing the explorer/ and installers/
directories so that it's easier to review the side-effects.
Note for the main README, I didn't think it was worth keeping a mention
of "used to have a prototype interpreter, now archived" versus focusing
on the existing toolchain (for "Currently, we have fleshed out").
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Inconsistent execution environments make using a path-as-define
difficult, so switch to an embedded file.
Also fixes the lldb launch so that passing tests run cleanly, and adds
TEST_TARGET to gdb (but without testing there). I'm dropping `sourceMap`
because it's not handled quite correctly (also not great to be trying to
pass source mappings in two different ways), and `env` didn't seem to be
working as intended either; maybe specifying `initCommands` causes other
things to not be evaluated. But the straight `initCommands` looks like
it's working. I used lldb to validate execution of these changes.
```
Running initCommands:
(lldb) command script import external/+llvm_project+llvm-project/llvm/utils/lldbDataFormatters.py
(lldb) settings set target.source-map "." "/usr/local/google/home/jperkins/dev/carbon-lang"
(lldb) settings set target.source-map "/proc/self/cwd" "/usr/local/google/home/jperkins/dev/carbon-lang"
(lldb) env TEST_TARGET=//toolchain/testing:file_test
(lldb) env TEST_TMPDIR=/tmp
Running tests with 128 thread(s)
.
Done!
Note: Google Test filter = ToolchainFileTest.toolchain/check/testdata/const/collapse.carbon
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from ToolchainFileTest
[ RUN ] ToolchainFileTest.toolchain/check/testdata/const/collapse.carbon
[ OK ] ToolchainFileTest.toolchain/check/testdata/const/collapse.carbon (0 ms)
[----------] 1 test from ToolchainFileTest (0 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (1 ms total)
[ PASSED ] 1 test.
Process 3869310 exited with status = 0 (0x00000000)
```
- The actual reason I started this: minor lowering updates in the golden
LLVM IR
- Process.inc changed enough to need a patch context update.
- https://github.com/llvm/llvm-project/pull/123126 added `proto_library`
uses without a `load`, which is broken in bazel 8
- Just commenting these out because we don't use them. I'll follow up
separately about a possible fix, but continuing to use `WORKSPACE` is a
bigger issue LLVM probably should address.
- Note this update is also triggering removal of `migrate_cpp`, in #4887
Although migration tooling is intended for Carbon, this tooling was
primarily a proof-of-concept prototype. The last time it got significant
work was in 2022
(https://github.com/carbon-language/carbon-lang/commits/trunk/migrate_cpp);
since then, we've mainly been doing small cleanups to keep it building.
We're now hitting an issue in #4886 that `TypeNodes.inc` isn't exposed
as a `#include`.
We already had thoughts about rewriting this as a `RecursiveASTVisitor`,
per `rewriter.cpp`/`rewriter.h`. That may justify a significantly
different approach than had been set up in `cpp_refactoring`. But, it's
hard to tell.
Either way, my sense is that rather than incrementally trying to keep
this code building, we should revisit it when we're ready and have a
long-term strategy for how migration should work.
The WORKSPACE file is deprecated; support is already off by default, and
it'll be removed in the next major bazel release. Our main dependency is
tree-sitter, and I'm trying to address that here.
We're currently using https://github.com/elliottt/rules_tree_sitter, but
that hasn't been updated in a couple years, meaning it lacks
MODULE.bazel support. In the registry, there's
https://registry.bazel.build/modules/tree-sitter-bazel, but this is only
the *parser* libraries of tree-sitter, not the *generator*. I'm using it
for that much, at least.
For the *generator*, which transforms grammar.js to parser.c/h, I'm just
requiring a non-hermetic invocation (i.e., people who want to work on it
will need to install tree-sitter; see the README.md updates). I tried
running it manually, but parser.c is about 600 KB; pre-commit rejects
files that large and I don't think an exception makes sense to override
for this (it'd probably also grow substantially if the grammar were
updated to cover more syntax). In order to make the non-hermetic call
not break "bazel build //..." for most developers, I'm marking most
targets in the package as manual.
Note, I did look long and hard at using `aspect_rules_js`/`rules_nodejs`
to invoke npm. This took a lot of time, and I have a commit that's
mostly working, except I hit a point where it uses `declare_symlink`
which we disallow for compatibility reasons (commit "Lots of work for
figuring out rule_js uses declare_symlink" on the PR). As a consequence,
I think we can't use the primary supported ways to have hermetic npm
calls.
Also, `treesitter` -> `tree_sitter` because it's generally called
`tree-sitter`, two words. We even had a `treesitter/src/tree_sitter`
directory so it's a bit inconsistent.
As far as bugs here, the parser library breaks bazel queries, e.g. the
error:
```
ERROR: Evaluation of query "somepath(//..., @llvm-project//third-party/unittest:gtest)" failed: preloading transitive closure failed: no such package '@@[unknown repo 'platforms' requested from @@tree-sitter-bazel+]//': The repository '@@[unknown repo 'platforms' requested from @@tree-sitter-bazel+]' could not be resolved: No repository visible as '@platforms' from repository '@@tree-sitter-bazel+'
```
I'm just excluding tree_sitter from queries where I can to work around
the error.
Uses bazel 8 flags since there's the update in #4729. bazelisk was used
to generate the list of flags (see bazelrc comment). The overall
approach is trying to do our best to follow
https://bazel.build/release/backward-compatibility, in particular since
`--incompatible_strict_action_env` had come up (essentially just
adopting as much as we can now).
The default visibility changes in `carbon_rules/BUILD` and
`cc_toolchains/BUILD` files are for
`--incompatible_config_setting_private_default_visibility`. That's
enough for fastbuild, but we use tcmalloc in opt, and it has an issue.
In `clang_toolchain.BUILD` it's for
`--incompatible_check_visibility_for_toolchains`, even though
`rules_shell` then breaks on it. `manifest/defs.bzl` changes are for
`--incompatible_disable_target_default_provider_fields` which
`rules_pkg` breaks on. Even though these flags are off, I'm keeping the
changes since we should eventually enable the flags.
I'm still looking at the tree sitter rules due to the WORKSPACE issue,
but I think that needs more substantial work.
This updates to bazel 8.0.0, also updating bazel mod deps and tools to
make that function. The release is a couple weeks old, and we haven't
updated in a while, and it's a major release. Note it includes some
incompatible flag flips that this is trying to update with respect to.
I'll try generally enabling incompatible support separately.
The most visible bazel behavior change here will be the change from `~`
to `+` in repo path names. (If you're curious,
https://github.com/bazelbuild/bazel/issues/23127 indicates this fixes a
Windows performance issue)
Note that this is building on top of both the action env update in #4728
(which got me started down this path) and the proto removal in #4731
(which would add significant work to this update). Only the commit
starting at "Work towards bazel 8.0.0" is specifically part of this PR.
As part of migrating to the latest bazel configurations in #4729, I'm
running into proto toolchain issues. For example:
"Error: <target @@protobuf+//:cc_toolchain> (rule
'proto_lang_toolchain') doesn't contain declared provider
'ProtoLangToolchainInfo'"
Although we may eventually want more use of proto, right now the only
use is for the explorer fuzzer. The explorer codebase is essentially
frozen, so continuing to run it isn't gaining us much (in fact, we've
already disabled autofuzzing for it).
So, rather than trying to fix the proto setup, this change:
1. Deletes `explorer/fuzzing`
2. Removes proto portions of `testing/fuzzing`, which were only in-use
by the explorer
3. Removes some ancillary proto support, which would otherwise break
from the bazel changes and would be difficult to validate as "still
working"
This change is partly isolated in order to make it easier to revive bits
of (3).
Instead of treating `Core.Int` as the toolchain's builtin `IntType`,
model it as a class that adapts the builtin type. This aligns us better
with the intended language model, gives an associated library for
`impl`s involving `Core.Int` to live within, and opens the door adding
member functions to `Core.Int` if we decide that is desirable.
Remarkably it also seems to make the formatted SemIR a little smaller,
because a call to a generic class generates less IR than a call to a
function.
Also make `-gsimple-template-names` lldb-only due to it tripping up gdb
in some cases (I came across it breaking SmallVector pretty printing
where gdb wouldn't associate a simplified type named declaration with a
simplified type named definition in another translation unit - seems gdb
can associate a type decl/def when it sees both (if you step into both
translation units or otherwise trigger gdb loading/parsing them) but it
doesn't seem able to /search/ for the type definition). Filed
https://sourceware.org/bugzilla/show_bug.cgi?id=32421 for this.
A couple of other things I'd like to do, but don't know how:
* It might be nice to allow opting into or out of gdb_index (with the
default being 'on' for gdb_flags, but you could opt out). But doesn't
seem super important.
* We should turn off fission by default, it seems - bazel has trouble
making the .dwo files available at the same path as is in the binary
especially on partial rebuilds. (not sure if we can do that, I guess
we can make fission a no-op/doesn't add any flags, even if we can't
change the fission default in bazel itself)
* can we have gdb_flags imply/disable lldb_flags? (so you can use
--features=gdb_flags without always having to add
--features=-lldb_flags)
In llvm 17 the _LIBCPP_ENABLE_ASSERTIONS flag was split into two:
- _LIBCPP_ENABLE_HARDENED_MODE for fast checks
- _LIBCPP_ENABLE_DEBUG_MODE for expensive checks
We kept HARDENED_MODE enabled in debug, but we can also turn it on for
opt builds.
In llvm 18, the _LIBCPP_ENABLE_HARDENED_MODE was further split into 4
settings, NONE, FAST, EXTENSIVE, DEBUG. As seen in the recent blog post
https://security.googleblog.com/2024/11/retrofitting-spatial-safety-to-hundreds.html
the FAST hardening mode is indeed very fast and has minimal impact,
while helping to catch a lot of bugs.
So we enable the FAST checks in opt builds, and EXTENSIVE checks in
debug builds.
The EXTENSIVE checks are the same that Chrome enables in every build
configuration, so we could consider enabling it in opt builds as well:
https://source.chromium.org/chromium/chromium/src/+/main:build/config/compiler/BUILD.gn;l=1127;drc=a8260dee097dde71ca4464c0c8d897a80c353db2
Came up due to [libc++ install
issues](https://discord.com/channels/655572317891461132/655577725347561492/1302023663155023905)
We've discussed clang version verification, and adding that as long as
I'm in here. The more significant bit is the libc++ check, which if it's
not installed should fail like:
```
(tons of output)
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
/usr/lib/llvm-16/lib/clang/16/include
/usr/local/include
/usr/include/x86_64-linux-gnu
/usr/include
End of search list.
/usr/local/google/home/jperkins/.cache/bazel/_bazel_jperkins/85deb7d9d96f7e0e80b42618a55969d7/external/_main~clang_toolchain_extension~bazel_cc_toolchain/_temp:6:2: error: "No libc++ install found!"
#error "No libc++ install found!"
^
1 error generated.
ERROR: Analysis of target '//toolchain:toolchain' failed; build aborted: Analysis failed
INFO: Elapsed time: 0.265s, Critical Path: 0.08s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully
```
pre-commit runs bazel, and GitHub runners have an old clang by default
(caught by the new check), so I'm installing here for a consistent
version.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Removes the separate language server binary; I'm not sure we need to
provide it. Instead, `carbon language-server` is added as a subcommand.
Moves //language_server to //toolchain/language_server. Splits into a
trivial language_server.h, and a substantive server.h. I wasn't sure
about a better name, but wanted the split similar to check/check.h,
lex/lex.h, etc. At the same time, the class is probably going to be a
little big so not a good fit to through into just a cpp file.
This fixes some style issues with the language server class, but
generally I'm trying to not address things here in order to keep it
simpler.
For reference, we're going down the busyboxing route because Carbon
depends on Clang, and we want both to be available as binaries.
Busyboxing allows this while avoiding duplicating symbols between
multiple binaries.
I'm removing the `cc_binary` for `driver:carbon` because I want to avoid
a significant increase in binary outputs; `bazel run //toolchain` still
works great.
This still doesn't have great test coverage (but non-zero:
`//examples:sieve` still builds/runs, for example). The problem is that
we want to avoid subprocessing for performance, but this mainly deals
with subprocessing. I'm still thinking about good approaches for that,
since we'll probably want more significant testing for `clang`
interaction... the solution might involve busyboxing `file_test` too.
Note development on this ran into the argv issue being fixed in #4405
I was looking at the documentation again, and I realized that while this
works, I think the intention is that `runfiles` is set _instead_ of both
fields (I was a little confused by the error I was getting when setting
`runfiles` _with_ `default_runfiles`)
I believe I've verified this works in necessary situations.
Use the diagnostic kind printing in #4425 to catch when we have
diagnostics with no tests.
This merges a couple other use-cases of filegroup manifests into a
common rule.
Note I do add a few tests for things, and also some things are
_actually_ unit tested (just not in the file_test structure). But I
stopped when I realized that dealing with merge conflicts is going to be
a pain. I might end up reverting test changes (as part of merge conflict
resolution) and doing narrow test additions in a separate PR, after both
this and #4425 are merged.
Undoes most of #4347, because of [performance
complaints](https://discord.com/channels/655572317891461132/707150492370862090/1295527235133898772).
With a 30-ish frame stack trace and `-c dbg`, my installed
`llvm-symbolizer` still seems slow (~6s), but the hermetic
`llvm-symbolizer` adds ~4s (i.e., ~10s total). I don't think we can
easily force the hermetic version to build in opt configuration, so I'm
backing it out.
@bazel_tools//tools/cpp:malloc is equivalent and comes from
https://bazel.build/reference/be/c-cpp#cc_binary.malloc. This also
avoids some confusion in the documentation, since while system_malloc
_can_ be used with `malloc`, it has no effect when it's used with
`--custom_malloc`. But, rather than trying to adjust that, maybe we can
just use @bazel_tools//tools/cpp:malloc
This should substantially reduce the total build size of `-c dbg`
builds, and especially improve cache hits during incremental
development. I'm seeing over 50% reduction total on Linux in just the
raw size of a complete debug build. Even on macOS where we can't use
split DWARF there are substantial reductions.
Note that LLDB and GDB want slightly different flags to have the best
experience with split debug information, and so the build and
documentation have been updated to enable LLDB's flags by default but
provide clear instructions for switching to GDB's flags, and they are
structured so that this can be done persistently for an individual
developer.
Causes the default development (`fastbuild`) Bazel build and test to
both use ASan, minimal optimizations, and produce good backtraces with
source locations.
This also fixes all of the non-host configs that were deeply broken for
the latest releases of Bazel going back quite some time.
The intent of our our Bazel toolchain config was for a minimally
optimized, minimal debug info, and ASan + UBSan configuration to be the
`fastbuild`, or the default development build of the project. This
matches its inclusion of asserts, etc.
At some point quite some time ago, all of this stopped working. Bazel no
longer has a `nonhost` feature. This was disabled a long time ago,
briefly argued to be re-enabled, but has persistently been removed.
However, since then the host and non-host features have been separated
including the compilation mode, and so none of that is needed now.
Instead, we can use a much simpler and more principled approach to all
of the feature configuration now which this PR implements.
However, we added TCMalloc _after_ all of the ASan stuff became broken,
and so we never saw that it is fundamentally incompatible with ASan. So
this PR also reworks how TCMalloc is used to only apply to `-c opt`
builds on Linux, and it also explicitly disables it when using
`--config=asan`. I've not found a convenient way to tie the malloc
library choice to a toolchain feature in Bazel, so this relies on the
config being used rather than the feature in isolation.
Last but not least, with this change `fastbuild` creates substantially
larger output and so this change also passes several new flags to reduce
the size costs. One is a general improvement from outlining ASan
instrumentation. The others are in a special feature as they reduce the
error message quality for two of the more expensive UBSan checks in
favor of small generated code size. Keeping these last two separate
allows disabling this locally if needed to debug a failure.
This only sets the symbolizer for our more used targets; not sure if
there's a great way to set it everywhere (I suppose I could try wrapping
cc_binary etc rules if there's a strong preference).
There is a downside here, symbolizing a fastbuild crash seems to take
about 3s. Not sure if there's a good way to get a faster llvm-symbolizer
execution...?
I tried running with the new LLVM update without the
LLVM_SYMBOLIZER_PATH, and it looks like that's insufficient. With the
settings, I now get readable crashes:
```
#9 0x000055dc007d1724 void Carbon::Internal::CheckFail<Carbon::TemplateString<5>{"FATAL"}, Carbon::TemplateString<27>{"toolchain/driver/driver.cpp"}, 84, Carbon::TemplateString<0>{}, Carbon::TemplateString<3>{"err"}>() (/usr/local/google/home/jperkins/.cache/bazel/_bazel_jperkins/85deb7d9d96f7e0e80b42618a55969d7/sandbox/linux-sandbox/9383/execroot/_main/bazel-out/k8-fastbuild/bin/toolchain/testing/file_test.runfiles/_main/toolchain/testing/file_test+0x2894724)
```
Note the LLVM update is for
https://github.com/llvm/llvm-project/pull/109021
I think `paths.parents[2]` is easier to read than
`path.parent.parent.parent`, just applying uniformly. I'd noticed this
while looking at #4227
Also remove a couple `resolve()` calls that shouldn't be necessary since
`__file__` is absolute (elsewhere in the same files, `resolve()` is used
to resolve potentially relative paths)
This improve the toolchain's performance by about 10%.
It will also allow us to leverage TCMalloc's extensions to do heap
profiling and get other information about how efficiently we're using
the heap.
Note that currently this causes all of our builds to produce a warning
due to an issue with `rules_python` and multiple modules registering
python toolchains:
https://github.com/bazelbuild/rules_python/issues/1818
This is also only enabled on Linux as there is no support for other OSes
at the moment.
Chandler Carruth
Richard Smithandjosh11b
David Blaikie