Fixes#6895
Note this is just a short-term fix to avoid confusion, as the compile
structure needs to change on the whole.
Assisted-by: Google Antigravity with Gemini
Add an optional additional set of positional parameters that can be
passed to the `link` subcommand for Clang-style (or GCC-style)
`LDFLAGS`. These can _also_ contain object files, etc., and in fact it
is useful to allow them to contain object files in order to integrate
the `carbon link` subcommand into a build system that mixes both link
flags and object files. This at least happens with Bazel, and I suspect
is common.
Eventually, it would be nice to have sufficient semantics to handle all
the varieties of links we want without resorting to this escape hatch,
but that's likely a long way away and so it seems especially useful to
allow falling back to Clang's flags as needed for now.
This does somewhat directly surface the Clang implementation detail in
the command line syntax, but I don't see a lot of good alternatives.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
The goal here is to be able to construct a build of the runtimes
directly in Bazel, or by emitting `BUILD` files, or by emitting into C++
code and using that on-demand. For that, we want a single source of
truth, and that source in Starlark.
This should also make the information more generally useful, and so I'm
moving as much as I can into the LLVM Bazel build. Apologies as that
makes the diffs extra annoying.
I do plan on upstreaming the Bazel parts of this, but would like to get
everything working in Carbon and stabilized first.
While here, I've also made a change suggested for the future in the
initial review by lifting the C++ template out of a string literal in
the `.bzl` file, and into an actual separate C++ file.
This only moves libc++, libc++abi, and libunwind. I want to get those
three working end-to-end before I work on the builtins or `crtbegin` and
`crtend`, as those have a bunch of additional complexity.
This also only uses the info in the C++ on-demand build. It seemed like
a reasonable increment to start code review, and my plan is to work on
other build strategies in a follow-up PR. If that doesn't work, let me
know and I'll come back once I have at least a second use of the info
here.
Multiple subcommands all need the ability to disable on-demand runtime
building, and this may be needed outside of using _prebuilt_ runtimes.
For example, with Bazel the plan is to not build runtimes at all and
have Bazel provide them as native Bazel libraries.
Updates the `link` subcommand to respect this flag when running Clang to
perform links.
We didn't have any real testing of the `link` subcommand, in part
because it was difficult -- it would try to link runtime libraries. Now
that we can prevent building them on demand, we can use that to test the
link command. That in turn helped uncover a couple of bugs that are
fixed here.
1) The `driver_env_` member of the `Driver` was re-used across
`RunCommand` invocations. Some of its fields are constant across
these, others can be updated, and still more are not necessarily
something we would expect to be re-used. This fixes that by removing
the `driver_env_` member, and replacing it with members for just the
fields of `DriverEnv` that we want to set initially based on the
construction of the `Driver` object. This causes multiple, sequential
`RunCommand` calls to not clobber or erroneously inherit state.
2) The temporary directory support in the driver unittest didn't allow
the driver to observe the things it wrote to the temporary directory.
This PR updates the test logic to create an overlay VFS so that both
the in-memory test inputs are observed, but so are the real files
written into the temporary directory.
3) The Clang runner, when asked to run Clang without runtimes would
still attempt to include runtimes in any link command. This isn't
quite what we want, as the whole reason to use this without building
runtimes is to reuse ones built in some other way and potentially in
some other location. For now, this PR uses a hack to suppress these
issues so that we can have a basic test, but in the future we'll need
a better solution here.
4) The driver test didn't include the actual driver in the install data.
The test even worked around this, but it makes it impossible to link
reliably as the `lld` binary isn't available. This adds the data
dependency and updates the test to the available digest, etc.
Mainly because "sorting_diagnostic_consumer" is legacy, since
`SortingDiagnosticConsumer` became `SortingConsumer`. Also better
reflecting contents of these files.
Where I'm not renaming, I'm less positive about dropping "diagnostics"
from "file_diagnostics" and "null_diagnostics" (which contain both a
consumer and emitter, and "null.h" seems like poor naming), so not doing
that here. Also "diagnostic.h" contains `struct Diagnostic`, so is a
decent fit.
Assisted-by: Google Antigravity with Gemini 3 Flash
This makes it easy to wire up build systems like Bazel that need to know
the actual include paths used. It also gives us a convenient place to
export any other information that build systems or integrations need,
and to get debugging info from users.
Most of the complexity is computing the Clang header search paths, but
I couldn't see a direct way to get closer to the source-of-truth than
this, and it doesn't seem _too_ unreasonable.
Depends on #6636 - start review at commit
[643fdab1](6637/commits/643fdab1)
Background:
https://docs.google.com/document/d/1wi85FRiWh4X9A-gCYMVGKR40-q5fM6-3JaSpePk-XCY/edit?usp=sharing
And specifically this work is essentially an alternative to #5543
Clang's code generation is implemented through an ASTListener
(clang::CodeGenerator) that is attached throughout Clang's
parsing/sema/code
generation phases and acts on Clang AST incrementally throughout that
process.
Prior to this patch, Carbon has only created the CodeGenerator during
Carbon's
`lower` phase, missing out on key callbacks that would be made by Clang
during
`check`. Some of these issues were addressed by #6237 and #6483 - but
there were
still remaining cases where the delayed processing lead to missing
functionality.
With #6483 much of the Clang code that made multithreaded complexity of
#5543 is
no longer present, and we have access to the point of ASTListener
registration
so we can register the CodeGenerator there and consume its resulting
llvm::Module during lower.
Examples of some of the bugs this addresses are seen in the linked doc,
and
checked in as tests in this change in
`clang_code_generator_callbacks.carbon`
An indicental bug that's also fixed, and caused all the other test case
churn,
is that the `CodeGenerator` created during `lower` wasn't getting passed
the
Clang `CodeGenOpts` and was creating its own default - so, most notably,
optimization flags were not respected. This meant that the LLVM IR from
Clang
was always -O0 style IR (optnone, no inlinehint, no TBAA, etc). With
this
change, now the Clang IRGen gets the real `CodeGenOpts` and respects
optimization/other flags specified there.
This is only meant to be a rough proof of concept - I'm totally open to
reworking this in any way (even quite substantially) if folks have ideas
about
how this should be implemented most generally/elegantly/etc.
This uses the existing Clang driver APIs for expanding response files
and so should be pretty carefully accurate to what is needed here.
Note that this doesn't try to generalize the expansion more widely for
the interop Clang invocation, but it would be straightforward to do so
if needed at some point.
This allows us to re-use the on-demand runtimes building, but in
a framework that is (much) more Bazel compatible:
- It creates a Bazel rule to generate the runtimes tree
- The generated runtimes tree is adjusted to integrate with Bazel's
output tracking and caching infrastructure so it doesn't need to be
rebuilt when a cached set of runtimes is available
- The build occurs during the build phase and the action informs Bazel
about the CPU usage to give Bazel a chance to not run other parts of
the build when there are no execution resources available
- The binary is factored into a stand-alone program for the Clang
runtimes, which depends on a minimal amount of Carbon and notably
avoids the busybox or installation. This should cause almost all
builds to get a cache hit here unless Clang itself is updated.
Some refactoring of the codegen options was done to support this. I've
tried to factor some of the code between this and the `build-runtimes`
subcommand, but it was challenging to do more without adding substantial
complexity or dependencies on more Carbon infrastructure than is
necessary. I think the result is tolerable, but open to suggestions
here if folks see specific changes that would improve things.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This unifies the default Clang arguments between the `clang` subcommand,
the `link` subcommand, and the `ClangInvocation` built for C++ interop.
This sets the stage to integrate either pre-built or on-demand runtimes
flags for both of these. However, this PR should have very little
practical difference. The biggest functional change is wrapping the
default arguments in flags to allow unused flags so that we can build a
collection of flags viable across compile and link.
This extracts the C-string `argv`-like building routine to a more
broadly reusable location. It also sinks the verbose logging logic out
of it and into the relevant runners. In turn, it simplifies the verbose
logging logic significantly.
The biggest functional change is removing the implicit synthesis of a
tool's `-v` verbose flag from the presence of a `vlog` stream. I thought
this would be helpful, but in practice of debugging these layers it has
been more of a hindrance than a help -- I pretty often only want verbose
logging on one side or the other, and we have ways of explicitly passing
a `-v` flag to the underlying tools already. I think my instinct to do
this was just wrong, so rip it out and simplify.
This does add an unused feature -- prepending a prefix of arguments
while building the C-string variant. This isn't used in this PR but will
be used in subsequent PRs and it seemed more disruptive to undo that
logic and then re-do it in a later PR. Let me know if it's too confusing
here.
Assisted-by: Gemini Code Assist
---------
Co-authored-by: Geoff Romer <gromer@google.com>
Previously this was kept in `//toolchain/install` so it would be near to
the code that actually defines the installation layout. However, that
creates somewhat unfortunate dependency cycles between
`//toolchain/install` and other directories. Exacerbating this, a
subsequent PR is likely to add dependencies on it from
`//toolchain/base` itself that suggests that is the correct layering.
This PR just moves the code mechanically with as few other edits as
possible.
This builds on the previous work to flesh out more on-demand runtimes
building. It adds building of the `libc++.a` archive runtime.
A number of changes are required for this to work:
- The runtimes build infrastructure needs to support building sources
from multiple parts of LLVM rather than a single part. We do this by
lifting the root of the runtimes source paths up a level to a common
runtimes tree, and installing the runtimes sources below this
directory.
- Both libc++ and libc++abi runtimes sources need to be installed, and
we even need to install some interesting parts of llvm-libc that are
used in the build of libc++.
- We need to generate the site configuration header file for libc++ from
the CMake template. This includes both setting up a set of
platform-independent defines and introducing some basic Bazel support
for processing the CMake template itself.
Doing all of this also exposed some missing features and limitations of
the runtimes building infrastructure that are addressed here.
One note is that all of this just adds libc++ to the explicit
`build-runtimes` command for testing. It doesn't yet trigger
automatically building these prior to linking, or configuring any of the
other subcommands to automatically use these runtimes. All of that will
come in follow-up PRs.
Also, this makes the `clang_runtimes_test` ... _very_ slow in our
default build configuration. Compiling libc++, even with many threads on
a large Linux server requires up to 50 seconds. I'm open to any
suggestions on how to handle this, including disabling the test in
non-optimized builds. I have some ideas to speed this up, but
fundamentally building libc++ is... not cheap.
I did look at some of the existing Bazel tools to process the CMake
template, but they all seemed significantly more complex than what we
need and didn't have broad adoption. Given that, it seemed slightly
better to just roll our own given the simple format.
Two of the new LLVM patch are currently under review upstream and so
hopefully temporary:
- https://github.com/llvm/llvm-project/pull/169155
- https://github.com/llvm/llvm-project/pull/169292
This clarifies that the CC1 logic is directly extracted from Clang.
There are probably some other places in the toolchain we should extract
code like this where we're replicating and customizing logic from LLVM,
but wanted to start here.
This is the first real step towards building libc++ itself, and fleshes
out both the core runtimes management logic and the archive-based
runtimes logic for a quite simple runtime.
Nothing here causes us to _use_ libunwind, and in fact this doesn't
include even the "on-demand" aspect of building `libunwind`. Instead,
this just wires it up to the explicit `build-runtimes` subcommand for
simple testing. The full integration along side the target directory is
future work.
Previously, the Clang runtimes building only considered building the
target resource directory, and was only _internally_ asynchronous.
Because the asynchrony was only internal, it could use the function
frame as a context object throughout the build of the resource dir. This
is simple but doesn't generalize well to more runtimes: if we want to
add 2 or 3 more runtimes, we want them to _all_ build asynchronously.
That means using some asynchronous builder that maintains the context
and allows them to proceed concurrently with other work.
This also factors all the runtimes building code into a separate set of
files. These aren't separate libraries at this point due to the
`ClangRunner` in some cases wanting to build runtimes on-demand, but it
at least lets us organize the code more cleanly.
Because this splits code between `clang_runner.*` and
`clang_runtimes.*`, it also works to update the `#include`s for both to
be roughly accurate. I used ClangD's include cleaner for this and it
probably also did some latent cleaning as it went, but that's the reason
for the churn of `#include` lines.
The archive building is also factored out into a re-usable helper. This
is a bit "over factored" in this PR, but supports the next PR that uses
the same code to build archives for other runtimes.
This also overhauls the synchronization used -- it uses a simple `Latch`
construct introduced in a previous PR to coordinate between the steps of
building the runtimes.
Last but not least, it factors the "enable leaking" state out of a
boolean in the runner to a parameter. This is important in the face of
concurrent calls as otherwise toggling this boolean can create a race.
The next PR will layer building more runtimes on top of this new
factoring.
---------
Co-authored-by: David Blaikie <dblaikie@gmail.com>
Adds a flag `--optimize=<mode>` that specifies what to optimize for:
* `--optimize=none` turns off the optimizer as much as possible, but
still respects always_inline.
* `--optimize=debug` aims to be the equivalent of `-Og` / `-O1`, and
provides optimizations that don't affect the ability to debug the
program. This is the default.
* `--optimize=size` optimizes for the size of the produced program, and
aims to be the equivalent of `-Oz`.
* `--optimize=speed` optimizes for the execution time of the produced
program, and aims to be the equivalent of `-O3`.
Following the approach taken by Clang, the optimization level feeds into
both the configuration of the LLVM pass pipeline and the attributes
added to function definitions generated by the frontend.
Optimization is performed in a new phase, `optimize`, which runs between
`lower` and `codegen`.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
This parallelizes the compilations and dramatically reduces the time to
build runtimes.
As part of this, teach the driver infrastructure to have an option to
control the use of threads and to build the relevant thread pool and
thread it into the various APIs.
However, it requires our `ClangRunner` to become thread-safe and to
invoke Clang in a way that is thread-safe. This is somewhat challenging
as the code in `clang_main` is distinctly _not_ thread-safe.
To address this, the relevant logic of `clang_main`, especially the CC1
execution, is extracted into our runner and cleaned up to be much more
appropriate in a multithreaded context. Much of this code should
eventually be factored back into Clang, but that will be a follow-up
patch to upstream.
Last but not least, this rearranges the `ClangRunner` API to make a bit
more sense out of the different options for building runtimes, and have
a clean model for which things need to be passed in at which points.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
This layer allows runtimes to be built on-demand but cached in a
consistent and re-usable location on the system. It handles careful
filesystem operations to ensure consistency even in the face of multiple
versions and build configurations.
This addresses a number of TODOs from the initial runtimes building
on-demand, and sets the stage to scale up to more runtimes.
This doesn't switch on-demand runtimes to be on by default, I wanted to
wait and make that change as a separate step.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
This is the first step to having Clang's runtime libraries fully
available for the Carbon toolchain. This PR focuses on the lowest level
runtimes, the CRT files and the builtins library.
The goal is to intercept Clang runs where it needs these
target-dependent pieces to be available, and build them on demand using
our Clang-running infrastructure. This avoids most of the subprocess
overhead, but there is still some due to missing features in Clang.
This requires exporting the sources for these runtimes from the Bazel
build, and installing them in our target-independent resource directory.
We then build a simplified "build" of these sources within the
`ClangRunner` itself to produce the specific artifacts and layout
expected by Clang.
It also required fixing our use of Clang on macOS to have a default
system root in order to successfully compile or link.
It also required cleaning up how the `ClangRunner` used target
information more generally -- instead of taking the target as
a constructor parameter, it manages its target internally and relies on
the Clang target-specifying command line flags.
I looked at whether we could split this into another layer separate from
the `ClangRunner`, but that proved frustratingly difficult to manage.
While we support building these on-demand as part of a detected link,
that doesn't seem feasible as we don't have the necessary separation
between compilation runs of Clang and link runs of Clang. However,
I have tried to factor the internals to provide as clear of separation
as I could across these.
I have also created a stand-alone subcommand to directly build the
runtimes which allows for easy testing. It also supports building them
into a specific directory, and that directory can in turn be passed to
a Clang invocation. This is designed to work both at the API level with
`ClangRunner` and at the subcommand level.
Currently, the only part of the commandline that is detected and
forwarded to the runtimes build is the target. Eventually, the plan is
to expand this so that we can build a maximally tailored set of runtimes
for a given compilation.
The other big TODO here is to actually implement caching storage of
these runtimes so they aren't built on every execution. Right now, this
uses a somewhat hack-y build of a temporary directory, but this isn't
expected to be suitable long-term. Building these runtimes on *every*
link makes those commands take approximately 15 seconds with an ASan
build like our default development build, and just over 2 seconds in an
optimized build. Because of this, I've kept all of this disabled by
default for now. The goal is that once caching and some other
improvements land, we can enable this by default.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This is trying to make it clearer when vectors are being indexed with
`CheckIRId`.
The only one that I still kind of want to change is the
`SmallVector<std::unique_ptr<CompilationUnit>>`, but because it's a
`unique_ptr` that's a little more complex. I may not bother.
Note, some of the changes around nuanced `SmallVector` interactions were
based on trying to copy the way `SmallVector` itself takes arguments,
like with range passing.
By moving dumping, we can have dumping occur before verification that
might CHECK-fail (e.g. parse tree and llvm IR verification).
I'm dropping vlogging of raw semir. It was only done when dumping, so
`-v` would print zero copies and `-v --dump-raw-sem-ir` would print two
copies. The lack of complaints about this suggests it's not needed.
I'm making a small change to drop newlines between textual and raw
semir. This is an edge case so I don't expect people to really notice in
general, but it seemed unusually aware of what's on a stream, and it
made it harder to do the dump_stream/raw_dump_stream approach, which I
felt would be decent in general, since check is the only phase that can
emit two different things (which I could also just drop -- we don't
really use raw semir anymore, it doesn't seem like a big need to be able
to print it with textual semir, but I'm assuming to just maintain
existing behavior).
In parse, we were previously dumping the tree on verification errors.
I'm removing that because now `--dump-parse-tree` should work fine,
where previously it wouldn't.
- Updates incompatible flags.
- `rules_flex` is no longer used, so enable its flag.
- Fixes `sh_test` deps for
`--incompatible_disable_autoloads_in_main_repo`
- Broadens the exception for `rules_cc` and `bazel_tools` due to changes
to runfiles deps; trying to avoid minutiae that shouldn't affect the
decision.
Rules executed by bazel don't necessarily have the right environment to
find the symbolizer, which was the intent of `cc_env` setting
`LLVM_SYMBOLIZER_PATH`. So far, this has kind of been a case-by-case
fix, but every so often I'm trying to debug a crash in a test that
doesn't provide it. Rather continuing down this route, instead add
drop-in wrappers for cc rules so that it's hard to forget.
Note `bazel/cc_rules` is intended to mirror `bazel/carbon_rules` and
`bazel/cc_toolchains`, rather than `@rules_cc`.
AFAICT there isn't a great way to add this as a default for the `bazel
run` environment. It's not typically going to be set on its own,
forwarding `$PATH` would be too broad, and the [action
`env_sets`](https://bazel.build/docs/cc-toolchain-config-reference#using-action-config)
I think are not quite what we need (I think those don't include output
execution, only compilation).
This adds support for most of the remaining LLVM command line tools
using a generic, generated wrapper. The subcommand interface for these
is (much) less interesting than our other subcommands, but it gives us
a uniform and consistent layer.
Note that I structured these as nested sub-sub-commands below an `llvm`
subcommand because of an expectation that we will want to add more, and
ones that don't use this generic layer. Some concrete future work:
- Add the `opt` and `llc` tools as subcommands for easier debugging and
experimentation with LLVM IR output from Carbon's toolchain.
- Potentially sink `lld` below the `llvm` layer given that it has
significantly less user visibility than commands like `clang`.
Unfortunately, the current driver subcommand APIs make nested
subcommands awkward. I've added a somewhat rough hack here to let the
LLVM tools go in, but there are some TODOs that I want to address in
a follow-up that works to adjust the structure of this code to be more
conducive to nesting like this.
Depends on #5048 -- only the last commit should be reviewed here.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
We end up needing to do this in any driver subcommand that reaches into
external code that may not be fully fuzz-clean. No need to grow multiple
different diagnostics for each, we can use a common diagnostic.
This removes the separately built and installed LLD binary. The symlinks
used by Clang when directly invoking LLD now point back to the main
`carbon-busybox` binary and dispatch through the newly added subcommand.
With this change we're down to shipping a single busybox binary in the
toolchain, removing duplicate installed copies of LLD and all its LLVM
dependencies. =]
The LLD subcommand works a bit differently from the `clang` subcommand
because the CLI for LLD is specific to which platform flavor of linker
is being invoked.
As part of this, I've extracted some of the common functionality in the
Clang runner into a base class that can be re-used. I expect to use this
again in a follow-up change to add subcommands to run other LLVM tools.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This should make it easy to add more tests which need the specific
behavior here. It also isolates where we reach into GoogleTest's
internals to a single common place.
Small cleanup for code sharing. Note, `*ReadFile` will trigger a
CHECK-failure on error; the relevant implementations would previously
have failed silently (empty string).
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
The language server needs stdin, and for tests we should be passing it
around. My intent is to pass in a faux stdin to Driver for language
server tests.
As long as I'm adding a new parameter, I was looking at also changing
the way streams are passed in to Driver for style (pointers since
they're held past construction lifetime). Since these are all stored in
DriverEnv, I thought it might be a net improvement to use the struct
directly, getting more explicit parameter names and also removing the
need for `SetFuzzing`.
I'm trying here to avoid functional changes, but there are a couple
additional fixes like removing an obsolete `find_insensitive` and
refactoring how `ValidateOptions` handles errors (because it reduces the
number of spots that operate on error_stream).
This adds a RawStringOstream. Versus TestRawOstream, which is
consolidated over to RawStringOstream, it uses a string for storage
instead of a vector, mainly to support move-to-string semantics. Versus
llvm::raw_string_ostream, it owns the string and supports pwrite (which
is needed for driver and its fd_ostream compatibility requirement).
This converts most uses of llvm::raw_string_ostream, leaving behind a
few in InstNamer that explicitly cannot own the string, such as:
```
llvm::raw_string_ostream(name)
<< "_" << tree.tokens().GetColumnNumber(token);
```
I have this as its own library so that it can use CHECK.
Yes this doesn't save much code, but it's code we repeatedly write.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
Follow-on to #4669 . Did some manual testing, but may have not exercised
all code paths.
---------
Co-authored-by: Josh L <josh11b@users.noreply.github.com>
These tests typically take 10-20s, but I'm seeing some timeouts
[here](https://github.com/carbon-language/carbon-lang/actions/runs/11899548036/job/33158400417).
This seemed particularly suspicious due to the _absence_ of output
(copied below). That got me looking, and maybe the subprocessing tickles
a cpu bottleneck, so proposing this approach to remove the exec. Even if
this doesn't solve the flakiness, I think it's a simpler implementation.
Note I believe this is intended to work. The `sh` rules rely on shebangs
(as noted at https://bazel.build/reference/be/shell#sh_test), and are
essentially just subprocessing to the input. Note this could've also had
`args` on a `cc_test` rule, but I'd expect the same args to be passed to
`run` where instead the benchmark behavior should be default (and I'm
assuming you'd rather not have args there). Fundamentally this becomes a
symlink:
```
bazel-bin/common/map_benchmark_test -> .../execroot/_main/bazel-out/k8-fastbuild/bin/common/map_benchmark
```
Copying snippet from timeout below:
```
==================== Test output for //common:map_benchmark_test:
/private/var/tmp/_bazel_runner/e591f63ed099023de1f206992dfce127/execroot/_main/bazel-out/darwin_arm64-fastbuild/testlogs/common/map_benchmark_test/test.log
-- Test timed out at 2024-11-18 19:32:13 UTC --
INFO: From Testing //common:map_benchmark_test:
================================================================================
```
This PR adds a `--dump-timings` flag to the `compile` subcommand
(similar to the existing `--dump-mem-usage` flag), which collects timing
data per compilation unit for each compilation phase. For example, on my
2020 M1 MacBook:
```
$ bazel build -c opt //toolchain
$ bazel-bin/toolchain/install/run_carbon compile --phase=lower --dump-timings examples/sieve.carbon | tail
...
---
filename: 'examples/sieve.carbon'
nanoseconds:
lex: 30792
parse: 25458
check: 226625
lower: 1136958
Total: 1419833
...
```
Most of the changes are pretty straightforward. There were a couple I
wasn't sure about though; let me know if I should change:
- new `Timings` class in its own file, pretty similar to the existing
`MemUsage` class
- added a `timings_` field to the `CompilationUnit` class
- added a `timings` field to the `Check::Unit` struct
- renamed `CheckParseTree` function to `CheckParseTreeInner` for ease of
timing with early `return`
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This is in anticipation of making the integer value store be customized
heavily. I'd like to extract it from the common code when doing that, so
first disentangling them here without any intended change in
functionality or behavior to enable that.
I've tried to update `#include`s to be as minimal as I can and added a
few missing includes spotted in the process.
I've split the test for value store to include what was easy focused on
just the value store templates rather than the unified shared value
stores.
This might surface some opportunities for adding more tests, but for
this PR, just doing the minimal restructuring.
Removes the separate language server binary; I'm not sure we need to
provide it. Instead, `carbon language-server` is added as a subcommand.
Moves //language_server to //toolchain/language_server. Splits into a
trivial language_server.h, and a substantive server.h. I wasn't sure
about a better name, but wanted the split similar to check/check.h,
lex/lex.h, etc. At the same time, the class is probably going to be a
little big so not a good fit to through into just a cpp file.
This fixes some style issues with the language server class, but
generally I'm trying to not address things here in order to keep it
simpler.
For reference, we're going down the busyboxing route because Carbon
depends on Clang, and we want both to be available as binaries.
Busyboxing allows this while avoiding duplicating symbols between
multiple binaries.
I'm removing the `cc_binary` for `driver:carbon` because I want to avoid
a significant increase in binary outputs; `bazel run //toolchain` still
works great.
This still doesn't have great test coverage (but non-zero:
`//examples:sieve` still builds/runs, for example). The problem is that
we want to avoid subprocessing for performance, but this mainly deals
with subprocessing. I'm still thinking about good approaches for that,
since we'll probably want more significant testing for `clang`
interaction... the solution might involve busyboxing `file_test` too.
Note development on this ran into the argv issue being fixed in #4405
Undoes most of #4347, because of [performance
complaints](https://discord.com/channels/655572317891461132/707150492370862090/1295527235133898772).
With a 30-ish frame stack trace and `-c dbg`, my installed
`llvm-symbolizer` still seems slow (~6s), but the hermetic
`llvm-symbolizer` adds ~4s (i.e., ~10s total). I don't think we can
easily force the hermetic version to build in opt configuration, so I'm
backing it out.
This extracts out the SourceBuffer handling of `-` in order to trivially
share it.
Note this still has a number of TODOs, it's just setting up the
essential subcommand infrastructure, with some tests demonstrating that
it at least does something.
This only sets the symbolizer for our more used targets; not sure if
there's a great way to set it everywhere (I suppose I could try wrapping
cc_binary etc rules if there's a strong preference).
There is a downside here, symbolizing a fastbuild crash seems to take
about 3s. Not sure if there's a good way to get a faster llvm-symbolizer
execution...?
I tried running with the new LLVM update without the
LLVM_SYMBOLIZER_PATH, and it looks like that's insufficient. With the
settings, I now get readable crashes:
```
#9 0x000055dc007d1724 void Carbon::Internal::CheckFail<Carbon::TemplateString<5>{"FATAL"}, Carbon::TemplateString<27>{"toolchain/driver/driver.cpp"}, 84, Carbon::TemplateString<0>{}, Carbon::TemplateString<3>{"err"}>() (/usr/local/google/home/jperkins/.cache/bazel/_bazel_jperkins/85deb7d9d96f7e0e80b42618a55969d7/sandbox/linux-sandbox/9383/execroot/_main/bazel-out/k8-fastbuild/bin/toolchain/testing/file_test.runfiles/_main/toolchain/testing/file_test+0x2894724)
```
Note the LLVM update is for
https://github.com/llvm/llvm-project/pull/109021
One of the things that ClangRunnerTest is doing is capturing
stderr/stdout because clang prints to it directly. This adds support for
that to FileTest.
I'm renaming the current `capture_output` field to `dump_output` because
the name is ambiguous after this change, and the flag is already named
`--dump_output`. It's still not great, but at least it's more distinct.
Note ClangRunner still doesn't use the vfs; that still needs work. I'm
just moving the NoArgs test over as a trivial test of the functionality.
Jon Ross-Perkins
Chandler CarruthandJon Ross-Perkins
David Blaikie
josh11bandJosh L
Boaz Brickner
Sam EstepandJon Ross-Perkins