This parallelizes the compilations and dramatically reduces the time to
build runtimes.
As part of this, teach the driver infrastructure to have an option to
control the use of threads and to build the relevant thread pool and
thread it into the various APIs.
However, it requires our `ClangRunner` to become thread-safe and to
invoke Clang in a way that is thread-safe. This is somewhat challenging
as the code in `clang_main` is distinctly _not_ thread-safe.
To address this, the relevant logic of `clang_main`, especially the CC1
execution, is extracted into our runner and cleaned up to be much more
appropriate in a multithreaded context. Much of this code should
eventually be factored back into Clang, but that will be a follow-up
patch to upstream.
Last but not least, this rearranges the `ClangRunner` API to make a bit
more sense out of the different options for building runtimes, and have
a clean model for which things need to be passed in at which points.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
This layer allows runtimes to be built on-demand but cached in a
consistent and re-usable location on the system. It handles careful
filesystem operations to ensure consistency even in the face of multiple
versions and build configurations.
This addresses a number of TODOs from the initial runtimes building
on-demand, and sets the stage to scale up to more runtimes.
This doesn't switch on-demand runtimes to be on by default, I wanted to
wait and make that change as a separate step.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
This is the first step to having Clang's runtime libraries fully
available for the Carbon toolchain. This PR focuses on the lowest level
runtimes, the CRT files and the builtins library.
The goal is to intercept Clang runs where it needs these
target-dependent pieces to be available, and build them on demand using
our Clang-running infrastructure. This avoids most of the subprocess
overhead, but there is still some due to missing features in Clang.
This requires exporting the sources for these runtimes from the Bazel
build, and installing them in our target-independent resource directory.
We then build a simplified "build" of these sources within the
`ClangRunner` itself to produce the specific artifacts and layout
expected by Clang.
It also required fixing our use of Clang on macOS to have a default
system root in order to successfully compile or link.
It also required cleaning up how the `ClangRunner` used target
information more generally -- instead of taking the target as
a constructor parameter, it manages its target internally and relies on
the Clang target-specifying command line flags.
I looked at whether we could split this into another layer separate from
the `ClangRunner`, but that proved frustratingly difficult to manage.
While we support building these on-demand as part of a detected link,
that doesn't seem feasible as we don't have the necessary separation
between compilation runs of Clang and link runs of Clang. However,
I have tried to factor the internals to provide as clear of separation
as I could across these.
I have also created a stand-alone subcommand to directly build the
runtimes which allows for easy testing. It also supports building them
into a specific directory, and that directory can in turn be passed to
a Clang invocation. This is designed to work both at the API level with
`ClangRunner` and at the subcommand level.
Currently, the only part of the commandline that is detected and
forwarded to the runtimes build is the target. Eventually, the plan is
to expand this so that we can build a maximally tailored set of runtimes
for a given compilation.
The other big TODO here is to actually implement caching storage of
these runtimes so they aren't built on every execution. Right now, this
uses a somewhat hack-y build of a temporary directory, but this isn't
expected to be suitable long-term. Building these runtimes on *every*
link makes those commands take approximately 15 seconds with an ASan
build like our default development build, and just over 2 seconds in an
optimized build. Because of this, I've kept all of this disabled by
default for now. The goal is that once caching and some other
improvements land, we can enable this by default.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This removes the separately built and installed LLD binary. The symlinks
used by Clang when directly invoking LLD now point back to the main
`carbon-busybox` binary and dispatch through the newly added subcommand.
With this change we're down to shipping a single busybox binary in the
toolchain, removing duplicate installed copies of LLD and all its LLVM
dependencies. =]
The LLD subcommand works a bit differently from the `clang` subcommand
because the CLI for LLD is specific to which platform flavor of linker
is being invoked.
As part of this, I've extracted some of the common functionality in the
Clang runner into a base class that can be re-used. I expect to use this
again in a follow-up change to add subcommands to run other LLVM tools.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Sadly, the Clang driver unconditionally injects the `-disable-free` flag
to CC1 invocations. =/ This ends up with us leaking memory when invoking
Clang programmatically, for example in unit tests.
I've fixed this by post-processing the flags. The alternatives I see all
involve duplicating even more code from within Clang's internals into
our runner, and we already have a lot of that. I have left a TODO to try
and follow up with upstream about fixing this in a more sustainable way,
but wanted to get the testing in place anyways.
I've also threaded a flag through the various layers so that when we're
on the command line we can actually skip this and get the same compile
time benefits that Clang itself gets from disabling freeing all of the
internal data structures.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This is caused by setting a temporary lambda to a `llvm::function_ref`.
The fix is to assign the lambda into a variable that outlives the
`llvm::function_ref`.
As discussed in #4530 . This required switching to using
`llvm::IntrusiveRefCntPtr` in a number of places.
---------
Co-authored-by: Josh L <josh11b@users.noreply.github.com>
For reference, we're going down the busyboxing route because Carbon
depends on Clang, and we want both to be available as binaries.
Busyboxing allows this while avoiding duplicating symbols between
multiple binaries.
I'm removing the `cc_binary` for `driver:carbon` because I want to avoid
a significant increase in binary outputs; `bazel run //toolchain` still
works great.
This still doesn't have great test coverage (but non-zero:
`//examples:sieve` still builds/runs, for example). The problem is that
we want to avoid subprocessing for performance, but this mainly deals
with subprocessing. I'm still thinking about good approaches for that,
since we'll probably want more significant testing for `clang`
interaction... the solution might involve busyboxing `file_test` too.
Note development on this ran into the argv issue being fixed in #4405
`ToolContext` should be explicitly initialized.
`-c` can still require a valid, writable `-o` path.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
This makes something like `bazel run :toolchain -- clang -- -c test.cpp`
work, because that can be run in-process. Note that `bazel run
:toolchain -- clang -- test.cpp` still requires subprocessing, and does
not work.
Note, the vision here is that we are trying to align how clang and
carbon compile c++ code. This is work towards intertwining command
execution.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
This switches `DCHECK` and `FATAL` as well.
The goal is to reduce the code size impact of these assertions so that
we can keep more of them enabled. Currently, the largest cost I see from
`CHECK` is not the actual check or the cold code itself, but actually
the failure to inline trivial functions due to the presence of the cold
code. This means that our goal isn't to reduce apparent code size in the
final binary but the LLVM IR cost assessed for these routines in the
inliner, which closely correlates with code size but is a bit different.
As discussed in #4283, experimentation shows that a single function call
with a minimal number of arguments is the lowest cost model for these.
This is easily achieved with a format-string API that internally uses
`llvm::formatv`. This PR is essentially the `CHECK` version of #4283.
However, the check macros are substantially harder to make work with
both format strings and streaming because they also take a condition.
Also, unexpectedly, I was very successful at devising a regular
expression based automated rewrite from the streaming to the format
string form with only low 10s of manual fixes. This includes compacting
strings broken up across lines, etc. Given how well that went, I've
prepared this PR which just directly switches to the format string API
and migrate everything to use it.
One nice side-effect is that the format string approach ends up greatly
simplifying the implementation here as well.
This is ... *shockingly* effective. Parsing speeds up by more than 3%
with just this change. And checking speeds up by **8%** with this change
alone:
```
BM_CompileAPIFileDenseDecls<Phase::Parse>/256 86.3µs ± 1% 82.9µs ± 1% -3.94% (p=0.000 n=17+19)
BM_CompileAPIFileDenseDecls<Phase::Parse>/1024 431µs ± 1% 415µs ± 1% -3.76% (p=0.000 n=18+19)
BM_CompileAPIFileDenseDecls<Phase::Parse>/4096 1.77ms ± 1% 1.71ms ± 1% -3.18% (p=0.000 n=18+19)
BM_CompileAPIFileDenseDecls<Phase::Parse>/16384 7.44ms ± 1% 7.17ms ± 2% -3.56% (p=0.000 n=18+20)
BM_CompileAPIFileDenseDecls<Phase::Parse>/65536 30.7ms ± 1% 29.7ms ± 1% -3.15% (p=0.000 n=18+20)
BM_CompileAPIFileDenseDecls<Phase::Parse>/262144 131ms ± 1% 127ms ± 1% -2.81% (p=0.000 n=18+18)
BM_CompileAPIFileDenseDecls<Phase::Check>/256 878µs ± 2% 800µs ± 1% -8.91% (p=0.000 n=19+20)
BM_CompileAPIFileDenseDecls<Phase::Check>/1024 1.88ms ± 2% 1.72ms ± 1% -8.56% (p=0.000 n=19+20)
BM_CompileAPIFileDenseDecls<Phase::Check>/4096 5.78ms ± 2% 5.28ms ± 1% -8.70% (p=0.000 n=20+18)
BM_CompileAPIFileDenseDecls<Phase::Check>/16384 21.9ms ± 1% 20.1ms ± 1% -8.02% (p=0.000 n=18+20)
BM_CompileAPIFileDenseDecls<Phase::Check>/65536 90.4ms ± 2% 83.1ms ± 1% -8.04% (p=0.000 n=19+20)
BM_CompileAPIFileDenseDecls<Phase::Check>/262144 381ms ± 2% 352ms ± 1% -7.79% (p=0.000 n=19+19)
```
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com>
The install directory contains the BUILD logic for creating an
installable tree of data files and executables for the toolchain, and
a library to facilitate toolchain code accessing the paths to their data
within this installation.
Then adds an installation of LLD in a synthetic LLVM installation, and
teaches the Clang runner to configure this and use it for linking
instead of the system linker.
Currently, the install paths only really manage access to the LLVM
binaries installed and used by the Clang runner for linking, but
eventually other data files like the prelude and runtime libraries will
be fleshed out as well. There are TODOs for moving more things over here
such as the prelude.
One interesting aspect of this is where to put helpers like parts of
LLVM in our install. This PR suggests nesting those files under
`lib/carbon`. While using a `lib` subdirectory isn't a perfect fit for
the FHS (Filesystem Hierarchy Standard), having a single location where
private data is collected is significantly superior to spreading them
across the system. This also matches similar patterns used by Clang
itself and several other language toolchains and standard libraries.
The install directory also provides a natural place for us to build out
packaging rules to create installable packages in various formats, but
that remains future work.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This adds the most rudimentary support for linking in the Carbon driver
by calling out to the Clang driver and letting it do the linking. This
is a super minimal version to start with, just enough to be somewhat
useful and to exercise calling into Clang for this.
Also adds a Clang runner to help run the Clang driver. This is a bit
more complete, although it sill needs some significant work. For
example, it will need some significant enhancements to be able to
compile C++ code, etc.
One thing that will likely change here is to better manage streams and
work better as a library. For now, this is a bit limited because Clang
and LLVM don't provide the necessary support for this.
Chandler CarruthandDana Jansens
josh11bandJosh L
Thomas Köppe
Boaz Brickner