This parallelizes the compilations and dramatically reduces the time to
build runtimes.
As part of this, teach the driver infrastructure to have an option to
control the use of threads and to build the relevant thread pool and
thread it into the various APIs.
However, it requires our `ClangRunner` to become thread-safe and to
invoke Clang in a way that is thread-safe. This is somewhat challenging
as the code in `clang_main` is distinctly _not_ thread-safe.
To address this, the relevant logic of `clang_main`, especially the CC1
execution, is extracted into our runner and cleaned up to be much more
appropriate in a multithreaded context. Much of this code should
eventually be factored back into Clang, but that will be a follow-up
patch to upstream.
Last but not least, this rearranges the `ClangRunner` API to make a bit
more sense out of the different options for building runtimes, and have
a clean model for which things need to be passed in at which points.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
This layer allows runtimes to be built on-demand but cached in a
consistent and re-usable location on the system. It handles careful
filesystem operations to ensure consistency even in the face of multiple
versions and build configurations.
This addresses a number of TODOs from the initial runtimes building
on-demand, and sets the stage to scale up to more runtimes.
This doesn't switch on-demand runtimes to be on by default, I wanted to
wait and make that change as a separate step.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
This is the first step to having Clang's runtime libraries fully
available for the Carbon toolchain. This PR focuses on the lowest level
runtimes, the CRT files and the builtins library.
The goal is to intercept Clang runs where it needs these
target-dependent pieces to be available, and build them on demand using
our Clang-running infrastructure. This avoids most of the subprocess
overhead, but there is still some due to missing features in Clang.
This requires exporting the sources for these runtimes from the Bazel
build, and installing them in our target-independent resource directory.
We then build a simplified "build" of these sources within the
`ClangRunner` itself to produce the specific artifacts and layout
expected by Clang.
It also required fixing our use of Clang on macOS to have a default
system root in order to successfully compile or link.
It also required cleaning up how the `ClangRunner` used target
information more generally -- instead of taking the target as
a constructor parameter, it manages its target internally and relies on
the Clang target-specifying command line flags.
I looked at whether we could split this into another layer separate from
the `ClangRunner`, but that proved frustratingly difficult to manage.
While we support building these on-demand as part of a detected link,
that doesn't seem feasible as we don't have the necessary separation
between compilation runs of Clang and link runs of Clang. However,
I have tried to factor the internals to provide as clear of separation
as I could across these.
I have also created a stand-alone subcommand to directly build the
runtimes which allows for easy testing. It also supports building them
into a specific directory, and that directory can in turn be passed to
a Clang invocation. This is designed to work both at the API level with
`ClangRunner` and at the subcommand level.
Currently, the only part of the commandline that is detected and
forwarded to the runtimes build is the target. Eventually, the plan is
to expand this so that we can build a maximally tailored set of runtimes
for a given compilation.
The other big TODO here is to actually implement caching storage of
these runtimes so they aren't built on every execution. Right now, this
uses a somewhat hack-y build of a temporary directory, but this isn't
expected to be suitable long-term. Building these runtimes on *every*
link makes those commands take approximately 15 seconds with an ASan
build like our default development build, and just over 2 seconds in an
optimized build. Because of this, I've kept all of this disabled by
default for now. The goal is that once caching and some other
improvements land, we can enable this by default.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
The old function name caused some confusion during the review of #5338,
sending this to see if it provides a less surprising function name and
boolean result. Happy to try other names / approaches as well.
We end up needing to do this in any driver subcommand that reaches into
external code that may not be fully fuzz-clean. No need to grow multiple
different diagnostics for each, we can use a common diagnostic.
Sadly, the Clang driver unconditionally injects the `-disable-free` flag
to CC1 invocations. =/ This ends up with us leaking memory when invoking
Clang programmatically, for example in unit tests.
I've fixed this by post-processing the flags. The alternatives I see all
involve duplicating even more code from within Clang's internals into
our runner, and we already have a lot of that. I have left a TODO to try
and follow up with upstream about fixing this in a more sustainable way,
but wanted to get the testing in place anyways.
I've also threaded a flag through the various layers so that when we're
on the command line we can actually skip this and get the same compile
time benefits that Clang itself gets from disabling freeing all of the
internal data structures.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This switches most error printing to use diagnostics instead of direct
stream writes, even when not a specific file diagnostic. I'm allowing
empty filenames for this use-case.
This allows a little more specific testing to validate coverage of
output using the diagnostic coverage test. I'm adding a few tests to
cover things that weren't previously tested.
Separately, this also forces a little more standardization in format...
considering how changes like #4568 show effort being spent to _mirror_
diagnostic style, my thought is now to just use diagnostic code where
possible.
Note this also allows incrementally better testing of the language
server; I'm changing the crash fix from #4847 in favor of diagnostic
testing.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
The language server needs stdin, and for tests we should be passing it
around. My intent is to pass in a faux stdin to Driver for language
server tests.
As long as I'm adding a new parameter, I was looking at also changing
the way streams are passed in to Driver for style (pointers since
they're held past construction lifetime). Since these are all stored in
DriverEnv, I thought it might be a net improvement to use the struct
directly, getting more explicit parameter names and also removing the
need for `SetFuzzing`.
I'm trying here to avoid functional changes, but there are a couple
additional fixes like removing an obsolete `find_insensitive` and
refactoring how `ValidateOptions` handles errors (because it reduces the
number of spots that operate on error_stream).
As discussed in #4530 . This required switching to using
`llvm::IntrusiveRefCntPtr` in a number of places.
---------
Co-authored-by: Josh L <josh11b@users.noreply.github.com>
This teaches the driver library to track when its being used with
fuzzing and disables the `clang` subcommand from actually running Clang.
The Clang libraries have a large backlog of fuzzer-found issues that
isn't being actively reduced, so we can't productively fuzz into it.
This lets us more productively fuzz at the top level.
This is also available on the command line itself, which should be
useful if anyone wants to fuzz Carbon from the command line using tools
like AFL -- they can inject this flag to avoid getting noise from the
fuzzer hitting known issues in Clang.
This makes something like `bazel run :toolchain -- clang -- -c test.cpp`
work, because that can be run in-process. Note that `bazel run
:toolchain -- clang -- test.cpp` still requires subprocessing, and does
not work.
Note, the vision here is that we are trying to align how clang and
carbon compile c++ code. This is work towards intertwining command
execution.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Chandler CarruthandDana Jansens
Thomas Köppe
josh11bandJosh L