Multiple subcommands all need the ability to disable on-demand runtime
building, and this may be needed outside of using _prebuilt_ runtimes.
For example, with Bazel the plan is to not build runtimes at all and
have Bazel provide them as native Bazel libraries.
Updates the `link` subcommand to respect this flag when running Clang to
perform links.
We didn't have any real testing of the `link` subcommand, in part
because it was difficult -- it would try to link runtime libraries. Now
that we can prevent building them on demand, we can use that to test the
link command. That in turn helped uncover a couple of bugs that are
fixed here.
1) The `driver_env_` member of the `Driver` was re-used across
`RunCommand` invocations. Some of its fields are constant across
these, others can be updated, and still more are not necessarily
something we would expect to be re-used. This fixes that by removing
the `driver_env_` member, and replacing it with members for just the
fields of `DriverEnv` that we want to set initially based on the
construction of the `Driver` object. This causes multiple, sequential
`RunCommand` calls to not clobber or erroneously inherit state.
2) The temporary directory support in the driver unittest didn't allow
the driver to observe the things it wrote to the temporary directory.
This PR updates the test logic to create an overlay VFS so that both
the in-memory test inputs are observed, but so are the real files
written into the temporary directory.
3) The Clang runner, when asked to run Clang without runtimes would
still attempt to include runtimes in any link command. This isn't
quite what we want, as the whole reason to use this without building
runtimes is to reuse ones built in some other way and potentially in
some other location. For now, this PR uses a hack to suppress these
issues so that we can have a basic test, but in the future we'll need
a better solution here.
4) The driver test didn't include the actual driver in the install data.
The test even worked around this, but it makes it impossible to link
reliably as the `lld` binary isn't available. This adds the data
dependency and updates the test to the available digest, etc.
This enables on-demand building of runtimes by default, and enables
their header files for all of the Clang invocations. This also switches
the default flags to use the LLVM-provided runtimes (compiler-rt,
libunwind, and libcxx).
This also switches even `llvm_symlinks_test` to use the Bazel prebuilt
runtimes, which requires having a way to pass a Carbon flag even when
invoking the busybox as `clang` or `clang++`. This uses the pattern that
has worked for other Clang wrappers of spelling flags:
`-X<tool-name>=--flag=value`
Last but not least, this updates the Carbon Bazel rules to use our
installed and the Bazel prebuilt runtimes. With that, we make the C++
interop hello-world be enabled by default as this should pass reliably
on both Linux and macOS now.
Previously, the Clang runtimes building only considered building the
target resource directory, and was only _internally_ asynchronous.
Because the asynchrony was only internal, it could use the function
frame as a context object throughout the build of the resource dir. This
is simple but doesn't generalize well to more runtimes: if we want to
add 2 or 3 more runtimes, we want them to _all_ build asynchronously.
That means using some asynchronous builder that maintains the context
and allows them to proceed concurrently with other work.
This also factors all the runtimes building code into a separate set of
files. These aren't separate libraries at this point due to the
`ClangRunner` in some cases wanting to build runtimes on-demand, but it
at least lets us organize the code more cleanly.
Because this splits code between `clang_runner.*` and
`clang_runtimes.*`, it also works to update the `#include`s for both to
be roughly accurate. I used ClangD's include cleaner for this and it
probably also did some latent cleaning as it went, but that's the reason
for the churn of `#include` lines.
The archive building is also factored out into a re-usable helper. This
is a bit "over factored" in this PR, but supports the next PR that uses
the same code to build archives for other runtimes.
This also overhauls the synchronization used -- it uses a simple `Latch`
construct introduced in a previous PR to coordinate between the steps of
building the runtimes.
Last but not least, it factors the "enable leaking" state out of a
boolean in the runner to a parameter. This is important in the face of
concurrent calls as otherwise toggling this boolean can create a race.
The next PR will layer building more runtimes on top of this new
factoring.
---------
Co-authored-by: David Blaikie <dblaikie@gmail.com>
This parallelizes the compilations and dramatically reduces the time to
build runtimes.
As part of this, teach the driver infrastructure to have an option to
control the use of threads and to build the relevant thread pool and
thread it into the various APIs.
However, it requires our `ClangRunner` to become thread-safe and to
invoke Clang in a way that is thread-safe. This is somewhat challenging
as the code in `clang_main` is distinctly _not_ thread-safe.
To address this, the relevant logic of `clang_main`, especially the CC1
execution, is extracted into our runner and cleaned up to be much more
appropriate in a multithreaded context. Much of this code should
eventually be factored back into Clang, but that will be a follow-up
patch to upstream.
Last but not least, this rearranges the `ClangRunner` API to make a bit
more sense out of the different options for building runtimes, and have
a clean model for which things need to be passed in at which points.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
This layer allows runtimes to be built on-demand but cached in a
consistent and re-usable location on the system. It handles careful
filesystem operations to ensure consistency even in the face of multiple
versions and build configurations.
This addresses a number of TODOs from the initial runtimes building
on-demand, and sets the stage to scale up to more runtimes.
This doesn't switch on-demand runtimes to be on by default, I wanted to
wait and make that change as a separate step.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
This is the first step to having Clang's runtime libraries fully
available for the Carbon toolchain. This PR focuses on the lowest level
runtimes, the CRT files and the builtins library.
The goal is to intercept Clang runs where it needs these
target-dependent pieces to be available, and build them on demand using
our Clang-running infrastructure. This avoids most of the subprocess
overhead, but there is still some due to missing features in Clang.
This requires exporting the sources for these runtimes from the Bazel
build, and installing them in our target-independent resource directory.
We then build a simplified "build" of these sources within the
`ClangRunner` itself to produce the specific artifacts and layout
expected by Clang.
It also required fixing our use of Clang on macOS to have a default
system root in order to successfully compile or link.
It also required cleaning up how the `ClangRunner` used target
information more generally -- instead of taking the target as
a constructor parameter, it manages its target internally and relies on
the Clang target-specifying command line flags.
I looked at whether we could split this into another layer separate from
the `ClangRunner`, but that proved frustratingly difficult to manage.
While we support building these on-demand as part of a detected link,
that doesn't seem feasible as we don't have the necessary separation
between compilation runs of Clang and link runs of Clang. However,
I have tried to factor the internals to provide as clear of separation
as I could across these.
I have also created a stand-alone subcommand to directly build the
runtimes which allows for easy testing. It also supports building them
into a specific directory, and that directory can in turn be passed to
a Clang invocation. This is designed to work both at the API level with
`ClangRunner` and at the subcommand level.
Currently, the only part of the commandline that is detected and
forwarded to the runtimes build is the target. Eventually, the plan is
to expand this so that we can build a maximally tailored set of runtimes
for a given compilation.
The other big TODO here is to actually implement caching storage of
these runtimes so they aren't built on every execution. Right now, this
uses a somewhat hack-y build of a temporary directory, but this isn't
expected to be suitable long-term. Building these runtimes on *every*
link makes those commands take approximately 15 seconds with an ASan
build like our default development build, and just over 2 seconds in an
optimized build. Because of this, I've kept all of this disabled by
default for now. The goal is that once caching and some other
improvements land, we can enable this by default.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
The old function name caused some confusion during the review of #5338,
sending this to see if it provides a less surprising function name and
boolean result. Happy to try other names / approaches as well.
We end up needing to do this in any driver subcommand that reaches into
external code that may not be fully fuzz-clean. No need to grow multiple
different diagnostics for each, we can use a common diagnostic.
Sadly, the Clang driver unconditionally injects the `-disable-free` flag
to CC1 invocations. =/ This ends up with us leaking memory when invoking
Clang programmatically, for example in unit tests.
I've fixed this by post-processing the flags. The alternatives I see all
involve duplicating even more code from within Clang's internals into
our runner, and we already have a lot of that. I have left a TODO to try
and follow up with upstream about fixing this in a more sustainable way,
but wanted to get the testing in place anyways.
I've also threaded a flag through the various layers so that when we're
on the command line we can actually skip this and get the same compile
time benefits that Clang itself gets from disabling freeing all of the
internal data structures.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This switches most error printing to use diagnostics instead of direct
stream writes, even when not a specific file diagnostic. I'm allowing
empty filenames for this use-case.
This allows a little more specific testing to validate coverage of
output using the diagnostic coverage test. I'm adding a few tests to
cover things that weren't previously tested.
Separately, this also forces a little more standardization in format...
considering how changes like #4568 show effort being spent to _mirror_
diagnostic style, my thought is now to just use diagnostic code where
possible.
Note this also allows incrementally better testing of the language
server; I'm changing the crash fix from #4847 in favor of diagnostic
testing.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
The language server needs stdin, and for tests we should be passing it
around. My intent is to pass in a faux stdin to Driver for language
server tests.
As long as I'm adding a new parameter, I was looking at also changing
the way streams are passed in to Driver for style (pointers since
they're held past construction lifetime). Since these are all stored in
DriverEnv, I thought it might be a net improvement to use the struct
directly, getting more explicit parameter names and also removing the
need for `SetFuzzing`.
I'm trying here to avoid functional changes, but there are a couple
additional fixes like removing an obsolete `find_insensitive` and
refactoring how `ValidateOptions` handles errors (because it reduces the
number of spots that operate on error_stream).
As discussed in #4530 . This required switching to using
`llvm::IntrusiveRefCntPtr` in a number of places.
---------
Co-authored-by: Josh L <josh11b@users.noreply.github.com>
This teaches the driver library to track when its being used with
fuzzing and disables the `clang` subcommand from actually running Clang.
The Clang libraries have a large backlog of fuzzer-found issues that
isn't being actively reduced, so we can't productively fuzz into it.
This lets us more productively fuzz at the top level.
This is also available on the command line itself, which should be
useful if anyone wants to fuzz Carbon from the command line using tools
like AFL -- they can inject this flag to avoid getting noise from the
fuzzer hitting known issues in Clang.
This makes something like `bazel run :toolchain -- clang -- -c test.cpp`
work, because that can be run in-process. Note that `bazel run
:toolchain -- clang -- test.cpp` still requires subprocessing, and does
not work.
Note, the vision here is that we are trying to align how clang and
carbon compile c++ code. This is work towards intertwining command
execution.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Chandler Carruth
Thomas Köppe
josh11bandJosh L