Multiple subcommands all need the ability to disable on-demand runtime
building, and this may be needed outside of using _prebuilt_ runtimes.
For example, with Bazel the plan is to not build runtimes at all and
have Bazel provide them as native Bazel libraries.
Updates the `link` subcommand to respect this flag when running Clang to
perform links.
We didn't have any real testing of the `link` subcommand, in part
because it was difficult -- it would try to link runtime libraries. Now
that we can prevent building them on demand, we can use that to test the
link command. That in turn helped uncover a couple of bugs that are
fixed here.
1) The `driver_env_` member of the `Driver` was re-used across
`RunCommand` invocations. Some of its fields are constant across
these, others can be updated, and still more are not necessarily
something we would expect to be re-used. This fixes that by removing
the `driver_env_` member, and replacing it with members for just the
fields of `DriverEnv` that we want to set initially based on the
construction of the `Driver` object. This causes multiple, sequential
`RunCommand` calls to not clobber or erroneously inherit state.
2) The temporary directory support in the driver unittest didn't allow
the driver to observe the things it wrote to the temporary directory.
This PR updates the test logic to create an overlay VFS so that both
the in-memory test inputs are observed, but so are the real files
written into the temporary directory.
3) The Clang runner, when asked to run Clang without runtimes would
still attempt to include runtimes in any link command. This isn't
quite what we want, as the whole reason to use this without building
runtimes is to reuse ones built in some other way and potentially in
some other location. For now, this PR uses a hack to suppress these
issues so that we can have a basic test, but in the future we'll need
a better solution here.
4) The driver test didn't include the actual driver in the install data.
The test even worked around this, but it makes it impossible to link
reliably as the `lld` binary isn't available. This adds the data
dependency and updates the test to the available digest, etc.
This layer allows runtimes to be built on-demand but cached in a
consistent and re-usable location on the system. It handles careful
filesystem operations to ensure consistency even in the face of multiple
versions and build configurations.
This addresses a number of TODOs from the initial runtimes building
on-demand, and sets the stage to scale up to more runtimes.
This doesn't switch on-demand runtimes to be on by default, I wanted to
wait and make that change as a separate step.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
This is the first step to having Clang's runtime libraries fully
available for the Carbon toolchain. This PR focuses on the lowest level
runtimes, the CRT files and the builtins library.
The goal is to intercept Clang runs where it needs these
target-dependent pieces to be available, and build them on demand using
our Clang-running infrastructure. This avoids most of the subprocess
overhead, but there is still some due to missing features in Clang.
This requires exporting the sources for these runtimes from the Bazel
build, and installing them in our target-independent resource directory.
We then build a simplified "build" of these sources within the
`ClangRunner` itself to produce the specific artifacts and layout
expected by Clang.
It also required fixing our use of Clang on macOS to have a default
system root in order to successfully compile or link.
It also required cleaning up how the `ClangRunner` used target
information more generally -- instead of taking the target as
a constructor parameter, it manages its target internally and relies on
the Clang target-specifying command line flags.
I looked at whether we could split this into another layer separate from
the `ClangRunner`, but that proved frustratingly difficult to manage.
While we support building these on-demand as part of a detected link,
that doesn't seem feasible as we don't have the necessary separation
between compilation runs of Clang and link runs of Clang. However,
I have tried to factor the internals to provide as clear of separation
as I could across these.
I have also created a stand-alone subcommand to directly build the
runtimes which allows for easy testing. It also supports building them
into a specific directory, and that directory can in turn be passed to
a Clang invocation. This is designed to work both at the API level with
`ClangRunner` and at the subcommand level.
Currently, the only part of the commandline that is detected and
forwarded to the runtimes build is the target. Eventually, the plan is
to expand this so that we can build a maximally tailored set of runtimes
for a given compilation.
The other big TODO here is to actually implement caching storage of
these runtimes so they aren't built on every execution. Right now, this
uses a somewhat hack-y build of a temporary directory, but this isn't
expected to be suitable long-term. Building these runtimes on *every*
link makes those commands take approximately 15 seconds with an ASan
build like our default development build, and just over 2 seconds in an
optimized build. Because of this, I've kept all of this disabled by
default for now. The goal is that once caching and some other
improvements land, we can enable this by default.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This makes something like `bazel run :toolchain -- clang -- -c test.cpp`
work, because that can be run in-process. Note that `bazel run
:toolchain -- clang -- test.cpp` still requires subprocessing, and does
not work.
Note, the vision here is that we are trying to align how clang and
carbon compile c++ code. This is work towards intertwining command
execution.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Chandler Carruth
