This is the first draft of the implementation of
[p6333](https://docs.carbon-lang.dev/proposals/p6333.html).
The `build` subcommand shared logic with the `compile` and `link`
subcommands,
so I've moved some of the functionality in `compile` and `link` to
shared
`CompileDriver` and `LinkDriver` classes, respectively. This also
required exposing the
`CompileOptions` and `LinkOptions` subcommand structs for re-use.
There's still some work to do on the proposal, most notably the package
include automatic path resolution and import, and the refactors to
`carbon compile`.
Add an optional additional set of positional parameters that can be
passed to the `link` subcommand for Clang-style (or GCC-style)
`LDFLAGS`. These can _also_ contain object files, etc., and in fact it
is useful to allow them to contain object files in order to integrate
the `carbon link` subcommand into a build system that mixes both link
flags and object files. This at least happens with Bazel, and I suspect
is common.
Eventually, it would be nice to have sufficient semantics to handle all
the varieties of links we want without resorting to this escape hatch,
but that's likely a long way away and so it seems especially useful to
allow falling back to Clang's flags as needed for now.
This does somewhat directly surface the Clang implementation detail in
the command line syntax, but I don't see a lot of good alternatives.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Multiple subcommands all need the ability to disable on-demand runtime
building, and this may be needed outside of using _prebuilt_ runtimes.
For example, with Bazel the plan is to not build runtimes at all and
have Bazel provide them as native Bazel libraries.
Updates the `link` subcommand to respect this flag when running Clang to
perform links.
We didn't have any real testing of the `link` subcommand, in part
because it was difficult -- it would try to link runtime libraries. Now
that we can prevent building them on demand, we can use that to test the
link command. That in turn helped uncover a couple of bugs that are
fixed here.
1) The `driver_env_` member of the `Driver` was re-used across
`RunCommand` invocations. Some of its fields are constant across
these, others can be updated, and still more are not necessarily
something we would expect to be re-used. This fixes that by removing
the `driver_env_` member, and replacing it with members for just the
fields of `DriverEnv` that we want to set initially based on the
construction of the `Driver` object. This causes multiple, sequential
`RunCommand` calls to not clobber or erroneously inherit state.
2) The temporary directory support in the driver unittest didn't allow
the driver to observe the things it wrote to the temporary directory.
This PR updates the test logic to create an overlay VFS so that both
the in-memory test inputs are observed, but so are the real files
written into the temporary directory.
3) The Clang runner, when asked to run Clang without runtimes would
still attempt to include runtimes in any link command. This isn't
quite what we want, as the whole reason to use this without building
runtimes is to reuse ones built in some other way and potentially in
some other location. For now, this PR uses a hack to suppress these
issues so that we can have a basic test, but in the future we'll need
a better solution here.
4) The driver test didn't include the actual driver in the install data.
The test even worked around this, but it makes it impossible to link
reliably as the `lld` binary isn't available. This adds the data
dependency and updates the test to the available digest, etc.
This enables on-demand building of runtimes by default, and enables
their header files for all of the Clang invocations. This also switches
the default flags to use the LLVM-provided runtimes (compiler-rt,
libunwind, and libcxx).
This also switches even `llvm_symlinks_test` to use the Bazel prebuilt
runtimes, which requires having a way to pass a Carbon flag even when
invoking the busybox as `clang` or `clang++`. This uses the pattern that
has worked for other Clang wrappers of spelling flags:
`-X<tool-name>=--flag=value`
Last but not least, this updates the Carbon Bazel rules to use our
installed and the Bazel prebuilt runtimes. With that, we make the C++
interop hello-world be enabled by default as this should pass reliably
on both Linux and macOS now.
This unifies the default Clang arguments between the `clang` subcommand,
the `link` subcommand, and the `ClangInvocation` built for C++ interop.
This sets the stage to integrate either pre-built or on-demand runtimes
flags for both of these. However, this PR should have very little
practical difference. The biggest functional change is wrapping the
default arguments in flags to allow unused flags so that we can build a
collection of flags viable across compile and link.
This parallelizes the compilations and dramatically reduces the time to
build runtimes.
As part of this, teach the driver infrastructure to have an option to
control the use of threads and to build the relevant thread pool and
thread it into the various APIs.
However, it requires our `ClangRunner` to become thread-safe and to
invoke Clang in a way that is thread-safe. This is somewhat challenging
as the code in `clang_main` is distinctly _not_ thread-safe.
To address this, the relevant logic of `clang_main`, especially the CC1
execution, is extracted into our runner and cleaned up to be much more
appropriate in a multithreaded context. Much of this code should
eventually be factored back into Clang, but that will be a follow-up
patch to upstream.
Last but not least, this rearranges the `ClangRunner` API to make a bit
more sense out of the different options for building runtimes, and have
a clean model for which things need to be passed in at which points.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
This layer allows runtimes to be built on-demand but cached in a
consistent and re-usable location on the system. It handles careful
filesystem operations to ensure consistency even in the face of multiple
versions and build configurations.
This addresses a number of TODOs from the initial runtimes building
on-demand, and sets the stage to scale up to more runtimes.
This doesn't switch on-demand runtimes to be on by default, I wanted to
wait and make that change as a separate step.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
This is the first step to having Clang's runtime libraries fully
available for the Carbon toolchain. This PR focuses on the lowest level
runtimes, the CRT files and the builtins library.
The goal is to intercept Clang runs where it needs these
target-dependent pieces to be available, and build them on demand using
our Clang-running infrastructure. This avoids most of the subprocess
overhead, but there is still some due to missing features in Clang.
This requires exporting the sources for these runtimes from the Bazel
build, and installing them in our target-independent resource directory.
We then build a simplified "build" of these sources within the
`ClangRunner` itself to produce the specific artifacts and layout
expected by Clang.
It also required fixing our use of Clang on macOS to have a default
system root in order to successfully compile or link.
It also required cleaning up how the `ClangRunner` used target
information more generally -- instead of taking the target as
a constructor parameter, it manages its target internally and relies on
the Clang target-specifying command line flags.
I looked at whether we could split this into another layer separate from
the `ClangRunner`, but that proved frustratingly difficult to manage.
While we support building these on-demand as part of a detected link,
that doesn't seem feasible as we don't have the necessary separation
between compilation runs of Clang and link runs of Clang. However,
I have tried to factor the internals to provide as clear of separation
as I could across these.
I have also created a stand-alone subcommand to directly build the
runtimes which allows for easy testing. It also supports building them
into a specific directory, and that directory can in turn be passed to
a Clang invocation. This is designed to work both at the API level with
`ClangRunner` and at the subcommand level.
Currently, the only part of the commandline that is detected and
forwarded to the runtimes build is the target. Eventually, the plan is
to expand this so that we can build a maximally tailored set of runtimes
for a given compilation.
The other big TODO here is to actually implement caching storage of
these runtimes so they aren't built on every execution. Right now, this
uses a somewhat hack-y build of a temporary directory, but this isn't
expected to be suitable long-term. Building these runtimes on *every*
link makes those commands take approximately 15 seconds with an ASan
build like our default development build, and just over 2 seconds in an
optimized build. Because of this, I've kept all of this disabled by
default for now. The goal is that once caching and some other
improvements land, we can enable this by default.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
As discussed in #4530 . This required switching to using
`llvm::IntrusiveRefCntPtr` in a number of places.
---------
Co-authored-by: Josh L <josh11b@users.noreply.github.com>
Note the purpose here is to make it simpler to add more subcommands,
without adding a lot of things to Driver.
This creates a copy of CodegenOptions, but it was double-registered at
present which felt odd. It's also fairly small right now. If this
becomes an issue, maybe we can look into using optional for delayed
initialization, or just go back to straight sharing.
---------
Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com>
I'm separating the options out so that it's easier to review. They
include a lot of boilerplate text that I think won't change much, and
makes it harder to review changes.
To explain filename differences, whereas `CodegenOptions` is shared (by
link and compile), `LinkOptions` and `CompileOptions` are
subcommand-specific. I'm planning to separate out the subcommands, so
I'm putting those in respective subcommand files. I'm still going to try
to use the `.h` to declare the interface, `.cpp` for bigger
implementation details (for better or worse, including comments on
options).
I'm also moving out corresponding Driver members to help shrink deltas
when refactoring. That is, the bodies aren't changing here, but a
refactoring of commands will make some changes. By moving the code to
different files now, it should be easier to identify what's changing
later.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
Lucile Rose Nihlen
Richard Smith
Chandler CarruthandJon Ross-Perkins
josh11bandJosh L