This parallelizes the compilations and dramatically reduces the time to
build runtimes.
As part of this, teach the driver infrastructure to have an option to
control the use of threads and to build the relevant thread pool and
thread it into the various APIs.
However, it requires our `ClangRunner` to become thread-safe and to
invoke Clang in a way that is thread-safe. This is somewhat challenging
as the code in `clang_main` is distinctly _not_ thread-safe.
To address this, the relevant logic of `clang_main`, especially the CC1
execution, is extracted into our runner and cleaned up to be much more
appropriate in a multithreaded context. Much of this code should
eventually be factored back into Clang, but that will be a follow-up
patch to upstream.
Last but not least, this rearranges the `ClangRunner` API to make a bit
more sense out of the different options for building runtimes, and have
a clean model for which things need to be passed in at which points.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
This layer allows runtimes to be built on-demand but cached in a
consistent and re-usable location on the system. It handles careful
filesystem operations to ensure consistency even in the face of multiple
versions and build configurations.
This addresses a number of TODOs from the initial runtimes building
on-demand, and sets the stage to scale up to more runtimes.
This doesn't switch on-demand runtimes to be on by default, I wanted to
wait and make that change as a separate step.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
What this really does is avoids shadowing names, so that we can
comfortable have things like `Check::DiagnosticEmitter` or
`Check::DiagnosticLoc` without shadowing being a concern.
Note, down this path I'm also thinking about:
- Renaming misc DiagnosticConsumer/DiagnosticEmitter classes, possibly
just to DiagnosticConsumer/DiagnosticEmitter (so
`Check::DiagnosticEmitter` instead of `SemIRLocDiagnosticEmitter`).
- Dropping `Diagnostic` from `Emitter::DiagnosticBuilder`.
- But not for `Check::DiagnosticBuilder`, because `Check::Builder` would
be ambiguous.
- Renaming diagnostics/diagnostic_* to drop "diagnostic".
[Discussion about SemIRLoc ->
DiagnosticLoc](https://discord.com/channels/655572317891461132/655578254970716160/1353771570463768698)
reminded me of this (in particular the older [Check::DiagnosticBuilder
discussion](https://discord.com/channels/655572317891461132/655578254970716160/1344363562608627763)),
but I'd only do that rename if there's matching consensus about a path
forward where we keep SemIRLoc, and in a way that it's only ever used
for diagnostics (the divergence from which is at the root of current
LocId discussion).
I'm trying to keep that separate from a namespace addition for clarity.
Sadly, the Clang driver unconditionally injects the `-disable-free` flag
to CC1 invocations. =/ This ends up with us leaking memory when invoking
Clang programmatically, for example in unit tests.
I've fixed this by post-processing the flags. The alternatives I see all
involve duplicating even more code from within Clang's internals into
our runner, and we already have a lot of that. I have left a TODO to try
and follow up with upstream about fixing this in a more sustainable way,
but wanted to get the testing in place anyways.
I've also threaded a flag through the various layers so that when we're
on the command line we can actually skip this and get the same compile
time benefits that Clang itself gets from disabling freeing all of the
internal data structures.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This switches most error printing to use diagnostics instead of direct
stream writes, even when not a specific file diagnostic. I'm allowing
empty filenames for this use-case.
This allows a little more specific testing to validate coverage of
output using the diagnostic coverage test. I'm adding a few tests to
cover things that weren't previously tested.
Separately, this also forces a little more standardization in format...
considering how changes like #4568 show effort being spent to _mirror_
diagnostic style, my thought is now to just use diagnostic code where
possible.
Note this also allows incrementally better testing of the language
server; I'm changing the crash fix from #4847 in favor of diagnostic
testing.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
The language server needs stdin, and for tests we should be passing it
around. My intent is to pass in a faux stdin to Driver for language
server tests.
As long as I'm adding a new parameter, I was looking at also changing
the way streams are passed in to Driver for style (pointers since
they're held past construction lifetime). Since these are all stored in
DriverEnv, I thought it might be a net improvement to use the struct
directly, getting more explicit parameter names and also removing the
need for `SetFuzzing`.
I'm trying here to avoid functional changes, but there are a couple
additional fixes like removing an obsolete `find_insensitive` and
refactoring how `ValidateOptions` handles errors (because it reduces the
number of spots that operate on error_stream).
As discussed in #4530 . This required switching to using
`llvm::IntrusiveRefCntPtr` in a number of places.
---------
Co-authored-by: Josh L <josh11b@users.noreply.github.com>
This teaches the driver library to track when its being used with
fuzzing and disables the `clang` subcommand from actually running Clang.
The Clang libraries have a large backlog of fuzzer-found issues that
isn't being actively reduced, so we can't productively fuzz into it.
This lets us more productively fuzz at the top level.
This is also available on the command line itself, which should be
useful if anyone wants to fuzz Carbon from the command line using tools
like AFL -- they can inject this flag to avoid getting noise from the
fuzzer hitting known issues in Clang.
I want to split commands out so that we don't keep piling onto driver
(particularly as I'm eyeing clang-related commands). This extracts out
the DriverEnv so that it can be easily shared, with the CompilationUnit
as an example.
CARBON_VLOG_TO is to remove the vlog_stream_ requirement of CARBON_VLOG.
Chandler CarruthandDana Jansens
Jon Ross-Perkins
josh11bandJosh L