Instead of building one Clang `ASTContext` per compilation, the
`--share-cpp-ast` flag causes us to build a single `ASTContext` and
share it across all contexts. One new abstraction is added: `CppDomain`
represents the Carbon-side view of a Clang AST that might be shared
across multiple `SemIR::File`s. This object owns the Clang instance and
the AST.
For now, we have no isolation between the C++ state exposed to different
Carbon compilations, and we have no multiplexing of generated LLVM IR
from C++ into different Carbon compilations, so the mode is not usable
yet. The plan is to keep it behind a flag until it's ready.
Assisted-by: Gemini via Antigravity
Split the diagnostic emitter into a separate emitter (regietered with
Clang) and listener (registered with the emitter). The purpose of this
split is to make the Clang emitter not depend on the `Check::Context`,
so that we can use it, and hence the same Clang instance, with multiple
`Check::Context`s. A fallback listener is registered to collect and emit
any diagnostics produced while we don't have a `Check::Context`
registered with the emitter.
Assisted-by: Gemini via Antigravity
When a Carbon virtual function overrides a C++ virtual function, we need
to export it with the C++ signature in order for it to work as an
override. Instead of mapping the C++ signature into Carbon and then back
again, use the original C++ signature from the base class as the
signature exported to C++.
Also add documentation explaining how we use thunks in C++ interop,
including in this new virtual function handling logic.
This allows C++ to call Carbon functions with generic type parameters,
with some conditions. Example:
```carbon
interface I {
fn Doit(self);
}
class A {
impl as I { fn Doit(unused self) {} }
}
class B {
impl as I { fn Doit(unused self) {} }
}
fn F[T:! I](t: T) {
t.Doit();
}
inline Cpp '''
void G() {
Carbon::A a;
Carbon::B b;
Carbon::F(a);
Carbon::F(b);
}
''';
```
The initial support is limited; only explicit parameters are handled
currently.
`CarbonExternalASTSource::GetOrExportFunctionToCpp` now generates a
`clang::FunctionTemplateDecl` for generic Carbon functions. If C++ code
attempts to call that templated function,
`CarbonExternalASTSource::LoadExternalSpecializations` will be called
with the template argument types of that call site. Then we can generate
a specialized thunk for those argument types for C++ to call.
The function to set the visible declarations with a given name
overwrites any existing declarations imported from an AST file, so we
need to avoid calling that for declaration contexts whose names are
managed by Clang to avoid clobbering names imported from modules.
Assisted-by: Gemini via Antigravity
Copy MultiplexExternalSemaSource.h and MultiplexExternalSemaSource.cpp
from https://github.com/llvm/llvm-project/pull/204458 into
third_party/llvm, and apply a few minor changes to allow them to compile
and pass precommit checks.
This allows
`0011-Add-empty-constructor-and-GetSources-method-to-Multi.patch` to be
removed, which brings Carbon closer to being able to compile on an
unmodified LLVM toolchain.
In generate_ast.cpp, an `CarbonExternalASTSource` is installed that has
a `Check::Context` pointer. During lowering, this `ExternalASTSource` is
still installed, and using it can cause a crash if the now-invalid
pointer is dereferenced.
Fix by adding a new `ReadOnlyASTSource` in sem_ir, and using that during
lowering.
`CarbonExternalASTSource` now inherits from `ReadOnlyASTSource` to avoid
some code duplication.
In generate_ast.cpp, we now always install a multiplex source, even if
there's only one child source. Clang internally keeps pointers to the
top-level `ExternalASTSource` installed via `setExternalSource`, and
those pointers aren't updated if `setExternalSource` is called again. By
using `MultiplexExternalSemaSource`, we can keep the top-level
`ExternalASTSource` pointer the same, and only update its children.
Using `MultiplexExternalSemaSource` this way requires a new constructor
and a method to modify its child sources; added a new LLVM patch adding
those.
Originally landed in #7335, reverted in #7353 due to ASAN errors.
Changes since original:
* Use LLVM RTTI to make `Lower::Context::Finalize` less brittle.
Add LLVM RTTI to `ReadOnlyASTSource` (and `CarbonExternalASTSource`).
Change Finalize so that instead of just deleting the last multiplex
child source, it erases any multiplex child sources that match
`ReadOnlyASTSource`; this includes `CarbonExternalASTSource` since it's
a subclass.
* Fix ASAN error by updating the `MultiplexExternalSemaSource` earlier
in lowering. It is sometimes accessed during PrepareToLower, so update
it in `Context::GetFileContext` rather than `Context::Finalize`.
Fixes https://github.com/carbon-language/carbon-lang/issues/7142
In generate_ast.cpp, an `CarbonExternalASTSource` is installed that has
a `Check::Context` pointer. During lowering, this `ExternalASTSource` is
still installed, and using it can cause a crash if the now-invalid
pointer is dereferenced.
Fix by adding a new `ReadOnlyASTSource` in sem_ir, and using that during
lowering.
`CarbonExternalASTSource` now inherits from `ReadOnlyASTSource` to avoid
some code duplication.
In generate_ast.cpp, we now always install a multiplex source, even if
there's only one child source. Clang internally keeps pointers to the
top-level `ExternalASTSource` installed via `setExternalSource`, and
those pointers aren't updated if `setExternalSource` is called again. By
using `MultiplexExternalSemaSource`, we can keep the top-level
`ExternalASTSource` pointer the same, and only update its children.
Using `MultiplexExternalSemaSource` this way requires a new constructor
and a method to modify its child sources; added a new LLVM patch adding
those.
https://github.com/carbon-language/carbon-lang/issues/7142
This helps us move away from the clone-with-modifications approach to
thunking, which gets unwieldy as signatures get more complex.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Change `Lookup` by InstId to return a ClangDecl pointer. All callers
were immediately calling `Get` anyway, so this makes call sites a little
shorter. The other `Lookup` method, by ClangDeclKey, is sometimes called
without calling `Get`, so left that as-is, but renamed to `LookupId`.
Also add a `decl` method to ClangDecl so that the commonly repeated
`clang_decl->key.decl` can be written `clang_decl->decl()`.
This ensures the clang_decls map is used as a cache - without this,
visiting the same entity twice could cause it to be
re-exported/duplicated. See attached test case.
Replace all uses of CppGlobalVarStore store with ClangDeclStore.
Adding a VarStorage->VarDecl mapping to ClangDeclStore is now done with
the `AddVar` method, which takes an extra `pattern_id` arg. While the
corresponding `ClangDecl` is unchanged from before, the reverse mapping
in `inst_id_to_clang_decl_id_` now uses the `pattern_id` as the key.
This is necessary because in some places the original VarStorage
instructions gets replaced (e.g. by a call to `Convert`). The
`pattern_id` remains stable in those cases.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Removing the clang_decl_id on SemIR::Function - using only the
clang_decls map to create the association between SemIR::Function and
clang::FunctionDecls.
This adds an `is_external` flag to ClangDecl to indicate whether the
entity originated from Carbon or was imported from another language.
(I'm open to names - I guess for now we mostly use "is this from C++" to
be more specific than "is this external" - eg: NameScope::is_cpp_scope)
Add `ExportVarToCpp`. This checks the `clang_decls` mapping and returns
an existing decl if found. Otherwise, it creates a new `VarDecl` and
adds it to the `clang_decls` mapping.
When lowering, in `FileContext::BuildGlobalVariableDecl`, the
`clang_decls` mapping is used to lookup an existing
`llvm::GlobalVariable` for the instruction. If found, use that rather
than creating a new one to avoid an unwanted second definition in the
llvm IR.
Carbon-side thunks (for example the `Copy`/`Destroy` witness thunks
generated for imported C++ types) are mangled by Carbon, and their names
incorporate a fingerprint of the involved types. The instruction
fingerprinter identifies a class only by its name and parent scope,
which is sufficient for Carbon classes but not for imported C++ classes:
different specializations of one class template (and other cases such as
types in anonymous namespaces) share a Carbon name and parent scope. As
a result, the thunks for two distinct specializations could mangle to
the same name, producing a single LLVM function with two definitions and
failing `verifyModule` during lowering.
When fingerprinting a class imported from C++, also include the Clang
mangled name of its type.
Test: toolchain/lower/testdata/interop/cpp/thunks.carbon gains a split
with two specializations of one class template, each requiring a thunk;
their thunks now get distinct mangled names instead of colliding.
Assisted-by: Claude Code
---------
Co-authored-by: Christopher Di Bella <cjdb.ns@gmail.com>
Flush diagnostics before destroying Clang. If we see an `inline Cpp` and
`Cpp` initialization failed, recover by skipping the inline code rather
than CHECK-failing.
This allows Clang to correctly generate the vtable for the exported
class.
There's still something wrong with new virtual functions in the Carbon
type (left a TODO) - I thought it might be related to not flagging
the CXXMethodDecl as virtual, but my initial experiments don't seem to
back that up, so I'll look into it further separately.
There's also a test regression due to an virtual (well, abstract
specifically, but I think it'd happen with a virtual one too) function
in an abstract class taking `self` by value being rejected since
the abstract class can't be instantiated. Not sure if this is a correct
change - the test's behavior could be preserved by using `ref self`
instnead of `self` in this function. Is that reasonable/expected? Should
we not require a type to be complete when passing by value if we can
compute the value representation without such completeness?
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
* For Carbon `base class C`, export as a regular C++ class.
* For Carbon `class C`, export with the C++ `final` keyword attribute.
* For Carbon `abstract C`, mark the destructor as pure virtual in cases
where no member function is abstract, or emit an error if the destructor
is not virtual.
To support the final point, mark the destructor of an exported class as
virtual if it overrides a virtual destructor from the base class.
In passing, fix a crash exporting fields if the class has an invalid
base type.
When an outer type defines an `extend` relationship to an inner type, we
require that inner type to be complete so that we can know that name
lookup can search both scopes as soon as the outer type is complete.
When doing name lookup, we require the type in which we are looking to
be complete. Then, we recursively add extended scopes, but then also
require each of them to be complete again, which inserts
RequireCompleteType instructions into the block doing lookup.
While these new instructions may differ in terms of their specifics,
they are redundant since we already required the type to be complete,
and specifics can not change the completeness of a type. They are also
problematic because a named constraint or interface can extend a scope
with a symbolic specific, by using `Self` as an argument. This inserts a
symbolic instruction into the block doing name lookup, even though that
block may not be generic.
A destructor is added to the C++ class definition in
`CarbonExternalASTSource::CompleteType`. The destructor calls a Carbon
function that calls the `Destroy` operator.
When any field of a Carbon class is access from C++ for the first time,
all fields are exported as `clang::FieldDecl`s (this is necessary
because clang fields have an internal index that is initialized on first
use).
`ClangDeclStore` now provides bidirectional mapping. This allows looking
up a `ClangDeclId` by `InstId`, so when Carbon class fields are exported
they can be looked up that way.
Allow any type that has a mapping from Carbon to C++ to be exposed to
C++ via name lookup. This also exposes the logic to export Carbon
classes to C++ to apply during type mapping, which gives very slight
support for passing Carbon types to C++ functions from Carbon, but not
really enough to sensibly test yet.
Depends on #7042.
Instead of exporting a class or namespace each time a new C++ name
lookup discovers it, track that we have exported the entity on its name
scope, and if a new name lookup finds the same entity, produce the same
clang declaration.
We don't yet populate the bases or fields, so the class types show up as
empty classes in C++ for now. But we do allow calls to static member
functions.
This works by generating two thunks, one in C++ and one in Carbon. For
example, given this input:
```c++
// Carbon:
fn Callme(f: f32) {}
// C++:
void F() {
// This will call `Callme__cpp_thunk`
Carbon::Callme(1.0);
}
```
These functions are generated:
```c++
// Carbon:
fn Callme__carbon_thunk(ref f: f32) {
// Call the target function.
Callme(f);
}
// C++:
// C++ declaration for the Carbon thunk.
void Callme__carbon_thunk(float& f);
void Callme__cpp_thunk(float f) {
// Call the Carbon thunk with args passed by reference.
Callme__carbon_thunk(f);
}
```
For now, all arguments are passed by reference, even if they are simple
types like pointers or i32.
Functions with non-void return types are not supported yet.
Use the same C++ -> Carbon map for both interop directions, and when
importing an entity from Carbon -> C++, check whether it was originally
a C++ entity and if so return the original.
Assisted-by: Gemini via Google Antigravity
For #6830, add support for inline C++ fragments as a declaration rather
than as a packaging directive. For now, this uses `inline Cpp
<string-literal>;` as syntax. The prior `import Cpp inline
<string-literal>;` is left alone for the time being. We can decide
separately whether to remove that.
`inline Cpp` requires that there was at least one `import Cpp`. It's not
clear to me if that's the right design long-term, but it seems
reasonable for now.
Assisted-by: Gemini via Google Antigravity
Since this requires using the `Mangler` class from `toolchain/check`,
moved it from `toolchain/lower` to `toolchain/sem_ir`.
The mangled name is then attached to the `FunctionDecl` with an
`AsmLabelAttr`.
Start recording the clang::DeclContext* -> InstId mapping for use in
later operations.
The test update includes removing the initial fail_* test because I
hadn't thought about the use of namespace aliases as a way to test for
the presence of a namespace without the failure caused by not finding
the thing inside the namespace.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Instead of injecting code to declare an `operator new`, generate AST for
it directly. In order to use this, directly generate a `CXXNewExpr`
rather than asking Clang to build one.
This is less of a hack, and doesn't visibly leak an `operator new`
declaration that inline C++ code or template instantiations might see.
It also avoids generating a warning in C++26 and later that the
`constexpr` declaration of `operator new` is used but not defined.
Assisted-by: Gemini 3.1 Pro via Antigravity
Add a clang::ExternalASTSource to begin exposing Carbon entities to
Clang - initially only a single `Carbon` top level namespace.
Subsequent work will add Carbon entities to this namespace.
Likely this CarbonExternalASTSource will be refactored into another
file, tie into/reference SemIR::File and CppFile, etc eventually - but
that'll wait for future patches.
If there's mechanical problems with the current implementation - how I'm
creating the new NamespaceDecl, etc - I'm all ears. It's very much in
the "it seems to work" state, not much more than that.
This does break Clang Modules (header modules, C++20 modules,
precompiled headers, etc) since they're implemented as an
ExternalASTSource as well, and Clang's ASTContext only supports one
ExternalASTSource at a time. To fix that regression we'll need to
implement some kind of ExternalASTSource multiplexing support - either
in Clang or Carbon (unclear which).
This regression of modules support can be observed by the following:
`A.h`
```
inline void f1() { }
```
`module.modulemap`
```
module A {
header "A.h"
export *
}
```
`test.carbon`
```
import Cpp inline '''
// Hardcode the pragma to ensure this isn't silently falling back to
// textual inclusion.
void f2() {
f1();
}
''';
```
```
carbon compile test.carbon -- -I . -fmodules -fimplicit-modules -fmodules-cache-path=module_cache
```
I wrote a `file_test` test for this, but it doesn't /quite/ work because
`file_test` provides an in-memory filesystem for tests to make them more
hermetic, but Clang's Filesystem abstrtaction is for reading only - so
the module that's written out successfully can't be found when it needs
to be read back in - so the test doesn't pass as a baseline. Clang does
have support for `llvm::vfs::OutputBackend` which allows virtualizing
output - which I guess we could tie together with the InMemoryFilesystem
we use for input to make such a test work. But I guess that's not worth
the effort here?
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Defer creating the CppContext until we have all of its components, so
that we know they're not null. Don't track the action on the context,
since it's not a reliable way of getting back to the compiler invocation
on failure. Don't flush the diagnostics emitter from the emitter
destructor since the derived class emitter will already have been
destroyed at that point. Distinguish between clang setup failing and
clang merely producing errors, and don't connect the check context to
clang if clang setup failed.
---------
Co-authored-by: David Blaikie <dblaikie@gmail.com>
Mainly because "sorting_diagnostic_consumer" is legacy, since
`SortingDiagnosticConsumer` became `SortingConsumer`. Also better
reflecting contents of these files.
Where I'm not renaming, I'm less positive about dropping "diagnostics"
from "file_diagnostics" and "null_diagnostics" (which contain both a
consumer and emitter, and "null.h" seems like poor naming), so not doing
that here. Also "diagnostic.h" contains `struct Diagnostic`, so is a
decent fit.
Assisted-by: Google Antigravity with Gemini 3 Flash
Some module metadata changed - because rather than linking one module
with one module metadata value (eg: PIC Level 0, or unspecified) and one
module with a different one (PIC level 2, in clang) - we use Clang's
Module as-is, no merging required, so Clang's module metadata sticks
rather than being merged with default values from Carbon.
Also tweaked the name we use for Clang's module name so it matches the
carbon file name.
Otherwise the IR changes seem to be just reorderings - C++ interop goes
first, then Carbon, rather than the other way around.
Background:
https://docs.google.com/document/d/1wi85FRiWh4X9A-gCYMVGKR40-q5fM6-3JaSpePk-XCY/edit?usp=sharing
And specifically this work is essentially an alternative to #5543
Clang's code generation is implemented through an ASTListener
(clang::CodeGenerator) that is attached throughout Clang's
parsing/sema/code
generation phases and acts on Clang AST incrementally throughout that
process.
Prior to this patch, Carbon has only created the CodeGenerator during
Carbon's
`lower` phase, missing out on key callbacks that would be made by Clang
during
`check`. Some of these issues were addressed by #6237 and #6483 - but
there were
still remaining cases where the delayed processing lead to missing
functionality.
With #6483 much of the Clang code that made multithreaded complexity of
#5543 is
no longer present, and we have access to the point of ASTListener
registration
so we can register the CodeGenerator there and consume its resulting
llvm::Module during lower.
Examples of some of the bugs this addresses are seen in the linked doc,
and
checked in as tests in this change in
`clang_code_generator_callbacks.carbon`
An indicental bug that's also fixed, and caused all the other test case
churn,
is that the `CodeGenerator` created during `lower` wasn't getting passed
the
Clang `CodeGenOpts` and was creating its own default - so, most notably,
optimization flags were not respected. This meant that the LLVM IR from
Clang
was always -O0 style IR (optnone, no inlinehint, no TBAA, etc). With
this
change, now the Clang IRGen gets the real `CodeGenOpts` and respects
optimization/other flags specified there.
This is only meant to be a rough proof of concept - I'm totally open to
reworking this in any way (even quite substantially) if folks have ideas
about
how this should be implemented most generally/elegantly/etc.
Instead of parsing a complete C++ translation unit and then interacting
with the translation unit further after the fact, delay finishing the
translation unit until we finish the Carbon check phase. This fixes some
issues where we would produce duplicated or incorrect diagnostics at the
end of the C++ translation unit, particularly for unused declarations.
Now we're in control of how we parse the translation unit, also disable
parsing of C++20 modules if the syntax appears within `import Cpp
inline` code.
Keep the same clang parser alive throughout check, and use it instead of
building a new one when parsing macros. This resolves issues where the
translation unit scope was destroyed too early, resulting in unqualified
lookup within macros being unable to find global scope entities.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
* Move `Sema` access from `CppFile` into `CppContext`.
* Move the mangle context from `SemIR::File` into `CppContext`.
* Move source location mapping state from `Context` into `CppContext`.
Also factor out the `GenerateAst` function that builds the `CppContext`
and `CppFile` into its own file.
Richard Smith
Geoff RomerandNicholas Bishop
Nicholas Bishop
David Blaikie
Chandler CarruthandChristopher Di Bella
Dana Jansens
