This removes a bunch of manual filesystem helpers and complexity that
are directly provided by the new library.
It also moves all of the install paths detection to use
`std::filesystem::path` instead of the LLVM path library. The goal is to
consolidate all our logic onto a single stack, and the standard one
seems the best for that purpose. This does give up some of the
optimizations of this code to avoid memory allocation, but in practice
that likely isn't a critical issue. And with the new filesystem library
we can likely do more to avoid that by using directory-object-relative
filesystem access. However, that will have to wait for moving more parts
of the toolchain over to use this set of filesystem abstractions. There
is a related TODO left in the manifest handling code.
Use it to dump the AST that includes a generated C++ thunk.
Based on #5917.
Also added printing of the full actual text when check fails to make
debugging easier.
Changed line replacement to allow removing complete lines.
Part of #5514.
toolchain/check/testdata/builtins/char/basics.carbon and
toolchain/lower/testdata/builtins/char.carbon are probably the most
interesting tests here. The parse tests is required because this adds a
new node kind, and we need coverage of it; but the attached info is
minor. There's a fair amount of test churn here because I'm adding the
Core.Char and Core.CharLiteral types as new singletons.
My intent here is that `CharId` is always a unicode code point, even
when the type is a `Char` and thus must be a single UTF-8 code unit
(single byte). This mainly means the stored value of a `CharValue` can
be printed internally without knowing the type.
---------
Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Although this focused on `Destroy` support, some choices here around
`implicit_type_impls` are because copy/move will likely follow a similar
approach. I'm trying not to predict too much about how we'll structure
those, but I'm putting `Destroy` impl logic in a file that could perhaps
be shared with those. They'd likely be interested in similar things,
e.g. traversing members of types (particularly class, struct literal,
tuple literal).
At present this sets the destroy function as `no_op` which is consistent
with current logic, but has a TODO to correctly define.
Constant importing for functions changes slightly due to some issues I
was having with `GetFunctionType`. zygoloid suggested this approach to
avoid `EvalInst` logic.
Adds a flag for controlling whether to generating these impls. While
this does generation for `class`, as noted above this'll also need to be
done for tuples and struct literals, which would leave the `none.carbon`
min_prelude unable to use any types. Note if destruction *would* occur,
it'll still look up `Core.Destroy` for that and fail, but that's already
true of any test using `none.carbon`. I'm trying to use the flag to see
if we can keep `none.carbon` working mostly-consistently.
I'd tried separating out the flag to #5852, but that got a lot of
pushback over whether the behavior was appropriate. I'm hoping that the
interactions here make it clearer why the particular approach -- the
goal is not to enable advanced testing, or create some new end-user
behavior that we really support, it's just to keep no-prelude tests
functional. The main question raised there was why not just keep
generating `impl T as Core.Destroy` if `fn destroy` is present -- but I
think here it should be apparent that would require additional
complexity, as the generation of `impl T as Core.Destroy` is not
currently conditioned based on the implementation of `fn destroy`. I'd
rather add complexity to this flag only if it's enabling interesting
test functionality.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
This is trying to make it clearer when vectors are being indexed with
`CheckIRId`.
The only one that I still kind of want to change is the
`SmallVector<std::unique_ptr<CompilationUnit>>`, but because it's a
`unique_ptr` that's a little more complex. I may not bother.
Note, some of the changes around nuanced `SmallVector` interactions were
based on trying to copy the way `SmallVector` itself takes arguments,
like with range passing.
This changes ConstantValueStore to use ValueStore so that we get the
allocation flow that we're leaning towards there. This adds
`ConstantId::SymbolicId` to represent what was previously an int32
"symbolic_index" (although called an index, it's consistent with ID
usages).
I'm also changing Chunk::at/push to be consistent with the wrapping
Get/Add naming; the naming difference sticks out more with
UninitializedFill being added.
Stop using the clang tooling library to build an ASTUnit; that library
is set up to process clang frontend arguments, assuming that something
has already built frontend arguments from the compiler arguments. It is
also too encapsulated and doesn't let us inspect and modify the compiler
invocation before it's executed.
Instead, build the AST unit directly in two phases:
* FIrst, take a list of clang driver arguments and convert them into a
list of compiler arguments, using `clang::createInvocation`. Internally,
this uses the clang driver to build a frontend invocation, including
building system-specific include paths as needed.
* Then, directly build an ASTUnit from this compiler invocation.
I've factored this so that we can split out the `createInvocation` step,
with the intention that we may want to move it out of check and into the
carbon driver with the rest of the driver-level argument handling, and
we may want to customize some of the clang options before we invoke the
clang frontend with that set of options.
In order to make the invocation reusable, it no longer depends on the
name of the carbon file importing the C++ code. In place of synthesizing
a header file name as `<foo.carbon>.generated.cpp_imports.h`, we now
insert line marker directives into the generated header so that errors
in that header cause Clang to point a diagnostic back at the Carbon
source file itself. This results in a minor improvement in the
diagnostic output: we no longer refer to a nonexistent generated file.
But the snippet still contains text that doesn't match the source code,
so it remains imperfect.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
By moving dumping, we can have dumping occur before verification that
might CHECK-fail (e.g. parse tree and llvm IR verification).
I'm dropping vlogging of raw semir. It was only done when dumping, so
`-v` would print zero copies and `-v --dump-raw-sem-ir` would print two
copies. The lack of complaints about this suggests it's not needed.
I'm making a small change to drop newlines between textual and raw
semir. This is an edge case so I don't expect people to really notice in
general, but it seemed unusually aware of what's on a stream, and it
made it harder to do the dump_stream/raw_dump_stream approach, which I
felt would be decent in general, since check is the only phase that can
emit two different things (which I could also just drop -- we don't
really use raw semir anymore, it doesn't seem like a big need to be able
to print it with textual semir, but I'm assuming to just maintain
existing behavior).
In parse, we were previously dumping the tree on verification errors.
I'm removing that because now `--dump-parse-tree` should work fine,
where previously it wouldn't.
I've been mulling this mainly for the parameter complexity of
check/lower, but doing lex/parse for symmetry.
I'm motivated by the plan to move dumping for all of them into the
respective functions, because of discussion about llvm-verifier. That
basically would add another bool parameter (or more) to each of these.
My instinct is we're going to probably accrue a little more over time,
so I'm suggesting this as maybe adding the boundary a little simpler
and/or easier to read.
Note it may make sense to refactor a little further, e.g. maybe
Lower::Context could receive the full set of options and pick out what
it wants, but I figured creating the struct itself would be a decent
start.
I'm trying to put things into options when we can produce a reasonable
default if the user doesn't assign a value. I'm using an explicit
constructor so that values can be added without affecting every caller.
A different factoring would be to pass in everything through the param
struct, but that just felt weird when I was trying it out.
Removing `inst_namer` and `module_name` from `LowerToLLVM` params --
both of these can be inferred from `sem_ir`, and I'm not seeing a
particular reason to maintain them at the call site.
This changes `Destroy` to use an interface for its implementation.
Note that this change includes a lot of test updates. Even when
`Destroy` is a no-op, it still causes code generation as part of
determining that.
Originally I was trying to use ranges to cut down the scope of this, and
to a degree I think they have. But a flipside here is that cases where
no destructors should be generated -- particularly globals -- would be
needed to completely remove destructor calls. Even for ranges, the range
can often include the destructor placement. So I've shifted
frame-of-thought a little: accept a bunch of destructor churn, because
destructors are needed and will be prevalent. The verbosity is a feature
of the design to make desugaring apparent in IR, not a bug.
Suggested by zygoloid while looking at #5678
```
CHECK failure at toolchain/lower/context.cpp:62: !llvm::verifyModule(*llvm_module_, &errs): Verifier errors: Instruction does not dominate all uses!
%.loc17_46.1.temp = alloca { i1, i32, i32 }, align 8, !dbg !13
%tuple.elem0.loc17_46.2.tuple.elem = getelementptr inbounds nuw { i1, i32, i32 }, ptr %.loc17_46.1.temp, i32 0, i32 0, !dbg !13
Instruction does not dominate all uses!
%.loc17_46.1.temp = alloca { i1, i32, i32 }, align 8, !dbg !13
%tuple.elem1.loc17_46.2.tuple.elem = getelementptr inbounds nuw { i1, i32, i32 }, ptr %.loc17_46.1.temp, i32 0, i32 1, !dbg !13
Instruction does not dominate all uses!
%.loc17_46.1.temp = alloca { i1, i32, i32 }, align 8, !dbg !13
%tuple.elem2.loc17_46.2.tuple.elem = getelementptr inbounds nuw { i1, i32, i32 }, ptr %.loc17_46.1.temp, i32 0, i32 2, !dbg !13
```
Adds a `--llvm-verifier` flag to be able to turn this off easily,
particularly for debugging the LLVM IR.
The call workaround is due to a verifier requirement `inlinable function
call in a function with debug info must have a !dbg location`. It
specifically comes up for the `++x` case, with `%1 = call i32
@"_CConvert.8b3d5d6a6c17be04:ImplicitAs.Core.b88d1103f417c6d4"(i32
%other)`. I think #5397 is in the direction of a fix for that, but #5397
was set aside because it puts the debug info in too many places.
Instead, address this by adding a stub location for calls that don't
have a good location. I'm deliberately putting this next to the TODO so
that it's easier to understand the association.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
The `full.carbon` prelude just sets a flag indicating an explicit intent
to include the full prelude. Once all tests include some prelude file,
an error can be enabled (currently it's commented out) that requires an
`INCLUDE-FILE` of some min-prelude to be present in all `check/` and
`lower/` file tests.
Factor out the logic for mapping from a `LocId` into a diagnostic
location from check into sem_ir so it can be reused by lowering. Include
the function and instruction being lowered in the pretty stack trace.
Example stack trace:
```carbon
2. filename: examples/sieve.carbon
3. core/prelude/types/int.carbon:213:3: lowering function Core.Op(Core.IntLiteral as Core.ImplicitAs(i32))
fn Op[addr self: Self*](other: Self) = "int.sadd_assign";
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4. core/prelude/operators/arithmetic.carbon:22:27: lowering call
fn Op[addr self: Self*](other: Other);
^~~~~~~~~~~~
```
As we've been discussing stack trace behavior, I was thinking having the
version in crashes would be helpful. e.g.:
```
1. Carbon version: 0.0.0-0.dev+bdcef04bb.dirty
```
To make it easier to identify crashing files when testing multiple.
```
(elided)
3. Check::Context
filename: duplicate_name_same_line.carbon
NodeStack:
(elided)
```
I've been mulling the name of this, changing it and updating comments to
try and better reflect the current semantic. "Imports" reflects how
we're currently printing this in SemIR.
Adds minimal defaults for codegen so that those tests don't need to
specify the full command line.
Moves driver/testdata/compile to check/testdata/basics/raw_sem_ir, which
seems like it better reflects the focus. Removes the textual IR test
because we have plenty of those now.
Moves a multiline token diagnostic test to parse because (a) diagnostics
doesn't have other tests and (b) this is really testing the way that
parse structures the location.
Drops `--include-diagnostic-kind` from some driver tests that aren't
testing a diagnostic, so the flag felt a bit like noise (even before
this change, there are a few tests that don't specify it because they're
not intended to test an emitted diagnostic, just high-level diagnostic
behaviors).
Adds some comments to reflect my understanding of why a test exists,
when I was pausing to think about it.
Modifies the stdin test to do more dumps; which happens to expose we
currently misbehave.
In preparation for lowering information from multiple `SemIR::File`s
into a single `llvm::Module`. The primary purpose of this is to support
lowering a local specific for an imported generic function, where the
instructions for the generic function are in a different file than the
instructions for the specific. See #5475 for a draft PR implementing
that functionality on top of this.
The per-`llvm::Module` state now lives in `Lower::Context`, and
`Lower::FileContext` tracks only the per-`SemIR::File` information.
`Lower::Context` should not mention any `SemIR` IDs that are
file-specific. For now, the C++ lowering and the specific coalescing
logic are kept per-file for simplicity.
This requires:
* Making `FunctionDecl` mutable since generating code
(`HandleTopLevelDecl()`) requires a mutable declaration and since we
manually add `used` attribute to force code generation.
* Passing the file system to `Lower` since it's needed by Clang code
generation.
* Creating an internal Clang LLVM module and link it against the Carbon
LLVM module.
Demo:
```c++
// hello_world.h
extern int puts;
inline void hello_world() {
((int (*)(const char*))&puts)("hello world");
}
```
```carbon
// main.carbon
library "Main";
import Cpp library "hello_world.h";
fn Run() -> i32 {
Cpp.hello_world();
return 0;
}
```
```shell
$ bazel-bin/toolchain/carbon compile main.carbon
$ bazel-bin/toolchain/carbon link main.o --output=demo
$ ./demo
hello world
```
Based on https://github.com/carbon-language/carbon-lang/pull/5406.
Part of #5405.
Right now, a lot of tests have started setting `--no-dump-sem-ir`. My
thought is that we can look at:
1. Put ranges in a bunch more files.
2. Shift more towards `--dump-sem-ir-ranges=only` instead of
`--no-dump-sem-ir`, because it allows mixing fail-tests with no IR
alongside tests that contain IR.
3. Evaluate switching the default to `--dump-sem-ir-ranges=only`, and
instead set `--dump-sem-ir-ranges=if-present` only in files that want to
typically show all IR (particularly import-related tests, where ranges
don't work well).
In real-world use, my thought is also that it'd be helpful to be able to
add the dump range comments to files, see the output (i.e., the default
behavior of `if-present`) but then also be able to pass `ignore` in
order to see the full IR without modifying the file (possibly also
useful in tests). That model is why I went for tri-state handling.
Note `only` can also have an interesting side-effect. Because core files
(including min_prelude versions) typically won't have ranges, they'd be
implicitly excluded.
- Updates incompatible flags.
- `rules_flex` is no longer used, so enable its flag.
- Fixes `sh_test` deps for
`--incompatible_disable_autoloads_in_main_repo`
- Broadens the exception for `rules_cc` and `bazel_tools` due to changes
to runfiles deps; trying to avoid minutiae that shouldn't affect the
decision.
Instead of crashing when run outside of `bazel`, make the toolchain's
`file_test` binary work properly when no test-specific environment
variables are set. This makes it a lot easier to run `file_test` under a
debugger.
There are two main changes here:
- Don't crash if `$TEST_TMPDIR` is unset. Instead, fall back to LLVM's
temporary directory (typically `$TMPDIR`). We already did this in some
places in tests. We now do it in more places.
- Don't fall back to a target label of `<target>` in the reproduction
commands if `$TEST_TARGET` is unset, because this causes all the tests
to fail because their output doesn't match the expected output due to a
differing bazel run command. Instead explicitly specify the target from
the `FileTestBase`-derived class.
Infrastructure for this has been added generally, but only rolled out to
the toolchain `file_test` binary for now.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Right now we construct `tree_and_subtrees_getters` a couple different
ways, it's just not obvious because one's abstracted in `check`. But
also, when formatting IR, we'll repeatedly do the `IncludeInDumps`
string check, which felt odd to me since it only needs to be calculated
once per IR.
This also shifts `CheckIRId` selection a little earlier, and in doing so
makes `CheckParseTrees` accept a sparse `units` argument. I actually
think this is a positive: it makes `CheckIRId` a little more stable
across possible command lines, when file loading fails (which is the
only time that a file will have a `CompilationUnit` but not a
`Check::Unit`).
Trying to build on the shared issue between these, I'm adding a
`MultiUnitCache` to store the calculated arrays. For the subtree
getters, this is very minor and avoids at most one incremental array
construction (moving logic out of `CompileSubcommand::Run` might be the
bigger benefit). For `include_in_dumps`, when dumping SemIR, this is
changing a calculation run once per entity (in each IR) to be calculated
once per IR (globally), i.e. O(M*N) -> O(N).
Note this seems to be marginal for performance of file_test:
- Before: Stats over 10 runs: max = 5.3s, min = 4.7s, avg = 4.9s, dev =
0.2s
- After: Stats over 10 runs: max = 4.9s, min = 4.7s, avg = 4.8s, dev =
0.1s
I was mainly thinking about this in the context of dumping SemIR ranges.
There, the impact may actually decrease because a range won't do any
cross-IR printing. But, I'm expecting to add another layer for whether
we're printing IR for a file, and that made the `should_format_entity`
callback stick out for me.
Instead of building an eval block as a separate pass at the end of a
generic, build the eval block incrementally.
The larger change here is that asking for the type or constant value of
an instruction now always returns an unattached type or constant value,
in order to preserve the behavior that we previously achieved by doing
the rewrite to attached types and constant values at the end of handling
the generic.
This also incidentally fixes some subtle issues where attached types and
constant values would leak out into check and cause it to get confused
about differences between attached and unattached values. Check should
no longer see attached values except where it explicitly asks for them.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
Don't import `ImplWitnessTable` into the `constants` block, because we
generally don't put `Unique` constants there. This matches the handling
of the other kinds of `Unique` constants. In order to keep the
instruction visible in formatted SemIR, add it to the `imports` block
instead.
Also fix a bug in the instruction formatter that resulted in
instructions in the `imports` block being omitted from the output if
they were only referenced by earlier instructions in the `imports` block
and by instructions in the `constants` block. This was already resulting
in some referenced instructions being omitted from the output, but also
occurred frequently for `impl_witness_table` instructions after this
change because it is common for the only reference to those instructions
to be from `impl_witness` instructions in the `constants` block.
This prints instructions that are inside the range, and entities that
overlap with the range. Note this can lead to incomplete printing of
entity contents.
Trying to make it easier to see the API at a glance. The class has
become really long, and this doesn't fundamentally change that, but
hopefully makes it easier to navigate. The entry structure also had some
cruft that I'm removing.
I'm trying to keep functions in the same order as they currently are.
The delta still looks unhappy because of the churn, but hopefully this
at least explains the ordering in formatter.h. You can try using the
"Add indent" commit on the PR to see a better before-after delta.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
This was originally needed to support constant evaluation of name
expressions, but that's now done in a different way.
This is actually a step toward treating all patterns as constants. The
upcoming change will do so in a slightly different way, and so it will
simplify the review to start from a baseline where patterns are never
constant.
Adds an empty `min_prelude/destroy.carbon` in anticipation of turning it
into an interface. Update `no_prelude` tests to be `min_prelude` and
import it where needed; in some cases, modify the file to remove the
dependency (i.e., rewrite code to have nothing to destruct).
Rules executed by bazel don't necessarily have the right environment to
find the symbolizer, which was the intent of `cc_env` setting
`LLVM_SYMBOLIZER_PATH`. So far, this has kind of been a case-by-case
fix, but every so often I'm trying to debug a crash in a test that
doesn't provide it. Rather continuing down this route, instead add
drop-in wrappers for cc rules so that it's hard to forget.
Note `bazel/cc_rules` is intended to mirror `bazel/carbon_rules` and
`bazel/cc_toolchains`, rather than `@rules_cc`.
AFAICT there isn't a great way to add this as a default for the `bazel
run` environment. It's not typically going to be set on its own,
forwarding `$PATH` would be too broad, and the [action
`env_sets`](https://bazel.build/docs/cc-toolchain-config-reference#using-action-config)
I think are not quite what we need (I think those don't include output
execution, only compilation).
The main goal of this is to collapse the LocId and SemIRLoc types into a
single type, eliminating the need for APIs to decide which to use. This
originated from discussion about UnwrapSemIRLoc in #5169. Although that
was removed in #5202, it's probably still a good direction for LocId.
This changes the packing of LocId to allow adding InstId, making it
tri-modal: ImportIRInstId, InstId, or NodeId. This has a side-effect of
reducing the available space for ImportIRInstId, although not by much
due to the pre-existing `ImplicitBit` behavior. If needed, we could also
probably play with packing a bit more since `ImplicitBit` really only
applies to `NodeId`, but I was trying to keep the logic a little
simpler. Note `TokenOnlyBit` can still apply to `ImportIRInstId`.
This leaves in place a typedef for SemIRLoc -- I intend to clean that up
separately.
Some Discord discussion is
[here](https://discord.com/channels/655572317891461132/655578254970716160/1353755830058745959).
TypeInstId is an InstId whose constant value has a type of TypeType.
This includes:
- Type value instructions, the `ClassType` or `IntLiteralType`
instructions.
- Constraint value instructions, which are the `FacetType` and
`TypeType` instructions, each of which also have type TypeType.
TypeInstId encodes in the type system that it is safe to convert the
instruction's value to a TypeId, and CHECKs at construction that this
invariant is maintained.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
In preparation for shifting from `TypeId`s potentially representing
attached types to always representing unattached types, using
[terminology suggested on
Discord](https://discord.com/channels/655572317891461132/963846118964350976/1359286326779973712).
This change causes us to track slightly more type spelling information
through SemIR.
One change that has significant impact on the SemIR output is that we
now build a `struct_type` instruction in each class representing the
types of the fields, including the spelling used for those types. This
is now no longer always identical to the corresponding canonical
`struct_type` for the object representation, so it's built separately
and owned by the class.
Also remove `TypeBlock` support entirely, as its only use was
representing `TupleType`s, which now use an `InstBlock`.