This adds just the debug info metadata for Compilation Units (the top
level container of debug info) - but without anything in them, LLVM
won't emit them at all, so while this is testable at the IR level, it
isn't observable at the object level until more debug info is added.
A couple of starting points in this patch:
* A flag (`--debug-info`, seems to match the naming/style of other flags
in the carbon driver, though this is different from the naming
conventions of clang/gcc) that enables debug info when lowering. Open to
other names/approaches (on by default? historically debug info's been to
large/expensive to do this, so sticking with that precedent for now).
* Enabling that flag by default in the lowering tests - I do find the
churn on golden tests a bit rough, and adding more features to all the
tests means more churn, but it seems consistent with the approach so far
- keep an eye on this and perhaps revisit this if the churn gets too
annoying
---------
Co-authored-by: Carbon Infra Bot <carbon-external-infra@google.com>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Support for types (particularly classes) is left as a TODO.
There's also an issue I'm observing with a "define in impl" test, but
this is probably an issue with resolving the prior declaration which is
imported indirectly. The PR was already feeling big, so I'm choosing to
cut here.
Note, this does not implement the rule "The owning library's API file
must import the `extern` declaration, and must also contain a
declaration."
The current `toolchain/autoupdate_testdata.py` script assumes the
correct bazel version is already installed. This change uses
`scripts/scripts_utils.py` to fallback to `bazelisk`.
I'm trying to make the line of code more clearly nested in crash output
(the way it is, I sometimes forget about it). Also,
`Check::HandleFunctionDecl` is the old naming scheme, it's now all
`Check::HandleParseNode`, so I'm replacing that.
Before:
```
3. extern_library_owner.carbon:6:1: Check::HandleFunctionDecl
extern fn F();
^~~~~~~~~~~~~~
#0 0x0000564c0057ef1d llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) ...
```
After:
```
3. extern_library_owner.carbon:6:1: checking FunctionDecl
extern fn F();
^~~~~~~~~~~~~~
#0 0x00005629029ffd9d llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) ...
```
Consolidates both main libraries into `//testing/base`, and factors out
the exe path handling for benchmarks and unit tests into a common
library to remove duplication. Refactors how that logic is managed to be
cleaner and avoid a confusing bool that came up in code review.
Updates all the tests and benchmarks that use these. I still need to
update other benchmarks to use the same main, but I wanted to keep this
PR somewhat minimal.
This also fixes a bug noticed in passing that the compilation benchmark
didn't have the required dependency on the benchmark library itself,
just the benchmark main library.
The big addition here is a very, very rough and very early skeleton of a
source code generator framework. This builds upon the lexers identifier
synthesis logic, improving on its framework and wiring it up with the
most rudimentary of source file generation. This is just enough to
roughly replicate my "big API file" source code benchmarks.
The source generation works *very* hard to both vary the structure and
content of the source as much as possible while ensuring the same
*total* amount of each construct is in use, from bytes in identifiers to
line breaks, parameters, etc. This lets us generate randomly structure
inputs that should consistently take the exact same amount of total work
to compile.
The complex identifier synthesis logic from the lexer's benchmark is
moved over here and the lexer uses APIs in the source generator for
identifiers. The other source synthesis in the lexer's benchmark isn't
yet moved over, but should likely be slowly absorbed here as it can be
refactored into a more principled and re-usable form. Some bits may stay
of course if they're just too lexer-specific.
Next, this adds a simple end-to-end compile benchmark for the driver
that directly and much more clearly reproduces all the measurements I've
done manually up until now. It should also be easy to extend to more
patterns over time as we add support to the source generator to produce
those patterns.
Last but not least, I've added a tiny CLI to the source generator so
that you can generate source code manually. This is especially nice for
generating demo source code to actually run through the driver or look
at in an editor. The CLI can also generate C++ source code which lets us
do some minimal comparative benchmarking between Carbon and C++/Clang.
There are huge number of TODOs in the source generation framework. This
is going to be a large ongoing effort I suspect.
There are also a bunch of rough edges I've left to try and get this out
for review sooner. I've left TODOs for refactorings that really need to
be done here, but hoping these can maybe be follow-ups. If not, please
flag and I'll try to layer them on here.
Sample compile benchmark output, nicely showing where we are w.r.t. our
goal speeds (2x behind on lex and check, 5x on parse) at least on a
recent AMD server CPU:
```
------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations Lines
------------------------------------------------------------------------------------------------------
BM_CompileAPIFileDenseDecls<Phase::Lex>/256 29420 ns 29419 ns 22860 6.62847M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/1024 146130 ns 146128 ns 4840 6.69959M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/4096 601584 ns 601577 ns 1020 6.69573M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/16384 2547578 ns 2547313 ns 280 6.404M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/65536 10816591 ns 10816389 ns 80 6.05193M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/262144 52191320 ns 52189828 ns 20 5.02261M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/256 101706 ns 101698 ns 6900 1.91745M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/1024 512161 ns 512162 ns 1380 1.9115M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/4096 2078426 ns 2078430 ns 340 1.938M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/16384 8795786 ns 8795583 ns 100 1.85468M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/65536 35073596 ns 35072973 ns 20 1.86639M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/262144 151100688 ns 151097370 ns 20 1.73483M/s
BM_CompileAPIFileDenseDecls<Phase::Check>/256 957059 ns 957049 ns 740 203.751k/s
BM_CompileAPIFileDenseDecls<Phase::Check>/1024 1956134 ns 1955985 ns 360 500.515k/s
BM_CompileAPIFileDenseDecls<Phase::Check>/4096 5797864 ns 5797417 ns 120 694.792k/s
BM_CompileAPIFileDenseDecls<Phase::Check>/16384 21219608 ns 21217584 ns 40 768.843k/s
BM_CompileAPIFileDenseDecls<Phase::Check>/65536 96311116 ns 96302334 ns 20 679.734k/s
BM_CompileAPIFileDenseDecls<Phase::Check>/262144 371637963 ns 371609964 ns 20 705.387k/s
```
Lest someone think this is *bad*, the fact that we're already within 2x
of our rather audacious goals makes me quite happy. =D
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
The TODO for switching to ReusingLoc (previously Untyped) had been there
for a while, so I'm trying to address it here. The intent had been to be
clearer about when the construction is validated, particularly so that
we aren't accidentally accepting an incorrect NodeId. Note, this does
fix an incorrect use of InvalidNodeId where NoLoc should've been called.
Since this changes the semantics of when `Parse::NodeId` is helpful in
`typed_nodes.h`, I'm doing a pass to either refine or switch to
`Parse::InvalidNodeId` where it compiles. I think most remaining
`Parse::NodeId` examples are things we _should_ be able to refine with a
little more work (versus before where `Parse::NodeId` also indicated
`LocId` construction might be used).
I'm also changing context.h to use `requires` that match what
`LocIdAndInst` has, I think it makes the diagnostics a little better.
And note I do add an overload for `ImportIRInstId`, also matching
`LocIdAndInst`, and widely used for import refs.
This adds fields to `EntityWithParamsBase` to reflect the intention with
`extern library` design. I'm renaming `decl_id` because it shouldn't be
expected to be assigned anymore. import_ref.cpp I'm deliberately keeping
on `first_owning_decl_id` (which will break when importing `extern
library` declarations). Most other cases are for diagnostics, and I'm
using `latest_decl_id` to try and get the closest declaration to the
error. Note I'm partly splitting out this PR to show the test effect,
which apparently we don't test related cases.
This avoids import cycles, and reduces the number of temporary vectors
we build (and potentially throw away on retry). Import the self specific
when importing a generic, now that there's no risk that will introduce
cycles.
Note that we could take the same approach to import classes, interfaces,
and so on, instead of the current third phase of resolution for those
instructions, but in this PR I'm just addressing the import cycle I'm
currently seeing in a work-in-progress PR.
TryEvalInst was assuming this to be the case when forming canonical
constants, but it previously wasn't.
This fixes an issue where we can end up with two identical-looking
constants for an empty struct value: one with an `Empty` block and
another with the canonical empty block.
The prior code crashed when trying to find the function's `Self`
parameter, I believe. `fail_redefine_with_dependents.carbon` handles
this case. It wasn't caught by the prior case because the `F` didn't
have any dependent parameters.
Note this also ran into a formatter crash, with invalid constants. I'm
fixing that here, but will also note it on #4145 (the crash in
FinishGenericDecl was muddled by a crash in Formatter code).
Instead of always forcing an extra pass when we need to import a generic
ID for a generic that isn't already imported, attempt to import the
rest of the instruction in the same pass. There are then three
possibilities:
- The instruction needs a retry anyway to form its constant value, and
we avoid an extra pass.
- The instruction produces its constant value on the first pass but
still needs a retry. In this case, the handler for that instruction
is expected to retry itself, before building its constant value. The
third pass in this case can't be avoided.
- The instruction succeeds on its first pass. We still need an extra
pass; track the constant produced by resolution separately.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
It's actually possible to get into all four combinations of having
parameter lists versus being generic:
- An entity nested within a generic, such as a member class, can be
generic even if it has no parameters.
- As a corner case, an entity with an *empty* parameter list has
parameter lists, but isn't a generic because it doesn't have any generic
parameters.
This prepares us for modeling associated entities of parameterized
interfaces.
We don't use the interface parameters when type-checking `impl`s or uses
of interface members yet, but we do now check interface arguments during
`impl` lookup.
A symbolic constant has an instruction to compute the constant value, as
well as potentially also having a generic ID and an index within that
generic to indicate where corresponding values can be found in a
specific. Import those pieces of information when importing such a
constant.
We try to import the generic before we start the main work of importing
the constant, and retry the import process if importing the generic adds
work to the worklist. This means that the first time we import anything
within a generic, we can now perform three passes calling
`TryResolveInst` instead of two, but the first pass is very lightweight
and only looks up and adds a single instruction, so the added overhead
of the extra pass should be minimal.
To avoid introducing cycles when importing a generic function, make the
import of a function declaration build the new `Function`,
`FunctionDecl`, and `FunctionType` in the first pass, like classes and
interfaces do.
Import generics and specifics when they are referenced by imported
entities.
When importing a generic, we import the symbolic constants required by
its eval block, and then rebuild the eval block itself given the list of
constants it needs to compute. This is likely a bit less efficient than
directly importing the contents of the eval block, but avoids needing to
either extend the importer code to be able to import the instructions
that can appear in the eval block or extend the evaluator to cope with
instructions from a different `SemIR::File`.
Importing a symbolic constant is unaffected, and does not yet preserve
the associated generic and index within that generic, so uses of a
generic from an imported IR still don't pick up values from the
specific, but the improved functionality can be seen in the changes to
the SemIR in the testcases.
Move subtree sizes over to TreeAndSubtrees, using the different
structure to represent the additional parse work that occurs, as well as
making it clear which functions require the extra information. My intent
is to make it hard to use this by accident.
The subtree size is still tracked during Parse::Tree construction. I
think a lot of that can be cleaned up, although we use it during
placeholder assignment so it may take some work. I wanted to see what
people thought about this before taking action on such a change.
I'm using a 1m line source file generated by #4124 for testing. Command
is `time bazel-bin/toolchain/install/prefix_root/bin/carbon compile
--phase=check --dump-mem-usage ~/tmp/data.carbon`
At head, what I'm seeing is:
```
...
parse_tree_.node_impls_:
used_bytes: 61516116
reserved_bytes: 61516116
...
Total:
used_bytes: 447814230
reserved_bytes: 551663894
...
1.43s user 0.14s system 99% cpu 1.565 total
```
With `Tree::Verify` disabled completely, it looks like:
```
parse_tree_.node_impls_:
used_bytes: 41010744
reserved_bytes: 41010744
...
Total:
used_bytes: 427308858
reserved_bytes: 531158522
...
1.20s user 0.13s system 99% cpu 1.332 total
```
Re-enabling just the basic verification (what is now `Tree::Verify`),
I'm seeing maybe 0.05s slower, but that's within noise for my system. I
do see variability in my timing results, and overall I think this is a
0.2s +/- 0.1s improvement versus the earlier (always testing `Extract`
code) implementation. That's opt; debug builds will be unaffected,
because the same checking occurs as before.
Note, the subtree size is a third of the node representation, which is
why I'm showing the decrease in memory usage here.
Instead of updating the `value_id` on the canonical constant
`BindSymbolicName` to refer to some particular instance of that
constant, create a new instruction, and attach the proper location to
it.
Allow directly constructing a PostorderIterator, to get rid of
`tree.postorder(node_id).end()` indirect construction. For ranges that
don't need tree data, make it clearer that they're not validated.
Note, this subtly gets rid of a subtree size use in the
`tree.postorder(node_id).end()` case (to get the discarded `begin()`
value).
Rename `ReturnInfo` to `ReturnTypeInfo`. Move it and `InitRepr` into
`type_info.h` alongside `ValueRepr`. Replace `ReturnSlot` with
`InitRepr`, and extend `InitRepr` to be able to represent the
incomplete-type case instead of CHECK-failing. Remove `has_return_slot`
from `InitRepr` and instead only provide that as part of
`ReturnTypeInfo`.
The caching isn't buying us much, and is adding complexity and
divergence between the codepaths for generic and non-generic functions.
This means we no longer suppress diagnostics for the second or
subsequent time we call a function with an incomplete return type. If we
want to add that back, it might be worth considering moving the
suppression to `TryToCompleteType` and only diagnosing that a type is
incomplete once, regardless of why we're requiring it to be complete.
As discussed in toolchain meeting, we want to avoid overloading the
meaning of "instance", and "specific" was the best name we found. It's a
little unorthodox and inventive, but hopefully over time will become as
unsurprising as the term "generic" is.
I'd considered moving DeclParams uses over, but when handling qualified
names, there's a parse node instead of an instruction. I did try to
unify a couple other uses though, including adding MergeDefinition. I
expect `interface` will use a little more once it's more completely
implemented, but maybe I'm wrong about that.
This implements a few closely related features:
- Starts merging namespaces discovered inside imports.
- Stores results of cross-package name lookup as an entry inside the
scope.
- Note this is particularly visible with `i32`.
- Moves more of the imported instructions to the import scope.
Note this is primarily for executing the namespace TODO in check.cpp,
which is removed here.
`testdata/namespace/merging_with_indirections.carbon` tests key
behavior.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
When evaluating within the context of a specific, we can encounter uses
of bindings that are nested within that specific, for example parts of
the declaration of a nested generic. Those bindings should evaluate to
the canonical form of themselves, as they would when evaluating outside
the context of the specific.
Fixes#4157.
Instead of reusing instructions from the generic entity in the eval
block, rebuild constants in the same way we rebuild types. The previous
attempt to not rebuild these constants assumed that every constant used
in a generic would be built in that generic, and not referenced directly
or referenced from some enclosing scope, which isn't true in practice
and is a fragile assumption in any case.
We could add back some reuse of instructions from the generic -- if we
happen to see the right instruction to build a constant, we could
opportunistically reuse it -- but given the complexity added by doing
so, I'm not pursuing that here.
Now that the eval block for a generic consists of instructions uniquely
owned by that generic, rather than often being shared with another
entity, include the generic in the formatted SemIR output. I'm using the
same scope name for the generic object itself as for the parameterized
class / function / interface, because there are very frequently
references between them and this keeps the IR simpler and more readable,
and avoids needing to invent a second name for the scope.
Note this isn't implementing checking through imports. The parse node
there is harder to access through the context, so would require
examining the entity in order to get the import declaration, to get at
the ImportIR. We also don't have a parse tree attached in that case, and
would need to add one to SemIR::File. But I believe we do want to add
that, so it's explicitly a TODO.
Note GetTokenText re-lexes literal values, so there's a bit of potential
overhead there. Not sure if we want a more efficient manner for
comparing in cases like this.
Also adds import_ir_scope to namespace formatting. I'd done this as an
aid for #4153, and am splitting it out.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
R B
Brymer Meneses
Chandler Carruth
josh11bandJosh L
