The particular test this focused on is indirect_two_file in
toolchain/check/testdata/function/definition/no_prelude/extern_library.carbon.
This removes `parent_scope_id_for_new_inst` because I think it's
returning unhelpful results. The use was at the root of incorrect
results for the indirect import chain. `name_id_for_new_inst` is
actually wrapping a union, so it's more important.
The merging of `is_extern` and `first_owning_decl_id` in
`handle_function.cpp` feels like it's less correct with the changes
that've been made to `extern`. This ripples in tests, because the error
recovery shifts.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Support for types (particularly classes) is left as a TODO.
There's also an issue I'm observing with a "define in impl" test, but
this is probably an issue with resolving the prior declaration which is
imported indirectly. The PR was already feeling big, so I'm choosing to
cut here.
Note, this does not implement the rule "The owning library's API file
must import the `extern` declaration, and must also contain a
declaration."
This avoids import cycles, and reduces the number of temporary vectors
we build (and potentially throw away on retry). Import the self specific
when importing a generic, now that there's no risk that will introduce
cycles.
Note that we could take the same approach to import classes, interfaces,
and so on, instead of the current third phase of resolution for those
instructions, but in this PR I'm just addressing the import cycle I'm
currently seeing in a work-in-progress PR.
This prepares us for modeling associated entities of parameterized
interfaces.
We don't use the interface parameters when type-checking `impl`s or uses
of interface members yet, but we do now check interface arguments during
`impl` lookup.
A symbolic constant has an instruction to compute the constant value, as
well as potentially also having a generic ID and an index within that
generic to indicate where corresponding values can be found in a
specific. Import those pieces of information when importing such a
constant.
We try to import the generic before we start the main work of importing
the constant, and retry the import process if importing the generic adds
work to the worklist. This means that the first time we import anything
within a generic, we can now perform three passes calling
`TryResolveInst` instead of two, but the first pass is very lightweight
and only looks up and adds a single instruction, so the added overhead
of the extra pass should be minimal.
To avoid introducing cycles when importing a generic function, make the
import of a function declaration build the new `Function`,
`FunctionDecl`, and `FunctionType` in the first pass, like classes and
interfaces do.
Import generics and specifics when they are referenced by imported
entities.
When importing a generic, we import the symbolic constants required by
its eval block, and then rebuild the eval block itself given the list of
constants it needs to compute. This is likely a bit less efficient than
directly importing the contents of the eval block, but avoids needing to
either extend the importer code to be able to import the instructions
that can appear in the eval block or extend the evaluator to cope with
instructions from a different `SemIR::File`.
Importing a symbolic constant is unaffected, and does not yet preserve
the associated generic and index within that generic, so uses of a
generic from an imported IR still don't pick up values from the
specific, but the improved functionality can be seen in the changes to
the SemIR in the testcases.
The caching isn't buying us much, and is adding complexity and
divergence between the codepaths for generic and non-generic functions.
This means we no longer suppress diagnostics for the second or
subsequent time we call a function with an incomplete return type. If we
want to add that back, it might be worth considering moving the
suppression to `TryToCompleteType` and only diagnosing that a type is
incomplete once, regardless of why we're requiring it to be complete.
This implements a few closely related features:
- Starts merging namespaces discovered inside imports.
- Stores results of cross-package name lookup as an entry inside the
scope.
- Note this is particularly visible with `i32`.
- Moves more of the imported instructions to the import scope.
Note this is primarily for executing the namespace TODO in check.cpp,
which is removed here.
`testdata/namespace/merging_with_indirections.carbon` tests key
behavior.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Instead of reusing instructions from the generic entity in the eval
block, rebuild constants in the same way we rebuild types. The previous
attempt to not rebuild these constants assumed that every constant used
in a generic would be built in that generic, and not referenced directly
or referenced from some enclosing scope, which isn't true in practice
and is a fragile assumption in any case.
We could add back some reuse of instructions from the generic -- if we
happen to see the right instruction to build a constant, we could
opportunistically reuse it -- but given the complexity added by doing
so, I'm not pursuing that here.
Now that the eval block for a generic consists of instructions uniquely
owned by that generic, rather than often being shared with another
entity, include the generic in the formatted SemIR output. I'm using the
same scope name for the generic object itself as for the parameterized
class / function / interface, because there are very frequently
references between them and this keeps the IR simpler and more readable,
and avoids needing to invent a second name for the scope.
Note this isn't implementing checking through imports. The parse node
there is harder to access through the context, so would require
examining the entity in order to get the import declaration, to get at
the ImportIR. We also don't have a parse tree attached in that case, and
would need to add one to SemIR::File. But I believe we do want to add
that, so it's explicitly a TODO.
Note GetTokenText re-lexes literal values, so there's a bit of potential
overhead there. Not sure if we want a more efficient manner for
comparing in cases like this.
Also adds import_ir_scope to namespace formatting. I'd done this as an
aid for #4153, and am splitting it out.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
Changes crash messages to start printing verbose forms of instructions,
rather than just the ID. Fixes some indentation issues with stacks. Also
switches unexpected inst formatting, because now there are lots, and
it'd be helpful to know where they are.
This uses a pimpl pattern for Formatter due to the number of member
functions on Formatter. Maybe we should refactor that, but this didn't
feel like a good place to do so.
Note, I have two concerns about this change... to note them here, to
make sure others are considering them when evaluating the
implementation:
1. Some instructions are very verbose to print, as evidenced by the
fn_decl printing (which includes function params) or scope printing
(which includes scope members).
- I'm not sure whether there's a way to simply reduce this, as it seems
essential to the requested printing of instructions.
- Long-term, we may at least want to limit the number of lines printed
here. However, I've already spent a fair amount of time here and I think
it's in a good state to evaluate.
2. Increased complexity in the crash handler may result in crash
messages failing to generate.
- For example, a crash in Formatter (and its deps, such as InstNamer or
location handling) prevents a stack from being printed. I'm pretty sure
I've written crashes in Formatter before.
Here's an example crash snippet (generated by adding a crash inside
`return` handling) before:
```
2. NodeStack:
0. FunctionDefinitionStart -> function2
1. ReturnStatementStart -> no value
2. IntLiteral -> inst+26
inst_block_stack_:
0. block<invalid> {inst+0, inst+1, inst+2, inst+23}
1. block9 {inst+26}
param_and_arg_refs_stack:
args_type_info_stack_:
```
And after:
```
2. Check::Context
NodeStack:
0. FunctionDefinitionStart: function2
1. ReturnStatementStart: no value
2. IntLiteral:
unexpected.inst+26.loc12_10: i32 = int_literal 0 [template = constants.%.2]
inst_block_stack_:
0. block<invalid> {
package: <namespace> = namespace [template] {
.Core = unexpected.inst+2
.F = unexpected.inst+23.loc11_22
}
unexpected.inst+1 = import Core
unexpected.inst+2: <namespace> = namespace unexpected.inst+1, [template] {}
unexpected.inst+23.loc11_22: %F.type = fn_decl @F [template = constants.%F] {
unexpected.inst+9.loc11_9: init type = call constants.%Bool() [template = bool]
unexpected.inst+10.loc11_9: type = value_of_initializer unexpected.inst+9.loc11_9 [template = bool]
unexpected.inst+11.loc11_9: type = converted unexpected.inst+9.loc11_9, unexpected.inst+10.loc11_9 [template = bool]
unexpected.inst+12.loc11_6: bool = param b
@F.%b: bool = bind_name b, unexpected.inst+12.loc11_6
unexpected.inst+19.loc11_18: init type = call constants.%Int32() [template = i32]
unexpected.inst+20.loc11_18: type = value_of_initializer unexpected.inst+19.loc11_18 [template = i32]
unexpected.inst+21.loc11_18: type = converted unexpected.inst+19.loc11_18, unexpected.inst+20.loc11_18 [template = i32]
@F.%return: ref i32 = var <return slot>
}
}
1. block9 {
unexpected.inst+26.loc12_10: i32 = int_literal 0 [template = constants.%.2]
}
param_and_arg_refs_stack:
args_type_info_stack_:
```
When forming a specific (previously called a generic instance), evaluate
the eval block of the generic to determine the values of any constants
used in that specific. The majority of the work here is updating
eval.cpp so that it can use the results of prior evaluations in the same
block when computing later values.
Include the computed results in the formatted SemIR output.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
For each generic, build a list of instructions describing the
computations we need to do when resolving an instance of the generic:
this is a list of the instance-specific constants and types that the
generic uses. Another way of viewing this list is as a block of Carbon
SemIR code that is evaluated in order to form an instance of the generic
-- this is referenced in the code as the "eval block" for the generic.
For each instruction in the generic whose type or value is a symbolic
constant, replace that type or constant value with a symbolic reference
that says "to find the actual type or value, look at index N in the list
of values for the generic instance".
For an instruction with a symbolic constant value, we can just add that
instruction to our list. For an instruction with a symbolic constant
type, however, we may not have a corresponding instruction computing the
type within the generic and may need to build a new instruction, but
will reuse one where possible. In the case where we build a new
instruction, we use the existing substitution code to build the type
within the eval block.
For now, this transformation is only done in the declaration region of
the generic, not in the definition region. Also, we map back from the
symbolic references to the underlying constant value in a few places
where we will eventually need to do a lookup into a generic instance, in
order to avoid regressing the tests.
This executes on a TODO in AddImportRef to add instructions to their own
block instead of the File block. This has an important consequence of
removing a pattern from InstBlockStack that added to blocks not
currently at the top, cleaning up an issue for ArrayStack. The delta
here is then mostly in different formatting of the import refs, a
consequence of the separation.
By adding an `ImportDecl` instruction, this creates something that can
be referenced through `ImportIRInst`.
packages/no_prelude/implicit_imports_entities.carbon is getting a test
of this (import_conflict and import_conflict_reverse).
Also re-packs ImportIR from 24 bytes to 16 on 64-bit, since I'm touching
everywhere that makes one anyways.
This refactors how the implicit import is handled in order to retain
more name scope information. As a consequence, private access control
works better between api files and implementation files. Note though
that this will also be essential for name poisoning between the API and
implementation, as discussed in #3763.
In implementing this, I ran into a couple issues with namespaces that I
think point to flaws in their handling. I've fixed some and added a TODO
for the biggest issue (in check.cpp line 281-288), which relates to the
handling of namespaces of direct imports which are first evaluated
indirectly.
In `ClassType`s and `InterfaceType`s, track a `GenericInstanceId` for
the instance rather than just the argument list.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Build a `Generic` object for generic functions. This object tracks the
generic parameters that are in scope for the generic entity. Eventually
it will track other information about the generic too.
Add basic SemIR formatting support for generic functions.
Name scopes store the names in their scope in a `DenseMap`. Several
places reasonably avoid depending on the iteration order by sorting the
names -- they're in the formatting code path where that's a solid
approach.
Unfortunately, when we're importing one scope into another, we also need
to walk the entire scope and do something for each name. =[ This doesn't
seem like a great place to sort things to stabilize them.
I've switched to a fairly simplistic solution of having a vector of name
entries that can be iterated stably, and a separate map for lookups. I
didn't use the set-of-indices trick here because it's not clear that's
the right trade-off for a scope: likely a lot of small scopes here with
relatively hot name lookups. And the key here isn't a large or
dynamically sized thing that we're canonicalizing, it's a `NameId`. That
made me lean towards duplicating the name in the hashtable for lookup
and the vector for iteration.
I thought about a fancy approach of sorting the hashtable keys by their
values (the indices), but that would still require a bit of copying and
more code.
I also thought a bit about other optimizations, but decided to leave a
comment for now -- it's not obvious to me exactly how hot this is and
whether it's better served by faster lookups, being more memory dense,
etc. And that might involve more of an SOA layout change or some other
approach. Rather than do that here, and especially before switching
hashtables, I stuck with a simple approach to address the ordering.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Don't use the pretty-printed type name, because that's intended for
diagnostics, not for a theoretically machine-readable format like SemIR.
Types are always constants, so omit the leading `constant.` on the type
instruction name.
Don't format the *type of* the function `F` as `"F"`. Instead use
`"<type of F>"`.
Also improve instruction naming for function and generic class name
values: name the value after the function or generic class, and add a
`.type` suffix to the type.
Adds access to the name lookup table in name scopes. This is so that we
can quickly check access during name lookup without resolving the entity
itself. Does this for names in general, but does not implement handling
for entity-scoped names, only namespace-scoped names (where they're
essentially just not exported).
Excludes `private` names from exports. Although names should be
accessible to `impl` files, that's not implemented here because we'll
probably want to do it by directly copying name lookup tables.
This has is a nice-to-have for me. Frequently I want to run a specific
test, and end up digging through output to be able to copy-paste the run
line. This uses TIP lines to inject the command into the file when using
AUTOUPDATE.
Note, one of the reasons I want this is because "bazel test
//toolchain/testing:file_test --test_output=all" has been regularly
exceeding bazel's output limit for me (workaround is either opening the
output file or specifying an obscure output limit flag), making it a
little harder for me to get the commands. However, frequently I'm adding
a file and want to iterate on it, so that's really the use case I have
in mind here.
Parse the name of a declaration as a sequence of `NameQualifier`s --
which have a name, possibly parameters, and a trailing period --
followed by a name and possibly parameters. This prepares us for parsing
declarations of members of generic classes and similar cases, but
actually supporting such member redeclarations is left to a future
change.
We previously required functions to have parameters, but no longer do,
following the direction of #3848. Cases like namespaces that can't
actually have parameters are now diagnosed in check instead of in parse.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Modifies the core package and literal handling to use factory functions
for standard type literals.
Updates function/builtin/import.carbon to stop depending on the prelude,
since it would list all the impls in the core file. Updates
alias/builtins.carbon to be failing (the type values cannot be aliased).
Note, I assume this doesn't affect all the TODOs (like for i32.carbon),
but does cut some things down (and requires updating some tests that
have prelude name conflicts, but I think they were intended to be
updated this way).
Starts warning on duplicate "export name" because I'm updating name
lookup to make it work.
Need to think harder about how to handle cross-package "export import".
But adding some failing tests to sketch out what *should* work.
Change syntax for package declaration to put the `impl` keyword at the
start and remove the `api` keyword.
To support this, rearrange processing of package, library, and import
declarations to use the general modifier handling support in declaration
parsing rather than special-case logic.
There is an ambiguity in `impl package.Foo as Bar`, which we resolve by
treating `package` as an introducer after a modifier only if it's not
followed by `.`.
This adds a `BindExport` instruction in order to better track the
location of the `export` itself, but a `bind_name_id` is also added to
`ImportRef` so that we know quickly where to put it in name lookup.
Merging identical names is a TODO. I haven't quite decided how best to
achieve that, because I do think the BindExport should be what's
actually added to name lookup.
Also, I will probably add a mode to DeclNameStack that blocks
non-namespace scopes. This seems to already be an error, but the wrong
one (maybe due to lack of support for cross-file decl/def support).
This addresses the const reliance of ResolvePrevInstForMerge by checking
the imported instruction, testing with `alias` name conflicts. Given the
need to examine structure, the helper function feels like it now gets in
the way, so I'm just removing it.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This was to track use of a declaration after import, prior to a
redeclaration. Per [discussion on
Discord](https://discord.com/channels/655572317891461132/1217182321933815820/1236016521059237962),
we likely don't need this check due to the change in behavior of
`extern`.
Rather than potentially getting one of many `extern` decls and depending
on it by accident, it is now planned to be _required_ to be imported,
and the library doing a non-`extern` decl must _know_ it's importing the
`extern` decl. The stricter requirement on the library means it now
seems more reasonable to use the `extern` decl.
So kind of rolling back #3831, though keeping `ImportIRInstId` (at least
for now) and keeping `Loaded`/`Unloaded` terminology (seems a nicer
fit).
This removes the builtin FunctionType, replacing it with a FunctionType
instruction. The constant for a FunctionDecl is now a StructValue with
type of FunctionType.
Note this means a function declaration produces _both_ a type, and a
value of the type. This has some consequences in terms of circularity,
and makes the importing of function declarations a little more complex.
It'll get particularly peculiar for imports because of the behavior of
the reference, but that's a known issue due to other things such as
`alias`. The impact will hopefully be contained to
ResolvePrevInstForMerge (and ImportRefs).
To note a small formatting change in diagnostics:
```
- // CHECK:STDERR: fail_member_lookup.carbon:[[@LINE+4]]:3: ERROR: Value of type `<associated <function> in Interface>` is not callable.
+ // CHECK:STDERR: fail_member_lookup.carbon:[[@LINE+4]]:3: ERROR: Value of type `<associated F in Interface>` is not callable.
- // CHECK:STDERR: fail_todo_facet_lookup.carbon:[[@LINE+4]]:3: ERROR: Value of type `<associated <function> in Interface>` is not callable.
+ // CHECK:STDERR: fail_todo_facet_lookup.carbon:[[@LINE+4]]:3: ERROR: Value of type `<associated F in Interface>` is not callable.
```
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Change the names for emitted globals for constants with storage to
include both the name of the constant and the name of the use.
This causes the instructions to also be named in SemIR and in LLVM IR
constants.
This could probably go further, but I'm just trying to get the ability
to run some significant tests without the prelude. This already requires
some migration of `i32` and `bool` to non-prelude-dependent types
(technically not at the moment, but I'm trying to stick with the
philosophical model of the prelude).
Note this undoes part of #3895 which had made all lex/parse tests depend
on the prelude -- I don't think that was a necessary change. Rather, it
seems better to isolate individual lex/parse tests from the prelude. I'd
been on the fence in that PR (it was fixing one of the issues that
contributed to wanting to not include the prelude), but now it feels
more consistent. The prelude could have a parse error and we shouldn't
break every parse test on that.
We don't need it any more, and removing it simplifies a few things:
- One fewer predefined `File` and reserved ID.
- We now have simply `Builtin` instructions for builtins, instead of
having an `ImportRef` that indirectly references a `Builtin`.
- `ConstantId`s now always refer directly to a local constant, instead
of sometimes referring to an `ImportRef` for a constant in the builtins
IR.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Use a level comparison during substitution to determine whether we're
substituting a particular binding. Evaluate symbolic bindings with the
same name and the same level to the same symbolic constant, for example
across redeclarations of a generic function.
In handle_class and handle_interface we assign TypeType, so is more
consistent. I think this had been missed because we haven't really been
using these declarations (historically, declarations didn't have a
type). It seems not to significantly affect output at present, although
I found this while trying to change merge behavior.
Jon Ross-PerkinsandRichard Smith
Brymer Meneses
Chandler CarruthandRichard Smith