This implements a few closely related features:
- Starts merging namespaces discovered inside imports.
- Stores results of cross-package name lookup as an entry inside the
scope.
- Note this is particularly visible with `i32`.
- Moves more of the imported instructions to the import scope.
Note this is primarily for executing the namespace TODO in check.cpp,
which is removed here.
`testdata/namespace/merging_with_indirections.carbon` tests key
behavior.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Also adds import_ir_scope to namespace formatting. I'd done this as an
aid for #4153, and am splitting it out.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
This is for consistency with #4120. Similar to that, we can use
overloads on the typed NodeId rather than individually named handlers.
There isn't the same caller benefit here though, since the calls from
check.cpp are already boilerplate.
We can't use the instruction from the generic directly, because it
doesn't have the right constant value. Instead add an instruction that
models the transition from the constant value in the generic to the
constant value in the generic instance.
Also start associating the self generic instance with unqualified
lookups that find results in an enclosing generic, so that we track the
information necessary to create the new instruction.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
I'm trying to increase the distinction between BuiltinKind and
BuiltinFunctionKind. BuiltinKind is for instructions,
BuiltinFunctionKind is for function definitions. To get to this point,
I'm doing a few changes:
- BuiltinKind -> BuiltinInstKind
- builtin_kind.* -> builtin_inst_kind.*: filename consistency
- Builtin -> BuiltinInst: mainly for consistency with the above
- Builtin::builtin_kind -> BuiltinInst::builtin_inst_kind: somewhat
repetitive but seems like a consistent edit
- Function::builtin_kind -> Function::builtin_function_kind: seems a
useful distinction
I'm leaving alone things like (and mentioning in case there's a desire
for more renames):
- InstId::BuiltinError, InstId::ForBuiltin: these I think are more
apparent because they're directly associated with Inst.
- GetBuiltinICmpPredicate in lowering: maybe builtin function handling
should be in its own file, but these local names don't feel problematic
to me.
- GetBuiltinType, BuildBuiltinValueRepr, PerformBuiltinIntComparison:
similar to the above, names don't feel too problematic
This executes on a TODO in AddImportRef to add instructions to their own
block instead of the File block. This has an important consequence of
removing a pattern from InstBlockStack that added to blocks not
currently at the top, cleaning up an issue for ArrayStack. The delta
here is then mostly in different formatting of the import refs, a
consequence of the separation.
Creates a `GlobalInit` class for storing relevant values, pulling
functions off `InstBlockStack` and `Context`. Adds a `Context` pointer
just so that it doesn't need to be passed in on each call (`Finalize` in
particular uses several members).
Note we have several different `InstBlockStack` instances, so several
copies of the relevant members were simply unused.
By adding an `ImportDecl` instruction, this creates something that can
be referenced through `ImportIRInst`.
packages/no_prelude/implicit_imports_entities.carbon is getting a test
of this (import_conflict and import_conflict_reverse).
Also re-packs ImportIR from 24 bytes to 16 on 64-bit, since I'm touching
everywhere that makes one anyways.
This works to leverage the capabilities of the hashtable as much as
possible, for example using the key context in the value stores.
However, there may still be opportunities to refactor more deeply and
use the functionality even better. Hopefully this is at least
a reasonable start and gets us a clean baseline.
On an Arm M1, this is a 15% improvement on my large lexing stress test,
but ends up a wash on my x86-64 server. This is a smaller benefit than
I expected, and it's because we're using a set-of-IDs and looking up
values with a key context for things like identifiers. This pattern has
a surprising tradeoff. The new hashtable uses significantly less memory,
a 10% peak RSS reduction just from the hashtable change. But indirecting
through the vector of values makes growing the hashtable dramatically
less cache-friendly: it causes growth to randomly access every key when
rehashing. On x86, everything gained by the faster hashtable is lost in
even slower growth. And even on Arm, this eats into the benefits.
But I have a plan to tweak how identifiers specifically work to avoid
most of the growth, and so I suspect this is the right tradeoff on the
whole. It gives us significant working set size reduction and we can
likely avoid the regressed operation (growth with rehash) in most cases
by clever reserving and if necessary by adding a hash caching layer to
the table infrastructure.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This refactors how the implicit import is handled in order to retain
more name scope information. As a consequence, private access control
works better between api files and implementation files. Note though
that this will also be essential for name poisoning between the API and
implementation, as discussed in #3763.
In implementing this, I ran into a couple issues with namespaces that I
think point to flaws in their handling. I've fixed some and added a TODO
for the biggest issue (in check.cpp line 281-288), which relates to the
handling of namespaces of direct imports which are first evaluated
indirectly.
Following up on discussion from #3948, doing a general rename of
"enclosing scope" to "parent scope" (and "enclosing scopes" to "ancestor
scopes"). The intent is to improve understandability and collide less
with C++ terminology for "enclosing scope". Note this changes most uses
of "enclosing", but leaves behind a few like "enclosing function" and
"enclosing block".
Note this does create some "parent class" mentions for "adapt" and "var"
(the class they're within), which is maybe unfortunate, but we'd
probably say "base class" if we meant inheritance so perhaps that's
okay. Along the same lines, these are the only `parent_class` uses I see
now, and we do have a few `base_class`.
Trying to conform with #4009. Changes SemIR::LocIdAndInst construction
to root out struct init cases with AddInst and related functions. I'm
using templating of AddInst functions in order to avoid `AddInst(loc_id,
InstName{...})` and instead have `AddInst<InstName>(loc_id, {...})` with
I think similar readability results. There are a couple cases where inst
construction is templated and so designated initializers couldn't be
used, so this may be better for those in particular due to the extra
type enforcement.
This probably doesn't clean up every last case, but I was trying to get
the bulk at once without bleeding over into less related changes.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Change syntax for package declaration to put the `impl` keyword at the
start and remove the `api` keyword.
To support this, rearrange processing of package, library, and import
declarations to use the general modifier handling support in declaration
parsing rather than special-case logic.
There is an ambiguity in `impl package.Foo as Bar`, which we resolve by
treating `package` as an introducer after a modifier only if it's not
followed by `.`.
The toolchain was iterating a map of package name to import list in a
couple of places to do things beyond counting or other order-invariant
operations. The result was that the specific hashtable iteration order
influenced the order of SemIR generated (and other behaviors like
diagnostic emission I suspect, but the source location sorting probably
hid this). Switching to a hashtable implementation with any order
seeding that changes run-to-run immediately shows the SemIR order
fluctuating without this.
This PR fixes that by instead accumulating the package imports data in a
vector and using a map to vector indices. This is also slightly more
efficient, although that seems unlikely to be an important factor here.
There was only one insertion point so I've just hand coded the
management of the indices and map, but happy to take a different
approach or use an abstraction here if desired.
One awkward aspect of this is that some of the loops need access to the
package name as well. I've just added storage for that as these seem
unlikely to be huge arrays of 10s of 1000s of imported packages, so the
double storage of the identifier ID seems likely OK. But again, happy to
take a different approach here if desired. It also seems like it might
be possible to work out the identifier from the node, but I kept the
patch more direct for simplicity.
I've also not used the LLVM `MapVector` abstraction of this pattern.
This was mostly to avoid adding another layer of abstractions to our
data structures, and because this is the first time we've hit this
really. My experience is also that it is reasonably often that there is
a more efficient way to orient the vector and map than what is
automatically provided. But that may just be my experience.
This adds a `BindExport` instruction in order to better track the
location of the `export` itself, but a `bind_name_id` is also added to
`ImportRef` so that we know quickly where to put it in name lookup.
Merging identical names is a TODO. I haven't quite decided how best to
achieve that, because I do think the BindExport should be what's
actually added to name lookup.
Also, I will probably add a mode to DeclNameStack that blocks
non-namespace scopes. This seems to already be an error, but the wrong
one (maybe due to lack of support for cross-file decl/def support).
I think the new name is more consistent for how `enclosing_scope_id` is
used relative to `name_id` (even removing the clarifying note on
`enclosing_scope_id_for_new_inst`). Suggesting `initial_scope_index` as
a replacing for the old `enclosing_scope`, hoping it's a little clearer.
I'm replacing `target_scope_id` uses in modifier logic because they
seemed to be based on the NameContext use.
This provides `export import` logic in lex, parse, and check; `export
name` logic is only in lex and parse, not check.
I think with `export name` I'm going to need to modify import_ref and
some consolidation logic, whereas `export import` seems feasible to keep
as primarily import logic. Given the implementations were looking like
they'd diverge more substantially, I thought it'd be helpful to cut the
PR here.
When we crash, include the source location of the parse node that we
were handling, as well as the name of the check function that we were
calling. Also include the source snippet, since it's easy to do so, and
may avoid the need to look at the input file in some cases.
As we talk about how imports should work, we want to start having
indirectly imported IRs in `import_irs`, where it's currently only
directly imported IRs. The tracking of `CheckIRId` is set up to be able
to determine whether an IR being indirectly imported is actually already
tracked (and may be a direct import). Changes here to how ApiForImpl is
handled start using the logic.
Note that indirect imports may be added while processing even the first
import from the current library. As a consequence, it's necessary to
prepare this map before starting imports, in particular, setting
ApiForImpl before any indirect imports may encounter the same import.
Also, prior validation that `num_irs` matches the final size are no
longer relevant.
check_ir_id is tracked on SemIR because I want to be able to figure it
out given an ImportIR, but the mapping is ephemeral after checking and
so I store that in Context.
I'm putting relevant logic into import_ref.cpp because the primary
alternative is import.cpp, and as more logic is added, ImportRefResolver
will be adding import IRs.
This allows ImportRefs to point at a potentially distant instruction,
while still providing a LocId that can be used for diagnostics. I think
this'll be used if we're trying to point ImportRefs at canonical
instructions in distant IRs. It conveniently eliminates a special-case
in check.cpp.
Restructures LocIdAndInst::Untyped because I think it's not really
needed after this change (the key non-import use was GetWithLocId, which
I just give friend access for). Adding NoLoc for things that don't
provide _any_ location because it turns out Inst can easily be passed in
this way which was not what I had intended, but is used.
We don't need it any more, and removing it simplifies a few things:
- One fewer predefined `File` and reserved ID.
- We now have simply `Builtin` instructions for builtins, instead of
having an `ImportRef` that indirectly references a `Builtin`.
- `ConstantId`s now always refer directly to a local constant, instead
of sometimes referring to an `ImportRef` for a constant in the builtins
IR.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This is mainly to prevent a crash. We should typically have a valid
location, but in cases of bugs I think this is better than a simple
check-failure (namely, provides the diagnostic to give better context of
why the diagnostic location looks bad). I also think we might have more
invalid locations in the future.
Note I ran into this because we weren't setting the definition_id for a
class in import_ref; I think this made it much easier to examine.
While most of the examples in our tests are made clear by the code
snippet, sometimes that's not the case. This makes it more clear when
the implicit `Main` package is used for example. I added this because I
got an error message without any source code to show in the snippet
(importing the prelude for example) and this makes the error much more
understandable.
- Adds an empty prelude.carbon file
- Imports that file in any non-Core package file
- Adds --disable-prelude-import to avoid that
- Adds --exclude-dump-file-prefix to be able to hide files from dumping
- Used to hide core files (we can't do this by package name due to lex
dumps, for example)
- Restructures some tests to not rely on `i32`, particularly `alias`
tests (which rely on a name ref) and tests with no prelude.
I'm adding the framework for switching i32 to calling Int32 in the
prelude, but I'm running into a separate error actually switching over.
So that *mostly* works, but isn't quite ready for prime time. However,
maybe the current state of this PR is still useful to review since it
does a lot of the infrastructure work and adds the %Core everywhere?
This allows impl lookup to find such impls.
Eventually we'll want to do this more lazily, and filter to the relevant
subset of `impl`s needed for a query. But for simplicity, for now just
import all `impl`s.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Adds ImportIRId::ApiForImpl to reserve a specific slot for the `api`
import, so that the code can trivially determine whether an import is
from the same library. This is then used for merging function
declarations, because the rules for redeclarations in the same library
slightly differ as compared to other imports (note they're also not
identical to same-file rules).
The main thing this leaves from the recent #3762 is verifying that
entities forward declared in the `impl` file are also defined, but
that's not in-scope for merging; it's moreso post-checking validation.
Note, a lot of our `invalid <entity> ID` comments in ids.h were
incorrectly copy-pasted, so I've cut `<entity>`.
Instead of attempting to reconstruct this based on the properties of the
target scope of the declaration name, check whether the enclosing scope
of the inner deferred definition scope is the enclosing deferred
definition scope.
Also add verbose logging to the deferred definition handling code to
make it easier to see when we switch between handling the regular parse
nodes and the deferred function bodies.
This doesn't significantly change logic, although I'm trying to add the
location to used state.
The issue I'm trying to address is how to identify a declaration as
"allowed to be redeclared". Consider:
```
library "a" api;
extern fn F();
```
```
library "b" api;
extern fn F();
```
```
library "c" api;
import library "a";
import library "b";
var x: auto = F();
fn F();
```
What currently happens is:
1. On import of "a", `F` becomes ImportRefUnused
2. On import of "b", `F` becomes ImportRefUsed in order to merge.
3. In "c", the call `F()` doesn't change the state.
4. In "c", the declaration `fn F();` needs some breadcrumb to understand
whether "F" has been referenced, as in step (3) here.
What I want to happen is:
1. On import of "a", `F` becomes ImportRefUnloaded
2. On import of "b", `F` becomes ImportRefLoaded in order to merge.
3. In "c", the call `F()` causes `F` to become ImportRefUsed
4. In "c", the declaration `fn F();` detects that `F` is already
ImportRefUsed, and can use the associated `used_id` for a diagnostic
about why redeclaring is invalid.
Note this PR isn't implementing (4). I'm focused on the refactoring to
add a new ImportRef state here.
In parse, form a list of methods that are defined inline, tracking where
they start, where they end, and which other inline methods are nested
within them.
In check, when we reach an inline method body, skip it and add it to a
worklist to be processed later. We also track when we reach the start
and end of a context in which inline method bodies are deferred, so that
we know when to replay the bodies.
When suspending a function definition to be processed later, the
`DeclNameStack` entry is moved to separate storage, including popping
the corresponding scopes from the scope stack and removing the
corresponding lexical names from lexical lookup. Later, when we return
to the function and parse its definition, the `DeclNameStack` entry is
restored. The same is done when we reach the end of a nested context
that can have inline methods, so that we can reenter the nested scope
before processing its members.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This replaces all invalid node IDs in import_ref.cpp with references to
the imported instruction.
This splits out ReplaceInstBeforeConstantUse into a separate function
when the LocationId is replaced, as for splicing. That's the less common
case, whereas others would need to provide the LocationId in order just
to not change the value.
The purpose of this change is to allow something such as a FunctionDecl
instruction to note an imported instruction as the "loc_id". Note that
doesn't occur here: this change is already very sweeping in edits. There
is no testdata affected, intended to show equivalent behavior.
We might want to consolidate NodeId references towards LocationId, but
if that's preferred, I'd still like to split it out. A lot of this just
piping through LocationId where it's a build error otherwise, enough
that imports should be able to start using it for diagnostics.
ValueStores are added but still unused -- just flushing out structure
for review.
Restructuring SemIRLocation is necessary to use LocationId this way. For
TokenOnly, it's not getting used in Parse, so I migrated it to Check and
it's now specific to SemIRLocation.
I also considered making LocationId reference an InstId (which would
need to be an ImportRef) instead of an ImportIRInstId. However, that
would've required import.cpp to add instructions for decls which are
reached during resolution -- we typically don't have an inst ready for
use. An extra inst is essentially 16 bytes in InstId's ValueStore + 4
bytes in LocationId's ValueStore, whereas this is 8 bytes per.
As discussed around #3792, identify the import a diagnostic message came
from prior to the diagnostic message itself. This occurs during location
translation so that the logic can be central.
I'd considered associating the parse node with ImportRef instructions,
but I realized about halfway through that because I need to store the
ImportDirectiveId on the ImportIR for cross-package imports, it's there
for use in location translation without extra work. That saves a fair
amount of stringing it through declarations, as well as an oddity where
ImportRef instructions would have a node that didn't really represent
them.
Since the addition of TranslateArg, I don't think this type is going to
go away (cutting a TODO). Refactoring names slightly to fit the current
role, and adding const to ConvertLocation.
Fix a collection of issues that were preventing lowering for overloaded
operators from working.
Instead of creating `import_ref` instructions during name lookup in the
current block, whatever that might be, we now create them in the `file`
block always. This avoids inserting them into blocks that might not be
intended to contain them, such as functions, and avoids the IR generated
for a function depending on which names we happen to have looked up
first.
When importing a class, function, or interface, import its enclosing
scope ID. This is necessary to allow us to distinguish between functions
at interface scope, which shouldn't be lowered, and other functions, and
will also be used in future to provide qualified names for declarations
when printing types. In order to support this:
- Track the constant values of namespaces created during importing so
that we can find them when resolving an import ref. Use those constant
values to convert an enclosing scope ID from the imported IR into a
corresponding ID in the current IR.
- Change how we do two-pass import of classes and namespaces so that we
can do two-pass import even for non-defining declarations, so that we
can import the enclosing scope.
While working on the final point above, I reworked `TryResolveInst` to
return a flag indicating whether another pass is necessary instead of
implicitly encoding this in the `ConstantId`. This permits the handling
of classes to be simplified; now `import_ir_constant_values` is only
accessed in a single place.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This revamps the support for cross-package imports, making them look
more like a namespace. The planned model is mentioned on
[#toolchain](https://discord.com/channels/655572317891461132/655578254970716160/1217586076022210670).
This does not implement name lookup into the new namespace structure.
A few key changes in this PR (it's a little sprawling) are:
- Moves logic for adding package imports from context.* to import.*
- Remove SemIR::Import, which was the prior model. This is instead now a
SemIR::Namespace with the NameScope getting a new import_ir_scopes
field.
- Allow SemIR::Namespace to use Parse::ImportDirectiveId in addition to
the prior Parse::NamespaceId
- The import_ir_scopes field includes a NameScopeId so that as we
traverse to child namespaces, we can directly perform name lookup in the
other IR.
- is_closed_import now tracks whether a namespace comes from a different
package. This has a diagnostic implemented in decl_name_stack.
This was previously discussed at
https://discord.com/channels/655572317891461132/655578254970716160/1209975051588210729.
I'm initiating this mainly because we typically use "id" suffixes to
indicate an `IdBase` being passed around and the non-id suffix of
`parse_node` suggests at it carrying more data than it actually does.
There used to be more reason for avoiding `node_id` because
`SemIR::InstId` used to be named `NodeId`, but that's no longer
necessary. As a consequence, I'd like to rename `parse_node` to more
precisely reflect its type.
In full, this is doing:
```
parse_node_kind -> node_kind
parse_node -> node_id
ParseNodeCategory -> NodeCategory
ParseNodeKind -> NodeKind
ParseNode -> NodeId
```
This is primarily in check and sem_ir, but with some `parse_node_kind`
references in parse too.
Pluralization is consistent with name forms on both sides, so that
wasn't part of my replacements.
I'm proposing a different split, along the line of "what does this
relate to". I view impl.h as having started down this route. Moving the
inst store stuff to inst.h feels odd to me given how much else is there
right now, but maybe it's still the best approach. Some files only
contain a store, no structured class, but I felt the consistency in file
naming (without _store suffixes) might help.
Note we may also want to do this with NameId, maybe some other things,
but the TypeId use is pretty broad and repetitive -- I thought I'd start
with it first.
Jon Ross-PerkinsandRichard Smith
Chandler CarruthandJon Ross-Perkins