Following up on discussion yesterday regarding this split.
Note, I'm expecting #3341 to do IdentifierId -> NameId in SemIR. It
might be worth adding NameId creation directly to StringStore if you're
content with this setup though.
Per the design, field access into a class value expression is a value
expression, even though we could produce an ephemeral reference
expression instead and avoid performing a value binding. This slightly
pessimizes class member access in some cases, but we should be able to
restore the old generated code by deferring actually performing the
value binding until a value expression is needed.
The value in the name lookup table is the class declaration. Map it to
the class type when it's found by lookup in an expression. This differs
from the behavior in a declaration name, where we want to find the class
declaration itself.
Advantages:
- Allows lexing/parsing in parallel, since they are modifying fully
separate ValueStores.
- Allows SemIR to reliably be stored hermetically.
Disadvantages:
- Creates overlapping storage of duplicate strings when multiple files
are compiled together.
- Prevents Ids from being uniquely compared cross-file.
Per discussion, the decision is that the advantages are more important.
The looser ownership remains because both SemIR checking and
metaprogramming may still generate things we would want to deduplicate.
It would be somewhat odd if TokenizedBuffer owned something that
checking modified.
This detects ordering issues with the `package` and `import` statements.
`library` is changed from package-specific to instead be generic between
the two, since structurally it's non-specific.
The next step would be to start exposing the results for the driver to
make ordering decisions for checking. That'll involve further
modifications to this code, but this felt like a reasonable change point
because it's the extent of the parser enforcement, and still causes
significant refactoring.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
If the initializing representation is the same as the value
representation, don't materialize a temporary and perform a value
binding. Instead, directly extract the value, using a new
`value_of_initializer` node.
This removes a lot of redundant `alloca`s from our generated LLVM IR.
I'm looking at this due to the conversation on #3341. Although
diagnostics aren't where they should be, I thought it may help to start
adding raw identifier support (which may also help show how I was
thinking about this).
Note regarding the TODO on how to form the token, `GetTokenText` returns
the `string_id`'s reference value for an `Identifier`. So to make
`GetTokenText` work in a way that returns `r#foo` for a raw identifier,
I think there are a few options:
1. Add additional data indicating the end of the identifier.
2. Add `RawIdentifier` as a token kind to indicate that it's raw and
should be prefixed with `r#` (but also giving later stages one more
token kind to handle)
3. Make the `string_id` correspond to `r#foo`, and have later stages add
`foo` to the strings table whenever `r#foo` is encountered (with map
lookups leading to deduplication).
4. Add `StringId::RawKeyword` special values for each keyword.
- This would mean `self` prints as `self`, `r#self` prints as `r#self`,
but `r#foo` is not a keyword so prints as `foo`.
- This means keywords would need to be listed in a place `StringId` can
depend on them, one way or the other (e.g., a `keywords.def` file in
`base/` should work).
5. Say that it _is_ an `Identifier`, and if it's a keyword spelling, it
must have been a raw identifier.
- Same limitation as above: This would mean `self` prints as `self`,
`r#self` prints as `r#self`, but `r#foo` is not a keyword so prints as
`foo`.
I'm hoping to resolve this issue separately though. :)
For now, we treat class types and `String` as non-copyable, because we
don't know how to emit SemIR to copy them yet. This will change as we
add support for copying those types when appropriate.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Adds a `BoundMethod` SemIR node to represent an `x.F` bound method, with
a new builtin type `BoundMethodType`. Reorganized conversion of call
expression arguments to also check and convert a `self` parameter in the
implicit parameters list.
In passing, improved diagnostics and error recovery for bad call
expressions. We now build a `call` node with the appropriate type and
value category, but with invalid arguments, if the argument conversion
failed, and diagnose calls to non-callable expressions.
`addr self` methods don't work properly yet; the `addr` is ignored for
now.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
It warns on anonymous unions even when they have a field initialized.
Reported at https://github.com/llvm/llvm-project/issues/70384
Depending on how it's fixed, we may be able to remove this. If it's
fixed at head but still released in clang-18, we'd probably just change
the conditional.
Reported by guille2718; this is a version-dependent approach from #3339
Co-authored-by: Guillermo Rey
<10690205+guille2718@users.noreply.github.com>
Provides an adapter for the llvm::yaml API because it otherwise needs a
bunch of const/non-const definitions, and the traits are difficult to
diagnose issues with. The current approach is pretty simple to use, even
if it's not super efficient (which, yaml output is more of a debugging
thing so I'm not really expecting it to be an issue).
Changes the format of yaml output to provide more index information,
just as reminders when seeing something like `node+0`. Note this would
create more churn in deltas if we were reliant on the output yaml in
tests, but we aren't so it should be okay.
Add support for member access into classes, for both non-instance
members and for fields.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Track the fields in a class, and generate a corresponding struct type as
the object representation for the class. For now, we always use a
pointer as the value representation for a class.
This replaces the use of `VarStorage` in this case.
Add an `UnboundFieldType` type as the type of a field, in cases where
it's referenced without an accompanying object.
Add a `BindName` node to describe the name binding performed for both
variables and fields so that we can handle them more uniformly.
This standardizes on having ValueStore and related structures provide
printing, removing the handlers in file.cpp.
The `[]` is provided for empty sequences versus if there was simply
nothing, in which case it would be a sequence when non-empty, and a null
value when empty. Consistently (and explicitly) providing sequences
feels easier to understand.
The changes to the output yaml are overall more terse. My hope is that
this is an improvement for most readers.
Also fixes printing of APInt, defaulting to unsigned for consistency
with Carbon's use.
1. In general, `semantics_ir` -> `sem_ir`, to match the directory name.
2. For the list of `ValueStore`-related accessors on `SemIR::File`, add
them to `check`'s `Context` object, shortening access.
Building on #3311, which started moving the result string into a
`unique_ptr`, instead have `StringLiteral` use a `BumpPtrAllocator` to
manage memory. But also, detect when a string is really trivial during
`Lex` and, if so, return `contents_` directly.
Finishing what #3316 started, add more bespoke ValueStore-like
structures to File. With this, the things which previously had somewhat
boilerplate Add/Get functions are now all on side classes, giving a
uniform style of API for calling.
Note, I was on the fence about making things public on ValueStore. If
it's preferred that I make some things there protected I certainly can,
there's just a trade-off that may mean more distinct child/wrapper
types.
I spent (a lot) of time working to see if there was any profitable way
to port the SIMD code that scans for identifier length to Arm. There
isn't really. =/ While working on these, I made some cleanups to the
SIMD code that seemed worth landing, and added some benchmarks. All this
PR does is the cleanups, benchmarks, and documents that Arm isn't just
waiting to get attention but doesn't really have good options (so far).
For posterity, here are the core techniques I tried:
1) Direct 32-byte SIMD scanning using pair-wise add trees to build
a 32-bit mask of valid identifier and then `clz` to compute the
distance. This is a very good analog to the 16-byte SIMD structure
used on x86-64. The pair-wise summing technique is the one used in
simdjson for similar purposes.
2) A 16-byte SIMD scanning similar to the x86 version but using `shrn`
to produce a 64-bit scalar bitmask with 4 bits per byte, and then
scaling the bit-count distance.
3) Various hybrid versions of (1) and (2) with short scalar scans to
identify short identifiers before paying the SIMD start-up cost.
4) A much fancier version of (1) that scanned 64-bytes at a time, but
cached the resulting 64-bit mask and re-used it until exhausted.
Some good background on these techniques on Arm CPUs is in this blog
post:
https://community.arm.com/arm-community-blogs/b/infrastructure-solutions-blog/posts/porting-x86-vector-bitmask-optimizations-to-arm-neon
Sadly, both (1) and (2) were significantly slower than a scalar loop
over the bytes. Even (3) was consistently slower.
The only approach that came close was (4) and it was very *slightly*
slower in typical examples and very *slightly* faster in extremely
difficult cases like huge identifiers.
Ultimately, the only path I see (suggested by Dougall on a Mastodon
discussion of this whole problem space) is to take (4) to the limit of
computing an identifier-or-not bitmask *for the entire source file*
using a deeply throughput optimized routine (maybe as part of the line
scanning). That should be able to manage the high latency you end up
with when handling these patterns in SIMD on Arm.
The good news is that at least the M1 is *so* fast in the byte-scanning
loop that this isn't hurting nearly as much as I feared.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Running `bazel test //...` reported:
```
Test execution time outside of range for MODERATE tests.
Consider setting timeout="short" or size="small".
```
This change adds size="small" to avoid such warnings being reported.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Retain the `ClassDeclaration` node to represent a syntactic declaration
of a class (including possibly a declaration of a generic class), but
use a separate SemIR node to represent the class type itself. This
allows us to give the two separate treatment.
The `ClassDeclaration` is still entered into the name lookup table for
its enclosing scope, but when it is named in an expression, the class
type is produced instead. When the class declaration is named in a
declaration name, it can be used to define members of the class, but an
expression that resolves to the class type cannot be used to define
members of the class.
In order to distinguish these cases, use `Name` rather than
`NameExpression` for the left-hand side of a `QualifiedName` parse node.
This removes the only use of the `Expression` form of a declaration
name, so that is also removed.
In the future, `ClassType` will also be used to describe types such as
`Vector(T)`, for which there is no corresponding `ClassDeclaration`.
This is a prerequisite for class support, where a class can be
referenced as a type before it becomes complete. For example, given:
```carbon
class A {
fn F(a: A);
class B {}
var b: B;
}
fn A.F(a: A) {}
```
we need to lower `B` before we lower `A`, even though `A` is used as a
type first.
This will also start catching some cases where we don't require a type
to be complete despite using it, as we now only lower types that are
required to be complete.
Remove the poison values for struct and tuple literals. We don't need
those any more, because we never generate references to those literals
as values, and we don't have a type to use for them because we never
require the type of a literal to be complete, only the type of the
entity initialized by the literal, which can be different, for example
when initializing an array from a tuple literal or a class from a struct
literal.
This doesn't affect the output: `llvm::Type` objects that are not
referenced by an LLVM module don't affect the IR for that module, and
the order in which `llvm::Type`s are created doesn't affect anything
either.
Building on #3313, start using ValueStore on File. Functions and classes
are straightforward. Types here I present as a borderline case where
maybe we want a more bespoke API, but maybe this is okay? Most other
things probably need a slightly different API, which although I might do
that for a consistent interface, felt more out-of-scope for this change.
This replaces the printing that was removed from SemIR's raw dump. It's
separate because (for example) lexing generates shared values, and so
reviewing them is not specific to any particular phase.
Split `node.h` into separate files for ID types (`id.h`) and for typed
nodes (`typed_nodes.h`). The per-node-kind data is now specified as part
of declaring the typed nodes, and is removed from the node kinds
x-macros, which now simply enumerate the node kinds.
Building on #3311, change SemIR to use the SharedValueStore. Since this
removes hermeticity, raw output no longer prints ints, reals, and
strings. TokenizedBuffer accessors are modified to return IDs because
values are often passed through in semantics without needing to read
them.
I would've put SharedValueStores on Context, except for the
GetArrayBoundValue convenience method. I felt awkward removing that, so
it's on File, at least for now. That's then used by the formatter and
Lower too. The flipside of this is that TokenizedBuffer has a
SharedValueStores only for printing, so maybe that's similar enough to
what File is doing.
This doesn't start shifting other SemIR members to ValueStore, but that
seems like a next step.
This updates lexing to use the data. I'll do checking separately, just
to split changes.
Note the ValueStore structure is also set up such that SemIR::File can
use it for other fields.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This includes being able to define a class that was previously
forward-declared, and being able to define a member function out-of-line
that was previously declared inside a class.
No support for fields or methods yet, and a class definition doesn't yet
cause the class to be treated as a complete type.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Piyush Udhao
josh11b
Richard SmithandJon Ross-Perkins
Jonathan B. CoeandChandler Carruth
Geoff Romer