I'm looking at this due to the conversation on #3341. Although
diagnostics aren't where they should be, I thought it may help to start
adding raw identifier support (which may also help show how I was
thinking about this).
Note regarding the TODO on how to form the token, `GetTokenText` returns
the `string_id`'s reference value for an `Identifier`. So to make
`GetTokenText` work in a way that returns `r#foo` for a raw identifier,
I think there are a few options:
1. Add additional data indicating the end of the identifier.
2. Add `RawIdentifier` as a token kind to indicate that it's raw and
should be prefixed with `r#` (but also giving later stages one more
token kind to handle)
3. Make the `string_id` correspond to `r#foo`, and have later stages add
`foo` to the strings table whenever `r#foo` is encountered (with map
lookups leading to deduplication).
4. Add `StringId::RawKeyword` special values for each keyword.
- This would mean `self` prints as `self`, `r#self` prints as `r#self`,
but `r#foo` is not a keyword so prints as `foo`.
- This means keywords would need to be listed in a place `StringId` can
depend on them, one way or the other (e.g., a `keywords.def` file in
`base/` should work).
5. Say that it _is_ an `Identifier`, and if it's a keyword spelling, it
must have been a raw identifier.
- Same limitation as above: This would mean `self` prints as `self`,
`r#self` prints as `r#self`, but `r#foo` is not a keyword so prints as
`foo`.
I'm hoping to resolve this issue separately though. :)
For now, we treat class types and `String` as non-copyable, because we
don't know how to emit SemIR to copy them yet. This will change as we
add support for copying those types when appropriate.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Adds a `BoundMethod` SemIR node to represent an `x.F` bound method, with
a new builtin type `BoundMethodType`. Reorganized conversion of call
expression arguments to also check and convert a `self` parameter in the
implicit parameters list.
In passing, improved diagnostics and error recovery for bad call
expressions. We now build a `call` node with the appropriate type and
value category, but with invalid arguments, if the argument conversion
failed, and diagnose calls to non-callable expressions.
`addr self` methods don't work properly yet; the `addr` is ignored for
now.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
It warns on anonymous unions even when they have a field initialized.
Reported at https://github.com/llvm/llvm-project/issues/70384
Depending on how it's fixed, we may be able to remove this. If it's
fixed at head but still released in clang-18, we'd probably just change
the conditional.
Reported by guille2718; this is a version-dependent approach from #3339
Co-authored-by: Guillermo Rey
<10690205+guille2718@users.noreply.github.com>
Provides an adapter for the llvm::yaml API because it otherwise needs a
bunch of const/non-const definitions, and the traits are difficult to
diagnose issues with. The current approach is pretty simple to use, even
if it's not super efficient (which, yaml output is more of a debugging
thing so I'm not really expecting it to be an issue).
Changes the format of yaml output to provide more index information,
just as reminders when seeing something like `node+0`. Note this would
create more churn in deltas if we were reliant on the output yaml in
tests, but we aren't so it should be okay.
Add support for member access into classes, for both non-instance
members and for fields.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Track the fields in a class, and generate a corresponding struct type as
the object representation for the class. For now, we always use a
pointer as the value representation for a class.
This replaces the use of `VarStorage` in this case.
Add an `UnboundFieldType` type as the type of a field, in cases where
it's referenced without an accompanying object.
Add a `BindName` node to describe the name binding performed for both
variables and fields so that we can handle them more uniformly.
This standardizes on having ValueStore and related structures provide
printing, removing the handlers in file.cpp.
The `[]` is provided for empty sequences versus if there was simply
nothing, in which case it would be a sequence when non-empty, and a null
value when empty. Consistently (and explicitly) providing sequences
feels easier to understand.
The changes to the output yaml are overall more terse. My hope is that
this is an improvement for most readers.
Also fixes printing of APInt, defaulting to unsigned for consistency
with Carbon's use.
1. In general, `semantics_ir` -> `sem_ir`, to match the directory name.
2. For the list of `ValueStore`-related accessors on `SemIR::File`, add
them to `check`'s `Context` object, shortening access.
Building on #3311, which started moving the result string into a
`unique_ptr`, instead have `StringLiteral` use a `BumpPtrAllocator` to
manage memory. But also, detect when a string is really trivial during
`Lex` and, if so, return `contents_` directly.
Finishing what #3316 started, add more bespoke ValueStore-like
structures to File. With this, the things which previously had somewhat
boilerplate Add/Get functions are now all on side classes, giving a
uniform style of API for calling.
Note, I was on the fence about making things public on ValueStore. If
it's preferred that I make some things there protected I certainly can,
there's just a trade-off that may mean more distinct child/wrapper
types.
I spent (a lot) of time working to see if there was any profitable way
to port the SIMD code that scans for identifier length to Arm. There
isn't really. =/ While working on these, I made some cleanups to the
SIMD code that seemed worth landing, and added some benchmarks. All this
PR does is the cleanups, benchmarks, and documents that Arm isn't just
waiting to get attention but doesn't really have good options (so far).
For posterity, here are the core techniques I tried:
1) Direct 32-byte SIMD scanning using pair-wise add trees to build
a 32-bit mask of valid identifier and then `clz` to compute the
distance. This is a very good analog to the 16-byte SIMD structure
used on x86-64. The pair-wise summing technique is the one used in
simdjson for similar purposes.
2) A 16-byte SIMD scanning similar to the x86 version but using `shrn`
to produce a 64-bit scalar bitmask with 4 bits per byte, and then
scaling the bit-count distance.
3) Various hybrid versions of (1) and (2) with short scalar scans to
identify short identifiers before paying the SIMD start-up cost.
4) A much fancier version of (1) that scanned 64-bytes at a time, but
cached the resulting 64-bit mask and re-used it until exhausted.
Some good background on these techniques on Arm CPUs is in this blog
post:
https://community.arm.com/arm-community-blogs/b/infrastructure-solutions-blog/posts/porting-x86-vector-bitmask-optimizations-to-arm-neon
Sadly, both (1) and (2) were significantly slower than a scalar loop
over the bytes. Even (3) was consistently slower.
The only approach that came close was (4) and it was very *slightly*
slower in typical examples and very *slightly* faster in extremely
difficult cases like huge identifiers.
Ultimately, the only path I see (suggested by Dougall on a Mastodon
discussion of this whole problem space) is to take (4) to the limit of
computing an identifier-or-not bitmask *for the entire source file*
using a deeply throughput optimized routine (maybe as part of the line
scanning). That should be able to manage the high latency you end up
with when handling these patterns in SIMD on Arm.
The good news is that at least the M1 is *so* fast in the byte-scanning
loop that this isn't hurting nearly as much as I feared.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Running `bazel test //...` reported:
```
Test execution time outside of range for MODERATE tests.
Consider setting timeout="short" or size="small".
```
This change adds size="small" to avoid such warnings being reported.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Retain the `ClassDeclaration` node to represent a syntactic declaration
of a class (including possibly a declaration of a generic class), but
use a separate SemIR node to represent the class type itself. This
allows us to give the two separate treatment.
The `ClassDeclaration` is still entered into the name lookup table for
its enclosing scope, but when it is named in an expression, the class
type is produced instead. When the class declaration is named in a
declaration name, it can be used to define members of the class, but an
expression that resolves to the class type cannot be used to define
members of the class.
In order to distinguish these cases, use `Name` rather than
`NameExpression` for the left-hand side of a `QualifiedName` parse node.
This removes the only use of the `Expression` form of a declaration
name, so that is also removed.
In the future, `ClassType` will also be used to describe types such as
`Vector(T)`, for which there is no corresponding `ClassDeclaration`.
This is a prerequisite for class support, where a class can be
referenced as a type before it becomes complete. For example, given:
```carbon
class A {
fn F(a: A);
class B {}
var b: B;
}
fn A.F(a: A) {}
```
we need to lower `B` before we lower `A`, even though `A` is used as a
type first.
This will also start catching some cases where we don't require a type
to be complete despite using it, as we now only lower types that are
required to be complete.
Remove the poison values for struct and tuple literals. We don't need
those any more, because we never generate references to those literals
as values, and we don't have a type to use for them because we never
require the type of a literal to be complete, only the type of the
entity initialized by the literal, which can be different, for example
when initializing an array from a tuple literal or a class from a struct
literal.
This doesn't affect the output: `llvm::Type` objects that are not
referenced by an LLVM module don't affect the IR for that module, and
the order in which `llvm::Type`s are created doesn't affect anything
either.
Building on #3313, start using ValueStore on File. Functions and classes
are straightforward. Types here I present as a borderline case where
maybe we want a more bespoke API, but maybe this is okay? Most other
things probably need a slightly different API, which although I might do
that for a consistent interface, felt more out-of-scope for this change.
This replaces the printing that was removed from SemIR's raw dump. It's
separate because (for example) lexing generates shared values, and so
reviewing them is not specific to any particular phase.
Split `node.h` into separate files for ID types (`id.h`) and for typed
nodes (`typed_nodes.h`). The per-node-kind data is now specified as part
of declaring the typed nodes, and is removed from the node kinds
x-macros, which now simply enumerate the node kinds.
Building on #3311, change SemIR to use the SharedValueStore. Since this
removes hermeticity, raw output no longer prints ints, reals, and
strings. TokenizedBuffer accessors are modified to return IDs because
values are often passed through in semantics without needing to read
them.
I would've put SharedValueStores on Context, except for the
GetArrayBoundValue convenience method. I felt awkward removing that, so
it's on File, at least for now. That's then used by the formatter and
Lower too. The flipside of this is that TokenizedBuffer has a
SharedValueStores only for printing, so maybe that's similar enough to
what File is doing.
This doesn't start shifting other SemIR members to ValueStore, but that
seems like a next step.
This updates lexing to use the data. I'll do checking separately, just
to split changes.
Note the ValueStore structure is also set up such that SemIR::File can
use it for other fields.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This includes being able to define a class that was previously
forward-declared, and being able to define a member function out-of-line
that was previously declared inside a class.
No support for fields or methods yet, and a class definition doesn't yet
cause the class to be treated as a complete type.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Incomplete types may be nested within other types; for example, a tuple
type might have an incomplete type as an element. Handle such cases by
walking through nested incomplete types when completing a type. This is
done non-recursively in case a very complex type is formed.
Types are generally no longer completed at the point where they're
formed. Instead, we attempt to complete a type when it is used in a
context that requires a complete type, and diagnose if the type cannot
be completed at that point. This will be necessary for classes, which
can become complete after their first use, and helps tease out bugs
where a type completeness check is missing.
Specifically, after lexing a comment line, look at the next line and see
if it starts with an identical sequence of indent, comment '/'s and
character after the '/'s. If so, skip it as part of a block of comments.
This skips repeatedly diagnosing the same erroneous comment introducer
after the first one in a block, but that seems like a feature rather
than a bug.
The big motivation is to make sure the lexer is minimally impacted by
the length of comment blocks and skips them as efficiently as possible.
While they aren't exactly common, large block comments do come up and
it'd be unfortunate for those to actually slow down the toolchain.
It also happens that this is particularly easy to do because we're just
looking to see if we see the same prefix byte sequence. With SIMD we can
typically handle the most common indents with just a few instructions.
Because of the diagnostic differences, I've included a scalar fallback
that replicates the functionality but has no limit on indent size or CPU
features. I've also added testing to cover this behavior.
The only non-noise benchmark changes are as expected the comment ones,
with a nice improvement across the board:
```
BM_CommentLines/1/0/0 15.8ms ± 2% 15.6ms ± 2% -0.87% (p=0.004 n=19+19)
BM_CommentLines/4/0/0 20.2ms ± 1% 18.8ms ± 1% -6.75% (p=0.000 n=18+18)
BM_CommentLines/128/0/0 221ms ± 1% 167ms ± 1% -24.44% (p=0.000 n=20+19)
BM_CommentLines/1/30/0 16.6ms ± 3% 16.5ms ± 3% ~ (p=0.175 n=19+20)
BM_CommentLines/4/30/0 26.1ms ± 1% 24.8ms ± 2% -5.05% (p=0.000 n=18+19)
BM_CommentLines/128/30/0 233ms ± 1% 185ms ± 1% -20.38% (p=0.000 n=19+20)
BM_CommentLines/1/70/0 19.2ms ± 1% 19.0ms ± 2% -0.66% (p=0.016 n=19+20)
BM_CommentLines/4/70/0 27.9ms ± 1% 26.6ms ± 1% -4.63% (p=0.000 n=19+19)
BM_CommentLines/128/70/0 251ms ± 1% 213ms ± 1% -15.18% (p=0.000 n=20+18)
BM_CommentLines/1/0/2 15.9ms ± 1% 15.8ms ± 2% ~ (p=0.061 n=19+19)
BM_CommentLines/4/0/2 20.5ms ± 2% 19.0ms ± 2% -7.53% (p=0.000 n=20+20)
BM_CommentLines/128/0/2 213ms ± 1% 153ms ± 1% -28.18% (p=0.000 n=19+20)
BM_CommentLines/1/30/2 16.8ms ± 2% 16.7ms ± 3% ~ (p=0.134 n=20+20)
BM_CommentLines/4/30/2 26.6ms ± 1% 25.2ms ± 3% -5.50% (p=0.000 n=20+20)
BM_CommentLines/128/30/2 238ms ± 1% 187ms ± 2% -21.49% (p=0.000 n=17+19)
BM_CommentLines/1/70/2 19.3ms ± 1% 19.4ms ± 3% ~ (p=0.407 n=17+20)
BM_CommentLines/4/70/2 28.2ms ± 1% 26.9ms ± 2% -4.70% (p=0.000 n=19+19)
BM_CommentLines/128/70/2 257ms ± 2% 214ms ± 1% -16.52% (p=0.000 n=20+18)
BM_CommentLines/1/0/8 16.3ms ± 2% 16.1ms ± 2% -1.22% (p=0.001 n=20+20)
BM_CommentLines/4/0/8 22.7ms ± 2% 20.4ms ± 2% -10.20% (p=0.000 n=20+20)
BM_CommentLines/128/0/8 244ms ± 1% 153ms ± 1% -37.26% (p=0.000 n=20+18)
BM_CommentLines/1/30/8 17.3ms ± 2% 17.2ms ± 3% ~ (p=0.192 n=20+20)
BM_CommentLines/4/30/8 28.0ms ± 2% 25.6ms ± 3% -8.46% (p=0.000 n=19+18)
BM_CommentLines/128/30/8 272ms ± 1% 196ms ± 2% -27.90% (p=0.000 n=18+20)
BM_CommentLines/1/70/8 19.9ms ± 2% 19.9ms ± 2% ~ (p=0.531 n=20+19)
BM_CommentLines/4/70/8 29.3ms ± 1% 27.3ms ± 1% -6.87% (p=0.000 n=19+19)
BM_CommentLines/128/70/8 292ms ± 1% 228ms ± 1% -21.97% (p=0.000 n=20+19)
```
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Using the computed value representation, fix lowering of struct and
tuple values to use the value representation rather than the object
representation. Fixes an issue found in the review of #3257.
This currently causes us to compute value representations of all types
as they are created, which generates substantially more SemIR to
represent types. We can get some of that back by deferring computation
of the value representation until the type is required to be complete,
but some of the additional cost here will persist with this approach.
I also considered making the computation of the value representation
type be something that lives entirely within the lowering phase, but I
think that's not the right approach in the longer term, because the
value representation will be semantically visible and relevant once we
start allowing it to be customized.
We should consider moving the nodes that exist to compute canonical
non-local types, including value representations, out into a separate
global block. That will clean up the SemIR representation substantially,
and make the SemIR produced for a function not depend on which types we
happen to have encountered beforehand. But that's not being done in this
PR.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Track the callee expression in full, instead of only tracking the
callee's FunctionId. This results in the `name_reference` denoting the
function actually being used.
Lowering now propagates a `llvm::Function*` as the value associated with
expressions of type `<function>`.
We were not creating `NameReference` node for names produced by member
access into a namespace, such as the second name in
`Namespace.Function`, which caused lowering of calls to such names to
fail. This is now fixed, but the resulting `NameReference` node only
refers to the name and the lookup result, not to the `Namespace.`
qualifier. We'll need to decide how to fit a third operand into that
node (perhaps we can stop storing the `name_id`, since it can be derived
from the lookup result) but for now the qualifier is not tracked.
While this test is testing //toolchain/sem_ir:file, the interesting
thing about it is that it's testing the YAML output from File printing.
Calling this test `:file_test` is also deeply confusing because that's
also the name we give to all of our tests that are testing the files in
`testdata/`, which is not what this test does.
Can specify which tokens are allowed generally, and any additional
tokens that only occur when the parse node has an error.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Don't produce an error diagnostic if any of the information leading to
identifying that error is itself known to be affected by an
already-diagnosed error.
Jon Ross-Perkins
Richard SmithandJon Ross-Perkins
josh11bandJon Ross-Perkins
Chandler CarruthandJon Ross-Perkins
Jonathan B. CoeandChandler Carruth
Geoff Romer