mirror of
https://github.com/carbon-language/carbon-lang.git
synced 2026-10-05 18:41:08 +01:00
7d9340880ea2bcf8ea371e1a35bd9ddd658d3c24
2072
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Richard Smith
|
7d9340880e |
Separate ClassType from ClassDeclaration. (#3329)
Retain the `ClassDeclaration` node to represent a syntactic declaration of a class (including possibly a declaration of a generic class), but use a separate SemIR node to represent the class type itself. This allows us to give the two separate treatment. The `ClassDeclaration` is still entered into the name lookup table for its enclosing scope, but when it is named in an expression, the class type is produced instead. When the class declaration is named in a declaration name, it can be used to define members of the class, but an expression that resolves to the class type cannot be used to define members of the class. In order to distinguish these cases, use `Name` rather than `NameExpression` for the left-hand side of a `QualifiedName` parse node. This removes the only use of the `Expression` form of a declaration name, so that is also removed. In the future, `ClassType` will also be used to describe types such as `Vector(T)`, for which there is no corresponding `ClassDeclaration`. |
||
 Jon Ross-Perkins
|
ce248239d4 |
Fix missing include for ids.h (#3328)
I think this is only noticeable in a more modular build, but we try to keep that working. |
||
|
Richard Smith
|
85e9642d18 |
Lower types in the order they were completed. (#3324)
This is a prerequisite for class support, where a class can be
referenced as a type before it becomes complete. For example, given:
```carbon
class A {
fn F(a: A);
class B {}
var b: B;
}
fn A.F(a: A) {}
```
we need to lower `B` before we lower `A`, even though `A` is used as a
type first.
This will also start catching some cases where we don't require a type
to be complete despite using it, as we now only lower types that are
required to be complete.
Remove the poison values for struct and tuple literals. We don't need
those any more, because we never generate references to those literals
as values, and we don't have a type to use for them because we never
require the type of a literal to be complete, only the type of the
entity initialized by the literal, which can be different, for example
when initializing an array from a tuple literal or a class from a struct
literal.
This doesn't affect the output: `llvm::Type` objects that are not
referenced by an LLVM module don't affect the IR for that module, and
the order in which `llvm::Type`s are created doesn't affect anything
either.
|
||
 josh11b
|
5971f80826 |
Add consistency test between NodeKind and Definition (#3322)
|
||
 Geoff Romer
|
9e0d831092 |
Handle let with no closing semicolon (#3323)
Also add a test for the `var` case |
||
|
josh11b
|
b08f8bf589 |
Remove unused LiteralStringId (#3321)
|
||
|
Jon Ross-Perkins
|
7e9d644e1f |
Switch File functions, classes, and types to ValueStores (#3316)
Building on #3313, start using ValueStore on File. Functions and classes are straightforward. Types here I present as a borderline case where maybe we want a more bespoke API, but maybe this is okay? Most other things probably need a slightly different API, which although I might do that for a consistent interface, felt more out-of-scope for this change. |
||
|
Jon Ross-Perkins
|
843dd40f22 |
Adding ValueStore printing and --dump-shared-values (#3320)
This replaces the printing that was removed from SemIR's raw dump. It's separate because (for example) lexing generates shared values, and so reviewing them is not specific to any particular phase. |
||
|
josh11b
|
5b3b7aa3ed |
Test for id overflow and string deduplication in SharedValueStores (#3319)
Follow-on to #3311 |
||
|
Richard Smith
|
a46e7dd967 |
Remove most of the metaprogramming in node.h in favor of listing all the members in the typed node structs. (#3310)
Split `node.h` into separate files for ID types (`id.h`) and for typed nodes (`typed_nodes.h`). The per-node-kind data is now specified as part of declaring the typed nodes, and is removed from the node kinds x-macros, which now simply enumerate the node kinds. |
||
|
Jon Ross-Perkins
|
e54deee525 |
Fix DenseMapInfo declarations for ValueStore's StringId use. (#3318)
The move of the DenseMap to value_store.h should have been matched with moving the declaration; this worked by happenstance. |
||
|
Jon Ross-Perkins
|
1b55ad86dd |
Extend SharedValueStores to SemIR (#3313)
Building on #3311, change SemIR to use the SharedValueStore. Since this removes hermeticity, raw output no longer prints ints, reals, and strings. TokenizedBuffer accessors are modified to return IDs because values are often passed through in semantics without needing to read them. I would've put SharedValueStores on Context, except for the GetArrayBoundValue convenience method. I felt awkward removing that, so it's on File, at least for now. That's then used by the formatter and Lower too. The flipside of this is that TokenizedBuffer has a SharedValueStores only for printing, so maybe that's similar enough to what File is doing. This doesn't start shifting other SemIR members to ValueStore, but that seems like a next step. |
||
|
Jon Ross-PerkinsandRichard Smith
|
d13f76e001 |
Add value store to be shared across compile stages. (#3311)
This updates lexing to use the data. I'll do checking separately, just to split changes. Note the ValueStore structure is also set up such that SemIR::File can use it for other fields. --------- Co-authored-by: Richard Smith <richard@metafoo.co.uk> |
||
|
629c63c7d1 |
Port the comment block scanning SIMD to Arm. (#3300)
This adds an Arm Neon code path for comment block scanning. It also tries to tightened up the exact architecture-specific coding pattern for SIMD code here. Notably, moving to always having architecture-specific code inside a macro that can be used to globally disable SIMD, but any place where *all* architectures will require some custom code, an `else` branch with an error. Overall, this makes large-block lexing >5% faster on an ARM server I have access to, but that's a bit misleading. The Neon performance there doesn't seem very good. On my M1 laptop the difference is *much* larger. There, even 4-line comments show a noticable improvement and the block speed looks well over 20%. Sadly, I don't have the same nice scripts to generate good statistical data. Raw benchmark data from an ARM sever for reference: ``` BM_CommentLines/1/0/0 19.4ms ± 7% 19.7ms ± 6% ~ (p=0.121 n=20+20) BM_CommentLines/4/0/0 24.8ms ± 7% 24.8ms ± 6% ~ (p=0.904 n=20+20) BM_CommentLines/128/0/0 251ms ± 2% 234ms ± 3% -6.70% (p=0.000 n=20+20) BM_CommentLines/1/30/0 21.6ms ±10% 21.9ms ±10% ~ (p=0.157 n=20+20) BM_CommentLines/4/30/0 28.8ms ± 9% 28.8ms ±11% ~ (p=0.779 n=20+20) BM_CommentLines/128/30/0 248ms ± 2% 233ms ± 2% -5.72% (p=0.000 n=20+20) BM_CommentLines/1/70/0 23.4ms ±12% 23.7ms ±12% ~ (p=0.341 n=20+20) BM_CommentLines/4/70/0 30.8ms ± 9% 31.1ms ±11% ~ (p=0.602 n=20+20) BM_CommentLines/128/70/0 302ms ± 4% 292ms ± 4% -3.46% (p=0.000 n=18+20) BM_CommentLines/1/0/2 19.8ms ± 7% 20.0ms ± 6% ~ (p=0.149 n=20+20) BM_CommentLines/4/0/2 25.1ms ± 7% 25.3ms ± 8% ~ (p=0.659 n=20+20) BM_CommentLines/128/0/2 225ms ± 2% 212ms ± 2% -5.88% (p=0.000 n=20+20) BM_CommentLines/1/30/2 22.0ms ± 9% 22.2ms ±10% ~ (p=0.289 n=20+20) BM_CommentLines/4/30/2 29.0ms ±11% 29.1ms ±12% ~ (p=0.738 n=20+20) BM_CommentLines/128/30/2 261ms ±10% 243ms ± 3% -6.85% (p=0.000 n=20+20) BM_CommentLines/1/70/2 23.5ms ±11% 23.8ms ±15% ~ (p=0.429 n=20+20) BM_CommentLines/4/70/2 31.3ms ±10% 31.5ms ±11% ~ (p=0.478 n=20+20) BM_CommentLines/128/70/2 306ms ± 4% 292ms ± 4% -4.52% (p=0.000 n=18+19) BM_CommentLines/1/0/8 20.9ms ± 8% 21.2ms ± 7% ~ (p=0.127 n=20+20) BM_CommentLines/4/0/8 27.3ms ± 9% 27.5ms ±12% ~ (p=0.678 n=20+20) BM_CommentLines/128/0/8 227ms ± 2% 210ms ± 2% -7.35% (p=0.000 n=19+20) BM_CommentLines/1/30/8 22.6ms ±11% 23.0ms ±10% ~ (p=0.114 n=20+20) BM_CommentLines/4/30/8 29.4ms ±10% 29.4ms ±12% ~ (p=0.947 n=20+20) BM_CommentLines/128/30/8 275ms ± 4% 257ms ± 7% -6.59% (p=0.000 n=19+20) BM_CommentLines/1/70/8 23.9ms ±13% 24.3ms ±14% ~ (p=0.265 n=20+20) BM_CommentLines/4/70/8 32.3ms ±11% 32.4ms ± 9% ~ (p=0.478 n=20+20) BM_CommentLines/128/70/8 319ms ± 4% 307ms ± 4% -3.83% (p=0.000 n=18+19) ``` --------- Co-authored-by: Richard Smith <richard@metafoo.co.uk> Co-authored-by: Jon Ross-Perkins <jperkins@google.com> |
||
|
Richard Smith
|
a95e122123 |
Use the full name declaration logic when adding var names to scope. (#3312)
This causes the names of class members to get added to the class scope. |
||
|
Richard SmithandJon Ross-Perkins
|
b01dfb3f93 |
Support for class definitions with static member functions. (#3305)
This includes being able to define a class that was previously forward-declared, and being able to define a member function out-of-line that was previously declared inside a class. No support for fields or methods yet, and a class definition doesn't yet cause the class to be treated as a complete type. --------- Co-authored-by: Jon Ross-Perkins <jperkins@google.com> |
||
|
josh11bandJon Ross-Perkins
|
3b82ea96db |
Add & update comments in parse/node_kind.def (#3307)
Co-authored-by: Jon Ross-Perkins <jperkins@google.com> |
||
|
josh11b
|
6d4d05f68a |
SEMANTICS->SEM_IR in macro names. Update comments in sem_ir/node.h. (#3309)
* Shift to "sem_ir" continued from #3176. * Comments in `node.h` changed to reflect #3280. |
||
|
Richard Smith
|
efd8e1b144 |
Fix follow-on error for member access on an invalid expression. (#3303)
We were looking at the type of the base expression prior to conversion instead of the type after conversion. As noted in review of #3302. |
||
|
Richard Smith
|
69353ed271 |
Basic support for incomplete types. (#3302)
Incomplete types may be nested within other types; for example, a tuple type might have an incomplete type as an element. Handle such cases by walking through nested incomplete types when completing a type. This is done non-recursively in case a very complex type is formed. Types are generally no longer completed at the point where they're formed. Instead, we attempt to complete a type when it is used in a context that requires a complete type, and diagnose if the type cannot be completed at that point. This will be necessary for classes, which can become complete after their first use, and helps tease out bugs where a type completeness check is missing. |
||
|
Chandler CarruthandRichard Smith
|
3015135a52 |
Skip blocks of comments with identical prefixes. (#3299)
Specifically, after lexing a comment line, look at the next line and see if it starts with an identical sequence of indent, comment '/'s and character after the '/'s. If so, skip it as part of a block of comments. This skips repeatedly diagnosing the same erroneous comment introducer after the first one in a block, but that seems like a feature rather than a bug. The big motivation is to make sure the lexer is minimally impacted by the length of comment blocks and skips them as efficiently as possible. While they aren't exactly common, large block comments do come up and it'd be unfortunate for those to actually slow down the toolchain. It also happens that this is particularly easy to do because we're just looking to see if we see the same prefix byte sequence. With SIMD we can typically handle the most common indents with just a few instructions. Because of the diagnostic differences, I've included a scalar fallback that replicates the functionality but has no limit on indent size or CPU features. I've also added testing to cover this behavior. The only non-noise benchmark changes are as expected the comment ones, with a nice improvement across the board: ``` BM_CommentLines/1/0/0 15.8ms ± 2% 15.6ms ± 2% -0.87% (p=0.004 n=19+19) BM_CommentLines/4/0/0 20.2ms ± 1% 18.8ms ± 1% -6.75% (p=0.000 n=18+18) BM_CommentLines/128/0/0 221ms ± 1% 167ms ± 1% -24.44% (p=0.000 n=20+19) BM_CommentLines/1/30/0 16.6ms ± 3% 16.5ms ± 3% ~ (p=0.175 n=19+20) BM_CommentLines/4/30/0 26.1ms ± 1% 24.8ms ± 2% -5.05% (p=0.000 n=18+19) BM_CommentLines/128/30/0 233ms ± 1% 185ms ± 1% -20.38% (p=0.000 n=19+20) BM_CommentLines/1/70/0 19.2ms ± 1% 19.0ms ± 2% -0.66% (p=0.016 n=19+20) BM_CommentLines/4/70/0 27.9ms ± 1% 26.6ms ± 1% -4.63% (p=0.000 n=19+19) BM_CommentLines/128/70/0 251ms ± 1% 213ms ± 1% -15.18% (p=0.000 n=20+18) BM_CommentLines/1/0/2 15.9ms ± 1% 15.8ms ± 2% ~ (p=0.061 n=19+19) BM_CommentLines/4/0/2 20.5ms ± 2% 19.0ms ± 2% -7.53% (p=0.000 n=20+20) BM_CommentLines/128/0/2 213ms ± 1% 153ms ± 1% -28.18% (p=0.000 n=19+20) BM_CommentLines/1/30/2 16.8ms ± 2% 16.7ms ± 3% ~ (p=0.134 n=20+20) BM_CommentLines/4/30/2 26.6ms ± 1% 25.2ms ± 3% -5.50% (p=0.000 n=20+20) BM_CommentLines/128/30/2 238ms ± 1% 187ms ± 2% -21.49% (p=0.000 n=17+19) BM_CommentLines/1/70/2 19.3ms ± 1% 19.4ms ± 3% ~ (p=0.407 n=17+20) BM_CommentLines/4/70/2 28.2ms ± 1% 26.9ms ± 2% -4.70% (p=0.000 n=19+19) BM_CommentLines/128/70/2 257ms ± 2% 214ms ± 1% -16.52% (p=0.000 n=20+18) BM_CommentLines/1/0/8 16.3ms ± 2% 16.1ms ± 2% -1.22% (p=0.001 n=20+20) BM_CommentLines/4/0/8 22.7ms ± 2% 20.4ms ± 2% -10.20% (p=0.000 n=20+20) BM_CommentLines/128/0/8 244ms ± 1% 153ms ± 1% -37.26% (p=0.000 n=20+18) BM_CommentLines/1/30/8 17.3ms ± 2% 17.2ms ± 3% ~ (p=0.192 n=20+20) BM_CommentLines/4/30/8 28.0ms ± 2% 25.6ms ± 3% -8.46% (p=0.000 n=19+18) BM_CommentLines/128/30/8 272ms ± 1% 196ms ± 2% -27.90% (p=0.000 n=18+20) BM_CommentLines/1/70/8 19.9ms ± 2% 19.9ms ± 2% ~ (p=0.531 n=20+19) BM_CommentLines/4/70/8 29.3ms ± 1% 27.3ms ± 1% -6.87% (p=0.000 n=19+19) BM_CommentLines/128/70/8 292ms ± 1% 228ms ± 1% -21.97% (p=0.000 n=20+19) ``` --------- Co-authored-by: Richard Smith <richard@metafoo.co.uk> |
||
|
Chandler Carruth
|
7371354dc7 |
Consolidate indent handling to following newlines. (#3298)
The biggest advantage of this is reducing the repeated code in every non-whitespace code path of the lexer to set indent appropriately. Now we handle it cleanly at the start and after vertical whitespace. While this makes the generated code for all the other paths through the lexer quite a bit nicer, it doesn't actually move performance in interesting ways outside of making large blocks of blank lines slightly slower. While there are lots of fluctuations in the benchmark data, they mostly seem to be either noise or artifacts of loop alignment and not really due to an important change here. Raw benchmark data: ``` BM_ValidKeywords 3.10ms ± 1% 3.13ms ± 1% +1.02% (p=0.000 n=20+19) BM_ValidIdentifiers<1, 64, false> 10.8ms ± 3% 10.7ms ± 3% ~ (p=0.383 n=20+20) BM_ValidIdentifiers<1, 1, true> 3.71ms ± 1% 3.67ms ± 2% -1.05% (p=0.000 n=19+19) BM_ValidIdentifiers<3, 5, true> 13.1ms ± 2% 13.0ms ± 2% ~ (p=0.120 n=20+19) BM_ValidIdentifiers<3, 16, true> 13.1ms ± 2% 13.2ms ± 2% ~ (p=0.091 n=20+20) BM_ValidIdentifiers<12, 64, true> 15.1ms ± 1% 15.1ms ± 1% ~ (p=0.138 n=19+19) BM_HorizontalWhitespace/1 13.2ms ± 3% 13.2ms ± 0% ~ (p=0.458 n=20+15) BM_HorizontalWhitespace/4 13.4ms ± 3% 13.3ms ± 1% -1.12% (p=0.000 n=20+19) BM_HorizontalWhitespace/16 14.1ms ± 2% 14.0ms ± 2% -0.89% (p=0.010 n=20+20) BM_HorizontalWhitespace/64 17.9ms ± 2% 17.8ms ± 1% -0.73% (p=0.002 n=20+16) BM_HorizontalWhitespace/128 24.2ms ± 2% 24.1ms ± 1% ~ (p=0.346 n=19+17) BM_RandomSource 7.88ms ± 2% 7.83ms ± 3% ~ (p=0.052 n=20+20) BM_GroupingSymbols/1/0/0 6.25ms ± 2% 6.38ms ± 1% +1.99% (p=0.000 n=20+19) BM_GroupingSymbols/2/0/0 5.24ms ± 2% 5.27ms ± 2% ~ (p=0.065 n=20+19) BM_GroupingSymbols/3/0/0 4.09ms ± 1% 4.07ms ± 1% ~ (p=0.063 n=20+20) BM_GroupingSymbols/4/0/0 3.80ms ± 1% 3.79ms ± 2% ~ (p=0.820 n=20+20) BM_GroupingSymbols/8/0/0 3.05ms ± 1% 3.09ms ± 1% +1.41% (p=0.000 n=20+20) BM_GroupingSymbols/16/0/0 2.95ms ± 1% 3.01ms ± 1% +1.79% (p=0.000 n=20+20) BM_GroupingSymbols/32/0/0 3.67ms ± 1% 3.77ms ± 1% +2.75% (p=0.000 n=20+20) BM_GroupingSymbols/0/1/0 5.66ms ± 1% 5.61ms ± 1% -0.88% (p=0.000 n=20+18) BM_GroupingSymbols/0/2/0 4.43ms ± 2% 4.40ms ± 1% -0.74% (p=0.005 n=20+17) BM_GroupingSymbols/0/3/0 3.15ms ± 2% 3.12ms ± 2% -0.96% (p=0.002 n=20+19) BM_GroupingSymbols/0/4/0 2.79ms ± 2% 2.77ms ± 2% -0.88% (p=0.005 n=20+20) BM_GroupingSymbols/0/8/0 1.81ms ± 2% 1.79ms ± 2% ~ (p=0.056 n=20+20) BM_GroupingSymbols/0/16/0 1.26ms ± 2% 1.26ms ± 2% ~ (p=0.547 n=20+20) BM_GroupingSymbols/0/32/0 1.05ms ± 1% 0.96ms ± 1% -8.52% (p=0.000 n=19+19) BM_GroupingSymbols/0/0/1 5.68ms ± 2% 5.65ms ± 2% ~ (p=0.126 n=20+18) BM_GroupingSymbols/0/0/2 4.44ms ± 2% 4.40ms ± 2% -0.99% (p=0.001 n=20+18) BM_GroupingSymbols/0/0/3 3.15ms ± 1% 3.13ms ± 1% -0.74% (p=0.005 n=20+18) BM_GroupingSymbols/0/0/4 2.80ms ± 1% 2.77ms ± 2% -1.01% (p=0.000 n=20+19) BM_GroupingSymbols/0/0/8 1.81ms ± 2% 1.79ms ± 2% -0.97% (p=0.006 n=20+20) BM_GroupingSymbols/0/0/16 1.26ms ± 1% 1.26ms ± 2% ~ (p=0.678 n=20+20) BM_GroupingSymbols/0/0/32 1.05ms ± 1% 0.96ms ± 1% -8.59% (p=0.000 n=20+19) BM_GroupingSymbols/32/1/0 3.57ms ± 1% 3.69ms ± 1% +3.39% (p=0.000 n=19+20) BM_GroupingSymbols/32/2/0 3.49ms ± 1% 3.60ms ± 1% +3.16% (p=0.000 n=20+20) BM_GroupingSymbols/32/3/0 3.41ms ± 1% 3.52ms ± 1% +3.49% (p=0.000 n=19+20) BM_GroupingSymbols/32/4/0 3.35ms ± 1% 3.45ms ± 1% +3.13% (p=0.000 n=20+20) BM_GroupingSymbols/32/8/0 3.12ms ± 1% 3.22ms ± 1% +3.09% (p=0.000 n=19+20) BM_GroupingSymbols/32/16/0 2.75ms ± 1% 2.82ms ± 1% +2.71% (p=0.000 n=19+20) BM_GroupingSymbols/32/32/0 2.24ms ± 2% 2.24ms ± 1% ~ (p=0.311 n=20+17) BM_GroupingSymbols/32/32/1 2.20ms ± 1% 2.23ms ± 1% +1.23% (p=0.000 n=20+20) BM_GroupingSymbols/32/32/2 2.19ms ± 1% 2.21ms ± 1% +1.08% (p=0.000 n=20+20) BM_GroupingSymbols/32/32/3 2.16ms ± 0% 2.19ms ± 1% +1.09% (p=0.000 n=18+20) BM_GroupingSymbols/32/32/4 2.14ms ± 1% 2.19ms ± 1% +2.25% (p=0.000 n=20+20) BM_GroupingSymbols/32/32/8 2.07ms ± 1% 2.11ms ± 1% +2.04% (p=0.000 n=17+20) BM_GroupingSymbols/32/32/16 1.94ms ± 1% 1.98ms ± 1% +1.95% (p=0.000 n=20+20) BM_GroupingSymbols/32/32/32 1.74ms ± 1% 1.77ms ± 1% +1.88% (p=0.000 n=18+20) BM_BlankLines/1 14.2ms ± 1% 14.4ms ± 1% +1.47% (p=0.000 n=18+16) BM_BlankLines/4 17.3ms ± 1% 17.5ms ± 2% +1.32% (p=0.000 n=17+20) BM_BlankLines/16 31.3ms ± 1% 33.0ms ± 2% +5.35% (p=0.000 n=19+20) BM_BlankLines/64 89.6ms ± 1% 101.3ms ± 1% +12.98% (p=0.000 n=20+20) BM_BlankLines/128 167ms ± 3% 185ms ± 1% +11.11% (p=0.000 n=19+20) BM_CommentLines/1/0/0 15.7ms ± 1% 15.8ms ± 1% ~ (p=0.109 n=19+16) BM_CommentLines/4/0/0 19.8ms ± 1% 20.2ms ± 1% +2.12% (p=0.000 n=20+19) BM_CommentLines/128/0/0 208ms ± 1% 221ms ± 1% +6.29% (p=0.000 n=20+19) BM_CommentLines/1/30/0 16.6ms ± 1% 16.6ms ± 1% ~ (p=0.354 n=19+19) BM_CommentLines/4/30/0 25.8ms ± 2% 26.1ms ± 1% +1.39% (p=0.000 n=20+19) BM_CommentLines/128/30/0 222ms ± 2% 232ms ± 1% +4.46% (p=0.000 n=20+18) BM_CommentLines/1/70/0 19.2ms ± 2% 19.1ms ± 1% ~ (p=0.478 n=20+19) BM_CommentLines/4/70/0 27.6ms ± 2% 27.9ms ± 1% +0.95% (p=0.001 n=20+18) BM_CommentLines/128/70/0 245ms ± 2% 250ms ± 1% +1.96% (p=0.000 n=20+18) BM_CommentLines/1/0/2 16.0ms ± 1% 15.9ms ± 2% -0.68% (p=0.015 n=15+18) BM_CommentLines/4/0/2 20.6ms ± 1% 20.5ms ± 1% -0.53% (p=0.024 n=18+19) BM_CommentLines/128/0/2 216ms ± 2% 213ms ± 1% -1.70% (p=0.000 n=17+19) BM_CommentLines/1/30/2 16.9ms ± 2% 16.8ms ± 3% ~ (p=0.072 n=20+20) BM_CommentLines/4/30/2 26.8ms ± 2% 26.5ms ± 1% -0.91% (p=0.000 n=20+19) BM_CommentLines/128/30/2 244ms ± 2% 238ms ± 1% -2.30% (p=0.000 n=19+17) BM_CommentLines/1/70/2 19.4ms ± 2% 19.4ms ± 2% ~ (p=0.059 n=18+20) BM_CommentLines/4/70/2 28.4ms ± 2% 28.2ms ± 1% -0.81% (p=0.003 n=20+19) BM_CommentLines/128/70/2 261ms ± 2% 255ms ± 1% -2.33% (p=0.000 n=20+18) BM_CommentLines/1/0/8 16.2ms ± 2% 16.2ms ± 2% ~ (p=0.904 n=20+20) BM_CommentLines/4/0/8 25.0ms ± 1% 22.6ms ± 1% -9.40% (p=0.000 n=20+19) BM_CommentLines/128/0/8 246ms ± 2% 244ms ± 1% -1.14% (p=0.000 n=20+19) BM_CommentLines/1/30/8 17.3ms ± 2% 17.3ms ± 1% ~ (p=1.000 n=19+18) BM_CommentLines/4/30/8 30.3ms ± 2% 27.9ms ± 1% -7.83% (p=0.000 n=20+20) BM_CommentLines/128/30/8 277ms ± 3% 271ms ± 1% -2.26% (p=0.000 n=20+17) BM_CommentLines/1/70/8 19.9ms ± 2% 19.9ms ± 2% ~ (p=0.687 n=19+20) BM_CommentLines/4/70/8 31.7ms ± 2% 29.3ms ± 1% -7.55% (p=0.000 n=20+18) BM_CommentLines/128/70/8 296ms ± 2% 291ms ± 1% -1.50% (p=0.000 n=20+18) ``` |
||
|
Richard SmithandChandler Carruth
|
e4caf7d604 |
Compute and cache the value representation of a type when it becomes complete. (#3271)
Using the computed value representation, fix lowering of struct and tuple values to use the value representation rather than the object representation. Fixes an issue found in the review of #3257. This currently causes us to compute value representations of all types as they are created, which generates substantially more SemIR to represent types. We can get some of that back by deferring computation of the value representation until the type is required to be complete, but some of the additional cost here will persist with this approach. I also considered making the computation of the value representation type be something that lives entirely within the lowering phase, but I think that's not the right approach in the longer term, because the value representation will be semantically visible and relevant once we start allowing it to be customized. We should consider moving the nodes that exist to compute canonical non-local types, including value representations, out into a separate global block. That will clean up the SemIR representation substantially, and make the SemIR produced for a function not depend on which types we happen to have encountered beforehand. But that's not being done in this PR. --------- Co-authored-by: Chandler Carruth <chandlerc@gmail.com> |
||
|
Richard Smith
|
1ae5fe0cd5 |
Provide the callee expression to the Call node. (#3291)
Track the callee expression in full, instead of only tracking the callee's FunctionId. This results in the `name_reference` denoting the function actually being used. Lowering now propagates a `llvm::Function*` as the value associated with expressions of type `<function>`. We were not creating `NameReference` node for names produced by member access into a namespace, such as the second name in `Namespace.Function`, which caused lowering of calls to such names to fail. This is now fixed, but the resulting `NameReference` node only refers to the name and the lookup result, not to the `Namespace.` qualifier. We'll need to decide how to fit a third operand into that node (perhaps we can stop storing the `name_id`, since it can be derived from the lookup result) but for now the qualifier is not tracked. |
||
|
Richard Smith
|
c61b6a0e1d |
Rename //toolchain/sem_ir:file_test -> :yaml_test. (#3293)
While this test is testing //toolchain/sem_ir:file, the interesting thing about it is that it's testing the YAML output from File printing. Calling this test `:file_test` is also deeply confusing because that's also the name we give to all of our tests that are testing the files in `testdata/`, which is not what this test does. |
||
|
josh11bandRichard Smith
|
a112f2e802 |
Validate parse nodes correspond to expected tokens (#3295)
Can specify which tokens are allowed generally, and any additional tokens that only occur when the parse node has an error. --------- Co-authored-by: Richard Smith <richard@metafoo.co.uk> |
||
|
Chandler Carruth
|
95a1cc8cba |
Tweak lexer to improve generated code. (#3296)
This is a collection of tweaks and they are ones I'm less confident in FWIW. I set out to make the generated code cleaner and reduce loading pointers through pointers in a bunch of cases. It also works to reduce the working-set-size, and reduce the set of mutated values on each iteration. But I didn't benchmark at each step and it's not clear that incremental benchmarks will even be meaningful, so its hard to say which tweaks were load bearing and which weren't. For example, with this, the core lexer dispatch loop doesn't mutate anything in memory -- the position is passed in register and incremented in register throughout. The pointer to the source text, the size, and the pointer to the lexer are also passed in registers. By making the tokenized buffer a direct member of the lexer, all of its members can be accessed at a constant offset from the lexer pointer itself. The cost of this is a move at the end of lexing, but especially as the buffer size gets large this seems like a trivial cost compared to the previous double-indirection. This uses a (significantly) cheaper representation for the line index -- the `Line` type is optimized for dense *storage*, and is not a great type for using in a tight loop like the lexer. Once using a good index and once the line table is readily available from the above, we can simply index the line table rather than store (and update) a separate pointer. Last but not least, this removes the column as suggested in a previous review. Now it is computed from the position and the line information. My macro benchmark looks like it improves in the 2% - 5% range, but its getting into the noise sadly (or maybe this is good?). On a quiet AMD server with 20 runs before/after I get reasonably compelling across-the-board improvements on all the microbenchmarks, including 5% on `RandomSource` which is key: ``` BM_ValidKeywords 3.44ms ± 1% 3.11ms ± 1% -9.52% (p=0.000 n=19+20) BM_ValidIdentifiers<1, 64, false> 11.2ms ± 1% 10.8ms ± 2% -3.23% (p=0.000 n=19+20) BM_ValidIdentifiers<1, 1, true> 3.96ms ± 1% 3.72ms ± 1% -6.10% (p=0.000 n=18+20) BM_ValidIdentifiers<3, 5, true> 13.5ms ± 1% 13.2ms ± 3% -2.45% (p=0.000 n=19+20) BM_ValidIdentifiers<3, 16, true> 13.5ms ± 2% 13.1ms ± 2% -2.81% (p=0.000 n=19+20) BM_ValidIdentifiers<12, 64, true> 15.6ms ± 1% 15.2ms ± 3% -2.41% (p=0.000 n=19+20) BM_HorizontalWhitespace/1 13.6ms ± 2% 13.3ms ± 3% -2.34% (p=0.000 n=19+20) BM_HorizontalWhitespace/4 13.9ms ± 2% 13.6ms ± 3% -2.06% (p=0.000 n=19+20) BM_HorizontalWhitespace/16 14.5ms ± 2% 14.3ms ± 2% -1.17% (p=0.002 n=19+20) BM_HorizontalWhitespace/64 18.5ms ± 3% 18.1ms ± 2% -2.17% (p=0.000 n=19+19) BM_HorizontalWhitespace/128 24.7ms ± 1% 24.4ms ± 2% -1.29% (p=0.000 n=18+20) BM_RandomSource 8.38ms ± 2% 7.92ms ± 2% -5.50% (p=0.000 n=19+20) BM_GroupingSymbols/1/0/0 6.63ms ± 2% 6.30ms ± 2% -4.97% (p=0.000 n=19+20) BM_GroupingSymbols/2/0/0 5.65ms ± 2% 5.27ms ± 2% -6.75% (p=0.000 n=19+20) BM_GroupingSymbols/3/0/0 4.51ms ± 1% 4.12ms ± 1% -8.62% (p=0.000 n=19+18) BM_GroupingSymbols/4/0/0 4.23ms ± 1% 3.82ms ± 1% -9.65% (p=0.000 n=19+18) BM_GroupingSymbols/8/0/0 3.50ms ± 1% 3.08ms ± 1% -12.21% (p=0.000 n=19+19) BM_GroupingSymbols/16/0/0 3.38ms ± 1% 2.97ms ± 1% -11.98% (p=0.000 n=19+19) BM_GroupingSymbols/32/0/0 4.10ms ± 1% 3.69ms ± 1% -10.02% (p=0.000 n=17+19) BM_GroupingSymbols/0/1/0 5.85ms ± 2% 5.70ms ± 1% -2.54% (p=0.000 n=19+20) BM_GroupingSymbols/0/2/0 4.60ms ± 2% 4.46ms ± 1% -3.12% (p=0.000 n=19+19) BM_GroupingSymbols/0/3/0 3.32ms ± 2% 3.18ms ± 1% -4.21% (p=0.000 n=18+20) BM_GroupingSymbols/0/4/0 2.94ms ± 2% 2.82ms ± 1% -4.22% (p=0.000 n=19+20) BM_GroupingSymbols/0/8/0 1.95ms ± 1% 1.83ms ± 1% -6.47% (p=0.000 n=17+19) BM_GroupingSymbols/0/16/0 1.40ms ± 1% 1.27ms ± 1% -9.12% (p=0.000 n=19+20) BM_GroupingSymbols/0/32/0 1.19ms ± 1% 1.05ms ± 1% -11.50% (p=0.000 n=19+20) BM_GroupingSymbols/0/0/1 5.88ms ± 2% 5.71ms ± 1% -2.87% (p=0.000 n=19+20) BM_GroupingSymbols/0/0/2 4.62ms ± 1% 4.45ms ± 2% -3.58% (p=0.000 n=18+20) BM_GroupingSymbols/0/0/3 3.32ms ± 1% 3.17ms ± 1% -4.72% (p=0.000 n=16+17) BM_GroupingSymbols/0/0/4 2.95ms ± 1% 2.81ms ± 2% -4.50% (p=0.000 n=16+19) BM_GroupingSymbols/0/0/8 1.96ms ± 1% 1.82ms ± 2% -6.80% (p=0.000 n=19+20) BM_GroupingSymbols/0/0/16 1.40ms ± 1% 1.27ms ± 1% -8.96% (p=0.000 n=19+20) BM_GroupingSymbols/0/0/32 1.19ms ± 1% 1.05ms ± 1% -11.64% (p=0.000 n=19+20) BM_GroupingSymbols/32/1/0 4.04ms ± 1% 3.60ms ± 1% -10.81% (p=0.000 n=17+20) BM_GroupingSymbols/32/2/0 3.93ms ± 1% 3.51ms ± 1% -10.55% (p=0.000 n=18+20) BM_GroupingSymbols/32/3/0 3.84ms ± 1% 3.44ms ± 1% -10.38% (p=0.000 n=18+19) BM_GroupingSymbols/32/4/0 3.76ms ± 2% 3.38ms ± 1% -10.21% (p=0.000 n=19+20) BM_GroupingSymbols/32/8/0 3.51ms ± 1% 3.14ms ± 1% -10.61% (p=0.000 n=19+19) BM_GroupingSymbols/32/16/0 3.11ms ± 1% 2.77ms ± 1% -10.95% (p=0.000 n=19+20) BM_GroupingSymbols/32/32/0 2.54ms ± 2% 2.25ms ± 1% -11.77% (p=0.000 n=19+16) BM_GroupingSymbols/32/32/1 2.50ms ± 1% 2.22ms ± 2% -11.19% (p=0.000 n=19+20) BM_GroupingSymbols/32/32/2 2.48ms ± 1% 2.21ms ± 1% -11.15% (p=0.000 n=19+20) BM_GroupingSymbols/32/32/3 2.46ms ± 1% 2.18ms ± 1% -11.11% (p=0.000 n=18+20) BM_GroupingSymbols/32/32/4 2.43ms ± 1% 2.16ms ± 1% -11.04% (p=0.000 n=19+20) BM_GroupingSymbols/32/32/8 2.35ms ± 1% 2.09ms ± 1% -11.11% (p=0.000 n=19+20) BM_GroupingSymbols/32/32/16 2.20ms ± 1% 1.96ms ± 1% -11.30% (p=0.000 n=19+18) BM_GroupingSymbols/32/32/32 1.98ms ± 1% 1.75ms ± 1% -11.54% (p=0.000 n=19+19) BM_BlankLines/1 14.6ms ± 1% 14.2ms ± 3% -2.53% (p=0.000 n=19+19) BM_BlankLines/4 18.3ms ± 1% 17.4ms ± 2% -4.68% (p=0.000 n=19+20) BM_BlankLines/16 34.9ms ± 2% 31.6ms ± 3% -9.42% (p=0.000 n=19+20) BM_BlankLines/64 102ms ± 2% 91ms ± 3% -10.75% (p=0.000 n=19+20) BM_BlankLines/128 190ms ± 2% 169ms ± 3% -10.82% (p=0.000 n=19+20) BM_CommentLines/1/0/0 16.2ms ± 1% 15.8ms ± 2% -2.29% (p=0.000 n=19+19) BM_CommentLines/4/0/0 21.0ms ± 1% 19.9ms ± 2% -5.17% (p=0.000 n=19+19) BM_CommentLines/128/0/0 237ms ± 1% 211ms ± 2% -10.93% (p=0.000 n=19+20) BM_CommentLines/1/30/0 17.0ms ± 1% 16.8ms ± 3% -1.36% (p=0.001 n=17+20) BM_CommentLines/4/30/0 27.2ms ± 2% 26.0ms ± 2% -4.24% (p=0.000 n=19+20) BM_CommentLines/128/30/0 255ms ± 2% 223ms ± 2% -12.58% (p=0.000 n=19+20) BM_CommentLines/1/70/0 19.7ms ± 1% 19.2ms ± 1% -2.35% (p=0.000 n=18+18) BM_CommentLines/4/70/0 29.0ms ± 2% 27.6ms ± 1% -4.81% (p=0.000 n=18+17) BM_CommentLines/128/70/0 273ms ± 7% 249ms ± 3% -9.05% (p=0.000 n=20+20) BM_CommentLines/1/0/2 16.4ms ± 2% 16.0ms ± 1% -2.49% (p=0.000 n=19+15) BM_CommentLines/4/0/2 22.0ms ± 2% 20.7ms ± 1% -5.84% (p=0.000 n=19+17) BM_CommentLines/128/0/2 256ms ± 2% 218ms ± 1% -14.88% (p=0.000 n=19+17) BM_CommentLines/1/30/2 17.4ms ± 1% 17.0ms ± 2% -2.26% (p=0.000 n=19+19) BM_CommentLines/4/30/2 28.5ms ± 1% 26.9ms ± 2% -5.67% (p=0.000 n=19+18) BM_CommentLines/128/30/2 288ms ± 5% 246ms ± 3% -14.70% (p=0.000 n=19+20) BM_CommentLines/1/70/2 19.9ms ± 1% 19.4ms ± 1% -2.56% (p=0.000 n=19+18) BM_CommentLines/4/70/2 30.0ms ± 2% 28.5ms ± 2% -5.20% (p=0.000 n=19+19) BM_CommentLines/128/70/2 300ms ± 3% 265ms ± 3% -11.69% (p=0.000 n=18+20) BM_CommentLines/1/0/8 16.7ms ± 2% 16.3ms ± 2% -2.22% (p=0.000 n=19+19) BM_CommentLines/4/0/8 26.4ms ± 1% 25.2ms ± 1% -4.42% (p=0.000 n=19+18) BM_CommentLines/128/0/8 285ms ± 2% 248ms ± 1% -13.16% (p=0.000 n=19+18) BM_CommentLines/1/30/8 17.9ms ± 2% 17.4ms ± 1% -2.99% (p=0.000 n=19+17) BM_CommentLines/4/30/8 32.0ms ± 1% 30.5ms ± 2% -4.73% (p=0.000 n=19+20) BM_CommentLines/128/30/8 320ms ± 2% 280ms ± 3% -12.52% (p=0.000 n=18+20) BM_CommentLines/1/70/8 20.7ms ± 2% 20.0ms ± 2% -3.22% (p=0.000 n=19+19) BM_CommentLines/4/70/8 33.5ms ± 2% 31.8ms ± 1% -5.23% (p=0.000 n=19+19) BM_CommentLines/128/70/8 338ms ± 3% 298ms ± 2% -11.64% (p=0.000 n=19+20) ``` |
||
|
Richard Smith
|
4e64b1948d |
Bare-bones support for forward-declared classes. (#3294)
This is mostly scaffolding, but is just about enough for pointers to classes to work properly as types. |
||
|
Chandler Carruth
|
0e2b6c7f1a |
Optimize runs of horizontal whitespace. (#3288)
So, this is a somewhat fun, simple improvement. =] Just use a loop and count runs of whitespace. I didn't even work all that hard to make the loop fast, but it seems great. Makes long runs of indentation more than 2x faster at basically no code complexity. I thought about doing this for vertical whitespace as well but it's not easy to do, and didn't seem worth adding complexity. Huge runs of blank lines aren't nearly as common as lots of indentation. I do have a plan for an analogous optimization for comment blocks, but want to simplify some other code first. We could also make this (hilariously) faster with some judicious use of SIMD or clever use of a string function, but it doesn't seem worth it given how fast the simple loop is already. I didn't work to get a high N count and so there's plenty of noise here, but the benchmark data speaks for itself: ``` BM_ValidKeywords 2.71ms ± 1% 2.75ms ± 1% +1.32% (p=0.016 n=5+5) BM_ValidIdentifiers<1, 64, false> 9.71ms ± 2% 9.74ms ± 1% ~ (p=1.000 n=5+5) BM_ValidIdentifiers<1, 1, true> 3.17ms ± 1% 3.22ms ± 2% ~ (p=0.151 n=5+5) BM_ValidIdentifiers<3, 5, true> 11.5ms ± 2% 11.6ms ± 3% ~ (p=0.548 n=5+5) BM_ValidIdentifiers<3, 16, true> 11.3ms ± 3% 11.6ms ± 3% ~ (p=0.095 n=5+5) BM_ValidIdentifiers<12, 64, true> 12.8ms ± 1% 12.8ms ± 1% ~ (p=1.000 n=5+5) BM_HorizontalWhitespace/1 11.6ms ± 1% 11.7ms ± 2% ~ (p=0.310 n=5+5) BM_HorizontalWhitespace/4 12.9ms ± 1% 11.8ms ± 0% -8.50% (p=0.008 n=5+5) BM_HorizontalWhitespace/16 17.3ms ± 4% 12.5ms ± 1% -27.69% (p=0.008 n=5+5) BM_HorizontalWhitespace/64 28.9ms ± 3% 16.0ms ± 2% -44.88% (p=0.008 n=5+5) BM_HorizontalWhitespace/128 47.6ms ± 3% 21.0ms ± 0% -55.88% (p=0.016 n=5+4) BM_RandomSource 7.92ms ± 1% 7.70ms ± 3% -2.74% (p=0.016 n=5+5) BM_GroupingSymbols/1/0/0 5.92ms ± 2% 5.86ms ± 1% ~ (p=0.310 n=5+5) BM_GroupingSymbols/2/0/0 5.30ms ± 1% 5.01ms ± 1% -5.52% (p=0.008 n=5+5) BM_GroupingSymbols/3/0/0 4.48ms ± 0% 3.95ms ± 1% -11.69% (p=0.008 n=5+5) BM_GroupingSymbols/4/0/0 4.61ms ± 1% 3.73ms ± 1% -19.12% (p=0.008 n=5+5) BM_GroupingSymbols/8/0/0 5.34ms ± 6% 3.05ms ± 1% -42.82% (p=0.008 n=5+5) BM_GroupingSymbols/16/0/0 6.44ms ± 1% 3.20ms ± 2% -50.24% (p=0.008 n=5+5) BM_GroupingSymbols/32/0/0 10.3ms ± 5% 4.2ms ± 1% -59.81% (p=0.008 n=5+5) BM_GroupingSymbols/0/1/0 5.17ms ± 2% 5.17ms ± 1% ~ (p=0.690 n=5+5) BM_GroupingSymbols/0/2/0 4.04ms ± 1% 4.04ms ± 2% ~ (p=1.000 n=5+5) BM_GroupingSymbols/0/3/0 2.89ms ± 1% 2.89ms ± 1% ~ (p=1.000 n=5+5) BM_GroupingSymbols/0/4/0 2.54ms ± 1% 2.55ms ± 1% ~ (p=0.421 n=5+5) BM_GroupingSymbols/0/8/0 1.65ms ± 2% 1.67ms ± 1% ~ (p=0.310 n=5+5) BM_GroupingSymbols/0/16/0 1.18ms ± 1% 1.19ms ± 2% ~ (p=0.222 n=5+5) BM_GroupingSymbols/0/32/0 948µs ± 1% 957µs ± 2% ~ (p=0.310 n=5+5) BM_GroupingSymbols/0/0/1 5.17ms ± 1% 5.16ms ± 2% ~ (p=0.841 n=5+5) BM_GroupingSymbols/0/0/2 4.03ms ± 2% 4.04ms ± 2% ~ (p=0.548 n=5+5) BM_GroupingSymbols/0/0/3 2.89ms ± 1% 2.88ms ± 1% ~ (p=0.841 n=5+5) BM_GroupingSymbols/0/0/4 2.54ms ± 1% 2.54ms ± 2% ~ (p=0.548 n=5+5) BM_GroupingSymbols/0/0/8 1.66ms ± 3% 1.67ms ± 2% ~ (p=0.690 n=5+5) BM_GroupingSymbols/0/0/16 1.19ms ± 1% 1.18ms ± 2% ~ (p=0.548 n=5+5) BM_GroupingSymbols/0/0/32 949µs ± 2% 955µs ± 1% ~ (p=0.310 n=5+5) BM_GroupingSymbols/32/1/0 10.0ms ± 1% 4.0ms ± 1% -59.76% (p=0.008 n=5+5) BM_GroupingSymbols/32/2/0 9.73ms ± 2% 3.92ms ± 2% -59.73% (p=0.008 n=5+5) BM_GroupingSymbols/32/3/0 9.40ms ± 2% 3.84ms ± 3% -59.10% (p=0.008 n=5+5) BM_GroupingSymbols/32/4/0 9.20ms ± 2% 3.74ms ± 1% -59.35% (p=0.008 n=5+5) BM_GroupingSymbols/32/8/0 8.38ms ± 2% 3.50ms ± 1% -58.22% (p=0.008 n=5+5) BM_GroupingSymbols/32/16/0 7.18ms ± 2% 3.06ms ± 3% -57.41% (p=0.008 n=5+5) BM_GroupingSymbols/32/32/0 5.53ms ± 1% 2.44ms ± 2% -55.99% (p=0.008 n=5+5) BM_GroupingSymbols/32/32/1 5.47ms ± 2% 2.40ms ± 2% -56.08% (p=0.008 n=5+5) BM_GroupingSymbols/32/32/2 5.41ms ± 3% 2.40ms ± 2% -55.66% (p=0.008 n=5+5) BM_GroupingSymbols/32/32/3 5.28ms ± 2% 2.37ms ± 2% -55.18% (p=0.008 n=5+5) BM_GroupingSymbols/32/32/4 5.25ms ± 2% 2.34ms ± 2% -55.52% (p=0.008 n=5+5) BM_GroupingSymbols/32/32/8 5.03ms ± 3% 2.25ms ± 1% -55.29% (p=0.008 n=5+5) BM_GroupingSymbols/32/32/16 4.62ms ± 1% 2.15ms ± 3% -53.41% (p=0.008 n=5+5) BM_GroupingSymbols/32/32/32 3.99ms ± 1% 1.86ms ± 1% -53.24% (p=0.008 n=5+5) BM_BlankLines/1 12.8ms ± 1% 12.6ms ± 2% ~ (p=0.310 n=5+5) BM_BlankLines/4 16.0ms ± 3% 15.8ms ± 1% ~ (p=0.310 n=5+5) BM_BlankLines/16 33.1ms ± 2% 32.1ms ± 1% -3.29% (p=0.008 n=5+5) BM_BlankLines/64 84.6ms ± 3% 84.2ms ± 3% ~ (p=0.690 n=5+5) BM_BlankLines/128 159ms ± 4% 155ms ± 3% ~ (p=0.310 n=5+5) BM_CommentLines/1/0/0 14.5ms ± 2% 14.5ms ± 2% ~ (p=0.690 n=5+5) BM_CommentLines/4/0/0 19.0ms ± 1% 19.0ms ± 1% ~ (p=0.548 n=5+5) BM_CommentLines/128/0/0 188ms ± 2% 186ms ± 1% ~ (p=0.151 n=5+5) BM_CommentLines/1/30/0 14.8ms ± 3% 14.7ms ± 2% ~ (p=0.421 n=5+5) BM_CommentLines/4/30/0 21.8ms ± 1% 21.3ms ± 4% ~ (p=0.095 n=5+5) BM_CommentLines/128/30/0 201ms ± 1% 200ms ± 2% ~ (p=0.421 n=5+5) BM_CommentLines/1/70/0 15.5ms ± 3% 15.2ms ± 4% ~ (p=0.095 n=5+5) BM_CommentLines/4/70/0 23.2ms ± 1% 22.5ms ± 2% -2.76% (p=0.008 n=5+5) BM_CommentLines/128/70/0 213ms ± 1% 209ms ± 1% -1.54% (p=0.016 n=5+5) BM_CommentLines/1/0/2 15.3ms ± 1% 14.5ms ± 1% -4.99% (p=0.008 n=5+5) BM_CommentLines/4/0/2 21.4ms ± 1% 20.2ms ± 2% -5.48% (p=0.008 n=5+5) BM_CommentLines/128/0/2 242ms ± 2% 218ms ± 5% -10.13% (p=0.008 n=5+5) BM_CommentLines/1/30/2 15.7ms ± 4% 15.1ms ± 2% -3.37% (p=0.008 n=5+5) BM_CommentLines/4/30/2 24.3ms ± 1% 22.5ms ± 3% -7.42% (p=0.008 n=5+5) BM_CommentLines/128/30/2 268ms ± 2% 240ms ± 3% -10.22% (p=0.008 n=5+5) BM_CommentLines/1/70/2 16.1ms ± 2% 15.3ms ± 3% -5.24% (p=0.008 n=5+5) BM_CommentLines/4/70/2 25.7ms ± 3% 24.0ms ± 3% -6.69% (p=0.008 n=5+5) BM_CommentLines/128/70/2 272ms ± 1% 247ms ± 1% -9.21% (p=0.008 n=5+5) BM_CommentLines/1/0/8 17.2ms ± 5% 14.8ms ± 2% -14.32% (p=0.008 n=5+5) BM_CommentLines/4/0/8 30.4ms ± 2% 20.8ms ± 2% -31.47% (p=0.008 n=5+5) BM_CommentLines/128/0/8 463ms ± 1% 260ms ± 1% -43.89% (p=0.008 n=5+5) BM_CommentLines/1/30/8 17.2ms ± 2% 15.2ms ± 2% -12.01% (p=0.008 n=5+5) BM_CommentLines/4/30/8 32.4ms ± 1% 23.2ms ± 3% -28.61% (p=0.008 n=5+5) BM_CommentLines/128/30/8 498ms ± 3% 287ms ± 2% -42.32% (p=0.008 n=5+5) BM_CommentLines/1/70/8 17.6ms ± 2% 15.5ms ± 3% -11.61% (p=0.008 n=5+5) BM_CommentLines/4/70/8 34.0ms ± 4% 24.7ms ± 3% -27.31% (p=0.008 n=5+5) BM_CommentLines/128/70/8 497ms ± 2% 297ms ± 4% -40.29% (p=0.008 n=5+5) ``` Stacked on top of #3287 -- only the last commit should be reviewed here. |
||
|
Richard Smith
|
870ccf3eac |
Suppress follow-on errors for various forms of expression. (#3292)
Don't produce an error diagnostic if any of the information leading to identifying that error is itself known to be affected by an already-diagnosed error. |
||
|
Chandler Carruth
|
a79ea4b28d |
Do more precise table dispatch for symbols. (#3287)
When originally switching to the table dispatch approach we discussed that it'd be nice to disentangle the monolithic symbol lexing routine with this as we'll typically have fairly precise dispatch. This is especially true for grouping symbols, which in Carbon are all constructively one-character (at this point). I think this provides a substantial improvement to the clarity of the code by disentangling the different paths. It also allowed a bunch of simplifications / clarifications to exactly what the behavior with closing invalid groups actually involves currently. This was initially motivated by code organization improvements, and any performance wins were speculative. However, when benchmarking it surfaced a problem that hadn't been clear -- we're generating too many distinct functions here, and the table-based dispatch slows down in the face of that. So this PR also includes a fix for that, removing the template-generated fan-out of dispatch functions for distinct symbols. Instead, we have a dedicated table to translate one character into the token kinds. This seems to work quite well, avoiding the huge branch-y structure and just do fairly cheap table translation & dispatch for all one-character symbols. Building the table requires the token kinds to be default constructable, so this also enables that and arranges for the zero-value kind to be the error kind. Combined, this is a modest speedup for *non* grouping symbols (3-4%, a bit noisy). And in some cases it is a huge speedup for grouping symbols (>10%). Raw benchmark data with 20 runs before/after -- despite the # of runs, the grouping symbols benchmarks were frustratingly noisy in non-uniform ways that couldn't fully be accounted for here. Still, this seems like an overall improvement. ``` BM_RandomSource 7.98ms ± 2% 7.73ms ± 3% -3.12% (p=0.000 n=18+19) BM_GroupingSymbols/1/0/0 5.90ms ± 2% 5.82ms ± 4% -1.38% (p=0.001 n=20+20) BM_GroupingSymbols/2/0/0 5.21ms ± 2% 5.15ms ± 2% -1.14% (p=0.002 n=20+18) BM_GroupingSymbols/3/0/0 4.42ms ± 2% 4.34ms ± 2% -1.87% (p=0.000 n=19+18) BM_GroupingSymbols/4/0/0 4.29ms ± 2% 4.38ms ± 5% ~ (p=0.297 n=17+20) BM_GroupingSymbols/8/0/0 5.09ms ±10% 5.10ms ± 7% ~ (p=0.919 n=18+20) BM_GroupingSymbols/16/0/0 6.35ms ± 8% 6.29ms ± 6% ~ (p=0.201 n=20+20) BM_GroupingSymbols/32/0/0 9.88ms ± 2% 9.83ms ± 1% ~ (p=0.167 n=18+20) BM_GroupingSymbols/0/1/0 5.12ms ± 2% 5.01ms ± 2% -2.14% (p=0.000 n=20+19) BM_GroupingSymbols/0/2/0 4.01ms ± 2% 3.93ms ± 4% -2.03% (p=0.000 n=20+19) BM_GroupingSymbols/0/3/0 2.92ms ± 3% 2.81ms ± 2% -3.87% (p=0.000 n=20+19) BM_GroupingSymbols/0/4/0 2.61ms ± 3% 2.47ms ± 2% -5.30% (p=0.000 n=20+18) BM_GroupingSymbols/0/8/0 1.77ms ± 3% 1.61ms ± 2% -8.91% (p=0.000 n=18+19) BM_GroupingSymbols/0/16/0 1.41ms ± 3% 1.16ms ± 4% -17.66% (p=0.000 n=20+20) BM_GroupingSymbols/0/32/0 1.10ms ± 2% 0.92ms ± 3% -16.36% (p=0.000 n=20+17) BM_GroupingSymbols/0/0/1 5.09ms ± 2% 5.03ms ± 3% -1.11% (p=0.001 n=20+18) BM_GroupingSymbols/0/0/2 4.01ms ± 2% 3.91ms ± 2% -2.67% (p=0.000 n=20+18) BM_GroupingSymbols/0/0/3 2.93ms ± 3% 2.81ms ± 2% -4.23% (p=0.000 n=20+19) BM_GroupingSymbols/0/0/4 2.59ms ± 2% 2.48ms ± 3% -4.48% (p=0.000 n=20+19) BM_GroupingSymbols/0/0/8 1.75ms ± 1% 1.62ms ± 3% -7.65% (p=0.000 n=17+19) BM_GroupingSymbols/0/0/16 1.40ms ± 2% 1.15ms ± 3% -17.67% (p=0.000 n=19+20) BM_GroupingSymbols/0/0/32 1.10ms ± 2% 0.92ms ± 3% -15.91% (p=0.000 n=20+19) BM_GroupingSymbols/32/1/0 9.62ms ± 2% 9.65ms ± 2% ~ (p=0.654 n=18+20) BM_GroupingSymbols/32/2/0 9.41ms ± 2% 9.37ms ± 2% ~ (p=0.095 n=20+19) BM_GroupingSymbols/32/3/0 9.13ms ± 2% 9.13ms ± 3% ~ (p=0.687 n=19+20) BM_GroupingSymbols/32/4/0 8.93ms ± 1% 8.87ms ± 2% -0.69% (p=0.010 n=20+18) BM_GroupingSymbols/32/8/0 8.15ms ± 2% 8.14ms ± 3% ~ (p=0.729 n=19+19) BM_GroupingSymbols/32/16/0 7.04ms ± 3% 6.92ms ± 1% -1.71% (p=0.000 n=20+18) BM_GroupingSymbols/32/32/0 5.48ms ± 2% 5.38ms ± 3% -1.81% (p=0.000 n=20+20) BM_GroupingSymbols/32/32/1 5.39ms ± 2% 5.29ms ± 2% -1.87% (p=0.000 n=19+19) BM_GroupingSymbols/32/32/2 5.34ms ± 2% 5.21ms ± 1% -2.45% (p=0.000 n=20+18) BM_GroupingSymbols/32/32/3 5.27ms ± 3% 5.16ms ± 2% -2.18% (p=0.000 n=20+19) BM_GroupingSymbols/32/32/4 5.21ms ± 2% 5.10ms ± 3% -2.11% (p=0.000 n=19+20) BM_GroupingSymbols/32/32/8 4.98ms ± 2% 4.83ms ± 2% -2.85% (p=0.000 n=19+19) BM_GroupingSymbols/32/32/16 4.55ms ± 2% 4.45ms ± 2% -2.25% (p=0.000 n=18+20) BM_GroupingSymbols/32/32/32 3.95ms ± 2% 3.84ms ± 2% -2.98% (p=0.000 n=19+20) ``` |
||
|
Richard Smith
|
17e053798c |
Fix incorrect argc calculation. (#3290)
Fix off-by-one error: the terminating `nullptr` in `argv` is not included in `argc`. This is currently causing crashing tests to crash again in their crash handler, meaning we don't get a backtrace or CHECK-failure message. |
||
|
Chandler Carruth
|
9ce9572522 |
Add a benchmark to stress test grouping symbols. (#3286)
This isn't really representative of anything, but it should help make it obvious when the handling of grouping symbols improves or regresses. Notably, the random source microbenchmark has *no* grouping symbols (in order to let it be random but always lexically valid), and so it's especially useful to have something that checks grouping symbols given their prevalence in realistic source code. |
||
|
josh11bandRichard Smith
|
8d0831f431 |
Made function and namespace nodes typed to remove a crash (#3285)
Bug found by fuzzing. Problem was untyped SemIR nodes had an invalid type id, which was retrieved by `HandlePrefixOperator` and then passed to `context.GetUnqualifiedType`, ultimately performing an invalid access in `semantics_ir_->GetNode`. We prefer to make a placeholder type for functions and namespaces to remove the need for checking for the untyped case everywhere. Eventually functions will have their own types, but this approach will be needed for namespaces (and perhaps other non-first-class entities like unbound methods and interface members) long term. Co-authored-by: Richard Smith <richard@metafoo.co.uk> |
||
|
|
6ba8712fbd |
Predetermine all the line splits in the lexer. (#3278)
## Summary ## Restructures the lexer to first scan the entire source text for newlines and create all the line structures needed. Doing this up-front makes it easy to produce an optimized version with minimal complexity. Currently, it leverages the system `memchr`, but even when expanded to handle more complex cases like CR+LF line endings, being isolated in this way will result in a significantly simpler implementation. This change improves the lexing of comment lines significantly by skipping their contents immediately. The overhead of the pre-scan is unmeasurable in all realistic benchmarks, and 10-30% in benchmarks consisting almost entirely of blank lines or comments. The improvement of comment lexing with average length comment lines mixed with code starts at 20% and goes up. Regressing blank line handling for non-empty comments seems like the right tradeoff (by far). ## Background and details ## One weak point in the lexer implementation were large runs of comments. While those aren't terribly common, they shouldn't present a hazard to the lexer performance. A bit more common is a pattern of comments like the following: ```carbon // Some method comment here. fn SomeMethodName(...) -> ...; // Some other method comment here. fn SomeOtherMethodName(...) -> ...; ``` Here, the lexer spends an inordinate amount of time getting from the `\n` after the first semicolon to the `fn` token. It has to skip a blank line, scan a line, find the `//` comment start, then scan to find the next `\n`, and then scan horizontal whitespace, etc. It is tempting to build a scanner *exactly* for this. In fact, I built one, and I can publish it in a PR if folks are interested in what it looks like. For x86-64, the PSHUFB trick used for scanning identifiers technically works. But it is *complicated*. Amazingly so. 150 lines of very subtle code with subtle performance pitfalls at ever turn. I felt very uncomfortable submitting it, but we can always go back to it. Nothing I've come up with quite matches it for sheer speed. However, most of the complexity and time is spent walking from a `//` to the end of the line. And *that* is something we can do very simply. In fact, there is a tuned function for that in libc: `memchr`. Using this we can build a very fast and much simpler scanner to split lines up-front. This PR uses that and a carefully crafted fast loop to first build up all the line info we need. Getting this to be as fast as possible required some other subtle changes, for example always creating a line structure that goes from the last `\n` and the end of the file. We then back up the EOF token to avoid surfacing this to users. The nice thing is the EOF token isn't part of any hot loop, and so this removes branches everywhere else at modest complexity. Once we have that, the rest of the lexer just needs to keep track of its current line in order to record column offsets. I've taken some care to try and optimize the lexer's usage of the line structures but there are more opportunities here I suspect. Combined, this gets much but not all of the performance of a huge SIMD scanner for newline-through-to-next-token. For extreme cases (100s of blank lines or empty comment lines between tokens) the holistic scanner is of course still much faster, but those don't seem nearly worth the cost. I was initially worried about the overhead of taking two passes over the source text, but in practice I've not been able to measure any appreciable cost to this with realistic source files. In some cases benchmarks with no newlines get *faster* because we use a much more efficient approach to fetch the source text into cache as a happenstance. And that in turn makes the byte-wise dispatched loop run faster as it stalls less. I'm particularly happy with this approach because it seems very clear how to extend this to support CR+LF, bare CR, and even complex mixtures without any significant speed cost. That wasn't at all true for the other approaches explored. I may try some further PRs to smooth out the last bits of slowness here, but already this is working excellent for me in practice. My 10mloc test case is down to 2.3s to lex. ## Raw benchmark data Using a tool that runs benchmarks before and after and analyzes the results, the following summarizes the CPU-time impact, each of these for lexing 100k tokens: ``` BM_ValidKeywords 2.57ms ± 1% 2.58ms ± 0% ~ (p=0.190 n=5+4) BM_ValidIdentifiers<1, 64, false> 9.24ms ± 4% 9.31ms ± 4% ~ (p=0.421 n=5+5) BM_ValidIdentifiers<1, 1, true> 3.05ms ± 4% 3.11ms ± 4% ~ (p=0.222 n=5+5) BM_ValidIdentifiers<3, 5, true> 10.9ms ± 0% 11.1ms ± 1% +1.76% (p=0.016 n=4+5) BM_ValidIdentifiers<3, 16, true> 11.1ms ± 7% 11.0ms ± 1% ~ (p=0.310 n=5+5) BM_ValidIdentifiers<12, 64, true> 12.2ms ± 1% 12.3ms ± 2% ~ (p=0.111 n=4+5) BM_HorizontalWhitespace/1 11.2ms ± 6% 11.1ms ± 2% ~ (p=0.841 n=5+5) BM_HorizontalWhitespace/4 12.0ms ± 3% 12.0ms ± 2% ~ (p=0.548 n=5+5) BM_HorizontalWhitespace/16 16.2ms ± 6% 15.9ms ± 8% ~ (p=0.690 n=5+5) BM_HorizontalWhitespace/64 27.7ms ± 3% 28.4ms ± 3% ~ (p=0.151 n=5+5) BM_HorizontalWhitespace/128 44.3ms ± 1% 45.6ms ± 6% +3.15% (p=0.032 n=5+5) BM_RandomSource 7.75ms ± 2% 7.72ms ± 1% ~ (p=1.000 n=5+5) BM_BlankLines/1 11.7ms ± 1% 12.1ms ± 1% +3.46% (p=0.008 n=5+5) BM_BlankLines/4 14.0ms ± 2% 15.2ms ± 3% +8.12% (p=0.008 n=5+5) BM_BlankLines/16 23.5ms ± 2% 31.1ms ± 4% +32.26% (p=0.008 n=5+5) BM_BlankLines/64 75.3ms ± 1% 81.2ms ± 3% +7.83% (p=0.008 n=5+5) BM_BlankLines/128 133ms ± 3% 150ms ± 2% +12.74% (p=0.008 n=5+5) BM_CommentLines/1/0/0 13.1ms ± 0% 13.7ms ± 1% +5.11% (p=0.008 n=5+5) BM_CommentLines/4/0/0 16.6ms ± 1% 18.2ms ± 4% +9.56% (p=0.008 n=5+5) BM_CommentLines/128/0/0 169ms ± 4% 182ms ± 1% +7.24% (p=0.008 n=5+5) BM_CommentLines/1/30/0 18.7ms ± 5% 14.1ms ± 0% -24.84% (p=0.008 n=5+5) BM_CommentLines/4/30/0 36.5ms ± 6% 20.6ms ± 3% -43.59% (p=0.008 n=5+5) BM_CommentLines/128/30/0 525ms ± 4% 198ms ± 1% -62.38% (p=0.008 n=5+5) BM_CommentLines/1/70/0 23.4ms ± 6% 14.7ms ± 2% -37.15% (p=0.008 n=5+5) BM_CommentLines/4/70/0 53.3ms ± 7% 22.4ms ± 4% -57.99% (p=0.008 n=5+5) BM_CommentLines/128/70/0 1.05s ± 4% 0.21s ± 2% -80.31% (p=0.008 n=5+5) BM_CommentLines/1/0/2 14.1ms ± 6% 14.3ms ± 1% ~ (p=0.151 n=5+5) BM_CommentLines/4/0/2 19.4ms ± 5% 20.1ms ± 1% ~ (p=0.151 n=5+5) BM_CommentLines/128/0/2 238ms ± 8% 229ms ± 0% ~ (p=0.151 n=5+5) BM_CommentLines/1/30/2 19.2ms ± 7% 14.6ms ± 1% -23.87% (p=0.008 n=5+5) BM_CommentLines/4/30/2 40.3ms ±13% 22.3ms ± 4% -44.63% (p=0.008 n=5+5) BM_CommentLines/128/30/2 568ms ± 7% 254ms ± 3% -55.28% (p=0.008 n=5+5) BM_CommentLines/1/70/2 23.3ms ± 1% 15.0ms ± 3% -35.61% (p=0.016 n=4+5) BM_CommentLines/4/70/2 57.2ms ± 9% 24.1ms ± 2% -57.81% (p=0.008 n=5+5) BM_CommentLines/128/70/2 1.07s ± 0% 0.26s ± 2% -75.51% (p=0.016 n=4+5) BM_CommentLines/1/0/8 15.9ms ± 7% 16.0ms ± 1% ~ (p=0.151 n=5+5) BM_CommentLines/4/0/8 24.2ms ± 6% 27.9ms ± 2% +15.36% (p=0.008 n=5+5) BM_CommentLines/128/0/8 386ms ± 4% 445ms ± 1% +15.28% (p=0.008 n=5+5) BM_CommentLines/1/30/8 20.6ms ± 5% 16.3ms ± 1% -20.95% (p=0.008 n=5+5) BM_CommentLines/4/30/8 45.3ms ± 6% 30.3ms ± 3% -32.98% (p=0.008 n=5+5) BM_CommentLines/128/30/8 699ms ± 3% 477ms ± 3% -31.83% (p=0.008 n=5+5) BM_CommentLines/1/70/8 25.7ms ± 5% 16.8ms ± 2% -34.67% (p=0.008 n=5+5) BM_CommentLines/4/70/8 62.0ms ± 4% 31.6ms ± 2% -49.10% (p=0.008 n=5+5) BM_CommentLines/128/70/8 1.20s ± 2% 0.48s ± 4% -59.60% (p=0.008 n=5+5) ``` The horizontal whitespace benchmark (and all of the non-line-oriented ones) are noisier than they appear here but do show some improvements (surprisingly). My guess is that it has a lot to do with system load, as the advantage is that we're using a vectorized loop to scan the text first and then doing the byte-dispatched loop. So when the cache is a bit slower to populate, the vectorized version starts to be faster. --------- Co-authored-by: Richard Smith <richard@metafoo.co.uk> Co-authored-by: josh11b <josh11b@users.noreply.github.com> |
||
|
|
5041a14f59 |
Add whitespace- and comment-specific benchmarking. (#3276)
These benchmarks zero-in and stress test horizontal and vertical whitespace as well as comment lexing performance. They set up essentially a worst-case scenario of ramping up whitespace between very sparse tokens to show how the lexer copes with this. The horizontal whitespace benchmark is perhaps less important as frequent runs of 50-characters of horizontal whitespace are relatively rare already, and likely to be exceedingly rare without trailing comments. But its good to include for completeness and it shows reasonably strong performance with the current table-dispatch approach. The blank line and comment line benchmarks are much more important. Lots of code is relatively line-sparse, especially API files that are perhaps the most useful to parse quickly. And many of these are a mixture of sparse with blank lines and sparse with large comment blocks. The benchmarks show that there are some serious limits here, even falling below 100k tokens per second throughput on some of the stress tests here. --------- Co-authored-by: Richard Smith <richard@metafoo.co.uk> Co-authored-by: josh11b <josh11b@users.noreply.github.com> |
||
|
Chandler CarruthandRichard Smith
|
2d735bbc51 |
Enhance the main lexer benchmark. (#3275)
This generalizes the main lexer benchmark's source generation to be a bit more comprehensive, specifically including whitespace and comments. Without these, we're missing a key part of the lexer's performance. This also tidies up a bit of the code and adds a more specific distribution of the different factors in lexing based on analysis of LLVM's source code. The enhancements to the source statistics script that helped collect the data here will be in a separate PR. With this, the output on my AMD cloud instance is: ``` ------------------------------------------------------------------------------------------------------------------------- Benchmark Time CPU Iterations bytes_per_second tokens_per_second ------------------------------------------------------------------------------------------------------------------------- BM_ValidKeywords 2809011 ns 2808912 ns 243 212.616M/s 35.601M/s BM_ValidIdentifiers<1, 64, false> 11461783 ns 11461652 ns 61 128.499M/s 8.72475M/s BM_ValidIdentifiers<1, 1, true> 3397293 ns 3397240 ns 208 84.2158M/s 29.4357M/s BM_ValidIdentifiers<3, 5, true> 14165143 ns 14164962 ns 51 40.3956M/s 7.05967M/s BM_ValidIdentifiers<3, 16, true> 15283128 ns 15282583 ns 47 71.7623M/s 6.5434M/s BM_ValidIdentifiers<12, 64, true> 17417323 ns 17417109 ns 41 219.007M/s 5.74148M/s ------------------------------------------------------------------------------------------------------------------------------------------ Benchmark Time CPU Iterations bytes_per_second lines_per_second tokens_per_second ------------------------------------------------------------------------------------------------------------------------------------------ BM_RandomSource 8132211 ns 8132227 ns 84 137.223M/s 3.9036M/s 12.2968M/s BM_SpeedOfLightStrCpy 29610 ns 29608 ns 24631 36.8065G/s 1072.17M/s 3.37745G/s BM_SpeedOfLightDispatch<1> 2144418 ns 2144421 ns 327 520.387M/s 14.8035M/s 46.6326M/s BM_SpeedOfLightDispatch<2> 1945954 ns 1945827 ns 351 573.498M/s 16.3144M/s 51.392M/s BM_SpeedOfLightDispatch<4> 2519565 ns 2519467 ns 292 442.923M/s 12.5999M/s 39.6909M/s BM_SpeedOfLightDispatch<8> 3011965 ns 3011968 ns 238 370.498M/s 10.5396M/s 33.2009M/s BM_SpeedOfLightDispatch<16> 4379575 ns 4379579 ns 160 254.803M/s 7.24841M/s 22.8332M/s BM_SpeedOfLightDispatch<32> 6678423 ns 6678353 ns 102 167.096M/s 4.75342M/s 14.9738M/s BM_SpeedOfLightDispatch<MaxDispatchTargets> 9373075 ns 9372688 ns 75 119.062M/s 3.38697M/s 10.6693M/s ``` I've compared the profile of the `BM_RandomSource` benchmark with this change and it largely corresponds to what I expect based on profiling hand-crafted Carbon inputs. And as you can see, we're closing in on lexing at least hitting the 10-million-lines-per-second mark. =] --------- Co-authored-by: Richard Smith <richard@metafoo.co.uk> |
||
|
Richard Smithandjosh11b
|
9c3e25decf |
Add SemIR::Node::Is<TypedNode>(). (#3284)
This allows node kind checks to be written a little more concisely. --------- Co-authored-by: josh11b <josh11b@users.noreply.github.com> |
||
|
Richard Smith
|
8779fcd77c |
Switch from SemIR::NodeKind::Foo to SemIR::Foo::Kind wherever possible. (#3283)
This is shorter, more closely connected to code using the typed node types, and avoids using the ambiguous word `Node` in places referring to typed `SemIR` nodes. |
||
|
Richard Smith
|
6e9d83e746 |
Support for while statements. (#3258)
Basic support for type-checking and lowering of `while` statements, as well as `break` and `continue`. |
||
|
Richard Smith
|
c7a9e29a89 |
Add typed nodes to SemIR. (#3280)
Replace `SemIR::Node::GetAsFoo` and `SemIR::Node::Foo::Make` with `SemIR::Foo` class that represents a particular kind of node, with named fields. Rename `SemIR::IntegerLiteral` and `SemIR::RealLiteral` to `IntegerValue` / `RealValue` to better reflect their purpose and avoid a name collision with the corresponding `SemIR` node kinds. Remove `NodeKind::Invalid` and the `SemIR::Node` default constructor entirely, as they were not used for anything. |
||
|
Chandler Carruth
|
0e8e94d8bf |
Expand source stats to roughly cover more tokens. (#3277)
This is pretty rough both in code-quality and what it tries to cover. The coverage approximation is probably fine as it mostly just gives some vague idea of the distributions we should aim at when evaluating performance. We still want performance with unusual distributions to be good. Happy to improve the code quality if folks want, and especially with suggestions on what would help. For this kind of quick scripting, it seemed "fine" to me. |
||
|
Chandler Carruth
|
6f5934a505 |
Unblock more lexer inlining. (#3274)
The big change is to make the lexer helpers have internal linkage, making all of them easy to inline into single call sites. Looking at the profile showed several other cases of unfortunate out-of-line functions. Two were due to the code size produced for checks -- those are switched to `DCHECK`s to remove that code from optimized builds. The loss of coverage seems minor. A last one was closing open groups. This was a surprising routine to be hot, but it the paths to discover "nothing to do here" were intertwined into the code. This PR extracts this common trace into a separate function that delegates to the looping recovery path. This lets the hot path inline easily. At this point, for a large lexing benchmark I'm using, 50% of the time is in the identifier hash table at this point. The remaining improvements are to actually make some of the hot routines like symbol lexing and comment lexing faster. |
||
|
Chandler Carruth
|
d1e3749aa8 |
Workaround issues with sanitizers on macOS. (#3279)
The latest versions of Clang, at least on an ARM mac, enforce that we not use static sanitizer runtimes. The flag is also a bit frustrating: it has to be split out of other flags in order to remove it, you can't just disable it or ignore a warning about it being unused. This gets things to be build and run again. However, the runtimes (I'm guessing ones that ship with the OS?) its using dynamically don't actually work -- at least one check was hitting pretty obvious false positives. So I've added a set of sanitizer flag workarounds we can expand as needed to continue to work around the limitations of sanitizers on this platform. Together, this restores fastbuild on my ARM macOS with the latest Clang installed. |
||
|
Chandler Carruth
|
03c3b86758 |
Switch lexer to fully table-driven design. (#3273)
This uses the musttail dispatched table approach to drive the entire lexing. The result is that there is no main lexer loop at all in a traditional sense, now everything is driven through tail recursive dispatch on the next byte of the source text. This should be easy to extend still -- the design pattern is to add lexer methods for handling specific cases, and then add a dispatch function to dispatch to them from the table. For example, we can add a method that handles decoding UTF-8 outside of the ASCII subset and set the table entries used by non-ASCII initial bytes to dispatch to it. The performance is already surprisingly good, benchmarks show a modest improvement across the board. That's despite there still being some *serious* performance issues that I'll fix in a separate patch. There are also opportunities to leverage this structure more heavily as needed by putting more specialized dispatch targets in for specific bytes. A follow-up PR will re-organize the functions here, as almost all of the methods on the `Lexer` should become private, but I wanted to keep that a separate change since it will probably render the diff even more hard to read than it already is. |
||
 Hana Dusíková
|
48d40aa0f0 |
Remove unnecessory constexpr for lambdas in tests. (#3272)
After reading Chandler's code I asked him why he has these. Lambdas will implicitly constexpr anyway. |
||
|
Richard Smith
|
74d52738ff |
Support for let declarations. (#3257)
A `let` declaration is represented by a `bind_name` node in SemIR: ```carbon-semir %b: i32 = bind_name "b", %a ``` Because `Check` encounters the pattern before it sees the value, we first create the `bind_name` node with an unset value and don't add it to the block. Then, once we've seen and converted the initializer, we update the `bind_name` to have the value and add it to the current block, after the initializer code. |
||
|
Chandler Carruth
|
c7e6238fa8 |
Introduce two speed-of-light benchmarks. (#3270)
The goal of these kinds of benchmarks is to help calibrate other
benchmarks and expectations. They benchmark the underlying hardware
capabilities that we can't avoid, and help illustrate bounds for what is
possible. The term "speed-of-light benchmark" references the aspect of
measuring how fast thing could possible run.
The first is a simple memory bandwidth measurement in the best case
scenario -- using `strcpy` over the buffer. This still does a minimal
number of writes to memory and examines each byte of input to see if it
is null, but can cheat in every way possible to run at the maximum speed
of hardware. To a certain extent, we never expect to get close to this
speed, but it's a good illustration of how much headroom the hardware
has available.
The second is potentially more interesting. This illustrates how fast a
byte-by-byte dispatch loop can potentially be. It uses the technique
that I'm hoping to use in the lexer itself of guaranteed tail recursion
to achieve this with a very small code footprint. The performance of
this technique, even when running in this extremely minimal setting to
establish bounds, is hugely dependent on the number of distinct dispatch
targets, and so the benchmark includes a healthy range to show the range
of performance that we might expect when running in a byte-by-byte mode.
Note that we should expect the lexer to be *faster* than this
"speed-of-light" whenever it is able to lex in larger granules than
byte-wise. But for complex, dense token sequences that force looking at
every byte, this shows the "worst case" "speed-of-light" in a sense.
On my recent AMD cloud VM instance, I get the following results running
the main lexer benchmark with these changes included:
```
-------------------------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations bytes_per_second tokens_per_second
-------------------------------------------------------------------------------------------------------------------------
BM_ValidKeywords 3169403 ns 3169283 ns 221 188.44M/s 31.5529M/s
BM_ValidIdentifiers<1, 64, false> 12486725 ns 12486445 ns 51 117.953M/s 8.00868M/s
BM_ValidIdentifiers<1, 1, true> 3950455 ns 3950298 ns 178 72.4252M/s 25.3145M/s
BM_ValidIdentifiers<3, 5, true> 15562294 ns 15561178 ns 45 36.7712M/s 6.42625M/s
BM_ValidIdentifiers<3, 16, true> 16118656 ns 16118374 ns 44 68.0412M/s 6.2041M/s
BM_ValidIdentifiers<12, 64, true> 19116271 ns 19116258 ns 35 199.541M/s 5.23115M/s
BM_ValidMix/10/40 7074336 ns 7073795 ns 93 140.744M/s 14.1367M/s
BM_ValidMix/25/30 6790722 ns 6790006 ns 102 131.793M/s 14.7275M/s
BM_ValidMix/50/20 5960514 ns 5960443 ns 118 112.594M/s 16.7773M/s
BM_ValidMix/75/10 4325546 ns 4325556 ns 159 102.559M/s 23.1184M/s
BM_SpeedOfLightStrCpy 24339 ns 24339 ns 29650 35.9049G/s 4.10858G/s
BM_SpeedOfLightDispatch<1>
|
||
|
Richard Smith
|
d306a41453 |
Allow our test runners to be more easily run outside bazel. (#3269)
Don't assume that the bazel-supplied environment variable TEST_TARGET is present. We don't actually need it for anything other than providing feedback to the developer if the test fails. This makes it a bit easier to reproduce test failures under gdb, particularly for multi-file tests. |
||
|
Richard Smith
|
bfa5463e5b |
Add signature enforcement for Main.Run and give it the symbol name main. (#3266)
For #2550. |