mirror of
https://github.com/carbon-language/carbon-lang.git
synced 2026-10-06 08:54:47 +01:00
3af7eb26721000208aa1f2f753af0c19d9339dd0
2084
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jon Ross-Perkins
|
3af7eb2672 |
Refactor YAML handling to use the llvm::yaml API. (#3337)
Provides an adapter for the llvm::yaml API because it otherwise needs a bunch of const/non-const definitions, and the traits are difficult to diagnose issues with. The current approach is pretty simple to use, even if it's not super efficient (which, yaml output is more of a debugging thing so I'm not really expecting it to be an issue). Changes the format of yaml output to provide more index information, just as reminders when seeing something like `node+0`. Note this would create more churn in deltas if we were reliant on the output yaml in tests, but we aren't so it should be okay. |
||
 Richard Smith
|
387a1711af |
Rename Deduced parameters to Implicit parameters. (#3336)
In preparation for supporting `self`, which is implicit but not deduced. |
||
|
Richard SmithandJon Ross-Perkins
|
ab575cf32a |
Support for member access into classes. (#3335)
Add support for member access into classes, for both non-instance members and for fields. --------- Co-authored-by: Jon Ross-Perkins <jperkins@google.com> |
||
|
Richard Smith
|
620408b999 |
Basic lowering support for classes. (#3334)
Track the fields in a class, and generate a corresponding struct type as the object representation for the class. For now, we always use a pointer as the value representation for a class. |
||
|
Richard Smith
|
7f2f4bec4f |
Add a Field node for fields in a class. (#3332)
This replaces the use of `VarStorage` in this case. Add an `UnboundFieldType` type as the type of a field, in cases where it's referenced without an accompanying object. Add a `BindName` node to describe the name binding performed for both variables and fields so that we can handle them more uniformly. |
||
|
Jon Ross-Perkins
|
74c3c665fa |
Refactor SemIR YAML printing to use dashed lists. (#3330)
This standardizes on having ValueStore and related structures provide printing, removing the handlers in file.cpp. The `[]` is provided for empty sequences versus if there was simply nothing, in which case it would be a sequence when non-empty, and a null value when empty. Consistently (and explicitly) providing sequences feels easier to understand. The changes to the output yaml are overall more terse. My hope is that this is an improvement for most readers. Also fixes printing of APInt, defaulting to unsigned for consistency with Carbon's use. |
||
|
Richard Smith
|
35721dc3d0 |
sem-ir: Write references to the file block as file. not package. (#3333)
This matches the renaming of the `package { ... }` block to `file { ...
}`.
|
||
|
Jon Ross-Perkins
|
e6634d240f |
Make SemIR::File access more terse. (#3331)
1. In general, `semantics_ir` -> `sem_ir`, to match the directory name. 2. For the list of `ValueStore`-related accessors on `SemIR::File`, add them to `check`'s `Context` object, shortening access. |
||
|
Jon Ross-Perkins
|
b2cfd5a8a8 |
Change StringLiteral to less frequently allocate a new string. (#3314)
Building on #3311, which started moving the result string into a `unique_ptr`, instead have `StringLiteral` use a `BumpPtrAllocator` to manage memory. But also, detect when a string is really trivial during `Lex` and, if so, return `contents_` directly. |
||
|
Jon Ross-Perkins
|
1d6298290f |
Add more value store types to File. (#3317)
Finishing what #3316 started, add more bespoke ValueStore-like structures to File. With this, the things which previously had somewhat boilerplate Add/Get functions are now all on side classes, giving a uniform style of API for calling. Note, I was on the fence about making things public on ValueStore. If it's preferred that I make some things there protected I certainly can, there's just a trade-off that may mean more distinct child/wrapper types. |
||
 Chandler CarruthandJon Ross-Perkins
|
1b0e2d3a4b |
Cleanups of SIMD code and document no Arm port. (#3325)
I spent (a lot) of time working to see if there was any profitable way to port the SIMD code that scans for identifier length to Arm. There isn't really. =/ While working on these, I made some cleanups to the SIMD code that seemed worth landing, and added some benchmarks. All this PR does is the cleanups, benchmarks, and documents that Arm isn't just waiting to get attention but doesn't really have good options (so far). For posterity, here are the core techniques I tried: 1) Direct 32-byte SIMD scanning using pair-wise add trees to build a 32-bit mask of valid identifier and then `clz` to compute the distance. This is a very good analog to the 16-byte SIMD structure used on x86-64. The pair-wise summing technique is the one used in simdjson for similar purposes. 2) A 16-byte SIMD scanning similar to the x86 version but using `shrn` to produce a 64-bit scalar bitmask with 4 bits per byte, and then scaling the bit-count distance. 3) Various hybrid versions of (1) and (2) with short scalar scans to identify short identifiers before paying the SIMD start-up cost. 4) A much fancier version of (1) that scanned 64-bytes at a time, but cached the resulting 64-bit mask and re-used it until exhausted. Some good background on these techniques on Arm CPUs is in this blog post: https://community.arm.com/arm-community-blogs/b/infrastructure-solutions-blog/posts/porting-x86-vector-bitmask-optimizations-to-arm-neon Sadly, both (1) and (2) were significantly slower than a scalar loop over the bytes. Even (3) was consistently slower. The only approach that came close was (4) and it was very *slightly* slower in typical examples and very *slightly* faster in extremely difficult cases like huge identifiers. Ultimately, the only path I see (suggested by Dougall on a Mastodon discussion of this whole problem space) is to take (4) to the limit of computing an identifier-or-not bitmask *for the entire source file* using a deeply throughput optimized routine (maybe as part of the line scanning). That should be able to manage the high latency you end up with when handling these patterns in SIMD on Arm. The good news is that at least the M1 is *so* fast in the byte-scanning loop that this isn't hurting nearly as much as I feared. --------- Co-authored-by: Jon Ross-Perkins <jperkins@google.com> |
||
 Jonathan B. CoeandChandler Carruth
|
8c28a0494e |
Add size="small" to test targets where advised (#3326)
Running `bazel test //...` reported: ``` Test execution time outside of range for MODERATE tests. Consider setting timeout="short" or size="small". ``` This change adds size="small" to avoid such warnings being reported. --------- Co-authored-by: Chandler Carruth <chandlerc@gmail.com> |
||
|
Richard Smith
|
7d9340880e |
Separate ClassType from ClassDeclaration. (#3329)
Retain the `ClassDeclaration` node to represent a syntactic declaration of a class (including possibly a declaration of a generic class), but use a separate SemIR node to represent the class type itself. This allows us to give the two separate treatment. The `ClassDeclaration` is still entered into the name lookup table for its enclosing scope, but when it is named in an expression, the class type is produced instead. When the class declaration is named in a declaration name, it can be used to define members of the class, but an expression that resolves to the class type cannot be used to define members of the class. In order to distinguish these cases, use `Name` rather than `NameExpression` for the left-hand side of a `QualifiedName` parse node. This removes the only use of the `Expression` form of a declaration name, so that is also removed. In the future, `ClassType` will also be used to describe types such as `Vector(T)`, for which there is no corresponding `ClassDeclaration`. |
||
|
Jon Ross-Perkins
|
ce248239d4 |
Fix missing include for ids.h (#3328)
I think this is only noticeable in a more modular build, but we try to keep that working. |
||
|
Richard Smith
|
85e9642d18 |
Lower types in the order they were completed. (#3324)
This is a prerequisite for class support, where a class can be
referenced as a type before it becomes complete. For example, given:
```carbon
class A {
fn F(a: A);
class B {}
var b: B;
}
fn A.F(a: A) {}
```
we need to lower `B` before we lower `A`, even though `A` is used as a
type first.
This will also start catching some cases where we don't require a type
to be complete despite using it, as we now only lower types that are
required to be complete.
Remove the poison values for struct and tuple literals. We don't need
those any more, because we never generate references to those literals
as values, and we don't have a type to use for them because we never
require the type of a literal to be complete, only the type of the
entity initialized by the literal, which can be different, for example
when initializing an array from a tuple literal or a class from a struct
literal.
This doesn't affect the output: `llvm::Type` objects that are not
referenced by an LLVM module don't affect the IR for that module, and
the order in which `llvm::Type`s are created doesn't affect anything
either.
|
||
 josh11b
|
5971f80826 |
Add consistency test between NodeKind and Definition (#3322)
|
||
 Geoff Romer
|
9e0d831092 |
Handle let with no closing semicolon (#3323)
Also add a test for the `var` case |
||
|
josh11b
|
b08f8bf589 |
Remove unused LiteralStringId (#3321)
|
||
|
Jon Ross-Perkins
|
7e9d644e1f |
Switch File functions, classes, and types to ValueStores (#3316)
Building on #3313, start using ValueStore on File. Functions and classes are straightforward. Types here I present as a borderline case where maybe we want a more bespoke API, but maybe this is okay? Most other things probably need a slightly different API, which although I might do that for a consistent interface, felt more out-of-scope for this change. |
||
|
Jon Ross-Perkins
|
843dd40f22 |
Adding ValueStore printing and --dump-shared-values (#3320)
This replaces the printing that was removed from SemIR's raw dump. It's separate because (for example) lexing generates shared values, and so reviewing them is not specific to any particular phase. |
||
|
josh11b
|
5b3b7aa3ed |
Test for id overflow and string deduplication in SharedValueStores (#3319)
Follow-on to #3311 |
||
|
Richard Smith
|
a46e7dd967 |
Remove most of the metaprogramming in node.h in favor of listing all the members in the typed node structs. (#3310)
Split `node.h` into separate files for ID types (`id.h`) and for typed nodes (`typed_nodes.h`). The per-node-kind data is now specified as part of declaring the typed nodes, and is removed from the node kinds x-macros, which now simply enumerate the node kinds. |
||
|
Jon Ross-Perkins
|
e54deee525 |
Fix DenseMapInfo declarations for ValueStore's StringId use. (#3318)
The move of the DenseMap to value_store.h should have been matched with moving the declaration; this worked by happenstance. |
||
|
Jon Ross-Perkins
|
1b55ad86dd |
Extend SharedValueStores to SemIR (#3313)
Building on #3311, change SemIR to use the SharedValueStore. Since this removes hermeticity, raw output no longer prints ints, reals, and strings. TokenizedBuffer accessors are modified to return IDs because values are often passed through in semantics without needing to read them. I would've put SharedValueStores on Context, except for the GetArrayBoundValue convenience method. I felt awkward removing that, so it's on File, at least for now. That's then used by the formatter and Lower too. The flipside of this is that TokenizedBuffer has a SharedValueStores only for printing, so maybe that's similar enough to what File is doing. This doesn't start shifting other SemIR members to ValueStore, but that seems like a next step. |
||
|
Jon Ross-PerkinsandRichard Smith
|
d13f76e001 |
Add value store to be shared across compile stages. (#3311)
This updates lexing to use the data. I'll do checking separately, just to split changes. Note the ValueStore structure is also set up such that SemIR::File can use it for other fields. --------- Co-authored-by: Richard Smith <richard@metafoo.co.uk> |
||
|
|
629c63c7d1 |
Port the comment block scanning SIMD to Arm. (#3300)
This adds an Arm Neon code path for comment block scanning. It also tries to tightened up the exact architecture-specific coding pattern for SIMD code here. Notably, moving to always having architecture-specific code inside a macro that can be used to globally disable SIMD, but any place where *all* architectures will require some custom code, an `else` branch with an error. Overall, this makes large-block lexing >5% faster on an ARM server I have access to, but that's a bit misleading. The Neon performance there doesn't seem very good. On my M1 laptop the difference is *much* larger. There, even 4-line comments show a noticable improvement and the block speed looks well over 20%. Sadly, I don't have the same nice scripts to generate good statistical data. Raw benchmark data from an ARM sever for reference: ``` BM_CommentLines/1/0/0 19.4ms ± 7% 19.7ms ± 6% ~ (p=0.121 n=20+20) BM_CommentLines/4/0/0 24.8ms ± 7% 24.8ms ± 6% ~ (p=0.904 n=20+20) BM_CommentLines/128/0/0 251ms ± 2% 234ms ± 3% -6.70% (p=0.000 n=20+20) BM_CommentLines/1/30/0 21.6ms ±10% 21.9ms ±10% ~ (p=0.157 n=20+20) BM_CommentLines/4/30/0 28.8ms ± 9% 28.8ms ±11% ~ (p=0.779 n=20+20) BM_CommentLines/128/30/0 248ms ± 2% 233ms ± 2% -5.72% (p=0.000 n=20+20) BM_CommentLines/1/70/0 23.4ms ±12% 23.7ms ±12% ~ (p=0.341 n=20+20) BM_CommentLines/4/70/0 30.8ms ± 9% 31.1ms ±11% ~ (p=0.602 n=20+20) BM_CommentLines/128/70/0 302ms ± 4% 292ms ± 4% -3.46% (p=0.000 n=18+20) BM_CommentLines/1/0/2 19.8ms ± 7% 20.0ms ± 6% ~ (p=0.149 n=20+20) BM_CommentLines/4/0/2 25.1ms ± 7% 25.3ms ± 8% ~ (p=0.659 n=20+20) BM_CommentLines/128/0/2 225ms ± 2% 212ms ± 2% -5.88% (p=0.000 n=20+20) BM_CommentLines/1/30/2 22.0ms ± 9% 22.2ms ±10% ~ (p=0.289 n=20+20) BM_CommentLines/4/30/2 29.0ms ±11% 29.1ms ±12% ~ (p=0.738 n=20+20) BM_CommentLines/128/30/2 261ms ±10% 243ms ± 3% -6.85% (p=0.000 n=20+20) BM_CommentLines/1/70/2 23.5ms ±11% 23.8ms ±15% ~ (p=0.429 n=20+20) BM_CommentLines/4/70/2 31.3ms ±10% 31.5ms ±11% ~ (p=0.478 n=20+20) BM_CommentLines/128/70/2 306ms ± 4% 292ms ± 4% -4.52% (p=0.000 n=18+19) BM_CommentLines/1/0/8 20.9ms ± 8% 21.2ms ± 7% ~ (p=0.127 n=20+20) BM_CommentLines/4/0/8 27.3ms ± 9% 27.5ms ±12% ~ (p=0.678 n=20+20) BM_CommentLines/128/0/8 227ms ± 2% 210ms ± 2% -7.35% (p=0.000 n=19+20) BM_CommentLines/1/30/8 22.6ms ±11% 23.0ms ±10% ~ (p=0.114 n=20+20) BM_CommentLines/4/30/8 29.4ms ±10% 29.4ms ±12% ~ (p=0.947 n=20+20) BM_CommentLines/128/30/8 275ms ± 4% 257ms ± 7% -6.59% (p=0.000 n=19+20) BM_CommentLines/1/70/8 23.9ms ±13% 24.3ms ±14% ~ (p=0.265 n=20+20) BM_CommentLines/4/70/8 32.3ms ±11% 32.4ms ± 9% ~ (p=0.478 n=20+20) BM_CommentLines/128/70/8 319ms ± 4% 307ms ± 4% -3.83% (p=0.000 n=18+19) ``` --------- Co-authored-by: Richard Smith <richard@metafoo.co.uk> Co-authored-by: Jon Ross-Perkins <jperkins@google.com> |
||
|
Richard Smith
|
a95e122123 |
Use the full name declaration logic when adding var names to scope. (#3312)
This causes the names of class members to get added to the class scope. |
||
|
Richard SmithandJon Ross-Perkins
|
b01dfb3f93 |
Support for class definitions with static member functions. (#3305)
This includes being able to define a class that was previously forward-declared, and being able to define a member function out-of-line that was previously declared inside a class. No support for fields or methods yet, and a class definition doesn't yet cause the class to be treated as a complete type. --------- Co-authored-by: Jon Ross-Perkins <jperkins@google.com> |
||
|
josh11bandJon Ross-Perkins
|
3b82ea96db |
Add & update comments in parse/node_kind.def (#3307)
Co-authored-by: Jon Ross-Perkins <jperkins@google.com> |
||
|
josh11b
|
6d4d05f68a |
SEMANTICS->SEM_IR in macro names. Update comments in sem_ir/node.h. (#3309)
* Shift to "sem_ir" continued from #3176. * Comments in `node.h` changed to reflect #3280. |
||
|
Richard Smith
|
efd8e1b144 |
Fix follow-on error for member access on an invalid expression. (#3303)
We were looking at the type of the base expression prior to conversion instead of the type after conversion. As noted in review of #3302. |
||
|
Richard Smith
|
69353ed271 |
Basic support for incomplete types. (#3302)
Incomplete types may be nested within other types; for example, a tuple type might have an incomplete type as an element. Handle such cases by walking through nested incomplete types when completing a type. This is done non-recursively in case a very complex type is formed. Types are generally no longer completed at the point where they're formed. Instead, we attempt to complete a type when it is used in a context that requires a complete type, and diagnose if the type cannot be completed at that point. This will be necessary for classes, which can become complete after their first use, and helps tease out bugs where a type completeness check is missing. |
||
|
Chandler CarruthandRichard Smith
|
3015135a52 |
Skip blocks of comments with identical prefixes. (#3299)
Specifically, after lexing a comment line, look at the next line and see if it starts with an identical sequence of indent, comment '/'s and character after the '/'s. If so, skip it as part of a block of comments. This skips repeatedly diagnosing the same erroneous comment introducer after the first one in a block, but that seems like a feature rather than a bug. The big motivation is to make sure the lexer is minimally impacted by the length of comment blocks and skips them as efficiently as possible. While they aren't exactly common, large block comments do come up and it'd be unfortunate for those to actually slow down the toolchain. It also happens that this is particularly easy to do because we're just looking to see if we see the same prefix byte sequence. With SIMD we can typically handle the most common indents with just a few instructions. Because of the diagnostic differences, I've included a scalar fallback that replicates the functionality but has no limit on indent size or CPU features. I've also added testing to cover this behavior. The only non-noise benchmark changes are as expected the comment ones, with a nice improvement across the board: ``` BM_CommentLines/1/0/0 15.8ms ± 2% 15.6ms ± 2% -0.87% (p=0.004 n=19+19) BM_CommentLines/4/0/0 20.2ms ± 1% 18.8ms ± 1% -6.75% (p=0.000 n=18+18) BM_CommentLines/128/0/0 221ms ± 1% 167ms ± 1% -24.44% (p=0.000 n=20+19) BM_CommentLines/1/30/0 16.6ms ± 3% 16.5ms ± 3% ~ (p=0.175 n=19+20) BM_CommentLines/4/30/0 26.1ms ± 1% 24.8ms ± 2% -5.05% (p=0.000 n=18+19) BM_CommentLines/128/30/0 233ms ± 1% 185ms ± 1% -20.38% (p=0.000 n=19+20) BM_CommentLines/1/70/0 19.2ms ± 1% 19.0ms ± 2% -0.66% (p=0.016 n=19+20) BM_CommentLines/4/70/0 27.9ms ± 1% 26.6ms ± 1% -4.63% (p=0.000 n=19+19) BM_CommentLines/128/70/0 251ms ± 1% 213ms ± 1% -15.18% (p=0.000 n=20+18) BM_CommentLines/1/0/2 15.9ms ± 1% 15.8ms ± 2% ~ (p=0.061 n=19+19) BM_CommentLines/4/0/2 20.5ms ± 2% 19.0ms ± 2% -7.53% (p=0.000 n=20+20) BM_CommentLines/128/0/2 213ms ± 1% 153ms ± 1% -28.18% (p=0.000 n=19+20) BM_CommentLines/1/30/2 16.8ms ± 2% 16.7ms ± 3% ~ (p=0.134 n=20+20) BM_CommentLines/4/30/2 26.6ms ± 1% 25.2ms ± 3% -5.50% (p=0.000 n=20+20) BM_CommentLines/128/30/2 238ms ± 1% 187ms ± 2% -21.49% (p=0.000 n=17+19) BM_CommentLines/1/70/2 19.3ms ± 1% 19.4ms ± 3% ~ (p=0.407 n=17+20) BM_CommentLines/4/70/2 28.2ms ± 1% 26.9ms ± 2% -4.70% (p=0.000 n=19+19) BM_CommentLines/128/70/2 257ms ± 2% 214ms ± 1% -16.52% (p=0.000 n=20+18) BM_CommentLines/1/0/8 16.3ms ± 2% 16.1ms ± 2% -1.22% (p=0.001 n=20+20) BM_CommentLines/4/0/8 22.7ms ± 2% 20.4ms ± 2% -10.20% (p=0.000 n=20+20) BM_CommentLines/128/0/8 244ms ± 1% 153ms ± 1% -37.26% (p=0.000 n=20+18) BM_CommentLines/1/30/8 17.3ms ± 2% 17.2ms ± 3% ~ (p=0.192 n=20+20) BM_CommentLines/4/30/8 28.0ms ± 2% 25.6ms ± 3% -8.46% (p=0.000 n=19+18) BM_CommentLines/128/30/8 272ms ± 1% 196ms ± 2% -27.90% (p=0.000 n=18+20) BM_CommentLines/1/70/8 19.9ms ± 2% 19.9ms ± 2% ~ (p=0.531 n=20+19) BM_CommentLines/4/70/8 29.3ms ± 1% 27.3ms ± 1% -6.87% (p=0.000 n=19+19) BM_CommentLines/128/70/8 292ms ± 1% 228ms ± 1% -21.97% (p=0.000 n=20+19) ``` --------- Co-authored-by: Richard Smith <richard@metafoo.co.uk> |
||
|
Chandler Carruth
|
7371354dc7 |
Consolidate indent handling to following newlines. (#3298)
The biggest advantage of this is reducing the repeated code in every non-whitespace code path of the lexer to set indent appropriately. Now we handle it cleanly at the start and after vertical whitespace. While this makes the generated code for all the other paths through the lexer quite a bit nicer, it doesn't actually move performance in interesting ways outside of making large blocks of blank lines slightly slower. While there are lots of fluctuations in the benchmark data, they mostly seem to be either noise or artifacts of loop alignment and not really due to an important change here. Raw benchmark data: ``` BM_ValidKeywords 3.10ms ± 1% 3.13ms ± 1% +1.02% (p=0.000 n=20+19) BM_ValidIdentifiers<1, 64, false> 10.8ms ± 3% 10.7ms ± 3% ~ (p=0.383 n=20+20) BM_ValidIdentifiers<1, 1, true> 3.71ms ± 1% 3.67ms ± 2% -1.05% (p=0.000 n=19+19) BM_ValidIdentifiers<3, 5, true> 13.1ms ± 2% 13.0ms ± 2% ~ (p=0.120 n=20+19) BM_ValidIdentifiers<3, 16, true> 13.1ms ± 2% 13.2ms ± 2% ~ (p=0.091 n=20+20) BM_ValidIdentifiers<12, 64, true> 15.1ms ± 1% 15.1ms ± 1% ~ (p=0.138 n=19+19) BM_HorizontalWhitespace/1 13.2ms ± 3% 13.2ms ± 0% ~ (p=0.458 n=20+15) BM_HorizontalWhitespace/4 13.4ms ± 3% 13.3ms ± 1% -1.12% (p=0.000 n=20+19) BM_HorizontalWhitespace/16 14.1ms ± 2% 14.0ms ± 2% -0.89% (p=0.010 n=20+20) BM_HorizontalWhitespace/64 17.9ms ± 2% 17.8ms ± 1% -0.73% (p=0.002 n=20+16) BM_HorizontalWhitespace/128 24.2ms ± 2% 24.1ms ± 1% ~ (p=0.346 n=19+17) BM_RandomSource 7.88ms ± 2% 7.83ms ± 3% ~ (p=0.052 n=20+20) BM_GroupingSymbols/1/0/0 6.25ms ± 2% 6.38ms ± 1% +1.99% (p=0.000 n=20+19) BM_GroupingSymbols/2/0/0 5.24ms ± 2% 5.27ms ± 2% ~ (p=0.065 n=20+19) BM_GroupingSymbols/3/0/0 4.09ms ± 1% 4.07ms ± 1% ~ (p=0.063 n=20+20) BM_GroupingSymbols/4/0/0 3.80ms ± 1% 3.79ms ± 2% ~ (p=0.820 n=20+20) BM_GroupingSymbols/8/0/0 3.05ms ± 1% 3.09ms ± 1% +1.41% (p=0.000 n=20+20) BM_GroupingSymbols/16/0/0 2.95ms ± 1% 3.01ms ± 1% +1.79% (p=0.000 n=20+20) BM_GroupingSymbols/32/0/0 3.67ms ± 1% 3.77ms ± 1% +2.75% (p=0.000 n=20+20) BM_GroupingSymbols/0/1/0 5.66ms ± 1% 5.61ms ± 1% -0.88% (p=0.000 n=20+18) BM_GroupingSymbols/0/2/0 4.43ms ± 2% 4.40ms ± 1% -0.74% (p=0.005 n=20+17) BM_GroupingSymbols/0/3/0 3.15ms ± 2% 3.12ms ± 2% -0.96% (p=0.002 n=20+19) BM_GroupingSymbols/0/4/0 2.79ms ± 2% 2.77ms ± 2% -0.88% (p=0.005 n=20+20) BM_GroupingSymbols/0/8/0 1.81ms ± 2% 1.79ms ± 2% ~ (p=0.056 n=20+20) BM_GroupingSymbols/0/16/0 1.26ms ± 2% 1.26ms ± 2% ~ (p=0.547 n=20+20) BM_GroupingSymbols/0/32/0 1.05ms ± 1% 0.96ms ± 1% -8.52% (p=0.000 n=19+19) BM_GroupingSymbols/0/0/1 5.68ms ± 2% 5.65ms ± 2% ~ (p=0.126 n=20+18) BM_GroupingSymbols/0/0/2 4.44ms ± 2% 4.40ms ± 2% -0.99% (p=0.001 n=20+18) BM_GroupingSymbols/0/0/3 3.15ms ± 1% 3.13ms ± 1% -0.74% (p=0.005 n=20+18) BM_GroupingSymbols/0/0/4 2.80ms ± 1% 2.77ms ± 2% -1.01% (p=0.000 n=20+19) BM_GroupingSymbols/0/0/8 1.81ms ± 2% 1.79ms ± 2% -0.97% (p=0.006 n=20+20) BM_GroupingSymbols/0/0/16 1.26ms ± 1% 1.26ms ± 2% ~ (p=0.678 n=20+20) BM_GroupingSymbols/0/0/32 1.05ms ± 1% 0.96ms ± 1% -8.59% (p=0.000 n=20+19) BM_GroupingSymbols/32/1/0 3.57ms ± 1% 3.69ms ± 1% +3.39% (p=0.000 n=19+20) BM_GroupingSymbols/32/2/0 3.49ms ± 1% 3.60ms ± 1% +3.16% (p=0.000 n=20+20) BM_GroupingSymbols/32/3/0 3.41ms ± 1% 3.52ms ± 1% +3.49% (p=0.000 n=19+20) BM_GroupingSymbols/32/4/0 3.35ms ± 1% 3.45ms ± 1% +3.13% (p=0.000 n=20+20) BM_GroupingSymbols/32/8/0 3.12ms ± 1% 3.22ms ± 1% +3.09% (p=0.000 n=19+20) BM_GroupingSymbols/32/16/0 2.75ms ± 1% 2.82ms ± 1% +2.71% (p=0.000 n=19+20) BM_GroupingSymbols/32/32/0 2.24ms ± 2% 2.24ms ± 1% ~ (p=0.311 n=20+17) BM_GroupingSymbols/32/32/1 2.20ms ± 1% 2.23ms ± 1% +1.23% (p=0.000 n=20+20) BM_GroupingSymbols/32/32/2 2.19ms ± 1% 2.21ms ± 1% +1.08% (p=0.000 n=20+20) BM_GroupingSymbols/32/32/3 2.16ms ± 0% 2.19ms ± 1% +1.09% (p=0.000 n=18+20) BM_GroupingSymbols/32/32/4 2.14ms ± 1% 2.19ms ± 1% +2.25% (p=0.000 n=20+20) BM_GroupingSymbols/32/32/8 2.07ms ± 1% 2.11ms ± 1% +2.04% (p=0.000 n=17+20) BM_GroupingSymbols/32/32/16 1.94ms ± 1% 1.98ms ± 1% +1.95% (p=0.000 n=20+20) BM_GroupingSymbols/32/32/32 1.74ms ± 1% 1.77ms ± 1% +1.88% (p=0.000 n=18+20) BM_BlankLines/1 14.2ms ± 1% 14.4ms ± 1% +1.47% (p=0.000 n=18+16) BM_BlankLines/4 17.3ms ± 1% 17.5ms ± 2% +1.32% (p=0.000 n=17+20) BM_BlankLines/16 31.3ms ± 1% 33.0ms ± 2% +5.35% (p=0.000 n=19+20) BM_BlankLines/64 89.6ms ± 1% 101.3ms ± 1% +12.98% (p=0.000 n=20+20) BM_BlankLines/128 167ms ± 3% 185ms ± 1% +11.11% (p=0.000 n=19+20) BM_CommentLines/1/0/0 15.7ms ± 1% 15.8ms ± 1% ~ (p=0.109 n=19+16) BM_CommentLines/4/0/0 19.8ms ± 1% 20.2ms ± 1% +2.12% (p=0.000 n=20+19) BM_CommentLines/128/0/0 208ms ± 1% 221ms ± 1% +6.29% (p=0.000 n=20+19) BM_CommentLines/1/30/0 16.6ms ± 1% 16.6ms ± 1% ~ (p=0.354 n=19+19) BM_CommentLines/4/30/0 25.8ms ± 2% 26.1ms ± 1% +1.39% (p=0.000 n=20+19) BM_CommentLines/128/30/0 222ms ± 2% 232ms ± 1% +4.46% (p=0.000 n=20+18) BM_CommentLines/1/70/0 19.2ms ± 2% 19.1ms ± 1% ~ (p=0.478 n=20+19) BM_CommentLines/4/70/0 27.6ms ± 2% 27.9ms ± 1% +0.95% (p=0.001 n=20+18) BM_CommentLines/128/70/0 245ms ± 2% 250ms ± 1% +1.96% (p=0.000 n=20+18) BM_CommentLines/1/0/2 16.0ms ± 1% 15.9ms ± 2% -0.68% (p=0.015 n=15+18) BM_CommentLines/4/0/2 20.6ms ± 1% 20.5ms ± 1% -0.53% (p=0.024 n=18+19) BM_CommentLines/128/0/2 216ms ± 2% 213ms ± 1% -1.70% (p=0.000 n=17+19) BM_CommentLines/1/30/2 16.9ms ± 2% 16.8ms ± 3% ~ (p=0.072 n=20+20) BM_CommentLines/4/30/2 26.8ms ± 2% 26.5ms ± 1% -0.91% (p=0.000 n=20+19) BM_CommentLines/128/30/2 244ms ± 2% 238ms ± 1% -2.30% (p=0.000 n=19+17) BM_CommentLines/1/70/2 19.4ms ± 2% 19.4ms ± 2% ~ (p=0.059 n=18+20) BM_CommentLines/4/70/2 28.4ms ± 2% 28.2ms ± 1% -0.81% (p=0.003 n=20+19) BM_CommentLines/128/70/2 261ms ± 2% 255ms ± 1% -2.33% (p=0.000 n=20+18) BM_CommentLines/1/0/8 16.2ms ± 2% 16.2ms ± 2% ~ (p=0.904 n=20+20) BM_CommentLines/4/0/8 25.0ms ± 1% 22.6ms ± 1% -9.40% (p=0.000 n=20+19) BM_CommentLines/128/0/8 246ms ± 2% 244ms ± 1% -1.14% (p=0.000 n=20+19) BM_CommentLines/1/30/8 17.3ms ± 2% 17.3ms ± 1% ~ (p=1.000 n=19+18) BM_CommentLines/4/30/8 30.3ms ± 2% 27.9ms ± 1% -7.83% (p=0.000 n=20+20) BM_CommentLines/128/30/8 277ms ± 3% 271ms ± 1% -2.26% (p=0.000 n=20+17) BM_CommentLines/1/70/8 19.9ms ± 2% 19.9ms ± 2% ~ (p=0.687 n=19+20) BM_CommentLines/4/70/8 31.7ms ± 2% 29.3ms ± 1% -7.55% (p=0.000 n=20+18) BM_CommentLines/128/70/8 296ms ± 2% 291ms ± 1% -1.50% (p=0.000 n=20+18) ``` |
||
|
Richard SmithandChandler Carruth
|
e4caf7d604 |
Compute and cache the value representation of a type when it becomes complete. (#3271)
Using the computed value representation, fix lowering of struct and tuple values to use the value representation rather than the object representation. Fixes an issue found in the review of #3257. This currently causes us to compute value representations of all types as they are created, which generates substantially more SemIR to represent types. We can get some of that back by deferring computation of the value representation until the type is required to be complete, but some of the additional cost here will persist with this approach. I also considered making the computation of the value representation type be something that lives entirely within the lowering phase, but I think that's not the right approach in the longer term, because the value representation will be semantically visible and relevant once we start allowing it to be customized. We should consider moving the nodes that exist to compute canonical non-local types, including value representations, out into a separate global block. That will clean up the SemIR representation substantially, and make the SemIR produced for a function not depend on which types we happen to have encountered beforehand. But that's not being done in this PR. --------- Co-authored-by: Chandler Carruth <chandlerc@gmail.com> |
||
|
Richard Smith
|
1ae5fe0cd5 |
Provide the callee expression to the Call node. (#3291)
Track the callee expression in full, instead of only tracking the callee's FunctionId. This results in the `name_reference` denoting the function actually being used. Lowering now propagates a `llvm::Function*` as the value associated with expressions of type `<function>`. We were not creating `NameReference` node for names produced by member access into a namespace, such as the second name in `Namespace.Function`, which caused lowering of calls to such names to fail. This is now fixed, but the resulting `NameReference` node only refers to the name and the lookup result, not to the `Namespace.` qualifier. We'll need to decide how to fit a third operand into that node (perhaps we can stop storing the `name_id`, since it can be derived from the lookup result) but for now the qualifier is not tracked. |
||
|
Richard Smith
|
c61b6a0e1d |
Rename //toolchain/sem_ir:file_test -> :yaml_test. (#3293)
While this test is testing //toolchain/sem_ir:file, the interesting thing about it is that it's testing the YAML output from File printing. Calling this test `:file_test` is also deeply confusing because that's also the name we give to all of our tests that are testing the files in `testdata/`, which is not what this test does. |
||
|
josh11bandRichard Smith
|
a112f2e802 |
Validate parse nodes correspond to expected tokens (#3295)
Can specify which tokens are allowed generally, and any additional tokens that only occur when the parse node has an error. --------- Co-authored-by: Richard Smith <richard@metafoo.co.uk> |
||
|
Chandler Carruth
|
95a1cc8cba |
Tweak lexer to improve generated code. (#3296)
This is a collection of tweaks and they are ones I'm less confident in FWIW. I set out to make the generated code cleaner and reduce loading pointers through pointers in a bunch of cases. It also works to reduce the working-set-size, and reduce the set of mutated values on each iteration. But I didn't benchmark at each step and it's not clear that incremental benchmarks will even be meaningful, so its hard to say which tweaks were load bearing and which weren't. For example, with this, the core lexer dispatch loop doesn't mutate anything in memory -- the position is passed in register and incremented in register throughout. The pointer to the source text, the size, and the pointer to the lexer are also passed in registers. By making the tokenized buffer a direct member of the lexer, all of its members can be accessed at a constant offset from the lexer pointer itself. The cost of this is a move at the end of lexing, but especially as the buffer size gets large this seems like a trivial cost compared to the previous double-indirection. This uses a (significantly) cheaper representation for the line index -- the `Line` type is optimized for dense *storage*, and is not a great type for using in a tight loop like the lexer. Once using a good index and once the line table is readily available from the above, we can simply index the line table rather than store (and update) a separate pointer. Last but not least, this removes the column as suggested in a previous review. Now it is computed from the position and the line information. My macro benchmark looks like it improves in the 2% - 5% range, but its getting into the noise sadly (or maybe this is good?). On a quiet AMD server with 20 runs before/after I get reasonably compelling across-the-board improvements on all the microbenchmarks, including 5% on `RandomSource` which is key: ``` BM_ValidKeywords 3.44ms ± 1% 3.11ms ± 1% -9.52% (p=0.000 n=19+20) BM_ValidIdentifiers<1, 64, false> 11.2ms ± 1% 10.8ms ± 2% -3.23% (p=0.000 n=19+20) BM_ValidIdentifiers<1, 1, true> 3.96ms ± 1% 3.72ms ± 1% -6.10% (p=0.000 n=18+20) BM_ValidIdentifiers<3, 5, true> 13.5ms ± 1% 13.2ms ± 3% -2.45% (p=0.000 n=19+20) BM_ValidIdentifiers<3, 16, true> 13.5ms ± 2% 13.1ms ± 2% -2.81% (p=0.000 n=19+20) BM_ValidIdentifiers<12, 64, true> 15.6ms ± 1% 15.2ms ± 3% -2.41% (p=0.000 n=19+20) BM_HorizontalWhitespace/1 13.6ms ± 2% 13.3ms ± 3% -2.34% (p=0.000 n=19+20) BM_HorizontalWhitespace/4 13.9ms ± 2% 13.6ms ± 3% -2.06% (p=0.000 n=19+20) BM_HorizontalWhitespace/16 14.5ms ± 2% 14.3ms ± 2% -1.17% (p=0.002 n=19+20) BM_HorizontalWhitespace/64 18.5ms ± 3% 18.1ms ± 2% -2.17% (p=0.000 n=19+19) BM_HorizontalWhitespace/128 24.7ms ± 1% 24.4ms ± 2% -1.29% (p=0.000 n=18+20) BM_RandomSource 8.38ms ± 2% 7.92ms ± 2% -5.50% (p=0.000 n=19+20) BM_GroupingSymbols/1/0/0 6.63ms ± 2% 6.30ms ± 2% -4.97% (p=0.000 n=19+20) BM_GroupingSymbols/2/0/0 5.65ms ± 2% 5.27ms ± 2% -6.75% (p=0.000 n=19+20) BM_GroupingSymbols/3/0/0 4.51ms ± 1% 4.12ms ± 1% -8.62% (p=0.000 n=19+18) BM_GroupingSymbols/4/0/0 4.23ms ± 1% 3.82ms ± 1% -9.65% (p=0.000 n=19+18) BM_GroupingSymbols/8/0/0 3.50ms ± 1% 3.08ms ± 1% -12.21% (p=0.000 n=19+19) BM_GroupingSymbols/16/0/0 3.38ms ± 1% 2.97ms ± 1% -11.98% (p=0.000 n=19+19) BM_GroupingSymbols/32/0/0 4.10ms ± 1% 3.69ms ± 1% -10.02% (p=0.000 n=17+19) BM_GroupingSymbols/0/1/0 5.85ms ± 2% 5.70ms ± 1% -2.54% (p=0.000 n=19+20) BM_GroupingSymbols/0/2/0 4.60ms ± 2% 4.46ms ± 1% -3.12% (p=0.000 n=19+19) BM_GroupingSymbols/0/3/0 3.32ms ± 2% 3.18ms ± 1% -4.21% (p=0.000 n=18+20) BM_GroupingSymbols/0/4/0 2.94ms ± 2% 2.82ms ± 1% -4.22% (p=0.000 n=19+20) BM_GroupingSymbols/0/8/0 1.95ms ± 1% 1.83ms ± 1% -6.47% (p=0.000 n=17+19) BM_GroupingSymbols/0/16/0 1.40ms ± 1% 1.27ms ± 1% -9.12% (p=0.000 n=19+20) BM_GroupingSymbols/0/32/0 1.19ms ± 1% 1.05ms ± 1% -11.50% (p=0.000 n=19+20) BM_GroupingSymbols/0/0/1 5.88ms ± 2% 5.71ms ± 1% -2.87% (p=0.000 n=19+20) BM_GroupingSymbols/0/0/2 4.62ms ± 1% 4.45ms ± 2% -3.58% (p=0.000 n=18+20) BM_GroupingSymbols/0/0/3 3.32ms ± 1% 3.17ms ± 1% -4.72% (p=0.000 n=16+17) BM_GroupingSymbols/0/0/4 2.95ms ± 1% 2.81ms ± 2% -4.50% (p=0.000 n=16+19) BM_GroupingSymbols/0/0/8 1.96ms ± 1% 1.82ms ± 2% -6.80% (p=0.000 n=19+20) BM_GroupingSymbols/0/0/16 1.40ms ± 1% 1.27ms ± 1% -8.96% (p=0.000 n=19+20) BM_GroupingSymbols/0/0/32 1.19ms ± 1% 1.05ms ± 1% -11.64% (p=0.000 n=19+20) BM_GroupingSymbols/32/1/0 4.04ms ± 1% 3.60ms ± 1% -10.81% (p=0.000 n=17+20) BM_GroupingSymbols/32/2/0 3.93ms ± 1% 3.51ms ± 1% -10.55% (p=0.000 n=18+20) BM_GroupingSymbols/32/3/0 3.84ms ± 1% 3.44ms ± 1% -10.38% (p=0.000 n=18+19) BM_GroupingSymbols/32/4/0 3.76ms ± 2% 3.38ms ± 1% -10.21% (p=0.000 n=19+20) BM_GroupingSymbols/32/8/0 3.51ms ± 1% 3.14ms ± 1% -10.61% (p=0.000 n=19+19) BM_GroupingSymbols/32/16/0 3.11ms ± 1% 2.77ms ± 1% -10.95% (p=0.000 n=19+20) BM_GroupingSymbols/32/32/0 2.54ms ± 2% 2.25ms ± 1% -11.77% (p=0.000 n=19+16) BM_GroupingSymbols/32/32/1 2.50ms ± 1% 2.22ms ± 2% -11.19% (p=0.000 n=19+20) BM_GroupingSymbols/32/32/2 2.48ms ± 1% 2.21ms ± 1% -11.15% (p=0.000 n=19+20) BM_GroupingSymbols/32/32/3 2.46ms ± 1% 2.18ms ± 1% -11.11% (p=0.000 n=18+20) BM_GroupingSymbols/32/32/4 2.43ms ± 1% 2.16ms ± 1% -11.04% (p=0.000 n=19+20) BM_GroupingSymbols/32/32/8 2.35ms ± 1% 2.09ms ± 1% -11.11% (p=0.000 n=19+20) BM_GroupingSymbols/32/32/16 2.20ms ± 1% 1.96ms ± 1% -11.30% (p=0.000 n=19+18) BM_GroupingSymbols/32/32/32 1.98ms ± 1% 1.75ms ± 1% -11.54% (p=0.000 n=19+19) BM_BlankLines/1 14.6ms ± 1% 14.2ms ± 3% -2.53% (p=0.000 n=19+19) BM_BlankLines/4 18.3ms ± 1% 17.4ms ± 2% -4.68% (p=0.000 n=19+20) BM_BlankLines/16 34.9ms ± 2% 31.6ms ± 3% -9.42% (p=0.000 n=19+20) BM_BlankLines/64 102ms ± 2% 91ms ± 3% -10.75% (p=0.000 n=19+20) BM_BlankLines/128 190ms ± 2% 169ms ± 3% -10.82% (p=0.000 n=19+20) BM_CommentLines/1/0/0 16.2ms ± 1% 15.8ms ± 2% -2.29% (p=0.000 n=19+19) BM_CommentLines/4/0/0 21.0ms ± 1% 19.9ms ± 2% -5.17% (p=0.000 n=19+19) BM_CommentLines/128/0/0 237ms ± 1% 211ms ± 2% -10.93% (p=0.000 n=19+20) BM_CommentLines/1/30/0 17.0ms ± 1% 16.8ms ± 3% -1.36% (p=0.001 n=17+20) BM_CommentLines/4/30/0 27.2ms ± 2% 26.0ms ± 2% -4.24% (p=0.000 n=19+20) BM_CommentLines/128/30/0 255ms ± 2% 223ms ± 2% -12.58% (p=0.000 n=19+20) BM_CommentLines/1/70/0 19.7ms ± 1% 19.2ms ± 1% -2.35% (p=0.000 n=18+18) BM_CommentLines/4/70/0 29.0ms ± 2% 27.6ms ± 1% -4.81% (p=0.000 n=18+17) BM_CommentLines/128/70/0 273ms ± 7% 249ms ± 3% -9.05% (p=0.000 n=20+20) BM_CommentLines/1/0/2 16.4ms ± 2% 16.0ms ± 1% -2.49% (p=0.000 n=19+15) BM_CommentLines/4/0/2 22.0ms ± 2% 20.7ms ± 1% -5.84% (p=0.000 n=19+17) BM_CommentLines/128/0/2 256ms ± 2% 218ms ± 1% -14.88% (p=0.000 n=19+17) BM_CommentLines/1/30/2 17.4ms ± 1% 17.0ms ± 2% -2.26% (p=0.000 n=19+19) BM_CommentLines/4/30/2 28.5ms ± 1% 26.9ms ± 2% -5.67% (p=0.000 n=19+18) BM_CommentLines/128/30/2 288ms ± 5% 246ms ± 3% -14.70% (p=0.000 n=19+20) BM_CommentLines/1/70/2 19.9ms ± 1% 19.4ms ± 1% -2.56% (p=0.000 n=19+18) BM_CommentLines/4/70/2 30.0ms ± 2% 28.5ms ± 2% -5.20% (p=0.000 n=19+19) BM_CommentLines/128/70/2 300ms ± 3% 265ms ± 3% -11.69% (p=0.000 n=18+20) BM_CommentLines/1/0/8 16.7ms ± 2% 16.3ms ± 2% -2.22% (p=0.000 n=19+19) BM_CommentLines/4/0/8 26.4ms ± 1% 25.2ms ± 1% -4.42% (p=0.000 n=19+18) BM_CommentLines/128/0/8 285ms ± 2% 248ms ± 1% -13.16% (p=0.000 n=19+18) BM_CommentLines/1/30/8 17.9ms ± 2% 17.4ms ± 1% -2.99% (p=0.000 n=19+17) BM_CommentLines/4/30/8 32.0ms ± 1% 30.5ms ± 2% -4.73% (p=0.000 n=19+20) BM_CommentLines/128/30/8 320ms ± 2% 280ms ± 3% -12.52% (p=0.000 n=18+20) BM_CommentLines/1/70/8 20.7ms ± 2% 20.0ms ± 2% -3.22% (p=0.000 n=19+19) BM_CommentLines/4/70/8 33.5ms ± 2% 31.8ms ± 1% -5.23% (p=0.000 n=19+19) BM_CommentLines/128/70/8 338ms ± 3% 298ms ± 2% -11.64% (p=0.000 n=19+20) ``` |
||
|
Richard Smith
|
4e64b1948d |
Bare-bones support for forward-declared classes. (#3294)
This is mostly scaffolding, but is just about enough for pointers to classes to work properly as types. |
||
|
Chandler Carruth
|
0e2b6c7f1a |
Optimize runs of horizontal whitespace. (#3288)
So, this is a somewhat fun, simple improvement. =] Just use a loop and count runs of whitespace. I didn't even work all that hard to make the loop fast, but it seems great. Makes long runs of indentation more than 2x faster at basically no code complexity. I thought about doing this for vertical whitespace as well but it's not easy to do, and didn't seem worth adding complexity. Huge runs of blank lines aren't nearly as common as lots of indentation. I do have a plan for an analogous optimization for comment blocks, but want to simplify some other code first. We could also make this (hilariously) faster with some judicious use of SIMD or clever use of a string function, but it doesn't seem worth it given how fast the simple loop is already. I didn't work to get a high N count and so there's plenty of noise here, but the benchmark data speaks for itself: ``` BM_ValidKeywords 2.71ms ± 1% 2.75ms ± 1% +1.32% (p=0.016 n=5+5) BM_ValidIdentifiers<1, 64, false> 9.71ms ± 2% 9.74ms ± 1% ~ (p=1.000 n=5+5) BM_ValidIdentifiers<1, 1, true> 3.17ms ± 1% 3.22ms ± 2% ~ (p=0.151 n=5+5) BM_ValidIdentifiers<3, 5, true> 11.5ms ± 2% 11.6ms ± 3% ~ (p=0.548 n=5+5) BM_ValidIdentifiers<3, 16, true> 11.3ms ± 3% 11.6ms ± 3% ~ (p=0.095 n=5+5) BM_ValidIdentifiers<12, 64, true> 12.8ms ± 1% 12.8ms ± 1% ~ (p=1.000 n=5+5) BM_HorizontalWhitespace/1 11.6ms ± 1% 11.7ms ± 2% ~ (p=0.310 n=5+5) BM_HorizontalWhitespace/4 12.9ms ± 1% 11.8ms ± 0% -8.50% (p=0.008 n=5+5) BM_HorizontalWhitespace/16 17.3ms ± 4% 12.5ms ± 1% -27.69% (p=0.008 n=5+5) BM_HorizontalWhitespace/64 28.9ms ± 3% 16.0ms ± 2% -44.88% (p=0.008 n=5+5) BM_HorizontalWhitespace/128 47.6ms ± 3% 21.0ms ± 0% -55.88% (p=0.016 n=5+4) BM_RandomSource 7.92ms ± 1% 7.70ms ± 3% -2.74% (p=0.016 n=5+5) BM_GroupingSymbols/1/0/0 5.92ms ± 2% 5.86ms ± 1% ~ (p=0.310 n=5+5) BM_GroupingSymbols/2/0/0 5.30ms ± 1% 5.01ms ± 1% -5.52% (p=0.008 n=5+5) BM_GroupingSymbols/3/0/0 4.48ms ± 0% 3.95ms ± 1% -11.69% (p=0.008 n=5+5) BM_GroupingSymbols/4/0/0 4.61ms ± 1% 3.73ms ± 1% -19.12% (p=0.008 n=5+5) BM_GroupingSymbols/8/0/0 5.34ms ± 6% 3.05ms ± 1% -42.82% (p=0.008 n=5+5) BM_GroupingSymbols/16/0/0 6.44ms ± 1% 3.20ms ± 2% -50.24% (p=0.008 n=5+5) BM_GroupingSymbols/32/0/0 10.3ms ± 5% 4.2ms ± 1% -59.81% (p=0.008 n=5+5) BM_GroupingSymbols/0/1/0 5.17ms ± 2% 5.17ms ± 1% ~ (p=0.690 n=5+5) BM_GroupingSymbols/0/2/0 4.04ms ± 1% 4.04ms ± 2% ~ (p=1.000 n=5+5) BM_GroupingSymbols/0/3/0 2.89ms ± 1% 2.89ms ± 1% ~ (p=1.000 n=5+5) BM_GroupingSymbols/0/4/0 2.54ms ± 1% 2.55ms ± 1% ~ (p=0.421 n=5+5) BM_GroupingSymbols/0/8/0 1.65ms ± 2% 1.67ms ± 1% ~ (p=0.310 n=5+5) BM_GroupingSymbols/0/16/0 1.18ms ± 1% 1.19ms ± 2% ~ (p=0.222 n=5+5) BM_GroupingSymbols/0/32/0 948µs ± 1% 957µs ± 2% ~ (p=0.310 n=5+5) BM_GroupingSymbols/0/0/1 5.17ms ± 1% 5.16ms ± 2% ~ (p=0.841 n=5+5) BM_GroupingSymbols/0/0/2 4.03ms ± 2% 4.04ms ± 2% ~ (p=0.548 n=5+5) BM_GroupingSymbols/0/0/3 2.89ms ± 1% 2.88ms ± 1% ~ (p=0.841 n=5+5) BM_GroupingSymbols/0/0/4 2.54ms ± 1% 2.54ms ± 2% ~ (p=0.548 n=5+5) BM_GroupingSymbols/0/0/8 1.66ms ± 3% 1.67ms ± 2% ~ (p=0.690 n=5+5) BM_GroupingSymbols/0/0/16 1.19ms ± 1% 1.18ms ± 2% ~ (p=0.548 n=5+5) BM_GroupingSymbols/0/0/32 949µs ± 2% 955µs ± 1% ~ (p=0.310 n=5+5) BM_GroupingSymbols/32/1/0 10.0ms ± 1% 4.0ms ± 1% -59.76% (p=0.008 n=5+5) BM_GroupingSymbols/32/2/0 9.73ms ± 2% 3.92ms ± 2% -59.73% (p=0.008 n=5+5) BM_GroupingSymbols/32/3/0 9.40ms ± 2% 3.84ms ± 3% -59.10% (p=0.008 n=5+5) BM_GroupingSymbols/32/4/0 9.20ms ± 2% 3.74ms ± 1% -59.35% (p=0.008 n=5+5) BM_GroupingSymbols/32/8/0 8.38ms ± 2% 3.50ms ± 1% -58.22% (p=0.008 n=5+5) BM_GroupingSymbols/32/16/0 7.18ms ± 2% 3.06ms ± 3% -57.41% (p=0.008 n=5+5) BM_GroupingSymbols/32/32/0 5.53ms ± 1% 2.44ms ± 2% -55.99% (p=0.008 n=5+5) BM_GroupingSymbols/32/32/1 5.47ms ± 2% 2.40ms ± 2% -56.08% (p=0.008 n=5+5) BM_GroupingSymbols/32/32/2 5.41ms ± 3% 2.40ms ± 2% -55.66% (p=0.008 n=5+5) BM_GroupingSymbols/32/32/3 5.28ms ± 2% 2.37ms ± 2% -55.18% (p=0.008 n=5+5) BM_GroupingSymbols/32/32/4 5.25ms ± 2% 2.34ms ± 2% -55.52% (p=0.008 n=5+5) BM_GroupingSymbols/32/32/8 5.03ms ± 3% 2.25ms ± 1% -55.29% (p=0.008 n=5+5) BM_GroupingSymbols/32/32/16 4.62ms ± 1% 2.15ms ± 3% -53.41% (p=0.008 n=5+5) BM_GroupingSymbols/32/32/32 3.99ms ± 1% 1.86ms ± 1% -53.24% (p=0.008 n=5+5) BM_BlankLines/1 12.8ms ± 1% 12.6ms ± 2% ~ (p=0.310 n=5+5) BM_BlankLines/4 16.0ms ± 3% 15.8ms ± 1% ~ (p=0.310 n=5+5) BM_BlankLines/16 33.1ms ± 2% 32.1ms ± 1% -3.29% (p=0.008 n=5+5) BM_BlankLines/64 84.6ms ± 3% 84.2ms ± 3% ~ (p=0.690 n=5+5) BM_BlankLines/128 159ms ± 4% 155ms ± 3% ~ (p=0.310 n=5+5) BM_CommentLines/1/0/0 14.5ms ± 2% 14.5ms ± 2% ~ (p=0.690 n=5+5) BM_CommentLines/4/0/0 19.0ms ± 1% 19.0ms ± 1% ~ (p=0.548 n=5+5) BM_CommentLines/128/0/0 188ms ± 2% 186ms ± 1% ~ (p=0.151 n=5+5) BM_CommentLines/1/30/0 14.8ms ± 3% 14.7ms ± 2% ~ (p=0.421 n=5+5) BM_CommentLines/4/30/0 21.8ms ± 1% 21.3ms ± 4% ~ (p=0.095 n=5+5) BM_CommentLines/128/30/0 201ms ± 1% 200ms ± 2% ~ (p=0.421 n=5+5) BM_CommentLines/1/70/0 15.5ms ± 3% 15.2ms ± 4% ~ (p=0.095 n=5+5) BM_CommentLines/4/70/0 23.2ms ± 1% 22.5ms ± 2% -2.76% (p=0.008 n=5+5) BM_CommentLines/128/70/0 213ms ± 1% 209ms ± 1% -1.54% (p=0.016 n=5+5) BM_CommentLines/1/0/2 15.3ms ± 1% 14.5ms ± 1% -4.99% (p=0.008 n=5+5) BM_CommentLines/4/0/2 21.4ms ± 1% 20.2ms ± 2% -5.48% (p=0.008 n=5+5) BM_CommentLines/128/0/2 242ms ± 2% 218ms ± 5% -10.13% (p=0.008 n=5+5) BM_CommentLines/1/30/2 15.7ms ± 4% 15.1ms ± 2% -3.37% (p=0.008 n=5+5) BM_CommentLines/4/30/2 24.3ms ± 1% 22.5ms ± 3% -7.42% (p=0.008 n=5+5) BM_CommentLines/128/30/2 268ms ± 2% 240ms ± 3% -10.22% (p=0.008 n=5+5) BM_CommentLines/1/70/2 16.1ms ± 2% 15.3ms ± 3% -5.24% (p=0.008 n=5+5) BM_CommentLines/4/70/2 25.7ms ± 3% 24.0ms ± 3% -6.69% (p=0.008 n=5+5) BM_CommentLines/128/70/2 272ms ± 1% 247ms ± 1% -9.21% (p=0.008 n=5+5) BM_CommentLines/1/0/8 17.2ms ± 5% 14.8ms ± 2% -14.32% (p=0.008 n=5+5) BM_CommentLines/4/0/8 30.4ms ± 2% 20.8ms ± 2% -31.47% (p=0.008 n=5+5) BM_CommentLines/128/0/8 463ms ± 1% 260ms ± 1% -43.89% (p=0.008 n=5+5) BM_CommentLines/1/30/8 17.2ms ± 2% 15.2ms ± 2% -12.01% (p=0.008 n=5+5) BM_CommentLines/4/30/8 32.4ms ± 1% 23.2ms ± 3% -28.61% (p=0.008 n=5+5) BM_CommentLines/128/30/8 498ms ± 3% 287ms ± 2% -42.32% (p=0.008 n=5+5) BM_CommentLines/1/70/8 17.6ms ± 2% 15.5ms ± 3% -11.61% (p=0.008 n=5+5) BM_CommentLines/4/70/8 34.0ms ± 4% 24.7ms ± 3% -27.31% (p=0.008 n=5+5) BM_CommentLines/128/70/8 497ms ± 2% 297ms ± 4% -40.29% (p=0.008 n=5+5) ``` Stacked on top of #3287 -- only the last commit should be reviewed here. |
||
|
Richard Smith
|
870ccf3eac |
Suppress follow-on errors for various forms of expression. (#3292)
Don't produce an error diagnostic if any of the information leading to identifying that error is itself known to be affected by an already-diagnosed error. |
||
|
Chandler Carruth
|
a79ea4b28d |
Do more precise table dispatch for symbols. (#3287)
When originally switching to the table dispatch approach we discussed that it'd be nice to disentangle the monolithic symbol lexing routine with this as we'll typically have fairly precise dispatch. This is especially true for grouping symbols, which in Carbon are all constructively one-character (at this point). I think this provides a substantial improvement to the clarity of the code by disentangling the different paths. It also allowed a bunch of simplifications / clarifications to exactly what the behavior with closing invalid groups actually involves currently. This was initially motivated by code organization improvements, and any performance wins were speculative. However, when benchmarking it surfaced a problem that hadn't been clear -- we're generating too many distinct functions here, and the table-based dispatch slows down in the face of that. So this PR also includes a fix for that, removing the template-generated fan-out of dispatch functions for distinct symbols. Instead, we have a dedicated table to translate one character into the token kinds. This seems to work quite well, avoiding the huge branch-y structure and just do fairly cheap table translation & dispatch for all one-character symbols. Building the table requires the token kinds to be default constructable, so this also enables that and arranges for the zero-value kind to be the error kind. Combined, this is a modest speedup for *non* grouping symbols (3-4%, a bit noisy). And in some cases it is a huge speedup for grouping symbols (>10%). Raw benchmark data with 20 runs before/after -- despite the # of runs, the grouping symbols benchmarks were frustratingly noisy in non-uniform ways that couldn't fully be accounted for here. Still, this seems like an overall improvement. ``` BM_RandomSource 7.98ms ± 2% 7.73ms ± 3% -3.12% (p=0.000 n=18+19) BM_GroupingSymbols/1/0/0 5.90ms ± 2% 5.82ms ± 4% -1.38% (p=0.001 n=20+20) BM_GroupingSymbols/2/0/0 5.21ms ± 2% 5.15ms ± 2% -1.14% (p=0.002 n=20+18) BM_GroupingSymbols/3/0/0 4.42ms ± 2% 4.34ms ± 2% -1.87% (p=0.000 n=19+18) BM_GroupingSymbols/4/0/0 4.29ms ± 2% 4.38ms ± 5% ~ (p=0.297 n=17+20) BM_GroupingSymbols/8/0/0 5.09ms ±10% 5.10ms ± 7% ~ (p=0.919 n=18+20) BM_GroupingSymbols/16/0/0 6.35ms ± 8% 6.29ms ± 6% ~ (p=0.201 n=20+20) BM_GroupingSymbols/32/0/0 9.88ms ± 2% 9.83ms ± 1% ~ (p=0.167 n=18+20) BM_GroupingSymbols/0/1/0 5.12ms ± 2% 5.01ms ± 2% -2.14% (p=0.000 n=20+19) BM_GroupingSymbols/0/2/0 4.01ms ± 2% 3.93ms ± 4% -2.03% (p=0.000 n=20+19) BM_GroupingSymbols/0/3/0 2.92ms ± 3% 2.81ms ± 2% -3.87% (p=0.000 n=20+19) BM_GroupingSymbols/0/4/0 2.61ms ± 3% 2.47ms ± 2% -5.30% (p=0.000 n=20+18) BM_GroupingSymbols/0/8/0 1.77ms ± 3% 1.61ms ± 2% -8.91% (p=0.000 n=18+19) BM_GroupingSymbols/0/16/0 1.41ms ± 3% 1.16ms ± 4% -17.66% (p=0.000 n=20+20) BM_GroupingSymbols/0/32/0 1.10ms ± 2% 0.92ms ± 3% -16.36% (p=0.000 n=20+17) BM_GroupingSymbols/0/0/1 5.09ms ± 2% 5.03ms ± 3% -1.11% (p=0.001 n=20+18) BM_GroupingSymbols/0/0/2 4.01ms ± 2% 3.91ms ± 2% -2.67% (p=0.000 n=20+18) BM_GroupingSymbols/0/0/3 2.93ms ± 3% 2.81ms ± 2% -4.23% (p=0.000 n=20+19) BM_GroupingSymbols/0/0/4 2.59ms ± 2% 2.48ms ± 3% -4.48% (p=0.000 n=20+19) BM_GroupingSymbols/0/0/8 1.75ms ± 1% 1.62ms ± 3% -7.65% (p=0.000 n=17+19) BM_GroupingSymbols/0/0/16 1.40ms ± 2% 1.15ms ± 3% -17.67% (p=0.000 n=19+20) BM_GroupingSymbols/0/0/32 1.10ms ± 2% 0.92ms ± 3% -15.91% (p=0.000 n=20+19) BM_GroupingSymbols/32/1/0 9.62ms ± 2% 9.65ms ± 2% ~ (p=0.654 n=18+20) BM_GroupingSymbols/32/2/0 9.41ms ± 2% 9.37ms ± 2% ~ (p=0.095 n=20+19) BM_GroupingSymbols/32/3/0 9.13ms ± 2% 9.13ms ± 3% ~ (p=0.687 n=19+20) BM_GroupingSymbols/32/4/0 8.93ms ± 1% 8.87ms ± 2% -0.69% (p=0.010 n=20+18) BM_GroupingSymbols/32/8/0 8.15ms ± 2% 8.14ms ± 3% ~ (p=0.729 n=19+19) BM_GroupingSymbols/32/16/0 7.04ms ± 3% 6.92ms ± 1% -1.71% (p=0.000 n=20+18) BM_GroupingSymbols/32/32/0 5.48ms ± 2% 5.38ms ± 3% -1.81% (p=0.000 n=20+20) BM_GroupingSymbols/32/32/1 5.39ms ± 2% 5.29ms ± 2% -1.87% (p=0.000 n=19+19) BM_GroupingSymbols/32/32/2 5.34ms ± 2% 5.21ms ± 1% -2.45% (p=0.000 n=20+18) BM_GroupingSymbols/32/32/3 5.27ms ± 3% 5.16ms ± 2% -2.18% (p=0.000 n=20+19) BM_GroupingSymbols/32/32/4 5.21ms ± 2% 5.10ms ± 3% -2.11% (p=0.000 n=19+20) BM_GroupingSymbols/32/32/8 4.98ms ± 2% 4.83ms ± 2% -2.85% (p=0.000 n=19+19) BM_GroupingSymbols/32/32/16 4.55ms ± 2% 4.45ms ± 2% -2.25% (p=0.000 n=18+20) BM_GroupingSymbols/32/32/32 3.95ms ± 2% 3.84ms ± 2% -2.98% (p=0.000 n=19+20) ``` |
||
|
Richard Smith
|
17e053798c |
Fix incorrect argc calculation. (#3290)
Fix off-by-one error: the terminating `nullptr` in `argv` is not included in `argc`. This is currently causing crashing tests to crash again in their crash handler, meaning we don't get a backtrace or CHECK-failure message. |
||
|
Chandler Carruth
|
9ce9572522 |
Add a benchmark to stress test grouping symbols. (#3286)
This isn't really representative of anything, but it should help make it obvious when the handling of grouping symbols improves or regresses. Notably, the random source microbenchmark has *no* grouping symbols (in order to let it be random but always lexically valid), and so it's especially useful to have something that checks grouping symbols given their prevalence in realistic source code. |
||
|
josh11bandRichard Smith
|
8d0831f431 |
Made function and namespace nodes typed to remove a crash (#3285)
Bug found by fuzzing. Problem was untyped SemIR nodes had an invalid type id, which was retrieved by `HandlePrefixOperator` and then passed to `context.GetUnqualifiedType`, ultimately performing an invalid access in `semantics_ir_->GetNode`. We prefer to make a placeholder type for functions and namespaces to remove the need for checking for the untyped case everywhere. Eventually functions will have their own types, but this approach will be needed for namespaces (and perhaps other non-first-class entities like unbound methods and interface members) long term. Co-authored-by: Richard Smith <richard@metafoo.co.uk> |
||
|
|
6ba8712fbd |
Predetermine all the line splits in the lexer. (#3278)
## Summary ## Restructures the lexer to first scan the entire source text for newlines and create all the line structures needed. Doing this up-front makes it easy to produce an optimized version with minimal complexity. Currently, it leverages the system `memchr`, but even when expanded to handle more complex cases like CR+LF line endings, being isolated in this way will result in a significantly simpler implementation. This change improves the lexing of comment lines significantly by skipping their contents immediately. The overhead of the pre-scan is unmeasurable in all realistic benchmarks, and 10-30% in benchmarks consisting almost entirely of blank lines or comments. The improvement of comment lexing with average length comment lines mixed with code starts at 20% and goes up. Regressing blank line handling for non-empty comments seems like the right tradeoff (by far). ## Background and details ## One weak point in the lexer implementation were large runs of comments. While those aren't terribly common, they shouldn't present a hazard to the lexer performance. A bit more common is a pattern of comments like the following: ```carbon // Some method comment here. fn SomeMethodName(...) -> ...; // Some other method comment here. fn SomeOtherMethodName(...) -> ...; ``` Here, the lexer spends an inordinate amount of time getting from the `\n` after the first semicolon to the `fn` token. It has to skip a blank line, scan a line, find the `//` comment start, then scan to find the next `\n`, and then scan horizontal whitespace, etc. It is tempting to build a scanner *exactly* for this. In fact, I built one, and I can publish it in a PR if folks are interested in what it looks like. For x86-64, the PSHUFB trick used for scanning identifiers technically works. But it is *complicated*. Amazingly so. 150 lines of very subtle code with subtle performance pitfalls at ever turn. I felt very uncomfortable submitting it, but we can always go back to it. Nothing I've come up with quite matches it for sheer speed. However, most of the complexity and time is spent walking from a `//` to the end of the line. And *that* is something we can do very simply. In fact, there is a tuned function for that in libc: `memchr`. Using this we can build a very fast and much simpler scanner to split lines up-front. This PR uses that and a carefully crafted fast loop to first build up all the line info we need. Getting this to be as fast as possible required some other subtle changes, for example always creating a line structure that goes from the last `\n` and the end of the file. We then back up the EOF token to avoid surfacing this to users. The nice thing is the EOF token isn't part of any hot loop, and so this removes branches everywhere else at modest complexity. Once we have that, the rest of the lexer just needs to keep track of its current line in order to record column offsets. I've taken some care to try and optimize the lexer's usage of the line structures but there are more opportunities here I suspect. Combined, this gets much but not all of the performance of a huge SIMD scanner for newline-through-to-next-token. For extreme cases (100s of blank lines or empty comment lines between tokens) the holistic scanner is of course still much faster, but those don't seem nearly worth the cost. I was initially worried about the overhead of taking two passes over the source text, but in practice I've not been able to measure any appreciable cost to this with realistic source files. In some cases benchmarks with no newlines get *faster* because we use a much more efficient approach to fetch the source text into cache as a happenstance. And that in turn makes the byte-wise dispatched loop run faster as it stalls less. I'm particularly happy with this approach because it seems very clear how to extend this to support CR+LF, bare CR, and even complex mixtures without any significant speed cost. That wasn't at all true for the other approaches explored. I may try some further PRs to smooth out the last bits of slowness here, but already this is working excellent for me in practice. My 10mloc test case is down to 2.3s to lex. ## Raw benchmark data Using a tool that runs benchmarks before and after and analyzes the results, the following summarizes the CPU-time impact, each of these for lexing 100k tokens: ``` BM_ValidKeywords 2.57ms ± 1% 2.58ms ± 0% ~ (p=0.190 n=5+4) BM_ValidIdentifiers<1, 64, false> 9.24ms ± 4% 9.31ms ± 4% ~ (p=0.421 n=5+5) BM_ValidIdentifiers<1, 1, true> 3.05ms ± 4% 3.11ms ± 4% ~ (p=0.222 n=5+5) BM_ValidIdentifiers<3, 5, true> 10.9ms ± 0% 11.1ms ± 1% +1.76% (p=0.016 n=4+5) BM_ValidIdentifiers<3, 16, true> 11.1ms ± 7% 11.0ms ± 1% ~ (p=0.310 n=5+5) BM_ValidIdentifiers<12, 64, true> 12.2ms ± 1% 12.3ms ± 2% ~ (p=0.111 n=4+5) BM_HorizontalWhitespace/1 11.2ms ± 6% 11.1ms ± 2% ~ (p=0.841 n=5+5) BM_HorizontalWhitespace/4 12.0ms ± 3% 12.0ms ± 2% ~ (p=0.548 n=5+5) BM_HorizontalWhitespace/16 16.2ms ± 6% 15.9ms ± 8% ~ (p=0.690 n=5+5) BM_HorizontalWhitespace/64 27.7ms ± 3% 28.4ms ± 3% ~ (p=0.151 n=5+5) BM_HorizontalWhitespace/128 44.3ms ± 1% 45.6ms ± 6% +3.15% (p=0.032 n=5+5) BM_RandomSource 7.75ms ± 2% 7.72ms ± 1% ~ (p=1.000 n=5+5) BM_BlankLines/1 11.7ms ± 1% 12.1ms ± 1% +3.46% (p=0.008 n=5+5) BM_BlankLines/4 14.0ms ± 2% 15.2ms ± 3% +8.12% (p=0.008 n=5+5) BM_BlankLines/16 23.5ms ± 2% 31.1ms ± 4% +32.26% (p=0.008 n=5+5) BM_BlankLines/64 75.3ms ± 1% 81.2ms ± 3% +7.83% (p=0.008 n=5+5) BM_BlankLines/128 133ms ± 3% 150ms ± 2% +12.74% (p=0.008 n=5+5) BM_CommentLines/1/0/0 13.1ms ± 0% 13.7ms ± 1% +5.11% (p=0.008 n=5+5) BM_CommentLines/4/0/0 16.6ms ± 1% 18.2ms ± 4% +9.56% (p=0.008 n=5+5) BM_CommentLines/128/0/0 169ms ± 4% 182ms ± 1% +7.24% (p=0.008 n=5+5) BM_CommentLines/1/30/0 18.7ms ± 5% 14.1ms ± 0% -24.84% (p=0.008 n=5+5) BM_CommentLines/4/30/0 36.5ms ± 6% 20.6ms ± 3% -43.59% (p=0.008 n=5+5) BM_CommentLines/128/30/0 525ms ± 4% 198ms ± 1% -62.38% (p=0.008 n=5+5) BM_CommentLines/1/70/0 23.4ms ± 6% 14.7ms ± 2% -37.15% (p=0.008 n=5+5) BM_CommentLines/4/70/0 53.3ms ± 7% 22.4ms ± 4% -57.99% (p=0.008 n=5+5) BM_CommentLines/128/70/0 1.05s ± 4% 0.21s ± 2% -80.31% (p=0.008 n=5+5) BM_CommentLines/1/0/2 14.1ms ± 6% 14.3ms ± 1% ~ (p=0.151 n=5+5) BM_CommentLines/4/0/2 19.4ms ± 5% 20.1ms ± 1% ~ (p=0.151 n=5+5) BM_CommentLines/128/0/2 238ms ± 8% 229ms ± 0% ~ (p=0.151 n=5+5) BM_CommentLines/1/30/2 19.2ms ± 7% 14.6ms ± 1% -23.87% (p=0.008 n=5+5) BM_CommentLines/4/30/2 40.3ms ±13% 22.3ms ± 4% -44.63% (p=0.008 n=5+5) BM_CommentLines/128/30/2 568ms ± 7% 254ms ± 3% -55.28% (p=0.008 n=5+5) BM_CommentLines/1/70/2 23.3ms ± 1% 15.0ms ± 3% -35.61% (p=0.016 n=4+5) BM_CommentLines/4/70/2 57.2ms ± 9% 24.1ms ± 2% -57.81% (p=0.008 n=5+5) BM_CommentLines/128/70/2 1.07s ± 0% 0.26s ± 2% -75.51% (p=0.016 n=4+5) BM_CommentLines/1/0/8 15.9ms ± 7% 16.0ms ± 1% ~ (p=0.151 n=5+5) BM_CommentLines/4/0/8 24.2ms ± 6% 27.9ms ± 2% +15.36% (p=0.008 n=5+5) BM_CommentLines/128/0/8 386ms ± 4% 445ms ± 1% +15.28% (p=0.008 n=5+5) BM_CommentLines/1/30/8 20.6ms ± 5% 16.3ms ± 1% -20.95% (p=0.008 n=5+5) BM_CommentLines/4/30/8 45.3ms ± 6% 30.3ms ± 3% -32.98% (p=0.008 n=5+5) BM_CommentLines/128/30/8 699ms ± 3% 477ms ± 3% -31.83% (p=0.008 n=5+5) BM_CommentLines/1/70/8 25.7ms ± 5% 16.8ms ± 2% -34.67% (p=0.008 n=5+5) BM_CommentLines/4/70/8 62.0ms ± 4% 31.6ms ± 2% -49.10% (p=0.008 n=5+5) BM_CommentLines/128/70/8 1.20s ± 2% 0.48s ± 4% -59.60% (p=0.008 n=5+5) ``` The horizontal whitespace benchmark (and all of the non-line-oriented ones) are noisier than they appear here but do show some improvements (surprisingly). My guess is that it has a lot to do with system load, as the advantage is that we're using a vectorized loop to scan the text first and then doing the byte-dispatched loop. So when the cache is a bit slower to populate, the vectorized version starts to be faster. --------- Co-authored-by: Richard Smith <richard@metafoo.co.uk> Co-authored-by: josh11b <josh11b@users.noreply.github.com> |
||
|
|
5041a14f59 |
Add whitespace- and comment-specific benchmarking. (#3276)
These benchmarks zero-in and stress test horizontal and vertical whitespace as well as comment lexing performance. They set up essentially a worst-case scenario of ramping up whitespace between very sparse tokens to show how the lexer copes with this. The horizontal whitespace benchmark is perhaps less important as frequent runs of 50-characters of horizontal whitespace are relatively rare already, and likely to be exceedingly rare without trailing comments. But its good to include for completeness and it shows reasonably strong performance with the current table-dispatch approach. The blank line and comment line benchmarks are much more important. Lots of code is relatively line-sparse, especially API files that are perhaps the most useful to parse quickly. And many of these are a mixture of sparse with blank lines and sparse with large comment blocks. The benchmarks show that there are some serious limits here, even falling below 100k tokens per second throughput on some of the stress tests here. --------- Co-authored-by: Richard Smith <richard@metafoo.co.uk> Co-authored-by: josh11b <josh11b@users.noreply.github.com> |
||
|
Chandler CarruthandRichard Smith
|
2d735bbc51 |
Enhance the main lexer benchmark. (#3275)
This generalizes the main lexer benchmark's source generation to be a bit more comprehensive, specifically including whitespace and comments. Without these, we're missing a key part of the lexer's performance. This also tidies up a bit of the code and adds a more specific distribution of the different factors in lexing based on analysis of LLVM's source code. The enhancements to the source statistics script that helped collect the data here will be in a separate PR. With this, the output on my AMD cloud instance is: ``` ------------------------------------------------------------------------------------------------------------------------- Benchmark Time CPU Iterations bytes_per_second tokens_per_second ------------------------------------------------------------------------------------------------------------------------- BM_ValidKeywords 2809011 ns 2808912 ns 243 212.616M/s 35.601M/s BM_ValidIdentifiers<1, 64, false> 11461783 ns 11461652 ns 61 128.499M/s 8.72475M/s BM_ValidIdentifiers<1, 1, true> 3397293 ns 3397240 ns 208 84.2158M/s 29.4357M/s BM_ValidIdentifiers<3, 5, true> 14165143 ns 14164962 ns 51 40.3956M/s 7.05967M/s BM_ValidIdentifiers<3, 16, true> 15283128 ns 15282583 ns 47 71.7623M/s 6.5434M/s BM_ValidIdentifiers<12, 64, true> 17417323 ns 17417109 ns 41 219.007M/s 5.74148M/s ------------------------------------------------------------------------------------------------------------------------------------------ Benchmark Time CPU Iterations bytes_per_second lines_per_second tokens_per_second ------------------------------------------------------------------------------------------------------------------------------------------ BM_RandomSource 8132211 ns 8132227 ns 84 137.223M/s 3.9036M/s 12.2968M/s BM_SpeedOfLightStrCpy 29610 ns 29608 ns 24631 36.8065G/s 1072.17M/s 3.37745G/s BM_SpeedOfLightDispatch<1> 2144418 ns 2144421 ns 327 520.387M/s 14.8035M/s 46.6326M/s BM_SpeedOfLightDispatch<2> 1945954 ns 1945827 ns 351 573.498M/s 16.3144M/s 51.392M/s BM_SpeedOfLightDispatch<4> 2519565 ns 2519467 ns 292 442.923M/s 12.5999M/s 39.6909M/s BM_SpeedOfLightDispatch<8> 3011965 ns 3011968 ns 238 370.498M/s 10.5396M/s 33.2009M/s BM_SpeedOfLightDispatch<16> 4379575 ns 4379579 ns 160 254.803M/s 7.24841M/s 22.8332M/s BM_SpeedOfLightDispatch<32> 6678423 ns 6678353 ns 102 167.096M/s 4.75342M/s 14.9738M/s BM_SpeedOfLightDispatch<MaxDispatchTargets> 9373075 ns 9372688 ns 75 119.062M/s 3.38697M/s 10.6693M/s ``` I've compared the profile of the `BM_RandomSource` benchmark with this change and it largely corresponds to what I expect based on profiling hand-crafted Carbon inputs. And as you can see, we're closing in on lexing at least hitting the 10-million-lines-per-second mark. =] --------- Co-authored-by: Richard Smith <richard@metafoo.co.uk> |
||
|
Richard Smithandjosh11b
|
9c3e25decf |
Add SemIR::Node::Is<TypedNode>(). (#3284)
This allows node kind checks to be written a little more concisely. --------- Co-authored-by: josh11b <josh11b@users.noreply.github.com> |