carbon-language/carbon-lang
1
0
Fork 0
mirror of https://github.com/carbon-language/carbon-lang.git synced 2026-10-05 22:02:55 +01:00
Code Issues Packages Projects Releases 10 Wiki Activity

Speed up type literal lexing and make it more strict. (#4430)

Browse Source 
This rejects type literals with more digits than we can lex without
APInt's help, and using a custom diagnostic. This is a pretty arbitrary
implementation limit, I'm wide open to even more strict rules here.

Despite no special casing and a very simplistic approach, by not using
APInt this completely eliminates the lexing overhead for `i32` in the
generated compilation benchmark where that specific type literal is very
common. We see a 10% improvement in lexing there:
```
BM_CompileAPIFileDenseDecls<Phase::Lex>/256        39.0µs ± 4%  34.8µs ± 2%  -10.86%  (p=0.000 n=19+20)
BM_CompileAPIFileDenseDecls<Phase::Lex>/1024        180µs ± 1%   158µs ± 2%  -12.22%  (p=0.000 n=18+20)
BM_CompileAPIFileDenseDecls<Phase::Lex>/4096        731µs ± 2%   641µs ± 1%  -12.31%  (p=0.000 n=18+20)
BM_CompileAPIFileDenseDecls<Phase::Lex>/16384      3.20ms ± 2%  2.86ms ± 2%  -10.47%  (p=0.000 n=18+19)
BM_CompileAPIFileDenseDecls<Phase::Lex>/65536      13.8ms ± 1%  12.4ms ± 2%   -9.78%  (p=0.000 n=18+19)
BM_CompileAPIFileDenseDecls<Phase::Lex>/262144     64.0ms ± 2%  58.4ms ± 2%   -8.70%  (p=0.000 n=19+18)
```

This starts to fix a TODO in the diagnostic for these by giving a
reasonably good diagnostic about a very large type literal. However, in
practice it regresses the diagnostics because error tokens produce noisy
extraneous diagnostics from parse and check currently. Leaving the TODO
there, and I have a follow-up PR to start improving the extraneous
diagnostics.
This commit is contained in:
Chandler Carruth
2024-10-24 21:43:35 +00:00
committed by GitHub
parent b67d03126e
commit 577fda1ca2
6 changed files with 112 additions and 139 deletions
Download Patch File Download Diff File Expand all files Collapse all files
toolchain/testing/compile_helper.cpp
+3 -2
View File
@@ -17,9 +17,10 @@ auto CompileHelper::GetTokenizedBuffer(llvm::StringRef text,
return token_storage_.front();
}
auto CompileHelper::GetTokenizedBufferWithSharedValueStore(llvm::StringRef text)
auto CompileHelper::GetTokenizedBufferWithSharedValueStore(
llvm::StringRef text, DiagnosticConsumer* consumer)
-> std::pair<Lex::TokenizedBuffer&, SharedValueStores&> {
auto& tokens = GetTokenizedBuffer(text);
auto& tokens = GetTokenizedBuffer(text, consumer);
return {tokens, value_store_storage_.front()};
}