mirror of
https://github.com/carbon-language/carbon-lang.git
synced 2026-09-24 22:02:23 +01:00
acbe6530c39c2240a883419a0ffa42e79cff50e0
25
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jon Ross-Perkins
|
acbe6530c3 |
Move diagnostics into a namespace (#5173)
What this really does is avoids shadowing names, so that we can comfortable have things like `Check::DiagnosticEmitter` or `Check::DiagnosticLoc` without shadowing being a concern. Note, down this path I'm also thinking about: - Renaming misc DiagnosticConsumer/DiagnosticEmitter classes, possibly just to DiagnosticConsumer/DiagnosticEmitter (so `Check::DiagnosticEmitter` instead of `SemIRLocDiagnosticEmitter`). - Dropping `Diagnostic` from `Emitter::DiagnosticBuilder`. - But not for `Check::DiagnosticBuilder`, because `Check::Builder` would be ambiguous. - Renaming diagnostics/diagnostic_* to drop "diagnostic". [Discussion about SemIRLoc -> DiagnosticLoc](https://discord.com/channels/655572317891461132/655578254970716160/1353771570463768698) reminded me of this (in particular the older [Check::DiagnosticBuilder discussion](https://discord.com/channels/655572317891461132/655578254970716160/1344363562608627763)), but I'd only do that rename if there's matching consensus about a path forward where we keep SemIRLoc, and in a way that it's only ever used for diagnostics (the divergence from which is at the root of current LocId discussion). I'm trying to keep that separate from a namespace addition for clarity. |
||
|
Jon Ross-Perkins
|
2fef1cb713 |
Switch to trailing returns in toolchain and related code. (#4919)
Also makes the style guide explicitly comment on void, but this was the intent IIRC because it matches Carbon's `-> ()` (and "always" versus "except for void", which we definitely went back and forth on). Includes adjusting function pointers, which I definitely forget this syntax works sometimes. Excludes utils/tree_sitter/src/scanner.c because it claims to be C, but really we should probably fix that to be cpp. |
||
 Jon Ross-PerkinsandChandler Carruth
|
7befe2ce9f |
Switch custom error stream output to diagnostic (#4846)
This switches most error printing to use diagnostics instead of direct stream writes, even when not a specific file diagnostic. I'm allowing empty filenames for this use-case. This allows a little more specific testing to validate coverage of output using the diagnostic coverage test. I'm adding a few tests to cover things that weren't previously tested. Separately, this also forces a little more standardization in format... considering how changes like #4568 show effort being spent to _mirror_ diagnostic style, my thought is now to just use diagnostic code where possible. Note this also allows incrementally better testing of the language server; I'm changing the crash fix from #4847 in favor of diagnostic testing. --------- Co-authored-by: Chandler Carruth <chandlerc@gmail.com> |
||
 Jon Ross-PerkinsandGeoff Romer
|
4c4c4a4d2c |
Add RawStringOstream for slightly simpler streaming to strings (#4817)
This adds a RawStringOstream. Versus TestRawOstream, which is
consolidated over to RawStringOstream, it uses a string for storage
instead of a vector, mainly to support move-to-string semantics. Versus
llvm::raw_string_ostream, it owns the string and supports pwrite (which
is needed for driver and its fd_ostream compatibility requirement).
This converts most uses of llvm::raw_string_ostream, leaving behind a
few in InstNamer that explicitly cannot own the string, such as:
```
llvm::raw_string_ostream(name)
<< "_" << tree.tokens().GetColumnNumber(token);
```
I have this as its own library so that it can use CHECK.
Yes this doesn't save much code, but it's code we repeatedly write.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
|
||
|
Jon Ross-Perkins
|
c832d523be |
Update files and clang-tidy config to pass with clang-tidy-20 (#4691)
Disables three new warnings because they lean more towards style conflicts than fixes. I've brought these up on #style. Other than that, mostly fixing basic issues, and things that clang-tidy-20 seems to fire where clang-tiday-16 didn't. One particular curious case is `llvm::StringLiteral::data()` uses, which are flagged as not strictly null-terminated; I'm switching to `const char*` in those spots which matches `llvm::formatv`'s format argument, but feels worse. I'm removing `run_clang_tidy.py` here because I'm observing it give fewer warnings than `bazel build --config=clang-tidy -k //toolchain/...`. The latter matches how we enforce in GitHub actions (and also caches results, and suppresses output for files that have no issues), so I'm dropping the bespoke script. |
||
|
Chandler Carruth
|
4148161e24 |
Refactor value store code to use separate files. (#4477)
This is in anticipation of making the integer value store be customized heavily. I'd like to extract it from the common code when doing that, so first disentangling them here without any intended change in functionality or behavior to enable that. I've tried to update `#include`s to be as minimal as I can and added a few missing includes spotted in the process. I've split the test for value store to include what was easy focused on just the value store templates rather than the unified shared value stores. This might surface some opportunities for adding more tests, but for this PR, just doing the minimal restructuring. |
||
|
Jon Ross-Perkins
|
62c36eceda |
Support printing the diagnostic kind for verification. (#4425)
This is to help identify which diagnostics we're actually using. Note that driver/testdata still has tests which don't pass this flag, and so continue to test the kind-less (default) behavior. |
||
 
|
4845f40dff |
Switch CARBON_CHECK to a format string API (#4285)
This switches `DCHECK` and `FATAL` as well. The goal is to reduce the code size impact of these assertions so that we can keep more of them enabled. Currently, the largest cost I see from `CHECK` is not the actual check or the cold code itself, but actually the failure to inline trivial functions due to the presence of the cold code. This means that our goal isn't to reduce apparent code size in the final binary but the LLVM IR cost assessed for these routines in the inliner, which closely correlates with code size but is a bit different. As discussed in #4283, experimentation shows that a single function call with a minimal number of arguments is the lowest cost model for these. This is easily achieved with a format-string API that internally uses `llvm::formatv`. This PR is essentially the `CHECK` version of #4283. However, the check macros are substantially harder to make work with both format strings and streaming because they also take a condition. Also, unexpectedly, I was very successful at devising a regular expression based automated rewrite from the streaming to the format string form with only low 10s of manual fixes. This includes compacting strings broken up across lines, etc. Given how well that went, I've prepared this PR which just directly switches to the format string API and migrate everything to use it. One nice side-effect is that the format string approach ends up greatly simplifying the implementation here as well. This is ... *shockingly* effective. Parsing speeds up by more than 3% with just this change. And checking speeds up by **8%** with this change alone: ``` BM_CompileAPIFileDenseDecls<Phase::Parse>/256 86.3µs ± 1% 82.9µs ± 1% -3.94% (p=0.000 n=17+19) BM_CompileAPIFileDenseDecls<Phase::Parse>/1024 431µs ± 1% 415µs ± 1% -3.76% (p=0.000 n=18+19) BM_CompileAPIFileDenseDecls<Phase::Parse>/4096 1.77ms ± 1% 1.71ms ± 1% -3.18% (p=0.000 n=18+19) BM_CompileAPIFileDenseDecls<Phase::Parse>/16384 7.44ms ± 1% 7.17ms ± 2% -3.56% (p=0.000 n=18+20) BM_CompileAPIFileDenseDecls<Phase::Parse>/65536 30.7ms ± 1% 29.7ms ± 1% -3.15% (p=0.000 n=18+20) BM_CompileAPIFileDenseDecls<Phase::Parse>/262144 131ms ± 1% 127ms ± 1% -2.81% (p=0.000 n=18+18) BM_CompileAPIFileDenseDecls<Phase::Check>/256 878µs ± 2% 800µs ± 1% -8.91% (p=0.000 n=19+20) BM_CompileAPIFileDenseDecls<Phase::Check>/1024 1.88ms ± 2% 1.72ms ± 1% -8.56% (p=0.000 n=19+20) BM_CompileAPIFileDenseDecls<Phase::Check>/4096 5.78ms ± 2% 5.28ms ± 1% -8.70% (p=0.000 n=20+18) BM_CompileAPIFileDenseDecls<Phase::Check>/16384 21.9ms ± 1% 20.1ms ± 1% -8.02% (p=0.000 n=18+20) BM_CompileAPIFileDenseDecls<Phase::Check>/65536 90.4ms ± 2% 83.1ms ± 1% -8.04% (p=0.000 n=19+20) BM_CompileAPIFileDenseDecls<Phase::Check>/262144 381ms ± 2% 352ms ± 1% -7.79% (p=0.000 n=19+19) ``` --------- Co-authored-by: Richard Smith <richard@metafoo.co.uk> Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com> |
||
|
|
a9c815c9f4 |
Introduce a source generator and end-to-end compile benchmarks (#4124)
The big addition here is a very, very rough and very early skeleton of a source code generator framework. This builds upon the lexers identifier synthesis logic, improving on its framework and wiring it up with the most rudimentary of source file generation. This is just enough to roughly replicate my "big API file" source code benchmarks. The source generation works *very* hard to both vary the structure and content of the source as much as possible while ensuring the same *total* amount of each construct is in use, from bytes in identifiers to line breaks, parameters, etc. This lets us generate randomly structure inputs that should consistently take the exact same amount of total work to compile. The complex identifier synthesis logic from the lexer's benchmark is moved over here and the lexer uses APIs in the source generator for identifiers. The other source synthesis in the lexer's benchmark isn't yet moved over, but should likely be slowly absorbed here as it can be refactored into a more principled and re-usable form. Some bits may stay of course if they're just too lexer-specific. Next, this adds a simple end-to-end compile benchmark for the driver that directly and much more clearly reproduces all the measurements I've done manually up until now. It should also be easy to extend to more patterns over time as we add support to the source generator to produce those patterns. Last but not least, I've added a tiny CLI to the source generator so that you can generate source code manually. This is especially nice for generating demo source code to actually run through the driver or look at in an editor. The CLI can also generate C++ source code which lets us do some minimal comparative benchmarking between Carbon and C++/Clang. There are huge number of TODOs in the source generation framework. This is going to be a large ongoing effort I suspect. There are also a bunch of rough edges I've left to try and get this out for review sooner. I've left TODOs for refactorings that really need to be done here, but hoping these can maybe be follow-ups. If not, please flag and I'll try to layer them on here. Sample compile benchmark output, nicely showing where we are w.r.t. our goal speeds (2x behind on lex and check, 5x on parse) at least on a recent AMD server CPU: ``` ------------------------------------------------------------------------------------------------------ Benchmark Time CPU Iterations Lines ------------------------------------------------------------------------------------------------------ BM_CompileAPIFileDenseDecls<Phase::Lex>/256 29420 ns 29419 ns 22860 6.62847M/s BM_CompileAPIFileDenseDecls<Phase::Lex>/1024 146130 ns 146128 ns 4840 6.69959M/s BM_CompileAPIFileDenseDecls<Phase::Lex>/4096 601584 ns 601577 ns 1020 6.69573M/s BM_CompileAPIFileDenseDecls<Phase::Lex>/16384 2547578 ns 2547313 ns 280 6.404M/s BM_CompileAPIFileDenseDecls<Phase::Lex>/65536 10816591 ns 10816389 ns 80 6.05193M/s BM_CompileAPIFileDenseDecls<Phase::Lex>/262144 52191320 ns 52189828 ns 20 5.02261M/s BM_CompileAPIFileDenseDecls<Phase::Parse>/256 101706 ns 101698 ns 6900 1.91745M/s BM_CompileAPIFileDenseDecls<Phase::Parse>/1024 512161 ns 512162 ns 1380 1.9115M/s BM_CompileAPIFileDenseDecls<Phase::Parse>/4096 2078426 ns 2078430 ns 340 1.938M/s BM_CompileAPIFileDenseDecls<Phase::Parse>/16384 8795786 ns 8795583 ns 100 1.85468M/s BM_CompileAPIFileDenseDecls<Phase::Parse>/65536 35073596 ns 35072973 ns 20 1.86639M/s BM_CompileAPIFileDenseDecls<Phase::Parse>/262144 151100688 ns 151097370 ns 20 1.73483M/s BM_CompileAPIFileDenseDecls<Phase::Check>/256 957059 ns 957049 ns 740 203.751k/s BM_CompileAPIFileDenseDecls<Phase::Check>/1024 1956134 ns 1955985 ns 360 500.515k/s BM_CompileAPIFileDenseDecls<Phase::Check>/4096 5797864 ns 5797417 ns 120 694.792k/s BM_CompileAPIFileDenseDecls<Phase::Check>/16384 21219608 ns 21217584 ns 40 768.843k/s BM_CompileAPIFileDenseDecls<Phase::Check>/65536 96311116 ns 96302334 ns 20 679.734k/s BM_CompileAPIFileDenseDecls<Phase::Check>/262144 371637963 ns 371609964 ns 20 705.387k/s ``` Lest someone think this is *bad*, the fact that we're already within 2x of our rather audacious goals makes me quite happy. =D --------- Co-authored-by: Jon Ross-Perkins <jperkins@google.com> Co-authored-by: Richard Smith <richard@metafoo.co.uk> |
||
|
Jon Ross-Perkins
|
1974e44fd9 |
Rename factory functions from 'Create' to 'Make' (#3706)
Similar to #3705, we actually have a mix of `Make` and `Create` in factory functions too, so this PR is normalizing on `Make`. It's intended to be consistent with the naming choice for Carbon factory functions. Note, MakeSyntheticBlock is the only one I feel a little weird about because llvm's own APIs use Create, and this is essentially wrapping LLVM calls. But the flipside is it also feels like a vague line to draw, when we also differ from LLVM coding style in other ways. |
||
|
Jon Ross-Perkins
|
3d2b4f8ae7 |
Switch gnu::unused to maybe_unused (#3592)
Preferring the C+11 name, [reflecting old discussion](https://discord.com/channels/655572317891461132/655578254970716160/1171977169556230164) |
||
|
Jon Ross-Perkins
|
379d776084 |
Add support for '--config=clang-tidy' (#3559)
This sets things up to use `bazel` to run `clang-tidy` using https://github.com/erenon/bazel_clang_tidy. I'm fixing issues outside of explorer, and disabling clang-tidy for targets in explorer that have legacy issues. I was going to disable clang-tidy for targets in explorer such as interpreter anyways, because they're slow to parse, and just extended that to the currently failing targets. |
||
|
Jon Ross-PerkinsandChandler Carruth
|
cafcd88882 |
Split lexing logic and storage to separate files. (#3365)
Just reorganizing logic a little, trying to mirror the direction we've gone with check, lower, etc. That is, lex.h contains a function `Lex` that is used directly. Note, I'm avoiding making meaningful changes here. It could in theory still affect inlining in benchmarks, but I'm not seeing an impact. Before: ``` ------------------------------------------------------------------------------------------------------ Benchmark Time CPU Iterations UserCounters... ------------------------------------------------------------------------------------------------------ BM_ValidKeywords 2784949 ns 2784867 ns 249 bytes_per_second=214.452M/s tokens_per_second=35.9084M/s BM_ValidKeywordsAsRawIdentifiers 3222597 ns 3222551 ns 210 bytes_per_second=244.513M/s tokens_per_second=31.0313M/s BM_RawIdentifierFocus 5907836 ns 5907518 ns 103 bytes_per_second=264.873M/s tokens_per_second=16.9276M/s BM_ValidIdentifiers<1, 64, false> 6255128 ns 6254297 ns 105 bytes_per_second=235.488M/s tokens_per_second=15.989M/s BM_ValidIdentifiers<1, 1, true> |
||
|
Jon Ross-Perkins
|
6742d0d048 |
Add partial raw identifier support. (#3344)
I'm looking at this due to the conversation on #3341. Although diagnostics aren't where they should be, I thought it may help to start adding raw identifier support (which may also help show how I was thinking about this). Note regarding the TODO on how to form the token, `GetTokenText` returns the `string_id`'s reference value for an `Identifier`. So to make `GetTokenText` work in a way that returns `r#foo` for a raw identifier, I think there are a few options: 1. Add additional data indicating the end of the identifier. 2. Add `RawIdentifier` as a token kind to indicate that it's raw and should be prefixed with `r#` (but also giving later stages one more token kind to handle) 3. Make the `string_id` correspond to `r#foo`, and have later stages add `foo` to the strings table whenever `r#foo` is encountered (with map lookups leading to deduplication). 4. Add `StringId::RawKeyword` special values for each keyword. - This would mean `self` prints as `self`, `r#self` prints as `r#self`, but `r#foo` is not a keyword so prints as `foo`. - This means keywords would need to be listed in a place `StringId` can depend on them, one way or the other (e.g., a `keywords.def` file in `base/` should work). 5. Say that it _is_ an `Identifier`, and if it's a keyword spelling, it must have been a raw identifier. - Same limitation as above: This would mean `self` prints as `self`, `r#self` prints as `r#self`, but `r#foo` is not a keyword so prints as `foo`. I'm hoping to resolve this issue separately though. :) |
||
|
Chandler CarruthandJon Ross-Perkins
|
1b0e2d3a4b |
Cleanups of SIMD code and document no Arm port. (#3325)
I spent (a lot) of time working to see if there was any profitable way to port the SIMD code that scans for identifier length to Arm. There isn't really. =/ While working on these, I made some cleanups to the SIMD code that seemed worth landing, and added some benchmarks. All this PR does is the cleanups, benchmarks, and documents that Arm isn't just waiting to get attention but doesn't really have good options (so far). For posterity, here are the core techniques I tried: 1) Direct 32-byte SIMD scanning using pair-wise add trees to build a 32-bit mask of valid identifier and then `clz` to compute the distance. This is a very good analog to the 16-byte SIMD structure used on x86-64. The pair-wise summing technique is the one used in simdjson for similar purposes. 2) A 16-byte SIMD scanning similar to the x86 version but using `shrn` to produce a 64-bit scalar bitmask with 4 bits per byte, and then scaling the bit-count distance. 3) Various hybrid versions of (1) and (2) with short scalar scans to identify short identifiers before paying the SIMD start-up cost. 4) A much fancier version of (1) that scanned 64-bytes at a time, but cached the resulting 64-bit mask and re-used it until exhausted. Some good background on these techniques on Arm CPUs is in this blog post: https://community.arm.com/arm-community-blogs/b/infrastructure-solutions-blog/posts/porting-x86-vector-bitmask-optimizations-to-arm-neon Sadly, both (1) and (2) were significantly slower than a scalar loop over the bytes. Even (3) was consistently slower. The only approach that came close was (4) and it was very *slightly* slower in typical examples and very *slightly* faster in extremely difficult cases like huge identifiers. Ultimately, the only path I see (suggested by Dougall on a Mastodon discussion of this whole problem space) is to take (4) to the limit of computing an identifier-or-not bitmask *for the entire source file* using a deeply throughput optimized routine (maybe as part of the line scanning). That should be able to manage the high latency you end up with when handling these patterns in SIMD on Arm. The good news is that at least the M1 is *so* fast in the byte-scanning loop that this isn't hurting nearly as much as I feared. --------- Co-authored-by: Jon Ross-Perkins <jperkins@google.com> |
||
|
Jon Ross-PerkinsandRichard Smith
|
d13f76e001 |
Add value store to be shared across compile stages. (#3311)
This updates lexing to use the data. I'll do checking separately, just to split changes. Note the ValueStore structure is also set up such that SemIR::File can use it for other fields. --------- Co-authored-by: Richard Smith <richard@metafoo.co.uk> |
||
|
Chandler Carruth
|
9ce9572522 |
Add a benchmark to stress test grouping symbols. (#3286)
This isn't really representative of anything, but it should help make it obvious when the handling of grouping symbols improves or regresses. Notably, the random source microbenchmark has *no* grouping symbols (in order to let it be random but always lexically valid), and so it's especially useful to have something that checks grouping symbols given their prevalence in realistic source code. |
||
|
5041a14f59 |
Add whitespace- and comment-specific benchmarking. (#3276)
These benchmarks zero-in and stress test horizontal and vertical whitespace as well as comment lexing performance. They set up essentially a worst-case scenario of ramping up whitespace between very sparse tokens to show how the lexer copes with this. The horizontal whitespace benchmark is perhaps less important as frequent runs of 50-characters of horizontal whitespace are relatively rare already, and likely to be exceedingly rare without trailing comments. But its good to include for completeness and it shows reasonably strong performance with the current table-dispatch approach. The blank line and comment line benchmarks are much more important. Lots of code is relatively line-sparse, especially API files that are perhaps the most useful to parse quickly. And many of these are a mixture of sparse with blank lines and sparse with large comment blocks. The benchmarks show that there are some serious limits here, even falling below 100k tokens per second throughput on some of the stress tests here. --------- Co-authored-by: Richard Smith <richard@metafoo.co.uk> Co-authored-by: josh11b <josh11b@users.noreply.github.com> |
||
|
Chandler CarruthandRichard Smith
|
2d735bbc51 |
Enhance the main lexer benchmark. (#3275)
This generalizes the main lexer benchmark's source generation to be a bit more comprehensive, specifically including whitespace and comments. Without these, we're missing a key part of the lexer's performance. This also tidies up a bit of the code and adds a more specific distribution of the different factors in lexing based on analysis of LLVM's source code. The enhancements to the source statistics script that helped collect the data here will be in a separate PR. With this, the output on my AMD cloud instance is: ``` ------------------------------------------------------------------------------------------------------------------------- Benchmark Time CPU Iterations bytes_per_second tokens_per_second ------------------------------------------------------------------------------------------------------------------------- BM_ValidKeywords 2809011 ns 2808912 ns 243 212.616M/s 35.601M/s BM_ValidIdentifiers<1, 64, false> 11461783 ns 11461652 ns 61 128.499M/s 8.72475M/s BM_ValidIdentifiers<1, 1, true> 3397293 ns 3397240 ns 208 84.2158M/s 29.4357M/s BM_ValidIdentifiers<3, 5, true> 14165143 ns 14164962 ns 51 40.3956M/s 7.05967M/s BM_ValidIdentifiers<3, 16, true> 15283128 ns 15282583 ns 47 71.7623M/s 6.5434M/s BM_ValidIdentifiers<12, 64, true> 17417323 ns 17417109 ns 41 219.007M/s 5.74148M/s ------------------------------------------------------------------------------------------------------------------------------------------ Benchmark Time CPU Iterations bytes_per_second lines_per_second tokens_per_second ------------------------------------------------------------------------------------------------------------------------------------------ BM_RandomSource 8132211 ns 8132227 ns 84 137.223M/s 3.9036M/s 12.2968M/s BM_SpeedOfLightStrCpy 29610 ns 29608 ns 24631 36.8065G/s 1072.17M/s 3.37745G/s BM_SpeedOfLightDispatch<1> 2144418 ns 2144421 ns 327 520.387M/s 14.8035M/s 46.6326M/s BM_SpeedOfLightDispatch<2> 1945954 ns 1945827 ns 351 573.498M/s 16.3144M/s 51.392M/s BM_SpeedOfLightDispatch<4> 2519565 ns 2519467 ns 292 442.923M/s 12.5999M/s 39.6909M/s BM_SpeedOfLightDispatch<8> 3011965 ns 3011968 ns 238 370.498M/s 10.5396M/s 33.2009M/s BM_SpeedOfLightDispatch<16> 4379575 ns 4379579 ns 160 254.803M/s 7.24841M/s 22.8332M/s BM_SpeedOfLightDispatch<32> 6678423 ns 6678353 ns 102 167.096M/s 4.75342M/s 14.9738M/s BM_SpeedOfLightDispatch<MaxDispatchTargets> 9373075 ns 9372688 ns 75 119.062M/s 3.38697M/s 10.6693M/s ``` I've compared the profile of the `BM_RandomSource` benchmark with this change and it largely corresponds to what I expect based on profiling hand-crafted Carbon inputs. And as you can see, we're closing in on lexing at least hitting the 10-million-lines-per-second mark. =] --------- Co-authored-by: Richard Smith <richard@metafoo.co.uk> |
||
 Hana Dusíková
|
48d40aa0f0 |
Remove unnecessory constexpr for lambdas in tests. (#3272)
After reading Chandler's code I asked him why he has these. Lambdas will implicitly constexpr anyway. |
||
|
Chandler Carruth
|
c7e6238fa8 |
Introduce two speed-of-light benchmarks. (#3270)
The goal of these kinds of benchmarks is to help calibrate other
benchmarks and expectations. They benchmark the underlying hardware
capabilities that we can't avoid, and help illustrate bounds for what is
possible. The term "speed-of-light benchmark" references the aspect of
measuring how fast thing could possible run.
The first is a simple memory bandwidth measurement in the best case
scenario -- using `strcpy` over the buffer. This still does a minimal
number of writes to memory and examines each byte of input to see if it
is null, but can cheat in every way possible to run at the maximum speed
of hardware. To a certain extent, we never expect to get close to this
speed, but it's a good illustration of how much headroom the hardware
has available.
The second is potentially more interesting. This illustrates how fast a
byte-by-byte dispatch loop can potentially be. It uses the technique
that I'm hoping to use in the lexer itself of guaranteed tail recursion
to achieve this with a very small code footprint. The performance of
this technique, even when running in this extremely minimal setting to
establish bounds, is hugely dependent on the number of distinct dispatch
targets, and so the benchmark includes a healthy range to show the range
of performance that we might expect when running in a byte-by-byte mode.
Note that we should expect the lexer to be *faster* than this
"speed-of-light" whenever it is able to lex in larger granules than
byte-wise. But for complex, dense token sequences that force looking at
every byte, this shows the "worst case" "speed-of-light" in a sense.
On my recent AMD cloud VM instance, I get the following results running
the main lexer benchmark with these changes included:
```
-------------------------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations bytes_per_second tokens_per_second
-------------------------------------------------------------------------------------------------------------------------
BM_ValidKeywords 3169403 ns 3169283 ns 221 188.44M/s 31.5529M/s
BM_ValidIdentifiers<1, 64, false> 12486725 ns 12486445 ns 51 117.953M/s 8.00868M/s
BM_ValidIdentifiers<1, 1, true> 3950455 ns 3950298 ns 178 72.4252M/s 25.3145M/s
BM_ValidIdentifiers<3, 5, true> 15562294 ns 15561178 ns 45 36.7712M/s 6.42625M/s
BM_ValidIdentifiers<3, 16, true> 16118656 ns 16118374 ns 44 68.0412M/s 6.2041M/s
BM_ValidIdentifiers<12, 64, true> 19116271 ns 19116258 ns 35 199.541M/s 5.23115M/s
BM_ValidMix/10/40 7074336 ns 7073795 ns 93 140.744M/s 14.1367M/s
BM_ValidMix/25/30 6790722 ns 6790006 ns 102 131.793M/s 14.7275M/s
BM_ValidMix/50/20 5960514 ns 5960443 ns 118 112.594M/s 16.7773M/s
BM_ValidMix/75/10 4325546 ns 4325556 ns 159 102.559M/s 23.1184M/s
BM_SpeedOfLightStrCpy 24339 ns 24339 ns 29650 35.9049G/s 4.10858G/s
BM_SpeedOfLightDispatch<1>
|
||
|
Chandler Carruth
|
b8802035ed |
Switch benchmarks to match unit test namespacing. (#3268)
Specifically, rather than nesting them in `Carbon::Testing`, nest them in `Carbon::Foo` for whatever component they're benchmarking. All our current benchmarks are lexer benchmarks so its `Carbon::Lex`. This makes even more sense for benchmarks than unittests I think. |
||
|
Jon Ross-PerkinsandRichard Smith
|
bc63e6ae0a |
Switch SourceBuffer to diagnostics. (#3197)
This updates SourceBuffer to diagnostics. Some additional edits to diagnostics were necessary due to issues moving arguments around, which seems to stem from a compile error with clang 14 (fixed in later versions). --------- Co-authored-by: Richard Smith <richard@metafoo.co.uk> |
||
|
Jon Ross-Perkins
|
9ac92ad71b |
Add support for compiling multiple files at once. (#3182)
Rearranges driver logic into CompilationUnits in order to associate artifacts from the various stages of compilation. Note, I'm not totally sure what the right thing to do is for lower/codegen, so I'm just doing a rote change there for now that mirrors prior phases (this is all the code supports anyways, so is probably right for now regardless). SourceBuffer error output is moved local for consistency with other steps, and so that it's less ambiguous whether the error should be expected to already include a filename. |
||
|
Jon Ross-PerkinsandChandler Carruth
|
ec182fb00d |
Rename lexer dir to lex (#3179)
Continuing with #3070. Just a dir and file rename (only prefix change is lexer_file_test). Everything in the lex dir should be marked as a move. Note, I think this closes #3070. There may still be further cleanup later, but the organizational changes suggested there are being completed. --------- Co-authored-by: Chandler Carruth <chandlerc@gmail.com> |