mirror of
https://github.com/carbon-language/carbon-lang.git
synced 2026-09-24 22:02:23 +01:00
fa95892a37d3de00db8928245ee5a00e79d11606
30
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jon Ross-Perkins
|
fa95892a37 |
Add diagnostic coverage, remove possibly-unreachable unary op diagnostic (#4519)
I'm working to make sure remaining diagnostics have coverage, at least the ones I'd previously added a TODO for. Note in particular that I couldn't figure out a repro for UnaryOperatorRequiresWhitespace; if you have one, I can add a test, but otherwise maybe it's actually unreachable due to being diagnosed through infix logic (or, maybe this'll let fuzzing tell me an example). |
||
 Chandler Carruth
|
4148161e24 |
Refactor value store code to use separate files. (#4477)
This is in anticipation of making the integer value store be customized heavily. I'd like to extract it from the common code when doing that, so first disentangling them here without any intended change in functionality or behavior to enable that. I've tried to update `#include`s to be as minimal as I can and added a few missing includes spotted in the process. I've split the test for value store to include what was easy focused on just the value store templates rather than the unified shared value stores. This might surface some opportunities for adding more tests, but for this PR, just doing the minimal restructuring. |
||
|
Chandler Carruth
|
577fda1ca2 |
Speed up type literal lexing and make it more strict. (#4430)
This rejects type literals with more digits than we can lex without APInt's help, and using a custom diagnostic. This is a pretty arbitrary implementation limit, I'm wide open to even more strict rules here. Despite no special casing and a very simplistic approach, by not using APInt this completely eliminates the lexing overhead for `i32` in the generated compilation benchmark where that specific type literal is very common. We see a 10% improvement in lexing there: ``` BM_CompileAPIFileDenseDecls<Phase::Lex>/256 39.0µs ± 4% 34.8µs ± 2% -10.86% (p=0.000 n=19+20) BM_CompileAPIFileDenseDecls<Phase::Lex>/1024 180µs ± 1% 158µs ± 2% -12.22% (p=0.000 n=18+20) BM_CompileAPIFileDenseDecls<Phase::Lex>/4096 731µs ± 2% 641µs ± 1% -12.31% (p=0.000 n=18+20) BM_CompileAPIFileDenseDecls<Phase::Lex>/16384 3.20ms ± 2% 2.86ms ± 2% -10.47% (p=0.000 n=18+19) BM_CompileAPIFileDenseDecls<Phase::Lex>/65536 13.8ms ± 1% 12.4ms ± 2% -9.78% (p=0.000 n=18+19) BM_CompileAPIFileDenseDecls<Phase::Lex>/262144 64.0ms ± 2% 58.4ms ± 2% -8.70% (p=0.000 n=19+18) ``` This starts to fix a TODO in the diagnostic for these by giving a reasonably good diagnostic about a very large type literal. However, in practice it regresses the diagnostics because error tokens produce noisy extraneous diagnostics from parse and check currently. Leaving the TODO there, and I have a follow-up PR to start improving the extraneous diagnostics. |
||
 Richard SmithandJon Ross-Perkins
|
e68e54dae4 |
Issue a diagnostic if we try to parse a source file that is too large. (#4429)
Previously in an optimized build we'd produce bogus tokens, such as tokens with incorrect IdentifierIds, and in a debug build we would try to CHECK-fail -- but actually wouldn't, because we're incorrectly checking for `2 << bits` instead of `1 << bits`. I hit this while I was trying to do some profiling and was seeing some very strange diagnostics. The diagnostic is pointed at the first token that is beyond the limit to help people determine where to split their files. --------- Co-authored-by: Jon Ross-Perkins <jperkins@google.com> |
||
|
Jon Ross-PerkinsandChandler Carruth
|
0db96ebc52 |
Stitch together adjacent comments using the indent. (#4397)
This is improving the comment production to produce fewer distinct comments. At present, comment processing uses strict prefix matching. It either expects `// ` (with a space) for valid comments, or just `//` (without a space) for invalid comments that lacked the space. As a consequence, the following would be three comments: ``` // Comment 1 // // // Comment 4 ``` This is because a 3-character prefix is used for valid comments. The prefix switches between lines 1 and 2, and again between lines 3 and 4, each resulting in a separate comment. For contrast, this is one comment because only a 2-character prefix is used: ``` //Comment 1 // // //Comment 4 ``` That's because all lines lack a suffix space. Additionally, with SIMD 16-byte boundaries, further splits can occur if processing needs to transition to non-SIMD. Here, I'm trying to just address all of this by: 1. Stitching together adjacent comments. Since a lexed comment starts at the `//` excluding the indent, the delta from the prior comment must be precisely the indent. 2. Adding support for switching from SIMD to non-SIMD on file boundaries. I considered trying to have a separate `//\n` prefix for SIMD processing of `// `, but I wasn't sure about the tradeoff of doing both at the same time (in particular, it'd require constructing a string for the different prefix), thus this stitch approach. This does mean multiple passes will be required for a typical long comment structure using blank comment lines to separate paragraphs (for performance reasons, I will recommend engineers not write comm... nevermind). --------- Co-authored-by: Chandler Carruth <chandlerc@gmail.com> |
||
|
Jon Ross-Perkins
|
0f350255ce |
Refactor compile-related tests to share construction. (#4396)
Note in particular that this fixes an issue where SharedValueStore had been shared across files, when they should be per-file. This is only visible when doing multiple compilations in a single test, which was rare before. This also moves these tests into the Testing namespace. My memory of the various namespacing changes is that we'd generally agreed to have tests in Testing so that we'd see SemIR:: and similar, same as we would in a lot of the implementation. |
||
|
c43fa3a8a5 |
Bit-pack the lexer's token info (#4270)
This makes each token info consist of 8 bytes of data: - 1 byte of the kind - 1 bit for whitespace tracking - 23 bits of payload - 32 bits for byte offset in the file This builds directly on representing the location of the token as a single 32-bit offset, now compressing the rest of the data into a single 32-bit bitfield. This adds some implementation limits: we can no longer lex more than 2^23 tokens in a single source file. Nor can we have more than 2^23 string literals, integer literals, real literals, or identifiers. Only the first of these is even close to an issue, and even then seems unlikely to ever be a problem in practice. The memory efficiency here is great and the motivating goal. But to make this work well, we also need to streamline how we create the tokens. Otherwise, all the bit fiddling can end up erasing our gains. This PR adds a number of APIs to manage creating and accessing the now significantly more complex storage of token infos to try and help with this. One big change required to simplify the writes here is to switch from computing whether a token has trailing space after-the-fact to pre-computing whether a token will have leading space. That lets us have the leading space information available immediately when forming the token, and avoids doing a single bit flip afterward. Another change that helps with this representation is to minimize the updating of groups after-the-fact. The code now tries to set the opening index directly when creating the closing token and only updates the opening group afterward. Because of the bit packing, this is a reduction of 0.5% of dynamic instructions in the compile benchmark, and has dramatic improvements for the grouping symbol focused benchmarks. All combined, this is a significant improvement on the lexer-focused benchmarks despite the added complexity, and a significant win on our compile time benchmarks due to both the lexer improvements and downstream memory density improvements: 5-12% reduction in lex time, growing larger as files get larger. About a 4.5% reduction in parse time, and even a 1-2% reduction in total check time. =D --------- Co-authored-by: Jon Ross-Perkins <jperkins@google.com> Co-authored-by: Geoff Romer <gromer@google.com> |
||
|
Chandler CarruthandRichard Smith
|
565fc5cebb |
Initial support for CR+LF (DOS / Windows) line endings. (#4056)
It turns out we can make these work with very minimal complexity because the LF is still in the right place either way. This also lets us easily support mixtures of LF and CR+LF line endings gracefully. We create the line structures around the LF bytes and then have the byte-dispatch loop notice a CR followed by an LF and skip to the LF behavior. Rather than add the remaining complexity around supporting bare CR and LF+CR sequences (both of which are quite rare now), this just adds diagnostics when we encounter a CR byte that won't fall out of our CR+LF handling. This is a better experience for users than the alternative. We still have a TODO to handle the full complexity of vertical whitespace, but I've updated it to reflect that the common case should be handled already. This isn't complete though: we need to add support in string literal lexing, and we need to teach the diagnostic rendering to handle the error messages above better. But those will be future PRs, this is enough to unblock folks who happen to edit a Carbon source file with notepad on Windows which seems important. --------- Co-authored-by: Richard Smith <richard@metafoo.co.uk> |
||
|
Jon Ross-Perkins
|
97d6446bab |
Clean up struct init for the TokenizedBuffer test. (#4015)
Mostly structs, but note one fix for StringRef init. This is rooted in #4009 |
||
|
Jon Ross-Perkins
|
ec300ef3ad |
Restructure lex keyword tests to reduce duplication. (#3941)
Rather than having a file listing all keywords, do it programmatically. Conversely, move the limited keyword tests out to a file test. |
||
|
bb117aea3a |
Add support for iN and uN for all suitable N. (#3868)
`i32` is retained as a special case for now, for bootstrapping purposes, and maps to `BuiltinIntType`, which is distinct from `Core.Int(32)`. This will be removed later once we support `Core.BigInt`. For now this provides both the `iN` types and also the builtins to support `Core.Int(N)`. The intent is that we'll change the `iN` support to rewrite to calls here when we do that for the other type literals and type keywords. No conversions between integer types are supported yet, and all literals are of type `i32`, so we can't actually form values of any of these new types. --------- Co-authored-by: Jon Ross-Perkins <jperkins@google.com> Co-authored-by: Carbon Infra Bot <carbon-external-infra@google.com> |
||
|
Jon Ross-Perkins
|
93e40289dd |
Collapse diagnostic errors and notes into a single vector. (#3805)
On #3792 it was requested to put context-related diagnostic messages prior to the diagnostic error. In order to support that, remove the distinction that an error needs to come first on DiagnosticMessage. |
||
|
Jon Ross-Perkins
|
1974e44fd9 |
Rename factory functions from 'Create' to 'Make' (#3706)
Similar to #3705, we actually have a mix of `Make` and `Create` in factory functions too, so this PR is normalizing on `Make`. It's intended to be consistent with the naming choice for Carbon factory functions. Note, MakeSyntheticBlock is the only one I feel a little weird about because llvm's own APIs use Create, and this is essentially wrapping LLVM calls. But the flipside is it also feels like a vague line to draw, when we also differ from LLVM coding style in other ways. |
||
|
Richard Smith
|
0a06fceb5f |
Improve diagnosis of mismatched brackets. (#3282)
Move handling of mismatched brackets out of the main lexing loop into a separate pass that is only run if there are mismatched brackets This is done in preparation for using both lookahead and lookbehind to work out how to match brackets, and to get this code far away from the hot lexing loop. Fix bracket insertion location to be immediately after the token that we're inserting the bracket after, rather than potentially at the end of a comment. When there are open brackets at the end of the file, say that there are open brackets, not that there's a closing bracket without a matching opening bracket. |
||
|
Jon Ross-Perkins
|
0db63ff17a |
Abbreviate Integer and FloatingPoint (#3435)
I was suggesting this because `FloatingPoint` is pretty long. `int` and `float` should be familiar abbreviations. `unsigned` should be familiar to developers too, but `UnsignedInt` still feels usefully clearer for the additional chars. |
||
|
Richard Smith
|
eae630a3db |
Rename Lex::{Token,Line} -> Lex::{Token,Line}Index. (#3433)
As discussed [on discord](https://discord.com/channels/655572317891461132/655578254970716160/1178878128714678282) and today's toolchain discussion. |
||
|
Jon Ross-Perkins
|
3f208e27f9 |
Align on FileStart/FileEnd for naming. (#3428)
The lexer has been using EndOfFile form (stemming from EOF), parser went to FileEnd form. This consolidates on FileEnd form. |
||
|
Jon Ross-PerkinsandChandler Carruth
|
cafcd88882 |
Split lexing logic and storage to separate files. (#3365)
Just reorganizing logic a little, trying to mirror the direction we've gone with check, lower, etc. That is, lex.h contains a function `Lex` that is used directly. Note, I'm avoiding making meaningful changes here. It could in theory still affect inlining in benchmarks, but I'm not seeing an impact. Before: ``` ------------------------------------------------------------------------------------------------------ Benchmark Time CPU Iterations UserCounters... ------------------------------------------------------------------------------------------------------ BM_ValidKeywords 2784949 ns 2784867 ns 249 bytes_per_second=214.452M/s tokens_per_second=35.9084M/s BM_ValidKeywordsAsRawIdentifiers 3222597 ns 3222551 ns 210 bytes_per_second=244.513M/s tokens_per_second=31.0313M/s BM_RawIdentifierFocus 5907836 ns 5907518 ns 103 bytes_per_second=264.873M/s tokens_per_second=16.9276M/s BM_ValidIdentifiers<1, 64, false> 6255128 ns 6254297 ns 105 bytes_per_second=235.488M/s tokens_per_second=15.989M/s BM_ValidIdentifiers<1, 1, true> |
||
|
Jon Ross-Perkins
|
3401eed8d8 |
Split IdentifierId and StringLiteralId from StringId (#3352)
Following up on discussion yesterday regarding this split. Note, I'm expecting #3341 to do IdentifierId -> NameId in SemIR. It might be worth adding NameId creation directly to StringStore if you're content with this setup though. |
||
|
Jon Ross-Perkins
|
3af7eb2672 |
Refactor YAML handling to use the llvm::yaml API. (#3337)
Provides an adapter for the llvm::yaml API because it otherwise needs a bunch of const/non-const definitions, and the traits are difficult to diagnose issues with. The current approach is pretty simple to use, even if it's not super efficient (which, yaml output is more of a debugging thing so I'm not really expecting it to be an issue). Changes the format of yaml output to provide more index information, just as reminders when seeing something like `node+0`. Note this would create more churn in deltas if we were reliant on the output yaml in tests, but we aren't so it should be okay. |
||
|
Jon Ross-Perkins
|
1b55ad86dd |
Extend SharedValueStores to SemIR (#3313)
Building on #3311, change SemIR to use the SharedValueStore. Since this removes hermeticity, raw output no longer prints ints, reals, and strings. TokenizedBuffer accessors are modified to return IDs because values are often passed through in semantics without needing to read them. I would've put SharedValueStores on Context, except for the GetArrayBoundValue convenience method. I felt awkward removing that, so it's on File, at least for now. That's then used by the formatter and Lower too. The flipside of this is that TokenizedBuffer has a SharedValueStores only for printing, so maybe that's similar enough to what File is doing. This doesn't start shifting other SemIR members to ValueStore, but that seems like a next step. |
||
|
Jon Ross-PerkinsandRichard Smith
|
d13f76e001 |
Add value store to be shared across compile stages. (#3311)
This updates lexing to use the data. I'll do checking separately, just to split changes. Note the ValueStore structure is also set up such that SemIR::File can use it for other fields. --------- Co-authored-by: Richard Smith <richard@metafoo.co.uk> |
||
|
Chandler Carruth
|
03c3b86758 |
Switch lexer to fully table-driven design. (#3273)
This uses the musttail dispatched table approach to drive the entire lexing. The result is that there is no main lexer loop at all in a traditional sense, now everything is driven through tail recursive dispatch on the next byte of the source text. This should be easy to extend still -- the design pattern is to add lexer methods for handling specific cases, and then add a dispatch function to dispatch to them from the table. For example, we can add a method that handles decoding UTF-8 outside of the ASCII subset and set the table entries used by non-ASCII initial bytes to dispatch to it. The performance is already surprisingly good, benchmarks show a modest improvement across the board. That's despite there still being some *serious* performance issues that I'll fix in a separate patch. There are also opportunities to leverage this structure more heavily as needed by putting more specialized dispatch targets in for specific bytes. A follow-up PR will re-organize the functions here, as almost all of the methods on the `Lexer` should become private, but I wanted to keep that a separate change since it will probably render the diff even more hard to read than it already is. |
||
|
Chandler Carruth
|
a46ca6bf7a |
Add a start-of-file token and parse node. (#3263)
This removes a (very) hot branch in the lexer where we need to special case when a token is the first token and can't look at its previous token. It also seems like a generally nice change to the structure of both the token buffer and parse tree as there are now bracketing elements for both ends and we should be able to avoid similar branching in the future. Mostly mechanical updates to the lexer and parser code to handle this, but also needed to special case the location information in the autoupdate code. And then the usual large body of auto-updated tests. No benchmark data for this change alone as in isolation and in the current lexer structure it doesn't make a big difference. But this branch was particularly difficult to handle when trying to update the whitespace skipping code to be faster, and so I think it is worth systematically avoiding the special case here. |
||
|
Geoff Romer
|
e3d3122f1d |
Move tests to the namespace of the code under test (#3244)
Rationale: this convention avoids forcing closely-related code to be far apart in the namespace hierarchy, and vice versa. By the same token, it makes the namespace hierarchy more consistent with the directory hierarchy. |
||
|
Jon Ross-Perkins
|
32a1be3690 |
Detect invalid yaml. (#3239)
Previous code could fail silently, only caught with yaml output mismatches. This uses ErrorOr to be explicit about the error detection; recovery isn't supported because we should only print valid yaml (in tests, at least). Also moves the test helpers from //toolchain/base to //toolchain/testing. Switches tree_test to use the matchers so that, on mismatch, gtest prints a path to the mismatch (the straight value means it was just printing "not equal"). Addresses https://github.com/carbon-language/carbon-lang/pull/3217#discussion_r1323586836 |
||
|
Jon Ross-Perkins
|
2ecab78297 |
Support multi-file lex printing and testing. (#3214)
Lex now prints its yaml as: ``` - filename: name tokens: [ ... ] ``` New support in file_test allows the `filename` marker at the top to define the default file number for later lines, meaning multi-file output from lexing is now associated with the appropriate file. Similar support will probably also apply to lowering, semir, and other places that print a filename once for the full dump. This hammers a bit at how line number replacements work in file_test, allowing stacking them so that lex errors and stdout can both be line-associated properly. I've tried to make the autoupdate more frequently work in one pass, now also taking into account the file index when doing line replacements. There are still some issues with EndOfFile that it may be good to discuss: because CHECK lines are appended to the end of the file now, and the EndOfFile token points at the last line including comments, new lex tests now take two runs to autoupdate (because without CHECK lines, the EndOfFile points at a content line, which content is then inserted after). Note that removing CHECK lines from the test is not a solution: autoupdate also started inserting blank lines, which breaks this for a similar reason. One solution here might be to not have EndOfFile associate with a line or column, which has been a bit of an issue regardless. Also fixes a small issue with toolchain's autoupdate script. |
||
|
Jon Ross-PerkinsandRichard Smith
|
bc63e6ae0a |
Switch SourceBuffer to diagnostics. (#3197)
This updates SourceBuffer to diagnostics. Some additional edits to diagnostics were necessary due to issues moving arguments around, which seems to stem from a compile error with clang 14 (fixed in later versions). --------- Co-authored-by: Richard Smith <richard@metafoo.co.uk> |
||
|
Jon Ross-Perkins
|
9ac92ad71b |
Add support for compiling multiple files at once. (#3182)
Rearranges driver logic into CompilationUnits in order to associate artifacts from the various stages of compilation. Note, I'm not totally sure what the right thing to do is for lower/codegen, so I'm just doing a rote change there for now that mirrors prior phases (this is all the code supports anyways, so is probably right for now regardless). SourceBuffer error output is moved local for consistency with other steps, and so that it's less ambiguous whether the error should be expected to already include a filename. |
||
|
Jon Ross-PerkinsandChandler Carruth
|
ec182fb00d |
Rename lexer dir to lex (#3179)
Continuing with #3070. Just a dir and file rename (only prefix change is lexer_file_test). Everything in the lex dir should be marked as a move. Note, I think this closes #3070. There may still be further cleanup later, but the organizational changes suggested there are being completed. --------- Co-authored-by: Chandler Carruth <chandlerc@gmail.com> |