mirror of
https://github.com/carbon-language/carbon-lang.git
synced 2026-09-28 11:30:11 +01:00
81dfb2b29b115df27f22cd7e8564d40c360d4e22
34
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Richard Smith
|
28602a87c2 |
Fix handling of repeated tuple indexing. (#4733)
Per [the design](https://docs.carbon-lang.dev/docs/design/lexical_conventions/), `x.1.2` should lex as `(x.1).2`, not as `x.(1.2)`. |
||
 Jon Ross-Perkins
|
c832d523be |
Update files and clang-tidy config to pass with clang-tidy-20 (#4691)
Disables three new warnings because they lean more towards style conflicts than fixes. I've brought these up on #style. Other than that, mostly fixing basic issues, and things that clang-tidy-20 seems to fire where clang-tiday-16 didn't. One particular curious case is `llvm::StringLiteral::data()` uses, which are flagged as not strictly null-terminated; I'm switching to `const char*` in those spots which matches `llvm::formatv`'s format argument, but feels worse. I'm removing `run_clang_tidy.py` here because I'm observing it give fewer warnings than `bazel build --config=clang-tidy -k //toolchain/...`. The latter matches how we enforce in GitHub actions (and also caches results, and suppresses output for files that have no issues), so I'm dropping the bespoke script. |
||
 Jon Ross-PerkinsandChandler Carruth
|
08f24551ec |
Add bit packing to NodeImpl (#4651)
Just a small packing optimization. We currently have 222 `NodeKinds`, so this reduces us to just 30ish more we can add without needing to pack more. However, if we did, there would be a couple options for bringing the count down by reusing `NodeKinds` and disambiguating based on the token kind (the 29 infix operators as an example). Or we could just undo this. I'm expecting this to yield a small improvement. I'll see if I can get better numbers since my machine's not really reliable, but here are some basic values. Also suggesting to draw the use of `::RawEnumType` for `TokenKind`, since bit packing appears to work without it. Hoping the `static_assert` is easier for people to understand the size of the field. With the change: ``` ---------------------------------------------------------------------------------------------------------------------------- Benchmark Time CPU Iterations Bytes Lines Tokens ---------------------------------------------------------------------------------------------------------------------------- BM_CompileAPIFileDenseDecls<Phase::Parse>/256 50399 ns 50359 ns 14336 104.588M/s 3.87217M/s 21.8629M/s BM_CompileAPIFileDenseDecls<Phase::Parse>/1024 237823 ns 237629 ns 3072 136.721M/s 4.11986M/s 24.2058M/s BM_CompileAPIFileDenseDecls<Phase::Parse>/4096 997645 ns 996771 ns 768 142.343M/s 4.04105M/s 23.9363M/s BM_CompileAPIFileDenseDecls<Phase::Parse>/16384 4020308 ns 4018319 ns 192 152.041M/s 4.05966M/s 24.0874M/s BM_CompileAPIFileDenseDecls<Phase::Parse>/65536 16691390 ns 16683058 ns 48 151.317M/s 3.92374M/s 23.2936M/s BM_CompileAPIFileDenseDecls<Phase::Parse>/262144 75265735 ns 75233476 ns 8 135.842M/s 3.48421M/s 20.6862M/s ``` Without the change: ``` ---------------------------------------------------------------------------------------------------------------------------- Benchmark Time CPU Iterations Bytes Lines Tokens ---------------------------------------------------------------------------------------------------------------------------- BM_CompileAPIFileDenseDecls<Phase::Parse>/256 51515 ns 51480 ns 13312 102.312M/s 3.78789M/s 21.387M/s BM_CompileAPIFileDenseDecls<Phase::Parse>/1024 241040 ns 240900 ns 3072 134.865M/s 4.06392M/s 23.8771M/s BM_CompileAPIFileDenseDecls<Phase::Parse>/4096 985593 ns 984657 ns 768 144.094M/s 4.09077M/s 24.2308M/s BM_CompileAPIFileDenseDecls<Phase::Parse>/16384 4109327 ns 4105496 ns 192 148.813M/s 3.97345M/s 23.576M/s BM_CompileAPIFileDenseDecls<Phase::Parse>/65536 17459655 ns 17446006 ns 48 144.7M/s 3.75215M/s 22.275M/s BM_CompileAPIFileDenseDecls<Phase::Parse>/262144 80802815 ns 80737489 ns 8 126.581M/s 3.24668M/s 19.276M/s ``` --------- Co-authored-by: Chandler Carruth <chandlerc@gmail.com> |
||
|
Jon Ross-Perkins
|
61c0a8b676 |
Make more use of llvm STLExtras (#4668)
This is essentially the result of looking at `.begin()` uses. We also frequently do `std::shuffle`, but unfortunately STLExtras doesn't provide a wrapper for that. |
||
 
|
3ba4997855 |
Canonicalize away bit width and embed small integers into IntIds (#4487)
The first change here is to canonicalize away bit width when tracking integers in our shared value store. This lets us have a more definitive model of "what is the mathematical value". It also frees us to use more efficient bit widths when available, such as bits inside the ID itself. For canonicalizing, we try to minimize the width adjustments and maximize the use of the SSO in APInt, and so we never shrink belowe 64-bits and grow in multiples of the word bit width in the implementation. We also canonicalize to the signed 2s compliment representation so we can represent negative numbers in an intuitive way. The canonicalizing requires getting the bit width out of the type and adjusting to it within the toolchain when doing any kind of math, and this PR updates various places to do that, as well as adding some convenience APIs to assist. Then we take advantage of the canonical form and embed small integers into the ID itself rather than allocating storage for them and referencing them with an index. This is especially helpful for the pervasive small integers such as the sizes of types, arrays, etc. Those no longer require indirection at all. Various short-cut APIs to take advantage of this have also been added. This PR improves lexing by about 5% when there are lots of `i32` types. --------- Co-authored-by: Dana Jansens <danakj@orodu.net> Co-authored-by: Carbon Infra Bot <carbon-external-infra@google.com> Co-authored-by: Jon Ross-Perkins <jperkins@google.com> |
||
|
Chandler Carruth
|
4148161e24 |
Refactor value store code to use separate files. (#4477)
This is in anticipation of making the integer value store be customized heavily. I'd like to extract it from the common code when doing that, so first disentangling them here without any intended change in functionality or behavior to enable that. I've tried to update `#include`s to be as minimal as I can and added a few missing includes spotted in the process. I've split the test for value store to include what was easy focused on just the value store templates rather than the unified shared value stores. This might surface some opportunities for adding more tests, but for this PR, just doing the minimal restructuring. |
||
|
Chandler Carruth
|
577fda1ca2 |
Speed up type literal lexing and make it more strict. (#4430)
This rejects type literals with more digits than we can lex without APInt's help, and using a custom diagnostic. This is a pretty arbitrary implementation limit, I'm wide open to even more strict rules here. Despite no special casing and a very simplistic approach, by not using APInt this completely eliminates the lexing overhead for `i32` in the generated compilation benchmark where that specific type literal is very common. We see a 10% improvement in lexing there: ``` BM_CompileAPIFileDenseDecls<Phase::Lex>/256 39.0µs ± 4% 34.8µs ± 2% -10.86% (p=0.000 n=19+20) BM_CompileAPIFileDenseDecls<Phase::Lex>/1024 180µs ± 1% 158µs ± 2% -12.22% (p=0.000 n=18+20) BM_CompileAPIFileDenseDecls<Phase::Lex>/4096 731µs ± 2% 641µs ± 1% -12.31% (p=0.000 n=18+20) BM_CompileAPIFileDenseDecls<Phase::Lex>/16384 3.20ms ± 2% 2.86ms ± 2% -10.47% (p=0.000 n=18+19) BM_CompileAPIFileDenseDecls<Phase::Lex>/65536 13.8ms ± 1% 12.4ms ± 2% -9.78% (p=0.000 n=18+19) BM_CompileAPIFileDenseDecls<Phase::Lex>/262144 64.0ms ± 2% 58.4ms ± 2% -8.70% (p=0.000 n=19+18) ``` This starts to fix a TODO in the diagnostic for these by giving a reasonably good diagnostic about a very large type literal. However, in practice it regresses the diagnostics because error tokens produce noisy extraneous diagnostics from parse and check currently. Leaving the TODO there, and I have a follow-up PR to start improving the extraneous diagnostics. |
||
|
Richard SmithandJon Ross-Perkins
|
e68e54dae4 |
Issue a diagnostic if we try to parse a source file that is too large. (#4429)
Previously in an optimized build we'd produce bogus tokens, such as tokens with incorrect IdentifierIds, and in a debug build we would try to CHECK-fail -- but actually wouldn't, because we're incorrectly checking for `2 << bits` instead of `1 << bits`. I hit this while I was trying to do some profiling and was seeing some very strange diagnostics. The diagnostic is pointed at the first token that is beyond the limit to help people determine where to split their files. --------- Co-authored-by: Jon Ross-Perkins <jperkins@google.com> |
||
|
Jon Ross-PerkinsandChandler Carruth
|
0db96ebc52 |
Stitch together adjacent comments using the indent. (#4397)
This is improving the comment production to produce fewer distinct comments. At present, comment processing uses strict prefix matching. It either expects `// ` (with a space) for valid comments, or just `//` (without a space) for invalid comments that lacked the space. As a consequence, the following would be three comments: ``` // Comment 1 // // // Comment 4 ``` This is because a 3-character prefix is used for valid comments. The prefix switches between lines 1 and 2, and again between lines 3 and 4, each resulting in a separate comment. For contrast, this is one comment because only a 2-character prefix is used: ``` //Comment 1 // // //Comment 4 ``` That's because all lines lack a suffix space. Additionally, with SIMD 16-byte boundaries, further splits can occur if processing needs to transition to non-SIMD. Here, I'm trying to just address all of this by: 1. Stitching together adjacent comments. Since a lexed comment starts at the `//` excluding the indent, the delta from the prior comment must be precisely the indent. 2. Adding support for switching from SIMD to non-SIMD on file boundaries. I considered trying to have a separate `//\n` prefix for SIMD processing of `// `, but I wasn't sure about the tradeoff of doing both at the same time (in particular, it'd require constructing a string for the different prefix), thus this stitch approach. This does mean multiple passes will be required for a typical long comment structure using blank comment lines to separate paragraphs (for performance reasons, I will recommend engineers not write comm... nevermind). --------- Co-authored-by: Chandler Carruth <chandlerc@gmail.com> |
||
|
Jon Ross-Perkins
|
1338f9e0ad |
Add tracking of lexed comments, with skeletal formatting. (#4385)
In order to format comments, it's helpful if they're tracked. This tracks them separately from tokens in order to avoid interfering with parse; it'd be inconvenient if comment tokens could show up in arbitrary locations, albeit possible to support. This additionally extracts out the TokenIterator support into a template in order to generally have it available for IndexBase types. I'm only adding it for CommentInfo, not sure if we'll want it elsewhere, but this structure still felt like a good fit. |
||
|
Jon Ross-PerkinsandRichard Smith
|
e7aebbe581 |
Update basic diagnostic capitalization/punctuation (#4328)
This is a primarily automated change:
- Search & replace for capitalization
-
`(CARBON_DIAGNOSTIC\((?:\n\s+)?\w+,(?:\n\s+)?\s\w+,(?:\n\s+)?\s")([A-Z])`
- `$1\L$2`
- Search & replace for period
-
`(CARBON_DIAGNOSTIC\((?:\n\s+)?\w+,(?:\n\s+)?\s\w+,(?:\n\s+)?\s"(?:[^)]|\n)+)\.("[,)])`
- `$1$2`
- Limited search & replace for `ERROR: ` -> `error: ` in streamed things
- Leaving a TODO for command_line because there's more cleanup that can
be done there
- Modify diagnostic_consumer.cpp
- ERROR -> error
- WARNING -> warning
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
|
||
|
4845f40dff |
Switch CARBON_CHECK to a format string API (#4285)
This switches `DCHECK` and `FATAL` as well. The goal is to reduce the code size impact of these assertions so that we can keep more of them enabled. Currently, the largest cost I see from `CHECK` is not the actual check or the cold code itself, but actually the failure to inline trivial functions due to the presence of the cold code. This means that our goal isn't to reduce apparent code size in the final binary but the LLVM IR cost assessed for these routines in the inliner, which closely correlates with code size but is a bit different. As discussed in #4283, experimentation shows that a single function call with a minimal number of arguments is the lowest cost model for these. This is easily achieved with a format-string API that internally uses `llvm::formatv`. This PR is essentially the `CHECK` version of #4283. However, the check macros are substantially harder to make work with both format strings and streaming because they also take a condition. Also, unexpectedly, I was very successful at devising a regular expression based automated rewrite from the streaming to the format string form with only low 10s of manual fixes. This includes compacting strings broken up across lines, etc. Given how well that went, I've prepared this PR which just directly switches to the format string API and migrate everything to use it. One nice side-effect is that the format string approach ends up greatly simplifying the implementation here as well. This is ... *shockingly* effective. Parsing speeds up by more than 3% with just this change. And checking speeds up by **8%** with this change alone: ``` BM_CompileAPIFileDenseDecls<Phase::Parse>/256 86.3µs ± 1% 82.9µs ± 1% -3.94% (p=0.000 n=17+19) BM_CompileAPIFileDenseDecls<Phase::Parse>/1024 431µs ± 1% 415µs ± 1% -3.76% (p=0.000 n=18+19) BM_CompileAPIFileDenseDecls<Phase::Parse>/4096 1.77ms ± 1% 1.71ms ± 1% -3.18% (p=0.000 n=18+19) BM_CompileAPIFileDenseDecls<Phase::Parse>/16384 7.44ms ± 1% 7.17ms ± 2% -3.56% (p=0.000 n=18+20) BM_CompileAPIFileDenseDecls<Phase::Parse>/65536 30.7ms ± 1% 29.7ms ± 1% -3.15% (p=0.000 n=18+20) BM_CompileAPIFileDenseDecls<Phase::Parse>/262144 131ms ± 1% 127ms ± 1% -2.81% (p=0.000 n=18+18) BM_CompileAPIFileDenseDecls<Phase::Check>/256 878µs ± 2% 800µs ± 1% -8.91% (p=0.000 n=19+20) BM_CompileAPIFileDenseDecls<Phase::Check>/1024 1.88ms ± 2% 1.72ms ± 1% -8.56% (p=0.000 n=19+20) BM_CompileAPIFileDenseDecls<Phase::Check>/4096 5.78ms ± 2% 5.28ms ± 1% -8.70% (p=0.000 n=20+18) BM_CompileAPIFileDenseDecls<Phase::Check>/16384 21.9ms ± 1% 20.1ms ± 1% -8.02% (p=0.000 n=18+20) BM_CompileAPIFileDenseDecls<Phase::Check>/65536 90.4ms ± 2% 83.1ms ± 1% -8.04% (p=0.000 n=19+20) BM_CompileAPIFileDenseDecls<Phase::Check>/262144 381ms ± 2% 352ms ± 1% -7.79% (p=0.000 n=19+19) ``` --------- Co-authored-by: Richard Smith <richard@metafoo.co.uk> Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com> |
||
|
c43fa3a8a5 |
Bit-pack the lexer's token info (#4270)
This makes each token info consist of 8 bytes of data: - 1 byte of the kind - 1 bit for whitespace tracking - 23 bits of payload - 32 bits for byte offset in the file This builds directly on representing the location of the token as a single 32-bit offset, now compressing the rest of the data into a single 32-bit bitfield. This adds some implementation limits: we can no longer lex more than 2^23 tokens in a single source file. Nor can we have more than 2^23 string literals, integer literals, real literals, or identifiers. Only the first of these is even close to an issue, and even then seems unlikely to ever be a problem in practice. The memory efficiency here is great and the motivating goal. But to make this work well, we also need to streamline how we create the tokens. Otherwise, all the bit fiddling can end up erasing our gains. This PR adds a number of APIs to manage creating and accessing the now significantly more complex storage of token infos to try and help with this. One big change required to simplify the writes here is to switch from computing whether a token has trailing space after-the-fact to pre-computing whether a token will have leading space. That lets us have the leading space information available immediately when forming the token, and avoids doing a single bit flip afterward. Another change that helps with this representation is to minimize the updating of groups after-the-fact. The code now tries to set the opening index directly when creating the closing token and only updates the opening group afterward. Because of the bit packing, this is a reduction of 0.5% of dynamic instructions in the compile benchmark, and has dramatic improvements for the grouping symbol focused benchmarks. All combined, this is a significant improvement on the lexer-focused benchmarks despite the added complexity, and a significant win on our compile time benchmarks due to both the lexer improvements and downstream memory density improvements: 5-12% reduction in lex time, growing larger as files get larger. About a 4.5% reduction in parse time, and even a 1-2% reduction in total check time. =D --------- Co-authored-by: Jon Ross-Perkins <jperkins@google.com> Co-authored-by: Geoff Romer <gromer@google.com> |
||
|
Chandler CarruthandJon Ross-Perkins
|
97e98bcc5a |
Shrink the lexer's token location and line data structures. (#4269)
First, this replaces the separate line index and column index in the token information with a single 32-bit byte offset of the token. This is then used to compute line and column numbers with a binary search of the line structure and then using that to compute the column within the line. In practice, this is _much_ more efficient: - Smaller token data structure. This will hopefully combine with a subsequent optimization PR that shrinks the token data structure still further. - Fewer stores to form each token's information in the tight hot loop of the lexer. - Less state to maintain while lexing, fewer computations while lexing. We only have to search to build the line and column information off the hot lexing path, and so this ends up being a significant win and shrinks some of the more significant data structures. Second, this shrinks the line start to a 32-bit integer and removes the line length. Our source buffer already ensures we only have 2 GiB of source with a nice diagnostic. I've just added a check to help document this in the lexer. The line length can be avoided in all of the cases it was being used, largely by looking at the next line's start and working from there. This also precipitated cleaning up some code that dated from when lines were only built during lexing rather than being pre-built, which resulted in nice simplifications. With this PR, I think it makes sense to re-name a bunch of methods on `TokenizedBuffer`, but to an extent that was already needed as these methods somewhat predate the more pervasive style conventions. I avoided that here to keep this PR focused on the implementation change, I'll create a subsequent PR to update the API to both better nomenclature and remove deviations from our conventions. There may also be a way to de-duplicate the binary search in the diagnostic location conversion and the main line accessor binary search, but it wasn't obvious to me that it would be a net savings, so left it alone for now. The performance impact of this varies quite a bit... The lexer's benchmark improves pretty consistent across the board on both x86 and Arm. For x86, where I have nice comparison tools, it appears 3% to 20% faster depending on the specific pattern. For Arm server CPUs at least it seems a much smaller but still an improvement. The overall compilation benchmarks however don't improve much with these changes alone on x86. Significant reduction in instruction count required for lexing, but the overall performance is bottlenecked elsewhere in the overall compilation it seems. However, on Arm, despite the more modest gains in special cases of lexing, this shows fairly consistent 1-2% improvements in overall lexing performance on our compilation benchmark. And the expectaiton is these improvements will compound with subsequent work to further compact our representation. --------- Co-authored-by: Jon Ross-Perkins <jperkins@google.com> |
||
 David Blaikie
|
a17480133f | Remove excess use of auto on initializers (auto x = Y(z) -> Y x(z)) (#4239) | ||
|
Chandler Carruth
|
44c85e0872 |
Reserve memory for the identifiers hashtable. (#4107)
This uses a heuristic reserve to greatly reduce hashtable growth of the identifiers hashtable. The design of the hashtable itself is optimized around compact memory use and is especially slow to grow and so this has an outsized impact. The heuristic was computed using `scripts/source_stats.py` and looking at C++ codebases. We may want to periodically re-evaluate it as Carbon code emerges and we have better data on its distributions of tokens. This also required fixing the `Reserve` method on `CanonicalValueStore` that wasn't actually used anywhere and so didn't even compile correctly. I added it to the relevant unit test so it is at least compiled locally to its definition. |
||
|
Chandler CarruthandRichard Smith
|
565fc5cebb |
Initial support for CR+LF (DOS / Windows) line endings. (#4056)
It turns out we can make these work with very minimal complexity because the LF is still in the right place either way. This also lets us easily support mixtures of LF and CR+LF line endings gracefully. We create the line structures around the LF bytes and then have the byte-dispatch loop notice a CR followed by an LF and skip to the LF behavior. Rather than add the remaining complexity around supporting bare CR and LF+CR sequences (both of which are quite rare now), this just adds diagnostics when we encounter a CR byte that won't fall out of our CR+LF handling. This is a better experience for users than the alternative. We still have a TODO to handle the full complexity of vertical whitespace, but I've updated it to reflect that the common case should be handled already. This isn't complete though: we need to add support in string literal lexing, and we need to teach the diagnostic rendering to handle the error messages above better. But those will be future PRs, this is enough to unblock folks who happen to edit a Carbon source file with notepad on Windows which seems important. --------- Co-authored-by: Richard Smith <richard@metafoo.co.uk> |
||
|
Chandler Carruth
|
dd0890619a |
Enable a couple of boring warnings. (#4018)
Just spotted these while looking at warnings that seem to fire on our code are probably are things we'd fix if we saw them. None of these seem important FWIW. Also removes a redundant flag that is part of `-Wall`. I have a follow-up for the high-value warning I spotted that motivated me to look at all of this. But it's noisy so kept it as a separate PR. |
||
|
Jon Ross-Perkins
|
37aa309c81 |
Add recovery tokens using AddToken to get correct parse tree sizes. (#3851)
The expected parse tree size is validated for valid parse trees, which is possible even if lex encounters an error. In that case, we were missing recovery tokens, resulting in a crash. AddToken should be the only place we call `token_infos_.push_back`. |
||
|
Richard Smith
|
45c071f2af |
Rework raw identifier lexing to avoid slowing down regular identifiers. (#3855)
Instead of special-casing tokens starting with `r`, lex them as normal identifiers, and add a special case to `#` handling to detect if the previous token was an `r` identifier. This roughly doubles the time to lex a raw identifier, because we do two hash table insertions rather than one, and probably slightly slows down lexing string literals starting with `#`, but should remove the 2% overhead to identifier lexing from the previous approach. |
||
|
Richard SmithandJon Ross-Perkins
|
f9ce0b194d |
Defer parsing of method bodies until the end of a suitable enclosing scope. (#3832)
In parse, form a list of methods that are defined inline, tracking where they start, where they end, and which other inline methods are nested within them. In check, when we reach an inline method body, skip it and add it to a worklist to be processed later. We also track when we reach the start and end of a context in which inline method bodies are deferred, so that we know when to replay the bodies. When suspending a function definition to be processed later, the `DeclNameStack` entry is moved to separate storage, including popping the corresponding scopes from the scope stack and removing the corresponding lexical names from lexical lookup. Later, when we return to the function and parse its definition, the `DeclNameStack` entry is restored. The same is done when we reach the end of a nested context that can have inline methods, so that we can reenter the nested scope before processing its members. --------- Co-authored-by: Jon Ross-Perkins <jperkins@google.com> |
||
|
Jon Ross-Perkins
|
b5d28f2c4b | location -> loc abbreviation (#3826) | ||
|
Jon Ross-Perkins
|
0bd45f0d6b |
Rename DiagnosticLocationTranslator -> DiagnosticConverter (#3804)
Since the addition of TranslateArg, I don't think this type is going to go away (cutting a TODO). Refactoring names slightly to fit the current role, and adding const to ConvertLocation. |
||
|
Jon Ross-Perkins
|
1974e44fd9 |
Rename factory functions from 'Create' to 'Make' (#3706)
Similar to #3705, we actually have a mix of `Make` and `Create` in factory functions too, so this PR is normalizing on `Make`. It's intended to be consistent with the naming choice for Carbon factory functions. Note, MakeSyntheticBlock is the only one I feel a little weird about because llvm's own APIs use Create, and this is essentially wrapping LLVM calls. But the flipside is it also feels like a vague line to draw, when we also differ from LLVM coding style in other ways. |
||
|
Jon Ross-Perkins
|
379d776084 |
Add support for '--config=clang-tidy' (#3559)
This sets things up to use `bazel` to run `clang-tidy` using https://github.com/erenon/bazel_clang_tidy. I'm fixing issues outside of explorer, and disabling clang-tidy for targets in explorer that have legacy issues. I was going to disable clang-tidy for targets in explorer such as interpreter anyways, because they're slow to parse, and just extended that to the currently failing targets. |
||
|
Richard Smith
|
0a06fceb5f |
Improve diagnosis of mismatched brackets. (#3282)
Move handling of mismatched brackets out of the main lexing loop into a separate pass that is only run if there are mismatched brackets This is done in preparation for using both lookahead and lookbehind to work out how to match brackets, and to get this code far away from the hot lexing loop. Fix bracket insertion location to be immediately after the token that we're inserting the bracket after, rather than potentially at the end of a comment. When there are open brackets at the end of the file, say that there are open brackets, not that there's a closing bracket without a matching opening bracket. |
||
|
Richard Smith
|
d87fe8b532 |
Rename Carbon::StringLiteralId -> Carbon::StringLiteralValueId. (#3522)
We have `StringLiteral`s in multiple other `Carbon` sub-namespaces. Rename to a more specific name to avoid collisions. We should likely also rename `Carbon::IntId` -> `Carbon::IntValueId` and `Carbon::RealId` -> `Carbon::RealValueId`, but this collision is prioritized because it was blocking work on typed parse nodes which introduces a `Carbon::Parse::StringLiteralId`. |
||
 josh11b
|
9390e666ce |
startswith -> starts_with to match upstream (#3519)
This matches the change to upstream `llvm::StringRef`, see https://github.com/llvm/llvm-project/pull/75491 . |
||
|
Jon Ross-Perkins
|
ef0fa81a58 |
Upgrade clang-format version (#3471)
This apparently includes a fix for https://github.com/llvm/llvm-project/issues/47664 (previously https://bugs.llvm.org/show_bug.cgi?id=48320) Note this continues with a clang-format version that's different from the compiler versions we're often using, but the differences in formatting seem worthwhile. |
||
|
Jon Ross-Perkins
|
0db63ff17a |
Abbreviate Integer and FloatingPoint (#3435)
I was suggesting this because `FloatingPoint` is pretty long. `int` and `float` should be familiar abbreviations. `unsigned` should be familiar to developers too, but `UnsignedInt` still feels usefully clearer for the additional chars. |
||
|
Richard Smith
|
eae630a3db |
Rename Lex::{Token,Line} -> Lex::{Token,Line}Index. (#3433)
As discussed [on discord](https://discord.com/channels/655572317891461132/655578254970716160/1178878128714678282) and today's toolchain discussion. |
||
|
Jon Ross-Perkins
|
3f208e27f9 |
Align on FileStart/FileEnd for naming. (#3428)
The lexer has been using EndOfFile form (stemming from EOF), parser went to FileEnd form. This consolidates on FileEnd form. |
||
 Jacob Schneider
|
482d233def |
fix crash caused by unicode chars (#3387)
Prevent a crash during lexing for unicode chars. |
||
|
Jon Ross-PerkinsandChandler Carruth
|
cafcd88882 |
Split lexing logic and storage to separate files. (#3365)
Just reorganizing logic a little, trying to mirror the direction we've gone with check, lower, etc. That is, lex.h contains a function `Lex` that is used directly. Note, I'm avoiding making meaningful changes here. It could in theory still affect inlining in benchmarks, but I'm not seeing an impact. Before: ``` ------------------------------------------------------------------------------------------------------ Benchmark Time CPU Iterations UserCounters... ------------------------------------------------------------------------------------------------------ BM_ValidKeywords 2784949 ns 2784867 ns 249 bytes_per_second=214.452M/s tokens_per_second=35.9084M/s BM_ValidKeywordsAsRawIdentifiers 3222597 ns 3222551 ns 210 bytes_per_second=244.513M/s tokens_per_second=31.0313M/s BM_RawIdentifierFocus 5907836 ns 5907518 ns 103 bytes_per_second=264.873M/s tokens_per_second=16.9276M/s BM_ValidIdentifiers<1, 64, false> 6255128 ns 6254297 ns 105 bytes_per_second=235.488M/s tokens_per_second=15.989M/s BM_ValidIdentifiers<1, 1, true> |