Chandler Carruth and Jon Ross-Perkins
97e98bcc5a
Shrink the lexer's token location and line data structures. ( #4269 )
...
First, this replaces the separate line index and column index in the
token information with a single 32-bit byte offset of the token. This is
then used to compute line and column numbers with a binary search of the
line structure and then using that to compute the column within the
line. In practice, this is _much_ more efficient:
- Smaller token data structure. This will hopefully combine with a
subsequent optimization PR that shrinks the token data structure still
further.
- Fewer stores to form each token's information in the tight hot loop of
the lexer.
- Less state to maintain while lexing, fewer computations while lexing.
We only have to search to build the line and column information off the
hot lexing path, and so this ends up being a significant win and shrinks
some of the more significant data structures.
Second, this shrinks the line start to a 32-bit integer and removes the
line length. Our source buffer already ensures we only have 2 GiB of
source with a nice diagnostic. I've just added a check to help document
this in the lexer. The line length can be avoided in all of the cases it
was being used, largely by looking at the next line's start and working
from there. This also precipitated cleaning up some code that dated from
when lines were only built during lexing rather than being pre-built,
which resulted in nice simplifications.
With this PR, I think it makes sense to re-name a bunch of methods on
`TokenizedBuffer`, but to an extent that was already needed as these
methods somewhat predate the more pervasive style conventions. I avoided
that here to keep this PR focused on the implementation change, I'll
create a subsequent PR to update the API to both better nomenclature and
remove deviations from our conventions.
There may also be a way to de-duplicate the binary search in the
diagnostic location conversion and the main line accessor binary search,
but it wasn't obvious to me that it would be a net savings, so left it
alone for now.
The performance impact of this varies quite a bit...
The lexer's benchmark improves pretty consistent across the board on
both x86 and Arm. For x86, where I have nice comparison tools, it
appears 3% to 20% faster depending on the specific pattern. For Arm
server CPUs at least it seems a much smaller but still an improvement.
The overall compilation benchmarks however don't improve much with these
changes alone on x86. Significant reduction in instruction count
required for lexing, but the overall performance is bottlenecked
elsewhere in the overall compilation it seems. However, on Arm, despite
the more modest gains in special cases of lexing, this shows fairly
consistent 1-2% improvements in overall lexing performance on our
compilation benchmark. And the expectaiton is these improvements will
compound with subsequent work to further compact our representation.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com >
2024-09-03 23:56:44 +00:00
David Blaikie
a17480133f
Remove excess use of auto on initializers (auto x = Y(z) -> Y x(z)) ( #4239 )
2024-08-22 20:29:35 +00:00
Chandler Carruth
44c85e0872
Reserve memory for the identifiers hashtable. ( #4107 )
...
This uses a heuristic reserve to greatly reduce hashtable growth of the
identifiers hashtable. The design of the hashtable itself is optimized
around compact memory use and is especially slow to grow and so this has
an outsized impact.
The heuristic was computed using `scripts/source_stats.py` and looking
at C++ codebases. We may want to periodically re-evaluate it as Carbon
code emerges and we have better data on its distributions of tokens.
This also required fixing the `Reserve` method on `CanonicalValueStore`
that wasn't actually used anywhere and so didn't even compile correctly.
I added it to the relevant unit test so it is at least compiled locally
to its definition.
2024-07-18 15:41:21 +00:00
Chandler Carruth and Richard Smith
565fc5cebb
Initial support for CR+LF (DOS / Windows) line endings. ( #4056 )
...
It turns out we can make these work with very minimal complexity because
the LF is still in the right place either way. This also lets us easily
support mixtures of LF and CR+LF line endings gracefully. We create the
line structures around the LF bytes and then have the byte-dispatch loop
notice a CR followed by an LF and skip to the LF behavior.
Rather than add the remaining complexity around supporting bare CR and
LF+CR sequences (both of which are quite rare now), this just adds
diagnostics when we encounter a CR byte that won't fall out of our CR+LF
handling. This is a better experience for users than the alternative. We
still have a TODO to handle the full complexity of vertical whitespace,
but I've updated it to reflect that the common case should be handled
already.
This isn't complete though: we need to add support in string literal
lexing, and we need to teach the diagnostic rendering to handle the
error messages above better. But those will be future PRs, this is
enough to unblock folks who happen to edit a Carbon source file with
notepad on Windows which seems important.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk >
2024-06-18 02:23:25 +00:00
Chandler Carruth
dd0890619a
Enable a couple of boring warnings. ( #4018 )
...
Just spotted these while looking at warnings that seem to fire on our
code are probably are things we'd fix if we saw them. None of these seem
important FWIW.
Also removes a redundant flag that is part of `-Wall`.
I have a follow-up for the high-value warning I spotted that motivated
me to look at all of this. But it's noisy so kept it as a separate PR.
2024-06-03 05:02:26 +00:00
Jon Ross-Perkins
37aa309c81
Add recovery tokens using AddToken to get correct parse tree sizes. ( #3851 )
...
The expected parse tree size is validated for valid parse trees, which
is possible even if lex encounters an error. In that case, we were
missing recovery tokens, resulting in a crash. AddToken should be the
only place we call `token_infos_.push_back`.
2024-04-04 16:03:54 +00:00
Richard Smith
45c071f2af
Rework raw identifier lexing to avoid slowing down regular identifiers. ( #3855 )
...
Instead of special-casing tokens starting with `r`, lex them as normal
identifiers, and add a special case to `#` handling to detect if the
previous token was an `r` identifier.
This roughly doubles the time to lex a raw identifier, because we do two
hash table insertions rather than one, and probably slightly slows down
lexing string literals starting with `#`, but should remove the 2%
overhead to identifier lexing from the previous approach.
2024-04-03 23:42:28 +00:00
Richard Smith and Jon Ross-Perkins
f9ce0b194d
Defer parsing of method bodies until the end of a suitable enclosing scope. ( #3832 )
...
In parse, form a list of methods that are defined inline, tracking where
they start, where they end, and which other inline methods are nested
within them.
In check, when we reach an inline method body, skip it and add it to a
worklist to be processed later. We also track when we reach the start
and end of a context in which inline method bodies are deferred, so that
we know when to replay the bodies.
When suspending a function definition to be processed later, the
`DeclNameStack` entry is moved to separate storage, including popping
the corresponding scopes from the scope stack and removing the
corresponding lexical names from lexical lookup. Later, when we return
to the function and parse its definition, the `DeclNameStack` entry is
restored. The same is done when we reach the end of a nested context
that can have inline methods, so that we can reenter the nested scope
before processing its members.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com >
2024-04-01 18:25:27 +00:00
Jon Ross-Perkins
b5d28f2c4b
location -> loc abbreviation ( #3826 )
2024-03-28 18:15:18 +00:00
Jon Ross-Perkins
0bd45f0d6b
Rename DiagnosticLocationTranslator -> DiagnosticConverter ( #3804 )
...
Since the addition of TranslateArg, I don't think this type is going to
go away (cutting a TODO). Refactoring names slightly to fit the current
role, and adding const to ConvertLocation.
2024-03-21 23:35:04 +00:00
Jon Ross-Perkins
1974e44fd9
Rename factory functions from 'Create' to 'Make' ( #3706 )
...
Similar to #3705 , we actually have a mix of `Make` and `Create` in
factory functions too, so this PR is normalizing on `Make`. It's
intended to be consistent with the naming choice for Carbon factory
functions.
Note, MakeSyntheticBlock is the only one I feel a little weird about
because llvm's own APIs use Create, and this is essentially wrapping
LLVM calls. But the flipside is it also feels like a vague line to draw,
when we also differ from LLVM coding style in other ways.
2024-02-14 18:26:56 +00:00
Jon Ross-Perkins
379d776084
Add support for '--config=clang-tidy' ( #3559 )
...
This sets things up to use `bazel` to run `clang-tidy` using
https://github.com/erenon/bazel_clang_tidy .
I'm fixing issues outside of explorer, and disabling clang-tidy for
targets in explorer that have legacy issues. I was going to disable
clang-tidy for targets in explorer such as interpreter anyways, because
they're slow to parse, and just extended that to the currently failing
targets.
2024-01-04 18:09:52 +00:00
Richard Smith
0a06fceb5f
Improve diagnosis of mismatched brackets. ( #3282 )
...
Move handling of mismatched brackets out of the main lexing loop into a
separate pass that is only run if there are mismatched brackets This is
done in preparation for using both lookahead and lookbehind to work out
how to match brackets, and to get this code far away from the hot lexing
loop.
Fix bracket insertion location to be immediately after the token that
we're inserting the bracket after, rather than potentially at the end of
a comment. When there are open brackets at the end of the file, say that
there are open brackets, not that there's a closing bracket without a
matching opening bracket.
2023-12-21 08:49:37 +00:00
Richard Smith
d87fe8b532
Rename Carbon::StringLiteralId -> Carbon::StringLiteralValueId. ( #3522 )
...
We have `StringLiteral`s in multiple other `Carbon` sub-namespaces.
Rename to a more specific name to avoid collisions.
We should likely also rename `Carbon::IntId` -> `Carbon::IntValueId` and
`Carbon::RealId` -> `Carbon::RealValueId`, but this collision is
prioritized because it was blocking work on typed parse nodes which
introduces a `Carbon::Parse::StringLiteralId`.
2023-12-18 23:27:33 +00:00
josh11b
9390e666ce
startswith -> starts_with to match upstream (#3519 )
...
This matches the change to upstream `llvm::StringRef`, see
https://github.com/llvm/llvm-project/pull/75491 .
2023-12-18 17:47:02 +00:00
Jon Ross-Perkins
ef0fa81a58
Upgrade clang-format version ( #3471 )
...
This apparently includes a fix for
https://github.com/llvm/llvm-project/issues/47664 (previously
https://bugs.llvm.org/show_bug.cgi?id=48320 )
Note this continues with a clang-format version that's different from
the compiler versions we're often using, but the differences in
formatting seem worthwhile.
2023-12-07 19:09:36 +00:00
Jon Ross-Perkins
0db63ff17a
Abbreviate Integer and FloatingPoint ( #3435 )
...
I was suggesting this because `FloatingPoint` is pretty long. `int` and
`float` should be familiar abbreviations. `unsigned` should be familiar
to developers too, but `UnsignedInt` still feels usefully clearer for
the additional chars.
2023-11-29 23:29:48 +00:00
Richard Smith
eae630a3db
Rename Lex::{Token,Line} -> Lex::{Token,Line}Index. ( #3433 )
...
As discussed [on
discord](https://discord.com/channels/655572317891461132/655578254970716160/1178878128714678282 )
and today's toolchain discussion.
2023-11-29 20:33:58 +00:00
Jon Ross-Perkins
3f208e27f9
Align on FileStart/FileEnd for naming. ( #3428 )
...
The lexer has been using EndOfFile form (stemming from EOF), parser went
to FileEnd form. This consolidates on FileEnd form.
2023-11-29 16:36:57 +00:00
Jacob Schneider
482d233def
fix crash caused by unicode chars ( #3387 )
...
Prevent a crash during lexing for unicode chars.
2023-11-11 06:27:15 +00:00
Jon Ross-Perkins and Chandler Carruth
cafcd88882
Split lexing logic and storage to separate files. ( #3365 )
...
Just reorganizing logic a little, trying to mirror the direction we've
gone with check, lower, etc. That is, lex.h contains a function `Lex`
that is used directly.
Note, I'm avoiding making meaningful changes here. It could in theory
still affect inlining in benchmarks, but I'm not seeing an impact.
Before:
```
------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
------------------------------------------------------------------------------------------------------
BM_ValidKeywords 2784949 ns 2784867 ns 249 bytes_per_second=214.452M/s tokens_per_second=35.9084M/s
BM_ValidKeywordsAsRawIdentifiers 3222597 ns 3222551 ns 210 bytes_per_second=244.513M/s tokens_per_second=31.0313M/s
BM_RawIdentifierFocus 5907836 ns 5907518 ns 103 bytes_per_second=264.873M/s tokens_per_second=16.9276M/s
BM_ValidIdentifiers<1, 64, false> 6255128 ns 6254297 ns 105 bytes_per_second=235.488M/s tokens_per_second=15.989M/s
BM_ValidIdentifiers<1, 1, true> 36776305518589chandlerc@gmail.com >
2023-11-10 23:59:49 +00:00
Chandler CarruthandJon Ross-Perkins
David Blaikie
josh11b
Jacob Schneider