Commit Graph
57 Commits
Author SHA1 Message Date
Chandler CarruthandRichard Smith 68569d234d Improve lexer and parser diagnostic locations (#7990)
Lexer diagnostics used a single `const char*` as their source location,
which only allowed pointing at a single character rather than a run of
characters. This limited the underlining of unrecognized tokens, and
also made escape sequence diagnostics particularly strange by stopping
at whatever point the lexer discovered the error and pointing only at
that location. For example: `"\w"` was reported at the `w`, and `"\x"`
at the closing `"`.

This change tries to always keep the primary location of a diagnostic
the point at which typing (inserting or deleting) might fix the error.
So for `"\x"`, `\x` is underlined, but the position is at the `"` as
that's where you would type a hex code.

The lexer's emitter now takes a special lexing source location that
represents either a single position _between_ two characters, a single
character, or a range of the source buffer. The lexer diagnostics are
updated to construct these for the character range as appropriate.

Parser diagnostics for a missing token were located at the _next_ token,
which is particularly confusing for terminators like missing `;`s. Now
they are located at a position after the previous token, with or without
a space depending on the token, but typically on the same line.
Diagnostics still sort by how far the parser has read.

The parser diagnostics for unclosed groups also now attaches a label to
the opening token when it is on a separate line. And the diagnostics for
mismatched indent in a multi-line string attaches a label to the closing
`'''` that establishes that indent.

There are likely more location improvements we should do here, but this
seemed like a reasonable incremental improvement and is what Claude
found when doing an initial audit. One example where I think we should
make further improvements is around suggesting raw identifiers when we
find an unexpected `Cpp` or `Core` or `base`. We do this consistently
now, but raw identifiers should be rare. A TODO is added for these
cases.

Assisted-by: Claude Code

---------

Co-authored-by: Richard Smith <richard@metafoo.co.uk>
2026-11-01 06:22:00 +00:00
Richard Smith d87891f3a6 Fix diagnostic style issues (#7982)
Remove trailing `.`s and leading capitalization from diagnostics, and
separate clauses with `;` rather than `.`, per
https://docs.carbon-lang.dev/toolchain/docs/diagnostics.html#diagnostic-message-style-guide
2026-10-15 20:15:28 +00:00
Chandler Carruth 2c9739799b Render diagnostics through the terminal library (#7683)
Diagnostic rendering was part of the data model: `Loc::FormatLocation`
and
`Loc::FormatSnippet` wrote plain text to a stream, so no capability,
color, or
terminal width could reach them.

A diagnostic is now one message with labels, contexts, and location
information
attached, and `Diagnostics::Renderer` draws it into a
`Terminal::Buffer`. The
design is in `toolchain/docs/diagnostics_rendering.md` (#7668),
authoring in
`toolchain/docs/diagnostics.md`.

Before:

```
example.carbon:6:3: error: 1 argument passed to function expecting 0 arguments
  Run0(1);
  ^~~~~~~
example.carbon:3:1: note: calling function declared here
fn Run0() {}
^~~~~~~~~~~
```

After:

```
error: 1 argument passed to function expecting 0 arguments
       ╭─┤ example.carbon:6:3
       ┆
     3 │ fn Run0() {}
       · ─────┬─────
       ·      ╰──┤ calling function declared here
       ┆
->   6 │   Run0(1);
       ·   ───────
       ╰────────────
```

`--no-diagnostic-snippets` gives a compact form, `example.carbon:6:3-9`.
File
tests use it; the rendering tests under `toolchain/diagnostics/testdata`
ask for
snippets.

C++ interop diagnostics are data now rather than text Clang drew:
`SemIR::ClangSourceLocId` names a `clang::SourceRange`, and a Clang
location is
marked as the point it names.

`StreamConsumer` moves to a `:renderer` target that `:emitter` does not
depend
on, keeping `common/terminal` out of `consumer.h`. Capabilities come
from the
file behind the error stream, so tests get plain ASCII.

`renderer_fuzzer` draws diagnostics built from fuzzer bytes with every
capability combination, since a shape the renderer doesn't expect must
never be
why the compiler dies while reporting a problem.

TODOs mark the follow-ups: line weights and crossing lines, wrapping
that keeps
a quoted type whole, and ordering diagnostics that share a byte offset.

Also: `IncorrectExtension`'s hint moves into its message, since its note
had no
location of its own, and `strip_noise` in `check_diagnostics.py` blanks
multi-line `#define`s instead of deleting them.

Assisted-by: Claude Code
2026-10-15 06:08:57 +00:00
Richard Smith 72e425fe5c Clean up some leftover references to the old :! syntax. (#7921) 2026-10-07 23:07:01 +00:00
Richard Smith 2ec4bab59b Lex and parse support for ... statements and expand. (#7832)
No support for `each` yet, but this should be enough that we can try out
an implementation approach supporting expanding a statement across
elements of a tuple without building "real" variadics.

Assisted-by: Claude via Antigravity
2026-10-05 22:23:12 +00:00
Özgür T. Önsoy 5474347d7d Support lexing and parsing positional params (#7651)
This implements lexing and parsing positional parameters such as `$0`.
2026-09-03 19:06:20 +00:00
Richard Smith 6515090557 Don't provide the "start of file" token to the bracket fixer. (#7675)
We previously passed in the start of file token, classified as
BracketTokenKind::Other, which allowed the bracket fixer to consider
corrections where it inserted tokens (such as a `{`) *before* the
start-of-file token.

Fixes #7672.
2026-08-24 18:14:10 +00:00
Richard Smith 3bbc03f527 Add better algorithm for repairing mismatched brackets (#7574)
Adds an algorithm to compute where to insert brackets to repair
bracketing mismatches during lexing. This takes indentation, as well as
a number of other cues, into account to predict where the brackets
should have gone. Detects when there is ambiguity between solutions and
makes no suggestion in that case. Reduces the problem by splitting on
properly bracketed top-level constructs, then uses a beam search to find
good candidate solutions quickly.

This includes both a fuzzer and an eval tool that can be used to
determine how well the algorithm fares against a given corpus of valid
Carbon code, by damaging it in various ways and seeing whether the
algorithm can correctly fix it. On all the eval modes, this algorithm
can correctly infer the positions for over 80% of lost brackets (and can
correctly restore 95+% of brackets in some modes), with low rates of
incorrect suggestions.

See added documentation for full details.

Assisted-by: Gemini via Antigravity, Claude via Claude Code
2026-08-19 18:55:32 +00:00
Chandler Carruth f0848b1f5e Trailing comments (#7441)
Carbon currently requires a comment to be the only non-whitespace on its
line. A `//` comment that follows other content on a line, called a
_trailing comment_, is a lexer error. This proposal removes that
restriction, allowing a comment to follow other content on a line.
Everything else about comments is unchanged: a comment still begins with
`//`, still requires whitespace after the `//`, and still runs to the
end of the line. Carbon continues to provide only line comments; no
block or intra-line comments are added.

Three observations motivate the change. First, trailing comments are
well suited to short _annotations_ attached to a specific entity or
value on a line. Second, the lexer design now makes it trivial to lex
trailing comments, and in fact requires extra logic and potentially cost
to reject them. Third, C++ code routinely uses trailing comments, so
allowing them lets Carbon carry the layout of migrated code over
directly, rather than reworking each comment to read well in a different
structure.

Implementation notes (beyond the proposal's design):

Keeping trailing comments cheap to lex required a few supporting
changes, all of which keep the cost off the lexer's hot path:

- The lexer already dispatches `//` to comment lexing wherever it
appears, so classifying a comment as trailing is a single O(1) check of
whether the `//` is the line's first non-whitespace (`start + indent`).
The hot comment path is otherwise unchanged.

- That check relies on each line's recorded indentation being its real
leading whitespace. Multi-line string literals previously recorded the
column where the literal opened for the lines they span; they now record
the true (closing-delimiter) indentation instead.

- Parser error recovery (`SkipPastLikelyEnd`) had relied on that
opening-column indentation to keep tokens following a multi-line string
literal attached to the same construct. It now reconstructs that
relationship directly by consulting the line on which the literal
opened, including when other tokens follow the closing delimiter (such
as `''' + "more"`). This is on the cold recovery path.

- `CommentData` records the trailing bit in the high bit of its length
field, keeping it at 8 bytes.

Assisted-by: Claude Code
2026-07-04 06:42:02 +00:00
Richard Smith 862b1c91f8 Fix GetTokenText for raw identifiers. (#7250)
Include the `r#` in the spelling of the identifier.
2026-05-22 01:57:13 +00:00
Jon Ross-Perkins d47b6221ae Fix invalid digit caret (#6921)
Stumbled on this playing with numeric literals

Assisted-by: Google Antigravity with Gemini
2026-03-17 18:13:55 +00:00
Jon Ross-PerkinsandChandler Carruth 613a139bef Add support for octal numbers (#6909)
This implements the leads decision made in #6821, proposal #6910. The
proposal is pending, but I figured it's relatively safe to just do given
the decision.

Assisted-by: Google Antigravity with Gemini

---------

Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
2026-03-16 21:28:37 +00:00
Jon Ross-Perkins c5761d2d16 Improve parsing large integers (#6908)
Though I started this thinking about performance of parse of large
decimal integers, I extended it to generally improve performance of
integer values (TBH I hadn't expected such a difference for binary/hex,
but I'll take it).

Note I think tests change because I'm making subtle changes to bit
widths. The changes themselves appear harmless to me, but happy to make
changes if it'd help.

Bumping up the number of digits by 10x because it's not really a
performance issue anymore (eh, maybe somebody will want to specify a
256-byte value in binary). But, at a certain point it still seems like a
mistake if somebody has that many digits in a row.

Fixes #980

Highlighting benchmark differences:
```diff
- BM_ComputeValue_IntDecimalN/1           37.1 ns         37.1 ns     18887116
+ BM_ComputeValue_IntDecimalN/1           21.9 ns         21.9 ns     31902433
- BM_ComputeValue_IntDecimalN/10000 1251228680 ns   1250457559 ns            1
+ BM_ComputeValue_IntDecimalN/10000     458818 ns       458626 ns         1523
- BM_ComputeValue_IntBinaryN/1            29.0 ns         29.0 ns     24058533
+ BM_ComputeValue_IntBinaryN/1            22.2 ns         22.1 ns     31566949
- BM_ComputeValue_IntBinaryN/10000     1390557 ns      1389782 ns          506
+ BM_ComputeValue_IntBinaryN/10000       16402 ns        16396 ns        42744
- BM_ComputeValue_IntHexN/1               34.0 ns         34.0 ns     20562432
+ BM_ComputeValue_IntHexN/1               22.4 ns         22.4 ns     31238055
- BM_ComputeValue_IntHexN/10000        5387942 ns      5385262 ns          130
+ BM_ComputeValue_IntHexN/10000          39249 ns        39233 ns        17859
```

Benchmark before:
```
----------------------------------------------------------------------------
Benchmark                                  Time             CPU   Iterations
----------------------------------------------------------------------------
BM_Lex_Float                            10.6 ns         10.6 ns     66138191
BM_Lex_Int                              15.5 ns         15.4 ns     45149703
BM_Lex_IntDecimalN/1                    3.11 ns         3.11 ns    225524908
BM_Lex_IntDecimalN/10                   11.8 ns         11.8 ns     56719805
BM_Lex_IntDecimalN/100                   102 ns          102 ns      6867468
BM_Lex_IntDecimalN/1000                  943 ns          942 ns       745313
BM_Lex_IntDecimalN/10000                9465 ns         9461 ns        73970
BM_ComputeValue_Float                   61.6 ns         61.6 ns     11377463
BM_ComputeValue_Int                      106 ns          106 ns      6587381
BM_ComputeValue_IntDecimalN/1           37.1 ns         37.1 ns     18887116
BM_ComputeValue_IntDecimalN/10          87.7 ns         87.7 ns      7960837
BM_ComputeValue_IntDecimalN/100         7963 ns         7956 ns        88858
BM_ComputeValue_IntDecimalN/1000     1212577 ns      1211906 ns          578
BM_ComputeValue_IntDecimalN/10000 1251228680 ns   1250457559 ns            1
BM_ComputeValue_IntBinaryN/1            29.0 ns         29.0 ns     24058533
BM_ComputeValue_IntBinaryN/10           69.4 ns         69.4 ns     10108642
BM_ComputeValue_IntBinaryN/100           963 ns          962 ns       726982
BM_ComputeValue_IntBinaryN/1000        21562 ns        21551 ns        32506
BM_ComputeValue_IntBinaryN/10000     1390557 ns      1389782 ns          506
BM_ComputeValue_IntHexN/1               34.0 ns         34.0 ns     20562432
BM_ComputeValue_IntHexN/10              70.4 ns         70.4 ns      9953165
BM_ComputeValue_IntHexN/100             1474 ns         1473 ns       472776
BM_ComputeValue_IntHexN/1000           61818 ns        61762 ns        11363
BM_ComputeValue_IntHexN/10000        5387942 ns      5385262 ns          130
```

Benchmark after:
```
----------------------------------------------------------------------------
Benchmark                                  Time             CPU   Iterations
----------------------------------------------------------------------------
BM_Lex_Float                            10.9 ns         10.9 ns     63993114
BM_Lex_Int                              15.1 ns         15.1 ns     46869766
BM_Lex_IntDecimalN/1                    3.16 ns         3.16 ns    220923300
BM_Lex_IntDecimalN/10                   12.2 ns         12.2 ns     57731654
BM_Lex_IntDecimalN/100                   102 ns          102 ns      6875516
BM_Lex_IntDecimalN/1000                  942 ns          942 ns       742359
BM_Lex_IntDecimalN/10000                9353 ns         9350 ns        75096
BM_ComputeValue_Float                   44.9 ns         44.9 ns     15619691
BM_ComputeValue_Int                     48.9 ns         48.9 ns     14361507
BM_ComputeValue_IntDecimalN/1           21.9 ns         21.9 ns     31902433
BM_ComputeValue_IntDecimalN/10          30.3 ns         30.3 ns     23134117
BM_ComputeValue_IntDecimalN/100          224 ns          223 ns      3092567
BM_ComputeValue_IntDecimalN/1000        5834 ns         5830 ns       117469
BM_ComputeValue_IntDecimalN/10000     458818 ns       458626 ns         1523
BM_ComputeValue_IntBinaryN/1            22.2 ns         22.1 ns     31566949
BM_ComputeValue_IntBinaryN/10           32.9 ns         32.9 ns     21306927
BM_ComputeValue_IntBinaryN/100           198 ns          198 ns      3545277
BM_ComputeValue_IntBinaryN/1000         1671 ns         1669 ns       419656
BM_ComputeValue_IntBinaryN/10000       16402 ns        16396 ns        42744
BM_ComputeValue_IntHexN/1               22.4 ns         22.4 ns     31238055
BM_ComputeValue_IntHexN/10              47.8 ns         47.7 ns     14694407
BM_ComputeValue_IntHexN/100              436 ns          436 ns      1609794
BM_ComputeValue_IntHexN/1000            3966 ns         3962 ns       177109
BM_ComputeValue_IntHexN/10000          39249 ns        39233 ns        17859
```

Assisted-by: Google Antigravity with Gemini
2026-03-16 20:00:51 +00:00
Ilya 6304df1db9 Fix crash in character literal lexing (#6805)
When lexing a hash-prefixed character literal, the lexer assumed that
the hash level of escape sequences inside the literal was zero, which
allowed unclosed escape sequences inside the literal which crashed the
compiler.

Closes #6799
2026-03-10 20:25:29 +00:00
Richard Smith 1b2ae912fc Add basic support for eval fn and musteval fn. (#6694)
Add support for compile-time functions. `eval fn` is analogous to C++
`constexpr`, and is evaluated at compile time when it has compile-time
arguments. `musteval fn` is analogous to C++ `consteval`, and requires
that its arguments be available at compile time and is always evaluated
at compile time. For now we require the modifier to match across
redeclarations of the function. The specific modifier syntax here is a
placeholder and not yet part of an approved design.

Limitations: Only very basic support for evaluation is provided. So far
there's no support for mutable state or `if` expressions, but otherwise
control flow and passing and returning values should work. Carbon
evaluation recursion is modeled by C++ recursion for now, so you can
overflow the toolchain stack easily. Functions that use in-place
initialization will generally not work yet, as they are modeled as
passing a non-compile-time-constant reference to a temporary to the
call.

Add missing categorization of `name_binding_decl` as `NotExpr` to match
other similar declaration instructions like `FunctionDecl`, so that we
can uniformly skip over them when they occur within function bodies.

Assisted-by: Gemini 3 Pro and Flash via Antigravity
2026-02-11 02:08:16 +00:00
Richard Smith bac828d244 Add support for char keyword per #5903. (#6078)
Make inst namer and stringify print `Core.Char` and `Core.String` as
`char` and `str` respectively. Plus a few cleanups.
2025-09-16 20:45:01 +00:00
Richard Smith ad84e71acd Avoid non-UTF8-encoded test files. (#5965)
Add a content keyword to file_test, `[[@0xAB]]`, that expands to the
code unit 0xAB, and use that instead of putting raw malformed code units
in test files.

Instead of printing the raw input bytes in snippets in diagnostics,
replace non-printable characters with <AB> in the output, being careful
to still compute the location of the caret and underscore properly.
2025-08-20 21:07:58 +00:00
Richard Smith 41ed82e033 Add basic support for strings to core, check, and lower. (#5963)
Add a `Core.String` class to the prelude representing a string view, and
rename the `String` keyword to `str` and make it evaluate to
`Core.String`.

`Core.String` is represented as a pair of a pointer to a character
(actually, to the first character of a string, but we don't have a way
of modeling that yet) and a size (which should be pointer-width, but is
currently always a `u64` as we don't have a `usize` equivalent yet).
`Core.String` values are generated directly by the toolchain for string
literal expressions.

This follows the direction established at the recent summit, but the
design implemented here has not been through the proposal process yet.
2025-08-20 00:18:50 +00:00
Jon Ross-Perkins 8d08e774fc Add a feature to explicitly include a file's SemIR (#5961)
Trying to figure out an easy way to debug semir in the prelude, #5703
removed an option to set `--exclude-dump-file-prefix` to empty. But,
this is probably an improvement over that flow... With this change, it's
possible to add `//@dump-sem-ir-file` to a specific prelude file, and
its full IR will be printed. Additionally, it becomes an option with the
default `--dump-sem-ir-ranges=only` to add `//@dump-sem-ir-file` and get
the full file's IR.
2025-08-15 18:53:43 +00:00
Richard Smith 7727c62880 Enforce a couple of char literal restrictions from #1964: (#5960)
* `\x` escapes are not permitted in character literals
* ASCII control characters (U+0000 .. U+001F) are not permitted in
character literals unless specified with escape sequences.
2025-08-15 00:45:16 +00:00
Jon Ross-PerkinsandRichard Smith cae8aa3adf Support lexing characters (#5893)
Adapts `StringLiteral` to lex characters. Adds a `CharLiteral` token,
which contains a `CharLiteralValue` which is a straight unicode code
point (suggested by zygoloid).

---------

Co-authored-by: Richard Smith <richard@metafoo.co.uk>
2025-08-01 18:24:03 +00:00
Jon Ross-Perkins 1f268b5d8b Consolidate token-related range handling to one struct (#5399)
This consolidates Lex::TokenizedBuffer::DumpSemIRRange and
Parse::TreeAndSubtrees::TokenRange into a single InclusiveTokenRange,
also making the OverlapsWithDumpSemIRRange function take the new struct.

I considered switching to `llvm::iterator_range<Lex::TokenIterator>`,
but we often want to see if the range is size one. Using `TokenIterator`
just looked like it'd add a bunch of offsetting to make it work; I view
that as low-value overhead.

For example:

```
  Lex::InclusiveTokenRange token_range = GetSubtreeTokenRange(node_id);
  auto begin_loc = tree_->tokens().TokenToDiagnosticLoc(token_range.begin);
  if (token_range.begin == token_range.end) {
    return begin_loc;
  }
  auto end_loc = tree_->tokens().TokenToDiagnosticLoc(token_range.end);
```

would become:

```
  llvm::iterator_range<Lex::TokenIterator> token_range = GetSubtreeTokenRange(node_id);
  auto begin_loc = tree_->tokens().TokenToDiagnosticLoc(*token_range.begin());
  if (token_range.begin() + 1 == token_range.end()) {
    return begin_loc;
  }
  auto end_loc = tree_->tokens().TokenToDiagnosticLoc(*(token_range.end() - 1));
```

So I'm keeping the bespoke struct.
2025-05-02 18:47:02 +00:00
Jon Ross-Perkins 828eccebba Switch dump-sem-ir-start to dump-sem-ir-begin (#5378)
Simply about consistency with begin/end naming.
2025-04-29 20:06:58 +00:00
Jon Ross-PerkinsandDavid Blaikie a342e5c117 Add lexing for dump-sem-ir-start and end (#5357)
Syntax rationale is on `DumpSemIRRange` to try and record this, since
I'm not sure this belongs in the language design. The intent of this is
to be able to subset SemIR, which will be done separately in the
formatter.

---------

Co-authored-by: David Blaikie <dblaikie@gmail.com>
2025-04-24 22:17:04 +00:00
Dana Jansensandzygoloid 24bde46181 Change array syntax from [T; N] to array(T, N) (#4981)
In line with the proposal in #4682, this changes the array syntax to be
array(T, N). `array` is a builtin keyword which must be followed by
parens containing two expressions and a separating comma.

The array type expression is still fully builtin, it does not forward to
a Core.Array library type yet. It merely adds the `ArrayType`
instruction, as was done with the previous syntax.

Followup work will change the instruction to reference to Core.Array,
once the library type exists and can be used directly.

---------

Co-authored-by: zygoloid <richard@metafoo.co.uk>
2025-02-20 22:42:47 +00:00
Jon Ross-Perkins b06fcc97f6 Clean up a few details of lex yaml printing (#4845)
- Escape dumped token strings (what got me here)
- Change the quoting from backticks to quotes
- Also add a `FormatEscaped` helper function for this, updating other
`.write_escaped` uses
2025-01-24 21:52:02 +00:00
Jon Ross-Perkins aec951d7c3 Fix autoupdate handling of carriage return (#4840)
Switching from RE2 to StrReplaceAll because it seems a fair fit for what
actually needs to be done here. Also pick up \t for visibility reasons.

This came up because clangd's LSP-related APIs print carriage returns.
2025-01-24 16:41:57 +00:00
David BlaikieandJon Ross-Perkins e6c1f0630a Add a newline after diagnostic output when testing (#4818)
This removes some churn when adding new diagnostic cases to test files
(where previous to this change the newly added newline would cause the
previous diagnostic CHECKs to be updated including changes to the line
number because the CHECK for the blank line meant an extra line between
CHECK and source line).

A few alternatives discussed here:
https://discord.com/channels/655572317891461132/655578254970716160/1329573358475673723

---------

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
2025-01-16 23:23:57 +00:00
Richard Smith d41668350b Suppress testing SemIR in int builtin tests. (#4748)
Add `EXTRA-ARGS:` support to file_test, to add arguments without
overriding the default arguments. Use `EXTRA-ARGS: --no-dump-sem-ir` to
turn off SemIR dumping and thus SemIR testing in the int builtin tests,
which validate correct behavior through diagnostics instead.

This doesn't get us any closer to supporting more targeted SemIR dumping
/ testing, but this seems to be a generally useful feature anyway. Most
existing
tests using `ARGS` have been switched over to using `EXTRA-ARGS`.

Requested in review of #4716.
2024-12-31 00:35:39 +00:00
Richard Smith 28602a87c2 Fix handling of repeated tuple indexing. (#4733)
Per [the
design](https://docs.carbon-lang.dev/docs/design/lexical_conventions/),
`x.1.2` should lex as `(x.1).2`, not as `x.(1.2)`.
2024-12-21 06:57:14 +00:00
Jon Ross-Perkins fa95892a37 Add diagnostic coverage, remove possibly-unreachable unary op diagnostic (#4519)
I'm working to make sure remaining diagnostics have coverage, at least
the ones I'd previously added a TODO for. Note in particular that I
couldn't figure out a repro for UnaryOperatorRequiresWhitespace; if you
have one, I can add a test, but otherwise maybe it's actually
unreachable due to being diagnosed through infix logic (or, maybe
this'll let fuzzing tell me an example).
2024-11-13 18:04:25 +00:00
Jon Ross-PerkinsandGeoff Romer 06f4eec91e Modify lex yaml output to elide FileStart/End in tests. (#4433)
Trying to make split file tests of lex functionality shorter and easier
to read. numeric_literals.carbon in particular has an example of why I'm
interested in this (at the bottom). This also switches from `[]` list
format to `-` list format so that the trailing `]` is removed.

Trimming comments in tokenized_buffer.h because (1) it feels like it's
giving too much detail about what's printed, which has drifted slightly
and (2) it also feels like it's trying to justify YAML output, when
that's just what we're doing in general.

---------

Co-authored-by: Geoff Romer <gromer@google.com>
2024-10-23 18:56:41 +00:00
Richard Smith 2e63da1a40 Move diagnostic kind name to the end of the diagnostic. (#4437)
Also surround it in square brackets rather than parentheses. This
matches the format used by Clang and GCC, and means diagnostics will
still match the `file:line:col: error: ` pattern used by some IDE tools.

Before:
```console
fail_builtins.carbon:11:11: error(AliasRequiresNameRef): alias initializer must be a name reference
```

After:
```console
fail_builtins.carbon:11:11: error: alias initializer must be a name reference [AliasRequiresNameRef]
```

Also tighten up test regex to only match on `STDERR` lines that list a
file name.
2024-10-23 16:56:23 +00:00
Jon Ross-Perkins 9fefef162f Add tests to catch untested diagnostics. (#4426)
Use the diagnostic kind printing in #4425 to catch when we have
diagnostics with no tests.

This merges a couple other use-cases of filegroup manifests into a
common rule.

Note I do add a few tests for things, and also some things are
_actually_ unit tested (just not in the file_test structure). But I
stopped when I realized that dealing with merge conflicts is going to be
a pain. I might end up reverting test changes (as part of merge conflict
resolution) and doing narrow test additions in a separate PR, after both
this and #4425 are merged.
2024-10-21 20:16:18 +00:00
Jon Ross-Perkins 62c36eceda Support printing the diagnostic kind for verification. (#4425)
This is to help identify which diagnostics we're actually using.

Note that driver/testdata still has tests which don't pass this flag,
and so continue to test the kind-less (default) behavior.
2024-10-18 22:33:56 +00:00
Jon Ross-PerkinsandRichard Smith e7aebbe581 Update basic diagnostic capitalization/punctuation (#4328)
This is a primarily automated change:

- Search & replace for capitalization
-
`(CARBON_DIAGNOSTIC\((?:\n\s+)?\w+,(?:\n\s+)?\s\w+,(?:\n\s+)?\s")([A-Z])`
    - `$1\L$2`
- Search & replace for period
-
`(CARBON_DIAGNOSTIC\((?:\n\s+)?\w+,(?:\n\s+)?\s\w+,(?:\n\s+)?\s"(?:[^)]|\n)+)\.("[,)])`
    - `$1$2`
- Limited search & replace for `ERROR: ` -> `error: ` in streamed things
- Leaving a TODO for command_line because there's more cleanup that can
be done there
- Modify diagnostic_consumer.cpp
    - ERROR -> error
    - WARNING -> warning

---------

Co-authored-by: Richard Smith <richard@metafoo.co.uk>
2024-09-19 21:32:53 +00:00
c43fa3a8a5 Bit-pack the lexer's token info (#4270)
This makes each token info consist of 8 bytes of data:
- 1 byte of the kind
- 1 bit for whitespace tracking
- 23 bits of payload
- 32 bits for byte offset in the file

This builds directly on representing the location of the token as
a single 32-bit offset, now compressing the rest of the data into
a single 32-bit bitfield.

This adds some implementation limits: we can no longer lex more than
2^23 tokens in a single source file. Nor can we have more than 2^23
string literals, integer literals, real literals, or identifiers. Only
the first of these is even close to an issue, and even then seems
unlikely to ever be a problem in practice.

The memory efficiency here is great and the motivating goal. But to make
this work well, we also need to streamline how we create the tokens.
Otherwise, all the bit fiddling can end up erasing our gains. This PR
adds a number of APIs to manage creating and accessing the now
significantly more complex storage of token infos to try and help with
this.

One big change required to simplify the writes here is to switch from
computing whether a token has trailing space after-the-fact to
pre-computing whether a token will have leading space. That lets us have
the leading space information available immediately when forming the
token, and avoids doing a single bit flip afterward.

Another change that helps with this representation is to minimize the
updating of groups after-the-fact. The code now tries to set the opening
index directly when creating the closing token and only updates the
opening group afterward. Because of the bit packing, this is a reduction
of 0.5% of dynamic instructions in the compile benchmark, and has
dramatic improvements for the grouping symbol focused benchmarks.

All combined, this is a significant improvement on the lexer-focused
benchmarks despite the added complexity, and a significant win on our
compile time benchmarks due to both the lexer improvements and
downstream memory density improvements: 5-12% reduction in lex time,
growing larger as files get larger. About a 4.5% reduction in parse
time, and even a 1-2% reduction in total check time. =D

---------

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Geoff Romer <gromer@google.com>
2024-09-06 16:22:36 +00:00
Jon Ross-Perkins 83413479d7 Move some of the test information to TIP lines (#4007)
This has is a nice-to-have for me. Frequently I want to run a specific
test, and end up digging through output to be able to copy-paste the run
line. This uses TIP lines to inject the command into the file when using
AUTOUPDATE.

Note, one of the reasons I want this is because "bazel test
//toolchain/testing:file_test --test_output=all" has been regularly
exceeding bazel's output limit for me (workaround is either opening the
output file or specifying an obscure output limit flag), making it a
little harder for me to get the commands. However, frequently I'm adding
a file and want to iterate on it, so that's really the use case I have
in mind here.
2024-06-04 17:57:23 +00:00
Jon Ross-Perkins ec300ef3ad Restructure lex keyword tests to reduce duplication. (#3941)
Rather than having a file listing all keywords, do it programmatically.
Conversely, move the limited keyword tests out to a file test.
2024-05-08 16:45:57 +00:00
Richard Smith 45c071f2af Rework raw identifier lexing to avoid slowing down regular identifiers. (#3855)
Instead of special-casing tokens starting with `r`, lex them as normal
identifiers, and add a special case to `#` handling to detect if the
previous token was an `r` identifier.

This roughly doubles the time to lex a raw identifier, because we do two
hash table insertions rather than one, and probably slightly slows down
lexing string literals starting with `#`, but should remove the 2%
overhead to identifier lexing from the previous approach.
2024-04-03 23:42:28 +00:00
Jon Ross-Perkins b8ceb8dd8b Print a blank line after a diagnostic. (#3806)
The purpose of the newline is to make it clearer where a given
diagnostic begins and ends, particularly as the first message of a
diagnostic may not be the error.

This is a trivial code change, but ripples edits through test files.
2024-03-22 18:10:49 +00:00
Jon Ross-Perkins 551a6d385e Augment the file_test framework to allow per-file fail checks. (#3747)
This handles toolchain failures per-file. The intent is to allow placing
both "success" and "fail" tests in the same file, using splits. However,
this PR only adds support and updates existing tests to continue
passing.
2024-03-08 20:28:25 +00:00
Richard Smith 0a06fceb5f Improve diagnosis of mismatched brackets. (#3282)
Move handling of mismatched brackets out of the main lexing loop into a
separate pass that is only run if there are mismatched brackets This is
done in preparation for using both lookahead and lookbehind to work out
how to match brackets, and to get this code far away from the hot lexing
loop.

Fix bracket insertion location to be immediately after the token that
we're inserting the bracket after, rather than potentially at the end of
a comment. When there are open brackets at the end of the file, say that
there are open brackets, not that there's a closing bracket without a
matching opening bracket.
2023-12-21 08:49:37 +00:00
josh11bandJon Ross-Perkins fada410559 Support declaration modifier keywords (#3412)
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
2023-12-05 22:45:57 +00:00
Jon Ross-Perkins 0db63ff17a Abbreviate Integer and FloatingPoint (#3435)
I was suggesting this because `FloatingPoint` is pretty long. `int` and
`float` should be familiar abbreviations. `unsigned` should be familiar
to developers too, but `UnsignedInt` still feels usefully clearer for
the additional chars.
2023-11-29 23:29:48 +00:00
Jon Ross-Perkins 3f208e27f9 Align on FileStart/FileEnd for naming. (#3428)
The lexer has been using EndOfFile form (stemming from EOF), parser went
to FileEnd form. This consolidates on FileEnd form.
2023-11-29 16:36:57 +00:00
Jacob Schneider 482d233def fix crash caused by unicode chars (#3387)
Prevent a crash during lexing for unicode chars.
2023-11-11 06:27:15 +00:00
Jon Ross-Perkins d096655cc6 Split out the SharedValueStores to be per-compilation unit. (#3353)
Advantages:

- Allows lexing/parsing in parallel, since they are modifying fully
separate ValueStores.
- Allows SemIR to reliably be stored hermetically.

Disadvantages:

- Creates overlapping storage of duplicate strings when multiple files
are compiled together.
- Prevents Ids from being uniquely compared cross-file.

Per discussion, the decision is that the advantages are more important.

The looser ownership remains because both SemIR checking and
metaprogramming may still generate things we would want to deduplicate.
It would be somewhat odd if TokenizedBuffer owned something that
checking modified.
2023-11-01 19:44:17 +00:00
Jon Ross-Perkins 6742d0d048 Add partial raw identifier support. (#3344)
I'm looking at this due to the conversation on #3341. Although
diagnostics aren't where they should be, I thought it may help to start
adding raw identifier support (which may also help show how I was
thinking about this).

Note regarding the TODO on how to form the token, `GetTokenText` returns
the `string_id`'s reference value for an `Identifier`. So to make
`GetTokenText` work in a way that returns `r#foo` for a raw identifier,
I think there are a few options:

1. Add additional data indicating the end of the identifier.
2. Add `RawIdentifier` as a token kind to indicate that it's raw and
should be prefixed with `r#` (but also giving later stages one more
token kind to handle)
3. Make the `string_id` correspond to `r#foo`, and have later stages add
`foo` to the strings table whenever `r#foo` is encountered (with map
lookups leading to deduplication).
4. Add `StringId::RawKeyword` special values for each keyword.
- This would mean `self` prints as `self`, `r#self` prints as `r#self`,
but `r#foo` is not a keyword so prints as `foo`.
- This means keywords would need to be listed in a place `StringId` can
depend on them, one way or the other (e.g., a `keywords.def` file in
`base/` should work).
5. Say that it _is_ an `Identifier`, and if it's a keyword spelling, it
must have been a raw identifier.
- Same limitation as above: This would mean `self` prints as `self`,
`r#self` prints as `r#self`, but `r#foo` is not a keyword so prints as
`foo`.

I'm hoping to resolve this issue separately though. :)
2023-10-30 18:11:55 +00:00
Jon Ross-PerkinsandRichard Smith d13f76e001 Add value store to be shared across compile stages. (#3311)
This updates lexing to use the data. I'll do checking separately, just
to split changes.

Note the ValueStore structure is also set up such that SemIR::File can
use it for other fields.

---------

Co-authored-by: Richard Smith <richard@metafoo.co.uk>
2023-10-20 15:00:53 +00:00