Four regions had `end` patterns that could fail to match, so `return
var;`, a `fn` with no parameter list, and an unterminated `"` each
swallowed the rest of the file; 77 of 1696 testdata files lost their
highlighting partway through. Operators were wrapped in `\b`, which only
holds next to a word character, so `a + b` highlighted nothing. And the
keywords had drifted about two years behind the lexer.
Identifiers are now classified by naming convention plus a call-site
lookahead, the way the Rust grammar does it, so nothing carries between
lines. Regions survive only for strings and embedded C++, where a
terminator reliably turns up, and raw strings spell out hash levels 0
through 2, so `\n` is an escape in `"..."` and plain text in `#"..."#`.
Trailing comments, character literals, raw identifiers, `$0`, `0o`
octal, arbitrary integer widths and six missing keywords are covered
now, `destructor` is gone, and `i32` reads as a type rather than as a
keyword.
Highlighting stays forgiving rather than diagnostic: anything after `//`
is a comment and odd numeric spellings still read as numbers. Pointing
out mistakes is the toolchain's job, and lenient rules hold steady while
you are still typing.
Every keyword and symbol in `token_kind.def` is covered, and unscoped
tokens across examples and the toolchain drop from 39% to 21%.
Note that I haven't tried to read and reason about every minute change
here as there are just too many. But I'm working on a follow-up PR that
adds testing that should be significantly easier te review.
Assisted-by: Claude Code
The images beside the samples were screenshots taken by hand, so they
drifted: they still showed `package Carbon api;`, syntax the language
dropped in 2024.
`render_sample.py` renders a sample the way the grammar in this
repository actually highlights it, using `tmlanguage.py`, a small
TextMate tokenizer. VS Code runs grammars under Oniguruma, which we
cannot depend on here, but this grammar uses no Oniguruma-only syntax,
so `re` runs its regexes unchanged and the two agree on every character
of every Carbon file in the repository. The output is SVG, so
regenerating needs nothing but Python, and a later grammar change gets a
same-path image diff showing what it did to real code.
The samples change where a construct left the language (`api`, `Carbon`
as the package name, `StringView`, `destructor`), refresh the
`keywords.carbon` inventory against `token_kind.def`, and add sections
for octal, raw and block literals, character literals, lambdas, and raw
identifiers. `interop.carbon` is new, covering inline C++. These are
highlighting fixtures rather than programs, so the deliberately invalid
lines stay.
Assisted-by: Claude Code
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
Chandler Carruth
