Four regions had `end` patterns that could fail to match, so `return
var;`, a `fn` with no parameter list, and an unterminated `"` each
swallowed the rest of the file; 77 of 1696 testdata files lost their
highlighting partway through. Operators were wrapped in `\b`, which only
holds next to a word character, so `a + b` highlighted nothing. And the
keywords had drifted about two years behind the lexer.
Identifiers are now classified by naming convention plus a call-site
lookahead, the way the Rust grammar does it, so nothing carries between
lines. Regions survive only for strings and embedded C++, where a
terminator reliably turns up, and raw strings spell out hash levels 0
through 2, so `\n` is an escape in `"..."` and plain text in `#"..."#`.
Trailing comments, character literals, raw identifiers, `$0`, `0o`
octal, arbitrary integer widths and six missing keywords are covered
now, `destructor` is gone, and `i32` reads as a type rather than as a
keyword.
Highlighting stays forgiving rather than diagnostic: anything after `//`
is a comment and odd numeric spellings still read as numbers. Pointing
out mistakes is the toolchain's job, and lenient rules hold steady while
you are still typing.
Every keyword and symbol in `token_kind.def` is covered, and unscoped
tokens across examples and the toolchain drop from 39% to 21%.
Note that I haven't tried to read and reason about every minute change
here as there are just too many. But I'm working on a follow-up PR that
adds testing that should be significantly easier te review.
Assisted-by: Claude Code
Setting "type" to bright red to make comparison obvious:
(screenshots taken from VSCode)
Before
<img width="446" height="150" alt="Screenshot 2026-07-20 135558"
src="https://github.com/user-attachments/assets/81249005-99f1-47bb-af4f-ad6e8f2404fb"
/>
After
<img width="429" height="149" alt="Screenshot 2026-07-20 135738"
src="https://github.com/user-attachments/assets/acc83086-fe19-4a18-a833-968daed56229"
/>
I'm not super familiar with textmate/highlightjs, but I think what was
happening on line 5 is that the "type" scope is being extended to the
`=` after `.y`.
Adding `,` as an "end" token fixes this, and I think this is what
highlightjs already does.
I've also added "unused" as a keyword.
Implement the toolchain side of proposal #7254, removing the `:!`
binding
syntax for generic and template parameters in favor of the keywords
`generic`,
`template`, and `runtime` plus contextual defaults for phase.
For valid programs this is semantics-preserving: each binding resolves
to the
same phase, and produces the same SemIR, as it did under `:!`/`:`. The
parser
derives a binding's phase from its syntactic context plus any explicit
phase
keyword; new diagnostics and error recovery for misused keywords are
described
below.
Implementation details for each component:
- Lexer: remove the `:!` (`ColonExclaim`) token, move its virtual
parse-node
budget onto `:`, and add the `generic` and `runtime` keywords.
- Parser: thread a `BindingContext` (`ExplicitParam`, `DeducedParam`, or
`CompileTimeEntityParam`) from declaration introducers down through
parameter
lists to each binding pattern, using a one-token lookahead to
distinguish a
name-qualifier parameter list from a declaration's own final list.
Parameters
of a compile-time entity (`class`, `interface`, `constraint`, `choice`,
`alias`, `export`, `namespace`) and deduced `[]` parameters default to
checked
generic; explicit function parameters and local bindings default to
runtime.
`HandleBindingPattern` resolves the phase from that context plus the
keyword: a
`generic` keyword needs no node of its own (the phase is carried by the
binding's node kind), while a `runtime` keyword is preserved as a
`RuntimeBindingName` node so `check` can name it in a diagnostic. A
phase
keyword that is merely redundant with the contextual default is
diagnosed
here, without invalidating the parse tree.
- Check: a phase keyword that is invalid for its context (for example
`runtime`
on a checked-generic parameter) is diagnosed here, and recovers by
building an
error binding that still introduces the name so that later uses of it do
not
produce cascading errors.
The removed `:!` syntax is now rejected as an ordinary parse error.
The `form`/`:?`/`->?` ("extended types") portion of proposal #7254 is
left for a
separate change.
Assisted-by: Claude Code
TextMate doesn't support JSON grammars, so convert our JSON grammar to a
plist automatically as a pre-commit check. Fix malformed info.plist
file. Add missing uuid to grammar file.
Assisted-by: Gemini via Antigravity
Chandler Carruth
oli-ej
Richard Smith