This is a re-do of #7888
Outside a call, the grammar scoped a name as a type or a variable by its
capitalization. A theme could not distinguish a type from a value with
the same spelling, or a parameter, a member, or an implemented interface
from any other name. Pointer member access `->` was scoped as a result
arrow, which scoped `Lookup` in `regions->Lookup(x)` as a type.
`entity.name.type` now applies only in a type position (after `:`, `->`,
`as`, `adapt`, or `require`, or in an `impl`) and to the name a nominal
declaration declares. A qualifier, in a declared name like `fn N.Base(T:
type).F()` or in a type, is `entity.name.scope-resolution` nested in
`entity.name.type`. As in the parser, a name is a qualifier when its
parameter lists are followed by `.`. Parameters, `let` bindings, choice
alternatives, members, implemented interfaces, primitive types,
`import`, and the strings naming a builtin or a library have their own
scopes, and brackets use the C++ grammar's scopes. A `->` written
directly after its operand is member access; no result arrow in the
repository is written that way.
The sample palette renders each scope that depends on context
differently, and `tmlanguage.py` now supports nested captures.
In 17 of the 19 themes measured, the VS Code built-ins plus One Dark
Pro, Night Owl, Tokyo Night, and Nord, 13% to 27% of the characters in
hand-written Carbon change color or style. A theme loses a color where
`.` is now punctuation, where word operators like `as` are no longer
control-flow keywords, or where a name outside a type position is no
longer a type.
Assisted-by: Claude Code
The sample renderings used VS Code's Dark+, which renders many scopes
identically, so a grammar change often doesn't show up in the images.
Dark+ gives the 42 scope stacks the samples produce 10 distinct
renderings: types, namespaces, builtin types and classes share one
color, and operators and separators stay at the default foreground, so a
rule that moved a token between them changed nothing visible.
`render_sample.py` now uses its own palette. Each family of related
scopes, such as the numeric literals, gets one color, and bold and
italic distinguish the scopes within a family; the renderer now emits
font styles to support this. The samples' 34 scopes get 32 renderings;
only the two embedded C++ spellings and a string's opening and closing
quotes share one. Unscoped text is dimmer than any scope's color, so
`0o8` renders as an octal `0o` followed by an unscoped `8`. The colors
are Catppuccin Mocha's, and each has at least 4.4:1 contrast against the
background.
No existing theme does this: across 264 theme files, the 19 VS Code
built-ins plus 30 popular extensions, the best renders 19 of the 42
stacks distinctly.
Assisted-by: Claude Code
Outside a call, the grammar scoped a name as a type or a variable by its
capitalization. A theme could not distinguish a type from a value with
the same
spelling, or a parameter, a member, or an implemented interface from any
other
name. Pointer member access `->` was scoped as a result arrow, which
scoped
`Lookup` in `regions->Lookup(x)` as a type.
`entity.name.type` now applies only in a type position (after `:`, `->`,
`as`,
`adapt`, or `require`, or in an `impl`) and to the name a nominal
declaration
declares. A qualifier, in a declared name like `fn N.Base(T: type).F()`
or in a
type, is `entity.name.scope-resolution` nested in `entity.name.type`. As
in the
parser, a name is a qualifier when its parameter lists are followed by
`.`.
Parameters, `let` bindings, choice alternatives, members, implemented
interfaces, primitive types, `import`, and the strings naming a builtin
or a
library have their own scopes, and brackets use the C++ grammar's
scopes. A
`->` written directly after its operand is member access; no result
arrow in
the repository is written that way.
The sample palette renders each scope that depends on context
differently, and
`tmlanguage.py` now supports nested captures.
In 17 of the 19 themes measured, the VS Code built-ins plus One Dark
Pro, Night
Owl, Tokyo Night, and Nord, 13% to 27% of the characters in hand-written
Carbon
change color or style. A theme loses a color where `.` is now
punctuation, where
word operators like `as` are no longer control-flow keywords, or where a
name
outside a type position is no longer a type.
Assisted-by: Claude Code
Four regions had `end` patterns that could fail to match, so `return
var;`, a `fn` with no parameter list, and an unterminated `"` each
swallowed the rest of the file; 77 of 1696 testdata files lost their
highlighting partway through. Operators were wrapped in `\b`, which only
holds next to a word character, so `a + b` highlighted nothing. And the
keywords had drifted about two years behind the lexer.
Identifiers are now classified by naming convention plus a call-site
lookahead, the way the Rust grammar does it, so nothing carries between
lines. Regions survive only for strings and embedded C++, where a
terminator reliably turns up, and raw strings spell out hash levels 0
through 2, so `\n` is an escape in `"..."` and plain text in `#"..."#`.
Trailing comments, character literals, raw identifiers, `$0`, `0o`
octal, arbitrary integer widths and six missing keywords are covered
now, `destructor` is gone, and `i32` reads as a type rather than as a
keyword.
Highlighting stays forgiving rather than diagnostic: anything after `//`
is a comment and odd numeric spellings still read as numbers. Pointing
out mistakes is the toolchain's job, and lenient rules hold steady while
you are still typing.
Every keyword and symbol in `token_kind.def` is covered, and unscoped
tokens across examples and the toolchain drop from 39% to 21%.
Note that I haven't tried to read and reason about every minute change
here as there are just too many. But I'm working on a follow-up PR that
adds testing that should be significantly easier te review.
Assisted-by: Claude Code
The images beside the samples were screenshots taken by hand, so they
drifted: they still showed `package Carbon api;`, syntax the language
dropped in 2024.
`render_sample.py` renders a sample the way the grammar in this
repository actually highlights it, using `tmlanguage.py`, a small
TextMate tokenizer. VS Code runs grammars under Oniguruma, which we
cannot depend on here, but this grammar uses no Oniguruma-only syntax,
so `re` runs its regexes unchanged and the two agree on every character
of every Carbon file in the repository. The output is SVG, so
regenerating needs nothing but Python, and a later grammar change gets a
same-path image diff showing what it did to real code.
The samples change where a construct left the language (`api`, `Carbon`
as the package name, `StringView`, `destructor`), refresh the
`keywords.carbon` inventory against `token_kind.def`, and add sections
for octal, raw and block literals, character literals, lambdas, and raw
identifiers. `interop.carbon` is new, covering inline C++. These are
highlighting fixtures rather than programs, so the deliberately invalid
lines stay.
Assisted-by: Claude Code
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>