Makes following changes to Carbon::Format()
- TokenKind::Period (i.e. `.`) should never have a space before or after
it.
- TokenKind::CloseSquareParen (i.e. `]`) should be treated as packed
content (no space preceeding it)
- Only exception I can think of is `impl forall [...]`
- Remove preceeding space from `[` and `(` if previous token was an
identifier (or identifier-ish token)
- Remove seperator following `++` / `--` unary operators.
- Explicit gaps in source code should be retained, up to 2 new lines.
Multiple test files were added to test formatting.
I imagine eventually this will need to be updated to read parse tree to
gather more context but this atleast lets us get a decent-ish format for
many of our current sample files (e.g. sieve.carbon)
Assisted-With: Gemini / Antigravity
---------
Co-authored-by: David Blaikie <dblaikie@gmail.com>
- Subdivide toolchain/driver:driver into granular subcommand targets
(compile_subcommand, format_subcommand, lld_subcommand, etc.) to enable
independent compilation and linking.
- Decouple Target and TargetMachine options from driver headers.
- Prune unused direct dependencies across toolchain BUILD files.
Assisted-by: Antigravity with Gemini
Implement the toolchain side of proposal #7254, removing the `:!`
binding
syntax for generic and template parameters in favor of the keywords
`generic`,
`template`, and `runtime` plus contextual defaults for phase.
For valid programs this is semantics-preserving: each binding resolves
to the
same phase, and produces the same SemIR, as it did under `:!`/`:`. The
parser
derives a binding's phase from its syntactic context plus any explicit
phase
keyword; new diagnostics and error recovery for misused keywords are
described
below.
Implementation details for each component:
- Lexer: remove the `:!` (`ColonExclaim`) token, move its virtual
parse-node
budget onto `:`, and add the `generic` and `runtime` keywords.
- Parser: thread a `BindingContext` (`ExplicitParam`, `DeducedParam`, or
`CompileTimeEntityParam`) from declaration introducers down through
parameter
lists to each binding pattern, using a one-token lookahead to
distinguish a
name-qualifier parameter list from a declaration's own final list.
Parameters
of a compile-time entity (`class`, `interface`, `constraint`, `choice`,
`alias`, `export`, `namespace`) and deduced `[]` parameters default to
checked
generic; explicit function parameters and local bindings default to
runtime.
`HandleBindingPattern` resolves the phase from that context plus the
keyword: a
`generic` keyword needs no node of its own (the phase is carried by the
binding's node kind), while a `runtime` keyword is preserved as a
`RuntimeBindingName` node so `check` can name it in a diagnostic. A
phase
keyword that is merely redundant with the contextual default is
diagnosed
here, without invalidating the parse tree.
- Check: a phase keyword that is invalid for its context (for example
`runtime`
on a checked-generic parameter) is diagnosed here, and recovers by
building an
error binding that still introduces the name so that later uses of it do
not
produce cascading errors.
The removed `:!` syntax is now rejected as an ordinary parse error.
The `form`/`:?`/`->?` ("extended types") portion of proposal #7254 is
left for a
separate change.
Assisted-by: Claude Code
Carbon currently requires a comment to be the only non-whitespace on its
line. A `//` comment that follows other content on a line, called a
_trailing comment_, is a lexer error. This proposal removes that
restriction, allowing a comment to follow other content on a line.
Everything else about comments is unchanged: a comment still begins with
`//`, still requires whitespace after the `//`, and still runs to the
end of the line. Carbon continues to provide only line comments; no
block or intra-line comments are added.
Three observations motivate the change. First, trailing comments are
well suited to short _annotations_ attached to a specific entity or
value on a line. Second, the lexer design now makes it trivial to lex
trailing comments, and in fact requires extra logic and potentially cost
to reject them. Third, C++ code routinely uses trailing comments, so
allowing them lets Carbon carry the layout of migrated code over
directly, rather than reworking each comment to read well in a different
structure.
Implementation notes (beyond the proposal's design):
Keeping trailing comments cheap to lex required a few supporting
changes, all of which keep the cost off the lexer's hot path:
- The lexer already dispatches `//` to comment lexing wherever it
appears, so classifying a comment as trailing is a single O(1) check of
whether the `//` is the line's first non-whitespace (`start + indent`).
The hot comment path is otherwise unchanged.
- That check relies on each line's recorded indentation being its real
leading whitespace. Multi-line string literals previously recorded the
column where the literal opened for the lines they span; they now record
the true (closing-delimiter) indentation instead.
- Parser error recovery (`SkipPastLikelyEnd`) had relied on that
opening-column indentation to keep tokens following a multi-line string
literal attached to the same construct. It now reconstructs that
relationship directly by consulting the line on which the literal
opened, including when other tokens follow the closing delimiter (such
as `''' + "more"`). This is on the cold recovery path.
- `CommentData` records the trailing bit in the high bit of its length
field, keeping it at 8 bytes.
Assisted-by: Claude Code
Rules executed by bazel don't necessarily have the right environment to
find the symbolizer, which was the intent of `cc_env` setting
`LLVM_SYMBOLIZER_PATH`. So far, this has kind of been a case-by-case
fix, but every so often I'm trying to debug a crash in a test that
doesn't provide it. Rather continuing down this route, instead add
drop-in wrappers for cc rules so that it's hard to forget.
Note `bazel/cc_rules` is intended to mirror `bazel/carbon_rules` and
`bazel/cc_toolchains`, rather than `@rules_cc`.
AFAICT there isn't a great way to add this as a default for the `bazel
run` environment. It's not typically going to be set on its own,
forwarding `$PATH` would be too broad, and the [action
`env_sets`](https://bazel.build/docs/cc-toolchain-config-reference#using-action-config)
I think are not quite what we need (I think those don't include output
execution, only compilation).
This switches most error printing to use diagnostics instead of direct
stream writes, even when not a specific file diagnostic. I'm allowing
empty filenames for this use-case.
This allows a little more specific testing to validate coverage of
output using the diagnostic coverage test. I'm adding a few tests to
cover things that weren't previously tested.
Separately, this also forces a little more standardization in format...
considering how changes like #4568 show effort being spent to _mirror_
diagnostic style, my thought is now to just use diagnostic code where
possible.
Note this also allows incrementally better testing of the language
server; I'm changing the crash fix from #4847 in favor of diagnostic
testing.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
This removes some churn when adding new diagnostic cases to test files
(where previous to this change the newly added newline would cause the
previous diagnostic CHECKs to be updated including changes to the line
number because the CHECK for the blank line meant an extra line between
CHECK and source line).
A few alternatives discussed here:
https://discord.com/channels/655572317891461132/655578254970716160/1329573358475673723
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Here I'm trying to add some simple formatting based on the token kind,
aiming mainly to keep the implementation short for now.
This approach won't generalize to arbitrary structures (e.g., it doesn't
discern between braces for a function body and a struct literal). I
think we'll probably want to build a parse tree and associate parse
kinds with tokens in order to format, additionally doing something less
linear. But my essential goal at present is to just get a
proof-of-concept that the basics can yield something that looks okay.
This is improving the comment production to produce fewer distinct
comments.
At present, comment processing uses strict prefix matching. It either
expects `// ` (with a space) for valid comments, or just `//` (without a
space) for invalid comments that lacked the space.
As a consequence, the following would be three comments:
```
// Comment 1
//
//
// Comment 4
```
This is because a 3-character prefix is used for valid comments. The
prefix switches between lines 1 and 2, and again between lines 3 and 4,
each resulting in a separate comment.
For contrast, this is one comment because only a 2-character prefix is
used:
```
//Comment 1
//
//
//Comment 4
```
That's because all lines lack a suffix space.
Additionally, with SIMD 16-byte boundaries, further splits can occur if
processing needs to transition to non-SIMD.
Here, I'm trying to just address all of this by:
1. Stitching together adjacent comments. Since a lexed comment starts at
the `//` excluding the indent, the delta from the prior comment must be
precisely the indent.
2. Adding support for switching from SIMD to non-SIMD on file
boundaries.
I considered trying to have a separate `//\n` prefix for SIMD processing
of `// `, but I wasn't sure about the tradeoff of doing both at the same
time (in particular, it'd require constructing a string for the
different prefix), thus this stitch approach. This does mean multiple
passes will be required for a typical long comment structure using blank
comment lines to separate paragraphs (for performance reasons, I will
recommend engineers not write comm... nevermind).
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
In order to format comments, it's helpful if they're tracked. This
tracks them separately from tokens in order to avoid interfering with
parse; it'd be inconvenient if comment tokens could show up in arbitrary
locations, albeit possible to support.
This additionally extracts out the TokenIterator support into a template
in order to generally have it available for IndexBase types. I'm only
adding it for CommentInfo, not sure if we'll want it elsewhere, but this
structure still felt like a good fit.
Since formatting covers comments, and the CHECK lines are in comments,
it can create recursive behaviors. This introduces AUTOUPDATE-SPLIT as a
way of formally designating a split to exclusively be used for
autoupdate output.
This extracts out the SourceBuffer handling of `-` in order to trivially
share it.
Note this still has a number of TODOs, it's just setting up the
essential subcommand infrastructure, with some tests demonstrating that
it at least does something.
DavidLoftusandDavid Blaikie
Chandler Carruth
Jon Ross-Perkins
