The diagnostics documentation covers how a diagnostic is produced and
worded, not the form it takes on a terminal. This adds
`toolchain/docs/diagnostics_rendering.md`: what a diagnostic is made
of, its layout, color, character set, and width, and how each degrades
when the terminal can't render it.
A diagnostic is one message plus labels read against the code they
mark, `Primary` or `Info`; a context and a path are set out apart. It
renders as one frame, anchored per file, with no headline: the message
hangs off the range that is wrong, led by the level word on a heavy
mark, with `->` in the margin on the reported lines. Labels read in
the order of their ranges, connectors that cross earlier rows break
around them, and a path draws only for the message's own location.
Ten rows on the shared example, against Clang's six and rustc's eleven.
Nothing a diagnostic says is dropped for width: source is windowed,
words wrap past the level word, connectors slide, and a label that
still can't hang is out-dented. Under sixty columns the compact form
takes over, one located line per part. Color is named per element with
a ramp per level and a light-background palette; ASCII keeps every
distinction with `^`, `.`, and the level word. The `file:line:column:`
header is the road not taken: structure comes from the language
server, and the compact form covers a grep-able line.
Assisted-by: Claude Code
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This should handle over-long lines. I had tried to make the normalize
method work, but it doesn't seem promising and so let's at least enable
this version.
Assisted-by: Antigravity with Gemini
- Move formatted printing to `hashing.cpp` instead of `hashing.h`
- Separate APInt and APFloat hashing specializations into a new
`hashing_llvm.h`
- Update toolchain/base dependencies and include sites that hash LLVM
data types to include `hashing_llvm.h`
Combined, this reduces the transitive includes caused by `hashing.h`.
Assisted-by: Antigravity with Gemini
`-Wunused-template` was added to `-Wunused`, so clean up the things it
found. One of them was a bug in the Clang warning that I've worked
around and reported upstream:
https://github.com/llvm/llvm-project/issues/218429
`load_diagnostic_kind` matched `^\s+CARBON_DIAGNOSTIC_KIND` without
`re.MULTILINE` against a file whose entries start at column zero, so it
found nothing: `check_unused` was comparing an empty set of declarations
against every use and reporting nothing at all. The check has never run.
With it running, three kinds turn out to be registered and never
declared, and go: `BuildFailureRunningClangToLink`,
`BuildOutputFileOpenError`, and `BuildPreludeManifestError`.
`load_diagnostic_uses_in` looked only for `CARBON_DIAGNOSTIC`, so a
diagnostic declared with `CARBON_DIAGNOSTIC_ON_SCOPE` counted as unused,
and it read the two macros' own definitions in `diagnostic.h` as uses.
Both are why the kinds above could not simply be deleted before.
Assisted-by: Claude Code
---------
Co-authored-by: Geoff Romer <gromer@google.com>
Everything drawn into a buffer was checked against `columns()`. That is
right for wrapping and for line drawing, both of which have somewhere
else to put what doesn't fit, but wrong for `DrawText`, which exists for
text that must not be broken and sometimes has to run past the width
with no other answer available. It and `DrawCodePoint` now check only
that the column is non-negative and the row is one a grid can index, and
widen the buffer as far as the text needs; `DrawWrappedText`,
`DrawHorizontalLine`, `DrawVerticalLine`, and `DrawBox` are unchanged.
That also settles what a caller does after a drawing overhangs, since
`DrawEnd` exists so that a run can continue where the last one ended,
and that continuation was itself a checked error whenever the previous
run overhung. A column computed to be negative, such as a gutter
narrower than the line number it holds, still fails.
A color picked to read against black is hard to read against white, and
nothing in `Capabilities` said which a stream was going into.
`ChooseBackground` reads `COLORFGBG`, which `rxvt` and its derivatives
set to the foreground and background palette indices, and takes anything
it doesn't answer to be dark: guessing dark costs contrast, while
guessing light puts pale text on a pale background. Asking the terminal
itself with an `OSC 11` query is the accurate answer, and needs raw
mode, a timeout, and somewhere to put the reply, so there is a TODO for
it rather than an implementation.
Every corner, tee, and crossing came out of `Charset::Ascii` as `+`,
which left six of the shapes a diagnostic draws indistinguishable: the
rule closing a frame read as the one separating two snippets, and the
anchor opening a diagnostic as the one carrying it on. Each stand-in now
keeps the axis its line runs through, which leaves `+` meaning a
crossing and nothing else. A tee keeps its through-stroke and leaves the
branch to what is drawn beside it, and a corner is `.` where its line
leaves downward and `'` where it arrives from above, which is where
those characters sit in their cells. A box is a box again:
```
+--+ .--.
| | -> | |
+--+ '--'
```
Assisted-by: Claude Code
Building a `for` loop looks `Core.Iterate` up twice: once for
`NewCursor` to make the cursor, and again for `Next` to advance it. A
range that implements neither failed both lookups and reported both, so
a loop over something that isn't iterable produced two errors saying the
same thing about the same expression.
The second lookup is skipped when the first already failed, which is
what `BuildBinaryOperator`'s `diagnose` parameter is for. The
`ErrorInst` it returns instead does not reach the produced SemIR: the
loop is abandoned on the error either way.
Assisted-by: Claude Code
---------
Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
- Subdivide toolchain/driver:driver into granular subcommand targets
(compile_subcommand, format_subcommand, lld_subcommand, etc.) to enable
independent compilation and linking.
- Decouple Target and TargetMachine options from driver headers.
- Prune unused direct dependencies across toolchain BUILD files.
Assisted-by: Antigravity with Gemini
Rich diagnostic rendering needs a layer underneath it that knows what
the attached terminal can do and can position styled text in two
dimensions. This adds that layer, both as the foundation the diagnostics
rendering work will build on and as something usable directly for
ordinary CLI output. Nothing depends on it yet, so it lands and is
reviewed on its own.
Four libraries, each with its own tests:
- `color`: a color, either one of the 16 named ANSI colors or a 24-bit
RGB value, and the escape sequences that select it at a given color
depth.
- `style`: colors plus text attributes, and the escapes that move a
terminal from one style to another.
- `capabilities`: what the terminal behind a stream supports, detected
from the environment.
- `buffer`: a grid of styled cells that layout code draws into and that
renders itself once.
Rationale for the design decisions lives in the headers, next to what it
explains. Four things are worth review attention in particular:
- The color detection precedence documented on `ChooseColorMode`. It
settles how a `--color` flag, `NO_COLOR`, `CLICOLOR`, `FORCE_COLOR`, and
the terminal itself interact. The policy is a pure function of those
inputs, so the whole table is tested without touching the process
environment.
- `Charset`, which decides whether any UTF-8 processing happens at all.
Column counts only follow from code points if the terminal agrees about
the encoding, so anything short of a locale naming UTF-8 is treated as
bytes.
- `Buffer` owning column accounting instead of its callers, which is
what keeps double-width characters, combining marks, and stray bytes
from misaligning everything after them.
- The API surface, which is held to operations that nothing else covers.
Junctions in line art come only from lines overlapping, and turning a
style on or off is spelled as a transition to or from the default style.
`terminal_benchmark` covers style transitions, full-screen rendering,
and text drawing. On an M-series laptop, rendering an 80x24 screen in
which every cell changes style costs about 14us with color off and
76-95us with it, and drawing a 40-column line of source costs about
140ns without UTF-8 processing and 394ns with it.
Assisted-by: Gemini and Claude
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
A defaulted `operator==` or `operator<=>` compares every base class
subobject, so children of `Printable` couldn't default their
comparisons: `Printable` had no comparison operators of its own, which
made the defaulted operator implicitly deleted. Children that want
member-wise comparison had to write it out by hand instead.
`Printable` is empty, so it now provides comparisons that always compare
equal. Its operands are constrained template parameter rather than
`const Printable&` so that they're only viable for comparing the base
class subobjects themselves. An overload taking `const Printable&` would
also be viable when comparing two `DerivedT` objects by converting them
to the base class, and would then both make children that provide no
comparison silently compare equal and displace the comparisons of
children that provide them through a conversion of their own, as
`EnumBase` does.
Some hand-written comparisons stay, for reasons unrelated to
`Printable`: using `= default` would change their meaning.
Adds `common/ostream_test.cpp`, which covers both the member and friend
forms of defaulting, the resulting comparison categories, and both of
the hazards above.
Assisted-by: Claude Code
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Change '1)' and '2)' to '(1)' and '(2)' to prevent the reflow engine
from treating them as ordered list markers when the paragraph is
wrapped.
Assisted-by: Antigravity with Gemini
- Enhance toolchain/parse/node_kind.def with category x-macros
(CARBON_PARSE_NODE_KIND_DECLARATION, CARBON_PARSE_NODE_KIND_EXPRESSION,
CARBON_PARSE_NODE_KIND_PATTERN, CARBON_PARSE_NODE_KIND_STATEMENT) that
default to CARBON_PARSE_NODE_KIND, keeping node_kind.def as the single
source of truth without manual expansion divergence.
- Decompose the monolithic toolchain/parse/extract.cpp translation unit
into separate compilation units for declarations, expressions, patterns,
and statements by expanding respective category x-macros.
- Forward-declare Tree and TokenizedBuffer in parse and lex dump headers
to reduce header inclusion depth.
Assisted-by: Antigravity with Gemini
- Consolidate token kind test assertions into helper functions using
EXPECT_THAT(spelling.str(), MatchesRegex(...)) under a single test case
rather than generating ~130 separate TEST classes and redundant matcher
instantiations.
- Slashes individual translation unit compilation time from 59.1s to ~3s
while keeping EXPECT_THAT for clear diagnostic error output and full
token coverage.
Assisted-by: Antigravity with Gemini
Replace the llvm::map_range based anonymous ranges returned by values()
and enumerate() with custom range and iterator types. This avoids
exposing complex template return types and improves encapsulation. But
most importantly, it is *much* cheaper to compile.
This change:
- Defines ValueStoreIterator, ValueStoreEnumerateIterator, and
ValueStoreRange templates in the Internal namespace.
- Exposes them via aliases in ValueStore: Iterator, ConstIterator,
Range, MutableRange, and EnumerateRange.
- Supports safe implicit conversion from Iterator to ConstIterator.
- Supports operator-> even when the value type is returned by value
(e.g., llvm::StringRef) by conditionally const-qualifying the pointer
type.
- Ensures C++20 comparison consistency with custom operator<=> and
operator== definitions.
- Restricts construction of these iterators and ranges to ValueStore
methods by using private constructors and friend declarations.
Assisted-by: Antigravity with Gemini
The `//@include-in-dumps` and `//@dump-sem-ir-begin`/`-end` tooling
directives were consumed for their side effects without a comment
record, so the tokens and comments together no longer reconstructed the
source: tooling that re-emits a file from them, such as `carbon format`,
silently dropped the directive lines. Now each recognized directive line
is also recorded as an ordinary full-line comment alongside its side
effect.
Adjacent full-line comments coalesce into one comment record only within
a category, determined by the byte after the `//` introducer: ordinary
comments (whitespace, or the end of the line or file), `//@...`
directives, and invalid introducers. A transition between categories
starts a new record, so a directive next to a comment block is its own
comment, while all the invalid spellings lump together to keep the
diagnostic noise at one per run.
The category boundary also fixes a lost directive: the invalid-comment
bulk skip compared only the `//` prefix, so `//!x` directly above
`//@dump-sem-ir-begin` absorbed the directive line and its side effect
was never recorded. Invalid comment runs now skip line by line (they
start from a diagnosed error, so they are not hot) and stop at a
whitespace or `@` introducer; the SIMD bulk skip handles only ordinary
comment blocks, whose prefix comparison already includes the whitespace
byte.
Nothing outside the lexer and the formatter reads comment records, and
lex dumps do not include comments, so no other behavior changes.
Assisted-by: Claude Code
Remove toolchain/parse/typed_nodes.h from transitive imports of tree.h
and node_stack.h to reduce compile-time overhead in toolchain/check.
Add an explicit include of typed_nodes.h to tree_and_subtrees.cpp where
it is needed for instantiating templates.
Assisted-by: Antigravity with Gemini
Remove clang/Frontend/CompilerInvocation.h from check_unit.h and forward
declare clang::CompilerInvocation instead to reduce header exposure.
Add an explicit include of CompilerInvocation.h to check_unit.cpp.
Assisted-by: Antigravity with Gemini
Remove clang/CodeGen/ModuleBuilder.h from toolchain/lower/file_context.h
to reduce transitive dependencies. Forward declare clang::CodeGenerator
and clang::FunctionDecl instead.
Add an explicit include of ModuleBuilder.h to context.cpp where the
complete types are required.
Assisted-by: Antigravity with Gemini
Add extern template declarations and explicit template instantiations
for ValueStore<InstId, Inst> and ValueStore<EntityNameId, EntityName> to
prevent redundant template instantiations.
Place them in their respective domain files (inst.h/cpp and
entity_name.h/cpp) to maintain modularity.
Assisted-by: Antigravity with Gemini
Document the phase of associated constants, which the discussions of
contextual phase defaults never covered: an associated constant is
always a checked generic binding. This is deliberately not presented as
a contextual default, because no other phase is possible for the
construct; correspondingly, no phase keyword (including `template`) is
allowed on one. Also rework the associated constants section of the
design README where the migration left a non-sequitur ("set to
compile-time values ... and so are defined using a `let` declaration"):
describe the `let` syntax and the binding's contextual phase separately.
Use consistent terminology for bindings versus constants. Bindings are
"checked" or "template" (generic) bindings, replacing the "symbolic
binding" and bare "generic binding" terms, so "symbolic" now only
describes constants and values. The "Symbolic facet bindings" section of
the generics details becomes "Checked facet bindings". The
expression-phase terms "symbolic constant" and "template constant" are
unchanged, and the binding-pattern definitions now name the constant
each kind binds. Template bindings are additionally described as
dependent and late checked, with each instantiation providing the
binding's value.
Also normalize the remaining "regular parameter" mentions to "runtime
parameter" to match the binding terminology, and describe the
compile-time "let template" as introducing a template generic binding
"T: C" whose uses are template constants.
Assisted-by: Claude Code
Implement the toolchain side of proposal #7254, removing the `:!`
binding
syntax for generic and template parameters in favor of the keywords
`generic`,
`template`, and `runtime` plus contextual defaults for phase.
For valid programs this is semantics-preserving: each binding resolves
to the
same phase, and produces the same SemIR, as it did under `:!`/`:`. The
parser
derives a binding's phase from its syntactic context plus any explicit
phase
keyword; new diagnostics and error recovery for misused keywords are
described
below.
Implementation details for each component:
- Lexer: remove the `:!` (`ColonExclaim`) token, move its virtual
parse-node
budget onto `:`, and add the `generic` and `runtime` keywords.
- Parser: thread a `BindingContext` (`ExplicitParam`, `DeducedParam`, or
`CompileTimeEntityParam`) from declaration introducers down through
parameter
lists to each binding pattern, using a one-token lookahead to
distinguish a
name-qualifier parameter list from a declaration's own final list.
Parameters
of a compile-time entity (`class`, `interface`, `constraint`, `choice`,
`alias`, `export`, `namespace`) and deduced `[]` parameters default to
checked
generic; explicit function parameters and local bindings default to
runtime.
`HandleBindingPattern` resolves the phase from that context plus the
keyword: a
`generic` keyword needs no node of its own (the phase is carried by the
binding's node kind), while a `runtime` keyword is preserved as a
`RuntimeBindingName` node so `check` can name it in a diagnostic. A
phase
keyword that is merely redundant with the contextual default is
diagnosed
here, without invalidating the parse tree.
- Check: a phase keyword that is invalid for its context (for example
`runtime`
on a checked-generic parameter) is diagnosed here, and recovers by
building an
error binding that still introduces the name so that later uses of it do
not
produce cascading errors.
The removed `:!` syntax is now rejected as an ordinary parse error.
The `form`/`:?`/`->?` ("extended types") portion of proposal #7254 is
left for a
separate change.
Assisted-by: Claude Code
The `style = "fixed"` is partially implied by `indent = 4`, but not
fully in some specific cases. There were also bugs in fully applying it
that are now fixed, and so we can specify it explicitly to get rumdl to
canonicalize nested list indentation more thoroughly.
Assisted-by: Antigravity with Gemini
This follows the pattern of our main compile benchmarks, but instead of
running the compile through a library API, it does so by running a
separate process. This lets us see the performance that we would expect
from `make` or another build system invoking the compiler, as opposed to
a minimal view from the library API.
Assisted-by: Claude Code
In `SourceGenTest.IdentifierByteSumStableAcrossSeeds`, the `first`
variable was storing `llvm::StringRef`s pointing to memory allocated by
a temporary `SourceGen` instance. This memory was freed at the end of
the loop iteration, leaving `first` with dangling references that were
accessed in subsequent iterations.
Fix this by storing `std::string` copies of the identifiers in `first`
to own the memory, and use `llvm::equal` to compare them.
Assisted-by: Antigravity with Gemini
`GetIdentifiersImpl` unconditionally sliced the 64-entry
`IdentifierLengthCounts` table even for uniform distributions, which the
API documents as having no `max_length` limit. Requesting a uniform
distribution with `max_length > 64` therefore tripped an out-of-bounds
slice assertion. Only compute the table slice on the non-uniform path,
where `max_length <= 64` is already enforced.
Add `IdentifierByteSumStableAcrossSeeds`, which exercises this path (a
uniform request up to length 200) and checks the core invariant that the
total identifier byte count is independent of the random seed across a
spread of parameters.
Assisted-by: Claude Code
---------
Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com>
`ErrorRecoveryBuffer::Apply` flagged each inserted token via the
pre-insertion index of the token it was inserted before. Once any
earlier insertion had been applied, that index no longer matches the
token's position in the merged list: with two closers inserted at the
same point (`{((}`), the second recovery token was left unflagged, and
with insertions at two separate points (`{(} {(}`), a real token was
flagged in the second one's place.
`IsRecoveryToken` now reports exactly the inserted tokens. This will
matter for `carbon format`: a recovery token's text exists in no source
byte range, so a minimal-edit model must know not to anchor edits on it.
Assisted-by: Claude Code
Carbon currently requires a comment to be the only non-whitespace on its
line. A `//` comment that follows other content on a line, called a
_trailing comment_, is a lexer error. This proposal removes that
restriction, allowing a comment to follow other content on a line.
Everything else about comments is unchanged: a comment still begins with
`//`, still requires whitespace after the `//`, and still runs to the
end of the line. Carbon continues to provide only line comments; no
block or intra-line comments are added.
Three observations motivate the change. First, trailing comments are
well suited to short _annotations_ attached to a specific entity or
value on a line. Second, the lexer design now makes it trivial to lex
trailing comments, and in fact requires extra logic and potentially cost
to reject them. Third, C++ code routinely uses trailing comments, so
allowing them lets Carbon carry the layout of migrated code over
directly, rather than reworking each comment to read well in a different
structure.
Implementation notes (beyond the proposal's design):
Keeping trailing comments cheap to lex required a few supporting
changes, all of which keep the cost off the lexer's hot path:
- The lexer already dispatches `//` to comment lexing wherever it
appears, so classifying a comment as trailing is a single O(1) check of
whether the `//` is the line's first non-whitespace (`start + indent`).
The hot comment path is otherwise unchanged.
- That check relies on each line's recorded indentation being its real
leading whitespace. Multi-line string literals previously recorded the
column where the literal opened for the lines they span; they now record
the true (closing-delimiter) indentation instead.
- Parser error recovery (`SkipPastLikelyEnd`) had relied on that
opening-column indentation to keep tokens following a multi-line string
literal attached to the same construct. It now reconstructs that
relationship directly by consulting the line on which the literal
opened, including when other tokens follow the closing delimiter (such
as `''' + "more"`). This is on the cold recovery path.
- `CommentData` records the trailing bit in the high bit of its length
field, keeping it at 8 bytes.
Assisted-by: Claude Code
This proposal removes the `:!` syntax for generics and templates in
favor of keywords (`generic`, `template`, `runtime`) and contextual
defaults for phase. It also replaces `:?` with `fwd` and introduces
`exttype` for extended types.
Assisted-by: Antigravity with Gemini, and Claude
---------
Co-authored-by: Geoff Romer <gromer@google.com>
A lambda introducer (`fn` in expression context) with no parameters and
no body -- for example `(fn)`, where `)` immediately follows `fn` --
left `HandleLambdaAfterParams` calling `ReturnErrorOnState()` without
ever emitting a `Lambda` node. The orphaned `LambdaIntroducer` leaf was
then left where an expression was required, so `Parse` produced a tree
that failed its own verification and aborted via `CARBON_FATAL`.
Recover the way `HandleLambdaBody` already does for a missing body after
a return type: emit a placeholder `InvalidParse` body and finish a
complete `Lambda` node, so the lambda stays a valid expression and the
surrounding construct (here a `ParenExpr`) extracts cleanly.
Found by fuzzing.
Assisted-by: Claude Code
`IsRawIdentifier` checked `token_text.starts_with("r#")` and then read
`token_text[2]`, but `starts_with` only guarantees a length of two. An
`r` identifier immediately followed by `#` at the end of the source --
so the token text is exactly `r#` -- made the `token_text[2]` read run
off the end. Guard on the length first.
Found by fuzzing. The read is reached only via `GetTokenText`, so the
parser fuzzer, which does not request token text, never hit it.
Assisted-by: Claude Code
Adds `toolchain/benchmarking/prelude_benchmark.cpp`, which measures the
time to compile the Core prelude and reports the most interesting SemIR
memory statistics as benchmark counters.
The prelude is compiled by checking a file that imports it: the implicit
prelude import causes the check phase to lex, parse, and check the full
set of prelude files, so this is a direct measure of prelude compilation
cost. Four input variations exercise increasing amounts of the prelude:
an empty file, a minimal single-type use, an operator-heavy file that
hits many impls, and a compact file that pulls in a wide swath of the
prelude.
Memory usage is queried directly: `Driver::set_mem_usage` takes a
`MemUsage` that a compile merges each file's usage into; the benchmark
passes one, compiles, and sums the entries by label. A compilation unit
collects into its own `MemUsage` whenever usage is dumped or a sink is
provided (decided in `SetMultiUnitCache`); after a file is done it dumps
that `MemUsage` per-file as before and, if a sink was provided, merges
into it via a new `MemUsage::Add(const MemUsage&)` overload. `MemUsage`
also exposes its entries via a public `Entry` type and an `entries()`
accessor.
Also extends `scripts/bench_runner.py` to (1) treat Mem-prefixed
counters as cost metrics (smaller is better) and (2) tolerate metrics
that aren't reported by every benchmark in a binary.
Assisted-by: Claude Code
---------
Co-authored-by: Christopher Di Bella <cjdb.ns@gmail.com>
Rumdl already appears to have _significantly_ fewer bugs than prettier,
and a solid LSP for editor integration.
The tool is: https://github.com/rvben/rumdl/
I've separated out the change across three commits for easier review.
The configuration tries to match the existing formatting, the changes to
the all the files are to correct issues found by the new tool.
Assisted-by: Antigravity with Gemini
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This is based on an idea suggested during review of the original type
erasure PR.
The type-erased check failure path lowers each check through
`CheckFailFormat<Ts...>`, which builds one format adapter per value,
down to the out-of-line `CheckFailImpl`, which takes an array of
base-class `format_adapter*` pointers. Previously a separate variadic
`CheckFailWithAdapters<Adapters...>` template sat in between: it existed
only to bind the adapter temporaries to named parameters so that
pointers to their base class could be collected into a `std::array` and
outlive the call to `CheckFailImpl`.
This removes that layer. `CheckFailFormat` now builds the pointer array
directly, in the braced-init-list of the `CheckFailImpl` call, using a
small `CheckFailFormatAdapterAddr<T>` helper to take the base-class
address of each adapter. The adapter temporaries are materialized as
named parameters of that helper within the same full-expression as the
`CheckFailImpl` call, so they remain alive across the call without a
dedicated function for that purpose.
The win is in debug info, not code. `CheckFailWithAdapters<Adapters...>`
was instantiated once per distinct adapter-type sequence in every
translation unit that uses `CARBON_CHECK`, and each instantiation
carried its own DWARF records -- type entries, string-table offsets, and
range lists. Dropping it removes those records from every such
translation unit. The generated machine code is unchanged.
Measured impact (fastbuild, the inputs to the inner-loop links):
- First-party object files shrink by 173,728 bytes (-0.233%) across the
`carbon-busybox` link inputs; 232 of 270 objects get smaller and none
grow. The largest reductions are in the check-densest translation units
(`type_completion`, `type`, `import_ref`, `function_context`,
`constant`, ...).
- Per `bloaty`, the reduction is almost entirely DWARF (`.debug_info`,
`.debug_str`, `.rela.debug_str_offsets`, `.debug_rnglists`); the loaded
code (VM size) is unchanged.
Assisted-by: Claude Code
---------
Co-authored-by: Geoff Romer <gromer@google.com>
Implements proposal #7016: `self` moves from the deduced implicit list
(`fn F[self: Self]()`) to the front of the explicit list. Its type may
be written explicitly (`fn F(self: Self)`) or omitted, in which case it
defaults to `Self` (`fn F(self)`, `fn F(ref self)`); `self` in the
implicit list is rejected.
Throughout checking, `self` is modeled as the first explicit parameter.
Because a method is just a function whose first parameter is `self`, it
can also be called as an ordinary function with the receiver passed
explicitly (`Type.M(obj, ...)`), not only as `obj.M(...)`. A new
`SemIR::CallArgParamPatterns` helper chooses the parameters matched
against the explicit arguments, excluding a leading `self` only when it
is supplied as a method-call receiver; arity checking, conversion, and
generic deduction use it. The resulting SemIR and lowering are
unchanged: `self` is still `call_param0`, and witnesses, thunks, and
vtables are unaffected.
An omitted `self` type is parsed as a `SelfBindingPattern` node with no
type expression; checking synthesizes the `Self` type so it behaves
exactly like `self: Self`. However, the exact spelling used must match
between a forward declaration and a definition, following #3763's rules
around declaration matching.
Generated functions, thunks, and C++ interop import/export build `self`
as the first explicit parameter, and the `self`-type override (e.g.
Derived->Base for a virtual override) applies to the explicit `self`.
Placement is validated by new diagnostics: `SelfInImplicitParamList`,
`SelfNotFirstParam`, and `SelfOutsideParamList`. The benchmark source
generator and the documentation adopt the `(self)` shorthand; the
prelude, the examples, and the test data are migrated in the following
commits.
Assisted-by: Claude Code with Claude Opus 4.7
---------
Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com>
`CARBON_CHECK` and `CARBON_FATAL` messages are formatted with
`llvm::formatv`. Previously each check site that had a message
instantiated its own copy of the formatv machinery -- a `formatv_object`
over a tuple of per-argument format adapters, plus that tuple -- in
every translation unit, keyed on the site's file, line, condition, and
format strings. A translation unit with many checks paid for that
machinery over and over.
This restructures check failure so that the formatting machinery is
compiled exactly once, and only a single small adapter is instantiated
per distinct value type per translation unit:
- `CheckFailImpl` (out-of-line) now takes the message's format string
and an array of already-type-erased `format_adapter`s, and renders the
whole failure message -- prefix plus the extra message -- directly into
one stream. The extra message is rendered in place, so no separate
string is ever materialized for it.
- `FormatvInto` (in the `.cpp`) renders a format string over that
adapter array. Rather than instantiate `llvm::formatv`, it drives the
formatv replacement loop over the public
`formatv_object_base::parseFormatString`, so this rendering code exists
exactly once. (A TODO notes that we should add a type-erased entry point
upstream in LLVM rather than reimplement the loop here.)
- The lowering from the macro down to that out-of-line call is split so
that the only per-check-site instantiation is trivial:
- `CheckFail<...>` is instantiated once per site, since its file, line,
condition, and format template-string parameters are unique to the site.
It just lowers those compile-time strings to ordinary arguments and
forwards to `CheckFailFormat`.
- `CheckFailFormat<Ts...>` is instantiated once per distinct value-type
sequence and shared across sites; it builds one type-erased adapter per
value.
- `CheckFailWithAdapters<Adapters...>` collects pointers to those
adapters into an array and calls `CheckFailImpl`. It is a distinct
function so the adapter temporaries stay alive while pointers to their
base class are in flight.
Format semantics, including runtime format-string validation, are
unchanged, and the rendered message is byte-for-byte identical.
For `DCHECK` in optimized builds the check is dead code; its arguments
are now routed through a trivial `IgnoreDeadCheckArgs` no-op rather than
`CheckFail`. This still type-checks the arguments so they cannot bitrot,
without instantiating any formatting machinery for them and without
provoking unused-variable warnings.
Measured full-rebuild impact (353 first-party translation units,
fastbuild): -112.6s CPU, -6.9% relative to trunk.
Assisted-by: Claude
These were defined inline in headers reached by most of the toolchain,
and as non-template functions their bodies -- including some very
expensive template instantiations -- are compiled in every including TU
even though they are only cold debug/dump paths:
- BundleStore's YAML output entry points instantiate IdAndKind::Dispatch
over all ~50 ID kinds (two ~365ms instantiations per TU) plus
Yaml::OutputScalar/std::function machinery; moved to a new bundle.cpp.
- TypeStore, ConstantValueStore, NameStoreWrapper, and SharedValueStores
OutputYaml bodies wrap capturing lambdas in std::function via
Yaml::OutputMapping; moved to their existing .cpp files.
This change appears to be worth another 10% compile time reduction.
Assisted-by: Claude
Fundamentally, this uses forward declarations of Clang types to reduce
the overall compile time cost of Clang headers across the codebase.
Tracing and profiling showed ~2s of every check TU's ~8-12s compile time
going just to parsing Clang frontend and AST headers pulled in via a few
sem_ir and check headers that only use the Clang types by pointer or
reference:
- sem_ir/cpp_file.h (reached via sem_ir/file.h by ~150 TUs) included
clang/Frontend/CompilerInstance.h, clang/CodeGen/ModuleBuilder.h,
clang/AST/Mangle.h, and llvm/IR/Module.h. CppFile's accessors move out
of line to a new cpp_file.cpp and the header now forward-declares the
Clang types.
- check/cpp/context.h (reached via check/context.h by ~100 TUs) included
clang/Frontend/FrontendAction.h and clang/Parse/Parser.h, pulling in
clang's Sema.h and ASTUnit.h.
- sem_ir/clang_decl.h included clang/AST/Decl.h; the three small
functions that need complete Clang types move out of line.
- sem_ir/cpp_overload_set.h included clang/Sema/Overload.h solely for
the three-field OverloadCandidateSet::OperatorRewriteInfo, which is now
mirrored as CppOverloadSet::OperatorRewriteInfo, and clang/AST/Decl.h
solely for a pointer.
- sem_ir/name_scope.h's clang/AST/DeclBase.h include was vestigial.
TUs (and more narrowly included headers) that genuinely use the Clang
definitions now include the Clang headers directly.
Representative compile times (fastbuild, aarch64), combined with the
preceding instantiation-cost changes, relative to trunk:
- check/eval.cpp: 11.85s -> 6.94s (-41%)
- check/handle_operator.cpp: 7.71s -> 3.30s (-57%)
- language_server.cpp: 6.68s -> 3.16s (-53%)
- lower/handle.cpp: 6.75s -> 3.66s (-46%)
- sem_ir/file.cpp: 8.60s -> 6.11s (-29%)
- driver.cpp: 6.68s -> 4.78s (-28%)
Measured full-rebuild impact (316 first-party TUs, fastbuild): -689.5s
CPU, -29.9% relative to trunk.
Assisted-by: Claude
This provides a native harness with several advantages over
`pre-commit`:
- Faster when initializing the cache
- Smaller cache sizes: under 43mb compared to over 62mb
- Better integration with `uv` for Python usage
Steps for migrating for existing contributors:
1. Install `prek` following instructions in the updated docs.
2. Replace hooks in an existing checkout with a special flag:
```sh
prek install --overwrite
```
The `--overwrite` flag is what removes the old hooks.
If you used the pre-push variant:
```sh
prek install --hook-type pre-push --overwrite
```
3. Optional cleanups:
```sh
rm -rf ~/.cache/pre-commit # reclaim the old hook-environment cache
pipx uninstall pre-commit # if installed via pipx; or `brew uninstall
pre-commit`
```
Assisted-by: Antigravity with Gemini
Not sure how these got missed when moving other things to `uv`, but this
should clean them up.
The bump to Python 3.12 is so that we can use `@override` with the
simple import from `typing`. This is needed by the newest versions of
`ty` to do type checking. Added the relevant `@override` annotations.
Assisted-by: Antigravity with Gemini
The `unused` modifier is rejected on parameters of a function
declaration, but the check only covered the explicit parameter list, so
an implicit parameter (such as self or a compile-time binding) could
carry unused without a definition. Check the implicit parameter list
too.
The code changes and the test updates are split into two commits for
easier review.
Assisted-by: Claude Code with Claude Opus 4.7
---------
Co-authored-by: Christopher Di Bella <cjdb.ns@gmail.com>
Carbon-side thunks (for example the `Copy`/`Destroy` witness thunks
generated for imported C++ types) are mangled by Carbon, and their names
incorporate a fingerprint of the involved types. The instruction
fingerprinter identifies a class only by its name and parent scope,
which is sufficient for Carbon classes but not for imported C++ classes:
different specializations of one class template (and other cases such as
types in anonymous namespaces) share a Carbon name and parent scope. As
a result, the thunks for two distinct specializations could mangle to
the same name, producing a single LLVM function with two definitions and
failing `verifyModule` during lowering.
When fingerprinting a class imported from C++, also include the Clang
mangled name of its type.
Test: toolchain/lower/testdata/interop/cpp/thunks.carbon gains a split
with two specializations of one class template, each requiring a thunk;
their thunks now get distinct mangled names instead of colliding.
Assisted-by: Claude Code
---------
Co-authored-by: Christopher Di Bella <cjdb.ns@gmail.com>
The `--dump-cpp-ast` file tests strip references to Clang builtins so
that the expected output is target-independent. The filter anchored a
`__`-prefixed builtin identifier on a preceding space or quote, which
matches the x86-64 `__va_list_tag` spelling but not the AArch64
`std::__va_list`, where `__` is preceded by the `::` namespace
qualifier. That left a single `RecordType 'std::__va_list'` line
unfiltered on AArch64, producing a spurious autoupdate diff for
`thunk_ast.carbon`.
Anchor the match on a preceding `:` as well so namespace-qualified
builtins are also filtered.
Carbon's test workflow covered Linux on x86-64 and macOS on AArch64, but
had no Linux AArch64 coverage, so AArch64-specific issues that don't
reproduce on macOS could land unnoticed. Add an `ubuntu-22.04-arm`
runner to the matrix.
The release used for Linux does not publish the monolithic
`LLVM-*-Linux-ARM64` package, only a `clang+llvm-*-aarch64-linux-gnu`
community build with a smaller tool set, so the Ubuntu setup now selects
the tarball by `runner.arch`. The prune step uses `rm -f` since the two
packages do not ship an identical set of tools to remove.
Assisted-by: Claude Code with Claude Opus 4.7
We've talked about adding the title to the filename several times over
the years and it seems really valuable. This requires us to compute a
"slug" for the title spelling that can be part of the filename.
Beyond that, we crossed 7000 recently, and so it seems likely that we
will need to add digits sooner rather than later here, so this goes
ahead and moves us to 6 digits so we don't have to adjust again for a
reasonable length of time.
To implement this and ensure we can sustain it going forward this adds a
tool to our pre-commit that validates (and corrects if needed) the
filename.
In order to update everything and keep links working, there are a _lot_
of changes, but the most interesting for direct review are in
`proposals/scripts`.
Assisted-by: Antigravity with Gemini
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This removes the need to install any specific version of Python or
figure out how to configure it by instead asking users to install `uv`
and letting it manage Python. Among other advantages, `uv` is designed
to be fast enough to embed directly into our scripts.
We were already using this in `bench_runner.py` so that the script could
import non standard library dependencies. Moving to it for the rest of
our Python unifies the approach and will also enable dependencies
whenever needed.
I've left `github_tools` alone as it has special handling with its own
Bazel setup.
I've updated the contributing tools to explain the approach here.
A lot of this is based on a few experiments using tools to help
synthesize a cohesive proposal or improve them. Likely more that can be
done here to get the most out of our tools in this space.
Assisted-by: Antigravity with Gemini
Previously, this relied on the subtlety that `-- ` (with the trailing
space) didn't get parsed as a flag. But there is support already for
escaping a leading `-` in a format argument, so use that to make the
code more obviously correct.
Assisted-by: Antigravity with Gemini
First, this makes the Bazel invocations not try to uses curses which
prevents running them with `pre-commit run ... -v` showing the timings
for each check. The curses display overwrote the output.
Second, this fixes the main slowdown I was seeing. Because we passed
_all_ files to the check-build-graph hook and there are large number of
files, pre-commit would run the tool over and over on a subset of the
files. This is especially wasteful as the build graph check already
doesn't do anything with the files, it just checks `//...` on each
invocation. So this just added a (large) constant factor of cost.
Third, this tries to reduce the cost of `fix_cc_deps.py` in the case of
large numbers of files. This still isn't _super_ fast -- but the rest of
the cost is in running the `bazel query` and parsing the output. I tried
switching it to jsonproto and it wasn't any faster. I think this would
need to be in a non-Python language and use `proto` directly to
significantly improve the cost here.
Assisted-by: Antigravity with Gemini
We had source generation support for some time, but needed to get all
the runtimes set up correctly so that standard library headers are
available. Now that this is in place, we can benchmark both languages.
Also fixes a bug in the C++ source generation causing compile failures.
Assisted-by: Antigravity with Gemini
The benchmarks themselves aren't really specific to `driver`.
Keeping the source generation near to the primary use case of
benchmarking also seems like a more discoverable location.
I feel a little bad doing this reorganization right after I gave a talk
with links to a bunch of this code, but seems good to reorganize a bit
before doing some work to extend things now that we have full standard
library support for C++ benchmarking and other improvements.
Assisted-by: Antigravity with Gemini
As part of this, move functions that seem reasonable to make out-of-line
to a separate `_impl.h` header file that is only included where the
explicit instantiation _definition_ is provided.
By using explicit instantiation we can make these templates behave more
like non-template classes in terms of supporting out-of-line definitions
that don't need to be compiled by every translation unit. The set of
eventual instantiations here is fundamentally known, and there tend to
be headers that define a canonical "leaf" type where it makes sense to
trigger the explicit instantiation.
Where we already had a `.cpp` file to put the explicit instantiation
definition, use it. But in some places we didn't have such a `.cpp` file
so this PR adds those.
This also requires that we have precise constraints on APIs that _can't_
be instantiated for specific argument types, as now we don't do this
lazily.
Combined, this appears to reduce the sum of object file sizes in the
`check` directory by almost 40% (122mb -> 74mb) in my measurement.
My actual goal was to improve compile times, but so far I don't have a
great methodology for measuring these... But the object file size
reduction seems to confirm this is a net win and likely represents a
non-trivial improvement in compile time.
Assisted-by: Antigravity with Gemini
Update the syntax for class (and interface/`impl`) methods to move
`self` into
the parameter parentheses `()` and make the type in its binding optional
(defaulting to `Self`). Introduce the `static` keyword for non-instance
member
variables to indicate static storage. Reflects the decision in leads
issue
[#6931](https://github.com/carbon-language/carbon-lang/issues/6931).
Updates the directly relevant design, but leaves a systematic update of
examples
to a future PR.
Assisted-by: Antigravity with Gemini
---------
Co-authored-by: Geoff Romer <gromer@google.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Also cleans up redundant code in the `sem_ir` dump methods that I missed
initially. Now we share as much logic as we can for dumping the non-ID
and ID components.
Assisted-by: Antigravity with Gemini
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
The labeling script will now directly set a check status for the PR as
`pending` when it marks something as dependent, and clear it when it no
longer is. This emulates a check that starts when marked as dependent
and runs until the last dependency lands, allowing automerge and other
workflows to work cleanly.
The branch protection rule will have to be updated to the new spelling.
This should do the same key thing as #7113, but integrated to the new
script.
Assisted-by: Antigravity with Gemini
Because this is an `__attribute__((used))` method in a templated base
class it forces a _huge_ amount of template instantiation in every
translation unit.
Often this was just printing the members of the type, which is still
useful in some cases (such as test output), but adds no value in the
debugger.
A more successful pattern for dumping has been namespace level
functions, and particularly static ones that more transparently don't
expand the non-debugger API surface. Add the few missing functions there
that cover `Printable` types with more interesting contents.
For several of these, it just gives us a "dump the whole thing" function
as a compliment to "dump this entity in the thing". These probably
aren't especially high value, but moving them here they become cheap, so
I've left them in.
For a couple, this expands the rich dumping support of SemIR constructs,
which should be substantially more useful than the previous `Dump`
behavior.
This reduces `check` cumulative object file size by another 14%.
Assisted-by: Antigravity with Gemini
This should detect when a PR has a dependency of another open PR and add
a comment and label describing it. The comment will even do a
best-effort to compute the best starting commit for review.
Whenever PRs are closed, it will also scan the open depnedent PRs and
try to either remove the PRs in the comment or if it reaches zero the
label.
It works to update a single comment on a PR rather than adding more
comments.
Assisted-by: Antigravity with Gemini
There are dependencies that kept this from upgrading automatically, but
while the new version technically includes breaking changes, they aren't
ones that cause any problems for VSCode.
Upgrading this is helpful as the old version has an irrelevant (for us)
security issue. With this we should be able to avoid distracting
security scanners.
Also updates other packages where relevant, all those automatically.
Assisted-by: Antigravity with Gemini
Without this, basic `bazel test //...` style wildcards would build a
bunch of extra configurations because of gaps excluding things. With
this, the action count of a normal build should be much more reasonable.
The switch from `target_compatible_with` to tagging is based on looking
at what ends up being most idiomatic and easiest -- trying to articulate
the complex and convoluted compatible with restrictions that would avoid
extraneous build configurations was really painful and this seems much
simpler and easier to deploy in a systematic way.
While here, also change the name of a rule that confused me to no end
while debugging this -- the rule that installs a `.bzl` file that
happens to be spelled `carbon_runtimes` is very different from all of
the other "installed carbon runtimes" kind of things in the tree. Adding
the file extension helps make that (much) more obvious.
Note that this is essentially a re-do of #7088 but now without any
dependencies that can mess up the merge.
Assisted-by: Antigravity with Gemini
This shouldn't change any functionality, but simplifies (significantly)
the logic in the installed toolchain, and also provides a better
conceptual balance between these.
I've tried to minimize the changes beyond a pure refactoring, but it was
a bit tricky to get everything working so some things have been mixed
in...
Assisted-by: Antigravity with Gemini
This reverts commit 4babfdbf22.
This was a stacked PR that was merged by accident, losing the commit and
description of the base change. Reverting and will re-land
independently.
Without this, basic `bazel test //...` style wildcards would build
a bunch of extra configurations because of gaps excluding things. With
this, the action count of a normal build should be much more reasonable.
The switch from `target_compatible_with` to tagging is based on looking
at what ends up being most idiomatic and easiest -- trying to articulate
the complex and convoluted compatible with restrictions that would avoid
extraneous build configurations was really painful and this seems much
simpler and easier to deploy in a systematic way.
While here, also change the name of a rule that confused me to no end
while debugging this -- the rule that installs a `.bzl` file that
happens to be spelled `carbon_runtimes` is very different from all of
the other "installed carbon runtimes" kind of things in the tree. Adding
the file extension helps make that (much) more obvious.
Assisted-by: Antigravity with Gemini
This is (far) from robust -- particularly with repeated compiles in the
same address space. But it appears to be sufficient in the short term,
and we already have the relevant TODOs to factor this upstream into
something that we can use here.
This also somehow uncovered a bug in how we were logging failed commands
-- the failed commands are destroyed when we destroy the driver object
(and its diagnostics object), so we simply cannot do the logging _after_
flushing diagnostics. That's probably ok, and just doing this in the
other order makes the code simpler.
Assisted-by: Antigravity with Gemini
Rather than defining our own action groups, work to re-use the
`rules_cc` ones, as they are (much) more comprehensive. Also, completely
eliminate the `codegen` action group as it was not well used. For
example, `-march` flags and `-O` flags change the preprocessor macros
defined. There isn't a really great "codegen" heuristic, so just pass
those flags to all compiles which is simpler anyways.
I'm tempted to do the same with preprocessor actions, but maybe it makes
sense to have that one stay separate.
Assisted-by: Antigravity with Gemini
This takes the bootstrap support that was added and makes it available
under convenient user-facing flags for while we're doing development.
For example, to build a bootstrap compiler and use it to build and run
the tests under `//common/...` you can now use:
```
bazel test --//:bootstrap_stage=1 --//:bootstrap_exec_config=true //common/...
```
This will use the stage1 bootstrap compiler, and it will build that
compiler in the exec config (so it is optimized and the above even works
when cross-building with Bazel).
Assisted-by: Antigravity with Gemini
Noticed this when testing the Carbon toolchain with a more complex
environment, don't have any way to observe this at the moment in Bazel
though.
Assisted-by: Antigravity with Gemini
This worked correctly in the system Clang toolchain, but was not
configured correctly in the Carbon toolchains. The test is designed to
let us cover all of these.
Assisted-by: Antigravity with Gemini
Hopefully this significantly reduces how often agents try to run `bazel`
directly without repeatedly including that in prompts. Also tried to
generally give useful skills for building, testing, and running things.
Also added a specific admonition to the `AGENTS.md` as there is a chance
that agents don't think they need to look at any skills for "standard"
build system commands like `bazel`, as those are "trivial". It seems
like a small chunk of context to spend to avoid churning with bad build
commands.
Assisted-by: Antigravity with Gemini
The runtimes and bootstrap Bazel logic was previously built around
defining custom Bazel platforms constrained with `constraint_settings`.
The use of platforms added significant complexity, including the need to
"save" and "restore" the original platform, and other complexity
stemming from changing the platform as a whole.
This PR switches to use the simpler tool of build settings, and
`target_settings` on the toolchain rather than platform compatibility.
This remove the entire need to save and restore the platform, and also
generally simplifies things.
This PR also fixes some bugs in the bootstrap that were hidden by the
use of platforms, such as the need to carefully manage the different
inputs to the runtimes build so that generated inputs pick up the
correct exec configuration -- the exec transition happened to do this
"automatically", but it seems better to handle explicitly. And it cleans
up an extraneous copy of `carbon_runtimes.bzl` that snuck in somehow.
Assisted-by: Antigravity with Gemini
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
This has been working really well for me, is incredibly faster than the
other approach, and some commits continue to hit bugs in the old system
where files that aren't even going to be run through `clangd-tidy` end
up tripping up the execution. Hopefully all of that is resolved with the
new version.
Fixes#7031
Also switches the previous symlinks test to be a more full integration
test. While a bit slow, it does seem worthwhile to have something that
tests things end-to-end, both with the prebuilt runtimes and the
on-demand runtimes. This test is already reasonably well separated from
the rest of the toolchain so incremental development shouldn't be
negatively impacted. And since we turned off ASan by default, it isn't
completely infeasibly expensive.
Assisted-by: Antigravity with Gemini
Note that this will require changing the branch protections to use new
names for all of the checks and be somewhat disruptive. There aren't any
really good ways I could find of fixing this. Some options that I
explored:
- Have a single `pre-merge` workflow file that contains all of the other
workflows, splitting as much of the logic as we can into re-usable
files. This would basically merge testing, `pre-commit`, and
`clangd-tidy` checking into a single workflow file. However, it would
also delay the pre-commit suggestions action to only run once _all_ of
these finish, rather than as soon as pre-commit finishes.
- Serialize `pre-commit` and the rest of `pre-merge` to get the effect
of the above option but without the downside. Instead, the downside
would be serializing some of our actions.
- Have a single `pre-merge` workflow that triggers whenever any of the
other workflows completes, and have it check whether all the others have
completed. It will fail until it reaches that point. This requires
passing in GitHub keys to the workflow so that it can check the status
of other checks, and documentation online seems to indicate it is
sometimes flaky, I assume because of racing triggers of events or
check-status not being guaranteed consistent in the queries.
- Have a single `pre-merge` workflow that polls, waiting for all the
other workflows to finish using some Python logic. This requires
building and maintaining code to poll GitHub, keys to authorize that
polling, and handling all of the failure modes of a polling operation --
timeouts, network issues, etc.
Maybe there are others, but not sure what they look like. Suggestions
welcome here.
I'm hesitant to either delay the pre-commit suggestions or serialize
pre-commit execution. And the complexity or flakiness of the other two
options seem worse than having to re-work the branch protections each
time the naming here changes. But interested if folks think a different
direction would be better.
Assisted-by: Antigravity with Gemini
This implements p7016 for tree_sitter. It also updates the build and
source file to allow this to build successfully and documents how to
successfully run these tests with Bazel given that it is fundamentally
not hermetic.
Assisted-by: Antigravity with Gemini
This covers basic usage and using it to make API calls to GitHub. It
also works to establish some reasonable safety guards to avoid
inappropriate commands.
Also introduces a skill specifically for ingesting the content in GitHub
issues using the command line tool. This is especially useful as
otherwise agents may try to browse the web version of an issues that is
significantly slower and harder to ensure the agent correctly gets all
of the context into its window and is able to leverage it.
This also disables the Google documentation style checking for agent
skills, as we want to instead try to follow the conventions, phrasing,
and other patterns that map best for agents' training sets. For example,
this avoids replacing `repo` with `repository` and avoids replacing
`e.g.` with `for example`. While these replacements make lots of sense
for our human-facing documentation, the agent-facing docs probably
benefit from being terse and using the exact patterns that agents are
trained on.
Assisted-by: Antigravity with Gemini
Also increases the default optimization to `-Og` which is likely to give
faster turn-around time which is what we want to optimize for here. This
should also _substantially_ shrink binary sizes, etc.
ASan is still available via `--config=asan`, and is added to the CI
infrastructure. However, my current thought is to only run it after push
rather than in PRs and in the merge queue.
This also switches to a more Bazel-based install layout, skipping the
FHS-based synthetic layout. The FHS-based layout is still reconstructed
explicitly when building an installable tar-ball.
The biggest change is to configure the just-built install as a Bazel
toolchain, including allowing it to build its own runtime libraries as
native Bazel libraries. This removes the need for a monolithic runtimes
build, all of that code logic is removed.
This should also pave the way to using the just-built toolchain for
doing a full 3-stage bootstrap. Building the 2nd stage is included here
as it was a particularly effective way to test that the Bazel
integration was fully working. Adding a 3rd-stage check for stability is
future work, but should be pretty easy.
There is a down-side: this uses the busybox to do the runtimes
compilation, which means they will be re-built after ~any change to
Carbon. However, the integration with Bazel should largely pay for this,
and we can continue to factor the tests away from depending on built
runtimes in most cases.
Now that we're building and testing the runtimes more directly, this
surfaced a problem with the layout of runtimes on macOS that is fixed
here. All of the Darwin OSes use a custom layout for their resource
directory compared to other targets. We now model this in both the C++
built runtimes and the Bazel built runtimes.
Assisted-by: Gemini via Antigravity
directory
This cleans up the `//toolchain/install/BUILD` file and the tree
generally to be more focused on arranging the actual installation rather
than preparing inputs to that installation.
I picke `//toolchain/runtimes` so we can put other runtimes preparation
logic there, but open to any other suggested organization.
There are other runtimes things that would in theory make sense to move
such as the `prebuilt_runtimes` logic, but a subsequent PR will delete
those and so I'm leaving them where they are for now.
This shifts the Bazel toolchain configuration of our installation to
build all of the Clang runtimes Carbon uses on-demand natively in Bazel.
We export the information about how to build into a generated Starlark
file, and emit BUILD files and Starlark logic into the installation to
orchestrate the build.
This requires some complex management of Bazel toolchains -- we need to
first set-up a "runtimes toolchain" that doesn't have runtimes of its
own, but can be used to _build_ runtimes. Then we build the runtimes
using that toolchain, and assemble them into the standard layout for a
Carbon runtimes tree. Finally we configure the _actual_ toolchain with
this built tree.
Currently, this is only setup for the installed toolchain, but I plan to
factor this runtimes build into one that can be used directly as well to
break up the monolithic runtimes build step into Bazel-integrated build
of the runtimes. This will also serve as the foundation for adding
bootstrapping support directly to our Bazel build.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
This moves the most complex of the logic fully into Starlark: both the
many different platform sources list, and the overriding of generic
files with architecture specific files.
This also fixes significant bugs in the AArch64 build where we were
skipping numerous files: all of the outlined atomics and `emupac.cpp`.
This PR forcibly disables `emupac.cpp` as fixing that will require a
more significant change.
This isn't as interesting as others, as it only involves compile
options.
It also adds a missing flag of `-fno-lto` as these objects can't be
LTO-ed.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Add an optional additional set of positional parameters that can be
passed to the `link` subcommand for Clang-style (or GCC-style)
`LDFLAGS`. These can _also_ contain object files, etc., and in fact it
is useful to allow them to contain object files in order to integrate
the `carbon link` subcommand into a build system that mixes both link
flags and object files. This at least happens with Bazel, and I suspect
is common.
Eventually, it would be nice to have sufficient semantics to handle all
the varieties of links we want without resorting to this escape hatch,
but that's likely a long way away and so it seems especially useful to
allow falling back to Clang's flags as needed for now.
This does somewhat directly surface the Clang implementation detail in
the command line syntax, but I don't see a lot of good alternatives.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
The goal here is to be able to construct a build of the runtimes
directly in Bazel, or by emitting `BUILD` files, or by emitting into C++
code and using that on-demand. For that, we want a single source of
truth, and that source in Starlark.
This should also make the information more generally useful, and so I'm
moving as much as I can into the LLVM Bazel build. Apologies as that
makes the diffs extra annoying.
I do plan on upstreaming the Bazel parts of this, but would like to get
everything working in Carbon and stabilized first.
While here, I've also made a change suggested for the future in the
initial review by lifting the C++ template out of a string literal in
the `.bzl` file, and into an actual separate C++ file.
This only moves libc++, libc++abi, and libunwind. I want to get those
three working end-to-end before I work on the builtins or `crtbegin` and
`crtend`, as those have a bunch of additional complexity.
This also only uses the info in the C++ on-demand build. It seemed like
a reasonable increment to start code review, and my plan is to work on
other build strategies in a follow-up PR. If that doesn't work, let me
know and I'll come back once I have at least a second use of the info
here.
Chandler CarruthandRichard Smith
