Chandler Carruth
d0badecd2a
Historically, we disabled the rendered form in tests outside of `toolchain/diagnostics`. The idea was to keep tests shorter and avoid very long chains of autoupdate before stabilizing due to all the line numbers and other artifacts in the rendered form. This PR adds `--diagnostic-line-numbers=[all|first]` and defaults to `first` in tests. This keeps the rendered form so that we can easily code review that form and the visual cues it does or doesn't provide. But it removes all but the first location's line numbers to provide significantly more stability in tests. By default, tests get the full unicode line-drawing form of diagnostics so we can ensure all of the artifacts there make sense with the specific case being diagnosed, but we do retain some focused ASCII-rendering tests to ensure that fallback remains usable. There are some issues here that will be fixed in subsequent PRs. Notably, we try to keep the frame together in the check lines, and so force some parts to use the first location. This largely works, but fails in some edge cases, causing the entire frame to be located incorrectly. There is a TODO about this that we need to fix, and it only occurs 6 times across the test suite, so it didn't seem worth front-loading a fix for here. Assisted-by: Claude Code
Toolchain architecture
Table of contents
Goals
The toolchain represents the production portion of Carbon. At a high level, the toolchain's top priorities are:
- Correctness.
- Quality of generated code, including performance.
- Compilation performance.
- Quality of diagnostics for incorrect or questionable code.
TODO: Add an expanded document that details the goals and priorities and link to it here.
High-level architecture
The main components are:
-
Driver: Provides commands and ties together compilation flow.
-
Diagnostics: Produces diagnostic output.
-
Diagnostics rendering: Draws diagnostics on a terminal.
-
Compilation flow:
- Source: Load the file into a SourceBuffer.
- Lex: Transform a SourceBuffer into a Lex::TokenizedBuffer.
- Parse: Transform a TokenizedBuffer into a Parse::Tree.
- Check: Transform a Tree to produce SemIR::File.
- Lower: Transform the SemIR to an LLVM Module.
- CodeGen: Transform the LLVM Module into an Object File.
Design patterns
A few common design patterns are:
-
Distinct steps: Each step of processing produces an output structure, avoiding callbacks passing data between structures.
-
For example, the parser takes a
Lex::TokenizedBufferas input and produces aParse::Treeas output. -
Performance: It should yield better locality versus a callback approach.
-
Understandability: Each step has a clear input and output, versus callbacks which obscure the flow of data.
-
-
Vectorized storage: Data is stored in vectors and flyweights are passed around, avoiding more typical heap allocation with pointers.
-
For example, the parse tree is stored as a
llvm::SmallVector<Parse::Tree::NodeImpl>indexed byParse::Nodewhich wraps anint32_t. -
Performance: Vectorization both minimizes memory allocation overhead and enables better read caching because adjacent entries will be cached together.
-
-
Iterative processing: We rely on state stacks and iterative loops for parsing, avoiding recursive function calls.
-
For example, the parser has a
Parse::Stateenum tracked instate_stack_, and loops inParse::Tree::Parse. -
Scalability: Complex code must not cause recursion issues. We have experience in Clang seeing stack frame recursion limits being hit in unexpected ways, and non-recursive approaches largely avoid that risk.
-
See also Idioms for abbreviations and more implementation techniques.
Adding features
We have a walkthrough for adding features.
Design docs
We have design docs.
Videos
Talks
These talks are focused on implementation details of the toolchain, and can be helpful for learning how the toolchain internals work.
2025
Implementation walkthroughs
These are recordings of implementing PRs.