The INCLUDE-FILE option is only used in the toolchain tests for now. If specified in a file test, the given file path is added to the test's arguments. For toolchain tests this makes the file's package available to the test. The `--custom-core` command line flag is added to the driver, which avoids adding the production `Core` package to the command line. Together, these allow a test to provide their own minimal `Core` package. For example, this would replace `Core` with the package and prelude in `facet_types.carbon`. ``` // INCLUDE-FILE: toolchain/testing/min_prelude/facet_types.carbon // EXTRA-ARGS: --custom-core ``` To support this: * //testing knows how to parse INCLUDE-FILE out of the header of a test file. * //testing adds the file to the virtual file system, and includes it in the test's arguments. * //toolchain/driver grows the --custom-core command line flag to avoid loading the production `Core` package. Tests that were creating their own minimal prelude to define BitAnd on types are now pointed to toolchain/testing/min_prelude/facet_types.carbon as the prelude. They no longer need to `import Core` in each test as a result. Such tests are no longer `no_prelude`, but instead have their own prelude. So they are moved to a `min_prelude` subdirectory. Closes #5076
Toolchain architecture
Table of contents
Goals
The toolchain represents the production portion of Carbon. At a high level, the toolchain's top priorities are:
- Correctness.
- Quality of generated code, including performance.
- Compilation performance.
- Quality of diagnostics for incorrect or questionable code.
TODO: Add an expanded document that details the goals and priorities and link to it here.
High-level architecture
The main components are:
-
Driver: Provides commands and ties together compilation flow.
-
Diagnostics: Produces diagnostic output.
-
Compilation flow:
- Source: Load the file into a SourceBuffer.
- Lex: Transform a SourceBuffer into a Lex::TokenizedBuffer.
- Parse: Transform a TokenizedBuffer into a Parse::Tree.
- Check: Transform a Tree to produce SemIR::File.
- Lower: Transform the SemIR to an LLVM Module.
- CodeGen: Transform the LLVM Module into an Object File.
Design patterns
A few common design patterns are:
-
Distinct steps: Each step of processing produces an output structure, avoiding callbacks passing data between structures.
-
For example, the parser takes a
Lex::TokenizedBufferas input and produces aParse::Treeas output. -
Performance: It should yield better locality versus a callback approach.
-
Understandability: Each step has a clear input and output, versus callbacks which obscure the flow of data.
-
-
Vectorized storage: Data is stored in vectors and flyweights are passed around, avoiding more typical heap allocation with pointers.
-
For example, the parse tree is stored as a
llvm::SmallVector<Parse::Tree::NodeImpl>indexed byParse::Nodewhich wraps anint32_t. -
Performance: Vectorization both minimizes memory allocation overhead and enables better read caching because adjacent entries will be cached together.
-
-
Iterative processing: We rely on state stacks and iterative loops for parsing, avoiding recursive function calls.
-
For example, the parser has a
Parse::Stateenum tracked instate_stack_, and loops inParse::Tree::Parse. -
Scalability: Complex code must not cause recursion issues. We have experience in Clang seeing stack frame recursion limits being hit in unexpected ways, and non-recursive approaches largely avoid that risk.
-
See also Idioms for abbreviations and more implementation techniques.
Adding features
We have a walkthrough for adding features.