Chandler CarruthandJon Ross-Perkins
13502b7c89
This restores the symlinks for the installation, but teaches the busybox info search to look for a relative path to the busybox binary itself before walking through symlinks. This let's it find the tree structure when directly invoking `prefix_root/bin/carbon` or similar, either inside of a Bazel rule or from the command line, and mirrors how we expect the installed tree to look. This works even when Bazel resolves the symlink target fully, and potentially to something nonsensical like a CAS file. In order to make a convenient Bazel target that can be used with `bazel run //toolchain`, this adds an override to explicitly set the desired argv[0] to use when selecting a mode for the busybox and a busybox binary. Currently, the workaround uses an environment variable because that required the least amount of plumbing, and seems a useful override mechanism generally, but I'm open to other approaches. This should allow a few things to work a bit more nicely: - It should handle sibling symlinks like `clang++` to `clang` or `ld.lld` to `lld`, where that symlink in turn points at the busybox. We want to use *initial* `argv[0]` value to select the mode there. - It avoids bouncing through Python (or other subprocesses) when invoking the `carbon` binary in Bazel rules, which will be nice for building the example code and benchmarking. It does come at a cost of removing one feature: the initial symlink can't be some unrelated alias like `my_carbon_symlink` -- we expect the *first* argv[0] name to have the meaningful filename for selecting a busybox mode. It also trades the complexity of the Python script for some complexity in the busybox search in order to look for a relative `carbon-busybox` binary. On the whole, I think that tradeoff is worthwhile, but it isn't free. --------- Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Toolchain architecture
Table of contents
Goals
The toolchain represents the production portion of Carbon. At a high level, the toolchain's top priorities are:
- Correctness.
- Quality of generated code, including performance.
- Compilation performance.
- Quality of diagnostics for incorrect or questionable code.
TODO: Add an expanded document that details the goals and priorities and link to it here.
High-level architecture
The main components are:
-
Driver: Provides commands and ties together compilation flow.
-
Diagnostics: Produces diagnostic output.
-
Compilation flow:
- Source: Load the file into a SourceBuffer.
- Lex: Transform a SourceBuffer into a Lex::TokenizedBuffer.
- Parse: Transform a TokenizedBuffer into a Parse::Tree.
- Check: Transform a Tree to produce SemIR::File.
- Lower: Transform the SemIR to an LLVM Module.
- CodeGen: Transform the LLVM Module into an Object File.
Design patterns
A few common design patterns are:
-
Distinct steps: Each step of processing produces an output structure, avoiding callbacks passing data between structures.
-
For example, the parser takes a
Lex::TokenizedBufferas input and produces aParse::Treeas output. -
Performance: It should yield better locality versus a callback approach.
-
Understandability: Each step has a clear input and output, versus callbacks which obscure the flow of data.
-
-
Vectorized storage: Data is stored in vectors and flyweights are passed around, avoiding more typical heap allocation with pointers.
-
For example, the parse tree is stored as a
llvm::SmallVector<Parse::Tree::NodeImpl>indexed byParse::Nodewhich wraps anint32_t. -
Performance: Vectorization both minimizes memory allocation overhead and enables better read caching because adjacent entries will be cached together.
-
-
Iterative processing: We rely on state stacks and iterative loops for parsing, avoiding recursive function calls.
-
For example, the parser has a
Parse::Stateenum tracked instate_stack_, and loops inParse::Tree::Parse. -
Scalability: Complex code must not cause recursion issues. We have experience in Clang seeing stack frame recursion limits being hit in unexpected ways, and non-recursive approaches largely avoid that risk.
-
See also Idioms for abbreviations and more implementation techniques.
Adding features
We have a walkthrough for adding features.