At present, `var *;` is a crash because it has errors that weren't being properly stored/handled. This changes the handling to give more to track invalid parses in patterns.
This handles the basics of type and value for structs. Structurally, these look like parameters and arguments (respectively) because expressions/generics may result in multiple IR nodes being generated.
Because `{}` needs to be cast to a type for storage, I'm also adding some validation that's not specific to `{}`, e.g. that `1` shouldn't be valid as a type for storage (previously, nothing errored for that).
This adds more stringification of types, particularly literals, because they come up in value errors now.
ImplicitAs is the result of me mulling whether I'm taking the right approach on type conversions. I think it needs to return a value so that if the implicit cast rewrites the value, the result is accessible to the caller. I may reorient the current TryTypeConversion logic to be more based on the ImplicitAs logic.
I'm trying to reduce the amount of per-SemanticsNodeKind boilerplate, and make mistakes (e.g., SemanticsNodeKind not matching the Make name, misplacing the type, or Get/Make type mismatches) easier to see.
I could've done this with (more) macros, but felt that the template approach was reasonable enough and likely easier to understand/debug. I'm not sure whether there's more that I could be doing with variadics to reduce the amount of factory code, but this feels good right now.
These changes should make output more stable when builtins are added to semantics. By omitting them from nodes and printing nodes as "relative to the last builtin", I should be able to add and remove builtins without automatically affecting every test. Also by printing builtin nodes as `nodeNameOfBuiltin`, it's a little easier to understand what's going on (for me, at least).
`self` wasn't being handled in expression logic, only parameter logic. This is needed for `self.x`.
Also renames SelfDeducedParameter to SelfIdentifier because it's parsed fairly consistently with Identifier; this better represents the situations where `self` will need to be handled, it's where `Identifier` is allowed (although `Identifier` could also refer to a type... but I still think `SelfDeducedParameter` is a more difficult to understand name).
This was bugging me after I saw all the strings; it feels like this is why we have EnumBase on the toolchain side.
I've included the move of EnumBase to //common because I figured it's reasonable to evaluate together; if we don't want EnumBase in this case, it doesn't make sense to move.
Also cleans up some comments about related parse nodes. Currently basic and not heavily validated.
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
We could similarly add others -- this is intended to make it easy to add more that parse essentially the same.
The functionality expected is that types will use GetDeclarationContext in order to error on certain functionality in the declaration scope loop. e.g., with how constraints and interfaces currently don't allow definitions.
I've only moved out `package` because it's only valid on the top line. It might still be good to parse it later, but with slightly different logic because it would always be an error, and the declaration context isn't quite the right framing for that.
Also unifies some errors with `fn`.
This is something we'd discussed. I added a TODO that we may want to eventually make this a map, but I remain uncertain and think it's not something that's going to really cost us if we end up switching back. In the meantime, I think this approach does offer simplicity. As discussed too, the performance overhead of a map may ultimately not be worthwhile here versus the relative memory costs.
The comments in parse_node_kind.def capture the change being made here.
Before:
```
// _external_: DeclaredName
// InterfaceBodyStart
// _external_: statements
// InterfaceBodyEnd
// InterfaceDefinition
```
After:
```
// InterfaceIntroducer
// DeclaredName
// InterfaceDefinitionStart
// _external_: declarations
// InterfaceDefinition
```
Really I just want to treat introduced things consistently. `var` defines my philosophy here: it doesn't always have a `DeclaredName`, so the `VarIntroducer` _must_ be the bounding node. By being consistent with that, I believe that overall the structure becomes easier to understand (that is, there are fewer inconsistencies to understand).
This also adds InterfaceDeclaration, since I think it can be predicted we'll have that, and it's helpful for making recover consistent with HandleDeclarationError.
Similarly, I'm also trying to standardize the loop processing a little with HandleDeclarationLoop. In the current approach, InterfaceDefinitionFinish isn't a necessary state, so I'm removing it.
This is starting to build out actual lowering logic, for a really simple `fn Main() -> i32 { return 0; }`
Notes for achieving this:
- In semantics, currently function names are bound separate from the signature. When emitting IR, this turns out to be inconvenient because we want to know the name when we process the declaration and the definition. This change addresses that by merging the name into the FunctionDeclaration node, which is also accessible from the definition. It removes the separate BindName. This should be the cause of all the test changes in semantics, because the IR generated changes.
- Add a "Lowering" class which I'm using to hold the llvm builder state. This class now has minimal support for the SemanticsIR generated by the above example.
- In the "Lowering", values from expressions are stored in a DenseMap. I'll keep thinking about whether there's a cleaner way to achieve this, and I'd call it a temporary solution for now. However, this is how the `0` in `return 0` gets properly associated across SemanticsIR instructions, and it'll frequently be an issue in less trivial cases.
Adjusts handling of class constants (`static const`) to use CamelCase. This probably better reflects how we use it in C++ code, treating as appropriate for CamelCase instead of under_score.
Fixes adding_children to be preorder in caller (not sure why this wasn't automated).
No automated changes.
Just opening files in vscode and seeing what clangd flags.
Some edits to ostream.h to stop it from getting flagged (the usage pattern means it's not always obviously used).
This starts handling return types on functions, and comparing types with `return` statements.
Note, errors remain poor because the type literal is currently associated with a builtin, losing the parse_node that specified it. This means we don't have the original source location to associate with, even though it may be helpful to point at the type in source. We could point at the signature overall, but my leaning is that we wouldn't want that long-term, so TODOs for now and may want to change a little about how the parse node is tracked once things are a little further along.
The intent here is that changes to prelude.carbon shouldn't break every test that expects some error from prelude.carbon; that would be too fragile. As a consequence, this effectively ignores the line number in prelude.carbon.
This is a little complex because we don't know which line in the original source file is actually causing the error, just that there is an error. Also, the previous look-behind approach required a fixed-with prefix, whereas we want a little more than that in order to capture the filename for comparison.
This would be hard to do with extra_check_replacement because the path to bazel.runfiles is complex to calculate. As a consequence, this is basically all new code.
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This adds tracking of call information plus basic type checking. It adds a builtin for the empty tuple, mainly so that I have the basis for a default function return type.
As an aside, it also unifies printing within SemanticsIR, fixing a missing comma after callables.
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Reals were mostly handled, but this PR adds storage of them. It also switches a little towards the FloatingPointType semantic from TokenizedBuffer.
While real literals like `1.0` were handled, the type literals were not. This just adds `f64`, similar to how I also only support `i32`.
The String type literal wasn't used, so I've added support in lexer and parser. Per discussion with @zygoloid String might be renamed based on the newer type literal plan, but it's still String in explorer and the design, so this is just consistent.
The builtin_types.carbon tests the three basic types that are there right now. The test is added to both parser and semantics so that it's clear what the state is in both stages.
There were some notes about default constructors being required in parse_tree.h, but they don't seem to be. Removing the default constructor forces the explicit `::Invalid` where there's no value immediately being assigned, which works fine for existing code. I think this actually reduces the chance of accidents more than the prior default-construct-as-invalid approach.
It feels like every time I add a builtin, I forget to add it to the builtin IR. This approach enforces the addition of builtin nodes, and also takes advantage of the normalized structure to set types appropriately (I think the new approach is simpler on this aspect).
Using the same const/constexpr done in EnumBase, adds Invalid and Builtin* values to replace Make functions that produced the same. This should make it clearer at call sites what the cost actually is, and reduces the syntactic overhead for MakeBuiltinReference in particular.
Really, this is that MakeBuiltinReference has been feeling pretty verbose, so I did that, and then one MakeInvalid is right next to it, and then obviously I should replace the other MakeInvalid for consistency...
Lazy allocation means that we can use a single node block for _all_ empty node blocks. The change in timing for when the definition node is emitted shouldn't affect semantic correctness; the signature is already present for recursive calls.
This change is to make it clearer what is a TODO versus unexpected behavior. I'm doing this now because I feel it's been getting a little confusing in code.
So for example if I write the code a return type `-> i32`, I get the diagnostic output plus the dump of the (invalid) IR:
```
/carbon-lang/toolchain/semantics/testdata/function/basic.carbon:37:10: Semantics TODO: HandleReturnType
cross_reference_irs_size: 1
callables: [
]
integer_literals: [
]
strings: [
]
nodes: [
{kind: CrossReference, arg0: ir0, arg1: block0, type: node0},
{kind: CrossReference, arg0: ir0, arg1: block1, type: node1},
{kind: CrossReference, arg0: ir0, arg1: block2, type: node0},
{kind: CrossReference, arg0: ir0, arg1: block3, type: node0},
]
node_blocks: [
[
],
]
```
For parameters (and in the future, arguments too; generally comma-separated lists) track two node blocks:
1. param_ir: The complete IR.
2. param_refs: Nodes within the IR that are the "root" parameter.
param_refs should allow quick counting of the # of parameters, and more efficient comparison of call args with function parameters. param_ir should be necessary to generate the actual signature.
In order to construct this, this refactors the node_block_stack into its own class, which is reused in params_stack. These carry references to the underlying SmallVector for lazy modification in order to avoid a dependency cycle with SemanticsIR (also see notes on empty node blocks below).
When finalized, the block pair is pushed onto finished_params_stack. That's because node_stack only has space for one thing, and this is two things -- so I'm essentially choosing a trade-off of adding another stack in order to avoid consuming more space in the expectation that most parse nodes have 0 or 1 things to return, and 2 will be very rare.
As factored, this currently consolidates most empty node blocks into a single canonical empty node block. This is because I think empty blocks, i.e. `()`, will be very common. In order to achieve this, SemanticsNodeBlockStack does lazy creation.
An alternative approach would have been to use 1 node block per parameter. We decided against this in order to reduce the number of vectors being created.
The parser and lexer already produce YAML, so this is fundamentally a consistency issue. I've been thinking about this, and was looking again because I'm working on adding callables, and figured I'd just fix it now.
Fundamentally this `.Print()` is wrong for debug output at present because `.fixed_spelling()` can be empty. It's also inconsistent with other enums to use it. We frequently print tokens for debugging, and it's easy to forget to specify `.name()` there.
Diagnostics use formatv, so we can provide a format_provider and address it in one spot that way. It also makes it harder to just forget to do the right thing.
Currently, there's a mix of accessing node_stack_ both directly and indirectly, and there are already several Push/Pop methods to help wrap the behavior. However, there's also direct access because of shifting over time, as well as variations in _how_ the stack is used.
This migrates to a separate class in order to make a more specific contract for the API. It cleans up existing uses and adds APIs where needed.
This somewhat abuses control-flow macro handling in `clang-format` to get the behavior we want. Fortunately, I don't think we're ever likely to want this for its intended purpose so it seems harmless to co-opt it like this. If we really wanted, we can narrow it to `if`-macros, but I picked the simpler option to start.
FWIW, I tried all the other macro formatting special cases to see if one would work but it didn't. Might be worth filing a feature request to get a `TypeDefinitionMacros` setting to compliment `TypenameMacros` and format like this does, but it seems (very) low priority.
The goal here is to (significantly) reduce the boilerplate needed when defining classes that wrap enums, especially those managed with the `.def`-file style X-macros that are common in the toolchain.
This should also provide both better and more consistent functionality to those classes once ported over to it.
Initially, only `ParserState`, `SemanticsNodeKind`, and `SemanticsBuiltinKind` are ported as these were also the three that JonMeow ported in his original pull/2453 "option 5". This is heavily based on that version of the code.
Goals I was considering that influenced the design:
- Keep the individual enum-wrapping classes as simple and easy to read as possible. Especially important is keeping the `.def` files that are often filled with really important documentation clean and easy to maintain over time.
- Don't rely on computed `#include`s as that is an especially dark corner of the preprocessor and breaks some build systems.
- Have a really good API of the enum-wrapping class, including nice constant names for the values, easy printing, and even easy debugger-callable methods to get the name (as opposed to the integer value).
- Keep the API that users interact with in the base class as clean and easy to read as possible.
- Reduce the boiler plate for each instance of these as much as possible.
- Avoid excessive inline generated code or constants that would result in steady growth in object file sizes and linker effort doing deduplication.
These goals aren't always compatible, so we end up needing to pick a compromise between them when in tension. I think this version is a pretty good compromise.
The original version I started with already pull most of the API into a CRTP-style base class. This version pulls *all* of the common API. This is the main tool for getting consistency and avoiding duplication. However, connecting this base class to the individual enum wrappers is still difficult. Some specific changes here that try to do as much as possible there:
- Use a slightly fancier macro pattern to reduce the boilerplate of defining the raw `enum class` prior to the wrapper class.
- Use a macro to simplify naming the base class.
- Move the name table to a `.cpp` file to avoid every inclusion generating a complete copy of the strings (that the linker has to deduplicate). This is done with some care to sharply reduce the boilerplate needed in that `.cpp` file.
- Sink the name _API_ fully into the CRTP base class. This requires some significant complexity in the implementation, but all of that is hidden behind a single implementation detail macro, and the API itself is simple and readable. This also makes it much more reasonable to test the entire system a single time next to the base class.
This version also moves from constant factory functions to normal constants. This requires two batches -- first a declaration, and then a definition -- but the API result is significantly better and similar to the original option, the macro structure reduces the cost of these. Unfortunately that makes the adoption a bit noisy, but I think its worth the churn.
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
The problem I'm trying to solve is: `var x: i32 = x;`. This change makes it so that name lookup fails, by removing `x` from name lookup between the `=` and `;`.
`var x: i32` still adds to name lookup to handle future situations like `var (x: i32, x: i32);` which is still a redefinition of `x`; if we don't add `x` to name lookup, it gets harder to catch that example.
The VariableDeclaration/VariableInitializer refactor in ParseTree supports this by given a bracketing-like structure for semantics to cue that it's entering an initialization expression. With this, VariableInitializer can remove the name lookup and queue it to be restored. VariableDeclaration doesn't need to change too much since it's still bracketed by VariableIntroducer, and so we just traverse slightly differently.
Note this also incidentally changes a little about NameReference, that it's returning the storage consistently instead of the name. You can see this e.g. in global_lookup.carbon, `Assign(node8, node4): node2;` using node4 (VarStorage) instead of Node5 (BindName). Really either _could_ work, since from a BindName we can get to the VarStorage, and that may be reason to switch later if we find it preferable to have the BindName for whatever reason.
But the *actual* value in NameLookup is a BindName so that errors can associate with the _name_ instead of the "storage" parse node, which is currently the `:`. This is mainly for fail_duplicate_decl.carbon, which has a "Previous definition" note that points at the storage's parse node.
This is currently used once for PreviousDefinition in semantics.
This PR doesn't just add a builder, it also adds support to the emitter itself to collect notes attached to a diagnostic, and to the consumers and emitters to print all of them.
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
This switches to single list storage of SemanticsNode. The driving motivation behind this is to simplify cross-references within a given IR. Types of nodes will frequently refer to other blocks. This causes a significant increase in the number of cross-references, which can become difficult to manage (and reason about). By reducing to a single list of nodes, cross-references are only needed when crossing IR boundaries.
Because cross-references now only have 2 things to track (IR and index), they can be a regular SemanticsNode and don't need further indirection. This wasn't motivating, but feels like it reinforces the simplification.
Note this isn't being used to deduplicate nodes, at least right now. That could lead to difficult-to-update situations, but also most nodes are associated with the underlying ParseTree::Node in order to track sources for diagnostics; as a consequence, nodes representing equal text in different source locations wouldn't be the same node. There may be future opportunities here, discussed with @zygoloid, but no action is taken at present.
We may eventually want to switch the storage of NodeBlocks to have `[start, end)` ranges instead of individual numbers, but I'm leaving that alone for now.
As an aside, I noticed I was accidentally overloading the copy constructor on SemanticsIR. I've added some disambiguation on that, but am not deleting the copy constructor per style advice (even though the type should never be copied due to storage size).
codespell tries to change `CrossReference -> cross-reference` so disabling it there.
When binding a name, add it to name lookup. On NameReference nodes, use name lookup.
- Switches from "identifiers" to the more generic "strings". Not strictly necessary here, but it's the overall direction I think we've agreed upon and wanted to do it while building more support out.
- Starts doing deduplication of strings.
- On BindName, registers names with name lookup.
- Does name lookup based on the deduplicated string.
- Per discussion with zygoloid, design is intended to be constant-time lookup regardless of the number of parent scopes.
- Adds scopes so that we can track names which will be deregistered from lookup.
Jon Ross-Perkins
