At present, `var *;` is a crash because it has errors that weren't being properly stored/handled. This changes the handling to give more to track invalid parses in patterns.
Reals were mostly handled, but this PR adds storage of them. It also switches a little towards the FloatingPointType semantic from TokenizedBuffer.
While real literals like `1.0` were handled, the type literals were not. This just adds `f64`, similar to how I also only support `i32`.
The String type literal wasn't used, so I've added support in lexer and parser. Per discussion with @zygoloid String might be renamed based on the newer type literal plan, but it's still String in explorer and the design, so this is just consistent.
The builtin_types.carbon tests the three basic types that are there right now. The test is added to both parser and semantics so that it's clear what the state is in both stages.
The problem I'm trying to solve is: `var x: i32 = x;`. This change makes it so that name lookup fails, by removing `x` from name lookup between the `=` and `;`.
`var x: i32` still adds to name lookup to handle future situations like `var (x: i32, x: i32);` which is still a redefinition of `x`; if we don't add `x` to name lookup, it gets harder to catch that example.
The VariableDeclaration/VariableInitializer refactor in ParseTree supports this by given a bracketing-like structure for semantics to cue that it's entering an initialization expression. With this, VariableInitializer can remove the name lookup and queue it to be restored. VariableDeclaration doesn't need to change too much since it's still bracketed by VariableIntroducer, and so we just traverse slightly differently.
Note this also incidentally changes a little about NameReference, that it's returning the storage consistently instead of the name. You can see this e.g. in global_lookup.carbon, `Assign(node8, node4): node2;` using node4 (VarStorage) instead of Node5 (BindName). Really either _could_ work, since from a BindName we can get to the VarStorage, and that may be reason to switch later if we find it preferable to have the BindName for whatever reason.
But the *actual* value in NameLookup is a BindName so that errors can associate with the _name_ instead of the "storage" parse node, which is currently the `:`. This is mainly for fail_duplicate_decl.carbon, which has a "Previous definition" note that points at the storage's parse node.
I think it'd be helpful to examine what the parse tree looks like in failure cases.
Note, I'm not sure that the behavior of the recovery situations is correct; the test had been asserting that the parse tree should indicate it's error free. However, this means that the only signal the the driver that the input is invalid is that the diagnostic emitter was used. I think it may be important to have it return a non-zero error to prevent compile, or we can turn these into warnings but then "requiring a space" is wrong.
Either way, that's a concern I have with the pre-existing recovery behavior: I'm just trying to highlight it as I make this change, because now the test is really that nodes aren't tagged with has_error.
- Finishes remaining "todo" parse nodes.
- Improving error recovery for invalid designators and structs, so that the parse tree still looks similar to a valid parse tree.
- Call expressions now have the thing being called as a child (of the start) instead of a sibling.
- Use of Start is replacing use of End in several parse nodes, like structs and call expressions.
- Adjusting documentation of parse node structures in an attempt to make it more consistent and understandable.
- The current state for interfaces and if/else is mostly being documented, not altered.
This works on multiple statements to make them better for the bracketing model. Stub nodes are added in more cases of invalid syntax, simply so that the semantics has reliably structured input. Comments in parse_node_kind.def now try to show the expected parse tree structure in postorder form.
This labels If, While, and For a little differently in parse nodes so that at the start of the postorder traversal, it'll already be available to semantics which structure is being processed. I need to do a little more with If in particular, but this felt like a reasonable stopping point.
While this makes significant parser changes, the changes to parser_state.def are minimal, mostly naming-related. The actual flow isn't substantively changed, just a couple minor names and the new As(If|While) state which allows distinguishing IfCondition and WhileCondition.
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
When there's no semicolon for an invalid EmptyDeclaration, rather than producing nothing, produce an EmptyDeclaration with the original location that led to the error.
Note this removes a direct edit (the only one) of the parse tree's error state. Elsewhere it's an indirection from adding an error node.
This is necessary in order to use a bracketing approach; parsing doesn't know the contained format until it parses the first element, which we don't want to do look-ahead for. I think the bracketing is higher value than knowing the format before adding the node.
This changes `return`, `break`, and `continue` to treat the keyword as the "start" and semicolon as the "parent", essentially bracketing the keyword.
Pragmatically this is focusing on making `return` work with only one ParseNodeKind: because `return` and `;` now bracket the expression, we can tightly determine whether the `return` has arguments without looking at subtree size. However, it's possible that `break` and `continue` may in the future take some kind of label as an argument, so the consistency seems beneficial there too.
Note this eliminates the StatementEnd ParseNodeKind, as it's obsolete with this change.
The ParseTree comments say that preorder is "easier to visualize and read". The problem is, both the ParseTree and Semantics need to operate on the postorder traversal: the ParseTree during construction, and the Semantics during processing. As a consequence, understanding the postorder traversal is important, but it's also very hard to decipher when presented preorder. This PR provides a way to see the postorder, with helpful indents to show subtrees.
This retains the preorder printing as an option for people who prefer that. I'm pretty sure it'll be easier to debug tests if we can see the postorder, so I'm making that the default.
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
In summary - some of the changes here are focused on producing the same test results for SemanticsIR, but I think the next step will be to change the SemanticsIR structure to reduce how much is added to the traversal stack.
Switching semantics to a postorder traversal is intended to be more efficient. The traversal stack is to eliminate risk of recursion limits within the semantic analysis that could come from layered code structures. However, we need to start considering the implications for type-checking and what the ParseTree looks like, as well as copying of data here.
As we start thinking about type-checking in SemanticsIR, it's helpful for a function to know its own signature in order to perform lookup recursive calls. The challenge in the post-order walk without this change is it doesn't know it's in a function definition (or similar) until it reaches the FunctionDeclaration; this restructures so that either:
1. For a declaration, the signature is a child of FunctionDeclaration(";")
2. For a definition, the signature is a child of FunctionDefinitionStart("{") which pairs with FunctionDefinition("}"), replacing CodeBlock.
This similarly reorients CodeBlock to be CodeBlockStart("{") as the first child of CodeBlock("}"). I'm not doing that with ParameterList here just because it affects a bit more, and felt like it could be delayed.
Overall, my goal is making the postorder traversal more intuitive along scope boundaries. I think we may also not need subtree_size, so I'm avoiding use of that now.
Currently the SemanticsIRFactory implementation is less clean than I might like (there are a couple comments to this point), but I was starting to feel like a more complete rewrite would be appropriate rather than trying to clean it up further: in particular, I think the node structures are off, but changing them is significant and also changes test output; in turn it may also warrant more substantial ParseTree changes. If you prefer from a reviewer POV, I can do a more complete rewrite.
This switch is being done in order to make it easier to update tests when the parse tree structure changes. I've been spending a lot of time doing such updates as part of refactoring the parse tree, and I expect more, so that's really at the root of all the refactoring I've been doing to lit updates.
Jon Ross-Perkins
