The comment on FunctionParameterFinish is actually correct (`1. FunctionParameter`), just a typo in the implementation. It specifically failed parsing at the comma after the 2nd param, regardless of whether there were more params.
Fundamentally this `.Print()` is wrong for debug output at present because `.fixed_spelling()` can be empty. It's also inconsistent with other enums to use it. We frequently print tokens for debugging, and it's easy to forget to specify `.name()` there.
Diagnostics use formatv, so we can provide a format_provider and address it in one spot that way. It also makes it harder to just forget to do the right thing.
This somewhat abuses control-flow macro handling in `clang-format` to get the behavior we want. Fortunately, I don't think we're ever likely to want this for its intended purpose so it seems harmless to co-opt it like this. If we really wanted, we can narrow it to `if`-macros, but I picked the simpler option to start.
FWIW, I tried all the other macro formatting special cases to see if one would work but it didn't. Might be worth filing a feature request to get a `TypeDefinitionMacros` setting to compliment `TypenameMacros` and format like this does, but it seems (very) low priority.
Also make minor updates to the skeletal design in
docs/design/name_lookup.md following #2113, as there are no longer any prelude names that are made available to unqualified name lookup by default.
Add `type` to the keyword list in
docs/design/lexical_conventions/words.md, following #2360.
The goal here is to (significantly) reduce the boilerplate needed when defining classes that wrap enums, especially those managed with the `.def`-file style X-macros that are common in the toolchain.
This should also provide both better and more consistent functionality to those classes once ported over to it.
Initially, only `ParserState`, `SemanticsNodeKind`, and `SemanticsBuiltinKind` are ported as these were also the three that JonMeow ported in his original pull/2453 "option 5". This is heavily based on that version of the code.
Goals I was considering that influenced the design:
- Keep the individual enum-wrapping classes as simple and easy to read as possible. Especially important is keeping the `.def` files that are often filled with really important documentation clean and easy to maintain over time.
- Don't rely on computed `#include`s as that is an especially dark corner of the preprocessor and breaks some build systems.
- Have a really good API of the enum-wrapping class, including nice constant names for the values, easy printing, and even easy debugger-callable methods to get the name (as opposed to the integer value).
- Keep the API that users interact with in the base class as clean and easy to read as possible.
- Reduce the boiler plate for each instance of these as much as possible.
- Avoid excessive inline generated code or constants that would result in steady growth in object file sizes and linker effort doing deduplication.
These goals aren't always compatible, so we end up needing to pick a compromise between them when in tension. I think this version is a pretty good compromise.
The original version I started with already pull most of the API into a CRTP-style base class. This version pulls *all* of the common API. This is the main tool for getting consistency and avoiding duplication. However, connecting this base class to the individual enum wrappers is still difficult. Some specific changes here that try to do as much as possible there:
- Use a slightly fancier macro pattern to reduce the boilerplate of defining the raw `enum class` prior to the wrapper class.
- Use a macro to simplify naming the base class.
- Move the name table to a `.cpp` file to avoid every inclusion generating a complete copy of the strings (that the linker has to deduplicate). This is done with some care to sharply reduce the boilerplate needed in that `.cpp` file.
- Sink the name _API_ fully into the CRTP base class. This requires some significant complexity in the implementation, but all of that is hidden behind a single implementation detail macro, and the API itself is simple and readable. This also makes it much more reasonable to test the entire system a single time next to the base class.
This version also moves from constant factory functions to normal constants. This requires two batches -- first a declaration, and then a definition -- but the API result is significantly better and similar to the original option, the macro structure reduces the cost of these. Unfortunately that makes the adoption a bit noisy, but I think its worth the churn.
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
The problem I'm trying to solve is: `var x: i32 = x;`. This change makes it so that name lookup fails, by removing `x` from name lookup between the `=` and `;`.
`var x: i32` still adds to name lookup to handle future situations like `var (x: i32, x: i32);` which is still a redefinition of `x`; if we don't add `x` to name lookup, it gets harder to catch that example.
The VariableDeclaration/VariableInitializer refactor in ParseTree supports this by given a bracketing-like structure for semantics to cue that it's entering an initialization expression. With this, VariableInitializer can remove the name lookup and queue it to be restored. VariableDeclaration doesn't need to change too much since it's still bracketed by VariableIntroducer, and so we just traverse slightly differently.
Note this also incidentally changes a little about NameReference, that it's returning the storage consistently instead of the name. You can see this e.g. in global_lookup.carbon, `Assign(node8, node4): node2;` using node4 (VarStorage) instead of Node5 (BindName). Really either _could_ work, since from a BindName we can get to the VarStorage, and that may be reason to switch later if we find it preferable to have the BindName for whatever reason.
But the *actual* value in NameLookup is a BindName so that errors can associate with the _name_ instead of the "storage" parse node, which is currently the `:`. This is mainly for fail_duplicate_decl.carbon, which has a "Previous definition" note that points at the storage's parse node.
In theory we're doing a central registry so that we can ensure there's at least one test for each. This isn't doing that, but I'm trying to validate that the central registry isn't leading to duplicates or abandoned checks (and catching a couple of each).
This switches to single list storage of SemanticsNode. The driving motivation behind this is to simplify cross-references within a given IR. Types of nodes will frequently refer to other blocks. This causes a significant increase in the number of cross-references, which can become difficult to manage (and reason about). By reducing to a single list of nodes, cross-references are only needed when crossing IR boundaries.
Because cross-references now only have 2 things to track (IR and index), they can be a regular SemanticsNode and don't need further indirection. This wasn't motivating, but feels like it reinforces the simplification.
Note this isn't being used to deduplicate nodes, at least right now. That could lead to difficult-to-update situations, but also most nodes are associated with the underlying ParseTree::Node in order to track sources for diagnostics; as a consequence, nodes representing equal text in different source locations wouldn't be the same node. There may be future opportunities here, discussed with @zygoloid, but no action is taken at present.
We may eventually want to switch the storage of NodeBlocks to have `[start, end)` ranges instead of individual numbers, but I'm leaving that alone for now.
As an aside, I noticed I was accidentally overloading the copy constructor on SemanticsIR. I've added some disambiguation on that, but am not deleting the copy constructor per style advice (even though the type should never be copied due to storage size).
codespell tries to change `CrossReference -> cross-reference` so disabling it there.
Doing some sorting of functions / enums (generally speaking, I've been trying to keep these loosely lexically sorted for lack of a better ordering).
Also removes some code that seems to be dead, and a minor TODO comment fix.
A quick check suggests this isn't necessary -- probably because the implicit enum cast is used for comparisons. I think this adds a lot to the boilerplate feel of these types, so if we can remove it there's a lot less sharing to do.
This is somewhat based on the name vs Name difference, but I figured I'd split it out and just sweep up the API on the whole while looking at a different approach to #2453
Printing intermediate state should be helpful to be able to examine the input when debugging later steps.
Intermediate state is hard to trace from the individual libraries since they don't know whether `dump` is going to print the state, so this moves some trace logic into the driver which is better equipped to make the decision.
I think it'd be helpful to examine what the parse tree looks like in failure cases.
Note, I'm not sure that the behavior of the recovery situations is correct; the test had been asserting that the parse tree should indicate it's error free. However, this means that the only signal the the driver that the input is invalid is that the diagnostic emitter was used. I think it may be important to have it return a non-zero error to prevent compile, or we can turn these into warnings but then "requiring a space" is wrong.
Either way, that's a concern I have with the pre-existing recovery behavior: I'm just trying to highlight it as I make this change, because now the test is really that nodes aren't tagged with has_error.
Initially I'd added this to lexer, this includes parser and semantics. Also adds ComparableIndexBase to unify a few common cases where <> comparisons are supported.
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
LLVM's bazel build has changed a bit, so this updates the tree for that.
LLVM is also moving `llvm::Optional` to match the standard API, but it seemed simpler to just switch to `std::optional`.
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
- Finishes remaining "todo" parse nodes.
- Improving error recovery for invalid designators and structs, so that the parse tree still looks similar to a valid parse tree.
- Call expressions now have the thing being called as a child (of the start) instead of a sibling.
- Use of Start is replacing use of End in several parse nodes, like structs and call expressions.
- Adjusting documentation of parse node structures in an attempt to make it more consistent and understandable.
- The current state for interfaces and if/else is mostly being documented, not altered.
This works on multiple statements to make them better for the bracketing model. Stub nodes are added in more cases of invalid syntax, simply so that the semantics has reliably structured input. Comments in parse_node_kind.def now try to show the expected parse tree structure in postorder form.
This labels If, While, and For a little differently in parse nodes so that at the start of the postorder traversal, it'll already be available to semantics which structure is being processed. I need to do a little more with If in particular, but this felt like a reasonable stopping point.
While this makes significant parser changes, the changes to parser_state.def are minimal, mostly naming-related. The actual flow isn't substantively changed, just a couple minor names and the new As(If|While) state which allows distinguishing IfCondition and WhileCondition.
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
First change towards re-implementing interfaces using the new parser. I kept it small to make sure we are on the same page regarding stack states and how the parse tree should look like.
Co-authored-by: ergawy <kareem.ergawy@guardsquare.com>
This switches the Verify method to walk postorder so that we can see how much subtree_size is really used, and shift towards removing it. It also starts calling Verify.
Also, I think I'd lost the reserve/size check during Parser refactoring, so I'm putting that back in as part of Verify.
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
When there's no semicolon for an invalid EmptyDeclaration, rather than producing nothing, produce an EmptyDeclaration with the original location that led to the error.
Note this removes a direct edit (the only one) of the parse tree's error state. Elsewhere it's an indirection from adding an error node.
This is a first pass at what semantic type checking might look like. Types propagate along nodes, we use an InvalidType object when there's an error, and once there's an InvalidType we stop doing so much type checking.
This adds some RealLiteral handling in order to get type mismatches. I'm cautious about creating some real value for SemanticsIR (since the tokenized buffer version is a bit constrained), so I'm not doing that yet. But I will probably need to in order to maintain SemanticsIR having hermetic copies of its data, without a parse tree dependency.
This is necessary in order to use a bracketing approach; parsing doesn't know the contained format until it parses the first element, which we don't want to do look-ahead for. I think the bracketing is higher value than knowing the format before adding the node.
This changes `return`, `break`, and `continue` to treat the keyword as the "start" and semicolon as the "parent", essentially bracketing the keyword.
Pragmatically this is focusing on making `return` work with only one ParseNodeKind: because `return` and `;` now bracket the expression, we can tightly determine whether the `return` has arguments without looking at subtree size. However, it's possible that `break` and `continue` may in the future take some kind of label as an argument, so the consistency seems beneficial there too.
Note this eliminates the StatementEnd ParseNodeKind, as it's obsolete with this change.
While the Parser has similar divergent states, lists of Expressions tend to be handled more like this. I'm keeping the divergent start state in order to continue support of a distinct error, but I think this organization of FunctionParameter/FunctionParameterFinish will be less surprising.
Adds remaining expression support, and switches the default to Parser2.
Note, this doesn't delete the current Parser yet. I'll just do that in its own PR.
Adds `while` and most of the expression support. Splits apart the fixity test to demonstrate more closely which bits are still failing.
```
//toolchain/parser/testdata:basics/fail_paren_match_regression.carbon.test FAILED in 0.8s
//toolchain/parser/testdata:basics/function_call.carbon.test FAILED in 0.7s
//toolchain/parser/testdata:basics/package.carbon.test FAILED in 0.7s
//toolchain/parser/testdata:basics/structs.carbon.test FAILED in 0.7s
//toolchain/parser/testdata:basics/tuples.carbon.test FAILED in 0.6s
//toolchain/parser/testdata:basics/var.carbon.test FAILED in 0.7s
//toolchain/parser/testdata:for/fail_colon_instead_of_in.carbon.test FAILED in 0.7s
//toolchain/parser/testdata:for/fail_missing_in.carbon.test FAILED in 0.7s
//toolchain/parser/testdata:for/fail_missing_var.carbon.test FAILED in 0.7s
//toolchain/parser/testdata:for/nested.carbon.test FAILED in 0.6s
//toolchain/parser/testdata:for/simple.carbon.test FAILED in 0.8s
//toolchain/parser/testdata:function/definition/with_params.carbon.test FAILED in 0.8s
//toolchain/parser/testdata:operators/fixity_in_call.carbon.test FAILED in 0.7s
//toolchain/parser/testdata:operators/fixity_in_var.carbon.test FAILED in 0.8s
```
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
The ParseTree comments say that preorder is "easier to visualize and read". The problem is, both the ParseTree and Semantics need to operate on the postorder traversal: the ParseTree during construction, and the Semantics during processing. As a consequence, understanding the postorder traversal is important, but it's also very hard to decipher when presented preorder. This PR provides a way to see the postorder, with helpful indents to show subtrees.
This retains the preorder printing as an option for people who prefer that. I'm pretty sure it'll be easier to debug tests if we can see the postorder, so I'm making that the default.
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Also does `if` support, discussed refactorings, like PopState/PushState instead of edits.
With these changes, I'm to where I can start talking about what's still missing:
```
//toolchain/parser/testdata:basics/fail_invalid_designators.carbon.test FAILED in 0.7s
//toolchain/parser/testdata:basics/fail_paren_match_regression.carbon.test FAILED in 0.6s
//toolchain/parser/testdata:basics/function_call.carbon.test FAILED in 0.7s
//toolchain/parser/testdata:basics/package.carbon.test FAILED in 0.7s
//toolchain/parser/testdata:basics/structs.carbon.test FAILED in 0.7s
//toolchain/parser/testdata:basics/tuples.carbon.test FAILED in 0.6s
//toolchain/parser/testdata:basics/var.carbon.test FAILED in 0.7s
//toolchain/parser/testdata:for/fail_colon_instead_of_in.carbon.test FAILED in 0.6s
//toolchain/parser/testdata:for/fail_missing_in.carbon.test FAILED in 0.6s
//toolchain/parser/testdata:for/fail_missing_var.carbon.test FAILED in 0.7s
//toolchain/parser/testdata:for/nested.carbon.test FAILED in 0.6s
//toolchain/parser/testdata:for/simple.carbon.test FAILED in 0.6s
//toolchain/parser/testdata:function/definition/with_params.carbon.test FAILED in 0.7s
//toolchain/parser/testdata:operators/associative.carbon.test FAILED in 0.6s
//toolchain/parser/testdata:operators/fail_missing_precedence_and_or.carbon.test FAILED in 0.6s
//toolchain/parser/testdata:operators/fail_missing_precedence_or_and.carbon.test FAILED in 0.6s
//toolchain/parser/testdata:operators/fail_variety.carbon.test FAILED in 0.6s
//toolchain/parser/testdata:operators/fixity.carbon.test FAILED in 0.9s
//toolchain/parser/testdata:operators/missing_precedence_not.carbon.test FAILED in 0.6s
//toolchain/parser/testdata:operators/postfix_unary.carbon.test FAILED in 0.7s
//toolchain/parser/testdata:operators/prefix_unary.carbon.test FAILED in 0.7s
//toolchain/parser/testdata:while/basic.carbon.test FAILED in 0.6s
//toolchain/parser/testdata:while/fail_unbraced.carbon.test FAILED in 0.6s
```
This does modify a couple `if` tests to not test so much expression syntax -- that just seems like unrelated syntax.
At present, CHECK/FATAL print their own stack trace. This switches to just using std::abort for the stack trace, as well as the CHECK printing more completely.
This has a few consequences:
1) I'm now buffering the FATAL strings in order to print it later.
2) We now print the bug report message and program arguments on failure. This is part of pretty printing and was elided before.
3) We can now have pretty printing on FATAL, e.g. to show the stacks we're building in the parser.
The intent of this approach is to eliminate recursion limits as a barrier for the parser. While it may not be urgent to address, I want to avoid pouring effort into a parser approach that we don't think will be usable long-term.
Right now this is passing a minor set of tests. It's intended to be enough to show how I'm thinking about flow control for the parser. I'm manually switching back and forth because it seemed like the easiest approach that avoids duplicating tests.
This rewrites semantics towards a more pure instruction model, in pursuit of the simple instruction-style output.
I think I can get this approach to type-check as it goes along, but obviously this change doesn't prove that yet. I'm separating it out because it's a large rewrite of the semantics structure, tossing out a lot of what was there before. But I think it does help towards several requests, like setting up a clear path for consolidating duplicate identifiers and making the node style more standardized.
I expect to need to pass multiple args to function calls, that'd probably be storing vectors of args similar to how I'm showing identifiers and integer literals stored.
This removes the semantics namespace because (a) it was getting annoying writing the `::` everywhere, and (b) I think the leaning with Carbon is to avoid namespaces (@chandlerc asked not to put SemanticsIR/SemanticsFactory in a namespace, which is the crux of the issue). But, it's still necessary to avoid name conflicts so I just prefix everything with "Semantics" (still a lot of typing, but no `::`).
In summary - some of the changes here are focused on producing the same test results for SemanticsIR, but I think the next step will be to change the SemanticsIR structure to reduce how much is added to the traversal stack.
Switching semantics to a postorder traversal is intended to be more efficient. The traversal stack is to eliminate risk of recursion limits within the semantic analysis that could come from layered code structures. However, we need to start considering the implications for type-checking and what the ParseTree looks like, as well as copying of data here.
As we start thinking about type-checking in SemanticsIR, it's helpful for a function to know its own signature in order to perform lookup recursive calls. The challenge in the post-order walk without this change is it doesn't know it's in a function definition (or similar) until it reaches the FunctionDeclaration; this restructures so that either:
1. For a declaration, the signature is a child of FunctionDeclaration(";")
2. For a definition, the signature is a child of FunctionDefinitionStart("{") which pairs with FunctionDefinition("}"), replacing CodeBlock.
This similarly reorients CodeBlock to be CodeBlockStart("{") as the first child of CodeBlock("}"). I'm not doing that with ParameterList here just because it affects a bit more, and felt like it could be delayed.
Overall, my goal is making the postorder traversal more intuitive along scope boundaries. I think we may also not need subtree_size, so I'm avoiding use of that now.
Currently the SemanticsIRFactory implementation is less clean than I might like (there are a couple comments to this point), but I was starting to feel like a more complete rewrite would be appropriate rather than trying to clean it up further: in particular, I think the node structures are off, but changing them is significant and also changes test output; in turn it may also warrant more substantial ParseTree changes. If you prefer from a reviewer POV, I can do a more complete rewrite.
This switch is being done in order to make it easier to update tests when the parse tree structure changes. I've been spending a lot of time doing such updates as part of refactoring the parse tree, and I expect more, so that's really at the root of all the refactoring I've been doing to lit updates.
Jon Ross-Perkins
Richard Smith
Kareem Ergawy
