In #3064, code was changed to look at a future token. This is an issue
because the parser is set up to enforce that tokens aren't used without
being consumed. That's part of #3118; related validation fails. Also,
since it's not necessarily the open paren that was consumed, it could be
a different opening symbol, which the closing symbol handling doesn't
check.
Under this approach, it's tracked whether an open paren was consumed,
and the open paren is associated with the state. That's more aligned
with how the parser expects to be fed information.
In paren condition handling for if and while, I'm also adding some
special casing for `if {` in particular to not assume the `{` is a
struct. I just think that this will come up somewhat often and the
resulting output is better this way (an error either way). I'm not doing
similar with `for` because there's already some `var` handling there,
and I'd need a little more time to think about structure -- whereas
right now I'm just trying to fix the crashes (`if {}`, `if []`, etc).
Fixes#3118
Also indicate what can appear within parentheses.
This is intended to be a clarification, not a design change. Note that
while we previously described the operand of `++` or `--` as being
simply an expression, the operand can never be anything other than the
kinds of expression the diagram now shows due to the expression category
rules added in #2006.
Fixes#3079.
- Only allow assignment at the top level in an expression statement.
- Allow both negation and complement as subexpressions of both bitwise
and numeric operators.
- Remove parsing support for postincrement and postdecrement.
- Add parsing support for `as` operator.
- Use the same ambient precedence for types and non-type expressions.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Refactor type canonicalization so that we can reuse the same code for
building a `T*` expression and for forming the type of an `&x`
expression.
Add basic computation of expression category in order to check that we
only take the address of durable reference expressions. This is
currently computed on demand rather than being tracked as part of the
semantics node, but in most cases can be determined by looking at only a
single expression, so caching it in the node doesn't seem worthwhile
yet. This decision should be revisited if we start doing more complex
category calculations.
Also add trivial lowering support, but it doesn't work properly yet
because lowering doesn't yet take the expression category into account.
Allow interleaving of STDOUT and STDERR check lines. Put STDOUT lines
after the line they're attached to, and STDERR lines before. If no
STDOUT check line is attached to any line, then put them all at the end
of the file instead.
This is intended to better handle the case where stdout contains
unreplaced mentions of line numbers, and also reflects that stdout is
typically a consequence of the test rather than commentary on it, so
placing it after the test seems likely to read better.
This also makes the filename a reference to the buffer since the line
seems better to have as a reference (versus copying a string per error).
Most tests now have different line deltas due to the extra output, but
the actual errors should overall stay the same.
Some of the error locations look like they could be improved, but this
change is only making it clear where they were before.
This started with cleaning up the remaining Name/expression type punning
in the node stack, and grew. I'm factoring out a class because we've
previously expressed the desire to factor logic out of SemanticsContext
where possible, and this seemed like a reasonable cut.
NameExpression as the first node as a QualifiedExpression allows the
qualifier handling to consider Name in one less spot, an incremental
simplification. However, the additional complexity caused by this makes
me split ApplyNameQualifier/ApplyExpressionQualifier in order to avoid
repeat checks of the parse node's kind. The logic is still largely
shared, thus a couple helper functions. I think this is all fairly well
structured in the isolated class.
I can see that we may want to avoid passing SemanticsContext as an
argument in the future if it elides a step of lookup.
This handles namespacing of functions. Parsing and semantics are changed
significantly, while lowering works without changes. Variables can't be
namespaced yet because they're dealing with patterns, and I didn't dig
through that code.
Most of the logic is done through the new name declaration stack, which
is necessary because semantics isn't quite sure where the declaration
name ends. It'd be complex for parsing to send a signal about this,
probably involving node variants and rewrites of the tree, and this
solution seems to work well. Unfortunately this means a new stack, but
that may be inevitable due to the extra information needing to be
tracked.
Note this doesn't deal with scoped lookups of non-namespace things,
which we'll need for generics. That'll probably involve pushing resolved
scopes onto a stack (or maybe just setting a singleton value?) to affect
contextual name lookup. But, I think the basics are there to make it
work when we can test the behavior.
This renames "designator expression" to "qualified expression" and adds
"qualified declaration" in order to use terminology more consistent with
C++.
Namespaces will probably need to be considered for name mangling down
the line, but this still uses the basic name.
- `xor` keyword is removed, with `^` used in its place.
- For consistency, also replaced unary `~` with unary `^`, as those
changes come from the same proposal.
- `type` keyword is added.
- To keep existing tests working, parsing support for `type` literal is
added too.
- `is` keyword is removed; we'd already added the `impls` keyword to the
list.
- Several other missing keywords added.
This PR renames parse nodes on a Name/NameExpression taxonomy. NameExpressions occur in a name context. The difference is that in non-expression contexts it's useful to return the identifier / string ID for adding to name lookup, whereas in expression contexts it's useful to return the resolved node ID for consistency with other expressions.
In the code, I do note SelfValueName is returned in the expression context: I'd expect this to change, as `self` in `[self: Self]` versus `self.Foo()` will probably be best handled similarly to the above. That means that, in the proposed taxonomy, both `SelfValueName` and `SelfValueNameExpression` will exist in order to assist semantics.
To contrast choices:
Original | Current | [zygoloid suggestion](https://discord.com/channels/655572317891461132/655578254970716160/1121581663399464970) | [This PR](https://discord.com/channels/655572317891461132/655578254970716160/1121814551789318215)
--- | --- | --- | ---
DeclaredName/DesignatedName | Identifier | NameComponent | Name
NameReference | NameReference | NameReference | NameExpression
SelfValueIdentifier | SelfValueIdentifier | SelfValueReference | SelfValueName
SelfTypeIdentifier | SelfTypeIdentiifer | SelfTypeReference | SelfTypeNameExpression
This is just a simplification: I think these different forms are getting in the way more than they're helping, particularly as I was looking into namespace functionality. The handling in semantics can be identical, providing a more uniform behavior.
Add semantic analysis and semantics IR building for `if` expressions, and add the first parts of control flow handling to semantics IR. After discussion with @chandlerc, use [block arguments](https://en.wikipedia.org/wiki/Static_single-assignment_form#Block_arguments) to convey values from the two arms of the `if` to the result. For now, only a single block argument is supported, but we should revisit this as we explore more of the requirements of the Semantics IR form.
Functions can now contain multiple code blocks, so grab the entry block up-front instead of assuming the entry block will be at the top of the block stack when we reach the end of function emission.
Add trivial support for `bool` type literal, because without it we can't write testcases.
We model `if a then b else` as a prefix operator for parsing precedence purposes. The rule that a statement starting with `if` is never an `if` expression is handled implicitly because the statement parser never invokes the expression parser for a statement starting with `if`.
This exposed a bug in our diagnosis of the whitespace rule for prefix operators, which was incorrectly being applied to non-symbolic operators in some cases, and was producing a bogus second diagnostic in some cases, which is also fixed here.
This is addressing an issue left behind by the context switch, removing a few diagnostics that had been in the header rather than figuring out proper homes. I'm splitting one for semis a little further, sharing one, and then the other two are actually able to be moved into more specific homes as-is (one is only used in one place, clearly an oversight that it wasn't there already).
The intent of this change is that instead of paths looking like `explorer/testdata/foo/bar.carbon` (repo-relative), they're now just `bar.carbon` (local). The consequence is that paths should be a bit more durable in various environments, and just paths should be shorter and easier to read.
The explorer's prelude is an exception to this since it comes from data, rather than being the test target. Due to the change in approaches, it needs the regex again.
Uses #2829
This builds on #2814 by adding the test framework to the toolchain. On Linux, this is 7s -> 4s for me. #2811 has more detailed timing for the explorer, which also had more dramatic changes because it's about 3x more tests run in 2x as many ways (6x total).
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
At present, `var *;` is a crash because it has errors that weren't being properly stored/handled. This changes the handling to give more to track invalid parses in patterns.
`self` wasn't being handled in expression logic, only parameter logic. This is needed for `self.x`.
Also renames SelfDeducedParameter to SelfIdentifier because it's parsed fairly consistently with Identifier; this better represents the situations where `self` will need to be handled, it's where `Identifier` is allowed (although `Identifier` could also refer to a type... but I still think `SelfDeducedParameter` is a more difficult to understand name).
Also cleans up some comments about related parse nodes. Currently basic and not heavily validated.
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This makes it possible to specify both deduced and regular parameters on types. It reorganizes the handling of parameter lists in order to allow more reuse of code in this approach. Both functions and types use the new DeclarationNameAndParams handling. Overall the goal here is to take advantage of commonality in structure.
Regarding destructors, the likely approach would be to use ParameterListAsDeduced directly because `destructor` is a keyword with no declaration name and no regular parameters.
We could similarly add others -- this is intended to make it easy to add more that parse essentially the same.
The functionality expected is that types will use GetDeclarationContext in order to error on certain functionality in the declaration scope loop. e.g., with how constraints and interfaces currently don't allow definitions.
I've only moved out `package` because it's only valid on the top line. It might still be good to parse it later, but with slightly different logic because it would always be an error, and the declaration context isn't quite the right framing for that.
Also unifies some errors with `fn`.
Depends on #2646
Right now, deduced parameter handling is very narrow to `self` support. This folds it into pattern handling, which should eventually be a superset of deduced parameter support, so this will avoid more duplication of logic.
Note, this subtly adds handling of multiple deduced parameters, but not generic parameters (`:!`) so it's still not quite right.
The comments in parse_node_kind.def capture the change being made here.
Before:
```
// _external_: DeclaredName
// InterfaceBodyStart
// _external_: statements
// InterfaceBodyEnd
// InterfaceDefinition
```
After:
```
// InterfaceIntroducer
// DeclaredName
// InterfaceDefinitionStart
// _external_: declarations
// InterfaceDefinition
```
Really I just want to treat introduced things consistently. `var` defines my philosophy here: it doesn't always have a `DeclaredName`, so the `VarIntroducer` _must_ be the bounding node. By being consistent with that, I believe that overall the structure becomes easier to understand (that is, there are fewer inconsistencies to understand).
This also adds InterfaceDeclaration, since I think it can be predicted we'll have that, and it's helpful for making recover consistent with HandleDeclarationError.
Similarly, I'm also trying to standardize the loop processing a little with HandleDeclarationLoop. In the current approach, InterfaceDefinitionFinish isn't a necessary state, so I'm removing it.
Reals were mostly handled, but this PR adds storage of them. It also switches a little towards the FloatingPointType semantic from TokenizedBuffer.
While real literals like `1.0` were handled, the type literals were not. This just adds `f64`, similar to how I also only support `i32`.
The String type literal wasn't used, so I've added support in lexer and parser. Per discussion with @zygoloid String might be renamed based on the newer type literal plan, but it's still String in explorer and the design, so this is just consistent.
The builtin_types.carbon tests the three basic types that are there right now. The test is added to both parser and semantics so that it's clear what the state is in both stages.
The comment on FunctionParameterFinish is actually correct (`1. FunctionParameter`), just a typo in the implementation. It specifically failed parsing at the comma after the 2nd param, regardless of whether there were more params.
Also make minor updates to the skeletal design in
docs/design/name_lookup.md following #2113, as there are no longer any prelude names that are made available to unqualified name lookup by default.
Add `type` to the keyword list in
docs/design/lexical_conventions/words.md, following #2360.
The problem I'm trying to solve is: `var x: i32 = x;`. This change makes it so that name lookup fails, by removing `x` from name lookup between the `=` and `;`.
`var x: i32` still adds to name lookup to handle future situations like `var (x: i32, x: i32);` which is still a redefinition of `x`; if we don't add `x` to name lookup, it gets harder to catch that example.
The VariableDeclaration/VariableInitializer refactor in ParseTree supports this by given a bracketing-like structure for semantics to cue that it's entering an initialization expression. With this, VariableInitializer can remove the name lookup and queue it to be restored. VariableDeclaration doesn't need to change too much since it's still bracketed by VariableIntroducer, and so we just traverse slightly differently.
Note this also incidentally changes a little about NameReference, that it's returning the storage consistently instead of the name. You can see this e.g. in global_lookup.carbon, `Assign(node8, node4): node2;` using node4 (VarStorage) instead of Node5 (BindName). Really either _could_ work, since from a BindName we can get to the VarStorage, and that may be reason to switch later if we find it preferable to have the BindName for whatever reason.
But the *actual* value in NameLookup is a BindName so that errors can associate with the _name_ instead of the "storage" parse node, which is currently the `:`. This is mainly for fail_duplicate_decl.carbon, which has a "Previous definition" note that points at the storage's parse node.
Doing some sorting of functions / enums (generally speaking, I've been trying to keep these loosely lexically sorted for lack of a better ordering).
Also removes some code that seems to be dead, and a minor TODO comment fix.
I think it'd be helpful to examine what the parse tree looks like in failure cases.
Note, I'm not sure that the behavior of the recovery situations is correct; the test had been asserting that the parse tree should indicate it's error free. However, this means that the only signal the the driver that the input is invalid is that the diagnostic emitter was used. I think it may be important to have it return a non-zero error to prevent compile, or we can turn these into warnings but then "requiring a space" is wrong.
Either way, that's a concern I have with the pre-existing recovery behavior: I'm just trying to highlight it as I make this change, because now the test is really that nodes aren't tagged with has_error.
- Finishes remaining "todo" parse nodes.
- Improving error recovery for invalid designators and structs, so that the parse tree still looks similar to a valid parse tree.
- Call expressions now have the thing being called as a child (of the start) instead of a sibling.
- Use of Start is replacing use of End in several parse nodes, like structs and call expressions.
- Adjusting documentation of parse node structures in an attempt to make it more consistent and understandable.
- The current state for interfaces and if/else is mostly being documented, not altered.
This works on multiple statements to make them better for the bracketing model. Stub nodes are added in more cases of invalid syntax, simply so that the semantics has reliably structured input. Comments in parse_node_kind.def now try to show the expected parse tree structure in postorder form.
This labels If, While, and For a little differently in parse nodes so that at the start of the postorder traversal, it'll already be available to semantics which structure is being processed. I need to do a little more with If in particular, but this felt like a reasonable stopping point.
While this makes significant parser changes, the changes to parser_state.def are minimal, mostly naming-related. The actual flow isn't substantively changed, just a couple minor names and the new As(If|While) state which allows distinguishing IfCondition and WhileCondition.
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
First change towards re-implementing interfaces using the new parser. I kept it small to make sure we are on the same page regarding stack states and how the parse tree should look like.
Co-authored-by: ergawy <kareem.ergawy@guardsquare.com>
When there's no semicolon for an invalid EmptyDeclaration, rather than producing nothing, produce an EmptyDeclaration with the original location that led to the error.
Note this removes a direct edit (the only one) of the parse tree's error state. Elsewhere it's an indirection from adding an error node.
This is necessary in order to use a bracketing approach; parsing doesn't know the contained format until it parses the first element, which we don't want to do look-ahead for. I think the bracketing is higher value than knowing the format before adding the node.
This changes `return`, `break`, and `continue` to treat the keyword as the "start" and semicolon as the "parent", essentially bracketing the keyword.
Pragmatically this is focusing on making `return` work with only one ParseNodeKind: because `return` and `;` now bracket the expression, we can tightly determine whether the `return` has arguments without looking at subtree size. However, it's possible that `break` and `continue` may in the future take some kind of label as an argument, so the consistency seems beneficial there too.
Note this eliminates the StatementEnd ParseNodeKind, as it's obsolete with this change.
Jon Ross-Perkins
Richard Smith
Farzana Ahmed SiddiqueandFarzana Ahmed Siddique
Geoff RomerandRichard Smith
Kareem Ergawy
