The lack of comments makes it difficult to be sure, but if any of these
are necessary, I'd like to make sure there are comments to explain why a
specific value was shown (which I do add for diagnostics_ in this
change).
Handles tuples (including nested tuples) in the semantic phase of the
tool chain. Does not handle tuple element access yet.
---------
Co-authored-by: Farzana Ahmed Siddique <fasiddique@google.com>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This started with cleaning up the remaining Name/expression type punning
in the node stack, and grew. I'm factoring out a class because we've
previously expressed the desire to factor logic out of SemanticsContext
where possible, and this seemed like a reasonable cut.
NameExpression as the first node as a QualifiedExpression allows the
qualifier handling to consider Name in one less spot, an incremental
simplification. However, the additional complexity caused by this makes
me split ApplyNameQualifier/ApplyExpressionQualifier in order to avoid
repeat checks of the parse node's kind. The logic is still largely
shared, thus a couple helper functions. I think this is all fairly well
structured in the isolated class.
I can see that we may want to avoid passing SemanticsContext as an
argument in the future if it elides a step of lookup.
I think there's more we can do here, but this seemed like a good
checkpoint to make sure the path I'm going down is roughly what you
expected. There's one actual edit in if expression structure to match
the increased enforcement.
This handles namespacing of functions. Parsing and semantics are changed
significantly, while lowering works without changes. Variables can't be
namespaced yet because they're dealing with patterns, and I didn't dig
through that code.
Most of the logic is done through the new name declaration stack, which
is necessary because semantics isn't quite sure where the declaration
name ends. It'd be complex for parsing to send a signal about this,
probably involving node variants and rewrites of the tree, and this
solution seems to work well. Unfortunately this means a new stack, but
that may be inevitable due to the extra information needing to be
tracked.
Note this doesn't deal with scoped lookups of non-namespace things,
which we'll need for generics. That'll probably involve pushing resolved
scopes onto a stack (or maybe just setting a singleton value?) to affect
contextual name lookup. But, I think the basics are there to make it
work when we can test the behavior.
This renames "designator expression" to "qualified expression" and adds
"qualified declaration" in order to use terminology more consistent with
C++.
Namespaces will probably need to be considered for name mangling down
the line, but this still uses the basic name.
- `xor` keyword is removed, with `^` used in its place.
- For consistency, also replaced unary `~` with unary `^`, as those
changes come from the same proposal.
- `type` keyword is added.
- To keep existing tests working, parsing support for `type` literal is
added too.
- `is` keyword is removed; we'd already added the `impls` keyword to the
list.
- Several other missing keywords added.
This PR renames parse nodes on a Name/NameExpression taxonomy. NameExpressions occur in a name context. The difference is that in non-expression contexts it's useful to return the identifier / string ID for adding to name lookup, whereas in expression contexts it's useful to return the resolved node ID for consistency with other expressions.
In the code, I do note SelfValueName is returned in the expression context: I'd expect this to change, as `self` in `[self: Self]` versus `self.Foo()` will probably be best handled similarly to the above. That means that, in the proposed taxonomy, both `SelfValueName` and `SelfValueNameExpression` will exist in order to assist semantics.
To contrast choices:
Original | Current | [zygoloid suggestion](https://discord.com/channels/655572317891461132/655578254970716160/1121581663399464970) | [This PR](https://discord.com/channels/655572317891461132/655578254970716160/1121814551789318215)
--- | --- | --- | ---
DeclaredName/DesignatedName | Identifier | NameComponent | Name
NameReference | NameReference | NameReference | NameExpression
SelfValueIdentifier | SelfValueIdentifier | SelfValueReference | SelfValueName
SelfTypeIdentifier | SelfTypeIdentiifer | SelfTypeReference | SelfTypeNameExpression
This is just a simplification: I think these different forms are getting in the way more than they're helping, particularly as I was looking into namespace functionality. The handling in semantics can be identical, providing a more uniform behavior.
Replacing direct raw_string_ostream uses. I figure the wrapper should be used more consistently.
There are still remaining raw_string_ostream uses that weren't compatible -- I'm continuing to look at those, but felt it was cleaner to have this on its own.
This is just a cleanup. Since we now have a testing directory, I think this is a better home for testonly libraries than //common. (I was thinking about this when I was considering adding more test_raw_ostream deps)
Add semantic analysis and semantics IR building for `if` expressions, and add the first parts of control flow handling to semantics IR. After discussion with @chandlerc, use [block arguments](https://en.wikipedia.org/wiki/Static_single-assignment_form#Block_arguments) to convey values from the two arms of the `if` to the result. For now, only a single block argument is supported, but we should revisit this as we explore more of the requirements of the Semantics IR form.
Functions can now contain multiple code blocks, so grab the entry block up-front instead of assuming the entry block will be at the top of the block stack when we reach the end of function emission.
Add trivial support for `bool` type literal, because without it we can't write testcases.
This adds vfs support to the toolchain, allowing Driver to take in-memory inputs in tests. As a consequence, I'm simplifying SourceBuffer: rather than allowing tests to pass in their own memory buffer, I'm using InMemoryFileSystem to push for greater consistency with production code. This does hit a quirk where I need to be careful about null terminator handling because fuzzer imports don't always have one, but that's probably more robust anyways.
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
We model `if a then b else` as a prefix operator for parsing precedence purposes. The rule that a statement starting with `if` is never an `if` expression is handled implicitly because the statement parser never invokes the expression parser for a statement starting with `if`.
This exposed a bug in our diagnosis of the whitespace rule for prefix operators, which was incorrectly being applied to non-symbolic operators in some cases, and was producing a bogus second diagnostic in some cases, which is also fixed here.
In explorer, we already support parsing a string_view, so use that. In toolchain, we need to build support, probably using vfs, so that's a todo.
bazel test //explorer:file_test --runs_per_test=5
- branch: Stats over 250 runs: max = 18.3s, min = 5.2s, avg = 11.2s, dev = 2.9s
- trunk: Stats over 250 runs: max = 22.3s, min = 5.9s, avg = 12.1s, dev = 2.8s
Not a dramatic improvement, but maybe more effective long-term, and this'd been requested on #2876
I'm looking at this as I start thinking about handling `import`. Syntax is based on llvm's `split-file` tool.
The `std::vector` -> `llvm::SmallVector` switch is minor, I'm doing it here because I had to touch everything anyways and I think for tests I'll lean slightly more towards the toolchain's way of doing things versus explorer's.
This is addressing an issue left behind by the context switch, removing a few diagnostics that had been in the header rather than figuring out proper homes. I'm splitting one for semis a little further, sharing one, and then the other two are actually able to be moved into more specific homes as-is (one is only used in one place, clearly an oversight that it wasn't there already).
The intent of this change is that instead of paths looking like `explorer/testdata/foo/bar.carbon` (repo-relative), they're now just `bar.carbon` (local). The consequence is that paths should be a bit more durable in various environments, and just paths should be shorter and easier to read.
The explorer's prelude is an exception to this since it comes from data, rather than being the test target. Due to the change in approaches, it needs the regex again.
Uses #2829
- Moves most parts to //testing/lit_test to be consistent with //testing/file_test.
- Separates the autoupdate script out because it's shared between lit_test and file_test now, not lit-specific.
- Renames scripts to autoupdate_testdata (or autoupdate_lit_testdata for explorer's extra) to be more consistent with the non-lit-specific setup.
- Switches from execv to subprocess.call to head off a subtle issue regarding execution of multiple scripts, which we're likely to want in the future. Mostly in this PR because everything was already being touched.
- Removes autoupdate's dependency on merge_output in order to (a) better support the division of lit and non-lit logic and (b) remove a subprocess, for reasons similar to file_test's removal of subprocesses.
The goal of this change is to start refactoring the monolithic file into separate files that will hopefully pose fewer conflicts for developers, and make it easier to skip to handling of specific functionality. It additionally addresses a scaling issue with parser.cpp where the file would continue to get larger as more features are added.
Switches Parser to a ParserContext, moves handlers to be free functions, and moves the controller logic into ParseTree. parser_handle_states.h does the declarations for handlers and little else; handlers are split out to individual files based on prefix (which is deliberately authored to cluster).
A couple things I'm avoiding based on historical discussion are:
- Having a subdirectory for all the handlers, such as `toolchain/parser/handlers/call_expression.cpp`
- Putting handlers in a namespace, such as `Carbon::ParserHandler::CallExpression`.
- The name of `Carbon::ParserHandlerCallExpression` is then necessary to minimize the chance of conflicts with semantics and lowering, where everything can be expected to be named similarly.
I'm globbing handlers because it seems hard to see missed ones under this approach -- names are too boilerplate.
I think this current setup could be split into target-per-file, but I'm not sure that's needed, so I'd delay until it becomes a build-time issue.
This builds on #2814 by adding the test framework to the toolchain. On Linux, this is 7s -> 4s for me. #2811 has more detailed timing for the explorer, which also had more dramatic changes because it's about 3x more tests run in 2x as many ways (6x total).
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
At present, `var *;` is a crash because it has errors that weren't being properly stored/handled. This changes the handling to give more to track invalid parses in patterns.
I'm suspicious this is responsible for some OOMs in fuzzing... Specifically, it dumps a really big parse tree, then the auto fuzzing system OOMs trying to cache the entire output in memory.
`self` wasn't being handled in expression logic, only parameter logic. This is needed for `self.x`.
Also renames SelfDeducedParameter to SelfIdentifier because it's parsed fairly consistently with Identifier; this better represents the situations where `self` will need to be handled, it's where `Identifier` is allowed (although `Identifier` could also refer to a type... but I still think `SelfDeducedParameter` is a more difficult to understand name).
This was bugging me after I saw all the strings; it feels like this is why we have EnumBase on the toolchain side.
I've included the move of EnumBase to //common because I figured it's reasonable to evaluate together; if we don't want EnumBase in this case, it doesn't make sense to move.
I've been heading this route with other parts of the parser because the overhead of adding enums and then switching on them felt tedious, and odd from a performance perspective to make calls when the caller knew the value to use. My leaning is towards this approach that makes it clearer what's actually different between the modes, and allows removing BraceExpressionKindToParserState. It's a mild code size decrease.
Also cleans up some comments about related parse nodes. Currently basic and not heavily validated.
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This makes it possible to specify both deduced and regular parameters on types. It reorganizes the handling of parameter lists in order to allow more reuse of code in this approach. Both functions and types use the new DeclarationNameAndParams handling. Overall the goal here is to take advantage of commonality in structure.
Regarding destructors, the likely approach would be to use ParameterListAsDeduced directly because `destructor` is a keyword with no declaration name and no regular parameters.
This is just a mild simplification to address repeat macro use. I'm hoping it makes it clearer and easier in parser_state.def to write down the multiple variants.
I'm writing these macros in a simple form that I think is easy to read, versus some complex macro recursion which I think is _possible_ but would be harder to reason. And, we don't really need arbitrary arg counts -- this is probably going to stay fairly limited long-term, although I could easily see something like a half dozen in some cases so maybe I'll be wrong and it'll go higher. But, I feel like these macros still make it easier to focus on the _intent_ of cases, rather than visually comparing each line for differences.
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
We could similarly add others -- this is intended to make it easy to add more that parse essentially the same.
The functionality expected is that types will use GetDeclarationContext in order to error on certain functionality in the declaration scope loop. e.g., with how constraints and interfaces currently don't allow definitions.
I've only moved out `package` because it's only valid on the top line. It might still be good to parse it later, but with slightly different logic because it would always be an error, and the declaration context isn't quite the right framing for that.
Also unifies some errors with `fn`.
Depends on #2646
Right now, deduced parameter handling is very narrow to `self` support. This folds it into pattern handling, which should eventually be a superset of deduced parameter support, so this will avoid more duplication of logic.
Note, this subtly adds handling of multiple deduced parameters, but not generic parameters (`:!`) so it's still not quite right.
The comments in parse_node_kind.def capture the change being made here.
Before:
```
// _external_: DeclaredName
// InterfaceBodyStart
// _external_: statements
// InterfaceBodyEnd
// InterfaceDefinition
```
After:
```
// InterfaceIntroducer
// DeclaredName
// InterfaceDefinitionStart
// _external_: declarations
// InterfaceDefinition
```
Really I just want to treat introduced things consistently. `var` defines my philosophy here: it doesn't always have a `DeclaredName`, so the `VarIntroducer` _must_ be the bounding node. By being consistent with that, I believe that overall the structure becomes easier to understand (that is, there are fewer inconsistencies to understand).
This also adds InterfaceDeclaration, since I think it can be predicted we'll have that, and it's helpful for making recover consistent with HandleDeclarationError.
Similarly, I'm also trying to standardize the loop processing a little with HandleDeclarationLoop. In the current approach, InterfaceDefinitionFinish isn't a necessary state, so I'm removing it.
Per [discussion](https://discord.com/channels/655572317891461132/655578254970716160/1078427629427904563), there's a preference for having the support this enables in the parser. For example, that the parser should detect and error on a non-default interface function's definition.
However, we do need to handle nesting of declarations. We could do that by making this a stack. I think though that it'll be more efficient to keep using state_stack_, since it'll be called in places which are a limited number of steps from the actual state. That may already be in cache since we frequently look at state_stack_, so I'm uncertain that maintaining an additional stack would be a net benefit.
Adjusts handling of class constants (`static const`) to use CamelCase. This probably better reflects how we use it in C++ code, treating as appropriate for CamelCase instead of under_score.
Fixes adding_children to be preorder in caller (not sure why this wasn't automated).
No automated changes.
Just opening files in vscode and seeing what clangd flags.
Some edits to ostream.h to stop it from getting flagged (the usage pattern means it's not always obviously used).
The intent here is that changes to prelude.carbon shouldn't break every test that expects some error from prelude.carbon; that would be too fragile. As a consequence, this effectively ignores the line number in prelude.carbon.
This is a little complex because we don't know which line in the original source file is actually causing the error, just that there is an error. Also, the previous look-behind approach required a fixed-with prefix, whereas we want a little more than that in order to capture the filename for comparison.
This would be hard to do with extra_check_replacement because the path to bazel.runfiles is complex to calculate. As a consequence, this is basically all new code.
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Reals were mostly handled, but this PR adds storage of them. It also switches a little towards the FloatingPointType semantic from TokenizedBuffer.
While real literals like `1.0` were handled, the type literals were not. This just adds `f64`, similar to how I also only support `i32`.
The String type literal wasn't used, so I've added support in lexer and parser. Per discussion with @zygoloid String might be renamed based on the newer type literal plan, but it's still String in explorer and the design, so this is just consistent.
The builtin_types.carbon tests the three basic types that are there right now. The test is added to both parser and semantics so that it's clear what the state is in both stages.
There were some notes about default constructors being required in parse_tree.h, but they don't seem to be. Removing the default constructor forces the explicit `::Invalid` where there's no value immediately being assigned, which works fine for existing code. I think this actually reduces the chance of accidents more than the prior default-construct-as-invalid approach.
Using the same const/constexpr done in EnumBase, adds Invalid and Builtin* values to replace Make functions that produced the same. This should make it clearer at call sites what the cost actually is, and reduces the syntactic overhead for MakeBuiltinReference in particular.
Really, this is that MakeBuiltinReference has been feeling pretty verbose, so I did that, and then one MakeInvalid is right next to it, and then obviously I should replace the other MakeInvalid for consistency...
Jon Ross-Perkins
Richard Smith
