Right now we expect crashes on invalid parses, we just don't try to
handle it in general even though the long-term intent is we should
handle semantics for bad parses. However, in theory, we should be
correctly handling code that parses as valid and that's probably more
interesting to fix bugs for. So this starts trying to fuzz that space of
valid parses.
Fuzzer corpus is based on explorer tests, with one merge run.
This also makes the filename a reference to the buffer since the line
seems better to have as a reference (versus copying a string per error).
Most tests now have different line deltas due to the extra output, but
the actual errors should overall stay the same.
Some of the error locations look like they could be improved, but this
change is only making it clear where they were before.
Handles tuples (including nested tuples) in the semantic phase of the
tool chain. Does not handle tuple element access yet.
---------
Co-authored-by: Farzana Ahmed Siddique <fasiddique@google.com>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This started with cleaning up the remaining Name/expression type punning
in the node stack, and grew. I'm factoring out a class because we've
previously expressed the desire to factor logic out of SemanticsContext
where possible, and this seemed like a reasonable cut.
NameExpression as the first node as a QualifiedExpression allows the
qualifier handling to consider Name in one less spot, an incremental
simplification. However, the additional complexity caused by this makes
me split ApplyNameQualifier/ApplyExpressionQualifier in order to avoid
repeat checks of the parse node's kind. The logic is still largely
shared, thus a couple helper functions. I think this is all fairly well
structured in the isolated class.
I can see that we may want to avoid passing SemanticsContext as an
argument in the future if it elides a step of lookup.
This adds a distinction between Unused and SoloParseNode, rather than
equating the two. This is intended to help identify nodes which are
getting pushed but maybe don't need to be.
Not totally done because I want to adjust declaration name handling due
to a quirk with how it mixes Name with Expression, but almost done. Once
that's done the type punning will be completely gone.
I think there's more we can do here, but this seemed like a good
checkpoint to make sure the path I'm going down is roughly what you
expected. There's one actual edit in if expression structure to match
the increased enforcement.
This handles namespacing of functions. Parsing and semantics are changed
significantly, while lowering works without changes. Variables can't be
namespaced yet because they're dealing with patterns, and I didn't dig
through that code.
Most of the logic is done through the new name declaration stack, which
is necessary because semantics isn't quite sure where the declaration
name ends. It'd be complex for parsing to send a signal about this,
probably involving node variants and rewrites of the tree, and this
solution seems to work well. Unfortunately this means a new stack, but
that may be inevitable due to the extra information needing to be
tracked.
Note this doesn't deal with scoped lookups of non-namespace things,
which we'll need for generics. That'll probably involve pushing resolved
scopes onto a stack (or maybe just setting a singleton value?) to affect
contextual name lookup. But, I think the basics are there to make it
work when we can test the behavior.
This renames "designator expression" to "qualified expression" and adds
"qualified declaration" in order to use terminology more consistent with
C++.
Namespaces will probably need to be considered for name mangling down
the line, but this still uses the basic name.
`semantics_handle.cpp` is a little on the large side, and is going to grow as we add new nodes. Some of the statement-specific parts have already been split into their own files. Split out the remaining such parts.
This PR renames parse nodes on a Name/NameExpression taxonomy. NameExpressions occur in a name context. The difference is that in non-expression contexts it's useful to return the identifier / string ID for adding to name lookup, whereas in expression contexts it's useful to return the resolved node ID for consistency with other expressions.
In the code, I do note SelfValueName is returned in the expression context: I'd expect this to change, as `self` in `[self: Self]` versus `self.Foo()` will probably be best handled similarly to the above. That means that, in the proposed taxonomy, both `SelfValueName` and `SelfValueNameExpression` will exist in order to assist semantics.
To contrast choices:
Original | Current | [zygoloid suggestion](https://discord.com/channels/655572317891461132/655578254970716160/1121581663399464970) | [This PR](https://discord.com/channels/655572317891461132/655578254970716160/1121814551789318215)
--- | --- | --- | ---
DeclaredName/DesignatedName | Identifier | NameComponent | Name
NameReference | NameReference | NameReference | NameExpression
SelfValueIdentifier | SelfValueIdentifier | SelfValueReference | SelfValueName
SelfTypeIdentifier | SelfTypeIdentiifer | SelfTypeReference | SelfTypeNameExpression
Add validation that every code block in a function is terminated by a sequence of terminating instructions, and that terminators don't appear anywhere else in code blocks.
This required tracking whether we're in a reachable code block. That's done on the fly when we create a new code block; the new `SemanticsNodeBlockId::Unreachable` is used to represent the case where we're not actually creating a code block because we're in unreachable code.
This is just a simplification: I think these different forms are getting in the way more than they're helping, particularly as I was looking into namespace functionality. The handling in semantics can be identical, providing a more uniform behavior.
This follows the same structure as `if` expressions, except that no
result value is needed.
Semantics IR building for code blocks is also added. Rather than popping
all the node stack entries we push for statements within a code block,
change statements and declarations to not push themselves onto the
stack. We're not notionally performing work recursively within prior
statements, and we don't need their value for anything, so it seems
cleaner to not push them. This also allows statements and declarations
to determine what syntactic context they're in by peeking at the top of
the stack, though that's not used in this patch.
Replacing direct raw_string_ostream uses. I figure the wrapper should be used more consistently.
There are still remaining raw_string_ostream uses that weren't compatible -- I'm continuing to look at those, but felt it was cleaner to have this on its own.
This is just a cleanup. Since we now have a testing directory, I think this is a better home for testonly libraries than //common. (I was thinking about this when I was considering adding more test_raw_ostream deps)
Add semantic analysis and semantics IR building for `if` expressions, and add the first parts of control flow handling to semantics IR. After discussion with @chandlerc, use [block arguments](https://en.wikipedia.org/wiki/Static_single-assignment_form#Block_arguments) to convey values from the two arms of the `if` to the result. For now, only a single block argument is supported, but we should revisit this as we explore more of the requirements of the Semantics IR form.
Functions can now contain multiple code blocks, so grab the entry block up-front instead of assuming the entry block will be at the top of the block stack when we reach the end of function emission.
Add trivial support for `bool` type literal, because without it we can't write testcases.
Now that the driver uses vfs, there's less reason for tests to do their own flow. Switch SemanticsIRTest to use the driver directly as an example simplification.
This adds vfs support to the toolchain, allowing Driver to take in-memory inputs in tests. As a consequence, I'm simplifying SourceBuffer: rather than allowing tests to pass in their own memory buffer, I'm using InMemoryFileSystem to push for greater consistency with production code. This does hit a quirk where I need to be careful about null terminator handling because fuzzer imports don't always have one, but that's probably more robust anyways.
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
We model `if a then b else` as a prefix operator for parsing precedence purposes. The rule that a statement starting with `if` is never an `if` expression is handled implicitly because the statement parser never invokes the expression parser for a statement starting with `if`.
This exposed a bug in our diagnosis of the whitespace rule for prefix operators, which was incorrectly being applied to non-symbolic operators in some cases, and was producing a bogus second diagnostic in some cases, which is also fixed here.
In explorer, we already support parsing a string_view, so use that. In toolchain, we need to build support, probably using vfs, so that's a todo.
bazel test //explorer:file_test --runs_per_test=5
- branch: Stats over 250 runs: max = 18.3s, min = 5.2s, avg = 11.2s, dev = 2.9s
- trunk: Stats over 250 runs: max = 22.3s, min = 5.9s, avg = 12.1s, dev = 2.8s
Not a dramatic improvement, but maybe more effective long-term, and this'd been requested on #2876
I'm looking at this as I start thinking about handling `import`. Syntax is based on llvm's `split-file` tool.
The `std::vector` -> `llvm::SmallVector` switch is minor, I'm doing it here because I had to touch everything anyways and I think for tests I'll lean slightly more towards the toolchain's way of doing things versus explorer's.
This shifts logic so that bindings are added to name lookup only after the scope is complete, removing logic around adding/removing/re-adding names in certain scopes.
This does mean that things like a function's forward declaration will need to go through an extra hoop for name conflict checks, because under this approach a function definition does conflict checking when it adds names for the body's use. But, that seems easy to address, and better than the current hoops.
I'm looking at adding CallableId, so figured I'd do this API refactoring which should reduce code duplication. This increases the amount of cross-calls between APIs because it may not be an issue for performance after optimizations, and should simplify reading of the API.
Also note the prior static_asserts on layout were missing a couple types, which is why I'm moving them into Entry where it's going to be more obvious when something's added.
This is a rename of a commonly used accessor. I tend to prefer shorter names, but it's semantics_ir in LoweringContext, and it feels odder to shorten to semantics there than to lengthen to semantics_ir here. I already have builtins_ir around here, too. I think it's helpful to use consistent names for semantics_ir since they're dealing with essentially the same thing.
This adds canonicalization of struct types based on their type fields. It obsoletes the current CanImplicitAsStruct because the type ids should now be identical when they're structurally identical; there's only a reason to implicit CanImplicitAsStruct to detect _compatible_ conversions.
The type fields themselves aren't canonicalized because it would need to be done during the first parse, and could yield name conflicts being associated with the wrong location. i.e.:
```
var x: {a: i32, a: i32};
var y: {a: i32, b: i32, a: i32};
```
This should yield two separate name conflict diagnostics pointing at the type fields for each respective line, but if struct type fields were canonicalized then both would point at the first `a: i32` field definition. This isn't expected to be an issue for types because I'm trying to print those, but we may also end up with a "first defined at" situation in some cases (still, less confusing because the type should match). Regardless, I think individual fields gets much more awkward.
This switches types to using SemanticsTypeId instead of SemanticsNodeId, and lowering pre-builds its list of types. The empty tuple type is special-cased because we don't want to emit it unless it's in-use, but as the implicit return for functions, it's frequently used. Callables use invalid to indicate the implicit return, and that seems undesirable to change due to the size increase.
We do use the empty tuple type for function returns, so I don't want to get rid of it, but the value is unused. With this change, builtins are all types. Long-term the empty tuple value should have a representation similar to structs; just a tuple value that's empty, not a built-in.
This removes special-casing of empty structs, handling them as just a regular value instead of a builtin. Note the `{} as Type` is still special-cased.
In lowering, removes the test of calling a function using `{}` because it's missing the proper load/store. This setup notices that error whereas the prior worked due to said special-casing. Fixing this will need to be done as part of generally adding loads for variable uses.
This is the first step to refactoring types into a SemanticsTypeId. This only tracks what's in-use, but as a consequence starts funneling type information through in ways similar to how I'd want it to do SemanticsTypeId.
Previously, IR for arguments in calls and struct values was separated out. This merges it back in. Additionally, parameters for functions and struct types had their own IR; the block is still there, but there's a TODO to decide what to do with it.
In the LLVM IR, this has the consequence of emitting expressions that are inputs to a call or struct value within the scope of the function, which is pretty much where it should be. Importantly it happens before the call is encountered.
This change also tinkers with the int and real literal lowering. I'm pretty sure both are still wrong, but was having trouble figuring out a "better" way to do it, and this seems like it'll work for now.
The intent of this change is that instead of paths looking like `explorer/testdata/foo/bar.carbon` (repo-relative), they're now just `bar.carbon` (local). The consequence is that paths should be a bit more durable in various environments, and just paths should be shorter and easier to read.
The explorer's prelude is an exception to this since it comes from data, rather than being the test target. Due to the change in approaches, it needs the regex again.
Uses #2829
- Moves most parts to //testing/lit_test to be consistent with //testing/file_test.
- Separates the autoupdate script out because it's shared between lit_test and file_test now, not lit-specific.
- Renames scripts to autoupdate_testdata (or autoupdate_lit_testdata for explorer's extra) to be more consistent with the non-lit-specific setup.
- Switches from execv to subprocess.call to head off a subtle issue regarding execution of multiple scripts, which we're likely to want in the future. Mostly in this PR because everything was already being touched.
- Removes autoupdate's dependency on merge_output in order to (a) better support the division of lit and non-lit logic and (b) remove a subprocess, for reasons similar to file_test's removal of subprocesses.
This echoes #2818 and the philosophy is mostly covered there.
Versus parsing, semantics uses fewer separate handler files (for now) because the logic has been shorter. However, the design is still intended to make it easy to split files along boundaries similar to the parser, as I've done for a couple more complex/inter-related sections.
This builds on #2814 by adding the test framework to the toolchain. On Linux, this is 7s -> 4s for me. #2811 has more detailed timing for the explorer, which also had more dramatic changes because it's about 3x more tests run in 2x as many ways (6x total).
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>