#2569 added PrintAsID to //common/ostream.h, but given it's
explorer-specific behavior, I don't think it's the right home for it.
Noticed this while pondering better ostream interfaces.
The main motivation for this is to get python loads in using the
`native-py` lint fix. However, enabling that made me wonder, maybe we
should fix in general?
`native-cc` is delayed, but not wholly cancelled (and `native-py`
picking up might indicate `native-cc` won't be too far behind). There's
also some automated fixes for `.append` and dict sorting -- this felt
okay to me, maybe not something to eagerly add but probably not worth
stopping buildifier from fixing (I've noticed the warnings in the past
and had been ignoring them).
Running everything does mean that load orders are sorted automatically
now, which I think is a positive. Most generally, I think these fixes
aren't _harmful_, and having them done automatically seems beneficial:
my biggest concern about `native-py` and `native-cc` was actually that
regressions wouldn't be caught, but this addresses that issue
automatically.
Add state (stack and heap information) to the trace stream only when
there is a change in state, this would prevent consecutive repetition of
state information in the trace output.
Checks it based on change in size of the stack and heap, added
`std::pair<int, int> stack_heap_sizes` in `Interpreter` to store size of
stack and heap for comparison.
Adds the following information about name resolution to the trace
output:
* Name resolution process
```
** resolving stmt | decl `<stmt | decl>` (<source_location>)
...
** finished resolving stmt | decl `<stmt | decl>` (<source_location>)
```
* When a name is added to the static scope
```
--- declared `<name>` as `<entity>` in `<scope>` (<source_location>)
```
* When a name is resolved
```
--- resolved `<name>` as `<result>` in `<scope>` (<source_location>)
```
* Marking a name declared/usable
```
--- marked `<name>` <usable | declared but not usable> in `<scope>`
```
Prevent copies when initializing value expression from reference
expression. This is based on
https://github.com/carbon-language/carbon-lang/pull/2006, which
introduces expression categories, and how it is possible to convert
to/from those different categories. Continuation of
https://github.com/carbon-language/carbon-lang/pull/2907
## Functional changes
* Initializing a value expression from a reference expression takes its
value without a copy
* Reading from the value expression causes an error if the value changed
from the time it was initialized
* In this situation, prevents a copy both for variable definitions, and
call parameter bindings
## Main implementation changes
* Add new `ExpressionCategoryAction`, which evaluates an expression and
returns an `ExpressionValue` containing its category and address (if
any), in addition to the resulting `Value*`
* `ExpressionAction`s now invokes `ExpressionCategoryAction` and unwraps
the returned `ExpressionValue`
* `RuntimeScope::BindAndPin` method, and corresponding when attempting
to read a `value_node`.
## Next work
* Avoid unnecessary copies from value expression to value expression,
after ensuring that even value expression temporaries are registered for
destruction (https://github.com/Pixep/carbon-lang/pull/9)
Add partial support for initializing expressions for variable declaration. This is based on https://github.com/carbon-language/carbon-lang/pull/2006, which introduces expression categories, and how it is possible to convert to/from those different categories.
## Functional changes
* Initializing expressions initialize directly the provided storage when used to initialize a variable.
* Allows initializing expressions to avoid a copy when using `[var|let] name: type = call_expression(...)` by initializing `name` in-place.
* Support `returned var: ...` and `return <expr>`
* Support nested initializing expressions
## Main implementation changes
* Updated PatternMatch logic to handle expression categories
* Updated `VariableDefinition` interpreter statement to allocate and pass a location to initializing expressions
* Update statement actions to allow passing an allocation, used by return expr or returned var
* Modified the RuntimeScope API to be one step closer to the memory model we want to have
* Remove `GetAllocationId` and older `Bind` which don't apply
* New set of tests to highlight those different situations
* Added a new intrinsic to print the allocation stack (and make sure we behave correctly, beyond visible side effects)
## Next work
* Dedicated `Action` to retrieve expression category information in the interpreter (https://github.com/carbon-language/carbon-lang/pull/2927)
* Avoid copies when initializing value expression from reference expression and prevent mutations for the duration of the "pinning" (https://github.com/carbon-language/carbon-lang/pull/2927)
* Avoid unnecessary copies from value expression to value expression, after ensuring that even value expression temporaries are registered for destruction.
* Avoid unnecessary copies when binding function arguments
By implementing these improvements, users will have the ability to choose specific parts of the trace output.
Currently, when executing a file using the explorer with the --trace_file=- or --trace_file=filename.txt flag, the resulting output is an extensive and verbose log containing all the information.
In this PR, I have introduced the `ProgramPhase` enum class, which have distinct phases encountered during the compilation of a program in the explorer. Each member of this enum class corresponds to a specific phase, signifying the relevant information to be included in the trace output.
The phases covered by the `ProgramPhase` enum class are as follows:
1. Printing the source program
2. Name resolution
3. Control flow resolution
4. Type checking
5. Unformed variable resolution
6. Printing declarations
7. Printing the timings
8. Printing whole output.
These phases can be selected by passing the following compiler flags along with `--trace_file=-`.
`-trace_source_program`, `-trace_name_resolution`, `-trace_control_flow_resolution`, `-trace_type_checking`, `-trace_unformed_variables_resolution`, `-trace_declarations`, `-trace_execution`, `-trace_timing` and `-trace_all`. If none of these flags is passed only execution trace will be added to the output.
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Per #257, we should be treating unformedness as all-or-nothing, rather than being a per-field or per-array-element property. Previously we initialized an array with no explicit initializer as containing a sequence of uninitialized values, but that led to crashes when attempting to access those values, as the checks for reading an uninitialized value only expected values to be uninitialized at the top level.
Also, we had existing tests that attempt to store to an element of an uninitialized array. We now detect that and treat it as UB during evaluation, rather than crashing due to trying to perform field access into an uninitialized value.
Finally, many of these problems can be detected statically, but the resolve_unformed pass wasn't catching them because it missed a few expression and declaration forms. Support for those cases has been added too. This causes the pass to recurse more often, and in particular our existing recursion test started hitting a stack overflow after this, so resolve_unformed now uses `RunWithExtraStack`. In passing, remove the need to explicitly tell `RunWithExtraStack` the return type, and infer it as the return type of the callable instead.
Adds stack space handling similar to Clang's approach, but with more support for forwarding return values. Refactors ParseAndExecute for better sharing, and for centralization of the InitStackSpace handling.
Fixes#2795
Currently ParseAndExecute-style logic is done in main.cpp and a test. #2799 is adding another test that needs it, as is #2811.
Also more clearly marks fuzzing as testing.
This is being extracted out of #2799 in order to try unblocking progress while review continues.
Removes `__continuation`, `__await`, and `__run`.
In part here, the discussion was that while the feature had been useful for validating the early explorer design, it's no longer needed for that role as the explorer is now quite robust. Continuations have been experimental and, at this point, don't have an owner pushing to a proposal.
The triggering factor is that, as we push to address fuzzer issues, I ran into a crash bug in this code; basically, `fn Main() -> i32 { __await; return 0; }`. When I mentioned this, the reaction seemed to trend towards removal of the feature.
As described in [the generics design](https://github.com/carbon-language/carbon-lang/blob/trunk/docs/design/generics/details.md#type-structure-of-an-impl-declaration), `impl` declarations are prioritized by type structure. Given two `impl` declarations that match a `type as interface` query, the one that describes the longest prefix of the query without using placeholders is preferred.
We implement this by putting all impls in a total order, first by type structure equivalence classes and then by lexical order. When matching an impl, we walk this total order, and stop once we find a match and reach the end of its equivalence class.
Equivalence classes are determined by finding the locations of the "holes" (the positions where deduced parameters appear) within the type structure, viewed as a tree. Two impls are in the same equivalence class if their holes are in the same place, and equivalence classes are ordered based on a reverse lexicographical ordering of their holes.
Explorer doesn't keep the `Bindings` list for a parameterized type in any particular order, but the type structure rule requires that we consider them in lexical order. In order to support this, we now track an index on the declared parameters of each generic. This is a simple numbering of enclosing generic parameters, both on that generic and on all lexically enclosing generics.
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This was bugging me after I saw all the strings; it feels like this is why we have EnumBase on the toolchain side.
I've included the move of EnumBase to //common because I figured it's reasonable to evaluate together; if we don't want EnumBase in this case, it doesn't make sense to move.
These are used by the AST in lots of ways, and this resolves various layering issues.
This means that `AllocationId` also lives in ast/, but is managed by interpreter/. A better layering here would be desirable, but this seems good enough for the time being.
This is intended to address currently flaky timeouts that are likely caused by the size of the prelude. I'm addressing a performance bottleneck in AnalyzeProgram with trace output. Trying to omit prelude traces reduces most trace output significantly, and I think it'll scale better as the prelude size increases.
The basic mechanics here are:
- In order to consistently track whether tracing is on, I've added a TraceStream class, explorer/interpreter/trace_stream.h.
- The AST now has a num_prelude_declarations field, so that it's provided where the boundary is.
- In order to mark where we try to skip prelude output, I've added calls to set_in_prelude in type_checker.
- In exec_program, I just use num_prelude_declarations directly to skip over.
- Everywhere checks TraceStream::is_enabled before printing, similar to the std::optional check that was previously used.
This does add some timing output in order to better diagnose where slowness is coming from, when tracing. It also adds "verbose" targets to make it easier to get the trace output.
So for example, here's a timing for zero.carbon:
```
Timings:
- Parse: 13ms
- AddPrelude: 25ms
- AnalyzeProgram: 116ms
- ExecProgram: 12ms
```
If I make a small change to just not set skipping_prelude (essentially getting back to current output):
```
- Parse: 13ms
- AddPrelude: 25ms
- AnalyzeProgram: 2359ms
- ExecProgram: 57ms
```
Thus in this trivial example, I'm eliminating about 95% of the execution time.
Note this approach could still be refined in a few ways:
- We could add a flag to allow overriding in_prelude. It should be a small amount of work after this change. But it's a little consistent with how parser_debug works, that it won't print prelude output by default (unless there's an error).
- Execution could skip messages involving initialization of globals declared in the prelude. This is a little noisy right now, but I don't think it's significant for performance because ExecProgram is tiny.
- Once files are more separated, we should be able to change the num_prelude_declarations/set_in_prelude approach.
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Detect when evaluating an impl recursively tries to evaluate the same impl for the same or a more complex set of parameters.
In order to perform the check after we have tested that the type structure matches and before we check that constraints are recursively satisfied, argument deduction is extended to check structural matching properties earlier.
This requires us to separate match failures into two kinds: hard failures that produce errors that should never be swallowed, and soft failures such as a missing impl that lead us to merely discard an impl as a candidate. A flag has been added to `ImplScope` and `ArgumentDeduction` to specify whether soft failures should produce an error message or not.
Addresses comments from this discussion: https://github.com/carbon-language/carbon-lang/pull/2460#discussion_r1046444433
Features:
* Move subtyping logic from Interpreter to TypeChecker, exposing subtyping as a series of access to `.base`.
* Excludes function parameter conversion, which is still done in ::Convert due to parameters conversion being handled differently.
Changes:
* Add new `class BaseAccessExpression : public MemberAccessExpression`, allowing rewrites
* Handle `BaseAccessExpression` expression type in Interpreter
* Move subtyping logic to `TypeChecker::ImplicitlyConvert`
Add a generic mechanism to decompose a `Value` and rebuild it, and use that to implement `Substitute`'s recursive transformation of values instead of a hand-rolled decomposition. This means `Substitute` now covers all kinds of values, whereas previously it used to be unable to transform some values, and should be less work to add new kinds of value.
We can use the same mechanism for various other things: structural dumping of values, equality comparisons, and value instantiation in the interpreter would all benefit from this. But in this change I'm just switching `Substitute` to this as a first step.
Depends on #2361, #2421
Add support for destructors of base classes.
Features:
- Call destructors from derived to base class
- Support addressing `TupleValue` using a new `Member` variant
Changes:
- Update `StepDestroy()` to recursively call destructors from derived to base class
- Add new `Member` variant and `IndexedValue` struct to be usable with `TupleValue`
Features:
* Split `Member` class into 3 dedicated classes covering named, positional, and a new "base class" element
Changes:
* Rename `Member` to `Element` to better reflect the variants covered
* Add `NamedElement`, `PositionalElement`, and `BaseElement` child classes for `Element`
* Split `GetMember` into 3 function variants based on available attributes (index, name, nothing).
* Add some unit tests to provide coverage of core features
Motivation:
This changeset splits Member into (currently 2) classes, as we see the need for more Member variants (base class access needed for #2378, possibly unnamed mixins, ...), which in addition to the current ones, also have significantly different attributes. This will allow supporting more Member types in the future cleanly.
Alternatives considered:
The alternative solution, "one class for positional, named & base class access", would expose unused or unavailable attributes depending on the Member actual type (`index()` only for positional, `name()` only for named, and neither for base class access).
Relates to https://github.com/carbon-language/carbon-lang/issues/1881
- Add support for `.base` field in structs for [parent class initialization](https://github.com/carbon-language/carbon-lang/blob/trunk/docs/design/classes.md#constructors)
- Disabling base class initialization without `.base`
- Support class constructors (`Create() -> Self`) for base classes
- Direct access to base class(es) attributes with `object.var` remains unaffected
Changes:
- Add `TypeChecker::FieldTypesWithBase` to help assessing if a struct with `base` fields can be converted to a class
- Add a new `base_type()` attribute+getter to `NominalClassDeclaration` to as a first step to allow resolving parametrized classes
- Add a new `base` attribute+getter to `NominalClassValue` that contains the base class `NominalClassValue`. It is currently used mainly to get and set members of a class object.
- Add `Interpreter::ConvertClassWithBase` to build `NominalClassValue` from a init struct, that contains `.base` fields with either `NominalClassValue` or `StructValue`
- Add `FindClassField` to find a field in a class or its base classes
- Remove superfluous `ClassDeclaration::base()` in favor of `ClassDeclaration::base_type()`
Limitations;
- Though some work is done in that direction, parametrized base class where time is not know at the declaration site are not supported. Namely the example below does not compile
```
base class A(T:! Type) {}
class B(T:! Type) extends A(T) {}
```
But this one is functional already
```
base class A(T:! Type) {}
class B extends A(i32) {}
```
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Change explorer's handling of witnesses to track them wherever possible. Associated constant support needs witnesses to be available much more pervasively.
The main changes here are:
- `Substitute` now takes a set of bindings covering both `Value`s for `GenericBinding`s and `Witness`es for `ImplBinding`s, and substitutes both.
- Witnesses are now tracked within `ConstraintType`s. The `.Self` type has an `ImplBinding` that self-references from parts of the constraint to other parts of the constraint can use to access the witness for the constraint.
- The constraint type for an `impl` and for a `GenericBinding` are now stored in pre-substituted form, with references to the actual constrained type rather than symbolic references to `.Self`. This results in minor changes in diagnostic text.
Argument deduction still performs substitutions without remapping witnesses.
When checking for control flow falling off a function after a `match`, determine whether it's possible for no case to have matched. Using the same implementation, also detect whether `case`s in a `match` are unreachable.
For now, the implementation never considers a match against specific values for any type other than tuples, alternatives, and `bool` to be exhaustive. In particular, matching against the sole value `{}` of type `{}` is not considered exhaustive. This is probably best left until explorer supports matching on struct and maybe class types more generally.
This implementation closely follows the algorithm described in the paper [Warnings for pattern matching](http://moscova.inria.fr/~maranget/papers/warn/warn.pdf) by Luc Maranget. Various optimizations are possible, such as reducing the amount of copying done, but for the purposes of explorer, comprehensibility is being favored over efficiency.
The problem is, perhaps surprisingly, co-NP-hard (by reduction to the tautology problem for disjunctive normal form, which is in turn dual to the satisfaction problem for conjunctive normal form, which is well-known to be NP-hard). The algorithm is therefore exponential-time in the worst case, but seems to be well-studied and performs well enough on non-pathological examples. Nonetheless, a depth limit has been imposed to prevent pathological examples such as those generated by a fuzzer from causing long runtimes.
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
- An introprocedural forward analysis that checks the may-be-formed states on local variables.
- Returns compilation error on usage of must-be-unformed variables.
- Implemented as a pass of `ASTNode` traversal.
- Currently supports detection of: function parameter, return expression and rhs of assign.
We fundamentally have cyclic references between declarations,
statements, expressions, and patterns, and there's no meaningful
layering between them. Combine them into a single build target.
Components such as paren_contents, static_scope, and library_name that
are defined without reference to specific AST nodes are kept as separate
BUILD targets.
This avoids the need to make copies of potentially large maps when
the same bindings are used in multiple values, such as when forming
a bound method value.
Type checking now stores information on a member access identifying the member that was accessed, not only its name. This parallels what we do for other similar constructs whose meaning is resolved by name lookup or type-checking, and will allow us to avoid redoing lookups in some cases during interpretation.
`.Self` is modeled as a new kind of expression, `DotSelfExpression`. Name resolution associates each occurrence of a `DotSelfExpression` with a generic binding, much like for an `IdentifierExpression`.
Both `:!` bindings and `where` expressions bring `.Self` into scope. The tentative intent is that if there are multiple `.Self`s in scope and they refer to different bindings, the result of using a `.Self` expression is an ambiguity error, but that is not implemented in this patch. Instead, like for `IdentifierExpression`s, we find the innermost enclosing definition of `.Self`.
`.Foo` expressions are rewritten to `.Self.Foo`, but this isn't enough to make them do anything useful, because associated constants aren't supported in general yet.
Support is added for all of the non-type expression contexts where we currently accept built-in conversions, such as reordering the fields in a struct.
- Makes `Self` a keyword
- Inside `class Foo { `... `}`, `Self` means `Foo`
- Inside `class Foo(T:! Type, U:! Type) {` ...` }`, `Self` means `Foo(T, U)`
- `impl as Bar` means `impl Self as Bar`
- Inside `external impl Foo as Bar {`...`}`, not in a scope already defining `Self`, `Self` means `Foo`
Implemented by introducing a new kind of declaration, a `SelfDeclaration`, that is automatically added to class and impl declarations.
Co-authored-by: Jon Meow <jperkins@google.com>