Right now //explorer/fuzzing:explorer_fuzzer takes my machine 80s to run, just because of the corpus size. The corpus is actually pretty small compared to the toolchain fuzzer, so reducing the corpus size doesn't feel quite right.
This adds support for sharding fuzz tests, and with 8 shards each is closer to 10s. This should put it closer to the noise of other explorer tests in terms of runtime.
Unfortunately I'm not seeing a sharding flag in the llvm library, which seems fair. However, that's why I'm working around it by creating separate test targets per shard, then a suite to merge them back together. The use of `shard_count` for this is idiomatic for bazel rules; I'm using it so that switching implementations should be low-impact if that's ever needed.
Co-authored-by: Adrien Leravat <Pixep@users.noreply.github.com>
This is intended to address currently flaky timeouts that are likely caused by the size of the prelude. I'm addressing a performance bottleneck in AnalyzeProgram with trace output. Trying to omit prelude traces reduces most trace output significantly, and I think it'll scale better as the prelude size increases.
The basic mechanics here are:
- In order to consistently track whether tracing is on, I've added a TraceStream class, explorer/interpreter/trace_stream.h.
- The AST now has a num_prelude_declarations field, so that it's provided where the boundary is.
- In order to mark where we try to skip prelude output, I've added calls to set_in_prelude in type_checker.
- In exec_program, I just use num_prelude_declarations directly to skip over.
- Everywhere checks TraceStream::is_enabled before printing, similar to the std::optional check that was previously used.
This does add some timing output in order to better diagnose where slowness is coming from, when tracing. It also adds "verbose" targets to make it easier to get the trace output.
So for example, here's a timing for zero.carbon:
```
Timings:
- Parse: 13ms
- AddPrelude: 25ms
- AnalyzeProgram: 116ms
- ExecProgram: 12ms
```
If I make a small change to just not set skipping_prelude (essentially getting back to current output):
```
- Parse: 13ms
- AddPrelude: 25ms
- AnalyzeProgram: 2359ms
- ExecProgram: 57ms
```
Thus in this trivial example, I'm eliminating about 95% of the execution time.
Note this approach could still be refined in a few ways:
- We could add a flag to allow overriding in_prelude. It should be a small amount of work after this change. But it's a little consistent with how parser_debug works, that it won't print prelude output by default (unless there's an error).
- Execution could skip messages involving initialization of globals declared in the prelude. This is a little noisy right now, but I don't think it's significant for performance because ExecProgram is tiny.
- Once files are more separated, we should be able to change the num_prelude_declarations/set_in_prelude approach.
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
The design of choice types expects the declaration of an alternative to match the usage: if an alternative is declared as `None`, then it should be used as `None` not `None()`, and if it is declared as `None()` then it should be used as `None()` not `None`. Update explorer to match.
Also clean up the handling of choice types and alternative values a little in general, by moving away from identifying choice types and alternatives as strings and towards identifying them symbolically.
Closes#2422
Support for global variables is still missing; they're a bit more tricky because they use a pattern to introduce their name.
Prior to this change, explorer heavily relied on name comparisons to determine whether two declarations declare the same entity. Some of those instances are fixed in this PR, but more remain to be fixed, and some TODOs are added for some harder-to-fix instances.
First step towards permitting declarations within namespaces. Supports only functions within namespaces for now, with no way to call those functions except from within other such functions. Unqualified lookups within a function in a namespace look in that namespace first.
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Add support for user-defined assignment, as well as compound assignment and increment, following the design direction in pending proposal #2511.
Some of this isn't fully testable yet: because explorer doesn't properly support `impl` specialization, the blanket `impl`s in the prelude prevent types from customizing assignment.
Add basic support for `match_first` declarations to explorer, as described in https://github.com/carbon-language/carbon-lang/blob/trunk/docs/design/generics/details.md#prioritization-rule. The concrete syntax here is not yet approved, so `match_first` is renamed to `__match_first` for now, but the semantics are necessary to implement other approved features in explorer so we're intentionally getting a little ahead of the design here.
Addresses comments from this discussion: https://github.com/carbon-language/carbon-lang/pull/2460#discussion_r1046444433
Features:
* Move subtyping logic from Interpreter to TypeChecker, exposing subtyping as a series of access to `.base`.
* Excludes function parameter conversion, which is still done in ::Convert due to parameters conversion being handled differently.
Changes:
* Add new `class BaseAccessExpression : public MemberAccessExpression`, allowing rewrites
* Handle `BaseAccessExpression` expression type in Interpreter
* Move subtyping logic to `TypeChecker::ImplicitlyConvert`
In particular, this means that a type can implement `ImplicitAs(Type)` and have values of that type behave like types.
This implies that `()` and `{}` are no longer types. They are now values whose type is the result of converting `()` or `{}` to type `Type`, as has been discussed recently and seems to be the supported direction. This fixes various cases where these types were previously mishandled.
Note, not trying to address every last error, just some obvious/easy ones.
```
/usr/local/google/home/jperkins/dev/carbon-lang/common/string_helpers.cpp:200:13: warning: prefer transparent functors 'less_equal<>' [modernize-use-transparent-functors]
auto le = std::less_equal<const char*>();
^
/usr/local/google/home/jperkins/dev/carbon-lang/toolchain/semantics/nodes/function.h:22:45: warning: pass by value and use std::move [modernize-pass-by-value]
Function(ParseTree::Node node, NodeId id, llvm::SmallVector<NodeRef> body)
^
/usr/local/google/home/jperkins/dev/carbon-lang/explorer/ast/declaration.cpp:230:14: warning: static member accessed through instance [readability-static-accessed-through-instance]
return cast<SelfDeclaration>(declaration).name();
^
/usr/local/google/home/jperkins/dev/carbon-lang/explorer/interpreter/type_checker.cpp:1104:33: warning: std::move of the variable 'impl' of the trivially-copyable type 'ConstraintType::ImplConstraint' has no effect [performance-move-const-arg]
impl_constraints_.push_back(std::move(impl));
^
/usr/local/google/home/jperkins/dev/carbon-lang/explorer/interpreter/type_checker.cpp:1145:36: warning: std::move of the variable 'rewrite' of the trivially-copyable type 'ConstraintType::RewriteConstraint' has no effect [performance-move-const-arg]
rewrite_constraints_.push_back(std::move(rewrite));
^
/usr/local/google/home/jperkins/dev/carbon-lang/explorer/interpreter/type_checker.cpp:1156:32: warning: std::move of the variable 'context' of the trivially-copyable type 'ConstraintType::LookupContext' has no effect [performance-move-const-arg]
lookup_contexts_.push_back(std::move(context));
^
/usr/local/google/home/jperkins/dev/carbon-lang/explorer/fuzzing/fuzzverter.cpp:74:42: warning: argument name 'trace' in comment does not match parameter name 'parser_debug' [bugprone-argument-comment]
/*trace=*/false);
^
./explorer/syntax/parse.h:19:17: note: 'parser_debug' declared here
bool parser_debug) -> ErrorOr<Carbon::AST>;
^
/usr/local/google/home/jperkins/dev/carbon-lang/explorer/fuzzing/proto_to_carbon_test.cpp:34:55: warning: argument name 'trace' in comment does not match parameter name 'parser_debug' [bugprone-argument-comment]
const ErrorOr<AST> ast = Carbon::Parse(&arena, f, /*trace=*/false);
^
./explorer/syntax/parse.h:19:17: note: 'parser_debug' declared here
bool parser_debug) -> ErrorOr<Carbon::AST>;
^
/usr/local/google/home/jperkins/dev/carbon-lang/explorer/fuzzing/proto_to_carbon_test.cpp:42:41: warning: argument name 'trace' in comment does not match parameter name 'parser_debug' [bugprone-argument-comment]
&arena, f, source_from_proto, /*trace=*/false);
^
./explorer/syntax/parse.h:25:59: note: 'parser_debug' declared here
std::string_view file_contents, bool parser_debug)
^
/usr/local/google/home/jperkins/dev/carbon-lang/explorer/syntax/parse_test.cpp:27:60: warning: argument name 'trace' in comment does not match parameter name 'parser_debug' [bugprone-argument-comment]
ParseFromString(&arena, "file.carbon", FileContents, /*trace=*/false);
^
./explorer/syntax/parse.h:25:59: note: 'parser_debug' declared here
std::string_view file_contents, bool parser_debug)
^
/usr/local/google/home/jperkins/dev/carbon-lang/migrate_cpp/cpp_refactoring/var_decl.cpp:57:58: warning: string concatenation results in allocation of unnecessary temporary strings; consider using 'operator+=' or 'string::append()' instead [performance-inefficient-string-concatenation]
segments.push_back({type_loc_class, qual_str + " " + range_str});
^
/usr/local/google/home/jperkins/dev/carbon-lang/explorer/fuzzing/ast_to_proto_test.cpp:105:55: warning: argument name 'trace' in comment does not match parameter name 'parser_debug' [bugprone-argument-comment]
const ErrorOr<AST> ast = Carbon::Parse(&arena, f, /*trace=*/false);
^
./explorer/syntax/parse.h:19:17: note: 'parser_debug' declared here
bool parser_debug) -> ErrorOr<Carbon::AST>;
^
```
This does some more work to the run_clang_tidy.py wrapper script, and runs an example pass.
"again" because it's really the proto fuzzer changes that broke it, it had been working before.
"mostly" because there's still an issue within the proto fuzzer that it can't find "port/protobuf.h", i.e. https://github.com/google/libprotobuf-mutator/tree/master/port, but I'm still hesitant to add an include path there.
Implements basic support for rewrite constraints as proposed in #2173.
Support for associated constants and complex constraints in general is also made more robust: argument deduction now properly computes and substitutes witnesses, and interface declarations track a more correct description of the constraint that they introduce than the one we previously built.
Instead of forming a `SymbolicWitness` that contains an `Expression` when we can't directly resolve a witness to an `impl`, form different kinds of `Witness` values for the various situations:
- A `BindingWitness` witnesses that a type implements a constraint by reference to an `ImplBinding`.
- A `ConstraintWitness` witnesses that a type implements a constraint by reference to witnesses for each of the impl constraints within the constraint.
- A `ConstraintImplWitness` witnesses that a type implements a constraint by reference to a larger constraint which contains that constraint as an impl constraint.
Remove `SymbolicWitness` values and `InstantiateImpl` expressions, which are now unused. In order to remove the final usage of `SymbolicWitness`, I fixed a TODO to give `Self` the proper type within an interface declaration. I don't think this is an observable change.
No functional change intended.
* Multiplication and division have the same priority.
* A new builtin interface DivWith is added.
* In some tests expecting a compilation error (syntax error), the error
message now says it is expecting SLASH or binary *.
When checking for control flow falling off a function after a `match`, determine whether it's possible for no case to have matched. Using the same implementation, also detect whether `case`s in a `match` are unreachable.
For now, the implementation never considers a match against specific values for any type other than tuples, alternatives, and `bool` to be exhaustive. In particular, matching against the sole value `{}` of type `{}` is not considered exhaustive. This is probably best left until explorer supports matching on struct and maybe class types more generally.
This implementation closely follows the algorithm described in the paper [Warnings for pattern matching](http://moscova.inria.fr/~maranget/papers/warn/warn.pdf) by Luc Maranget. Various optimizations are possible, such as reducing the amount of copying done, but for the purposes of explorer, comprehensibility is being favored over efficiency.
The problem is, perhaps surprisingly, co-NP-hard (by reduction to the tautology problem for disjunctive normal form, which is in turn dual to the satisfaction problem for conjunctive normal form, which is well-known to be NP-hard). The algorithm is therefore exponential-time in the worst case, but seems to be well-studied and performs well enough on non-pathological examples. Nonetheless, a depth limit has been imposed to prevent pathological examples such as those generated by a fuzzer from causing long runtimes.
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
* Replay changes from pre-force-push mixin branch
* Base MixinPseudoType off of the new InterfaceType
* Add more test cases.
Also removed an unnecessary check that would have already been
handled by the parser.
* WIP detect member clashes during mixing mixins
* Implement fuzzer changes
* Implement member name clash check when mixing mixins
* Modify parser and lexer for experimental mixin feature
* Add comments
* Update explorer/testdata/mixin/simple-mix-in-mixin.carbon
Co-authored-by: josh11b <josh11b@users.noreply.github.com>
* Update explorer/testdata/mixin/use-mixin-method-in-class-method.carbon
Co-authored-by: josh11b <josh11b@users.noreply.github.com>
* Make code review changes
Co-authored-by: josh11b <josh11b@users.noreply.github.com>
To pick up proto updates (field renames etc.) and new carbon source samples added to testdata.
```
$ rm explorer/fuzzing/fuzzer_corpus/*
$ explorer/fuzzing/regen_corpus.py
```
Following #1191, add initial support for bitwise operators. Support for both
integer operations and for operator overloading via the interfaces specified in
#1191 is provided.
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Allows unformed state for local variables. Reports a run-time error when an unformed local variable is used.
- Added declaration without initialization for local variables in the parser.
- Made the init expression of VariableDefinition optional.
- Expanded (alive, dead) to (uninitialized, alive, dead) in the Heap.
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Basic support for declaring, specifying the values of, and using associated constants.
This is incomplete in various ways. For example, when checking whether a type satisfies a constraint, there is no check that its associated constants match those in the constraint, and name lookup into a value whose type is an associated constant is not supported yet.
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This PR implements the `returned var` for the explorer.
- Enabled `returned var` and `return var` syntax in lexer and parser.
- Split `Return` statement into `ReturnVar` and `ReturnExpression` to distinguish the two types of returns.
- Added logic in name resolution, type checking and interpretation to process `returned var` definition and `ReturnVar` statement.
We fundamentally have cyclic references between declarations,
statements, expressions, and patterns, and there's no meaningful
layering between them. Combine them into a single build target.
Components such as paren_contents, static_scope, and library_name that
are defined without reference to specific AST nodes are kept as separate
BUILD targets.
Type checking now stores information on a member access identifying the member that was accessed, not only its name. This parallels what we do for other similar constructs whose meaning is resolved by name lookup or type-checking, and will allow us to avoid redoing lookups in some cases during interpretation.
`.Self` is modeled as a new kind of expression, `DotSelfExpression`. Name resolution associates each occurrence of a `DotSelfExpression` with a generic binding, much like for an `IdentifierExpression`.
Both `:!` bindings and `where` expressions bring `.Self` into scope. The tentative intent is that if there are multiple `.Self`s in scope and they refer to different bindings, the result of using a `.Self` expression is an ambiguity error, but that is not implemented in this patch. Instead, like for `IdentifierExpression`s, we find the innermost enclosing definition of `.Self`.
`.Foo` expressions are rewritten to `.Self.Foo`, but this isn't enough to make them do anything useful, because associated constants aren't supported in general yet.
* initial implementation of new and delete
* remove requirement for empty heap from trace.carbon
* more tests
* fixed substitution for generic function types
Build constraint types from `where` expressions. This is very bare-bones; there's no support for constraining `.Self` or associated types yet, but degenerate cases not involving self-reference work.
`==` constraints are syntactically supported but not semantically verified yet, and they aren't used for anything.
Add scaffolding for constraints in general, and support specifically constraints formed by applying a `&` operator.
No support for constraints formed with `where` nor for named constraints at this point.