* test cases for raw string literals * raw string literal implementation * match as block string if starting with triple ", and better error message for simple string except for *#"""#* * fix broken test case block string literal cannot be one line * test cases for raw string literals * raw string literal implementation * match as block string if starting with triple ", and better error message for simple string except for *#"""#* * fix broken test case block string literal cannot be one line * removed unused initial value * rename flag to indicate multi-line string and remove comment * use * to get value from std::optional * clean-ups * removed skip_scan flag and directly return in case of a single line string starting with #+\'\'\' * Updated error message: simple string -> single-line string. Co-authored-by: josh11b <josh11b@users.noreply.github.com> * Updated test cases according to changes in error message * Removed counting_hashtag flag. * Implemented ScanHelper class to handle scanning * Fixed explanation of ReadHashTags. * Addressed PR comment. * Clarify that scan_helper holds the source text. * Addressed PR comments. * Updated error messages in test cases. * Added const keyword to return type of GetCurrentStr(). * addressed PR comments. 1. Moved ScanHelper class to lex_scan_helper.h and lex_scan_helper.cpp. 2. Moved ReadHashTags and Process* functions to lex_scan_helper.cpp. Moved YY_USER_ACTION, SIMPLE_TOKEN and ARG_TOKEN to lex_helper.h. Added a wrapper function YyinputWrapper to call static function yyinput in lexer.lpp. 3. Renamed ScanHelper with StringLexHelper. 4. Modified BUILD accordingly. 5. Renamed data members and functions. * Addressed PR comments. 1. Adjusted order to keep ret usage close. 2. Used resize to construct the string to avoid creation of temp string. Co-authored-by: Jon Ross-Perkins <jperkins@google.com> * Removed the multi_line flag and skip_read field to improve readability. * Copied default parameter value to definition of UnescapeStringLiteral. Co-authored-by: Jon Ross-Perkins <jperkins@google.com> * Copied default parameter value to definition of ParseBlockStringLiteral. Co-authored-by: Jon Ross-Perkins <jperkins@google.com> * Prefix CARBON_ to SIMPLE_TOKEN and ARG_TOKEN macros. * Rollback redefinition of arguments. * Updated comment on the flex macro. Co-authored-by: Jon Ross-Perkins <jperkins@google.com> * Updated wording. Co-authored-by: Jon Ross-Perkins <jperkins@google.com> * Moved the EOF error out of the loop. * Removed duplicated declaration. * Changed type of `hashtag_num` and `leading_quotes` to int. * Minor fix: string copy. Co-authored-by: Jon Ross-Perkins <jperkins@google.com> * Added comment on YyinputWrapper. Co-authored-by: Jon Ross-Perkins <jperkins@google.com> * Garmmar in comment. Co-authored-by: Jon Ross-Perkins <jperkins@google.com> * Added check of eof before readling next char. * Minor updates based on PR comments. * Minor changes to address PR comments. * Used a clearer way to calculate `hashtag_num` and `leading_quotes`. Switched back to indicate muti-line string with a flag. * Directly copy StringRef for compilation error message. * Make str_with_quote const as we don't change it. Co-authored-by: josh11b <josh11b@users.noreply.github.com> * Added TODO for unsupported cases. * Fixed a typo. Co-authored-by: Jon Ross-Perkins <jperkins@google.com> Co-authored-by: josh11b <josh11b@users.noreply.github.com> Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Explorer
explorer is an implementation of Carbon whose primary purpose is to act as a
clear specification of the language. As an extension of that goal, it can also
be used as a platform for prototyping and validating changes to the language.
Consequently, it prioritizes straightforward, readable code over performance,
diagnostic quality, and other conventional implementation priorities. In other
words, its intended audience is people working on the design of Carbon, and it
is not intended for real-world Carbon programming on any scale. See the
toolchain directory for a separate implementation that's
focused on the needs of Carbon users.
Overview
explorer represents Carbon code using an abstract syntax tree (AST), which is
defined in the ast directory. The syntax directory
contains lexer and parser, which define how the AST is generated from Carbon
code. The interpreter directory contains the remainder of the
implementation.
explorer is an interpreter rather than a compiler, although it attempts to
separate compile time from run time, since that separation is an important
constraint on Carbon's design.
Programming conventions
The class hierarchies in explorer are built to support
LLVM-style RTTI, and
define a kind accessor that returns an enum identifying the concrete type.
explorer typically relies less on virtual dispatch, and more on using kind
as the key of a switch and then down-casting in the individual cases. As a
result, adding a new derived class to a hierarchy requires updating existing
code to handle it. It is generally better to avoid defining default cases for
RTTI switches, so that the compiler can help ensure the code is updated when a
new type is added.
explorer never uses plain pointer types directly. Instead, we use the
Nonnull<T*> alias for pointers that are not nullable, or
std::optional<Nonnull<T*>> for pointers that are nullable.
Many of the most commonly-used objects in explorer have lifetimes that are
tied to the lifespan of the entire Carbon program. We manage the lifetimes of
those objects by allocating them through an Arena object,
which can allocate objects of arbitrary types, and retains ownership of them. As
of this writing, all of explorer uses a single Arena object, we may
introduce multiple Arenas for different lifetime groups in the future.
For simplicity, explorer generally treats all errors as fatal. Errors caused
by bugs in the user-provided Carbon code should be reported with the error
builders in error_builders.h. Errors caused by bugs
in explorer itself should be reported with
CHECK or FATAL.
Example Programs (Regression Tests)
The testdata/ subdirectory includes some example programs with
expected output.
These tests make use of LLVM's lit and FileCheck. Tests have boilerplate at the top:
// Part of the Carbon Language project, under the Apache License v2.0 with LLVM
// Exceptions. See /LICENSE for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
// RUN: explorer %s 2>&1 | \
// RUN: FileCheck --match-full-lines --allow-unused-prefixes=false %s
// RUN: explorer --trace %s 2>&1 | \
// RUN: FileCheck --match-full-lines --allow-unused-prefixes %s
// AUTOUPDATE: explorer %s
// CHECK: result: 0
package ExplorerTest api;
To explain this boilerplate:
- The standard copyright is expected.
- The
RUNlines indicate two commands forlitto execute using the file: one without--traceoutput, one with.- Output is piped to
FileCheckfor verification. - Setting
-allow-unused-prefixesto false when processing the ordinary output, and true when handling the--traceoutput, allows us to omit the tracing output from theCHECKlines, while ensuring they cover all non-tracing output. - Setting
-match-full-linesin both cases indicates that eachCHECKline must match a complete output line, with no extra characters before or after theCHECKpattern. RUN:will be followed by thenotcommand when failure is expected. In particular,RUN: not explorer ....%sis alitsubstitution for the path to the given test file.
- Output is piped to
- The
AUTOUPDATEline indicates thatCHECKlines will be automatically inserted immediately below by the./update_checks.pyscript. - The
CHECKlines indicate expected output, verified byFileCheck.- Where a
CHECKline contains text like{{.*}}, the double curly braces indicate a contained regular expression.
- Where a
- The
packageis required in all test files, per normal Carbon syntax rules.
Useful commands
./update_checks.py-- Updates expected output.bazel test ... --test_output=errors-- Runs tests and prints any errors.
Updating fuzzer logic after making AST changes
Please refer to Fuzzer documentation.
Trace Program Execution
When tracing is turned on (using the --trace_file=... option), explorer
prints the state of the program and each step that is performed during
execution.
State of the Program
The state of the program is printed in the following format, which consists of two components: (1) a stack of actions and (2) a memory.
{
stack: action1 ## action2 ## ...
memory: 0: valueA, 1: valueB, 2: valueC, ...
}
The memory is a mapping of addresses to values. The memory is used to represent
both heap-allocated objects and also mutable parts of the procedure call stack,
for example, for local variables. When an address is deallocated, it stays in
memory but !! is printed before its value.
The stack is list of actions separated by double pound signs (##). Each action
has the format:
syntax .position. [[ results ]] { scope }
which can have up to four parts.
- The
syntaxfor the part of the program to be executed such as an expression or statement. - The
positionof execution (an integer) for this action (each action can take multiple steps to complete). - The
resultsfrom subexpressions of this part. - The
scopeis the variables whose lifetimes are associated with this part of the program.
The stack always begins with a function call to Main.
In the special case of a function call, when the function call finishes, the
result value appears at the end of the results.
Step of Execution
Each step of execution is printed in the following format:
--- step kind syntax .position. (file-location) --->
- The
syntaxis the part of the program being executed. - The
kindis the syntactic category of the part, such asexp,stmt, ordecl. - The
positionsays how far alongexploreris in executing this action. - The
file-locationgives the filename and line number for thesyntax.
Each step of execution can push new actions on the stack, pop actions, increment the position number of an action, and add result values to an action.
Experimental feature: Delimited Continuations
Delimited continuations provide a kind of resumable exception with first-class continuations. The point of experimenting with this feature is not to say that we want delimited continuations in Carbon, but this represents a place-holder for other powerful control-flow features that might eventually be in Carbon, such as coroutines, threads, exceptions, etc. As we refactor the executable semantics, having this feature in place will keep us honest and prevent us from accidentally simplifying the interpreter to the point where it can't handle features like this one.
Instead of delimited continuations, we could have instead done regular continuations with callcc. However, there seems to be a consensus amongst the experts that delimited continuations are better than regular ones.
So what are delimited continuations? Recall that a continuation is a
representation of what happens next in a computation. In the abstract machine,
the procedure call stack represents the current continuation. A delimited
continuation is also about what happens next, but it doesn't go all the way to
the end of the execution. Instead it represents what happens up until control
reaches the nearest enclosing __continuation statement.
The statement
__continuation <identifier> <statement>
creates a continuation object from the given statement and binds the continuation object to the given identifier. The given statement is not yet executed.
The statement
__run <expression>;
starts or resumes execution of the continuation object that results from the given expression.
The statement
__await;
pauses the current continuation, saving the control state in the continuation
object. Control is then returned to the statement after the __run that
initiated the current continuation.
These three language features are demonstrated in the following example, where
we create a continuation and bind it to k. We then run the continuation twice.
The first time increments x to 1 and the second time increments x to 2,
so the expected result of this program is 2.
fn Main() -> Int {
var Int: x = 0;
__continuation k {
x = x + 1;
__await;
x = x + 1;
}
__run k;
__run k;
return x;
}
Note that the control state of the continuation object bound to k mutates as
the program executes. Upon creation, the control state is at the beginning of
the continuation. After the first __run, the control state is just after the
__await. After the second __run, the control state is at the end of the
continuation.
Continuation variables are currently copyable, but that operation is "shallow": the two values are aliases for the same underlying continuation object.
The delimited continuation feature described here is based on the
shift/reset style of delimited continuations created by Danvy and Filinsky
(Abstracting control, ACM Conference on Lisp and Functional Programming, 1990).
We adapted the feature to operate in a more imperative manner. The
__continuation feature is equivalent to a reset followed immediately by a
shift to pause and capture the continuation object. The __run feature is
equivalent to calling the continuation. The __await feature is equivalent to a
shift except that it updates the continuation in place.