Files
carbon-lang/explorer/syntax
8d0f3364d8 Initial implementation of raw string literals (#1304)
* test cases for raw string literals

* raw string literal implementation

* match as block string if starting with triple ", and better error message for simple string

except for *#"""#*

* fix broken test case

block string  literal cannot be one line

* test cases for raw string literals

* raw string literal implementation

* match as block string if starting with triple ", and better error message for simple string

except for *#"""#*

* fix broken test case

block string  literal cannot be one line

* removed unused initial value

* rename flag to indicate multi-line string and remove comment

* use * to get value from std::optional

* clean-ups

* removed skip_scan flag and directly return in case of a single line string starting with #+\'\'\'

* Updated error message: simple string -> single-line string.

Co-authored-by: josh11b <josh11b@users.noreply.github.com>

* Updated test cases according to changes in error message

* Removed counting_hashtag flag.

* Implemented ScanHelper class to handle scanning

* Fixed explanation of ReadHashTags.

* Addressed PR comment.

* Clarify that scan_helper holds the source text.

* Addressed PR comments.

* Updated error messages in test cases.

* Added const keyword to return type of GetCurrentStr().

* addressed PR comments.

1. Moved ScanHelper class to lex_scan_helper.h and lex_scan_helper.cpp.
2. Moved ReadHashTags and Process* functions to lex_scan_helper.cpp. Moved YY_USER_ACTION, SIMPLE_TOKEN and ARG_TOKEN to lex_helper.h. Added a wrapper function YyinputWrapper to call static function yyinput in lexer.lpp.
3. Renamed ScanHelper with StringLexHelper.
4. Modified BUILD accordingly.
5. Renamed data members and functions.

* Addressed PR comments.

1. Adjusted order to keep ret usage close.
2. Used resize to construct the string to avoid creation of temp string.

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>

* Removed the multi_line flag and skip_read field to improve readability.

* Copied default parameter value to definition of UnescapeStringLiteral.

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>

* Copied default parameter value to definition of ParseBlockStringLiteral.

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>

* Prefix CARBON_ to SIMPLE_TOKEN and ARG_TOKEN macros.

* Rollback redefinition of arguments.

* Updated comment on the flex macro.

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>

* Updated wording.

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>

* Moved the EOF error out of the loop.

* Removed duplicated declaration.

* Changed type of `hashtag_num` and `leading_quotes` to int.

* Minor fix: string copy.

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>

* Added comment on YyinputWrapper.

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>

* Garmmar in comment.

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>

* Added check of eof before readling next char.

* Minor updates based on PR comments.

* Minor changes to address PR comments.

* Used a clearer way to calculate `hashtag_num` and `leading_quotes`. Switched back to indicate muti-line string with a flag.

* Directly copy StringRef for compilation error message.

* Make str_with_quote const as we don't change it.

Co-authored-by: josh11b <josh11b@users.noreply.github.com>

* Added TODO for unsupported cases.

* Fixed a typo.

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>

Co-authored-by: josh11b <josh11b@users.noreply.github.com>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
2022-06-22 15:38:47 -07:00
..
2022-05-12 17:25:43 -07:00
2022-05-12 17:25:43 -07:00
2022-05-12 17:25:43 -07:00

The code in this directory is responsible for translating Carbon source code to the AST defined in ast. It consists primarily of a Flex lexer defined in lexer.lpp and a Bison grammar defined in parser.ypp.

It is possible to define and test a new expression syntax without defining its semantics by using the UnimplementedExpression AST node type and the same techniques can be applied to other kinds of AST nodes as needed. See the handling of the UNIMPL_EXAMPLE token for an example of how this is done, and see unimplemented_example_test.cpp for an example of how to test it.

Precedence and associativity

The Bison expression grammar uses the precedence climbing method to model precedence and associativity, suitably modified to handle Carbon's partial precedence order without grammar ambiguities.

Consider this example precedence diagram:

graph BT
    %%{init: {'themeVariables': {'fontFamily': 'monospace'}}}%%
    minus["minus<br>-x"]
    mul>"mul<br>x * y"]
    add>"add<br>x + y"]
    mod["mod<br>x % y"]
    eq["eq<br>x = y"]

    eq --> add & mod
    add --> mul
    mul & mod --> minus

For each precedence level, we have up to three grammar productions:

  • foo_expression represents an expression at that precedence level or higher, and includes as productions all of the expression kinds that are immediately higher in the precedence graph:
    add_expression:
      mul_expression | add_lhs '+' add_operand ;
    
  • foo_operand represents an operand of a foo_expression that is not itself a foo_expression.
    eq_operand:
      add_expression | mod_expression ;
    
  • For left-associative operators, foo_lhs represents either a foo_operand or a foo_expression.
    add_lhs:
      add_operand | add_expression ;
    

The above approach leads to (benign) reduce-reduce conflicts. In our example precedence diagram, the expression -x == y has two different parses:

  • eq_expression
    • eq_operand
      • add_expression
        • mul_expression
          • minus_expression
            • -
            • x
    • ==
    • eq_operand
      • ...
        • y

and

  • eq_expression
    • eq_operand
      • mod_expression
        • minus_expression
          • -
          • x
    • ==
    • eq_operand
      • ...
        • y

These would invoke the same parsing actions, so the states can be combined, but Bison isn't smart enough to see that.

In order to eliminate these conflicts, if there are multiple paths through the precedence graph between a higher-precedence level foo and some lower precedence level bar -- that is, if there's a diamond in the precedence graph with foo at the top and bar at the bottom -- foo_expressions are excluded from all intermediate _expression productions on the diamond between foo and bar, and are added back in the downstream _operand productions in the diamond instead:

minus_expression:
  identifier | '-' identifier ;

// In the real grammar, trivial productions like this are inlined.
mul_operand:
  minus_expression ;
mul_lhs:
  mul_operand | mul_expression ;
// A minus_expression is not a mul_expression, even though it's a
// higher-precedence expression, because there are multiple paths from
// eq_expression to minus_expression, and this production is on such a path.
mul_expression:
  mul_lhs '*' mul_operand

// minus_expression is listed here because it is excluded from mul_expression.
add_operand:
  minus_expression | mul_expression ;
// This is notionally
//   add_operand | add_expression
// but that introduces another kind of reduce-reduce conflict, because there
// would be two ways to interpret a mul_expression as an add_lhs.
add_lhs:
  minus_expression | add_expression ;
// A mul_expression is an add_expression, because multiplication is
// higher-precedence, and mul is not at the top of a diamond in the precedence
// graph. minus_expression is excluded because we are within a diamond with it
// at the top.
add_expression:
  mul_expression | add_lhs '+' add_operand ;

mod_operand:
  minus_expression ;
mod_expression:
  mod_operand '%' mod_operand ;

// We add back minus_expression here because it was excluded from add_expression
// and mod_expression.
eq_operand:
  minus_expression | add_expression | mod_expression ;
// We also include minus_expression here because this is the bottom of the
// precedence diamond.
eq_expression:
  minus_expression | add_expression | mod_expression | eq_operand '=' eq_operand ;