Commit Graph
5 Commits
Author SHA1 Message Date
Vineeth B V 8c85fa2744 Added unicode support (#2027)
Can parse Unicode literals following the format \u{HHHH...} in a string
2022-08-19 13:39:11 -07:00
8d0f3364d8 Initial implementation of raw string literals (#1304)
* test cases for raw string literals

* raw string literal implementation

* match as block string if starting with triple ", and better error message for simple string

except for *#"""#*

* fix broken test case

block string  literal cannot be one line

* test cases for raw string literals

* raw string literal implementation

* match as block string if starting with triple ", and better error message for simple string

except for *#"""#*

* fix broken test case

block string  literal cannot be one line

* removed unused initial value

* rename flag to indicate multi-line string and remove comment

* use * to get value from std::optional

* clean-ups

* removed skip_scan flag and directly return in case of a single line string starting with #+\'\'\'

* Updated error message: simple string -> single-line string.

Co-authored-by: josh11b <josh11b@users.noreply.github.com>

* Updated test cases according to changes in error message

* Removed counting_hashtag flag.

* Implemented ScanHelper class to handle scanning

* Fixed explanation of ReadHashTags.

* Addressed PR comment.

* Clarify that scan_helper holds the source text.

* Addressed PR comments.

* Updated error messages in test cases.

* Added const keyword to return type of GetCurrentStr().

* addressed PR comments.

1. Moved ScanHelper class to lex_scan_helper.h and lex_scan_helper.cpp.
2. Moved ReadHashTags and Process* functions to lex_scan_helper.cpp. Moved YY_USER_ACTION, SIMPLE_TOKEN and ARG_TOKEN to lex_helper.h. Added a wrapper function YyinputWrapper to call static function yyinput in lexer.lpp.
3. Renamed ScanHelper with StringLexHelper.
4. Modified BUILD accordingly.
5. Renamed data members and functions.

* Addressed PR comments.

1. Adjusted order to keep ret usage close.
2. Used resize to construct the string to avoid creation of temp string.

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>

* Removed the multi_line flag and skip_read field to improve readability.

* Copied default parameter value to definition of UnescapeStringLiteral.

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>

* Copied default parameter value to definition of ParseBlockStringLiteral.

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>

* Prefix CARBON_ to SIMPLE_TOKEN and ARG_TOKEN macros.

* Rollback redefinition of arguments.

* Updated comment on the flex macro.

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>

* Updated wording.

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>

* Moved the EOF error out of the loop.

* Removed duplicated declaration.

* Changed type of `hashtag_num` and `leading_quotes` to int.

* Minor fix: string copy.

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>

* Added comment on YyinputWrapper.

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>

* Garmmar in comment.

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>

* Added check of eof before readling next char.

* Minor updates based on PR comments.

* Minor changes to address PR comments.

* Used a clearer way to calculate `hashtag_num` and `leading_quotes`. Switched back to indicate muti-line string with a flag.

* Directly copy StringRef for compilation error message.

* Make str_with_quote const as we don't change it.

Co-authored-by: josh11b <josh11b@users.noreply.github.com>

* Added TODO for unsupported cases.

* Fixed a typo.

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>

Co-authored-by: josh11b <josh11b@users.noreply.github.com>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
2022-06-22 15:38:47 -07:00
pk19604014 b36c006a01 Change proto_to_carbon to use \x for escaping in string literals to be compatible with Carbon string parsing (#1266) 2022-05-18 08:57:00 -07:00
Richard SmithandJon Meow e85f45de10 Move CHECK lines in tests next to the line that caused the output. (#1224)
Use FileCheck's `[[@LINE+n]]` mechanism to refer to the next line.

This is made awkward by a couple of things:

* We want to keep the `CHECK` lines in the original order.
* Errors are sometimes more than one line long.

The approach we use is to interleave the original lines and the check lines, putting each check line as early as possible subject to two rules:

1) Check lines never precede the 'AUTOUPDATE' line
2) Except when required by rule (1), a check line that refers to a source line by line number is never placed earlier than a source line that precedes that source line.

The actual `[[@LINE+n]]` annotations are created in a second pass after we've interleaved the lines so that we can work out the correct offsets.

Co-authored-by: Jon Meow <jperkins@google.com>
2022-05-03 15:48:56 -07:00
Jon Meow 309ec35f95 Rename executable_semantics to explorer (#1188)
Change generated with:

```
#!/usr/bin/bash -eux

# Helper script for renaming pending work.
# Run from the repo root.

# Rename executable_semantics in code.
sed -i 's/executable_semantics/explorer/g' \
  $(git grep -l 'executable_semantics' . | grep -v proposals)
sed -i 's/executable semantics/explorer/g' \
  $(git grep -l 'executable semantics' . | grep -v proposals)
sed -i 's/Executable semantics/Explorer/g' \
  $(git grep -l 'Executable semantics' . |  grep -v proposals)
sed -i 's/Executable Semantics/Explorer/g' \
  $(git grep -l 'Executable Semantics' . |  grep -v proposals)
sed -i 's/EXECUTABLE_SEMANTICS/EXPLORER/g' \
  $(git grep -l 'EXECUTABLE_SEMANTICS' . | grep -v proposals)
sed -i 's/ExecutableSemantics/Explorer/g' \
  $(git grep -l 'ExecutableSemantics' . | grep -v proposals)
sed -i 's/executable-semantics/explorer/g' \
  $(git grep -l 'executable-semantics' . | grep -v proposals)

# This is only needed for the initial move.
mv executable_semantics explorer
mv explorer/fuzzing/executable_semantics_fuzzer.cpp explorer/fuzzing/explorer_fuzzer.cpp
```

Verified with `bazel test ...`
2022-04-29 13:20:25 -07:00