* test cases for raw string literals
* raw string literal implementation
* match as block string if starting with triple ", and better error message for simple string
except for *#"""#*
* fix broken test case
block string literal cannot be one line
* test cases for raw string literals
* raw string literal implementation
* match as block string if starting with triple ", and better error message for simple string
except for *#"""#*
* fix broken test case
block string literal cannot be one line
* removed unused initial value
* rename flag to indicate multi-line string and remove comment
* use * to get value from std::optional
* clean-ups
* removed skip_scan flag and directly return in case of a single line string starting with #+\'\'\'
* Updated error message: simple string -> single-line string.
Co-authored-by: josh11b <josh11b@users.noreply.github.com>
* Updated test cases according to changes in error message
* Removed counting_hashtag flag.
* Implemented ScanHelper class to handle scanning
* Fixed explanation of ReadHashTags.
* Addressed PR comment.
* Clarify that scan_helper holds the source text.
* Addressed PR comments.
* Updated error messages in test cases.
* Added const keyword to return type of GetCurrentStr().
* addressed PR comments.
1. Moved ScanHelper class to lex_scan_helper.h and lex_scan_helper.cpp.
2. Moved ReadHashTags and Process* functions to lex_scan_helper.cpp. Moved YY_USER_ACTION, SIMPLE_TOKEN and ARG_TOKEN to lex_helper.h. Added a wrapper function YyinputWrapper to call static function yyinput in lexer.lpp.
3. Renamed ScanHelper with StringLexHelper.
4. Modified BUILD accordingly.
5. Renamed data members and functions.
* Addressed PR comments.
1. Adjusted order to keep ret usage close.
2. Used resize to construct the string to avoid creation of temp string.
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
* Removed the multi_line flag and skip_read field to improve readability.
* Copied default parameter value to definition of UnescapeStringLiteral.
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
* Copied default parameter value to definition of ParseBlockStringLiteral.
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
* Prefix CARBON_ to SIMPLE_TOKEN and ARG_TOKEN macros.
* Rollback redefinition of arguments.
* Updated comment on the flex macro.
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
* Updated wording.
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
* Moved the EOF error out of the loop.
* Removed duplicated declaration.
* Changed type of `hashtag_num` and `leading_quotes` to int.
* Minor fix: string copy.
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
* Added comment on YyinputWrapper.
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
* Garmmar in comment.
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
* Added check of eof before readling next char.
* Minor updates based on PR comments.
* Minor changes to address PR comments.
* Used a clearer way to calculate `hashtag_num` and `leading_quotes`. Switched back to indicate muti-line string with a flag.
* Directly copy StringRef for compilation error message.
* Make str_with_quote const as we don't change it.
Co-authored-by: josh11b <josh11b@users.noreply.github.com>
* Added TODO for unsupported cases.
* Fixed a typo.
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: josh11b <josh11b@users.noreply.github.com>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
I'm doing this to avoid macro name conflicts, following https://google.github.io/styleguide/cppguide.html#Preprocessor_Macros: "If you do export a macro from a header, it must have a globally unique name. To achieve this, it must be named with a prefix consisting of your project's namespace name (but upper case)."
Commands run:
```
sed -i 's/\(DCHECK\|CHECK\|FATAL\|MAKE_UNIQUE_NAME\|MAKE_UNIQUE_NAME_IMPL\|RAW_EXITING_STREAM\|RETURN_IF_ERROR\|RETURN_IF_ERROR_IMPL\|ASSIGN_OR_RETURN\|ASSIGN_OR_RETURN_IMPL\|DIAGNOSTIC_KIND\|RETURN_IF_STACK_LIMITED\)(/CARBON_\1(/g' $(git ls-files *.cpp *.h *.lpp *.ypp *.def ':!third_party')
sed -i 's/#undef DIAGNOSTIC_KIND/#undef CARBON_DIAGNOSTIC_KIND/' toolchain/diagnostics/diagnostic_registry.def
```
Note this isn't *quite* everything, but it's intended to be a large pass at everything:
```
╚╡git grep '#define ' *.cpp *.h *.lpp *.ypp *.def ':!third_party' | grep -v '#define CARBON' | grep -v _H_
explorer/syntax/lexer.lpp: #define YY_USER_ACTION \
explorer/syntax/lexer.lpp: #define SIMPLE_TOKEN(name) \
explorer/syntax/lexer.lpp: #define ARG_TOKEN(name, arg) \
explorer/syntax/parse_and_lex_context.h:#define YY_DECL \
migrate_cpp/cpp_refactoring/var_decl.cpp:#define ABSTRACT_TYPE(Class, Base)
migrate_cpp/cpp_refactoring/var_decl.cpp:#define TYPE(Class, Base) \
```
We may in particular want to do a pass to clean up #ifdef guards and make them be CARBON_ rooted.
* Initial implementation of block string literals following lexical_conventions/string_literals.md.
Enabled yyinput() in flex to implement parsing.
Added ParseBlockStringLiteral() helper to handler further transformations such as indenting.
Modified formar_grammar to support single-quoted strings to prevent a failure on lexer.lpp.
* Fixed _find_string_end quote parameter type int -> str.
* Update executable_semantics/syntax/BUILD
Co-authored-by: Geoff Romer <gromer@google.com>
* Addressed code review comments - split table-drived test into individual tests, renamed constants to match style guide.
* Addressed code review comments -- using EXPECT_THAT_EXPECTED() in tests, lexer comments and code cleanup.
Co-authored-by: Geoff Romer <gromer@google.com>
This is based on discussion on #732: that we should probably parse the invalid whitespace, then reject it as part of string validation, rather than having different parses. I worry the question of "how is this parsed" may lead to subtly unexpected results if we aren't consistent, so I'm switching the logic from the lexer to the unescape library (and also adjusting the list of rejected whitespace).
It was in my mind to add String in order to support libraries in `package`. `print` is added in order to have a String go to stdout. I've tried to do `print` in a way that won't be too hard to add other printable types, but it's probably also somewhat optional here -- that is, if desired, I could remove it. But it was a lot easier to doublecheck `\n` behavior with it, and I suspect it'll be helpful in other tests if it supports more value types.
On the side, this also fixes dereferencing in Pattern/Expression Print() calls, which I was noticing printing pointers instead of values. This may be another argument for moving away from passing pointers, since this seems to be a difficult-to-catch error.
Co-authored-by: Geoff Romer <gromer@google.com>
Jon Meow
pk19604014andGeoff Romer