From 30682750ce7befc2eb8bfb8f54901a6c8b029a1d Mon Sep 17 00:00:00 2001 From: Jon Meow <46229924+jonmeow@users.noreply.github.com> Date: Fri, 27 Aug 2021 15:43:14 -0700 Subject: [PATCH] Updated string literal design based on #199 (#732) I've modified the text from the proposal slightly, focusing the overview more on a design setup, but mostly kept the details. One important thing here is I noticed that raw tab characters are disallowed -- this was a little buried before, and I've now updated the list of characters allowed in a string to exclude tabs. Additionally, I've noted `\0D` in the list of escapes as explicitly invalid. Co-authored-by: Richard Smith --- docs/design/lexical_conventions/README.md | 2 +- .../lexical_conventions/string_literals.md | 348 ++++++++++++++++++ 2 files changed, 349 insertions(+), 1 deletion(-) create mode 100644 docs/design/lexical_conventions/string_literals.md diff --git a/docs/design/lexical_conventions/README.md b/docs/design/lexical_conventions/README.md index 8903ab698e52..413fdccb8076 100644 --- a/docs/design/lexical_conventions/README.md +++ b/docs/design/lexical_conventions/README.md @@ -37,7 +37,7 @@ A _lexical element_ is one of the following: - a [word](words.md) - a literal: - a [numeric literal](numeric_literals.md) - - TODO: string literals + - a [string literal](string_literals.md) - TODO: operators, comments, ... The sequence of lexical elements is formed by repeatedly removing the longest diff --git a/docs/design/lexical_conventions/string_literals.md b/docs/design/lexical_conventions/string_literals.md new file mode 100644 index 000000000000..fadd1d91455a --- /dev/null +++ b/docs/design/lexical_conventions/string_literals.md @@ -0,0 +1,348 @@ +# String literals + + + + + +## Table of contents + +- [Overview](#overview) +- [Details](#details) + - [Simple and block string literals](#simple-and-block-string-literals) + - [Escape sequences](#escape-sequences) + - [Raw string literals](#raw-string-literals) + - [Encoding](#encoding) +- [Alternatives considered](#alternatives-considered) +- [References](#references) + + + +## Overview + +Carbon supports both simple literals that are single-line using one double +quotation mark (`"`) and block literals that are multi-line using three double +quotation marks (`"""`). A block string literal may have a file type indicator +after the first `"""`; this does not affect the string itself, but may assist +other tooling. For example: + +```carbon +// Simple string literal: +var simple: String = "example"; + +// Block string literal: +var block: String = """ + The winds grow high; so do your stomachs, lords. + How irksome is this music to my heart! + When such strings jar, what hope of harmony? + I pray, my lords, let me compound this strife. + -- History of Henry VI, Part II, Act II, Scene 1, W. Shakespeare + """; + +// Block string literal with file type indicator: +var code_block: String = """cpp + #include + int main() { + std::cout << "Hello world!"; + return 0; + } + """ +``` + +The indentation of a block string literal's terminating line is removed from all +preceding lines. As a consequence, in the above `code_block` example, only +`std::cout` and `return` are indented in the resulting string, and by 4 spaces +each. + +Escape sequences introduced by a backslash (`\`) and are used to express special +character or code unit sequences, such as `\n` for a newline character. Raw +string literals are additionally delimited with one or more `#`; these require +an equal number of hash symbols (`#`) after the `\` to indicate an escape +sequence. Raw string literals are used to more easily write literal `\`s in +strings. Both simple and block string literals have raw forms. For example: + +```carbon +// Raw simple string literal with newline escape sequence: +var newline: String = "line one\nline two"; + +// Raw simple string literal with literal `\n`, not a newline: +var raw: String = #"line one\nstill line one"#; + +// Raw simple string literal with newline escape sequence: +var raw_newline: String = "#line one\#nline two"#; +``` + +## Details + +### Simple and block string literals + +A _simple string literal_ is formed of a sequence of: + +- Characters other than `\` and `"`. + - Only space characters (U+0020) are valid whitespace in a string literal. + Other whitespace, including tabs and newlines, are disallowed but parse + as part of the string for error recovery purposes. +- [Escape sequences](#escape-sequences). + - Each escape sequence is replaced with the corresponding character + sequence or code unit sequence. + - Similarly to invalid whitespace, invalid escape sequences such as `\z` + parse as part of the string. + +This sequence is enclosed in `"`s. For example, this is a simple string literal: + +```carbon +var String: lucius = "The strings, my lord, are false."; +``` + +A _block string literal_ starts with `"""`, followed by an optional file type +indicator, followed by a newline, and ends at the next instance of three double +quotation marks whose first `"` is not part of a `\"` escape sequence. The +closing `"""` shall be the first non-whitespace characters on that line. The +lines between the opening line and the closing line (exclusive) are _content +lines_. The content lines shall not contain `\` characters that do not form part +of an escape sequence. + +The _indentation_ of a block string literal is the sequence of horizontal +whitespace preceding the closing `"""`. Each non-empty content line shall begin +with the indentation of the string literal. The content of the literal is formed +as follows: + +- The indentation of the closing line is removed from each non-empty content + line. +- All trailing whitespace on each line, including the line terminator, is + replaced with a single line feed (U+000A) character. +- The resulting lines are concatenated. +- Each [escape sequence](#escape-sequences) is replaced with the corresponding + character sequence or code unit sequence. + +A content line is considered empty if it contains only whitespace characters. + +```carbon +var String: w = """ + This is a string literal. Its first character is 'T' and its last character is + a newline character. It contains another newline between 'is' and 'a'. + """; + +// This string literal is invalid because the """ after 'closing' terminates +// the literal, but is not at the start of the line. +var String: invalid = """ + error: closing """ is not on its own line. + """; +``` + +A _file type indicator_ is any sequence of non-whitespace characters other than +`"` or `#`. The file type indicator has no semantic meaning to the Carbon +compiler, but some file type indicators are understood by the language tooling +(for example, syntax highlighter, code formatter) as indicating the structure of +the string literal's content. + +```carbon +// This is a block string literal. Its first two characters are spaces, and its +// last character is a line feed. It has a file type of 'c++'. +var String: starts_with_whitespace = """c++ + int x = 1; // This line starts with two spaces. + int y = 2; // This line starts with two spaces. + """; +``` + +The file type indicator might contain semantic information beyond the file type +itself, such as instructions to the code formatter to disable formatting for the +code block. + +**Open question:** There is no concrete set of recognized file type indicators. +It would be useful to informally specify a set of well-known indicators, so that +tools have a common understanding of what those indicators mean, perhaps in a +best practices guide. + +#### Escape sequences + +Within a string literal, the following escape sequences are recognized: + +| Escape | Meaning | +| ------------- | -------------------------------------------------------- | +| `\t` | U+0009 CHARACTER TABULATION | +| `\n` | U+000A LINE FEED | +| `\r` | U+000D CARRIAGE RETURN | +| `\"` | U+0022 QUOTATION MARK (`"`) | +| `\'` | U+0027 APOSTROPHE (`'`) | +| `\\` | U+005C REVERSE SOLIDUS (`\`) | +| `\0` | Code unit with value 0 | +| `\0D` | Invalid, reserved for evolution | +| `\xHH` | Code unit with value HH16 | +| `\u{HHHH...}` | Unicode code point U+HHHH... | +| `\` | No string literal content produced (block literals only) | + +Hex characters (`H`) must be uppercase (`\xAA`, not `\xaa`). + +This includes all C++ escape sequences except: + +- `\?`, which was historically used to escape trigraphs in string literals, + and no longer serves any purpose. +- `\ooo` octal escapes, which are removed because Carbon does not support + octal literals; `\0` is retained as a special case, which is expected to be + important for C interoperability. +- `\uABCD`, which is replaced by `\u{ABCD}`. +- `\U0010FFFF`, which is replaced by `\u{10FFFF}`. +- `\a` (bell), `\b` (backspace), `\v` (vertical tab), and `\f` (form feed). + `\a` and `\b` are obsolescent, and `\f` and `\v` are largely obsolete. These + characters can be expressed with `\x07`, `\x08`, `\x0B`, and `\x0C` + respectively if needed. + +Note that this is the same set of escape sequences supported by +[Swift](https://docs.swift.org/swift-book/LanguageGuide/StringsAndCharacters.html#ID295) +and [Rust](https://doc.rust-lang.org/reference/tokens.html), except that, unlike +in Swift, support for `\xHH` is provided. + +While octal escape sequences are expected to remain not permitted (even though +`\0D` is reserved), the decision to not support `\1`..`\7` or more generally +`\DDDD` is _experimental_. + +In the above table, `H` represents an arbitrary hexadecimal character, `0`-`9` +or `A`-`F` (case-sensitive). Unlike in C++, but like in Python, `\x` expects +exactly two hexadecimal digits. As in JavaScript, Rust, and Swift, Unicode code +points can be expressed by number using `\u{10FFFF}` notation, which accepts any +number of hexadecimal characters. Any numeric code point in the ranges +016-D7FF16 or E00016-10FFFF16 can be +expressed this way. + +_Open question:_ Some programming languages (notably Python) support a +`\N{unicode character name}` syntax. We could add such an escape sequence. +Future proposals considering adding such support should pay attention to work +done by C++'s Unicode study group in this area. + +The escape sequence `\0` shall not be followed by a decimal digit. In cases +where a null byte should be followed by a decimal digit, `\x00` can be used +instead: `"foo\x00123"`. The intent is to preserve the possibility of permitting +decimal escape sequences in the future. + +A backslash followed by a line feed character is an escape sequence that +produces no string contents. This escape sequence is _experimental_, and can +only appear in block string literals. This escape sequence is processed after +trailing whitespace is replaced by a line feed character, so a `\` followed by +horizontal whitespace followed by a line terminator removes the whitespace up to +and including the line terminator. Unlike in Rust, but like in Swift, leading +whitespace on the line after an escaped newline is not removed, other than +whitespace that matches the indentation of the terminating `"""`. + +A character sequence starting with a backslash that doesn't match any known +escape sequence is invalid. Whitespace characters other than space and, for +block string literals, new line optionally preceded by carriage return are +disallowed. All other characters (including non-printable characters) are +preserved verbatim. Because all Carbon source files are required to be valid +sequences of Unicode characters, code unit sequences that are not valid UTF-8 +can only be produced by `\x` escape sequences. + +The decision to disallow raw tab characters in string literals is +_experimental_. + +```carbon +var String: fret = "I would 'twere something that would fret the string,\n" + + "The master-cord on's \u{2764}\u{FE0F}!"; + +// This string contains two characters (prior to encoding in UTF-8): +// U+1F3F9 (BOW AND ARROW) followed by U+0032 (DIGIT TWO) +var String: password = "\u{1F3F9}2"; + +// This string contains no newline characters. +var String: type_mismatch = """ + Shall I compare thee to a summer's day? Thou art \ + more lovely and more temperate.\ + """; + +var String: trailing_whitespace = """ + This line ends in a space followed by a newline. \n\ + This line starts with four spaces. + """; +``` + +### Raw string literals + +In order to allow strings whose contents include `\`s and `"`s, the delimiters +of string literals can be customized by prefixing the opening delimiter with _N_ +`#` characters. A closing delimiter for such a string is only recognized if it +is followed by _N_ `#` characters, and similarly, escape sequences in such +string literals are recognized only if the `\` is also followed by _N_ `#` +characters. A `\`, `"`, or `"""` not followed by _N_ `#` characters has no +special meaning. + +| Opening delimiter | Escape sequence introducer | Closing delimiter | +| ----------------- | ----------------------------- | ----------------- | +| `"` / `"""` | `\` (for example, `\n`) | `"` / `"""` | +| `#"` / `#"""` | `\#` (for example, `\#n`) | `"#` / `"""#` | +| `##"` / `##"""` | `\##` (for example, `\##n`) | `"##` / `"""##` | +| `###"` / `###"""` | `\###` (for example, `\###n`) | `"###` / `"""###` | +| ... | ... | ... | + +For example: + +```carbon +var String: x = #""" + This is the content of the string. The 'T' is the first character + of the string. + """ <-- This is not the end of the string. + """#; + // But the preceding line does end the string. +// OK, final character is \ +var String: y = #"Hello\"#; +var String: z = ##"Raw strings #"nesting"#"##; +var String: w = #"Tab is expressed as \t. Example: '\#t'"#; +``` + +Note that both a raw simple string literal and a raw block string literal can +begin with `#"""`. These cases can be distinguished by the presence or absence +of additional `"`s later in the same line: + +- In a raw simple string literal, there must be a `"` and one or more `#`s + later in the same line terminating the string. +- In a raw block string literal, the rest of the line is a file type + indicator, which can contain neither `"` nor `#`. + +```carbon +// This string is a single-line raw string literal. +// The contents of this string start and end with exactly two "s. +var String: ambig1 = #"""This is a raw string literal starting with """#; + +// This string is a raw block string literal with file-type 'This', whose +// contents start with "is a ". +var String: ambig2 = #"""This + is a block string literal with file type 'This', first character 'i', + and last character 'X': X\# + """#; + +// This is a single-line raw string literal, equivalent to "\"". +var String: ambig3 = #"""#; +``` + +### Encoding + +A string literal results in a sequence of 8-bit bytes. Like Carbon source files, +string literals are encoded in UTF-8. There is no guarantee that the string is +valid UTF-8, however, because arbitrary byte sequences can be inserted by way of +`\xHH` escape sequences. + +This is _experimental_, and should be revisited if we find sufficient motivation +for directly expressing string literals in other encodings. Similarly, as +library support for a string type evolves, we should consider including string +literal syntax (perhaps as the default) that guarantees the string content is a +valid UTF-8 encoding, so that valid UTF-8 can be distinguished from an arbitrary +string in the type system. In such string literals, we should consider rejecting +`\xHH` escapes in which HH is greater than 7F16, as in Rust. + +## Alternatives considered + +- [Block string literals](/proposals/p0199.md#block-string-literals) + - [Leading whitespace removal](/proposals/p0199.md#leading-whitespace-removal) + - [Terminating newline](/proposals/p0199.md#terminating-newline) +- [Escape sequences](/proposals/p0199.md#escape-sequences-1) +- [Raw string literals](/proposals/p0199.md#raw-string-literals-1) + - [Trailing whitespace](/proposals/p0199.md#trailing-whitespace) + - [Line separators](/proposals/p0199.md#line-separators) +- [Internal whitespace](/proposals/p0199.md#internal-whitespace) + +## References + +- Proposal + [#199: String literals](https://github.com/carbon-language/carbon-lang/pull/199)