carbon-language/carbon-lang
1
0
Fork 0
mirror of https://github.com/carbon-language/carbon-lang.git synced 2026-10-06 07:34:45 +01:00
Code Issues Packages Projects Releases 10 Wiki Activity

Updated string literal design based on #199 (#732)

Browse Source 
I've modified the text from the proposal slightly, focusing the overview more on a design setup, but mostly kept the details. One important thing here is I noticed that raw tab characters are disallowed -- this was a little buried before, and I've now updated the list of characters allowed in a string to exclude tabs. Additionally, I've noted `\0D` in the list of escapes as explicitly invalid.

Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This commit is contained in:
Jon Meow
2021-08-27 15:43:14 -07:00
committed by GitHub
co-authored by Richard Smith
parent 821daa220a
commit 30682750ce
2 changed files with 349 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/design/lexical_conventions/README.md
+1 -1
View File
@@ -37,7 +37,7 @@ A _lexical element_ is one of the following:
- a [word](words.md)
- a literal:
- a [numeric literal](numeric_literals.md)
- TODO: string literals
- a [string literal](string_literals.md)
- TODO: operators, comments, ...
The sequence of lexical elements is formed by repeatedly removing the longest
docs/design/lexical_conventions/string_literals.md
+348
View File
@@ -0,0 +1,348 @@
# String literals
<!--
Part of the Carbon Language project, under the Apache License v2.0 with LLVM
Exceptions. See /LICENSE for license information.
SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
-->
<!-- toc -->
## Table of contents
- [Overview](#overview)
- [Details](#details)
- [Simple and block string literals](#simple-and-block-string-literals)
- [Escape sequences](#escape-sequences)
- [Raw string literals](#raw-string-literals)
- [Encoding](#encoding)
- [Alternatives considered](#alternatives-considered)
- [References](#references)
<!-- tocstop -->
## Overview
Carbon supports both simple literals that are single-line using one double
quotation mark (`"`) and block literals that are multi-line using three double
quotation marks (`"""`). A block string literal may have a file type indicator
after the first `"""`; this does not affect the string itself, but may assist
other tooling. For example:
```carbon
// Simple string literal:
var simple: String = "example";
// Block string literal:
var block: String = """
The winds grow high; so do your stomachs, lords.
How irksome is this music to my heart!
When such strings jar, what hope of harmony?
I pray, my lords, let me compound this strife.
-- History of Henry VI, Part II, Act II, Scene 1, W. Shakespeare
""";
// Block string literal with file type indicator:
var code_block: String = """cpp
#include <iostream>
int main() {
std::cout << "Hello world!";
return 0;
}
"""
```
The indentation of a block string literal's terminating line is removed from all
preceding lines. As a consequence, in the above `code_block` example, only
`std::cout` and `return` are indented in the resulting string, and by 4 spaces
each.
Escape sequences introduced by a backslash (`\`) and are used to express special
character or code unit sequences, such as `\n` for a newline character. Raw
string literals are additionally delimited with one or more `#`; these require
an equal number of hash symbols (`#`) after the `\` to indicate an escape
sequence. Raw string literals are used to more easily write literal `\`s in
strings. Both simple and block string literals have raw forms. For example:
```carbon
// Raw simple string literal with newline escape sequence:
var newline: String = "line one\nline two";
// Raw simple string literal with literal `\n`, not a newline:
var raw: String = #"line one\nstill line one"#;
// Raw simple string literal with newline escape sequence:
var raw_newline: String = "#line one\#nline two"#;
```
## Details
### Simple and block string literals
A _simple string literal_ is formed of a sequence of:
- Characters other than `\` and `"`.
- Only space characters (U+0020) are valid whitespace in a string literal.
Other whitespace, including tabs and newlines, are disallowed but parse
as part of the string for error recovery purposes.
- [Escape sequences](#escape-sequences).
- Each escape sequence is replaced with the corresponding character
sequence or code unit sequence.
- Similarly to invalid whitespace, invalid escape sequences such as `\z`
parse as part of the string.
This sequence is enclosed in `"`s. For example, this is a simple string literal:
```carbon
var String: lucius = "The strings, my lord, are false.";
```
A _block string literal_ starts with `"""`, followed by an optional file type
indicator, followed by a newline, and ends at the next instance of three double
quotation marks whose first `"` is not part of a `\"` escape sequence. The
closing `"""` shall be the first non-whitespace characters on that line. The
lines between the opening line and the closing line (exclusive) are _content
lines_. The content lines shall not contain `\` characters that do not form part
of an escape sequence.
The _indentation_ of a block string literal is the sequence of horizontal
whitespace preceding the closing `"""`. Each non-empty content line shall begin
with the indentation of the string literal. The content of the literal is formed
as follows:
- The indentation of the closing line is removed from each non-empty content
line.
- All trailing whitespace on each line, including the line terminator, is
replaced with a single line feed (U+000A) character.
- The resulting lines are concatenated.
- Each [escape sequence](#escape-sequences) is replaced with the corresponding
character sequence or code unit sequence.
A content line is considered empty if it contains only whitespace characters.
```carbon
var String: w = """
This is a string literal. Its first character is 'T' and its last character is
a newline character. It contains another newline between 'is' and 'a'.
""";
// This string literal is invalid because the """ after 'closing' terminates
// the literal, but is not at the start of the line.
var String: invalid = """
error: closing """ is not on its own line.
""";
```
A _file type indicator_ is any sequence of non-whitespace characters other than
`"` or `#`. The file type indicator has no semantic meaning to the Carbon
compiler, but some file type indicators are understood by the language tooling
(for example, syntax highlighter, code formatter) as indicating the structure of
the string literal's content.
```carbon
// This is a block string literal. Its first two characters are spaces, and its
// last character is a line feed. It has a file type of 'c++'.
var String: starts_with_whitespace = """c++
int x = 1; // This line starts with two spaces.
int y = 2; // This line starts with two spaces.
""";
```
The file type indicator might contain semantic information beyond the file type
itself, such as instructions to the code formatter to disable formatting for the
code block.
**Open question:** There is no concrete set of recognized file type indicators.
It would be useful to informally specify a set of well-known indicators, so that
tools have a common understanding of what those indicators mean, perhaps in a
best practices guide.
#### Escape sequences
Within a string literal, the following escape sequences are recognized:
| Escape | Meaning |
| ------------- | -------------------------------------------------------- |
| `\t` | U+0009 CHARACTER TABULATION |
| `\n` | U+000A LINE FEED |
| `\r` | U+000D CARRIAGE RETURN |
| `\"` | U+0022 QUOTATION MARK (`"`) |
| `\'` | U+0027 APOSTROPHE (`'`) |
| `\\` | U+005C REVERSE SOLIDUS (`\`) |
| `\0` | Code unit with value 0 |
| `\0D` | Invalid, reserved for evolution |
| `\xHH` | Code unit with value HH<sub>16</sub> |
| `\u{HHHH...}` | Unicode code point U+HHHH... |
| `\<newline>` | No string literal content produced (block literals only) |
Hex characters (`H`) must be uppercase (`\xAA`, not `\xaa`).
This includes all C++ escape sequences except:
- `\?`, which was historically used to escape trigraphs in string literals,
and no longer serves any purpose.
- `\ooo` octal escapes, which are removed because Carbon does not support
octal literals; `\0` is retained as a special case, which is expected to be
important for C interoperability.
- `\uABCD`, which is replaced by `\u{ABCD}`.
- `\U0010FFFF`, which is replaced by `\u{10FFFF}`.
- `\a` (bell), `\b` (backspace), `\v` (vertical tab), and `\f` (form feed).
`\a` and `\b` are obsolescent, and `\f` and `\v` are largely obsolete. These
characters can be expressed with `\x07`, `\x08`, `\x0B`, and `\x0C`
respectively if needed.
Note that this is the same set of escape sequences supported by
[Swift](https://docs.swift.org/swift-book/LanguageGuide/StringsAndCharacters.html#ID295)
and [Rust](https://doc.rust-lang.org/reference/tokens.html), except that, unlike
in Swift, support for `\xHH` is provided.
While octal escape sequences are expected to remain not permitted (even though
`\0D` is reserved), the decision to not support `\1`..`\7` or more generally
`\DDDD` is _experimental_.
In the above table, `H` represents an arbitrary hexadecimal character, `0`-`9`
or `A`-`F` (case-sensitive). Unlike in C++, but like in Python, `\x` expects
exactly two hexadecimal digits. As in JavaScript, Rust, and Swift, Unicode code
points can be expressed by number using `\u{10FFFF}` notation, which accepts any
number of hexadecimal characters. Any numeric code point in the ranges
0<sub>16</sub>-D7FF<sub>16</sub> or E000<sub>16</sub>-10FFFF<sub>16</sub> can be
expressed this way.
_Open question:_ Some programming languages (notably Python) support a
`\N{unicode character name}` syntax. We could add such an escape sequence.
Future proposals considering adding such support should pay attention to work
done by C++'s Unicode study group in this area.
The escape sequence `\0` shall not be followed by a decimal digit. In cases
where a null byte should be followed by a decimal digit, `\x00` can be used
instead: `"foo\x00123"`. The intent is to preserve the possibility of permitting
decimal escape sequences in the future.
A backslash followed by a line feed character is an escape sequence that
produces no string contents. This escape sequence is _experimental_, and can
only appear in block string literals. This escape sequence is processed after
trailing whitespace is replaced by a line feed character, so a `\` followed by
horizontal whitespace followed by a line terminator removes the whitespace up to
and including the line terminator. Unlike in Rust, but like in Swift, leading
whitespace on the line after an escaped newline is not removed, other than
whitespace that matches the indentation of the terminating `"""`.
A character sequence starting with a backslash that doesn't match any known
escape sequence is invalid. Whitespace characters other than space and, for
block string literals, new line optionally preceded by carriage return are
disallowed. All other characters (including non-printable characters) are
preserved verbatim. Because all Carbon source files are required to be valid
sequences of Unicode characters, code unit sequences that are not valid UTF-8
can only be produced by `\x` escape sequences.
The decision to disallow raw tab characters in string literals is
_experimental_.
```carbon
var String: fret = "I would 'twere something that would fret the string,\n" +
"The master-cord on's \u{2764}\u{FE0F}!";
// This string contains two characters (prior to encoding in UTF-8):
// U+1F3F9 (BOW AND ARROW) followed by U+0032 (DIGIT TWO)
var String: password = "\u{1F3F9}2";
// This string contains no newline characters.
var String: type_mismatch = """
Shall I compare thee to a summer's day? Thou art \
more lovely and more temperate.\
""";
var String: trailing_whitespace = """
This line ends in a space followed by a newline. \n\
This line starts with four spaces.
""";
```
### Raw string literals
In order to allow strings whose contents include `\`s and `"`s, the delimiters
of string literals can be customized by prefixing the opening delimiter with _N_
`#` characters. A closing delimiter for such a string is only recognized if it
is followed by _N_ `#` characters, and similarly, escape sequences in such
string literals are recognized only if the `\` is also followed by _N_ `#`
characters. A `\`, `"`, or `"""` not followed by _N_ `#` characters has no
special meaning.
| Opening delimiter | Escape sequence introducer | Closing delimiter |
| ----------------- | ----------------------------- | ----------------- |
| `"` / `"""` | `\` (for example, `\n`) | `"` / `"""` |
| `#"` / `#"""` | `\#` (for example, `\#n`) | `"#` / `"""#` |
| `##"` / `##"""` | `\##` (for example, `\##n`) | `"##` / `"""##` |
| `###"` / `###"""` | `\###` (for example, `\###n`) | `"###` / `"""###` |
| ... | ... | ... |
For example:
```carbon
var String: x = #"""
This is the content of the string. The 'T' is the first character
of the string.
""" <-- This is not the end of the string.
"""#;
// But the preceding line does end the string.
// OK, final character is \
var String: y = #"Hello\"#;
var String: z = ##"Raw strings #"nesting"#"##;
var String: w = #"Tab is expressed as \t. Example: '\#t'"#;
```
Note that both a raw simple string literal and a raw block string literal can
begin with `#"""`. These cases can be distinguished by the presence or absence
of additional `"`s later in the same line:
- In a raw simple string literal, there must be a `"` and one or more `#`s
later in the same line terminating the string.
- In a raw block string literal, the rest of the line is a file type
indicator, which can contain neither `"` nor `#`.
```carbon
// This string is a single-line raw string literal.
// The contents of this string start and end with exactly two "s.
var String: ambig1 = #"""This is a raw string literal starting with """#;
// This string is a raw block string literal with file-type 'This', whose
// contents start with "is a ".
var String: ambig2 = #"""This
is a block string literal with file type 'This', first character 'i',
and last character 'X': X\#
"""#;
// This is a single-line raw string literal, equivalent to "\"".
var String: ambig3 = #"""#;
```
### Encoding
A string literal results in a sequence of 8-bit bytes. Like Carbon source files,
string literals are encoded in UTF-8. There is no guarantee that the string is
valid UTF-8, however, because arbitrary byte sequences can be inserted by way of
`\xHH` escape sequences.
This is _experimental_, and should be revisited if we find sufficient motivation
for directly expressing string literals in other encodings. Similarly, as
library support for a string type evolves, we should consider including string
literal syntax (perhaps as the default) that guarantees the string content is a
valid UTF-8 encoding, so that valid UTF-8 can be distinguished from an arbitrary
string in the type system. In such string literals, we should consider rejecting
`\xHH` escapes in which HH is greater than 7F<sub>16</sub>, as in Rust.
## Alternatives considered
- [Block string literals](/proposals/p0199.md#block-string-literals)
- [Leading whitespace removal](/proposals/p0199.md#leading-whitespace-removal)
- [Terminating newline](/proposals/p0199.md#terminating-newline)
- [Escape sequences](/proposals/p0199.md#escape-sequences-1)
- [Raw string literals](/proposals/p0199.md#raw-string-literals-1)
- [Trailing whitespace](/proposals/p0199.md#trailing-whitespace)
- [Line separators](/proposals/p0199.md#line-separators)
- [Internal whitespace](/proposals/p0199.md#internal-whitespace)
## References
- Proposal
[#199: String literals](https://github.com/carbon-language/carbon-lang/pull/199)