mirror of
https://github.com/carbon-language/carbon-lang.git
synced 2026-09-24 19:50:14 +01:00
Unicode char limit (#2040)
The `\u{HHHH...}` can be an arbitrary length, potentially including `\u{}`.
Restrict to 1 to 8 characters.
This commit is contained in:
Jon Ross-Perkins
@@ -204,8 +204,8 @@ While octal escape sequences are expected to remain not permitted (even though
|
||||
In the above table, `H` represents an arbitrary hexadecimal character, `0`-`9`
|
||||
or `A`-`F` (case-sensitive). Unlike in C++, but like in Python, `\x` expects
|
||||
exactly two hexadecimal digits. As in JavaScript, Rust, and Swift, Unicode code
|
||||
points can be expressed by number using `\u{10FFFF}` notation, which accepts any
|
||||
number of hexadecimal characters. Any numeric code point in the ranges
|
||||
points can be expressed by number using `\u{10FFFF}` notation. This accepts
|
||||
between 1 and 8 hexadecimal characters. Any numeric code point in the ranges
|
||||
0<sub>16</sub>-D7FF<sub>16</sub> or E000<sub>16</sub>-10FFFF<sub>16</sub> can be
|
||||
expressed this way.
|
||||
|
||||
@@ -338,6 +338,10 @@ string in the type system. In such string literals, we should consider rejecting
|
||||
- [Leading whitespace removal](/proposals/p0199.md#leading-whitespace-removal)
|
||||
- [Terminating newline](/proposals/p0199.md#terminating-newline)
|
||||
- [Escape sequences](/proposals/p0199.md#escape-sequences-1)
|
||||
- Unicode escape sequences:
|
||||
- [Allow zero digits](/proposals/p2040.md#allow-zero-digits)
|
||||
- [Allow any number of hexadecimal characters](/proposals/p2040.md#allow-any-number-of-hexadecimal-characters)
|
||||
- [Limiting to 6 digits versus 8](/proposals/p2040.md#limiting-to-6-digits-versus-8)
|
||||
- [Raw string literals](/proposals/p0199.md#raw-string-literals-1)
|
||||
- [Trailing whitespace](/proposals/p0199.md#trailing-whitespace)
|
||||
- [Line separators](/proposals/p0199.md#line-separators)
|
||||
@@ -347,3 +351,5 @@ string in the type system. In such string literals, we should consider rejecting
|
||||
|
||||
- Proposal
|
||||
[#199: String literals](https://github.com/carbon-language/carbon-lang/pull/199)
|
||||
- Proposal
|
||||
[#2040: Unicode escape code length](https://github.com/carbon-language/carbon-lang/pull/2040)
|
||||
|
||||
@@ -0,0 +1,109 @@
|
||||
# Unicode escape code length
|
||||
|
||||
<!--
|
||||
Part of the Carbon Language project, under the Apache License v2.0 with LLVM
|
||||
Exceptions. See /LICENSE for license information.
|
||||
SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
|
||||
-->
|
||||
|
||||
[Pull request](https://github.com/carbon-language/carbon-lang/pull/2040)
|
||||
|
||||
<!-- toc -->
|
||||
|
||||
## Table of contents
|
||||
|
||||
- [Abstract](#abstract)
|
||||
- [Problem](#problem)
|
||||
- [Background](#background)
|
||||
- [Proposal](#proposal)
|
||||
- [Rationale](#rationale)
|
||||
- [Alternatives considered](#alternatives-considered)
|
||||
- [Allow zero digits](#allow-zero-digits)
|
||||
- [Allow any number of hexadecimal characters](#allow-any-number-of-hexadecimal-characters)
|
||||
- [Limiting to 6 digits versus 8](#limiting-to-6-digits-versus-8)
|
||||
|
||||
<!-- tocstop -->
|
||||
|
||||
## Abstract
|
||||
|
||||
The `\u{HHHH...}` can be an arbitrary length, potentially including `\u{}`.
|
||||
Restrict to 1 to 8 characters.
|
||||
|
||||
## Problem
|
||||
|
||||
[Proposal #199: String literals](https://github.com/carbon-language/carbon-lang/pull/199)
|
||||
says "any number of hexadecimal characters" is valid for `\u{HHHH}`. This is
|
||||
undesirable, because it means `\u{000 ... 000E9}` is a valid escape sequence,
|
||||
for any number of `0` characters. Additionally, it's not clear if `\u{}` is
|
||||
meant to be valid.
|
||||
|
||||
## Background
|
||||
|
||||
[Proposal #199: String literals](https://github.com/carbon-language/carbon-lang/pull/199)
|
||||
says:
|
||||
|
||||
> As in JavaScript, Rust, and Swift, Unicode code points can be expressed by
|
||||
> number using `\u{10FFFF}` notation, which accepts any number of hexadecimal
|
||||
> characters. Any numeric code point in the ranges
|
||||
> 0<sub>16</sub>-D7FF<sub>16</sub> or E000<sub>16</sub>-10FFFF<sub>16</sub> can
|
||||
> be expressed this way.
|
||||
|
||||
When it comes to the number of digits, the languages differ:
|
||||
|
||||
- In [JavaScript](https://262.ecma-international.org/13.0/#prod-CodePoint),
|
||||
between 1 and 6 digits are supported, and it must be less than or equal to
|
||||
`10FFFF`.
|
||||
- In [Rust](https://doc.rust-lang.org/reference/tokens.html), between 1 and 6
|
||||
digits are supported.
|
||||
- In
|
||||
[Swift](https://docs.swift.org/swift-book/LanguageGuide/StringsAndCharacters.html),
|
||||
between 1 and 8 digits are supported.
|
||||
|
||||
Unicode's codespace is 0 to [`10FFFF`](https://unicode.org/glossary/#codespace).
|
||||
|
||||
## Proposal
|
||||
|
||||
The `\u{H...}` syntax is only valid for 1 to 8 unicode characters.
|
||||
|
||||
## Rationale
|
||||
|
||||
- [Code that is easy to read, understand, and write](/docs/project/goals.md#code-that-is-easy-to-read-understand-and-write)
|
||||
- This restriction does not affect the ability to write valid Unicode.
|
||||
Instead, it restricts the ability to write confusing or invalid unicode,
|
||||
which should make it easier to detect errors.
|
||||
- [Fast and scalable development](/docs/project/goals.md#fast-and-scalable-development)
|
||||
- Simplifies tooling by reducing the number of syntaxes that need to be
|
||||
supported, and allowing early failure on obviously invalid inputs.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
### Allow zero digits
|
||||
|
||||
We could allow `\u{}` as a version of `\u{0}`. However, as shorthand, it doesn't
|
||||
save much and `\x00` is both equal length and clearer.
|
||||
|
||||
Rather than allowing this syntax, we prefer to disallow it for consistency with
|
||||
other languages.
|
||||
|
||||
### Allow any number of hexadecimal characters
|
||||
|
||||
We could allow any number of digits in the `\u` escape. However, this has the
|
||||
consequence of requiring parsing of escapes of completely arbitrary length.
|
||||
|
||||
This creates unnecessary complexity in the parser because we need to consider
|
||||
what happens if the result is greater than 32 bits, significantly larger than
|
||||
unicode's current `10FFFF` limit. One way to do this would be to store the
|
||||
result in a 32-bit integer and keep parsing until the value goes above `10FFFF`,
|
||||
then error as invalid if that's exceeded. This would allow an arbitrary number
|
||||
of leading `0`'s to correctly parse.
|
||||
|
||||
It should make it easier to write a simple parser if we instead limit the number
|
||||
of digits to a reasonable amount.
|
||||
|
||||
### Limiting to 6 digits versus 8
|
||||
|
||||
A limit of 6 digits offers a reasonable limit as the minimum needed to represent
|
||||
Unicode's codespace. A limit of 8 digits offers a reasonable limit as a standard
|
||||
4-byte value, and roughly matches UTF-32.
|
||||
|
||||
While it seems a weak advantage, this proposal leans towards 8.
|
||||
Reference in New Issue
Block a user