mirror of
https://github.com/carbon-language/carbon-lang.git
synced 2026-11-22 10:22:39 +00:00
This is based on discussion on #732: that we should probably parse the invalid whitespace, then reject it as part of string validation, rather than having different parses. I worry the question of "how is this parsed" may lead to subtly unexpected results if we aren't consistent, so I'm switching the logic from the lexer to the unescape library (and also adjusting the list of rejected whitespace).
Lexical conventions
Table of contents
Lexical elements
The first stage of processing a source file is the division of the source file into lexical elements.
A lexical element is one of the following:
- a maximal sequence of whitespace characters
- a word
- a literal:
- TODO: operators, comments, ...
The sequence of lexical elements is formed by repeatedly removing the longest initial sequence of characters that forms a valid lexical element.