Jon Ross-PerkinsandGeoff Romer
a24816a1f4
Note I'm mostly trying to capture [the docs](https://docs.google.com/document/d/1RRYMm42osyqhI2LyjrjockYCutQ5dOf8Abu50kTrkX0/edit?resourcekey=0-kHyqOESbOHmzZphUbtLrTw&tab=t.0) as they exist today, not fixing issues with the docs. I think the doc itself hasn't changed much lately (i.e., for months). Trying to organize it a little better though, particularly so that it shows up reasonably when looking in github or the website. --------- Co-authored-by: Geoff Romer <gromer@google.com>
1.5 KiB
Lex
Table of contents
Overview
Lexing converts input source code into tokenized output. Literals, such as string literals, have their value parsed and form a single token at this stage.
Bracket matching
The lexer handles matching for (), [], and {}. When a bracket lacks a
match, it will insert a "recovery" token to produce a match. As a consequence,
the lexer's output should always have matched brackets, even with invalid code.
While bracket matching could use hints such as contextual clues from indentation, that is not yet implemented.
Alternatives considered
Bracket matching in parser
Bracket matching could have also been implemented in the parser, with some awareness of parse state. However, that would shift some of the complexity of recovery in other error situations, such as where the parser searches for the next comma in a list. That needs to skip over bracketed ranges. We don't think the trade-offs would yield a net benefit, so any change in this direction would need to show concrete improvement, for example better diagnostics for common issues.