mirror of
https://github.com/carbon-language/carbon-lang.git
synced 2026-10-02 09:05:49 +01:00
Jon Ross-PerkinsandGeoff Romer
a24816a1f4
Note I'm mostly trying to capture [the docs](https://docs.google.com/document/d/1RRYMm42osyqhI2LyjrjockYCutQ5dOf8Abu50kTrkX0/edit?resourcekey=0-kHyqOESbOHmzZphUbtLrTw&tab=t.0) as they exist today, not fixing issues with the docs. I think the doc itself hasn't changed much lately (i.e., for months). Trying to organize it a little better though, particularly so that it shows up reasonably when looking in github or the website. --------- Co-authored-by: Geoff Romer <gromer@google.com>
45 lines
1.5 KiB
Markdown
45 lines
1.5 KiB
Markdown
# Lex
|
|
|
|
<!--
|
|
Part of the Carbon Language project, under the Apache License v2.0 with LLVM
|
|
Exceptions. See /LICENSE for license information.
|
|
SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
|
|
-->
|
|
|
|
<!-- toc -->
|
|
|
|
## Table of contents
|
|
|
|
- [Overview](#overview)
|
|
- [Bracket matching](#bracket-matching)
|
|
- [Alternatives considered](#alternatives-considered)
|
|
- [Bracket matching in parser](#bracket-matching-in-parser)
|
|
|
|
<!-- tocstop -->
|
|
|
|
## Overview
|
|
|
|
Lexing converts input source code into tokenized output. Literals, such as
|
|
string literals, have their value parsed and form a single token at this stage.
|
|
|
|
## Bracket matching
|
|
|
|
The lexer handles matching for `()`, `[]`, and `{}`. When a bracket lacks a
|
|
match, it will insert a "recovery" token to produce a match. As a consequence,
|
|
the lexer's output should always have matched brackets, even with invalid code.
|
|
|
|
While bracket matching could use hints such as contextual clues from
|
|
indentation, that is not yet implemented.
|
|
|
|
## Alternatives considered
|
|
|
|
### Bracket matching in parser
|
|
|
|
Bracket matching could have also been implemented in the parser, with some
|
|
awareness of parse state. However, that would shift some of the complexity of
|
|
recovery in other error situations, such as where the parser searches for the
|
|
next comma in a list. That needs to skip over bracketed ranges. We don't think
|
|
the trade-offs would yield a net benefit, so any change in this direction would
|
|
need to show concrete improvement, for example better diagnostics for common
|
|
issues.
|