mirror of
https://github.com/carbon-language/carbon-lang.git
synced 2026-10-04 07:21:04 +01:00
Add sentinel Eof token at the end of the tokenized buffer. (#388)
This has two goals: 1) It allows us to simplify and remove special cases from the parser: when we expect a particular token next, we can just check for it without needing a special case for end-of-file. 2) It gives us a token to use as a position when emitting diagnostics at the end of the file. Centralize all updating of `position` to `Consume` and `SkipTo`, so that we can in a single place ensure that we never go past the EOF token.
This commit is contained in:
@@ -21,5 +21,6 @@ CARBON_PARSE_NODE_KIND(FunctionDeclaration)
|
||||
CARBON_PARSE_NODE_KIND(Identifier)
|
||||
CARBON_PARSE_NODE_KIND(ParameterListEnd)
|
||||
CARBON_PARSE_NODE_KIND(ParameterList)
|
||||
CARBON_PARSE_NODE_KIND(FileEnd)
|
||||
|
||||
#undef CARBON_PARSE_NODE_KIND
|
||||
|
||||
Reference in New Issue
Block a user