mirror of
https://github.com/carbon-language/carbon-lang.git
synced 2026-11-18 16:30:30 +00:00
Add sentinel Eof token at the end of the tokenized buffer. (#388)
This has two goals: 1) It allows us to simplify and remove special cases from the parser: when we expect a particular token next, we can just check for it without needing a special case for end-of-file. 2) It gives us a token to use as a position when emitting diagnostics at the end of the file. Centralize all updating of `position` to `Consume` and `SkipTo`, so that we can in a single place ensure that we never go past the EOF token.
This commit is contained in:
1 parent
d1757d9979
commit
1e6c7e3963
7 files changed
+415
-284
No files matched your search
@@ -21,5 +21,6 @@ CARBON_PARSE_NODE_KIND(FunctionDeclaration)
|
||||
CARBON_PARSE_NODE_KIND(Identifier)
|
||||
CARBON_PARSE_NODE_KIND(ParameterListEnd)
|
||||
CARBON_PARSE_NODE_KIND(ParameterList)
|
||||
CARBON_PARSE_NODE_KIND(FileEnd)
|
||||
|
||||
#undef CARBON_PARSE_NODE_KIND
|
||||
Reference in new issue
Block a user