Add sentinel Eof token at the end of the tokenized buffer. (#388)

This has two goals:

1) It allows us to simplify and remove special cases from the parser:
   when we expect a particular token next, we can just check for it
   without needing a special case for end-of-file.

2) It gives us a token to use as a position when emitting diagnostics at
   the end of the file.

Centralize all updating of `position` to `Consume` and `SkipTo`, so that we can in a single place ensure that we never go past the EOF token.
This commit is contained in:
Richard Smith
2021-03-18 21:39:10 -07:00
committed by GitHub
parent d1757d9979
commit 1e6c7e3963
7 changed files with 415 additions and 284 deletions
+1
View File
@@ -21,5 +21,6 @@ CARBON_PARSE_NODE_KIND(FunctionDeclaration)
CARBON_PARSE_NODE_KIND(Identifier)
CARBON_PARSE_NODE_KIND(ParameterListEnd)
CARBON_PARSE_NODE_KIND(ParameterList)
CARBON_PARSE_NODE_KIND(FileEnd)
#undef CARBON_PARSE_NODE_KIND