carbon-language/carbon-lang
1
0
Fork 0
mirror of https://github.com/carbon-language/carbon-lang.git synced 2026-10-05 22:02:55 +01:00
Code Issues Packages Projects Releases 10 Wiki Activity

Move information on the token associated with a parse node from the .def file into the typed node. (#4001)

Browse Source 
Instead of tracking the token associated with a parse node in the `.def`
file macro, track it on the typed node instead. List the token as a
field inside the node structure to show the order of the token relative
to the other components of the grammar production, and to allow the
token index to be accessed when the node is extracted.

Remove the corresponding information from the `.def` file, leaving
behind just a list of parse node kinds in the majority of cases.

This also removes the checking of the token kind associated with a parse
node in the case where the parse node has errors. Previously we had a
flag on the node kind to indicate whether we should check this, but per
[discord
discussion](https://discord.com/channels/655572317891461132/655578254970716160/1246214418979881052),
we have decided to remove this.

---------

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This commit is contained in:
Richard Smith
2024-05-31 23:11:51 +00:00
committed by GitHub
co-authored by Jon Ross-Perkins
parent 5bb318cae6
commit 3c01ee69ed
14 changed files with 921 additions and 927 deletions
Download Patch File Download Diff File Expand all files Collapse all files
toolchain/parse/context.cpp
-2
View File
@@ -68,7 +68,6 @@ Context::Context(Tree& tree, Lex::TokenizedBuffer& tokens,
auto Context::AddLeafNode(NodeKind kind, Lex::TokenIndex token, bool has_error)
-> void {
kind.CheckMatchesTokenKind(tokens_->GetKind(token), has_error);
tree_->node_impls_.push_back(
Tree::NodeImpl(kind, has_error, token, /*subtree_size=*/1));
if (has_error) {
@@ -78,7 +77,6 @@ auto Context::AddLeafNode(NodeKind kind, Lex::TokenIndex token, bool has_error)
auto Context::AddNode(NodeKind kind, Lex::TokenIndex token, int subtree_start,
bool has_error) -> void {
kind.CheckMatchesTokenKind(tokens_->GetKind(token), has_error);
int subtree_size = tree_->size() - subtree_start + 1;
tree_->node_impls_.push_back(
Tree::NodeImpl(kind, has_error, token, subtree_size));