Separate subtree size information from parse nodes. (#4174)

Move subtree sizes over to TreeAndSubtrees, using the different
structure to represent the additional parse work that occurs, as well as
making it clear which functions require the extra information. My intent
is to make it hard to use this by accident.

The subtree size is still tracked during Parse::Tree construction. I
think a lot of that can be cleaned up, although we use it during
placeholder assignment so it may take some work. I wanted to see what
people thought about this before taking action on such a change.

I'm using a 1m line source file generated by #4124 for testing. Command
is `time bazel-bin/toolchain/install/prefix_root/bin/carbon compile
--phase=check --dump-mem-usage ~/tmp/data.carbon`

At head, what I'm seeing is:

```
...
parse_tree_.node_impls_:
  used_bytes:      61516116
  reserved_bytes:  61516116
...
Total:
  used_bytes:      447814230
  reserved_bytes:  551663894
...
1.43s user 0.14s system 99% cpu 1.565 total
```

With `Tree::Verify` disabled completely, it looks like:
```
parse_tree_.node_impls_:
  used_bytes:      41010744
  reserved_bytes:  41010744
...
Total:
  used_bytes:      427308858
  reserved_bytes:  531158522
...
1.20s user 0.13s system 99% cpu 1.332 total
```

Re-enabling just the basic verification (what is now `Tree::Verify`),
I'm seeing maybe 0.05s slower, but that's within noise for my system. I
do see variability in my timing results, and overall I think this is a
0.2s +/- 0.1s improvement versus the earlier (always testing `Extract`
code) implementation. That's opt; debug builds will be unaffected,
because the same checking occurs as before.

Note, the subtree size is a third of the node representation, which is
why I'm showing the decrease in memory usage here.
This commit is contained in:
Jon Ross-Perkins
2024-07-31 19:39:45 +00:00
committed by GitHub
parent e6e61e14ae
commit f67791cfee
40 changed files with 767 additions and 692 deletions
+2 -4
View File
@@ -18,8 +18,7 @@ static auto FinishAndSkipInvalidDecl(Context& context, int32_t subtree_start)
context.ReplacePlaceholderNode(subtree_start, NodeKind::InvalidParseStart,
cursor, /*has_error=*/true);
context.AddNode(NodeKind::InvalidParseSubtree,
context.SkipPastLikelyEnd(cursor), subtree_start,
/*has_error=*/true);
context.SkipPastLikelyEnd(cursor), /*has_error=*/true);
}
// Prints a diagnostic and calls FinishAndSkipInvalidDecl.
@@ -226,12 +225,11 @@ static auto TryHandleAsModifier(Context& context) -> bool {
auto extern_token = context.Consume();
if (context.PositionIs(Lex::TokenKind::Library)) {
// `extern library <owning_library>` syntax.
auto subtree_start = context.tree().size();
context.ParseLibrarySpecifier(/*accept_default=*/true);
// TODO: Consider error recovery when a non-declaration token is next,
// like a typo of the library name.
context.AddNode(NodeKind::ExternModifierWithLibrary, extern_token,
subtree_start, /*has_error=*/false);
/*has_error=*/false);
} else {
// `extern` syntax without a library.
context.AddLeafNode(NodeKind::ExternModifier, extern_token);