Instead of adding a `Token<Kind>` field to typed parse nodes that
contains a `TokenIndex`, make the parse node directly hold a
`TokenIndex` of a more specific kind.
Instead of tracking the token associated with a parse node in the `.def`
file macro, track it on the typed node instead. List the token as a
field inside the node structure to show the order of the token relative
to the other components of the grammar production, and to allow the
token index to be accessed when the node is extracted.
Remove the corresponding information from the `.def` file, leaving
behind just a list of parse node kinds in the majority of cases.
This also removes the checking of the token kind associated with a parse
node in the case where the parse node has errors. Previously we had a
flag on the node kind to indicate whether we should check this, but per
[discord
discussion](https://discord.com/channels/655572317891461132/655578254970716160/1246214418979881052),
we have decided to remove this.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Mirroring #4003 for NodeCategory.
Note we template a lot more on NodeCategory's enum, so this is a
slightly more awkward delta.
Also, switch from Enum in KeywordModifierSet to RawEnumType for
consistency with EnumBase. The templating on NodeCategory had me
thinking about that more.
For now, a builtin function is defined by specifying a string literal
initializer in a function declaration:
```carbon
fn MyBuiltin(a: i32) -> i32 = "builtin.name";
```
End-to-end support is included for a sample `"int.add"` builtin
performing integer addition, covering constant evaluation and code
generation.
The implementation here needs substantial refactoring before we'll be
ready to start adding more builtins. That refactoring work will be
coming next. This change is aiming to checkpoint some incremental
progress.
This sets things up to use `bazel` to run `clang-tidy` using
https://github.com/erenon/bazel_clang_tidy.
I'm fixing issues outside of explorer, and disabling clang-tidy for
targets in explorer that have legacy issues. I was going to disable
clang-tidy for targets in explorer such as interpreter anyways, because
they're slow to parse, and just extended that to the currently failing
targets.
Don't first form a `std::tuple` and then convert it to a struct. Speeds
up compilation of extract.cpp by 15-20%.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: josh11b <josh11b@users.noreply.github.com>
Tests previously uncovered code. Fix uncovered problems:
* formatting of trace output
* package & import directives need to be classified as declarations
* the problem that meant the previous problem wasn't caught by existing
tests (since `Tree::Verify` didn't check that top-level declarations
match `AnyDeclId`, as required by `Tree::ExtractFile()`).
These are intended to allow the structure of a parse tree node to be
described more precisely in code, to support these use cases:
- Automated checking that the parse tree conforms to the expected
structure. (Added to `Tree::Verify`.)
- Easier reading and understanding of the structure of the parse tree by
toolchain developers. (See `parse/typed_nodes.h`.)
- Easier navigation of the parse tree, for example for tooling uses and
for use when forming diagnostics.
On this last point, an object representing the file may be inspecting
using `Tree::ExtractFile`, as in:
```
auto file = tree->ExtractFile();
for (AnyDeclId decl_id : file.decls) {
// `decl_id` is convertible to a `NodeId`.
if (std::optional<FunctionDecl> fn_decl =
tree->ExtractAs<FunctionDecl>(decl_id)) {
// fn_decl->params is a `TuplePatternId` (which extends `NodeId`)
// that is guaranteed to reference a `TuplePattern`.
std::optional<TuplePattern> params = tree->Extract(fn_decl->params);
// `params` has a value unless there was an error in that node.
} else if (auto class_def = tree->ExtractAs<ClassDefinition>(decl_id)) {
// ...
}
}
```
The `Extract...` functions collect the child nodes into the typed parse
node's fields (internally using a `Tree::SiblingIterator`) for easy
access. However, this is not as fast as directly observing the tree
structure using the postorder strategy being used by the check stage.
These functions rely on using struct reflection on the typed parse node
definitions from `parse/typed_nodes.h` to get the expected structure of
child nodes and then populate them.
Note that validating these in `Tree::Verify` adds significant cost to
it, and is currently included in the parsing stage. Without this change,
a 10 mloc test case of lex & parse takes 4.129 s ± 0.041 s. With this
change, it takes 5.768 s ± 0.036 s.
This builds upon and completes #3393.
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Richard Smith
