We already do this with things like llvm::DenseMapInfo, I don't know why
I was doing this with format_provider. But this should be more
consistent, and slightly better for not entering another library's
namespace.
Use two different nodes for "<type> followed by `as`" and "<type>
omitted before `as`, use `self`", so it is easier to determine which
case. Later the second case will push the type id for `self` onto the
node stack, making the two paths more similar.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Consume the components of the `impl` declaration, and set up scopes for
the child elements. We don't yet build a representation for the impl
itself.
Also, add an interface type value. This is necessary so that we have a
value for the expression on the right-hand side of `as` in an `impl`.
This doesn't matter too much as these were expanded by the LLVM iterator
facade, but eventually this should enable that facade to be a little
less fancy and should also simplify the dispatch to directly use the
three-way comparison.
One case is a bit subtle and didn't have a comment so I added one to
explain a bit what is going on there.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This required adding a few headers that were found transitively before,
but not too many. This is sadly a fairly manual process of opening every
file in my IDE, but I think I got everything in `//common` and
`//toolchain`.
There are a few cases where technically we don't need `foo.h` to be
included into `foo.cpp`, but I've forced those to stay with a pragma.
I've tried to catch the places where we can cut deps in Bazel as well,
but not sure I got all of those.
I had been noticing these in other PRs and it seemed better to isolate
the change.
The categories `Expr`, `MemberName`, `Decl`, `Statement`, and `Modifier`
are usable since they are associated with a consistent `IdKind`. The
mapping to `IdKind` for NodeKinds that have those categories are no
longer listed explicitly, ensuring that the `NodeCategory` mapping is
the source of truth.
Also: fixes the category of the `FunctionDefinitionStart` and
`ArrayExprStart` node kinds.
Note: I've added [a section on defining constexpr constants to the
Toolchain architecture
doc](https://docs.google.com/document/d/1RRYMm42osyqhI2LyjrjockYCutQ5dOf8Abu50kTrkX0/edit?resourcekey=0-kHyqOESbOHmzZphUbtLrTw&tab=t.0#heading=h.f7682a2tpvxr).
FUTURE:
* We should switch `TuplePattern` to put an `InstId` on the `NodeStack`
instead of an `InstBlockId`, so we can handle the pattern category.
* We should make a category for names to replace uses of the `NameId`
`IdKind`.
* We should make use of these new APIs more, and propagate more-precise
types through the codebase.
QUESTION: Should I use a different approach for determining the number
of members of the `NodeKind` enum?
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Recent runs of `clang-tidy` for me started showing more errors, and this
is a collection of changes to address them.
First, I've systematically applied the disabling tag to all C++ rules
under //explorer/... with `buildozer` so we don't spend time analyzing
this code or reporting errors from it. Not sure this was strictly
necessary, but it seemed like a nice consistency improvement.
Next, I disabled a buggy check for missing `default` cases in
`switch`es. It seems to get confused by the fancy conversions in our
`enum_base.h`. We don't miss much with this as the Clang compiler
warnings for `switch` catch most of our actual bugs. I also removed the
local disabling of this now that it is turned off centrally.
Lastly, I added error checking to two file descriptor manipulating calls
in the `file_test` infrastructure.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This patches bazel_clang_tidy handling of headers. I found an equivalent
change at https://github.com/erenon/bazel_clang_tidy/pull/13, but that
was [already
rejected](https://github.com/erenon/bazel_clang_tidy/pull/13#issuecomment-1047007424).
Per the criticism, this will result in redundant processing of headers.
The project instead uses `HeaderFilterRegex: ".*"`, but that results in
two problems:
1. When running with `-k`, errors are repeated when a header is included
more than once, which is common.
2. clang-tidy including errors from headers that are included from other
modules (e.g., abseil-cpp); filtering correctly is difficult.
Given the trade-offs and options (including forking), I thought patching
was preferable so long as it remains narrow.
Also stop supporting `var` with initializer inside `for`.
Resolves TODO in `handle_variable.cpp`
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This change adds a `BindSymbolicName` instruction for generic bindings,
paralleling the existing `BindName`. A mechanism is also added to allow
both kinds of binding to be accessed uniformly, for convenience in the
case where the two different kinds of binding are treated the same.
Generic bindings of type `type` are allowed to be used as types,
although no operations are provided for such types. For now lowering
treats these types as empty structs, which seems like a reasonable
lowering for non-monomorphized unconstrained types.
This sets things up to use `bazel` to run `clang-tidy` using
https://github.com/erenon/bazel_clang_tidy.
I'm fixing issues outside of explorer, and disabling clang-tidy for
targets in explorer that have legacy issues. I was going to disable
clang-tidy for targets in explorer such as interpreter anyways, because
they're slow to parse, and just extended that to the currently failing
targets.
Don't first form a `std::tuple` and then convert it to a struct. Speeds
up compilation of extract.cpp by 15-20%.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: josh11b <josh11b@users.noreply.github.com>
This involves a number of supporting changes:
* The `parse_node;` member of instruction types may now have any type
derived from `Parse::NodeId` and is no longer required to have that
exact type.
* `Parse::Node::Invalid` is now a singleton object of a separate type
that is convertible to `Parse::NodeId` and its descendants. This
replaces the `Invalid` member of its descendants, and avoids having to
write long `NodeIdOneOf<...>` types when initializing variables to
invalid.
* `IndexBase` now allows `==` and `!=` comparisons between its derived
classes and types that are convertible to those types.
* A number of functions in the check stage have been changed to preserve
more type information instead of using `Parse::NodeId`.
* `NodeIdForKind<K>` (also known as `KId`) now has a `Kind` member so it
may be used to declare `NodeIdOneOf<T, U>` types without #including
`parse/typed_nodes.h`.
* `NodeIdForKind<K>` (also known as `KId`) may be implicitly converted
to `NodeIdOneOf<T, U>` if `T::Kind == K` or `U::Kind == K` (executing a
TODO).
Many of the `parse_node` members were not converted since they would
have required more extensive changes. They have been marked with "TODO"
comments.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This is mirroring the structure of codegen/codegen.h, lower/lower.h, and
check/check.h. I recently did lex/lex.h, so parse/parse.h is the last.
Now, the directory's main API file is eponymous with the directory.
I could've used a friend function to avoid making the Tree constructor
public, but in other places we make less use of `friend`, just leaving
things public. This felt more consistent, and simple because it only
affects the constructor.
Goal is to increase type safety, though more work needs to be done (see
added TODOs).
Note that, after this change, check handlers corresponding to deleted
parse node kinds will no longer compile.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Use this new category to replace the unconstrained `NodeId` child of
`MemberAccessExpr ` and `PointerMemberAccessExpr`. For now this new
category matches `IdentifierName` and `BaseName`, but later this will be
expanded to support `a.(b.c)` and `p->(b.c)` syntactic forms.
QUESTION: Is it time to make a `node_category.def` x-macro file?
ANSWER: Not yet.
Tests previously uncovered code. Fix uncovered problems:
* formatting of trace output
* package & import directives need to be classified as declarations
* the problem that meant the previous problem wasn't caught by existing
tests (since `Tree::Verify` didn't check that top-level declarations
match `AnyDeclId`, as required by `Tree::ExtractFile()`).
Previously would fail with:
```
CHECK failure at toolchain/parse/node_kind.cpp:74: Lex::TokenKind::Error == expected_token_kind || token_kind == expected_token_kind: Created parse node with NodeKind NamedConstraintDecl and has_error 1 for lexical token kind Constraint, but expected token kind Semi
```
Issue found by fuzzing.
These are intended to allow the structure of a parse tree node to be
described more precisely in code, to support these use cases:
- Automated checking that the parse tree conforms to the expected
structure. (Added to `Tree::Verify`.)
- Easier reading and understanding of the structure of the parse tree by
toolchain developers. (See `parse/typed_nodes.h`.)
- Easier navigation of the parse tree, for example for tooling uses and
for use when forming diagnostics.
On this last point, an object representing the file may be inspecting
using `Tree::ExtractFile`, as in:
```
auto file = tree->ExtractFile();
for (AnyDeclId decl_id : file.decls) {
// `decl_id` is convertible to a `NodeId`.
if (std::optional<FunctionDecl> fn_decl =
tree->ExtractAs<FunctionDecl>(decl_id)) {
// fn_decl->params is a `TuplePatternId` (which extends `NodeId`)
// that is guaranteed to reference a `TuplePattern`.
std::optional<TuplePattern> params = tree->Extract(fn_decl->params);
// `params` has a value unless there was an error in that node.
} else if (auto class_def = tree->ExtractAs<ClassDefinition>(decl_id)) {
// ...
}
}
```
The `Extract...` functions collect the child nodes into the typed parse
node's fields (internally using a `Tree::SiblingIterator`) for easy
access. However, this is not as fast as directly observing the tree
structure using the postorder strategy being used by the check stage.
These functions rely on using struct reflection on the typed parse node
definitions from `parse/typed_nodes.h` to get the expected structure of
child nodes and then populate them.
Note that validating these in `Tree::Verify` adds significant cost to
it, and is currently included in the parsing stage. Without this change,
a 10 mloc test case of lex & parse takes 4.129 s ± 0.041 s. With this
change, it takes 5.768 s ± 0.036 s.
This builds upon and completes #3393.
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Move handling of mismatched brackets out of the main lexing loop into a
separate pass that is only run if there are mismatched brackets This is
done in preparation for using both lookahead and lookbehind to work out
how to match brackets, and to get this code far away from the hot lexing
loop.
Fix bracket insertion location to be immediately after the token that
we're inserting the bracket after, rather than potentially at the end of
a comment. When there are open brackets at the end of the file, say that
there are open brackets, not that there's a closing bracket without a
matching opening bracket.
We have `StringLiteral`s in multiple other `Carbon` sub-namespaces.
Rename to a more specific name to avoid collisions.
We should likely also rename `Carbon::IntId` -> `Carbon::IntValueId` and
`Carbon::RealId` -> `Carbon::RealValueId`, but this collision is
prioritized because it was blocking work on typed parse nodes which
introduces a `Carbon::Parse::StringLiteralId`.
This builds on the series of changes to NodeKinds, aiming to simplify
the NodeKind implementation a little, also making it clearer that
there's a single associated token for each parse node (or, for
placeholders/invalid parses, not validated).
Note that prior to the relevant changes, there were nodes with multiple
tokens. This change is also locking in the approach of one token per
parse node, by refactoring macros to stop supporting multiple.
Note #3486 rewrites the macro behavior, and is already approved: so this
PR is only for the changed enforcement during error. Also, #3493 already
changed several things to allow any token while this PR was awaiting
review, but this still changes enforcement for `For` and `If`.
This was brought up on
[#toolchain](https://discord.com/channels/655572317891461132/655578254970716160/1182066616456970251),
and I think this any-on-error approach gets at least some support. We
could try setting it to the introducer, but it's quite possible we want
it to be something like the token which led to the parse error, rather
than a static token. That leads to a conclusion that, most typically,
we'll expect arbitrary tokens when error conditions may lead to tokens
which aren't the expected token.
A couple related, recent `CARBON_IF_ERROR` crash fixes can be found in
#3404 and #3424. Something like #3404 would've been needed regardless
because `namespace` didn't have `CARBON_IF_ERROR` before, although I
might've missed the underlying issue with declarations because only
`namespace` had a relevant test (that is, if #3404 had added
`CARBON_ANY_TOKEN_ON_ERROR`, I wouldn't have had a crash in #3462).
#3424 would've been avoided with this change because there was a
`CARBON_IF_ERROR`, and it was just too restrictive.
This leaves a single state for each in the expr loop. I was trying to
think through ways to have per-token states, but they felt sort of
bulky.
Note this is more verbose: but I think the long-term is going to be that
when we start wanting to add handlers, we're going to need to switch to
different names based on the token found. As a consequence, the parse
state logic will end up diverging a little, and we'll just want to align
towards boilerplate handlers.
Short-term, this opens up a path for saying that each parse node
corresponds to precisely one token in success states, and separates out
what were becoming big handler functions in check.
Another incremental change to diagnostic formatting. I simply recurse
over all the tokens in the subtree of a parse node and construct a
`DiagnosticLocation` that covers all of the tokens.
I believe it's nicer for the user to be directed at the entire chunk of
source where the error is occurring rather then just pointing at the
bracketing/terminator tokens, but let me know if you all agree.
StructFieldUnknown was used previously for invalid parses. But we have
added other, more common ways of talking about those; so this is
removing the special-case.
This remains structurally valid although we don't use this operator or
have a design for it, this at least fixes a fuzzer-found crash.
---------
Co-authored-by: josh11b <josh11b@users.noreply.github.com>
This is also doing the parse node split, allowing lower reliance in
formatter on the tokenized buffer (something that I may be touching more
due to import handling).
(Split out of #3410)
Consistently use `ParenExpr` solely for parenthesized single
expressions, and use more syntax-oriented terminology for states and
nodes that might represent either a `ParenExpr` or a tuple literal.
Building on #3463. The PushState+PopState to construct a state feels
worth cleanup. The rest is just kind of making it easier to do without
adding another PushState overload.
Building on #3462, trying to make the flow easier to see, also making a
little more use of macros for boilerplate modifier handling.
---------
Co-authored-by: josh11b <josh11b@users.noreply.github.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Jon Ross-Perkins
josh11bandJon Ross-Perkins
Richard Smith
Chandler CarruthandJon Ross-Perkins
kshokhin
Geoff Romer
Jacob Schneider