This involves a number of supporting changes:
* The `parse_node;` member of instruction types may now have any type
derived from `Parse::NodeId` and is no longer required to have that
exact type.
* `Parse::Node::Invalid` is now a singleton object of a separate type
that is convertible to `Parse::NodeId` and its descendants. This
replaces the `Invalid` member of its descendants, and avoids having to
write long `NodeIdOneOf<...>` types when initializing variables to
invalid.
* `IndexBase` now allows `==` and `!=` comparisons between its derived
classes and types that are convertible to those types.
* A number of functions in the check stage have been changed to preserve
more type information instead of using `Parse::NodeId`.
* `NodeIdForKind<K>` (also known as `KId`) now has a `Kind` member so it
may be used to declare `NodeIdOneOf<T, U>` types without #including
`parse/typed_nodes.h`.
* `NodeIdForKind<K>` (also known as `KId`) may be implicitly converted
to `NodeIdOneOf<T, U>` if `T::Kind == K` or `U::Kind == K` (executing a
TODO).
Many of the `parse_node` members were not converted since they would
have required more extensive changes. They have been marked with "TODO"
comments.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This is mirroring the structure of codegen/codegen.h, lower/lower.h, and
check/check.h. I recently did lex/lex.h, so parse/parse.h is the last.
Now, the directory's main API file is eponymous with the directory.
I could've used a friend function to avoid making the Tree constructor
public, but in other places we make less use of `friend`, just leaving
things public. This felt more consistent, and simple because it only
affects the constructor.
When formatting special values, we currently have `NameId` doing `if
(*this == SelfValue)`, `BoolValue` doing `case 0:`, and `TypeId` doing
`if (index == TypeType.index)`. I'm suggesting we consolidate onto the
`*this == SelfValue` approach for consistency, it seems the easiest to
see the mapping of values.
This mixes in #3552 because I'm adding `Exports` there. I'm suggesting
reformatting `Empty` and `Exports` consistent with `Unreachable`.
This adds a block for exported InstIds, rather than scanning the package
scope. This working down a path discussed last month, which we'll need
to add enclosing namespaces to the Inst in order to complete import of
something like `namespace NS; var NS.a;`
Exports could've been a separate `vector<InstId>` on `SemIR::File`, but
using an entry in `inst_blocks` felt more consistent.
I think this check doesn't offer enough value to try to work around the
false positives. We probably will, at times, check the contents of an
optional without validating because structurally we know it must have a
value. Here though, I believe the `while` will be doing a check of the
value; we shouldn't need a more explicit check.
Goal is to increase type safety, though more work needs to be done (see
added TODOs).
Note that, after this change, check handlers corresponding to deleted
parse node kinds will no longer compile.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Use this new category to replace the unconstrained `NodeId` child of
`MemberAccessExpr ` and `PointerMemberAccessExpr`. For now this new
category matches `IdentifierName` and `BaseName`, but later this will be
expanded to support `a.(b.c)` and `p->(b.c)` syntactic forms.
QUESTION: Is it time to make a `node_category.def` x-macro file?
ANSWER: Not yet.
Tests previously uncovered code. Fix uncovered problems:
* formatting of trace output
* package & import directives need to be classified as declarations
* the problem that meant the previous problem wasn't caught by existing
tests (since `Tree::Verify` didn't check that top-level declarations
match `AnyDeclId`, as required by `Tree::ExtractFile()`).
The explorer encoded the old runfiles layout in a string in one place
that didn't get updated. This causes our instructions for running the
explorer to break. The fix is at least quite simple.
Fixes#3540Fixes#3541
Previously would fail with:
```
CHECK failure at toolchain/parse/node_kind.cpp:74: Lex::TokenKind::Error == expected_token_kind || token_kind == expected_token_kind: Created parse node with NodeKind NamedConstraintDecl and has_error 1 for lexical token kind Constraint, but expected token kind Semi
```
Issue found by fuzzing.
These are intended to allow the structure of a parse tree node to be
described more precisely in code, to support these use cases:
- Automated checking that the parse tree conforms to the expected
structure. (Added to `Tree::Verify`.)
- Easier reading and understanding of the structure of the parse tree by
toolchain developers. (See `parse/typed_nodes.h`.)
- Easier navigation of the parse tree, for example for tooling uses and
for use when forming diagnostics.
On this last point, an object representing the file may be inspecting
using `Tree::ExtractFile`, as in:
```
auto file = tree->ExtractFile();
for (AnyDeclId decl_id : file.decls) {
// `decl_id` is convertible to a `NodeId`.
if (std::optional<FunctionDecl> fn_decl =
tree->ExtractAs<FunctionDecl>(decl_id)) {
// fn_decl->params is a `TuplePatternId` (which extends `NodeId`)
// that is guaranteed to reference a `TuplePattern`.
std::optional<TuplePattern> params = tree->Extract(fn_decl->params);
// `params` has a value unless there was an error in that node.
} else if (auto class_def = tree->ExtractAs<ClassDefinition>(decl_id)) {
// ...
}
}
```
The `Extract...` functions collect the child nodes into the typed parse
node's fields (internally using a `Tree::SiblingIterator`) for easy
access. However, this is not as fast as directly observing the tree
structure using the postorder strategy being used by the check stage.
These functions rely on using struct reflection on the typed parse node
definitions from `parse/typed_nodes.h` to get the expected structure of
child nodes and then populate them.
Note that validating these in `Tree::Verify` adds significant cost to
it, and is currently included in the parsing stage. Without this change,
a 10 mloc test case of lex & parse takes 4.129 s ± 0.041 s. With this
change, it takes 5.768 s ± 0.036 s.
This builds upon and completes #3393.
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
The goal here is to make the representation more uniform so that we can
start adding different kinds of binding -- checked generic bindings and
template bindings -- across both function parameters and local `let`
declarations.
With this change, the entry in the parameter list for the function is
the name binding, not the Param itself, which has some ripple effects on
consumers of that list that want to access the parameter rather than the
binding. This is expected to change again when we start adding more of
the pattern matching SemIR, but this seems good enough for now.
By adding an InstId to the NameScope, we can determine whether the
declaration is being added to a scoped entity (versus a namespace).
The choice of InstId on NameScope is chosen versus other solutions
because, for imports, we want to just have a list of InstIds to import
and, from those, get the containing namespaces for addition. Similar may
also be desirable for printing fully qualified names given a singular
InstId. That means an InstId must have a path to find enclosing name
scopes.
What we're looking at here is:
- NameScopeId knows its InstId. (done here)
- Inst knows the enclosing NameScopeId. (future work)
- To walk up enclosing scopes for an Inst:
1. Fetch the Inst.
2. Find its enclosing NameScopeId (which will be per-declaration due to
Function etc complexity).
3. Fetch the NameScope if not Package scope. (if Package scope, done)
4. Use the InstId on the NameScope to go back to step 1.
Move handling of mismatched brackets out of the main lexing loop into a
separate pass that is only run if there are mismatched brackets This is
done in preparation for using both lookahead and lookbehind to work out
how to match brackets, and to get this code far away from the hot lexing
loop.
Fix bracket insertion location to be immediately after the token that
we're inserting the bracket after, rather than potentially at the end of
a comment. When there are open brackets at the end of the file, say that
there are open brackets, not that there's a closing bracket without a
matching opening bracket.
Directories in `external/` now include versions of dependencies, so
continuing to make that work would require version-locking the
compile_flags.txt. We've previously seen issues with directories being
forgotten; this switches to letting bazel generate everything.
Setup follows instructions at
https://github.com/hedronvision/bazel-compile-commands-extractor?tab=readme-ov-file#first-add-this-tool-to-your-bazel-setup
Maintains `./scripts/create_compdb.py`. Per discussion, new files seem
to work okay -- I'm getting #include completion that clearly detects
files.
Note this creates an `external` directory:
```
╚╡ls -ld external
lrwxrwxrwx 1 jperkins primarygroup 27 Dec 19 14:02 external -> bazel-out/../../../external
```
I felt sort of weird checking that in though, at least alongside
`bazel-execroot` it feels like it should have a `bazel-` prefix.
However, I can't rename it. So omitting it, and leaving it to the script
to generate, felt like a reasonable compromise.
Running takes a few minutes (about 3 for me), and should be faster on
reruns. It does print a couple warnings, just tree_sitter missing
parser.c and _GNUC_PREREQ errors from m4.
Proposal to focus implementation effort for the next 1-2 years on the
Carbon
toolchain instead of the explorer. This will impact the explorer in a
few ways:
- We will keep the explorer's code in place, building, and passing its
basic
tests. It can remain a good baseline for exploring Carbon's language
features.
- We won't prioritize expanding the explorer's coverage of Carbon
features or
other improvements -- it is good enough for what we need until the
toolchain
catches up.
- We will stop actively fuzzing and expanding test coverage for the
explorer.
- Eventually, when we want to resume work on the explorer, we'll
evaluate the
best platform to build on -- the current explorer codebase or on top of
the
toolchain's semantic IR.
Also tries to update the core readme and contributing docs to reflect
this.
Now that both zlib and zstd are handled with bzlmod stuff, they use
their own repository names rather than the repo-rules spelling.
Sadly, the target determinator tool didn't detect that this test could
be impacted so this slipped past our merge queue.
The use of zlib-ng seemed to work previously, but may have only done so
because of some combination of an implementation detail of
`strip_include_prefix` in older versions of Bazel and some happenstance
of a system installed `zlib.h` being possible to find. With Bazel 7 it
started breaking on my machine which doesn't have a system `zlib.h`.
Using the "normal" zlib and the bzlmod rigging for it is simple and
seems to work.
Also adds the Linux AArch64 component of the Bazel lock file which was
necessary to test this on my Arm Linux machine.
We have `StringLiteral`s in multiple other `Carbon` sub-namespaces.
Rename to a more specific name to avoid collisions.
We should likely also rename `Carbon::IntId` -> `Carbon::IntValueId` and
`Carbon::RealId` -> `Carbon::RealValueId`, but this collision is
prioritized because it was blocking work on typed parse nodes which
introduces a `Carbon::Parse::StringLiteralId`.
Building on #3505, the toolchain and llvm rules require a little more
special-casing to get them to work well. This also moves
libprotobuffer_mutator, but that one's more minor. The migration
encounters more quirks in repo naming as seen by various queries.
This changes some of the toolchain work that was recently done for bazel
7 in #3496, dropping a bzl file I'd suggested to add, instead using
`:all` for toolchain registration. (somewhat as an improvement, somewhat
just to avoid a `load`)
Remaining in the WORKSPACE are example code repos and tree sitter rules.
Neither of these are part of the main toolchain builds, and so will
probably be lower impact if there's a good solution for them.
Some background information is at https://bazel.build/external/migration
Trying to handle the simple cases first. This adds a requirement for
bazel 7 due to differences in bzlmod handling between 6 and 7 (also
discussed on
[#infra](https://discord.com/channels/655572317891461132/707150492370862090/1184942191412510720)).
Bazel seems to be okay with a partial migration such as ths.
The python import behavior has subtly shifted, so `carbon.` is no longer
part of import paths. There's a version-incompatible change for `@@`.
bzlmod makes repos sometimes show as `name~version`.
`target-determinator` seems to be okay with `@@` after a version update.
Things not moved here are things that basically need more dep work:
- clang_register_toolchains because I need to dive into its format.
- llvm-project because we need something slightly atypical, I need to
make sure patching and the repo work carries over.
- com_google_libprotobuf_mutator is sufficiently atypical that it
doesn't have a module already, but should be one of the easier things to
fix.
- brotli/woff2: I think we should actually consider removing these. But
again, they're not trivial moves.
- treesitter due to toolchain registration, which has shifted a bit.
- rules_nodejs because treesitter depends on it in an awkward way to
migrate.
This builds out a little infrastructure for one name scope to `extend`
another. We'll need more refinement here to cover other cases, but this
should provide some foundation for that future work.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This builds on the series of changes to NodeKinds, aiming to simplify
the NodeKind implementation a little, also making it clearer that
there's a single associated token for each parse node (or, for
placeholders/invalid parses, not validated).
Note that prior to the relevant changes, there were nodes with multiple
tokens. This change is also locking in the approach of one token per
parse node, by refactoring macros to stop supporting multiple.
Note #3486 rewrites the macro behavior, and is already approved: so this
PR is only for the changed enforcement during error. Also, #3493 already
changed several things to allow any token while this PR was awaiting
review, but this still changes enforcement for `For` and `If`.
This was brought up on
[#toolchain](https://discord.com/channels/655572317891461132/655578254970716160/1182066616456970251),
and I think this any-on-error approach gets at least some support. We
could try setting it to the introducer, but it's quite possible we want
it to be something like the token which led to the parse error, rather
than a static token. That leads to a conclusion that, most typically,
we'll expect arbitrary tokens when error conditions may lead to tokens
which aren't the expected token.
A couple related, recent `CARBON_IF_ERROR` crash fixes can be found in
#3404 and #3424. Something like #3404 would've been needed regardless
because `namespace` didn't have `CARBON_IF_ERROR` before, although I
might've missed the underlying issue with declarations because only
`namespace` had a relevant test (that is, if #3404 had added
`CARBON_ANY_TOKEN_ON_ERROR`, I wouldn't have had a crash in #3462).
#3424 would've been avoided with this change because there was a
`CARBON_IF_ERROR`, and it was just too restrictive.
This leaves a single state for each in the expr loop. I was trying to
think through ways to have per-token states, but they felt sort of
bulky.
Note this is more verbose: but I think the long-term is going to be that
when we start wanting to add handlers, we're going to need to switch to
different names based on the token found. As a consequence, the parse
state logic will end up diverging a little, and we'll just want to align
towards boilerplate handlers.
Short-term, this opens up a path for saying that each parse node
corresponds to precisely one token in success states, and separates out
what were becoming big handler functions in check.
Another incremental change to diagnostic formatting. I simply recurse
over all the tokens in the subtree of a parse node and construct a
`DiagnosticLocation` that covers all of the tokens.
I believe it's nicer for the user to be directed at the entire chunk of
source where the error is occurring rather then just pointing at the
bracketing/terminator tokens, but let me know if you all agree.
josh11b
Jon Ross-Perkins
Nicolas Gonzalez
Geoff Romer
Christopher Di Bella
Uriel García
Jacob Schneider