Adds subset conversion of `NodeIdOneOf` due to the choice usage, plus
the pre-existing TODO. Fixes incorrect information about nodes on
StructLiteral and TupleLiteral.
After this change, `UncheckedLoc` is only used in a couple import
contexts (hard to verify) plus `InstStore::GetWithLocId`.
Building on #5120, make a variant of `AddNode` that returns typed nodes,
and replace `UnsafeMake` uses with it. This switches to templating in
import parsing so that we can get type validation.
With this change, `UnsafeMake` ends up used in three places: `Tree::As`,
`Tree::TryAs`, and `Context::AddNode`. That should mean that all typed
nodes are verified.
This flows out of #5084 and trying to reduce UnsafeMake use. It turns
out imports and namespaces were using unexpected node kinds (previously
ImportIntroducer instead of ImportDecl, for example). This fixes and
adds validation.
I was uncertain about whether to just remove the is_convertible check,
since I don't see it as motivating creation of a conversion between
NodeIdOneOf types. So I've just left a TODO for now.
According to approved syntax at
https://github.com/carbon-language/carbon-lang/blob/trunk/proposals/p3848.md#syntax-defined,
`fn F[]` without explicit parameters should be valid. This makes it
work, then adds some validation to prevent `class C[]` in check.
Note that for `fn`, positional parameters are a TODO -- but this allows
me to test validation in `fn destroy[]` which is rejected, not just a
TODO.
This doesn't change functionality, but I was seeing better diagnostics
in VS Code.
This also changes the NodeId constructors for related types (also
NodeCategory and NodeIdForKind) to use UnsafeMake for construction. That
originated from avoiding ambiguity coming from `requires`, but the
constructor mode is also one we should typically avoid (e.g., preferring
`Parse::Tree::As`).
Use CRTP to eliminate per-type declarations of `None`. Note this adds
`None` to a few that may not need it, but eliminates a lot off
boilerplate.
Note this leaves `GenericInstIndex::None` because it has a more complex
construction.
Also fix `InstId::InitTombstone` to be `NoneIndex - 1`
We had a long discussion of this, so trying to document what seems to be
the conclusion... and also clean up the exceptions that I could find.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
Noted CopyNameFromImportIR while glancing around (this one's interesting
because it's NameId, not void nor auto), did a scan just for a few other
cases. Not an exhaustive fix, and TBH assuming we'd prefer `auto ... ->
auto` since equivalent Carbon syntax would probably be `fn ... -> auto`
In line with the proposal in #4682, this changes the array syntax to be
array(T, N). `array` is a builtin keyword which must be followed by
parens containing two expressions and a separating comma.
The array type expression is still fully builtin, it does not forward to
a Core.Array library type yet. It merely adds the `ArrayType`
instruction, as was done with the previous syntax.
Followup work will change the instruction to reference to Core.Array,
once the library type exists and can be used directly.
---------
Co-authored-by: zygoloid <richard@metafoo.co.uk>
Parameter lists need substantially different treatment than tuple
patterns in other contexts, so this change splits them into separate
parse node kinds.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Rearrange `NodeCategory` printing so we get a compile-time error for
missing switch cases if it's missing any categories. Add several missing
categories.
In passing, fix some minor things in the `NodeCategory` class
definition, and fix an overly-permissive typed node.
This adds support for choice types at a similar level to that of a C
enum, where each alternative has a name but no additional
data/parameters attached to it. We generate a TODO diagnostic if
parameters are specified.
Because there's no extra data, the storage is a simple unsigned integer
discriminant of the smallest possible size.
A choice without any alternatives is not constructible. A choice with a
single alternative is, and has an empty tuple in place of a discriminant
since it has only one state. The empty tuple is used to make the class
non-constructible. This can be improved.
Each alternative is turned into a let binding on the choice that is a
value of the choice with that alternative set as the active one in the
discriminant. This isn't possible to write in user code with a class
right now, since the let binding has the same type as the choice
(which is a class) it is within. It's possible to generate it in semir
however by adding the binding after the class is marked complete.
When I open a .def file, there are often 4 errors:
- The #error
- The #define is not defined
- Missing `;`
- Identifier naming
This PR is meant to disable all of these, since they can be distracting
from fixable diagnostics.
Change parse tree from `template (T:! type)` to `(template T):! type`,
so that we have information about whether a binding is a template
binding available when forming the representation of the binding
pattern. This incidentally fixes a bug that we would accept `template
addr A:! B` instead of the intended `addr template A:! B`.
Track whether a symbolic binding is a template binding on the
`EntityName` object. I'm borrowing a bit from the `CompileTimeBindIndex`
for this in order to avoid making `EntityName`s larger. Longer-term, we
should think about using a different representation for symbolic
bindings, to avoid including these fields in all `EntityName`s, but
that's out of scope for this change.
So far, template bindings are treated as having the same phase as
checked bindings, but that will change in a future PR.
To do this, we restructure the parse tree to make `forall` a leaf node
that comes before the parameter list.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Change representation of package names from `IdentifierId` to
`PackageNameId`, and add a special value `PackageNameId::Core` for the
Core package. Add a `Core` expression to name the Core package, and
support for parsing the `Core` keyword in `package` and `import`
declarations.
For now, I've made no changes to instruction fingerprinting or name
mangling. This means that fingerprints and mangled names will collide
between names in the `Core` package and names in a `r#Core` package. See
#4908.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Also makes the style guide explicitly comment on void, but this was the
intent IIRC because it matches Carbon's `-> ()` (and "always" versus
"except for void", which we definitely went back and forth on).
Includes adjusting function pointers, which I definitely forget this
syntax works sometimes.
Excludes utils/tree_sitter/src/scanner.c because it claims to be C, but
really we should probably fix that to be cpp.
When a node kind's Id::Kind is determined from its category, don't also
require it to be listed in the switch over all node kinds. This was both
redundant and also error prone -- and in practice for several node
kinds, the Id::Kind computed in the two different ways was different.
Instead, have the switch over node kinds handle only special cases that
can't be handled by their category, and enforce that each node kind has
an Id::Kind specified in exactly one way via checks in the .cpp file.
This refines the previous change in #4280 -- we still get the improved
errors for missing updates, but now also don't require redundant
additions to the switch.
At present, we typically define a DiagnosticConverter, then store an
instance of it and a DiagnosticEmitter that wraps it. This is relatively
minor in general, but I've been trying to create more self-contained
DiagnosticEmitter classes (which hold their own DiagnosticConverter,
similar to NullDiagnosticEmitter), and there it just gets in the way.
Since we don't reuse DiagnosticConverter instances, this combines the
definition into DiagnosticEmitter. Mainly this means we don't have a
separate object in play, and less to carry around.
The most impact is probably to SemIRDiagnosticConverter, which was also
the most complex. Now `SemIRLocDiagnosticEmitter`, this gets some
different construction flow. Note in the PR I've split the file rename
to its own commit, to try to help delta views. However, the most
substantial parts of the refactoring are split into #4876, which this
depends upon.
At present, lower depends on `Check::SemIRDiagnosticConverter` for debug
info. That was to support a quick implementation of debug info, but
isn't great because it's both an unusual dependency on check's
implementation, and relying on diagnostic structures for debug info.
This cleans that up by splitting relevant logic out to a library in
sem_ir, and having lowering use sem_ir's library instead of check's.
Additionally, a small refactoring of `Parse::TreeAndSubtrees` to allow
getting locations in lowering without going through a `DiagnosticLoc`.
I'm adding `Parse::GetTreeAndSubtreesFn` in because it's a complex
signature to have in so many spots.
I chose to have `ResolveNodeId` return a `SmallVector` because it seemed
likely to be fairly compact, but that could also be using an optional
callback to handle resolved node IDs, possibly just returning the last
entry. This could be switched if preferred.
Note this change shouldn't affect behavior, it's just moving code
around.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
I'm planning on eliminating DiagnosticConverter. As part of this,
collapse it into diagnostic_emitter.h, and refactoring the header a
little so that it's more readable when making changes.
I'm looking at eliminating `DiagnosticConverter`. This change removes
`NodeLocConverter` (albeit adding `UnitAndImportsDiagnosticConverter`),
and in doing so, refactors lex conversion functions to extract them out
from the `DiagnosticConverter` functions.
I'll be following up with changes that collapse `DiagnosticConverter`
logic into `DiagnosticEmitter` locations. The intent is that we
shouldn't need separate ownership of both types.
Based on [the lastest thinking on
#4672](https://github.com/carbon-language/carbon-lang/issues/4672#issuecomment-2606209281)
, require a full syntactic match for impl redeclaration, instead of
excluding the `where` restriction. This means no updates to the impl
witness on redeclaration, and no diagnostics that those updates are
consistent.
Not included in this PR, but will need to be done in the future:
* Support for assigning values to associated constants in the body of
the impl definition. This will require moving the checking that
non-function associated constants are set from the definition start to
definition end.
* Identify semantic redeclarations that are not syntactic matches to
give a failed redeclaration diagnostic. This should be done once we are
already identifying impl declarations with the same type structure in
order to require they be identified in an impl_priority/match_first
block.
* Merging of the functions in `check/impl.cpp` that are now always
called together.
Also add some test coverage of `where` parsing I developed in PR I've
now abandoned because of this new simplification of the impl
redeclaration semantics.
---------
Co-authored-by: Josh L <josh11b@users.noreply.github.com>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Came up on #4831, style:
"For a type T whose values can be compared for equality, define a
non-member operator== and document when two values of type T are
considered equal."
https://google.github.io/styleguide/cppguide.html#Operator_Overloading
Note while we could put some of these out-of-line, it's helpful to keep
them inside the braces:
- For private member access
- For templated cases so that we aren't duplicating templates
- Very mild preference for keeping class's API documented within the
braces
This is a followup from #4834, I searched for "invalid" uses in our
codebase. This is mostly changing comments, and a couple debug
functions, but shouldn't affect testable behavior.
Note a couple things I'll highlight as not changing (but could) are:
- `ReturnTypeInfo::is_valid`
- `"invalid"` uses in the formatter
- `AddInvalid` for `!has_value` in `inst_fingerprinter` (because the
cases it's called sound invalid-ish)
High level, replacing `Id::Invalid` with `Id::None` and `Id::is_valid`
with `Id::has_value` for clarity, as discussed
[here](https://discord.com/channels/655572317891461132/655578254970716160/1331664574545395794).
The `IntId` refactoring is needed together with `AnyIdBase` because it's
also used with `ValueStore`.
Note, trying to be careful not to rewrite `EnumBase::InvalidIndex`, or
`is_valid` in general (e.g., `IdKind::is_valid`).
I've tried to sequence commits here:
1. Automatic replacements:
- `((?:Id|Index)(?: |::|\(|Base(?:\(|::)))Invalid((?:Index)?\W)` ->
`$1None$2`
- `<invalid>` -> `<none>`
- `InvalidNodeId` -> `NoneNodeId`
- `/\*invalid\*/` -> `/*none*/`
- `id((?:_|\(\))(?:\.|->))is_valid` -> `id$1has_value`
2. Manual edits:
- In `int.h` and `int_test.cpp`
- `IntT` has `is_value`, which I'm renaming to `is_embedded_value`.
- Manual edits to comments in this file.
- `AnyIdBase` and `IdBase`
- Declaration of `is_valid` -> `has_value`, `InvalidIndex` ->
`NoneIndex`.
- In `ids.h` and `ids.cpp`
- `is_valid` -> `has_value`
- `// An explicitly invalid ID.` -> `// An ID with no value.`; similar
for index
- Various math on `InvalidIndex` -> `NoneIndex`
- Various mentions of "valid" in comments
- In `value_store.h`, for `IdT::Invalid`, plus one comment
- In `impl.h` and `tokenized_buffer.h`, we had different initialization
of `::None` values (versus `ids.h` syntax) that I fixed manually.
- Spot checks to compile
- Particularly where `is_valid` replacements didn't catch spots due to
different naming.
3. Autoupdate tests
4. verbose.carbon (NOAUTOUPDATE)
5. Comment spot checks
Note there are probably other mentions of "Invalid" that should be swept
up, but I'd like to argue for merging and separating out remaining
cleanup since this is so sweeping (and likely to hit merge conflicts
from churn). We'll probably have lingering mentions of "invalid" for a
bit regardless, just because there are uses of "invalid" in non-Id APIs.
This adds a RawStringOstream. Versus TestRawOstream, which is
consolidated over to RawStringOstream, it uses a string for storage
instead of a vector, mainly to support move-to-string semantics. Versus
llvm::raw_string_ostream, it owns the string and supports pwrite (which
is needed for driver and its fd_ostream compatibility requirement).
This converts most uses of llvm::raw_string_ostream, leaving behind a
few in InstNamer that explicitly cannot own the string, such as:
```
llvm::raw_string_ostream(name)
<< "_" << tree.tokens().GetColumnNumber(token);
```
I have this as its own library so that it can use CHECK.
Yes this doesn't save much code, but it's code we repeatedly write.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
This removes some churn when adding new diagnostic cases to test files
(where previous to this change the newly added newline would cause the
previous diagnostic CHECKs to be updated including changes to the line
number because the CHECK for the blank line meant an extra line between
CHECK and source line).
A few alternatives discussed here:
https://discord.com/channels/655572317891461132/655578254970716160/1329573358475673723
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This change splits `NodeKind::IdentifierName` into separate node kinds
depending on whether the identifier is followed by parameters, and
similarly splits `NameQualifier` based on whether the qualifier has
parameters. This enables us to only push a pattern block when it's
actually needed, rather than "defensively" pushing one when it might be
needed.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This change deliberately breaks away from the line/column ordering, and
instead focuses on a last byte offset corresponding to the final token
processed as part of producing the message. Where that's equal, this
maintains stable ordering in order to reflect the order that diagnostics
were produced.
The intent of this approach is that lex, parse, and check diagnostics
are interleaved based on where they are produced, but that
subexpressions still have diagnostics emitted prior to containing
expressions. In particular, the prior line/column sort essentially
sorted on the _start_ of where a diagnostic was associated, and this is
closer to sorting based on the _end_. As a consequence, something like
`F(1 2)` will have the error for `1 2` emitted _before_ a diagnostic for
`F(1 2)` not matching parameters, instead of _after_.
In check, we track the last handled node. This provides a
last_byte_offset _separate_ from where a diagnostic is associated. The
intent is that this creates an ordering of diagnostics which may be
associated with earlier code, to cause the diagnostics to be emitted
later. An example consequence of this is the change in ordering of
modifier diagnostics: we are diagnosing those from the same place, but
they have the same last_byte_offset, so we print them out in the order
produced.
I've added similar tracking to parse, but cannot identify any test which
is affected by it (note the separate commit, I thought about this late).
I'm not sure whether we have good out-of-order errors we could produce
for this.
A significant number of tests have reordered diagnostics as a
consequence of this change, so this change does not add further testing.
These don't fully work in check and beyond yet, because they're not
added into lexical lookup, but already mostly do the right thing.
Per #3407, disallow namespace declarations anywhere other than at file
scope for now.
We don't treat statements starting with a packaging introducer keyword
(`package`, `library`, `import`) as declarations because they're
sufficiently unlikely to occur that the error recovery doesn't seem
important, and this avoids needing to disambiguate `package.` at the
start of an expression.
Pursuant to discussion regarding #4699, turn on
`misc-non-private-member-variables-in-classes` using the
`IgnoreClassesWithAllMemberVariablesBeingPublic` flag (the check treats
structs as classes, so we need this for structs with all-public
members). Updates the style guide notes to match, which should be pretty
minor due to the scoping of test fixtures.
Also fixes some underscore uses in test files on the way. Basically this
is keeping the style for [class data member
naming](https://google.github.io/styleguide/cppguide.html#Variable_Names)
even while making them public.
Disables three new warnings because they lean more towards style
conflicts than fixes. I've brought these up on #style.
Other than that, mostly fixing basic issues, and things that
clang-tidy-20 seems to fire where clang-tiday-16 didn't. One particular
curious case is `llvm::StringLiteral::data()` uses, which are flagged as
not strictly null-terminated; I'm switching to `const char*` in those
spots which matches `llvm::formatv`'s format argument, but feels worse.
I'm removing `run_clang_tidy.py` here because I'm observing it give
fewer warnings than `bazel build --config=clang-tidy -k
//toolchain/...`. The latter matches how we enforce in GitHub actions
(and also caches results, and suppresses output for files that have no
issues), so I'm dropping the bespoke script.
Just a small packing optimization. We currently have 222 `NodeKinds`, so
this reduces us to just 30ish more we can add without needing to pack
more. However, if we did, there would be a couple options for bringing
the count down by reusing `NodeKinds` and disambiguating based on the
token kind (the 29 infix operators as an example). Or we could just undo
this.
I'm expecting this to yield a small improvement. I'll see if I can get
better numbers since my machine's not really reliable, but here are some
basic values.
Also suggesting to draw the use of `::RawEnumType` for `TokenKind`,
since bit packing appears to work without it. Hoping the `static_assert`
is easier for people to understand the size of the field.
With the change:
```
----------------------------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations Bytes Lines Tokens
----------------------------------------------------------------------------------------------------------------------------
BM_CompileAPIFileDenseDecls<Phase::Parse>/256 50399 ns 50359 ns 14336 104.588M/s 3.87217M/s 21.8629M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/1024 237823 ns 237629 ns 3072 136.721M/s 4.11986M/s 24.2058M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/4096 997645 ns 996771 ns 768 142.343M/s 4.04105M/s 23.9363M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/16384 4020308 ns 4018319 ns 192 152.041M/s 4.05966M/s 24.0874M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/65536 16691390 ns 16683058 ns 48 151.317M/s 3.92374M/s 23.2936M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/262144 75265735 ns 75233476 ns 8 135.842M/s 3.48421M/s 20.6862M/s
```
Without the change:
```
----------------------------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations Bytes Lines Tokens
----------------------------------------------------------------------------------------------------------------------------
BM_CompileAPIFileDenseDecls<Phase::Parse>/256 51515 ns 51480 ns 13312 102.312M/s 3.78789M/s 21.387M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/1024 241040 ns 240900 ns 3072 134.865M/s 4.06392M/s 23.8771M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/4096 985593 ns 984657 ns 768 144.094M/s 4.09077M/s 24.2308M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/16384 4109327 ns 4105496 ns 192 148.813M/s 3.97345M/s 23.576M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/65536 17459655 ns 17446006 ns 48 144.7M/s 3.75215M/s 22.275M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/262144 80802815 ns 80737489 ns 8 126.581M/s 3.24668M/s 19.276M/s
```
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
- Provide `Check::Dump(context, arg)` and similar.
- gdb and lldb should do contextual lookup, and `call Dump(*this,
Lex::TokenIndex::Invalid)` has been tested with gdb.
- Since this is only for debug, keeps the functions fully separated from
code.
- Uses alwayslink to ensure objects are correctly linked, even though
there are no calls.
- `-Wno-missing-prototypes` is needed when we don't have forward
declarations.
- Code is not linked in opt builds, using `#ifndef NDEBUG`.
- This probably could be doing something in BUILD files with a
`select()`, but the `#ifndef` seemed easier.
This is based on #4620, but uses free functions instead of member
functions.
Co-authored-by: Dana Jansens <danakj@orodu.net>
---------
Co-authored-by: danakj <danakj@orodu.net>
Jon Ross-Perkins
Geoff RomerandJon Ross-Perkins
czapiga
David BlaikieandJon Ross-Perkins