Lexer diagnostics used a single `const char*` as their source location,
which only allowed pointing at a single character rather than a run of
characters. This limited the underlining of unrecognized tokens, and
also made escape sequence diagnostics particularly strange by stopping
at whatever point the lexer discovered the error and pointing only at
that location. For example: `"\w"` was reported at the `w`, and `"\x"`
at the closing `"`.
This change tries to always keep the primary location of a diagnostic
the point at which typing (inserting or deleting) might fix the error.
So for `"\x"`, `\x` is underlined, but the position is at the `"` as
that's where you would type a hex code.
The lexer's emitter now takes a special lexing source location that
represents either a single position _between_ two characters, a single
character, or a range of the source buffer. The lexer diagnostics are
updated to construct these for the character range as appropriate.
Parser diagnostics for a missing token were located at the _next_ token,
which is particularly confusing for terminators like missing `;`s. Now
they are located at a position after the previous token, with or without
a space depending on the token, but typically on the same line.
Diagnostics still sort by how far the parser has read.
The parser diagnostics for unclosed groups also now attaches a label to
the opening token when it is on a separate line. And the diagnostics for
mismatched indent in a multi-line string attaches a label to the closing
`'''` that establishes that indent.
There are likely more location improvements we should do here, but this
seemed like a reasonable incremental improvement and is what Claude
found when doing an initial audit. One example where I think we should
make further improvements is around suggesting raw identifiers when we
find an unexpected `Cpp` or `Core` or `base`. We do this consistently
now, but raw identifiers should be rare. A TODO is added for these
cases.
Assisted-by: Claude Code
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Malformed `base` declarations with an omitted colon need two different
recovery paths. For `extend base`, the consumed `extend` modifier
requires the parse tree to retain its `BaseColon` and base expression
children, so this synthesizes an errored `BaseColon` and continues
parsing the expression.
Other malformed forms, such as `base calss X {}`, now use the standard
declaration-error recovery: emit `ExpectedAfterBase`, skip past the
likely declaration end, and form an errored `BaseDecl` without inventing
a colon or cascading diagnostics.
The regression covers `extend base Foo;`, bare `base;`, and the reviewer
counterexample `base calss X {}`.
Tests:
- `prek run --files toolchain/parse/handle_base.cpp
toolchain/parse/testdata/class/fail_base.carbon`
- `./scripts/run_bazelisk.py test -c dbg //toolchain/parse/...`
- `./scripts/run_bazelisk.py test -c dbg //toolchain/testing:file_test`
AI assistance: OpenAI Codex helped inspect the parser recovery path,
implement the change, and run verification. The operator reviewed and
authorized the contribution.
Assisted-by: OpenAI Codex
Also renames variables of that type to match. A follow-up PR will rename
`Parse::StateStackEntry` to `Parse::State`. This is more consistent with
the naming of similar enums elsewhere in the toolchain, and with the
prevailing practice of using `state` rather than e.g. `entry` as the
name of a `StateStackEntry`.
See also discussion
[here](https://discord.com/channels/655572317891461132/963846118964350976/1326280585592700990)
This is a primarily automated change:
- Search & replace for capitalization
-
`(CARBON_DIAGNOSTIC\((?:\n\s+)?\w+,(?:\n\s+)?\s\w+,(?:\n\s+)?\s")([A-Z])`
- `$1\L$2`
- Search & replace for period
-
`(CARBON_DIAGNOSTIC\((?:\n\s+)?\w+,(?:\n\s+)?\s\w+,(?:\n\s+)?\s"(?:[^)]|\n)+)\.("[,)])`
- `$1$2`
- Limited search & replace for `ERROR: ` -> `error: ` in streamed things
- Leaving a TODO for command_line because there's more cleanup that can
be done there
- Modify diagnostic_consumer.cpp
- ERROR -> error
- WARNING -> warning
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Most of these are places where we failed to include a header file and
simply never got an error about this. The fix is to include the header
file.
Most other cases are functions that should have been marked `static` but
were not. Finding all of these was a main motivation for me enabling the
warning despite how much work it is.
One complicating factor was that we weren't including the `handle.h` for
all the state-based handler functions. While this isn't a tiny amount of
code, it is just declarations and doesn't add any extra dependencies. It
also lets us have the checking for which functions need to be `static`
and which don't. For the `parse` library I had to add the `handle.h`
header as well, I tried to match the design of it in `check`.
I have also had to work around a bug in the warning, but given the value
it seems to be providing, that seems reasonable. I've filed the bug
upstream: https://github.com/llvm/llvm-project/issues/94138
I also had to use some hacks to work around limitations of Bazel rules
that wrap `cc_library` rules and don't expose `copts`. I filed a bug for
`cc_proto_library` specifically:
~https://github.com/bazelbuild/bazel/issues/22610~https://github.com/bazelbuild/bazel/issues/4446
Change syntax for package declaration to put the `impl` keyword at the
start and remove the `api` keyword.
To support this, rearrange processing of package, library, and import
declarations to use the general modifier handling support in declaration
parsing rather than special-case logic.
There is an ambiguity in `impl package.Foo as Bar`, which we resolve by
treating `package` as an introducer after a modifier only if it's not
followed by `.`.
When we expect a semicolon, we should be able to provide more standard
recovery.
Note this affects a test of `base`, but it looks like a partial
improvement (the prior line was moving too far). Still, it looks like
recovery isn't handling `{}` quite right. I'm not trying to address that
here though -- leaving a TODO.
This is something I noticed working on
https://github.com/carbon-language/carbon-lang/pull/3884; I think we
have more functions named Diagnose* at present than Emit* or Report*, so
just trying to consolidate. Note a couple Diagnose* functions do a
little more validation, but maybe those should actually be renamed away
(zygoloid had mentioned wanting to generally split out diagnostics to
their own function, and then we'd have it be a more common pattern).
Chandler CarruthandRichard Smith
mstr-six
Geoff Romer
Jon Ross-PerkinsandRichard Smith