Files
carbon-lang/toolchain/lex/string_literal.h
Chandler CarruthandRichard Smith 68569d234d Improve lexer and parser diagnostic locations (#7990)
Lexer diagnostics used a single `const char*` as their source location,
which only allowed pointing at a single character rather than a run of
characters. This limited the underlining of unrecognized tokens, and
also made escape sequence diagnostics particularly strange by stopping
at whatever point the lexer discovered the error and pointing only at
that location. For example: `"\w"` was reported at the `w`, and `"\x"`
at the closing `"`.

This change tries to always keep the primary location of a diagnostic
the point at which typing (inserting or deleting) might fix the error.
So for `"\x"`, `\x` is underlined, but the position is at the `"` as
that's where you would type a hex code.

The lexer's emitter now takes a special lexing source location that
represents either a single position _between_ two characters, a single
character, or a range of the source buffer. The lexer diagnostics are
updated to construct these for the character range as appropriate.

Parser diagnostics for a missing token were located at the _next_ token,
which is particularly confusing for terminators like missing `;`s. Now
they are located at a position after the previous token, with or without
a space depending on the token, but typically on the same line.
Diagnostics still sort by how far the parser has read.

The parser diagnostics for unclosed groups also now attaches a label to
the opening token when it is on a separate line. And the diagnostics for
mismatched indent in a multi-line string attaches a label to the closing
`'''` that establishes that indent.

There are likely more location improvements we should do here, but this
seemed like a reasonable incremental improvement and is what Claude
found when doing an initial audit. One example where I think we should
make further improvements is around suggesting raw identifiers when we
find an unexpected `Cpp` or `Core` or `base`. We do this consistently
now, but raw identifiers should be rare. A TODO is added for these
cases.

Assisted-by: Claude Code

---------

Co-authored-by: Richard Smith <richard@metafoo.co.uk>
2026-11-01 06:22:00 +00:00

119 lines
4.2 KiB
C++

// Part of the Carbon Language project, under the Apache License v2.0 with LLVM
// Exceptions. See /LICENSE for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#ifndef CARBON_TOOLCHAIN_LEX_STRING_LITERAL_H_
#define CARBON_TOOLCHAIN_LEX_STRING_LITERAL_H_
#include <optional>
#include "llvm/ADT/StringRef.h"
#include "llvm/Support/Allocator.h"
#include "toolchain/diagnostics/emitter.h"
#include "toolchain/lex/source_loc.h"
#include "toolchain/lex/token_info.h"
namespace Carbon::Lex {
class StringLiteral {
public:
// A string literal's kind.
enum class Kind : int8_t {
// A character literal is still handled through string literal lexing.
Char,
// A single-line string, `"<content>"`.
SingleLine,
// A multi-line string, `'''<content>'''`.
MultiLine,
// An incorrectly double-quoted multi-line string, `"""<content>"""`.
MultiLineWithDoubleQuotes,
};
// Extract a string literal token from the given text, if it has a suitable
// form. Returning std::nullopt indicates no string literal was found;
// returning an invalid literal indicates a string prefix was found, but it's
// malformed and is returning a partial string literal to assist error
// construction.
static auto Lex(llvm::StringRef source_text) -> std::optional<StringLiteral>;
// Expand any escape sequences and compute the resulting character. This
// handles error recovery internally, but can return nullopt for an invalid
// character.
auto ComputeCharLiteralValue(Diagnostics::Emitter<SourceLoc>& emitter) const
-> std::optional<CharLiteralValue>;
// Expand any escape sequences in the given string literal and compute the
// resulting value. This handles error recovery internally and cannot fail.
//
// When content_needs_validation_ is false and the string has no indent to
// deal with, this can return the content directly. Otherwise, the allocator
// will be used for the StringRef.
auto ComputeStringValue(llvm::BumpPtrAllocator& allocator,
Diagnostics::Emitter<SourceLoc>& emitter) const
-> llvm::StringRef;
// Get the text corresponding to this literal.
auto text() const -> llvm::StringRef { return text_; }
// Determine whether this is a multi-line string literal.
auto kind() const -> Kind { return kind_; }
// Returns true if the string has a valid terminator.
auto is_terminated() const -> bool { return is_terminated_; }
// Returns true if this is a multi-line string literal whose introducer line
// is malformed. Such a literal covers just the introducer line and is never
// terminated.
auto has_invalid_introducer() const -> bool {
return has_invalid_introducer_;
}
private:
struct Introducer;
explicit StringLiteral(llvm::StringRef text, llvm::StringRef content,
bool content_needs_validation, int hash_level,
Kind kind, bool is_terminated,
bool has_invalid_introducer = false)
: text_(text),
content_(content),
content_needs_validation_(content_needs_validation),
hash_level_(hash_level),
kind_(kind),
is_terminated_(is_terminated),
has_invalid_introducer_(has_invalid_introducer) {}
// The complete text of the string literal.
llvm::StringRef text_;
// The content of the literal. For a multi-line literal, this begins
// immediately after the newline following the file type indicator, and ends
// at the start of the closing `"""`. Leading whitespace is not removed from
// either end.
llvm::StringRef content_;
// Whether content needs validation, in particular due to either an escape
// (which needs modifications) or a tab character (which may cause a warning).
bool content_needs_validation_;
// The number of `#`s preceding the opening `"` or `"""`.
int hash_level_;
// Whether this was a single-line string literal, multi-line string literal,
// or a char literal.
Kind kind_;
// Whether the literal is valid, or should only be used for errors.
bool is_terminated_;
// Whether this is a multi-line literal whose introducer line is malformed.
bool has_invalid_introducer_;
};
} // namespace Carbon::Lex
#endif // CARBON_TOOLCHAIN_LEX_STRING_LITERAL_H_