mirror of
https://github.com/carbon-language/carbon-lang.git
synced 2026-10-04 22:02:52 +01:00
Chandler Carruth
f0848b1f5e
Carbon currently requires a comment to be the only non-whitespace on its line. A `//` comment that follows other content on a line, called a _trailing comment_, is a lexer error. This proposal removes that restriction, allowing a comment to follow other content on a line. Everything else about comments is unchanged: a comment still begins with `//`, still requires whitespace after the `//`, and still runs to the end of the line. Carbon continues to provide only line comments; no block or intra-line comments are added. Three observations motivate the change. First, trailing comments are well suited to short _annotations_ attached to a specific entity or value on a line. Second, the lexer design now makes it trivial to lex trailing comments, and in fact requires extra logic and potentially cost to reject them. Third, C++ code routinely uses trailing comments, so allowing them lets Carbon carry the layout of migrated code over directly, rather than reworking each comment to read well in a different structure. Implementation notes (beyond the proposal's design): Keeping trailing comments cheap to lex required a few supporting changes, all of which keep the cost off the lexer's hot path: - The lexer already dispatches `//` to comment lexing wherever it appears, so classifying a comment as trailing is a single O(1) check of whether the `//` is the line's first non-whitespace (`start + indent`). The hot comment path is otherwise unchanged. - That check relies on each line's recorded indentation being its real leading whitespace. Multi-line string literals previously recorded the column where the literal opened for the lines they span; they now record the true (closing-delimiter) indentation instead. - Parser error recovery (`SkipPastLikelyEnd`) had relied on that opening-column indentation to keep tokens following a multi-line string literal attached to the same construct. It now reconstructs that relationship directly by consulting the line on which the literal opened, including when other tokens follow the closing delimiter (such as `''' + "more"`). This is on the cold recovery path. - `CommentData` records the trailing bit in the high bit of its length field, keeping it at 8 bytes. Assisted-by: Claude Code
118 lines
4.2 KiB
C++
118 lines
4.2 KiB
C++
// Part of the Carbon Language project, under the Apache License v2.0 with LLVM
|
|
// Exceptions. See /LICENSE for license information.
|
|
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
|
|
|
|
#ifndef CARBON_TOOLCHAIN_LEX_STRING_LITERAL_H_
|
|
#define CARBON_TOOLCHAIN_LEX_STRING_LITERAL_H_
|
|
|
|
#include <optional>
|
|
|
|
#include "llvm/ADT/StringRef.h"
|
|
#include "llvm/Support/Allocator.h"
|
|
#include "toolchain/diagnostics/emitter.h"
|
|
#include "toolchain/lex/token_info.h"
|
|
|
|
namespace Carbon::Lex {
|
|
|
|
class StringLiteral {
|
|
public:
|
|
// A string literal's kind.
|
|
enum class Kind : int8_t {
|
|
// A character literal is still handled through string literal lexing.
|
|
Char,
|
|
|
|
// A single-line string, `"<content>"`.
|
|
SingleLine,
|
|
|
|
// A multi-line string, `'''<content>'''`.
|
|
MultiLine,
|
|
|
|
// An incorrectly double-quoted multi-line string, `"""<content>"""`.
|
|
MultiLineWithDoubleQuotes,
|
|
};
|
|
|
|
// Extract a string literal token from the given text, if it has a suitable
|
|
// form. Returning std::nullopt indicates no string literal was found;
|
|
// returning an invalid literal indicates a string prefix was found, but it's
|
|
// malformed and is returning a partial string literal to assist error
|
|
// construction.
|
|
static auto Lex(llvm::StringRef source_text) -> std::optional<StringLiteral>;
|
|
|
|
// Expand any escape sequences and compute the resulting character. This
|
|
// handles error recovery internally, but can return nullopt for an invalid
|
|
// character.
|
|
auto ComputeCharLiteralValue(Diagnostics::Emitter<const char*>& emitter) const
|
|
-> std::optional<CharLiteralValue>;
|
|
|
|
// Expand any escape sequences in the given string literal and compute the
|
|
// resulting value. This handles error recovery internally and cannot fail.
|
|
//
|
|
// When content_needs_validation_ is false and the string has no indent to
|
|
// deal with, this can return the content directly. Otherwise, the allocator
|
|
// will be used for the StringRef.
|
|
auto ComputeStringValue(llvm::BumpPtrAllocator& allocator,
|
|
Diagnostics::Emitter<const char*>& emitter) const
|
|
-> llvm::StringRef;
|
|
|
|
// Get the text corresponding to this literal.
|
|
auto text() const -> llvm::StringRef { return text_; }
|
|
|
|
// Determine whether this is a multi-line string literal.
|
|
auto kind() const -> Kind { return kind_; }
|
|
|
|
// Returns true if the string has a valid terminator.
|
|
auto is_terminated() const -> bool { return is_terminated_; }
|
|
|
|
// Returns true if this is a multi-line string literal whose introducer line
|
|
// is malformed. Such a literal covers just the introducer line and is never
|
|
// terminated.
|
|
auto has_invalid_introducer() const -> bool {
|
|
return has_invalid_introducer_;
|
|
}
|
|
|
|
private:
|
|
struct Introducer;
|
|
|
|
explicit StringLiteral(llvm::StringRef text, llvm::StringRef content,
|
|
bool content_needs_validation, int hash_level,
|
|
Kind kind, bool is_terminated,
|
|
bool has_invalid_introducer = false)
|
|
: text_(text),
|
|
content_(content),
|
|
content_needs_validation_(content_needs_validation),
|
|
hash_level_(hash_level),
|
|
kind_(kind),
|
|
is_terminated_(is_terminated),
|
|
has_invalid_introducer_(has_invalid_introducer) {}
|
|
|
|
// The complete text of the string literal.
|
|
llvm::StringRef text_;
|
|
|
|
// The content of the literal. For a multi-line literal, this begins
|
|
// immediately after the newline following the file type indicator, and ends
|
|
// at the start of the closing `"""`. Leading whitespace is not removed from
|
|
// either end.
|
|
llvm::StringRef content_;
|
|
|
|
// Whether content needs validation, in particular due to either an escape
|
|
// (which needs modifications) or a tab character (which may cause a warning).
|
|
bool content_needs_validation_;
|
|
|
|
// The number of `#`s preceding the opening `"` or `"""`.
|
|
int hash_level_;
|
|
|
|
// Whether this was a single-line string literal, multi-line string literal,
|
|
// or a char literal.
|
|
Kind kind_;
|
|
|
|
// Whether the literal is valid, or should only be used for errors.
|
|
bool is_terminated_;
|
|
|
|
// Whether this is a multi-line literal whose introducer line is malformed.
|
|
bool has_invalid_introducer_;
|
|
};
|
|
|
|
} // namespace Carbon::Lex
|
|
|
|
#endif // CARBON_TOOLCHAIN_LEX_STRING_LITERAL_H_
|