mirror of
https://github.com/carbon-language/carbon-lang.git
synced 2026-09-24 21:20:11 +01:00
Chandler Carruth
f0848b1f5e
Carbon currently requires a comment to be the only non-whitespace on its line. A `//` comment that follows other content on a line, called a _trailing comment_, is a lexer error. This proposal removes that restriction, allowing a comment to follow other content on a line. Everything else about comments is unchanged: a comment still begins with `//`, still requires whitespace after the `//`, and still runs to the end of the line. Carbon continues to provide only line comments; no block or intra-line comments are added. Three observations motivate the change. First, trailing comments are well suited to short _annotations_ attached to a specific entity or value on a line. Second, the lexer design now makes it trivial to lex trailing comments, and in fact requires extra logic and potentially cost to reject them. Third, C++ code routinely uses trailing comments, so allowing them lets Carbon carry the layout of migrated code over directly, rather than reworking each comment to read well in a different structure. Implementation notes (beyond the proposal's design): Keeping trailing comments cheap to lex required a few supporting changes, all of which keep the cost off the lexer's hot path: - The lexer already dispatches `//` to comment lexing wherever it appears, so classifying a comment as trailing is a single O(1) check of whether the `//` is the line's first non-whitespace (`start + indent`). The hot comment path is otherwise unchanged. - That check relies on each line's recorded indentation being its real leading whitespace. Multi-line string literals previously recorded the column where the literal opened for the lines they span; they now record the true (closing-delimiter) indentation instead. - Parser error recovery (`SkipPastLikelyEnd`) had relied on that opening-column indentation to keep tokens following a multi-line string literal attached to the same construct. It now reconstructs that relationship directly by consulting the line on which the literal opened, including when other tokens follow the closing delimiter (such as `''' + "more"`). This is on the cold recovery path. - `CommentData` records the trailing bit in the high bit of its length field, keeping it at 8 bytes. Assisted-by: Claude Code
105 lines
3.8 KiB
C++
105 lines
3.8 KiB
C++
// Part of the Carbon Language project, under the Apache License v2.0 with LLVM
|
|
// Exceptions. See /LICENSE for license information.
|
|
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
|
|
|
|
#ifndef CARBON_TOOLCHAIN_FORMAT_FORMATTER_H_
|
|
#define CARBON_TOOLCHAIN_FORMAT_FORMATTER_H_
|
|
|
|
#include <cstdint>
|
|
|
|
#include "common/ostream.h"
|
|
#include "toolchain/lex/tokenized_buffer.h"
|
|
|
|
namespace Carbon::Format {
|
|
|
|
// Implements Format(); see format.h. It's intended to be constructed and
|
|
// `Run()` once, then destructed.
|
|
//
|
|
// TODO: This will probably need to work less linearly in the future, for
|
|
// example to handle smart wrapping of arguments. This is a simple
|
|
// implementation that only handles simple code. Before adding too much more
|
|
// complexity, it should be rewritten.
|
|
//
|
|
// TODO: Add retention of blank lines between original code.
|
|
//
|
|
// TODO: Add support for formatting line ranges (will need flags too).
|
|
class Formatter {
|
|
public:
|
|
explicit Formatter(const Lex::TokenizedBuffer* tokens, llvm::raw_ostream* out)
|
|
: tokens_(tokens),
|
|
out_(out),
|
|
next_comment_(tokens->comments().begin()),
|
|
comments_end_(tokens->comments().end()) {}
|
|
|
|
// See class comments.
|
|
auto Run() -> bool;
|
|
|
|
private:
|
|
// Tracks the status of the current line of output.
|
|
enum class LineState : uint8_t {
|
|
// There is no output for the current line.
|
|
Empty,
|
|
// The current line has content (possibly just an indent), and does not need
|
|
// a separator added.
|
|
HasSeparator,
|
|
// The current line has content, and will need a separator, typically a
|
|
// single space or newline.
|
|
NeedsSeparator,
|
|
// The current line has content and is complete; a newline is pending but
|
|
// has not yet been emitted. We defer the newline so that a trailing comment
|
|
// can still be attached to this line before it is broken. The newline is
|
|
// materialized by the next content emitted (see `PrepareForPackedContent`)
|
|
// or when the file ends.
|
|
EndOfLine,
|
|
};
|
|
|
|
// Marks the current line as complete, so the next content starts a new line.
|
|
// The newline is deferred rather than emitted immediately, allowing a
|
|
// trailing comment to be attached first. Does not indent, allowing blank
|
|
// lines.
|
|
auto RequireEmptyLine() -> void;
|
|
|
|
// Emits the comment at `next_comment_` and advances past it. A trailing
|
|
// comment is kept on the current line (separated by a space) when there is
|
|
// still content to attach it to; otherwise the comment is emitted on its own
|
|
// line.
|
|
auto EmitComment() -> void;
|
|
|
|
// Ensures there is a separator before adding new content. May do
|
|
// `PrepareForPackedContent` or output a separator space, dependent on line
|
|
// state. Always results in line_state_ being HasSeparator; the caller is
|
|
// responsible for adjusting state if needed.
|
|
auto PrepareForSpacedContent() -> void;
|
|
|
|
// Requires that the current line is indented, but not necessarily a separator
|
|
// space. May output spaces for `indent_`, dependent on line state. Only
|
|
// guarantees the line_state_ is not Empty; the caller is responsible for
|
|
// adjusting state if needed.
|
|
auto PrepareForPackedContent() -> void;
|
|
|
|
// Returns the next token index.
|
|
static auto NextToken(Lex::TokenIndex token) -> Lex::TokenIndex {
|
|
return *(Lex::TokenIterator(token) + 1);
|
|
}
|
|
|
|
// The tokens being formatted.
|
|
const Lex::TokenizedBuffer* tokens_;
|
|
|
|
// The output stream for formatted content.
|
|
llvm::raw_ostream* out_;
|
|
|
|
// The next comment to emit, and one past the last comment.
|
|
Lex::CommentIterator next_comment_;
|
|
Lex::CommentIterator comments_end_;
|
|
|
|
// The state of the line currently written to output.
|
|
LineState line_state_ = LineState::Empty;
|
|
|
|
// The current code indent level, to be added to new lines.
|
|
int indent_ = 0;
|
|
};
|
|
|
|
} // namespace Carbon::Format
|
|
|
|
#endif // CARBON_TOOLCHAIN_FORMAT_FORMATTER_H_
|