mirror of
https://github.com/carbon-language/carbon-lang.git
synced 2026-11-16 14:48:11 +00:00
## Summary ## Restructures the lexer to first scan the entire source text for newlines and create all the line structures needed. Doing this up-front makes it easy to produce an optimized version with minimal complexity. Currently, it leverages the system `memchr`, but even when expanded to handle more complex cases like CR+LF line endings, being isolated in this way will result in a significantly simpler implementation. This change improves the lexing of comment lines significantly by skipping their contents immediately. The overhead of the pre-scan is unmeasurable in all realistic benchmarks, and 10-30% in benchmarks consisting almost entirely of blank lines or comments. The improvement of comment lexing with average length comment lines mixed with code starts at 20% and goes up. Regressing blank line handling for non-empty comments seems like the right tradeoff (by far). ## Background and details ## One weak point in the lexer implementation were large runs of comments. While those aren't terribly common, they shouldn't present a hazard to the lexer performance. A bit more common is a pattern of comments like the following: ```carbon // Some method comment here. fn SomeMethodName(...) -> ...; // Some other method comment here. fn SomeOtherMethodName(...) -> ...; ``` Here, the lexer spends an inordinate amount of time getting from the `\n` after the first semicolon to the `fn` token. It has to skip a blank line, scan a line, find the `//` comment start, then scan to find the next `\n`, and then scan horizontal whitespace, etc. It is tempting to build a scanner *exactly* for this. In fact, I built one, and I can publish it in a PR if folks are interested in what it looks like. For x86-64, the PSHUFB trick used for scanning identifiers technically works. But it is *complicated*. Amazingly so. 150 lines of very subtle code with subtle performance pitfalls at ever turn. I felt very uncomfortable submitting it, but we can always go back to it. Nothing I've come up with quite matches it for sheer speed. However, most of the complexity and time is spent walking from a `//` to the end of the line. And *that* is something we can do very simply. In fact, there is a tuned function for that in libc: `memchr`. Using this we can build a very fast and much simpler scanner to split lines up-front. This PR uses that and a carefully crafted fast loop to first build up all the line info we need. Getting this to be as fast as possible required some other subtle changes, for example always creating a line structure that goes from the last `\n` and the end of the file. We then back up the EOF token to avoid surfacing this to users. The nice thing is the EOF token isn't part of any hot loop, and so this removes branches everywhere else at modest complexity. Once we have that, the rest of the lexer just needs to keep track of its current line in order to record column offsets. I've taken some care to try and optimize the lexer's usage of the line structures but there are more opportunities here I suspect. Combined, this gets much but not all of the performance of a huge SIMD scanner for newline-through-to-next-token. For extreme cases (100s of blank lines or empty comment lines between tokens) the holistic scanner is of course still much faster, but those don't seem nearly worth the cost. I was initially worried about the overhead of taking two passes over the source text, but in practice I've not been able to measure any appreciable cost to this with realistic source files. In some cases benchmarks with no newlines get *faster* because we use a much more efficient approach to fetch the source text into cache as a happenstance. And that in turn makes the byte-wise dispatched loop run faster as it stalls less. I'm particularly happy with this approach because it seems very clear how to extend this to support CR+LF, bare CR, and even complex mixtures without any significant speed cost. That wasn't at all true for the other approaches explored. I may try some further PRs to smooth out the last bits of slowness here, but already this is working excellent for me in practice. My 10mloc test case is down to 2.3s to lex. ## Raw benchmark data Using a tool that runs benchmarks before and after and analyzes the results, the following summarizes the CPU-time impact, each of these for lexing 100k tokens: ``` BM_ValidKeywords 2.57ms ± 1% 2.58ms ± 0% ~ (p=0.190 n=5+4) BM_ValidIdentifiers<1, 64, false> 9.24ms ± 4% 9.31ms ± 4% ~ (p=0.421 n=5+5) BM_ValidIdentifiers<1, 1, true> 3.05ms ± 4% 3.11ms ± 4% ~ (p=0.222 n=5+5) BM_ValidIdentifiers<3, 5, true> 10.9ms ± 0% 11.1ms ± 1% +1.76% (p=0.016 n=4+5) BM_ValidIdentifiers<3, 16, true> 11.1ms ± 7% 11.0ms ± 1% ~ (p=0.310 n=5+5) BM_ValidIdentifiers<12, 64, true> 12.2ms ± 1% 12.3ms ± 2% ~ (p=0.111 n=4+5) BM_HorizontalWhitespace/1 11.2ms ± 6% 11.1ms ± 2% ~ (p=0.841 n=5+5) BM_HorizontalWhitespace/4 12.0ms ± 3% 12.0ms ± 2% ~ (p=0.548 n=5+5) BM_HorizontalWhitespace/16 16.2ms ± 6% 15.9ms ± 8% ~ (p=0.690 n=5+5) BM_HorizontalWhitespace/64 27.7ms ± 3% 28.4ms ± 3% ~ (p=0.151 n=5+5) BM_HorizontalWhitespace/128 44.3ms ± 1% 45.6ms ± 6% +3.15% (p=0.032 n=5+5) BM_RandomSource 7.75ms ± 2% 7.72ms ± 1% ~ (p=1.000 n=5+5) BM_BlankLines/1 11.7ms ± 1% 12.1ms ± 1% +3.46% (p=0.008 n=5+5) BM_BlankLines/4 14.0ms ± 2% 15.2ms ± 3% +8.12% (p=0.008 n=5+5) BM_BlankLines/16 23.5ms ± 2% 31.1ms ± 4% +32.26% (p=0.008 n=5+5) BM_BlankLines/64 75.3ms ± 1% 81.2ms ± 3% +7.83% (p=0.008 n=5+5) BM_BlankLines/128 133ms ± 3% 150ms ± 2% +12.74% (p=0.008 n=5+5) BM_CommentLines/1/0/0 13.1ms ± 0% 13.7ms ± 1% +5.11% (p=0.008 n=5+5) BM_CommentLines/4/0/0 16.6ms ± 1% 18.2ms ± 4% +9.56% (p=0.008 n=5+5) BM_CommentLines/128/0/0 169ms ± 4% 182ms ± 1% +7.24% (p=0.008 n=5+5) BM_CommentLines/1/30/0 18.7ms ± 5% 14.1ms ± 0% -24.84% (p=0.008 n=5+5) BM_CommentLines/4/30/0 36.5ms ± 6% 20.6ms ± 3% -43.59% (p=0.008 n=5+5) BM_CommentLines/128/30/0 525ms ± 4% 198ms ± 1% -62.38% (p=0.008 n=5+5) BM_CommentLines/1/70/0 23.4ms ± 6% 14.7ms ± 2% -37.15% (p=0.008 n=5+5) BM_CommentLines/4/70/0 53.3ms ± 7% 22.4ms ± 4% -57.99% (p=0.008 n=5+5) BM_CommentLines/128/70/0 1.05s ± 4% 0.21s ± 2% -80.31% (p=0.008 n=5+5) BM_CommentLines/1/0/2 14.1ms ± 6% 14.3ms ± 1% ~ (p=0.151 n=5+5) BM_CommentLines/4/0/2 19.4ms ± 5% 20.1ms ± 1% ~ (p=0.151 n=5+5) BM_CommentLines/128/0/2 238ms ± 8% 229ms ± 0% ~ (p=0.151 n=5+5) BM_CommentLines/1/30/2 19.2ms ± 7% 14.6ms ± 1% -23.87% (p=0.008 n=5+5) BM_CommentLines/4/30/2 40.3ms ±13% 22.3ms ± 4% -44.63% (p=0.008 n=5+5) BM_CommentLines/128/30/2 568ms ± 7% 254ms ± 3% -55.28% (p=0.008 n=5+5) BM_CommentLines/1/70/2 23.3ms ± 1% 15.0ms ± 3% -35.61% (p=0.016 n=4+5) BM_CommentLines/4/70/2 57.2ms ± 9% 24.1ms ± 2% -57.81% (p=0.008 n=5+5) BM_CommentLines/128/70/2 1.07s ± 0% 0.26s ± 2% -75.51% (p=0.016 n=4+5) BM_CommentLines/1/0/8 15.9ms ± 7% 16.0ms ± 1% ~ (p=0.151 n=5+5) BM_CommentLines/4/0/8 24.2ms ± 6% 27.9ms ± 2% +15.36% (p=0.008 n=5+5) BM_CommentLines/128/0/8 386ms ± 4% 445ms ± 1% +15.28% (p=0.008 n=5+5) BM_CommentLines/1/30/8 20.6ms ± 5% 16.3ms ± 1% -20.95% (p=0.008 n=5+5) BM_CommentLines/4/30/8 45.3ms ± 6% 30.3ms ± 3% -32.98% (p=0.008 n=5+5) BM_CommentLines/128/30/8 699ms ± 3% 477ms ± 3% -31.83% (p=0.008 n=5+5) BM_CommentLines/1/70/8 25.7ms ± 5% 16.8ms ± 2% -34.67% (p=0.008 n=5+5) BM_CommentLines/4/70/8 62.0ms ± 4% 31.6ms ± 2% -49.10% (p=0.008 n=5+5) BM_CommentLines/128/70/8 1.20s ± 2% 0.48s ± 4% -59.60% (p=0.008 n=5+5) ``` The horizontal whitespace benchmark (and all of the non-line-oriented ones) are noisier than they appear here but do show some improvements (surprisingly). My guess is that it has a lot to do with system load, as the advantage is that we're using a vectorized loop to scan the text first and then doing the byte-dispatched loop. So when the cache is a bit slower to populate, the vectorized version starts to be faster. --------- Co-authored-by: Richard Smith <richard@metafoo.co.uk> Co-authored-by: josh11b <josh11b@users.noreply.github.com>
429 lines
14 KiB
C++
429 lines
14 KiB
C++
// Part of the Carbon Language project, under the Apache License v2.0 with LLVM
|
|
// Exceptions. See /LICENSE for license information.
|
|
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
|
|
|
|
#ifndef CARBON_TOOLCHAIN_LEX_TOKENIZED_BUFFER_H_
|
|
#define CARBON_TOOLCHAIN_LEX_TOKENIZED_BUFFER_H_
|
|
|
|
#include <cstdint>
|
|
#include <iterator>
|
|
|
|
#include "common/ostream.h"
|
|
#include "llvm/ADT/APInt.h"
|
|
#include "llvm/ADT/DenseMap.h"
|
|
#include "llvm/ADT/SmallVector.h"
|
|
#include "llvm/ADT/StringRef.h"
|
|
#include "llvm/ADT/iterator.h"
|
|
#include "llvm/ADT/iterator_range.h"
|
|
#include "llvm/Support/raw_ostream.h"
|
|
#include "toolchain/base/index_base.h"
|
|
#include "toolchain/diagnostics/diagnostic_emitter.h"
|
|
#include "toolchain/lex/token_kind.h"
|
|
#include "toolchain/source/source_buffer.h"
|
|
|
|
namespace Carbon::Lex {
|
|
|
|
class TokenizedBuffer;
|
|
|
|
// A lightweight handle to a lexed token in a `TokenizedBuffer`.
|
|
//
|
|
// `Token` objects are designed to be passed by value, not reference or
|
|
// pointer. They are also designed to be small and efficient to store in data
|
|
// structures.
|
|
//
|
|
// `Token` objects from the same `TokenizedBuffer` can be compared with each
|
|
// other, both for being the same token within the buffer, and to establish
|
|
// relative position within the token stream that has been lexed out of the
|
|
// buffer. `Token` objects from different `TokenizedBuffer`s cannot be
|
|
// meaningfully compared.
|
|
//
|
|
// All other APIs to query a `Token` are on the `TokenizedBuffer`.
|
|
struct Token : public ComparableIndexBase {
|
|
using ComparableIndexBase::ComparableIndexBase;
|
|
};
|
|
|
|
// A lightweight handle to a lexed line in a `TokenizedBuffer`.
|
|
//
|
|
// `Line` objects are designed to be passed by value, not reference or
|
|
// pointer. They are also designed to be small and efficient to store in data
|
|
// structures.
|
|
//
|
|
// Each `Line` object refers to a specific line in the source code that was
|
|
// lexed. They can be compared directly to establish that they refer to the
|
|
// same line or the relative position of different lines within the source.
|
|
//
|
|
// All other APIs to query a `Line` are on the `TokenizedBuffer`.
|
|
struct Line : public ComparableIndexBase {
|
|
using ComparableIndexBase::ComparableIndexBase;
|
|
|
|
static const Line Invalid;
|
|
};
|
|
|
|
constexpr Line Line::Invalid(Line::InvalidIndex);
|
|
|
|
// A lightweight handle to a lexed identifier in a `TokenizedBuffer`.
|
|
//
|
|
// `Identifier` objects are designed to be passed by value, not reference or
|
|
// pointer. They are also designed to be small and efficient to store in data
|
|
// structures.
|
|
//
|
|
// Each identifier lexed is canonicalized to a single entry in the identifier
|
|
// table. `Identifier` objects will compare equal if they refer to the same
|
|
// identifier spelling. Where the identifier was written is not preserved.
|
|
//
|
|
// All other APIs to query a `Identifier` are on the `TokenizedBuffer`.
|
|
struct Identifier : public IndexBase {
|
|
using IndexBase::IndexBase;
|
|
|
|
static const Identifier Invalid;
|
|
};
|
|
|
|
constexpr Identifier Identifier::Invalid = Identifier(Identifier::InvalidIndex);
|
|
|
|
// Random-access iterator over tokens within the buffer.
|
|
class TokenIterator
|
|
: public llvm::iterator_facade_base<
|
|
TokenIterator, std::random_access_iterator_tag, const Token, int>,
|
|
public Printable<TokenIterator> {
|
|
public:
|
|
TokenIterator() = delete;
|
|
|
|
explicit TokenIterator(Token token) : token_(token) {}
|
|
|
|
auto operator==(const TokenIterator& rhs) const -> bool {
|
|
return token_ == rhs.token_;
|
|
}
|
|
auto operator<(const TokenIterator& rhs) const -> bool {
|
|
return token_ < rhs.token_;
|
|
}
|
|
|
|
auto operator*() const -> const Token& { return token_; }
|
|
|
|
using iterator_facade_base::operator-;
|
|
auto operator-(const TokenIterator& rhs) const -> int {
|
|
return token_.index - rhs.token_.index;
|
|
}
|
|
|
|
auto operator+=(int n) -> TokenIterator& {
|
|
token_.index += n;
|
|
return *this;
|
|
}
|
|
auto operator-=(int n) -> TokenIterator& {
|
|
token_.index -= n;
|
|
return *this;
|
|
}
|
|
|
|
// Prints the raw token index.
|
|
auto Print(llvm::raw_ostream& output) const -> void;
|
|
|
|
private:
|
|
friend class TokenizedBuffer;
|
|
|
|
Token token_;
|
|
};
|
|
|
|
// The value of a real literal.
|
|
//
|
|
// This is either a dyadic fraction (mantissa * 2^exponent) or a decadic
|
|
// fraction (mantissa * 10^exponent).
|
|
//
|
|
// `RealLiteralValue` carries a reference back to `TokenizedBuffer` which can be
|
|
// invalidated if the buffer is edited or destroyed.
|
|
class RealLiteralValue : public Printable<RealLiteralValue> {
|
|
public:
|
|
auto Print(llvm::raw_ostream& output_stream) const -> void {
|
|
mantissa.print(output_stream, /*isSigned=*/false);
|
|
output_stream << "*" << (is_decimal ? "10" : "2") << "^" << exponent;
|
|
}
|
|
|
|
// The mantissa, represented as an unsigned integer.
|
|
const llvm::APInt& mantissa;
|
|
|
|
// The exponent, represented as a signed integer.
|
|
const llvm::APInt& exponent;
|
|
|
|
// If false, the value is mantissa * 2^exponent.
|
|
// If true, the value is mantissa * 10^exponent.
|
|
bool is_decimal;
|
|
};
|
|
|
|
// A diagnostic location translator that maps token locations into source
|
|
// buffer locations.
|
|
class TokenLocationTranslator : public DiagnosticLocationTranslator<Token> {
|
|
public:
|
|
explicit TokenLocationTranslator(const TokenizedBuffer* buffer)
|
|
: buffer_(buffer) {}
|
|
|
|
// Map the given token into a diagnostic location.
|
|
auto GetLocation(Token token) -> DiagnosticLocation override;
|
|
|
|
private:
|
|
const TokenizedBuffer* buffer_;
|
|
};
|
|
|
|
// A buffer of tokenized Carbon source code.
|
|
//
|
|
// This is constructed by lexing the source code text into a series of tokens.
|
|
// The buffer provides lightweight handles to tokens and other lexed entities,
|
|
// as well as iterations to walk the sequence of tokens found in the buffer.
|
|
//
|
|
// Lexing errors result in a potentially incomplete sequence of tokens and
|
|
// `HasError` returning true.
|
|
class TokenizedBuffer : public Printable<TokenizedBuffer> {
|
|
public:
|
|
// Lexes a buffer of source code into a tokenized buffer.
|
|
//
|
|
// The provided source buffer must outlive any returned `TokenizedBuffer`
|
|
// which will refer into the source.
|
|
static auto Lex(SourceBuffer& source, DiagnosticConsumer& consumer)
|
|
-> TokenizedBuffer;
|
|
|
|
[[nodiscard]] auto GetKind(Token token) const -> TokenKind;
|
|
[[nodiscard]] auto GetLine(Token token) const -> Line;
|
|
|
|
// Returns the 1-based line number.
|
|
[[nodiscard]] auto GetLineNumber(Token token) const -> int;
|
|
|
|
// Returns the 1-based column number.
|
|
[[nodiscard]] auto GetColumnNumber(Token token) const -> int;
|
|
|
|
// Returns the source text lexed into this token.
|
|
[[nodiscard]] auto GetTokenText(Token token) const -> llvm::StringRef;
|
|
|
|
// Returns the identifier associated with this token. The token kind must be
|
|
// an `Identifier`.
|
|
[[nodiscard]] auto GetIdentifier(Token token) const -> Identifier;
|
|
|
|
// Returns the value of an `IntegerLiteral()` token.
|
|
[[nodiscard]] auto GetIntegerLiteral(Token token) const -> const llvm::APInt&;
|
|
|
|
// Returns the value of an `RealLiteral()` token.
|
|
[[nodiscard]] auto GetRealLiteral(Token token) const -> RealLiteralValue;
|
|
|
|
// Returns the value of a `StringLiteral()` token.
|
|
[[nodiscard]] auto GetStringLiteral(Token token) const -> llvm::StringRef;
|
|
|
|
// Returns the size specified in a `*TypeLiteral()` token.
|
|
[[nodiscard]] auto GetTypeLiteralSize(Token token) const
|
|
-> const llvm::APInt&;
|
|
|
|
// Returns the closing token matched with the given opening token.
|
|
//
|
|
// The given token must be an opening token kind.
|
|
[[nodiscard]] auto GetMatchedClosingToken(Token opening_token) const -> Token;
|
|
|
|
// Returns the opening token matched with the given closing token.
|
|
//
|
|
// The given token must be a closing token kind.
|
|
[[nodiscard]] auto GetMatchedOpeningToken(Token closing_token) const -> Token;
|
|
|
|
// Returns whether the given token has leading whitespace.
|
|
[[nodiscard]] auto HasLeadingWhitespace(Token token) const -> bool;
|
|
// Returns whether the given token has trailing whitespace.
|
|
[[nodiscard]] auto HasTrailingWhitespace(Token token) const -> bool;
|
|
|
|
// Returns whether the token was created as part of an error recovery effort.
|
|
//
|
|
// For example, a closing paren inserted to match an unmatched paren.
|
|
[[nodiscard]] auto IsRecoveryToken(Token token) const -> bool;
|
|
|
|
// Returns the 1-based line number.
|
|
[[nodiscard]] auto GetLineNumber(Line line) const -> int;
|
|
|
|
// Returns the 1-based indentation column number.
|
|
[[nodiscard]] auto GetIndentColumnNumber(Line line) const -> int;
|
|
|
|
// Returns the next line handle.
|
|
[[nodiscard]] auto GetNextLine(Line line) const -> Line;
|
|
|
|
// Returns the previous line handle.
|
|
[[nodiscard]] auto GetPrevLine(Line line) const -> Line;
|
|
|
|
// Returns the text for an identifier.
|
|
[[nodiscard]] auto GetIdentifierText(Identifier id) const -> llvm::StringRef;
|
|
|
|
// Prints a description of the tokenized stream to the provided `raw_ostream`.
|
|
//
|
|
// It prints one line of information for each token in the buffer, including
|
|
// the kind of token, where it occurs within the source file, indentation for
|
|
// the associated line, the spelling of the token in source, and any
|
|
// additional information tracked such as which unique identifier it is or any
|
|
// matched grouping token.
|
|
//
|
|
// Each line is formatted as a YAML record:
|
|
//
|
|
// clang-format off
|
|
// ```
|
|
// token: { index: 0, kind: 'Semi', line: 1, column: 1, indent: 1, spelling: ';' }
|
|
// ```
|
|
// clang-format on
|
|
//
|
|
// This can be parsed as YAML using tools like `python-yq` combined with `jq`
|
|
// on the command line. The format is also reasonably amenable to other
|
|
// line-oriented shell tools from `grep` to `awk`.
|
|
auto Print(llvm::raw_ostream& output_stream) const -> void;
|
|
|
|
// Prints a description of a single token. See `Print` for details on the
|
|
// format.
|
|
auto PrintToken(llvm::raw_ostream& output_stream, Token token) const -> void;
|
|
|
|
// Returns true if the buffer has errors that were detected at lexing time.
|
|
[[nodiscard]] auto has_errors() const -> bool { return has_errors_; }
|
|
|
|
[[nodiscard]] auto tokens() const -> llvm::iterator_range<TokenIterator> {
|
|
return llvm::make_range(TokenIterator(Token(0)),
|
|
TokenIterator(Token(token_infos_.size())));
|
|
}
|
|
|
|
[[nodiscard]] auto size() const -> int { return token_infos_.size(); }
|
|
|
|
[[nodiscard]] auto expected_parse_tree_size() const -> int {
|
|
return expected_parse_tree_size_;
|
|
}
|
|
|
|
auto filename() const -> llvm::StringRef { return source_->filename(); }
|
|
|
|
private:
|
|
// Implementation detail struct implementing the actual lexer logic.
|
|
class Lexer;
|
|
friend Lexer;
|
|
|
|
friend class TokenLocationTranslator;
|
|
|
|
// A diagnostic location translator that maps token locations into source
|
|
// buffer locations.
|
|
class SourceBufferLocationTranslator
|
|
: public DiagnosticLocationTranslator<const char*> {
|
|
public:
|
|
explicit SourceBufferLocationTranslator(const TokenizedBuffer* buffer)
|
|
: buffer_(buffer) {}
|
|
|
|
// Map the given position within the source buffer into a diagnostic
|
|
// location.
|
|
auto GetLocation(const char* loc) -> DiagnosticLocation override;
|
|
|
|
private:
|
|
const TokenizedBuffer* buffer_;
|
|
};
|
|
|
|
// Specifies minimum widths to use when printing a token's fields via
|
|
// `printToken`.
|
|
struct PrintWidths {
|
|
// Widens `this` to the maximum of `this` and `new_width` for each
|
|
// dimension.
|
|
auto Widen(const PrintWidths& widths) -> void;
|
|
|
|
int index;
|
|
int kind;
|
|
int line;
|
|
int column;
|
|
int indent;
|
|
};
|
|
|
|
struct TokenInfo {
|
|
TokenKind kind;
|
|
|
|
// Whether the token has trailing whitespace.
|
|
bool has_trailing_space = false;
|
|
|
|
// Whether the token was injected artificially during error recovery.
|
|
bool is_recovery = false;
|
|
|
|
// Line on which the Token starts.
|
|
Line token_line;
|
|
|
|
// Zero-based byte offset of the token within its line.
|
|
int32_t column;
|
|
|
|
// We may have up to 32 bits of payload, based on the kind of token.
|
|
union {
|
|
static_assert(
|
|
sizeof(Token) <= sizeof(int32_t),
|
|
"Unable to pack token and identifier index into the same space!");
|
|
|
|
Identifier id = Identifier::Invalid;
|
|
int32_t literal_index;
|
|
Token closing_token;
|
|
Token opening_token;
|
|
int32_t error_length;
|
|
};
|
|
};
|
|
|
|
struct LineInfo {
|
|
// The length will always be assigned later. Indent may be assigned if
|
|
// non-zero.
|
|
explicit LineInfo(int64_t start)
|
|
: start(start),
|
|
length(static_cast<int32_t>(llvm::StringRef::npos)),
|
|
indent(0) {}
|
|
|
|
explicit LineInfo(int64_t start, int32_t length)
|
|
: start(start), length(length), indent(0) {}
|
|
|
|
// Zero-based byte offset of the start of the line within the source buffer
|
|
// provided.
|
|
int64_t start;
|
|
|
|
// The byte length of the line. Does not include the newline character (or a
|
|
// nul-terminator or EOF).
|
|
int32_t length;
|
|
|
|
// The byte offset from the start of the line of the first non-whitespace
|
|
// character.
|
|
int32_t indent;
|
|
};
|
|
|
|
struct IdentifierInfo {
|
|
llvm::StringRef text;
|
|
};
|
|
|
|
// The constructor is merely responsible for trivial initialization of
|
|
// members. A working object of this type is built with the `lex` function
|
|
// above so that its return can indicate if an error was encountered while
|
|
// lexing.
|
|
explicit TokenizedBuffer(SourceBuffer& source) : source_(&source) {}
|
|
|
|
auto GetLineInfo(Line line) -> LineInfo&;
|
|
[[nodiscard]] auto GetLineInfo(Line line) const -> const LineInfo&;
|
|
auto AddLine(LineInfo info) -> Line;
|
|
auto GetTokenInfo(Token token) -> TokenInfo&;
|
|
[[nodiscard]] auto GetTokenInfo(Token token) const -> const TokenInfo&;
|
|
auto AddToken(TokenInfo info) -> Token;
|
|
[[nodiscard]] auto GetTokenPrintWidths(Token token) const -> PrintWidths;
|
|
auto PrintToken(llvm::raw_ostream& output_stream, Token token,
|
|
PrintWidths widths) const -> void;
|
|
|
|
SourceBuffer* source_;
|
|
|
|
llvm::SmallVector<TokenInfo> token_infos_;
|
|
|
|
llvm::SmallVector<LineInfo> line_infos_;
|
|
|
|
llvm::SmallVector<IdentifierInfo> identifier_infos_;
|
|
|
|
// Storage for integers that form part of the value of a numeric or type
|
|
// literal.
|
|
llvm::SmallVector<llvm::APInt> literal_int_storage_;
|
|
|
|
llvm::SmallVector<std::string> literal_string_storage_;
|
|
|
|
llvm::DenseMap<llvm::StringRef, Identifier> identifier_map_;
|
|
|
|
// The number of parse tree nodes that we expect to be created for the tokens
|
|
// in this buffer.
|
|
int expected_parse_tree_size_ = 0;
|
|
|
|
bool has_errors_ = false;
|
|
};
|
|
|
|
// A diagnostic emitter that uses positions within a source buffer's text as
|
|
// its source of location information.
|
|
using LexerDiagnosticEmitter = DiagnosticEmitter<const char*>;
|
|
|
|
// A diagnostic emitter that uses tokens as its source of location information.
|
|
using TokenDiagnosticEmitter = DiagnosticEmitter<Token>;
|
|
|
|
} // namespace Carbon::Lex
|
|
|
|
#endif // CARBON_TOOLCHAIN_LEX_TOKENIZED_BUFFER_H_
|