Files
carbon-lang/lexer/tokenized_buffer.h
T
Chandler CarruthandJon Meow 3995fc2d6c Merge lexer from the toolchain repository. (#213)
The only change here is to update the fuzzer build extension path.

The main original commit message:

> Add an initial lexer. (#17)
>
> The specific logic here hasn't been updated to track the latest
> discussed changes, much less implement many aspects of things like
> Unicode support.
>
> However, this should lay out a reasonable framework and set of APIs.
> It gives an idea of the overall lexer architecture being proposed. The
> actual lexing algorithm is a relatively boring and naive hand written
> loop. It may make sense to replace this with something generated or
> other more advanced approach in the future, getting the implementation
> right was not the primary goal here. Instead, the focus was entirely
> on the architecture, encapsulation, APIs, and the testing
> infrastructure.
>
> The architecture of the lexer differs from "classical" high
> performance lexers in compilers. A high level summary:
>
> -   It is eager rather than lazy, lexing an entire file.
> -   Tokens intrinsically know their source location.
> -   Grouping lexical symbols are tracked within the lexer.
> -   Indentation is tracked within the lexer.
>
> Tracking of grouping and indentation is intended to simplify the
> strategies used for recovery of mismatched grouping tokens, and
> eventually use indentation.
>
> Folding source location into the token itself simplifies the data
> structures significantly, and doesn't lose any fidelity due to the
> absence of a preprocessor with token pasting.
>
> The fact that this is an eager lexer instead of a lazy lexer is
> designed to simplify the implementation and testing of the lexer (and
> subsequent components). There is no reason to expect Carbon to lex so
> many tokens that there are significant locality advantages of lazy
> lexing. Moreover, if we want comparable performance benefits, I think
> pipelining is a much more promising architecture than laziness. For
> now, the simplicity is a huge win.
>
> Being eager also makes it easy for us to use extremely dense memory
> encodings for the information about lexed tokens. Everything is
> created in a dense array, and small indices are used to identify each
> token within the array.
>
> There is a fuzzer included here that we have run extensively over the
> code, but currently toolchain bugs and Bazel limitations prevent it
> from easily building. I'm hoping myself or someone else can push on
> this soon and enable the fuzzer to at least build if not run fuzz
> tests automatically. We have a significant fuzzing corpus that I'll
> add in a subsequent commit as well.

This also includes the fuzzer whose commit message was:

> Add fuzz testing infrastructure and the lexer's fuzzer. (#21)
>
> This adds a fairly simple `cc_fuzz_test` macro that is specialized for
> working with LLVM's LibFuzzer. In addition to building the fuzzer
> binary with the toolchain's `fuzzer` feature, it also sets up the test
> execution to pass the corpus as file arguments which is a simple
> mechanism to enable regression testing against the fuzz corpus.
>
> I've included an initial fuzzer corpus as well. To run the fuzzer in
> an open ended fashion, and build up a larger corpus:
> ```shell
> mkdir /tmp/new_corpus
> cp lexer/fuzzer_corpus/* /tmp/new_corpus
> ./bazel-bin/lexer/tokenized_buffer_fuzzer /tmp/new_corpus
> ```
>
> You can parallelize the fuzzer by adding `-jobs=N` for N threads. For
> more details about running fuzzers, see the documentation:
> http://llvm.org/docs/LibFuzzer.html
>
> To minimize and merge any interesting new inputs:
> ```shell
> ./bazel-bin/lexer/tokenized_buffer_fuzzer -merge=1 \
>     lexer/fuzzer_corpus /tmp/new_corpus
> ```

Co-authored-by: Jon Meow <46229924+jonmeow@users.noreply.github.com>
2020-12-08 01:49:25 -08:00

344 lines
11 KiB
C++

// Part of the Carbon Language project, under the Apache License v2.0 with LLVM
// Exceptions. See /LICENSE for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#ifndef LEXER_TOKENIZED_BUFFER_H_
#define LEXER_TOKENIZED_BUFFER_H_
#include <stdint.h>
#include <iterator>
#include "diagnostics/diagnostic_emitter.h"
#include "lexer/token_kind.h"
#include "llvm/ADT/APInt.h"
#include "llvm/ADT/DenseMap.h"
#include "llvm/ADT/Optional.h"
#include "llvm/ADT/SmallVector.h"
#include "llvm/ADT/StringRef.h"
#include "llvm/ADT/iterator.h"
#include "llvm/ADT/iterator_range.h"
#include "source/source_buffer.h"
namespace Carbon {
// A buffer of tokenized Carbon source code.
//
// This is constructed by lexing the source code text into a series of tokens.
// The buffer provides lightweight handles to tokens and other lexed entities,
// as well as iterations to walk the sequence of tokens found in the buffer.
//
// Lexing errors result in a potentially incomplete sequence of tokens and
// `HasError` returning true.
class TokenizedBuffer {
public:
// A lightweight handle to a lexed token in a `TokenizedBuffer`.
//
// `Token` objects are designed to be passed by value, not reference or
// pointer. They are also designed to be small and efficient to store in data
// structures.
//
// `Token` objects from the same `TokenizedBuffer` can be compared with each
// other, both for being the same token within the buffer, and to establish
// relative position within the token stream that has been lexed out of the
// buffer.
//
// All other APIs to query a `Token` are on the `TokenizedBuffer`.
class Token {
public:
Token() = default;
bool operator==(const Token& rhs) const { return index == rhs.index; }
bool operator!=(const Token& rhs) const { return index != rhs.index; }
bool operator<(const Token& rhs) const { return index < rhs.index; }
bool operator<=(const Token& rhs) const { return index <= rhs.index; }
bool operator>(const Token& rhs) const { return index > rhs.index; }
bool operator>=(const Token& rhs) const { return index >= rhs.index; }
private:
friend class TokenizedBuffer;
explicit Token(int index) : index(index) {}
int32_t index;
};
// A lightweight handle to a lexed line in a `TokenizedBuffer`.
//
// `Line` objects are designed to be passed by value, not reference or
// pointer. They are also designed to be small and efficient to store in data
// structures.
//
// Each `Line` object refers to a specific line in the source code that was
// lexed. They can be compared directly to establish that they refer to the
// same line or the relative position of different lines within the source.
//
// All other APIs to query a `Line` are on the `TokenizedBuffer`.
class Line {
public:
Line() = default;
bool operator==(const Line& rhs) const { return index == rhs.index; }
bool operator!=(const Line& rhs) const { return index != rhs.index; }
bool operator<(const Line& rhs) const { return index < rhs.index; }
bool operator<=(const Line& rhs) const { return index <= rhs.index; }
bool operator>(const Line& rhs) const { return index > rhs.index; }
bool operator>=(const Line& rhs) const { return index >= rhs.index; }
private:
friend class TokenizedBuffer;
explicit Line(int index) : index(index) {}
int32_t index;
};
// A lightweight handle to a lexed identifier in a `TokenizedBuffer`.
//
// `Identifier` objects are designed to be passed by value, not reference or
// pointer. They are also designed to be small and efficient to store in data
// structures.
//
// Each identifier lexed is canonicalized to a single entry in the identifier
// table. `Identifier` objects will compare equal if they refer to the same
// identifier spelling. Where the identifier was written is not preserved.
//
// All other APIs to query a `Identifier` are on the `TokenizedBuffer`.
class Identifier {
public:
Identifier() = default;
// Most normal APIs are provided by the `TokenizedBuffer`, we just support
// basic comparison operations.
bool operator==(const Identifier& rhs) const { return index == rhs.index; }
bool operator!=(const Identifier& rhs) const { return index != rhs.index; }
private:
friend class TokenizedBuffer;
explicit Identifier(int index) : index(index) {}
int32_t index;
};
// Random-access iterator over tokens within the buffer.
class TokenIterator
: public llvm::iterator_facade_base<
TokenIterator, std::random_access_iterator_tag, Token, int> {
public:
TokenIterator() = default;
explicit TokenIterator(Token token) : token(token) {}
bool operator==(const TokenIterator& rhs) const {
return token == rhs.token;
}
bool operator<(const TokenIterator& rhs) const { return token < rhs.token; }
const Token& operator*() const { return token; }
Token& operator*() { return token; }
int operator-(const TokenIterator& rhs) const {
return token.index - rhs.token.index;
}
TokenIterator& operator+=(int n) {
token.index += n;
return *this;
}
TokenIterator& operator-=(int n) {
token.index -= n;
return *this;
}
private:
friend class TokenizedBuffer;
Token token;
};
// Lexes a buffer of source code into a tokenized buffer.
//
// The provided source buffer must outlive any returned `TokenizedBuffer`
// which will refer into the source.
//
// FIXME: Need to pass in some diagnostic machinery to report the details of
// the error! Right now it prints to stderr.
static TokenizedBuffer Lex(SourceBuffer& source, DiagnosticEmitter& emitter);
// Returns true if the buffer has errors that are detectable at lexing time.
auto HasErrors() const -> bool { return has_errors; }
llvm::iterator_range<TokenIterator> Tokens() const {
return llvm::make_range(TokenIterator(Token(0)),
TokenIterator(Token(token_infos.size())));
}
auto Size() const -> int { return token_infos.size(); }
auto GetKind(Token token) const -> TokenKind;
auto GetLine(Token token) const -> Line;
// Returns the 1-based line number.
auto GetLineNumber(Token token) const -> int;
// Returns the 1-based column number.
auto GetColumnNumber(Token token) const -> int;
// Returns the source text lexed into this token.
auto GetTokenText(Token token) const -> llvm::StringRef;
// Returns the identifier associated with this token. The token kind must be
// an `Identifier`.
auto GetIdentifier(Token token) const -> Identifier;
// Returns the value of an `IntegerLiteral()` token.
auto GetIntegerLiteral(Token token) const -> llvm::APInt;
// Returns the closing token matched with the given opening token.
//
// The given token must be an opening token kind.
auto GetMatchedClosingToken(Token opening_token) const -> Token;
// Returns the opening token matched with the given closing token.
//
// The given token must be a closing token kind.
auto GetMatchedOpeningToken(Token closing_token) const -> Token;
// Returns whether the token was created as part of an error recovery effort.
//
// For example, a closing paren inserted to match an unmatched paren.
auto IsRecoveryToken(Token token) const -> bool;
// Returns the 1-based line number.
auto GetLineNumber(Line line) const -> int;
// Returns the 1-based indentation column number.
auto GetIndentColumnNumber(Line line) const -> int;
// Returns the text for an identifier.
auto GetIdentifierText(Identifier id) const -> llvm::StringRef;
// Prints a description of the tokenized stream to the provided `raw_ostream`.
//
// It prints one line of information for each token in the buffer, including
// the kind of token, where it occurs within the source file, indentation for
// the associated line, the spelling of the token in source, and any
// additional information tracked such as which unique identifier it is or any
// matched grouping token.
//
// Each line is formatted as a YAML record:
//
// clang-format off
// ```
// token: { index: 0, kind: 'Semi', line: 1, column: 1, indent: 1, spelling: ';' }
// ```
// clang-format on
//
// This can be parsed as YAML using tools like `python-yq` combined with `jq`
// on the command line. The format is also reasonably amenable to other
// line-oriented shell tools from `grep` to `awk`.
auto Print(llvm::raw_ostream& output_stream) const -> void;
// Prints a description of a single token. See `print` for details on the
// format.
auto PrintToken(llvm::raw_ostream& output_stream, Token token) const -> void;
private:
// Implementation detail struct implementing the actual lexer logic.
class Lexer;
friend Lexer;
// Specifies minimum widths to use when printing a token's fields via
// `printToken`.
struct PrintWidths {
int index;
int kind;
int column;
int line;
int indent;
// Widens `this` to the maximum of `this` and `new_width` for each
// dimension.
void Widen(const PrintWidths& new_width);
};
struct TokenInfo {
TokenKind kind;
// Whether the token was injected artificially during error recovery.
bool is_recovery = false;
// Line on which the Token starts.
Line token_line;
// Zero-based byte offset of the token within its line.
int32_t column;
// We may have up to 32 bits of payload, based on the kind of token.
union {
static_assert(
sizeof(Token) <= sizeof(int32_t),
"Unable to pack token and identifier index into the same space!");
Identifier id;
int32_t literal_index;
Token closing_token;
Token opening_token;
int32_t error_length;
};
};
struct LineInfo {
// Zero-based byte offset of the start of the line within the source buffer
// provided.
int64_t start;
// The byte length of the line. Does not include the newline character (or a
// null terminator or EOF).
int32_t length;
// The byte offset from the start of the line of the first non-whitespace
// character.
int32_t indent;
};
struct IdentifierInfo {
llvm::StringRef text;
};
// The constructor is merely responsible for trivial initialization of
// members. A working object of this type is built with the `lex` function
// above so that its return can indicate if an error was encountered while
// lexing.
explicit TokenizedBuffer(SourceBuffer& source) : source(&source) {}
auto GetLineInfo(Line line) -> LineInfo&;
auto GetLineInfo(Line line) const -> const LineInfo&;
auto AddLine(LineInfo info) -> Line;
auto GetTokenInfo(Token token) -> TokenInfo&;
auto GetTokenInfo(Token token) const -> const TokenInfo&;
auto AddToken(TokenInfo info) -> Token;
auto GetTokenPrintWidths(Token token) const -> PrintWidths;
auto PrintToken(llvm::raw_ostream& output_stream, Token token,
PrintWidths widths) const -> void;
SourceBuffer* source;
llvm::SmallVector<TokenInfo, 16> token_infos;
llvm::SmallVector<LineInfo, 16> line_infos;
llvm::SmallVector<IdentifierInfo, 16> identifier_infos;
llvm::SmallVector<llvm::APInt, 16> int_literals;
llvm::DenseMap<llvm::StringRef, Identifier> identifier_map;
bool has_errors = false;
};
} // namespace Carbon
#endif // LEXER_TOKENIZED_BUFFER_H_