mirror of
https://github.com/carbon-language/carbon-lang.git
synced 2026-11-15 22:38:10 +00:00
Chandler CarruthandJon Meow
3995fc2d6c
The only change here is to update the fuzzer build extension path. The main original commit message: > Add an initial lexer. (#17) > > The specific logic here hasn't been updated to track the latest > discussed changes, much less implement many aspects of things like > Unicode support. > > However, this should lay out a reasonable framework and set of APIs. > It gives an idea of the overall lexer architecture being proposed. The > actual lexing algorithm is a relatively boring and naive hand written > loop. It may make sense to replace this with something generated or > other more advanced approach in the future, getting the implementation > right was not the primary goal here. Instead, the focus was entirely > on the architecture, encapsulation, APIs, and the testing > infrastructure. > > The architecture of the lexer differs from "classical" high > performance lexers in compilers. A high level summary: > > - It is eager rather than lazy, lexing an entire file. > - Tokens intrinsically know their source location. > - Grouping lexical symbols are tracked within the lexer. > - Indentation is tracked within the lexer. > > Tracking of grouping and indentation is intended to simplify the > strategies used for recovery of mismatched grouping tokens, and > eventually use indentation. > > Folding source location into the token itself simplifies the data > structures significantly, and doesn't lose any fidelity due to the > absence of a preprocessor with token pasting. > > The fact that this is an eager lexer instead of a lazy lexer is > designed to simplify the implementation and testing of the lexer (and > subsequent components). There is no reason to expect Carbon to lex so > many tokens that there are significant locality advantages of lazy > lexing. Moreover, if we want comparable performance benefits, I think > pipelining is a much more promising architecture than laziness. For > now, the simplicity is a huge win. > > Being eager also makes it easy for us to use extremely dense memory > encodings for the information about lexed tokens. Everything is > created in a dense array, and small indices are used to identify each > token within the array. > > There is a fuzzer included here that we have run extensively over the > code, but currently toolchain bugs and Bazel limitations prevent it > from easily building. I'm hoping myself or someone else can push on > this soon and enable the fuzzer to at least build if not run fuzz > tests automatically. We have a significant fuzzing corpus that I'll > add in a subsequent commit as well. This also includes the fuzzer whose commit message was: > Add fuzz testing infrastructure and the lexer's fuzzer. (#21) > > This adds a fairly simple `cc_fuzz_test` macro that is specialized for > working with LLVM's LibFuzzer. In addition to building the fuzzer > binary with the toolchain's `fuzzer` feature, it also sets up the test > execution to pass the corpus as file arguments which is a simple > mechanism to enable regression testing against the fuzz corpus. > > I've included an initial fuzzer corpus as well. To run the fuzzer in > an open ended fashion, and build up a larger corpus: > ```shell > mkdir /tmp/new_corpus > cp lexer/fuzzer_corpus/* /tmp/new_corpus > ./bazel-bin/lexer/tokenized_buffer_fuzzer /tmp/new_corpus > ``` > > You can parallelize the fuzzer by adding `-jobs=N` for N threads. For > more details about running fuzzers, see the documentation: > http://llvm.org/docs/LibFuzzer.html > > To minimize and merge any interesting new inputs: > ```shell > ./bazel-bin/lexer/tokenized_buffer_fuzzer -merge=1 \ > lexer/fuzzer_corpus /tmp/new_corpus > ``` Co-authored-by: Jon Meow <46229924+jonmeow@users.noreply.github.com>
94 lines
4.8 KiB
C++
94 lines
4.8 KiB
C++
// Part of the Carbon Language project, under the Apache License v2.0 with LLVM
|
|
// Exceptions. See /LICENSE for license information.
|
|
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
|
|
|
|
#include "lexer/token_kind.h"
|
|
|
|
#include <string.h>
|
|
|
|
#include "gmock/gmock.h"
|
|
#include "gtest/gtest.h"
|
|
#include "llvm/ADT/StringRef.h"
|
|
|
|
namespace Carbon {
|
|
namespace {
|
|
|
|
using ::testing::MatchesRegex;
|
|
|
|
// We restrict symbols to punctuation characters that are expected to be widely
|
|
// available on modern keyboards used for programming.
|
|
constexpr llvm::StringLiteral SymbolRegex = "[][{}!@#%^&*()/?\\|;:.,<>=+~-]+";
|
|
|
|
// We restrict keywords to be lowercase ASCII letters and underscores.
|
|
constexpr llvm::StringLiteral KeywordRegex = "[a-z_]+";
|
|
|
|
#define CARBON_TOKEN(TokenName) \
|
|
TEST(TokenKindTest, TokenName) { \
|
|
EXPECT_EQ(#TokenName, TokenKind::TokenName().Name()); \
|
|
EXPECT_FALSE(TokenKind::TokenName().IsSymbol()); \
|
|
EXPECT_FALSE(TokenKind::TokenName().IsKeyword()); \
|
|
EXPECT_EQ("", TokenKind::TokenName().GetFixedSpelling()); \
|
|
}
|
|
#define CARBON_SYMBOL_TOKEN(TokenName, Spelling) \
|
|
TEST(TokenKindTest, TokenName) { \
|
|
EXPECT_EQ(#TokenName, TokenKind::TokenName().Name()); \
|
|
EXPECT_TRUE(TokenKind::TokenName().IsSymbol()); \
|
|
EXPECT_FALSE(TokenKind::TokenName().IsGroupingSymbol()); \
|
|
EXPECT_FALSE(TokenKind::TokenName().IsOpeningSymbol()); \
|
|
EXPECT_FALSE(TokenKind::TokenName().IsClosingSymbol()); \
|
|
EXPECT_FALSE(TokenKind::TokenName().IsKeyword()); \
|
|
EXPECT_EQ(Spelling, TokenKind::TokenName().GetFixedSpelling()); \
|
|
EXPECT_THAT(Spelling, MatchesRegex(SymbolRegex.str())); \
|
|
}
|
|
#define CARBON_OPENING_GROUP_SYMBOL_TOKEN(TokenName, Spelling, ClosingName) \
|
|
TEST(TokenKindTest, TokenName) { \
|
|
EXPECT_EQ(#TokenName, TokenKind::TokenName().Name()); \
|
|
EXPECT_TRUE(TokenKind::TokenName().IsSymbol()); \
|
|
EXPECT_TRUE(TokenKind::TokenName().IsGroupingSymbol()); \
|
|
EXPECT_TRUE(TokenKind::TokenName().IsOpeningSymbol()); \
|
|
EXPECT_EQ(TokenKind::ClosingName(), \
|
|
TokenKind::TokenName().GetClosingSymbol()); \
|
|
EXPECT_FALSE(TokenKind::TokenName().IsClosingSymbol()); \
|
|
EXPECT_FALSE(TokenKind::TokenName().IsKeyword()); \
|
|
EXPECT_EQ(Spelling, TokenKind::TokenName().GetFixedSpelling()); \
|
|
EXPECT_THAT(Spelling, MatchesRegex(SymbolRegex.str())); \
|
|
}
|
|
#define CARBON_CLOSING_GROUP_SYMBOL_TOKEN(TokenName, Spelling, OpeningName) \
|
|
TEST(TokenKindTest, TokenName) { \
|
|
EXPECT_EQ(#TokenName, TokenKind::TokenName().Name()); \
|
|
EXPECT_TRUE(TokenKind::TokenName().IsSymbol()); \
|
|
EXPECT_TRUE(TokenKind::TokenName().IsGroupingSymbol()); \
|
|
EXPECT_FALSE(TokenKind::TokenName().IsOpeningSymbol()); \
|
|
EXPECT_TRUE(TokenKind::TokenName().IsClosingSymbol()); \
|
|
EXPECT_EQ(TokenKind::OpeningName(), \
|
|
TokenKind::TokenName().GetOpeningSymbol()); \
|
|
EXPECT_FALSE(TokenKind::TokenName().IsKeyword()); \
|
|
EXPECT_EQ(Spelling, TokenKind::TokenName().GetFixedSpelling()); \
|
|
EXPECT_THAT(Spelling, MatchesRegex(SymbolRegex.str())); \
|
|
}
|
|
#define CARBON_KEYWORD_TOKEN(TokenName, Spelling) \
|
|
TEST(TokenKindTest, TokenName) { \
|
|
EXPECT_EQ(#TokenName, TokenKind::TokenName().Name()); \
|
|
EXPECT_FALSE(TokenKind::TokenName().IsSymbol()); \
|
|
EXPECT_TRUE(TokenKind::TokenName().IsKeyword()); \
|
|
EXPECT_EQ(Spelling, TokenKind::TokenName().GetFixedSpelling()); \
|
|
EXPECT_THAT(Spelling, MatchesRegex(KeywordRegex.str())); \
|
|
}
|
|
#include "lexer/token_registry.def"
|
|
|
|
// Verify that the symbol tokens are sorted from longest to shortest. This is
|
|
// important to ensure that simply in-order testing will identify tokens
|
|
// following the max-munch rule.
|
|
TEST(TokenKindTest, SymbolsInDescendingLength) {
|
|
int previous_length = INT_MAX;
|
|
#define CARBON_SYMBOL_TOKEN(TokenName, Spelling) \
|
|
EXPECT_LE(llvm::StringRef(Spelling).size(), previous_length) \
|
|
<< "Symbol token not in descending length order: " << #TokenName; \
|
|
previous_length = llvm::StringRef(Spelling).size();
|
|
#include "lexer/token_registry.def"
|
|
EXPECT_GT(previous_length, 0);
|
|
}
|
|
|
|
} // namespace
|
|
} // namespace Carbon
|