Merge lexer from the toolchain repository. (#213)

The only change here is to update the fuzzer build extension path.

The main original commit message:

> Add an initial lexer. (#17)
>
> The specific logic here hasn't been updated to track the latest
> discussed changes, much less implement many aspects of things like
> Unicode support.
>
> However, this should lay out a reasonable framework and set of APIs.
> It gives an idea of the overall lexer architecture being proposed. The
> actual lexing algorithm is a relatively boring and naive hand written
> loop. It may make sense to replace this with something generated or
> other more advanced approach in the future, getting the implementation
> right was not the primary goal here. Instead, the focus was entirely
> on the architecture, encapsulation, APIs, and the testing
> infrastructure.
>
> The architecture of the lexer differs from "classical" high
> performance lexers in compilers. A high level summary:
>
> -   It is eager rather than lazy, lexing an entire file.
> -   Tokens intrinsically know their source location.
> -   Grouping lexical symbols are tracked within the lexer.
> -   Indentation is tracked within the lexer.
>
> Tracking of grouping and indentation is intended to simplify the
> strategies used for recovery of mismatched grouping tokens, and
> eventually use indentation.
>
> Folding source location into the token itself simplifies the data
> structures significantly, and doesn't lose any fidelity due to the
> absence of a preprocessor with token pasting.
>
> The fact that this is an eager lexer instead of a lazy lexer is
> designed to simplify the implementation and testing of the lexer (and
> subsequent components). There is no reason to expect Carbon to lex so
> many tokens that there are significant locality advantages of lazy
> lexing. Moreover, if we want comparable performance benefits, I think
> pipelining is a much more promising architecture than laziness. For
> now, the simplicity is a huge win.
>
> Being eager also makes it easy for us to use extremely dense memory
> encodings for the information about lexed tokens. Everything is
> created in a dense array, and small indices are used to identify each
> token within the array.
>
> There is a fuzzer included here that we have run extensively over the
> code, but currently toolchain bugs and Bazel limitations prevent it
> from easily building. I'm hoping myself or someone else can push on
> this soon and enable the fuzzer to at least build if not run fuzz
> tests automatically. We have a significant fuzzing corpus that I'll
> add in a subsequent commit as well.

This also includes the fuzzer whose commit message was:

> Add fuzz testing infrastructure and the lexer's fuzzer. (#21)
>
> This adds a fairly simple `cc_fuzz_test` macro that is specialized for
> working with LLVM's LibFuzzer. In addition to building the fuzzer
> binary with the toolchain's `fuzzer` feature, it also sets up the test
> execution to pass the corpus as file arguments which is a simple
> mechanism to enable regression testing against the fuzz corpus.
>
> I've included an initial fuzzer corpus as well. To run the fuzzer in
> an open ended fashion, and build up a larger corpus:
> ```shell
> mkdir /tmp/new_corpus
> cp lexer/fuzzer_corpus/* /tmp/new_corpus
> ./bazel-bin/lexer/tokenized_buffer_fuzzer /tmp/new_corpus
> ```
>
> You can parallelize the fuzzer by adding `-jobs=N` for N threads. For
> more details about running fuzzers, see the documentation:
> http://llvm.org/docs/LibFuzzer.html
>
> To minimize and merge any interesting new inputs:
> ```shell
> ./bazel-bin/lexer/tokenized_buffer_fuzzer -merge=1 \
>     lexer/fuzzer_corpus /tmp/new_corpus
> ```

Co-authored-by: Jon Meow <46229924+jonmeow@users.noreply.github.com>

lexer/token_kind.cpp

@@ -0,0 +1,103 @@
+// Part of the Carbon Language project, under the Apache License v2.0 with LLVM
+// Exceptions. See /LICENSE for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+#include "lexer/token_kind.h"
+
+#include "llvm/ADT/StringRef.h"
+
+namespace Carbon {
+
+auto TokenKind::Name() const -> llvm::StringRef {
+  static constexpr llvm::StringLiteral Names[] = {
+#define CARBON_TOKEN(TokenName) #TokenName,
+#include "lexer/token_registry.def"
+  };
+  return Names[static_cast<int>(kind_value)];
+}
+
+auto TokenKind::IsSymbol() const -> bool {
+  static constexpr bool Table[] = {
+#define CARBON_TOKEN(TokenName) false,
+#define CARBON_SYMBOL_TOKEN(TokenName, Spelling) true,
+#include "lexer/token_registry.def"
+  };
+  return Table[static_cast<int>(kind_value)];
+}
+
+auto TokenKind::IsGroupingSymbol() const -> bool {
+  static constexpr bool Table[] = {
+#define CARBON_TOKEN(TokenName) false,
+#define CARBON_OPENING_GROUP_SYMBOL_TOKEN(TokenName, Spelling, ClosingName) \
+  true,
+#define CARBON_CLOSING_GROUP_SYMBOL_TOKEN(TokenName, Spelling, OpeningName) \
+  true,
+#include "lexer/token_registry.def"
+  };
+  return Table[static_cast<int>(kind_value)];
+}
+
+auto TokenKind::IsOpeningSymbol() const -> bool {
+  static constexpr bool Table[] = {
+#define CARBON_TOKEN(TokenName) false,
+#define CARBON_OPENING_GROUP_SYMBOL_TOKEN(TokenName, Spelling, ClosingName) \
+  true,
+#include "lexer/token_registry.def"
+  };
+  return Table[static_cast<int>(kind_value)];
+}
+
+auto TokenKind::GetClosingSymbol() const -> TokenKind {
+  static constexpr TokenKind Table[] = {
+#define CARBON_TOKEN(TokenName) Error(),
+#define CARBON_OPENING_GROUP_SYMBOL_TOKEN(TokenName, Spelling, ClosingName) \
+  ClosingName(),
+#include "lexer/token_registry.def"
+  };
+  auto result = Table[static_cast<int>(kind_value)];
+  assert(result != Error() && "Only opening symbols are valid!");
+  return result;
+}
+
+auto TokenKind::IsClosingSymbol() const -> bool {
+  static constexpr bool Table[] = {
+#define CARBON_TOKEN(TokenName) false,
+#define CARBON_CLOSING_GROUP_SYMBOL_TOKEN(TokenName, Spelling, OpeningName) \
+  true,
+#include "lexer/token_registry.def"
+  };
+  return Table[static_cast<int>(kind_value)];
+}
+
+auto TokenKind::GetOpeningSymbol() const -> TokenKind {
+  static constexpr TokenKind Table[] = {
+#define CARBON_TOKEN(TokenName) Error(),
+#define CARBON_CLOSING_GROUP_SYMBOL_TOKEN(TokenName, Spelling, OpeningName) \
+  OpeningName(),
+#include "lexer/token_registry.def"
+  };
+  auto result = Table[static_cast<int>(kind_value)];
+  assert(result != Error() && "Only closing symbols are valid!");
+  return result;
+}
+
+auto TokenKind::IsKeyword() const -> bool {
+  static constexpr bool Table[] = {
+#define CARBON_TOKEN(TokenName) false,
+#define CARBON_KEYWORD_TOKEN(TokenName, Spelling) true,
+#include "lexer/token_registry.def"
+  };
+  return Table[static_cast<int>(kind_value)];
+}
+
+auto TokenKind::GetFixedSpelling() const -> llvm::StringRef {
+  static constexpr llvm::StringLiteral Table[] = {
+#define CARBON_TOKEN(TokenName) "",
+#define CARBON_SYMBOL_TOKEN(TokenName, Spelling) Spelling,
+#define CARBON_KEYWORD_TOKEN(TokenName, Spelling) Spelling,
+#include "lexer/token_registry.def"
+  };
+  return Table[static_cast<int>(kind_value)];
+}
+
+}  // namespace Carbon