This has two goals:
1) It allows us to simplify and remove special cases from the parser:
when we expect a particular token next, we can just check for it
without needing a special case for end-of-file.
2) It gives us a token to use as a position when emitting diagnostics at
the end of the file.
Centralize all updating of `position` to `Consume` and `SkipTo`, so that we can in a single place ensure that we never go past the EOF token.
This patch tries to make everything adhere to the Google declaration
order. This is tricky as there doesn't appear to be a `clang-tidy` check
that helps us here at all.
One of the complex cases are the `enum`-wrapping classes we use. These
need to have a *public* conversion operator to the nested `enum` type to support
`switch` statements and the like. However, this makes it impossible to
fully respect the Google declaration order. We need to define the enum
type before the public API in order to use it in contexts like the
conversion operator. Even defining out-of-line won't help avoid this. It
is weird to have a type in the public API that is private, but again
this is only intended to be used for implicit conversions within
a `switch` statement or a `case` label. In one case, we had the
non-conforming order of declaration. I've added a comment to explain why
there. In the other case, the conversion operator is actually *private*
rather than public, which doesn't actually work in practice. I've made
this public and moved the declaration order to match with a matching
comment.
Only change is to update the path to the fuzzer build extension.
Original main commit message:
> Add an initial parser library. (#30)
>
> This library builds a parse tree, very similar to a concrete syntax
> tree. There are no semantics here, simply introducing the basic
> syntactic structure.
>
> The current focus has been on the APIs and the data structures used to
> represent the parse tree, and not on the actual code doing the
> parsing. The code doing the parsing tries to be reasonably efficient
> and reasonably easy to understand recursive descent parser. But there
> is likely much that can be done to improve this code path. A notable
> area where very little thought has been given yet are emitting good
> diagnostics and doing good recovery in the event of parse errors.
>
> Also, this code does not try to match the current under-discussion
> grammar closely. It is only partial and reflects discussions from some
> time ago. It should be updated incrementally to reflect the current
> expected grammar.
>
> The data structure used for the parse tree is unusual. The first
> constraint is that there is a precise one-to-one correspondence
> between the tokens produced by the lexer and the nodes in the parse
> tree. Every token results in exactly one node. In that way, the parse
> tree can be thought of as merely shaping the token stream into a tree.
>
> Each node is also represented with a fixed set of data that is densely
> packed. Combined with the exact relationship to tokens, this allows us
> to fully allocate the parse tree's storage, and to use a dense array
> rather than a pointer-based tree structure.
>
> The tree structure itself is implicitly defined by tracking the size
> of each subtree rooted at a particular node. See the code comments for
> more details (and I'm happy to add more comments where necessary). The
> goal is to minimize both the allocations (one), the working set size
> of the tree as a whole, and optimize common iteration patterns. The
> tree is stored in postorder. This allows depth-first postorder
> iteration as well as topological iteration by walking in reverse.
>
> Building the parse tree in postorder is a natural consequence of the
> grammar being LR rather than LL, which is a consequence of supporting
> infix operators.
>
> As with the Lexer, the parser supports an API for operating on the
> parse tree, as well as the ability to print the tree in both
> a human-readable and machine-readable format (YAML-based). It includes
> significant unit tests and a fuzz tester. The fuzzer's corpus will be
> in a follow-up commit.
>
> This is the largest chunk of code already written by several of us
> prior to open sourcing. (There are a few more pieces, but they are
> significantly smaller and less interesting.) If there are major things
> that folks would like to see happen here, it may make sense to move
> them into issues for tracking. I have tried to update the code to
> follow the style guidelines, but apologies if I missed anything, just
> let me know. We also have issues #19 and #29 to track things that
> already came up with the lexer.
Co-authored-by: Jon Meow <46229924+jonmeow@users.noreply.github.com>
Richard Smith
Chandler Carruth
