mirror of
https://github.com/carbon-language/carbon-lang.git
synced 2026-09-24 22:02:23 +01:00
Chandler CarruthandJon Meow
3512c2218f
Only change is to update the path to the fuzzer build extension. Original main commit message: > Add an initial parser library. (#30) > > This library builds a parse tree, very similar to a concrete syntax > tree. There are no semantics here, simply introducing the basic > syntactic structure. > > The current focus has been on the APIs and the data structures used to > represent the parse tree, and not on the actual code doing the > parsing. The code doing the parsing tries to be reasonably efficient > and reasonably easy to understand recursive descent parser. But there > is likely much that can be done to improve this code path. A notable > area where very little thought has been given yet are emitting good > diagnostics and doing good recovery in the event of parse errors. > > Also, this code does not try to match the current under-discussion > grammar closely. It is only partial and reflects discussions from some > time ago. It should be updated incrementally to reflect the current > expected grammar. > > The data structure used for the parse tree is unusual. The first > constraint is that there is a precise one-to-one correspondence > between the tokens produced by the lexer and the nodes in the parse > tree. Every token results in exactly one node. In that way, the parse > tree can be thought of as merely shaping the token stream into a tree. > > Each node is also represented with a fixed set of data that is densely > packed. Combined with the exact relationship to tokens, this allows us > to fully allocate the parse tree's storage, and to use a dense array > rather than a pointer-based tree structure. > > The tree structure itself is implicitly defined by tracking the size > of each subtree rooted at a particular node. See the code comments for > more details (and I'm happy to add more comments where necessary). The > goal is to minimize both the allocations (one), the working set size > of the tree as a whole, and optimize common iteration patterns. The > tree is stored in postorder. This allows depth-first postorder > iteration as well as topological iteration by walking in reverse. > > Building the parse tree in postorder is a natural consequence of the > grammar being LR rather than LL, which is a consequence of supporting > infix operators. > > As with the Lexer, the parser supports an API for operating on the > parse tree, as well as the ability to print the tree in both > a human-readable and machine-readable format (YAML-based). It includes > significant unit tests and a fuzz tester. The fuzzer's corpus will be > in a follow-up commit. > > This is the largest chunk of code already written by several of us > prior to open sourcing. (There are a few more pieces, but they are > significantly smaller and less interesting.) If there are major things > that folks would like to see happen here, it may make sense to move > them into issues for tracking. I have tried to update the code to > follow the style guidelines, but apologies if I missed anything, just > let me know. We also have issues #19 and #29 to track things that > already came up with the lexer. Co-authored-by: Jon Meow <46229924+jonmeow@users.noreply.github.com>
26 lines
982 B
Modula-2
26 lines
982 B
Modula-2
// Part of the Carbon Language project, under the Apache License v2.0 with LLVM
|
|
// Exceptions. See /LICENSE for license information.
|
|
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
|
|
//
|
|
// Note that this is an X-macro header.
|
|
//
|
|
// It does not use `#include` guards, and instead is designed to be `#include`ed
|
|
// after the x-macro is defined in order for its inclusion to expand to the
|
|
// desired output. The x-macro for this header is `CARBON_PARSE_NODE_KIND`. The
|
|
// definition provided will be removed at the end of this file to clean up.
|
|
|
|
#ifndef CARBON_PARSE_NODE_KIND
|
|
#error "Must define the x-macro to use this file."
|
|
#endif
|
|
|
|
CARBON_PARSE_NODE_KIND(CodeBlockEnd)
|
|
CARBON_PARSE_NODE_KIND(CodeBlock)
|
|
CARBON_PARSE_NODE_KIND(DeclarationEnd)
|
|
CARBON_PARSE_NODE_KIND(EmptyDeclaration)
|
|
CARBON_PARSE_NODE_KIND(FunctionDeclaration)
|
|
CARBON_PARSE_NODE_KIND(Identifier)
|
|
CARBON_PARSE_NODE_KIND(ParameterListEnd)
|
|
CARBON_PARSE_NODE_KIND(ParameterList)
|
|
|
|
#undef CARBON_PARSE_NODE_KIND
|