Add tracking of lexed comments, with skeletal formatting. (#4385)

In order to format comments, it's helpful if they're tracked. This
tracks them separately from tokens in order to avoid interfering with
parse; it'd be inconvenient if comment tokens could show up in arbitrary
locations, albeit possible to support.

This additionally extracts out the TokenIterator support into a template
in order to generally have it available for IndexBase types. I'm only
adding it for CommentInfo, not sure if we'll want it elsewhere, but this
structure still felt like a good fit.
This commit is contained in:
Jon Ross-Perkins
2024-10-09 21:05:53 +00:00
committed by GitHub
parent efb5d6d25a
commit 1338f9e0ad
8 changed files with 197 additions and 50 deletions
+12
View File
@@ -16,8 +16,20 @@ auto Format(const Lex::TokenizedBuffer& tokens, llvm::raw_ostream& out)
// TODO: Error recovery.
return false;
}
auto comments = tokens.comments();
auto comment_it = comments.begin();
llvm::ListSeparator sep(" ");
for (auto token : tokens.tokens()) {
while (comment_it != comments.end() &&
tokens.IsAfterComment(token, *comment_it)) {
// TODO: Fix newlines and indent.
out << "\n" << tokens.GetCommentText(*comment_it) << "\n";
++comment_it;
}
switch (tokens.GetKind(token)) {
case Lex::TokenKind::FileStart:
break;