carbon-language/carbon-lang
1
0
Fork 0
mirror of https://github.com/carbon-language/carbon-lang.git synced 2026-10-05 21:01:21 +01:00
Code Issues Packages Projects Releases 10 Wiki Activity

Add tracking of lexed comments, with skeletal formatting. (#4385)

Browse Source 
In order to format comments, it's helpful if they're tracked. This
tracks them separately from tokens in order to avoid interfering with
parse; it'd be inconvenient if comment tokens could show up in arbitrary
locations, albeit possible to support.

This additionally extracts out the TokenIterator support into a template
in order to generally have it available for IndexBase types. I'm only
adding it for CommentInfo, not sure if we'll want it elsewhere, but this
structure still felt like a good fit.
This commit is contained in:
Jon Ross-Perkins
2024-10-09 21:05:53 +00:00
committed by GitHub
parent efb5d6d25a
commit 1338f9e0ad
8 changed files with 197 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files
toolchain/lex/tokenized_buffer.cpp
+13 -4
View File
@@ -345,15 +345,24 @@ auto TokenizedBuffer::AddLine(LineInfo info) -> LineIndex {
return LineIndex(static_cast<int>(line_infos_.size()) - 1);
}
auto TokenizedBuffer::IsAfterComment(TokenIndex token,
CommentIndex comment_index) const -> bool {
const auto& comment_data = comments_[comment_index.index];
return GetTokenInfo(token).byte_offset() > comment_data.start;
}
auto TokenizedBuffer::GetCommentText(CommentIndex comment_index) const
-> llvm::StringRef {
const auto& comment_data = comments_[comment_index.index];
return source_->text().substr(comment_data.start, comment_data.length);
}
auto TokenizedBuffer::CollectMemUsage(MemUsage& mem_usage,
llvm::StringRef label) const -> void {
mem_usage.Add(MemUsage::ConcatLabel(label, "allocator_"), allocator_);
mem_usage.Add(MemUsage::ConcatLabel(label, "token_infos_"), token_infos_);
mem_usage.Add(MemUsage::ConcatLabel(label, "line_infos_"), line_infos_);
}
auto TokenIterator::Print(llvm::raw_ostream& output) const -> void {
output << token_.index;
mem_usage.Add(MemUsage::ConcatLabel(label, "comments_"), comments_);
}
auto TokenizedBuffer::SourceBufferDiagnosticConverter::ConvertLoc(