carbon-language/carbon-lang
1
0
Fork 0
mirror of https://github.com/carbon-language/carbon-lang.git synced 2026-10-04 10:21:04 +01:00
Code Issues Packages Projects Releases 10 Wiki Activity

Shrink the lexer's token location and line data structures. (#4269)

Browse Source 
First, this replaces the separate line index and column index in the
token information with a single 32-bit byte offset of the token. This is
then used to compute line and column numbers with a binary search of the
line structure and then using that to compute the column within the
line. In practice, this is _much_ more efficient:

- Smaller token data structure. This will hopefully combine with a
subsequent optimization PR that shrinks the token data structure still
further.
- Fewer stores to form each token's information in the tight hot loop of
the lexer.
- Less state to maintain while lexing, fewer computations while lexing.

We only have to search to build the line and column information off the
hot lexing path, and so this ends up being a significant win and shrinks
some of the more significant data structures.

Second, this shrinks the line start to a 32-bit integer and removes the
line length. Our source buffer already ensures we only have 2 GiB of
source with a nice diagnostic. I've just added a check to help document
this in the lexer. The line length can be avoided in all of the cases it
was being used, largely by looking at the next line's start and working
from there. This also precipitated cleaning up some code that dated from
when lines were only built during lexing rather than being pre-built,
which resulted in nice simplifications.

With this PR, I think it makes sense to re-name a bunch of methods on
`TokenizedBuffer`, but to an extent that was already needed as these
methods somewhat predate the more pervasive style conventions. I avoided
that here to keep this PR focused on the implementation change, I'll
create a subsequent PR to update the API to both better nomenclature and
remove deviations from our conventions.

There may also be a way to de-duplicate the binary search in the
diagnostic location conversion and the main line accessor binary search,
but it wasn't obvious to me that it would be a net savings, so left it
alone for now.

The performance impact of this varies quite a bit...

The lexer's benchmark improves pretty consistent across the board on
both x86 and Arm. For x86, where I have nice comparison tools, it
appears 3% to 20% faster depending on the specific pattern. For Arm
server CPUs at least it seems a much smaller but still an improvement.

The overall compilation benchmarks however don't improve much with these
changes alone on x86. Significant reduction in instruction count
required for lexing, but the overall performance is bottlenecked
elsewhere in the overall compilation it seems. However, on Arm, despite
the more modest gains in special cases of lexing, this shows fairly
consistent 1-2% improvements in overall lexing performance on our
compilation benchmark. And the expectaiton is these improvements will
compound with subsequent work to further compact our representation.

---------

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This commit is contained in:
Chandler Carruth
2024-09-03 23:56:44 +00:00
committed by GitHub
co-authored by Jon Ross-Perkins
parent 8b0154ce85
commit 97e98bcc5a
3 changed files with 120 additions and 124 deletions
Download Patch File Download Diff File Expand all files Collapse all files
toolchain/lex/tokenized_buffer.cpp
+58 -45
View File
@@ -4,6 +4,7 @@
#include "toolchain/lex/tokenized_buffer.h"
#include <algorithm>
#include <cmath>
#include "common/check.h"
@@ -24,7 +25,7 @@ auto TokenizedBuffer::GetKind(TokenIndex token) const -> TokenKind {
}
auto TokenizedBuffer::GetLine(TokenIndex token) const -> LineIndex {
return GetTokenInfo(token).token_line;
return FindLineIndex(GetTokenInfo(token).byte_offset);
}
auto TokenizedBuffer::GetLineNumber(TokenIndex token) const -> int {
@@ -32,7 +33,9 @@ auto TokenizedBuffer::GetLineNumber(TokenIndex token) const -> int {
}
auto TokenizedBuffer::GetColumnNumber(TokenIndex token) const -> int {
return GetTokenInfo(token).column + 1;
const auto& token_info = GetTokenInfo(token);
const auto& line_info = GetLineInfo(FindLineIndex(token_info.byte_offset));
return token_info.byte_offset - line_info.start + 1;
}
auto TokenizedBuffer::GetEndLoc(TokenIndex token) const
@@ -62,19 +65,16 @@ auto TokenizedBuffer::GetTokenText(TokenIndex token) const -> llvm::StringRef {
}
if (token_info.kind == TokenKind::Error) {
const auto& line_info = GetLineInfo(token_info.token_line);
int64_t token_start = line_info.start + token_info.column;
return source_->text().substr(token_start, token_info.error_length);
return source_->text().substr(token_info.byte_offset,
token_info.error_length);
}
// Refer back to the source text to preserve oddities like radix or digit
// separators the author included.
if (token_info.kind == TokenKind::IntLiteral ||
token_info.kind == TokenKind::RealLiteral) {
const auto& line_info = GetLineInfo(token_info.token_line);
int64_t token_start = line_info.start + token_info.column;
std::optional<NumericLiteral> relexed_token =
NumericLiteral::Lex(source_->text().substr(token_start));
NumericLiteral::Lex(source_->text().substr(token_info.byte_offset));
CARBON_CHECK(relexed_token) << "Could not reform numeric literal token.";
return relexed_token->text();
}
@@ -82,10 +82,8 @@ auto TokenizedBuffer::GetTokenText(TokenIndex token) const -> llvm::StringRef {
// Refer back to the source text to find the original spelling, including
// escape sequences etc.
if (token_info.kind == TokenKind::StringLiteral) {
const auto& line_info = GetLineInfo(token_info.token_line);
int64_t token_start = line_info.start + token_info.column;
std::optional<StringLiteral> relexed_token =
StringLiteral::Lex(source_->text().substr(token_start));
StringLiteral::Lex(source_->text().substr(token_info.byte_offset));
CARBON_CHECK(relexed_token) << "Could not reform string literal token.";
return relexed_token->text();
}
@@ -93,10 +91,9 @@ auto TokenizedBuffer::GetTokenText(TokenIndex token) const -> llvm::StringRef {
// Refer back to the source text to avoid needing to reconstruct the
// spelling from the size.
if (token_info.kind.is_sized_type_literal()) {
const auto& line_info = GetLineInfo(token_info.token_line);
int64_t token_start = line_info.start + token_info.column;
llvm::StringRef suffix =
source_->text().substr(token_start + 1).take_while(IsDecimalDigit);
llvm::StringRef suffix = source_->text()
.substr(token_info.byte_offset + 1)
.take_while(IsDecimalDigit);
return llvm::StringRef(suffix.data() - 1, suffix.size() + 1);
}
@@ -254,6 +251,7 @@ auto TokenizedBuffer::PrintToken(llvm::raw_ostream& output_stream,
widths.Widen(GetTokenPrintWidths(token));
int token_index = token.index;
const auto& token_info = GetTokenInfo(token);
LineIndex line_index = FindLineIndex(token_info.byte_offset);
llvm::StringRef token_text = GetTokenText(token);
// Output the main chunk using one format string. We have to do the
@@ -265,10 +263,9 @@ auto TokenizedBuffer::PrintToken(llvm::raw_ostream& output_stream,
llvm::format_decimal(token_index, widths.index),
llvm::right_justify(llvm::formatv("'{0}'", token_info.kind.name()).str(),
widths.kind + 2),
llvm::format_decimal(GetLineNumber(token_info.token_line), widths.line),
llvm::format_decimal(GetLineNumber(GetLine(token)), widths.line),
llvm::format_decimal(GetColumnNumber(token), widths.column),
llvm::format_decimal(GetIndentColumnNumber(token_info.token_line),
widths.indent),
llvm::format_decimal(GetIndentColumnNumber(line_index), widths.indent),
token_text);
switch (token_info.kind) {
@@ -313,6 +310,31 @@ auto TokenizedBuffer::PrintToken(llvm::raw_ostream& output_stream,
output_stream << " },";
}
// Find the line index corresponding to a specific byte offset within the source
// text for this tokenized buffer.
//
// This takes advantage of the lines being sorted by their starting byte offsets
// to do a binary search for the line that contains the provided offset.
auto TokenizedBuffer::FindLineIndex(int32_t byte_offset) const -> LineIndex {
CARBON_DCHECK(!line_infos_.empty());
const auto* line_it =
std::partition_point(line_infos_.begin(), line_infos_.end(),
[byte_offset](LineInfo line_info) {
return line_info.start <= byte_offset;
});
--line_it;
// If this isn't the first line but it starts past the end of the source, then
// this is a synthetic line added for simplicity of lexing. Step back one
// further to find the last non-synthetic line.
if (line_it != line_infos_.begin() &&
line_it->start == static_cast<int32_t>(source_->text().size())) {
--line_it;
}
CARBON_DCHECK(line_it->start <= byte_offset);
return LineIndex(line_it - line_infos_.begin());
}
auto TokenizedBuffer::GetLineInfo(LineIndex line) -> LineInfo& {
return line_infos_[line.index];
}
@@ -355,40 +377,32 @@ auto TokenizedBuffer::SourceBufferDiagnosticConverter::ConvertLoc(
const char* loc, ContextFnT /*context_fn*/) const -> DiagnosticLoc {
CARBON_CHECK(StringRefContainsPointer(buffer_->source_->text(), loc))
<< "location not within buffer";
int64_t offset = loc - buffer_->source_->text().begin();
int32_t offset = loc - buffer_->source_->text().begin();
// Find the first line starting after the given location. Note that we can't
// inspect `line.length` here because it is not necessarily correct for the
// final line during lexing (but will be correct later for the parse tree).
const auto* line_it = std::partition_point(
// Find the first line starting after the given location.
const auto* next_line_it = std::partition_point(
buffer_->line_infos_.begin(), buffer_->line_infos_.end(),
[offset](const LineInfo& line) { return line.start <= offset; });
// Step back one line to find the line containing the given position.
CARBON_CHECK(line_it != buffer_->line_infos_.begin())
CARBON_CHECK(next_line_it != buffer_->line_infos_.begin())
<< "location precedes the start of the first line";
--line_it;
const auto* line_it = std::prev(next_line_it);
int line_number = line_it - buffer_->line_infos_.begin();
int column_number = offset - line_it->start;
// Start by grabbing the line from the buffer. If the line isn't fully lexed,
// the length will be npos and the line will be grabbed from the known start
// to the end of the buffer; we'll then adjust the length.
llvm::StringRef line =
buffer_->source_->text().substr(line_it->start, line_it->length);
if (line_it->length == static_cast<int32_t>(llvm::StringRef::npos)) {
CARBON_CHECK(line.take_front(column_number).count('\n') == 0)
<< "Currently we assume no unlexed newlines prior to the error column, "
"but there was one when erroring at "
<< buffer_->source_->filename() << ":" << line_number << ":"
<< column_number;
// Look for the next newline since we don't know the length. We can start at
// the column because prior newlines will have been lexed.
auto end_newline_pos = line.find('\n', column_number);
if (end_newline_pos != llvm::StringRef::npos) {
line = line.take_front(end_newline_pos);
}
}
// Grab the line from the buffer by slicing from this line to the next
// minus the newline. When on the last line, instead use the start to the end
// of the buffer.
llvm::StringRef text = buffer_->source_->text();
llvm::StringRef line = next_line_it != buffer_->line_infos_.end()
? text.slice(line_it->start, next_line_it->start)
: text.substr(line_it->start);
// Remove a newline at the end of the line if present.
// TODO: This should expand to remove all vertical whitespace bytes at the
// tail of the line such as CR+LF, etc.
line.consume_back("\n");
return {.filename = buffer_->source_->filename(),
.line = line,
@@ -401,9 +415,8 @@ auto TokenDiagnosticConverter::ConvertLoc(TokenIndex token,
-> DiagnosticLoc {
// Map the token location into a position within the source buffer.
const auto& token_info = buffer_->GetTokenInfo(token);
const auto& line_info = buffer_->GetLineInfo(token_info.token_line);
const char* token_start =
buffer_->source_->text().begin() + line_info.start + token_info.column;
buffer_->source_->text().begin() + token_info.byte_offset;
// Find the corresponding file location.
// TODO: Should we somehow indicate in the diagnostic location if this token