carbon-language/carbon-lang
1
0
Fork 0
mirror of https://github.com/carbon-language/carbon-lang.git synced 2026-10-05 22:02:55 +01:00
Code Issues Packages Projects Releases 10 Wiki Activity

Introduce a source generator and end-to-end compile benchmarks (#4124)

Browse Source 
The big addition here is a very, very rough and very early skeleton of a
source code generator framework. This builds upon the lexers identifier
synthesis logic, improving on its framework and wiring it up with the
most rudimentary of source file generation. This is just enough to
roughly replicate my "big API file" source code benchmarks.

The source generation works *very* hard to both vary the structure and
content of the source as much as possible while ensuring the same
*total* amount of each construct is in use, from bytes in identifiers to
line breaks, parameters, etc. This lets us generate randomly structure
inputs that should consistently take the exact same amount of total work
to compile.

The complex identifier synthesis logic from the lexer's benchmark is
moved over here and the lexer uses APIs in the source generator for
identifiers. The other source synthesis in the lexer's benchmark isn't
yet moved over, but should likely be slowly absorbed here as it can be
refactored into a more principled and re-usable form. Some bits may stay
of course if they're just too lexer-specific.

Next, this adds a simple end-to-end compile benchmark for the driver
that directly and much more clearly reproduces all the measurements I've
done manually up until now. It should also be easy to extend to more
patterns over time as we add support to the source generator to produce
those patterns.

Last but not least, I've added a tiny CLI to the source generator so
that you can generate source code manually. This is especially nice for
generating demo source code to actually run through the driver or look
at in an editor. The CLI can also generate C++ source code which lets us
do some minimal comparative benchmarking between Carbon and C++/Clang.

There are huge number of TODOs in the source generation framework. This
is going to be a large ongoing effort I suspect.

There are also a bunch of rough edges I've left to try and get this out
for review sooner. I've left TODOs for refactorings that really need to
be done here, but hoping these can maybe be follow-ups. If not, please
flag and I'll try to layer them on here.

Sample compile benchmark output, nicely showing where we are w.r.t. our
goal speeds (2x behind on lex and check, 5x on parse) at least on a
recent AMD server CPU:
```
------------------------------------------------------------------------------------------------------
Benchmark                                                 Time             CPU   Iterations      Lines
------------------------------------------------------------------------------------------------------
BM_CompileAPIFileDenseDecls<Phase::Lex>/256           29420 ns        29419 ns        22860 6.62847M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/1024         146130 ns       146128 ns         4840 6.69959M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/4096         601584 ns       601577 ns         1020 6.69573M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/16384       2547578 ns      2547313 ns          280   6.404M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/65536      10816591 ns     10816389 ns           80 6.05193M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/262144     52191320 ns     52189828 ns           20 5.02261M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/256        101706 ns       101698 ns         6900 1.91745M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/1024       512161 ns       512162 ns         1380  1.9115M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/4096      2078426 ns      2078430 ns          340   1.938M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/16384     8795786 ns      8795583 ns          100 1.85468M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/65536    35073596 ns     35072973 ns           20 1.86639M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/262144  151100688 ns    151097370 ns           20 1.73483M/s
BM_CompileAPIFileDenseDecls<Phase::Check>/256        957059 ns       957049 ns          740 203.751k/s
BM_CompileAPIFileDenseDecls<Phase::Check>/1024      1956134 ns      1955985 ns          360 500.515k/s
BM_CompileAPIFileDenseDecls<Phase::Check>/4096      5797864 ns      5797417 ns          120 694.792k/s
BM_CompileAPIFileDenseDecls<Phase::Check>/16384    21219608 ns     21217584 ns           40 768.843k/s
BM_CompileAPIFileDenseDecls<Phase::Check>/65536    96311116 ns     96302334 ns           20 679.734k/s
BM_CompileAPIFileDenseDecls<Phase::Check>/262144  371637963 ns    371609964 ns           20 705.387k/s
```

Lest someone think this is *bad*, the fact that we're already within 2x
of our rather audacious goals makes me quite happy. =D

---------

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This commit is contained in:
Chandler Carruth
2024-08-13 19:37:55 +00:00
committed by GitHub
co-authored by Jon Ross-Perkins Richard Smith
parent e62973a8ef
commit a9c815c9f4
13 changed files with 1562 additions and 181 deletions
Download Patch File Download Diff File Expand all files Collapse all files
toolchain/lex/tokenized_buffer_benchmark.cpp
+18 -180
View File
@@ -11,6 +11,7 @@
#include "common/check.h"
#include "llvm/ADT/Sequence.h"
#include "llvm/ADT/StringExtras.h"
#include "testing/base/source_gen.h"
#include "toolchain/base/value_store.h"
#include "toolchain/diagnostics/diagnostic_emitter.h"
#include "toolchain/diagnostics/null_diagnostics.h"
@@ -26,180 +27,14 @@ namespace {
// and 1% itself needs to not be too tiny. This makes 100,000 a great balance.
constexpr int NumTokens = 100'000;
auto IdentifierStartChars() -> llvm::ArrayRef<char> {
static llvm::SmallVector<char> chars = [] {
llvm::SmallVector<char> chars;
chars.push_back('_');
for (char c : llvm::seq_inclusive('A', 'Z')) {
chars.push_back(c);
}
for (char c : llvm::seq_inclusive('a', 'z')) {
chars.push_back(c);
}
return chars;
}();
return chars;
}
auto IdentifierChars() -> llvm::ArrayRef<char> {
static llvm::SmallVector<char> chars = [] {
llvm::ArrayRef<char> start_chars = IdentifierStartChars();
llvm::SmallVector<char> chars(start_chars.begin(), start_chars.end());
for (char c : llvm::seq_inclusive('0', '9')) {
chars.push_back(c);
}
return chars;
}();
return chars;
}
// Generates a random identifier string of the specified length using the
// provided RNG BitGen.
auto GenerateRandomIdentifier(absl::BitGen& gen, int length) -> std::string {
llvm::ArrayRef<char> start_chars = IdentifierStartChars();
llvm::ArrayRef<char> chars = IdentifierChars();
std::string id_result;
llvm::raw_string_ostream os(id_result);
llvm::StringRef id;
do {
// Erase any prior attempts to find an identifier.
id_result.clear();
os << start_chars[absl::Uniform<int>(gen, 0, start_chars.size())];
for (int j : llvm::seq(0, length)) {
static_cast<void>(j);
os << chars[absl::Uniform<int>(gen, 0, chars.size())];
}
// Check if we ended up forming an integer type literal or a keyword, and
// try again.
id = llvm::StringRef(id_result);
} while (
llvm::any_of(TokenKind::KeywordTokens,
[id](auto token) { return id == token.fixed_spelling(); }) ||
((id.consume_front("i") || id.consume_front("u") ||
id.consume_front("f")) &&
llvm::all_of(id, [](const char c) { return llvm::isDigit(c); })));
return id_result;
}
// Get a static pool of random identifiers with the desired distribution.
template <int MinLength = 1, int MaxLength = 64, bool Uniform = false>
auto GetRandomIdentifiers() -> const std::array<std::string, NumTokens>& {
static_assert(MinLength <= MaxLength);
static_assert(
Uniform || MaxLength <= 64,
"Cannot produce a meaningful non-uniform distribution of lengths longer "
"than 64 as those are exceedingly rare in our observed data sets.");
static const std::array<std::string, NumTokens> id_storage = [] {
std::array<int, 64> id_length_counts;
// For non-uniform distribution, we simulate a distribution roughly based on
// the observed histogram of identifier lengths, but smoothed a bit and
// reduced to small counts so that we cycle through all the lengths
// reasonably quickly. We want sampling of even 10% of NumTokens from this
// in a round-robin form to not be skewed overly much. This still inherently
// compresses the long tail as we'd rather have coverage even though it
// distorts the distribution a bit.
//
// The distribution here comes from a script that analyzes source code run
// over a few directories of LLVM. The script renders a visual ascii-art
// histogram along with the data for each bucket, and that output is
// included in comments above each bucket size below to help visualize the
// rough shape we're aiming for.
//
// 1 characters [3976] ███████████████████████████████▊
id_length_counts[0] = 40;
// 2 characters [3724] █████████████████████████████▊
id_length_counts[1] = 40;
// 3 characters [4173] █████████████████████████████████▍
id_length_counts[2] = 40;
// 4 characters [5000] ████████████████████████████████████████
id_length_counts[3] = 50;
// 5 characters [1568] ████████████▌
id_length_counts[4] = 20;
// 6 characters [2226] █████████████████▊
id_length_counts[5] = 20;
// 7 characters [2380] ███████████████████
id_length_counts[6] = 20;
// 8 characters [1786] ██████████████▎
id_length_counts[7] = 18;
// 9 characters [1397] ███████████▏
id_length_counts[8] = 12;
// 10 characters [ 739] █████▉
id_length_counts[9] = 12;
// 11 characters [ 779] ██████▎
id_length_counts[10] = 12;
// 12 characters [1344] ██████████▊
id_length_counts[11] = 12;
// 13 characters [ 498] ████
id_length_counts[12] = 5;
// 14 characters [ 284] ██▎
id_length_counts[13] = 3;
// 15 characters [ 172] █▍
// 16 characters [ 278] ██▎
// 17 characters [ 191] █▌
// 18 characters [ 207] █▋
for (int i : llvm::seq(14, 18)) {
id_length_counts[i] = 2;
}
// 19 - 63 characters are all <100 but non-zero, and we map them to 1 for
// coverage despite slightly over weighting the tail.
for (int i : llvm::seq(18, 64)) {
id_length_counts[i] = 1;
}
// Used to track the different count buckets when in a non-uniform
// distribution.
int length_bucket_index = 0;
int length_count = 0;
std::array<std::string, NumTokens> ids;
absl::BitGen gen;
for (auto [i, id] : llvm::enumerate(ids)) {
if (Uniform) {
// Rather than using randomness, for a uniform distribution rotate
// lengths in round-robin to get a deterministic and exact size on every
// run. We will then shuffle them at the end to produce a random
// ordering.
int length = MinLength + i % (1 + MaxLength - MinLength);
id = GenerateRandomIdentifier(gen, length);
continue;
}
// For non-uniform distribution, walk through each each length bucket
// until our count matches the desired distribution, and then move to the
// next.
id = GenerateRandomIdentifier(gen, length_bucket_index + 1);
if (length_count < id_length_counts[length_bucket_index]) {
++length_count;
} else {
length_bucket_index =
(length_bucket_index + 1) % id_length_counts.size();
length_count = 0;
}
}
return ids;
}();
return id_storage;
}
// Compute a random sequence of just identifiers.
template <int MinLength = 1, int MaxLength = 64, bool Uniform = false>
auto RandomIdentifierSeq(llvm::StringRef separator = " ") -> std::string {
// Get a static pool of identifiers with the desired distribution.
const std::array<std::string, NumTokens>& ids =
GetRandomIdentifiers<MinLength, MaxLength, Uniform>();
// Shuffle tokens so we get exactly one of each identifier but in a random
// order.
std::array<llvm::StringRef, NumTokens> tokens;
for (int i : llvm::seq(NumTokens)) {
tokens[i] = ids[i];
}
std::shuffle(tokens.begin(), tokens.end(), absl::BitGen());
return llvm::join(tokens, separator);
static auto RandomIdentifierSeq(int min_length, int max_length, bool uniform,
llvm::StringRef separator = " ")
-> std::string {
auto& gen = Testing::SourceGen::Global();
llvm::SmallVector<llvm::StringRef> ids =
gen.GetShuffledIdentifiers(NumTokens, min_length, max_length, uniform);
return llvm::join(ids, separator);
}
auto GetSymbolTokenTable() -> llvm::ArrayRef<TokenKind> {
@@ -299,7 +134,6 @@ auto RandomSource(RandomSourceOptions options) -> std::string {
// Get static pools of symbols, keywords, and identifiers.
llvm::ArrayRef<TokenKind> symbols = GetSymbolTokenTable();
llvm::ArrayRef<TokenKind> keywords = TokenKind::KeywordTokens;
const std::array<std::string, NumTokens>& ids = GetRandomIdentifiers();
// Build a list of StringRefs from the different types with the desired
// distribution, then shuffle that list.
@@ -312,6 +146,8 @@ auto RandomSource(RandomSourceOptions options) -> std::string {
<< "We require at least 500 identifiers as we need to collect a "
"reasonable number of samples to end up with a reasonable "
"distribution of lengths.";
llvm::SmallVector<llvm::StringRef> ids =
Testing::SourceGen::Global().GetIdentifiers(num_identifiers);
for (int i : llvm::seq(num_symbols)) {
tokens[i] = symbols[i % symbols.size()].fixed_spelling();
@@ -454,7 +290,8 @@ BENCHMARK(BM_ValidKeywordsAsRawIdentifiers);
// This benchmark does a 50-50 split of r-prefixed and r#-prefixed identifiers
// to directly compare raw and non-raw performance.
void BM_RawIdentifierFocus(benchmark::State& state) {
const std::array<std::string, NumTokens>& ids = GetRandomIdentifiers();
llvm::SmallVector<llvm::StringRef> ids =
Testing::SourceGen::Global().GetIdentifiers(NumTokens / 2);
llvm::SmallVector<std::string> modified_ids;
// As we resize, start with the in-use prefix. Note that `r#` uses the first
@@ -490,7 +327,7 @@ BENCHMARK(BM_RawIdentifierFocus);
template <int MinLength, int MaxLength, bool Uniform>
void BM_ValidIdentifiers(benchmark::State& state) {
std::string source = RandomIdentifierSeq<MinLength, MaxLength, Uniform>();
std::string source = RandomIdentifierSeq(MinLength, MaxLength, Uniform);
LexerBenchHelper helper(source);
for (auto _ : state) {
@@ -525,7 +362,7 @@ BENCHMARK(BM_ValidIdentifiers<80, 80, /*Uniform=*/true>);
void BM_HorizontalWhitespace(benchmark::State& state) {
int num_spaces = state.range(0);
std::string separator(num_spaces, ' ');
std::string source = RandomIdentifierSeq<3, 5, /*Uniform=*/true>(separator);
std::string source = RandomIdentifierSeq(3, 5, /*uniform=*/true, separator);
LexerBenchHelper helper(source);
for (auto _ : state) {
@@ -579,7 +416,8 @@ void BM_GroupingSymbols(benchmark::State& state) {
// It should still let us look for specific pain points. We do include some
// whitespace and keywords to make sure *some* other parts of the benchmark
// are also active and have some reasonable icache pressure.
const std::array<std::string, NumTokens>& ids = GetRandomIdentifiers();
llvm::SmallVector<llvm::StringRef> ids =
Testing::SourceGen::Global().GetShuffledIdentifiers(NumTokens);
std::string source;
llvm::raw_string_ostream os(source);
int num_tokens_per_nest =
@@ -658,7 +496,7 @@ BENCHMARK(BM_GroupingSymbols)
void BM_BlankLines(benchmark::State& state) {
int num_blank_lines = state.range(0);
std::string separator(num_blank_lines, '\n');
std::string source = RandomIdentifierSeq<3, 5, /*Uniform=*/true>(separator);
std::string source = RandomIdentifierSeq(3, 5, /*uniform=*/true, separator);
LexerBenchHelper helper(source);
for (auto _ : state) {
@@ -693,7 +531,7 @@ void BM_CommentLines(benchmark::State& state) {
os << std::string(comment_indent, ' ') << "//"
<< std::string(comment_length, ' ') << "\n";
}
std::string source = RandomIdentifierSeq<3, 5, /*Uniform=*/true>(separator);
std::string source = RandomIdentifierSeq(3, 5, /*uniform=*/true, separator);
LexerBenchHelper helper(source);
for (auto _ : state) {