Files
carbon-lang/testing/base/source_gen_main.cpp
T
a9c815c9f4 Introduce a source generator and end-to-end compile benchmarks (#4124)
The big addition here is a very, very rough and very early skeleton of a
source code generator framework. This builds upon the lexers identifier
synthesis logic, improving on its framework and wiring it up with the
most rudimentary of source file generation. This is just enough to
roughly replicate my "big API file" source code benchmarks.

The source generation works *very* hard to both vary the structure and
content of the source as much as possible while ensuring the same
*total* amount of each construct is in use, from bytes in identifiers to
line breaks, parameters, etc. This lets us generate randomly structure
inputs that should consistently take the exact same amount of total work
to compile.

The complex identifier synthesis logic from the lexer's benchmark is
moved over here and the lexer uses APIs in the source generator for
identifiers. The other source synthesis in the lexer's benchmark isn't
yet moved over, but should likely be slowly absorbed here as it can be
refactored into a more principled and re-usable form. Some bits may stay
of course if they're just too lexer-specific.

Next, this adds a simple end-to-end compile benchmark for the driver
that directly and much more clearly reproduces all the measurements I've
done manually up until now. It should also be easy to extend to more
patterns over time as we add support to the source generator to produce
those patterns.

Last but not least, I've added a tiny CLI to the source generator so
that you can generate source code manually. This is especially nice for
generating demo source code to actually run through the driver or look
at in an editor. The CLI can also generate C++ source code which lets us
do some minimal comparative benchmarking between Carbon and C++/Clang.

There are huge number of TODOs in the source generation framework. This
is going to be a large ongoing effort I suspect.

There are also a bunch of rough edges I've left to try and get this out
for review sooner. I've left TODOs for refactorings that really need to
be done here, but hoping these can maybe be follow-ups. If not, please
flag and I'll try to layer them on here.

Sample compile benchmark output, nicely showing where we are w.r.t. our
goal speeds (2x behind on lex and check, 5x on parse) at least on a
recent AMD server CPU:
```
------------------------------------------------------------------------------------------------------
Benchmark                                                 Time             CPU   Iterations      Lines
------------------------------------------------------------------------------------------------------
BM_CompileAPIFileDenseDecls<Phase::Lex>/256           29420 ns        29419 ns        22860 6.62847M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/1024         146130 ns       146128 ns         4840 6.69959M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/4096         601584 ns       601577 ns         1020 6.69573M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/16384       2547578 ns      2547313 ns          280   6.404M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/65536      10816591 ns     10816389 ns           80 6.05193M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/262144     52191320 ns     52189828 ns           20 5.02261M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/256        101706 ns       101698 ns         6900 1.91745M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/1024       512161 ns       512162 ns         1380  1.9115M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/4096      2078426 ns      2078430 ns          340   1.938M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/16384     8795786 ns      8795583 ns          100 1.85468M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/65536    35073596 ns     35072973 ns           20 1.86639M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/262144  151100688 ns    151097370 ns           20 1.73483M/s
BM_CompileAPIFileDenseDecls<Phase::Check>/256        957059 ns       957049 ns          740 203.751k/s
BM_CompileAPIFileDenseDecls<Phase::Check>/1024      1956134 ns      1955985 ns          360 500.515k/s
BM_CompileAPIFileDenseDecls<Phase::Check>/4096      5797864 ns      5797417 ns          120 694.792k/s
BM_CompileAPIFileDenseDecls<Phase::Check>/16384    21219608 ns     21217584 ns           40 768.843k/s
BM_CompileAPIFileDenseDecls<Phase::Check>/65536    96311116 ns     96302334 ns           20 679.734k/s
BM_CompileAPIFileDenseDecls<Phase::Check>/262144  371637963 ns    371609964 ns           20 705.387k/s
```

Lest someone think this is *bad*, the fact that we're already within 2x
of our rather audacious goals makes me quite happy. =D

---------

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
2024-08-13 19:37:55 +00:00

116 lines
3.5 KiB
C++

// Part of the Carbon Language project, under the Apache License v2.0 with LLVM
// Exceptions. See /LICENSE for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include "common/bazel_working_dir.h"
#include "common/command_line.h"
#include "common/init_llvm.h"
#include "common/ostream.h"
#include "llvm/ADT/ArrayRef.h"
#include "llvm/ADT/SmallVector.h"
#include "llvm/ADT/StringRef.h"
#include "llvm/Support/FileSystem.h"
#include "testing/base/source_gen.h"
namespace Carbon::Testing {
namespace {
constexpr CommandLine::CommandInfo Info = {
.name = "source_gen",
.help = R"""(
A source generator for Carbon.
)""",
};
constexpr CommandLine::ArgInfo OutputArgInfo = {
.name = "output",
.value_name = "FILE",
.help = R"""(
Writes the generate source code to a file rather than stdout.
)""",
};
constexpr CommandLine::ArgInfo LinesArgInfo = {
.name = "lines",
.value_name = "N",
.help = R"""(
The number of lines of code to target for a generated source file.
)""",
};
constexpr CommandLine::ArgInfo LanguageArgInfo = {
.name = "language",
//.value_name = "[carbon|cpp]",
.help = R"""(
The language of source code to generate. The C++ source generation is best
effort to try to provide as much comparable benchmarking as possible, but the
primary language focus is generating Carbon.
)""",
};
auto Run(llvm::ArrayRef<llvm::StringRef> args) -> bool {
// Default to outputting to stdout and writing 10k lines of source code.
llvm::StringRef output_filename = "-";
int lines = 10'000;
SourceGen::Language language;
CommandLine::ParseResult parsed_args = CommandLine::Parse(
args, llvm::outs(), llvm::errs(), Info,
[&](CommandLine::CommandBuilder& b) {
b.AddStringOption(OutputArgInfo,
[&](auto& arg_b) { arg_b.Set(&output_filename); });
b.AddIntegerOption(LinesArgInfo,
[&](auto& arg_b) { arg_b.Set(&lines); });
b.AddOneOfOption(LanguageArgInfo, [&](auto& arg_b) {
arg_b.SetOneOf(
{
arg_b.OneOfValue("carbon", SourceGen::Language::Carbon)
.Default(true),
arg_b.OneOfValue("cpp", SourceGen::Language::Cpp),
},
&language);
});
// No-op action as there is only one operation for this command.
b.Do([] {});
});
if (parsed_args == CommandLine::ParseResult::Error) {
return false;
} else if (parsed_args == CommandLine::ParseResult::MetaSuccess) {
// Fully handled by the CLI library.
return true;
}
std::optional<llvm::raw_fd_ostream> output_file;
llvm::raw_fd_ostream* output = &llvm::outs();
if (output_filename != "-") {
std::error_code ec;
output_file.emplace(output_filename, ec, llvm::sys::fs::OF_None);
if (ec) {
llvm::errs() << "ERROR: Unable to open output file '" << output_filename
<< "': " << ec.message() << "\n";
return false;
}
output = &*output_file;
}
SourceGen gen(language);
*output << gen.GenAPIFileDenseDecls(lines, SourceGen::DenseDeclParams{});
output->flush();
return true;
}
} // namespace
} // namespace Carbon::Testing
auto main(int argc, char** argv) -> int {
// Do LLVM's initialization first, this will also transform UTF-16 to UTF-8.
Carbon::InitLLVM init_llvm(argc, argv);
Carbon::SetWorkingDirForBazel();
llvm::SmallVector<llvm::StringRef> args(argv + 1, argv + argc);
bool success = Carbon::Testing::Run(args);
return success ? EXIT_SUCCESS : EXIT_FAILURE;
}