Commit Graph
203 Commits
Author SHA1 Message Date
Chandler Carruth 4148161e24 Refactor value store code to use separate files. (#4477)
This is in anticipation of making the integer value store be customized
heavily. I'd like to extract it from the common code when doing that, so
first disentangling them here without any intended change in
functionality or behavior to enable that.

I've tried to update `#include`s to be as minimal as I can and added a
few missing includes spotted in the process.

I've split the test for value store to include what was easy focused on
just the value store templates rather than the unified shared value
stores.

This might surface some opportunities for adding more tests, but for
this PR, just doing the minimal restructuring.
2024-11-04 04:00:17 +00:00
Jon Ross-Perkins 145c44b66c Move the language server into toolchain's busybox. (#4469)
Removes the separate language server binary; I'm not sure we need to
provide it. Instead, `carbon language-server` is added as a subcommand.

Moves //language_server to //toolchain/language_server. Splits into a
trivial language_server.h, and a substantive server.h. I wasn't sure
about a better name, but wanted the split similar to check/check.h,
lex/lex.h, etc. At the same time, the class is probably going to be a
little big so not a good fit to through into just a cpp file.

This fixes some style issues with the language server class, but
generally I'm trying to not address things here in order to keep it
simpler.
2024-11-01 19:49:39 +00:00
Chandler Carruth 954441c358 Exempt the clang subcommand when fuzzing. (#4468)
This teaches the driver library to track when its being used with
fuzzing and disables the `clang` subcommand from actually running Clang.

The Clang libraries have a large backlog of fuzzer-found issues that
isn't being actively reduced, so we can't productively fuzz into it.
This lets us more productively fuzz at the top level.

This is also available on the command line itself, which should be
useful if anyone wants to fuzz Carbon from the command line using tools
like AFL -- they can inject this flag to avoid getting noise from the
fuzzer hitting known issues in Clang.
2024-11-01 15:35:29 +00:00
Jon Ross-Perkins 957599b2ab Implement a basic busybox for carbon/clang. (#4406)
For reference, we're going down the busyboxing route because Carbon
depends on Clang, and we want both to be available as binaries.
Busyboxing allows this while avoiding duplicating symbols between
multiple binaries.

I'm removing the `cc_binary` for `driver:carbon` because I want to avoid
a significant increase in binary outputs; `bazel run //toolchain` still
works great.

This still doesn't have great test coverage (but non-zero:
`//examples:sieve` still builds/runs, for example). The problem is that
we want to avoid subprocessing for performance, but this mainly deals
with subprocessing. I'm still thinking about good approaches for that,
since we'll probably want more significant testing for `clang`
interaction... the solution might involve busyboxing `file_test` too.

Note development on this ran into the argv issue being fixed in #4405
2024-10-30 17:07:55 +00:00
Jon Ross-PerkinsandGeoff Romer 06f4eec91e Modify lex yaml output to elide FileStart/End in tests. (#4433)
Trying to make split file tests of lex functionality shorter and easier
to read. numeric_literals.carbon in particular has an example of why I'm
interested in this (at the bottom). This also switches from `[]` list
format to `-` list format so that the trailing `]` is removed.

Trimming comments in tokenized_buffer.h because (1) it feels like it's
giving too much detail about what's printed, which has drifted slightly
and (2) it also feels like it's trying to justify YAML output, when
that's just what we're doing in general.

---------

Co-authored-by: Geoff Romer <gromer@google.com>
2024-10-23 18:56:41 +00:00
Richard Smith 2e63da1a40 Move diagnostic kind name to the end of the diagnostic. (#4437)
Also surround it in square brackets rather than parentheses. This
matches the format used by Clang and GCC, and means diagnostics will
still match the `file:line:col: error: ` pattern used by some IDE tools.

Before:
```console
fail_builtins.carbon:11:11: error(AliasRequiresNameRef): alias initializer must be a name reference
```

After:
```console
fail_builtins.carbon:11:11: error: alias initializer must be a name reference [AliasRequiresNameRef]
```

Also tighten up test regex to only match on `STDERR` lines that list a
file name.
2024-10-23 16:56:23 +00:00
Jon Ross-Perkins c25177658a Add more compile benchmark stats (#4408)
I was discussing some details of cross-compiler lex performance. Since
we were talking about LoC initially, and lex performance especially will
differ based on bytes and tokens being lexed, throwing in some stats for
how we're processing those. Here's some example output:

```
----------------------------------------------------------------------------------------------------------------------------
Benchmark                                                 Time             CPU   Iterations      Bytes      Lines     Tokens
----------------------------------------------------------------------------------------------------------------------------
BM_CompileAPIFileDenseDecls<Phase::Lex>/256           31828 ns        31798 ns        22528  165.64M/s 6.13247M/s 34.6249M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/1024         147513 ns       147434 ns         5120 220.363M/s 6.64025M/s  39.014M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/4096         611530 ns       610985 ns         1280  232.22M/s 6.59264M/s 39.0501M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/16384       2645671 ns      2643411 ns          320 231.122M/s 6.17119M/s  36.616M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/65536      11593324 ns     11587201 ns           64 217.864M/s 5.64934M/s 33.5378M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/262144     60338069 ns     60313976 ns           16 169.444M/s 4.34607M/s 25.8032M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/256         53355 ns        53308 ns        13312 98.8029M/s 3.65798M/s 20.6535M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/1024       253979 ns       253818 ns         3072 128.001M/s  3.8571M/s 22.6619M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/4096      1052984 ns      1052427 ns          768 134.815M/s 3.82734M/s 22.6705M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/16384     4364730 ns      4362756 ns          192 140.038M/s 3.73915M/s 22.1857M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/65536    19419562 ns     19413505 ns           48 130.035M/s 3.37188M/s 20.0175M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/262144   89023213 ns     88979387 ns            8 114.856M/s 2.94595M/s 17.4905M/s
BM_CompileAPIFileDenseDecls<Phase::Check>/256        676254 ns       675605 ns         1024 7.79597M/s  288.63k/s 1.62965M/s
BM_CompileAPIFileDenseDecls<Phase::Check>/1024      1412608 ns      1411876 ns         1024 23.0112M/s 693.404k/s 4.07401M/s
BM_CompileAPIFileDenseDecls<Phase::Check>/4096      4333665 ns      4331240 ns          256 32.7581M/s 929.988k/s 5.50858M/s
BM_CompileAPIFileDenseDecls<Phase::Check>/16384    16566625 ns     16553982 ns           64 36.9065M/s 985.443k/s 5.84699M/s
BM_CompileAPIFileDenseDecls<Phase::Check>/65536    68609701 ns     68542189 ns           16 36.8304M/s 955.032k/s 5.66963M/s
BM_CompileAPIFileDenseDecls<Phase::Check>/262144  302899379 ns    302596672 ns            8 33.7739M/s 866.265k/s 5.14313M/s
```

Also note, this is the discussion that led to [me looking at bytes per
token](https://discord.com/channels/655572317891461132/655578254970716160/1295803122844700786)
2024-10-22 00:22:46 +00:00
Jon Ross-Perkins 9fefef162f Add tests to catch untested diagnostics. (#4426)
Use the diagnostic kind printing in #4425 to catch when we have
diagnostics with no tests.

This merges a couple other use-cases of filegroup manifests into a
common rule.

Note I do add a few tests for things, and also some things are
_actually_ unit tested (just not in the file_test structure). But I
stopped when I realized that dealing with merge conflicts is going to be
a pain. I might end up reverting test changes (as part of merge conflict
resolution) and doing narrow test additions in a separate PR, after both
this and #4425 are merged.
2024-10-21 20:16:18 +00:00
Jon Ross-Perkins 62c36eceda Support printing the diagnostic kind for verification. (#4425)
This is to help identify which diagnostics we're actually using.

Note that driver/testdata still has tests which don't pass this flag,
and so continue to test the kind-less (default) behavior.
2024-10-18 22:33:56 +00:00
Jon Ross-Perkins 4a73b36688 Switch back to the system llvm-symbolizer. (#4410)
Undoes most of #4347, because of [performance
complaints](https://discord.com/channels/655572317891461132/707150492370862090/1295527235133898772).
With a 30-ish frame stack trace and `-c dbg`, my installed
`llvm-symbolizer` still seems slow (~6s), but the hermetic
`llvm-symbolizer` adds ~4s (i.e., ~10s total). I don't think we can
easily force the hermetic version to build in opt configuration, so I'm
backing it out.
2024-10-16 18:32:11 +00:00
Jon Ross-Perkins 434173b016 Add skeletal format subcommand. (#4383)
This extracts out the SourceBuffer handling of `-` in order to trivially
share it.

Note this still has a number of TODOs, it's just setting up the
essential subcommand infrastructure, with some tests demonstrating that
it at least does something.
2024-10-09 17:33:29 +00:00
Jon Ross-Perkins 9d1d8d75be Replace llvm_symbolizer with a cc_env_data() array. (#4354)
This is a minor adjustment to make it easier to modify (or omit) data
where needed. Also, makes it pair better with cc_env().
2024-09-27 21:50:59 +00:00
Jon Ross-Perkins 74d6fc0b9f Update LLVM and use a hermetic llvm-symbolizer (#4347)
This only sets the symbolizer for our more used targets; not sure if
there's a great way to set it everywhere (I suppose I could try wrapping
cc_binary etc rules if there's a strong preference).

There is a downside here, symbolizing a fastbuild crash seems to take
about 3s. Not sure if there's a good way to get a faster llvm-symbolizer
execution...?

I tried running with the new LLVM update without the
LLVM_SYMBOLIZER_PATH, and it looks like that's insufficient. With the
settings, I now get readable crashes:

```
 #9 0x000055dc007d1724 void Carbon::Internal::CheckFail<Carbon::TemplateString<5>{"FATAL"}, Carbon::TemplateString<27>{"toolchain/driver/driver.cpp"}, 84, Carbon::TemplateString<0>{}, Carbon::TemplateString<3>{"err"}>() (/usr/local/google/home/jperkins/.cache/bazel/_bazel_jperkins/85deb7d9d96f7e0e80b42618a55969d7/sandbox/linux-sandbox/9383/execroot/_main/bazel-out/k8-fastbuild/bin/toolchain/testing/file_test.runfiles/_main/toolchain/testing/file_test+0x2894724)
```

Note the LLVM update is for
https://github.com/llvm/llvm-project/pull/109021
2024-09-27 20:41:55 +00:00
Jon Ross-PerkinsandChandler Carruth 4332d8239d Fix invocation issues with clang runner (#4353)
`ToolContext` should be explicitly initialized.

`-c` can still require a valid, writable `-o` path.

---------

Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
2024-09-27 20:33:11 +00:00
Jon Ross-Perkins 73c6f67378 Add support for capturing console output to FileTest. (#4339)
One of the things that ClangRunnerTest is doing is capturing
stderr/stdout because clang prints to it directly. This adds support for
that to FileTest.

I'm renaming the current `capture_output` field to `dump_output` because
the name is ambiguous after this change, and the flag is already named
`--dump_output`. It's still not great, but at least it's more distinct.

Note ClangRunner still doesn't use the vfs; that still needs work. I'm
just moving the NoArgs test over as a trivial test of the functionality.
2024-09-27 16:10:44 +00:00
Jon Ross-Perkins ee383638bc Flush pending diagnostics on crash. (#4337)
This risks diagnsotic formatting crashing, but I think we more
frequently see cases where it'd be interesting to know what diagnostics
were being delayed as part of the default sorting.
2024-09-25 21:00:36 +00:00
Jon Ross-PerkinsandChandler Carruth bba4f8aa20 Refactor install structure to make changes easier. (#4332)
Instead of separately constructing file information for `filegroup` and
`pkg_filegroup`, this instead creates a single structure which is used
to generate both. Additionally, I'm unifying the `llvm_link_data` and
`install_lib_data` targets (though the `llvm_link_data` target is
problematic for busyboxing, I don't think it can keep working as it does
right now).

Note I'm also stopping reuse of llvm's binary_alias. We need to be able
to symlink non-binary files, so I'm going to just share logic there.
(plus, I admit I find the name "binary_alias" confusing since it's not
an alias in bazel terms)

---------

Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
2024-09-24 17:19:44 +00:00
Jon Ross-PerkinsandChandler Carruth c107aaad13 Add a clang subcommand. (#4322)
This makes something like `bazel run :toolchain -- clang -- -c test.cpp`
work, because that can be run in-process. Note that `bazel run
:toolchain -- clang -- test.cpp` still requires subprocessing, and does
not work.

Note, the vision here is that we are trying to align how clang and
carbon compile c++ code. This is work towards intertwining command
execution.

---------

Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
2024-09-23 23:21:20 +00:00
Jon Ross-Perkins c7057eff89 Improve install_paths handling for relative paths. (#4331)
Because install_paths is not presently validated, and it's resolved
after the `SetWorkingDirForBazel` call, if a relative path is used with
bazel then it would fail silently. This starts making the driver share
install path errors, and starts changing how `//toolchain` launches
`carbon`.

Note the implementation is still brittle and will break with symlinks.
That's something I plan to address as part of busyboxing.
2024-09-23 22:26:53 +00:00
Jon Ross-PerkinsandRichard Smith e7aebbe581 Update basic diagnostic capitalization/punctuation (#4328)
This is a primarily automated change:

- Search & replace for capitalization
-
`(CARBON_DIAGNOSTIC\((?:\n\s+)?\w+,(?:\n\s+)?\s\w+,(?:\n\s+)?\s")([A-Z])`
    - `$1\L$2`
- Search & replace for period
-
`(CARBON_DIAGNOSTIC\((?:\n\s+)?\w+,(?:\n\s+)?\s\w+,(?:\n\s+)?\s"(?:[^)]|\n)+)\.("[,)])`
    - `$1$2`
- Limited search & replace for `ERROR: ` -> `error: ` in streamed things
- Leaving a TODO for command_line because there's more cleanup that can
be done there
- Modify diagnostic_consumer.cpp
    - ERROR -> error
    - WARNING -> warning

---------

Co-authored-by: Richard Smith <richard@metafoo.co.uk>
2024-09-19 21:32:53 +00:00
Geoff Romer 7f6d684b29 Add usage tips to NOAUTOUPDATE tests (#4324)
These tips are especially valuable in these cases, because you can't use
`autoupdate_testdata.py` to identify the output difference, so dumping
the output is pretty much the only option.
2024-09-19 21:00:21 +00:00
Jon Ross-Perkinsandjosh11b c029931910 Refactor link and compile into subcommand objects. (#4303)
Note the purpose here is to make it simpler to add more subcommands,
without adding a lot of things to Driver.

This creates a copy of CodegenOptions, but it was double-registered at
present which felt odd. It's also fairly small right now. If this
becomes an issue, maybe we can look into using optional for delayed
initialization, or just go back to straight sharing.

---------

Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com>
2024-09-14 03:16:53 +00:00
Jon Ross-PerkinsandGeoff Romer 1b956e68fe Extract subcommand options from the driver file. (#4300)
I'm separating the options out so that it's easier to review. They
include a lot of boilerplate text that I think won't change much, and
makes it harder to review changes.

To explain filename differences, whereas `CodegenOptions` is shared (by
link and compile), `LinkOptions` and `CompileOptions` are
subcommand-specific. I'm planning to separate out the subcommands, so
I'm putting those in respective subcommand files. I'm still going to try
to use the `.h` to declare the interface, `.cpp` for bigger
implementation details (for better or worse, including comments on
options).

I'm also moving out corresponding Driver members to help shrink deltas
when refactoring. That is, the bodies aren't changing here, but a
refactoring of commands will make some changes. By moving the code to
different files now, it should be easier to identify what's changing
later.

---------

Co-authored-by: Geoff Romer <gromer@google.com>
2024-09-13 16:17:51 +00:00
4845f40dff Switch CARBON_CHECK to a format string API (#4285)
This switches `DCHECK` and `FATAL` as well.

The goal is to reduce the code size impact of these assertions so that
we can keep more of them enabled. Currently, the largest cost I see from
`CHECK` is not the actual check or the cold code itself, but actually
the failure to inline trivial functions due to the presence of the cold
code. This means that our goal isn't to reduce apparent code size in the
final binary but the LLVM IR cost assessed for these routines in the
inliner, which closely correlates with code size but is a bit different.

As discussed in #4283, experimentation shows that a single function call
with a minimal number of arguments is the lowest cost model for these.
This is easily achieved with a format-string API that internally uses
`llvm::formatv`. This PR is essentially the `CHECK` version of #4283.

However, the check macros are substantially harder to make work with
both format strings and streaming because they also take a condition.
Also, unexpectedly, I was very successful at devising a regular
expression based automated rewrite from the streaming to the format
string form with only low 10s of manual fixes. This includes compacting
strings broken up across lines, etc. Given how well that went, I've
prepared this PR which just directly switches to the format string API
and migrate everything to use it.

One nice side-effect is that the format string approach ends up greatly
simplifying the implementation here as well.

This is ... *shockingly* effective. Parsing speeds up by more than 3%
with just this change. And checking speeds up by **8%** with this change
alone:
```
BM_CompileAPIFileDenseDecls<Phase::Parse>/256      86.3µs ± 1%  82.9µs ± 1%  -3.94%  (p=0.000 n=17+19)
BM_CompileAPIFileDenseDecls<Phase::Parse>/1024      431µs ± 1%   415µs ± 1%  -3.76%  (p=0.000 n=18+19)
BM_CompileAPIFileDenseDecls<Phase::Parse>/4096     1.77ms ± 1%  1.71ms ± 1%  -3.18%  (p=0.000 n=18+19)
BM_CompileAPIFileDenseDecls<Phase::Parse>/16384    7.44ms ± 1%  7.17ms ± 2%  -3.56%  (p=0.000 n=18+20)
BM_CompileAPIFileDenseDecls<Phase::Parse>/65536    30.7ms ± 1%  29.7ms ± 1%  -3.15%  (p=0.000 n=18+20)
BM_CompileAPIFileDenseDecls<Phase::Parse>/262144    131ms ± 1%   127ms ± 1%  -2.81%  (p=0.000 n=18+18)
BM_CompileAPIFileDenseDecls<Phase::Check>/256       878µs ± 2%   800µs ± 1%  -8.91%  (p=0.000 n=19+20)
BM_CompileAPIFileDenseDecls<Phase::Check>/1024     1.88ms ± 2%  1.72ms ± 1%  -8.56%  (p=0.000 n=19+20)
BM_CompileAPIFileDenseDecls<Phase::Check>/4096     5.78ms ± 2%  5.28ms ± 1%  -8.70%  (p=0.000 n=20+18)
BM_CompileAPIFileDenseDecls<Phase::Check>/16384    21.9ms ± 1%  20.1ms ± 1%  -8.02%  (p=0.000 n=18+20)
BM_CompileAPIFileDenseDecls<Phase::Check>/65536    90.4ms ± 2%  83.1ms ± 1%  -8.04%  (p=0.000 n=19+20)
BM_CompileAPIFileDenseDecls<Phase::Check>/262144    381ms ± 2%   352ms ± 1%  -7.79%  (p=0.000 n=19+19)
```

---------

Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com>
2024-09-12 16:42:08 +00:00
Jon Ross-Perkins de57c9988c Break out driver environment info into its own type. (#4299)
I want to split commands out so that we don't keep piling onto driver
(particularly as I'm eyeing clang-related commands). This extracts out
the DriverEnv so that it can be easily shared, with the CompilationUnit
as an example.

CARBON_VLOG_TO is to remove the vlog_stream_ requirement of CARBON_VLOG.
2024-09-12 00:06:31 +00:00
Chandler Carruth 0c8ab663c9 Migrate all CARBON_VLOG to the format string variant. (#4284)
This mostly uses a hilarious set of regular expressions to mechanically
switch all but two uses, and then manually fixed the last two. There
weren't too many.

Also simplifies the `vlog` implementation now that it's all going
through a format string.

This alone has a nice impact on parse and check of about 2% and 1%
respectively. The impact on lex in my timings looks like noise (no
change in instruction count, unlike the other phases).
```
name                                               old cpu/op   new cpu/op   delta
BM_CompileAPIFileDenseDecls<Phase::Lex>/256        39.1µs ± 3%  38.1µs ± 2%  -2.42%  (p=0.000 n=20+19)
BM_CompileAPIFileDenseDecls<Phase::Lex>/1024        187µs ± 3%   183µs ± 1%  -2.30%  (p=0.000 n=20+20)
BM_CompileAPIFileDenseDecls<Phase::Lex>/4096        776µs ± 4%   756µs ± 1%  -2.62%  (p=0.000 n=20+20)
BM_CompileAPIFileDenseDecls<Phase::Lex>/16384      3.36ms ± 1%  3.33ms ± 1%  -0.90%  (p=0.000 n=18+18)
BM_CompileAPIFileDenseDecls<Phase::Lex>/65536      14.4ms ± 2%  14.2ms ± 1%  -1.41%  (p=0.000 n=20+20)
BM_CompileAPIFileDenseDecls<Phase::Lex>/262144     65.7ms ± 1%  65.2ms ± 2%  -0.86%  (p=0.002 n=19+20)
BM_CompileAPIFileDenseDecls<Phase::Parse>/256      87.5µs ± 1%  86.3µs ± 1%  -1.43%  (p=0.000 n=19+20)
BM_CompileAPIFileDenseDecls<Phase::Parse>/1024      438µs ± 2%   431µs ± 1%  -1.54%  (p=0.000 n=19+20)
BM_CompileAPIFileDenseDecls<Phase::Parse>/4096     1.81ms ± 2%  1.77ms ± 1%  -2.12%  (p=0.000 n=20+20)
BM_CompileAPIFileDenseDecls<Phase::Parse>/16384    7.54ms ± 1%  7.43ms ± 1%  -1.44%  (p=0.000 n=19+20)
BM_CompileAPIFileDenseDecls<Phase::Parse>/65536    31.2ms ± 1%  30.6ms ± 1%  -2.03%  (p=0.000 n=20+20)
BM_CompileAPIFileDenseDecls<Phase::Parse>/262144    133ms ± 1%   130ms ± 1%  -1.85%  (p=0.000 n=20+20)
BM_CompileAPIFileDenseDecls<Phase::Check>/256       882µs ± 1%   878µs ± 1%  -0.52%  (p=0.001 n=17+19)
BM_CompileAPIFileDenseDecls<Phase::Check>/1024     1.90ms ± 2%  1.88ms ± 1%  -1.17%  (p=0.000 n=19+19)
BM_CompileAPIFileDenseDecls<Phase::Check>/4096     5.85ms ± 2%  5.76ms ± 1%  -1.43%  (p=0.000 n=20+19)
BM_CompileAPIFileDenseDecls<Phase::Check>/16384    22.2ms ± 2%  21.9ms ± 2%  -1.20%  (p=0.000 n=20+19)
BM_CompileAPIFileDenseDecls<Phase::Check>/65536    91.2ms ± 2%  90.3ms ± 1%  -1.00%  (p=0.000 n=20+19)
BM_CompileAPIFileDenseDecls<Phase::Check>/262144    382ms ± 1%   380ms ± 1%  -0.51%  (p=0.003 n=18+19)
```
2024-09-11 12:11:23 +00:00
Jon Ross-Perkins 6311552fcc Generate and use a manifest for prelude files. (#4291)
This removes the directory crawl because bazel doesn't remove files from
execroot when the rule generating them would no longer generate them.

Fixes #4288
2024-09-10 20:59:49 +00:00
Jon Ross-Perkins e382e6fd97 Refactor FindPreludeFiles into InstallPaths (#4268)
From the driver's perspective, `FindPreludeFiles` is closely tied to
`compile`. This makes it difficult to refactor commands without
affecting the test dependencies on `FindPreludeFiles`. `InstallPaths`
seems like a decent home since it is responsible for the install
structure.

I'm switching to an `Error` return to allow callers to choose how to
handle it (e.g., in file tests, we typically don't want the direct error
stream).
2024-09-04 22:22:48 +00:00
David BlaikieandJon Ross-Perkins c5ada29ba9 Add filename and line number to function debug info metadata (#4243)
Refactors a bunch of the SemIRDiagnosticConverter to be able to use that
from Lower to access source locations there to use in debug info.

I assume some of this is a bit jank/would need to be fixed/improved in
the future - like the context functor that's passed into ConvertLoc?
(not totally clear what that's for/what the debug info will be missing
out on in its absence, I could throw a FIXME in there if you like)

---------

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
2024-08-23 00:05:26 +00:00
David Blaikie 1e1034ee81 Enable debug info by default (#4232)
Discussed in the toolchain meeting today - we'd like to try having this
on by default and see if the cost isn't too high.

The nodebug test is a bit verbose, because it doesn't have the
`--exclude-dump-file-prefix` that test_file would usually add. Is there
a nicer way I could write this test to verify that --no-debug-info does
what it's meant to?
2024-08-22 20:27:45 +00:00
David Blaikie 5a11048c34 Remove some explicit (Mutable)ArrayRef constructions (#4238)
Rely on implicit conversion in call sites and initialization.

Removing the explicit conversions is only code simplication.
Moving from `auto x = Y(z)` to `Y x = z;` helps ensure that only
implicit constructors/conversions are happening (whereas the prior
syntax allows explicit conversions) which can help with readability
since implicit conversions are generally "less
complex"/risky/attention-requiring.
2024-08-22 19:40:54 +00:00
d57aa57215 Add rudimentry debug info metadata emission (#4225)
This adds just the debug info metadata for Compilation Units (the top
level container of debug info) - but without anything in them, LLVM
won't emit them at all, so while this is testable at the IR level, it
isn't observable at the object level until more debug info is added.

A couple of starting points in this patch:
* A flag (`--debug-info`, seems to match the naming/style of other flags
in the carbon driver, though this is different from the naming
conventions of clang/gcc) that enables debug info when lowering. Open to
other names/approaches (on by default? historically debug info's been to
large/expensive to do this, so sticking with that precedent for now).
* Enabling that flag by default in the lowering tests - I do find the
churn on golden tests a bit rough, and adding more features to all the
tests means more churn, but it seems consistent with the approach so far
- keep an eye on this and perhaps revisit this if the churn gets too
annoying

---------

Co-authored-by: Carbon Infra Bot <carbon-external-infra@google.com>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
2024-08-20 20:00:35 +00:00
Chandler Carruth 72cb9d0d06 Refactor testing exe path and benchmark main handling. (#4216)
Consolidates both main libraries into `//testing/base`, and factors out
the exe path handling for benchmarks and unit tests into a common
library to remove duplication. Refactors how that logic is managed to be
cleaner and avoid a confusing bool that came up in code review.

Updates all the tests and benchmarks that use these. I still need to
update other benchmarks to use the same main, but I wanted to keep this
PR somewhat minimal.

This also fixes a bug noticed in passing that the compilation benchmark
didn't have the required dependency on the benchmark library itself,
just the benchmark main library.
2024-08-14 17:50:08 +00:00
a9c815c9f4 Introduce a source generator and end-to-end compile benchmarks (#4124)
The big addition here is a very, very rough and very early skeleton of a
source code generator framework. This builds upon the lexers identifier
synthesis logic, improving on its framework and wiring it up with the
most rudimentary of source file generation. This is just enough to
roughly replicate my "big API file" source code benchmarks.

The source generation works *very* hard to both vary the structure and
content of the source as much as possible while ensuring the same
*total* amount of each construct is in use, from bytes in identifiers to
line breaks, parameters, etc. This lets us generate randomly structure
inputs that should consistently take the exact same amount of total work
to compile.

The complex identifier synthesis logic from the lexer's benchmark is
moved over here and the lexer uses APIs in the source generator for
identifiers. The other source synthesis in the lexer's benchmark isn't
yet moved over, but should likely be slowly absorbed here as it can be
refactored into a more principled and re-usable form. Some bits may stay
of course if they're just too lexer-specific.

Next, this adds a simple end-to-end compile benchmark for the driver
that directly and much more clearly reproduces all the measurements I've
done manually up until now. It should also be easy to extend to more
patterns over time as we add support to the source generator to produce
those patterns.

Last but not least, I've added a tiny CLI to the source generator so
that you can generate source code manually. This is especially nice for
generating demo source code to actually run through the driver or look
at in an editor. The CLI can also generate C++ source code which lets us
do some minimal comparative benchmarking between Carbon and C++/Clang.

There are huge number of TODOs in the source generation framework. This
is going to be a large ongoing effort I suspect.

There are also a bunch of rough edges I've left to try and get this out
for review sooner. I've left TODOs for refactorings that really need to
be done here, but hoping these can maybe be follow-ups. If not, please
flag and I'll try to layer them on here.

Sample compile benchmark output, nicely showing where we are w.r.t. our
goal speeds (2x behind on lex and check, 5x on parse) at least on a
recent AMD server CPU:
```
------------------------------------------------------------------------------------------------------
Benchmark                                                 Time             CPU   Iterations      Lines
------------------------------------------------------------------------------------------------------
BM_CompileAPIFileDenseDecls<Phase::Lex>/256           29420 ns        29419 ns        22860 6.62847M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/1024         146130 ns       146128 ns         4840 6.69959M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/4096         601584 ns       601577 ns         1020 6.69573M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/16384       2547578 ns      2547313 ns          280   6.404M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/65536      10816591 ns     10816389 ns           80 6.05193M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/262144     52191320 ns     52189828 ns           20 5.02261M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/256        101706 ns       101698 ns         6900 1.91745M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/1024       512161 ns       512162 ns         1380  1.9115M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/4096      2078426 ns      2078430 ns          340   1.938M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/16384     8795786 ns      8795583 ns          100 1.85468M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/65536    35073596 ns     35072973 ns           20 1.86639M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/262144  151100688 ns    151097370 ns           20 1.73483M/s
BM_CompileAPIFileDenseDecls<Phase::Check>/256        957059 ns       957049 ns          740 203.751k/s
BM_CompileAPIFileDenseDecls<Phase::Check>/1024      1956134 ns      1955985 ns          360 500.515k/s
BM_CompileAPIFileDenseDecls<Phase::Check>/4096      5797864 ns      5797417 ns          120 694.792k/s
BM_CompileAPIFileDenseDecls<Phase::Check>/16384    21219608 ns     21217584 ns           40 768.843k/s
BM_CompileAPIFileDenseDecls<Phase::Check>/65536    96311116 ns     96302334 ns           20 679.734k/s
BM_CompileAPIFileDenseDecls<Phase::Check>/262144  371637963 ns    371609964 ns           20 705.387k/s
```

Lest someone think this is *bad*, the fact that we're already within 2x
of our rather audacious goals makes me quite happy. =D

---------

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
2024-08-13 19:37:55 +00:00
Jon Ross-Perkins f67791cfee Separate subtree size information from parse nodes. (#4174)
Move subtree sizes over to TreeAndSubtrees, using the different
structure to represent the additional parse work that occurs, as well as
making it clear which functions require the extra information. My intent
is to make it hard to use this by accident.

The subtree size is still tracked during Parse::Tree construction. I
think a lot of that can be cleaned up, although we use it during
placeholder assignment so it may take some work. I wanted to see what
people thought about this before taking action on such a change.

I'm using a 1m line source file generated by #4124 for testing. Command
is `time bazel-bin/toolchain/install/prefix_root/bin/carbon compile
--phase=check --dump-mem-usage ~/tmp/data.carbon`

At head, what I'm seeing is:

```
...
parse_tree_.node_impls_:
  used_bytes:      61516116
  reserved_bytes:  61516116
...
Total:
  used_bytes:      447814230
  reserved_bytes:  551663894
...
1.43s user 0.14s system 99% cpu 1.565 total
```

With `Tree::Verify` disabled completely, it looks like:
```
parse_tree_.node_impls_:
  used_bytes:      41010744
  reserved_bytes:  41010744
...
Total:
  used_bytes:      427308858
  reserved_bytes:  531158522
...
1.20s user 0.13s system 99% cpu 1.332 total
```

Re-enabling just the basic verification (what is now `Tree::Verify`),
I'm seeing maybe 0.05s slower, but that's within noise for my system. I
do see variability in my timing results, and overall I think this is a
0.2s +/- 0.1s improvement versus the earlier (always testing `Extract`
code) implementation. That's opt; debug builds will be unaffected,
because the same checking occurs as before.

Note, the subtree size is a third of the node representation, which is
why I'm showing the decrease in memory usage here.
2024-07-31 19:39:45 +00:00
Jon Ross-Perkins 65d6e3e221 Use verbose formatting of instructions on crash messages. (#4125)
Changes crash messages to start printing verbose forms of instructions,
rather than just the ID. Fixes some indentation issues with stacks. Also
switches unexpected inst formatting, because now there are lots, and
it'd be helpful to know where they are.

This uses a pimpl pattern for Formatter due to the number of member
functions on Formatter. Maybe we should refactor that, but this didn't
feel like a good place to do so.

Note, I have two concerns about this change... to note them here, to
make sure others are considering them when evaluating the
implementation:

1. Some instructions are very verbose to print, as evidenced by the
fn_decl printing (which includes function params) or scope printing
(which includes scope members).
- I'm not sure whether there's a way to simply reduce this, as it seems
essential to the requested printing of instructions.
- Long-term, we may at least want to limit the number of lines printed
here. However, I've already spent a fair amount of time here and I think
it's in a good state to evaluate.
2. Increased complexity in the crash handler may result in crash
messages failing to generate.
- For example, a crash in Formatter (and its deps, such as InstNamer or
location handling) prevents a stack from being printed. I'm pretty sure
I've written crashes in Formatter before.

Here's an example crash snippet (generated by adding a crash inside
`return` handling) before:

```
2.	NodeStack:
	0.	FunctionDefinitionStart -> function2
	1.	ReturnStatementStart -> no value
	2.	IntLiteral -> inst+26
inst_block_stack_:
	0.	block<invalid>	{inst+0, inst+1, inst+2, inst+23}
	1.	block9	{inst+26}
param_and_arg_refs_stack:
args_type_info_stack_:
```

And after:

```
2.	Check::Context
          NodeStack:
            0. FunctionDefinitionStart: function2
            1. ReturnStatementStart: no value
            2. IntLiteral:
              unexpected.inst+26.loc12_10: i32 = int_literal 0 [template = constants.%.2]
          inst_block_stack_:
            0. block<invalid> {
                package: <namespace> = namespace [template] {
                  .Core = unexpected.inst+2
                  .F = unexpected.inst+23.loc11_22
                }
                unexpected.inst+1 = import Core
                unexpected.inst+2: <namespace> = namespace unexpected.inst+1, [template] {}
                unexpected.inst+23.loc11_22: %F.type = fn_decl @F [template = constants.%F] {
                  unexpected.inst+9.loc11_9: init type = call constants.%Bool() [template = bool]
                  unexpected.inst+10.loc11_9: type = value_of_initializer unexpected.inst+9.loc11_9 [template = bool]
                  unexpected.inst+11.loc11_9: type = converted unexpected.inst+9.loc11_9, unexpected.inst+10.loc11_9 [template = bool]
                  unexpected.inst+12.loc11_6: bool = param b
                  @F.%b: bool = bind_name b, unexpected.inst+12.loc11_6
                  unexpected.inst+19.loc11_18: init type = call constants.%Int32() [template = i32]
                  unexpected.inst+20.loc11_18: type = value_of_initializer unexpected.inst+19.loc11_18 [template = i32]
                  unexpected.inst+21.loc11_18: type = converted unexpected.inst+19.loc11_18, unexpected.inst+20.loc11_18 [template = i32]
                  @F.%return: ref i32 = var <return slot>
                }
              }
            1. block9 {
                unexpected.inst+26.loc12_10: i32 = int_literal 0 [template = constants.%.2]
              }
          param_and_arg_refs_stack:
          args_type_info_stack_:
```
2024-07-17 22:05:19 +00:00
Jon Ross-Perkinsandjosh11b f1190a4792 Add basic output of where memory is stored after a compile. (#4136)
The output is really basic, I'm just adding this to help track how
memory is allocated.

```
---
filename:        'check/testdata/expr_category/in_place_tuple_init.carbon'
source_:
  used_bytes:      8057
  reserved_bytes:  8057
tokens_.allocator_:
  used_bytes:      0
  reserved_bytes:  0
tokens_.token_infos_:
  used_bytes:      1040
  reserved_bytes:  2032

(eliding)

value_stores_.string_literals_.set_:
  used_bytes:      320
  reserved_bytes:  320
Total:
  used_bytes:      20609
  reserved_bytes:  29437
...
```

---------

Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com>
2024-07-16 23:11:01 +00:00
Jon Ross-Perkins 006e31238e Adjust driver references to use the install version. (#4126)
Also makes an alias so that this is easier to find. Verified that
running the alias still finds prelude files.

Note the actual target is printed when building (although a symlink is
also created, using that symlink confuses file-finding).

```
╚╡bazel build :carbon
...
Target //toolchain/install:prefix_root/bin/carbon up-to-date:
  bazel-bin/toolchain/install/prefix_root/bin/carbon
```
2024-07-12 22:06:55 +00:00
Chandler Carruth e71e6ca07f Use separate value stores for identifiers and string literals (#4106)
This undoes a previous change to unify them, and I think at my advice.
=[ Sorry about that, I think I was just wrong.

Specifically, I think I had suggested that it would be more efficient to
have a single shared hashtable of strings. The more I look at profiles
of the toolchain, the less likely that seems. Specifically for
identifiers and string literals it seems especially problematic.

Using a single, joint hashtable is likely a good idea when all of the
different querying code paths are equally likely, the strings follow the
same distribution of sizes, and either there is no clustering of access
to different sets of strings or none of the sets are meaningfully small
enough to fit into a lower level of resident cache.

I think essentially none of these predicates actually hold for
identifiers vs. string literals:
- Identifiers are *much* more hot
- They have wildly different size distributions.
- The access patterns are very clustered

Sorry for the misleading advice on that one.

While splitting them, I've worked to simplify the code a bit by building
a way to have the `StringRef` holding canonical value stores not require
specializations, and so we get a pretty large code cleanup in the
process here.
2024-07-03 15:54:04 +00:00
Chandler Carruth 7b0ae725fa Fix linking to not look for a system libstdc++. (#4060)
Not sure what changed (I think an upstream LLVM change, but maybe a
Linux distro change), but several folks have been running into problems
finding a standard C++ library when running the Carbon link step. It was
actually breaking our example build when LLD didn't find the right
libstdc++ install, but it finds one reliably on our build bots and
sometimes for some of the developers.

For now, just remove the C++ standard library from the link. We're not
doing that level of interop, and the plan is really to do that not with
the system standard C++ library but by building and bundling libc++ with
the installed toolchain.

Left a comment explaining what's going on here.
2024-06-18 14:56:56 +00:00
Chandler CarruthandJon Ross-Perkins b51dc7f8e2 Introduce version and build info stamping. (#4054)
This adds a defined Carbon version to the Bazel build and codebase that
can be used both to implement features like version checks and to report
a meaningful version on the command line. This replaces a hard-coded
string and a TODO in the driver.

As part of this, it adds support for defining the version in Bazel, and
special build flags for overriding relevant parts such as the
pre-release marker used. The exact structure and meaning of our version
string, including the pre-release parts, is implemented here in line
with the draft proposal:

https://docs.google.com/document/d/11S5VAPe5Pm_BZPlajWrqDDVr9qc7-7tS2VshqO0wWkk/edit?resourcekey=0-2YFC9Uvl4puuDnWlr2MmYw

This also introduces a workspace status command to the repository to
extract the git commit SHA and other information when building, and the
logic to stamp that into binaries as part of the version string when
useful. The technique used leverages weak symbols with whole archive
linking to allow a link-time override of unstamped data with stamped
data in the leaf executable. This makes building with `--stamp` a
reasonable default, especially for development builds. The CI system is
explicitly opted out of this as there it has no benefit.

Last but not least, all of these are wired into the install rules so
that we build installable packages with the version number in a
conventional place in the directory and filename.

---------

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
2024-06-18 00:33:05 +00:00
Chandler Carruth 268dd04511 Teach fuzzers to initialize their installs from runfiles. (#4030)
This also allows us to default construct installs in an error state,
which is useful for cases like fuzzers where we want to default
construct a global, but then re-initilaize it That said, happy to
consider alternative designs here.
2024-06-05 17:04:17 +00:00
Chandler Carruth 8c64f0bfdd Add -Wmissing-prototypes and fix issues it finds. (#4019)
Most of these are places where we failed to include a header file and
simply never got an error about this. The fix is to include the header
file.

Most other cases are functions that should have been marked `static` but
were not. Finding all of these was a main motivation for me enabling the
warning despite how much work it is.

One complicating factor was that we weren't including the `handle.h` for
all the state-based handler functions. While this isn't a tiny amount of
code, it is just declarations and doesn't add any extra dependencies. It
also lets us have the checking for which functions need to be `static`
and which don't. For the `parse` library I had to add the `handle.h`
header as well, I tried to match the design of it in `check`.

I have also had to work around a bug in the warning, but given the value
it seems to be providing, that seems reasonable. I've filed the bug
upstream: https://github.com/llvm/llvm-project/issues/94138

I also had to use some hacks to work around limitations of Bazel rules
that wrap `cc_library` rules and don't expose `copts`. I filed a bug for
`cc_proto_library` specifically:
~https://github.com/bazelbuild/bazel/issues/22610~ 
https://github.com/bazelbuild/bazel/issues/4446
2024-06-04 20:04:45 +00:00
Jon Ross-Perkins 83413479d7 Move some of the test information to TIP lines (#4007)
This has is a nice-to-have for me. Frequently I want to run a specific
test, and end up digging through output to be able to copy-paste the run
line. This uses TIP lines to inject the command into the file when using
AUTOUPDATE.

Note, one of the reasons I want this is because "bazel test
//toolchain/testing:file_test --test_output=all" has been regularly
exceeding bazel's output limit for me (workaround is either opening the
output file or specifying an obscure output limit flag), making it a
little harder for me to get the commands. However, frequently I'm adding
a file and want to iterate on it, so that's really the use case I have
in mind here.
2024-06-04 17:57:23 +00:00
Chandler Carruth e34e240263 Move the core standard library to the installation. (#3995)
This removes the `data_dir` from the driver favoring the installation
abstraction for the both locating the prelude and linking utilities.

With this, an installed toolchain should also be able to compile and
link Carbon successfully, and the build of the examples should exercise
this path almost exactly. (The only difference is using the driver
`cc_binary` directly rather than relying on the symlink from inside the
install tree.)
2024-06-01 05:16:44 +00:00
Chandler CarruthandJon Ross-Perkins d3a5b0eee7 Add utilities for managing a toolchain install, and install and use LLD. (#3993)
The install directory contains the BUILD logic for creating an
installable tree of data files and executables for the toolchain, and
a library to facilitate toolchain code accessing the paths to their data
within this installation.

Then adds an installation of LLD in a synthetic LLVM installation, and
teaches the Clang runner to configure this and use it for linking
instead of the system linker.

Currently, the install paths only really manage access to the LLVM
binaries installed and used by the Clang runner for linking, but
eventually other data files like the prelude and runtime libraries will
be fleshed out as well. There are TODOs for moving more things over here
such as the prelude.

One interesting aspect of this is where to put helpers like parts of
LLVM in our install. This PR suggests nesting those files under
`lib/carbon`. While using a `lib` subdirectory isn't a perfect fit for
the FHS (Filesystem Hierarchy Standard), having a single location where
private data is collected is significantly superior to spreading them
across the system. This also matches similar patterns used by Clang
itself and several other language toolchains and standard libraries.

The install directory also provides a natural place for us to build out
packaging rules to create installable packages in various formats, but
that remains future work.

---------

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
2024-06-01 03:19:52 +00:00
Jon Ross-Perkins 16942e4d07 Move errs tie to LLVM to include test infrastructure. (#4006)
The lack of flushing can sometimes be observed when streaming test
output, particularly with --dump_output. My thought is moving it into
our LLVM init is probably reasonable, given the driver's already doing
this for similar reasons; this means it's used in tests through our
gtest_main.cpp in addition to the driver.
2024-05-30 21:08:44 +00:00
Chandler Carruth 582b2c04dd Allow GoogleTest tests to locate their executable path. (#3992)
This also factors out the code for doing this location from the driver
to a tiny helper library.

The motivation for this is letting tests locate data files like the
prelude or other files needed by the toolchain more easily. A subsequent
PR will use this heavily in the Clang runner and related logic for
example.
2024-05-28 23:50:57 +00:00
Richard Smith 5c8fa6ad5c Replace FoldingSet with DenseMap for instruction canonicalization. (#3979)
Switch from recursing into non-canonical instruction fields to
separately canonicalizing those fields. This means we now form canonical
`InstBlockId`s, `TypeBlockId`s, `IntId`s, `FloatId`s, and `BindNameId`s
at least in the cases when they're referenced by a constant instruction.

This reduces the overall runtime for @chandlerc's 10MLoC example by
27.5% on my machine.
2024-05-23 00:48:49 +00:00
Chandler Carruth 36d8d2960a Introduce basic linking with the Clang driver. (#3922)
This adds the most rudimentary support for linking in the Carbon driver
by calling out to the Clang driver and letting it do the linking. This
is a super minimal version to start with, just enough to be somewhat
useful and to exercise calling into Clang for this.

Also adds a Clang runner to help run the Clang driver. This is a bit
more complete, although it sill needs some significant work. For
example, it will need some significant enhancements to be able to
compile C++ code, etc.

One thing that will likely change here is to better manage streams and
work better as a library. For now, this is a bit limited because Clang
and LLVM don't provide the necessary support for this.
2024-04-30 18:01:39 +00:00