As part of migrating to the latest bazel configurations in #4729, I'm
running into proto toolchain issues. For example:
"Error: <target @@protobuf+//:cc_toolchain> (rule
'proto_lang_toolchain') doesn't contain declared provider
'ProtoLangToolchainInfo'"
Although we may eventually want more use of proto, right now the only
use is for the explorer fuzzer. The explorer codebase is essentially
frozen, so continuing to run it isn't gaining us much (in fact, we've
already disabled autofuzzing for it).
So, rather than trying to fix the proto setup, this change:
1. Deletes `explorer/fuzzing`
2. Removes proto portions of `testing/fuzzing`, which were only in-use
by the explorer
3. Removes some ancillary proto support, which would otherwise break
from the bazel changes and would be difficult to validate as "still
working"
This change is partly isolated in order to make it easier to revive bits
of (3).
As discussed in #4530 . This required switching to using
`llvm::IntrusiveRefCntPtr` in a number of places.
---------
Co-authored-by: Josh L <josh11b@users.noreply.github.com>
Undoes most of #4347, because of [performance
complaints](https://discord.com/channels/655572317891461132/707150492370862090/1295527235133898772).
With a 30-ish frame stack trace and `-c dbg`, my installed
`llvm-symbolizer` still seems slow (~6s), but the hermetic
`llvm-symbolizer` adds ~4s (i.e., ~10s total). I don't think we can
easily force the hermetic version to build in opt configuration, so I'm
backing it out.
This only sets the symbolizer for our more used targets; not sure if
there's a great way to set it everywhere (I suppose I could try wrapping
cc_binary etc rules if there's a strong preference).
There is a downside here, symbolizing a fastbuild crash seems to take
about 3s. Not sure if there's a good way to get a faster llvm-symbolizer
execution...?
I tried running with the new LLVM update without the
LLVM_SYMBOLIZER_PATH, and it looks like that's insufficient. With the
settings, I now get readable crashes:
```
#9 0x000055dc007d1724 void Carbon::Internal::CheckFail<Carbon::TemplateString<5>{"FATAL"}, Carbon::TemplateString<27>{"toolchain/driver/driver.cpp"}, 84, Carbon::TemplateString<0>{}, Carbon::TemplateString<3>{"err"}>() (/usr/local/google/home/jperkins/.cache/bazel/_bazel_jperkins/85deb7d9d96f7e0e80b42618a55969d7/sandbox/linux-sandbox/9383/execroot/_main/bazel-out/k8-fastbuild/bin/toolchain/testing/file_test.runfiles/_main/toolchain/testing/file_test+0x2894724)
```
Note the LLVM update is for
https://github.com/llvm/llvm-project/pull/109021
One of the things that ClangRunnerTest is doing is capturing
stderr/stdout because clang prints to it directly. This adds support for
that to FileTest.
I'm renaming the current `capture_output` field to `dump_output` because
the name is ambiguous after this change, and the flag is already named
`--dump_output`. It's still not great, but at least it's more distinct.
Note ClangRunner still doesn't use the vfs; that still needs work. I'm
just moving the NoArgs test over as a trivial test of the functionality.
This switches `DCHECK` and `FATAL` as well.
The goal is to reduce the code size impact of these assertions so that
we can keep more of them enabled. Currently, the largest cost I see from
`CHECK` is not the actual check or the cold code itself, but actually
the failure to inline trivial functions due to the presence of the cold
code. This means that our goal isn't to reduce apparent code size in the
final binary but the LLVM IR cost assessed for these routines in the
inliner, which closely correlates with code size but is a bit different.
As discussed in #4283, experimentation shows that a single function call
with a minimal number of arguments is the lowest cost model for these.
This is easily achieved with a format-string API that internally uses
`llvm::formatv`. This PR is essentially the `CHECK` version of #4283.
However, the check macros are substantially harder to make work with
both format strings and streaming because they also take a condition.
Also, unexpectedly, I was very successful at devising a regular
expression based automated rewrite from the streaming to the format
string form with only low 10s of manual fixes. This includes compacting
strings broken up across lines, etc. Given how well that went, I've
prepared this PR which just directly switches to the format string API
and migrate everything to use it.
One nice side-effect is that the format string approach ends up greatly
simplifying the implementation here as well.
This is ... *shockingly* effective. Parsing speeds up by more than 3%
with just this change. And checking speeds up by **8%** with this change
alone:
```
BM_CompileAPIFileDenseDecls<Phase::Parse>/256 86.3µs ± 1% 82.9µs ± 1% -3.94% (p=0.000 n=17+19)
BM_CompileAPIFileDenseDecls<Phase::Parse>/1024 431µs ± 1% 415µs ± 1% -3.76% (p=0.000 n=18+19)
BM_CompileAPIFileDenseDecls<Phase::Parse>/4096 1.77ms ± 1% 1.71ms ± 1% -3.18% (p=0.000 n=18+19)
BM_CompileAPIFileDenseDecls<Phase::Parse>/16384 7.44ms ± 1% 7.17ms ± 2% -3.56% (p=0.000 n=18+20)
BM_CompileAPIFileDenseDecls<Phase::Parse>/65536 30.7ms ± 1% 29.7ms ± 1% -3.15% (p=0.000 n=18+20)
BM_CompileAPIFileDenseDecls<Phase::Parse>/262144 131ms ± 1% 127ms ± 1% -2.81% (p=0.000 n=18+18)
BM_CompileAPIFileDenseDecls<Phase::Check>/256 878µs ± 2% 800µs ± 1% -8.91% (p=0.000 n=19+20)
BM_CompileAPIFileDenseDecls<Phase::Check>/1024 1.88ms ± 2% 1.72ms ± 1% -8.56% (p=0.000 n=19+20)
BM_CompileAPIFileDenseDecls<Phase::Check>/4096 5.78ms ± 2% 5.28ms ± 1% -8.70% (p=0.000 n=20+18)
BM_CompileAPIFileDenseDecls<Phase::Check>/16384 21.9ms ± 1% 20.1ms ± 1% -8.02% (p=0.000 n=18+20)
BM_CompileAPIFileDenseDecls<Phase::Check>/65536 90.4ms ± 2% 83.1ms ± 1% -8.04% (p=0.000 n=19+20)
BM_CompileAPIFileDenseDecls<Phase::Check>/262144 381ms ± 2% 352ms ± 1% -7.79% (p=0.000 n=19+19)
```
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com>
Demo site: https://jonmeow.carbon-lang.dev/
I'm trying to keep work under the `/website` subdirectory so that the
misc files don't interfere with unrelated views of the repository. The
`prebuild.py` script does some work to move things around and add
frontmatter, helping the jekyll generation.
I'm using the "just-the-docs" theme because I think it's a decent match
for what we want, and getting jekyll up and running with it wasn't too
difficult. Note #1526 proposed using Docusaurus; I started out there,
but was having trouble getting it working with newer versions. The
plugins in particular I got stuck trying to make work, which sent me
looking for options that we could have working with less customization.
I do lean towards jekyll though, because it's what GH uses so hopefully
we can get a more consistent experience.
Having a website has been approved for a while under #1492, but hasn't
been a priority. I'm mainly doing this because I want to just be able to
point people to carbon-lang.dev and have easy links that way.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
With bazel 7.2.0, there are some dependency changes, resulting in:
```
WARNING: For repository 'bazel_skylib', the root module requires module version bazel_skylib@1.5.0, but got bazel_skylib@1.6.1 in the resolved dependency graph.
WARNING: For repository 'platforms', the root module requires module version platforms@0.0.8, but got platforms@0.0.9 in the resolved dependency graph.
```
This goes around doing some updates... protobuf's include structure has
changed a little too.
Most of these are places where we failed to include a header file and
simply never got an error about this. The fix is to include the header
file.
Most other cases are functions that should have been marked `static` but
were not. Finding all of these was a main motivation for me enabling the
warning despite how much work it is.
One complicating factor was that we weren't including the `handle.h` for
all the state-based handler functions. While this isn't a tiny amount of
code, it is just declarations and doesn't add any extra dependencies. It
also lets us have the checking for which functions need to be `static`
and which don't. For the `parse` library I had to add the `handle.h`
header as well, I tried to match the design of it in `check`.
I have also had to work around a bug in the warning, but given the value
it seems to be providing, that seems reasonable. I've filed the bug
upstream: https://github.com/llvm/llvm-project/issues/94138
I also had to use some hacks to work around limitations of Bazel rules
that wrap `cc_library` rules and don't expose `copts`. I filed a bug for
`cc_proto_library` specifically:
~https://github.com/bazelbuild/bazel/issues/22610~https://github.com/bazelbuild/bazel/issues/4446
Just replacing AUTOUPDATE with NOAUTOUPDATE, and removing the autoupdate
script. Tests will still run, but autoupdate may need to be fixed if
significant changes are made.
I noticed this while trying to autoupdate for #4007 (because we verify
that tests have been autoupdated). Autoupdate was likely broken by
#3449.
Trying to run with no changes gives:
```
CHECK failure at testing/file_test/file_test_base.cpp:801: !absl::GetFlag(FLAGS_test_targets_file).empty(): Missing --test_targets_file.
```
This is because the `file_test` rule creates a file with test inputs
that it runs with, which the prebuilt binary doesn't provide.
A local kludge to create a `file_test` target not using prebuilt_binary
showed another error:
```
: CommandLine Error: Option ': CommandLine Error: Option 'parser_debug' registered more than once!
LLVM ERROR: inconsistency in registered CommandLine options
trace_phase' registered more than once!
LLVM ERROR: inconsistency in registered CommandLine options
```
I'm thinking here that the explorer code hasn't been in enough use, and
we're seeing some rot as a consequence. Rather than trying to maintain
it, I'm suggesting to go with NOAUTOUPDATE.
Just spotted these while looking at warnings that seem to fire on our
code are probably are things we'd fix if we saw them. None of these seem
important FWIW.
Also removes a redundant flag that is part of `-Wall`.
I have a follow-up for the high-value warning I spotted that motivated
me to look at all of this. But it's noisy so kept it as a separate PR.
The install directory contains the BUILD logic for creating an
installable tree of data files and executables for the toolchain, and
a library to facilitate toolchain code accessing the paths to their data
within this installation.
Then adds an installation of LLD in a synthetic LLVM installation, and
teaches the Clang runner to configure this and use it for linking
instead of the system linker.
Currently, the install paths only really manage access to the LLVM
binaries installed and used by the Clang runner for linking, but
eventually other data files like the prelude and runtime libraries will
be fleshed out as well. There are TODOs for moving more things over here
such as the prelude.
One interesting aspect of this is where to put helpers like parts of
LLVM in our install. This PR suggests nesting those files under
`lib/carbon`. While using a `lib` subdirectory isn't a perfect fit for
the FHS (Filesystem Hierarchy Standard), having a single location where
private data is collected is significantly superior to spreading them
across the system. This also matches similar patterns used by Clang
itself and several other language toolchains and standard libraries.
The install directory also provides a natural place for us to build out
packaging rules to create installable packages in various formats, but
that remains future work.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
I'd kept these in to separate the bazel module update from the BUILD
file changes, then forgot about it. I think all of these can be cleanly
removed now. I think it's something we should clean up for consistency
with the bazel central repository names; I think it's best to reduce
that divergence.
llvm_zlib and llvm_zstd remain because of how llvm depends on the
particular names.
This handles toolchain failures per-file. The intent is to allow placing
both "success" and "fail" tests in the same file, using splits. However,
this PR only adds support and updates existing tests to continue
passing.
Recent runs of `clang-tidy` for me started showing more errors, and this
is a collection of changes to address them.
First, I've systematically applied the disabling tag to all C++ rules
under //explorer/... with `buildozer` so we don't spend time analyzing
this code or reporting errors from it. Not sure this was strictly
necessary, but it seemed like a nice consistency improvement.
Next, I disabled a buggy check for missing `default` cases in
`switch`es. It seems to get confused by the fancy conversions in our
`enum_base.h`. We don't miss much with this as the Clang compiler
warnings for `switch` catch most of our actual bugs. I also removed the
local disabling of this now that it is turned off centrally.
Lastly, I added error checking to two file descriptor manipulating calls
in the `file_test` infrastructure.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
These changes are aimed at improving code readability and
maintainability within the `StaticScope` class of the AST folder.
- **Optimized Print and PrintID Methods:**
Introduced a template function `PrintCommon` to handle common logic in
`Print` and `PrintID`. This change simplifies the class interface and
avoids repetition, enhancing code readability.
- **Simplified TryResolveHere Method:**
Streamlined the `TryResolveHere` method by simplifying the conditional
logic. The refactored code is more readable and easier to understand,
improving overall code quality.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This patches bazel_clang_tidy handling of headers. I found an equivalent
change at https://github.com/erenon/bazel_clang_tidy/pull/13, but that
was [already
rejected](https://github.com/erenon/bazel_clang_tidy/pull/13#issuecomment-1047007424).
Per the criticism, this will result in redundant processing of headers.
The project instead uses `HeaderFilterRegex: ".*"`, but that results in
two problems:
1. When running with `-k`, errors are repeated when a header is included
more than once, which is common.
2. clang-tidy including errors from headers that are included from other
modules (e.g., abseil-cpp); filtering correctly is difficult.
Given the trade-offs and options (including forking), I thought patching
was preferable so long as it remains narrow.
This sets things up to use `bazel` to run `clang-tidy` using
https://github.com/erenon/bazel_clang_tidy.
I'm fixing issues outside of explorer, and disabling clang-tidy for
targets in explorer that have legacy issues. I was going to disable
clang-tidy for targets in explorer such as interpreter anyways, because
they're slow to parse, and just extended that to the currently failing
targets.
The explorer encoded the old runfiles layout in a string in one place
that didn't get updated. This causes our instructions for running the
explorer to break. The fix is at least quite simple.
Fixes#3540Fixes#3541
Some background information is at https://bazel.build/external/migration
Trying to handle the simple cases first. This adds a requirement for
bazel 7 due to differences in bzlmod handling between 6 and 7 (also
discussed on
[#infra](https://discord.com/channels/655572317891461132/707150492370862090/1184942191412510720)).
Bazel seems to be okay with a partial migration such as ths.
The python import behavior has subtly shifted, so `carbon.` is no longer
part of import paths. There's a version-incompatible change for `@@`.
bzlmod makes repos sometimes show as `name~version`.
`target-determinator` seems to be okay with `@@` after a version update.
Things not moved here are things that basically need more dep work:
- clang_register_toolchains because I need to dive into its format.
- llvm-project because we need something slightly atypical, I need to
make sure patching and the repo work carries over.
- com_google_libprotobuf_mutator is sufficiently atypical that it
doesn't have a module already, but should be one of the easier things to
fix.
- brotli/woff2: I think we should actually consider removing these. But
again, they're not trivial moves.
- treesitter due to toolchain registration, which has shifted a bit.
- rules_nodejs because treesitter depends on it in an awkward way to
migrate.
Per their documentation, the `llvm::Initialize*` functions are only
supposed to be called by the main program, not by a library like
toolchain/codegen. Fixes a hang due to a data race in multithreaded
autoupdate.
Add a utility class `Carbon::InitLLVM` to do the common LLVM
initialization shared by all Carbon tools, optionally including
initializing the LLVM targets. Because the LLVM targets add a lot of
binary size, only initialize them for binaries that opt in by depending
on a new target `//common:all_llvm_targets`.
Also fix `//explorer:file_test` and `//explorer:file_test.trace` to
share a binary rather than linking an identical binary twice.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
The version of `flake8` was too old to support with Python 3.12 -- there
is new F-string support that caused false positives sadly. The updated
version has fixes for all of these.
This in turn updates codespell which has picked up several new fixes
that actually fire in our code, so also fix everything. While we don't
do more in-depth updates to old proposals, similar to simply fixing
broken links, fixing automatically detected typos seems scalable and
fine.
All edits were automatically generated here.
As a follow up to #2825 this pr implements size deduction for nested
arrays.
E.g. `var x: [[i32;];] = ((1,2), (3,4));`.
Also updated the pattern matching logic for arrays, now it also checks
element types of the tuple size being deduced from. As a result code
like `var x: [i32;] = ("foo", "bar");`(note, that it tries to init an
array of i32 with a tuple of strings) fails to compile with a pattern
match error instead of an implicit cast failed error.
Moved some common type-related logic used in type_checker.cpp and
interpreter.cpp to the separate file.
Add validation to `CheckAddrSelfAccess` to additionally check for
situations where `addr` is potentially missing. I also updated the name
of the function since `me` was renamed to `self`.
Closes#3367
Running `bazel test //...` reported:
```
Test execution time outside of range for MODERATE tests.
Consider setting timeout="short" or size="small".
```
This change adds size="small" to avoid such warnings being reported.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Minor changes to the blaze command line executed by our autoupdate
scripts:
- Don't change the convenience symlinks. Running autoupdate shouldn't
cause `./bazel-bin/...` to switch to running a different binary.
- Don't produce so much spam. Bazel will still log its build progress if
necessary, and still report compile and runtime errors, but won't
produce half a dozen lines of INFO at the start of the command.
Rationale: this convention avoids forcing closely-related code to be far
apart in the namespace hierarchy, and vice versa. By the same token, it
makes the namespace hierarchy more consistent with the directory
hierarchy.
Lex now prints its yaml as:
```
- filename: name
tokens: [ ... ]
```
New support in file_test allows the `filename` marker at the top to
define the default file number for later lines, meaning multi-file
output from lexing is now associated with the appropriate file. Similar
support will probably also apply to lowering, semir, and other places
that print a filename once for the full dump.
This hammers a bit at how line number replacements work in file_test,
allowing stacking them so that lex errors and stdout can both be
line-associated properly. I've tried to make the autoupdate more
frequently work in one pass, now also taking into account the file index
when doing line replacements.
There are still some issues with EndOfFile that it may be good to
discuss: because CHECK lines are appended to the end of the file now,
and the EndOfFile token points at the last line including comments, new
lex tests now take two runs to autoupdate (because without CHECK lines,
the EndOfFile points at a content line, which content is then inserted
after). Note that removing CHECK lines from the test is not a solution:
autoupdate also started inserting blank lines, which breaks this for a
similar reason. One solution here might be to not have EndOfFile
associate with a line or column, which has been a bit of an issue
regardless.
Also fixes a small issue with toolchain's autoupdate script.
When printing callable declarations, it does not print the self pattern
in the deduced bindings.
Example:
User code:
```
class A {
fn Fun[self: Self]() {}
}
```
Print output is missing the self pattern
```
class A {
fn Fun ()
{
}
}
```
This PR fixes it and includes the self pattern in the print output.
```
class A {
fn Fun [self: Self]()
{
}
}
```
---------
Co-authored-by: Geoff Romer <gromer@google.com>
Specifically this should improve linking by producing one large binary
instead of one-per-directory. The inclusion of the driver hits the size
issue. Separating out things which have more llvm deps has been
discussed, but I'm not doing that here because I think the semantics
layer will need to depend on clang for interop, and we'd lose a lot of
the benefits that way. Also, having just one place to look seems
simpler.
Includes supporting changes to file_test infrastructure, the most
significant of which is probably passing tests via file instead of a
really large args thing, using a custom rule to do that. That's because
dealing with the layered filegroups that allow the toolchain setup is
more complicated, and this approach scales well.
Combined test time is ~9s, so not sharding right now.
I wasn't sure if people would prefer having the autoupdate script under
testing, so I left it alone for now.
With the toolchain splitting namespaces, ostream.h's `operator<<`
templates aren't reliably found with name lookup, likely due to the loss
of associated namespaces (zygoloid commented on this at
https://github.com/carbon-language/carbon-lang/pull/3161#discussion_r1307941999).
This is especially a barrier to moving the lex files into `Carbon::Lex`;
versus other parts of the toolchain, they contain more printable types
which are used cross-namespace, including `Carbon::Testing`. As a
consequence, I'm looking at migrating ostream.h to a more reliable
approach that doesn't rely as much on everything being in the `Carbon`
namespace.
This function hid the `Value::Print` function from the base class, and
provided different output. As far as I can tell, the only caller is the
implementation of `Value::Print`.
Refactors `CallDestructor` and `CallFunction` to both call a new method
`BindSelfIfPresent`, which includes support for binding `addr self` if
specified. Fixes the associated unit test.
Closes#2802.
josh11bandJosh L
Jon Ross-Perkins
Richard Smith
David Blaikie
cui fliter
Daniel James SzelogowskiandRichard Smith
Uriel García
kshokhin
Jacob Schneider
Jonathan B. CoeandChandler Carruth
Geoff Romer
Prabhat SachdevaandGeoff Romer
Ikko Eltociear Ashimine
Lucile Rose Nihlen