Because we merge cleanups across scopes in various cases, and want to
use linear indexes into the complete stack, the ArrayStack abstraction
is getting in the way more than it's helping. Switch to just a
SmallVector.
This loses the unit testing of the MergeIntoGrandparent logic. This is
covered indirectly by check tests still, but direct testing of it is a
bit tricky given that ScopeStack isn't set up for use without a Context.
Destroy local variables and temporaries at each `}`, and when branching
with `break` and `continue`. In `for` statements, destroy loop variables
along with anything created within the loop at the end of each loop
iteration, and destroy the cursor and range object when the loop
terminates.
Assisted-by: Gemini via Antigravity
---------
Co-authored-by: Geoff Romer <gromer@google.com>
Fix a few API issues. There's also a newly-added file in compiler-rt
that is not supposed to be built by default but is not being excluded
properly by a glob. Added a patch to exclude that and sent
https://github.com/llvm/llvm-project/pull/208861 upstream.
This is based on an idea suggested during review of the original type
erasure PR.
The type-erased check failure path lowers each check through
`CheckFailFormat<Ts...>`, which builds one format adapter per value,
down to the out-of-line `CheckFailImpl`, which takes an array of
base-class `format_adapter*` pointers. Previously a separate variadic
`CheckFailWithAdapters<Adapters...>` template sat in between: it existed
only to bind the adapter temporaries to named parameters so that
pointers to their base class could be collected into a `std::array` and
outlive the call to `CheckFailImpl`.
This removes that layer. `CheckFailFormat` now builds the pointer array
directly, in the braced-init-list of the `CheckFailImpl` call, using a
small `CheckFailFormatAdapterAddr<T>` helper to take the base-class
address of each adapter. The adapter temporaries are materialized as
named parameters of that helper within the same full-expression as the
`CheckFailImpl` call, so they remain alive across the call without a
dedicated function for that purpose.
The win is in debug info, not code. `CheckFailWithAdapters<Adapters...>`
was instantiated once per distinct adapter-type sequence in every
translation unit that uses `CARBON_CHECK`, and each instantiation
carried its own DWARF records -- type entries, string-table offsets, and
range lists. Dropping it removes those records from every such
translation unit. The generated machine code is unchanged.
Measured impact (fastbuild, the inputs to the inner-loop links):
- First-party object files shrink by 173,728 bytes (-0.233%) across the
`carbon-busybox` link inputs; 232 of 270 objects get smaller and none
grow. The largest reductions are in the check-densest translation units
(`type_completion`, `type`, `import_ref`, `function_context`,
`constant`, ...).
- Per `bloaty`, the reduction is almost entirely DWARF (`.debug_info`,
`.debug_str`, `.rela.debug_str_offsets`, `.debug_rnglists`); the loaded
code (VM size) is unchanged.
Assisted-by: Claude Code
---------
Co-authored-by: Geoff Romer <gromer@google.com>
`CARBON_CHECK` and `CARBON_FATAL` messages are formatted with
`llvm::formatv`. Previously each check site that had a message
instantiated its own copy of the formatv machinery -- a `formatv_object`
over a tuple of per-argument format adapters, plus that tuple -- in
every translation unit, keyed on the site's file, line, condition, and
format strings. A translation unit with many checks paid for that
machinery over and over.
This restructures check failure so that the formatting machinery is
compiled exactly once, and only a single small adapter is instantiated
per distinct value type per translation unit:
- `CheckFailImpl` (out-of-line) now takes the message's format string
and an array of already-type-erased `format_adapter`s, and renders the
whole failure message -- prefix plus the extra message -- directly into
one stream. The extra message is rendered in place, so no separate
string is ever materialized for it.
- `FormatvInto` (in the `.cpp`) renders a format string over that
adapter array. Rather than instantiate `llvm::formatv`, it drives the
formatv replacement loop over the public
`formatv_object_base::parseFormatString`, so this rendering code exists
exactly once. (A TODO notes that we should add a type-erased entry point
upstream in LLVM rather than reimplement the loop here.)
- The lowering from the macro down to that out-of-line call is split so
that the only per-check-site instantiation is trivial:
- `CheckFail<...>` is instantiated once per site, since its file, line,
condition, and format template-string parameters are unique to the site.
It just lowers those compile-time strings to ordinary arguments and
forwards to `CheckFailFormat`.
- `CheckFailFormat<Ts...>` is instantiated once per distinct value-type
sequence and shared across sites; it builds one type-erased adapter per
value.
- `CheckFailWithAdapters<Adapters...>` collects pointers to those
adapters into an array and calls `CheckFailImpl`. It is a distinct
function so the adapter temporaries stay alive while pointers to their
base class are in flight.
Format semantics, including runtime format-string validation, are
unchanged, and the rendered message is byte-for-byte identical.
For `DCHECK` in optimized builds the check is dead code; its arguments
are now routed through a trivial `IgnoreDeadCheckArgs` no-op rather than
`CheckFail`. This still type-checks the arguments so they cannot bitrot,
without instantiating any formatting machinery for them and without
provoking unused-variable warnings.
Measured full-rebuild impact (353 first-party translation units,
fastbuild): -112.6s CPU, -6.9% relative to trunk.
Assisted-by: Claude
Because this is an `__attribute__((used))` method in a templated base
class it forces a _huge_ amount of template instantiation in every
translation unit.
Often this was just printing the members of the type, which is still
useful in some cases (such as test output), but adds no value in the
debugger.
A more successful pattern for dumping has been namespace level
functions, and particularly static ones that more transparently don't
expand the non-debugger API surface. Add the few missing functions there
that cover `Printable` types with more interesting contents.
For several of these, it just gives us a "dump the whole thing" function
as a compliment to "dump this entity in the thing". These probably
aren't especially high value, but moving them here they become cheap, so
I've left them in.
For a couple, this expands the rich dumping support of SemIR constructs,
which should be substantially more useful than the previous `Dump`
behavior.
This reduces `check` cumulative object file size by another 14%.
Assisted-by: Antigravity with Gemini
This still only works if the hash of the distinct types are identical
(so it still doesn't address the derived pointer v base pointer case -
well, not in the way we would want to address it, we could use this
change to make derived pointer and base pointer not compare equal, but
that's not very ergonomic)
I think in a follow up maybe I can use a `TranslatingKeyContext` to
translate `Derived*` to `Base*` in general.
No test coverage for this change, since it's a no-compile situation and
we don't seem to generally do no-compile tests.
Discovered while working on #6940
---------
Co-authored-by: Geoff Romer <gromer@google.com>
When the symlink target length from lstat was 0, the code resized the
buffer using status.size() instead of buffer_size, so the first
allocation stayed empty instead of using MinBufferSize. Align the resize
with the buffer_size path used for readlinkat.
This is a mostly routine update, with some edits for a benchmark API
change.
I'm not updating LLVM here, since that could conflict with other ongoing
work.
Most versions are through `pre-commit autoupdate --freeze`, clang-format
was manually updated to the latest at
https://github.com/ssciwr/clang-format-wheel
My read of the style changes here are that they seem fine, none of them
look like regressions (which has caused me to delay/adjust updates in
the past).
The YAML test helpers didn't use the `Printable` abstraction in one
place and instead directly used `<<` with a `std::ostream`. This matches
the `require`s expression in the `error_test_helpers.h` printing logic
for `ErrorOr`, but fails to provide the necessary implementation for
`llvm::formatv` to succeed with the `Yaml::Value` type.
The main fix is to use `Printable` and to define the `Print` method in
terms of `llvm::raw_ostream`. We already have all the mapping hooks in
place to also support `std::ostream` when needed based on that
definition.
This also adds some constraints to the printing in
`error_test_helpers.h` so it is a bit less under-constrained and more
understandable when it is correctly being used. These are just tidying
though, they aren't what makes these headers work together.
I've added a test to try and make sure these test helpers compose as
well.
OwningArrayRef is being removed upstream, per
https://github.com/llvm/llvm-project/pull/169126. This replaces uses
with `SmallVector`.
I've also made a separate commit which does init changes; these aren't
strictly necessary, but I added to make it a little more idiomatic in
spots.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
When building in Bazel actions, notably building runtimes, using
absolute paths makes the results non-hermetic and generally less
cache-friendly.
This restructures the code to only form an absolute path as part of the
`bazel run` change of working directory. It also tries to make the API
for doing this a bit more clear by taking the `exe_path` and
transforming it internally.
To support this, this PR also generalizes the `RemovingDir` to support
relative paths. While these can be tricky -- the working directory needs
to not change while they exist -- that isn't a reason to fully exclude
them and they're useful for implementing relative-path runtimes, etc.
This extracts the C-string `argv`-like building routine to a more
broadly reusable location. It also sinks the verbose logging logic out
of it and into the relevant runners. In turn, it simplifies the verbose
logging logic significantly.
The biggest functional change is removing the implicit synthesis of a
tool's `-v` verbose flag from the presence of a `vlog` stream. I thought
this would be helpful, but in practice of debugging these layers it has
been more of a hindrance than a help -- I pretty often only want verbose
logging on one side or the other, and we have ways of explicitly passing
a `-v` flag to the underlying tools already. I think my instinct to do
this was just wrong, so rip it out and simplify.
This does add an unused feature -- prepending a prefix of arguments
while building the C-string variant. This isn't used in this PR but will
be used in subsequent PRs and it seemed more disruptive to undo that
logic and then re-do it in a later PR. Let me know if it's too confusing
here.
Assisted-by: Gemini Code Assist
---------
Co-authored-by: Geoff Romer <gromer@google.com>
There are two uses I'm not converting here, that seem to want the
"shortest" behavior. For everything else, I'm going to `zip_equal` since
it's more restrictive.
I wish `zip` were named `zip_shortest`.
The standard `std::latch` is very restrictive in how it can be used, and
this makes it hard to easily leverage for simple coordination between a
set of dynamically scheduled tasks, where there isn't an interesting
synchronizing "merge" or future result.
This tool makes it easy to establish a latch, hand out handles to it,
and once all are destroyed, take whatever relevant action.
Note: this is split out of a larger change that uses it. I can wait
until the use case is ready, but seemed nice to review this separately.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
This fixes various violations of C++'s One Definition Rule, where we
accidentally gave the same static data member multiple definitions in
different translation units. Clang happens to emit such definitions with
weak linkage, which allows us to get away with this without link errors,
but it's still formally incorrect.
Also switch keyword order around for a handful of instances of
`constexpr inline`, per agreement in open discussion.
This happens to reduce the size of a `-c dbg` toolchain binary by 7.2
MiB, presumably by making more of our symbols and especially debug info
discardable.
This is just reducing boilerplate in `typed_insts.h` because we have a
number of singleton types, and keep adding more.
The changes to `TemplateString` allow `TemplateString IrName` to be used
as a `StringLiteral`.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
This layer allows runtimes to be built on-demand but cached in a
consistent and re-usable location on the system. It handles careful
filesystem operations to ensure consistency even in the face of multiple
versions and build configurations.
This addresses a number of TODOs from the initial runtimes building
on-demand, and sets the stage to scale up to more runtimes.
This doesn't switch on-demand runtimes to be on by default, I wanted to
wait and make that change as a separate step.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
Adds a unit test, and some smaller edits:
- Remove the `=` when defining names, in order to change `}` placement
by clang-format on uses.
- context:
https://github.com/carbon-language/carbon-lang/pull/6053#discussion_r2343423178
- I believe with `EnumBase` that keeping the `=` had been a deliberate
choice, so this PR is intended to confirm that removing it is okay.
- Delete `EnumMaskBase::name`
- context:
https://github.com/carbon-language/carbon-lang/pull/6053#discussion_r2344233707
- We can't just do nothing because `EnumBase::name` uses indexing that's
incompatible with `EnumMaskBase`.
- Some small comment cleanups.
- Tests don't need to be in the `Carbon` namespace anymore, macros work
fine in other namespaces, but it's still the right namespace.
- Documentation on `EnumBase::name` seems to be referring to a prior
structure, wherein we had a macro defining the function instead of the
`Names` array.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This is a bit of an experiment to see if there's a reasonable way to
write a shared enum type, rather than writing per-case wrappers for
things like `HasTypeQualifiers` or the printing. I think it's a bit
borderline complexity right now, but I'm not sure I can reduce it much
further.
This changes from things like `Internal::EnumClassName##RawEnum` to
`Internal::EnumClassName##Data::RawEnum` so that the enum entries can
have back references to bit shifts without needing to know the
containing type name. Because I'm trying to reduce duplication between
mask and non-mask enums, I did this to non-mask enums too.
This was motivated by #6035 adding another enum mask (which will grow
more entries, and is intended to switch if this is accepted), but I'm
not using that PR as a base here because I didn't want the merge
dependency.
Our style guide suggests using `<stdint.h>` and not the `std::`
qualifiers, and this is consistent with other headers like `<time.h>`.
The `clang-tidy` check enforces the reverse pattern, so disable it to
allow us to continue following our style pattern.
This is a collection of improvements to the filesystem library motivated
by using it to build a runtimes cache. It adds several core features:
- Advisory file locking
- Renaming of entries
- Testing for things being open
- File timestamp querying and updating
It also makes several more minor improvements such as improving the
names of functions and making them work in a more predictable fashion.
For example, the functions to read and write an entire file to/from
strings now actually handle the entire file rather than potentially
composing with other reads or writes, and adding the word `File` to
their name makes that more clear. Similarly, directory reading is more
robust in the face of repeatedly reading the same directory, and several
convenience functions were added to handle common patterns of reading
directories.
There is also a small fix to `ostream` uncovered by the tests added
here.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
Echoing what was added in #5608, updating existing uses. Unfortunately
there's divergent behavior for operators versus constructors, so keeping
the nolint on those.
They're essentially equivalent, we just typically write `typename`; even
in the examples here, most have other templates in the same file that
use `typename`.
Previously, this matcher mostly worked, but the `DescribeTo` functions
wouldn't compile when another polymorphic matcher was nested to match
the value.
The updated code uses the same polymorphic matcher design as used by
`Not` and others in Google Test itself.
I've added a test that uses `VariantWith` to nest matchers more deeply
with `IsSuccess`. This test doesn't compile prior to this change.
Specifically this adds `WriteStream` to get an LLVM-style
`raw_fd_ostream` for an open file, and `Rename` corresponding to
`rename` and `renameat` Unix-like system calls.
Some basic testing for both is added as well.
This was split out of work to switch the runtimes building to use the
new filesystem library.
Tidies up extraneous move, unnecessary function style type cast, and
simplifies the temporary directory string construction. These were
noticed during another PR review.
Also corrects support for older glibc versions, including the
GNU-specific quirks of `strerror_r`. Restricts the fancier formatting
with the name of the error number to when a recent glibc is available.
Lastly, filters the benchmarks in the benchmark test down to smaller
ones to avoid test timeout flakiness.
The standard filesystem API lacks significant functionality, ranging
from correct and secure creation of directories and files within them by
using `openat` and avoiding [TOCTOU] issues, to support for filesystem
locking.
[TOCTOU]: https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use
The LLVM filesystem library has more functionality, but uses an API that
is increasingly diverging from the standard, and also fails to defend
against TOCTOU.
This library is designed to carefully model the Unix or POSIX filesystem
concepts of `openat` to avoid TOCTOU. However, it also tries to limit
itself to an API subset that LLVM's filesystem library has also
implemneted and so we have a strong reason to expect to be possible to
port to Windows reasonably.
This PR included several benchmarks that show that this implementation
is also faster for the majority of operations than the C++ standard
library. The only places where there is a consistent regression is in
recursively creating directories, and this is directly connected to the
approach of using `openat` as the basis. Even there, while the wall time
regresses, the cycles and instructions are significantly improved.
There are a number of operations not yet included here, I've focused on
a core set of opening, closing, creating, and removing, and then adding
those that I saw the current toolchain code using actively. I'll plan to
expand the operations as needed going forward.
A follow-up PR that I'll finish polishing and send next ports
`//toolchain/install` to consistently use this library and
`std::filesystem::path` to both exercise the library and showcase its
use. I'll be working systematically across the toolchain to converge all
the code, extending this library as needed.
For reference, benchmark results on my macOS laptop:
https://gist.github.com/chandlerc/29d1f4d465a835b8be5174a48dad2e8f
Benchmark results on a Asahi Linux M1 Mac Mini:
https://gist.github.com/chandlerc/c42d43dd6b9b91746ab314b2afa152f7
Benchmark results on a Linux server with weirdly slow FS operations:
https://gist.github.com/chandlerc/48301a7383eb3972d53351b7e35e0561
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
When using this with filesystem errors, a few issues came up that I'm
fixing here. They're small enough and near enough in code that it didn't
seem worth splitting part.
- It's nice to forward declare custom error types and an API using them
and then define both later. That doesn't work with `requires` but works
fine with `static_assert`, so go back to that pattern here. A test is
added that checks this pattern compiles.
- The `operator*` didn't support moving out of `ErrorOr`, which is
especially important when writing code that is happy with just
`CARBON_CHECK`-failing on any errors. For example, we have a lot of
filesystem code in tests that is made *much* more concise by just using
`*` on a function return and letting the built-in checking ensure no
errors were present. But when the value is move-only, this requires
special overloading. Add that and add a test with a move-only value.
- There wasn't an idiomatic way to do something like `operator*` for
`ErrorOr<Success, ...>`. This PR factors out the checking for `ok()`
into a `Check()` method that can be used to make code more readable that
is intentionally just verifying no error. Also makes the result of
`operator*` `[[nodiscard]]` to improve error messages and help void
accidental bugs.
- The `IsError` and `IsSuccess` test helpers required printable values
which isn't always realistic. Teach the printing logic to be conditional
on some indication of a printable value and gracefully fall back to a
generic string otherwise for testing output.
- The use of the `listener` in `IsError` and `IsSuccess` assumed a
non-null stream. Instead, streaming should go directly to the `listener`
as it is configured to only actually do the output when a stream is
installed. When a stream isn't installed, the previous code would crash
if the `MatchAndExplain` method ended up called without an 'interesting'
stream attached to the listener.
- When doing a `CARBON_CHECK` that there isn't an error, print the error
out as the check failure message. Without this, all the nice error
message work doesn't end up helping the debugging of test code that hits
these errors, etc.
This doesn't split apart the current error type into one that tracks
location and one that doesn't, although that might be easier to do once
we have this.
Instead, this is primarily intended to support custom error types that
lazily materialize the error message in case that can be avoided by
completely handling the error. For example, many file system operations
are *expected* to produce errors even in the hot path and we don't want
to render `ENOENT` (for example) to a pretty string and instead will
directly query the error to understand and handle it in code.
The type parameter ordering isn't the most obvious, but helpfully allows
us to default the error type in a useful way.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
These have unique challenges for our hashing scheme, and so its useful
to make sure the hash functions we use can handle them.
Some other work on Abseil's hash tables uncovered that this might be
risky and may have surfaced some improvements to reduce the impact here,
but the first step seems to try and start covering this path in the
benchmarks.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
When finding an executable, this validates that the returned binary is a
symlink back to the same thing as /proc/self/exe, also using that as a
fallback for different things.
Looking back at #3912, we started using `findProgramByName` in order to
avoid path canonicalization done by `GetMainExecutable`. That created
issues as in #5096, wherein an `argv[0]` that's not explicit enough
(`llvm-symbolizer` instead of the full path, done in [LLVM's
Signals.cpp](https://github.com/llvm/llvm-project/blob/4f60f45130c6bd96c79e468fe9927a29af760f56/llvm/lib/Support/Signals.cpp#L198))
leads to incorrect results (finding an `llvm-symbolizer` in `$PATH`).
One option to fix this would be to patch LLVM to provide an absolute
path for `llvm-symbolizer`. However, I'll suggest that passing a
filename in `argv[0]` is not terribly uncommon, and could be a migration
limitation if we force it. The failure mode is also opaque; for example:
```
$ /bin/sh -c "exec -a llvm-symbolizer ./bazel-bin/toolchain/carbon"
error: expected carbon-busybox symlink at `/usr/lib/llvm-19/bin/llvm-symbolizer`
```
Combined with the `setenv` of `LLVM_SYMBOLIZER_PATH` in
`busybox_main.cpp`, this is intended to fix#5096.
Use it to replace most existing modernize-loop-convert lints with
range-based for loops. As requested in review of #5475.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
We already go to some effort to avoid moving these, but we end up still
moving them twice: once when adding to the worklist and again when
reversing a chunk of the worklist.
* To avoid a move when constructing the worklist, add an `EmplaceResult`
utility that allows the result of a function call to be emplaced into a
container.
* To avoid moves when reversing the list, stop reversing it. Instead of
reversing the list and popping tasks as we run them, we accumulate a
sequence of tasks for a deferred definition region, run them in the
order they were enqueued, then pop them all at the end. This will in
some cases increase the high-water-mark of the size of the worklist, but
not asymptotically. The same high-water-mark could be reached with the
old approach by reordering the declarations in the source file.
In passing, we no longer create `LeaveDeferredDefinitionRegion` tasks
for non-nested regions. We don't need them, because we can detect that
condition by our reaching the end of the worklist. This means that the
enter / leave region actions are now always in correspondence -- we only
create them for *nested* regions. The tasks have been renamed to convey
this.
We still move the suspended function states around if the worklist grows
to over 64 entries and gets reallocated. We could potentially address
that issue too by switching to a chunked allocation strategy as is used
by `ValueStore` and then make the tasks noncopyable, but I'm not
attempting that in this PR.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
This reverts commit 1889ee3904.
We have identified that this is causing ODR violations, because the
`fuzzer` feature is being added `cc_fuzz_test` targets, and thus any
includes they make, but not to the rest of the build. Any include that
is seen from both places has ODR violations if it branches on
FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION.
We need to apply fuzzer globally when building fuzz targets somehow, or
not set different defines in fuzzer.
Richard Smith
Chandler CarruthandGeoff Romer
David BlaikieandGeoff Romer
cui
Jon Ross-Perkins
