This layer allows runtimes to be built on-demand but cached in a
consistent and re-usable location on the system. It handles careful
filesystem operations to ensure consistency even in the face of multiple
versions and build configurations.
This addresses a number of TODOs from the initial runtimes building
on-demand, and sets the stage to scale up to more runtimes.
This doesn't switch on-demand runtimes to be on by default, I wanted to
wait and make that change as a separate step.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
Adds a unit test, and some smaller edits:
- Remove the `=` when defining names, in order to change `}` placement
by clang-format on uses.
- context:
https://github.com/carbon-language/carbon-lang/pull/6053#discussion_r2343423178
- I believe with `EnumBase` that keeping the `=` had been a deliberate
choice, so this PR is intended to confirm that removing it is okay.
- Delete `EnumMaskBase::name`
- context:
https://github.com/carbon-language/carbon-lang/pull/6053#discussion_r2344233707
- We can't just do nothing because `EnumBase::name` uses indexing that's
incompatible with `EnumMaskBase`.
- Some small comment cleanups.
- Tests don't need to be in the `Carbon` namespace anymore, macros work
fine in other namespaces, but it's still the right namespace.
- Documentation on `EnumBase::name` seems to be referring to a prior
structure, wherein we had a macro defining the function instead of the
`Names` array.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This is a bit of an experiment to see if there's a reasonable way to
write a shared enum type, rather than writing per-case wrappers for
things like `HasTypeQualifiers` or the printing. I think it's a bit
borderline complexity right now, but I'm not sure I can reduce it much
further.
This changes from things like `Internal::EnumClassName##RawEnum` to
`Internal::EnumClassName##Data::RawEnum` so that the enum entries can
have back references to bit shifts without needing to know the
containing type name. Because I'm trying to reduce duplication between
mask and non-mask enums, I did this to non-mask enums too.
This was motivated by #6035 adding another enum mask (which will grow
more entries, and is intended to switch if this is accepted), but I'm
not using that PR as a base here because I didn't want the merge
dependency.
Our style guide suggests using `<stdint.h>` and not the `std::`
qualifiers, and this is consistent with other headers like `<time.h>`.
The `clang-tidy` check enforces the reverse pattern, so disable it to
allow us to continue following our style pattern.
This is a collection of improvements to the filesystem library motivated
by using it to build a runtimes cache. It adds several core features:
- Advisory file locking
- Renaming of entries
- Testing for things being open
- File timestamp querying and updating
It also makes several more minor improvements such as improving the
names of functions and making them work in a more predictable fashion.
For example, the functions to read and write an entire file to/from
strings now actually handle the entire file rather than potentially
composing with other reads or writes, and adding the word `File` to
their name makes that more clear. Similarly, directory reading is more
robust in the face of repeatedly reading the same directory, and several
convenience functions were added to handle common patterns of reading
directories.
There is also a small fix to `ostream` uncovered by the tests added
here.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
Echoing what was added in #5608, updating existing uses. Unfortunately
there's divergent behavior for operators versus constructors, so keeping
the nolint on those.
They're essentially equivalent, we just typically write `typename`; even
in the examples here, most have other templates in the same file that
use `typename`.
Previously, this matcher mostly worked, but the `DescribeTo` functions
wouldn't compile when another polymorphic matcher was nested to match
the value.
The updated code uses the same polymorphic matcher design as used by
`Not` and others in Google Test itself.
I've added a test that uses `VariantWith` to nest matchers more deeply
with `IsSuccess`. This test doesn't compile prior to this change.
Specifically this adds `WriteStream` to get an LLVM-style
`raw_fd_ostream` for an open file, and `Rename` corresponding to
`rename` and `renameat` Unix-like system calls.
Some basic testing for both is added as well.
This was split out of work to switch the runtimes building to use the
new filesystem library.
Tidies up extraneous move, unnecessary function style type cast, and
simplifies the temporary directory string construction. These were
noticed during another PR review.
Also corrects support for older glibc versions, including the
GNU-specific quirks of `strerror_r`. Restricts the fancier formatting
with the name of the error number to when a recent glibc is available.
Lastly, filters the benchmarks in the benchmark test down to smaller
ones to avoid test timeout flakiness.
The standard filesystem API lacks significant functionality, ranging
from correct and secure creation of directories and files within them by
using `openat` and avoiding [TOCTOU] issues, to support for filesystem
locking.
[TOCTOU]: https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use
The LLVM filesystem library has more functionality, but uses an API that
is increasingly diverging from the standard, and also fails to defend
against TOCTOU.
This library is designed to carefully model the Unix or POSIX filesystem
concepts of `openat` to avoid TOCTOU. However, it also tries to limit
itself to an API subset that LLVM's filesystem library has also
implemneted and so we have a strong reason to expect to be possible to
port to Windows reasonably.
This PR included several benchmarks that show that this implementation
is also faster for the majority of operations than the C++ standard
library. The only places where there is a consistent regression is in
recursively creating directories, and this is directly connected to the
approach of using `openat` as the basis. Even there, while the wall time
regresses, the cycles and instructions are significantly improved.
There are a number of operations not yet included here, I've focused on
a core set of opening, closing, creating, and removing, and then adding
those that I saw the current toolchain code using actively. I'll plan to
expand the operations as needed going forward.
A follow-up PR that I'll finish polishing and send next ports
`//toolchain/install` to consistently use this library and
`std::filesystem::path` to both exercise the library and showcase its
use. I'll be working systematically across the toolchain to converge all
the code, extending this library as needed.
For reference, benchmark results on my macOS laptop:
https://gist.github.com/chandlerc/29d1f4d465a835b8be5174a48dad2e8f
Benchmark results on a Asahi Linux M1 Mac Mini:
https://gist.github.com/chandlerc/c42d43dd6b9b91746ab314b2afa152f7
Benchmark results on a Linux server with weirdly slow FS operations:
https://gist.github.com/chandlerc/48301a7383eb3972d53351b7e35e0561
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
When using this with filesystem errors, a few issues came up that I'm
fixing here. They're small enough and near enough in code that it didn't
seem worth splitting part.
- It's nice to forward declare custom error types and an API using them
and then define both later. That doesn't work with `requires` but works
fine with `static_assert`, so go back to that pattern here. A test is
added that checks this pattern compiles.
- The `operator*` didn't support moving out of `ErrorOr`, which is
especially important when writing code that is happy with just
`CARBON_CHECK`-failing on any errors. For example, we have a lot of
filesystem code in tests that is made *much* more concise by just using
`*` on a function return and letting the built-in checking ensure no
errors were present. But when the value is move-only, this requires
special overloading. Add that and add a test with a move-only value.
- There wasn't an idiomatic way to do something like `operator*` for
`ErrorOr<Success, ...>`. This PR factors out the checking for `ok()`
into a `Check()` method that can be used to make code more readable that
is intentionally just verifying no error. Also makes the result of
`operator*` `[[nodiscard]]` to improve error messages and help void
accidental bugs.
- The `IsError` and `IsSuccess` test helpers required printable values
which isn't always realistic. Teach the printing logic to be conditional
on some indication of a printable value and gracefully fall back to a
generic string otherwise for testing output.
- The use of the `listener` in `IsError` and `IsSuccess` assumed a
non-null stream. Instead, streaming should go directly to the `listener`
as it is configured to only actually do the output when a stream is
installed. When a stream isn't installed, the previous code would crash
if the `MatchAndExplain` method ended up called without an 'interesting'
stream attached to the listener.
- When doing a `CARBON_CHECK` that there isn't an error, print the error
out as the check failure message. Without this, all the nice error
message work doesn't end up helping the debugging of test code that hits
these errors, etc.
This doesn't split apart the current error type into one that tracks
location and one that doesn't, although that might be easier to do once
we have this.
Instead, this is primarily intended to support custom error types that
lazily materialize the error message in case that can be avoided by
completely handling the error. For example, many file system operations
are *expected* to produce errors even in the hot path and we don't want
to render `ENOENT` (for example) to a pretty string and instead will
directly query the error to understand and handle it in code.
The type parameter ordering isn't the most obvious, but helpfully allows
us to default the error type in a useful way.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
These have unique challenges for our hashing scheme, and so its useful
to make sure the hash functions we use can handle them.
Some other work on Abseil's hash tables uncovered that this might be
risky and may have surfaced some improvements to reduce the impact here,
but the first step seems to try and start covering this path in the
benchmarks.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
When finding an executable, this validates that the returned binary is a
symlink back to the same thing as /proc/self/exe, also using that as a
fallback for different things.
Looking back at #3912, we started using `findProgramByName` in order to
avoid path canonicalization done by `GetMainExecutable`. That created
issues as in #5096, wherein an `argv[0]` that's not explicit enough
(`llvm-symbolizer` instead of the full path, done in [LLVM's
Signals.cpp](https://github.com/llvm/llvm-project/blob/4f60f45130c6bd96c79e468fe9927a29af760f56/llvm/lib/Support/Signals.cpp#L198))
leads to incorrect results (finding an `llvm-symbolizer` in `$PATH`).
One option to fix this would be to patch LLVM to provide an absolute
path for `llvm-symbolizer`. However, I'll suggest that passing a
filename in `argv[0]` is not terribly uncommon, and could be a migration
limitation if we force it. The failure mode is also opaque; for example:
```
$ /bin/sh -c "exec -a llvm-symbolizer ./bazel-bin/toolchain/carbon"
error: expected carbon-busybox symlink at `/usr/lib/llvm-19/bin/llvm-symbolizer`
```
Combined with the `setenv` of `LLVM_SYMBOLIZER_PATH` in
`busybox_main.cpp`, this is intended to fix#5096.
Use it to replace most existing modernize-loop-convert lints with
range-based for loops. As requested in review of #5475.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
We already go to some effort to avoid moving these, but we end up still
moving them twice: once when adding to the worklist and again when
reversing a chunk of the worklist.
* To avoid a move when constructing the worklist, add an `EmplaceResult`
utility that allows the result of a function call to be emplaced into a
container.
* To avoid moves when reversing the list, stop reversing it. Instead of
reversing the list and popping tasks as we run them, we accumulate a
sequence of tasks for a deferred definition region, run them in the
order they were enqueued, then pop them all at the end. This will in
some cases increase the high-water-mark of the size of the worklist, but
not asymptotically. The same high-water-mark could be reached with the
old approach by reordering the declarations in the source file.
In passing, we no longer create `LeaveDeferredDefinitionRegion` tasks
for non-nested regions. We don't need them, because we can detect that
condition by our reaching the end of the worklist. This means that the
enter / leave region actions are now always in correspondence -- we only
create them for *nested* regions. The tasks have been renamed to convey
this.
We still move the suspended function states around if the worklist grows
to over 64 entries and gets reallocated. We could potentially address
that issue too by switching to a chunked allocation strategy as is used
by `ValueStore` and then make the tasks noncopyable, but I'm not
attempting that in this PR.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
This reverts commit 1889ee3904.
We have identified that this is causing ODR violations, because the
`fuzzer` feature is being added `cc_fuzz_test` targets, and thus any
includes they make, but not to the rest of the build. Any include that
is seen from both places has ODR violations if it branches on
FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION.
We need to apply fuzzer globally when building fuzz targets somehow, or
not set different defines in fuzzer.
I'm trying to make the offsetting a little easier to understand, and
also get a better `requires` structure on calls. The second is for an
attempt to refactor the `Formatter` API, but also changing the `InstId`
`derived_from` requires seems helpful for clarity on what's really
happening.
Trying to make repeated `std::same_as` easier to write. Calling it
"concepts.h" because I figure we'll maybe have a couple more things like
this.
Was looking at this because I may add a couple more similar constructs.
The `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` flag is a standard flag
proposed by LibFuzzer that is meant to inform compiled code that it is
being built for fuzzing, as described here:
https://llvm.org/docs/LibFuzzer.html#fuzzer-friendly-build-mode
We add the flag to our `fuzzing` feature/config, and enable DCHECKs when
under fuzzing so that we can catch bugs that currently are caught on the
other side of DCHECK, even if they don't cause ASAN to trap a read/write
beyond the capacity of a value store.
Teach CARBON_KIND_SWITCH to handle mutable lvalues and rvalues, and
CARBON_KIND to forward along rvalues so that it's possible to write
`case CARBON_KIND(const T& t)`, `case CARBON_KIND(T& t)`, and `case
CARBON_KIND(T&& t)`, depending on the type that was passed to
CARBON_KIND_SWITCH.
Replace all uses of VariantMatch with their equivalent of a switch using
CARBON_KIND_SWITCH, and remove the VariantMatch helper from the
codebase.
The version of clangd/clang-tidy on developer machines has slowly
diverged from the one on the CI builders, which is causing a slowly
increasing amount of pain as clang-tidy CI runs fail (incorrectly) over
things that a newer clangd/clang-tidy was perfectly fine with locally.
This bumps the Clang version used in the ubuntu builders to 19, which is
the most recent in Debian stable.
We use https://apt.llvm.org instead of LLVM's GitHub releases
(https://github.com/llvm/llvm-project/releases) as the former more
reliably has packages for newer Clang/LLVM versions on x64. The
community-build releases binaries on LLVM's GitHub have stopped
including Ubuntu packages that match the GitHub x64 Ubuntu workers for
some time (for at least the 18 and 19 releases).
By moving to apt.llvm.org packages we only download and install the
headers and libraries needed for development, rather than every output
of building llvm, which is much faster and saves lots of disk space. We
also remove the system installations of other versions of clang/llvm so
we should end up using negative disk space. We can no longer easily
cache the installation but apt.llvm.org is a reliable end point.
We bump the ubuntu image version for the github workers to 24.04, as
apt.llvm.org has stopped building images for 22.10 in 2022 at its end of
life.
The `pre_commit` workflow disabled sudo unlike the other workflows that
install Clang/LLVM, including the `clang-tidy` workflow (which is also
run on `pull_request`). We bring it into alignment with the other
workflows so that we can install the llvm packages. And we lock its
ubuntu image to 24.04 so that it can be moved in lockstep with the other
workflows that depend on Clang/LLVM.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
#5445 updates to bazel 8.2.1, this does more updates (including to
buildifier, which does autofixes like the `sh_test` loads in the other
PR).
Note I'm using the latest available clang-format wheel. That's not
really something I expect people to have installed, but should mostly be
consistent. I'm specifically skipping clang-format 18 because it had
some broad regressions, and 19 got really confused by a `requires` on a
trailing return. Using the latest seemed probably okay since most people
won't see the difference. Do note that trailing returns in macros,
https://github.com/llvm/llvm-project/issues/47664, seems to be cropping
up again as an issue.
- Updates incompatible flags.
- `rules_flex` is no longer used, so enable its flag.
- Fixes `sh_test` deps for
`--incompatible_disable_autoloads_in_main_repo`
- Broadens the exception for `rules_cc` and `bazel_tools` due to changes
to runfiles deps; trying to avoid minutiae that shouldn't affect the
decision.
Instead of building the definition of a thunk immediately when we
generate the thunk declaration, wait until we reach the `}` of the
outermost class, interface, etc. -- at the same time when we would parse
the definition of the thunk if it were defined inline.
This fixes issues where we fail to define the thunk because it requires
an enclosing class to be complete, or its definition depends on
something declared later in the enclosing class.
Make the representation of a suspended function scope, and its
constituent suspended components, be move-only, and switch to passing it
around by rvalue reference instead of by value because it's expensive
both to move and especially to copy.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
`FindIfOrNull` returns a pointer to the element in the range if it's
found, and nullptr otherwise. `FindIfOrNone` returns a copy of the
element in the range if it's found, and `T::None` (for a range of
elements of type `T`) otherwise. `Contains` returns a bool indicating
whether the element in the range is found.
These functions replace `llvm::find()` and `llvm::find_if()` when you
want a single answer back instead of an iterator. This avoids the need
to check against `end()`, allowing the return condition to be tested as
a standard bool.
We replace uses of `find()` and `find_if()` that did not require an
iterator with these new helpers.
Note that the return type of `FindIfOrNull` is a pointer since we can
not write `optional<T&>`, which must be tested for null. If the null
check is omitted, UB occurs and the resulting code may end up with an
incorrect pointer (https://crbug.com/40153300) into the range (or
elsewhere), rather than a null dereference. And this would be very
confusing to debug. Hopefully debug builds and sanitizers keep this from
being an issue we sink a bunch of time into debugging.
Rules executed by bazel don't necessarily have the right environment to
find the symbolizer, which was the intent of `cc_env` setting
`LLVM_SYMBOLIZER_PATH`. So far, this has kind of been a case-by-case
fix, but every so often I'm trying to debug a crash in a test that
doesn't provide it. Rather continuing down this route, instead add
drop-in wrappers for cc rules so that it's hard to forget.
Note `bazel/cc_rules` is intended to mirror `bazel/carbon_rules` and
`bazel/cc_toolchains`, rather than `@rules_cc`.
AFAICT there isn't a great way to add this as a default for the `bazel
run` environment. It's not typically going to be set on its own,
forwarding `$PATH` would be too broad, and the [action
`env_sets`](https://bazel.build/docs/cc-toolchain-config-reference#using-action-config)
I think are not quite what we need (I think those don't include output
execution, only compilation).
I was thinking about this after `seq` changes in #5182, and looked for
other uses that might be replaceable. Here's the resulting cleanup
around `seq`:
- Switch to `enumerate` or `zip` when possible.
- `int _` -> `auto _` (it's typically a `size_t`, but there's no reason
to cast when unused)
- Fix a case of cast style `(size_t)...` -> `static_cast<size_t>(...)`
- Switch `(void)close_children_count` to `[[maybe_unused]]`
This fixes a `copy constructor must pass its first argument by
reference` compilation error when compiled with a recent enough Clang
(after
https://github.com/llvm/llvm-project/commit/fe0d3e3764961b62f43f1b129f30aaec5f30bc16,
targeted for LLVM 21 release).
```
carbon/lang/common/set.h:81:59: error: copy constructor must pass its first argument by reference
81 | SetView(SetView<std::remove_const_t<KeyT>, KeyContextT> other_view)
| ^
```
This makes it friendlier in interactive debuggers. If you want to print
a value without a newline from code, you will have to be calling Print()
anyway since Dump() is private, and Print() does not add a newline.
We had a long discussion of this, so trying to document what seems to be
the conclusion... and also clean up the exceptions that I could find.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
Noted CopyNameFromImportIR while glancing around (this one's interesting
because it's NameId, not void nor auto), did a scan just for a few other
cases. Not an exhaustive fix, and TBH assuming we'd prefer `auto ... ->
auto` since equivalent Carbon syntax would probably be `fn ... -> auto`
When I open a .def file, there are often 4 errors:
- The #error
- The #define is not defined
- Missing `;`
- Identifier naming
This PR is meant to disable all of these, since they can be distracting
from fixable diagnostics.
Jon Ross-Perkinsandjosh11b
Chandler CarruthandGeoff Romer
Thomas Köppe
Alexander Kornienko
Boaz Brickner