The YAML test helpers didn't use the `Printable` abstraction in one
place and instead directly used `<<` with a `std::ostream`. This matches
the `require`s expression in the `error_test_helpers.h` printing logic
for `ErrorOr`, but fails to provide the necessary implementation for
`llvm::formatv` to succeed with the `Yaml::Value` type.
The main fix is to use `Printable` and to define the `Print` method in
terms of `llvm::raw_ostream`. We already have all the mapping hooks in
place to also support `std::ostream` when needed based on that
definition.
This also adds some constraints to the printing in
`error_test_helpers.h` so it is a bit less under-constrained and more
understandable when it is correctly being used. These are just tidying
though, they aren't what makes these headers work together.
I've added a test to try and make sure these test helpers compose as
well.
When building in Bazel actions, notably building runtimes, using
absolute paths makes the results non-hermetic and generally less
cache-friendly.
This restructures the code to only form an absolute path as part of the
`bazel run` change of working directory. It also tries to make the API
for doing this a bit more clear by taking the `exe_path` and
transforming it internally.
To support this, this PR also generalizes the `RemovingDir` to support
relative paths. While these can be tricky -- the working directory needs
to not change while they exist -- that isn't a reason to fully exclude
them and they're useful for implementing relative-path runtimes, etc.
The standard `std::latch` is very restrictive in how it can be used, and
this makes it hard to easily leverage for simple coordination between a
set of dynamically scheduled tasks, where there isn't an interesting
synchronizing "merge" or future result.
This tool makes it easy to establish a latch, hand out handles to it,
and once all are destroyed, take whatever relevant action.
Note: this is split out of a larger change that uses it. I can wait
until the use case is ready, but seemed nice to review this separately.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
Adds a unit test, and some smaller edits:
- Remove the `=` when defining names, in order to change `}` placement
by clang-format on uses.
- context:
https://github.com/carbon-language/carbon-lang/pull/6053#discussion_r2343423178
- I believe with `EnumBase` that keeping the `=` had been a deliberate
choice, so this PR is intended to confirm that removing it is okay.
- Delete `EnumMaskBase::name`
- context:
https://github.com/carbon-language/carbon-lang/pull/6053#discussion_r2344233707
- We can't just do nothing because `EnumBase::name` uses indexing that's
incompatible with `EnumMaskBase`.
- Some small comment cleanups.
- Tests don't need to be in the `Carbon` namespace anymore, macros work
fine in other namespaces, but it's still the right namespace.
- Documentation on `EnumBase::name` seems to be referring to a prior
structure, wherein we had a macro defining the function instead of the
`Names` array.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This is a bit of an experiment to see if there's a reasonable way to
write a shared enum type, rather than writing per-case wrappers for
things like `HasTypeQualifiers` or the printing. I think it's a bit
borderline complexity right now, but I'm not sure I can reduce it much
further.
This changes from things like `Internal::EnumClassName##RawEnum` to
`Internal::EnumClassName##Data::RawEnum` so that the enum entries can
have back references to bit shifts without needing to know the
containing type name. Because I'm trying to reduce duplication between
mask and non-mask enums, I did this to non-mask enums too.
This was motivated by #6035 adding another enum mask (which will grow
more entries, and is intended to switch if this is accepted), but I'm
not using that PR as a base here because I didn't want the merge
dependency.
Tidies up extraneous move, unnecessary function style type cast, and
simplifies the temporary directory string construction. These were
noticed during another PR review.
Also corrects support for older glibc versions, including the
GNU-specific quirks of `strerror_r`. Restricts the fancier formatting
with the name of the error number to when a recent glibc is available.
Lastly, filters the benchmarks in the benchmark test down to smaller
ones to avoid test timeout flakiness.
The standard filesystem API lacks significant functionality, ranging
from correct and secure creation of directories and files within them by
using `openat` and avoiding [TOCTOU] issues, to support for filesystem
locking.
[TOCTOU]: https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use
The LLVM filesystem library has more functionality, but uses an API that
is increasingly diverging from the standard, and also fails to defend
against TOCTOU.
This library is designed to carefully model the Unix or POSIX filesystem
concepts of `openat` to avoid TOCTOU. However, it also tries to limit
itself to an API subset that LLVM's filesystem library has also
implemneted and so we have a strong reason to expect to be possible to
port to Windows reasonably.
This PR included several benchmarks that show that this implementation
is also faster for the majority of operations than the C++ standard
library. The only places where there is a consistent regression is in
recursively creating directories, and this is directly connected to the
approach of using `openat` as the basis. Even there, while the wall time
regresses, the cycles and instructions are significantly improved.
There are a number of operations not yet included here, I've focused on
a core set of opening, closing, creating, and removing, and then adding
those that I saw the current toolchain code using actively. I'll plan to
expand the operations as needed going forward.
A follow-up PR that I'll finish polishing and send next ports
`//toolchain/install` to consistently use this library and
`std::filesystem::path` to both exercise the library and showcase its
use. I'll be working systematically across the toolchain to converge all
the code, extending this library as needed.
For reference, benchmark results on my macOS laptop:
https://gist.github.com/chandlerc/29d1f4d465a835b8be5174a48dad2e8f
Benchmark results on a Asahi Linux M1 Mac Mini:
https://gist.github.com/chandlerc/c42d43dd6b9b91746ab314b2afa152f7
Benchmark results on a Linux server with weirdly slow FS operations:
https://gist.github.com/chandlerc/48301a7383eb3972d53351b7e35e0561
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
These have unique challenges for our hashing scheme, and so its useful
to make sure the hash functions we use can handle them.
Some other work on Abseil's hash tables uncovered that this might be
risky and may have surfaced some improvements to reduce the impact here,
but the first step seems to try and start covering this path in the
benchmarks.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Use it to replace most existing modernize-loop-convert lints with
range-based for loops. As requested in review of #5475.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
We already go to some effort to avoid moving these, but we end up still
moving them twice: once when adding to the worklist and again when
reversing a chunk of the worklist.
* To avoid a move when constructing the worklist, add an `EmplaceResult`
utility that allows the result of a function call to be emplaced into a
container.
* To avoid moves when reversing the list, stop reversing it. Instead of
reversing the list and popping tasks as we run them, we accumulate a
sequence of tasks for a deferred definition region, run them in the
order they were enqueued, then pop them all at the end. This will in
some cases increase the high-water-mark of the size of the worklist, but
not asymptotically. The same high-water-mark could be reached with the
old approach by reordering the declarations in the source file.
In passing, we no longer create `LeaveDeferredDefinitionRegion` tasks
for non-nested regions. We don't need them, because we can detect that
condition by our reaching the end of the worklist. This means that the
enter / leave region actions are now always in correspondence -- we only
create them for *nested* regions. The tasks have been renamed to convey
this.
We still move the suspended function states around if the worklist grows
to over 64 entries and gets reallocated. We could potentially address
that issue too by switching to a chunked allocation strategy as is used
by `ValueStore` and then make the tasks noncopyable, but I'm not
attempting that in this PR.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
I'm trying to make the offsetting a little easier to understand, and
also get a better `requires` structure on calls. The second is for an
attempt to refactor the `Formatter` API, but also changing the `InstId`
`derived_from` requires seems helpful for clarity on what's really
happening.
Trying to make repeated `std::same_as` easier to write. Calling it
"concepts.h" because I figure we'll maybe have a couple more things like
this.
Was looking at this because I may add a couple more similar constructs.
Teach CARBON_KIND_SWITCH to handle mutable lvalues and rvalues, and
CARBON_KIND to forward along rvalues so that it's possible to write
`case CARBON_KIND(const T& t)`, `case CARBON_KIND(T& t)`, and `case
CARBON_KIND(T&& t)`, depending on the type that was passed to
CARBON_KIND_SWITCH.
Replace all uses of VariantMatch with their equivalent of a switch using
CARBON_KIND_SWITCH, and remove the VariantMatch helper from the
codebase.
- Updates incompatible flags.
- `rules_flex` is no longer used, so enable its flag.
- Fixes `sh_test` deps for
`--incompatible_disable_autoloads_in_main_repo`
- Broadens the exception for `rules_cc` and `bazel_tools` due to changes
to runfiles deps; trying to avoid minutiae that shouldn't affect the
decision.
Instead of building the definition of a thunk immediately when we
generate the thunk declaration, wait until we reach the `}` of the
outermost class, interface, etc. -- at the same time when we would parse
the definition of the thunk if it were defined inline.
This fixes issues where we fail to define the thunk because it requires
an enclosing class to be complete, or its definition depends on
something declared later in the enclosing class.
Make the representation of a suspended function scope, and its
constituent suspended components, be move-only, and switch to passing it
around by rvalue reference instead of by value because it's expensive
both to move and especially to copy.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
`FindIfOrNull` returns a pointer to the element in the range if it's
found, and nullptr otherwise. `FindIfOrNone` returns a copy of the
element in the range if it's found, and `T::None` (for a range of
elements of type `T`) otherwise. `Contains` returns a bool indicating
whether the element in the range is found.
These functions replace `llvm::find()` and `llvm::find_if()` when you
want a single answer back instead of an iterator. This avoids the need
to check against `end()`, allowing the return condition to be tested as
a standard bool.
We replace uses of `find()` and `find_if()` that did not require an
iterator with these new helpers.
Note that the return type of `FindIfOrNull` is a pointer since we can
not write `optional<T&>`, which must be tested for null. If the null
check is omitted, UB occurs and the resulting code may end up with an
incorrect pointer (https://crbug.com/40153300) into the range (or
elsewhere), rather than a null dereference. And this would be very
confusing to debug. Hopefully debug builds and sanitizers keep this from
being an issue we sink a bunch of time into debugging.
Rules executed by bazel don't necessarily have the right environment to
find the symbolizer, which was the intent of `cc_env` setting
`LLVM_SYMBOLIZER_PATH`. So far, this has kind of been a case-by-case
fix, but every so often I'm trying to debug a crash in a test that
doesn't provide it. Rather continuing down this route, instead add
drop-in wrappers for cc rules so that it's hard to forget.
Note `bazel/cc_rules` is intended to mirror `bazel/carbon_rules` and
`bazel/cc_toolchains`, rather than `@rules_cc`.
AFAICT there isn't a great way to add this as a default for the `bazel
run` environment. It's not typically going to be set on its own,
forwarding `$PATH` would be too broad, and the [action
`env_sets`](https://bazel.build/docs/cc-toolchain-config-reference#using-action-config)
I think are not quite what we need (I think those don't include output
execution, only compilation).
This adds a RawStringOstream. Versus TestRawOstream, which is
consolidated over to RawStringOstream, it uses a string for storage
instead of a vector, mainly to support move-to-string semantics. Versus
llvm::raw_string_ostream, it owns the string and supports pwrite (which
is needed for driver and its fd_ostream compatibility requirement).
This converts most uses of llvm::raw_string_ostream, leaving behind a
few in InstNamer that explicitly cannot own the string, such as:
```
llvm::raw_string_ostream(name)
<< "_" << tree.tokens().GetColumnNumber(token);
```
I have this as its own library so that it can use CHECK.
Yes this doesn't save much code, but it's code we repeatedly write.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
This changes to an `Error` return to let the driver do the "error: "
prefix, except for one case with `help` that needs more work to change
(I'm not planning on picking up that TODO). It also changes
capitalization, backtick use, and a few minor punctuation things to try
to better match the diagnostic style.
This also adds `Error` matchers so that the changes to command line
testing are clearer.
After some poking, it would take a more significant change to
restructure the string generation to take less time when run under ASan,
and it's not worth it at the moment.
For future reference, nearly half the time here is in building the
global data structures of random string contents, not in the actual
benchmark functions. If/when we want to improve this, we should switch
to a growing pool of random strings similar to what `SourceGen` uses.
That lets it not allocate the full size of data when just testing that
the benchmark doesn't crash.
I thought about having these benchmarks switch to use `SourceGen`, but
I'd like to keep them stand-alone if easy, and there are some important
differences that would have to be adapted around which wouldn't be
trivial. I'd rather come back in with a better generation strategy than
re-use the source code one here.
These tests typically take 10-20s, but I'm seeing some timeouts
[here](https://github.com/carbon-language/carbon-lang/actions/runs/11899548036/job/33158400417).
This seemed particularly suspicious due to the _absence_ of output
(copied below). That got me looking, and maybe the subprocessing tickles
a cpu bottleneck, so proposing this approach to remove the exec. Even if
this doesn't solve the flakiness, I think it's a simpler implementation.
Note I believe this is intended to work. The `sh` rules rely on shebangs
(as noted at https://bazel.build/reference/be/shell#sh_test), and are
essentially just subprocessing to the input. Note this could've also had
`args` on a `cc_test` rule, but I'd expect the same args to be passed to
`run` where instead the benchmark behavior should be default (and I'm
assuming you'd rather not have args there). Fundamentally this becomes a
symlink:
```
bazel-bin/common/map_benchmark_test -> .../execroot/_main/bazel-out/k8-fastbuild/bin/common/map_benchmark
```
Copying snippet from timeout below:
```
==================== Test output for //common:map_benchmark_test:
/private/var/tmp/_bazel_runner/e591f63ed099023de1f206992dfce127/execroot/_main/bazel-out/darwin_arm64-fastbuild/testlogs/common/map_benchmark_test/test.log
-- Test timed out at 2024-11-18 19:32:13 UTC --
INFO: From Testing //common:map_benchmark_test:
================================================================================
```
This switches `DCHECK` and `FATAL` as well.
The goal is to reduce the code size impact of these assertions so that
we can keep more of them enabled. Currently, the largest cost I see from
`CHECK` is not the actual check or the cold code itself, but actually
the failure to inline trivial functions due to the presence of the cold
code. This means that our goal isn't to reduce apparent code size in the
final binary but the LLVM IR cost assessed for these routines in the
inliner, which closely correlates with code size but is a bit different.
As discussed in #4283, experimentation shows that a single function call
with a minimal number of arguments is the lowest cost model for these.
This is easily achieved with a format-string API that internally uses
`llvm::formatv`. This PR is essentially the `CHECK` version of #4283.
However, the check macros are substantially harder to make work with
both format strings and streaming because they also take a condition.
Also, unexpectedly, I was very successful at devising a regular
expression based automated rewrite from the streaming to the format
string form with only low 10s of manual fixes. This includes compacting
strings broken up across lines, etc. Given how well that went, I've
prepared this PR which just directly switches to the format string API
and migrate everything to use it.
One nice side-effect is that the format string approach ends up greatly
simplifying the implementation here as well.
This is ... *shockingly* effective. Parsing speeds up by more than 3%
with just this change. And checking speeds up by **8%** with this change
alone:
```
BM_CompileAPIFileDenseDecls<Phase::Parse>/256 86.3µs ± 1% 82.9µs ± 1% -3.94% (p=0.000 n=17+19)
BM_CompileAPIFileDenseDecls<Phase::Parse>/1024 431µs ± 1% 415µs ± 1% -3.76% (p=0.000 n=18+19)
BM_CompileAPIFileDenseDecls<Phase::Parse>/4096 1.77ms ± 1% 1.71ms ± 1% -3.18% (p=0.000 n=18+19)
BM_CompileAPIFileDenseDecls<Phase::Parse>/16384 7.44ms ± 1% 7.17ms ± 2% -3.56% (p=0.000 n=18+20)
BM_CompileAPIFileDenseDecls<Phase::Parse>/65536 30.7ms ± 1% 29.7ms ± 1% -3.15% (p=0.000 n=18+20)
BM_CompileAPIFileDenseDecls<Phase::Parse>/262144 131ms ± 1% 127ms ± 1% -2.81% (p=0.000 n=18+18)
BM_CompileAPIFileDenseDecls<Phase::Check>/256 878µs ± 2% 800µs ± 1% -8.91% (p=0.000 n=19+20)
BM_CompileAPIFileDenseDecls<Phase::Check>/1024 1.88ms ± 2% 1.72ms ± 1% -8.56% (p=0.000 n=19+20)
BM_CompileAPIFileDenseDecls<Phase::Check>/4096 5.78ms ± 2% 5.28ms ± 1% -8.70% (p=0.000 n=20+18)
BM_CompileAPIFileDenseDecls<Phase::Check>/16384 21.9ms ± 1% 20.1ms ± 1% -8.02% (p=0.000 n=18+20)
BM_CompileAPIFileDenseDecls<Phase::Check>/65536 90.4ms ± 2% 83.1ms ± 1% -8.04% (p=0.000 n=19+20)
BM_CompileAPIFileDenseDecls<Phase::Check>/262144 381ms ± 2% 352ms ± 1% -7.79% (p=0.000 n=19+19)
```
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com>
The goal is to replace our stream operator APIs with format string APIs
that can be made to have much less impact on inlining and other
optimizations of the performance critical path through the code.
Several experiments show that the most compact representation we can
arrange for is one that calls an uninlined function and passes a minimal
number of arguments to it. It doesn't help to do any work to minimize
the arguments such as building a lambda -- the cost of extra code to
merge the arguments is likely to outweigh the benefit.
Initial experiments showed that switching a hot but uninlined function
to this new API enabled inlining and the subsequent performance
improvement.
This also adds a 'TemplateString` utility that allows using a string
literal as a template parameter. This is useful to remove the format
string itself from the arguments passed to the function by passing it as
a template argument instead.
Currently, support is left in place for both APIs because with
`CARBON_VLOG` we can detect whether or not any message was provided
expecting a format string. This should allow incrementally migrating
code to this API. I've added some test coverage in this PR, but I'll
separate out any switching of parts of the codebase over.
The goal is to eventually replace all the usages and remove the
streaming support entirely.
This PR doesn't update `CARBON_CHECK` in the same way because it is
substantially more complex to switch. I have a few experimental PRs
looking at that and will discuss how best to approach this with the
specific challenges check presents separately. But the goal is for all
of the macro-based output APIs to move to format strings rather than
streams.
---------
Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Consolidates both main libraries into `//testing/base`, and factors out
the exe path handling for benchmarks and unit tests into a common
library to remove duplication. Refactors how that logic is managed to be
cleaner and avoid a confusing bool that came up in code review.
Updates all the tests and benchmarks that use these. I still need to
update other benchmarks to use the same main, but I wanted to keep this
PR somewhat minimal.
This also fixes a bug noticed in passing that the compilation benchmark
didn't have the required dependency on the benchmark library itself,
just the benchmark main library.
The big addition here is a very, very rough and very early skeleton of a
source code generator framework. This builds upon the lexers identifier
synthesis logic, improving on its framework and wiring it up with the
most rudimentary of source file generation. This is just enough to
roughly replicate my "big API file" source code benchmarks.
The source generation works *very* hard to both vary the structure and
content of the source as much as possible while ensuring the same
*total* amount of each construct is in use, from bytes in identifiers to
line breaks, parameters, etc. This lets us generate randomly structure
inputs that should consistently take the exact same amount of total work
to compile.
The complex identifier synthesis logic from the lexer's benchmark is
moved over here and the lexer uses APIs in the source generator for
identifiers. The other source synthesis in the lexer's benchmark isn't
yet moved over, but should likely be slowly absorbed here as it can be
refactored into a more principled and re-usable form. Some bits may stay
of course if they're just too lexer-specific.
Next, this adds a simple end-to-end compile benchmark for the driver
that directly and much more clearly reproduces all the measurements I've
done manually up until now. It should also be easy to extend to more
patterns over time as we add support to the source generator to produce
those patterns.
Last but not least, I've added a tiny CLI to the source generator so
that you can generate source code manually. This is especially nice for
generating demo source code to actually run through the driver or look
at in an editor. The CLI can also generate C++ source code which lets us
do some minimal comparative benchmarking between Carbon and C++/Clang.
There are huge number of TODOs in the source generation framework. This
is going to be a large ongoing effort I suspect.
There are also a bunch of rough edges I've left to try and get this out
for review sooner. I've left TODOs for refactorings that really need to
be done here, but hoping these can maybe be follow-ups. If not, please
flag and I'll try to layer them on here.
Sample compile benchmark output, nicely showing where we are w.r.t. our
goal speeds (2x behind on lex and check, 5x on parse) at least on a
recent AMD server CPU:
```
------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations Lines
------------------------------------------------------------------------------------------------------
BM_CompileAPIFileDenseDecls<Phase::Lex>/256 29420 ns 29419 ns 22860 6.62847M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/1024 146130 ns 146128 ns 4840 6.69959M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/4096 601584 ns 601577 ns 1020 6.69573M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/16384 2547578 ns 2547313 ns 280 6.404M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/65536 10816591 ns 10816389 ns 80 6.05193M/s
BM_CompileAPIFileDenseDecls<Phase::Lex>/262144 52191320 ns 52189828 ns 20 5.02261M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/256 101706 ns 101698 ns 6900 1.91745M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/1024 512161 ns 512162 ns 1380 1.9115M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/4096 2078426 ns 2078430 ns 340 1.938M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/16384 8795786 ns 8795583 ns 100 1.85468M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/65536 35073596 ns 35072973 ns 20 1.86639M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/262144 151100688 ns 151097370 ns 20 1.73483M/s
BM_CompileAPIFileDenseDecls<Phase::Check>/256 957059 ns 957049 ns 740 203.751k/s
BM_CompileAPIFileDenseDecls<Phase::Check>/1024 1956134 ns 1955985 ns 360 500.515k/s
BM_CompileAPIFileDenseDecls<Phase::Check>/4096 5797864 ns 5797417 ns 120 694.792k/s
BM_CompileAPIFileDenseDecls<Phase::Check>/16384 21219608 ns 21217584 ns 40 768.843k/s
BM_CompileAPIFileDenseDecls<Phase::Check>/65536 96311116 ns 96302334 ns 20 679.734k/s
BM_CompileAPIFileDenseDecls<Phase::Check>/262144 371637963 ns 371609964 ns 20 705.387k/s
```
Lest someone think this is *bad*, the fact that we're already within 2x
of our rather audacious goals makes me quite happy. =D
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Based on discussion around the region handling in generic_region_stack,
create a generic structure for the stack-of-vectors support. I also want
to add this to InstBlockStack, but that's a little more complex due to
GlobalInit, so cutting a PR here to check with review.
My work here is how I noticed #4099; I want to be sure that I'm correct
about the issue, but it's the difference between being able to use
PeekArray or not.
Note in scope_stack.h, I believe we could remove next_compile_time_index
and make it just based on elements_size(). However, I want to verify
with you before I make further changes there.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
This injects a customization point for hashtable-specific equality
testing that the key context uses by default. While this is rarely
needed, there are LLVM types where it is necessary and it seems a good
general tool to have to avoid unnecessary complexity from custom key
contexts when a simple customization of equality is all that is
required.
This also adds a CRTP mixin for implementing a common pattern of key
contexts where the context provides translation of some key types into
another type, potentially using state. Rather than having to implement
the entire key context API, code can derive from this template and
simply provide a set of overloads for the types it wants to translate.
Any key types used which can be passed to one of those overloads will
get translated before following the same logic as the default key
context. While this updates the only usage so far of this pattern, a
subsequent PR will add several more users making the pattern worth
abstracting here.
This adds a defined Carbon version to the Bazel build and codebase that
can be used both to implement features like version checks and to report
a meaningful version on the command line. This replaces a hard-coded
string and a TODO in the driver.
As part of this, it adds support for defining the version in Bazel, and
special build flags for overriding relevant parts such as the
pre-release marker used. The exact structure and meaning of our version
string, including the pre-release parts, is implemented here in line
with the draft proposal:
https://docs.google.com/document/d/11S5VAPe5Pm_BZPlajWrqDDVr9qc7-7tS2VshqO0wWkk/edit?resourcekey=0-2YFC9Uvl4puuDnWlr2MmYw
This also introduces a workspace status command to the repository to
extract the git commit SHA and other information when building, and the
logic to stamp that into binaries as part of the version string when
useful. The technique used leverages weak symbols with whole archive
linking to allow a link-time override of unstamped data with stamped
data in the leaf executable. This makes building with `--stamp` a
reasonable default, especially for development builds. The CI system is
explicitly opted out of this as there it has no benefit.
Last but not least, all of these are wired into the install rules so
that we build installable packages with the version number in a
conventional place in the directory and filename.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This uses a header-only extraction of the Boost unordered hashtable
project to allow a trivial Bazel build and for us to benchmark against
it effectively.
The hash table design is heavily based on Abseil's ["Swiss
Tables"][swiss-tables] design. It uses an array of bytes storing
metadata about each entry and an array of entries where each is a pair
of key and value. The metadata byte consists of 7-bits of hash of the
key (distinct from the bits used to index the table), and one bit
indicating the presence of a special entry -- either empty or deleted.
[swiss-tables]: https://abseil.io/about/design/swisstables
There are a large range of optimizations and other nuanced aspects of
this hash table design and implementation, a good point to understand
that context is `raw_hashtable.h` which has an overview of the design
and references to various other files for relevant details.
---------
Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com>
This also factors out the code for doing this location from the driver
to a tiny helper library.
The motivation for this is letting tests locate data files like the
prelude or other files needed by the toolchain more easily. A subsequent
PR will use this heavily in the Clang runner and related logic for
example.
This largely reproduces the upstream one, but has a few advantages:
1) It initializes LLVM which can be important if we use the backends.
2) It parses commandline flags with Abseil allowing the use of Abseil
flags in benchmarks.
3) It flips the default for tabular display of results which we use
pretty heavily.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Co-authored-by: Carbon Infra Bot <carbon-external-infra@google.com>
In parse, form a list of methods that are defined inline, tracking where
they start, where they end, and which other inline methods are nested
within them.
In check, when we reach an inline method body, skip it and add it to a
worklist to be processed later. We also track when we reach the start
and end of a context in which inline method bodies are deferred, so that
we know when to replay the bodies.
When suspending a function definition to be processed later, the
`DeclNameStack` entry is moved to separate storage, including popping
the corresponding scopes from the scope stack and removing the
corresponding lexical names from lexical lookup. Later, when we return
to the function and parse its definition, the `DeclNameStack` entry is
restored. The same is done when we reach the end of a nested context
that can have inline methods, so that we can reenter the nested scope
before processing its members.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
I'd kept these in to separate the bazel module update from the BUILD
file changes, then forgot about it. I think all of these can be cleanly
removed now. I think it's something we should clean up for consistency
with the bazel central repository names; I think it's best to reduce
that divergence.
llvm_zlib and llvm_zstd remain because of how llvm depends on the
particular names.
Per their documentation, the `llvm::Initialize*` functions are only
supposed to be called by the main program, not by a library like
toolchain/codegen. Fixes a hang due to a data race in multithreaded
autoupdate.
Add a utility class `Carbon::InitLLVM` to do the common LLVM
initialization shared by all Carbon tools, optionally including
initializing the LLVM targets. Because the LLVM targets add a lot of
binary size, only initialize them for binaries that opt in by depending
on a new target `//common:all_llvm_targets`.
Also fix `//explorer:file_test` and `//explorer:file_test.trace` to
share a binary rather than linking an identical binary twice.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Running `bazel test //...` reported:
```
Test execution time outside of range for MODERATE tests.
Consider setting timeout="short" or size="small".
```
This change adds size="small" to avoid such warnings being reported.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Replace `SemIR::Node::GetAsFoo` and `SemIR::Node::Foo::Make` with
`SemIR::Foo` class that represents a particular kind of node, with named
fields.
Rename `SemIR::IntegerLiteral` and `SemIR::RealLiteral` to
`IntegerValue` / `RealValue` to better reflect their purpose and avoid a
name collision with the corresponding `SemIR` node kinds.
Remove `NodeKind::Invalid` and the `SemIR::Node` default constructor
entirely, as they were not used for anything.
This file group exists to allow a `genquery` rule and a Python test to
verify our non-test dependency graph. We don't actually need to build
the binaries in the file group as part of that. The `genquery` rule
seems to do the right thing -- building it directly doesn't cause the
binaries in the group to be built. But without a manual tag, the group
itself is part of `:all` and thus part of `//...` and part of the rules
that will be built even with PR #3106. A consequence is that any change
to the toolchain causes several other binaries to be built as well
because this file group is in the impacted set. Making it manual should
avoid all of this, and without breaking the actual use from `genquery`.
For example, before this change, in a fully cached build after a `bazel
clean`:
```
> bazel test //bazel/check_deps:all
INFO: Invocation ID: 2d83ebee-4c00-425d-be33-23f42b079614
INFO: Analyzed 3 targets (103 packages loaded, 7137 targets configured).
INFO: Found 2 targets and 1 test target...
INFO: Elapsed time: 4.081s, Critical Path: 2.61s
INFO: 3111 processes: 2796 disk cache hit, 315 internal.
INFO: Build completed successfully, 3111 total actions
```
After this change:
```
> bazel test //bazel/check_deps:al
INFO: Invocation ID: c94089e8-a420-4d3c-9902-134e6b55b297
INFO: Analyzed 2 targets (92 packages loaded, 568 targets configured).
INFO: Found 1 target and 1 test target...
INFO: Elapsed time: 0.700s, Critical Path: 0.01s
INFO: 7 processes: 2 disk cache hit, 5 internal.
INFO: Build completed successfully, 7 total actions
```
While here, re-generate the file group, and fix several issues it
uncovers: mark test utilities as `testonly` and update our LLVM package
allowlist to include `clangd`'s package.
With the toolchain splitting namespaces, ostream.h's `operator<<`
templates aren't reliably found with name lookup, likely due to the loss
of associated namespaces (zygoloid commented on this at
https://github.com/carbon-language/carbon-lang/pull/3161#discussion_r1307941999).
This is especially a barrier to moving the lex files into `Carbon::Lex`;
versus other parts of the toolchain, they contain more printable types
which are used cross-namespace, including `Carbon::Testing`. As a
consequence, I'm looking at migrating ostream.h to a more reliable
approach that doesn't rely as much on everything being in the `Carbon`
namespace.
Continuing on #3070.
I moved ParseTree::Node to just Parse::Node, versus Parse::Tree::Node.
Other name changes are just removing "Parse" or "Parser" prefixes.
In EnumBase, I'm directly defining operator<< because the ostream.h
approach just isn't working, not for either of Parse::State nor
Parse::NodeKind. Errors look like:
```toolchain/parser/parser_context.cpp:449:34: error: invalid operands to binary expression ('llvm::raw_ostream' and 'const Carbon::Parse::State')
output << "\t" << i << ".\t" << entry.state;
~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~
```
The expected template in `Carbon::` is not in the error list; I only see
the:
```
./common/ostream.h:112:6: note: candidate template ignored: requirement 'std::is_base_of_v<std::ostream, llvm::raw_ostream>' was not satisfied [with S = llvm::raw_ostream, T = Carbon::Parse::State]
auto operator<<(S& standard_out, const T& value) -> S& {
^
```
I'm still prodding at this, but not seeing an obvious fix.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Jon Ross-Perkins
Chandler Carruth
Boaz Brickner
Jonathan B. CoeandChandler Carruth