At present, I think if we crash from multiple threads in parallel, it
can lead to the stack trace not being printed out. Using
CrashRecoveryContext here seems to more successfully print a stack trace
on errors, which I'm hoping will ease debugging.
i.e., before:
```
-----------------------------------------------------------------------------
.Please report issues to https://github.com/carbon-language/carbon-lang/issues and include the crash backtrace.
Stack dump:
0. performing autoupdate for toolchain/check/testdata/alias/no_prelude/import_order.carbon
1. Program arguments: compile --phase=check --dump-sem-ir --no-prelude-import --exclude-dump-file-prefix=/usr/local/google/home/jperkins/.cache/bazel/_bazel_jper.kins/85deb7d9d96f7e0e80b42618a55969d7/execroot/_main/bazel-out/k8-fastbuild/b.in/toolchain/install/prefix_root/lib/carbon/../../lib/carbon/core a.carbon b.carbon
.CHECK failure at ./toolchain/sem_ir/ids.h:140: is_valid()
CHECK failure at ./toolchain/sem_ir/ids.h:140: is_valid()
external/bazel_tools/tools/test/test-setup.sh: line 328: 1151859 Aborted "${TEST_PATH}" "$@" 2>&1
```
(EOF)
after:
```
-----------------------------------------------------------------------------
Please report issues to https://github.com/carbon-language/carbon-lang/issues and include the crash backtrace.
Stack dump:
0. performing autoupdate for toolchain/check/testdata/alias/no_prelude/import_order.carbon
1. Program arguments: compile --phase=check --dump-sem-ir --no-prelude-import --exclude-dump-file-prefix=/usr/local/google/home/jperkins/.cache/bazel/_bazel_jperkins/85deb7d9d96f7e0e80b42618a55969d7/execroot/_main/bazel-out/k8-fastbuild/bin/toolchain/install/prefix_root/lib/carbon/../../lib/carbon/core a.carbon b.carbon
..CHECK failure at ./toolchain/sem_ir/ids.h:140: is_valid()
CHECK failure at ./toolchain/sem_ir/ids.h:140: is_valid()
......CHECK failure at ./toolchain/sem_ir/ids.h:140: is_valid()
CHECK failure at ./toolchain/sem_ir/ids.h:140: is_valid()
.....................................................................................CHECK failure at ./toolchain/sem_ir/ids.h:140: is_valid()
.CHECK failure at ./toolchain/sem_ir/ids.h:140: is_valid()
............................ #0 0x000056045ee22d7d llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/usr/local/google/home/jperkins/.cache/bazel/_bazel_jperkins/85deb7d9d96f7e0e80b42618a55969d7/execroot/_main/bazel-out/k8-fastbuild/bin/toolchain/testing/file_test.runfiles/_main/toolchain/testing/file_test+0x731dd7d)
```
(elided the full stack trace)
This uses a heuristic reserve to greatly reduce hashtable growth of the
identifiers hashtable. The design of the hashtable itself is optimized
around compact memory use and is especially slow to grow and so this has
an outsized impact.
The heuristic was computed using `scripts/source_stats.py` and looking
at C++ codebases. We may want to periodically re-evaluate it as Carbon
code emerges and we have better data on its distributions of tokens.
This also required fixing the `Reserve` method on `CanonicalValueStore`
that wasn't actually used anywhere and so didn't even compile correctly.
I added it to the relevant unit test so it is at least compiled locally
to its definition.
This better follows the principle that types are simply constants of
type `type`, and allows more uniform treatment of types as just another
kind of constant from generics handling.
Use a hash table to map from `TypeId` to information about the complete
type. This makes basic operations on types a bit simpler, and operations
that actually need to access the complete class information a bit more
complex.
Changes crash messages to start printing verbose forms of instructions,
rather than just the ID. Fixes some indentation issues with stacks. Also
switches unexpected inst formatting, because now there are lots, and
it'd be helpful to know where they are.
This uses a pimpl pattern for Formatter due to the number of member
functions on Formatter. Maybe we should refactor that, but this didn't
feel like a good place to do so.
Note, I have two concerns about this change... to note them here, to
make sure others are considering them when evaluating the
implementation:
1. Some instructions are very verbose to print, as evidenced by the
fn_decl printing (which includes function params) or scope printing
(which includes scope members).
- I'm not sure whether there's a way to simply reduce this, as it seems
essential to the requested printing of instructions.
- Long-term, we may at least want to limit the number of lines printed
here. However, I've already spent a fair amount of time here and I think
it's in a good state to evaluate.
2. Increased complexity in the crash handler may result in crash
messages failing to generate.
- For example, a crash in Formatter (and its deps, such as InstNamer or
location handling) prevents a stack from being printed. I'm pretty sure
I've written crashes in Formatter before.
Here's an example crash snippet (generated by adding a crash inside
`return` handling) before:
```
2. NodeStack:
0. FunctionDefinitionStart -> function2
1. ReturnStatementStart -> no value
2. IntLiteral -> inst+26
inst_block_stack_:
0. block<invalid> {inst+0, inst+1, inst+2, inst+23}
1. block9 {inst+26}
param_and_arg_refs_stack:
args_type_info_stack_:
```
And after:
```
2. Check::Context
NodeStack:
0. FunctionDefinitionStart: function2
1. ReturnStatementStart: no value
2. IntLiteral:
unexpected.inst+26.loc12_10: i32 = int_literal 0 [template = constants.%.2]
inst_block_stack_:
0. block<invalid> {
package: <namespace> = namespace [template] {
.Core = unexpected.inst+2
.F = unexpected.inst+23.loc11_22
}
unexpected.inst+1 = import Core
unexpected.inst+2: <namespace> = namespace unexpected.inst+1, [template] {}
unexpected.inst+23.loc11_22: %F.type = fn_decl @F [template = constants.%F] {
unexpected.inst+9.loc11_9: init type = call constants.%Bool() [template = bool]
unexpected.inst+10.loc11_9: type = value_of_initializer unexpected.inst+9.loc11_9 [template = bool]
unexpected.inst+11.loc11_9: type = converted unexpected.inst+9.loc11_9, unexpected.inst+10.loc11_9 [template = bool]
unexpected.inst+12.loc11_6: bool = param b
@F.%b: bool = bind_name b, unexpected.inst+12.loc11_6
unexpected.inst+19.loc11_18: init type = call constants.%Int32() [template = i32]
unexpected.inst+20.loc11_18: type = value_of_initializer unexpected.inst+19.loc11_18 [template = i32]
unexpected.inst+21.loc11_18: type = converted unexpected.inst+19.loc11_18, unexpected.inst+20.loc11_18 [template = i32]
@F.%return: ref i32 = var <return slot>
}
}
1. block9 {
unexpected.inst+26.loc12_10: i32 = int_literal 0 [template = constants.%.2]
}
param_and_arg_refs_stack:
args_type_info_stack_:
```
This improve the toolchain's performance by about 10%.
It will also allow us to leverage TCMalloc's extensions to do heap
profiling and get other information about how efficiently we're using
the heap.
Note that currently this causes all of our builds to produce a warning
due to an issue with `rules_python` and multiple modules registering
python toolchains:
https://github.com/bazelbuild/rules_python/issues/1818
This is also only enabled on Linux as there is no support for other OSes
at the moment.
Most of this is enabled by default, but there is some that needs an
explicit flag.
Avoiding `-Wnon-virtual-dtor` itself for now because that warning can
require changes that carry overhead such as having an extra, unused
destructor entry in the vtable. Hopefully we don't have too many folks
who need our code to be `-Wnon-virtual-dtor` clean.
Querying a local constant with an invalid instruction triggers a crash,
this will prevent self_param_id to be used unless it is defined.
Closes#4071 and #4080
This will only fix the crash. The scenario where a declaration-only
interface is imported then defined still gives an error message, but it
won't crash this time.
When evaluating in a generic context, a constant with a symbolic type
might evaluate to a constant with a concrete type (or a more specific
symbolic type). This can't actually happen yet given the current state
of the toolchain, as far as I can determine, so this is more just a
refactoring for now, but will be relied upon by future generics work.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Carbon Infra Bot <carbon-external-infra@google.com>
When forming a specific (previously called a generic instance), evaluate
the eval block of the generic to determine the values of any constants
used in that specific. The majority of the work here is updating
eval.cpp so that it can use the results of prior evaluations in the same
block when computing later values.
Include the computed results in the formatted SemIR output.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
I was considering making a "/bazel" alias for this, but I'm really on
the fence about whether that's a good choice. However, I do think this
is good to have to simplify installs.
Fixes#3071 and #3896
Proposal for how Carbon version numbers work:
- A single version across language, standard library, compiler, linker,
etc.
- Semantic Versioning (SemVer) based
- Details of how SemVer criteria for major, minor, and patch should
apply to
Carbon
- Details of how we will operate before 1.0 and how this connects to
Carbon's
milestones
- Directional guidance for future work including post-1.0 versions, LTS
versions, and standardization
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This is for consistency with #4120. Similar to that, we can use
overloads on the typed NodeId rather than individually named handlers.
There isn't the same caller benefit here though, since the calls from
check.cpp are already boilerplate.
Adds scripts/calculate_release_shas.py to print the versions, and
updates tool versions. Consolidates target-determinator logic with the
other tool logic.
This is also intended to make it easier to add more tools.
Also makes an alias so that this is easier to find. Verified that
running the alias still finds prelude files.
Note the actual target is printed when building (although a symlink is
also created, using that symlink confuses file-finding).
```
╚╡bazel build :carbon
...
Target //toolchain/install:prefix_root/bin/carbon up-to-date:
bazel-bin/toolchain/install/prefix_root/bin/carbon
```
We can't use the instruction from the generic directly, because it
doesn't have the right constant value. Instead add an instruction that
models the transition from the constant value in the generic to the
constant value in the generic instance.
Also start associating the self generic instance with unqualified
lookups that find results in an enclosing generic, so that we track the
information necessary to create the new instruction.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Renames Lower::Handle* to Lower::LowerFunctionInst. This allows writing
a templated handler for instructions, moving code out of the macro
expansion and removing some of the redundancy in things like
`HandleAddrOf(..., SemIR::AddrOf inst)`
This adds `DefinitionInfo` for `Define`-based configuration so that
parameters are optional. It also makes it easier to provide the
equivalent functions on both `Definition` and `Define`.
A common pattern used here is to change from a `switch` with in-line
`case`s to instead have `case`s that call an overloaded function. What's
happening here is that the instruction type is used to select an
overload, and if an overload is not defined, a compiler error would
result. Meanwhile, clusters of overloads are being defined using
`requires`-based templating, so that equivalent implementations are not
copied. This addresses a limitation of a vanilla `switch` approach where
it's hard to have redundant cases using conditional logic, while also
getting compiler errors when adding new `InstKind` entries, which had
been a significant part of why we used macros previously.
This starts hitting some odd clang-format edge cases causing
`CARBON_KIND_SWITCH(inst){` (missing space), which I haven't seen
before. Adding `CARBON_KIND_SWITCH` to .clang-format works around it.
This is to remove all the FatalIfEncountered handlers in handle.cpp.
They just feel like noise when reading the file. Plus it's one less bit
of boilerplate to add for instructions that don't lower.
Note that I left HandleParam/HandleAddrPattern. I'd be happy to change
those to just set lowered=false too, but was hesitant to given the
separate logic.
Also, I'm separately considering migrating the macro logic into similar
constexpr things. If I do, I might switch Define to take in a struct.
But for how this particular parameter works, the overload felt
reasonable, particularly since is_lowered is not used in combination
with TerminatorKind.
I'm trying to increase the distinction between BuiltinKind and
BuiltinFunctionKind. BuiltinKind is for instructions,
BuiltinFunctionKind is for function definitions. To get to this point,
I'm doing a few changes:
- BuiltinKind -> BuiltinInstKind
- builtin_kind.* -> builtin_inst_kind.*: filename consistency
- Builtin -> BuiltinInst: mainly for consistency with the above
- Builtin::builtin_kind -> BuiltinInst::builtin_inst_kind: somewhat
repetitive but seems like a consistent edit
- Function::builtin_kind -> Function::builtin_function_kind: seems a
useful distinction
I'm leaving alone things like (and mentioning in case there's a desire
for more renames):
- InstId::BuiltinError, InstId::ForBuiltin: these I think are more
apparent because they're directly associated with Inst.
- GetBuiltinICmpPredicate in lowering: maybe builtin function handling
should be in its own file, but these local names don't feel problematic
to me.
- GetBuiltinType, BuildBuiltinValueRepr, PerformBuiltinIntComparison:
similar to the above, names don't feel too problematic
Require types into which qualified lookup is performed to be completely
defined. Eventually this will trigger substitution into the definition
for generic types.
Also reorder links so that the future/most recent items are first,
putting older videos lower down.
Noting the lightning talk because it's there, but also that it's a
lightning talk because most of the others are 40m/1h, so 5m is very
short by comparison (I was tempted to label the length of everything,
but not sure how valuable others view that).
For each generic, build a list of instructions describing the
computations we need to do when resolving an instance of the generic:
this is a list of the instance-specific constants and types that the
generic uses. Another way of viewing this list is as a block of Carbon
SemIR code that is evaluated in order to form an instance of the generic
-- this is referenced in the code as the "eval block" for the generic.
For each instruction in the generic whose type or value is a symbolic
constant, replace that type or constant value with a symbolic reference
that says "to find the actual type or value, look at index N in the list
of values for the generic instance".
For an instruction with a symbolic constant value, we can just add that
instruction to our list. For an instruction with a symbolic constant
type, however, we may not have a corresponding instruction computing the
type within the generic and may need to build a new instruction, but
will reuse one where possible. In the case where we build a new
instruction, we use the existing substitution code to build the type
within the eval block.
For now, this transformation is only done in the declaration region of
the generic, not in the definition region. Also, we map back from the
symbolic references to the underlying constant value in a few places
where we will eventually need to do a lookup into a generic instance, in
order to avoid regressing the tests.
The use of ArrayStack here is intended to simplify the logic, and also
make better use of the inst heap allocations. Prior changes #4101 and
#4103 removed the less related logic from InstBlockStack, although #4103
is the actual part that blocked using ArrayStack.
BTW, note the PrintForStackDump implementation was incorrect because it
didn't apply size_. This simplification fixes the issue.
This executes on a TODO in AddImportRef to add instructions to their own
block instead of the File block. This has an important consequence of
removing a pattern from InstBlockStack that added to blocks not
currently at the top, cleaning up an issue for ArrayStack. The delta
here is then mostly in different formatting of the import refs, a
consequence of the separation.
Creates a `GlobalInit` class for storing relevant values, pulling
functions off `InstBlockStack` and `Context`. Adds a `Context` pointer
just so that it doesn't need to be passed in on each call (`Finalize` in
particular uses several members).
Note we have several different `InstBlockStack` instances, so several
copies of the relevant members were simply unused.
By adding an `ImportDecl` instruction, this creates something that can
be referenced through `ImportIRInst`.
packages/no_prelude/implicit_imports_entities.carbon is getting a test
of this (import_conflict and import_conflict_reverse).
Also re-packs ImportIR from 24 bytes to 16 on 64-bit, since I'm touching
everywhere that makes one anyways.
Based on discussion around the region handling in generic_region_stack,
create a generic structure for the stack-of-vectors support. I also want
to add this to InstBlockStack, but that's a little more complex due to
GlobalInit, so cutting a PR here to check with review.
My work here is how I noticed #4099; I want to be sure that I'm correct
about the issue, but it's the difference between being able to use
PeekArray or not.
Note in scope_stack.h, I believe we could remove next_compile_time_index
and make it just based on elements_size(). However, I want to verify
with you before I make further changes there.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
This undoes a previous change to unify them, and I think at my advice.
=[ Sorry about that, I think I was just wrong.
Specifically, I think I had suggested that it would be more efficient to
have a single shared hashtable of strings. The more I look at profiles
of the toolchain, the less likely that seems. Specifically for
identifiers and string literals it seems especially problematic.
Using a single, joint hashtable is likely a good idea when all of the
different querying code paths are equally likely, the strings follow the
same distribution of sizes, and either there is no clustering of access
to different sets of strings or none of the sets are meaningfully small
enough to fit into a lower level of resident cache.
I think essentially none of these predicates actually hold for
identifiers vs. string literals:
- Identifiers are *much* more hot
- They have wildly different size distributions.
- The access patterns are very clustered
Sorry for the misleading advice on that one.
While splitting them, I've worked to simplify the code a bit by building
a way to have the `StringRef` holding canonical value stores not require
specializations, and so we get a pretty large code cleanup in the
process here.
This works to leverage the capabilities of the hashtable as much as
possible, for example using the key context in the value stores.
However, there may still be opportunities to refactor more deeply and
use the functionality even better. Hopefully this is at least
a reasonable start and gets us a clean baseline.
On an Arm M1, this is a 15% improvement on my large lexing stress test,
but ends up a wash on my x86-64 server. This is a smaller benefit than
I expected, and it's because we're using a set-of-IDs and looking up
values with a key context for things like identifiers. This pattern has
a surprising tradeoff. The new hashtable uses significantly less memory,
a 10% peak RSS reduction just from the hashtable change. But indirecting
through the vector of values makes growing the hashtable dramatically
less cache-friendly: it causes growth to randomly access every key when
rehashing. On x86, everything gained by the faster hashtable is lost in
even slower growth. And even on Arm, this eats into the benefits.
But I have a plan to tweak how identifiers specifically work to avoid
most of the growth, and so I suspect this is the right tradeoff on the
whole. It gives us significant working set size reduction and we can
likely avoid the regressed operation (growth with rehash) in most cases
by clever reserving and if necessary by adding a hash caching layer to
the table infrastructure.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
I found this through inspection, looking at an array stack data type.
Tests pass either way, not sure what a good test would be for
regressions (tests do fail if the size doesn't match, but either
approach gets an appropriate size). But this is followed by
`truncate(remaining_compile_time_bindings)`, so it seems like
`drop_back` is a better match than `drop_front`.
When forming a `ConstantId` for a symbolic constant, add storage to
track the generic in which the constant was formed and the index within
that generic. These fields are not yet populated.
This injects a customization point for hashtable-specific equality
testing that the key context uses by default. While this is rarely
needed, there are LLVM types where it is necessary and it seems a good
general tool to have to avoid unnecessary complexity from custom key
contexts when a simple customization of equality is all that is
required.
This also adds a CRTP mixin for implementing a common pattern of key
contexts where the context provides translation of some key types into
another type, potentially using state. Rather than having to implement
the entire key context API, code can derive from this template and
simply provide a set of overloads for the types it wants to translate.
Any key types used which can be passed to one of those overloads will
get translated before following the same logic as the default key
context. While this updates the only usage so far of this pattern, a
subsequent PR will add several more users making the pattern worth
abstracting here.
LLVM's `APInt` and `APFloat` need specialized handling to be used
effectively in hashtables. We can't inject overrides into LLVM so we
need to handle them in our hashing routine.
There were also problematic limits on hashing pairs and tuples. First,
the unique-object-representation hashing of pairs was more restricted
than tuples which was a problematic asymmetry and isn't needed. But the
larger issue is that we didn't support recursively hashing when
necessary. That requires a careful predicate to avoid infinite recursion
but lets us handle important use cases for hashtables with a tuple as a
key.
Also added support for hashing arrays that recurse in addition to arrays
where we can hash the raw storage, and added overloads to redirect to
common array handling from various array-like types.
Last but not least, re-worked the constraint model for hashing as raw
data to not override custom hashing functions.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Co-authored-by: Carbon Infra Bot <carbon-external-infra@google.com>
When checking a declaration or definition of a generic, track a list of
created instructions that depend on the generic's parameters in some
way, along with information on how they depend on the parameters. This
will eventually be used to determine what information we need to compute
when creating instances of the generic, but for now we're just building
the list.
Information is tracked separately for the declaration region and the
definition region of the generic, because in general these may be first
provided in separate declarations, and they should be substituted into
at different times.
This is awkward to track... Probably it would be best done by tracking
the ratio of unique IDs to lines as a floating point and plot them and
see what a best fit distribution curve looks like. But none of the
histogram printing or stats tracking stuff already in use here makes it
easy to do any of that...
So this does what I hope is a reasonable rough approximation by counting
the ceiling of unique identifiers per 10 lines of code, and plotting
that discreet histogram. Shape of the histogram is exactly what I would
expect: one centered distribution, vaguely normal looking. And the
center for a bunch of different codebases, including our toolchain, is
exactly at 5, which would mean 0.5 unique IDs per line. And the
distribution is pretty reliably bounded above by 10 or 1 unique ID per
line. Which almost seems to clean to be true? Slightly worried about
confirmation bias making me think this code is working because the
results look so pretty.
Here is the output for the toolchain:
```
## Unique IDs per 10 lines ## (median: 6)
2 ids [ 2] █▎
3 ids [19] ████████████▎
4 ids [32] ████████████████████▋
5 ids [55] ███████████████████████████████████▌
6 ids [62] ████████████████████████████████████████
7 ids [44] ████████████████████████████▍
8 ids [22] ██████████████▎
9 ids [11] ███████▏
10 ids [ 7] ████▌
11 ids [ 2] █▎
```
And here is the output for llvm-project/*/{lib,include} (to avoid
tests):
```
# Unique IDs per 10 lines ## (median: 5)
1 ids [ 29] ▍
2 ids [ 282] ███▊
3 ids [1492] ███████████████████▉
4 ids [2674] ███████████████████████████████████▌
5 ids [3011] ████████████████████████████████████████
6 ids [2267] ██████████████████████████████▏
7 ids [1549] ████████████████████▋
8 ids [ 817] ██████████▉
9 ids [ 301] ████
10 ids [ 98] █▎
11 ids [ 61] ▊
12 ids [ 50] ▋
13 ids [ 25] ▍
14 ids [ 33] ▌
15 ids [ 14] ▏
16 ids [ 15] ▎
17 ids [ 9] ▏
18 ids [ 8] ▏
19 ids [ 12] ▏
20 ids [ 15] ▎
21 ids [ 3]
22 ids [ 8] ▏
23 ids [ 3]
24 ids [ 3]
25 ids [ 6] ▏
26 ids [ 0]
27 ids [ 2]
28 ids [ 0]
29 ids [ 0]
30 ids [ 3]
31 ids [ 1]
32 ids [ 1]
```
Jon Ross-Perkins
Chandler Carruth
Richard Smith
Gıyaseddin Tanrıkulu
dependabot[bot]