I was considering making a "/bazel" alias for this, but I'm really on
the fence about whether that's a good choice. However, I do think this
is good to have to simplify installs.
Fixes#3071 and #3896
Proposal for how Carbon version numbers work:
- A single version across language, standard library, compiler, linker,
etc.
- Semantic Versioning (SemVer) based
- Details of how SemVer criteria for major, minor, and patch should
apply to
Carbon
- Details of how we will operate before 1.0 and how this connects to
Carbon's
milestones
- Directional guidance for future work including post-1.0 versions, LTS
versions, and standardization
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This is for consistency with #4120. Similar to that, we can use
overloads on the typed NodeId rather than individually named handlers.
There isn't the same caller benefit here though, since the calls from
check.cpp are already boilerplate.
Adds scripts/calculate_release_shas.py to print the versions, and
updates tool versions. Consolidates target-determinator logic with the
other tool logic.
This is also intended to make it easier to add more tools.
Also makes an alias so that this is easier to find. Verified that
running the alias still finds prelude files.
Note the actual target is printed when building (although a symlink is
also created, using that symlink confuses file-finding).
```
╚╡bazel build :carbon
...
Target //toolchain/install:prefix_root/bin/carbon up-to-date:
bazel-bin/toolchain/install/prefix_root/bin/carbon
```
We can't use the instruction from the generic directly, because it
doesn't have the right constant value. Instead add an instruction that
models the transition from the constant value in the generic to the
constant value in the generic instance.
Also start associating the self generic instance with unqualified
lookups that find results in an enclosing generic, so that we track the
information necessary to create the new instruction.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Renames Lower::Handle* to Lower::LowerFunctionInst. This allows writing
a templated handler for instructions, moving code out of the macro
expansion and removing some of the redundancy in things like
`HandleAddrOf(..., SemIR::AddrOf inst)`
This adds `DefinitionInfo` for `Define`-based configuration so that
parameters are optional. It also makes it easier to provide the
equivalent functions on both `Definition` and `Define`.
A common pattern used here is to change from a `switch` with in-line
`case`s to instead have `case`s that call an overloaded function. What's
happening here is that the instruction type is used to select an
overload, and if an overload is not defined, a compiler error would
result. Meanwhile, clusters of overloads are being defined using
`requires`-based templating, so that equivalent implementations are not
copied. This addresses a limitation of a vanilla `switch` approach where
it's hard to have redundant cases using conditional logic, while also
getting compiler errors when adding new `InstKind` entries, which had
been a significant part of why we used macros previously.
This starts hitting some odd clang-format edge cases causing
`CARBON_KIND_SWITCH(inst){` (missing space), which I haven't seen
before. Adding `CARBON_KIND_SWITCH` to .clang-format works around it.
This is to remove all the FatalIfEncountered handlers in handle.cpp.
They just feel like noise when reading the file. Plus it's one less bit
of boilerplate to add for instructions that don't lower.
Note that I left HandleParam/HandleAddrPattern. I'd be happy to change
those to just set lowered=false too, but was hesitant to given the
separate logic.
Also, I'm separately considering migrating the macro logic into similar
constexpr things. If I do, I might switch Define to take in a struct.
But for how this particular parameter works, the overload felt
reasonable, particularly since is_lowered is not used in combination
with TerminatorKind.
I'm trying to increase the distinction between BuiltinKind and
BuiltinFunctionKind. BuiltinKind is for instructions,
BuiltinFunctionKind is for function definitions. To get to this point,
I'm doing a few changes:
- BuiltinKind -> BuiltinInstKind
- builtin_kind.* -> builtin_inst_kind.*: filename consistency
- Builtin -> BuiltinInst: mainly for consistency with the above
- Builtin::builtin_kind -> BuiltinInst::builtin_inst_kind: somewhat
repetitive but seems like a consistent edit
- Function::builtin_kind -> Function::builtin_function_kind: seems a
useful distinction
I'm leaving alone things like (and mentioning in case there's a desire
for more renames):
- InstId::BuiltinError, InstId::ForBuiltin: these I think are more
apparent because they're directly associated with Inst.
- GetBuiltinICmpPredicate in lowering: maybe builtin function handling
should be in its own file, but these local names don't feel problematic
to me.
- GetBuiltinType, BuildBuiltinValueRepr, PerformBuiltinIntComparison:
similar to the above, names don't feel too problematic
Require types into which qualified lookup is performed to be completely
defined. Eventually this will trigger substitution into the definition
for generic types.
Also reorder links so that the future/most recent items are first,
putting older videos lower down.
Noting the lightning talk because it's there, but also that it's a
lightning talk because most of the others are 40m/1h, so 5m is very
short by comparison (I was tempted to label the length of everything,
but not sure how valuable others view that).
For each generic, build a list of instructions describing the
computations we need to do when resolving an instance of the generic:
this is a list of the instance-specific constants and types that the
generic uses. Another way of viewing this list is as a block of Carbon
SemIR code that is evaluated in order to form an instance of the generic
-- this is referenced in the code as the "eval block" for the generic.
For each instruction in the generic whose type or value is a symbolic
constant, replace that type or constant value with a symbolic reference
that says "to find the actual type or value, look at index N in the list
of values for the generic instance".
For an instruction with a symbolic constant value, we can just add that
instruction to our list. For an instruction with a symbolic constant
type, however, we may not have a corresponding instruction computing the
type within the generic and may need to build a new instruction, but
will reuse one where possible. In the case where we build a new
instruction, we use the existing substitution code to build the type
within the eval block.
For now, this transformation is only done in the declaration region of
the generic, not in the definition region. Also, we map back from the
symbolic references to the underlying constant value in a few places
where we will eventually need to do a lookup into a generic instance, in
order to avoid regressing the tests.
The use of ArrayStack here is intended to simplify the logic, and also
make better use of the inst heap allocations. Prior changes #4101 and
#4103 removed the less related logic from InstBlockStack, although #4103
is the actual part that blocked using ArrayStack.
BTW, note the PrintForStackDump implementation was incorrect because it
didn't apply size_. This simplification fixes the issue.
This executes on a TODO in AddImportRef to add instructions to their own
block instead of the File block. This has an important consequence of
removing a pattern from InstBlockStack that added to blocks not
currently at the top, cleaning up an issue for ArrayStack. The delta
here is then mostly in different formatting of the import refs, a
consequence of the separation.
Creates a `GlobalInit` class for storing relevant values, pulling
functions off `InstBlockStack` and `Context`. Adds a `Context` pointer
just so that it doesn't need to be passed in on each call (`Finalize` in
particular uses several members).
Note we have several different `InstBlockStack` instances, so several
copies of the relevant members were simply unused.
By adding an `ImportDecl` instruction, this creates something that can
be referenced through `ImportIRInst`.
packages/no_prelude/implicit_imports_entities.carbon is getting a test
of this (import_conflict and import_conflict_reverse).
Also re-packs ImportIR from 24 bytes to 16 on 64-bit, since I'm touching
everywhere that makes one anyways.
Based on discussion around the region handling in generic_region_stack,
create a generic structure for the stack-of-vectors support. I also want
to add this to InstBlockStack, but that's a little more complex due to
GlobalInit, so cutting a PR here to check with review.
My work here is how I noticed #4099; I want to be sure that I'm correct
about the issue, but it's the difference between being able to use
PeekArray or not.
Note in scope_stack.h, I believe we could remove next_compile_time_index
and make it just based on elements_size(). However, I want to verify
with you before I make further changes there.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
This undoes a previous change to unify them, and I think at my advice.
=[ Sorry about that, I think I was just wrong.
Specifically, I think I had suggested that it would be more efficient to
have a single shared hashtable of strings. The more I look at profiles
of the toolchain, the less likely that seems. Specifically for
identifiers and string literals it seems especially problematic.
Using a single, joint hashtable is likely a good idea when all of the
different querying code paths are equally likely, the strings follow the
same distribution of sizes, and either there is no clustering of access
to different sets of strings or none of the sets are meaningfully small
enough to fit into a lower level of resident cache.
I think essentially none of these predicates actually hold for
identifiers vs. string literals:
- Identifiers are *much* more hot
- They have wildly different size distributions.
- The access patterns are very clustered
Sorry for the misleading advice on that one.
While splitting them, I've worked to simplify the code a bit by building
a way to have the `StringRef` holding canonical value stores not require
specializations, and so we get a pretty large code cleanup in the
process here.
This works to leverage the capabilities of the hashtable as much as
possible, for example using the key context in the value stores.
However, there may still be opportunities to refactor more deeply and
use the functionality even better. Hopefully this is at least
a reasonable start and gets us a clean baseline.
On an Arm M1, this is a 15% improvement on my large lexing stress test,
but ends up a wash on my x86-64 server. This is a smaller benefit than
I expected, and it's because we're using a set-of-IDs and looking up
values with a key context for things like identifiers. This pattern has
a surprising tradeoff. The new hashtable uses significantly less memory,
a 10% peak RSS reduction just from the hashtable change. But indirecting
through the vector of values makes growing the hashtable dramatically
less cache-friendly: it causes growth to randomly access every key when
rehashing. On x86, everything gained by the faster hashtable is lost in
even slower growth. And even on Arm, this eats into the benefits.
But I have a plan to tweak how identifiers specifically work to avoid
most of the growth, and so I suspect this is the right tradeoff on the
whole. It gives us significant working set size reduction and we can
likely avoid the regressed operation (growth with rehash) in most cases
by clever reserving and if necessary by adding a hash caching layer to
the table infrastructure.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
I found this through inspection, looking at an array stack data type.
Tests pass either way, not sure what a good test would be for
regressions (tests do fail if the size doesn't match, but either
approach gets an appropriate size). But this is followed by
`truncate(remaining_compile_time_bindings)`, so it seems like
`drop_back` is a better match than `drop_front`.
When forming a `ConstantId` for a symbolic constant, add storage to
track the generic in which the constant was formed and the index within
that generic. These fields are not yet populated.
This injects a customization point for hashtable-specific equality
testing that the key context uses by default. While this is rarely
needed, there are LLVM types where it is necessary and it seems a good
general tool to have to avoid unnecessary complexity from custom key
contexts when a simple customization of equality is all that is
required.
This also adds a CRTP mixin for implementing a common pattern of key
contexts where the context provides translation of some key types into
another type, potentially using state. Rather than having to implement
the entire key context API, code can derive from this template and
simply provide a set of overloads for the types it wants to translate.
Any key types used which can be passed to one of those overloads will
get translated before following the same logic as the default key
context. While this updates the only usage so far of this pattern, a
subsequent PR will add several more users making the pattern worth
abstracting here.
LLVM's `APInt` and `APFloat` need specialized handling to be used
effectively in hashtables. We can't inject overrides into LLVM so we
need to handle them in our hashing routine.
There were also problematic limits on hashing pairs and tuples. First,
the unique-object-representation hashing of pairs was more restricted
than tuples which was a problematic asymmetry and isn't needed. But the
larger issue is that we didn't support recursively hashing when
necessary. That requires a careful predicate to avoid infinite recursion
but lets us handle important use cases for hashtables with a tuple as a
key.
Also added support for hashing arrays that recurse in addition to arrays
where we can hash the raw storage, and added overloads to redirect to
common array handling from various array-like types.
Last but not least, re-worked the constraint model for hashing as raw
data to not override custom hashing functions.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Co-authored-by: Carbon Infra Bot <carbon-external-infra@google.com>
When checking a declaration or definition of a generic, track a list of
created instructions that depend on the generic's parameters in some
way, along with information on how they depend on the parameters. This
will eventually be used to determine what information we need to compute
when creating instances of the generic, but for now we're just building
the list.
Information is tracked separately for the declaration region and the
definition region of the generic, because in general these may be first
provided in separate declarations, and they should be substituted into
at different times.
This is awkward to track... Probably it would be best done by tracking
the ratio of unique IDs to lines as a floating point and plot them and
see what a best fit distribution curve looks like. But none of the
histogram printing or stats tracking stuff already in use here makes it
easy to do any of that...
So this does what I hope is a reasonable rough approximation by counting
the ceiling of unique identifiers per 10 lines of code, and plotting
that discreet histogram. Shape of the histogram is exactly what I would
expect: one centered distribution, vaguely normal looking. And the
center for a bunch of different codebases, including our toolchain, is
exactly at 5, which would mean 0.5 unique IDs per line. And the
distribution is pretty reliably bounded above by 10 or 1 unique ID per
line. Which almost seems to clean to be true? Slightly worried about
confirmation bias making me think this code is working because the
results look so pretty.
Here is the output for the toolchain:
```
## Unique IDs per 10 lines ## (median: 6)
2 ids [ 2] █▎
3 ids [19] ████████████▎
4 ids [32] ████████████████████▋
5 ids [55] ███████████████████████████████████▌
6 ids [62] ████████████████████████████████████████
7 ids [44] ████████████████████████████▍
8 ids [22] ██████████████▎
9 ids [11] ███████▏
10 ids [ 7] ████▌
11 ids [ 2] █▎
```
And here is the output for llvm-project/*/{lib,include} (to avoid
tests):
```
# Unique IDs per 10 lines ## (median: 5)
1 ids [ 29] ▍
2 ids [ 282] ███▊
3 ids [1492] ███████████████████▉
4 ids [2674] ███████████████████████████████████▌
5 ids [3011] ████████████████████████████████████████
6 ids [2267] ██████████████████████████████▏
7 ids [1549] ████████████████████▋
8 ids [ 817] ██████████▉
9 ids [ 301] ████
10 ids [ 98] █▎
11 ids [ 61] ▊
12 ids [ 50] ▋
13 ids [ 25] ▍
14 ids [ 33] ▌
15 ids [ 14] ▏
16 ids [ 15] ▎
17 ids [ 9] ▏
18 ids [ 8] ▏
19 ids [ 12] ▏
20 ids [ 15] ▎
21 ids [ 3]
22 ids [ 8] ▏
23 ids [ 3]
24 ids [ 3]
25 ids [ 6] ▏
26 ids [ 0]
27 ids [ 2]
28 ids [ 0]
29 ids [ 0]
30 ids [ 3]
31 ids [ 1]
32 ids [ 1]
```
So, I noticed that ImportIRInst didn't print properly while trying to
debug an issue, and that's where this started. Then I was sort of trying
to figure out why we have "type_blocks" but "typeBlock", so trying to
make that more consistent. "importIRInst0" felt more odd than
"import_ir_inst0" which is why I'm suggesting down this particular
route, but let me know if you'd prefer the reverse (but then do we also
do "typeBlocks", etc, removing the consistency with the member name?)
Also starting to print more detail on import_irs, and added
import_ir_insts (adjusting formatting for that too).
Technically, any small size buffer's lifetime has ended by the time we
get to the base's destructor. This means its no longer valid to access
table contents if stored there from the base destructor. We need to
handle destruction in the table class instead.
This ends up being a trivial change because the logic is already
factored out, we just need to call it from a different point.
This refactors how the implicit import is handled in order to retain
more name scope information. As a consequence, private access control
works better between api files and implementation files. Note though
that this will also be essential for name poisoning between the API and
implementation, as discussed in #3763.
In implementing this, I ran into a couple issues with namespaces that I
think point to flaws in their handling. I've fixed some and added a TODO
for the biggest issue (in check.cpp line 281-288), which relates to the
handling of namespaces of direct imports which are first evaluated
indirectly.
In a `class C(T:! type)`, the type `Self` should be `C(T)`, not merely
`C`. Similarly, in an `interface I(T:! type)`, the type of self should
be `I(T)`, not merely `I`.
In `ClassType`s and `InterfaceType`s, track a `GenericInstanceId` for
the instance rather than just the argument list.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Also add a corresponding value store and YAML output.
We don't create any generic instances in this change; this is just
adding infrastructure for future changes.
Build a `Generic` object for generic functions. This object tracks the
generic parameters that are in scope for the generic entity. Eventually
it will track other information about the generic too.
Add basic SemIR formatting support for generic functions.
Python [PEP-585](https://peps.python.org/pep-0585/) replaces a number of
`typing` module types with built-in equivalents and `collections.abc`
versions as of Python 3.9, with the aim of eventually removing the
`typing` module versions of these classes altogether. Since the minimum
required version of Python listed in the [Contribution Tools
document](https://github.com/carbon-language/carbon-lang/blob/trunk/docs/project/contribution_tools.md#main-tools)
is 3.9, the type hints in the various python files in the repo can be
updated to this style of type hint without a need for backwards
compatibility.
Feel free to close if this isn't a desired change at this time!
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
When constant evaluation produces a known non-symbolic value, treat the
result as a symbolic constant anyway if the type of the value is
symbolic.
We don't yet have many ways to produce a constant that has a known value
but a symbolic type. The added test case is one such way: an array `[T;
0]` initialized from `()` is a symbolic constant only because its type
is symbolic -- we know its value is always `()`. More ways to form such
constants will be appearing soon as we start to support generics: for
example, a method of a generic class has a symbolic type but a known
constant value of `{}`.
When substituting into a symbolic constant, also substitute into its
type.
We'd been discussing that explorer remains necessary for print, and I
was wondering if this kind of approach would be okay (we _probably_ want
this to work, based on #2110, albeit with more overloads -- but I don't
think there's a good way to support overloads at the moment).
```
╚╡../bazel-bin/examples/sieve
2
3
5
7
11
13
17
19
23
29
31
37
41
43
...
```
This switches from a macro that simply wraps genrules to a proper
Starlark rule that runs first compile and then link actions.
Most interestingly, this uses the rule structure to allow using the
Carbon toolchain built either in the target config or the exec config.
While the exec config is more principled and even necessary in a
cross-compile situaiton, it is dramatically less efficient when
developing Carbon as all the binaries and tests outside of our examples
will be built with the target config. This triggers a complete second
build of the toolchain in the exec config for examples before this PR.
It is tempting to try to keep the exec config but make it not cause
redundant actions, but the way Bazel sets up exec and target config
makes it essentially impossible to share their artifacts. There used to
be a hack in Bazel itself to force sharing but it was removed due to it
violating the principled design. Instead, these rules are explicit about
their intent to use the target config, much like a test would be.
I have rigged up a flag that is carefully threaded through a wrapper
macro with `select`s to allow easily switching to the exec configuration
in case it is desired or needed. But the the `.bazelrc` sets the default
to the target config. The `BUILD` file default is the principled `exec`
in case these rules are used by importing into some other Bazel
workspace where we might *only* need the exec config.
The net outcome of this is shaving over 2500 actions off of a clean
rebuild such as is triggered by a version bump to LLVM, including some
of the very slow and expensive compiles of LLVM and Clang themselves.
These would only be triggered if you built the examples so this may
mostly impact our CI latency.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This configures the nightly build to generate release notes and provides
a template for organizing them. The organization is done through
labeling of PRs and categorizing them based on those labels. I've tried
to provide a rough categorization that seems reasonable for folks.
While doing this I looked at our PR labeling and found a few bugs that
were preventing many labels form being attached. I've fixed those and
significantly expanded the coverage of file-based labeling. I've also
added code to do author-based labeling for automated PRs so those can be
separated out from human PRs.
Updating to 1.8.4 breaks the patch file, so I'm looking at solutions
that don't require maintaining a patch.
Verifying this is working with `bazel build
//toolchain/lex:tokenized_buffer_benchmark && strings
bazel-bin/toolchain/lex/tokenized_buffer_benchmark |& grep pfm`
Jon Ross-Perkins
Chandler Carruth
dependabot[bot]
Jack McCluskeyandJon Ross-Perkins
Geoff Romer