This PR adds filetype highlighting support to block string literals in
the TextMate grammar.
After adding the regex in the `begin` of the block string literal,
`beginCaptures` is used to access the first capturing group of the regex
(which captures the filetype) and apply the `constant.character.escape`
rule.
Closes#3641
Building on #3636 which handles the general import case, add special
casing for namespaces. Namespaces can be combined cross-IR, so it's a
little more complex.
The implementation adds import_id to the Namespace instruction as a
reference to find the original using the normal structure. This is
achieved by moving the name_id to NameScope to free up space.
---
I considered a few alternatives...
I considered adding import_id to NameScope, but:
1. It's more consistent with things such as Function or Class that
provide name_id on the info object rather than the instruction.
2. I thought it more likely that there would be more NameScope cases
that might want a name_id rather than the import_id, since ImportRef
will typically be used.
I considered putting the import source (cross-ref IR id + inst id) on
the NameScope versus a separate ImportRef, which seems like the
strongest argument towards the NameScope approach because it removes an
instruction. That just felt inconsistent though, and the overhead of
instruction-per-imported-namespace should be low (theoretically few
namespaces should be used). Plus I feel a bit odd adding two
generally-unused ids to NameScope.
A specialized Namespace structure could also have been created to store
the import_id, but that would add an indirection to the NameScope.
Do not create runtime name bindings for `FieldDecl`s even though they're
declared with `:`, so that we can still constant-evaluate references to
fields.
In array indexing, move the check for an out-of-bounds index into the
constant evaluation logic, so that we will also benefit from it when
constant evaluating a compile-time function.
In tuple indexing, require a template constant index instead of an
integer literal.
The pip dependencies in github_tools are the reason the
MODULE.bazel.lock is platform-dependent. Following complaints about the
platform-dependence, split apart github_tools from the rest of the bazel
repo and make it not track the lockfile: while there's an incremental
safety risk due to not tracking checksums, it's unlikely the tools there
would ever be part of a Carbon release process. If we eventually add
Python tools that need pip to the release, it might be desirable to go
back to re-unify the bazezl repos.
This does make running pr_comments incrementally more inconvenient
because a "bazel run" needs to be run from the github_tools subdir.
As a consequence of separating the dependency, this means tests will not
be continuously run in github_tools. They're now a separate repo, and we
cannot add a dependency without restoring the platform-dependent issue.
I think pr_comments is sufficiently low value and unchanging that it is
not worth building separate CI for it.
Cleans up some legacy references to third_party/llvm-project, since now
github_tools needs to be added to the main bazelignore.
Testing this hash function with representative hash table
implementations showed significant differences in quality between
different multiplicative hashing constants. The constants used and
documented were OK but had clear limitations when merely using a single
64-bit multiplication. However, a search uncovered (partly by luck)
a constant that has empirically been shown to be both significantly
better than other constants and generally not have problematic
weaknesses. There are still plenty of collisions for string keys and
heavily loaded hash tables of course, but no examples of severe outlier
collision rates as observed with all other constants we have tried.
There is a slightly longer comment explaining some of this context and
the other constants we have tried in the PR as well.
To this day, we still don't fully understand why the constant used here
behaves so much better than other constants we have tried, including all
of those we've found in other hashing algorithms.
Also, form `addr_of error` instead of `addr_of operand` when `operand`
is not a reference expression, so that we don't try to constant-evaluate
a meaningless expression.
Also, expect a constant for an array bound rather than specifically an
integer literal. This allows constant evaluation results to be more
easily tested by inspecting array bounds.
The constant value we associate with an initializing representation is
the object representation that the initializing expression will store to
its destination.
Also include the type in the profile of an instruction. This is now
necessary for array values, which are represented as tuple_value
instructions with array type, to avoid instructions with different types
being merged by constant canonicalization.
This PR is changing the color of the var keyword in the `PrintTotalArea`
source example which is referenced in the README. The original color was
white and changed to `#FF7B72`.
---------
Co-authored-by: jonmeow <jperkins@google.com>
Name conflicts weren't previously tested, and the diagnostics were just
a TODO, so this is also adding testing for that. But handling too.
Removing AddEntry because I think it's hard to make helpful for this
use-case when we want to do a diagnostic followup (because really,
callers want the full `.insert` result of pointer + success), and unused
otherwise.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Remove the type canonicalization mechanism and instead rely on constant
canonicalization to deduplicate types.
Rename the `Canonicalize*Type` functions to reflect that they're no
longer performing canonicalization. Switch code that creates types due
to semantic checking, rather than due to source syntax, to directly
create type constants through evaluation rather than creating an
instruction and evaluating it to produce a separate constant
representation.
The mapping from `const (const T)` that was previously performed by type
canonicalization is now implemented in expression evaluation instead.
The value `<error>` is now treated as a constant value, with a special
property that an instruction involving `<error>` that could possibly be
constant evaluates to `<error>`. This helps avoid producing follow-on
errors when an error occurs as a subexpression of an expression, such as
a type, that is intended to be constant.
This makes duplicate and previous definition handling match. While we
may want to make both point more fine-grained at the name, the necessary
logic seems likely to be equivalent.
Note, I'm looking at this mainly due to duplicate names in imports,
where it's especially helpful to take an instruction instead of a parse
node. We'll eventually want to handle parse nodes from other imports
better, and I think this is the way it would most likely work.
Rather than producing multiple constants with the same value, fold all
instances of a given constant to the same constant instruction.
A future PR will use this to replace the current type canonicalization
system.
The `libpfm` in the Bazel central repository uses `make` to build it,
which is difficult to integrate with our toolchain. Rather than try to
fix all the issues there, it's easy to just add a native Bazel build for
the library. I don't know that any of the relevant upstream folks are
interested in this kind of build, but it seems easy for us to maintain
as a Carbon project build configuration. I've also not tried to port all
of the different configurations as a consequence, and only 64-bit x86
and Arm as that seems the only likely architectures we'll care about in
the near term.
I've kept this using the `bzlmod` stuff as best I can, and I *think* I'm
holding all of those pieces correctly, but if not, happy for suggestions
on adjustments.
The `google_benchmark` package also has an awkward way of enabling
`libpfm` support using a top-level `bazel` command line flag. I think
this is because of how brittle the Bazel build of `libpfm` is, but I'm
not sure. With the new build, it seems easy to patch `google_benchmark`
to detect the same conditions as we build `libpfm` under, and enable it
there. So I've done this to avoid folks having to pass a command line
flag on platforms where it is supported.
The result is that we now get really nice CPU counter support in our
benchmarks out-of-the-box on Linux x86-64 and AArch64. For example on my
Fedora Asahi install on a Mac Mini I get:
```console
$ bazel run -c opt --copt=-gmlt //common:hashing_benchmark --run_under="taskset -c 4" -- --benchmark_counters_tabular=true --benchmark_perf_counters=CYCLES,INSTRUCTIONS
INFO: Invocation ID: 4aaeb9e9-7df5-4f1f-b56b-c03411790268
INFO: Analyzed target //common:hashing_benchmark (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //common:hashing_benchmark up-to-date:
bazel-bin/common/hashing_benchmark
INFO: Elapsed time: 0.360s, Critical Path: 0.02s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: /bin/bash -c 'taskset -c 4 bazel-bin/common/hashing_benchmark '\''--benchmark_counters_tabular=true'\'' '\''--benchmark_perf_counters=CYCLES,INSTRUCTIONS'\'''
2024-01-15T00:10:50-08:00
Running /home/chandlerc/.cache/bazel/_bazel_chandlerc/b686aa8910e0845b88c21d715819b076/execroot/_main/bazel-out/aarch64-opt/bin/common/hashing_benchmark
Run on (8 X 2064 MHz CPU s)
CPU Caches:
L1 Data 64 KiB (x8)
L1 Instruction 128 KiB (x8)
L2 Unified 4096 KiB (x2)
Load Average: 0.01, 0.08, 0.08
--------------------------------------------------------------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations CYCLES INSTRUCTIONS bytes_per_second
--------------------------------------------------------------------------------------------------------------------------------------------------------------
BM_LatencyHash<RandValues<uint8_t>, CarbonHashBench> 4.11 ns 4.11 ns 170200064 13.1321 9.00587 232.116Mi/s
BM_LatencyHash<RandValues<uint8_t>, AbseilHashBench> 4.82 ns 4.82 ns 145643520 15.3657 12.0059 197.946Mi/s
BM_LatencyHash<RandValues<uint8_t>, LLVMHashBench> 7.96 ns 7.95 ns 87956480 25.3737 17.0068 119.991Mi/s
BM_LatencyHash<RandValues<uint16_t>, CarbonHashBench> 4.11 ns 4.11 ns 170365952 13.1247 9.00587 464.573Mi/s
BM_LatencyHash<RandValues<uint16_t>, AbseilHashBench> 5.51 ns 5.51 ns 127568896 17.5578 14.0059 346.225Mi/s
BM_LatencyHash<RandValues<uint16_t>, LLVMHashBench> 8.00 ns 7.99 ns 87085056 25.377 17.0068 238.834Mi/s
BM_LatencyHash<RandValues<std::pair<uint8_t, uint8_t>>, CarbonHashBench> 4.91 ns 4.90 ns 136013824 15.6456 14.0059 389.006Mi/s
BM_LatencyHash<RandValues<std::pair<uint8_t, uint8_t>>, AbseilHashBench> 6.85 ns 6.85 ns 102630400 21.8041 18.0059 278.637Mi/s
BM_LatencyHash<RandValues<std::pair<uint8_t, uint8_t>>, LLVMHashBench> 7.57 ns 7.56 ns 92798976 24.1437 20.0068 252.151Mi/s
BM_LatencyHash<RandValues<uint32_t>, CarbonHashBench> 4.12 ns 4.12 ns 170229760 13.1272 9.00587 926.444Mi/s
BM_LatencyHash<RandValues<uint32_t>, AbseilHashBench> 4.93 ns 4.92 ns 145304576 15.3738 12.0059 775.224Mi/s
BM_LatencyHash<RandValues<uint32_t>, LLVMHashBench> 8.11 ns 8.10 ns 87127040 25.373 17.0068 470.98Mi/s
```
Instructions created by splices during conversion are now evaluated, as
are instructions created in cases where we first create a placeholder
instruction and later replace it by a different instruction.
This also removes the ability to set a parse node and instruction
independently after creating an `InstId`, which could lead to them
accidentally not matching.
These changes are aimed at improving code readability and
maintainability within the `StaticScope` class of the AST folder.
- **Optimized Print and PrintID Methods:**
Introduced a template function `PrintCommon` to handle common logic in
`Print` and `PrintID`. This change simplifies the class interface and
avoids repetition, enhancing code readability.
- **Simplified TryResolveHere Method:**
Streamlined the `TryResolveHere` method by simplifying the conditional
logic. The refactored code is more readable and easier to understand,
improving overall code quality.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
The parse nodes are still tracked as part of the same value store
interface in order to ensure parity, but they're split out from Inst
itself in order to reduce the size of Inst -- the expectation is that
they don't need to be passed around quite as much.
This change doesn't actually reduce the passing very much, although
there are hints of it: AddInstAndPush doesn't typically need a separate
parse node from the one on the Inst itself, for example. In a couple
spots I changed code to rely a little more on the InstId until the
ParseNode is needed, but it's very low hanging fruit where done. I think
convert could do more to not eagerly fetch the parse node before its
use, but more cleanup felt it would be easier to handle separately. I'm
currently viewing this as making such cleanup _possible_ rather than
executing on it up-front.
But also, I want to make sure there's a consensus to head in this
direction before pulling the trigger. We speculated that this would
result in the parse node being passed around less, and I do think that's
the case, although it's a bit fuzzy in the change.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This is accomplished by tracking an extra bit on the ID we store in the
constant values table, and propagating that from subexpressions to the
enclosing expression. This extra bit is not yet computed correctly for
types; that will be addressed in later PRs.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
We propose a roadmap for 2024 focused on a working Carbon toolchain that
supports Carbon ↔ C++ interop.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Form a side table with constant values for each instruction. Evaluation
is only supported for a few very simple kinds of instruction for now.
This is not observable outside of the SemIR output, because nothing
depends on expressions having a constant value phase yet.
yash is mainly for flex/bison formatting, which was for explorer: we're
refocusing away from that, so it doesn't make as much sense to include
now.
black I think is relatively new, it was previously part of the python
plugin. I'm hoping inclusion helps with setup.
Per discussion, unqualified lookup should only occur on identifiers that
existed at the time Context was initialized. As a consequence, the
resize logic in LexicalLookup may not be necessary. Even considering
metaprogramming, if metaprogramming is restricted to qualified name
lookup, it may not be necessary in the future. Support should be easy to
add if we need it. Trying to clearly document in the CHECK message what
the rationale is.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This is primarily being done for performance reasons, removing hash
lookups. The increased memory consumption is accepted.
Refactors LexicalLookup out to its own structure.
Karthik Prakash
Jon Ross-Perkins
czapiga
Richard Smith
kshokhin
Chandler Carruth
Rayyan Kandjonmeow
josh11b
Daniel James SzelogowskiandRichard Smith