Instead of running on the "current" `@` (the state at which the previous
`jj` command happened to be run), intentionally trigger a working copy
snapshot from `jj_prek`.
Assisted-by: Claude Opus 5 via Antigravity
Since we started bootstrapping, our stage-1 toolchain binaries have been
inputs to stage-2 compilations, and to the compilation database logic,
they're indistinguishable from generated sources. This caused us to get
compilation command lines for them, which clangd would try to index by
parsing the binary, and would either run incredibly slowly or crash.
Filter out non-source files from the compilation database.
In passing, also emit the JSON dump without whitespace, which makes the
database a bit smaller and faster for clangd to parse.
Assisted-by: Gemini via Antigravity
---------
Co-authored-by: Geoff Romer <gromer@google.com>
Our development instructions now recommend installing a number of
binaries through `cargo`. Updating these binaries has to be done by
hand. So we can provide a script that users can use to update them
easily and regularly.
Adds `toolchain/benchmarking/prelude_benchmark.cpp`, which measures the
time to compile the Core prelude and reports the most interesting SemIR
memory statistics as benchmark counters.
The prelude is compiled by checking a file that imports it: the implicit
prelude import causes the check phase to lex, parse, and check the full
set of prelude files, so this is a direct measure of prelude compilation
cost. Four input variations exercise increasing amounts of the prelude:
an empty file, a minimal single-type use, an operator-heavy file that
hits many impls, and a compact file that pulls in a wide swath of the
prelude.
Memory usage is queried directly: `Driver::set_mem_usage` takes a
`MemUsage` that a compile merges each file's usage into; the benchmark
passes one, compiles, and sums the entries by label. A compilation unit
collects into its own `MemUsage` whenever usage is dumped or a sink is
provided (decided in `SetMultiUnitCache`); after a file is done it dumps
that `MemUsage` per-file as before and, if a sink was provided, merges
into it via a new `MemUsage::Add(const MemUsage&)` overload. `MemUsage`
also exposes its entries via a public `Entry` type and an `entries()`
accessor.
Also extends `scripts/bench_runner.py` to (1) treat Mem-prefixed
counters as cost metrics (smaller is better) and (2) tolerate metrics
that aren't reported by every benchmark in a binary.
Assisted-by: Claude Code
---------
Co-authored-by: Christopher Di Bella <cjdb.ns@gmail.com>
Our mirror repositories contain vim scripts and textmate grammars,
neither of which have any need for the LLVM exception, so we can use the
base Apache 2.0 license there with no loss of relevant permissions. This
is a compatible license, so providing these repositories under these
license terms is valid.
Other than removing the LLVM exception, the new license file has one
other change compared to LICENSE: the amount of whitespace on some lines
is reduced from four spaces to three. This makes the new license
*exactly* match the Apache 2.0 license, byte-for-byte.
This is important because the license checking done by GitHub's linguist
project doesn't recognize Apache-2.0-with-LLVM-exception as an
acceptable license, but does allow plain Apache-2.0.
Add support to the dump command to `dump context import_ir_inst123`
Support for dumping the ids was previously added, but missed support for
parsing and creating a C++ id from its printed id.
TextMate doesn't support JSON grammars, so convert our JSON grammar to a
plist automatically as a pre-commit check. Fix malformed info.plist
file. Add missing uuid to grammar file.
Assisted-by: Gemini via Antigravity
Provide a cut-down repository containing just our textmate bundle, both
for easy installation in general and so that github's linguist in
particular can pick it up and use it for highlighting Carbon files.
The sync_repos script is automatically run by our github workflow
whenever utils/ changes.
Split out the `prek` tool usage instructions into a separate skill. This
should make the agent more likely to realize the skill is relevant to a
particular task and consult it. Extend the skill to include instructions
for using `prek` in a jj workspace, and add a helper script for that
situation.
Add a `jj` skill, with the main purpose being to instruct the agent to
use `jj` not `git`, and to use `--no-pager` when running it.
Extend the `bazel` tool description slightly to more strongly encourage
agents to read and follow it.
Assisted-by: Gemini via Antigravity
Mostly these errors were around `dict` missing arguments, and they are
almost always `[str, Any]`.
But a real thorn here was `xml.etree.ElementTree.Element`. `ty` insists
that this is a generic type, and indeed it appears to be one, or
becoming one, in some python version. But it is not generic in python
3.12. So we are stuck in an unsolvable land where:
- `ty` gives an error unless you write `[str]` on the type, because it
thinks it is generic.
- python3.12 gives an error if you do write `[str]` on the type, because
it thinks it is not generic.
Forcing `ty` to target exactly python 3.12 does not help. So I have just
used a linter-ignore comment on that line.
This provides a native harness with several advantages over
`pre-commit`:
- Faster when initializing the cache
- Smaller cache sizes: under 43mb compared to over 62mb
- Better integration with `uv` for Python usage
Steps for migrating for existing contributors:
1. Install `prek` following instructions in the updated docs.
2. Replace hooks in an existing checkout with a special flag:
```sh
prek install --overwrite
```
The `--overwrite` flag is what removes the old hooks.
If you used the pre-push variant:
```sh
prek install --hook-type pre-push --overwrite
```
3. Optional cleanups:
```sh
rm -rf ~/.cache/pre-commit # reclaim the old hook-environment cache
pipx uninstall pre-commit # if installed via pipx; or `brew uninstall
pre-commit`
```
Assisted-by: Antigravity with Gemini
Not sure how these got missed when moving other things to `uv`, but this
should clean them up.
The bump to Python 3.12 is so that we can use `@override` with the
simple import from `typing`. This is needed by the newest versions of
`ty` to do type checking. Added the relevant `@override` annotations.
Assisted-by: Antigravity with Gemini
This removes the need to install any specific version of Python or
figure out how to configure it by instead asking users to install `uv`
and letting it manage Python. Among other advantages, `uv` is designed
to be fast enough to embed directly into our scripts.
We were already using this in `bench_runner.py` so that the script could
import non standard library dependencies. Moving to it for the rest of
our Python unifies the approach and will also enable dependencies
whenever needed.
I've left `github_tools` alone as it has special handling with its own
Bazel setup.
I've updated the contributing tools to explain the approach here.
First, this makes the Bazel invocations not try to uses curses which
prevents running them with `pre-commit run ... -v` showing the timings
for each check. The curses display overwrote the output.
Second, this fixes the main slowdown I was seeing. Because we passed
_all_ files to the check-build-graph hook and there are large number of
files, pre-commit would run the tool over and over on a subset of the
files. This is especially wasteful as the build graph check already
doesn't do anything with the files, it just checks `//...` on each
invocation. So this just added a (large) constant factor of cost.
Third, this tries to reduce the cost of `fix_cc_deps.py` in the case of
large numbers of files. This still isn't _super_ fast -- but the rest of
the cost is in running the `bazel query` and parsing the output. I tried
switching it to jsonproto and it wasn't any faster. I think this would
need to be in a non-Python language and use `proto` directly to
significantly improve the cost here.
Assisted-by: Antigravity with Gemini
In particular, bazel builds would previously fail in
`workspace_status.py` if you didn't have a `.git` with this error:
> ```
> ERROR: <builtin>: BazelWorkspaceStatusAction stable-status.txt failed:
Failed to determine workspace status: Process exited with status 1
> fatal: not a git repository (or any of the parent directories): .git
> ```
Assisted-by: Google Antigravity
---------
Co-authored-by: Josh L <josh11b@users.noreply.github.com>
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
The `split(" ")` function will split two consecutive spaces apart,
giving an empty string in its output. So `dump context inst5` was
mis-parsed to have arguments `["context", "", "inst5"]`. If `split()` is
called with no arguments, it splits on whitespace but ignores
consecutive whitespace, so we correctly parse the args to be
`["context", "inst5"]`.
`dump context facet_type` was an error before since we expected that to
be followed with an id value. While `dump context facet_type 5` still
works, if there's no id value, try to use `facet_type` as a variable
name. This allows us to dump an inst id if it happens to be named
`inst`, etc, without having to use `--` to disambiguate.
This has been working really well for me, is incredibly faster than the
other approach, and some commits continue to hit bugs in the old system
where files that aren't even going to be run through `clangd-tidy` end
up tripping up the execution. Hopefully all of that is resolved with the
new version.
This also switches to a more Bazel-based install layout, skipping the
FHS-based synthetic layout. The FHS-based layout is still reconstructed
explicitly when building an installable tar-ball.
The biggest change is to configure the just-built install as a Bazel
toolchain, including allowing it to build its own runtime libraries as
native Bazel libraries. This removes the need for a monolithic runtimes
build, all of that code logic is removed.
This should also pave the way to using the just-built toolchain for
doing a full 3-stage bootstrap. Building the 2nd stage is included here
as it was a particularly effective way to test that the Bazel
integration was fully working. Adding a 3rd-stage check for stability is
future work, but should be pretty easy.
There is a down-side: this uses the busybox to do the runtimes
compilation, which means they will be re-built after ~any change to
Carbon. However, the integration with Bazel should largely pay for this,
and we can continue to factor the tests away from depending on built
runtimes in most cases.
Now that we're building and testing the runtimes more directly, this
surfaced a problem with the layout of runtimes on macOS that is fixed
here. All of the Darwin OSes use a custom layout for their resource
directory compared to other targets. We now model this in both the C++
built runtimes and the Bazel built runtimes.
Assisted-by: Gemini via Antigravity
Updates the way to access vscode marketplace for publishing. I've
adjusted CarbonInfraBot's attached email to match.
The `#editor-integrations` change is for inconsistent markdown handling
by MS...
https://marketplace.visualstudio.com/items?itemName=carbon-lang.carbon-vscode
looks fine at the moment, but I was seeing rendering as a title -- maybe
a bug that won't be rolled out, but backticks seem fair here.
Assisted-by: Google Antigravity with Gemini
The goal here is to be able to construct a build of the runtimes
directly in Bazel, or by emitting `BUILD` files, or by emitting into C++
code and using that on-demand. For that, we want a single source of
truth, and that source in Starlark.
This should also make the information more generally useful, and so I'm
moving as much as I can into the LLVM Bazel build. Apologies as that
makes the diffs extra annoying.
I do plan on upstreaming the Bazel parts of this, but would like to get
everything working in Carbon and stabilized first.
While here, I've also made a change suggested for the future in the
initial review by lifting the C++ template out of a string literal in
the `.bzl` file, and into an actual separate C++ file.
This only moves libc++, libc++abi, and libunwind. I want to get those
three working end-to-end before I work on the builtins or `crtbegin` and
`crtend`, as those have a bunch of additional complexity.
This also only uses the info in the C++ on-demand build. It seemed like
a reasonable increment to start code review, and my plan is to work on
other build strategies in a follow-up PR. If that doesn't work, let me
know and I'll come back once I have at least a second use of the info
here.
This brings some fixes:
- The handling of `zlib` and `zstd` are much cleaner
- Three of our patches are no longer needed
This also includes the fixes from #6562
It also moves us from `zlib` to `zlib-ng` which is a much better basis
for what we want, and likely makes our toolchain faster when generating
debug info at least.
It fixes another API change in terms of which headers provide the
`createInvocation` we use.
Lastly, it cleans up the deps test to correctly recognize the wrappers
for `zlib-ng` and `zstd`, as well as improving the documentation for why
we allow dependencies on them.
Also consolidate on using `//bazel/cc_rules:defs.bzl` where appropriate.
Also update a couple of Bazel modules deps of `@rules_cc` to the latest
versions.
The dump of a block looks like:
```
(lldb) dump context require_impls_block_id
require_block60000001
- require0: {self_id: inst60000019, facet_type_inst_id: inst6000001D, extend_self: true, parent_scope: name_scope60000002}
```
The dump of an individual RequireImplsId is shown above for `require0`.
This re-adds the ability to put a space between the id type and number.
In particular, when copy/pasting large hex-encoded id numbers that are
retrieved from `p/x`, such as an array of InstIds, putting a space
between allows faster editing. The space allows the previous command to
be reused, and then to delete the id in a single key command.
For example, when developing against a checkout of LLVM, it is useful to
be able to consistently pass an override flag to Bazel for that
repository.
This lets:
```console
bazel test --override_repository=+_repo_rules+llvm-raw=$HOME/src/llvm/llvm-project //toolchain/...
```
and
```console
./scripts/create_compdb.py --extra-bazel-flag=--override_repository=+_repo_rules+llvm-raw=$HOME/src/llvm/llvm-project
```
Share the same Bazel cache and use the same flags.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
Adds support to the `dump` debugger command for named constraint ids,
which are printed as `constraint<number>`. While doing so, we print
whether the `constraint` is complete or not, and add the same to
`interface` to match.
And we noticed that the printing of name and name scope ids, which are
not tagged, are very verbose by adding 7 `0`s to them for no reason. So
make the dump output easier to read by dropping 0 prefixes.
Before:
```
name_scope00000000: {inst: inst0000000E, parent_scope: name_scope<none>, has_error: false, extended_scopes: [], names: {name00000000: inst6000000F, name00000001: inst60000011}} {kind: Namespace, arg0: name_scope00000000, arg1: inst<none>, type: type(inst(NamespaceType))} `package`
```
After:
```
name_scope0: {inst: instE, parent_scope: name_scope<none>, has_error: false, extended_scopes: [], names: {name0: inst6000000F, name1: inst60000011}} {kind: Namespace, arg0: name_scope0, arg1: inst<none>, type: type(inst(NamespaceType))} `package`
```
This change makes dumping and debugging work again with InstIds that are
now tagged with the CheckIRId. The textual representation of an InstId
is changed from `irN.instM` back to `instM` but the `M` is now a hex
value with the tag as part of it, which is the same number that is
physically in the `InstId::index` field. This prevents any cases where
we would potentially print incorrect values for large InstIds.
We teach the `dump` command in lldb to parse hex values for InstId so
that we can paste these numbers back into the debugger.
This restores the original approach in #6046 as it appears
`--notool_deps` isn't sufficient in some situations. It still isn't
clear to me why it seemed to work initially, but I can easily reproduce
the issue now even with that flag.
I've tried to address the feedback in the original PR on the Python
code.
We have grown more generation rules, so try to use a regex instead of
listing all of them.
Also, manually add the runfiles C++ library that isn't "generated", but
is symlinked into the source tree only when built.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Currently hitting ^C prints out two stack traces, requiring scrolling up
though multiple screens of scrollback to get back to the autoupdate
results. This primarily shows up when hitting ^C while it's symbolizing
a C++ stack trace.
Richard Smith
Dana Jansens
Özgür T. Önsoy
Chandler CarruthandChristopher Di Bella
Jon Ross-Perkins
David BlaikieandJon Ross-Perkins