This removes the need to install any specific version of Python or
figure out how to configure it by instead asking users to install `uv`
and letting it manage Python. Among other advantages, `uv` is designed
to be fast enough to embed directly into our scripts.
We were already using this in `bench_runner.py` so that the script could
import non standard library dependencies. Moving to it for the rest of
our Python unifies the approach and will also enable dependencies
whenever needed.
I've left `github_tools` alone as it has special handling with its own
Bazel setup.
I've updated the contributing tools to explain the approach here.
This let's you point Bazel at an installed toolchain or download one of
our release archives. When you do, it will configure itself as a C++
Bazel toolchain. This toolchain works reasonably well, but doesn't cache
the C++ runtimes, and so linking is inefficient. The next step will be
to pivot the runtimes from the implicitly on-demand (which can't cache
when using a sandboxed build system like Bazel) to _explicit_ on-demand
runtimes directly with Bazel support.
I've included an example Bazel project that uses this and provides a
bunch of documentation and an example script that should let folks try
this out easily.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
Co-authored-by: David Blaikie <dblaikie@gmail.com>
The WORKSPACE file is deprecated; support is already off by default, and
it'll be removed in the next major bazel release. Our main dependency is
tree-sitter, and I'm trying to address that here.
We're currently using https://github.com/elliottt/rules_tree_sitter, but
that hasn't been updated in a couple years, meaning it lacks
MODULE.bazel support. In the registry, there's
https://registry.bazel.build/modules/tree-sitter-bazel, but this is only
the *parser* libraries of tree-sitter, not the *generator*. I'm using it
for that much, at least.
For the *generator*, which transforms grammar.js to parser.c/h, I'm just
requiring a non-hermetic invocation (i.e., people who want to work on it
will need to install tree-sitter; see the README.md updates). I tried
running it manually, but parser.c is about 600 KB; pre-commit rejects
files that large and I don't think an exception makes sense to override
for this (it'd probably also grow substantially if the grammar were
updated to cover more syntax). In order to make the non-hermetic call
not break "bazel build //..." for most developers, I'm marking most
targets in the package as manual.
Note, I did look long and hard at using `aspect_rules_js`/`rules_nodejs`
to invoke npm. This took a lot of time, and I have a commit that's
mostly working, except I hit a point where it uses `declare_symlink`
which we disallow for compatibility reasons (commit "Lots of work for
figuring out rule_js uses declare_symlink" on the PR). As a consequence,
I think we can't use the primary supported ways to have hermetic npm
calls.
Also, `treesitter` -> `tree_sitter` because it's generally called
`tree-sitter`, two words. We even had a `treesitter/src/tree_sitter`
directory so it's a bit inconsistent.
As far as bugs here, the parser library breaks bazel queries, e.g. the
error:
```
ERROR: Evaluation of query "somepath(//..., @llvm-project//third-party/unittest:gtest)" failed: preloading transitive closure failed: no such package '@@[unknown repo 'platforms' requested from @@tree-sitter-bazel+]//': The repository '@@[unknown repo 'platforms' requested from @@tree-sitter-bazel+]' could not be resolved: No repository visible as '@platforms' from repository '@@tree-sitter-bazel+'
```
I'm just excluding tree_sitter from queries where I can to work around
the error.
The pip dependencies in github_tools are the reason the
MODULE.bazel.lock is platform-dependent. Following complaints about the
platform-dependence, split apart github_tools from the rest of the bazel
repo and make it not track the lockfile: while there's an incremental
safety risk due to not tracking checksums, it's unlikely the tools there
would ever be part of a Carbon release process. If we eventually add
Python tools that need pip to the release, it might be desirable to go
back to re-unify the bazezl repos.
This does make running pr_comments incrementally more inconvenient
because a "bazel run" needs to be run from the github_tools subdir.
As a consequence of separating the dependency, this means tests will not
be continuously run in github_tools. They're now a separate repo, and we
cannot add a dependency without restoring the platform-dependent issue.
I think pr_comments is sufficiently low value and unchanging that it is
not worth building separate CI for it.
Cleans up some legacy references to third_party/llvm-project, since now
github_tools needs to be added to the main bazelignore.
Directories in `external/` now include versions of dependencies, so
continuing to make that work would require version-locking the
compile_flags.txt. We've previously seen issues with directories being
forgotten; this switches to letting bazel generate everything.
Setup follows instructions at
https://github.com/hedronvision/bazel-compile-commands-extractor?tab=readme-ov-file#first-add-this-tool-to-your-bazel-setup
Maintains `./scripts/create_compdb.py`. Per discussion, new files seem
to work okay -- I'm getting #include completion that clearly detects
files.
Note this creates an `external` directory:
```
╚╡ls -ld external
lrwxrwxrwx 1 jperkins primarygroup 27 Dec 19 14:02 external -> bazel-out/../../../external
```
I felt sort of weird checking that in though, at least alongside
`bazel-execroot` it feels like it should have a `bazel-` prefix.
However, I can't rename it. So omitting it, and leaving it to the script
to generate, felt like a reasonable compromise.
Running takes a few minutes (about 3 for me), and should be faster on
reruns. It does print a couple warnings, just tree_sitter missing
parser.c and _GNUC_PREREQ errors from m4.
### Description:
This pull request introduces two significant changes:
**Improved Error Handling in `remap_file` Function:**
- The error handling in the `remap_file` function has been simplified
for improved clarity and user-friendliness. The original code included a
combination of `assert` and `exit`, which could be perplexing during
runtime.
**Changes Made:**
- Replaced the original `assert` and `exit` with a more straightforward
`if-else` block for handling errors.
- Utilized `sys.exit` for clear error messages and proper program
termination when an error occurs.
**Why I Did It:**
- Improved Clarity: The original code's `assert` primarily served
debugging purposes and might not behave as expected during runtime. The
new code offers clarity and predictability.
- User-Friendly Errors: The updated code provides users with easily
understandable error messages.
**Added `.DS_Store` to `.gitignore`:**
- `.DS_Store` files are commonly generated by macOS Finder to store
folder-specific metadata. They do not need to be tracked in version
control.
**Changes Made:**
- Added `.DS_Store` to the `.gitignore` file.
**Why I Did It:**
- To prevent `.DS_Store` files from being tracked in the Git repository,
ensuring a cleaner repository and avoiding unintended commits of
macOS-specific files.
### Improvements:
- **Better Error Handling:** The new code in the `remap_file` function
ensures the program exits gracefully with informative error messages
when errors occur.
- **Enhanced Code Clarity:** The code now communicates its intentions
more clearly with `sys.exit` for handling expected runtime errors.
- **Cleaner Git Repository:** The addition of `.DS_Store` to
`.gitignore` prevents the tracking of macOS-specific files in version
control.
VSCode uses a `.vscode/launch.json` file for configuring the debugger.
This PR adds a line to the `.gitignore` file to also ignore it along
with the already-ignored `.vscode/settings.json` file.
---------
Co-authored-by: jonmeow <jperkins@google.com>
Following #875, diagnose any use of a name prior to the point where it is introduced and its type is known.
This works by performing name resolution on a top-level class, interface, or impl twice: the first pass performs name-resolution for everything other than nested function bodies, and the second pass performs name resolution on function bodies. At the moment, the second pass does a superset of the work done by the first pass, and as a consequence, some identifier expressions now have their target set twice to the same thing.
The Bazel disk cache provides a stable, durable, and supported
persistent cache of the intermediate steps of Bazel builds. This is like
`ccache` but covering more steps and more reliable.
It is based on the same fundamental infrastructure that enables
a *remote* build cache, and at some point we may want to enable that so
that our CI can pre-seed a cache for users. But for now, this just
enables local support for this.
For local builds this will ensure that intermediate artifacts are
re-used when changing branches, syncing, across Bazel server restarts or
even upgrades. It should dramatically reduce the need to re-build large
and slow components like all of Clang and LLVM. Largely, those should
only be rebuilt when we update our pinned revision of upstream and then
cached persistently on a given machine. Enabling this by default
requires hard-coding a path, and so I've also added built-in support for
importing a user config file if present so that folks can customize this
(and any other Bazel features) as desired.
This also wires the disk cache up for the GitHub cache in our CI. By
using the designated Bazel disk cache for this, we get a *much* simpler
and *much* more robust solution. So much so that I've re-arranged the
key structure so we can more aggressively re-use old caches. The only
downside is that nothing ever gets removed automatically from the cache.
I've added a layer that will trim any unused file roughly once a day.
For CI, the benefits of this are huge. None of the previous hacks are
needed to locate and preserve the cache. The design of the cache is
deeply durable and so we don't need to aggressively invalidate it. Last
but not least, it focuses the cache on the raw artifacts, which compress
exceptionally well. With the other changes to shrink the build outputs
and this one combined, I expect we will be able to enable all of the
build targets across all four config/os combinations without any
significant pressure on our space quota for GitHub action caches.
This restructures the `compile_flags.txt` to use the downloaded libc++ system
headers and avoid needing a virtual include directory to be built. It still
needs _some_ Bazel build to complete before working in order to have the libc++
system headers downloaded and the symlink to the Bazel tree created.
One (very) tricky part of making this work is to work around bugs in Clang's
tooling layer that incorrectly handle `..` path components after traversing
symlinks. To avoid this, we add a custom symlinks (`bazel-execroot` and
`bazel-clang-toolchain`) that hide the relevant traversal of the Bazel layout to
find build artifacts and the downloaded toolchain. These symlinks will be broken
until a build with Bazel downloads the toolchain and creates the basic output
tree structure.
It also adds a `create_compdb.py` script. Running this script improves the
tooling fidelity by taking a few steps:
1. It queries Bazel to find all the relevant files and adds them to a
`compile_commands.json` database that allows `clangd` and other tools to
index the entire project for improved cross-references, etc.
2. It builds all the generated files with Bazel so that they can be included
successfully. This is very fast in my testing, taking only 10s of seconds. It
is also very likely to be cached effectively.
3. It translates the arguments from `compile_flags.txt` to make them
persistently use the built generated files include paths so that nothing
breaks even as different targets are built potentially with different
configurations.
There are still some limitations.
- It still requires running Bazel before anything works, even if a fast run.
- It will require re-running if new generated files are added and needed but not
built.
- It assumes that the standard Bazel symlink names are used and available.
Much of the Python here was written by @geoffromer in #384 -- I've adapted it
here after discussing to try to fill in some of the blanks and use a slightly
different approach to querying Bazel. I use the normal `bazel query` rather than
`bazel aquery`. This, for example, allows the index to reliably cover header
files in header-only libraries more directly (rather than relying on transitive
inclusion). It also seems a bit simpler too parse, but that is a pretty minor
difference.
Co-authored-by: Geoffrey Romer <gromer@google.com>
We'd tried to skip this before, but VSCode keeps wanting to create the directory for me, and I think both you and I are using it (maybe me more than you right now?). I'd like to just ignore it -- seems like a cheap add.
The Bazel bits are collected into a directory and given less confusing
names (I hope). Other than names, everything is a direct copy from the
toolchain repository without any edits.
A few files needed to be merged in:
- `.gitignore`
- `.pre-commit-config.yaml`
Subsequent commits will add relevant C++ infrastructure and then the
source code itself.
Richard Smith
Chandler Carruth
Dana Jansens
Jon Ross-Perkins
m3ndax
mx42andjosh11b
Lucile Rose Nihlenandjonmeow
FZC
Jon Meow