Commit Graph
22 Commits
Author SHA1 Message Date
Jon Ross-Perkins ccf51cef23 Update to bazel 8.0.0 (#4729)
This updates to bazel 8.0.0, also updating bazel mod deps and tools to
make that function. The release is a couple weeks old, and we haven't
updated in a while, and it's a major release. Note it includes some
incompatible flag flips that this is trying to update with respect to.
I'll try generally enabling incompatible support separately.

The most visible bazel behavior change here will be the change from `~`
to `+` in repo path names. (If you're curious,
https://github.com/bazelbuild/bazel/issues/23127 indicates this fixes a
Windows performance issue)

Note that this is building on top of both the action env update in #4728
(which got me started down this path) and the proto removal in #4731
(which would add significant work to this update). Only the commit
starting at "Work towards bazel 8.0.0" is specifically part of this PR.
2025-01-06 23:50:12 +00:00
Jon Ross-Perkins 266fd6aa75 Remove explorer's proto fuzzer and proto dependencies (#4731)
As part of migrating to the latest bazel configurations in #4729, I'm
running into proto toolchain issues. For example:
"Error: <target @@protobuf+//:cc_toolchain> (rule
'proto_lang_toolchain') doesn't contain declared provider
'ProtoLangToolchainInfo'"

Although we may eventually want more use of proto, right now the only
use is for the explorer fuzzer. The explorer codebase is essentially
frozen, so continuing to run it isn't gaining us much (in fact, we've
already disabled autofuzzing for it).

So, rather than trying to fix the proto setup, this change:

1. Deletes `explorer/fuzzing`
2. Removes proto portions of `testing/fuzzing`, which were only in-use
by the explorer
3. Removes some ancillary proto support, which would otherwise break
from the bazel changes and would be difficult to validate as "still
working"

This change is partly isolated in order to make it easier to revive bits
of (3).
2024-12-20 23:55:06 +00:00
Jon Ross-Perkins 145c44b66c Move the language server into toolchain's busybox. (#4469)
Removes the separate language server binary; I'm not sure we need to
provide it. Instead, `carbon language-server` is added as a subcommand.

Moves //language_server to //toolchain/language_server. Splits into a
trivial language_server.h, and a substantive server.h. I wasn't sure
about a better name, but wanted the split similar to check/check.h,
lex/lex.h, etc. At the same time, the class is probably going to be a
little big so not a good fit to through into just a cpp file.

This fixes some style issues with the language server class, but
generally I'm trying to not address things here in order to keep it
simpler.
2024-11-01 19:49:39 +00:00
Jon Ross-Perkins 957599b2ab Implement a basic busybox for carbon/clang. (#4406)
For reference, we're going down the busyboxing route because Carbon
depends on Clang, and we want both to be available as binaries.
Busyboxing allows this while avoiding duplicating symbols between
multiple binaries.

I'm removing the `cc_binary` for `driver:carbon` because I want to avoid
a significant increase in binary outputs; `bazel run //toolchain` still
works great.

This still doesn't have great test coverage (but non-zero:
`//examples:sieve` still builds/runs, for example). The problem is that
we want to avoid subprocessing for performance, but this mainly deals
with subprocessing. I'm still thinking about good approaches for that,
since we'll probably want more significant testing for `clang`
interaction... the solution might involve busyboxing `file_test` too.

Note development on this ran into the argv issue being fixed in #4405
2024-10-30 17:07:55 +00:00
Jon Ross-Perkins 4293c9f25f Switch python syntax to use Path.parents (#4235)
I think `paths.parents[2]` is easier to read than
`path.parent.parent.parent`, just applying uniformly. I'd noticed this
while looking at #4227

Also remove a couple `resolve()` calls that shouldn't be necessary since
`__file__` is absolute (elsewhere in the same files, `resolve()` is used
to resolve potentially relative paths)
2024-08-22 08:29:04 +00:00
Chandler CarruthandJon Ross-Perkins d3a5b0eee7 Add utilities for managing a toolchain install, and install and use LLD. (#3993)
The install directory contains the BUILD logic for creating an
installable tree of data files and executables for the toolchain, and
a library to facilitate toolchain code accessing the paths to their data
within this installation.

Then adds an installation of LLD in a synthetic LLVM installation, and
teaches the Clang runner to configure this and use it for linking
instead of the system linker.

Currently, the install paths only really manage access to the LLVM
binaries installed and used by the Clang runner for linking, but
eventually other data files like the prelude and runtime libraries will
be fleshed out as well. There are TODOs for moving more things over here
such as the prelude.

One interesting aspect of this is where to put helpers like parts of
LLVM in our install. This PR suggests nesting those files under
`lib/carbon`. While using a `lib` subdirectory isn't a perfect fit for
the FHS (Filesystem Hierarchy Standard), having a single location where
private data is collected is significantly superior to spreading them
across the system. This also matches similar patterns used by Clang
itself and several other language toolchains and standard libraries.

The install directory also provides a natural place for us to build out
packaging rules to create installable packages in various formats, but
that remains future work.

---------

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
2024-06-01 03:19:52 +00:00
Jon Ross-Perkins a3b1c433be Remove legacy repo_name settings (#3772)
I'd kept these in to separate the bazel module update from the BUILD
file changes, then forgot about it. I think all of these can be cleanly
removed now. I think it's something we should clean up for consistency
with the bazel central repository names; I think it's best to reduce
that divergence.

llvm_zlib and llvm_zstd remain because of how llvm depends on the
particular names.
2024-03-13 22:58:56 +00:00
Jon Ross-Perkins 39bc76595b Remove zlib/zstd version dependence in check_deps (#3530) 2023-12-19 19:13:54 +00:00
Chandler Carruth 9fe3a94ff6 Update the deps test to use the new spellings. (#3527)
Now that both zlib and zstd are handled with bzlmod stuff, they use
their own repository names rather than the repo-rules spelling.

Sadly, the target determinator tool didn't detect that this test could
be impacted so this slipped past our merge queue.
2023-12-19 15:37:25 +00:00
Jon Ross-Perkins a5eb04eedc Move toolchain and llvm WORKSPACE rules to bzlmod (#3514)
Building on #3505, the toolchain and llvm rules require a little more
special-casing to get them to work well. This also moves
libprotobuffer_mutator, but that one's more minor. The migration
encounters more quirks in repo naming as seen by various queries.

This changes some of the toolchain work that was recently done for bazel
7 in #3496, dropping a bzl file I'd suggested to add, instead using
`:all` for toolchain registration. (somewhat as an improvement, somewhat
just to avoid a `load`)

Remaining in the WORKSPACE are example code repos and tree sitter rules.
Neither of these are part of the main toolchain builds, and so will
probably be lower impact if there's a good solution for them.
2023-12-18 17:12:04 +00:00
Jon Ross-Perkins 6204a27ea9 Start adapting to bzlmod configurations. (#3505)
Some background information is at https://bazel.build/external/migration

Trying to handle the simple cases first. This adds a requirement for
bazel 7 due to differences in bzlmod handling between 6 and 7 (also
discussed on
[#infra](https://discord.com/channels/655572317891461132/707150492370862090/1184942191412510720)).
Bazel seems to be okay with a partial migration such as ths.

The python import behavior has subtly shifted, so `carbon.` is no longer
part of import paths. There's a version-incompatible change for `@@`.
bzlmod makes repos sometimes show as `name~version`.

`target-determinator` seems to be okay with `@@` after a version update.

Things not moved here are things that basically need more dep work:

- clang_register_toolchains because I need to dive into its format.
- llvm-project because we need something slightly atypical, I need to
make sure patching and the repo work carries over.
- com_google_libprotobuf_mutator is sufficiently atypical that it
doesn't have a module already, but should be one of the easier things to
fix.
- brotli/woff2: I think we should actually consider removing these. But
again, they're not trivial moves.
- treesitter due to toolchain registration, which has shifted a bit.
- rules_nodejs because treesitter depends on it in an awkward way to
migrate.
2023-12-15 01:05:35 +00:00
Jonathan B. CoeandChandler Carruth 8c28a0494e Add size="small" to test targets where advised (#3326)
Running `bazel test //...` reported:

```
Test execution time outside of range for MODERATE tests.
Consider setting timeout="short" or size="small".
```

This change adds size="small" to avoid such warnings being reported.

---------

Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
2023-10-24 06:52:00 +00:00
Chandler Carruth 4596cd230d Avoid building the non-test file group in :all. (#3191)
This file group exists to allow a `genquery` rule and a Python test to
verify our non-test dependency graph. We don't actually need to build
the binaries in the file group as part of that. The `genquery` rule
seems to do the right thing -- building it directly doesn't cause the
binaries in the group to be built. But without a manual tag, the group
itself is part of `:all` and thus part of `//...` and part of the rules
that will be built even with PR #3106. A consequence is that any change
to the toolchain causes several other binaries to be built as well
because this file group is in the impacted set. Making it manual should
avoid all of this, and without breaking the actual use from `genquery`.

For example, before this change, in a fully cached build after a `bazel
clean`:
```
> bazel test //bazel/check_deps:all
INFO: Invocation ID: 2d83ebee-4c00-425d-be33-23f42b079614
INFO: Analyzed 3 targets (103 packages loaded, 7137 targets configured).
INFO: Found 2 targets and 1 test target...
INFO: Elapsed time: 4.081s, Critical Path: 2.61s
INFO: 3111 processes: 2796 disk cache hit, 315 internal.
INFO: Build completed successfully, 3111 total actions
```

After this change:
```
> bazel test //bazel/check_deps:al
INFO: Invocation ID: c94089e8-a420-4d3c-9902-134e6b55b297
INFO: Analyzed 2 targets (92 packages loaded, 568 targets configured).
INFO: Found 1 target and 1 test target...
INFO: Elapsed time: 0.700s, Critical Path: 0.01s
INFO: 7 processes: 2 disk cache hit, 5 internal.
INFO: Build completed successfully, 7 total actions
```

While here, re-generate the file group, and fix several issues it
uncovers: mark test utilities as `testonly` and update our LLVM package
allowlist to include `clangd`'s package.
2023-10-05 01:30:41 +00:00
Jon Ross-Perkins 605763d62d Add lint fixes to the buildifier setup. (#3109)
The main motivation for this is to get python loads in using the
`native-py` lint fix. However, enabling that made me wonder, maybe we
should fix in general?

`native-cc` is delayed, but not wholly cancelled (and `native-py`
picking up might indicate `native-cc` won't be too far behind). There's
also some automated fixes for `.append` and dict sorting -- this felt
okay to me, maybe not something to eagerly add but probably not worth
stopping buildifier from fixing (I've noticed the warnings in the past
and had been ignoring them).

Running everything does mean that load orders are sorted automatically
now, which I think is a positive. Most generally, I think these fixes
aren't _harmful_, and having them done automatically seems beneficial:
my biggest concern about `native-py` and `native-cc` was actually that
regressions wouldn't be caught, but this addresses that issue
automatically.
2023-08-22 21:01:42 +00:00
Jon Ross-Perkins c27f41c94b Update the LLVM version in use. (#2895)
The prior terminfo and zlib calls are obsolete. I'm adding the zstd library myself as a quick fix, although I want to investigate if we can make better use of [LLVM's workspace](https://github.com/llvm/llvm-project/blob/main/utils/bazel/WORKSPACE) (where the zstd dep comes from).

```
ERROR: .../external/llvm-project/llvm/BUILD.bazel:184:11: no such package '@llvm_zstd//': The repository '@llvm_zstd' could not be resolved: Repository '@llvm_zstd' is not defined and referenced by '@llvm-project//llvm:Support'
```

The particular commit in use fixes a macos build error. https://github.com/llvm/llvm-project/commit/c5f6a287499a816cba5585708999e2c8b134290f
2023-06-15 16:58:12 -07:00
Jon Ross-Perkins d51ca271af Refactory the IR/builder relationship (#2342)
I'm thinking about how to handle multiple files, and I think the current IRFactory is useful as a file-focused thing. So shifting/renaming accordingly. (doing this in its own PR to make the history a little cleaner for git's move detection)
2022-10-25 12:55:26 -07:00
Jon Meow 107848e30f Switch mypy approach to pre-commit (#1234)
I'm treating my PR to mypy bazel integration as dead in the water. Abandoning that approach, this seems to work reasonably well as a replacement. It's actually a slightly newer version of mypy too -- the bazel integration was stuck on an old version.

Note I don't think we're losing much: we shouldn't be writing too much python.
2022-05-09 09:54:12 -07:00
Jon Meow 309ec35f95 Rename executable_semantics to explorer (#1188)
Change generated with:

```
#!/usr/bin/bash -eux

# Helper script for renaming pending work.
# Run from the repo root.

# Rename executable_semantics in code.
sed -i 's/executable_semantics/explorer/g' \
  $(git grep -l 'executable_semantics' . | grep -v proposals)
sed -i 's/executable semantics/explorer/g' \
  $(git grep -l 'executable semantics' . | grep -v proposals)
sed -i 's/Executable semantics/Explorer/g' \
  $(git grep -l 'Executable semantics' . |  grep -v proposals)
sed -i 's/Executable Semantics/Explorer/g' \
  $(git grep -l 'Executable Semantics' . |  grep -v proposals)
sed -i 's/EXECUTABLE_SEMANTICS/EXPLORER/g' \
  $(git grep -l 'EXECUTABLE_SEMANTICS' . | grep -v proposals)
sed -i 's/ExecutableSemantics/Explorer/g' \
  $(git grep -l 'ExecutableSemantics' . | grep -v proposals)
sed -i 's/executable-semantics/explorer/g' \
  $(git grep -l 'executable-semantics' . | grep -v proposals)

# This is only needed for the initial move.
mv executable_semantics explorer
mv explorer/fuzzing/executable_semantics_fuzzer.cpp explorer/fuzzing/explorer_fuzzer.cpp
```

Verified with `bazel test ...`
2022-04-29 13:20:25 -07:00
Jon Meow 770555d8c9 Update check_deps, make reusing scripts easier (#1187)
I went down this route because I figured the installer binary should be in `check_deps`. The result:

- Add thin wrapper scripts for `buildozer` and `bazel`, similar to the `buildifier` wrapper that's already there.
  - These are mainly to avoid deduplicating logic, and dodging intricacies of Python imports that I don't understand well through instead subprocessing.
- Fix update_roots.py (it was missing a `.parent`, must've been relocated without a re-run at some point)
- Fix fuzzer proto code to be `testonly`
- Update the `check_deps` targets
2022-04-14 19:48:51 -07:00
pk19604014andJon Meow 5ae6d22415 Carbon fuzzing 1/3: added a proto for representing Carbon AST + ast_to_proto library (#1128)
* initial fuzzer proto

* visibility change

* use newer protocol buffer version which has the defs.bzl bug fixed

* Adjusted fix_cc_deps to work with protobuf external repo

* explicitly load rules_cc to avoid a frozenset bug in the version loaded by protobuf

* use explit deps, use llvm's zlib

* restored cxx settings

* deps change

* Cleaned up WORKSPACE and changed the test to read carbon sources from testdata

* updated comment

* adapted to new ErrorOr return value

* proto buffer 3.19.2 -> 3.19.4

* changed comment

* Apply suggestions from code review

Co-authored-by: Jon Meow <jperkins@google.com>

* Apply suggestions from code review

Co-authored-by: Jon Meow <jperkins@google.com>

* Update executable_semantics/fuzzing/BUILD

Co-authored-by: Jon Meow <jperkins@google.com>

* addressed review comments

* updated Unimplemented error message

* Addressed review comments

* more review comments

* switched to loading protobuf via rules_proto()

* Ignore protobuf headers in fix_cc_deps.py until the script supports alias rules

* renamed repeated proto fields to be plural

* added @zlib to check_non_test_cc_deps

* Update common/fuzzing/BUILD

Co-authored-by: Jon Meow <jperkins@google.com>

* review comments

* set is_omitted_expression for return value

Co-authored-by: Jon Meow <jperkins@google.com>
2022-03-25 15:29:31 -04:00
Chandler Carruth 65ac59eb2a Add a script to compute minimal roots for check_deps. (#932)
This avoids needing to have nearly as many rules here which should
reduce its churn.

I've tested that this reaches the exact same set of transitive
dependencies.

Note, only the last commit here is new.
2021-11-03 12:53:54 -07:00
Chandler Carruth 5f67029479 Use upstream GoogleTest and add related test utils. (#876)
This moves over to the vanilla upstream GoogleTest pulled in the more
expected manner with Bazel. It also adds Abseil and Google Benchmark
libraries in the same fashion (there are cross dependencies here).

As part of this, also introduce a dependency check test that can enforce
basic layering of dependencies. For example, this lets us ensure that
non-test Carbon code only depends on LLVM and Clang despite having other
libraries available. There remains some cleanup to improve the way these
dependency tests work, but this at least ensures we don't regress.

I've also provided workarounds to allow both Carbon code and LLVM code
to freely be used with GoogleTest (and other `std::ostream` based
output code). This is done by extending the code in
`//common/ostream.h`. One downside is that it requires opening the
`llvm` namespace and adding an ADL_found overload there. I think on
balance this is still a win and doesn't make me too nervous.

The new version of GoogleTest requires printing more often from matchers
and so I've also added several printing routines to types that
previously didn't require them. Otherwise, most of the updates are just
using the more conventional upstream style of including the headers and
adding `ostream.h` where it is needed.

I did consider moving code over to use `std::ostream` instead of LLVM's
`raw_ostream`, but the advantages of not doing virtual dispatch still
seem significant, and it also seems good to retain access to LLVM's
formatting utilities built around `raw_ostream` given that we can't pull
arbitrary dependencies into Carbon code outside of test code.

All of this was slightly motivated by requests for newer features in
GoogleTest, but much more-so by my desire to have access to Google
Benchmark and Abseil when writing benchmarks. For example, using
Abseil's random number generator seems extremely helpful when generating
inputs for benchmarks. The growing dependencies between these packages
further motivated me to just pull them all in and ensure they worked
well.
2021-11-02 20:14:12 -07:00