On macOS, Homebrew recommends `pipx` to install Python binaries, or
installing `pre-commit` directly, since it is available.
Here is the message in full.
```console
❯ pip3 install pre-commit
error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try brew install
xyz, where xyz is the package you are trying to
install.
If you wish to install a non-brew-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip.
If you wish to install a non-brew packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.
```
~~Also, unless there is a compelling reason to install a particular 3.x
version of Python, it may be better to let Homebrew use whatever version
people may already have installed (or let it automatically install the
version they've tested).~~
Here are the details of the pre-commit Homebrew package:
```
❯ brew info pre-commit
==> pre-commit: stable 3.6.2 (bottled), HEAD
Framework for managing multi-language pre-commit hooks
https://pre-commit.com/
/opt/homebrew/Cellar/pre-commit/3.6.2 (679 files, 9.9MB) *
Poured from bottle using the formulae.brew.sh API on 2024-03-08 at 22:22:11
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/p/pre-commit.rb
License: MIT
==> Dependencies
Required: libyaml ✔, python@3.12 ✔
```
Installing directly with Homebrew is a supported option in the
pre-commit documentation. https://pre-commit.com/#install
This should also unblock our switch to C++20 and other improvements.
There are three core parts of the change --
1) Updating our infrastructure to fetch and find Clang-16.
2) Updating our documentation to reflect this and help folks with any
system issues they encounter.
The infrastructure change is unfortunately tricky. We can't get Clang 16
easily on GitHub's runner images, and in the past we've had persistent
problems with flakiness when our actions download this much during their
runs. Due to the flakiness, we've previously removed all downloading of
dependencies outside of Bazel itself, and added retry loops around Bazel
specifically to overcome flaky downloads.
This change tries to address these problems by populating the Clang and
LLVM toolchain in a place that we can then cache using the built-in
GitHub action caching infrastructure. This seems like by far the least
likely to flake way of downloading extra things into our runs. And since
these are relatively slow moving dependencies, we should populate this
cache very, very rarely.
For Linux, this downloads the binary release artifact from GitHub,
prunes out large parts of it that we don't need, and then caches this as
a local toolchain. This proves both small and fast.
For macOS, this uses a trick to cache the destination of Homebrew
installs. It unfortunately caches the *entire* Homebrew installation
though, and so it also goes to some lengths to prune and minimize how
much is installed from Homebrew. The result is "only" a 2gb cache image.
Because of the size and slower download and filesystem, the macOS runs
see a 1 - 2 minute slowdown.
We might extend the Linux infrastructure here usefully if we want to
test multiple LLVM versions. We might also extend the macOS version to
get a cheaper way to prune parts of the system and free up disk space,
or to cache other Homebrew installed tools if needed.
Last but not least, this brought to the forefront an issue with our C++
toolchain integration which relied on a specific CMake build option
being set in the LLVM toolchain install. This option isn't used in the
official release artifacts. Instead, switch to a more robust approach to
linking libc++abi statically that shouldn't have these problems.
Beyond the infrastructure changes, this also updates the documentation
to reflect requiring Clang 16 or newer, and adds some extra tips for
folks that are missing this.
The documentation is also updated to address a problem with getting the
right libc++abi files installed to support the more robust linking
strategy. This may reduce the problems we've seen in the past around
libc++abi and linking on other Linux distros as well.
---------
Co-authored-by: josh11b <josh11b@users.noreply.github.com>
Proposal to focus implementation effort for the next 1-2 years on the
Carbon
toolchain instead of the explorer. This will impact the explorer in a
few ways:
- We will keep the explorer's code in place, building, and passing its
basic
tests. It can remain a good baseline for exploring Carbon's language
features.
- We won't prioritize expanding the explorer's coverage of Carbon
features or
other improvements -- it is good enough for what we need until the
toolchain
catches up.
- We will stop actively fuzzing and expanding test coverage for the
explorer.
- Eventually, when we want to resume work on the explorer, we'll
evaluate the
best platform to build on -- the current explorer codebase or on top of
the
toolchain's semantic IR.
Also tries to update the core readme and contributing docs to reflect
this.
This Pull Request added support for Bazel installation and setup in the
contribution_tools.md file un the docs directory. It added a link to the
releases page of the bazel repository for debian/ubuntu users to
download the binary and execute the commands provided in the
documentation.
Closes#3439
Adds some notes about the required build flags for lldb to successfully
find the symbols on macOS debug builds. Also adds a recommended debugger
configuration for interactive debugging in VSCode on macOS.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
You need bazel version 6.0.0 (with --host_per_file_copt) to build
carbon. This adds versioning info and adds that Linux, Mac and Windows
can use bazelisk.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Ubuntu 22.04 base image and examples using github, a copy instruction and volumes.
I was using this to run examples on a windows machine since it is more comfortable than switching OS or using the WSL subsystem directly.
If this is something that other people would like to have I can add other versions of ubuntu and other linux distributions.
Key rewrite points:
- Making more use of apt now, since we no longer need `brew` for llvm on Linux.
- Gets ahead of issues with brew's llvm 15 on Linux.
- Clear list of commands for a typical setup.
- Less verbose text about various tool options (I think this was just too much).
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
The toolchain embeds the assumption that `clang` and `llvm-ar` are
adjacent, which may not be true on all host platforms. Moreover,
in #1842 and #1843, we test if Homebrow LLVM is in `PATH` by checking
that `llvm-ar` is adjacent to `clang` and if it isn't, we `fail()` the
build, even if `llvm-ar` *is* in `PATH`.
Instead, actually check `PATH` with `repository_ctx.which`. This however
necessitates the assumption that `llvm-ar` and other LLVM binutils
are adjacent, and subsequently that `clang` and `ld.lld` are adjacent.
It appears that we don't seem to always be using these tools, but we
should avoid embedding wrong assumptions regardless.
Update docs to reflect this change.
> The only issue that I currently have with this one is that the `clangd` extension is having some hiccups, not being able to find includes like `<gmock/gmock.h>` and `<benchmark/benchmark.h>` and thus showing extra errors. However, this only affects the VSCode extension and bazel seems to work fine.
This might be an issue with the compilation database: you might try ./scripts/create_compdb.py to help.
Fixes#1527
On Debian, the zlib1g-dev package is required to build and should be given as a requirement.
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Older versions of GDB (before version 10.1) don't work with the
DWARF v5 debugging format the LLVM uses. The error message
which GDB shows when this happens is a bit misleading though.
Provide documentation about how to deal with that error message.
This proposal establishes a plan for moving away from the embedded copy
of LLVM and instead downloading it with Bazel.
The goal is that after this lands, we will do a history-rewrite to
cleanup the repository. There are instructions on how folks can move any
in-flight work over to the newly tidied repo.
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: josh11b <josh11b@users.noreply.github.com>
Moves common script logic into utils.py (not a great name, but couldn't come up with better). This is in particular to make the buildifier.py script really trivial, allowing that pre-commit to be easily added. However, scripts have also been diverging on how we find bazel, so I'm trying to unify that.
The advantage of reimplementing buildifier's pre-commit is that (a) we can now run buildifier server-side, and (b) we can stop advising installing it manually. Then the only Linux-specific package manager is Cargo, which is only used for watchman, which is optional -- so stop highlighting Linux-specific package managers in the tool instructions.
- brew instead of pyenv to centralize package management
- can't recall why we used pyenv before, may not have been the best choice.
- pyenv has been a burden in updating versions, a single `brew upgrade` works better
- python3.9 because it's the latest and greatest, feels weird going to old versions if we're recommending installs.
- bazel 5.0.0 pre-release due to https://github.com/bazelbuild/bazel/commit/b9fc66d327debcfbdb2964afdba35a1cc8919b81
- earlier versions want `python` to be on the path, which pyenv did but brew does not
For most people this will mean: `rm -rf ~/.pyenv && brew uninstall pyenv && brew install python@3.9 && pip3 install -U pip && pip3 install pre-commit`, plus removing any `pyenv` invocations from the shell `rc` file.
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Now that LLVM 12 has been released we no longer have any need to
bootstrap LLVM to get the desired featureset. LLVM 12 is available
widely, including in Homebrew across multiple platforms and in the
GitHub action runners.
Sadly, the Linux distribution builds of LLVM-12 are largely broken and
not as useful for us. The Homebrew Linux install was also broken
originally, but I've worked extensively with the Homebrew folks to get
the Linux install into a really good shape. It should now work reliably.
There are two primary bugs in Linux LLVM packages that need to be fixed
before we can just use them:
- https://bugs.llvm.org/show_bug.cgi?id=43604
- https://bugs.llvm.org/show_bug.cgi?id=46321
Once those are addressed and point releases with the fixes widely
available we can further simplify things.
Even with the need to use Homebrew installs, using the released LLVM has
the extra advantage of making it easy to properly support Darwin ARM and
I've added that configuration so that I can test things there.
Last but not least, this will significantly shrink our build outputs
which should allow building much more in continuous integration on
GitHub actions without exceeding the action cache size limits. I've even
added several tweaks and adjustments to the compile and build flags to
improve the build performance and reduce the build output size.
Once this is landed and stable, we can consider adding the refactoring
tooling back to our CI.
One of the biggest downsides of this path is that our CI has to download
and install the LLVM toolchain from Homebrew on each run. This is pretty
slow (takes a couple of minutes). But it is a fixed overhead -- it won't
get worse over time. Eventually, we can either look at a much fancier
action configuration to avoid this or hopefully the Debian packages will
get updated and we can move back to those.
The bootstrapping has served us long enough at this point. We can
resurrect it if we ever find a compelling reason for breaking off of the
latest LLVM release as our host toolchain.
Co-authored-by: Jon Meow <46229924+jonmeow@users.noreply.github.com>
This matches the version on Ubuntu LTS and other OSes. The only problem
I found with it in our testing is that Bazel confusingly sets the locale
to use `LANG=en_US` by default which breaks UTF-8 support. We may need
to shift this on Windows, but this seems like a reasonable first step.
Co-authored-by: Jon Meow <46229924+jonmeow@users.noreply.github.com>
Notes versus what jsiek wrote:
- This adopts Bazel for building.
- System-local versions of bison/flex are used. I found https://github.com/jmillikin/rules_bison, but those print a lot of warnings (things like -Wsign-compare IIRC) which makes builds hard to read. Plus I think the underlying bison_cc_library rule didn't work, so this would really only get a hermetic bison/flex build (helpful, but didn't seem worth more time).
- I'm adding in a .bazeliskrc to push a somewhat more standard choice of bazel versions. I noticed I was getting unstable versions by default, possible Google-specific, but seemed good to include.
- The `-lpthread` kludge.
- Turn all of the examples into golden tests.
- Including adding a golden test rule.
- Fixed various style guide issues. For example:
- Fixing function names to be CamelCase instead of snake_case (https://google.github.io/styleguide/cppguide.html#Function_Names)
- Removed exception use (https://google.github.io/styleguide/cppguide.html#Exceptions)
- File name fixes (https://github.com/carbon-language/carbon-lang/blob/trunk/docs/project/cpp_style_guide.md#file-names)
- Dropped `using` of `std` names -- I believe this is preferred (maybe we should be explicit about this in the Carbon style guide)
- Switched `enum` uses to `enum class` for ease-of-identification.
- Spent some time breaking out files to hopefully be easier to read/edit pieces, and understand relations between structs.
- Added `code requires` to `syntax.ypp` to address include issues
Possibly other things -- but the fundamental structure is, I believe, unchanged. I put in the golden tests pretty early to ensure I wasn't mutating output/results.
Per chandlerc's comment:
First, this builds inside the Bazel tree rather than in the source
repository. This ensures we start in a clean directory each time the
repository rule is run again. We don't need to detect an existing build
with this, and it will only be rebuilt when the workspace file or the
repository rule implementation is changed. This can be forced by using:
```
bazel sync --configure
```
Second, we use an implicit dependency on the `WORKSPACE` file to locate
the workspace directory automatically, and the `HEAD` file from the
`llvm-project` submodule to trigger a rebuild if the submodule is
updated.
Third, teach the CMake script to try to use a system-installed `clang`
if installed and not overridden by the `CC` environment variable. This
is very different from the prior logic -- this is only used with the
CMake build, and so should work with any system C++ compiler that can
build Clang and LLVM.
Lastly, this tweaks the CMake options to tune this build given that we
now fully control it and it will only be used in this context. This
still results in a 1.5gb build for me. =/ But its as small as I can make
it really. It's a frustrating long list, but I couldn't find a more
brief way of representing this.
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
- bazel/bazelisk because it's kinda needed
- buildifier per #236
- Clang/LLVM I know a number of people are familiar with, but I wasn't...
- `go get` and `pip` because it's feeling more and more appropriate to separate out package managers vs other tools
Adopts new copyright and markdown toc checks.
The new toc check generates the toc header, so that's why all the md files changed (this had felt better to me for the long-term, more auto-generated content)
Nathan Youngman
Chandler Carruthandjosh11b
Richard Smith
Jon Ross-Perkins
Sree Dananjay S
Lucile Rose NihlenandChandler Carruth
Samiur KhanandJon Ross-Perkins
Avi levy
Nico Z
cabmeurer
3405691582
Vitalii
yale
SADIK KUZU
Joshua Peterson
Parisa KhaleghiandJon Ross-Perkins
Wolfgang E SanyerandChandler Carruth
mconst
Jon Meow
Dave Abrahams
Matt Godbolt
Richard Smith