In practice, this means that `executable_semantics` will be built and run using the configuration specified on the command line, rather than e.g. always using `-c opt`.
Also fix a bug exposed by this change.
Now that LLVM 12 has been released we no longer have any need to
bootstrap LLVM to get the desired featureset. LLVM 12 is available
widely, including in Homebrew across multiple platforms and in the
GitHub action runners.
Sadly, the Linux distribution builds of LLVM-12 are largely broken and
not as useful for us. The Homebrew Linux install was also broken
originally, but I've worked extensively with the Homebrew folks to get
the Linux install into a really good shape. It should now work reliably.
There are two primary bugs in Linux LLVM packages that need to be fixed
before we can just use them:
- https://bugs.llvm.org/show_bug.cgi?id=43604
- https://bugs.llvm.org/show_bug.cgi?id=46321
Once those are addressed and point releases with the fixes widely
available we can further simplify things.
Even with the need to use Homebrew installs, using the released LLVM has
the extra advantage of making it easy to properly support Darwin ARM and
I've added that configuration so that I can test things there.
Last but not least, this will significantly shrink our build outputs
which should allow building much more in continuous integration on
GitHub actions without exceeding the action cache size limits. I've even
added several tweaks and adjustments to the compile and build flags to
improve the build performance and reduce the build output size.
Once this is landed and stable, we can consider adding the refactoring
tooling back to our CI.
One of the biggest downsides of this path is that our CI has to download
and install the LLVM toolchain from Homebrew on each run. This is pretty
slow (takes a couple of minutes). But it is a fixed overhead -- it won't
get worse over time. Eventually, we can either look at a much fancier
action configuration to avoid this or hopefully the Debian packages will
get updated and we can move back to those.
The bootstrapping has served us long enough at this point. We can
resurrect it if we ever find a compelling reason for breaking off of the
latest LLVM release as our host toolchain.
Co-authored-by: Jon Meow <46229924+jonmeow@users.noreply.github.com>
The recovery is mostly useful when triaging multiple failures, it seems
easy for us to skip. Giving up the nice diagnostics doesn't lose much
for some types of error where there isn't any real information to
convey, so seems worth doing that for a few cases.
Removing the nice diagnostics from the rest of UBSan saves another
20%-ish of output size on LLVM, but seems like it would be giving up
usability. I think there are other approaches we can use instead so this
PR just focuses on the easy wins.
What this does:
- Sets up a `migrate_cpp` tool which currently only runs `clang-tidy`.
- This is intended to have more transformations in the future.
- Sets up a `migrate_cpp.sh` script.
- This copies the original woff2 code into a `carbon` directory and runs the `migrate_cpp` tool on it there.
- Adds the initial `carbon` directory of woff2
- To be clear, this is currently only updated via `clang-tidy`.
- More transformations should be expected in the future.
- Minor related edits. For example:
- Adjust pre-commit to skip the `carbon` directory, because it's third-party code and shouldn't be edited in the same way.
- Adds `@brotli_carbon` as a local repository so that we can "build" outputs.
- Makes clang-tidy from the bootstrap toolchain accessible for BUILD dependencies, as it's then used for `migrate_cpp`.
What this does not do:
- Any actual transformation of C++ code to Carbon
Co-authored-by: Matthew Riley <mdriley@gmail.com>
Adds all the necessary machinery to our toolchain and Bazel
configuration to support ASan. This includes ensuring sufficient debug
information is available for backtraces, etc.
As part of ASan, it enables UBSan to catch more basic undefined behavior
in C++. It also enables more complete checking in ASan for lifetime
bugs.
These configs can be enabled in any build mode with `--config=asan`.
They are also enabled by default in `-c fastbuild` where asserts are
also enabled. The goal is to have a single build mode that catches the
overwhelming majority of correctness issues.
Leak checking is part of ASan and finds leaks in `executable_semantics`
code that probably aren't interesting to fix right now. I've disabled
leak checking in the `BUILD` file for the test that showed this --
everything else passed. If more things need this disabled, the same
`BUILD` change should be easily replicated.
If you see unsymbolized backtraces, you may need to either put
`llvm-symbolizer` on your path, or point the `ASAN_SYMBOLIZER_PATH`
environment variable at it. For example, in the project root you could
do something like the following to use the downloaded toolchain's
symbolizer:
```bash export
ASAN_SYMBOLIZER_PATH=$PWD/bazel-clang-toolchain/bin/llvm-symbolizer
```
I'll try to update documentation soon with this as well.
There was a lot of redundant noise in the older form. It also had
several issues that made the ordering and mixing together of different
flags much less obvious.
With this change, the user-provided flags are reliably placed in
a useful position, along with fundamental flags like the source file and
output.
All of this is largely in preparation for trying to add the first
sanitizer configurations (as well as enabling them by default).
Tested on Ubuntu 20.04 and macOS (both Intel and M1 BigSur).
The bootstrap option is still available with `--config=force_local_bootstrap`.
Note that while this works on the ARM M1 for macOS, it is using the x86-based
toolchain and Rosetta, not doing an ARM build at the moment.
You can also point at a locally installed Clang and LLVM toolchain by using
the `--override_repository` flag to Bazel.
Co-authored-by: Matthew Riley <mdriley@gmail.com>
Notes versus what jsiek wrote:
- This adopts Bazel for building.
- System-local versions of bison/flex are used. I found https://github.com/jmillikin/rules_bison, but those print a lot of warnings (things like -Wsign-compare IIRC) which makes builds hard to read. Plus I think the underlying bison_cc_library rule didn't work, so this would really only get a hermetic bison/flex build (helpful, but didn't seem worth more time).
- I'm adding in a .bazeliskrc to push a somewhat more standard choice of bazel versions. I noticed I was getting unstable versions by default, possible Google-specific, but seemed good to include.
- The `-lpthread` kludge.
- Turn all of the examples into golden tests.
- Including adding a golden test rule.
- Fixed various style guide issues. For example:
- Fixing function names to be CamelCase instead of snake_case (https://google.github.io/styleguide/cppguide.html#Function_Names)
- Removed exception use (https://google.github.io/styleguide/cppguide.html#Exceptions)
- File name fixes (https://github.com/carbon-language/carbon-lang/blob/trunk/docs/project/cpp_style_guide.md#file-names)
- Dropped `using` of `std` names -- I believe this is preferred (maybe we should be explicit about this in the Carbon style guide)
- Switched `enum` uses to `enum class` for ease-of-identification.
- Spent some time breaking out files to hopefully be easier to read/edit pieces, and understand relations between structs.
- Added `code requires` to `syntax.ypp` to address include issues
Possibly other things -- but the fundamental structure is, I believe, unchanged. I put in the golden tests pretty early to ensure I wasn't mutating output/results.
Per chandlerc's comment:
First, this builds inside the Bazel tree rather than in the source
repository. This ensures we start in a clean directory each time the
repository rule is run again. We don't need to detect an existing build
with this, and it will only be rebuilt when the workspace file or the
repository rule implementation is changed. This can be forced by using:
```
bazel sync --configure
```
Second, we use an implicit dependency on the `WORKSPACE` file to locate
the workspace directory automatically, and the `HEAD` file from the
`llvm-project` submodule to trigger a rebuild if the submodule is
updated.
Third, teach the CMake script to try to use a system-installed `clang`
if installed and not overridden by the `CC` environment variable. This
is very different from the prior logic -- this is only used with the
CMake build, and so should work with any system C++ compiler that can
build Clang and LLVM.
Lastly, this tweaks the CMake options to tune this build given that we
now fully control it and it will only be used in this context. This
still results in a 1.5gb build for me. =/ But its as small as I can make
it really. It's a frustrating long list, but I couldn't find a more
brief way of representing this.
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
This updates the Bazel toolchain logic to work on macOS. Much like on
Linux, I'm not testing this against a *released* LLVM, but against
a from-source build. You can build and install LLVM from top-of-tree
locally with CMake, or on macOS maybe use Homebrew like:
```
brew install llvm --HEAD
```
You then need to point Bazel to the installed `clang` executable if it
is not placed onto your PATH (Homebrew doesn't):
```
bazel test --repo_env=CC=$HOME/homebrew/opt/llvm/bin/clang //parser:all
```
This builds and passes tests for me at least.
The fuzzer feature may not work (yet) with this setup, but that can be
improved incrementally as we proceed.
The Bazel bits are collected into a directory and given less confusing
names (I hope). Other than names, everything is a direct copy from the
toolchain repository without any edits.
A few files needed to be merged in:
- `.gitignore`
- `.pre-commit-config.yaml`
Subsequent commits will add relevant C++ infrastructure and then the
source code itself.
Geoff Romer
Jon Meow
Chandler CarruthandJon Meow
