The main motivation for this is to get python loads in using the
`native-py` lint fix. However, enabling that made me wonder, maybe we
should fix in general?
`native-cc` is delayed, but not wholly cancelled (and `native-py`
picking up might indicate `native-cc` won't be too far behind). There's
also some automated fixes for `.append` and dict sorting -- this felt
okay to me, maybe not something to eagerly add but probably not worth
stopping buildifier from fixing (I've noticed the warnings in the past
and had been ignoring them).
Running everything does mean that load orders are sorted automatically
now, which I think is a positive. Most generally, I think these fixes
aren't _harmful_, and having them done automatically seems beneficial:
my biggest concern about `native-py` and `native-cc` was actually that
regressions wouldn't be caught, but this addresses that issue
automatically.
I'm partly doing this because the current setup would be difficult to share with the toolchain. e.g., ProtoToCarbon isn't explorer-specific, but the only way to run it via CLI is the explorer's fuzzverter. I want a separate tool.
This change:
- Adds a //common/fuzzing:proto_to_carbon tool.
- The rest of fuzzverter is now just //explorer/fuzzing:ast_to_proto.
- The change simplifies overall handling and removes a LLVM CLI dependency.
- Stops allowing unknown fields in the proto.
- This has mostly led to forgetting to remove fuzzer inputs that were for removed features.
- Moves more non-explorer-specific bits to //common/fuzzing.
- Cleans up remaining pieces in //explorer/fuzzing
- Merges the //explorer/fuzzing proto tests, which deduplicates AstToString copies.
- These tests also had duplicate dependencies, etc -- and all complete in ~6s.
- Updates and fixes regen_corpus which was previously broken by other changes.
- Updates the README to reflect changes.
- Removes obsolete proto-fuzzer build configuration (AFAICT this is no longer needed).
Protobufs code hits a warning with the latest system headers on macOS.
I figured this may have been fixed so I updated protobufs and Bazel to
the latest releases. This generally cleaned things up.
However, it actually added *more* warnings. This clearly isn't a really
well tested path. In fact, we already have a disabled warning that we'd
like for Carbon code because LLVM isn't clean for that warning.
So I've switched our warning strategy to a more durable approach of
suppressing all warnings for external repository headers and source
files. This lets us re-enable the missing warning and should fix the
protobuf warning that started me down this twisty path.
Sadly, we *have* to update to Bazel 6 in order to have the necessary
flag to use this approach to suppressing warnings, so I couldn't do this
as two PRs cleanly. =/ That's why I've bundled both the Bazel (and
protobuf) updates with the warning strategy change.
Last but not least, I've fixed several unused parameters in Carbon's
code that our warnings now catch.
This makes more modern CPU instructions available. I noticed that we weren't already doing this when working on another bit of code where its actually relevant. This doesn't make a big difference for any of the few benchmarks we have at the moment, but it seems like a good idea.
Modern Clang/LLVM support this exact spelling both on x86 and ARM CPUs, so its surprisingly portable. I've tested it on my ARM mac just in case.
I've picked specific arch flags here because using detection with `native` seems to run into issues in the GitHub actions. Sadly, the x86 macOS runners force a somewhat minimal set of features for x86, but it will still give us consistent results.
LLVM_SYMBOLIZER_PATH is required if `llvm-symbolizer` isn't in the developer's PATH. This sets it via bazel instead of having a developer handle it. I noticed this because the apt install of clang doesn't put llvm-symbolizer in the PATH.
I'd like to make this the default without putting it everywhere, but I don't see a way to do this intrinsically through [the toolchain](https://bazel.build/docs/cc-toolchain-config-reference), and the [rules_cc/defs.bzl](https://github.com/bazelbuild/rules_cc/blob/main/cc/defs.bzl) remains a thin wrapper around the native cc_binary.
Since I'm adding another env, it seems undesirable to have the macos asan workaround separate. As a consequence, this merges it in. Note bazel doesn't support merging a dict and a select, so it's also necessary to have the two env vars at least mildly aware there's something up (and this could get worse if we end up having more selects).
This had already hit one bug in LLVM, and now I've hit another:
https://github.com/llvm/llvm-project/issues/58385
Since this isn't specific to a config, and it seems to be a clear sign
that this isn't the best tested path, let's just drop to the more fully
tested flags. Probably should have done this rather than the more
targeted workaround last time...
We've been having issues with asan builds on linux. This change should fix all of that. A build run can be found at:
https://github.com/carbon-language/carbon-lang/actions/runs/3093378863/jobs/5005683059
(currently in progress, but I'm expecting it to succeed at this point)
It may be that the issues with asan builds were actually related to caching. That is, maybe the brew build command didn't change enough between v14 and v15 that the cache hits were still an issue. We did notice this with 15.0.0 versus 15.0.1 include paths (that is, bazel wasn't happy using the cached results of a 15.0.0 build due to the skew in include paths). In order to address this, I've added CACHE_VERSION to the remote_cache setup. I've also set up corresponding buckets in Cloud.
However, I'm also switching Linux to llvm-15 and apt. I'd originally been looking at this because the issues were linux-specific, and we've previously had linux-specific issues with Homebrew. Although it may have been the cache all along, I would prefer to keep this setup (if nothing else, it made the caching issues more obvious, even though we were still confused by the include path manifestation).
The debug flag change is discussed at https://github.com/llvm/llvm-project/issues/57637
This modifies the devcontainer Dockerfile to switch to an ubuntu and apt-based llvm-15. That was used in testing of these changes. The move away from brew is partly necessary if we want llvm-15, but also installs much faster (roughly 90s setup).
This was based in part on #1618
This PR fixes issue #298. I am attempting to build Carbon using Bazel on Windows, without the need to go through WSL. This is still a work in progress, below is a list of issues I have discovered and my fixes for them.
- Windows does not support Homebrew; however, I was able to properly install all of the required packages through [Chocolatey](https://chocolatey.org/). This was my first time using the Chocolatey package manager, yet it was fairly easy to use. I believe there are other options as well for Windows.
- As is mentioned in issue #298, your Windows installation must be running in [Developer Mode](https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development). This will allow unprivileged users to create symlinks during the build process.
- As it currently stands in trunk, clang_configuration.bzl will fail when attempting to run the method `_compute_clang_cpp_include_search_paths` on Windows. I have discovered that this is because Clang++.exe fails to execute if you provide it with an input file that does not exist. Thus, I have the build script generate an empty temp file in the Bazel repo for Clang to use when running the above method.
- A cc toolchain specifically for Windows was created in clang_toolchain.BUILD. I simply mimicked what was done for other platforms, I apologize if this is incorrect since I am fairly new to Bazel.
As I understand it, the next step is to configure clang_cc_toolchain_config.bzl to support the new Windows toolchain. I intend to continue working on this, however as I stated I am quite new so help is seriously appreciated!
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
The toolchain embeds the assumption that `clang` and `llvm-ar` are
adjacent, which may not be true on all host platforms. Moreover,
in #1842 and #1843, we test if Homebrow LLVM is in `PATH` by checking
that `llvm-ar` is adjacent to `clang` and if it isn't, we `fail()` the
build, even if `llvm-ar` *is* in `PATH`.
Instead, actually check `PATH` with `repository_ctx.which`. This however
necessitates the assumption that `llvm-ar` and other LLVM binutils
are adjacent, and subsequently that `clang` and `ld.lld` are adjacent.
It appears that we don't seem to always be using these tools, but we
should avoid embedding wrong assumptions regardless.
Update docs to reflect this change.
By adding a small, proactive safety check, the hope is that this will better point developers in the direction of a fix and reduce the volume of duplicate issues filed for this common troubleshooting question.
This change seems to be safe in the present as this check mirrors the current behavior in `clang_cc_toolchain_config.bzl`, finding these tools in the same directory beside `clang`. Non-standard builds using the same bazel toolchain config should not be affected for this reason. If that behavior ever changes in the future, this check will become out of sync.
Open to ideas for better heuristics here! However, checking for `llvm-ar` seems to satisfactory for right now. Verified this change worked locally (macOS) with and without Homebrew's LLVM in `PATH`.
Might want to wait on review from @chandlerc.
Fixes#1842.
There were compile actions that should use `clang` that we didn't
include because they aren't *technically* C compile actions. For our
toolchain though, we treat them as such, so create a list of actions
that we compile equivalently to C, and use that to set it up.
This will fix a problem with top-of-tree LLVM where we need to handle
preprocessed assembly files.
Fixes#1173
There is a long standing crash in the LLVM code generator that we manage
to hit when fuzzing. Disable the fast instruction selector in the
fuzzing config to avoid it. I reduced a test case and filed the LLVM bug
here: https://github.com/llvm/llvm-project/issues/56133
* finished fuzzer and added fuzzverter util
* fixed typo
* renamed cmd line params
* fixed libproto_mutator download path
* small fixes
* small fixes
* small fixes
* renamed sample corpus proto
* small fixes
* try building on github with LIBCPP_DEBUG enabled
* temporarily marked proto fuzzer as a manual test
* code review
* use a dedicated proto-fuzzer feature to work around LIBCPP_DEBUG=1 crash in proto code
* code review comments, added README.md
* minor fixes to the text
* Update bazel/cc_toolchains/clang_cc_toolchain_config.bzl
Co-authored-by: Jon Meow <jperkins@google.com>
* use Carbon source representation for "empty Main()" instead of text format proto representation
* fixed typo
* made FuzzerUtil produce the full carbon source (proto converted + Main if needed) to decrease code duplication a bit
* typo
* switched to text proto format per code review
* Update executable_semantics/prelude.h
Co-authored-by: Jon Meow <jperkins@google.com>
* Update executable_semantics/fuzzing/README.md
Co-authored-by: Jon Meow <jperkins@google.com>
* Update executable_semantics/fuzzing/README.md
Co-authored-by: Jon Meow <jperkins@google.com>
* Update executable_semantics/fuzzing/README.md
Co-authored-by: Jon Meow <jperkins@google.com>
* Update executable_semantics/fuzzing/README.md
Co-authored-by: Jon Meow <jperkins@google.com>
* Update executable_semantics/fuzzing/README.md
Co-authored-by: Jon Meow <jperkins@google.com>
* review comments
* removed unnecessary file mode variables
* Update executable_semantics/fuzzing/fuzzverter.cpp
Co-authored-by: Jon Meow <jperkins@google.com>
* Update executable_semantics/fuzzing/README.md
Co-authored-by: Jon Meow <jperkins@google.com>
* Update executable_semantics/fuzzing/README.md
Co-authored-by: Jon Meow <jperkins@google.com>
* code review comments
* Update executable_semantics/syntax/BUILD
Co-authored-by: Jon Meow <jperkins@google.com>
* buildifier
Co-authored-by: Jon Meow <jperkins@google.com>
The LLVM-13 release is now on Homebrew, so switch to it. This is
important because our CI only installs the latest version currently.
If you hit issues after this, make sure to update your Homebrew install
to get the latest Clang release. You can always directly set `CC` in
your environment to use a specific `clang` compiler.
The big change here is to enable libc++'s debug mode outside of `opt`
builds on Linux where it seems to work well with the Homebrew installed
toolchain. I'm restricting it to Linux as the system libc++ install on
x86 macOS doesn't seem to work. This is based on and subsumes #811.
This also tidies up how `-NDEBUG` is set to include non-codegen compile
actions, and consolidates some optimization flags in a single location.
This part has no functionality change, but would likely invalidate
caches so bundled here where both a) I noticed and b) we already had
a cache invalidation.
Only the special nullability attributes (`_Nonnull`) work correctly
through type aliases like we're using with `Ptr`. But they aren't
strictly UB and so have to be specially enabled in our sanitizer config
in order to usefully catch nullness errors early. Turn on those
sanitizers as well.
Also, now that we are using fully remote build output caching for our
CI and not trying to squeeze under an arbitrary size limit, re-enable
the nice error messages for all the UBSan checks.
Note that this will have a (very) slow CI run as it will have to
recompile ~everything and upload fresh artifacts. But those should then
be effective cache hits going forward.
Now that LLVM 12 has been released we no longer have any need to
bootstrap LLVM to get the desired featureset. LLVM 12 is available
widely, including in Homebrew across multiple platforms and in the
GitHub action runners.
Sadly, the Linux distribution builds of LLVM-12 are largely broken and
not as useful for us. The Homebrew Linux install was also broken
originally, but I've worked extensively with the Homebrew folks to get
the Linux install into a really good shape. It should now work reliably.
There are two primary bugs in Linux LLVM packages that need to be fixed
before we can just use them:
- https://bugs.llvm.org/show_bug.cgi?id=43604
- https://bugs.llvm.org/show_bug.cgi?id=46321
Once those are addressed and point releases with the fixes widely
available we can further simplify things.
Even with the need to use Homebrew installs, using the released LLVM has
the extra advantage of making it easy to properly support Darwin ARM and
I've added that configuration so that I can test things there.
Last but not least, this will significantly shrink our build outputs
which should allow building much more in continuous integration on
GitHub actions without exceeding the action cache size limits. I've even
added several tweaks and adjustments to the compile and build flags to
improve the build performance and reduce the build output size.
Once this is landed and stable, we can consider adding the refactoring
tooling back to our CI.
One of the biggest downsides of this path is that our CI has to download
and install the LLVM toolchain from Homebrew on each run. This is pretty
slow (takes a couple of minutes). But it is a fixed overhead -- it won't
get worse over time. Eventually, we can either look at a much fancier
action configuration to avoid this or hopefully the Debian packages will
get updated and we can move back to those.
The bootstrapping has served us long enough at this point. We can
resurrect it if we ever find a compelling reason for breaking off of the
latest LLVM release as our host toolchain.
Co-authored-by: Jon Meow <46229924+jonmeow@users.noreply.github.com>
The recovery is mostly useful when triaging multiple failures, it seems
easy for us to skip. Giving up the nice diagnostics doesn't lose much
for some types of error where there isn't any real information to
convey, so seems worth doing that for a few cases.
Removing the nice diagnostics from the rest of UBSan saves another
20%-ish of output size on LLVM, but seems like it would be giving up
usability. I think there are other approaches we can use instead so this
PR just focuses on the easy wins.
What this does:
- Sets up a `migrate_cpp` tool which currently only runs `clang-tidy`.
- This is intended to have more transformations in the future.
- Sets up a `migrate_cpp.sh` script.
- This copies the original woff2 code into a `carbon` directory and runs the `migrate_cpp` tool on it there.
- Adds the initial `carbon` directory of woff2
- To be clear, this is currently only updated via `clang-tidy`.
- More transformations should be expected in the future.
- Minor related edits. For example:
- Adjust pre-commit to skip the `carbon` directory, because it's third-party code and shouldn't be edited in the same way.
- Adds `@brotli_carbon` as a local repository so that we can "build" outputs.
- Makes clang-tidy from the bootstrap toolchain accessible for BUILD dependencies, as it's then used for `migrate_cpp`.
What this does not do:
- Any actual transformation of C++ code to Carbon
Co-authored-by: Matthew Riley <mdriley@gmail.com>
Adds all the necessary machinery to our toolchain and Bazel
configuration to support ASan. This includes ensuring sufficient debug
information is available for backtraces, etc.
As part of ASan, it enables UBSan to catch more basic undefined behavior
in C++. It also enables more complete checking in ASan for lifetime
bugs.
These configs can be enabled in any build mode with `--config=asan`.
They are also enabled by default in `-c fastbuild` where asserts are
also enabled. The goal is to have a single build mode that catches the
overwhelming majority of correctness issues.
Leak checking is part of ASan and finds leaks in `executable_semantics`
code that probably aren't interesting to fix right now. I've disabled
leak checking in the `BUILD` file for the test that showed this --
everything else passed. If more things need this disabled, the same
`BUILD` change should be easily replicated.
If you see unsymbolized backtraces, you may need to either put
`llvm-symbolizer` on your path, or point the `ASAN_SYMBOLIZER_PATH`
environment variable at it. For example, in the project root you could
do something like the following to use the downloaded toolchain's
symbolizer:
```bash export
ASAN_SYMBOLIZER_PATH=$PWD/bazel-clang-toolchain/bin/llvm-symbolizer
```
I'll try to update documentation soon with this as well.
There was a lot of redundant noise in the older form. It also had
several issues that made the ordering and mixing together of different
flags much less obvious.
With this change, the user-provided flags are reliably placed in
a useful position, along with fundamental flags like the source file and
output.
All of this is largely in preparation for trying to add the first
sanitizer configurations (as well as enabling them by default).
Tested on Ubuntu 20.04 and macOS (both Intel and M1 BigSur).
The bootstrap option is still available with `--config=force_local_bootstrap`.
Note that while this works on the ARM M1 for macOS, it is using the x86-based
toolchain and Rosetta, not doing an ARM build at the moment.
You can also point at a locally installed Clang and LLVM toolchain by using
the `--override_repository` flag to Bazel.
Co-authored-by: Matthew Riley <mdriley@gmail.com>
Notes versus what jsiek wrote:
- This adopts Bazel for building.
- System-local versions of bison/flex are used. I found https://github.com/jmillikin/rules_bison, but those print a lot of warnings (things like -Wsign-compare IIRC) which makes builds hard to read. Plus I think the underlying bison_cc_library rule didn't work, so this would really only get a hermetic bison/flex build (helpful, but didn't seem worth more time).
- I'm adding in a .bazeliskrc to push a somewhat more standard choice of bazel versions. I noticed I was getting unstable versions by default, possible Google-specific, but seemed good to include.
- The `-lpthread` kludge.
- Turn all of the examples into golden tests.
- Including adding a golden test rule.
- Fixed various style guide issues. For example:
- Fixing function names to be CamelCase instead of snake_case (https://google.github.io/styleguide/cppguide.html#Function_Names)
- Removed exception use (https://google.github.io/styleguide/cppguide.html#Exceptions)
- File name fixes (https://github.com/carbon-language/carbon-lang/blob/trunk/docs/project/cpp_style_guide.md#file-names)
- Dropped `using` of `std` names -- I believe this is preferred (maybe we should be explicit about this in the Carbon style guide)
- Switched `enum` uses to `enum class` for ease-of-identification.
- Spent some time breaking out files to hopefully be easier to read/edit pieces, and understand relations between structs.
- Added `code requires` to `syntax.ypp` to address include issues
Possibly other things -- but the fundamental structure is, I believe, unchanged. I put in the golden tests pretty early to ensure I wasn't mutating output/results.
Per chandlerc's comment:
First, this builds inside the Bazel tree rather than in the source
repository. This ensures we start in a clean directory each time the
repository rule is run again. We don't need to detect an existing build
with this, and it will only be rebuilt when the workspace file or the
repository rule implementation is changed. This can be forced by using:
```
bazel sync --configure
```
Second, we use an implicit dependency on the `WORKSPACE` file to locate
the workspace directory automatically, and the `HEAD` file from the
`llvm-project` submodule to trigger a rebuild if the submodule is
updated.
Third, teach the CMake script to try to use a system-installed `clang`
if installed and not overridden by the `CC` environment variable. This
is very different from the prior logic -- this is only used with the
CMake build, and so should work with any system C++ compiler that can
build Clang and LLVM.
Lastly, this tweaks the CMake options to tune this build given that we
now fully control it and it will only be used in this context. This
still results in a 1.5gb build for me. =/ But its as small as I can make
it really. It's a frustrating long list, but I couldn't find a more
brief way of representing this.
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
This updates the Bazel toolchain logic to work on macOS. Much like on
Linux, I'm not testing this against a *released* LLVM, but against
a from-source build. You can build and install LLVM from top-of-tree
locally with CMake, or on macOS maybe use Homebrew like:
```
brew install llvm --HEAD
```
You then need to point Bazel to the installed `clang` executable if it
is not placed onto your PATH (Homebrew doesn't):
```
bazel test --repo_env=CC=$HOME/homebrew/opt/llvm/bin/clang //parser:all
```
This builds and passes tests for me at least.
The fuzzer feature may not work (yet) with this setup, but that can be
improved incrementally as we proceed.
The Bazel bits are collected into a directory and given less confusing
names (I hope). Other than names, everything is a direct copy from the
toolchain repository without any edits.
A few files needed to be merged in:
- `.gitignore`
- `.pre-commit-config.yaml`
Subsequent commits will add relevant C++ infrastructure and then the
source code itself.
Jon Ross-Perkins
Samiur KhanandJon Ross-Perkins
Chandler Carruth
micttyl
ethangandJon Ross-Perkins
Metarsit
3405691582
jkamins7andJon Ross-Perkins
Calvin
Josh Soref
pk19604014
Geoff Romer
