I'm partly doing this because the current setup would be difficult to share with the toolchain. e.g., ProtoToCarbon isn't explorer-specific, but the only way to run it via CLI is the explorer's fuzzverter. I want a separate tool.
This change:
- Adds a //common/fuzzing:proto_to_carbon tool.
- The rest of fuzzverter is now just //explorer/fuzzing:ast_to_proto.
- The change simplifies overall handling and removes a LLVM CLI dependency.
- Stops allowing unknown fields in the proto.
- This has mostly led to forgetting to remove fuzzer inputs that were for removed features.
- Moves more non-explorer-specific bits to //common/fuzzing.
- Cleans up remaining pieces in //explorer/fuzzing
- Merges the //explorer/fuzzing proto tests, which deduplicates AstToString copies.
- These tests also had duplicate dependencies, etc -- and all complete in ~6s.
- Updates and fixes regen_corpus which was previously broken by other changes.
- Updates the README to reflect changes.
- Removes obsolete proto-fuzzer build configuration (AFAICT this is no longer needed).
Protobufs code hits a warning with the latest system headers on macOS.
I figured this may have been fixed so I updated protobufs and Bazel to
the latest releases. This generally cleaned things up.
However, it actually added *more* warnings. This clearly isn't a really
well tested path. In fact, we already have a disabled warning that we'd
like for Carbon code because LLVM isn't clean for that warning.
So I've switched our warning strategy to a more durable approach of
suppressing all warnings for external repository headers and source
files. This lets us re-enable the missing warning and should fix the
protobuf warning that started me down this twisty path.
Sadly, we *have* to update to Bazel 6 in order to have the necessary
flag to use this approach to suppressing warnings, so I couldn't do this
as two PRs cleanly. =/ That's why I've bundled both the Bazel (and
protobuf) updates with the warning strategy change.
Last but not least, I've fixed several unused parameters in Carbon's
code that our warnings now catch.
* finished fuzzer and added fuzzverter util
* fixed typo
* renamed cmd line params
* fixed libproto_mutator download path
* small fixes
* small fixes
* small fixes
* renamed sample corpus proto
* small fixes
* try building on github with LIBCPP_DEBUG enabled
* temporarily marked proto fuzzer as a manual test
* code review
* use a dedicated proto-fuzzer feature to work around LIBCPP_DEBUG=1 crash in proto code
* code review comments, added README.md
* minor fixes to the text
* Update bazel/cc_toolchains/clang_cc_toolchain_config.bzl
Co-authored-by: Jon Meow <jperkins@google.com>
* use Carbon source representation for "empty Main()" instead of text format proto representation
* fixed typo
* made FuzzerUtil produce the full carbon source (proto converted + Main if needed) to decrease code duplication a bit
* typo
* switched to text proto format per code review
* Update executable_semantics/prelude.h
Co-authored-by: Jon Meow <jperkins@google.com>
* Update executable_semantics/fuzzing/README.md
Co-authored-by: Jon Meow <jperkins@google.com>
* Update executable_semantics/fuzzing/README.md
Co-authored-by: Jon Meow <jperkins@google.com>
* Update executable_semantics/fuzzing/README.md
Co-authored-by: Jon Meow <jperkins@google.com>
* Update executable_semantics/fuzzing/README.md
Co-authored-by: Jon Meow <jperkins@google.com>
* Update executable_semantics/fuzzing/README.md
Co-authored-by: Jon Meow <jperkins@google.com>
* review comments
* removed unnecessary file mode variables
* Update executable_semantics/fuzzing/fuzzverter.cpp
Co-authored-by: Jon Meow <jperkins@google.com>
* Update executable_semantics/fuzzing/README.md
Co-authored-by: Jon Meow <jperkins@google.com>
* Update executable_semantics/fuzzing/README.md
Co-authored-by: Jon Meow <jperkins@google.com>
* code review comments
* Update executable_semantics/syntax/BUILD
Co-authored-by: Jon Meow <jperkins@google.com>
* buildifier
Co-authored-by: Jon Meow <jperkins@google.com>
I didn't fully configure the new dependencies correctly or fully get
them working with our tooling rigging for compilation databases.
- I needed to fix the sha256 of the benchmark. I pasted the wrong
one, but didn't test it effectively.
- Didn't successfully enable the use of Abseil from GoogleTest
(including nice things like its symbolization, etc). Doing this is
a bit awkward as it needs to go into our `.bazelrc`, but it works.
- Didn't add libraries other that GoogleTest to the compile flags.
- Didn't teach the compilation database creation step to cause these
external repositories to be linked in and populated nicely.
All of these are fixed. As I was making changes to the Python script
here, I've added a test to at least type check it and fixed the type
errors reported.
Also add one more flag to our `.bazelrc` to improve the overall build
efficiency.
While the first build with this will be very slow (likely a couple of
hours), the caching layer we are using now should effectively cache this
for subsequent builds so that only the changed files have to be
recompiled. Updates to Clang and LLVM themselves will still result in
a slow rebuild, but normal development should remain fast.
Currently, Bazel's disk cache grows without bound:
https://github.com/bazelbuild/bazel/issues/5139
Until this is fixed, provide a script to remove entries after 30 days.
If things are changing so rapidly that we need a more aggressive
threshold, we can adjust as we go. I'm somewhat hoping that we don't end
up with *that* many build artifacts, but let's see.
The Bazel disk cache provides a stable, durable, and supported
persistent cache of the intermediate steps of Bazel builds. This is like
`ccache` but covering more steps and more reliable.
It is based on the same fundamental infrastructure that enables
a *remote* build cache, and at some point we may want to enable that so
that our CI can pre-seed a cache for users. But for now, this just
enables local support for this.
For local builds this will ensure that intermediate artifacts are
re-used when changing branches, syncing, across Bazel server restarts or
even upgrades. It should dramatically reduce the need to re-build large
and slow components like all of Clang and LLVM. Largely, those should
only be rebuilt when we update our pinned revision of upstream and then
cached persistently on a given machine. Enabling this by default
requires hard-coding a path, and so I've also added built-in support for
importing a user config file if present so that folks can customize this
(and any other Bazel features) as desired.
This also wires the disk cache up for the GitHub cache in our CI. By
using the designated Bazel disk cache for this, we get a *much* simpler
and *much* more robust solution. So much so that I've re-arranged the
key structure so we can more aggressively re-use old caches. The only
downside is that nothing ever gets removed automatically from the cache.
I've added a layer that will trim any unused file roughly once a day.
For CI, the benefits of this are huge. None of the previous hacks are
needed to locate and preserve the cache. The design of the cache is
deeply durable and so we don't need to aggressively invalidate it. Last
but not least, it focuses the cache on the raw artifacts, which compress
exceptionally well. With the other changes to shrink the build outputs
and this one combined, I expect we will be able to enable all of the
build targets across all four config/os combinations without any
significant pressure on our space quota for GitHub action caches.
Now that LLVM 12 has been released we no longer have any need to
bootstrap LLVM to get the desired featureset. LLVM 12 is available
widely, including in Homebrew across multiple platforms and in the
GitHub action runners.
Sadly, the Linux distribution builds of LLVM-12 are largely broken and
not as useful for us. The Homebrew Linux install was also broken
originally, but I've worked extensively with the Homebrew folks to get
the Linux install into a really good shape. It should now work reliably.
There are two primary bugs in Linux LLVM packages that need to be fixed
before we can just use them:
- https://bugs.llvm.org/show_bug.cgi?id=43604
- https://bugs.llvm.org/show_bug.cgi?id=46321
Once those are addressed and point releases with the fixes widely
available we can further simplify things.
Even with the need to use Homebrew installs, using the released LLVM has
the extra advantage of making it easy to properly support Darwin ARM and
I've added that configuration so that I can test things there.
Last but not least, this will significantly shrink our build outputs
which should allow building much more in continuous integration on
GitHub actions without exceeding the action cache size limits. I've even
added several tweaks and adjustments to the compile and build flags to
improve the build performance and reduce the build output size.
Once this is landed and stable, we can consider adding the refactoring
tooling back to our CI.
One of the biggest downsides of this path is that our CI has to download
and install the LLVM toolchain from Homebrew on each run. This is pretty
slow (takes a couple of minutes). But it is a fixed overhead -- it won't
get worse over time. Eventually, we can either look at a much fancier
action configuration to avoid this or hopefully the Debian packages will
get updated and we can move back to those.
The bootstrapping has served us long enough at this point. We can
resurrect it if we ever find a compelling reason for breaking off of the
latest LLVM release as our host toolchain.
Co-authored-by: Jon Meow <46229924+jonmeow@users.noreply.github.com>
Per https://docs.bazel.build/versions/master/guide.html#option-defaults, test inherits from build, thus why this fixes caching.
The issue can be observed with `bazel build :all && bazel test :all`, the line:
`INFO: Build option --test_env has changed, discarding analysis cache.`
Adds all the necessary machinery to our toolchain and Bazel
configuration to support ASan. This includes ensuring sufficient debug
information is available for backtraces, etc.
As part of ASan, it enables UBSan to catch more basic undefined behavior
in C++. It also enables more complete checking in ASan for lifetime
bugs.
These configs can be enabled in any build mode with `--config=asan`.
They are also enabled by default in `-c fastbuild` where asserts are
also enabled. The goal is to have a single build mode that catches the
overwhelming majority of correctness issues.
Leak checking is part of ASan and finds leaks in `executable_semantics`
code that probably aren't interesting to fix right now. I've disabled
leak checking in the `BUILD` file for the test that showed this --
everything else passed. If more things need this disabled, the same
`BUILD` change should be easily replicated.
If you see unsymbolized backtraces, you may need to either put
`llvm-symbolizer` on your path, or point the `ASAN_SYMBOLIZER_PATH`
environment variable at it. For example, in the project root you could
do something like the following to use the downloaded toolchain's
symbolizer:
```bash export
ASAN_SYMBOLIZER_PATH=$PWD/bazel-clang-toolchain/bin/llvm-symbolizer
```
I'll try to update documentation soon with this as well.
This matches the version on Ubuntu LTS and other OSes. The only problem
I found with it in our testing is that Bazel confusingly sets the locale
to use `LANG=en_US` by default which breaks UTF-8 support. We may need
to shift this on Windows, but this seems like a reasonable first step.
Co-authored-by: Jon Meow <46229924+jonmeow@users.noreply.github.com>
Tested on Ubuntu 20.04 and macOS (both Intel and M1 BigSur).
The bootstrap option is still available with `--config=force_local_bootstrap`.
Note that while this works on the ARM M1 for macOS, it is using the x86-based
toolchain and Rosetta, not doing an ARM build at the moment.
You can also point at a locally installed Clang and LLVM toolchain by using
the `--override_repository` flag to Bazel.
Co-authored-by: Matthew Riley <mdriley@gmail.com>
The Bazel bits are collected into a directory and given less confusing
names (I hope). Other than names, everything is a direct copy from the
toolchain repository without any edits.
A few files needed to be merged in:
- `.gitignore`
- `.pre-commit-config.yaml`
Subsequent commits will add relevant C++ infrastructure and then the
source code itself.
Jon Ross-Perkins
Chandler Carruth
pk19604014andJon Meow
