This makes more modern CPU instructions available. I noticed that we weren't already doing this when working on another bit of code where its actually relevant. This doesn't make a big difference for any of the few benchmarks we have at the moment, but it seems like a good idea.
Modern Clang/LLVM support this exact spelling both on x86 and ARM CPUs, so its surprisingly portable. I've tested it on my ARM mac just in case.
I've picked specific arch flags here because using detection with `native` seems to run into issues in the GitHub actions. Sadly, the x86 macOS runners force a somewhat minimal set of features for x86, but it will still give us consistent results.
Makes explorer/testdata/assoc_const/rewrite_large_type.carbon NOAUTOUPDATE and no-trace because otherwise it takes ~130s to run. With this it's sub-second, explorer is just dumping a lot of trace output (maybe still something to fix).
The intent of this approach is to eliminate recursion limits as a barrier for the parser. While it may not be urgent to address, I want to avoid pouring effort into a parser approach that we don't think will be usable long-term.
Right now this is passing a minor set of tests. It's intended to be enough to show how I'm thinking about flow control for the parser. I'm manually switching back and forth because it seemed like the easiest approach that avoids duplicating tests.
I'm thinking about how to handle multiple files, and I think the current IRFactory is useful as a file-focused thing. So shifting/renaming accordingly. (doing this in its own PR to make the history a little cleaner for git's move detection)
LLVM_SYMBOLIZER_PATH is required if `llvm-symbolizer` isn't in the developer's PATH. This sets it via bazel instead of having a developer handle it. I noticed this because the apt install of clang doesn't put llvm-symbolizer in the PATH.
I'd like to make this the default without putting it everywhere, but I don't see a way to do this intrinsically through [the toolchain](https://bazel.build/docs/cc-toolchain-config-reference), and the [rules_cc/defs.bzl](https://github.com/bazelbuild/rules_cc/blob/main/cc/defs.bzl) remains a thin wrapper around the native cc_binary.
Since I'm adding another env, it seems undesirable to have the macos asan workaround separate. As a consequence, this merges it in. Note bazel doesn't support merging a dict and a select, so it's also necessary to have the two env vars at least mildly aware there's something up (and this could get worse if we end up having more selects).
This had already hit one bug in LLVM, and now I've hit another:
https://github.com/llvm/llvm-project/issues/58385
Since this isn't specific to a config, and it seems to be a clear sign
that this isn't the best tested path, let's just drop to the more fully
tested flags. Probably should have done this rather than the more
targeted workaround last time...
There are lots of ways a declaration can be used before we have the information necessary to handle that use. Issue diagnostics for these.
Interleave declaration and type-checking of global declarations so that declaring a later declaration can depend on the results of type-checking an earlier one.
Incorporates tests added in #2266.
Fixes#1394, fixes#1395, fixes#1396.
Co-authored-by: pmqtt <51272730+pmqtt@users.noreply.github.com>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Adds a simple script to merge stdout/stderr and put on labels. This is hidden to the RUN line using lit.cfg.py.
This is my solution to addressing how errors printed by the toolchain break sorting of stdout output; just put stdout first. We could also have toggles for interleaving output or such, which might help test whether we do it properly.
This also moves some previous-distributed replacement logic into lit_autoupdate_base.py: I think having that adjacent to lit.cfg.py is probably the better choice, and it reduces duplication in toolchain scripts. It happens here because I need to change the resulting commands to include the merge.
I've refactored the script in order to make it work in more contexts, which is why the delta is lost. I've actually refactored a significant amount with the intent of making the logic easier to understand, because I was also adjusting bits of it.
Some key notes:
- Removes the multi-pass update that was dealing with unfixed line numbers in explorer (I think the current script should work in one pass)
- Fixed explorer to handle multiple line numbers on the same line (turns out we can rely on local format for line numbers).
- Using execv instead of imports because making Python imports work in a setup like this feels like it's not worth it; only a nuisance.
- Adding __init__.py to satisfy mypy, which otherwise considers the lit_autoupdate.py scripts to be issues.
- Using py because I was thinking sh would be more platform-dependent. py should port better to Windows.
- Getting rid of [[ID#]] capture groups in the semantics-ir tests because with the autoupdate it's kind of moot (also, hard to autogenerate the pairs without relying on the %### value).
Note this does mean tests switch to more of a "make a change, see which tests change" setup. I don't know that that's a _bad_ thing though -- it's pretty much how tests are being written right now, which is why I went down this rabbit hole. It's a nuisance to make a change then _manually_ have to update a bunch of code.
My intent is to use this for to convert parse-tree tests to lit, but I wanted to do this with _existing_ tests first as a proof of concept and to make sure there's agreement.
On #2224 @zygoloid pointed out we needed --implicit-check-not to ensure we were correctly matching output. This is the standard way we're writing explorer tests, so I was looking at unifying our lit approaches.
This is one take on it, making more use of substitutions to bring various testing into alignment, as well as symlinks to avoid config skew (maybe I'll eventually figure out a better solution than symlinks).
Makes a couple small fixes in explorer to remove end-of-line whitespace on output.
Dashes in bazel package names in particular can cause problems (which renaming `llvm-patches` to `llvm_patches` will fix for me). But we've generally named files with underscores, so there's also consistency.
We've been having issues with asan builds on linux. This change should fix all of that. A build run can be found at:
https://github.com/carbon-language/carbon-lang/actions/runs/3093378863/jobs/5005683059
(currently in progress, but I'm expecting it to succeed at this point)
It may be that the issues with asan builds were actually related to caching. That is, maybe the brew build command didn't change enough between v14 and v15 that the cache hits were still an issue. We did notice this with 15.0.0 versus 15.0.1 include paths (that is, bazel wasn't happy using the cached results of a 15.0.0 build due to the skew in include paths). In order to address this, I've added CACHE_VERSION to the remote_cache setup. I've also set up corresponding buckets in Cloud.
However, I'm also switching Linux to llvm-15 and apt. I'd originally been looking at this because the issues were linux-specific, and we've previously had linux-specific issues with Homebrew. Although it may have been the cache all along, I would prefer to keep this setup (if nothing else, it made the caching issues more obvious, even though we were still confused by the include path manifestation).
The debug flag change is discussed at https://github.com/llvm/llvm-project/issues/57637
This modifies the devcontainer Dockerfile to switch to an ubuntu and apt-based llvm-15. That was used in testing of these changes. The move away from brew is partly necessary if we want llvm-15, but also installs much faster (roughly 90s setup).
This was based in part on #1618
Note this will need to be set per-cc_binary, but I don't think there's a good way to avoid that.
I didn't try hijacking the `cc_binary` rule name because that felt a bit excessive. We probably will have a few binaries we want to run directly, but I don't think it needs to be addressed on every last one.
This is part of addressing #1404
This PR fixes issue #298. I am attempting to build Carbon using Bazel on Windows, without the need to go through WSL. This is still a work in progress, below is a list of issues I have discovered and my fixes for them.
- Windows does not support Homebrew; however, I was able to properly install all of the required packages through [Chocolatey](https://chocolatey.org/). This was my first time using the Chocolatey package manager, yet it was fairly easy to use. I believe there are other options as well for Windows.
- As is mentioned in issue #298, your Windows installation must be running in [Developer Mode](https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development). This will allow unprivileged users to create symlinks during the build process.
- As it currently stands in trunk, clang_configuration.bzl will fail when attempting to run the method `_compute_clang_cpp_include_search_paths` on Windows. I have discovered that this is because Clang++.exe fails to execute if you provide it with an input file that does not exist. Thus, I have the build script generate an empty temp file in the Bazel repo for Clang to use when running the above method.
- A cc toolchain specifically for Windows was created in clang_toolchain.BUILD. I simply mimicked what was done for other platforms, I apologize if this is incorrect since I am fairly new to Bazel.
As I understand it, the next step is to configure clang_cc_toolchain_config.bzl to support the new Windows toolchain. I intend to continue working on this, however as I stated I am quite new so help is seriously appreciated!
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
The toolchain embeds the assumption that `clang` and `llvm-ar` are
adjacent, which may not be true on all host platforms. Moreover,
in #1842 and #1843, we test if Homebrow LLVM is in `PATH` by checking
that `llvm-ar` is adjacent to `clang` and if it isn't, we `fail()` the
build, even if `llvm-ar` *is* in `PATH`.
Instead, actually check `PATH` with `repository_ctx.which`. This however
necessitates the assumption that `llvm-ar` and other LLVM binutils
are adjacent, and subsequently that `clang` and `ld.lld` are adjacent.
It appears that we don't seem to always be using these tools, but we
should avoid embedding wrong assumptions regardless.
Update docs to reflect this change.
By adding a small, proactive safety check, the hope is that this will better point developers in the direction of a fix and reduce the volume of duplicate issues filed for this common troubleshooting question.
This change seems to be safe in the present as this check mirrors the current behavior in `clang_cc_toolchain_config.bzl`, finding these tools in the same directory beside `clang`. Non-standard builds using the same bazel toolchain config should not be affected for this reason. If that behavior ever changes in the future, this check will become out of sync.
Open to ideas for better heuristics here! However, checking for `llvm-ar` seems to satisfactory for right now. Verified this change worked locally (macOS) with and without Homebrew's LLVM in `PATH`.
Might want to wait on review from @chandlerc.
Fixes#1842.
There were compile actions that should use `clang` that we didn't
include because they aren't *technically* C compile actions. For our
toolchain though, we treat them as such, so create a list of actions
that we compile equivalently to C, and use that to set it up.
This will fix a problem with top-of-tree LLVM where we need to handle
preprocessed assembly files.
This proposal establishes a plan for moving away from the embedded copy
of LLVM and instead downloading it with Bazel.
The goal is that after this lands, we will do a history-rewrite to
cleanup the repository. There are instructions on how folks can move any
in-flight work over to the newly tidied repo.
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: josh11b <josh11b@users.noreply.github.com>
Fixes#1173
There is a long standing crash in the LLVM code generator that we manage
to hit when fuzzing. Disable the fast instruction selector in the
fuzzing config to avoid it. I reduced a test case and filed the LLVM bug
here: https://github.com/llvm/llvm-project/issues/56133
I'm treating my PR to mypy bazel integration as dead in the water. Abandoning that approach, this seems to work reasonably well as a replacement. It's actually a slightly newer version of mypy too -- the bazel integration was stuck on an old version.
Note I don't think we're losing much: we shouldn't be writing too much python.
I went down this route because I figured the installer binary should be in `check_deps`. The result:
- Add thin wrapper scripts for `buildozer` and `bazel`, similar to the `buildifier` wrapper that's already there.
- These are mainly to avoid deduplicating logic, and dodging intricacies of Python imports that I don't understand well through instead subprocessing.
- Fix update_roots.py (it was missing a `.parent`, must've been relocated without a re-run at some point)
- Fix fuzzer proto code to be `testonly`
- Update the `check_deps` targets
* finished fuzzer and added fuzzverter util
* fixed typo
* renamed cmd line params
* fixed libproto_mutator download path
* small fixes
* small fixes
* small fixes
* renamed sample corpus proto
* small fixes
* try building on github with LIBCPP_DEBUG enabled
* temporarily marked proto fuzzer as a manual test
* code review
* use a dedicated proto-fuzzer feature to work around LIBCPP_DEBUG=1 crash in proto code
* code review comments, added README.md
* minor fixes to the text
* Update bazel/cc_toolchains/clang_cc_toolchain_config.bzl
Co-authored-by: Jon Meow <jperkins@google.com>
* use Carbon source representation for "empty Main()" instead of text format proto representation
* fixed typo
* made FuzzerUtil produce the full carbon source (proto converted + Main if needed) to decrease code duplication a bit
* typo
* switched to text proto format per code review
* Update executable_semantics/prelude.h
Co-authored-by: Jon Meow <jperkins@google.com>
* Update executable_semantics/fuzzing/README.md
Co-authored-by: Jon Meow <jperkins@google.com>
* Update executable_semantics/fuzzing/README.md
Co-authored-by: Jon Meow <jperkins@google.com>
* Update executable_semantics/fuzzing/README.md
Co-authored-by: Jon Meow <jperkins@google.com>
* Update executable_semantics/fuzzing/README.md
Co-authored-by: Jon Meow <jperkins@google.com>
* Update executable_semantics/fuzzing/README.md
Co-authored-by: Jon Meow <jperkins@google.com>
* review comments
* removed unnecessary file mode variables
* Update executable_semantics/fuzzing/fuzzverter.cpp
Co-authored-by: Jon Meow <jperkins@google.com>
* Update executable_semantics/fuzzing/README.md
Co-authored-by: Jon Meow <jperkins@google.com>
* Update executable_semantics/fuzzing/README.md
Co-authored-by: Jon Meow <jperkins@google.com>
* code review comments
* Update executable_semantics/syntax/BUILD
Co-authored-by: Jon Meow <jperkins@google.com>
* buildifier
Co-authored-by: Jon Meow <jperkins@google.com>
* initial fuzzer proto
* visibility change
* use newer protocol buffer version which has the defs.bzl bug fixed
* Adjusted fix_cc_deps to work with protobuf external repo
* explicitly load rules_cc to avoid a frozenset bug in the version loaded by protobuf
* use explit deps, use llvm's zlib
* restored cxx settings
* deps change
* Cleaned up WORKSPACE and changed the test to read carbon sources from testdata
* updated comment
* adapted to new ErrorOr return value
* proto buffer 3.19.2 -> 3.19.4
* changed comment
* Apply suggestions from code review
Co-authored-by: Jon Meow <jperkins@google.com>
* Apply suggestions from code review
Co-authored-by: Jon Meow <jperkins@google.com>
* Update executable_semantics/fuzzing/BUILD
Co-authored-by: Jon Meow <jperkins@google.com>
* addressed review comments
* updated Unimplemented error message
* Addressed review comments
* more review comments
* switched to loading protobuf via rules_proto()
* Ignore protobuf headers in fix_cc_deps.py until the script supports alias rules
* renamed repeated proto fields to be plural
* added @zlib to check_non_test_cc_deps
* Update common/fuzzing/BUILD
Co-authored-by: Jon Meow <jperkins@google.com>
* review comments
* set is_omitted_expression for return value
Co-authored-by: Jon Meow <jperkins@google.com>
This switches to generating per-file targets such as `//executable_semantics/testdata:tuple/equality_false.carbon.test`, instead of one test covering all files. Essentially this relies on bazel instead of lit to handle test parallelization.
I think lit_test accrued a bit of cruft as I switched approaches... I still may go further, but this is particularly fixing a bug where `FileCheck` (instead of `%{FileCheck}`) should've failed in tests.
This avoids needing to have nearly as many rules here which should
reduce its churn.
I've tested that this reaches the exact same set of transitive
dependencies.
Note, only the last commit here is new.
This moves over to the vanilla upstream GoogleTest pulled in the more
expected manner with Bazel. It also adds Abseil and Google Benchmark
libraries in the same fashion (there are cross dependencies here).
As part of this, also introduce a dependency check test that can enforce
basic layering of dependencies. For example, this lets us ensure that
non-test Carbon code only depends on LLVM and Clang despite having other
libraries available. There remains some cleanup to improve the way these
dependency tests work, but this at least ensures we don't regress.
I've also provided workarounds to allow both Carbon code and LLVM code
to freely be used with GoogleTest (and other `std::ostream` based
output code). This is done by extending the code in
`//common/ostream.h`. One downside is that it requires opening the
`llvm` namespace and adding an ADL_found overload there. I think on
balance this is still a win and doesn't make me too nervous.
The new version of GoogleTest requires printing more often from matchers
and so I've also added several printing routines to types that
previously didn't require them. Otherwise, most of the updates are just
using the more conventional upstream style of including the headers and
adding `ostream.h` where it is needed.
I did consider moving code over to use `std::ostream` instead of LLVM's
`raw_ostream`, but the advantages of not doing virtual dispatch still
seem significant, and it also seems good to retain access to LLVM's
formatting utilities built around `raw_ostream` given that we can't pull
arbitrary dependencies into Carbon code outside of test code.
All of this was slightly motivated by requests for newer features in
GoogleTest, but much more-so by my desire to have access to Google
Benchmark and Abseil when writing benchmarks. For example, using
Abseil's random number generator seems extremely helpful when generating
inputs for benchmarks. The growing dependencies between these packages
further motivated me to just pull them all in and ensure they worked
well.
Chandler Carruth
Jon Ross-Perkins
micttyl
ethangandJon Ross-Perkins
Metarsit
3405691582
jkamins7andJon Ross-Perkins
Calvin
Krishna Agarwal
Josh Soref
pk19604014
Jon Meow