I'd kept these in to separate the bazel module update from the BUILD
file changes, then forgot about it. I think all of these can be cleanly
removed now. I think it's something we should clean up for consistency
with the bazel central repository names; I think it's best to reduce
that divergence.
llvm_zlib and llvm_zstd remain because of how llvm depends on the
particular names.
This handles toolchain failures per-file. The intent is to allow placing
both "success" and "fail" tests in the same file, using splits. However,
this PR only adds support and updates existing tests to continue
passing.
Update the compiler-rt patch (maybe this could be upstreamed, or try out
https://github.com/google/fuzztest?) and handle the ThreadPool ->
DefaultThreadPool rename.
When we always had a single file in the test file, we could run the
driver over it to see output. This adds the ability to do something
similar for multi-file tests. So for example in test output, there's
now:
```
[ RUN ] ToolchainFileTest.toolchain/check/testdata/class/fail_todo_import.carbon
To test this file alone, run:
bazel test //toolchain/testing:file_test --test_arg=--file_tests=toolchain/check/testdata/class/fail_todo_import.carbon
To view output, run:
bazel run //toolchain/testing:file_test --test_arg=--file_tests=toolchain/check/testdata/class/fail_todo_import.carbon --test_arg=--dump_output
[ OK ] ToolchainFileTest.toolchain/check/testdata/class/fail_todo_import.carbon (64 ms)
```
And here's what a run looks like:
```
$ bazel run //toolchain/testing:file_test --test_arg=--file_tests=toolchain/check/testdata/class/fail_todo_import.carbon --test_arg=--dump_output
(elided bazel output)
Executing tests from //toolchain/testing:file_test
-----------------------------------------------------------------------------
===============================================================================
= toolchain/check/testdata/class/fail_todo_import.carbon
===============================================================================
= stderr
===============================================================================
b.carbon: ERROR: Semantics TODO: `TryResolveImportRefUnused on ClassDecl`.
b.carbon: ERROR: Semantics TODO: `TryResolveImportRefUnused on ClassDecl`.
b.carbon: ERROR: Semantics TODO: `TryResolveInst on ClassType`.
b.carbon: ERROR: Semantics TODO: `TryResolveInst on ClassType`.
b.carbon:18:29: ERROR: Name `d_ref` not found.
var d: (ForwardDeclared,) = d_ref;
^~~~~
===============================================================================
= stdout
===============================================================================
--- a.carbon
(eliding semir output)
}
===============================================================================
= Exit with success: false
===============================================================================
Done!
```
This also switches from PrettyStackTraceFormat to
PrettyStackTraceString. This is partly so that we can share the produced
command (avoiding two different format strings corresponding to the same
value), but also it's in my mind to step away from
https://github.com/llvm/llvm-project/pull/77351.
Recent runs of `clang-tidy` for me started showing more errors, and this
is a collection of changes to address them.
First, I've systematically applied the disabling tag to all C++ rules
under //explorer/... with `buildozer` so we don't spend time analyzing
this code or reporting errors from it. Not sure this was strictly
necessary, but it seemed like a nice consistency improvement.
Next, I disabled a buggy check for missing `default` cases in
`switch`es. It seems to get confused by the fancy conversions in our
`enum_base.h`. We don't miss much with this as the Clang compiler
warnings for `switch` catch most of our actual bugs. I also removed the
local disabling of this now that it is turned off centrally.
Lastly, I added error checking to two file descriptor manipulating calls
in the `file_test` infrastructure.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Choosing to make the constructor explicit in the test, rather than
NOLINT, because it seems to better reflect how our code is usually
written (and may be more likely to trip an issue).
Per their documentation, the `llvm::Initialize*` functions are only
supposed to be called by the main program, not by a library like
toolchain/codegen. Fixes a hang due to a data race in multithreaded
autoupdate.
Add a utility class `Carbon::InitLLVM` to do the common LLVM
initialization shared by all Carbon tools, optionally including
initializing the LLVM targets. Because the LLVM targets add a lot of
binary size, only initialize them for binaries that opt in by depending
on a new target `//common:all_llvm_targets`.
Also fix `//explorer:file_test` and `//explorer:file_test.trace` to
share a binary rather than linking an identical binary twice.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Also switch how we ensure that stdin is closed for tests, so that `bazel
run` doesn't hang if invoked manually.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This now puts file content into a string, allowing conflict markers to
be elided from file content. When code executes, this means it executes
without seeing conflict markers, without a temporary update to the file
that would only remove conflict markers.
Also refactors the main process flow, because it was getting a little
too lengthy. This means passing a bunch of parameters passed around
(partly because TestContext is private on the test class, and I don't
want to change that). I'm hoping that overall it's easier to read the
core loop now.
Mostly tested with some manually added conflict markers, and that
current tests don't change.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Running `bazel test //...` reported:
```
Test execution time outside of range for MODERATE tests.
Consider setting timeout="short" or size="small".
```
This change adds size="small" to avoid such warnings being reported.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Fix off-by-one error: the terminating `nullptr` in `argv` is not
included in `argc`. This is currently causing crashing tests to crash
again in their crash handler, meaning we don't get a backtrace or
CHECK-failure message.
Don't assume that the bazel-supplied environment variable TEST_TARGET is
present. We don't actually need it for anything other than providing
feedback to the developer if the test fails.
This makes it a bit easier to reproduce test failures under gdb,
particularly for multi-file tests.
Minor changes to the blaze command line executed by our autoupdate
scripts:
- Don't change the convenience symlinks. Running autoupdate shouldn't
cause `./bazel-bin/...` to switch to running a different binary.
- Don't produce so much spam. Bazel will still log its build progress if
necessary, and still report compile and runtime errors, but won't
produce half a dozen lines of INFO at the start of the command.
This is just aiming for an API simplification. Instead of the typical
`non_check_line_->line_number(), non_check_line_->indent()` with an
`INT_MAX, ""` special-case, instead pass `to_file_end` and let the
functions infer whether `non_check_line_` should be examined. (I'm also
hoping removing `INT_MAX` improves understandability)
Note this depends on #3234
Trying to more proactively catch when autoupdate is missed. Most of the
execution time of these tests should be in running the program under
test, not processing output, so this should have marginal overhead in
order to produce a useful reminder.
I'm finding the current autoupdate difficult to reason about. What I'm
trying to do here is use the class to make it easier to add helper
functions.
For example, I merge the vector+cursor for stdout/stderr into an object,
passed to helpers together instead of as two parameters.
ShouldAddCheckLine can check against output_file_number_ without passing
that through a couple levels of function calls. In turn,
ShouldAddCheckLine is shared with the end-of-file logic instead of that
having its own comparison from what AddCheckLines does.
Also, I'm trying to get the pre-AUTOUPDATE edits in their own loop,
distinct from the main code. The class means AddRemappedNonCheckLine is
a helper function to share code, instead of a lambda (which I was
thinking would just confuse the flow further).
I'm also changing the input non_check_lines to a single vector to match
stdout/stderr. Because of the AUTOUPDATE + SPLIT lines, we're guaranteed
to have at least one line per file.
I realized file_offset_in_new_lines is redundant with output_line_number
so code now uses the latter (because it's older -- I think one's as good
as the other, otherwise).
Note, this change deliberately does not affect output. I'm only trying
to make it easier to read for the next changes, using the lack of change
in results as a good indicator that this is getting it right.
Include information in the test crash output to identify which test we
were running, and the corresponding resolved argument list, if either
`bazel test` or autoupdate crashes.
Lex now prints its yaml as:
```
- filename: name
tokens: [ ... ]
```
New support in file_test allows the `filename` marker at the top to
define the default file number for later lines, meaning multi-file
output from lexing is now associated with the appropriate file. Similar
support will probably also apply to lowering, semir, and other places
that print a filename once for the full dump.
This hammers a bit at how line number replacements work in file_test,
allowing stacking them so that lex errors and stdout can both be
line-associated properly. I've tried to make the autoupdate more
frequently work in one pass, now also taking into account the file index
when doing line replacements.
There are still some issues with EndOfFile that it may be good to
discuss: because CHECK lines are appended to the end of the file now,
and the EndOfFile token points at the last line including comments, new
lex tests now take two runs to autoupdate (because without CHECK lines,
the EndOfFile points at a content line, which content is then inserted
after). Note that removing CHECK lines from the test is not a solution:
autoupdate also started inserting blank lines, which breaks this for a
similar reason. One solution here might be to not have EndOfFile
associate with a line or column, which has been a bit of an issue
regardless.
Also fixes a small issue with toolchain's autoupdate script.
toolchain/driver/testdata/fail_missing_file.carbon has no content after
its AUTOUPDATE line. This caused it to oscillate between three states:
1) If there were no content lines after AUTOUPDATE, autoupdate would add
a blank line and then its STDERR CHECK line.
2) If there was a (blank) content line after AUTOUPDATE, autoupdate
would move the STDERR CHECK line to immediately after AUTOUPDATE,
leaving a trailing blank line.
3) pre-commit would remove the trailing blank line, leaving no content
lines after AUTOUPDATE.
Handle the special case of no content lines after AUTOUPDATE by
producing "early" STDERR check lines immediately after the AUTOUPDATE
line, rather than before the next line, which might not exist. We
already did this in the case where there were early STDOUT check lines.
Specifically this should improve linking by producing one large binary
instead of one-per-directory. The inclusion of the driver hits the size
issue. Separating out things which have more llvm deps has been
discussed, but I'm not doing that here because I think the semantics
layer will need to depend on clang for interop, and we'd lose a lot of
the benefits that way. Also, having just one place to look seems
simpler.
Includes supporting changes to file_test infrastructure, the most
significant of which is probably passing tests via file instead of a
really large args thing, using a custom rule to do that. That's because
dealing with the layered filegroups that allow the toolchain setup is
more complicated, and this approach scales well.
Combined test time is ~9s, so not sharding right now.
I wasn't sure if people would prefer having the autoupdate script under
testing, so I left it alone for now.
Write unattached CHECK:STDOUT lines at the end of the complete test
file, not at the end of the first split file.
Also, perform line number remappings for the current file even if we see
a check line for an earlier file first. We used to stop performing
remapping after the first check line that referred to a previous file.
To facilitate this, instead of splitting the check lines up by output
file prior to forming the output, we instead form a single list of check
lines and have the check lines track which file they refer to.
The main motivation for this is to get python loads in using the
`native-py` lint fix. However, enabling that made me wonder, maybe we
should fix in general?
`native-cc` is delayed, but not wholly cancelled (and `native-py`
picking up might indicate `native-cc` won't be too far behind). There's
also some automated fixes for `.append` and dict sorting -- this felt
okay to me, maybe not something to eagerly add but probably not worth
stopping buildifier from fixing (I've noticed the warnings in the past
and had been ignoring them).
Running everything does mean that load orders are sorted automatically
now, which I think is a positive. Most generally, I think these fixes
aren't _harmful_, and having them done automatically seems beneficial:
my biggest concern about `native-py` and `native-cc` was actually that
regressions wouldn't be caught, but this addresses that issue
automatically.
Just some small refactorings stemming from #3073.
AddCheckLines -> BuildCheckLines because the two lists are now fully
separate. Adding is_blank to be more direct about behavior than the
Print call.
Allow interleaving of STDOUT and STDERR check lines. Put STDOUT lines
after the line they're attached to, and STDERR lines before. If no
STDOUT check line is attached to any line, then put them all at the end
of the file instead.
This is intended to better handle the case where stdout contains
unreplaced mentions of line numbers, and also reflects that stdout is
typically a consequence of the test rather than commentary on it, so
placing it after the test seems likely to read better.
This migrates explorer tests to file_test, using the new --autoupdate
functionality. Per discussion, the trace tests that were using "not"
output are mostly migrated to checking full output. The main exception
is tests that were including trace output from the prelude: the prelude
output is pretty long (multiple MB already) and I think it wasn't the
intent to include, only trace output from the small program.
This is the remaining use of lit support, so the supporting libraries
are also removed here.
I've migrated the toolchain autoupdate scripts here, I just need a
little more time to do the explorer side (which I need to check
performance, that may require multithreading as we do in the current
script). However, this felt substantial enough to share and it means I
can handle autoupdate in more of the toolchain, including preparatory
work for autoupdate on multi-file tests.
Once explorer is done I'll remove the old script.
This is a simplification of the construction, although somewhat limiting
(it means that the caller can't register the same file multiple times,
though I stopped doing that anyways since it was causing confusion).
What this more importantly _allows_ is logic on the FileTestBase child
itself that's not test-specific -- in particular, autoupdate
functionality which wouldn't use RUN_ALL_TESTS.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
I'm integrating absl flag support with a few thoughts here...
1. It simplifies the handling of files in file_test.
- Removes the need for a separate subset target.
3. Looking forward, I'm planning to add a flag to allow for autoupdate
of golden files.
4. In explorer, there's been confusion about having file_test run tests
"twice" so hopefully it's clearer when it's a separate target with a
different flag on the target.
This is sufficient to convert remaining toolchain tests to file_test. %s
and %t are currently used. While #2978 might use %T with lit, I think
that's resolving a lit-specific issue that's not necessary when
TEST_TMPDIR is readily available for any necessary operations (i.e.,
setting the working directory). With ARGS, I think it's feasible to
switch to file_test without %T.
Also adds documentation to file_test_base.h, which was starting to feel
like a significant gap.
This is not yet handled by autoupdate, but I'm eyeing that next.
Putting fuzzer files under //testing to emphasize the testonly aspect
(consolidates bazel and common subdirectories). The attributes on
explorer_fuzzer are also a little skewed from what's desirable; it's
been working okay, but this should still be a refinement.
This goes back to my comment at the end of #2934. Rather than
implementing support in the shared script, this only affects one
explorer test so would be better to have in the respective invocation.
It's coming up now because #3018 has me thinking about autoupdate
behaviors again.
Adding `name` to `glob_lit_tests` will make it conform with other
implementations of `glob_lit_tests` out there. If someone uses this repo
while providing a different version of `glob_lit_tests` that requires a
name, those build rules will become invalid.
Created `explorer/trace_testdata/full_trace.carbon` to test the whole trace and some changes in `explorer/file_test.cpp` to treat `/trace_testdata/` tests differently.
Overall, cleaning up remaining lit uses.
#2851 had removed FileCheck invocations from some of the explorer tests; this starts as just restoring that. But, now that we have far fewer `lit` tests, it seems best to refine `lit.cfg.py` to focus on providing fewer commands (not all were even used).
Also, adding testing of an error to explorer (which I noticed due to a change that would've broken that) made me notice that autoupdate_lit_test's for_lit logic didn't actually work, so I'm just cutting it and going to manual updates. Really, we might want to just remove lit autoupdate support altogether since it's only a couple tests using it, but I'm not ready to make that change right now.
Fixes#2912
Replacing direct raw_string_ostream uses. I figure the wrapper should be used more consistently.
There are still remaining raw_string_ostream uses that weren't compatible -- I'm continuing to look at those, but felt it was cleaner to have this on its own.
This is just a cleanup. Since we now have a testing directory, I think this is a better home for testonly libraries than //common. (I was thinking about this when I was considering adding more test_raw_ostream deps)
In explorer, we already support parsing a string_view, so use that. In toolchain, we need to build support, probably using vfs, so that's a todo.
bazel test //explorer:file_test --runs_per_test=5
- branch: Stats over 250 runs: max = 18.3s, min = 5.2s, avg = 11.2s, dev = 2.9s
- trunk: Stats over 250 runs: max = 22.3s, min = 5.9s, avg = 12.1s, dev = 2.8s
Not a dramatic improvement, but maybe more effective long-term, and this'd been requested on #2876
By implementing these improvements, users will have the ability to choose specific parts of the trace output.
Currently, when executing a file using the explorer with the --trace_file=- or --trace_file=filename.txt flag, the resulting output is an extensive and verbose log containing all the information.
In this PR, I have introduced the `ProgramPhase` enum class, which have distinct phases encountered during the compilation of a program in the explorer. Each member of this enum class corresponds to a specific phase, signifying the relevant information to be included in the trace output.
The phases covered by the `ProgramPhase` enum class are as follows:
1. Printing the source program
2. Name resolution
3. Control flow resolution
4. Type checking
5. Unformed variable resolution
6. Printing declarations
7. Printing the timings
8. Printing whole output.
These phases can be selected by passing the following compiler flags along with `--trace_file=-`.
`-trace_source_program`, `-trace_name_resolution`, `-trace_control_flow_resolution`, `-trace_type_checking`, `-trace_unformed_variables_resolution`, `-trace_declarations`, `-trace_execution`, `-trace_timing` and `-trace_all`. If none of these flags is passed only execution trace will be added to the output.
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Jon Ross-Perkins
Chandler CarruthandJon Ross-Perkins
Richard SmithandJon Ross-Perkins
Jonathan B. CoeandChandler Carruth
Jordan Rupprecht
Prabhat Sachdeva
