Adds a simple script to merge stdout/stderr and put on labels. This is hidden to the RUN line using lit.cfg.py.
This is my solution to addressing how errors printed by the toolchain break sorting of stdout output; just put stdout first. We could also have toggles for interleaving output or such, which might help test whether we do it properly.
This also moves some previous-distributed replacement logic into lit_autoupdate_base.py: I think having that adjacent to lit.cfg.py is probably the better choice, and it reduces duplication in toolchain scripts. It happens here because I need to change the resulting commands to include the merge.
I've refactored the script in order to make it work in more contexts, which is why the delta is lost. I've actually refactored a significant amount with the intent of making the logic easier to understand, because I was also adjusting bits of it.
Some key notes:
- Removes the multi-pass update that was dealing with unfixed line numbers in explorer (I think the current script should work in one pass)
- Fixed explorer to handle multiple line numbers on the same line (turns out we can rely on local format for line numbers).
- Using execv instead of imports because making Python imports work in a setup like this feels like it's not worth it; only a nuisance.
- Adding __init__.py to satisfy mypy, which otherwise considers the lit_autoupdate.py scripts to be issues.
- Using py because I was thinking sh would be more platform-dependent. py should port better to Windows.
- Getting rid of [[ID#]] capture groups in the semantics-ir tests because with the autoupdate it's kind of moot (also, hard to autogenerate the pairs without relying on the %### value).
Note this does mean tests switch to more of a "make a change, see which tests change" setup. I don't know that that's a _bad_ thing though -- it's pretty much how tests are being written right now, which is why I went down this rabbit hole. It's a nuisance to make a change then _manually_ have to update a bunch of code.
My intent is to use this for to convert parse-tree tests to lit, but I wanted to do this with _existing_ tests first as a proof of concept and to make sure there's agreement.
On #2224 @zygoloid pointed out we needed --implicit-check-not to ensure we were correctly matching output. This is the standard way we're writing explorer tests, so I was looking at unifying our lit approaches.
This is one take on it, making more use of substitutions to bring various testing into alignment, as well as symlinks to avoid config skew (maybe I'll eventually figure out a better solution than symlinks).
Makes a couple small fixes in explorer to remove end-of-line whitespace on output.
This builds on semantics-ir lit support added by #2222
The googletest setup was feeling cumbersome, especially as I'm thinking about how to add more testing: I feel like I'm wrestling with the infrastructure.
The `[[ID1]]` and so on in tests is one advantage of switching: it's easier to do matching of IDs for verification. This is also more agnostic about the numbers than before, something which I'm concerned will be important as I think about builtins.
To explain my builtins thought, I think that needs to be another SemanticsIR with basically names pointing at builtin things. But this (a) creates multiple SemanticsIRs, which would confuse the current singleton approach and (b) starts creating more fluctuation for IDs, potentially impacting the numbers used (also, chandlerc's suggested pointers for some use-cases).
Overall it felt like I was heading towards a situation with googletest where writing the tests would be really difficult, and it was adding to my hesitance to write more code in the toolchain. I'm hoping this acts as a simplification.
Note, the "cp" commit has some incremental changes to googletest that I'd considered for making it easier to add matchers, but ultimately I ended up with this outcome.
Jon Ross-Perkins