Rearranges driver logic into CompilationUnits in order to associate
artifacts from the various stages of compilation.
Note, I'm not totally sure what the right thing to do is for
lower/codegen, so I'm just doing a rote change there for now that
mirrors prior phases (this is all the code supports anyways, so is
probably right for now regardless).
SourceBuffer error output is moved local for consistency with other
steps, and so that it's less ambiguous whether the error should be
expected to already include a filename.
There were two issues contributing to this crash:
- Primarily, the issue is that we queue up diagnostics and don't format
them into a string until we reach the end of compilation. In some code
paths in the driver, we destroyed the Semantics IR object before this
happened. But diagnostics can contain references to Semantics IR
objects, such as strings stored in the string table, which can lead to a
use after destruction bug.
This is fixed by ensuring the diagnotics consumer is flushed before
destroying any of the objects that it can refer to. The current approach
to this is not especially clean, unfortunately, but this requires
fighting C++ as this isn't the order in which it wants to destroy
things.
- This issue was obscured by the Semantics IR's string table holding a
reference to whatever underlying storage it was given rather than its
own string storage, so sometimes it would hold a reference to a string
from the source file, and sometimes a string from the tokenized buffer's
string table. The diagnostics were always flushed before the source file
was destroyed, but not before the tokenized buffer was destroyed. So to
see the issue, you'd need to have a string literal with certain contents
followed by an identifier with a name that matched those contents.
The crash is made more reliable by holding references to the Semantics
IR's string map in its string table, rather than references to someone
else's strings. This also fixes a latent bug where passing a string
temporary to SemanticsIR::AddString would store a dangling reference in
the string table. Incidentally, AddString is also changed to perform
only one hash table lookup rather than two for each added string.
This also tries to restructure the command line interface to the
toolchain a bit to make it start operating more like a compiler that
could be integrated into a build system rather than primarily as
a testing tool.
1) This switches form a `dump` subcommand to a `compile` subcommand
which has "dump" actions that can be enabled within it.
2) A distinct set of compile _phases_ that match the toolchain
structure:
- `lex` to run the lexer
- `parse` to run the parser
- `check` to fully check that the code is valid
- `lower` to lower to LLVM's IR
- `codegen` to generate executable code
3) The codegen phase has two output formats: textual assembly and
a binary object. These outputs can be configured, with a default for
an object when writing to a file and more firm default for textual
assembly when writing to stdout.
4) Select and expose the use of the LLVM host detection to compute
a default code generation target in the driver so that the command
line interface can reflect this. For example, the `help` output will
include the default target.
5) The `//toolchain/codegen` library APIs have been restructured a bit
to make the code flow a bit more naturally when implementing the new
command line structure. No real changes to the logic though.
There are also some minor tweaks to the command line interface based on
trying to use the shortest names for things that still seem likely to be
learnable for users:
- Switched `target-triple` to just `target`: the "triple" component to
this name is historical and can be confusing. For example, almost all
"triple" strings have more than three components today.
- Switched to just `--output` as now the fact that it is a file can be
configured in the documentation -- it will render as `--output=FILE`.
This also adds support for two custom output filename modes. First, when
no output is specified, we now compute one in the conventional way for
compilers by removing the file extension of the input file and replacing
it with `.o` for an object file output or `.s` for an assembly file
output. This matches the behavior of Clang and GCC for example.
Second, output to stdout is enabled with the special output file name of
`-` since it is no longer the default. This also follows the convention
of most compilers and many other command line tools to use `-` as a file
name to signify using standard in/out pipes.
There are still some rough edges here that I suspect could be improved,
but this seems like a good start of switching over to a complete
argument parser.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Lucile Rose Nihlen <luci.the.rose@gmail.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Allow interleaving of STDOUT and STDERR check lines. Put STDOUT lines
after the line they're attached to, and STDERR lines before. If no
STDOUT check line is attached to any line, then put them all at the end
of the file instead.
This is intended to better handle the case where stdout contains
unreplaced mentions of line numbers, and also reflects that stdout is
typically a consequence of the test rather than commentary on it, so
placing it after the test seems likely to read better.
I've migrated the toolchain autoupdate scripts here, I just need a
little more time to do the explorer side (which I need to check
performance, that may require multithreading as we do in the current
script). However, this felt substantial enough to share and it means I
can handle autoupdate in more of the toolchain, including preparatory
work for autoupdate on multi-file tests.
Once explorer is done I'll remove the old script.
This is sufficient to convert remaining toolchain tests to file_test. %s
and %t are currently used. While #2978 might use %T with lit, I think
that's resolving a lit-specific issue that's not necessary when
TEST_TMPDIR is readily available for any necessary operations (i.e.,
setting the working directory). With ARGS, I think it's feasible to
switch to file_test without %T.
Also adds documentation to file_test_base.h, which was starting to feel
like a significant gap.
This is not yet handled by autoupdate, but I'm eyeing that next.
This also makes the filename a reference to the buffer since the line
seems better to have as a reference (versus copying a string per error).
Most tests now have different line deltas due to the extra output, but
the actual errors should overall stay the same.
Some of the error locations look like they could be improved, but this
change is only making it clear where they were before.
Adding `name` to `glob_lit_tests` will make it conform with other
implementations of `glob_lit_tests` out there. If someone uses this repo
while providing a different version of `glob_lit_tests` that requires a
name, those build rules will become invalid.
This pr creates the object file for the carbon code that returns either
0 or 1. The command to convert the object code to binary is `clang
<object_file_name> -o <binary_name>`
---------
Co-authored-by: Farzana Ahmed Siddique <fasiddique@google.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Overall, cleaning up remaining lit uses.
#2851 had removed FileCheck invocations from some of the explorer tests; this starts as just restoring that. But, now that we have far fewer `lit` tests, it seems best to refine `lit.cfg.py` to focus on providing fewer commands (not all were even used).
Also, adding testing of an error to explorer (which I noticed due to a change that would've broken that) made me notice that autoupdate_lit_test's for_lit logic didn't actually work, so I'm just cutting it and going to manual updates. Really, we might want to just remove lit autoupdate support altogether since it's only a couple tests using it, but I'm not ready to make that change right now.
Fixes#2912
Add semantic analysis and semantics IR building for `if` expressions, and add the first parts of control flow handling to semantics IR. After discussion with @chandlerc, use [block arguments](https://en.wikipedia.org/wiki/Static_single-assignment_form#Block_arguments) to convey values from the two arms of the `if` to the result. For now, only a single block argument is supported, but we should revisit this as we explore more of the requirements of the Semantics IR form.
Functions can now contain multiple code blocks, so grab the entry block up-front instead of assuming the entry block will be at the top of the block stack when we reach the end of function emission.
Add trivial support for `bool` type literal, because without it we can't write testcases.
This shifts logic so that bindings are added to name lookup only after the scope is complete, removing logic around adding/removing/re-adding names in certain scopes.
This does mean that things like a function's forward declaration will need to go through an extra hoop for name conflict checks, because under this approach a function definition does conflict checking when it adds names for the body's use. But, that seems easy to address, and better than the current hoops.
This switches types to using SemanticsTypeId instead of SemanticsNodeId, and lowering pre-builds its list of types. The empty tuple type is special-cased because we don't want to emit it unless it's in-use, but as the implicit return for functions, it's frequently used. Callables use invalid to indicate the implicit return, and that seems undesirable to change due to the size increase.
We do use the empty tuple type for function returns, so I don't want to get rid of it, but the value is unused. With this change, builtins are all types. Long-term the empty tuple value should have a representation similar to structs; just a tuple value that's empty, not a built-in.
This removes special-casing of empty structs, handling them as just a regular value instead of a builtin. Note the `{} as Type` is still special-cased.
In lowering, removes the test of calling a function using `{}` because it's missing the proper load/store. This setup notices that error whereas the prior worked due to said special-casing. Fixing this will need to be done as part of generally adding loads for variable uses.
This is the first step to refactoring types into a SemanticsTypeId. This only tracks what's in-use, but as a consequence starts funneling type information through in ways similar to how I'd want it to do SemanticsTypeId.
Previously, IR for arguments in calls and struct values was separated out. This merges it back in. Additionally, parameters for functions and struct types had their own IR; the block is still there, but there's a TODO to decide what to do with it.
In the LLVM IR, this has the consequence of emitting expressions that are inputs to a call or struct value within the scope of the function, which is pretty much where it should be. Importantly it happens before the call is encountered.
This change also tinkers with the int and real literal lowering. I'm pretty sure both are still wrong, but was having trouble figuring out a "better" way to do it, and this seems like it'll work for now.
- Moves most parts to //testing/lit_test to be consistent with //testing/file_test.
- Separates the autoupdate script out because it's shared between lit_test and file_test now, not lit-specific.
- Renames scripts to autoupdate_testdata (or autoupdate_lit_testdata for explorer's extra) to be more consistent with the non-lit-specific setup.
- Switches from execv to subprocess.call to head off a subtle issue regarding execution of multiple scripts, which we're likely to want in the future. Mostly in this PR because everything was already being touched.
- Removes autoupdate's dependency on merge_output in order to (a) better support the division of lit and non-lit logic and (b) remove a subprocess, for reasons similar to file_test's removal of subprocesses.
This builds on #2814 by adding the test framework to the toolchain. On Linux, this is 7s -> 4s for me. #2811 has more detailed timing for the explorer, which also had more dramatic changes because it's about 3x more tests run in 2x as many ways (6x total).
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
This makes TokenizedBuffer more consistent with ParseTree and SemanticsIR, which also wrap with [] to produce a sequence value.
It also makes it possible in driver.cpp to just prefix the line with a name, so it ends up with:
var_name: [
(content)
]
Noticed this due to bracketing comments on #2443 and trying to think of better answers. With this, we can also say that the [] bracket a variable.
Adds a simple script to merge stdout/stderr and put on labels. This is hidden to the RUN line using lit.cfg.py.
This is my solution to addressing how errors printed by the toolchain break sorting of stdout output; just put stdout first. We could also have toggles for interleaving output or such, which might help test whether we do it properly.
This also moves some previous-distributed replacement logic into lit_autoupdate_base.py: I think having that adjacent to lit.cfg.py is probably the better choice, and it reduces duplication in toolchain scripts. It happens here because I need to change the resulting commands to include the merge.
I've refactored the script in order to make it work in more contexts, which is why the delta is lost. I've actually refactored a significant amount with the intent of making the logic easier to understand, because I was also adjusting bits of it.
Some key notes:
- Removes the multi-pass update that was dealing with unfixed line numbers in explorer (I think the current script should work in one pass)
- Fixed explorer to handle multiple line numbers on the same line (turns out we can rely on local format for line numbers).
- Using execv instead of imports because making Python imports work in a setup like this feels like it's not worth it; only a nuisance.
- Adding __init__.py to satisfy mypy, which otherwise considers the lit_autoupdate.py scripts to be issues.
- Using py because I was thinking sh would be more platform-dependent. py should port better to Windows.
- Getting rid of [[ID#]] capture groups in the semantics-ir tests because with the autoupdate it's kind of moot (also, hard to autogenerate the pairs without relying on the %### value).
Note this does mean tests switch to more of a "make a change, see which tests change" setup. I don't know that that's a _bad_ thing though -- it's pretty much how tests are being written right now, which is why I went down this rabbit hole. It's a nuisance to make a change then _manually_ have to update a bunch of code.
My intent is to use this for to convert parse-tree tests to lit, but I wanted to do this with _existing_ tests first as a proof of concept and to make sure there's agreement.
On #2224 @zygoloid pointed out we needed --implicit-check-not to ensure we were correctly matching output. This is the standard way we're writing explorer tests, so I was looking at unifying our lit approaches.
This is one take on it, making more use of substitutions to bring various testing into alignment, as well as symlinks to avoid config skew (maybe I'll eventually figure out a better solution than symlinks).
Makes a couple small fixes in explorer to remove end-of-line whitespace on output.
This unifies `dump-tokens` and `dump-parse-tree` so that we don't keep writing basically the same code repeatedly.
I'll need to modify the output of SemanticsIR::Print more, and may soon unify printing multiple IRs this way (particularly including the builtins SemanticsIR) but this is intended to offer a starting point.
I may eventually try to unify lit.cfg.py files, but I was thinking about whether that works in various contexts we may run in and eventually decided copying the driver/testdata/lit.cfg.py file would be the easiest solution.
I was mainly looking at keywords trying to figure out what needs work and the amount to which it doesn't reflect the design confused me (including some things that we've decided not to include, and some things I'm not aware of discussion about). I figured this cleanup would at least make it somewhat clearer why things are in there.
I'm treating https://github.com/carbon-language/carbon-lang/blob/trunk/docs/design/lexical_conventions/words.md as canonical, with `_` and `xor` as presumably deliberate exceptions. Similarly avoiding symbol tokens because I assume you'll push proposals for the difference.
I dropped the `Keyword` qualifier because `is` makes `IsKeyword` a name conflict, and dropping the qualifier seemed like the more consistent solution (it doesn't do `AmpSymbol`, after all). If we need clarity I might lean towards a separate namespace to avoid naming conflicts.
This switches to generating per-file targets such as `//executable_semantics/testdata:tuple/equality_false.carbon.test`, instead of one test covering all files. Essentially this relies on bazel instead of lit to handle test parallelization.
Jon Ross-Perkins
Richard Smith
Farzana Ahmed SiddiqueandFarzana Ahmed Siddique
Jordan Rupprecht
Jon Meow