Add a textual IR format to the toolchain.
The exact details of the format are somewhat arbitrary right now, and I
expect them to change as we refine the semantics IR model, but at the
moment they're somewhat directly following the current structure of the
IR.
Semantics tests currently test both the "raw" format, which shows the
details of the representation, and the textual format, which is somewhat
higher level. We may want to revisit that decision once the textual
format is a bit more stable, and test only the textual format in most of
these tests, but for now it seems prudent to keep both sets of tests.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Allow interleaving of STDOUT and STDERR check lines. Put STDOUT lines
after the line they're attached to, and STDERR lines before. If no
STDOUT check line is attached to any line, then put them all at the end
of the file instead.
This is intended to better handle the case where stdout contains
unreplaced mentions of line numbers, and also reflects that stdout is
typically a consequence of the test rather than commentary on it, so
placing it after the test seems likely to read better.
I've migrated the toolchain autoupdate scripts here, I just need a
little more time to do the explorer side (which I need to check
performance, that may require multithreading as we do in the current
script). However, this felt substantial enough to share and it means I
can handle autoupdate in more of the toolchain, including preparatory
work for autoupdate on multi-file tests.
Once explorer is done I'll remove the old script.
This is a simplification of the construction, although somewhat limiting
(it means that the caller can't register the same file multiple times,
though I stopped doing that anyways since it was causing confusion).
What this more importantly _allows_ is logic on the FileTestBase child
itself that's not test-specific -- in particular, autoupdate
functionality which wouldn't use RUN_ALL_TESTS.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
This shifts explorer's file_test to use ExplorerMain in order to be able
to pass arguments similar to how the command line would. It also means
that various output wrapping done by the command line is shared, no
longer copied by file_test.
In order to help make this work, I plugged vfs::FileSystem into
explorer, similar to how we're doing this on the toolchain side (might
try to share more code later). I was having trouble getting an overlay
working, I think due to how paths are specified -- but due to the issues
in fixing this, I'm just loading the prelude into the InMemoryFilesystem
for now.
Under this approach, I'm unifying more of the file versus string
handling. I think we should shift towards an approach where, like
toolchain code, anyone trying to use Explorer should use a vfs
implementation to supply code. This should reduce the number of distinct
code paths which we're maintaining/testing.
Short-term, this allows the trace testdata to be merged into file_test
with less special casing, using ARGS:. Long-term, it means that the
file_test knows the ARGS to pass to generate checks to match against,
which is the direction I'm planning for autoupdate.
This is sufficient to convert remaining toolchain tests to file_test. %s
and %t are currently used. While #2978 might use %T with lit, I think
that's resolving a lit-specific issue that's not necessary when
TEST_TMPDIR is readily available for any necessary operations (i.e.,
setting the working directory). With ARGS, I think it's feasible to
switch to file_test without %T.
Also adds documentation to file_test_base.h, which was starting to feel
like a significant gap.
This is not yet handled by autoupdate, but I'm eyeing that next.
Putting fuzzer files under //testing to emphasize the testonly aspect
(consolidates bazel and common subdirectories). The attributes on
explorer_fuzzer are also a little skewed from what's desirable; it's
been working okay, but this should still be a refinement.
This also makes the filename a reference to the buffer since the line
seems better to have as a reference (versus copying a string per error).
Most tests now have different line deltas due to the extra output, but
the actual errors should overall stay the same.
Some of the error locations look like they could be improved, but this
change is only making it clear where they were before.
The lack of comments makes it difficult to be sure, but if any of these
are necessary, I'd like to make sure there are comments to explain why a
specific value was shown (which I do add for diagnostics_ in this
change).
Adding `name` to `glob_lit_tests` will make it conform with other
implementations of `glob_lit_tests` out there. If someone uses this repo
while providing a different version of `glob_lit_tests` that requires a
name, those build rules will become invalid.
This pr creates the object file for the carbon code that returns either
0 or 1. The command to convert the object code to binary is `clang
<object_file_name> -o <binary_name>`
---------
Co-authored-by: Farzana Ahmed Siddique <fasiddique@google.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Overall, cleaning up remaining lit uses.
#2851 had removed FileCheck invocations from some of the explorer tests; this starts as just restoring that. But, now that we have far fewer `lit` tests, it seems best to refine `lit.cfg.py` to focus on providing fewer commands (not all were even used).
Also, adding testing of an error to explorer (which I noticed due to a change that would've broken that) made me notice that autoupdate_lit_test's for_lit logic didn't actually work, so I'm just cutting it and going to manual updates. Really, we might want to just remove lit autoupdate support altogether since it's only a couple tests using it, but I'm not ready to make that change right now.
Fixes#2912
These tests are doing string comparisons on output that don't seem to be meaningfully different from a file_test.
I'm tempted to migrate lexer tests in general, but I'm not doing that here since others may find more value in the current approach.
Replacing direct raw_string_ostream uses. I figure the wrapper should be used more consistently.
There are still remaining raw_string_ostream uses that weren't compatible -- I'm continuing to look at those, but felt it was cleaner to have this on its own.
This is just a cleanup. Since we now have a testing directory, I think this is a better home for testonly libraries than //common. (I was thinking about this when I was considering adding more test_raw_ostream deps)
Add semantic analysis and semantics IR building for `if` expressions, and add the first parts of control flow handling to semantics IR. After discussion with @chandlerc, use [block arguments](https://en.wikipedia.org/wiki/Static_single-assignment_form#Block_arguments) to convey values from the two arms of the `if` to the result. For now, only a single block argument is supported, but we should revisit this as we explore more of the requirements of the Semantics IR form.
Functions can now contain multiple code blocks, so grab the entry block up-front instead of assuming the entry block will be at the top of the block stack when we reach the end of function emission.
Add trivial support for `bool` type literal, because without it we can't write testcases.
This adds vfs support to the toolchain, allowing Driver to take in-memory inputs in tests. As a consequence, I'm simplifying SourceBuffer: rather than allowing tests to pass in their own memory buffer, I'm using InMemoryFileSystem to push for greater consistency with production code. This does hit a quirk where I need to be careful about null terminator handling because fuzzer imports don't always have one, but that's probably more robust anyways.
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
In explorer, we already support parsing a string_view, so use that. In toolchain, we need to build support, probably using vfs, so that's a todo.
bazel test //explorer:file_test --runs_per_test=5
- branch: Stats over 250 runs: max = 18.3s, min = 5.2s, avg = 11.2s, dev = 2.9s
- trunk: Stats over 250 runs: max = 22.3s, min = 5.9s, avg = 12.1s, dev = 2.8s
Not a dramatic improvement, but maybe more effective long-term, and this'd been requested on #2876
This shifts logic so that bindings are added to name lookup only after the scope is complete, removing logic around adding/removing/re-adding names in certain scopes.
This does mean that things like a function's forward declaration will need to go through an extra hoop for name conflict checks, because under this approach a function definition does conflict checking when it adds names for the body's use. But, that seems easy to address, and better than the current hoops.
This switches types to using SemanticsTypeId instead of SemanticsNodeId, and lowering pre-builds its list of types. The empty tuple type is special-cased because we don't want to emit it unless it's in-use, but as the implicit return for functions, it's frequently used. Callables use invalid to indicate the implicit return, and that seems undesirable to change due to the size increase.
We do use the empty tuple type for function returns, so I don't want to get rid of it, but the value is unused. With this change, builtins are all types. Long-term the empty tuple value should have a representation similar to structs; just a tuple value that's empty, not a built-in.
This removes special-casing of empty structs, handling them as just a regular value instead of a builtin. Note the `{} as Type` is still special-cased.
In lowering, removes the test of calling a function using `{}` because it's missing the proper load/store. This setup notices that error whereas the prior worked due to said special-casing. Fixing this will need to be done as part of generally adding loads for variable uses.
This is the first step to refactoring types into a SemanticsTypeId. This only tracks what's in-use, but as a consequence starts funneling type information through in ways similar to how I'd want it to do SemanticsTypeId.
Previously, IR for arguments in calls and struct values was separated out. This merges it back in. Additionally, parameters for functions and struct types had their own IR; the block is still there, but there's a TODO to decide what to do with it.
In the LLVM IR, this has the consequence of emitting expressions that are inputs to a call or struct value within the scope of the function, which is pretty much where it should be. Importantly it happens before the call is encountered.
This change also tinkers with the int and real literal lowering. I'm pretty sure both are still wrong, but was having trouble figuring out a "better" way to do it, and this seems like it'll work for now.
This is a convenient test helper for anything that can use injected
streams. Extract this to where it can be used by other tests and add
some basic tests, mostly documenting how it works.
- Moves most parts to //testing/lit_test to be consistent with //testing/file_test.
- Separates the autoupdate script out because it's shared between lit_test and file_test now, not lit-specific.
- Renames scripts to autoupdate_testdata (or autoupdate_lit_testdata for explorer's extra) to be more consistent with the non-lit-specific setup.
- Switches from execv to subprocess.call to head off a subtle issue regarding execution of multiple scripts, which we're likely to want in the future. Mostly in this PR because everything was already being touched.
- Removes autoupdate's dependency on merge_output in order to (a) better support the division of lit and non-lit logic and (b) remove a subprocess, for reasons similar to file_test's removal of subprocesses.
This builds on #2814 by adding the test framework to the toolchain. On Linux, this is 7s -> 4s for me. #2811 has more detailed timing for the explorer, which also had more dramatic changes because it's about 3x more tests run in 2x as many ways (6x total).
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
These changes should make output more stable when builtins are added to semantics. By omitting them from nodes and printing nodes as "relative to the last builtin", I should be able to add and remove builtins without automatically affecting every test. Also by printing builtin nodes as `nodeNameOfBuiltin`, it's a little easier to understand what's going on (for me, at least).
This is starting to build out actual lowering logic, for a really simple `fn Main() -> i32 { return 0; }`
Notes for achieving this:
- In semantics, currently function names are bound separate from the signature. When emitting IR, this turns out to be inconvenient because we want to know the name when we process the declaration and the definition. This change addresses that by merging the name into the FunctionDeclaration node, which is also accessible from the definition. It removes the separate BindName. This should be the cause of all the test changes in semantics, because the IR generated changes.
- Add a "Lowering" class which I'm using to hold the llvm builder state. This class now has minimal support for the SemanticsIR generated by the above example.
- In the "Lowering", values from expressions are stored in a DenseMap. I'll keep thinking about whether there's a cleaner way to achieve this, and I'd call it a temporary solution for now. However, this is how the `0` in `return 0` gets properly associated across SemanticsIR instructions, and it'll frequently be an issue in less trivial cases.
This makes TokenizedBuffer more consistent with ParseTree and SemanticsIR, which also wrap with [] to produce a sequence value.
It also makes it possible in driver.cpp to just prefix the line with a name, so it ends up with:
var_name: [
(content)
]
Noticed this due to bracketing comments on #2443 and trying to think of better answers. With this, we can also say that the [] bracket a variable.
Printing intermediate state should be helpful to be able to examine the input when debugging later steps.
Intermediate state is hard to trace from the individual libraries since they don't know whether `dump` is going to print the state, so this moves some trace logic into the driver which is better equipped to make the decision.
This is a first pass at what semantic type checking might look like. Types propagate along nodes, we use an InvalidType object when there's an error, and once there's an InvalidType we stop doing so much type checking.
This adds some RealLiteral handling in order to get type mismatches. I'm cautious about creating some real value for SemanticsIR (since the tokenized buffer version is a bit constrained), so I'm not doing that yet. But I will probably need to in order to maintain SemanticsIR having hermetic copies of its data, without a parse tree dependency.
The ParseTree comments say that preorder is "easier to visualize and read". The problem is, both the ParseTree and Semantics need to operate on the postorder traversal: the ParseTree during construction, and the Semantics during processing. As a consequence, understanding the postorder traversal is important, but it's also very hard to decipher when presented preorder. This PR provides a way to see the postorder, with helpful indents to show subtrees.
This retains the preorder printing as an option for people who prefer that. I'm pretty sure it'll be easier to debug tests if we can see the postorder, so I'm making that the default.
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
This starts adding builtins with TypeType and IntegerLiteralType. Note, structurally that's all they are, and not directly accessible in any way.
Adds a type field to SemanticsNode. Now, IntegerLiteralType can be identified as having type=TypeType, and IntegerLiteral as type=IntegerLiteralType. The current iteration doesn't do anything for type propagation, because I wanted to avoid making this too big.
This also switches the Identifier IR to instead BindName, with some side-effects. I'd been trying to think how to provide a name for TypeType, and switching around how things worked seemed like a better approach. And while I think it's the right direction (e.g., alias should just be a BindName), I also realized I don't need to name TypeType: there's probably a keyword to refer to the builtin, so it shouldn't use regular name lookup.
As a step towards builtins, provide more blocks. The intent is that any significant scope change will become its own NodeBlock. Builtins should produce the first set of node blocks.
Note, SemanticsIR as set up here isn't handling ordering of import processing -- I haven't thought that through much beyond that we probably want some lighter-weight processing of the parse tree to achieve it. But I think the essence of loading builtins first as their own IR block is... probably right?
I'm thinking about how to handle multiple files, and I think the current IRFactory is useful as a file-focused thing. So shifting/renaming accordingly. (doing this in its own PR to make the history a little cleaner for git's move detection)
Richard SmithandChandler Carruth
Jon Ross-Perkins
Farzana Ahmed SiddiqueandFarzana Ahmed Siddique
Jordan Rupprecht
Zhou