* Change `InterfaceWitness` -> `ImplWitness`
* Include a `SpecificId` in the `ImplWitness`. This allows the
`InstBlock` it contains to have its own identity, allowing it to be
changed as the impl is processed. Evaluation only updates the specific.
* Create the `ImplWitness` at the start of the impl definition. In the
future, this will be populated with the values of non-function
associated constants. For now, it starts full of invalid instruction
ids.
* Implements the model suggested in #4672 .
Note that the non-SemIR testdata changes are to these file:
* `toolchain/check/testdata/impl/lookup/fail_todo_undefined_impl.carbon`
* `toolchain/check/testdata/struct/import.carbon`
* `toolchain/check/testdata/tuple/import.carbon`
The last two are due to an import of generics bug exposed by this PR,
which will be fixed in a follow-on.
---------
Co-authored-by: Josh L <josh11b@users.noreply.github.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Add `EXTRA-ARGS:` support to file_test, to add arguments without
overriding the default arguments. Use `EXTRA-ARGS: --no-dump-sem-ir` to
turn off SemIR dumping and thus SemIR testing in the int builtin tests,
which validate correct behavior through diagnostics instead.
This doesn't get us any closer to supporting more targeted SemIR dumping
/ testing, but this seems to be a generally useful feature anyway. Most
existing
tests using `ARGS` have been switched over to using `EXTRA-ARGS`.
Requested in review of #4716.
This changes to an `Error` return to let the driver do the "error: "
prefix, except for one case with `help` that needs more work to change
(I'm not planning on picking up that TODO). It also changes
capitalization, backtick use, and a few minor punctuation things to try
to better match the diagnostic style.
This also adds `Error` matchers so that the changes to command line
testing are clearer.
This is really a set of closely related changes:
1) We really shouldn't be creating a Check::Unit for _Lower_. This fixes
that by storing the diagnostic converter for reuse.
2) Rather than passing in node converters to Check as their own array,
pass diagnostic converters as part of Check::Unit.
3) To support creating the converters early, pass SemIR::File
pre-constructed.
4) Since SemIR::File construction was used to track "checked", add
`is_checked` for that.
5) Clarifies a subtle edge case around `input_filename_` use with `-`.
Note the key consequence of this change, where I actually started, is
that `Check` only has one array of `Check::Unit` instead of receiving
`NodeLocConverter` as a separate array.
The first change here is to canonicalize away bit width when tracking
integers in our shared value store. This lets us have a more definitive
model of "what is the mathematical value". It also frees us to use more
efficient bit widths when available, such as bits inside the ID itself.
For canonicalizing, we try to minimize the width adjustments and
maximize the use of the SSO in APInt, and so we never shrink belowe
64-bits and grow in multiples of the word bit width in the
implementation. We also canonicalize to the signed 2s compliment
representation so we can represent negative numbers in an intuitive way.
The canonicalizing requires getting the bit width out of the type and
adjusting to it within the toolchain when doing any kind of math, and
this PR updates various places to do that, as well as adding some
convenience APIs to assist.
Then we take advantage of the canonical form and embed small integers
into the ID itself rather than allocating storage for them and
referencing them with an index. This is especially helpful for the
pervasive small integers such as the sizes of types, arrays, etc. Those
no longer require indirection at all. Various short-cut APIs to take
advantage of this have also been added.
This PR improves lexing by about 5% when there are lots of `i32` types.
---------
Co-authored-by: Dana Jansens <danakj@orodu.net>
Co-authored-by: Carbon Infra Bot <carbon-external-infra@google.com>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This PR adds a `--dump-timings` flag to the `compile` subcommand
(similar to the existing `--dump-mem-usage` flag), which collects timing
data per compilation unit for each compilation phase. For example, on my
2020 M1 MacBook:
```
$ bazel build -c opt //toolchain
$ bazel-bin/toolchain/install/run_carbon compile --phase=lower --dump-timings examples/sieve.carbon | tail
...
---
filename: 'examples/sieve.carbon'
nanoseconds:
lex: 30792
parse: 25458
check: 226625
lower: 1136958
Total: 1419833
...
```
Most of the changes are pretty straightforward. There were a couple I
wasn't sure about though; let me know if I should change:
- new `Timings` class in its own file, pretty similar to the existing
`MemUsage` class
- added a `timings_` field to the `CompilationUnit` class
- added a `timings` field to the `Check::Unit` struct
- renamed `CheckParseTree` function to `CheckParseTreeInner` for ease of
timing with early `return`
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Also surround it in square brackets rather than parentheses. This
matches the format used by Clang and GCC, and means diagnostics will
still match the `file:line:col: error: ` pattern used by some IDE tools.
Before:
```console
fail_builtins.carbon:11:11: error(AliasRequiresNameRef): alias initializer must be a name reference
```
After:
```console
fail_builtins.carbon:11:11: error: alias initializer must be a name reference [AliasRequiresNameRef]
```
Also tighten up test regex to only match on `STDERR` lines that list a
file name.
Use the diagnostic kind printing in #4425 to catch when we have
diagnostics with no tests.
This merges a couple other use-cases of filegroup manifests into a
common rule.
Note I do add a few tests for things, and also some things are
_actually_ unit tested (just not in the file_test structure). But I
stopped when I realized that dealing with merge conflicts is going to be
a pain. I might end up reverting test changes (as part of merge conflict
resolution) and doing narrow test additions in a separate PR, after both
this and #4425 are merged.
One of the things that ClangRunnerTest is doing is capturing
stderr/stdout because clang prints to it directly. This adds support for
that to FileTest.
I'm renaming the current `capture_output` field to `dump_output` because
the name is ambiguous after this change, and the flag is already named
`--dump_output`. It's still not great, but at least it's more distinct.
Note ClangRunner still doesn't use the vfs; that still needs work. I'm
just moving the NoArgs test over as a trivial test of the functionality.
This is a primarily automated change:
- Search & replace for capitalization
-
`(CARBON_DIAGNOSTIC\((?:\n\s+)?\w+,(?:\n\s+)?\s\w+,(?:\n\s+)?\s")([A-Z])`
- `$1\L$2`
- Search & replace for period
-
`(CARBON_DIAGNOSTIC\((?:\n\s+)?\w+,(?:\n\s+)?\s\w+,(?:\n\s+)?\s"(?:[^)]|\n)+)\.("[,)])`
- `$1$2`
- Limited search & replace for `ERROR: ` -> `error: ` in streamed things
- Leaving a TODO for command_line because there's more cleanup that can
be done there
- Modify diagnostic_consumer.cpp
- ERROR -> error
- WARNING -> warning
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
These tips are especially valuable in these cases, because you can't use
`autoupdate_testdata.py` to identify the output difference, so dumping
the output is pretty much the only option.
Discussed in the toolchain meeting today - we'd like to try having this
on by default and see if the cost isn't too high.
The nodebug test is a bit verbose, because it doesn't have the
`--exclude-dump-file-prefix` that test_file would usually add. Is there
a nicer way I could write this test to verify that --no-debug-info does
what it's meant to?
This adds just the debug info metadata for Compilation Units (the top
level container of debug info) - but without anything in them, LLVM
won't emit them at all, so while this is testable at the IR level, it
isn't observable at the object level until more debug info is added.
A couple of starting points in this patch:
* A flag (`--debug-info`, seems to match the naming/style of other flags
in the carbon driver, though this is different from the naming
conventions of clang/gcc) that enables debug info when lowering. Open to
other names/approaches (on by default? historically debug info's been to
large/expensive to do this, so sticking with that precedent for now).
* Enabling that flag by default in the lowering tests - I do find the
churn on golden tests a bit rough, and adding more features to all the
tests means more churn, but it seems consistent with the approach so far
- keep an eye on this and perhaps revisit this if the churn gets too
annoying
---------
Co-authored-by: Carbon Infra Bot <carbon-external-infra@google.com>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This undoes a previous change to unify them, and I think at my advice.
=[ Sorry about that, I think I was just wrong.
Specifically, I think I had suggested that it would be more efficient to
have a single shared hashtable of strings. The more I look at profiles
of the toolchain, the less likely that seems. Specifically for
identifiers and string literals it seems especially problematic.
Using a single, joint hashtable is likely a good idea when all of the
different querying code paths are equally likely, the strings follow the
same distribution of sizes, and either there is no clustering of access
to different sets of strings or none of the sets are meaningfully small
enough to fit into a lower level of resident cache.
I think essentially none of these predicates actually hold for
identifiers vs. string literals:
- Identifiers are *much* more hot
- They have wildly different size distributions.
- The access patterns are very clustered
Sorry for the misleading advice on that one.
While splitting them, I've worked to simplify the code a bit by building
a way to have the `StringRef` holding canonical value stores not require
specializations, and so we get a pretty large code cleanup in the
process here.
This has is a nice-to-have for me. Frequently I want to run a specific
test, and end up digging through output to be able to copy-paste the run
line. This uses TIP lines to inject the command into the file when using
AUTOUPDATE.
Note, one of the reasons I want this is because "bazel test
//toolchain/testing:file_test --test_output=all" has been regularly
exceeding bazel's output limit for me (workaround is either opening the
output file or specifying an obscure output limit flag), making it a
little harder for me to get the commands. However, frequently I'm adding
a file and want to iterate on it, so that's really the use case I have
in mind here.
Switch from recursing into non-canonical instruction fields to
separately canonicalizing those fields. This means we now form canonical
`InstBlockId`s, `TypeBlockId`s, `IntId`s, `FloatId`s, and `BindNameId`s
at least in the cases when they're referenced by a constant instruction.
This reduces the overall runtime for @chandlerc's 10MLoC example by
27.5% on my machine.
The driver now looks for all files under core/prelude/ and considers
them all to be part of the prelude. The driver also now only processes
the prelude in `--phase=check` and later, when it would actually be
imported.
With that done, add a simple `carbon_binary` build rule and use it to
build the example in `//examples`. This should cause the example to be
built as part of our continuous integration.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
The `file_test` layer still adds it to the VFS and filters it from the
dump output, but the driver now injects it into the compilation unites
depending on the `--import-prelude` file (enabled by default).
This doesn't address improvements to how we find the prelude code, just
sinking the needed logic into the driver itself.
- Adds an empty prelude.carbon file
- Imports that file in any non-Core package file
- Adds --disable-prelude-import to avoid that
- Adds --exclude-dump-file-prefix to be able to hide files from dumping
- Used to hide core files (we can't do this by package name due to lex
dumps, for example)
- Restructures some tests to not rely on `i32`, particularly `alias`
tests (which rely on a name ref) and tests with no prelude.
I'm adding the framework for switching i32 to calling Int32 in the
prelude, but I'm running into a separate error actually switching over.
So that *mostly* works, but isn't quite ready for prime time. However,
maybe the current state of this PR is still useful to review since it
does a lot of the infrastructure work and adds the %Core everywhere?
The purpose of the newline is to make it clearer where a given
diagnostic begins and ends, particularly as the first message of a
diagnostic may not be the error.
This is a trivial code change, but ripples edits through test files.
This handles toolchain failures per-file. The intent is to allow placing
both "success" and "fail" tests in the same file, using splits. However,
this PR only adds support and updates existing tests to continue
passing.
Building on #3636 which handles the general import case, add special
casing for namespaces. Namespaces can be combined cross-IR, so it's a
little more complex.
The implementation adds import_id to the Namespace instruction as a
reference to find the original using the normal structure. This is
achieved by moving the name_id to NameScope to free up space.
---
I considered a few alternatives...
I considered adding import_id to NameScope, but:
1. It's more consistent with things such as Function or Class that
provide name_id on the info object rather than the instruction.
2. I thought it more likely that there would be more NameScope cases
that might want a name_id rather than the import_id, since ImportRef
will typically be used.
I considered putting the import source (cross-ref IR id + inst id) on
the NameScope versus a separate ImportRef, which seems like the
strongest argument towards the NameScope approach because it removes an
instruction. That just felt inconsistent though, and the overhead of
instruction-per-imported-namespace should be low (theoretically few
namespaces should be used). Plus I feel a bit odd adding two
generally-unused ids to NameScope.
A specialized Namespace structure could also have been created to store
the import_id, but that would add an indirection to the NameScope.
Namespaces are copied, which means also adding their name to the
underlying instruction. It happened not to be done previously; the name
was only in name lookup.
Since the only import supported right now is the default import,
functionality is limited; in the future I'll need to deal with namespace
vs package conflicts.
Tests of namespace imports are under "namespace" -- I figured this would
be best for scaling as more instructions get support.
This also improves some debugging-related output that I was trying to
use while trying to build the support.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Move handling of mismatched brackets out of the main lexing loop into a
separate pass that is only run if there are mismatched brackets This is
done in preparation for using both lookahead and lookbehind to work out
how to match brackets, and to get this code far away from the hot lexing
loop.
Fix bracket insertion location to be immediately after the token that
we're inserting the bracket after, rather than potentially at the end of
a comment. When there are open brackets at the end of the file, say that
there are open brackets, not that there's a closing bracket without a
matching opening bracket.
This creates a namespace for `package` scope.
It looks like names of class_decls in namespaces lead to an unexpected
instref. This is already true, as best as I can tell. I'm not sure if
there's a preferred approach to address that, so I've left a TODO for
now.
This removes the filename from the file-scoped block, and places it
above to make it clear where the full SemIR begins (with multifile,
providing a barrier between).
I was suggesting this because `FloatingPoint` is pretty long. `int` and
`float` should be familiar abbreviations. `unsigned` should be familiar
to developers too, but `UnsignedInt` still feels usefully clearer for
the additional chars.
This is an incremental improvement on our diagnostic messages that
simply underlines an entire token if the token is larger than 1 char
(else it points to the single char with a caret like it used to).
- Treat an input of `-` as meaning stdin.
- Fix building of an llvm::MemoryBuffer from a non-regular file.
- Do not enforce filename restrictions on non-regular files.
- Do not invent an output file name based on the name of a non-regular
file.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Advantages:
- Allows lexing/parsing in parallel, since they are modifying fully
separate ValueStores.
- Allows SemIR to reliably be stored hermetically.
Disadvantages:
- Creates overlapping storage of duplicate strings when multiple files
are compiled together.
- Prevents Ids from being uniquely compared cross-file.
Per discussion, the decision is that the advantages are more important.
The looser ownership remains because both SemIR checking and
metaprogramming may still generate things we would want to deduplicate.
It would be somewhat odd if TokenizedBuffer owned something that
checking modified.
Provides an adapter for the llvm::yaml API because it otherwise needs a
bunch of const/non-const definitions, and the traits are difficult to
diagnose issues with. The current approach is pretty simple to use, even
if it's not super efficient (which, yaml output is more of a debugging
thing so I'm not really expecting it to be an issue).
Changes the format of yaml output to provide more index information,
just as reminders when seeing something like `node+0`. Note this would
create more churn in deltas if we were reliant on the output yaml in
tests, but we aren't so it should be okay.
This standardizes on having ValueStore and related structures provide
printing, removing the handlers in file.cpp.
The `[]` is provided for empty sequences versus if there was simply
nothing, in which case it would be a sequence when non-empty, and a null
value when empty. Consistently (and explicitly) providing sequences
feels easier to understand.
The changes to the output yaml are overall more terse. My hope is that
this is an improvement for most readers.
Also fixes printing of APInt, defaulting to unsigned for consistency
with Carbon's use.
This replaces the printing that was removed from SemIR's raw dump. It's
separate because (for example) lexing generates shared values, and so
reviewing them is not specific to any particular phase.
This makes the difference between errors and lower-level diagnostics
visible to users, and aligns the toolchain's behavior with the
expectations in `driver_fuzzer.cpp`.
toolchain/driver/testdata/fail_missing_file.carbon has no content after
its AUTOUPDATE line. This caused it to oscillate between three states:
1) If there were no content lines after AUTOUPDATE, autoupdate would add
a blank line and then its STDERR CHECK line.
2) If there was a (blank) content line after AUTOUPDATE, autoupdate
would move the STDERR CHECK line to immediately after AUTOUPDATE,
leaving a trailing blank line.
3) pre-commit would remove the trailing blank line, leaving no content
lines after AUTOUPDATE.
Handle the special case of no content lines after AUTOUPDATE by
producing "early" STDERR check lines immediately after the AUTOUPDATE
line, rather than before the next line, which might not exist. We
already did this in the case where there were early STDOUT check lines.
This updates SourceBuffer to diagnostics. Some additional edits to
diagnostics were necessary due to issues moving arguments around, which
seems to stem from a compile error with clang 14 (fixed in later
versions).
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Rearranges driver logic into CompilationUnits in order to associate
artifacts from the various stages of compilation.
Note, I'm not totally sure what the right thing to do is for
lower/codegen, so I'm just doing a rote change there for now that
mirrors prior phases (this is all the code supports anyways, so is
probably right for now regardless).
SourceBuffer error output is moved local for consistency with other
steps, and so that it's less ambiguous whether the error should be
expected to already include a filename.
There were two issues contributing to this crash:
- Primarily, the issue is that we queue up diagnostics and don't format
them into a string until we reach the end of compilation. In some code
paths in the driver, we destroyed the Semantics IR object before this
happened. But diagnostics can contain references to Semantics IR
objects, such as strings stored in the string table, which can lead to a
use after destruction bug.
This is fixed by ensuring the diagnotics consumer is flushed before
destroying any of the objects that it can refer to. The current approach
to this is not especially clean, unfortunately, but this requires
fighting C++ as this isn't the order in which it wants to destroy
things.
- This issue was obscured by the Semantics IR's string table holding a
reference to whatever underlying storage it was given rather than its
own string storage, so sometimes it would hold a reference to a string
from the source file, and sometimes a string from the tokenized buffer's
string table. The diagnostics were always flushed before the source file
was destroyed, but not before the tokenized buffer was destroyed. So to
see the issue, you'd need to have a string literal with certain contents
followed by an identifier with a name that matched those contents.
The crash is made more reliable by holding references to the Semantics
IR's string map in its string table, rather than references to someone
else's strings. This also fixes a latent bug where passing a string
temporary to SemanticsIR::AddString would store a dangling reference in
the string table. Incidentally, AddString is also changed to perform
only one hash table lookup rather than two for each added string.
This also tries to restructure the command line interface to the
toolchain a bit to make it start operating more like a compiler that
could be integrated into a build system rather than primarily as
a testing tool.
1) This switches form a `dump` subcommand to a `compile` subcommand
which has "dump" actions that can be enabled within it.
2) A distinct set of compile _phases_ that match the toolchain
structure:
- `lex` to run the lexer
- `parse` to run the parser
- `check` to fully check that the code is valid
- `lower` to lower to LLVM's IR
- `codegen` to generate executable code
3) The codegen phase has two output formats: textual assembly and
a binary object. These outputs can be configured, with a default for
an object when writing to a file and more firm default for textual
assembly when writing to stdout.
4) Select and expose the use of the LLVM host detection to compute
a default code generation target in the driver so that the command
line interface can reflect this. For example, the `help` output will
include the default target.
5) The `//toolchain/codegen` library APIs have been restructured a bit
to make the code flow a bit more naturally when implementing the new
command line structure. No real changes to the logic though.
There are also some minor tweaks to the command line interface based on
trying to use the shortest names for things that still seem likely to be
learnable for users:
- Switched `target-triple` to just `target`: the "triple" component to
this name is historical and can be confusing. For example, almost all
"triple" strings have more than three components today.
- Switched to just `--output` as now the fact that it is a file can be
configured in the documentation -- it will render as `--output=FILE`.
This also adds support for two custom output filename modes. First, when
no output is specified, we now compute one in the conventional way for
compilers by removing the file extension of the input file and replacing
it with `.o` for an object file output or `.s` for an assembly file
output. This matches the behavior of Clang and GCC for example.
Second, output to stdout is enabled with the special output file name of
`-` since it is no longer the default. This also follows the convention
of most compilers and many other command line tools to use `-` as a file
name to signify using standard in/out pipes.
There are still some rough edges here that I suspect could be improved,
but this seems like a good start of switching over to a complete
argument parser.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Lucile Rose Nihlen <luci.the.rose@gmail.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Allow interleaving of STDOUT and STDERR check lines. Put STDOUT lines
after the line they're attached to, and STDERR lines before. If no
STDOUT check line is attached to any line, then put them all at the end
of the file instead.
This is intended to better handle the case where stdout contains
unreplaced mentions of line numbers, and also reflects that stdout is
typically a consequence of the test rather than commentary on it, so
placing it after the test seems likely to read better.
I've migrated the toolchain autoupdate scripts here, I just need a
little more time to do the explorer side (which I need to check
performance, that may require multithreading as we do in the current
script). However, this felt substantial enough to share and it means I
can handle autoupdate in more of the toolchain, including preparatory
work for autoupdate on multi-file tests.
Once explorer is done I'll remove the old script.
This is sufficient to convert remaining toolchain tests to file_test. %s
and %t are currently used. While #2978 might use %T with lit, I think
that's resolving a lit-specific issue that's not necessary when
TEST_TMPDIR is readily available for any necessary operations (i.e.,
setting the working directory). With ARGS, I think it's feasible to
switch to file_test without %T.
Also adds documentation to file_test_base.h, which was starting to feel
like a significant gap.
This is not yet handled by autoupdate, but I'm eyeing that next.
This also makes the filename a reference to the buffer since the line
seems better to have as a reference (versus copying a string per error).
Most tests now have different line deltas due to the extra output, but
the actual errors should overall stay the same.
Some of the error locations look like they could be improved, but this
change is only making it clear where they were before.
Jon Ross-Perkins
Sam EstepandJon Ross-Perkins
Geoff Romer
David Blaikie
Jacob Schneider
