This removes the directory crawl because bazel doesn't remove files from
execroot when the rule generating them would no longer generate them.
Fixes#4288
From the driver's perspective, `FindPreludeFiles` is closely tied to
`compile`. This makes it difficult to refactor commands without
affecting the test dependencies on `FindPreludeFiles`. `InstallPaths`
seems like a decent home since it is responsible for the install
structure.
I'm switching to an `Error` return to allow callers to choose how to
handle it (e.g., in file tests, we typically don't want the direct error
stream).
Discussed in the toolchain meeting today - we'd like to try having this
on by default and see if the cost isn't too high.
The nodebug test is a bit verbose, because it doesn't have the
`--exclude-dump-file-prefix` that test_file would usually add. Is there
a nicer way I could write this test to verify that --no-debug-info does
what it's meant to?
This adds just the debug info metadata for Compilation Units (the top
level container of debug info) - but without anything in them, LLVM
won't emit them at all, so while this is testable at the IR level, it
isn't observable at the object level until more debug info is added.
A couple of starting points in this patch:
* A flag (`--debug-info`, seems to match the naming/style of other flags
in the carbon driver, though this is different from the naming
conventions of clang/gcc) that enables debug info when lowering. Open to
other names/approaches (on by default? historically debug info's been to
large/expensive to do this, so sticking with that precedent for now).
* Enabling that flag by default in the lowering tests - I do find the
churn on golden tests a bit rough, and adding more features to all the
tests means more churn, but it seems consistent with the approach so far
- keep an eye on this and perhaps revisit this if the churn gets too
annoying
---------
Co-authored-by: Carbon Infra Bot <carbon-external-infra@google.com>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Just spotted these while looking at warnings that seem to fire on our
code are probably are things we'd fix if we saw them. None of these seem
important FWIW.
Also removes a redundant flag that is part of `-Wall`.
I have a follow-up for the high-value warning I spotted that motivated
me to look at all of this. But it's noisy so kept it as a separate PR.
This removes the `data_dir` from the driver favoring the installation
abstraction for the both locating the prelude and linking utilities.
With this, an installed toolchain should also be able to compile and
link Carbon successfully, and the build of the examples should exercise
this path almost exactly. (The only difference is using the driver
`cc_binary` directly rather than relying on the symlink from inside the
install tree.)
The install directory contains the BUILD logic for creating an
installable tree of data files and executables for the toolchain, and
a library to facilitate toolchain code accessing the paths to their data
within this installation.
Then adds an installation of LLD in a synthetic LLVM installation, and
teaches the Clang runner to configure this and use it for linking
instead of the system linker.
Currently, the install paths only really manage access to the LLVM
binaries installed and used by the Clang runner for linking, but
eventually other data files like the prelude and runtime libraries will
be fleshed out as well. There are TODOs for moving more things over here
such as the prelude.
One interesting aspect of this is where to put helpers like parts of
LLVM in our install. This PR suggests nesting those files under
`lib/carbon`. While using a `lib` subdirectory isn't a perfect fit for
the FHS (Filesystem Hierarchy Standard), having a single location where
private data is collected is significantly superior to spreading them
across the system. This also matches similar patterns used by Clang
itself and several other language toolchains and standard libraries.
The install directory also provides a natural place for us to build out
packaging rules to create installable packages in various formats, but
that remains future work.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This could probably go further, but I'm just trying to get the ability
to run some significant tests without the prelude. This already requires
some migration of `i32` and `bool` to non-prelude-dependent types
(technically not at the moment, but I'm trying to stick with the
philosophical model of the prelude).
Note this undoes part of #3895 which had made all lex/parse tests depend
on the prelude -- I don't think that was a necessary change. Rather, it
seems better to isolate individual lex/parse tests from the prelude. I'd
been on the fence in that PR (it was fixing one of the issues that
contributed to wanting to not include the prelude), but now it feels
more consistent. The prelude could have a parse error and we shouldn't
break every parse test on that.
The driver now looks for all files under core/prelude/ and considers
them all to be part of the prelude. The driver also now only processes
the prelude in `--phase=check` and later, when it would actually be
imported.
With that done, add a simple `carbon_binary` build rule and use it to
build the example in `//examples`. This should cause the example to be
built as part of our continuous integration.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This is just to be able to directly run the driver (e.g., `bazel run
//toolchain/driver:carbon compile foo.carbon`); it shouldn't affect
anything else.
The `file_test` layer still adds it to the VFS and filters it from the
dump output, but the driver now injects it into the compilation unites
depending on the `--import-prelude` file (enabled by default).
This doesn't address improvements to how we find the prelude code, just
sinking the needed logic into the driver itself.
- Adds an empty prelude.carbon file
- Imports that file in any non-Core package file
- Adds --disable-prelude-import to avoid that
- Adds --exclude-dump-file-prefix to be able to hide files from dumping
- Used to hide core files (we can't do this by package name due to lex
dumps, for example)
- Restructures some tests to not rely on `i32`, particularly `alias`
tests (which rely on a name ref) and tests with no prelude.
I'm adding the framework for switching i32 to calling Int32 in the
prelude, but I'm running into a separate error actually switching over.
So that *mostly* works, but isn't quite ready for prime time. However,
maybe the current state of this PR is still useful to review since it
does a lot of the infrastructure work and adds the %Core everywhere?
This handles toolchain failures per-file. The intent is to allow placing
both "success" and "fail" tests in the same file, using splits. However,
this PR only adds support and updates existing tests to continue
passing.
This removes a (very) hot branch in the lexer where we need to special
case when a token is the first token and can't look at its previous
token. It also seems like a generally nice change to the structure of
both the token buffer and parse tree as there are now bracketing
elements for both ends and we should be able to avoid similar branching
in the future.
Mostly mechanical updates to the lexer and parser code to handle this,
but also needed to special case the location information in the
autoupdate code. And then the usual large body of auto-updated tests.
No benchmark data for this change alone as in isolation and in the
current lexer structure it doesn't make a big difference. But this
branch was particularly difficult to handle when trying to update the
whitespace skipping code to be faster, and so I think it is worth
systematically avoiding the special case here.
Lex now prints its yaml as:
```
- filename: name
tokens: [ ... ]
```
New support in file_test allows the `filename` marker at the top to
define the default file number for later lines, meaning multi-file
output from lexing is now associated with the appropriate file. Similar
support will probably also apply to lowering, semir, and other places
that print a filename once for the full dump.
This hammers a bit at how line number replacements work in file_test,
allowing stacking them so that lex errors and stdout can both be
line-associated properly. I've tried to make the autoupdate more
frequently work in one pass, now also taking into account the file index
when doing line replacements.
There are still some issues with EndOfFile that it may be good to
discuss: because CHECK lines are appended to the end of the file now,
and the EndOfFile token points at the last line including comments, new
lex tests now take two runs to autoupdate (because without CHECK lines,
the EndOfFile points at a content line, which content is then inserted
after). Note that removing CHECK lines from the test is not a solution:
autoupdate also started inserting blank lines, which breaks this for a
similar reason. One solution here might be to not have EndOfFile
associate with a line or column, which has been a bit of an issue
regardless.
Also fixes a small issue with toolchain's autoupdate script.
Specifically this should improve linking by producing one large binary
instead of one-per-directory. The inclusion of the driver hits the size
issue. Separating out things which have more llvm deps has been
discussed, but I'm not doing that here because I think the semantics
layer will need to depend on clang for interop, and we'd lose a lot of
the benefits that way. Also, having just one place to look seems
simpler.
Includes supporting changes to file_test infrastructure, the most
significant of which is probably passing tests via file instead of a
really large args thing, using a custom rule to do that. That's because
dealing with the layered filegroups that allow the toolchain setup is
more complicated, and this approach scales well.
Combined test time is ~9s, so not sharding right now.
I wasn't sure if people would prefer having the autoupdate script under
testing, so I left it alone for now.
Jon Ross-Perkins
David Blaikie
Chandler Carruth
Richard SmithandJon Ross-Perkins
josh11bandJon Ross-Perkins