Commit Graph
192 Commits
Author SHA1 Message Date
Richard SmithandChandler Carruth 1013d1773c Model function calls as initializing expressions (#3089)
Start treating function calls as initializing expressions instead of as
value expressions.

This required adding support for expression categories. Value bindings
and temporary materialization conversions are created where necessary to
transition between expression categories. For a function call with a
return slot, we speculatively create a materialized temporary before the
call and either commit to it or replace it with something else later,
once we see how the function call expression is actually used.

This change follows the direction suggested in #3133 for initializing
expressions: depending on the return type of a function, the return
value will either be initialized in-place or returned directly. This is
visible in the semantics IR, which is a little unfortunate but is
probably necessary as this is part of the semantics of the program.

---------

Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
2023-08-24 19:35:07 +00:00
Richard Smith 0ff5e42b12 Put the IR for the return type in the same block as the IR for the parameters. (#3145)
Previously, IR generated for the computation of the return type ended up
in the package's node block. Put it with the IR for the parameters for
consistency and cleanliness.
2023-08-23 23:38:17 +00:00
Jon Ross-Perkins 67da700dd5 Split Semantics into Check and SemIR namespaces (#3138)
Splits IR files into SemIR, and logic files into Check. These will be
split into separate directories as part of a later move; the namespaces
are being done first in order to vet the switch, and hopefully make
conflicts a little easier to manage due to the substantial renames.

A lot of this is just automated removal of Semantics prefixes from
names, adding namespace references where needed. A few special-cases
are:

- SemanticsIR -> SemIR::File
- A few things were discussed, like Unit, CompileUnit, or CompiledUnit.
Unit was too vague for chandlerc, and I thought CompileUnit might lead
to incorrect inferences (CompilationUnit would be more precise, but
typically written as SemIR::CompilationUnit which is pretty long). File
seemed to be a short name that we could agree on.
- SemanticsIRFormatter -> SemIR::Formatter
- FormatSemanticsIR -> SemIR::FormatFile
- SemanticsFileTest -> CheckFileTest
- It remains in the Testing namespace, where just "FileTest" might be
too broad a name.
- SemanticsDeclarationNameStack::Context ->
Check::DeclarationNameStack::NameContext
  - This avoids a Check::Context name shadowing.

Changes check_internal.h to include ostream.h to improve finding of
Print/operator<< (otherwise it didn't compile).

This is part of #3070
2023-08-23 22:51:23 +00:00
Richard Smith a64ffea657 Model parameters as value expressions rather than reference expressions. (#3144)
Add a new node kind to distinguish parameters from local variables so
that we can give the two kinds of binding different expression
categories.
2023-08-23 20:23:04 +00:00
Jon Ross-Perkins 7330e6698c Switch more of toolchain to enumerate/seq (#3137)
These were already used in a few spots, just trying to switch more loops
for consistency and ease of reading.
2023-08-23 00:47:03 +00:00
Jon Ross-Perkins 605763d62d Add lint fixes to the buildifier setup. (#3109)
The main motivation for this is to get python loads in using the
`native-py` lint fix. However, enabling that made me wonder, maybe we
should fix in general?

`native-cc` is delayed, but not wholly cancelled (and `native-py`
picking up might indicate `native-cc` won't be too far behind). There's
also some automated fixes for `.append` and dict sorting -- this felt
okay to me, maybe not something to eagerly add but probably not worth
stopping buildifier from fixing (I've noticed the warnings in the past
and had been ignoring them).

Running everything does mean that load orders are sorted automatically
now, which I think is a positive. Most generally, I think these fixes
aren't _harmful_, and having them done automatically seems beneficial:
my biggest concern about `native-py` and `native-cc` was actually that
regressions wouldn't be caught, but this addresses that issue
automatically.
2023-08-22 21:01:42 +00:00
Jon Ross-Perkins 2425e28e3a Collapse names into VarStorage (#3116)
This removes BindName, putting name information directly on VarStorage.
As a side-effect of updating semantics_ir_test for this change, I also
noted that function bodies were being generated as invalid YAML so am
fixing that (just `{}` to `[]` bracketing, otherwise the test wouldn't
work anymore).

Because names are now available, I've updated lowering to use them for
vars.

In the SemIR formatter, the name is now repeated because it's a
parameter to VarStorage. I believe this is just default behavior, and
we'd have to special-case VarStorage to remove it because it's automatic
argument printing in action. On the balance, it felt like letting it
print was reasonable.

I've noted in places that the name on VarStorage is expected to be
optional, but am not adding support because I'd have no way of testing
it at present.
2023-08-22 18:55:31 +00:00
Richard Smith e05523db21 Fix a stack use after scope and a heap use after free found by fuzzing. (#3126)
There were two issues contributing to this crash:

- Primarily, the issue is that we queue up diagnostics and don't format
them into a string until we reach the end of compilation. In some code
paths in the driver, we destroyed the Semantics IR object before this
happened. But diagnostics can contain references to Semantics IR
objects, such as strings stored in the string table, which can lead to a
use after destruction bug.

This is fixed by ensuring the diagnotics consumer is flushed before
destroying any of the objects that it can refer to. The current approach
to this is not especially clean, unfortunately, but this requires
fighting C++ as this isn't the order in which it wants to destroy
things.

- This issue was obscured by the Semantics IR's string table holding a
reference to whatever underlying storage it was given rather than its
own string storage, so sometimes it would hold a reference to a string
from the source file, and sometimes a string from the tokenized buffer's
string table. The diagnostics were always flushed before the source file
was destroyed, but not before the tokenized buffer was destroyed. So to
see the issue, you'd need to have a string literal with certain contents
followed by an identifier with a name that matched those contents.

The crash is made more reliable by holding references to the Semantics
IR's string map in its string table, rather than references to someone
else's strings. This also fixes a latent bug where passing a string
temporary to SemanticsIR::AddString would store a dangling reference in
the string table. Incidentally, AddString is also changed to perform
only one hash table lookup rather than two for each added string.
2023-08-22 00:51:13 +00:00
Richard Smith d74b8f0497 Fix diagnostic messages that don't end in a period. (#3127) 2023-08-21 23:59:18 +00:00
a2b4cabeaa Switch the toolchain to the new CLI library. (#2979)
This also tries to restructure the command line interface to the
toolchain a bit to make it start operating more like a compiler that
could be integrated into a build system rather than primarily as
a testing tool.

1) This switches form a `dump` subcommand to a `compile` subcommand
   which has "dump" actions that can be enabled within it.

2) A distinct set of compile _phases_ that match the toolchain
   structure:
   - `lex` to run the lexer
   - `parse` to run the parser
   - `check` to fully check that the code is valid
   - `lower` to lower to LLVM's IR
   - `codegen` to generate executable code

3) The codegen phase has two output formats: textual assembly and
   a binary object. These outputs can be configured, with a default for
   an object when writing to a file and more firm default for textual
   assembly when writing to stdout.

4) Select and expose the use of the LLVM host detection to compute
   a default code generation target in the driver so that the command
   line interface can reflect this. For example, the `help` output will
   include the default target.

5) The `//toolchain/codegen` library APIs have been restructured a bit
   to make the code flow a bit more naturally when implementing the new
   command line structure. No real changes to the logic though.

There are also some minor tweaks to the command line interface based on
trying to use the shortest names for things that still seem likely to be
learnable for users:

- Switched `target-triple` to just `target`: the "triple" component to
  this name is historical and can be confusing. For example, almost all
  "triple" strings have more than three components today.

- Switched to just `--output` as now the fact that it is a file can be
  configured in the documentation -- it will render as `--output=FILE`.

This also adds support for two custom output filename modes. First, when
no output is specified, we now compute one in the conventional way for
compilers by removing the file extension of the input file and replacing
it with `.o` for an object file output or `.s` for an assembly file
output. This matches the behavior of Clang and GCC for example.

Second, output to stdout is enabled with the special output file name of
`-` since it is no longer the default. This also follows the convention
of most compilers and many other command line tools to use `-` as a file
name to signify using standard in/out pipes.

There are still some rough edges here that I suspect could be improved,
but this seems like a good start of switching over to a complete
argument parser.

---------

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Lucile Rose Nihlen <luci.the.rose@gmail.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
2023-08-18 01:06:09 +00:00
Jon Ross-Perkins 2e290b9e73 Fix missing APSInt include. (#3107)
Works by chance right now, but should be expected to break in future
LLVM versions.

The builtin kind include is removed as unused.
2023-08-16 18:17:57 +00:00
Jon Ross-PerkinsandRichard Smith 09242eeebd Run clang-tidy over toolchain code. (#3105)
Amongst other changes, this removes the implicit StringRef cast in
formatter code due to google-explicit-constructor. Only one line of code
relied on the implicit cast, and style says "Do not define implicit
conversions."
(https://google.github.io/styleguide/cppguide.html#Implicit_Conversions).

---------

Co-authored-by: Richard Smith <richard@metafoo.co.uk>
2023-08-16 15:21:58 +00:00
Jon Ross-Perkins fa857e42be Rename //testing/util to base (#3104)
Renaming per #3100
2023-08-15 21:28:56 +00:00
Farzana Ahmed SiddiqueandFarzana Ahmed Siddique a6ba7827cd Semantics for array index. (#3099)
Co-authored-by: Farzana Ahmed Siddique <fasiddique@google.com>
2023-08-15 18:17:42 +00:00
Jon Ross-Perkins a692fb89a3 Rename //toolchain/common to base (#3101)
Renaming per #3100
2023-08-15 17:47:39 +00:00
Jon Ross-Perkins 956d65e333 Remove stray semicolon (#3098) 2023-08-14 20:30:39 +00:00
Farzana Ahmed SiddiqueandFarzana Ahmed Siddique 9228487d46 Semantics for array type (#3087)
Co-authored-by: Farzana Ahmed Siddique <fasiddique@google.com>
2023-08-11 23:06:13 +00:00
Richard Smith 883888e22c Produce label names based on the reason for the branch. (#3086)
When producing textual semantics IR, map each branch instruction back to
the construct that produced it and use that to determine a name for the
corresponding block label.
2023-08-11 01:20:29 +00:00
Richard Smith 6e01b90394 Rename some semantics nodes and some IR names to make them match better. (#3085) 2023-08-10 21:48:50 +00:00
Richard Smith 9c516c40b9 Always create a StructType node for every syntactic struct type. (#3084)
Don't elide the StructType node if we've already created an equivalent
type. Its spelling and location may be interesting to diagnostics,
tooling, debug information, etc.
2023-08-10 20:30:39 +00:00
Richard SmithandChandler Carruth 6cbf280a68 Add formatted textual IR output (#3056)
Add a textual IR format to the toolchain.

The exact details of the format are somewhat arbitrary right now, and I
expect them to change as we refine the semantics IR model, but at the
moment they're somewhat directly following the current structure of the
IR.

Semantics tests currently test both the "raw" format, which shows the
details of the representation, and the textual format, which is somewhat
higher level. We may want to revisit that decision once the textual
format is a bit more stable, and test only the textual format in most of
these tests, but for now it seems prudent to keep both sets of tests.

---------

Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
2023-08-10 19:41:39 +00:00
Richard SmithandChandler Carruth fc5a9541ce Update precedence rules to match design. (#3081)
- Only allow assignment at the top level in an expression statement.
- Allow both negation and complement as subexpressions of both bitwise
  and numeric operators.
- Remove parsing support for postincrement and postdecrement.
- Add parsing support for `as` operator.
- Use the same ambient precedence for types and non-type expressions.

---------

Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
2023-08-10 18:39:37 +00:00
Richard Smith 5b45c2319f Only allow assignment to durable reference expressions. (#3077)
Now that we support expression categories, use them to determine whether
the left-hand side of an assignment expression is valid.
2023-08-09 19:46:34 +00:00
Farzana Ahmed SiddiqueandFarzana Ahmed Siddique a67aeb5724 Parser for array type. (#3075)
Co-authored-by: Farzana Ahmed Siddique <fasiddique@google.com>
2023-08-09 19:04:44 +00:00
Richard Smith 2947877518 Add support for dereference operator. (#3066)
Also fix a couple of error-recovery issues exposed by the tests for this
change.
2023-08-08 21:55:32 +00:00
Richard Smith 62205763a5 Add support for & operator. (#3055)
Refactor type canonicalization so that we can reuse the same code for
building a `T*` expression and for forming the type of an `&x`
expression.

Add basic computation of expression category in order to check that we
only take the address of durable reference expressions. This is
currently computed on demand rather than being tracked as part of the
semantics node, but in most cases can be determined by looking at only a
single expression, so caching it in the node doesn't seem worthwhile
yet. This decision should be revisited if we start doing more complex
category calculations.

Also add trivial lowering support, but it doesn't work properly yet
because lowering doesn't yet take the expression category into account.
2023-08-08 20:53:50 +00:00
Richard Smith 212188a922 Prefer to put STDOUT CHECK at the end of the file. (#3073)
Allow interleaving of STDOUT and STDERR check lines. Put STDOUT lines
after the line they're attached to, and STDERR lines before. If no
STDOUT check line is attached to any line, then put them all at the end
of the file instead.

This is intended to better handle the case where stdout contains
unreplaced mentions of line numbers, and also reflects that stdout is
typically a consequence of the test rather than commentary on it, so
placing it after the test seems likely to read better.
2023-08-08 19:51:52 +00:00
Farzana Ahmed SiddiqueandFarzana Ahmed Siddique ce459d4da7 Semantics & Lowering for simple assignment. (#3058)
Co-authored-by: Farzana Ahmed Siddique <fasiddique@google.com>
2023-08-04 20:23:46 +00:00
Farzana Ahmed SiddiqueandFarzana Ahmed Siddique 3a093980a6 Lowering for tuple indexing. (#3059)
Co-authored-by: Farzana Ahmed Siddique <fasiddique@google.com>
2023-08-03 21:54:58 +00:00
Farzana Ahmed SiddiqueandFarzana Ahmed Siddique 64051910c8 Toolchain: Semantics for indexing (#3044)
Co-authored-by: Farzana Ahmed Siddique <fasiddique@google.com>
2023-08-03 21:09:59 +00:00
Richard Smith c8b42379a4 Basic semantic checking for pointer types and const-qualified types. (#3038)
Semantic handling for use of `T*` and `const T` as types.

There's no way to form values of these types yet, and no conversions for
them are supported.

Factor out the common code to canonicalize types using a folding set,
and switch to using the same folding set for all kinds of type by adding
the kind as part of the folding set key.

Improve type printing to not include the `as type` portion when the type
is printed in a context within another type where a conversion to `type`
is implied, as in `{}*` and pre-existing cases like `({}, {}) as type`
(which we used to print as `({} as type, {} as type}) as type`.
2023-08-02 23:19:52 +00:00
Jon Ross-Perkins b5167b2d69 Implement autoupdate for file_test. (#3043)
I've migrated the toolchain autoupdate scripts here, I just need a
little more time to do the explorer side (which I need to check
performance, that may require multithreading as we do in the current
script). However, this felt substantial enough to share and it means I
can handle autoupdate in more of the toolchain, including preparatory
work for autoupdate on multi-file tests.

Once explorer is done I'll remove the old script.
2023-08-02 21:32:21 +00:00
Richard Smith 90d2d7ef65 Fix some minor errors in diagnostic text. (#3047) 2023-08-02 18:17:16 +00:00
Richard SmithandJon Ross-Perkins 0c33dead70 Remove the type field from semantics nodes that don't produce values of that type. (#3049)
For `Assign` and `ReturnExpression`, this field wasn't used for
anything. For `StructTypeField`, we stored the type of the field here,
and now store it as an argument of the node instead.

---------

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
2023-08-02 17:30:27 +00:00
Richard Smith ab709b4117 Fix crash type-checking member access into an invalid expression. (#3046)
Found by fuzzer.
2023-08-01 21:47:13 +00:00
Jon Ross-PerkinsandChandler Carruth eb05f618ce Refactor FileTest construction so that the test class is directly available. (#3035)
This is a simplification of the construction, although somewhat limiting
(it means that the caller can't register the same file multiple times,
though I stopped doing that anyways since it was causing confusion).
What this more importantly _allows_ is logic on the FileTestBase child
itself that's not test-specific -- in particular, autoupdate
functionality which wouldn't use RUN_ALL_TESTS.

---------

Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
2023-08-01 16:00:29 +00:00
Farzana Ahmed SiddiqueandFarzana Ahmed Siddique 6cca85534f Parser for index expression such as a[0] (#3033)
Co-authored-by: Farzana Ahmed Siddique <fasiddique@google.com>
2023-07-28 18:31:12 +00:00
db5e269097 Removed builtin empty tuple type (#3021)
Co-authored-by: Farzana Ahmed Siddique <fasiddique@google.com>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
2023-07-27 21:17:29 +00:00
Richard Smith c4b880c6ef Parsing for pointer types and pointer operators. (#3026)
This provides parsing support for the functionality added in #2006.
2023-07-26 21:24:54 +00:00
Jon Ross-Perkins f088a71cec Add file_test support for specifying arguments and only checking a subset of output. (#3018)
This is sufficient to convert remaining toolchain tests to file_test. %s
and %t are currently used. While #2978 might use %T with lit, I think
that's resolving a lit-specific issue that's not necessary when
TEST_TMPDIR is readily available for any necessary operations (i.e.,
setting the working directory). With ARGS, I think it's feasible to
switch to file_test without %T.

Also adds documentation to file_test_base.h, which was starting to feel
like a significant gap.

This is not yet handled by autoupdate, but I'm eyeing that next.
2023-07-25 22:47:56 +00:00
Richard Smith 35ccac5125 Semantics and lowering for type keyword and type values. (#3022)
Also switch from modeling type values as `i1` to modeling them as an
empty struct. We don't need any runtime representation for types, as
there are no runtime operations on them, so an empty struct seems like a
good representation.
2023-07-25 21:49:45 +00:00
Jon Ross-Perkins 60c9b202f6 Relocate fuzzer files and the explorer_fuzzer rule. (#3017)
Putting fuzzer files under //testing to emphasize the testonly aspect
(consolidates bazel and common subdirectories). The attributes on
explorer_fuzzer are also a little skewed from what's desirable; it's
been working okay, but this should still be a refinement.
2023-07-25 13:27:56 +00:00
Jon Ross-Perkins 907fa82fc6 Fuzz semantics when it passes parsing without errors. (#3003)
Right now we expect crashes on invalid parses, we just don't try to
handle it in general even though the long-term intent is we should
handle semantics for bad parses. However, in theory, we should be
correctly handling code that parses as valid and that's probably more
interesting to fix bugs for. So this starts trying to fuzz that space of
valid parses.

Fuzzer corpus is based on explorer tests, with one merge run.
2023-07-22 00:03:40 +00:00
Jon Ross-Perkins 65a4e006a2 Add line output to diagnostics to help identify error locations. (#3010)
This also makes the filename a reference to the buffer since the line
seems better to have as a reference (versus copying a string per error).
Most tests now have different line deltas due to the extra output, but
the actual errors should overall stay the same.

Some of the error locations look like they could be improved, but this
change is only making it clear where they were before.
2023-07-21 23:53:36 +00:00
Farzana Ahmed SiddiqueandFarzana Ahmed Siddique 6a81a3396e Updated tests for tool chain to incorporate type blocks (#3008)
Co-authored-by: Farzana Ahmed Siddique <fasiddique@google.com>
2023-07-21 17:36:24 +00:00
Jon Ross-Perkins 0acc62fec0 Clean up unnecessary Carbon:: uses in the node stack. (#3004)
Not sure how I let these creep in.
2023-07-21 15:41:16 +00:00
Jon Ross-Perkins e51f971bce Reduce special-casing of parameters in ParamOrArgSave. (#3001)
I think this is a simpler and more efficient way of achieving the same
end result.
2023-07-21 01:08:33 +00:00
0594a1aac7 Integrates tuple to semantic analysis (#2992)
Handles tuples (including nested tuples) in the semantic phase of the
tool chain. Does not handle tuple element access yet.

---------

Co-authored-by: Farzana Ahmed Siddique <fasiddique@google.com>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
2023-07-21 01:05:37 +00:00
Jon Ross-Perkins 446b0ce4ae Refactor declaration name context logic to its own class. (#2989)
This started with cleaning up the remaining Name/expression type punning
in the node stack, and grew. I'm factoring out a class because we've
previously expressed the desire to factor logic out of SemanticsContext
where possible, and this seemed like a reasonable cut.

NameExpression as the first node as a QualifiedExpression allows the
qualifier handling to consider Name in one less spot, an incremental
simplification. However, the additional complexity caused by this makes
me split ApplyNameQualifier/ApplyExpressionQualifier in order to avoid
repeat checks of the parse node's kind. The logic is still largely
shared, thus a couple helper functions. I think this is all fairly well
structured in the isolated class.

I can see that we may want to avoid passing SemanticsContext as an
argument in the future if it elides a step of lookup.
2023-07-17 21:55:29 +00:00
Jon Ross-Perkins 43065a1257 Finish refactoring Push/Pop for stronger type handling. (#2987)
This adds a distinction between Unused and SoloParseNode, rather than
equating the two. This is intended to help identify nodes which are
getting pushed but maybe don't need to be.

Not totally done because I want to adjust declaration name handling due
to a quirk with how it mixes Name with Expression, but almost done. Once
that's done the type punning will be completely gone.
2023-07-14 23:16:16 +00:00