The goal of this proposal is to provide a way for user-defined types to
support range-based iteration with `for`.
The current proposed solution exposes 3 interfaces that can be
implemented by user types to enable support for
ranged-for loops.
Co-authored-by: josh11b <josh11b@users.noreply.github.com>
---------
Co-authored-by: josh11b <josh11b@users.noreply.github.com>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Geoff Romer <gromer@google.com>
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Previously, accessing any property of a token's kind required both
a function call to an out-of-line method and loading the relevant data
out of a table. These method calls were often roughly as expensive as
the load out of the table -- they would push and pop registers, and so
would touch a stack cache line not unlike the table cache line.
We can inline them in a way that carefully leaves the table definitions
in the single TU so we don't get tons of copies of data that have to be
merged by the linker. But the access to the table can be in an inline
function that allows the call overhead to evaporate and these to turn
into just table loads in the callers as well.
While in theory we could rely on forms of LTO to acheive this, it
doesn't seem worth relying on that. This is an easy win with very little
cost in reality.
Several of the lexer benchmarks improve by 1% or 2% from removing the
function call overhead. It's not a huge win, but it's nice.
Even more nice is that the profile is cleaned up significantly, clearly
focusing on the core functions where time is spent in lexing and
parsing.
This also pulls my big-picture benchmark of 10mloc file down to under
4.3s to lex and parse excitingly. Basically, we're above 2.3mloc/s
lexing and parsing. Not *quite* the 10mloc/s that I'm hoping for, but
still progress.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
A lot of this is more boring "remove unused header", plus some other
minor cleanups. I think the most significant changes were:
- yaml_test_helpers.cpp is doing a switch on an unsigned int, comparing
to enum values.
-
[bugprone-switch-missing-default-case](https://clang.llvm.org/extra/clang-tidy/checks/bugprone/switch-missing-default-case.html)
is unhappy with EnumBase, but correctly identified
yaml_test_helpers.cpp, so I'm opting to address it rather than disabling
it even though it needs NOLINT in several locations as a result, in
addition to what I think are some low-value `default` cases. I'd be fine
going the other way with this too and disabling it globally (I could see
it being noisier in the explorer).
- MarkInitializerFor swaps the argument names between the .h and .cpp. I
think the .cpp had the order as intended.
- There's a new-ish
[performance-enum-size](https://clang.llvm.org/extra/clang-tidy/checks/performance/enum-size.html)
which I'm basically treating as "add int8_t to enums".
My main motivation here is to just clean up as many of these as I can so
that I stop seeing them in vscode.
This is a rough script that uses regexes to do a simple scan of source
code and extract some basic source code statistics. Things like column
width, comment line density, identifier lengths and densities.
After scanning, it prints out both raw stats and in a few cases renders
a quick histogram to the terminal to help visualize a relevant
distribution.
I threw this together pretty quickly, and this is an area of Python I
have very limited familiarity with, so happy to have any suggestions for
how to better approach this.
Trying to improve diagnostic names and code flow. I was thinking
"operand" might be a better name than "name" in the context of what `a`
is in `a[b]`, where we already call `b` the `index`.
Flow-wise, note this replaces pushing BuiltinError with instead creating
Index nodes that may contain an error. This is intended to aim towards
producing a more consistent SemIR in the face of errors. There are pros
and cons of both approaches, but I'm aiming for simpler control flow.
The successor strategy talk isn't up yet, but will be on Sep 8 -- just
trying to get them all in one pass.
Slides were previously linked, which I could keep, although it's weird
to have a long link with just `(Slides)` at the end because it implies
the whole linked text is slides (rather than a video). So if that's
desired, I might switch instead to something like:
```
- Talk name, con ([video](youtube link), [slides](slides link))
```
But, this current format (video-only full-name link) felt consistent
with the 2022 links.
This PR removes `PrintDepth` from statement and declaration.
Implements `PrintIndent` for better indented formatting along with
various changes to make printing of statements and declarations better.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Pass function parameters directly or indirectly, depending on their
value representation. For types such as empty tuples with an empty value
representation, don't pass them at all.
For types using a pointer value representation, emit an `llvm.memcpy`
call to perform assignment rather performing a first-class aggregate
load and store.
It was pointed out in review that the approach to building random inputs
for the benchmark would produce significantly varying input lengths and
cause unavoidable noise of as much as 3% between runs.
While initially, these were only used for very coarse measurements, this
is a definite problem, and so this PR restructures things along the
lines suggested. Rather than generating random lengths or random
selections according to distributions, instead the lengths and coverage
of sets are done deterministically. Then the sequences are randomized in
order to prevent getting stuck in a silent local minima or maxima.
The technique works to ensure that the randomness is done on every run
of the benchmark, not just every run of the program, so that we don't
have noise hidden from run-to-run. That means fresh memory allocation
and some wasted time computed freshly shuffled inputs, but makes sure
the unavoidable noise from things like ASLR show up (as much as
possible) even with simple repetitions of the benchmark runs.
As part of this, this PR moves towards building an entirely custom
distribution of identifier lengths based on the direct measurements of
the LLVM codebase. These measurements are provided by a script that will
be in a subsequent PR, but the exact distribution doesn't matter as much
as our ability to control it and build on it in a fully deterministic
way.
To help with analyzing all of these, we also start tracking the bytes
processed in addition to the tokens processed as rates. The naming is
changed to be similar, and this produces the following nice output from
the benchmark now:
```
---------------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations bytes_per_second tokens_per_second
---------------------------------------------------------------------------------------------------------------
BM_ValidKeywords 3365571 ns 3365577 ns 206 177.449M/s 29.7126M/s
BM_ValidIdentifiers<1, 64, false> 12646151 ns 12645834 ns 52 116.466M/s 7.90774M/s
BM_ValidIdentifiers<1, 1, true> 4388390 ns 4388141 ns 161 65.1988M/s 22.7887M/s
BM_ValidIdentifiers<3, 5, true> 16286551 ns 16286572 ns 43 35.1334M/s 6.14003M/s
BM_ValidIdentifiers<3, 16, true> 15798770 ns 15797567 ns 44 69.4229M/s 6.33009M/s
BM_ValidIdentifiers<12, 64, true> 15670499 ns 15670257 ns 38 243.421M/s 6.38152M/s
BM_ValidMix/10/40 7415940 ns 7415564 ns 94 134.257M/s 13.4852M/s
BM_ValidMix/25/30 7349454 ns 7349402 ns 95 121.762M/s 13.6065M/s
BM_ValidMix/50/20 6691223 ns 6690969 ns 105 100.3M/s 14.9455M/s
BM_ValidMix/75/10 5131517 ns 5131263 ns 135 86.4556M/s 19.4884M/s
```
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Start treating function calls as initializing expressions instead of as
value expressions.
This required adding support for expression categories. Value bindings
and temporary materialization conversions are created where necessary to
transition between expression categories. For a function call with a
return slot, we speculatively create a materialized temporary before the
call and either commit to it or replace it with something else later,
once we see how the function call expression is actually used.
This change follows the direction suggested in #3133 for initializing
expressions: depending on the return type of a function, the return
value will either be initialized in-place or returned directly. This is
visible in the semantics IR, which is a little unfortunate but is
probably necessary as this is part of the semantics of the program.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Previously, IR generated for the computation of the return type ended up
in the package's node block. Put it with the IR for the parameters for
consistency and cleanliness.
Splits IR files into SemIR, and logic files into Check. These will be
split into separate directories as part of a later move; the namespaces
are being done first in order to vet the switch, and hopefully make
conflicts a little easier to manage due to the substantial renames.
A lot of this is just automated removal of Semantics prefixes from
names, adding namespace references where needed. A few special-cases
are:
- SemanticsIR -> SemIR::File
- A few things were discussed, like Unit, CompileUnit, or CompiledUnit.
Unit was too vague for chandlerc, and I thought CompileUnit might lead
to incorrect inferences (CompilationUnit would be more precise, but
typically written as SemIR::CompilationUnit which is pretty long). File
seemed to be a short name that we could agree on.
- SemanticsIRFormatter -> SemIR::Formatter
- FormatSemanticsIR -> SemIR::FormatFile
- SemanticsFileTest -> CheckFileTest
- It remains in the Testing namespace, where just "FileTest" might be
too broad a name.
- SemanticsDeclarationNameStack::Context ->
Check::DeclarationNameStack::NameContext
- This avoids a Check::Context name shadowing.
Changes check_internal.h to include ostream.h to improve finding of
Print/operator<< (otherwise it didn't compile).
This is part of #3070
Previously, the code would try each form of lexing and let that
sub-lexing routine reject the code. This was very branch heavy and also
hard to optimize -- lots of hard to inline function calls, etc.
However, it's really nice to keep the different categories of lexing
broken out into their own functions rather than flattening this into
a huge state machine.
So this creates a miniature explicit state machine by building a table
of function pointers that wrap the methods on the lexer. The main lexing
loop simply indexes this table with the first byte of the source code,
and calls the dispatch function pointer returned.
The result is that the main lex loop is *incredibly* tight code, and the
only time spent appears to be stalls waiting on memory for the next
byte. =]
As part of this, optimize symbol lexing specifically by recognizing all
the symbols that are exactly one character -- IE, we don't even need to
look at the *next* character, there is no max-munch or anything else.
For these, we pre-detect the exact token kind and hand that into the
symbol lexing routine to avoid re-computing it. The symbols in this
category are really frequent symbols in practice like `,` and `;`, so
this seems likely worthwhile in practice.
The one-byte-dispatch should also be reasonably extendable in the
future. For example, I suspect this is the likely hot-path for non-ASCII
lexing, where we see the UTF-8 marker at the start of a token and most
(if not all) of the token is non-ASCII. We can use this table to
dispatch immediately to an optimized routine dedicated to UTF-8
processing, without any slowdown for other inputs.
The benchmark results are best for keyword lexing because that is the
fastest thing to lex -- it goes form 25 mt/s to 30 mt/s. Other
improvements are less dramatic, but I think this is still worthwhile
because it gives a really strong basis for both expanded functionality
without performance hit and further optimizations.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This adds a benchmark that tries to synthesize mixtures of symbols,
keywords, and identifiers. It tweaks the distribution of identifier
lengths based on some empirical measurements of, for example LLVM's
codebase.
Also establish a framework for skewing the symbol distribution, although
that one is based entirely on intuition and not measurements. It should
be adjusted as we have measurements.
The ratios between symbols, keywords, and identifiers is also
unmeasured, but several different ratios are covered.
Neither literals nor grouping symbols are included yet, as both present
some additional challenges in forming them, and this seemed like
a plausible increment in expanding the benchmark coverage.
Currently, for long identifiers, a huge (>30%) fraction of time is spent
finding the end of the identifier. We can speed this up with a fun
application of SIMD and in-register lookup tables.
With this, the BM_ValidIdentifiers/12/64 benchmark goes from around 4
million
tokens/second to around 6 mt/s, so roughly 1.5x improvement. However,
there was a decent amount of noise in the measurement and I didn't study
it too closely as I was very happy with the overall result. The profile
shifted from >30% of the time in this loop to <10% of the time, so the
scan itself is 3x or more faster with this.
One concern with optimizing the lexer right now is that we don't have
full Unicode support from the design. This PR takes some steps to at
least
try and avoid this pitfall -- the new routine works to classify UTF-8
code units, and has a fallback in that case that can grow the needed
logic.
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Currently this is focused on benchmarking the identifier and token
lexing paths, but should expand in the future to cover other parts of
the lexer.
In order to effectively benchmark tokens, this adds support to the
`token_kind` library to produce a list of all the tokens in Carbon that
the benchmark can use to create random inputs.
For identifiers, the benchmark has support for benchmarking different
distributions of identifier sizes so it is easy to zoom into the
performance specifically of short or long identifiers.
This benchmarking is motivated by profiling overall toolchain
performance and noticing that an unreasonable amount of time is spent in
the lexer. In turn, the identifier lexing was surprisingly hot. I have
performance improvements in the works following this, but wanted to
separately introduce the benchmarking framework as the review focus will
be completely different.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
I was initially looking at llvm::seq, but then looking through uses it
seemed like enumerate/zip would be best for these, allowing less direct
indexing and more range-based looping.
The main motivation for this is to get python loads in using the
`native-py` lint fix. However, enabling that made me wonder, maybe we
should fix in general?
`native-cc` is delayed, but not wholly cancelled (and `native-py`
picking up might indicate `native-cc` won't be too far behind). There's
also some automated fixes for `.append` and dict sorting -- this felt
okay to me, maybe not something to eagerly add but probably not worth
stopping buildifier from fixing (I've noticed the warnings in the past
and had been ignoring them).
Running everything does mean that load orders are sorted automatically
now, which I think is a positive. Most generally, I think these fixes
aren't _harmful_, and having them done automatically seems beneficial:
my biggest concern about `native-py` and `native-cc` was actually that
regressions wouldn't be caught, but this addresses that issue
automatically.
We already needed to manually flush diagnostics along every path out of
the driver, so switch to a CHECK-failure if we get this wrong rather
than a potential use-after-lifetime bug.
Switch the driver to flush explicitly rather than using a bunch of
cleanup lambdas, now that we have checking that we get this right.
This is a follow-up after #3126.
The semicolon is part of the statement's syntax, and including it will
hopefully help clarify that we're not using `var` as a placeholder and
talking about `return` _variablename_ `;`.
This removes BindName, putting name information directly on VarStorage.
As a side-effect of updating semantics_ir_test for this change, I also
noted that function bodies were being generated as invalid YAML so am
fixing that (just `{}` to `[]` bracketing, otherwise the test wouldn't
work anymore).
Because names are now available, I've updated lowering to use them for
vars.
In the SemIR formatter, the name is now repeated because it's a
parameter to VarStorage. I believe this is just default behavior, and
we'd have to special-case VarStorage to remove it because it's automatic
argument printing in action. On the balance, it felt like letting it
print was reasonable.
I've noted in places that the name on VarStorage is expected to be
optional, but am not adding support because I'd have no way of testing
it at present.
There were two issues contributing to this crash:
- Primarily, the issue is that we queue up diagnostics and don't format
them into a string until we reach the end of compilation. In some code
paths in the driver, we destroyed the Semantics IR object before this
happened. But diagnostics can contain references to Semantics IR
objects, such as strings stored in the string table, which can lead to a
use after destruction bug.
This is fixed by ensuring the diagnotics consumer is flushed before
destroying any of the objects that it can refer to. The current approach
to this is not especially clean, unfortunately, but this requires
fighting C++ as this isn't the order in which it wants to destroy
things.
- This issue was obscured by the Semantics IR's string table holding a
reference to whatever underlying storage it was given rather than its
own string storage, so sometimes it would hold a reference to a string
from the source file, and sometimes a string from the tokenized buffer's
string table. The diagnostics were always flushed before the source file
was destroyed, but not before the tokenized buffer was destroyed. So to
see the issue, you'd need to have a string literal with certain contents
followed by an identifier with a name that matched those contents.
The crash is made more reliable by holding references to the Semantics
IR's string map in its string table, rather than references to someone
else's strings. This also fixes a latent bug where passing a string
temporary to SemanticsIR::AddString would store a dangling reference in
the string table. Incidentally, AddString is also changed to perform
only one hash table lookup rather than two for each added string.
This matches what we do for all other dump output.
No test: this is just changing the behavior of a dump mode, and is
really awkward to exercise without adding back in something like a lit
test to observe the behavior when stdout and stderr go to the same
place.
Add a language server for carbon as part of GSoC.
This currently does code outline using toolchain parser.
See development steps in utils/vscode/README.md for running and using
language server.
In #3064, code was changed to look at a future token. This is an issue
because the parser is set up to enforce that tokens aren't used without
being consumed. That's part of #3118; related validation fails. Also,
since it's not necessarily the open paren that was consumed, it could be
a different opening symbol, which the closing symbol handling doesn't
check.
Under this approach, it's tracked whether an open paren was consumed,
and the open paren is associated with the state. That's more aligned
with how the parser expects to be fed information.
In paren condition handling for if and while, I'm also adding some
special casing for `if {` in particular to not assume the `{` is a
struct. I just think that this will come up somewhat often and the
resulting output is better this way (an error either way). I'm not doing
similar with `for` because there's already some `var` handling there,
and I'd need a little more time to think about structure -- whereas
right now I'm just trying to fix the crashes (`if {}`, `if []`, etc).
Fixes#3118
Adds some notes about the required build flags for lldb to successfully
find the symbols on macOS debug builds. Also adds a recommended debugger
configuration for interactive debugging in VSCode on macOS.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
This PR implements the lowering for array element access. Besides, this
creates a function for pointerTY to avoid code duplication.
---------
Co-authored-by: Farzana Ahmed Siddique <fasiddique@google.com>
This also tries to restructure the command line interface to the
toolchain a bit to make it start operating more like a compiler that
could be integrated into a build system rather than primarily as
a testing tool.
1) This switches form a `dump` subcommand to a `compile` subcommand
which has "dump" actions that can be enabled within it.
2) A distinct set of compile _phases_ that match the toolchain
structure:
- `lex` to run the lexer
- `parse` to run the parser
- `check` to fully check that the code is valid
- `lower` to lower to LLVM's IR
- `codegen` to generate executable code
3) The codegen phase has two output formats: textual assembly and
a binary object. These outputs can be configured, with a default for
an object when writing to a file and more firm default for textual
assembly when writing to stdout.
4) Select and expose the use of the LLVM host detection to compute
a default code generation target in the driver so that the command
line interface can reflect this. For example, the `help` output will
include the default target.
5) The `//toolchain/codegen` library APIs have been restructured a bit
to make the code flow a bit more naturally when implementing the new
command line structure. No real changes to the logic though.
There are also some minor tweaks to the command line interface based on
trying to use the shortest names for things that still seem likely to be
learnable for users:
- Switched `target-triple` to just `target`: the "triple" component to
this name is historical and can be confusing. For example, almost all
"triple" strings have more than three components today.
- Switched to just `--output` as now the fact that it is a file can be
configured in the documentation -- it will render as `--output=FILE`.
This also adds support for two custom output filename modes. First, when
no output is specified, we now compute one in the conventional way for
compilers by removing the file extension of the input file and replacing
it with `.o` for an object file output or `.s` for an assembly file
output. This matches the behavior of Clang and GCC for example.
Second, output to stdout is enabled with the special output file name of
`-` since it is no longer the default. This also follows the convention
of most compilers and many other command line tools to use `-` as a file
name to signify using standard in/out pipes.
There are still some rough edges here that I suspect could be improved,
but this seems like a good start of switching over to a complete
argument parser.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Lucile Rose Nihlen <luci.the.rose@gmail.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
The existing tuple index lowering was crashing while accessing the
element from return value. This PR fixes the bug.
Co-authored-by: Farzana Ahmed Siddique <fasiddique@google.com>
This library is designed around supporting the kinds of use cases we
expect in the toolchain and other Carbon tools. It supports subcommands,
options, and prints help.
There is still a decent chunk of work to be done to finish polishing
this, but it should give us a solid starting point.
For details about the library and a brief roadmap, see the main comment
in `command_line.h` which provides a comprehensive overview of the
library and a roadmap of the remaining work.
Porting the driver to this went fairly well, but did require some
changes. That port is separated into a follow-up PR #2979.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Lucile Rose Nihlen <luci.the.rose@gmail.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This reflects changes from a number of approved proposals:
- #2138 : "generic" -> "checked generic", "template" -> "template
generic"
- #2360 : "type", "facet type", "facet". Note: I am not using the term
"generic type" from #2360 since that meaning conflicts with the
generally accepted meaning of "generic type" of a type with a
compile-time parameter.
- #2760 / #2770 : internal/external impl -> extending impl
- #2964 : "symbolic constant" and "template constant"
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Prabhat SachdevaandJon Ross-Perkins
maan2003
Lucile Rose NihlenandChandler Carruth
Farzana Ahmed SiddiqueandFarzana Ahmed Siddique