This adds a benchmark that tries to synthesize mixtures of symbols,
keywords, and identifiers. It tweaks the distribution of identifier
lengths based on some empirical measurements of, for example LLVM's
codebase.
Also establish a framework for skewing the symbol distribution, although
that one is based entirely on intuition and not measurements. It should
be adjusted as we have measurements.
The ratios between symbols, keywords, and identifiers is also
unmeasured, but several different ratios are covered.
Neither literals nor grouping symbols are included yet, as both present
some additional challenges in forming them, and this seemed like
a plausible increment in expanding the benchmark coverage.
Currently, for long identifiers, a huge (>30%) fraction of time is spent
finding the end of the identifier. We can speed this up with a fun
application of SIMD and in-register lookup tables.
With this, the BM_ValidIdentifiers/12/64 benchmark goes from around 4
million
tokens/second to around 6 mt/s, so roughly 1.5x improvement. However,
there was a decent amount of noise in the measurement and I didn't study
it too closely as I was very happy with the overall result. The profile
shifted from >30% of the time in this loop to <10% of the time, so the
scan itself is 3x or more faster with this.
One concern with optimizing the lexer right now is that we don't have
full Unicode support from the design. This PR takes some steps to at
least
try and avoid this pitfall -- the new routine works to classify UTF-8
code units, and has a fallback in that case that can grow the needed
logic.
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Currently this is focused on benchmarking the identifier and token
lexing paths, but should expand in the future to cover other parts of
the lexer.
In order to effectively benchmark tokens, this adds support to the
`token_kind` library to produce a list of all the tokens in Carbon that
the benchmark can use to create random inputs.
For identifiers, the benchmark has support for benchmarking different
distributions of identifier sizes so it is easy to zoom into the
performance specifically of short or long identifiers.
This benchmarking is motivated by profiling overall toolchain
performance and noticing that an unreasonable amount of time is spent in
the lexer. In turn, the identifier lexing was surprisingly hot. I have
performance improvements in the works following this, but wanted to
separately introduce the benchmarking framework as the review focus will
be completely different.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
I was initially looking at llvm::seq, but then looking through uses it
seemed like enumerate/zip would be best for these, allowing less direct
indexing and more range-based looping.
The main motivation for this is to get python loads in using the
`native-py` lint fix. However, enabling that made me wonder, maybe we
should fix in general?
`native-cc` is delayed, but not wholly cancelled (and `native-py`
picking up might indicate `native-cc` won't be too far behind). There's
also some automated fixes for `.append` and dict sorting -- this felt
okay to me, maybe not something to eagerly add but probably not worth
stopping buildifier from fixing (I've noticed the warnings in the past
and had been ignoring them).
Running everything does mean that load orders are sorted automatically
now, which I think is a positive. Most generally, I think these fixes
aren't _harmful_, and having them done automatically seems beneficial:
my biggest concern about `native-py` and `native-cc` was actually that
regressions wouldn't be caught, but this addresses that issue
automatically.
We already needed to manually flush diagnostics along every path out of
the driver, so switch to a CHECK-failure if we get this wrong rather
than a potential use-after-lifetime bug.
Switch the driver to flush explicitly rather than using a bunch of
cleanup lambdas, now that we have checking that we get this right.
This is a follow-up after #3126.
The semicolon is part of the statement's syntax, and including it will
hopefully help clarify that we're not using `var` as a placeholder and
talking about `return` _variablename_ `;`.
This removes BindName, putting name information directly on VarStorage.
As a side-effect of updating semantics_ir_test for this change, I also
noted that function bodies were being generated as invalid YAML so am
fixing that (just `{}` to `[]` bracketing, otherwise the test wouldn't
work anymore).
Because names are now available, I've updated lowering to use them for
vars.
In the SemIR formatter, the name is now repeated because it's a
parameter to VarStorage. I believe this is just default behavior, and
we'd have to special-case VarStorage to remove it because it's automatic
argument printing in action. On the balance, it felt like letting it
print was reasonable.
I've noted in places that the name on VarStorage is expected to be
optional, but am not adding support because I'd have no way of testing
it at present.
There were two issues contributing to this crash:
- Primarily, the issue is that we queue up diagnostics and don't format
them into a string until we reach the end of compilation. In some code
paths in the driver, we destroyed the Semantics IR object before this
happened. But diagnostics can contain references to Semantics IR
objects, such as strings stored in the string table, which can lead to a
use after destruction bug.
This is fixed by ensuring the diagnotics consumer is flushed before
destroying any of the objects that it can refer to. The current approach
to this is not especially clean, unfortunately, but this requires
fighting C++ as this isn't the order in which it wants to destroy
things.
- This issue was obscured by the Semantics IR's string table holding a
reference to whatever underlying storage it was given rather than its
own string storage, so sometimes it would hold a reference to a string
from the source file, and sometimes a string from the tokenized buffer's
string table. The diagnostics were always flushed before the source file
was destroyed, but not before the tokenized buffer was destroyed. So to
see the issue, you'd need to have a string literal with certain contents
followed by an identifier with a name that matched those contents.
The crash is made more reliable by holding references to the Semantics
IR's string map in its string table, rather than references to someone
else's strings. This also fixes a latent bug where passing a string
temporary to SemanticsIR::AddString would store a dangling reference in
the string table. Incidentally, AddString is also changed to perform
only one hash table lookup rather than two for each added string.
This matches what we do for all other dump output.
No test: this is just changing the behavior of a dump mode, and is
really awkward to exercise without adding back in something like a lit
test to observe the behavior when stdout and stderr go to the same
place.
Add a language server for carbon as part of GSoC.
This currently does code outline using toolchain parser.
See development steps in utils/vscode/README.md for running and using
language server.
In #3064, code was changed to look at a future token. This is an issue
because the parser is set up to enforce that tokens aren't used without
being consumed. That's part of #3118; related validation fails. Also,
since it's not necessarily the open paren that was consumed, it could be
a different opening symbol, which the closing symbol handling doesn't
check.
Under this approach, it's tracked whether an open paren was consumed,
and the open paren is associated with the state. That's more aligned
with how the parser expects to be fed information.
In paren condition handling for if and while, I'm also adding some
special casing for `if {` in particular to not assume the `{` is a
struct. I just think that this will come up somewhat often and the
resulting output is better this way (an error either way). I'm not doing
similar with `for` because there's already some `var` handling there,
and I'd need a little more time to think about structure -- whereas
right now I'm just trying to fix the crashes (`if {}`, `if []`, etc).
Fixes#3118
Adds some notes about the required build flags for lldb to successfully
find the symbols on macOS debug builds. Also adds a recommended debugger
configuration for interactive debugging in VSCode on macOS.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
This PR implements the lowering for array element access. Besides, this
creates a function for pointerTY to avoid code duplication.
---------
Co-authored-by: Farzana Ahmed Siddique <fasiddique@google.com>
This also tries to restructure the command line interface to the
toolchain a bit to make it start operating more like a compiler that
could be integrated into a build system rather than primarily as
a testing tool.
1) This switches form a `dump` subcommand to a `compile` subcommand
which has "dump" actions that can be enabled within it.
2) A distinct set of compile _phases_ that match the toolchain
structure:
- `lex` to run the lexer
- `parse` to run the parser
- `check` to fully check that the code is valid
- `lower` to lower to LLVM's IR
- `codegen` to generate executable code
3) The codegen phase has two output formats: textual assembly and
a binary object. These outputs can be configured, with a default for
an object when writing to a file and more firm default for textual
assembly when writing to stdout.
4) Select and expose the use of the LLVM host detection to compute
a default code generation target in the driver so that the command
line interface can reflect this. For example, the `help` output will
include the default target.
5) The `//toolchain/codegen` library APIs have been restructured a bit
to make the code flow a bit more naturally when implementing the new
command line structure. No real changes to the logic though.
There are also some minor tweaks to the command line interface based on
trying to use the shortest names for things that still seem likely to be
learnable for users:
- Switched `target-triple` to just `target`: the "triple" component to
this name is historical and can be confusing. For example, almost all
"triple" strings have more than three components today.
- Switched to just `--output` as now the fact that it is a file can be
configured in the documentation -- it will render as `--output=FILE`.
This also adds support for two custom output filename modes. First, when
no output is specified, we now compute one in the conventional way for
compilers by removing the file extension of the input file and replacing
it with `.o` for an object file output or `.s` for an assembly file
output. This matches the behavior of Clang and GCC for example.
Second, output to stdout is enabled with the special output file name of
`-` since it is no longer the default. This also follows the convention
of most compilers and many other command line tools to use `-` as a file
name to signify using standard in/out pipes.
There are still some rough edges here that I suspect could be improved,
but this seems like a good start of switching over to a complete
argument parser.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Lucile Rose Nihlen <luci.the.rose@gmail.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
The existing tuple index lowering was crashing while accessing the
element from return value. This PR fixes the bug.
Co-authored-by: Farzana Ahmed Siddique <fasiddique@google.com>
This library is designed around supporting the kinds of use cases we
expect in the toolchain and other Carbon tools. It supports subcommands,
options, and prints help.
There is still a decent chunk of work to be done to finish polishing
this, but it should give us a solid starting point.
For details about the library and a brief roadmap, see the main comment
in `command_line.h` which provides a comprehensive overview of the
library and a roadmap of the remaining work.
Porting the driver to this went fairly well, but did require some
changes. That port is separated into a follow-up PR #2979.
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Lucile Rose Nihlen <luci.the.rose@gmail.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This reflects changes from a number of approved proposals:
- #2138 : "generic" -> "checked generic", "template" -> "template
generic"
- #2360 : "type", "facet type", "facet". Note: I am not using the term
"generic type" from #2360 since that meaning conflicts with the
generally accepted meaning of "generic type" of a type with a
compile-time parameter.
- #2760 / #2770 : internal/external impl -> extending impl
- #2964 : "symbolic constant" and "template constant"
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Just some small refactorings stemming from #3073.
AddCheckLines -> BuildCheckLines because the two lists are now fully
separate. Adding is_blank to be more direct about behavior than the
Print call.
Where there was an object initialization immediately followed by
assignment I condensed it to one line, which seems to be the convention
looking at other files. I also deleted the repetition of 'of' in some
c++ comments because it was grammatically incorrect.
---------
Co-authored-by: Clayton Gearhart <claytongearhart240@gmail.com>
Makes the trace output for pattern match more consistent with the rest
of the trace by using `Match()` and `Indent()` prefix methods and
wrapping the code in the trace with backticks.
Unlike in the toolchain, we have not been preferring LLVM facilities
over standard ones. Update the documentation to describe this and
provide some rationale.
When producing textual semantics IR, map each branch instruction back to
the construct that produced it and use that to determine a name for the
corresponding block label.
Instead of manually putting symbols before and after heading, this
introduces two methods `Heading` and `SubHeading` inside `TraceStream`.
Both methods take `llvm::StringRef` as parameter, formats the given text
as follows and adds it into the output stream.
**Heading**
```
* * * * * * * * * * heading * * * * * * * * * *
-------------------------------------------------
```
**Sub heading**
```
- - - - - sub heading - - - - -
---------------------------------
```
Note: both methods assert that tracing should be enabled.
Also indicate what can appear within parentheses.
This is intended to be a clarification, not a design change. Note that
while we previously described the operand of `++` or `--` as being
simply an expression, the operand can never be anything other than the
kinds of expression the diagram now shows due to the expression category
rules added in #2006.
Fixes#3079.
Don't elide the StructType node if we've already created an equivalent
type. Its spelling and location may be interesting to diagnostics,
tooling, debug information, etc.
Add a textual IR format to the toolchain.
The exact details of the format are somewhat arbitrary right now, and I
expect them to change as we refine the semantics IR model, but at the
moment they're somewhat directly following the current structure of the
IR.
Semantics tests currently test both the "raw" format, which shows the
details of the representation, and the textual format, which is somewhat
higher level. We may want to revisit that decision once the textual
format is a bit more stable, and test only the textual format in most of
these tests, but for now it seems prudent to keep both sets of tests.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
- Only allow assignment at the top level in an expression statement.
- Allow both negation and complement as subexpressions of both bitwise
and numeric operators.
- Remove parsing support for postincrement and postdecrement.
- Add parsing support for `as` operator.
- Use the same ambient precedence for types and non-type expressions.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
Defines methods into `TraceStream` for adding line prefixes. Instead of
directly using string literals.
Example usage,
```
trace_stream_->Start() << "declaring ... " << ... ;
```
will result in,
```
->> declaring ...
```
Richard Smith
Chandler Carruth
maan2003
Jon Ross-Perkins
Lucile Rose NihlenandChandler Carruth
Farzana Ahmed SiddiqueandFarzana Ahmed Siddique
Prabhat Sachdeva
josh11bandRichard Smith
Clayton GearhartandClayton Gearhart