Jon Ross-Perkins
f67791cfee
Separate subtree size information from parse nodes. ( #4174 )
...
Move subtree sizes over to TreeAndSubtrees, using the different
structure to represent the additional parse work that occurs, as well as
making it clear which functions require the extra information. My intent
is to make it hard to use this by accident.
The subtree size is still tracked during Parse::Tree construction. I
think a lot of that can be cleaned up, although we use it during
placeholder assignment so it may take some work. I wanted to see what
people thought about this before taking action on such a change.
I'm using a 1m line source file generated by #4124 for testing. Command
is `time bazel-bin/toolchain/install/prefix_root/bin/carbon compile
--phase=check --dump-mem-usage ~/tmp/data.carbon`
At head, what I'm seeing is:
```
...
parse_tree_.node_impls_:
used_bytes: 61516116
reserved_bytes: 61516116
...
Total:
used_bytes: 447814230
reserved_bytes: 551663894
...
1.43s user 0.14s system 99% cpu 1.565 total
```
With `Tree::Verify` disabled completely, it looks like:
```
parse_tree_.node_impls_:
used_bytes: 41010744
reserved_bytes: 41010744
...
Total:
used_bytes: 427308858
reserved_bytes: 531158522
...
1.20s user 0.13s system 99% cpu 1.332 total
```
Re-enabling just the basic verification (what is now `Tree::Verify`),
I'm seeing maybe 0.05s slower, but that's within noise for my system. I
do see variability in my timing results, and overall I think this is a
0.2s +/- 0.1s improvement versus the earlier (always testing `Extract`
code) implementation. That's opt; debug builds will be unaffected,
because the same checking occurs as before.
Note, the subtree size is a third of the node representation, which is
why I'm showing the decrease in memory usage here.
2024-07-31 19:39:45 +00:00
Richard Smith
202aae453f
Get the VS Code extension working again. ( #4037 )
...
VSCode's JSON parser no longer accepts comments in at least textmate
grammars, so remove the comments. It's unclear whether VSCode changed
here or whether this hasn't worked since the comments were added.
Clean up the textmate grammar: refactor, add some missing operators and
keywords, classify keywords as introducer / modifier as appropriate, fix
comment grammar to require space after `//`, and generalize recognition
of iN, fN, uN to recognize all such type literals.
Be more cautious in language server -- I was seeing frequent crashes for
`IdentifierName` parse nodes whose token was not actually an identifier,
presumably due to error recovery.
2024-06-06 23:27:33 +00:00
Chandler Carruth
dd0890619a
Enable a couple of boring warnings. ( #4018 )
...
Just spotted these while looking at warnings that seem to fire on our
code are probably are things we'd fix if we saw them. None of these seem
important FWIW.
Also removes a redundant flag that is part of `-Wall`.
I have a follow-up for the high-value warning I spotted that motivated
me to look at all of this. But it's noisy so kept it as a separate PR.
2024-06-03 05:02:26 +00:00
Jon Ross-Perkins
1974e44fd9
Rename factory functions from 'Create' to 'Make' ( #3706 )
...
Similar to #3705 , we actually have a mix of `Make` and `Create` in
factory functions too, so this PR is normalizing on `Make`. It's
intended to be consistent with the naming choice for Carbon factory
functions.
Note, MakeSyntheticBlock is the only one I feel a little weird about
because llvm's own APIs use Create, and this is essentially wrapping
LLVM calls. But the flipside is it also feels like a vague line to draw,
when we also differ from LLVM coding style in other ways.
2024-02-14 18:26:56 +00:00
Jon Ross-Perkins
785ab04ede
Address a likely issue with language_server ( #3610 )
...
The `node_kind` check is on `ch`, so the `node_token` probably should be
too. Trying to address the issue reported by [skoriop on
Discord](https://discord.com/channels/655572317891461132/993958660277555301/1197201736226574420 )
Note, I haven't tested this beyond building.
2024-01-17 19:52:29 +00:00
Jon Ross-Perkins
379d776084
Add support for '--config=clang-tidy' ( #3559 )
...
This sets things up to use `bazel` to run `clang-tidy` using
https://github.com/erenon/bazel_clang_tidy .
I'm fixing issues outside of explorer, and disabling clang-tidy for
targets in explorer that have legacy issues. I was going to disable
clang-tidy for targets in explorer such as interpreter anyways, because
they're slow to parse, and just extended that to the currently failing
targets.
2024-01-04 18:09:52 +00:00
Jon Ross-Perkins
c8b30d3eec
Split Parse out to its own target. ( #3556 )
...
This is mirroring the structure of codegen/codegen.h, lower/lower.h, and
check/check.h. I recently did lex/lex.h, so parse/parse.h is the last.
Now, the directory's main API file is eponymous with the directory.
I could've used a friend function to avoid making the Tree constructor
public, but in other places we make less use of `friend`, just leaving
things public. This felt more consistent, and simple because it only
affects the constructor.
2024-01-03 19:44:05 +00:00
Richard Smith
fe6f7b4330
Rename Name -> IdentifierName given that we have several other kinds of parse nodes that represent names. ( #3453 )
...
Factor out common checking handling for the different kinds of
unqualified names
2023-12-07 01:48:14 +00:00
Richard Smith
332a368cee
Rename Parse::Node -> Parse::NodeId. ( #3432 )
...
As discussed [on
discord](https://discord.com/channels/655572317891461132/655578254970716160/1178878128714678282 )
and today's toolchain discussion.
2023-11-29 18:53:12 +00:00
Jon Ross-Perkins and Chandler Carruth
cafcd88882
Split lexing logic and storage to separate files. ( #3365 )
...
Just reorganizing logic a little, trying to mirror the direction we've
gone with check, lower, etc. That is, lex.h contains a function `Lex`
that is used directly.
Note, I'm avoiding making meaningful changes here. It could in theory
still affect inlining in benchmarks, but I'm not seeing an impact.
Before:
```
------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
------------------------------------------------------------------------------------------------------
BM_ValidKeywords 2784949 ns 2784867 ns 249 bytes_per_second=214.452M/s tokens_per_second=35.9084M/s
BM_ValidKeywordsAsRawIdentifiers 3222597 ns 3222551 ns 210 bytes_per_second=244.513M/s tokens_per_second=31.0313M/s
BM_RawIdentifierFocus 5907836 ns 5907518 ns 103 bytes_per_second=264.873M/s tokens_per_second=16.9276M/s
BM_ValidIdentifiers<1, 64, false> 6255128 ns 6254297 ns 105 bytes_per_second=235.488M/s tokens_per_second=15.989M/s
BM_ValidIdentifiers<1, 1, true> 36776305518589chandlerc@gmail.com >
2023-11-10 23:59:49 +00:00
josh11b
5020fdb3be
Use abbreviation "decl" instead of "declaration" ( #3382 )
...
Part of switching to the [abbreviations we've decided to
use](https://docs.google.com/document/d/1RRYMm42osyqhI2LyjrjockYCutQ5dOf8Abu50kTrkX0/edit?resourcekey=0-kHyqOESbOHmzZphUbtLrTw#heading=h.pph7i5m5un7q ).
I will rename files in a follow-up PR.
2023-11-10 10:43:25 +00:00
Jon Ross-Perkins
3401eed8d8
Split IdentifierId and StringLiteralId from StringId ( #3352 )
...
Following up on discussion yesterday regarding this split.
Note, I'm expecting #3341 to do IdentifierId -> NameId in SemIR. It
might be worth adding NameId creation directly to StringStore if you're
content with this setup though.
2023-11-02 18:44:32 +00:00
Jon Ross-Perkins
1b55ad86dd
Extend SharedValueStores to SemIR ( #3313 )
...
Building on #3311 , change SemIR to use the SharedValueStore. Since this
removes hermeticity, raw output no longer prints ints, reals, and
strings. TokenizedBuffer accessors are modified to return IDs because
values are often passed through in semantics without needing to read
them.
I would've put SharedValueStores on Context, except for the
GetArrayBoundValue convenience method. I felt awkward removing that, so
it's on File, at least for now. That's then used by the formatter and
Lower too. The flipside of this is that TokenizedBuffer has a
SharedValueStores only for printing, so maybe that's similar enough to
what File is doing.
This doesn't start shifting other SemIR members to ValueStore, but that
seems like a next step.
2023-10-20 17:53:00 +00:00
Jon Ross-Perkins and Richard Smith
d13f76e001
Add value store to be shared across compile stages. ( #3311 )
...
This updates lexing to use the data. I'll do checking separately, just
to split changes.
Note the ValueStore structure is also set up such that SemIR::File can
use it for other fields.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk >
2023-10-20 15:00:53 +00:00
Jon Ross-Perkins and Richard Smith
bc63e6ae0a
Switch SourceBuffer to diagnostics. ( #3197 )
...
This updates SourceBuffer to diagnostics. Some additional edits to
diagnostics were necessary due to issues moving arguments around, which
seems to stem from a compile error with clang 14 (fixed in later
versions).
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk >
2023-09-07 23:02:10 +00:00
Jon Ross-Perkins
9ac92ad71b
Add support for compiling multiple files at once. ( #3182 )
...
Rearranges driver logic into CompilationUnits in order to associate
artifacts from the various stages of compilation.
Note, I'm not totally sure what the right thing to do is for
lower/codegen, so I'm just doing a rote change there for now that
mirrors prior phases (this is all the code supports anyways, so is
probably right for now regardless).
SourceBuffer error output is moved local for consistency with other
steps, and so that it's less ambiguous whether the error should be
expected to already include a filename.
2023-09-06 21:34:59 +00:00
Jon Ross-Perkins and Chandler Carruth
ec182fb00d
Rename lexer dir to lex ( #3179 )
...
Continuing with #3070 . Just a dir and file rename (only prefix change is
lexer_file_test). Everything in the lex dir should be marked as a move.
Note, I think this closes #3070 . There may still be further cleanup
later, but the organizational changes suggested there are being
completed.
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com >
2023-09-01 02:39:04 +00:00
Jon Ross-Perkins
c555b39a2c
Rename parser dir to parse ( #3178 )
...
Continuing with #3070 . Just a dir and file rename (mostly removing
prefixes, although for parse_tree_fuzzer and parse_tree_file_test I'm
dropping "tree" instead of "parse"). Everything in the parse dir should
be marked as a move.
2023-09-01 01:35:45 +00:00
Jon Ross-Perkins
3533850673
Moving lexer into a 'Lex' namespace. ( #3170 )
...
Continuing with #3070 , this creates a `Lex` namespace for lexer. This is
probably the last namespace addition right now, and will be followed by
file moves, although I'll also share a PR separately moving member
classes out of TokenizedBuffer.
2023-08-31 17:50:59 +00:00
Jon Ross-Perkins and Richard Smith
7157445f97
Set up a 'Parse' namespace. ( #3161 )
...
Continuing on #3070 .
I moved ParseTree::Node to just Parse::Node, versus Parse::Tree::Node.
Other name changes are just removing "Parse" or "Parser" prefixes.
In EnumBase, I'm directly defining operator<< because the ostream.h
approach just isn't working, not for either of Parse::State nor
Parse::NodeKind. Errors look like:
```toolchain/parser/parser_context.cpp:449:34: error: invalid operands to binary expression ('llvm::raw_ostream' and 'const Carbon::Parse::State')
output << "\t" << i << ".\t" << entry.state;
~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~
```
The expected template in `Carbon::` is not in the error list; I only see
the:
```
./common/ostream.h:112:6: note: candidate template ignored: requirement 'std::is_base_of_v<std::ostream, llvm::raw_ostream>' was not satisfied [with S = llvm::raw_ostream, T = Carbon::Parse::State]
auto operator<<(S& standard_out, const T& value) -> S& {
^
```
I'm still prodding at this, but not seeing an obvious fix.
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk >
2023-08-28 21:58:38 +00:00
maan2003
7c891fdacd
Language Server ( #3112 )
...
Add a language server for carbon as part of GSoC.
This currently does code outline using toolchain parser.
See development steps in utils/vscode/README.md for running and using
language server.
2023-08-21 18:39:31 +00:00
Jon Ross-Perkins
Richard Smith
Chandler Carruth
josh11b
maan2003