Pursuant to discussion regarding #4699, turn on
`misc-non-private-member-variables-in-classes` using the
`IgnoreClassesWithAllMemberVariablesBeingPublic` flag (the check treats
structs as classes, so we need this for structs with all-public
members). Updates the style guide notes to match, which should be pretty
minor due to the scoping of test fixtures.
Also fixes some underscore uses in test files on the way. Basically this
is keeping the style for [class data member
naming](https://google.github.io/styleguide/cppguide.html#Variable_Names)
even while making them public.
Disables three new warnings because they lean more towards style
conflicts than fixes. I've brought these up on #style.
Other than that, mostly fixing basic issues, and things that
clang-tidy-20 seems to fire where clang-tiday-16 didn't. One particular
curious case is `llvm::StringLiteral::data()` uses, which are flagged as
not strictly null-terminated; I'm switching to `const char*` in those
spots which matches `llvm::formatv`'s format argument, but feels worse.
I'm removing `run_clang_tidy.py` here because I'm observing it give
fewer warnings than `bazel build --config=clang-tidy -k
//toolchain/...`. The latter matches how we enforce in GitHub actions
(and also caches results, and suppresses output for files that have no
issues), so I'm dropping the bespoke script.
Just a small packing optimization. We currently have 222 `NodeKinds`, so
this reduces us to just 30ish more we can add without needing to pack
more. However, if we did, there would be a couple options for bringing
the count down by reusing `NodeKinds` and disambiguating based on the
token kind (the 29 infix operators as an example). Or we could just undo
this.
I'm expecting this to yield a small improvement. I'll see if I can get
better numbers since my machine's not really reliable, but here are some
basic values.
Also suggesting to draw the use of `::RawEnumType` for `TokenKind`,
since bit packing appears to work without it. Hoping the `static_assert`
is easier for people to understand the size of the field.
With the change:
```
----------------------------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations Bytes Lines Tokens
----------------------------------------------------------------------------------------------------------------------------
BM_CompileAPIFileDenseDecls<Phase::Parse>/256 50399 ns 50359 ns 14336 104.588M/s 3.87217M/s 21.8629M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/1024 237823 ns 237629 ns 3072 136.721M/s 4.11986M/s 24.2058M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/4096 997645 ns 996771 ns 768 142.343M/s 4.04105M/s 23.9363M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/16384 4020308 ns 4018319 ns 192 152.041M/s 4.05966M/s 24.0874M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/65536 16691390 ns 16683058 ns 48 151.317M/s 3.92374M/s 23.2936M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/262144 75265735 ns 75233476 ns 8 135.842M/s 3.48421M/s 20.6862M/s
```
Without the change:
```
----------------------------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations Bytes Lines Tokens
----------------------------------------------------------------------------------------------------------------------------
BM_CompileAPIFileDenseDecls<Phase::Parse>/256 51515 ns 51480 ns 13312 102.312M/s 3.78789M/s 21.387M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/1024 241040 ns 240900 ns 3072 134.865M/s 4.06392M/s 23.8771M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/4096 985593 ns 984657 ns 768 144.094M/s 4.09077M/s 24.2308M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/16384 4109327 ns 4105496 ns 192 148.813M/s 3.97345M/s 23.576M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/65536 17459655 ns 17446006 ns 48 144.7M/s 3.75215M/s 22.275M/s
BM_CompileAPIFileDenseDecls<Phase::Parse>/262144 80802815 ns 80737489 ns 8 126.581M/s 3.24668M/s 19.276M/s
```
---------
Co-authored-by: Chandler Carruth <chandlerc@gmail.com>
- Provide `Check::Dump(context, arg)` and similar.
- gdb and lldb should do contextual lookup, and `call Dump(*this,
Lex::TokenIndex::Invalid)` has been tested with gdb.
- Since this is only for debug, keeps the functions fully separated from
code.
- Uses alwayslink to ensure objects are correctly linked, even though
there are no calls.
- `-Wno-missing-prototypes` is needed when we don't have forward
declarations.
- Code is not linked in opt builds, using `#ifndef NDEBUG`.
- This probably could be doing something in BUILD files with a
`select()`, but the `#ifndef` seemed easier.
This is based on #4620, but uses free functions instead of member
functions.
Co-authored-by: Dana Jansens <danakj@orodu.net>
---------
Co-authored-by: danakj <danakj@orodu.net>
This is essentially the result of looking at `.begin()` uses. We also
frequently do `std::shuffle`, but unfortunately STLExtras doesn't
provide a wrapper for that.
The offsets were originally added to deal with churn from builtins in
the raw semir. In textual semir, we mostly see instruction IDs for
imports, and builtins have also settled down more.
On imports, where possible, use the `EntityNameId` for an import instead
of printing an instruction. Next, show the source location if we have a
node. Only show the instruction if there's no location.
This also exposes `Parse::Tree` and `TokenizedBuffer`, so that we can
pass a `SemIR::File` without the component parts. In particular this
allows us to get the `TokenizedBuffer` for import IRs without
substantial structural modifications. We may want to make these optional
for serialized `SemIR` later, but the nodes/tokens contain source
location, which we'd need for debug information -- so it's not clear how
much we can really make them optional without substantial information
loss.
Reduce arguments to just `File` in a few spots, as a result of the
accompanying `TokenizedBuffer` and `Parse::Tree`. Also updates style to
pass around `const File*` where the reference is maintained, instead of
`const File&`.
I was considering keeping a direct reference to the tree and tokens on
`Context`, but initially my thought was it wouldn't make much
difference. I can re-add those if desired, just as direct caching of the
`File` fields.
This removes a lot of boilerplate `Print` functions in favor of a
CRTP-based approach that uses a `Label` field as an automatic prefix.
This `Label` is also made available for other purposes, particularly
`IdKind` crash messages in this change. In particular, for
`RequireIdKind` in node_stack.h from using numeric IdKinds (e.g., 5 and
24) to something that will print `IdKind(<label>)` (this came up
recently on #toolchain).
While I'm in here, also doing some other tinkering:
- Moving operators to be `friend` members, to reduce the extra
templating now that the base types are templated.
- Adjusts IntId diagnostics from `int [...]` to `int(...)` for
consistency with other id printing.
- Changes InstBlockId's label from "block" to "inst_block", since we
have multiple blocks now.
- Fixes StructTypeFieldsId to use "struct_type_fields" instead of
"type_block" (from `TypeBlockId`)
- Does some more adjustments from camelCase to snake_case for
consistency
I'm working to make sure remaining diagnostics have coverage, at least
the ones I'd previously added a TODO for. Note in particular that I
couldn't figure out a repro for UnaryOperatorRequiresWhitespace; if you
have one, I can add a test, but otherwise maybe it's actually
unreachable due to being diagnosed through infix logic (or, maybe
this'll let fuzzing tell me an example).
Previously Collect() was used for types that implemented
CollectMemUsage() but otherwise Add() was used. This required the caller
to think about the type of the field and know/decide which method to
use.
Now, the caller always uses Collect() unless they are adding specific
byte values, in which case Add is used. Typically then, Add will only be
used to implement the CollectMemUsage() function.
To do this we require all Collect() methods to be templates so that they
all be a single overload set. The Collect on BumpPtrAllocator is
converted to a template that checks
`std::same_as<llvm::BumpPtrAllocator, T>`.
Split StructComma into StructLiteralComma and StructTypeLiteralComma in
order to easily differentiate handling (remains the same in this PR).
Add "Literal" to StructField and StructTypeField because it feels
inconsistent versus the other non-shared things. StructFieldDesignator
remains shared between value literals and type literals.
Note I probably would've made StructFieldDesignator non-shared too, but
that'd require either a lookahead of 2 (to see the separator`) or a
writeback after parsing the separator, neither of which felt especially
crucial for this, when what I'm really trying to do is split type
literal handling a little further.
This is in anticipation of making the integer value store be customized
heavily. I'd like to extract it from the common code when doing that, so
first disentangling them here without any intended change in
functionality or behavior to enable that.
I've tried to update `#include`s to be as minimal as I can and added a
few missing includes spotted in the process.
I've split the test for value store to include what was easy focused on
just the value store templates rather than the unified shared value
stores.
This might surface some opportunities for adding more tests, but for
this PR, just doing the minimal restructuring.
An invalid parse due to an error token isn't likely a great diagnostic
as it will already have been diagnosed by the lexer. A common case to
start handling that is when the parser encounters an invalid token when
expecting an expression.
This removes a number of unhelpful diagnostics after the lexer has done
a good job diagnosing.
This also means that there may be parse tree errors that aren't
diagnosed when there are lexer-diagnosed errors, so track that.
Follow-up to #4430 that almost finishes addressing its diagnostic TODO.
AFAICT https://github.com/carbon-language/carbon-lang/pull/4363 made
builds of extract.cpp go from ~15s to ~35s. I'm not sure how to really
improve on this, short of adding boilerplate to the types in order to
reduce template use (e.g., instead of using struct reflection to return
fields, we could have something that directly returns fields). But, this
switch to `MaybeTrace` seems to be about a 20% build time improvement
(down to ~30s), with `noinline` accounting for a part of that.
This refactors the diagnostic kind coverage check into something that
also works for node kinds. Then, since this points out a few node kinds
that aren't having their parse verified, I'm adding minor tests for
those.
Also surround it in square brackets rather than parentheses. This
matches the format used by Clang and GCC, and means diagnostics will
still match the `file:line:col: error: ` pattern used by some IDE tools.
Before:
```console
fail_builtins.carbon:11:11: error(AliasRequiresNameRef): alias initializer must be a name reference
```
After:
```console
fail_builtins.carbon:11:11: error: alias initializer must be a name reference [AliasRequiresNameRef]
```
Also tighten up test regex to only match on `STDERR` lines that list a
file name.
Building on #4411, avoid using StringLiteral in format strings. This
includes a diagnostic check to prevent regressions (which is also how I
gathered issues).
Note, I haven't looked at `std::string` uses yet, but we might need
things like that to be able to pass strings in code back to the user.
StringLiteral though means that it's literally written down in the
toolchain, at which point it should probably be written in the format
string instead of separately.
---------
Co-authored-by: Geoff Romer <gromer@google.com>
This is to help identify which diagnostics we're actually using.
Note that driver/testdata still has tests which don't pass this flag,
and so continue to test the kind-less (default) behavior.
Note, this supports plurals, but doesn't apply it anywhere. I'm mainly
doing that to demonstrate the approach regarding syntax. See
format_providers.h for details.
Note in particular that this fixes an issue where SharedValueStore had
been shared across files, when they should be per-file. This is only
visible when doing multiple compilations in a single test, which was
rare before.
This also moves these tests into the Testing namespace. My memory of the
various namespacing changes is that we'd generally agreed to have tests
in Testing so that we'd see SemIR:: and similar, same as we would in a
lot of the implementation.
Omit the `note: ` prefix and the snippet from the "in import" note that
precedes a diagnostic.
This makes our diagnostic output more closely match that of Clang and
GCC.
This is a primarily automated change:
- Search & replace for capitalization
-
`(CARBON_DIAGNOSTIC\((?:\n\s+)?\w+,(?:\n\s+)?\s\w+,(?:\n\s+)?\s")([A-Z])`
- `$1\L$2`
- Search & replace for period
-
`(CARBON_DIAGNOSTIC\((?:\n\s+)?\w+,(?:\n\s+)?\s\w+,(?:\n\s+)?\s"(?:[^)]|\n)+)\.("[,)])`
- `$1$2`
- Limited search & replace for `ERROR: ` -> `error: ` in streamed things
- Leaving a TODO for command_line because there's more cleanup that can
be done there
- Modify diagnostic_consumer.cpp
- ERROR -> error
- WARNING -> warning
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
This switches `DCHECK` and `FATAL` as well.
The goal is to reduce the code size impact of these assertions so that
we can keep more of them enabled. Currently, the largest cost I see from
`CHECK` is not the actual check or the cold code itself, but actually
the failure to inline trivial functions due to the presence of the cold
code. This means that our goal isn't to reduce apparent code size in the
final binary but the LLVM IR cost assessed for these routines in the
inliner, which closely correlates with code size but is a bit different.
As discussed in #4283, experimentation shows that a single function call
with a minimal number of arguments is the lowest cost model for these.
This is easily achieved with a format-string API that internally uses
`llvm::formatv`. This PR is essentially the `CHECK` version of #4283.
However, the check macros are substantially harder to make work with
both format strings and streaming because they also take a condition.
Also, unexpectedly, I was very successful at devising a regular
expression based automated rewrite from the streaming to the format
string form with only low 10s of manual fixes. This includes compacting
strings broken up across lines, etc. Given how well that went, I've
prepared this PR which just directly switches to the format string API
and migrate everything to use it.
One nice side-effect is that the format string approach ends up greatly
simplifying the implementation here as well.
This is ... *shockingly* effective. Parsing speeds up by more than 3%
with just this change. And checking speeds up by **8%** with this change
alone:
```
BM_CompileAPIFileDenseDecls<Phase::Parse>/256 86.3µs ± 1% 82.9µs ± 1% -3.94% (p=0.000 n=17+19)
BM_CompileAPIFileDenseDecls<Phase::Parse>/1024 431µs ± 1% 415µs ± 1% -3.76% (p=0.000 n=18+19)
BM_CompileAPIFileDenseDecls<Phase::Parse>/4096 1.77ms ± 1% 1.71ms ± 1% -3.18% (p=0.000 n=18+19)
BM_CompileAPIFileDenseDecls<Phase::Parse>/16384 7.44ms ± 1% 7.17ms ± 2% -3.56% (p=0.000 n=18+20)
BM_CompileAPIFileDenseDecls<Phase::Parse>/65536 30.7ms ± 1% 29.7ms ± 1% -3.15% (p=0.000 n=18+20)
BM_CompileAPIFileDenseDecls<Phase::Parse>/262144 131ms ± 1% 127ms ± 1% -2.81% (p=0.000 n=18+18)
BM_CompileAPIFileDenseDecls<Phase::Check>/256 878µs ± 2% 800µs ± 1% -8.91% (p=0.000 n=19+20)
BM_CompileAPIFileDenseDecls<Phase::Check>/1024 1.88ms ± 2% 1.72ms ± 1% -8.56% (p=0.000 n=19+20)
BM_CompileAPIFileDenseDecls<Phase::Check>/4096 5.78ms ± 2% 5.28ms ± 1% -8.70% (p=0.000 n=20+18)
BM_CompileAPIFileDenseDecls<Phase::Check>/16384 21.9ms ± 1% 20.1ms ± 1% -8.02% (p=0.000 n=18+20)
BM_CompileAPIFileDenseDecls<Phase::Check>/65536 90.4ms ± 2% 83.1ms ± 1% -8.04% (p=0.000 n=19+20)
BM_CompileAPIFileDenseDecls<Phase::Check>/262144 381ms ± 2% 352ms ± 1% -7.79% (p=0.000 n=19+19)
```
---------
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Co-authored-by: josh11b <15258583+josh11b@users.noreply.github.com>
Includes support for the `impls`, `=`, and `==` requirement operators to
the right of a `where`, but `and` to allow multiple requirements is
still a TODO.
---------
Co-authored-by: Josh L <josh11b@users.noreply.github.com>
Co-authored-by: Richard Smith <richard@metafoo.co.uk>
Just a small factoring thing. The `if` logic is sizeable, and there's no
need for it to be in handle_expr (we do also have handle_brace_expr,
handle_index_expr, etc)
Applies #4278 TEST_NAME substitution to tests. Note I've tried to
structure commits as:
1. Do all the replacements.
2. autoupdate (nothing else) -- this shows incorrect updates.
3. Fix up manually, including autoupdates to get back to original
output.
These appear in `where` clauses, as in:
```
U:! InterfaceB where .C = Vector(.D)
V:! type where Vector(.Self) impls Sortable
```
`.Self` can additionally appear in type expressions in a binding pattern
such as `T:! InterfaceA(.Self)`.
---------
Co-authored-by: Josh L <josh11b@users.noreply.github.com>
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Most of these are about enabling inlining, in a couple of cases moving
code to a header and throughout switching to `CARBON_DCHECK`. The code
size of `CARBON_CHECK` seems to make inliing quite unreliable. I'm going
to think about whether there are ways to improve this, but a reasonably
small number of these seem worth switching for now to get some compile
time savings.
Also moves VLOG out of the hot path which helps a bit as well.
All combined, this net a bit over 10%, although it varies a bit exactly
how much. We're now pretty consistently over 800k lines/second for check
in the compilation benchmark for files >=4k lines, which makes me happy.
That's remarkably close to our original target.
Not really planning to keep optimizing here, just was glancing at the
profile and many of these stood out to me and were easy to fix.
Move subtree sizes over to TreeAndSubtrees, using the different
structure to represent the additional parse work that occurs, as well as
making it clear which functions require the extra information. My intent
is to make it hard to use this by accident.
The subtree size is still tracked during Parse::Tree construction. I
think a lot of that can be cleaned up, although we use it during
placeholder assignment so it may take some work. I wanted to see what
people thought about this before taking action on such a change.
I'm using a 1m line source file generated by #4124 for testing. Command
is `time bazel-bin/toolchain/install/prefix_root/bin/carbon compile
--phase=check --dump-mem-usage ~/tmp/data.carbon`
At head, what I'm seeing is:
```
...
parse_tree_.node_impls_:
used_bytes: 61516116
reserved_bytes: 61516116
...
Total:
used_bytes: 447814230
reserved_bytes: 551663894
...
1.43s user 0.14s system 99% cpu 1.565 total
```
With `Tree::Verify` disabled completely, it looks like:
```
parse_tree_.node_impls_:
used_bytes: 41010744
reserved_bytes: 41010744
...
Total:
used_bytes: 427308858
reserved_bytes: 531158522
...
1.20s user 0.13s system 99% cpu 1.332 total
```
Re-enabling just the basic verification (what is now `Tree::Verify`),
I'm seeing maybe 0.05s slower, but that's within noise for my system. I
do see variability in my timing results, and overall I think this is a
0.2s +/- 0.1s improvement versus the earlier (always testing `Extract`
code) implementation. That's opt; debug builds will be unaffected,
because the same checking occurs as before.
Note, the subtree size is a third of the node representation, which is
why I'm showing the decrease in memory usage here.
Allow directly constructing a PostorderIterator, to get rid of
`tree.postorder(node_id).end()` indirect construction. For ranges that
don't need tree data, make it clearer that they're not validated.
Note, this subtly gets rid of a subtree size use in the
`tree.postorder(node_id).end()` case (to get the discarded `begin()`
value).
Richard Smith
Jon Ross-Perkins
Dana Jansens
josh11bandJosh L
David Blaikie
Brymer Meneses