Files
carbon-lang/toolchain/lex/helpers.cpp
T
Jon Ross-Perkins c5761d2d16 Improve parsing large integers (#6908)
Though I started this thinking about performance of parse of large
decimal integers, I extended it to generally improve performance of
integer values (TBH I hadn't expected such a difference for binary/hex,
but I'll take it).

Note I think tests change because I'm making subtle changes to bit
widths. The changes themselves appear harmless to me, but happy to make
changes if it'd help.

Bumping up the number of digits by 10x because it's not really a
performance issue anymore (eh, maybe somebody will want to specify a
256-byte value in binary). But, at a certain point it still seems like a
mistake if somebody has that many digits in a row.

Fixes #980

Highlighting benchmark differences:
```diff
- BM_ComputeValue_IntDecimalN/1           37.1 ns         37.1 ns     18887116
+ BM_ComputeValue_IntDecimalN/1           21.9 ns         21.9 ns     31902433
- BM_ComputeValue_IntDecimalN/10000 1251228680 ns   1250457559 ns            1
+ BM_ComputeValue_IntDecimalN/10000     458818 ns       458626 ns         1523
- BM_ComputeValue_IntBinaryN/1            29.0 ns         29.0 ns     24058533
+ BM_ComputeValue_IntBinaryN/1            22.2 ns         22.1 ns     31566949
- BM_ComputeValue_IntBinaryN/10000     1390557 ns      1389782 ns          506
+ BM_ComputeValue_IntBinaryN/10000       16402 ns        16396 ns        42744
- BM_ComputeValue_IntHexN/1               34.0 ns         34.0 ns     20562432
+ BM_ComputeValue_IntHexN/1               22.4 ns         22.4 ns     31238055
- BM_ComputeValue_IntHexN/10000        5387942 ns      5385262 ns          130
+ BM_ComputeValue_IntHexN/10000          39249 ns        39233 ns        17859
```

Benchmark before:
```
----------------------------------------------------------------------------
Benchmark                                  Time             CPU   Iterations
----------------------------------------------------------------------------
BM_Lex_Float                            10.6 ns         10.6 ns     66138191
BM_Lex_Int                              15.5 ns         15.4 ns     45149703
BM_Lex_IntDecimalN/1                    3.11 ns         3.11 ns    225524908
BM_Lex_IntDecimalN/10                   11.8 ns         11.8 ns     56719805
BM_Lex_IntDecimalN/100                   102 ns          102 ns      6867468
BM_Lex_IntDecimalN/1000                  943 ns          942 ns       745313
BM_Lex_IntDecimalN/10000                9465 ns         9461 ns        73970
BM_ComputeValue_Float                   61.6 ns         61.6 ns     11377463
BM_ComputeValue_Int                      106 ns          106 ns      6587381
BM_ComputeValue_IntDecimalN/1           37.1 ns         37.1 ns     18887116
BM_ComputeValue_IntDecimalN/10          87.7 ns         87.7 ns      7960837
BM_ComputeValue_IntDecimalN/100         7963 ns         7956 ns        88858
BM_ComputeValue_IntDecimalN/1000     1212577 ns      1211906 ns          578
BM_ComputeValue_IntDecimalN/10000 1251228680 ns   1250457559 ns            1
BM_ComputeValue_IntBinaryN/1            29.0 ns         29.0 ns     24058533
BM_ComputeValue_IntBinaryN/10           69.4 ns         69.4 ns     10108642
BM_ComputeValue_IntBinaryN/100           963 ns          962 ns       726982
BM_ComputeValue_IntBinaryN/1000        21562 ns        21551 ns        32506
BM_ComputeValue_IntBinaryN/10000     1390557 ns      1389782 ns          506
BM_ComputeValue_IntHexN/1               34.0 ns         34.0 ns     20562432
BM_ComputeValue_IntHexN/10              70.4 ns         70.4 ns      9953165
BM_ComputeValue_IntHexN/100             1474 ns         1473 ns       472776
BM_ComputeValue_IntHexN/1000           61818 ns        61762 ns        11363
BM_ComputeValue_IntHexN/10000        5387942 ns      5385262 ns          130
```

Benchmark after:
```
----------------------------------------------------------------------------
Benchmark                                  Time             CPU   Iterations
----------------------------------------------------------------------------
BM_Lex_Float                            10.9 ns         10.9 ns     63993114
BM_Lex_Int                              15.1 ns         15.1 ns     46869766
BM_Lex_IntDecimalN/1                    3.16 ns         3.16 ns    220923300
BM_Lex_IntDecimalN/10                   12.2 ns         12.2 ns     57731654
BM_Lex_IntDecimalN/100                   102 ns          102 ns      6875516
BM_Lex_IntDecimalN/1000                  942 ns          942 ns       742359
BM_Lex_IntDecimalN/10000                9353 ns         9350 ns        75096
BM_ComputeValue_Float                   44.9 ns         44.9 ns     15619691
BM_ComputeValue_Int                     48.9 ns         48.9 ns     14361507
BM_ComputeValue_IntDecimalN/1           21.9 ns         21.9 ns     31902433
BM_ComputeValue_IntDecimalN/10          30.3 ns         30.3 ns     23134117
BM_ComputeValue_IntDecimalN/100          224 ns          223 ns      3092567
BM_ComputeValue_IntDecimalN/1000        5834 ns         5830 ns       117469
BM_ComputeValue_IntDecimalN/10000     458818 ns       458626 ns         1523
BM_ComputeValue_IntBinaryN/1            22.2 ns         22.1 ns     31566949
BM_ComputeValue_IntBinaryN/10           32.9 ns         32.9 ns     21306927
BM_ComputeValue_IntBinaryN/100           198 ns          198 ns      3545277
BM_ComputeValue_IntBinaryN/1000         1671 ns         1669 ns       419656
BM_ComputeValue_IntBinaryN/10000       16402 ns        16396 ns        42744
BM_ComputeValue_IntHexN/1               22.4 ns         22.4 ns     31238055
BM_ComputeValue_IntHexN/10              47.8 ns         47.7 ns     14694407
BM_ComputeValue_IntHexN/100              436 ns          436 ns      1609794
BM_ComputeValue_IntHexN/1000            3966 ns         3962 ns       177109
BM_ComputeValue_IntHexN/10000          39249 ns        39233 ns        17859
```

Assisted-by: Google Antigravity with Gemini
2026-03-16 20:00:51 +00:00

31 lines
1.0 KiB
C++

// Part of the Carbon Language project, under the Apache License v2.0 with LLVM
// Exceptions. See /LICENSE for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
#include "toolchain/lex/helpers.h"
namespace Carbon::Lex {
auto CanLexInt(Diagnostics::Emitter<const char*>& emitter, llvm::StringRef text)
-> bool {
// Integer parsing has poor scaling characteristics for extremely large digit
// amounts. We've done some performance work on this, but this limit exists to
// avoid really extreme cases.
//
// 2^128 would be 39 decimal digits or 128 binary. In either case, this limit
// is far above the threshold for normal ints.
constexpr size_t DigitLimit = 10000;
if (text.size() > DigitLimit) {
CARBON_DIAGNOSTIC(
TooManyDigits, Error,
"found a sequence of {0} digits, which is greater than the "
"limit of {1}",
size_t, size_t);
emitter.Emit(text.begin(), TooManyDigits, text.size(), DigitLimit);
return false;
}
return true;
}
} // namespace Carbon::Lex