Files
carbon-lang/toolchain/lex/testdata/fail_block_string_second_line.carbon
T
c43fa3a8a5 Bit-pack the lexer's token info (#4270)
This makes each token info consist of 8 bytes of data:
- 1 byte of the kind
- 1 bit for whitespace tracking
- 23 bits of payload
- 32 bits for byte offset in the file

This builds directly on representing the location of the token as
a single 32-bit offset, now compressing the rest of the data into
a single 32-bit bitfield.

This adds some implementation limits: we can no longer lex more than
2^23 tokens in a single source file. Nor can we have more than 2^23
string literals, integer literals, real literals, or identifiers. Only
the first of these is even close to an issue, and even then seems
unlikely to ever be a problem in practice.

The memory efficiency here is great and the motivating goal. But to make
this work well, we also need to streamline how we create the tokens.
Otherwise, all the bit fiddling can end up erasing our gains. This PR
adds a number of APIs to manage creating and accessing the now
significantly more complex storage of token infos to try and help with
this.

One big change required to simplify the writes here is to switch from
computing whether a token has trailing space after-the-fact to
pre-computing whether a token will have leading space. That lets us have
the leading space information available immediately when forming the
token, and avoids doing a single bit flip afterward.

Another change that helps with this representation is to minimize the
updating of groups after-the-fact. The code now tries to set the opening
index directly when creating the closing token and only updates the
opening group afterward. Because of the bit packing, this is a reduction
of 0.5% of dynamic instructions in the compile benchmark, and has
dramatic improvements for the grouping symbol focused benchmarks.

All combined, this is a significant improvement on the lexer-focused
benchmarks despite the added complexity, and a significant win on our
compile time benchmarks due to both the lexer improvements and
downstream memory density improvements: 5-12% reduction in lex time,
growing larger as files get larger. About a 4.5% reduction in parse
time, and even a 1-2% reduction in total check time. =D

---------

Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
Co-authored-by: Geoff Romer <gromer@google.com>
2024-09-06 16:22:36 +00:00

32 lines
2.2 KiB
Plaintext

// Part of the Carbon Language project, under the Apache License v2.0 with LLVM
// Exceptions. See /LICENSE for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
var s: String = '''
error here: '''
// Put all autoupdates below because otherwise is puts a CHECK in the middle of
// the block string.
//
// AUTOUPDATE
// TIP: To test this file alone, run:
// TIP: bazel test //toolchain/testing:file_test --test_arg=--file_tests=toolchain/lex/testdata/fail_block_string_second_line.carbon
// TIP: To dump output, run:
// TIP: bazel run //toolchain/testing:file_test -- --dump_output --file_tests=toolchain/lex/testdata/fail_block_string_second_line.carbon
// CHECK:STDERR: fail_block_string_second_line.carbon:[[@LINE-10]]:3: ERROR: Only whitespace is permitted before the closing `'''` of a multi-line string.
// CHECK:STDERR: error here: '''
// CHECK:STDERR: ^
// CHECK:STDOUT: - filename: fail_block_string_second_line.carbon
// CHECK:STDOUT: tokens: [
// CHECK:STDOUT: { index: 0, kind: 'FileStart', line: {{ *\d+}}, column: 1, indent: 1, spelling: '' },
// CHECK:STDOUT: { index: 1, kind: 'Var', line: {{ *}}[[@LINE-17]], column: 1, indent: 1, spelling: 'var', has_leading_space: true },
// CHECK:STDOUT: { index: 2, kind: 'Identifier', line: {{ *}}[[@LINE-18]], column: 5, indent: 1, spelling: 's', identifier: 0, has_leading_space: true },
// CHECK:STDOUT: { index: 3, kind: 'Colon', line: {{ *}}[[@LINE-19]], column: 6, indent: 1, spelling: ':' },
// CHECK:STDOUT: { index: 4, kind: 'StringTypeLiteral', line: {{ *}}[[@LINE-20]], column: 8, indent: 1, spelling: 'String', has_leading_space: true },
// CHECK:STDOUT: { index: 5, kind: 'Equal', line: {{ *}}[[@LINE-21]], column: 15, indent: 1, spelling: '=', has_leading_space: true },
// CHECK:STDOUT: { index: 6, kind: 'StringLiteral', line: {{ *}}[[@LINE-22]], column: 17, indent: 1, spelling: ''''
// CHECK:STDOUT: error here: '''', value: `error here: `, has_leading_space: true },
// CHECK:STDOUT: { index: 7, kind: 'FileEnd', line: {{ *}}[[@LINE+1]], column: {{ *\d+}}, indent: 1, spelling: '', has_leading_space: true },
// CHECK:STDOUT: ]