carbon-language/carbon-lang
1
0
Fork 0
mirror of https://github.com/carbon-language/carbon-lang.git synced 2026-10-05 13:11:05 +01:00
Code Issues Packages Projects Releases 10 Wiki Activity

Use new test framework in explorer to reduce per-test overhead (#2811)

Browse Source 
Isolating test execution time (no build time included):

- Linux, `lit` test-per-file: Elapsed time: 28.214s, Critical Path: 13.97s
- Linux, `cc_test`-per-file: Elapsed time: 11.534s, Critical Path: 6.05s
- Linux, merged `cc_test` with 50 shards: Elapsed time: 11.677s, Critical Path: 11.17s
- Mac, `lit` test-per-file: Elapsed time: 295.686s, Critical Path: 20.00s
- Mac, `cc_test`-per-file: Elapsed time: 55.788s, Critical Path: 3.81s
- Mac, merged `cc_test` with 50 shards: Elapsed time: 16.269s, Critical Path: 7.54s

In GH actions:

- [Before](https://github.com/carbon-language/carbon-lang/actions/runs/4866602695/jobs/8678306144?pr=2799):
  - test / test (ubuntu-22.04, fastbuild) (pull_request_target) Successful in 20m
  - test / test (ubuntu-22.04, opt) (pull_request_target) Successful in 15m
  - test / test (macos-12, fastbuild) (pull_request_target) Successful in 36m
  - test / test (macos-12, opt) (pull_request_target) Successful in 21m
- [After](https://github.com/carbon-language/carbon-lang/actions/runs/4875154751/jobs/8697004066?pr=2811):
  - test / test (ubuntu-22.04, fastbuild) (pull_request_target) Successful in 10m
  - test / test (ubuntu-22.04, opt) (pull_request_target) Successful in 9m
  - test / test (macos-12, fastbuild) (pull_request_target) Successful in 12m
  - test / test (macos-12, opt) (pull_request_target) Successful in 9m

I'm still leaving a handful of `lit` tests to test end-to-end binary execution. This is why testdata directories are split (`lit` tests next to the `explorer` binary, the `cc_test`s next to `ParseAndExecute`).
This commit is contained in:
Jon Ross-Perkins
2023-05-15 16:23:59 -07:00
committed by GitHub
parent ee9f883ef3
commit 82f33ff384
763 changed files with 556 additions and 1899 deletions
Download Patch File Download Diff File Expand all files Collapse all files
explorer/testdata/string/basic.carbon
Vendored
-2
View File
@@ -3,8 +3,6 @@
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
// AUTOUPDATE
// RUN: %{explorer-run}
// RUN: %{explorer-run-trace}
// CHECK:STDOUT: result: 0
package ExplorerTest api;
explorer/testdata/string/block.carbon
Vendored
-2
View File
@@ -3,8 +3,6 @@
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
// AUTOUPDATE
// RUN: %{explorer-run}
// RUN: %{explorer-run-trace}
// CHECK:STDOUT: result: 0
package ExplorerTest api;
explorer/testdata/string/block_escaped_triple_quotes.carbon
Vendored
-2
View File
@@ -3,8 +3,6 @@
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
// AUTOUPDATE
// RUN: %{explorer-run}
// RUN: %{explorer-run-trace}
// CHECK:STDOUT: result: 0
package ExplorerTest api;
explorer/testdata/string/block_file_type_indicator.carbon
Vendored
-2
View File
@@ -3,8 +3,6 @@
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
// AUTOUPDATE
// RUN: %{explorer-run}
// RUN: %{explorer-run-trace}
// CHECK:STDOUT: result: 0
package ExplorerTest api;
explorer/testdata/string/fail_block_quotes_not_on_own_line.carbon
Vendored
+1 -3
View File
@@ -3,13 +3,11 @@
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
// AUTOUPDATE
// RUN: %{not} %{explorer-run}
// RUN: %{not} %{explorer-run-trace}
package ExplorerTest api;
fn Main() -> i32 {
// CHECK:STDERR: SYNTAX ERROR: {{.*}}/explorer/testdata/string/fail_block_quotes_not_on_own_line.carbon:[[@LINE+1]]: Invalid block string: Should end with triple quotes: error: closing '''
// CHECK:STDERR: SYNTAX ERROR: explorer/testdata/string/fail_block_quotes_not_on_own_line.carbon:[[@LINE+1]]: Invalid block string: Should end with triple quotes: error: closing '''
var s: String = '''
error: closing ''' is not on its own line.
''';
explorer/testdata/string/fail_hex_lower.carbon
Vendored
+1 -3
View File
@@ -3,13 +3,11 @@
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
// AUTOUPDATE
// RUN: %{not} %{explorer-run}
// RUN: %{not} %{explorer-run-trace}
package ExplorerTest api;
fn Main() -> i32 {
// CHECK:STDERR: SYNTAX ERROR: {{.*}}/explorer/testdata/string/fail_hex_lower.carbon:[[@LINE+1]]: Invalid escaping in string: "str\xaa"
// CHECK:STDERR: SYNTAX ERROR: explorer/testdata/string/fail_hex_lower.carbon:[[@LINE+1]]: Invalid escaping in string: "str\xaa"
Print("str\xaa");
return 0;
}
explorer/testdata/string/fail_hex_truncated.carbon
Vendored
+1 -3
View File
@@ -3,13 +3,11 @@
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
// AUTOUPDATE
// RUN: %{not} %{explorer-run}
// RUN: %{not} %{explorer-run-trace}
package ExplorerTest api;
fn Main() -> i32 {
// CHECK:STDERR: SYNTAX ERROR: {{.*}}/explorer/testdata/string/fail_hex_truncated.carbon:[[@LINE+1]]: Invalid escaping in string: "str\x"
// CHECK:STDERR: SYNTAX ERROR: explorer/testdata/string/fail_hex_truncated.carbon:[[@LINE+1]]: Invalid escaping in string: "str\x"
Print("str\x");
return 0;
}
explorer/testdata/string/fail_invalid_escape.carbon
Vendored
+1 -3
View File
@@ -3,13 +3,11 @@
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
// AUTOUPDATE
// RUN: %{not} %{explorer-run}
// RUN: %{not} %{explorer-run-trace}
package ExplorerTest api;
fn Main() -> i32 {
// CHECK:STDERR: SYNTAX ERROR: {{.*}}/explorer/testdata/string/fail_invalid_escape.carbon:[[@LINE+1]]: Invalid escaping in string: "str\e"
// CHECK:STDERR: SYNTAX ERROR: explorer/testdata/string/fail_invalid_escape.carbon:[[@LINE+1]]: Invalid escaping in string: "str\e"
Print("str\e");
return 0;
}
explorer/testdata/string/fail_newline.carbon
Vendored
+1 -3
View File
@@ -3,13 +3,11 @@
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
// AUTOUPDATE
// RUN: %{not} %{explorer-run}
// RUN: %{not} %{explorer-run-trace}
package ExplorerTest api;
fn Main() -> i32 {
// CHECK:STDERR: SYNTAX ERROR: {{.*}}/explorer/testdata/string/fail_newline.carbon:[[@LINE+2]]: missing closing quote in single-line string: "new
// CHECK:STDERR: SYNTAX ERROR: explorer/testdata/string/fail_newline.carbon:[[@LINE+2]]: missing closing quote in single-line string: "new
// CHECK:STDERR:
Print("new
line");
explorer/testdata/string/fail_octal.carbon
Vendored
+1 -3
View File
@@ -3,13 +3,11 @@
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
// AUTOUPDATE
// RUN: %{not} %{explorer-run}
// RUN: %{not} %{explorer-run-trace}
package ExplorerTest api;
fn Main() -> i32 {
// CHECK:STDERR: SYNTAX ERROR: {{.*}}/explorer/testdata/string/fail_octal.carbon:[[@LINE+1]]: Invalid escaping in string: "str\01"
// CHECK:STDERR: SYNTAX ERROR: explorer/testdata/string/fail_octal.carbon:[[@LINE+1]]: Invalid escaping in string: "str\01"
Print("str\01");
return 0;
}
explorer/testdata/string/fail_raw_block_more_hash_tags_on_left.carbon
Vendored
+1 -3
View File
@@ -3,13 +3,11 @@
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
// AUTOUPDATE
// RUN: %{not} %{explorer-run}
// RUN: %{not} %{explorer-run-trace}
package ExplorerTest api;
fn Main() -> i32 {
// CHECK:STDERR: SYNTAX ERROR: {{.*}}/explorer/testdata/string/fail_raw_block_more_hash_tags_on_left.carbon:[[@LINE+1]]: Unexpected end of file
// CHECK:STDERR: SYNTAX ERROR: explorer/testdata/string/fail_raw_block_more_hash_tags_on_left.carbon:[[@LINE+1]]: Unexpected end of file
var s: String = ##'''
error: there are more #s on the left than the right.
'''#;
explorer/testdata/string/fail_raw_block_more_hash_tags_on_right.carbon
Vendored
+1 -3
View File
@@ -3,13 +3,11 @@
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
// AUTOUPDATE
// RUN: %{not} %{explorer-run}
// RUN: %{not} %{explorer-run-trace}
package ExplorerTest api;
fn Main() -> i32 {
// CHECK:STDERR: SYNTAX ERROR: {{.*}}/explorer/testdata/string/fail_raw_block_more_hash_tags_on_right.carbon:[[@LINE+1]]: invalid character '\x23' in source file.
// CHECK:STDERR: SYNTAX ERROR: explorer/testdata/string/fail_raw_block_more_hash_tags_on_right.carbon:[[@LINE+1]]: invalid character '\x23' in source file.
var s: String = #'''
error: there are more #s on the right than the left.
'''##;
explorer/testdata/string/fail_raw_block_quotes_not_on_own_line.carbon
Vendored
+1 -3
View File
@@ -3,13 +3,11 @@
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
// AUTOUPDATE
// RUN: %{not} %{explorer-run}
// RUN: %{not} %{explorer-run-trace}
package ExplorerTest api;
fn Main() -> i32 {
// CHECK:STDERR: SYNTAX ERROR: {{.*}}/explorer/testdata/string/fail_raw_block_quotes_not_on_own_line.carbon:[[@LINE+1]]: Invalid block string: Should end with triple quotes: error: closing '''
// CHECK:STDERR: SYNTAX ERROR: explorer/testdata/string/fail_raw_block_quotes_not_on_own_line.carbon:[[@LINE+1]]: Invalid block string: Should end with triple quotes: error: closing '''
var s: String = #'''
error: closing '''# is not on its own line.
'''#;
explorer/testdata/string/fail_raw_block_single_line.carbon
Vendored
+1 -3
View File
@@ -3,13 +3,11 @@
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
// AUTOUPDATE
// RUN: %{not} %{explorer-run}
// RUN: %{not} %{explorer-run-trace}
package ExplorerTest api;
fn CompareStr(s: String) -> i32 {
// CHECK:STDERR: SYNTAX ERROR: {{.*}}/explorer/testdata/string/fail_raw_block_single_line.carbon:[[@LINE+1]]: Invalid block string: Too few lines
// CHECK:STDERR: SYNTAX ERROR: explorer/testdata/string/fail_raw_block_single_line.carbon:[[@LINE+1]]: Invalid block string: Too few lines
if (s == #'''raw string literal starting with '''#) {
return 0;
}
explorer/testdata/string/fail_raw_more_hash_tags_on_left.carbon
Vendored
+1 -3
View File
@@ -3,13 +3,11 @@
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
// AUTOUPDATE
// RUN: %{not} %{explorer-run}
// RUN: %{not} %{explorer-run-trace}
package ExplorerTest api;
fn CompareStr(s: String) -> i32 {
// CHECK:STDERR: SYNTAX ERROR: {{.*}}/explorer/testdata/string/fail_raw_more_hash_tags_on_left.carbon:[[@LINE+2]]: missing closing quote in single-line string: ##"str"#) {
// CHECK:STDERR: SYNTAX ERROR: explorer/testdata/string/fail_raw_more_hash_tags_on_left.carbon:[[@LINE+2]]: missing closing quote in single-line string: ##"str"#) {
// CHECK:STDERR:
if (s == ##"str"#) {
return 0;
explorer/testdata/string/fail_raw_more_hash_tags_on_right.carbon
Vendored
+1 -3
View File
@@ -3,13 +3,11 @@
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
// AUTOUPDATE
// RUN: %{not} %{explorer-run}
// RUN: %{not} %{explorer-run-trace}
package ExplorerTest api;
fn CompareStr(s: String) -> i32 {
// CHECK:STDERR: SYNTAX ERROR: {{.*}}/explorer/testdata/string/fail_raw_more_hash_tags_on_right.carbon:[[@LINE+1]]: invalid character '\x23' in source file.
// CHECK:STDERR: SYNTAX ERROR: explorer/testdata/string/fail_raw_more_hash_tags_on_right.carbon:[[@LINE+1]]: invalid character '\x23' in source file.
if (s == "str"#) {
return 0;
}
explorer/testdata/string/fail_tab.carbon
Vendored
+1 -3
View File
@@ -3,13 +3,11 @@
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
// AUTOUPDATE
// RUN: %{not} %{explorer-run}
// RUN: %{not} %{explorer-run-trace}
package ExplorerTest api;
fn Main() -> i32 {
// CHECK:STDERR: SYNTAX ERROR: {{.*}}/explorer/testdata/string/fail_tab.carbon:[[@LINE+1]]: Invalid escaping in string: "new line"
// CHECK:STDERR: SYNTAX ERROR: explorer/testdata/string/fail_tab.carbon:[[@LINE+1]]: Invalid escaping in string: "new line"
Print("new line");
return 0;
}
explorer/testdata/string/hex.carbon
Vendored
-2
View File
@@ -3,8 +3,6 @@
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
// AUTOUPDATE
// RUN: %{explorer-run}
// RUN: %{explorer-run-trace}
// CHECK:STDOUT: result: 0
package ExplorerTest api;
explorer/testdata/string/newline.carbon
Vendored
-2
View File
@@ -3,8 +3,6 @@
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
// AUTOUPDATE
// RUN: %{explorer-run}
// RUN: %{explorer-run-trace}
// CHECK:STDOUT: result: 0
package ExplorerTest api;
explorer/testdata/string/raw_basic.carbon
Vendored
-2
View File
@@ -3,8 +3,6 @@
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
// AUTOUPDATE
// RUN: %{explorer-run}
// RUN: %{explorer-run-trace}
// CHECK:STDOUT: result: 0
package ExplorerTest api;
explorer/testdata/string/raw_basic_multi_hashtag.carbon
Vendored
-2
View File
@@ -3,8 +3,6 @@
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
// AUTOUPDATE
// RUN: %{explorer-run}
// RUN: %{explorer-run-trace}
// CHECK:STDOUT: result: 0
package ExplorerTest api;
explorer/testdata/string/raw_block.carbon
Vendored
-2
View File
@@ -3,8 +3,6 @@
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
// AUTOUPDATE
// RUN: %{explorer-run}
// RUN: %{explorer-run-trace}
// CHECK:STDOUT: result: 0
package ExplorerTest api;
explorer/testdata/string/raw_block_escaped_back_slash.carbon
Vendored
-2
View File
@@ -3,8 +3,6 @@
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
// AUTOUPDATE
// RUN: %{explorer-run}
// RUN: %{explorer-run-trace}
// CHECK:STDOUT: result: 0
package ExplorerTest api;
explorer/testdata/string/raw_block_escaped_triple_quotes.carbon
Vendored
-2
View File
@@ -3,8 +3,6 @@
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
// AUTOUPDATE
// RUN: %{explorer-run}
// RUN: %{explorer-run-trace}
// CHECK:STDOUT: result: 0
package ExplorerTest api;
explorer/testdata/string/raw_nesting.carbon
Vendored
-2
View File
@@ -3,8 +3,6 @@
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
// AUTOUPDATE
// RUN: %{explorer-run}
// RUN: %{explorer-run-trace}
// CHECK:STDOUT: result: 0
package ExplorerTest api;
explorer/testdata/string/unicode.carbon
Vendored
-2
View File
@@ -3,8 +3,6 @@
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
//
// AUTOUPDATE
// RUN: %{explorer-run}
// RUN: %{explorer-run-trace}
// CHECK:STDOUT: result: 0
package ExplorerTest api;