mirror of
https://github.com/carbon-language/carbon-lang.git
synced 2026-09-28 07:40:11 +01:00
Jon Ross-Perkins
d12583fc08
I'm partly doing this because the current setup would be difficult to share with the toolchain. e.g., ProtoToCarbon isn't explorer-specific, but the only way to run it via CLI is the explorer's fuzzverter. I want a separate tool.
This change:
- Adds a //common/fuzzing:proto_to_carbon tool.
- The rest of fuzzverter is now just //explorer/fuzzing:ast_to_proto.
- The change simplifies overall handling and removes a LLVM CLI dependency.
- Stops allowing unknown fields in the proto.
- This has mostly led to forgetting to remove fuzzer inputs that were for removed features.
- Moves more non-explorer-specific bits to //common/fuzzing.
- Cleans up remaining pieces in //explorer/fuzzing
- Merges the //explorer/fuzzing proto tests, which deduplicates AstToString copies.
- These tests also had duplicate dependencies, etc -- and all complete in ~6s.
- Updates and fixes regen_corpus which was previously broken by other changes.
- Updates the README to reflect changes.
- Removes obsolete proto-fuzzer build configuration (AFAICT this is no longer needed).
114 lines
3.9 KiB
Markdown
114 lines
3.9 KiB
Markdown
# Explorer structured fuzzer
|
|
|
|
<!--
|
|
Part of the Carbon Language project, under the Apache License v2.0 with LLVM
|
|
Exceptions. See /LICENSE for license information.
|
|
SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
|
|
-->
|
|
|
|
## Overview
|
|
|
|
Fuzz testing is based on generating a large amount of random inputs for a
|
|
software component in order to trigger bugs and unexpected behavior. Basic
|
|
fuzzing uses randomly generated arrays of bytes as inputs, which works great for
|
|
some applications but is problematic for testing the logic that operates on
|
|
highly structured data, as most random inputs are immediately rejected as
|
|
invalid before any interesting parts of the code get a chance to run.
|
|
|
|
Structured fuzzing addresses this issue by ensuring the randomly generated data
|
|
is itself structured, and as such has a high chance of presenting a valid input.
|
|
|
|
`explorer_fuzzer` is a structured fuzzer based on
|
|
[libprotobuf-mutator](https://github.com/google/libprotobuf-mutator), which is a
|
|
library to randomly mutate
|
|
[protobuffers](https://github.com/protocolbuffers/protobuf).
|
|
|
|
The input to the fuzzer is an instance of `Carbon::Fuzzing::Carbon` proto
|
|
randomly generated by the `libprotobuf-mutator` framework. `explorer_fuzzer`
|
|
converts the proto to a Carbon source code string, and tries to parse and
|
|
execute the code using `explorer` implementation.
|
|
|
|
## Fuzzer data format
|
|
|
|
`libprotobuf-mutator` supports fuzzer inputs in either text or binary protocol
|
|
buffer format. `explorer_fuzzer` uses text proto format with `Carbon` proto
|
|
message definition in `common/fuzzing/carbon.proto`.
|
|
|
|
## Incorporating AST changes into the fuzzer
|
|
|
|
Fuzzer AST representation in [carbon.proto](/common/fuzzing/carbon.proto) needs
|
|
to be updated when changes are made to the AST, like adding a new AST node
|
|
classes or changing relevant data members of existing nodes.
|
|
|
|
[ast_to_proto_test](ast_to_proto_test.cpp) normally should not require direct
|
|
changes, as tests work off of Carbon test files in
|
|
[testdata](/explorer/testdata).
|
|
|
|
To incorporate AST changes into fuzzing logic:
|
|
|
|
1. Add appropriate AST information to
|
|
[carbon.proto](/common/fuzzing/carbon.proto). Use existing similar cases as
|
|
examples.
|
|
|
|
2. Modify [proto_to_carbon.cpp](/common/fuzzing/proto_to_carbon.cpp) which
|
|
handles printing of a Carbon proto instance as a Carbon source string. For
|
|
example, add code to print newly introduced proto fields.
|
|
|
|
3. Add logic to populate the proto to
|
|
[ast_to_proto.cpp](/explorer/fuzzing/ast_to_proto.cpp).
|
|
|
|
4. Make sure `bazel test //explorer/fuzzing:ast_to_proto_test` passes with the
|
|
new changes.
|
|
|
|
## Running the fuzzer
|
|
|
|
The fuzzer can be run in 'unit test' mode, where the fuzzer executes on each
|
|
input file from the `fuzzer_corpus/` folder, or in 'fuzzing' mode, where the
|
|
fuzzer will keep generating random inputs and executing the logic on them until
|
|
a crash is triggered, or forever in a bug-free program ;).
|
|
|
|
To run in 'unit test' mode:
|
|
|
|
```bash
|
|
bazel test //explorer/fuzzing:explorer_fuzzer
|
|
```
|
|
|
|
To run in 'fuzzing' mode:
|
|
|
|
```bash
|
|
bazel build --config=fuzzer //explorer/fuzzing:explorer_fuzzer.full_corpus
|
|
|
|
bazel-bin/explorer/fuzzing/explorer_fuzzer.full_corpus
|
|
```
|
|
|
|
## Investigating a crash
|
|
|
|
Typically it's going to be easiest to run explorer on directly. You can do this
|
|
with:
|
|
|
|
````bash
|
|
# Convert the crash to a source file.
|
|
bazel run //common/fuzzing:proto_to_carbon -- /tmp/crash.textproto > crash.carbon
|
|
|
|
# Run explorer on the crash.
|
|
bazel run //explorer -- crash.carbon
|
|
```
|
|
|
|
It's also possible to run the fuzzer on a single input:
|
|
|
|
```bash
|
|
bazel-bin/explorer/fuzzing/explorer_fuzzer.full_corpus /tmp/crash.textproto
|
|
````
|
|
|
|
## Generating new fuzzer corpus entries
|
|
|
|
The ability of the fuzzing framework to generate 'interesting' inputs can be
|
|
improved by providing 'seed' inputs known as the fuzzer corpus. The inputs need
|
|
to be a `Fuzzing::Carbon` text proto.
|
|
|
|
To generate a text proto from Carbon source:
|
|
|
|
```bash
|
|
bazel run //explorer/fuzzing:ast_to_proto -- /tmp/crash.carbon > crash.textproto
|
|
```
|