A fuzz bug was hit last night where the root of the fuzz bug appears to
be exhaustion of the virtual address space. The specific case in
question instantiated a module with ~100 memories ~100 times, and each
memory reserved ~8gb of the virtual address space. This takes around 47
bits of addressable memory which is mighty close to the limit of what
can be done on x86_64, so this commit reduces the number of memories
that an instance may have when coming out of `wasm-smith`.
* fuzzing: Add a custom mutator based on `wasm-mutate`
* fuzz: Add a version of the `compile` fuzz target that uses `wasm-mutate`
* Update `wasmparser` dependencies
Addresses #3809: when we are asked to create a Cranelift backend with
shared flags that indicate support for SIMD, we should check that the
ISA level needed for our SIMD lowerings is present.
This commit improves the stability of the fuzz targets by ensuring the
generated configs and modules are congruent, especially when the pooling
allocator is being used.
For the `differential` target, this means both configurations must use the same
allocation strategy for now as one side generates the module that might not be
compatible with another arbitrary config now that we fuzz the pooling
allocator.
These changes also ensure that constraints put on the config are more
consistently applied, especially when using a fuel-based timeout.
* Add the instance allocation strategy to generated fuzzing configs.
This commit adds support for generating configs with arbitrary instance
allocation strategies.
With this, the pooling allocator will be fuzzed as part of the existing fuzz
targets.
* Refine maximum constants for arbitrary module limits.
* Add an `instantiate-many` fuzz target.
This commit adds a new `instantiate-many` fuzz target that will attempt to
instantiate and terminate modules in an arbitrary order.
It generates up to 5 modules, from which a random sequence of instances will be
created.
The primary benefactor of this fuzz target is the pooling instance allocator.
* Allow no aliasing in generated modules when using the pooling allocator.
This commit prevents aliases in the generated modules as they might count
against the configured import limits of the pooling allocator.
As the existing module linking proposal implementation will eventually be
deprecated in favor of the component model proposal, it isn't very important
that we test aliases in generated modules with the pooling allocator.
* Improve distribution of memory config in fuzzing.
The previous commit attempted to provide a 32-bit upper bound to 64-bit
arbitrary values, which skewed the distribution heavily in favor of the upper
bound.
This commit removes the constraint and instead uses arbitrary 32-bit values
that are converted to 64-bit values in the `Arbitrary` implementation.
* fuzz: Refactor Wasmtime's fuzz targets
A recent fuzz bug found is related to timing out when compiling a
module. This timeout, however, is predominately because Cranelift's
debug verifier is enabled and taking up over half the compilation time.
I wanted to fix this by disabling the verifier when input modules might
have a lot of functions, but this was pretty difficult to implement.
Over time we've grown a number of various fuzzers. Most are
`wasm-smith`-based at this point but there's various entry points for
configuring the wasm-smith module, the wasmtime configuration, etc. I've
historically gotten quite lost in trying to change defaults and feeling
like I have to touch a lot of different places. This is the motivation
for this commit, simplifying fuzzer default configuration.
This commit removes the ability to create a default `Config` for
fuzzing, instead only supporting generating a configuration via
`Arbitrary`. This then involved refactoring all targets and fuzzers to
ensure that configuration is generated through `Arbitrary`. This should
actually expand the coverage of some existing fuzz targets since
`Arbitrary for Config` will tweak options that don't affect runtime,
such as memory configuration or jump veneers.
All existing fuzz targets are refactored to use this new method of
configuration. Some fuzz targets were also shuffled around or
reimplemented:
* `compile` - this now directly calls `Module::new` to skip all the
fuzzing infrastructure. This is mostly done because this fuzz target
isn't too interesting and is largely just seeing what happens when
things are thrown at the wall for Wasmtime.
* `instantiate-maybe-invalid` - this fuzz target now skips instantiation
and instead simply goes into `Module::new` like the `compile` target.
The rationale behind this is that most modules won't instantiate
anyway and this fuzz target is primarily fuzzing the compiler. This
skips having to generate arbitrary configuration since
wasm-smith-generated-modules (or valid ones at least) aren't used
here.
* `instantiate` - this fuzz target was removed. In general this fuzz
target isn't too interesting in isolation. Almost everything it deals
with likely won't pass compilation and is covered by the `compile`
fuzz target, and otherwise interesting modules being instantiated can
all theoretically be created by `wasm-smith` anyway.
* `instantiate-wasm-smith` and `instantiate-swarm` - these were both merged
into a new `instantiate` target (replacing the old one from above).
There wasn't really much need to keep these separate since they really
only differed at this point in methods of timeout. Otherwise we much
more heavily use `SwarmConfig` than wasm-smith's built-in options.
The intention is that we should still have basically the same coverage
of fuzzing as before, if not better because configuration is now
possible on some targets. Additionally there is one centralized point of
configuration for fuzzing for wasmtime, `Arbitrary for ModuleConfig`.
This internally creates an arbitrary `SwarmConfig` from `wasm-smith` and
then further tweaks it for Wasmtime's needs, such as enabling various
wasm proposals by default. In the future enabling a wasm proposal on
fuzzing should largely just be modifying this one trait implementation.
* fuzz: Sometimes disable the cranelift debug verifier
This commit disables the cranelift debug verifier if the input wasm
module might be "large" for the definition of "more than 10 functions".
While fuzzing we disable threads (set them to 1) and enable the
cranelift debug verifier. Coupled with a 20-30x slowdown this means that
a module with the maximum number of functions, 100, gives:
60x / 100 functions / 30x slowdown = 20ms
With only 20 milliseconds per function this is even further halved by
the `differential` fuzz target compiling a module twice, which means
that, when compiling with a normal release mode Wasmtime, if any
function takes more than 10ms to compile then it's a candidate for
timing out while fuzzing. Given that the cranelift debug verifier can
more than double compilation time in fuzzing mode this actually means
that the real time budget for function compilation is more like 4ms.
The `wasm-smith` crate can pretty easily generate a large function that
takes 4ms to compile, and then when that function is multiplied 100x in
the `differential` fuzz target we trivially time out the fuzz target.
The hope of this commit is to buy back half our budget by disabling the
debug verifier for modules that may have many functions. Further
refinements can be implemented in the future such as limiting functions
for just the differential target as well.
* Fix the single-function-module fuzz configuration
* Tweak how features work in differential fuzzing
* Disable everything for baseline differential fuzzing
* Enable selectively for each engine afterwards
* Also forcibly enable reference types and bulk memory for spec tests
* Log wasms when compiling
* Add reference types support to v8 fuzzer
* Fix timeouts via fuel
The default store has "infinite" fuel so that needs to be consumed
before fuel is added back in.
* Remove fuzzing-specific tests
These no longer compile and also haven't been added to in a long time.
Most of the time a reduced form of original the fuzz test case is added
when a fuzz bug is fixed.
Alignment on all memory instructions in wasm is currently best-effort
and not actually required, meaning that whatever wasm actually uses as
an address should work regardless of whether the address is aligned or
not. This is theoretically tested in the fuzzers via
wasm-smith-generated code, but wasm-smith doesn't today have too too
high of a chance of generating an actual successful load/store.
This commit adds a new configuration option to the `Config` generator
for fuzzing which forces usage of a custom linear memory implementation
which is backed by Rust's `Vec<u8>` and forces the base address of
linear memory to be off-by-one relative to the base address of the
`Vec<u8>` itself. This should theoretically force host addresses to
almost always be unaligned, even if wasm addresses are otherwise
aligned.
The main interesting fuzz coverage here is likely to be in the existing
`differential` target which compares running the same module in wasmtime
with two different `Config` values to ensure the same results are
produced. This probably won't increase coverage all that much in the
near future due to wasm-smith rarely generating successful loads/stores,
but in the meantime by hooking this up into `Config` it also means that
we'll be running in comparison against v8 and also ensuring that all
spec tests succeed if misalignment is forced at the hardware level.
As a side effect this commit also cleans up the fuzzers slightly:
* The `DifferentialConfig` struct is removed and folded into `Config`
* The `init_hang_limit` processing is removed since we don't use
`-ttf`-generated modules from binaryen any more.
* Traps are now asserted to have the same trap code, otherwise
differential fuzzing fails.
* Some more debug logging was added to the differential fuzzer
This commit removes the `differential_spec` fuzz target for now,
although this removal is intended to be temporary. We have #3251 to
track re-enabling the spec interpreter in a way that it won't time out,
and additionally the spec interpreter is also failing to build with
ocaml on oss-fuzz so that will also need to be investigated when
re-enabling.
This commit removes the Lightbeam backend from Wasmtime as per [RFC 14].
This backend hasn't received maintenance in quite some time, and as [RFC
14] indicates this doesn't meet the threshold for keeping the code
in-tree, so this commit removes it.
A fast "baseline" compiler may still be added in the future. The
addition of such a backend should be in line with [RFC 14], though, with
the principles we now have for stable releases of Wasmtime. I'll close
out Lightbeam-related issues once this is merged.
[RFC 14]: https://github.com/bytecodealliance/rfcs/pull/14
We've got a large crop of fuzz-bugs from fuzzing with enabled-with-SIMD
on oss-fuzz but at this point the fuzz stats from oss-fuzz say that the
fuzzers like v8 are spending less than 50% of its time actually fuzzing
and presumably mostly hitting crashes and such. While we fix the other
issues this disables simd for fuzzing with v8 so we can try to see if we
can weed out other issues.
* Add differential fuzzing against V8
This commit adds a differential fuzzing target to Wasmtime along the
lines of the wasmi and spec interpreters we already have, but with V8
instead. The intention here is that wasmi is unlikely to receive updates
over time (e.g. for SIMD), and the spec interpreter is not suitable for
fuzzing against in general due to its performance characteristics. The
hope is that V8 is indeed appropriate to fuzz against because it's
naturally receiving updates and it also is expected to have good
performance.
Here the `rusty_v8` crate is used which provides bindings to V8 as well
as precompiled binaries by default. This matches exactly the use case we
need and at least for now I think the `rusty_v8` crate will be
maintained by the Deno folks as they continue to develop it. If it
becomes an issue though maintaining we can evaluate other options to
have differential fuzzing against.
For now this commit enables the SIMD and bulk-memory feature of
fuzz-target-generation which should enable them to get
differentially-fuzzed with V8 in addition to the compilation fuzzing
we're already getting.
* Use weak linkage for GDB jit helpers
This should help us deduplicate our symbol with other JIT runtimes, if
any. For now this leans on some C helpers to define the weak linkage
since Rust doesn't support that on stable yet.
* Don't use rusty_v8 on MinGW
They don't have precompiled libraries there.
* Fix msvc build
* Comment about execution
* Update wasm-smith to 0.7.0
* Canonicalize NaN with wasm-smith for differential fuzzing
This then also enables floating point executing in wasmi in addition to
the spec interpreter. With NaN canonicalization at the wasm level this
means that we should be producing deterministic results between Wasmtime
and these alternative implementations.
* Implement the memory64 proposal in Wasmtime
This commit implements the WebAssembly [memory64 proposal][proposal] in
both Wasmtime and Cranelift. In terms of work done Cranelift ended up
needing very little work here since most of it was already prepared for
64-bit memories at one point or another. Most of the work in Wasmtime is
largely refactoring, changing a bunch of `u32` values to something else.
A number of internal and public interfaces are changing as a result of
this commit, for example:
* Acessors on `wasmtime::Memory` that work with pages now all return
`u64` unconditionally rather than `u32`. This makes it possible to
accommodate 64-bit memories with this API, but we may also want to
consider `usize` here at some point since the host can't grow past
`usize`-limited pages anyway.
* The `wasmtime::Limits` structure is removed in favor of
minimum/maximum methods on table/memory types.
* Many libcall intrinsics called by jit code now unconditionally take
`u64` arguments instead of `u32`. Return values are `usize`, however,
since the return value, if successful, is always bounded by host
memory while arguments can come from any guest.
* The `heap_addr` clif instruction now takes a 64-bit offset argument
instead of a 32-bit one. It turns out that the legalization of
`heap_addr` already worked with 64-bit offsets, so this change was
fairly trivial to make.
* The runtime implementation of mmap-based linear memories has changed
to largely work in `usize` quantities in its API and in bytes instead
of pages. This simplifies various aspects and reflects that
mmap-memories are always bound by `usize` since that's what the host
is using to address things, and additionally most calculations care
about bytes rather than pages except for the very edge where we're
going to/from wasm.
Overall I've tried to minimize the amount of `as` casts as possible,
using checked `try_from` and checked arithemtic with either error
handling or explicit `unwrap()` calls to tell us about bugs in the
future. Most locations have relatively obvious things to do with various
implications on various hosts, and I think they should all be roughly of
the right shape but time will tell. I mostly relied on the compiler
complaining that various types weren't aligned to figure out
type-casting, and I manually audited some of the more obvious locations.
I suspect we have a number of hidden locations that will panic on 32-bit
hosts if 64-bit modules try to run there, but otherwise I think we
should be generally ok (famous last words). In any case I wouldn't want
to enable this by default naturally until we've fuzzed it for some time.
In terms of the actual underlying implementation, no one should expect
memory64 to be all that fast. Right now it's implemented with
"dynamic" heaps which have a few consequences:
* All memory accesses are bounds-checked. I'm not sure how aggressively
Cranelift tries to optimize out bounds checks, but I suspect not a ton
since we haven't stressed this much historically.
* Heaps are always precisely sized. This means that every call to
`memory.grow` will incur a `memcpy` of memory from the old heap to the
new. We probably want to at least look into `mremap` on Linux and
otherwise try to implement schemes where dynamic heaps have some
reserved pages to grow into to help amortize the cost of
`memory.grow`.
The memory64 spec test suite is scheduled to now run on CI, but as with
all the other spec test suites it's really not all that comprehensive.
I've tried adding more tests for basic things as I've had to implement
guards for them, but I wouldn't really consider the testing adequate
from just this PR itself. I did try to take care in one test to actually
allocate a 4gb+ heap and then avoid running that in the pooling
allocator or in emulation because otherwise that may fail or take
excessively long.
[proposal]: https://github.com/WebAssembly/memory64/blob/master/proposals/memory64/Overview.md
* Fix some tests
* More test fixes
* Fix wasmtime tests
* Fix doctests
* Revert to 32-bit immediate offsets in `heap_addr`
This commit updates the generation of addresses in wasm code to always
use 32-bit offsets for `heap_addr`, and if the calculated offset is
bigger than 32-bits we emit a manual add with an overflow check.
* Disable memory64 for spectest fuzzing
* Fix wrong offset being added to heap addr
* More comments!
* Clarify bytes/pages
This new target compares the outputs of executing the first exported
function of a Wasm module in Wasmtime and in the official Wasm spec
interpreter (using the `wasm-spec-interpreter` crate). This is an
initial step towards more fully-featured fuzzing (e.g. compare memories,
add `v128`, add references, add other proposals, etc.)
* Enable simd fuzzing on oss-fuzz
This commit generally enables the simd feature while fuzzing, which
should affect almost all fuzzers. For fuzzers that just throw random
data at the wall and see what sticks, this means that they'll now be
able to throw simd-shaped data at the wall and have it stick. For
wasm-smith-based fuzzers this commit also updates wasm-smith to 0.6.0
which allows further configuring the `SwarmConfig` after generation,
notably allowing `instantiate-swarm` to generate modules using simd
using `wasm-smith`. This should much more reliably feed simd-related
things into the fuzzers.
Finally, this commit updates wasmtime to avoid usage of the general
`wasm_smith::Module` generator to instead use a Wasmtime-specific custom
default configuration which enables various features we have
implemented.
* Allow dummy table creation to fail
Tables might creation for imports may exceed the memory limit on the
store, which we'll want to gracefully recover from and not fail the
fuzzers.
* cranelift: Initial fuzzer implementation
* cranelift: Generate multiple test cases in fuzzer
* cranelift: Separate function generator in fuzzer
* cranelift: Insert random instructions in fuzzer
* cranelift: Rename gen_testcase
* cranelift: Implement div for unsigned values in interpreter
* cranelift: Run all test cases in fuzzer
* cranelift: Comment options in function_runner
* cranelift: Improve fuzzgen README.md
* cranelift: Fuzzgen remove unused variable
* cranelift: Fuzzer code style fixes
Thanks! @bjorn3
* cranelift: Fix nits in CLIF fuzzer
Thanks @cfallin!
* cranelift: Implement Arbitrary for TestCase
* cranelift: Remove gen_testcase
* cranelift: Move fuzzers to wasmtime fuzz directory
* cranelift: CLIF-Fuzzer ignore tests that produce traps
* cranelift: CLIF-Fuzzer create new fuzz target to validate generated testcases
* cranelift: Store clif-fuzzer config in a separate struct
* cranelift: Generate variables upfront per function
* cranelift: Prevent publishing of fuzzgen crate
There are occasional timeouts in type checking where Z3 hangs. This is a known
issue[0] with the implementation of type checking in Peepmatic, and getting
these timeouts in the fuzzers is just annoying and adds noise to our fuzzing
results. When we fix [0] we can reintroduce these fuzz targets.
[0]: https://github.com/bytecodealliance/wasmtime/issues/2695
* Consume fuel during function execution
This commit adds codegen infrastructure necessary to instrument wasm
code to consume fuel as it executes. Currently nothing is really done
with the fuel, but that'll come in later commits.
The focus of this commit is to implement the codegen infrastructure
necessary to consume fuel and account for fuel consumed correctly.
* Periodically check remaining fuel in wasm JIT code
This commit enables wasm code to periodically check to see if fuel has
run out. When fuel runs out an intrinsic is called which can do what it
needs to do in the result of fuel running out. For now a trap is thrown
to have at least some semantics in synchronous stores, but another
planned use for this feature is for asynchronous stores to periodically
yield back to the host based on fuel running out.
Checks for remaining fuel happen in the same locations as interrupt
checks, which is to say the start of the function as well as loop
headers.
* Improve codegen by caching `*const VMInterrupts`
The location of the shared interrupt value and fuel value is through a
double-indirection on the vmctx (load through the vmctx and then load
through that pointer). The second pointer in this chain, however, never
changes, so we can alter codegen to account for this and remove some
extraneous load instructions and hopefully reduce some register
pressure even maybe.
* Add tests fuel can abort infinite loops
* More fuzzing with fuel
Use fuel to time out modules in addition to time, using fuzz input to
figure out which.
* Update docs on trapping instructions
* Fix doc links
* Fix a fuzz test
* Change setting fuel to adding fuel
* Fix a doc link
* Squelch some rustdoc warnings
This commit updates the various tooling used by wasmtime which has new
updates to the module linking proposal. This is done primarily to sync
with WebAssembly/module-linking#26. The main change implemented here is
that wasmtime now supports creating instances from a set of values, nott
just from instantiating a module. Additionally subtyping handling of
modules with respect to imports is now properly handled by desugaring
two-level imports to imports of instances.
A number of small refactorings are included here as well, but most of
them are in accordance with the changes to `wasmparser` and the updated
binary format for module linking.
Recent changes to fuzzers made expectations more strict about handling
errors while fuzzing, but this erroneously changed a module compilation
step to always assume that the input wasm is valid. Instead a flag is
now passed through indicating whether the wasm blob is known valid or
invalid, and only if compilation fails and it's known valid do we panic.
This PR adds a new fuzz target, `differential_wasmi`, that runs a
Cranelift-based Wasm backend alongside a simple third-party Wasm
interpeter crate (`wasmi`). The fuzzing runs the first function in a
given module to completion on each side, and then diffs the return value
and linear memory contents.
This strategy should provide end-to-end coverage including both the Wasm
translation to CLIF (which has seen some subtle and scary bugs at
times), the lowering from CLIF to VCode, the register allocation, and
the final code emission.
This PR also adds a feature `experimental_x64` to the fuzzing crate (and
the chain of dependencies down to `cranelift-codegen`) so that we can
fuzz the new x86-64 backend as well as the current one.
This commit removes the binaryen support for fuzzing from wasmtime,
instead switching over to `wasm-smith`. In general it's great to have
what fuzzing we can, but our binaryen support suffers from a few issues:
* The Rust crate, binaryen-sys, seems largely unmaintained at this
point. While we could likely take ownership and/or send PRs to update
the crate it seems like the maintenance is largely on us at this point.
* Currently the binaryen-sys crate doesn't support fuzzing anything
beyond MVP wasm, but we're interested at least in features like bulk
memory and reference types. Additionally we'll also be interested in
features like module-linking. New features would require either
implementation work in binaryen or the binaryen-sys crate to support.
* We have 4-5 fuzz-bugs right now related to timeouts simply in
generating a module for wasmtime to fuzz. One investigation along
these lines in the past revealed a bug in binaryen itself, and in any
case these bugs would otherwise need to get investigated, reported,
and possibly fixed ourselves in upstream binaryen.
Overall I'm not sure at this point if maintaining binaryen fuzzing is
worth it with the advent of `wasm-smith` which has similar goals for
wasm module generation, but is much more readily maintainable on our
end.
Additonally in this commit I've added a fuzzer for wasm-smith's
`SwarmConfig`-based fuzzer which should expand the coverage of tested
modules.
Closes#2163
This commit uses the new `MaybeInvalidModule` type in `wasm-smith` to
try to explore more points in the fuzz target space in the
`instantiate-maybe-invalid` fuzz target. The goal here is to use the raw
fuzz input as the body of a function to stress the validator/decoder a
bit more, and try to get inputs we might not otherwise generate.
This new fuzz target exercises sequences of `table.get`s, `table.set`s, and
GCs.
It already found a couple bugs:
* Some leaks due to ref count cycles between stores and host-defined functions
closing over those stores.
* If there are no live references for a PC, Cranelift can avoid emiting an
associated stack map. This was running afoul of a debug assertion.
* Add a spec test fuzzer for Config
This commit adds a new fuzzer which is intended to run on oss-fuzz. This
fuzzer creates and arbitrary `Config` which *should* pass spec tests and
then asserts that it does so. The goal here is to weed out any
accidental bugs in global configuration which could cause
non-spec-compliant behavior.
* Move implementation to `fuzzing` crate
... but turn it back on in CI by default. The `binaryen-sys` crate
builds binaryen from source, which is a drag on CI for a few reasons:
* This is quite large and takes a good deal of time to build
* The debug build directory for binaryen is 4GB large
In an effort to both save time and disk space on the builders this
commit adds a `binaryen` feature to the `wasmtime-fuzz` crate. This
feature is enabled specifically when running the fuzzers on CI, but it
is disabled during the typical `cargo test --all` command. This means
that the test builders should save an extra 4G of space and be a bit
speedier now that they don't build a giant wad of C++.
We'll need to update the OSS-fuzz integration to enable the `binaryen`
feature when executing `cargo fuzz build`, and I'll do that once this
gets closer to landing.
This commit removes the two fuzz targets that we imported from cranelift
when cranelift merged in. These have both uncovered a few issues in the
fuzz targets themselves, for example:
* `translate_module` - this doesn't verify the wasm is valid a head of
time and cranelift is known to panic on translating invalid wasm
modules. We also already do a lot of fuzzing of translation of wasm
modules, so this isn't necessarily buying us anything over what we're
already fuzzing.
* `reader_parse_test` - discovered in #1205 we already found some "bugs"
in this but it may not necessarily rise to the level of "needs to be
run on oss-fuzz for us to find more bugs" yet. It looks like this is
still somewhat internal so we can re-enable when we've got folks to
fix the fuzz bugs coming in.
Closes#1205
* Move compilation into Module from Instance.
* Fix fuzzing
* Use wasmtime::Module in fuzzing crates
Instead of wasmtime_jit.
* Compile eagerly.
* Review fixes.
* Always use the saved name.
* Preserve the former behavior for fuzzing oracle
* Remove usage of `CompilationStrategy` from `Config`
This commit removes the public API usage of the internal
`CompilationStrategy` enumeration from the `Config` type in the
`wasmtime` crate. To do this the `enum` was copied locally into the
crate and renamed `Strategy`. The high-level description of this change
is:
* The `Config::strategy` method now takes a locally-defined `Strategy`
enumeration instead of an internal type.
* The contents of `Strategy` are always the same, not relying on Cargo
features to indicate which variants are present. This avoids
unnecessary downstream `#[cfg]`.
* A `lightbeam` feature was added to the `wasmtime` crate itself to
lightbeam compilation support.
* The `Config::strategy` method is now fallible. It returns a runtime
error if support for the selected strategy wasn't compiled in.
* The `Strategy` enum is listed as `#[non_exhaustive]` so we can safely
add variants over time to it.
This reduces the public crate dependencies of the `wasmtime` crate
itself, removing the need to reach into internal crates even more!
cc #708
* Fix fuzz targets
* Update nightly used to build releases
* Run rustfmt
We only generate *valid* sequences of API calls. To do this, we keep track of
what objects we've already created in earlier API calls via the `Scope` struct.
To generate even-more-pathological sequences of API calls, we use [swarm
testing]:
> In swarm testing, the usual practice of potentially including all features
> in every test case is abandoned. Rather, a large “swarm” of randomly
> generated configurations, each of which omits some features, is used, with
> configurations receiving equal resources.
[swarm testing]: https://www.cs.utah.edu/~regehr/papers/swarm12.pdf
There are more public APIs and instance introspection APIs that we have than
this fuzzer exercises right now. We will need a better generator of valid Wasm
than `wasm-opt -ttf` to really get the most out of those currently-unexercised
APIs, since the Wasm modules generated by `wasm-opt -ttf` don't import and
export a huge variety of things.
When the test case that causes the failure can successfully be disassembled to
WAT, we get logs like this:
```
[2019-11-26T18:48:46Z INFO wasmtime_fuzzing] Wrote WAT disassembly to: /home/fitzgen/wasmtime/crates/fuzzing/target/scratch/8437-0.wat
[2019-11-26T18:48:46Z INFO wasmtime_fuzzing] If this fuzz test fails, copy `/home/fitzgen/wasmtime/crates/fuzzing/target/scratch/8437-0.wat` to `wasmtime/crates/fuzzing/tests/regressions/my-regression.wat` and add the following test to `wasmtime/crates/fuzzing/tests/regressions.rs`:
```
#[test]
fn my_fuzzing_regression_test() {
let data = wat::parse_str(
include_str!("./regressions/my-regression.wat")
).unwrap();
oracles::instantiate(data, CompilationStrategy::Auto)
}
```
```
If the test case cannot be disassembled to WAT, then we get logs like this:
```
[2019-11-26T18:48:46Z INFO wasmtime_fuzzing] Wrote Wasm test case to: /home/fitzgen/wasmtime/crates/fuzzing/target/scratch/8437-0.wasm
[2019-11-26T18:48:46Z INFO wasmtime_fuzzing] Failed to disassemble Wasm into WAT:
Bad magic number (at offset 0)
Stack backtrace:
Run with RUST_LIB_BACKTRACE=1 env variable to display a backtrace
[2019-11-26T18:48:46Z INFO wasmtime_fuzzing] If this fuzz test fails, copy `/home/fitzgen/wasmtime/crates/fuzzing/target/scratch/8437-0.wasm` to `wasmtime/crates/fuzzing/tests/regressions/my-regression.wasm` and add the following test to `wasmtime/crates/fuzzing/tests/regressions.rs`:
```
#[test]
fn my_fuzzing_regression_test() {
let data = include_bytes!("./regressions/my-regression.wasm");
oracles::instantiate(data, CompilationStrategy::Auto)
}
```
```
Alex Crichton
Conrad Watt
Nick Fitzgerald
Chris Fallin
Peter Huene
Afonso Bordado
Andrew Brown
Afonso Bordado
Sergei Pepyakin