Testing

This section describes how to run Wasmtime's tests and add new tests.

Before continuing, make sure you can build Wasmtime successfully. Can't run the tests if you can't build it!

Running All Tests

To run all of Wasmtime's tests (excluding WASI integration tests), execute this command:

cargo test --all

To include WASI integration tests, you'll need wasm32-wasi target installed, which, assuming you're using rustup.rs to manage your Rust versions, can be done as follows:

rustup target add wasm32-wasi

Next, to run all tests including the WASI integration tests, execute this command:

cargo test --features test_programs --all

You can also exclude a particular crate from testing with --exclude. For example, if you want to avoid testing the wastime-fuzzing crate — which requires that libclang is installed on your system, and for some reason maybe you don't have it — you can run:

cargo test --all --exclude wasmtime-fuzzing

Testing a Specific Crate

You can test a particular Wasmtime crate with cargo test -p wasmtime-whatever. For example, to test the wasmtime-environ crate, execute this command:

cargo test -p wasmtime-environ

Alternatively, you can cd into the crate's directory, and run cargo test there, without needing to supply the -p flag:

cd crates/environ/
cargo test

Running the Wasm Spec Tests

The spec testsuite itself is in a git submodule, so make sure you've checked it out and initialized its submodule:

git submodule update --init

When the submodule is checked out, Wasmtime runs the Wasm spec testsuite as part of testing the wasmtime-cli crate:

cargo test -p wasmtime-cli

Running WASI Integration Tests Only

WASI integration tests can be run separately from all other tests which can be useful when working on the wasi-common crate. This can be done by executing this command:

cargo test --features test_programs -p test-programs

Adding New Tests

Adding Rust's `#[test]`-Style Tests

For very "unit-y" tests, we add test modules in the same .rs file as the code that is being tested. These test modules are configured to only get compiled during testing with #[cfg(test)].

// some code...

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn some_test_for_that_code() {
        // ...
    }
}

If you're writing a unit test and a test module doesn't already exist, you can create one.

For more "integration-y" tests, we create a tests directory within the crate, and put the tests inside there. For example, there are various code cache-related tests at crates/environ/tests/cache_*.rs. Always feel free to add a tests directory to a crate, if you want to add a new test and there aren't any existing tests.

Adding Specification-Style Wast Tests

We use the spec testsuite as-is and without custom patches or a forked version. This probably isn't what you want to modify when adding a new Wasmtime test!

When you have a Wasmtime-specific test that you'd like to write in Wast and use the Wast-style assertions, you can add it to our "misc testsuite". The misc testsuite uses the same syntax and assertions as the spec testsuite, but lives in tests/misc_testsuite. Feel free to add new tests to existing tests/misc_testsuite/*.wast files or create new ones as needed. These tests are run as part of the wasmtime-cli crate's tests.

If you have a new test that you think really belongs in the spec testsuite, make sure it makes sense for every Wasm implementation to run your test (i.e. it isn't Wasmtime-specific) and send a pull request upstream. Once it is accepted in the upstream repo, we can update our git submodule and we'll start running the new tests.

Adding WASI Integration Tests

When you have a WASI-specific test program that you'd like to include as a test case to run against our WASI implementation, you can add it to our test-programs crate. In particular, you should drop a main-style Rust source file into crates/test-programs/wasi-tests/src/bin/some_new_test.rs with a name of your choice. And that's it! The build script included in the test-programs crate will automatically generate the necessary boilerplate code for your test program so that it's run on all supported hosts.

If you would like to tweak which host to run the test program against however (for instance, only on Unix but on Windows), you can tweak that in the build script located under crates/test-programs/build.rs.

4.6 KiB Raw Blame History