It was an alias, defined in `build.rs`, for
cfg(any(feature = "cranelift", feature = "winch"))
But it wasn't actually saving us much typing because we also have to have
#[cfg_attr(nightlydoc, doc(cfg(any(feature = "cranelift", feature = "winch"))))]
for all the public APIs that are gated on a compiler being present so that the
API docs show which methods require which features.
So I'm removing it and using the full `cfg(any(..))` form instead.
This allows us to remove a `build.rs` that is otherwise unused and helps
`rust-analyzer` give better code completions and jump-to-definitions and things.
* Fold guest profiling flags into `--profile` CLI option
This commit removes the `--profile-guest` and `--profile-guest-interval`
CLI flags and folds them into the preexisting `--profile` flag.
Specifying guest profiling can now be done with `--profile guest` which
has a default path/interval to write to. The path/interval can be
configured with `--profile guest,path.json` and `--profile
guest,path.json,10ms`.
* Update src/commands/run.rs
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
---------
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
* Reorganize profiling-related code
This commit is a bit of reorganization around profiling-related code in
Wasmtime with the aim of eventually simplifying it a bit more. The
changes here are:
* All exposed agents are removed and instead only constructor functions
returning trait objects are now exposed.
* All `*_disabled.rs` files and modules are removed in favor of a
function that returns a result (less trait impls).
* All `*_linux.rs` files where renamed to just `*.rs`. (less files in
play)
* The `pid` and `tid` arguments were removed as they were only used by
the jitdump profiler and now that manages it internally.
* Registering an entire ELF image is now part of the trait rather than
buried within the trampoline code in Wasmtime.
* Remove DWARF support from jitdump
In general Wasmtime's support for DWARF is not great so this is rarely
used and at least in my experience this hasn't been necessary to get
good information from perf. This commit removes the processing here
which while probably useful is probably not necessary and otherwise
makes the jidump agent a bit of an odd-one-out relative among the other
agents.
* Remove now no-longer-needed `dbg_image` argument
* Only grab the jitdump lock once-per-module
Refactor slightly to account for this.
* Fill in the `tid` argument to jitump
This has been the same as `self.pid` for quite some time but with
`rustix` it's pretty easy to get access to the current thread id.
* Merge module/trampoline registration for profilers
Add a second argument to registration of an entire module for custom
names to get functions named correctly, and otherwise profilers now only
need to look at individual functions.
* Fixup vtune support
* Delete no-longer-needed accessors
Closes#6328
* Remove unused import
* Fix a portability issue with u32-vs-i32
Bake in the `*mut VMContext` to `&mut Instance` translation into the
macro-generated trampolines to avoid the need to use
`Instance::from_vmctx` with an extra level of indentation everywhere.
Additionally some libcalls are now entirely safe code as their one
unsafe operation was the `VMContext` to `Instance` translation.
* Make Wasmtime compatible with Stacked Borrows in MIRI
The fact that Wasmtime executes correctly under Tree Borrows but not
Stacked Borrows is a bit suspect and given what I've since learned about
the aliasing models I wanted to give it a stab to get things working
with Stacked Borrows. It turns out that this wasn't all that difficult,
but required two underlying changes:
* First the implementation of `Instance::vmctx` is now specially crafted
in an intentional way to preserve the provenance of the returned
pointer. This way all `&Instance` pointers will return a `VMContext`
pointer with the same provenance and acquiring the pointer won't
accidentally invalidate all prior pointers.
* Second the conversion from `VMContext` to `Instance` has been updated
to work with provenance and such. Previously the conversion looked
like `&mut VMContext -> &mut Instance`, but I think this didn't play
well with MIRI because `&mut VMContext` has no provenance over any
data since it's zero-sized. Instead now the conversion is from `*mut
VMContext` to `&mut Instance` where we know that `*mut VMContext` has
provenance over the entire instance allocation. This shuffled a fair
bit around to handle the new closure-based API to prevent escaping
pointers, but otherwise no major change other than the structure and
the types in play.
This commit additionally picks up a dependency on the `sptr` crate which
is a crate for prototyping strict-provenance APIs in Rust. This is I
believe intended to be upstreamed into Rust one day (it's in the
standard library as a Nightly-only API right now) but in the meantime
this is a stable alternative.
* Clean up manual `unsafe impl Send` impls
This commit adds a new wrapper type `SendSyncPtr<T>` which automatically
impls the `Send` and `Sync` traits based on the `T` type contained.
Otherwise it works similarly to `NonNull<T>`. This helps clean up a
number of manual annotations of `unsafe impl {Send,Sync} for ...`
throughout the runtime.
* Remove pointer-to-integer casts with tables
In an effort to enable MIRI's "strict provenance" mode this commit
removes the integer-to-pointer casts in the runtime `Table`
implementation for Wasmtime. Most of the bits were already there to
track all this, so this commit plumbed around the various pointer types
and with the help of the `sptr` crate preserves the provenance of all
related pointers.
* Remove integer-to-pointer casts in CoW management
The `MemoryImageSlot` type stored a `base: usize` field mostly because I
was too lazy to have a `Send`/`Sync` type as a pointer, so this commit
updates it to use `SendSyncPtr<u8>` and then plumbs the pointer-ness
throughout the implementation. This removes all integer-to-pointer casts
and has pointers stores as actual pointers when they're at rest.
* Remove pointer-to-integer casts in "raw" representations
This commit changes the "raw" representation of `Func` and `ExternRef`
to a `*mut c_void` instead of the previous `usize`. This is done to
satisfy MIRI's requirements with strict provenance, properly marking the
intermediate value as a pointer rather than round-tripping through
integers.
* Minor remaining cleanups
* Switch to Stacked Borrows for MIRI on CI
Additionally enable the strict-provenance features to force warnings
emitted today to become errors.
* Fix a typo
* Replace a negative offset with `sub`
* Comment the sentinel value
* Use NonNull::dangling
In #6338 that PR is bouncing on CI due to Windows running out of disk
space. Locally a `./ci/run-tests.sh` compile produces a ~13G build
directory. Testing on CI it looks like Windows builders have ~13G of
free space on them for GitHub Actions. I think something in that PR has
tipped us just over the edge of requiring too much space, although I'm
not sure exactly what.
To address the issue this commit disables DWARF debug information
entirely on all builders on CI. Not only should this save a sliver of
compile time but this reduces a local build directory to ~4.7G, over a
50% reduction. That should keep us fitting in CI for more time to come
hopefully.
* x64: Add non-SSE 4.1 lowerings for `insertlane` instructions
This commit avoids the use of `pinsr*` when SSE 4.1 is enabled by using
alternative means of inserting values into vectors.
* Clarify comments
* Remove the initializer from a global's type information
This commit removes the `Global::initializer` field to instead only
store type information about the global rather than its initialization
state. Instead now the initializer is stored separately from the type of
the global, and only for defined globals. This removes the need in a few
locations to synthesize dummy initializers.
* Actually delete what I intended to delete
* Simplify global initializer loop
* Remove lingering references to `experimental_x64`
This hasn't been experimental for quite a long time now and these are
all mostly old vestiges of when the current backend was under
development, so remove them all as they're no longer necessary.
* Try to fix test
* Try again to fix tests
* add fuel parameter to control plane
Co-authored-by: Falk Zwimpfer <24669719+FalkZ@users.noreply.github.com>
Co-authored-by: Moritz Waser <mzrw.dev@pm.me>
* remove unused cranelift setting
Co-authored-by: Falk Zwimpfer <24669719+FalkZ@users.noreply.github.com>
Co-authored-by: Moritz Waser <mzrw.dev@pm.me>
* update fuel parameter documentation
Co-authored-by: Falk Zwimpfer <24669719+FalkZ@users.noreply.github.com>
Co-authored-by: Moritz Waser <mzrw.dev@pm.me>
* include control plane in printed test case
Co-authored-by: Falk Zwimpfer <24669719+FalkZ@users.noreply.github.com>
Co-authored-by: Moritz Waser <mzrw.dev@pm.me>
* print control plane as comment
Co-authored-by: Falk Zwimpfer <24669719+FalkZ@users.noreply.github.com>
Co-authored-by: Moritz Waser <mzrw.dev@pm.me>
* set fuel limit in cranelift settings
Co-authored-by: Falk Zwimpfer <24669719+FalkZ@users.noreply.github.com>
Co-authored-by: Moritz Waser <mzrw.dev@pm.me>
* separate cli arg parsing from arbitrary impl
`ControlPlane::new` is replaced with an `Arbitrary` implementation
and `ControlPlane::set_fuel`, as it was before.
The CLI argument parsing inside the fuzz target `cranelift-fuzzgen`
is separated from the `Arbitrary` implementation of `TestCase`.
To achieve this, the `TestCase` doesn't carry a build TargetIsa
anymore, but it it generated with an isa- and flags-builder.
The CLI argument can then modify the flags further before the
`TargetIsa` is built.
The `TestCase.isa_builder` is wrapper in an `Rc` such that optimized
test cases can share the same one and it doesn't need to be made
`Clone`.
Co-authored-by: Falk Zwimpfer <24669719+FalkZ@users.noreply.github.com>
Co-authored-by: Moritz Waser <mzrw.dev@pm.me>
* parse control planes with clif-util
Co-authored-by: Falk Zwimpfer <24669719+FalkZ@users.noreply.github.com>
Co-authored-by: Moritz Waser <mzrw.dev@pm.me>
* move parsing logic from cranelift-control to cranelift-reader
Co-authored-by: Falk Zwimpfer <24669719+FalkZ@users.noreply.github.com>
Co-authored-by: Moritz Waser <mzrw.dev@pm.me>
* remove parsing and settings plumbing
Co-authored-by: Falk Zwimpfer <24669719+FalkZ@users.noreply.github.com>
Co-authored-by: Moritz Waser <mzrw.dev@pm.me>
---------
Co-authored-by: Falk Zwimpfer <24669719+FalkZ@users.noreply.github.com>
Co-authored-by: Moritz Waser <mzrw.dev@pm.me>
* test: clean up `wasi_testsuite` test files
When running `cargo run wasi_testsuite -- --nocapture`, some test cases
generate files and folders that need to be removed for subsequent test
runs to pass. These paths are all suffixed with `*.cleanup` so we walk
the suite directory before and after execution to clean up these files.
* test: fix how `wasi_testsuite` maps directories
This change fixes how directories are mapped from the host environment
to the WebAssembly guest when the `wasi_testsuite` test is run.
Previously, Wasmtime's `--dir` flag was used which expects the passed
directory to be in the current working directory. In this case, though,
the directories were buried down `tests/wasi-testsuite/wasi-common/...`
so the correct flag is `--mapdir`. Switching to this flag allows a large
number of the ignored tests to now pass.
* wasi: add the `wasi-testsuite` tests for wasi-common
As described [here], this uses the `prod/testsuite-base` branch in which
the tests are built as `.wasm` files.
[here]: https://github.com/WebAssembly/wasi-testsuite/#getting-started
* chore: update `walkdir` everywhere to its latest version
This is done in order to use it for `wasi_testsuite` testing.
* vet: extend `walkdir`'s exemption
* test: factor out `get_wasmtime_command`
This will be helpful for `wasi_testsuite` testing.
* test: use all `wasi-testsuite` test cases
This change alters the `wasi_testsuite` test to run all of the available
test cases in [wasi-testsuite]. This involved making the test runner a
bit more robust to the various shapes of JSON specifications in that
project. Unfortunately, the `wasi_testsuite` test fails some of the
cases, so I added a `WASI_COMMON_IGNORE_LIST` to avoid these
temporarily. (This may remind some of the Wasm testsuite ignore lists in
Cranelift; those relied on `build.rs` to create a `#[test]` for each
test case, which I felt is not yet needed here).
It's unclear to me why the tests are failing. It could be because:
- wasi-common has a bug
- wasi-testsuite overspecifies (or incorrectly specifies) a test
- the test runner incorrectly configures Wasmtime's CLI execution.
But this change makes it easier to resolve this. Remove the file from
`WASI_COMMON_IGNORE_LIST` and run `cargo test wasi_testsuite --
--nocapture`. The printed output will show the expected result, the
actual result, and a command to replicate the failure from the command
line.
[wasi-testsuite]: https://github.com/WebAssembly/wasi-testsuite
* review: add "shrinking" comment
* Unify differential result comparison logic
Execution of functions handles the case where one instance stack
overflows and the other didn't, but instantiation didn't handle this
case. This commit fixes this issue by sharing the result comparison
logic between the two branches.
* Update crates/fuzzing/src/oracles.rs
Co-authored-by: bjorn3 <17426603+bjorn3@users.noreply.github.com>
---------
Co-authored-by: bjorn3 <17426603+bjorn3@users.noreply.github.com>
Changing LLVM and/or Rust to avoid special handling of `main` is a fair
amount of work, and there could be other toolchains with similar special
rules for functions named `main`, so rename the command entrypoint back
to `run`.
We could potentially re-evaluate this in the future, such as in a
preview3 timeframe, but for now, let's go with the simplest thing that
works.
Changing LLVM and/or Rust to avoid special handling of `main` is a fair
amount of work, and there could be other toolchains with similar special
rules for functions named `main`, so rename the command entrypoint back
to `run`.
We could potentially re-evaluate this in the future, such as in a
preview3 timeframe, but for now, let's go with the simplest thing that
works.
* Run some tests in MIRI on CI
This commit is an implementation of getting at least chunks of Wasmtime
to run in MIRI on CI. The full test suite is not possible to run in MIRI
because MIRI cannot run Cranelift-produced code at runtime (aka it
doesn't support JITs). Running MIRI, however, is still quite valuable if
we can manage it because it would have trivially detected
GHSA-ch89-5g45-qwc7, our most recent security advisory. The goal of this
PR is to select a subset of the test suite to execute in CI under MIRI
and boost our confidence in the copious amount of `unsafe` code in
Wasmtime's runtime.
Under MIRI's default settings, which is to use the [Stacked
Borrows][stacked] model, much of the code in `Instance` and `VMContext`
is considered invalid. Under the optional [Tree Borrows][tree] model,
however, this same code is accepted. After some [extremely helpful
discussion][discuss] on the Rust Zulip my current conclusion is that
what we're doing is not fundamentally un-sound but we need to model it
in a different way. This PR, however, uses the Tree Borrows model for
MIRI to get something onto CI sooner rather than later, and I hope to
follow this up with something that passed Stacked Borrows. Additionally
that'll hopefully make this diff smaller and easier to digest.
Given all that, the end result of this PR is to get 131 separate unit
tests executing on CI. These unit tests largely exercise the embedding
API where wasm function compilation is not involved. Some tests compile
wasm functions but don't run them, but compiling wasm through Cranelift
in MIRI is so slow that it doesn't seem worth it at this time. This does
mean that there's a pretty big hole in MIRI's test coverage, but that's
to be expected as we're a JIT compiler after all.
To get tests working in MIRI this PR uses a number of strategies:
* When platform-specific code is involved there's now `#[cfg(miri)]` for
MIRI's version. For example there's a custom-built "mmap" just for
MIRI now. Many of these are simple noops, some are `unimplemented!()`
as they shouldn't be reached, and some are slightly nontrivial
implementations such as mmaps and trap handling (for native-to-native
function calls).
* Many test modules are simply excluded via `#![cfg(not(miri))]` at the
top of the file. This excludes the entire module's worth of tests from
MIRI. Other modules have `#[cfg_attr(miri, ignore)]` annotations to
ignore tests by default on MIRI. The latter form is used in modules
where some tests work and some don't. This means that all future test
additions will need to be effectively annotated whether they work in
MIRI or not. My hope though is that there's enough precedent in the
test suite of what to do to not cause too much burden.
* A number of locations are fixed with respect to MIRI's analysis. For
example `ComponentInstance`, the component equivalent of
`wasmtime_runtime::Instance`, was actually left out from the fix for
the CVE by accident. MIRI dutifully highlighted the issues here and
I've fixed them locally. Some locations fixed for MIRI are changed to
something that looks similar but is subtly different. For example
retaining items in a `Store<T>` is now done with a Wasmtime-specific
`StoreBox<T>` type. This is because, with MIRI's analyses, moving a
`Box<T>` invalidates all pointers derived from this `Box<T>`. We don't
want these semantics, so we effectively have a custom `Box<T>` to suit
our needs in this regard.
* Some default configuration is different under MIRI. For example most
linear memories are dynamic with no guards and no space reserved for
growth. Settings such as parallel compilation are disabled. These are
applied to make MIRI "work by default" in more places ideally. Some
tests which perform N iterations of something perform fewer iterations
on MIRI to not take quite so long.
This PR is not intended to be a one-and-done-we-never-look-at-it-again
kind of thing. Instead this is intended to lay the groundwork to
continuously run MIRI in CI to catch any soundness issues. This feels,
to me, overdue given the amount of `unsafe` code inside of Wasmtime. My
hope is that over time we can figure out how to run Wasm in MIRI but
that may take quite some time. Regardless this will be adding nontrivial
maintenance work to contributors to Wasmtime. MIRI will be run on CI for
merges, MIRI will have test failures when everything else passes,
MIRI's errors will need to be deciphered by those who have probably
never run MIRI before, things like that. Despite all this to me it seems
worth the cost at this time. Just getting this running caught two
possible soundness bugs in the component implementation that could have
had a real-world impact in the future!
[stacked]: https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md
[tree]: https://perso.crans.org/vanille/treebor/
[discuss]: https://rust-lang.zulipchat.com/#narrow/stream/269128-miri/topic/Tree.20vs.20Stacked.20Borrows.20.26.20a.20debugging.20question
* Update alignment comment
This PR updates the link in Wasmtime's README at the root of the repo to
link to Cranelift's new website rather than its subdirectory; and also
updates the README in `cranelift/` to point to the new website,
https://cranelift.dev/.
We have only ever used Unknown for the stdio streams, and I don't expect
us to use it for anything else in the future, so rename it.
Set the returned filetype to character device: Closes#146.
Also, fix some warnings.
This pulls in Kerollmops/slice-group-by#20 which is necessary to get
Cranelift "clean" in MIRI with Stacked Borrows. I plan on leveraging
this in a subsequent commit to #6332 which turns on Stacked Borrows for
Wasmtime, but currently it fails due to this transitive dependency of
Cranelift, hence the update.
* wasmtime: Fix resetting stack-walking registers when entering/exiting Wasm
Fixes a regression from #6262, originally reported in
https://github.com/bytecodealliance/wasmtime-dotnet/pull/245
The issue was that we would enter Wasm and save the stack-walking registers but
never clear them after Wasm returns. Then if a host-to-host call tried to
capture a stack, we would mistakenly attempt to use those stale registers to
start the stack walk. This mistake would be caught by an assertion, triggering a
panic.
This commit fixes the issue by managing the save/restore in the
`CallThreadState` construction/drop, rather than in the old `set_prev`
method.
Co-Authored-By: Alex Crichton <alex@alexcrichton.com>
* Plumb through `VMRuntimeLimits` when capturing stack traces
This way we can differentiate between the same module loaded in different stores
and avoid leaking other stores' frames into our backtraces.
Co-Authored-By: Jamey Sharp <jsharp@fastly.com>
---------
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
Co-authored-by: Jamey Sharp <jsharp@fastly.com>
Pat Hickey
Nick Fitzgerald
Alex Crichton
theothergraham
Afonso Bordado
Juan Jose Medina
Remo Senekowitsch
Jamey Sharp
wasmtime-publish
Andrew Brown
T0b1-iOS
Dante Broggi
Dan Gohman
Chris Fallin
Pat Hickey
Trevor Elliott