cranelift

Commit Graph

Author	SHA1	Message	Date
Alex Crichton	c0bb341d95	Run some tests in MIRI on CI (#6332 ) * Run some tests in MIRI on CI This commit is an implementation of getting at least chunks of Wasmtime to run in MIRI on CI. The full test suite is not possible to run in MIRI because MIRI cannot run Cranelift-produced code at runtime (aka it doesn't support JITs). Running MIRI, however, is still quite valuable if we can manage it because it would have trivially detected GHSA-ch89-5g45-qwc7, our most recent security advisory. The goal of this PR is to select a subset of the test suite to execute in CI under MIRI and boost our confidence in the copious amount of `unsafe` code in Wasmtime's runtime. Under MIRI's default settings, which is to use the [Stacked Borrows][stacked] model, much of the code in `Instance` and `VMContext` is considered invalid. Under the optional [Tree Borrows][tree] model, however, this same code is accepted. After some [extremely helpful discussion][discuss] on the Rust Zulip my current conclusion is that what we're doing is not fundamentally un-sound but we need to model it in a different way. This PR, however, uses the Tree Borrows model for MIRI to get something onto CI sooner rather than later, and I hope to follow this up with something that passed Stacked Borrows. Additionally that'll hopefully make this diff smaller and easier to digest. Given all that, the end result of this PR is to get 131 separate unit tests executing on CI. These unit tests largely exercise the embedding API where wasm function compilation is not involved. Some tests compile wasm functions but don't run them, but compiling wasm through Cranelift in MIRI is so slow that it doesn't seem worth it at this time. This does mean that there's a pretty big hole in MIRI's test coverage, but that's to be expected as we're a JIT compiler after all. To get tests working in MIRI this PR uses a number of strategies: * When platform-specific code is involved there's now `#[cfg(miri)]` for MIRI's version. For example there's a custom-built "mmap" just for MIRI now. Many of these are simple noops, some are `unimplemented!()` as they shouldn't be reached, and some are slightly nontrivial implementations such as mmaps and trap handling (for native-to-native function calls). * Many test modules are simply excluded via `#![cfg(not(miri))]` at the top of the file. This excludes the entire module's worth of tests from MIRI. Other modules have `#[cfg_attr(miri, ignore)]` annotations to ignore tests by default on MIRI. The latter form is used in modules where some tests work and some don't. This means that all future test additions will need to be effectively annotated whether they work in MIRI or not. My hope though is that there's enough precedent in the test suite of what to do to not cause too much burden. * A number of locations are fixed with respect to MIRI's analysis. For example `ComponentInstance`, the component equivalent of `wasmtime_runtime::Instance`, was actually left out from the fix for the CVE by accident. MIRI dutifully highlighted the issues here and I've fixed them locally. Some locations fixed for MIRI are changed to something that looks similar but is subtly different. For example retaining items in a `Store<T>` is now done with a Wasmtime-specific `StoreBox<T>` type. This is because, with MIRI's analyses, moving a `Box<T>` invalidates all pointers derived from this `Box<T>`. We don't want these semantics, so we effectively have a custom `Box<T>` to suit our needs in this regard. * Some default configuration is different under MIRI. For example most linear memories are dynamic with no guards and no space reserved for growth. Settings such as parallel compilation are disabled. These are applied to make MIRI "work by default" in more places ideally. Some tests which perform N iterations of something perform fewer iterations on MIRI to not take quite so long. This PR is not intended to be a one-and-done-we-never-look-at-it-again kind of thing. Instead this is intended to lay the groundwork to continuously run MIRI in CI to catch any soundness issues. This feels, to me, overdue given the amount of `unsafe` code inside of Wasmtime. My hope is that over time we can figure out how to run Wasm in MIRI but that may take quite some time. Regardless this will be adding nontrivial maintenance work to contributors to Wasmtime. MIRI will be run on CI for merges, MIRI will have test failures when everything else passes, MIRI's errors will need to be deciphered by those who have probably never run MIRI before, things like that. Despite all this to me it seems worth the cost at this time. Just getting this running caught two possible soundness bugs in the component implementation that could have had a real-world impact in the future! [stacked]: https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md [tree]: https://perso.crans.org/vanille/treebor/ [discuss]: https://rust-lang.zulipchat.com/#narrow/stream/269128-miri/topic/Tree.20vs.20Stacked.20Borrows.20.26.20a.20debugging.20question * Update alignment comment	2 years ago
Alex Crichton	b0939f6626	Remove explicit `S` type parameters (#5275 ) * Remove explicit `S` type parameters This commit removes the explicit `S` type parameter on `Func::typed` and `Instance::get_typed_func`. Historical versions of Rust required that this be a type parameter but recent rustcs support a mixture of explicit type parameters and `impl Trait`. This removes, at callsites, a superfluous `, _` argument which otherwise never needs specification. * Fix mdbook examples	2 years ago
Alex Crichton	7a1b7cdf92	Implement RFC 11: Redesigning Wasmtime's APIs (#2897 ) Implement Wasmtime's new API as designed by RFC 11. This is quite a large commit which has had lots of discussion externally, so for more information it's best to read the RFC thread and the PR thread.	3 years ago
Alex Crichton	2697a18d2f	Redo the statically typed `Func` API (#2719 ) * Redo the statically typed `Func` API This commit reimplements the `Func` API with respect to statically typed dispatch. Previously `Func` had a `getN` and `getN_async` family of methods which were implemented for 0 to 16 parameters. The return value of these functions was an `impl Fn(..)` closure with the appropriate parameters and return values. There are a number of downsides with this approach that have become apparent over time: * The addition of `_async` doubled the API surface area (which is quite large here due to one-method-per-number-of-parameters). The [documentation of `Func`][old-docs] are quite verbose and feel "polluted" with all these getters, making it harder to understand the other methods that can be used to interact with a `Func`. * These methods unconditionally pay the cost of returning an owned `impl Fn` with a `'static` lifetime. While cheap, this is still paying the cost for cloning the `Store` effectively and moving data into the closed-over environment. * Storage of the return value into a struct, for example, always requires `Box`-ing the returned closure since it otherwise cannot be named. * Recently I had the desire to implement an "unchecked" path for invoking wasm where you unsafely assert the type signature of a wasm function. Doing this with today's scheme would require doubling (again) the API surface area for both async and synchronous calls, further polluting the documentation. The main benefit of the previous scheme is that by returning a `impl Fn` it was quite easy and ergonomic to actually invoke the function. In practice, though, examples would often have something akin to `.get0::<()>()?()?` which is a lot of things to interpret all at once. Note that `get0` means "0 parameters" yet a type parameter is passed. There's also a double function invocation which looks like a lot of characters all lined up in a row. Overall, I think that the previous design is starting to show too many cracks and deserves a rewrite. This commit is that rewrite. The new design in this commit is to delete the `getN{,_async}` family of functions and instead have a new API: impl Func { fn typed<P, R>(&self) -> Result<&Typed<P, R>>; } impl Typed<P, R> { fn call(&self, params: P) -> Result<R, Trap>; async fn call_async(&self, params: P) -> Result<R, Trap>; } This should entirely replace the current scheme, albeit by slightly losing ergonomics use cases. The idea behind the API is that the existence of `Typed<P, R>` is a "proof" that the underlying function takes `P` and returns `R`. The `Func::typed` method peforms a runtime type-check to ensure that types all match up, and if successful you get a `Typed` value. Otherwise an error is returned. Once you have a `Typed` then, like `Func`, you can either `call` or `call_async`. The difference with a `Typed`, however, is that the params/results are statically known and hence these calls can be much more efficient. This is a much smaller API surface area from before and should greatly simplify the `Func` documentation. There's still a problem where `Func::wrapN_async` produces a lot of functions to document, but that's now the sole offender. It's a nice benefit that the statically-typed-async verisons are now expressed with an `async` function rather than a function-returning-a-future which makes it both more efficient and easier to understand. The type `P` and `R` are intended to either be bare types (e.g. `i32`) or tuples of any length (including 0). At this time `R` is only allowed to be `()` or a bare `i32`-style type because multi-value is not supported with a native ABI (yet). The `P`, however, can be any size of tuples of parameters. This is also where some ergonomics are lost because instead of `f(1, 2)` you now have to write `f.call((1, 2))` (note the double-parens). Similarly `f()` becomes `f.call(())`. Overall I feel that this is a better tradeoff than before. While not universally better due to the loss in ergonomics I feel that this design is much more flexible in terms of what you can do with the return value and also understanding the API surface area (just less to take in). [old-docs]: https://docs.rs/wasmtime/0.24.0/wasmtime/struct.Func.html#method.get0 * Rename Typed to TypedFunc * Implement multi-value returns through `Func::typed` * Fix examples in docs * Fix some more errors * More test fixes * Rebasing and adding `get_typed_func` * Updating tests * Fix typo * More doc tweaks * Tweak visibility on `Func::invoke` * Fix tests again	4 years ago
Alex Crichton	a277cf5ee4	Store `WasmFuncType` in `FuncType` (#2365 ) This commit updates `wasmtime::FuncType` to exactly store an internal `WasmFuncType` from the cranelift crates. This allows us to remove a translation layer when we are given a `FuncType` and want to get an internal cranelift type out as a result. The other major change from this commit was changing the constructor and accessors of `FuncType` to be iterator-based instead of exposing implementation details.	4 years ago
Yury Delendik	15c68f2cc1	Disconnects `Store` state fields from `Compiler` (#1761 ) * Moves CodeMemory, VMInterrupts and SignatureRegistry from Compiler * CompiledModule holds CodeMemory and GdbJitImageRegistration * Store keeps track of its JIT code * Makes "jit_int.rs" stuff Send+Sync * Adds the threads example.	4 years ago
Dan Gohman	9364eb1d98	Refactor (#1524 ) * Compute instance exports on demand. Instead having instances eagerly compute a Vec of Externs, and bumping the refcount for each Extern, compute Externs on demand. This also enables `Instance::get_export` to avoid doing a linear search. This also means that the closure returned by `get0` and friends now holds an `InstanceHandle` to dynamically hold the instance live rather than being scoped to a lifetime. * Compute module imports and exports on demand too. And compute Extern::ty on demand too. * Add a utility function for computing an ExternType. * Add a utility function for looking up a function's signature. * Add a utility function for computing the ValType of a Global. * Rename wasmtime_environ::Export to EntityIndex. This helps differentiate it from other Export types in the tree, and describes what it is. * Fix a typo in a comment. * Simplify module imports and exports. * Make `Instance::exports` return the export names. This significantly simplifies the public API, as it's relatively common to need the names, and this avoids the need to do a zip with `Module::exports`. This also changes `ImportType` and `ExportType` to have public members instead of private members and accessors, as I find that simplifies the usage particularly in cases where there are temporary instances. * Remove `Instance::module`. This doesn't quite remove `Instance`'s `module` member, it gets a step closer. * Use a InstanceHandle utility function. * Don't consume self in the `Func::get` methods. Instead, just create a closure containing the instance handle and the export for them to call. Use `ExactSizeIterator` to avoid needing separate `num_` methods. Rename `Extern::func()` etc. to `into_func()` etc. * Revise examples to avoid using `nth`. * Add convenience methods to instance for getting specific extern types. * Use the convenience functions in more tests and examples. * Avoid cloning strings for `ImportType` and `ExportType`. * Remove more obviated clone() calls. * Simplify `Func`'s closure state. * Make wasmtime::Export's fields private. This makes them more consistent with ExportType. * Fix compilation error. * Make a lifetime parameter explicit, and use better lifetime names. Instead of 'me, use 'instance and 'module to make it clear what the lifetime is. * More lifetime cleanups.	5 years ago
Alex Crichton	4c82da440a	Move most wasmtime tests into one test suite (#1544 ) * Move most wasmtime tests into one test suite This commit moves most wasmtime tests into a single test suite which gets compiled into one executable instead of having lots of test executables. The goal here is to reduce disk space on CI, and this should be achieved by having fewer executables which means fewer copies of `libwasmtime.rlib` linked across binaries on the system. More importantly though this means that DWARF debug information should only be in one executable rather than duplicated across many. * Share more build caches Globally set `RUSTFLAGS` to `-Dwarnings` instead of individually so all build steps share the same value. * Allow some dead code in cranelift-codegen Prevents having to fix all warnings for all possible feature combinations, only the main ones which come up. * Update some debug file paths	5 years ago
Alex Crichton	afd980b4f6	Refactor the internals of `Func` to remove layers of indirection (#1363 ) * Remove `WrappedCallable` indirection At this point `Func` has evolved quite a bit since inception and the `WrappedCallable` trait I don't believe is needed any longer. This should help clean up a few entry points by having fewer traits in play. * Remove the `Callable` trait This commit removes the `wasmtime::Callable` trait, changing the signature of `Func::new` to take an appropriately typed `Fn`. Additionally the function now always takes `&Caller` like `Func::wrap` optionally can, to empower `Func::new` to have the same capabilities of `Func::wrap`. * Add a test for an already-fixed issue Closes #849 * rustfmt * Update more locations for `Callable` * rustfmt * Remove a stray leading borrow * Review feedback * Remove unneeded `wasmtime_call_trampoline` shim	5 years ago
Alex Crichton	16804673a2	Support parsing the text format in `wasmtime` crate (#813 ) * Support parsing the text format in `wasmtime` crate This commit adds support to the `wasmtime::Module` type to parse the text format. This is often quite convenient to support in testing or tinkering with the runtime. Additionally the `wat` parser is pretty lightweight and easy to add to builds, so it's relatively easy for us to support as well! The exact manner that this is now supported comes with a few updates to the existing API: * A new optional feature of the `wasmtime` crate, `wat`, has been added. This is enabled by default. * The `Module::new` API now takes `impl AsRef<[u8]>` instead of just `&[u8]`, and when the `wat` feature is enabled it will attempt to interpret it either as a wasm binary or as the text format. Note that this check is quite cheap since you just check the first byte. * A `Module::from_file` API was added as a convenience to parse a file from disk, allowing error messages for `.wat` files on disk to be a bit nicer. APIs like `Module::new_unchecked` and `Module::validate` remain unchanged, they require the binary format to be called. The intention here is to make this as convenient as possible for new developers of the `wasmtime` crate. By changing the default behavior though this has ramifications such as, for example, supporting the text format implicitly through the C API now. * Handle review comments * Update more tests to avoid usage of `wat` crate * Go back to unchecked for now in wasm_module_new Looks like C# tests rely on this?	5 years ago
Alex Crichton	e5afdd2ede	Document the `wasmtime::Instance` APIs (#814 ) * Document the `wasmtime::Instance` APIs This documents oddities like the import list and export list and how to match them all up. Addtionally this largely just expands all the docs related to `Instance` to get filled out. This also moves the `set_signal_handler` functions into platform-specific modules in order to follow Rust idioms about how to expose platform-specific information. Additionally the methods are marked `unsafe` because I figure anything having to do with signal handling is `unsafe` inherently. I don't actually know what these functions do, so they're currently still undocumented. * Fix build of python bindings * Fix some rebase conflicts	5 years ago
Alex Crichton	420dcd76fd	Don't require `Store` in `Instance` constructor (#810 ) * Don't require `Store` in `Instance` constructor This can be inferred from the `Module` argument. Additionally add a `store` accessor to an `Instance` in case it's needed to instantiate another `Module`. cc #708 * Update more constructors * Fix a doctest * Don't ignore store in `wasm_instance_new` * Run rustfmt	5 years ago
Alex Crichton	6571fb8f4f	Remove `HostRef` from the `wasmtime` public API (#788 ) * Remove `HostRef` from the `wasmtime` public API This commit removes all remaining usages of `HostRef` in the public API of the `wasmtime` crate. This involved a number of API decisions such as: * None of `Func`, `Global`, `Table`, or `Memory` are wrapped in `HostRef` * All of `Func`, `Global`, `Table`, and `Memory` implement `Clone` now. * Methods called `type` are renamed to `ty` to avoid typing `r#type`. * Methods requiring mutability for external items now no longer require mutability. The mutable reference here is sort of a lie anyway since the internals are aliased by the underlying module anyway. This affects: * `Table::set` * `Table::grow` * `Memory::grow` * `Instance::set_signal_handler` * The `Val::FuncRef` type is now no longer automatically coerced to `AnyRef`. This is technically a breaking change which is pretty bad, but I'm hoping that we can live with this interim state while we sort out the `AnyRef` story in general. * The implementation of the C API was refactored and updated in a few locations to account for these changes: * Accessing the exports of an instance are now cached to ensure we always hand out the same `HostRef` values. * `wasm__t` for external values no longer have internal cache, instead they all wrap `wasm_external_t` and have an unchecked accessor for the underlying variant (since the type is proof that it's there). This makes casting back and forth much more trivial. This is all related to #708 and while there's still more work to be done in terms of documentation, this is the major bulk of the rest of the implementation work on #708 I believe. More API updates * Run rustfmt * Fix a doc test * More test updates	5 years ago
Alex Crichton	41780fb1a6	Handle same-named imports with different signatures This commit fixes the `wasmtime::Instance` instantiation API when imports have the same name but might be imported under different types. This is handled in the API by listing imports as a list instead of as a name map, but they were interpreted as a name map under the hood causing collisions. This commit now keeps track of the index used to define each import, and the index is passed through in the `Resolver`. Existing implementaitons of `Resolver` all ignore this, but the API now uses it exclusivley to match up `Extern` definitions to imports.	5 years ago

8 Commits (c736c712e53d1e352a8c182507656e22413661bc)