cranelift

Commit Graph

Author	SHA1	Message	Date
Alex Crichton	e68aa99588	Implement the memory64 proposal in Wasmtime (#3153 ) * Implement the memory64 proposal in Wasmtime This commit implements the WebAssembly [memory64 proposal][proposal] in both Wasmtime and Cranelift. In terms of work done Cranelift ended up needing very little work here since most of it was already prepared for 64-bit memories at one point or another. Most of the work in Wasmtime is largely refactoring, changing a bunch of `u32` values to something else. A number of internal and public interfaces are changing as a result of this commit, for example: * Acessors on `wasmtime::Memory` that work with pages now all return `u64` unconditionally rather than `u32`. This makes it possible to accommodate 64-bit memories with this API, but we may also want to consider `usize` here at some point since the host can't grow past `usize`-limited pages anyway. * The `wasmtime::Limits` structure is removed in favor of minimum/maximum methods on table/memory types. * Many libcall intrinsics called by jit code now unconditionally take `u64` arguments instead of `u32`. Return values are `usize`, however, since the return value, if successful, is always bounded by host memory while arguments can come from any guest. * The `heap_addr` clif instruction now takes a 64-bit offset argument instead of a 32-bit one. It turns out that the legalization of `heap_addr` already worked with 64-bit offsets, so this change was fairly trivial to make. * The runtime implementation of mmap-based linear memories has changed to largely work in `usize` quantities in its API and in bytes instead of pages. This simplifies various aspects and reflects that mmap-memories are always bound by `usize` since that's what the host is using to address things, and additionally most calculations care about bytes rather than pages except for the very edge where we're going to/from wasm. Overall I've tried to minimize the amount of `as` casts as possible, using checked `try_from` and checked arithemtic with either error handling or explicit `unwrap()` calls to tell us about bugs in the future. Most locations have relatively obvious things to do with various implications on various hosts, and I think they should all be roughly of the right shape but time will tell. I mostly relied on the compiler complaining that various types weren't aligned to figure out type-casting, and I manually audited some of the more obvious locations. I suspect we have a number of hidden locations that will panic on 32-bit hosts if 64-bit modules try to run there, but otherwise I think we should be generally ok (famous last words). In any case I wouldn't want to enable this by default naturally until we've fuzzed it for some time. In terms of the actual underlying implementation, no one should expect memory64 to be all that fast. Right now it's implemented with "dynamic" heaps which have a few consequences: * All memory accesses are bounds-checked. I'm not sure how aggressively Cranelift tries to optimize out bounds checks, but I suspect not a ton since we haven't stressed this much historically. * Heaps are always precisely sized. This means that every call to `memory.grow` will incur a `memcpy` of memory from the old heap to the new. We probably want to at least look into `mremap` on Linux and otherwise try to implement schemes where dynamic heaps have some reserved pages to grow into to help amortize the cost of `memory.grow`. The memory64 spec test suite is scheduled to now run on CI, but as with all the other spec test suites it's really not all that comprehensive. I've tried adding more tests for basic things as I've had to implement guards for them, but I wouldn't really consider the testing adequate from just this PR itself. I did try to take care in one test to actually allocate a 4gb+ heap and then avoid running that in the pooling allocator or in emulation because otherwise that may fail or take excessively long. [proposal]: https://github.com/WebAssembly/memory64/blob/master/proposals/memory64/Overview.md * Fix some tests * More test fixes * Fix wasmtime tests * Fix doctests * Revert to 32-bit immediate offsets in `heap_addr` This commit updates the generation of addresses in wasm code to always use 32-bit offsets for `heap_addr`, and if the calculated offset is bigger than 32-bits we emit a manual add with an overflow check. * Disable memory64 for spectest fuzzing * Fix wrong offset being added to heap addr * More comments! * Clarify bytes/pages	3 years ago
Pat Hickey	8b4bdf92e2	make ResourceLimiter operate on Store data; add hooks for entering and exiting native code (#2952 ) * wasmtime_runtime: move ResourceLimiter defaults into this crate In preparation of changing wasmtime::ResourceLimiter to be a re-export of this definition, because translating between two traits was causing problems elsewhere. * wasmtime: make ResourceLimiter a re-export of wasmtime_runtime::ResourceLimiter * refactor Store internals to support ResourceLimiter as part of store's data * add hooks for entering and exiting native code to Store * wasmtime-wast, fuzz: changes to adapt ResourceLimiter API * fix tests * wrap calls into wasm with entering/exiting exit hooks as well * the most trivial test found a bug, lets write some more * store: mark some methods as #[inline] on Store, StoreInner, StoreInnerMost Co-authored-By: Alex Crichton <alex@alexcrichton.com> * improve tests for the entering/exiting native hooks Co-authored-by: Alex Crichton <alex@alexcrichton.com>	3 years ago
Alex Crichton	7a1b7cdf92	Implement RFC 11: Redesigning Wasmtime's APIs (#2897 ) Implement Wasmtime's new API as designed by RFC 11. This is quite a large commit which has had lots of discussion externally, so for more information it's best to read the RFC thread and the PR thread.	3 years ago
Alex Crichton	8384f3a347	Bring back `Module::deserialize` (#2858 ) * Bring back `Module::deserialize` I thought I was being clever suggesting that `Module::deserialize` was removed from #2791 by funneling all module constructors into `Module::new`. As our studious fuzzers have found, though, this means that `Module::new` is not safe currently to pass arbitrary user-defined input into. Now one might pretty reasonable expect to be able to do that, however, being a WebAssembly engine and all. This PR as a result separates the `deserialize` part of `Module::new` back into `Module::deserialize`. This means that binary blobs created with `Module::serialize` and `Engine::precompile_module` will need to be passed to `Module::deserialize` to "rehydrate" them back into a `Module`. This restores the property that it should be safe to pass arbitrary input to `Module::new` since it's always expected to be a wasm module. This also means that fuzzing will no longer attempt to fuzz `Module::deserialize` which isn't something we want to do anyway. * Fix an example * Mark `Module::deserialize` as `unsafe`	4 years ago
Peter Huene	f12b4c467c	Add resource limiting to the Wasmtime API. (#2736 ) * Add resource limiting to the Wasmtime API. This commit adds a `ResourceLimiter` trait to the Wasmtime API. When used in conjunction with `Store::new_with_limiter`, this can be used to monitor and prevent WebAssembly code from growing linear memories and tables. This is particularly useful when hosts need to take into account host resource usage to determine if WebAssembly code can consume more resources. A simple `StaticResourceLimiter` is also included with these changes that will simply limit the size of linear memories or tables for all instances created in the store based on static values. * Code review feedback. * Implemented `StoreLimits` and `StoreLimitsBuilder`. * Moved `max_instances`, `max_memories`, `max_tables` out of `Config` and into `StoreLimits`. * Moved storage of the limiter in the runtime into `Memory` and `Table`. * Made `InstanceAllocationRequest` use a reference to the limiter. * Updated docs. * Made `ResourceLimiterProxy` generic to remove a level of indirection. * Fixed the limiter not being used for `wasmtime::Memory` and `wasmtime::Table`. * Code review feedback and bug fix. * `Memory::new` now returns `Result<Self>` so that an error can be returned if the initial requested memory exceeds any limits placed on the store. * Changed an `Arc` to `Rc` as the `Arc` wasn't necessary. * Removed `Store` from the `ResourceLimiter` callbacks. Custom resource limiter implementations are free to capture any context they want, so no need to unnecessarily store a weak reference to `Store` from the proxy type. * Fixed a bug in the pooling instance allocator where an instance would be leaked from the pool. Previously, this would only have happened if the OS was unable to make the necessary linear memory available for the instance. With these changes, however, the instance might not be created due to limits placed on the store. We now properly deallocate the instance on error. * Added more tests, including one that covers the fix mentioned above. * Code review feedback. * Add another memory to `test_pooling_allocator_initial_limits_exceeded` to ensure a partially created instance is successfully deallocated. * Update some doc comments for better documentation of `Store` and `ResourceLimiter`.	4 years ago
Peter Huene	d1313b1291	Code review feedback. * Move `Module::compile` to `Engine::precompile_module`. * Remove `Module::deserialize` method. * Make `Module::serialize` the same format as `Engine::precompile_module`. * Make `Engine::precompile_module` return a `Vec<u8>`. * Move the remaining serialization-related code to `serialization.rs`.	4 years ago
Peter Huene	54c07d8f16	Implement shared host functions. (#2625 ) * Implement defining host functions at the Config level. This commit introduces defining host functions at the `Config` rather than with `Func` tied to a `Store`. The intention here is to enable a host to define all of the functions once with a `Config` and then use a `Linker` (or directly with `Store::get_host_func`) to use the functions when instantiating a module. This should help improve the performance of use cases where a `Store` is short-lived and redefining the functions at every module instantiation is a noticeable performance hit. This commit adds `add_to_config` to the code generation for Wasmtime's `Wasi` type. The new method adds the WASI functions to the given config as host functions. This commit adds context functions to `Store`: `get` to get a context of a particular type and `set` to set the context on the store. For safety, `set` cannot replace an existing context value of the same type. `Wasi::set_context` was added to set the WASI context for a `Store` when using `Wasi::add_to_config`. * Add `Config::define_host_func_async`. * Make config "async" rather than store. This commit moves the concept of "async-ness" to `Config` rather than `Store`. Note: this is a breaking API change for anyone that's already adopted the new async support in Wasmtime. Now `Config::new_async` is used to create an "async" config and any `Store` associated with that config is inherently "async". This is needed for async shared host functions to have some sanity check during their execution (async host functions, like "async" `Func`, need to be called with the "async" variants). * Update async function tests to smoke async shared host functions. This commit updates the async function tests to also smoke the shared host functions, plus `Func::wrap0_async`. This also changes the "wrap async" method names on `Config` to `wrap$N_host_func_async` to slightly better match what is on `Func`. * Move the instance allocator into `Engine`. This commit moves the instantiated instance allocator from `Config` into `Engine`. This makes certain settings in `Config` no longer order-dependent, which is how `Config` should ideally be. This also removes the confusing concept of the "default" instance allocator, instead opting to construct the on-demand instance allocator when needed. This does alter the semantics of the instance allocator as now each `Engine` gets its own instance allocator rather than sharing a single one between all engines created from a configuration. * Make `Engine::new` return `Result`. This is a breaking API change for anyone using `Engine::new`. As creating the pooling instance allocator may fail (likely cause is not enough memory for the provided limits), instead of panicking when creating an `Engine`, `Engine::new` now returns a `Result`. * Remove `Config::new_async`. This commit removes `Config::new_async` in favor of treating "async support" as any other setting on `Config`. The setting is `Config::async_support`. * Remove order dependency when defining async host functions in `Config`. This commit removes the order dependency where async support must be enabled on the `Config` prior to defining async host functions. The check is now delayed to when an `Engine` is created from the config. * Update WASI example to use shared `Wasi::add_to_config`. This commit updates the WASI example to use `Wasi::add_to_config`. As only a single store and instance are used in the example, it has no semantic difference from the previous example, but the intention is to steer users towards defining WASI on the config and only using `Wasi::add_to_linker` when more explicit scoping of the WASI context is required.	4 years ago
Alex Crichton	dccaa64962	Add knobs to limit memories/tables in a `Store` Fuzzing has turned up that module linking can create large amounts of tables and memories in addition to instances. For example if N instances are allowed and M tables are allowed per-instance, then currently wasmtime allows MxN tables (which is quite a lot). This is causing some wasm-smith-generated modules to exceed resource limits while fuzzing! This commits adds corresponding `max_tables` and `max_memories` functions to sit alongside the `max_instances` configuration. Additionally fuzzing now by default configures all of these to a somewhat low value to avoid too much resource usage while fuzzing.	4 years ago
Alex Crichton	8748cf5bd3	Add an instance limit to `Config` (#2593 ) * Add an instance limit to `Config` This commit adds a new parameter to `Config` which limits the number of instances that can be created within a store connected to that `Config`. The intention here is to provide a default safeguard against module-linking modules that recursively create too many instances. * Update crates/c-api/include/wasmtime.h Co-authored-by: Peter Huene <peter@huene.dev> Co-authored-by: Peter Huene <peter@huene.dev>	4 years ago
Alex Crichton	703762c49e	Update support for the module linking proposal This commit updates the various tooling used by wasmtime which has new updates to the module linking proposal. This is done primarily to sync with WebAssembly/module-linking#26. The main change implemented here is that wasmtime now supports creating instances from a set of values, nott just from instantiating a module. Additionally subtyping handling of modules with respect to imports is now properly handled by desugaring two-level imports to imports of instances. A number of small refactorings are included here as well, but most of them are in accordance with the changes to `wasmparser` and the updated binary format for module linking.	4 years ago
Alex Crichton	9ac7d01288	Implement the module linking alias section (#2451 ) This commit is intended to do almost everything necessary for processing the alias section of module linking. Most of this is internal refactoring, the highlights being: * Type contents are now stored separately from a `wasmtime_env::Module`. Given that modules can freely alias types and have them used all over the place, it seemed best to have one canonical location to type storage which everywhere else points to (with indices). A new `TypeTables` structure is produced during compilation which is shared amongst all member modules in a wasm blob. * Instantiation is heavily refactored to account for module linking. The main gotcha here is that imports are now listed as "initializers". We have a sort of pseudo-bytecode-interpreter which interprets the initialization of a module. This is more complicated than just matching imports at this point because in the module linking proposal the module, alias, import, and instance sections may all be interleaved. This means that imports aren't guaranteed to show up at the beginning of the address space for modules/instances. Otherwise most of the changes here largely fell out from these two design points. Aliases are recorded as initializers in this scheme. Copying around type information and/or just knowing type information during compilation is also pretty easy since everything is just a pointer into a `TypeTables` and we don't have to actually copy any types themselves. Lots of various refactorings were necessary to accomodate these changes. Tests are hoped to cover a breadth of functionality here, but not necessarily a depth. There's still one more piece of the module linking proposal missing which is exporting instances/modules, which will come in a future PR. It's also worth nothing that there's one large TODO which isn't implemented in this change that I plan on opening an issue for. With module linking when a set of modules comes back from compilation each modules has all the trampolines for the entire set of modules. This is quite a lot of duplicate trampolines across module-linking modules. We'll want to refactor this at some point to instead have only one set of trampolines per set of module linking modules and have them shared from there. I figured it was best to separate out this change, however, since it's purely related to resource usage, and doesn't impact non-module-linking modules at all. cc #2094	4 years ago
Alex Crichton	73cda83548	Propagate module-linking types to `wasmtime` (#2115 ) This commit adds lots of plumbing to get the type section from the module linking proposal plumbed all the way through to the `wasmtime` crate and the `wasmtime-c-api` crate. This isn't all that useful right now because Wasmtime doesn't support imported/exported modules/instances, but this is all necessary groundwork to getting that exported at some point. I've added some light tests but I suspect the bulk of the testing will come in a future commit. One major change in this commit is that `SignatureIndex` no longer follows type type index space in a wasm module. Instead a new `TypeIndex` type is used to track that. Function signatures, still indexed by `SignatureIndex`, are then packed together tightly.	4 years ago
Alex Crichton	77827a48a9	Start compiling module-linking modules (#2093 ) This commit is intended to be the first of many in implementing the module linking proposal. At this time this builds on #2059 so it shouldn't land yet. The goal of this commit is to compile bare-bones modules which use module linking, e.g. those with nested modules. My hope with module linking is that almost everything in wasmtime only needs mild refactorings to handle it. The goal is that all per-module structures are still per-module and at the top level there's just a `Vec` containing a bunch of modules. That's implemented currently where `wasmtime::Module` contains `Arc<[CompiledModule]>` and an index of which one it's pointing to. This should enable serialization/deserialization of any module in a nested modules scenario, no matter how you got it. Tons of features of the module linking proposal are missing from this commit. For example instantiation flat out doesn't work, nor does import/export of modules or instances. That'll be coming as future commits, but the purpose here is to start laying groundwork in Wasmtime for handling lots of modules in lots of places.	4 years ago

13 Commits (46ade3dab30e8dbf181245bc8877d519ae77bf70)