* Move `wasmtime explore` behind a Cargo feature
Enable this Cargo feature by default, but enable building the CLI
without the `explore` subcommand.
* Move the `wast` subcommand behind a Cargo feature
* Move support for `wat` behind a CLI feature
This was already conditional in crates such as `wasmtime` and this makes
it an optional dependency of the CLI as well.
* Move CLI cache support behind a Cargo feature
Additionally refactor `wasmtime-cli-flags` to not unconditionally pull
in caching support by removing its `default` feature and appropriately
enabling it from the CLI.
* Move `rayon` behind an optional feature
* Move `http-body-util` dependency behind `serve` feature
* Add a Cargo feature for compiling out log statements
This sets the static features of `log` and `tracing` to statically
remove all log statements from the binary to cut down on binary size.
* Move logging support behind a Cargo feature
Enables statically removing logging support in addition to the previous
compiling out log statements themselves.
* Move demangling support behind a Cargo feature
* Enable building the CLI without cranelift
Compile out the `compile` subcommand for example.
* Gate all profiling support behind one feature flag
This commit moves vtune/jitdump support behind a single `profiling`
feature flag that additionally includes the guest profiler dependencies
now too.
* Move support for core dumps behind a feature flag
* Move addr2line behind a feature
* Fix rebase
* Document cargo features and a minimal build
* Tidy up the source a bit
* Rename compile-out-logging
* Document disabling logging
* Note the host architecture as well
* Fix tests
* Fix tests
* Mention debuginfo stripping
* Fix CI configuration for checking features
* Fix book tests
* Update lock file after rebase
* Enable coredump feature by default
* Remove usage of `BTreeMap` for compiler flags
No need for a datastructure here really, a simple list with static
strings works alright.
* Fix winch compile and a warning
* Fix test compile
* Enable threads, multi-memory, and relaxed-simd by default
This commit enables by default three proposals that were advanced to
stage 4 in the proposal process at last week's in-person CG meeting.
These proposals are:
* `threads` - atomic instructions and shared memory
* `multi-memory` - the ability to have more than one linear memory
* `relaxed-simd` - new simd instructions that may differ across
platforms
These proposals are now all enabled by default in Wasmtime. Note that
they can still be selectively disabled via `Config` options.
* Fix an x64-specific test
* PCC: add facts to global values, parse and print them. No verification yet.
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
* PCC: propagate facts on GV loads and check them.
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
* PCC: support propagating facts on iteratively-elaborated GVs as well.
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
* PCC: fix up Wasmtime uses of GVs after refactors to memflags handling.
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
* PCC: working end-to-end for static memories!
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
* PCC: add toplevel Wasmtime option `-C enable-pcc=y`.
* Fix filetests build.
* Review feedback, and blessed test updates due to GV legalization changes.
---------
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
Some notable changes here are:
* The `wasm-tools` crates have enabled the `relaxed-simd`, `threads`,
and `multi-memory` proposals by default. For now I've left these
disabled-by-default in Wasmtime to get enabled in a future PR.
* The `wast` crate has support for parsing `thread` and `wait`
constructs from the `threads` proposal for WebAssembly. They're left
unimplemented for now and return errors. This will get filled in in a
future update.
* Support set_fuel in store APIs
Fixes: https://github.com/bytecodealliance/wasmtime/issues/5109
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* rename set_fuel to reset_fuel
To make it more clear that consumed fuel is being reset.
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* update out of date documentation for fuel in C API
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
---------
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* Introduce StackMemory and StackMemoryCreator
This allows custom implementations of stack memory to be plugged into
the async functionality for wasmtime. Currently, stacks are always
mmapped, and this custom allocator allows embedders to use any memory
they would like.
The new APIs are also exposed in the C api.
This has no effect on windows, as our hands are tied with fibers there.
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* Add test for custom host memory
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* fix allocator test
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* Address review comments
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* Fix lint warnings
prtest:full
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* fix windows lint warning
prtest:full
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
---------
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* Update syntax for `trappable_error_type` in `bindgen!`
Following through on a suggestion from #7152 this makes the syntax a bit
more consistent with the rest of `bindgen!` today. This additionally
updates the documentation on the `bindgen!` macro itself.
* Update parsing to afford for versions
* WIP veriwasm 2.0
Co-Authored-By: Chris Fallin <cfallin@fastly.com>
* PCC: successfully parse some simple facts.
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
* PCC: plumb facts through VCode and add framework on LowerBackend to check them.
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
* PCC: code is carrying some proofs! Very simple test-case.
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
* PCC: add a `safe` flag for checked memory accesses.
* PCC: add pretty-printing of facts to CLIF output.
* PCC: misc. cleanups.
* PCC: lots of cleanup.
* Post-rebase fixups and some misc. fixes.
* Add serde traits to facts.
* PCC: add succeed and fail tests.
* Review feedback: rename `safe` memflag to `checked`.
* Review feedback.
---------
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
To prepare this commit I've run `WIT_REQUIRE_SEMICOLONS=1 ./ci/run-tests.sh`
locally which configures `wit-parser` to generate an error for missing
semicolons in WIT files. This led me to add quite a few semicolons in
quite a few places in what is going to be the first of a few batches of
semicolons.
CI checks for this cannot be added just yet because the wasi-nn spec is
a submodule which needs to be updated with semicolons before this
repository can require semicolons. Nevertheless that doesn't stop us
from using semicolons in the meantime (yay gradual rollout of changes!)
so I figure this would be good to get in sooner rather than later.
* Rename `Host*` things to avoid name conflicts with bindings.
* Update to the latest resource-enabled wit files.
* Adapting the code to the new bindings.
* Update wasi-http to the resource-enabled wit deps.
* Start adapting the wasi-http code to the new bindings.
* Make `get_directories` always return new owned handles.
* Simplify the `poll_one` implementation.
* Update the wasi-preview1-component-adapter.
FIXME: temporarily disable wasi-http tests.
Add logging to the cli world, since stderr is now a reseource that
can only be claimed once.
* Work around a bug hit by poll-list, fix a bug in poll-one.
* Comment out `test_fd_readwrite_invalid_fd`, which panics now.
* Fix a few FIXMEs.
* Use `.as_ref().trapping_unwrap()` instead of `TrappingUnwrapRef`.
* Use `drop_in_place`.
* Remove `State::with_mut`.
* Remove the `RefCell` around the `State`.
* Update to wit-bindgen 0.12.
* Update wasi-http to use resources for poll and I/O.
This required making incoming-body and outgoing-body resourrces too, to
work with `push_input_stream_child` and `push_output_stream_child`.
* Re-enable disabled tests, remove logging from the worlds.
* Remove the `poll_list` workarounds that are no longer needed.
* Remove logging from the adapter.
That said, there is no replacement yet, so add a FIXME comment.
* Reenable a test that now passes.
* Remove `.descriptors_mut` and use `with_descriptors_mut` instead.
Replace `.descriptors()` and `.descriptors_mut()` with functions
that take closures, which limits their scope, to prevent them from
invalid aliasing.
* Implement dynamic borrow checking for descriptors.
* Add a cargo-vet audit for wasmtime-wmemcheck.
* Update cargo vet for wit-bindgen 0.12.
* Cut down on duplicate sync/async resource types (#1)
* Allow calling `get-directories` more than once (#2)
For now `Clone` the directories into new descriptor slots as needed.
* Start to lift restriction of stdio only once (#3)
* Start to lift restriction of stdio only once
This commit adds new `{Stdin,Stdout}Stream` traits which take over the
job of the stdio streams in `WasiCtxBuilder` and `WasiCtx`. These traits
bake in the ability to create a stream at any time to satisfy the API
of `wasi:cli`. The TTY functionality is folded into them as while I was
at it.
The implementation for stdin is relatively trivial since the stdin
implementation already handles multiple streams reading it. Built-in
impls of the `StdinStream` trait are also provided for helper types in
`preview2::pipe` which resulted in the implementation of
`MemoryInputPipe` being updated to support `Clone` where all clones read
the same original data.
* Get tests building
* Un-ignore now-passing test
* Remove unneeded argument from `WasiCtxBuilder::build`
* Fix tests
* Remove some workarounds
Stdio functions can now be called multiple times.
* If `poll_oneoff` fails part-way through, clean up properly.
Fix the `Drop` implementation for pollables to only drop the pollables
that have been successfully added to the list.
This fixes the poll_oneoff_files failure and removes a FIXME.
---------
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
* Add memory protection keys (MPK)
In order to use MPK on an x86_64 Linux system, we need access to the
underlying `pkey_*` system calls (`sys`), control of the x86 PKRU
register (`pkru`), and a way of determining if MPK is even supported
(`is_supported`). These various parts are wrapped in a `ProtectionKey`
abstraction along with a `ProtectionMask` that can be used `allow` the
CPU to access protected regions.
* Integrate MPK into the pooling allocator
This change adds "stripes" to the pooling allocator's `MemoryPool`. Now,
when requesting a slot in which to instantiate, the user (i.e.,
`InstanceAllocationRequest`) will be transparently assigned to one of
the stripes, each of which is associated with a protection key. The user
can also request a specific protection key to use, which will override
the original "find me a slot logic".
This has implications for how instances get allocated: once a store is
assigned a protection key, it will only allocate requests with that key,
limiting how many slots it has access to. E.g., if 15 keys are active,
the store can only ever access 1/15th of the slots.
This change also includes a tri-bool configuration field,
`memory_protection_keys`, which is disabled by default for the time
being.
* Address review comments
This is a rollup of 43 commits addressing review comments of various
kinds: bug fixes, refactorings, documentation improvements, etc. It also
ensures that CI runs all checks. A big thanks to @fitzgen and
@alexcrichton for the review!
prtest:full
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
---------
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
* Wasmtime: Move `Global` and `Table` to `externals` submodules
Just mechanical code motion, not functional changes.
* Wasmtime: Add `hash_key` methods for `Func` and `Table`
* Rename `wasmtime::Func::caller_checked_func_ref` to `wasmtime::Func::vm_func_ref`
We removed the "caller checked" part of the `VMFuncRef` type's name a while ago,
so update this method to be in line with that.
No functional changes, just mechanical renaming.
* cargo fmt
* Fix doc link
This commit makes it so that the library type for core dumps is serializable
into the standard binary format for core dumps.
Additionally, this commit makes it so that we use the library type for
generating core dumps in the CLI. We previously were using a one-off
implementation of core dump generation that only had backtrace information and
no instances, modules, globals, or memories included. The library type has all
that information, so the core dumps produced by our CLI will both be more
featureful and be generated by shared code paths going forward.
Along the way, implementing all this required some new helper methods sprinkled
throughout `wasmtime` and `wasmtime-runtime`:
* `wasmtime::Instance::module`: get the module that a `wasmtime::Instance` is an
instance of. This is public, since it seems generally useful. This involved
adding a new return value from `ModuleRegistry::register_module` that is an
identifier that can be used to recover a reference to the registered module.
* `wasmtime::Instance::all_{globals,memories}`: get the full global/memory index
space. I made these `pub(crate)` out of caution. I don't think we want to commit
to exposing non-exported things in the public API, even if we internally need
them for debugging-related features like core dumps. These also needed
corresponding methods inside `wasmtime-runtime`.
* `wasmtime::{Global,Memory}::hash_key`: this was needed to work around the fact
that each time you call `{Global,Memory}::from_wasmtime`, it creates a new
entry in the `StoreData` and so you can get duplicates. But we need to key some
hash maps on globals and memories when constructing core dumps, so we can't
treat the underlying `Stored<T>` as a hash key because it isn't stable across
duplicate `StoreData` entries. So we have these new methods. They are only
`pub(crate)`, are definitely implementation details, and aren't exposed in the
public API.
* `wasmtime::FrameInfo::module`: Each frame in a backtrace now keeps a handle to
its associated module instead of just the name. This is publicly exposed
because it seems generally useful. This means I also deprecated
`wasmtime::FrameInfo::module_name` since you can now instead do
`frame.module().name()` to get that exact same info. I updated callers inside
the repo.
* Include Wasm-defined globals and memories in core dumps
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
* Narrow scope of unsafe block
* Add missing skip-miri attribute for test that calls into Wasm
---------
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
* Support resource maps in `component::bindgen!`
This commit adds support to `component::bindgen!` to specify resource
types using the `with` key of the macro. This can be used to configure
the `T` of `Resource<T>` to use a preexisting type rather than
unconditionally generating a new empty enum to have a fresh type.
* Reenable tests
Instead of relying purely on the assumption that type handles can be compared
cheaply by pointer equality, fallback to a more expensive walk of the
type tree that recursively compares types structurally.
This allows different components to call into each other as long as
their types are structurally equivalent.
Signed-off-by: Ryan Levick <ryan.levick@fermyon.com>
* Fix documentation typos
While working on integrating MPK into Wasmtime, I found these typos and
line-wrapping issues. This fix is a separate commit to separate it from
the already-complex MPK implementation.
* fix: remove extra changes
* Add support for `v128` to the typed function API
This commit adds a Rust type `V128` which corresponds to the wasm
`v128` type. This is intended to perhaps one day have accessors for
lanes of various sizes but in the meantime only supports conversion back
and forth between `u128`. The intention of this type is to allow
platforms to perform typed between to functions that take or return
`v128` wasm values.
Previously this was not implemented because it's a bit tricky ABI-wise.
Typed functions work by passing arguments in registers which requires
the calling convention to match in both Cranelift and in Rust. This
should be the case for supported platforms and the default calling
convention, especially now that the wasm calling convention is separate
from the platform calling convention. This does mean, however, that this
feature can only be supported on x86_64 and AArch64. Currently neither
s390x nor RISC-V have a means of supporting the vector calling
convention since the vector types aren't available on stable in Rust
itself. This means that it's now unfortunately possible to write a
Wasmtime embedding that compiles on x86_64 that doesn't compile on s390x
for example, but given how niche this feature is that seems like an ok
tradeoff for now and by the time it's a problem Rust might have native
stable support for vector types on these platforms.
prtest:full
* Fix compile of C API
* Conditionally enable typed v128 tests
* Review comments
* Fix compiler warnings
* Redesign Wasmtime's CLI
This commit follows through on discussion from #6741 to redesign the
flags that the `wasmtime` binary accepts on the CLI. Almost all flags
have been renamed/moved and will require callers to update. The main
motivation here is to cut down on the forest of options in `wasmtime -h`
which are difficult to mentally group together and understand.
The main change implemented here is to move options behind "option
groups" which are intended to be abbreviated with a single letter:
* `-O foo` - an optimization or performance-tuning related option
* `-C foo` - a codegen option affecting the compilation process.
* `-D foo` - a debug-related option
* `-W foo` - a wasm-related option, for example changing wasm semantics
* `-S foo` - a WASI-related option, configuring various proposals for example
Each option group can be explored by passing `help`, for example `-O
help`. This will print all options within the group along with their
help message. Additionally `-O help-long` can be passed to print the
full comment for each option if desired.
Option groups can be specified multiple times on the command line, for
example `-Wrelaxed-simd -Wthreads`. They can also be combined together
with commas as `-Wrelaxed-simd,threads`. Configuration works as a "last
option wins" so `-Ccache,cache=n` would end up with a compilation
cache disabled.
Boolean options can be specified as `-C foo` to enable `foo`, or they
can be specified with `-Cfoo=$val` with any of `y`, `n`, `yes`, `no`,
`true`, or `false`. All other options require a `=foo` value to be
passed and the parsing depends on the type.
This commit additionally applies a few small refactorings to the CLI as
well. For example the help text no longer prints information about wasm
features after printing the option help. This is still available via
`-Whelp` as all wasm features have moved from `--wasm-features` to `-W`.
Additionally flags are no longer conditionally compiled in, but instead
all flags are always supported. A runtime error is returned if support
for a flag is not compiled in. Additionally the "experimental" name of
WASI proposals has been dropped in favor of just the name of the
proposal, for example `--wasi nn` instead of `--wasi-modules
experimental-wasi-nn`. This is intended to mirror how wasm proposals
don't have "experimental" in the name and an opt-in is required
regardless.
A full listing of flags and how they have changed is:
| old cli flag | new cli flag |
|-----------------------------------------------|-------------------------------------------------|
| `-O, --optimize` | removed |
| `--opt-level <LEVEL>` | `-O opt-level=N` |
| `--dynamic-memory-guard-size <SIZE>` | `-O dynamic-memory-guard-size=...` |
| `--static-memory-forced` | `-O static-memory-forced` |
| `--static-memory-guard-size <SIZE>` | `-O static-memory-guard-size=N` |
| `--static-memory-maximum-size <MAXIMUM>` | `-O static-memory-maximum-size=N` |
| `--dynamic-memory-reserved-for-growth <SIZE>` | `-O dynamic-memory-reserved-for-growth=...` |
| `--pooling-allocator` | `-O pooling-allocator` |
| `--disable-memory-init-cow` | `-O memory-init-cow=no` |
| `--compiler <COMPILER>` | `-C compiler=..` |
| `--enable-cranelift-debug-verifier` | `-C cranelift-debug-verifier` |
| `--cranelift-enable <SETTING>` | `-C cranelift-NAME` |
| `--cranelift-set <NAME=VALUE>` | `-C cranelift-NAME=VALUE` |
| `--config <CONFIG_PATH>` | `-C cache-config=..` |
| `--disable-cache` | `-C cache=no` |
| `--disable-parallel-compilation` | `-C parallel-compilation=no` |
| `-g` | `-D debug-info` |
| `--disable-address-map` | `-D address-map=no` |
| `--disable-logging` | `-D logging=no` |
| `--log-to-files` | `-D log-to-files` |
| `--coredump-on-trap <PATH>` | `-D coredump=..` |
| `--wasm-features all` | `-W all-proposals` |
| `--wasm-features -all` | `-W all-proposals=n` |
| `--wasm-features bulk-memory` | `-W bulk-memory` |
| `--wasm-features multi-memory` | `-W multi-memory` |
| `--wasm-features multi-value` | `-W multi-value` |
| `--wasm-features reference-types` | `-W reference-types` |
| `--wasm-features simd` | `-W simd` |
| `--wasm-features tail-call` | `-W tail-call` |
| `--wasm-features threads` | `-W threads` |
| `--wasm-features memory64` | `-W memory64` |
| `--wasm-features copmonent-model` | `-W component-model` |
| `--wasm-features function-references` | `-W function-references` |
| `--relaxed-simd-deterministic` | `-W relaxed-simd-deterministic` |
| `--enable-cranelift-nan-canonicalization` | `-W nan-canonicalization` |
| `--fuel <N>` | `-W fuel=N` |
| `--epoch-interruption` | `-W epoch-interruption` |
| `--allow-unknown-exports` | `-W unknown-exports-allow` |
| `--trap-unknown-imports` | `-W unknown-imports-trap` |
| `--default-values-unknown-imports` | `-W unknown-imports-default` |
| `--max-instances <MAX_INSTANCES>` | `-W max-instances=N` |
| `--max-memories <MAX_MEMORIES>` | `-W max-memories=N` |
| `--max-memory-size <BYTES>` | `-W max-memory-size=N` |
| `--max-table-elements <MAX_TABLE_ELEMENTS>` | `-W max-table-elements=N` |
| `--max-tables <MAX_TABLES>` | `-W max-tables=N` |
| `--max-wasm-stack <MAX_WASM_STACK>` | `-W max-wasm-stack=N` |
| `--trap-on-grow-failure` | `-W trap-on-grow-failure` |
| `--wasm-timeout <TIME>` | `-W timeout=N` |
| `--wmemcheck` | `-W wmemcheck` |
| `--wasi-modules default` | removed |
| `--wasi-modules -default` | removed |
| `--wasi-modules wasi-common` | `-S common` |
| `--wasi-modules -wasi-common` | `-S common=n` |
| `--wasi-modules experimental-wasi-nn` | `-S nn` |
| `--wasi-modules experimental-wasi-threads` | `-S threads` |
| `--wasi-modules experimental-wasi-http` | `-S http` |
| `--listenfd` | `-S listenfd` |
| `--tcplisten <SOCKET ADDRESS>` | `-S tcplisten=...` |
| `--wasi-nn-graph <FORMAT::HOST>` | `-S nn-graph=FORMAT::HOST` |
| `--preview2` | `-S preview2` |
| `--dir <DIRECTORY>` | `--dir ...` |
| `--mapdir <GUEST_DIR::HOST_DIR>` | `--dir a::b` |
* Be more descriptive with help text
* Document `=val` is optional for `-Ccranelift-xxx`
* Fix compile after rebase
* Fix rebase of `--inherit-network`
* Fix wasi-http test
* Fix compile without pooling allocator support
* Update some flags in docs
* Fix bench-api build
* Update flags for gdb/lldb tests
* Fixup optimization flags
prtest:full
This commit optimizes the lifting and lowering routines for `list<u8>`
and other similar primitive integer types. The current lifting/lowering
is intended to be general-purpose and correct but doesn't optimize well
with LLVM for a number of reasons. I first attempted to reshape the
general-purpose code to be easier for LLVM to optimize but in the end
was unable to convince LLVM that various pointers here don't alias which
meant that the general-purpose lowering/lifting never optimized well.
Instead, however, I've added new trait methods which are implemented the
same way as the general purpose methods beforehand. The
integer/primitive implementations overwrite these implementations with
more specialized versions given knowledge of primitives.
On a local benchmark this makes lifting/lowering disappear from a
profile since memcpy is generally much faster than per-item processing.
* feat(wasmtime-cli): add async support flag
Within the wasmtime CLI, the current default behavior
is to only inject the synchronous functions to linkers.
This will add a flag called `--async` that will inject
the asynchronous one instead.
* chore: refactor wasi http crate
* feat(wasmtime-wasi): make in_tokio function public
* feat(wasi-http): define default feature called sync
* Revert "feat(wasmtime-cli): add async support flag"
This reverts commit b743ff2003a2e391972330aa8e8437c6356e580a.
* chore: improve flaky tests for wasi http
* feat(wasi-http): expose sync api for components
* chore: add tests for sync api of wasi http components
* feat(wasmtime-cli): restore support for wasi http module
* chore: revert change to outbound http request invalid test
* chore: have extra tracing to help debugging
* feat(wasi-http): allow modules with sync functions in linker
* fix(wasi-http): missing response body in sync api
* feat: include blocking for io streams
* chore: add tests for wasi http module in cli
* chore: disable preview2 flag in wasi http test
* chore: use preview2 flag in wasi http test
* fix(wasi-http): missing stream output in sync api
* chore: fix tests for wasi http
* chore: add tracing for poll oneoff call
* chore: send exit signal on wasi http test
* chore: swap println to tracing debug
* chore: set http server timeout to 50 secs by default
* chore: add test posting large file
* chore: revert formatting in cargo toml
* chore: fix wasi-http feature and skip failing tests
prtest:full
---------
Co-authored-by: Eduardo Rodrigues <eduardomourar@users.noreply.github.com>
* Update Rust in CI to 1.72.0
* Update CI, tooling, and docs for MSRV
This commit codifies an MSRV policy for Wasmtime at "stable minus two"
meaning that the latest three releases of Rust will be supported. This
is enforced on CI with a full test suite job running on Linux x86_64
with the minimum supported Rust version. The full test suite will use
the latest stable version. A downside of this approach is that new
changes may break MSRV support on non-Linux or non-x86_64 platforms and
we won't know about it, but that's deemed a minor enough risk at this
time.
A minor fix is applied to Wasmtime's `Cargo.toml` to support Rust 1.70.0
instead of requiring Rust 1.71.0
* Fix installation of rust
* Scrape MSRV from Cargo.toml
* Cranelift is the same as Wasmtime's MSRV now, more words too
* Fix a typo
This commit takes a leaf out of `wiggle`'s book to enable bindings
generation for async host functions where only some host functions are
async instead of all of them. This enhances the `async` key with a few
more options:
async: {
except_imports: ["foo"],
only_imports: ["bar"],
}
This is beyond what `wiggle` supports where either an allow-list or
deny-list can be specified (although only one can be specified). This
can be useful if either the list of sync imports or the list of async
imports is small.
* added trap to resource dtor
* basic resources
* fixed signature and trait bounds
* fixed static function
* fixed trait signature
* basic resources
* added result to resource drop
* reverted formatting
* added doc comment to resource `dtor`
* fixed merge issue
* fixed another merge issue
* added resource import codegen test
* Revert changes to `Cargo.lock`
* Update `Cargo.lock` with the wit-bindgen update
* Add `cargo vet` entries for new crates
* Restore old-style of printing types in bindgen
Avoid collecting resources-as-we-go in favor of doing that more
declaratively elsewhere. Additionally handle imported-vs-exported
resource via the order that interfaces are visited.
* Update the shape of resource traits:
* Use `&mut self` instead of `StoreContextMut`
* Make resource traits as supertraits of the `Host` trait generated for
each resource.
* More uniformly handle types/returns with resource methods.
* Fix derivations of `Clone` and `Copy` for where handles are contained.
* Fix generation of handle typedefs
* Support resources-in-worlds
* Remove now-duplicate function
This is now the same as the preexisting `generate_function_trait_sig`
* Fix classifying handles as imported or exported
This determination happens by looking up the origin definition of a
resource, not the leaf possibly-aliased type.
* Fix chains-of-use of resources
Resources don't have their representation asserted since the bare type
itself doesn't implement `ComponentType`, but otherwise generate type
aliases the same way as other type aliases.
* Revert `Cargo.lock` changes
No need to update `wit-bindgen` to 0.10.0 just yet, that'll happen in a
future update if necessary.
* Add basic runtime tests for resources
* fixed merge issue
---------
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
By not activating the `derive` feature on `serde`, the compilation speed
can be improved by a lot. This is because `serde` can then compile in
parallel to `serde_derive`, allowing it to finish compilation possibly
even before `serde_derive`, unblocking all the crates waiting for
`serde` to start compiling much sooner.
As it turns out the main deciding factor for how long the compile time of a
project is, is primarly determined by the depth of dependencies rather
than the width. In other words, a crate's compile times aren't affected
by how many crates it depends on, but rather by the longest chain of
dependencies that it needs to wait on. In many cases `serde` is part of
that long chain, as it is part of a long chain if the `derive` feature
is active:
`proc-macro2` compile build script > `proc-macro2` run build script >
`proc-macro2` > `quote` > `syn` > `serde_derive` > `serde` >
`serde_json` (or any crate that depends on serde)
By decoupling it from `serde_derive`, the chain is shortened and compile
times get much better.
Check this issue for a deeper elaboration:
https://github.com/serde-rs/serde/issues/2584
For `wasmtime` I'm seeing a reduction from 24.75s to 22.45s when
compiling in `release` mode. This is because wasmtime through `gimli`
has a dependency on `indexmap` which can only start compiling when
`serde` is finished, which you want to happen as early as possible so
some of wasmtime's dependencies can start compiling.
To measure the full effect, the dependencies can't by themselves
activate the `derive` feature. I've upstreamed a patch for
`fxprof-processed-profile` which was the only dependency that activated
it for `wasmtime` (not yet published to crates.io). `wasmtime-cli` and
co. may need patches for their dependencies to see a similar
improvement.
* Handle more failures with `--trap-on-grow-failure`
This debugging tool was missing a few cases about `memory.grow` or
`table.grow` returning -1, namely when the OS reported failure in the
`memory.grow` case or in the case for tables when the computation
overflowed for the new table size. This commit updates the
`*_grow_failed` methods, which were already hooking these errors, to
return a `Result<()>` which enables returning a trap from them as well.
* Fix test
* Wasmtime: Rename `IndexAllocator` to `ModuleAffinityIndexAllocator`
We will have multiple kinds of index allocators soon, so clarify which one this
is.
* Wasmtime: Introduce a simple index allocator
This will be used in future commits refactoring the pooling allocator.
* Wasmtime: refactor the pooling allocator for components
We used to have one index allocator, an index per instance, and give out N
tables and M memories to every instance regardless how many tables and memories
they need.
Now we have an index allocator for memories and another for tables. An instance
isn't associated with a single instance, each of its memories and tables have an
index. We allocate exactly as many tables and memories as the instance actually
needs.
Ultimately, this gives us better component support, where a component instance
might have varying numbers of internal tables and memories.
Additionally, you can now limit the number of tables, memories, and core
instances a single component can allocate from the pooling allocator, even if
there is the capacity for that many available. This is to give embedders tools
to limit individual component instances and prevent them from hogging too much
of the pooling allocator's resources.
* Remove unused file
Messed up from rebasing, this code is actually just inline in the index
allocator module.
* Address review feedback
* Fix benchmarks build
* Fix ignoring test under miri
The `async_functions` module is not even compiled-but-ignored with miri, it is
completely `cfg`ed off. Therefore we ahve to do the same with this test that
imports stuff from that module.
* Fix doc links
* Allow testing utilities to be unused
The exact `cfg`s that unlock the tests that use these are platform and feature
dependent and ends up being like 5 things and super long. Simpler to just allow
unused for when we are testing on other platforms or don't have the compile time
features enabled.
* Debug assert that the pool is empty on drop, per Alex's suggestion
Also fix a couple scenarios where we could leak indices if allocating an index
for a memory/table succeeded but then creating the memory/table itself failed.
* Fix windows compile errors
This commit adds a new `Engine::detect_precompiled` API to inspect some
bytes and determine if they look like a precompiled artifact of either a
core wasm module or component. This is something I'll be using soon in
an upcoming refactor of the Wasmtime CLI to support components, but it's
something we've also talked about before which can be useful for systems
storing both precompiled modules and components.
Implementation-wise this looks at the ELF header of the input and
determines if it's got all the right flags that Wasmtime sets for the
various bits and bobs of our object format.
* Configure Mach ports vs signals via `Config`
This commit adds a `Config::macos_use_mach_ports` configuration option
to replace the old `posix-signals-on-macos` compile-time Cargo feature.
This'll make Wasmtime a tad larger on macOS but likely negligibly so.
Otherwise this is intended to provide a resolution to #6785 where
embedders will be able to use any build of Wasmtime and configure at
runtime how trap handling should happen.
Functionally this commit additionally registers a `pthread_atfork`
handler to cause any usage of Wasmtime in the child to panic. This
should help head off a known-invalid state in case anyone runs into it
in the future.
* Fix build on non-macOS
Sample code provided in Readme of `wasmtime` crate specifies three
generic arguments for `get_typed_func` API, while it needs only two.
This fixes the sample code by removing the last generic argument.
Returns a summary of the resources required to instantiate this module or
component.
Potential uses of the returned information:
* Determining whether your pooling allocator configuration supports
instantiating this module.
* Deciding how many of which `Module` you want to instantiate within a fixed
amount of resources, e.g. determining whether to create 5 instances of module X
or 10 instances of module Y.
Part of #6627.
* Add config method for whether or not to capture coredumps on trap
* Add core dump support to the runtime
This adds the machinery to capture a core dump when a trap occurs
and attach it to the resulting anyhow::Error that gets bubbled up to
the caller. I've created a CoreDumpStack structure in the runtime, which is
currently just a backtrace until we design a way to recover the locals
stack values when a trap occurs. When that CoreDumpStack gets converted to
a wasmtime::WasmCoreDump, we add additional information from the Store such
as globals, memories, and instance information.
A lot of this is mechanistically similar to how backtraces
are captured and attached to errors. Given that they both are attached as
context to anyhow::Errors, setting coredump_on_trap to true will supercede
any setting for wasm_backtrace.
* Address some PR feedback
* Fix docs
* Switch WasmCoreDump to have vec of module names rather than Modules
* Add smoketests for WasmCoreDump
* clean up tests
* Update memories and globals field to be store_
* add debug impl for wasmcoredump
* Remove duplicate imports
* ignore miri for wasm tests
* Wasmtime: Add support for Wasm tail calls
This adds the `Config::wasm_tail_call` method and `--wasm-features tail-call`
CLI flag to enable the Wasm tail calls proposal in Wasmtime.
This PR is mostly just plumbing and enabling tests, since all the prerequisite
work (Wasmtime trampoline overhauls and Cranelift tail calls) was completed in
earlier pull requests.
When Wasm tail calls are enabled, Wasm code uses the `tail` calling
convention. The `tail` calling convention is known to cause a 1-7% slow down for
regular code that isn't using tail calls, which is why it isn't used
unconditionally. This involved shepherding `Tunables` through to Wasm signature
construction methods. The eventual plan is for the `tail` calling convention to
be used unconditionally, but not until the performance regression is
addressed. This work is tracked in
https://github.com/bytecodealliance/wasmtime/issues/6759
Additionally while our x86-64, aarch64, and riscv64 backends support tail calls,
the s390x backend does not support them yet. Attempts to use tail calls on s390x
will return errors. Support for s390x is tracked in
https://github.com/bytecodealliance/wasmtime/issues/6530
* Store `Tunables` inside the `Compiler`
Instead of passing as an argument to every `Compiler` method.
* Cranelift: Support "direct" return calls on riscv64
They still use `jalr` instead of `jal` but this allows us to use the `RiscvCall`
reloc, which Wasmtime handles. Before we were using `LoadExternalName` which
produces an `Abs8` reloc, which Wasmtime intentionally does not handle since
that involves patching code at runtime, which makes loading code slower.
* Fix tests that assume tail call support on s390x
* Remove unused WIT files from Wasmtime
These files aren't actually read by anything currently. They were added
historically and a previous refactoring in #6390 forgot to remove them.
No tests or build process reads them so this deletes them to get them
out of the way.
* Update dependencies on wasm-tools crates.
This commit updates the deps on the wasm-tools family of crates to bring
in a few fixes for WIT/component-related things. Primarily though this
brings in an update to the component model where empty types are now
disallowed.
Some tests using empty types were adjusted to use non-empty types, but
many tests were also simply removed as they existed to test what would
happen with empty types which now no longer needs to be tested.
* Update `stream-error` in preview2
Add a `dummy` field to make it a non-empty structure. It's expected that
this will change to something else more "official" in the future, but
for now this is here to keep everything compiling.
* Update component fuzzing to avoid empty types
Empty types are no longer valid
* Update crates/wasi/wit/deps/io/streams.wit
Co-authored-by: Peter Huene <peter@huene.dev>
---------
Co-authored-by: Peter Huene <peter@huene.dev>
* Implement component-to-component calls with resources
This fills out support in FACT in Wasmtime to support
component-to-component calls that use resources. This ended up being
relatively simple as it's "just" a matter of moving resources between
tables which at this time bottoms out in calls to the host. These new
trampolines are are relatively easy to add after #6751 which helps keep
this change contained.
Closes#6696
* Review comments
In poking at #6696 I felt that the amount of boilerplate for defining
new kinds of trampolines in the component model was getting a bit
excessive. There was already 6 different types and I was adding two more
and I had to touch just a few too many places to get this done. In the
end I decided to refactor how trampolines are handled in the component
model to make it much easier to add new kinds of trampolines.
To that end the type-specific counts/lists/etc are all gone now in favor
of a single concept of a trampoline. This means components now track
trampolines in-bulk rather than individually by type. For example
compiling trampolines is now a loop over "compile this list of
trampolines" where previously it was N loops for the N types of
trampolines. The `Trampoline` definition is where the enum and dispatch
happens where that contains all possible trampolines that a component
could require.
This ended up being a large refactor to the Cranelift component
integration, but there is not intended to be any functional change from
this refactoring. Additionally all trampolines are now removed from the
global initializers list since there's nothing preventing them from
being initialized earlier on during the instantiation process.
Overall this should drastically reduce the number of locations that need
to have trampoline-specific knowledge to translation, the dataflow
graph, and compilation. Nearly everything else can operate over
everything in bulk and forward between these systems.
* Fix signatures registered with modules-in-components
This commit fixes a minor issue in
`FunctionIndices::link_and_append_code` which previously ended up only
filling out the `wasm_to_native_trampolines` field for the first module
rather than all the modules. Additionally the first module might have
too many entries that encompass all modules instead of just its own
entries. The fix in this commit is to refactor this logic to ensure that
the necessary maps are present for all translations.
While technically a bug that can be surfaced through the embedder API
it's pretty obscure. The given test here panics beforehand but succeeds
afterwards, but this is moreso prep for some future resource-related
work where this map will need persisting into the component metadata
side of things.
* Initial support for resources
Lots of bits and pieces squashed into this commit. Much to be done
still.
* Start supporting destructors
* Get some basic drop tests working
Also add a test which requires host-defined drop to be called which
isn't working.
* Fix rebase issue
* Fix a failing test
* I am zorthax, destroyer of resources
* Remove a branch in compiled code
No need to check for a null funcref when we already know ahead of time
if it's ever going to be null or not.
* Fix the test suite
* Add embedder API to destroy resources
* Add TODO for factc
* Fix a warning and leave a comment
* Integrate resources into `Type`
Plumb around dynamic information about resource types.
* Implement `Val::Own`
* Implement reentrance check for destructors
Implemented both in the raw wasm intrinsic as well as the host.
* Use cast instead of transmute
* Fill out some cranelift-shared comments
* Update codegen for resource.drop shim
The MAY_ENTER flag must always be checked, regardless of whether there's
an actual destructor or not.
* Update wasm-tools crates to latest `main`
* Update resource.drop binary format
* Add some docs
* Implement dynamic tracking for borrow resources
Not actually hooked up anywhere but this should at least be a first stab
at an implementation of the spec.
* Remove git overrides
* Remove no-longer-needed arms in wit-bindgen
* Prepare for mutability in `LiftContext`
* Change `&LiftContext` to `&mut LiftContext`
* Remove `store: &'a StoreOpaque` from `LiftContext`, instead storing
just `memory: &'a [u8]`
* Refactor methods to avoid needing the entire `StoreOpaque`
This'll enable `LiftContext` to store `&'a mut ResourceTable` in an
upcoming commit to refer to the host's resources.
* Lowering a borrow is infallible
* Use `ResourceAny` for both own/borrow
Rename `Val::Own` to `Val::Resource` accordingly.
* Initial implementation of borrowed resources
Lots of juggling of contexts here and there to try and get everything
working but this is hopefully a faithful implementation. Tests not
implemented yet and will come next and additionally likely update
implementation details as issues are weeded out.
* Add a suite of tests for borrowing resources
Code coverage was used to ensure that almost all of the various paths
through the code are taken to ensure all the basic bases are covered.
There's probably still lurking bugs, but this should be a solid enough
base to start from hopefully.
* Fill in an issue for bindgen todo
* Add docs, still more to go
* Fill out more documentation
* Fill out a test TODO
* Update the host `Resource<T>` type
* Add docs everywhere
* Don't require a `Store` for creating the resource or getting the
representation.
The latter point is the main refactoring in this commit. This is done in
preparation for `bindgen!` to use this type where host bindings
generally do not have access to the store.
* Document `ResourceAny`
* Debug assert dtor is non-null
* Review comments on loading libcalls
* Update some comments
* Update a comment
* Fix some typos
* Add a test that host types are the same when guest types differ
* Fix some typos
* Thread things through a bit less
* Undo CompileKey-related changes
* Gate an async function on the async feature
* Fix doc links
* Skip resources tests in miri
They all involve compilation which takes too long and doesn't currently
work
Currently wasmtime's error message for `wasmtime nonexistent.wasm`
doesn't actually refer to `nonexistent.wasm` anywhere, so this commit
fills that in as contextual information on the returned error.
Alex Crichton
Chris Fallin
Tyler Rockwood
Dan Gohman
Trevor Elliott
Andrew Brown
Nick Fitzgerald
Rainy Sinclair
Ryan Levick
Adam C. Foltzer
Eduardo de Moura Rodrigues
Timmy Silesmo
Christopher Serr
Lann
Timon
ssunkin-fastly
Anoop Elias
guregu