* Add compatibility shims for Wasmtime 13 CLI
This commit introduces a compatibility shim for the Wasmtime 13 CLI and
prior. The goal of this commit is to address concerns raised in #7336
and other locations as well. While the new CLI cannot be un-shipped at
this point this PR attempts to ameliorate the situation somewhat through
a few avenues:
* A complete copy of the old CLI parser is now included in `wasmtime` by
default.
* The `WASMTIME_NEW_CLI=0` environment variable can force usage of the
old CLI parser for the `run` and `compile` commands.
* The `WASMTIME_NEW_CLI=1` environment variable can force usage of the
new CLI parser.
* Otherwise both the old and the new CLI parser are executed. Depending
on the result one is selected to be executed, possibly with a warning
printed.
* If both CLI parsers succeed but produce the same result, then no
warning is emitted and execution continues as usual.
* If both CLI parsers succeed but produce different results then a
warning is emitted indicating this. The warning points to #7384 which
has further examples of how to squash the warning. The warning also
mentions the above env var to turn the warning off. In this situation
the old semantics are used at this time instead of the new semantics.
It's intended that eventually this will change in the future.
* If the new CLI succeeds and the old CLI fails then the new semantics
are executed without warning.
* If the old CLI succeeds and the new CLI fails then a warning is issued
and the old semantics are used.
* If both the old and the new CLI fail to parse then the error message
for the new CLI is emitted.
Note that this doesn't add up to a perfect transition. The hope is that
most of the major cases of change at the very least have something
printed. My plan is to land this on `main` and then backport it to the
14.0.0 branch as a 14.0.3 release.
* Wordsmith messages
* Update messages
* More wording updates
* Fix grammar
* More updates
Currently Wasmtime has two implementations of the `wasi_snapshot_preview1` set
of APIs. The now-historical implementation lives in the `wasi-common`
crate and the more recent implementation lives in the `wasmtime-wasi`
crate. The main difference is that the `wasmtime-wasi` implementation is
based on the implementation of preview2, meaning that the preview1
implementation is a shim in that direction. Additionally currently the
preview2 implementation of preview1 is accessible via the `-Spreview2`
flag on the CLI.
This commit updates the interpretation of the `-Spreview2` flag and the
defaults around which implementation to choose. By default the
preview1-built-on-preview2 implementation (the new `wasmtime-wasi`
implementation) is selected now. This means that the `wasi-common`
implementation is disabled by default. There are still two use cases to
keep running the `wasi-common` implementation, however:
* When running modules that import from `wasi_unstable`, the "snapshot"
before `wasi_snapshot_preview1`, currently `wasi-common` is required.
The shims to implement `wasi_unstable` have not yet been implemented
in the `wasmtime-wasi` crate.
* When running with WASI threads (`-Sthreads`) the preview2
implementation does not work. This is because the preview2
implementation expects mutable access to the table which is not
granted when threads are enabled.
Tests using `wasi_unstable` now pass `-Spreview2=n` to explicitly
request the old `wasi-common` implementation. Additionally the
`wasi-common` implementation is still selected by default when
`-Sthreads` is passed (enabling the WASI threads proposal).
* PCC: add memory type and fact annotations needed for dynamic-memory validation.
* PCC: update dynamic fact kinds, add sketch of dynamic-mem case, and add parser.
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
* Working dynamic-range verification on x64 and aarch64.
* Fix x64 shll: output range according to bitwidth, not always-64-bit.
* Review feedback.
* Missing backtick in doc comment.
---------
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
* PCC: support x86-64.
This PR extends the proof-carrying-code infrastructure to support x86-64
as well as aarch64. In the process, many of the mechanisms had to be
made a little more general.
One important change is that the PCC leaves more "breadcrumbs" on the
frontend now, avoiding the need for magic handling of facts on constant
values, etc., in the backend. For the first time a lowering rule also
gains the ability to add a fact to a vreg to preserve the chain as well.
With these changes, we can validate compilation of SpiderMonkey.wasm
with Wasm static memories on x86-64 and aarch64:
```
cfallin@fastly2:~/work/wasmtime% target/release/wasmtime compile -C pcc=yes --target x86_64 ../wasm-tests/spidermonkey.wasm
cfallin@fastly2:~/work/wasmtime% target/release/wasmtime compile -C pcc=yes --target aarch64 ../wasm-tests/spidermonkey.wasm
cfallin@fastly2:~/work/wasmtime%
```
* Don't run regalloc checker if not requested in addition to PCC; it's fairly expensive.
* Refactor x64 PCC code to avoid deep pattern matches on Gpr/Xmm types; explicitly match every instruction kind.
* Update some crates to align on latest `rustix`
This commit addresses some dependabot warnings showing up on the
Wasmtime repository by updating all dependencies to using the latest
`rustix` release.
* Debug CI
prtest:full
* Downgrade rustix to 0.38.14
Works around bytecodealliance/rustix#901
* mpk: add Wasm test
This adds a Wasmtime-level test checking that, when MPK is available on
the system, an MPK-configured memory pool obeys the same invariants as
other memory pool configurations (e.g., `guards_present` for normal
allocation, `guards_present_pooling` for non-MPK pooling allocation).
It also fixes a bug. A previous commit had updated the validation logic
of `MemoryPool` to check that the memory plan's bound would be reached
before the next slot of the same stripe. But `MemoryPool::allocate` has
some "double-check" logic that was triggered by this test. It now
matches the logic in `MemoryPool::validate`.
* mpk: explicitly disable MPK on certain tests
Some Wasmtime tests rely on specific limits to the memory pool. When MPK
is enabled, these tests fail because MPK splits access to the pool
slices among different stores. This change does not yet enable MPK,
though locally I run with MPK enabled. With this commit and MPK enabled
locally, all Wasmtime tests now pass.
* cranelift-wasm: Add max size to `HeapData`
* cranelift-wasm: Add a small optimization for dynamic memories with `min_size == max_size`
In this case, we don't need to load the dynamic heap bound from the vmctx
because it actually has a constant size. Instead, we can use the constant
directly.
* Review comments
* mpk: add `max_memory_protection_keys`
If Wasmtime is ever embedded in an application that also uses memory
protection keys, it could be useful to limit how many Wasmtime
allocates and uses. This came up while examining `*.wast` tests: if
there was no way limiting the number of keys used, then those tests
configured a pool that reserved too much memory. This change takes that
further to attempt to limit the initial number of keys allocated. The
unfortunate side effect of using a `OnceLock` is that the `max` setting
is only applicable on the first invocation, the one that sets the
`OnceLock`.
* mpk: use two protection keys for WAST tests
This change stems from how slicing memory slots into MPK-protected
regions limits the number of memories each store can access: e.g., with
fifteen keys in use, a store only has access to a fifteenth of the
available slots. If we simply multiple the number of memory slots needed
to run the `*.wast` spec tests by fifteen, we run out of available
memory. This limits the number of protection keys used to two, which
still allows us to test the functionality without reserving too much
memory.
* mpk: ensure `keys` only ever returns `max` items
This addresses a review comment to slice the list of keys down to the
`max` hint regardless of how many are allocated in the first invocation.
* fix: remove warning about unused parameter
This is required to compile for a target which doesn't have a cranelift
backend. Before this change using any of the cranelift crates that
depend on cranelift-codegen would forcefully enable all default features
and thus host-arch. With this change only the std and unwind features
are still forcefully enabled as cranelift-codegen doesn't compile with
either disabled.
This change fixes a bug with `ProtectionKey::protect`: previously it
initialized each stripe with read and write permissions (i.e.,
`pkey_mprotect(..., PROT_READ | PROT_WRITE)` under the mistaken
assumption that these permissions were MPK-specific, "what MPK
permissions will we be allowed to set in the PKRU for these regions in
the future?". This assumption is incorrect: the regions were immediately
made accessible for reading and writing. The fix is to initially protect
the regions with `PROT_NONE` and allow Wasmtime's `memory.grow`
implementation to mark pages with `mprotect(..., PROT_READ |
PROT_WRITE)` as usual. Whether a store can access a slice is still
determined by the CPU state set in `mpk::allow`.
This follows up #6807 and removes the last remaining reference to
the removed `posix-signals-on-macos` feature flag.
Note that `lib.rs` now imports `mod unix` on MacOS. This change
is similar to the change in `traphandlers.rs` in #6807. It is
needed for hosts that use signals instead of Mach ports on MacOs.
* Expand further minimization section of documentation
This commit fills out the page about producing minimal builds a bit
more. The intention here is to provide examples of ideas about how to
reduce size further as well as current limitations and how they can be
evaluated.
* Review comments
* Introduce UDP streams
Introduce new `inbound-datagram-stream` and `outbound-datagram-stream` types and moved `receive` and `send` methods to those respectively. These streams are returned by `bind` can be individually subscribed to. This resolves a design issue where a UDP server would end up in a spin loop because `receive` returned EWOULDBLOCK but poll_* always returned immediately because the socket was ready for sending. In this new setup, users can poll each direction separately. Fixes https://github.com/WebAssembly/wasi-sockets/issues/64
Additionally:
- Enable send-like behaviour by making `outbound-datagram::remote-address` optional. Fixes https://github.com/WebAssembly/wasi-sockets/pull/57
- Dropped the `network` parameter from the `connect` call, because `bind` is now _required_ to perform IO.
* Align names with wasi-http
* Revert previous changes to `bind`. Replace `connect` with `stream`
Remove the Mutex again. Instead allow `stream` to be called multiple times, but trap if the previous streams are still active.
* The code block was treated as Rust code.
* Align more closely to wasi-io's input&output-stream
* Use `send` instead of `sendto` on connected sockets.
prtest:full
Previously, we assumed that the Wasmtime engine would be able to
allocate keys 1-15 from the OS, in that order. (Recall that Linux
reserves key 0 for itself). While enabling various tests for MPK,
tests in `wasmtime_fuzzing::oracles` would fail because Wasmtime could
only start allocating at key 2, e.g.; it turns out that the
`diff_v8::smoke` test instantiates V8 which happens to allocate a key
for itself.
The reason for the "allocate keys 1-15 in order" assumption was that the
logic for calculating the stripe each key owned was very simple: `key -
1`. We needed some way to map each key ID to the stripe ID it is
associated with.
With this change, we maintain a little bit more state in order to make
the mapping less brittle. `ProtectionKey` stores the "key ID to slice
ID" mapping as an additional `u32` in the struct. This means that,
regardless of what other code in the process allocates MPK keys,
Wasmtime should be able to work fine with the remaining keys it can
allocate.
This commit introduces a wrapper crate which is now the new "real" C
API. The purpose of this change is to enable using LTO when building the
C API. Currently LTO is disabled because one of the crate types of the C
API is an "rlib" which means that it can't have LTO performed due to
rustc limitations. The solution here is to remove the "cdylib" and
"staticlib" crate types from the "wasmtime-c-api" crate, rename that
crate to "wasmtime-c-api-impl", and reintroduce 'wasmtime-c-api' as a
new crate which wraps the previous crate and reexports it.
This way LTO can be enabled when just building the artifacts and the use
case from #6765 is still satisfied by having a crate that can be linked
to from Rust. Locally this reduces the size of the C API artifact for me
by nearly 1M.
* riscv64: Specify rounding modes in instructions
This commit updates how floating-point instructions specify their float
rounding mode (FRM). Previously instructions stored `Option<FRM>` and
this would mostly be `None`. All floating-point instructions in RISC-V
have a 3-bit `rm` field, and most encode the FRM into this field but
some have a require encoding of this field. For example `fsgnj.s` uses
the `rm` field to differentiate between `fsgnj`, `fsgnjx`, and `fsgnjn`.
Instructions like `fadd` however use this field for a rounding mode.
All FPU instructions now store `FRM` directly. Instruction helpers like
`fadd` require this to be specified explicitly. Instructions helpers
like for `fsgnj` do not take this as an argument and hardcode the field
as necessary. This means that all lowerings of floating point
instructions, where relevant, now specify a rounding mode.
Previously the default rounding mode was to use the `fcsr` register,
meaning that the rounding mode would be determined dynamically at
runtime depending on the status of this register. Cranelift semantics,
however, are derivative of WebAssembly semantics which specify
round-to-nearest ties-to-even. This PR additionally fixes this
discrepancy by using `FRM::RNE` in all existing instructions instead of
`FRM::Fcsr`.
* riscv64: Refactor float-to-int conversions
This commit removes the `FcvtToInt` macro-instruction in the riscv64
backend in favor of decomposing it into individual operation for
`fcvt_to_{s,u}int*` instructions. This additionally provides a slightly
different lowering for the `*_sat` operations which doesn't use
branches. The non-saturating operations continue to have a number of
branches and their code has changed slightly due to how immediates are
loaded. Overall everything is in ISLE now instead of split a bit.
* riscv64: Clean up some dead code in the backend
Don't put `#![allow(dead_code)]` at the root, instead place it on some
smaller items.
* Fix emission tests
* Add regression tests and bless output
Closes#5992Closes#5993
* Enable i8/i16 saturating float-to-int in fuzzgen
* Better `fcvt_*_bound` implementations
* Fix typo in match orderings
* Fix tests on x64
Where float-to-int isn't implemented for i8/i16
With a custom standard library disabling all features means disabling
some support for feature-detection macros of the native platform. This
meant that `wasmtime compile` output locally wasn't runnable in
`wasmtime-min run` because it couldn't correctly detect that features
were in fact available.
In an effort to simplify the many fuel related APIs, simplify the
interface here to a single counter with get and set methods.
Additionally the async yield is reduced to an interval of the total fuel
instead of injecting fuel, so it's easy to still reason about how much
fuel is left even with yielding turned on.
Internally this works by keeping two counters - one the VM uses to
increment towards 0 for fuel, the other to track how much is in
"reserve". Then when we're out of gas, we pull from the reserve to
refuel and continue. We use the reserve in two cases: one for overflow
of the fuel (which is an i64 and the API expresses fuel as u64) and the
other for async yieling, which then the yield interval acts as a cap to
how much we can refuel with.
This also means that `get_fuel` can return the full range of `u64`
before this change it could only return up to `i64::MAX`. This is
important because this PR is removing the functionality to track fuel
consumption, and this makes the API less error prone for embedders to
track consumption themselves.
Careful to note that the VM counter that is stored as `i64` can be
positive if an instruction "costs" multiple units of fuel when the fuel
ran out.
prtest:full
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
This was mistakenly removed during #7300 with some local testing I was
doing. This will eventually be required to get a minimal build of the C
API but for now I didn't intend on deleting this so I wanted to rectify
my mistake.
While not a large amount of binary size if the purpose of the
`--no-default-features` build is to showcase "minimal Wasmtime" then may
as well try to make `clap` as small as possible.
* riscv64: Extend distance trampolines can jump
Use a PIC-friendly set of instructions to enable destination of the
trampoline to be more than 4k away from the tail call site of the
trampoline itself.
* Build "min" artifacts on CI
This commit updates the binary artifacts produced by CI to include "min"
builds where all default features are disabled. Additionally all the
stops are pulled in terms of build flags, nightly versions, etc, to get
a build that is as small as possible without actual source code changes.
This effectively codifies the instructions in #7282 into an easily
downloadable artifact.
No new tarballs are created for github releases but instead tarballs
that previously had a `wasmtime` executable for example now have a
`wasmtime-min` executable. Furthermore the C API which previously had
`libwasmtime.so` for example now has `libwasmtime-min.so`. The intention
is that the minimum-size artifacts are handy for determining a rough
size of Wasmtime but they're not so important that it seems worthwhile
to dedicate entire release entries for.
CI is refactored to support these minimum builds with separate builders.
This means that a single tarball produced as a final result is actually
two separate tarballs merged together, one for the normal build we do
today plus a new "min" tarball produced on the new "min" builders.
Various scripts and CI organization has been adjusted accordingly.
While here I went ahead and enabled `panic=abort` and debuginfo
stripping in our current release artifacts. While this doesn't affect a
whole lot it's less to upload to GitHub Actions all the time.
* Fix Windows unzip
This reworks the way that remat and LICM interact during aegraph
elaboration. In principle, both happen during the same single-pass "code
placement" algorithm: we decide where to place pure instructions (those
that are eligible for movement), and remat pushes them one way while
LICM pushes them the other.
The interaction is a little more subtle than simple heuristic priority,
though -- it's really a decision ordering issue. A remat'd value wants to sink
as deep into the loop nest as it can (to the use's block), but we don't
know *where* the uses go until we process them (and make LICM-related
choices), and we process uses after defs during elaboration. Or more
precisely, we have some work at the use before recursively processing
the def, and some work after the recursion returns; and the LICM
decision happens after recursion returns, because LICM wants to know
where the defs are to know how high we can hoist. (The recursion is
itself unrolled into a state machine on an explicit stack so that's a
little hard to see but that's what is happening in principle.)
The solution here is to make remat a separate just-in-time thing, once
we have arg values. Just before we plug the final arg values into the
elaborated instruction, we ask: is this a remat'd value, and if so, do
we have a copy of the computation in this block yet. If not, we make
one. This has to happen in two places (the main elab loop and the
toplevel driver from the skeleton).
The one downside of this solution is that it doesn't handle *recursive*
rematerialization by default. This means that if we, for example, decide
to remat single-constant-arg adds (as we actually do in our current
rules), we won't then also recursively remat the constant arg to those
adds. This can be seen in the `licm.clif` test case. This doesn't seem
to be a dealbreaker to me because most such cases will be able to fold
the constants anyway (they happen mostly because of pointer
pre-computations: a loop over structs in Wasm computes heap_base + p +
offset, and naive LICM pulls a `heap_base + offset` out of the loop for
every struct field accessed in the loop, with horrible register pressure
resulting; that's why we have that remat rule. Most such offsets are
pretty small.).
Fixes#7283.
@jameysharp and I noticed a couple of refactoring opportunities while
reading through the elaboration pass:
* The elaboration loop doesn't need to match on the top of the stack as
a reference, because each case pops it immediately. Instead we can pop
and match on the popped value.
* Computing the cost of a `Result` value that's not in the DFG was using
the block that contains the instruction to determine the level, but
since the instruction is already known to not be in the DFG, this
would default to `LoopLevel::root()` unconditionally. This also meant
that the `Cost::at_level` function turned into an identity function on
the cost given, making it unnecessary.
Co-authored-by: Jamey Sharp <jsharp@fastly.com>
Alex Crichton
Andrew Brown
Chris Fallin
Pat Hickey
Jeffrey Charles
Trevor Elliott
Steven
Nick Fitzgerald
bjorn3
Ulan Degenbaev
Colin D Murphy
Afonso Bordado
Dave Bakker
Xinzhao Xu
Rahul
Tyler Rockwood
wasmtime-publish