* ci: Upgrade QEMU to `8.1.1`
This adds support for RISC-V's Zcb extension that includes some
extra compressed instructions.
It also removes the current cpuinfo patch, that has been released
in 8.1
* wasmtime: Don't assert the exact faulting address for wasm traps
* x64: Fix false dependencies in int-to-float conversions
This commit is a result of the investigation on #7085. The int-to-float
conversion instructions used right now on the x64 backend will
implicitly source the upper bits of the result from a different
register. This implicitly creates a dependency on further consumers
using the conversion result on whatever previously defined the upper
bits, even though they aren't used. This false dependency is the primary
reason for the slowdown witnessed in #7085.
The fix chosen in this commit is to model the int-to-float instructions
with a new shape of instruction instead of the previous `GprToXmm{,Vex}`. This
previous shape was modeled as single-input and single-output, but this
does not reflect the actual nature of the `cvtsi2s{s,d}` instructions.
Instead these now use `CvtIntToFloat{,Vex}` which have two source
operands and one destination operand, modeling how the upper bits of a
different register are used. In lowerings using this instruction the
upper bits to preserver are always sourced from a zero'd out register to
force breaking dependencies between instructions.
Closes#7085
* Remove now dead code
* Remove outdated test
Golden test output covers this test case anyway nowadays
* Review comments
* Fix emit tests
* Add memory protection keys (MPK)
In order to use MPK on an x86_64 Linux system, we need access to the
underlying `pkey_*` system calls (`sys`), control of the x86 PKRU
register (`pkru`), and a way of determining if MPK is even supported
(`is_supported`). These various parts are wrapped in a `ProtectionKey`
abstraction along with a `ProtectionMask` that can be used `allow` the
CPU to access protected regions.
* Integrate MPK into the pooling allocator
This change adds "stripes" to the pooling allocator's `MemoryPool`. Now,
when requesting a slot in which to instantiate, the user (i.e.,
`InstanceAllocationRequest`) will be transparently assigned to one of
the stripes, each of which is associated with a protection key. The user
can also request a specific protection key to use, which will override
the original "find me a slot logic".
This has implications for how instances get allocated: once a store is
assigned a protection key, it will only allocate requests with that key,
limiting how many slots it has access to. E.g., if 15 keys are active,
the store can only ever access 1/15th of the slots.
This change also includes a tri-bool configuration field,
`memory_protection_keys`, which is disabled by default for the time
being.
* Address review comments
This is a rollup of 43 commits addressing review comments of various
kinds: bug fixes, refactorings, documentation improvements, etc. It also
ensures that CI runs all checks. A big thanks to @fitzgen and
@alexcrichton for the review!
prtest:full
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
---------
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
* delete preview2::pipe::pipe
this is not used anywhere, not documented, and trivial for the user to
build on their own if they need it. i suspect they probably only want
one end of their pipe wrapped with AsyncReadStream or AsyncWriteStream.
* fix cfg_attr to ignore forking tests on qemu
* delete "async fd stdin" test, which is just the worker thread one anyway
* Bump wasm-tools crates
Two major changes/reasons for this update:
* Primarily pulling in support for semicolons-in-WIT files. Semicolons are
not currently required, though, so I'll follow-up later with actual
semicolons.
* The syntax for parsing `(if ...)` was fixed in `wast`. Previously it
did not require `(then ...)` but this is required by the spec. New
spec tests require this as well. This breaks existing text format
tests which don't use `(then ...)` inside of an `(if ...)`. Most tests
were updated by hand but `embenchen_*` tests were updated by running
through the old parser to produce non-s-expression using code.
* Fix an example `*.wat`
For `XmmRmiRVex`-format instructions, when the opcode is commutative, the first
operand is one of xmm{0..7}, and the second operand is one of xmm{8..15}, then
we can swap the operands to save a byte on instruction encoding.
These registers require an additional byte to reference when encoded in certain
AVX instruction formats (and maybe other situations as well?) so prefer
xmm{0..7} when they are available and only fall back to xmm{8..15} when register
pressure is higher.
* Wasmtime: Move `Global` and `Table` to `externals` submodules
Just mechanical code motion, not functional changes.
* Wasmtime: Add `hash_key` methods for `Func` and `Table`
* Rename `wasmtime::Func::caller_checked_func_ref` to `wasmtime::Func::vm_func_ref`
We removed the "caller checked" part of the `VMFuncRef` type's name a while ago,
so update this method to be in line with that.
No functional changes, just mechanical renaming.
* cargo fmt
* Fix doc link
This commit makes it so that the library type for core dumps is serializable
into the standard binary format for core dumps.
Additionally, this commit makes it so that we use the library type for
generating core dumps in the CLI. We previously were using a one-off
implementation of core dump generation that only had backtrace information and
no instances, modules, globals, or memories included. The library type has all
that information, so the core dumps produced by our CLI will both be more
featureful and be generated by shared code paths going forward.
Along the way, implementing all this required some new helper methods sprinkled
throughout `wasmtime` and `wasmtime-runtime`:
* `wasmtime::Instance::module`: get the module that a `wasmtime::Instance` is an
instance of. This is public, since it seems generally useful. This involved
adding a new return value from `ModuleRegistry::register_module` that is an
identifier that can be used to recover a reference to the registered module.
* `wasmtime::Instance::all_{globals,memories}`: get the full global/memory index
space. I made these `pub(crate)` out of caution. I don't think we want to commit
to exposing non-exported things in the public API, even if we internally need
them for debugging-related features like core dumps. These also needed
corresponding methods inside `wasmtime-runtime`.
* `wasmtime::{Global,Memory}::hash_key`: this was needed to work around the fact
that each time you call `{Global,Memory}::from_wasmtime`, it creates a new
entry in the `StoreData` and so you can get duplicates. But we need to key some
hash maps on globals and memories when constructing core dumps, so we can't
treat the underlying `Stored<T>` as a hash key because it isn't stable across
duplicate `StoreData` entries. So we have these new methods. They are only
`pub(crate)`, are definitely implementation details, and aren't exposed in the
public API.
* `wasmtime::FrameInfo::module`: Each frame in a backtrace now keeps a handle to
its associated module instead of just the name. This is publicly exposed
because it seems generally useful. This means I also deprecated
`wasmtime::FrameInfo::module_name` since you can now instead do
`frame.module().name()` to get that exact same info. I updated callers inside
the repo.
* Include Wasm-defined globals and memories in core dumps
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
* Narrow scope of unsafe block
* Add missing skip-miri attribute for test that calls into Wasm
---------
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
* Support resource maps in `component::bindgen!`
This commit adds support to `component::bindgen!` to specify resource
types using the `with` key of the macro. This can be used to configure
the `T` of `Resource<T>` to use a preexisting type rather than
unconditionally generating a new empty enum to have a fresh type.
* Reenable tests
* Start refactoring wasi-http
* Checkpoint
* Initial implementation of response future handling
* Lazily initialize response headers and body
* make wasmtime-wasi-http compile
* wasi-http wit: make a way to reject outgoing-request in outgoing-handler
before waiting for the future to resolve
* wasi: sync wit from wasi-http
* outgoing handler impl: report errors to userland
* test-programs: get wasi-http-components kicking over, delete modules and components-sync tests
wasi-http-components-sync will come back once we get done with other
stuff, but its superfulous for now. wasi-http-modules will not be
returning.
* Process headers
* Add HostIncomingBody::new
* Add trailers functions
* Add TODO for body task outline
* Rework incoming-response-consume to return a future-trailers value as well
* Fix the wit
* First cut at the worker loop
* wasi-http: change how we represent bodies/trailers, and annotate own/borrow/child throughout
* Update types_impl.rs for wit changes
* Split body management into its own module
* Checkpoint
* more work on incoming body and future trailers
* Fill out some more functions
* Implement future-trailers-{subscribe,get}
* Implement drop-future-trailers
* Rework fields, but make the borrow checker mad
* Fix borrow error
* wasi-http-tests: fix build
* test-runner: report errors with stdout/stderr properly
* fix two trivial wasi-http tests
the error type here changed from a types::Error to an
outbound_handler::Error
* Remove unnecessary drops
* Convert a `bail!` to a `todo!`
* Remove a TODO that documented the body worker structure
* fill in a bunch more of OutputBody
* Remove the custom FrameFut future in favor of using http_body_util
* Move the outgoing body types to body.rs
* Rework the handling of outgoing bodies
* Fix the `outgoing request get` test
* Avoid deadlocking the post tests
* future_incoming_request_get shouldn't delete the resource
* Fix the invalid_dnsname test
* implement drop-future-incoming-response
* Fix invalid_port and invalid_dnsname tests
* Fix the post test
* Passing a too large string to println! caused the large post test to fail
* Format
* Plumb through `between_bytes_timeout`
* Downgrade hyper
* Revert "Downgrade hyper"
This reverts commit fa0750e0c42823cb785288fcf6c0507c5ae9fd64.
* Restore old https connection setup
* Sync the wasi and wasi-http http deps
* Fix tests
* Remove the module and component-sync tests, as they are currently not
supported
* Fix the reference to the large_post test in the components test
* Fix wasi-http integration
* sync implementation of wasi-http
* Slightly more robust error checking
* Ignore the wasi-http cli test
prtest:full
* Consistent ignore attributes between sync and async tests
* Fix doc errors
* code motion: introduce intermediate `HostIncomingBodyBuilder` rather than a tuple
* explain design
* Turn FieldMap into a type synonym
* Tidy up some future state (#7073)
Co-authored-by: Pat Hickey <phickey@fastly.com>
* body HostInputStream: report runtime errors with StreamRuntimeError
HostInputStream is designed wrong to need that in the first place. We
will fix it in a follow-up as soon as resources land.
---------
Co-authored-by: Pat Hickey <phickey@fastly.com>
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
Instead of relying purely on the assumption that type handles can be compared
cheaply by pointer equality, fallback to a more expensive walk of the
type tree that recursively compares types structurally.
This allows different components to call into each other as long as
their types are structurally equivalent.
Signed-off-by: Ryan Levick <ryan.levick@fermyon.com>
* Fix documentation typos
While working on integrating MPK into Wasmtime, I found these typos and
line-wrapping issues. This fix is a separate commit to separate it from
the already-complex MPK implementation.
* fix: remove extra changes
* Fix character boundary issues in preview1 host adapter
This fixes two separate issues in the preview1-to-preview2 host-side
adapter in the `wasmtime-wasi` crate. Both instances were copying a
truncated string to the guest program but the truncation was happening
in the middle of a unicode character so truncation now happens on bytes
instead of a string.
* Update `write_bytes` to take `&[u8]`
Try to weed out any accidental string-related issues for the future.
* wasi: Fix a few issues around stdin
This commit is intended to address #6986 and some other issues related
to stdin and reading it, notably:
* Previously once EOF was reached the `closed` flag was mistakenly not
set.
* Previously data would be infinitely buffered regardless of how fast
the guest program would consume it.
* Previously stdin would be immediately ready by Wasmtime regardless of
whether the guest wanted to read stdin or not.
* The host-side preview1-to-preview2 adapter didn't perform a blocking
read meaning that it never blocked.
These issues are addressed by refactoring the code in question.
Note that this is similar to the logic of `AsyncReadStream` somewhat but
that type is not appropriate in this context due to the singleton nature
of stdin meaning that the per-stream helper task and per-stream buffer
of `AsyncReadStream` are not appropriate.
Closees #6986
* Increase slop size for windows
This commit is a follow-up to #6833 to remove the `unix` module for
handling stdio which sets stdin to nonblocking mode. I've just now
discovered that on macOS at least configuring `O_NONBLOCK` for stdin
affects the stdout/stderr descriptors too. This program for example will
panic:
fn main() {
unsafe {
let r = libc::fcntl(
libc::STDIN_FILENO,
libc::F_SETFL,
libc::fcntl(libc::STDIN_FILENO, libc::F_GETFL) | libc::O_NONBLOCK,
);
assert_eq!(r, 0);
}
loop {
println!("hello");
}
}
It was originally assumed that updating the flags for stdin wouldn't
affect anything else except Wasmtime, but because this looks to not be
the case this commit removes the logic of registering stdin raw with
Tokio and instead unconditionally using the worker thread solution which
should work in all situations.
* Update wasm-tools family of crates
Mostly minor updates, but staying up-to-date.
* Update text format syntax
* Update cargo vet entries
* Update more old text syntax
* riscv64: Deduplicate Trap Instruction
* riscv64: Use `defer_trap` in TrapIf instruction
This places the actual trap opcode at the end.
* riscv64: Emit islands before `br_table` sequence
This fixes a slightly subtle issue with our island emission in BrTable.
We used to emit islands right before the jump table targets. This
causes issues because if the island is actually emitted, we have
the potential to jump right into the middle of it. This happens
because we have calculated a fixed offset from the `auipc` instruction
assuming no island is emitted.
This commit changes the island to be emitted right at the start of the
br_table sequence so that this cannot happen.
* riscv64: Add trapz and trapnz helpers
* riscv64: Emit inline traps on `TrapIf`
* cranelift: Add support for public labels
* cranelift: Allow targeting labels with relocations
* cranelift: Emit label related relocations in module
* riscv64: Implement TLS GD
* cranelift: Rename `label_is_public` field
* cranelift: Avoid making MachLabel part of the exposed API
* Implement support for exported resources in `bindgen!`
This commit updates the `wasmtime::component::bindgen!` to support
exported resources. Exported resources are themselves always modeled as
`ResourceAny` at runtime and are required to perform dynamic type checks
to ensure that the right type of resource is passed in. Exported
resources have their own `GuestXXX` structure exported to namespace all
of their methods. Otherwise, like imports, there's not a whole lot of
special treatment of exported resources.
* Work around `heck` issue
Looks like `to_snake_case` behaves differently if the `unicode` feature
is enabled or not so bypass that entirely.
* fix(wasi-http): honor permit in output stream
* chore: enable large body test for wasi http
* chore: disable large body test
* fix(wasi-http): ensure output stream is flushed
* fix(wasi-http): logical error in io check-write wrap
* chore: fix outbound issue in usize casting
* chore: enable large request test
* chore: fix warnings
* prtest:full
* chore: call unreachable only on wasm32 arch
prtest:full
* chore: disable wasi http tests failing on windows
* chore: use renamed large request test
I was curious to poke around the riscv64 backend and I wanted to touch
up the handling of `Imm{12,20}` a bit after reading it. This commit is a
refactoring of these two types with the following changes:
* The payload of these types is now unsigned and guarantees that
irrelevant bits are set to zero. For example `Imm12` is stored as
`u16` where the upper four bits are guaranteed to be zero. This fixes
a discrepancy where `Imm12::maybe_from_i64` was masked for example but
`Imm12::from_bits` wasn't.
* The `Neg for Imm12` impl was removed because -2048 is a valid
`Imm12` but 2048 is not in-range for `Imm12` meaning that it is not an
infallible operation.
* Accessors are now named `bits` to get the `u32` representation
suitable to be encoded into an instruction. Acquiring the underlying
value is now done with `as_i{16,32}` depending on the type. The signed
accessor does sign-extension as required to produce the semantically
equivalent value.
* Manual constructors were renamed to `from_{i16,i32}` instead of
`from_bits`. This in theory helps convey that they're constructors for
logical values rather than literal bit-wise values. Additionally
asserts are now placed in these constructors asserting that the
provided value is in-range.
* The `FALSE` and `TRUE` constants were renamed `ZERO` and `ONE` and
`Imm20::ZERO` was added.
This commit ended up changing many runtests, but only their CLIF
printing rather than their encoding. This change is due to the fact that
`Display` now prints the logical value of the immediate rather than the
raw bit representation as a base 10 integer. It's not intended that this
commit actually changes any behavior, instead it should purely be
internal refactorings.
* riscv64: Update funct4 field width on CR type
It's only 4 bits, not 5
* riscv64: Add CA Ops
* riscv64: Encode `c.j` instructions
* riscv64: Use uncompressed instructions where labels are used.
Not doing so could cause a compressed instruction to be emitted with the wrong label type.
* riscv64: Implement `c.jr` instruction
* riscv64: Implement `c.jalr` instruction
This also update CallIndirect to allow compressing Jalr
* riscv64: Update RVCJump label range
It is only +-2KiB instead of +-4KiB
This commit attempts to simplify the `preview2::bindings` module by
consolidating everything into a single invocation of `bindgen!`.
Previously this was separated out (I think) to handle some methods being
`async` and not others, but the `bindgen!` macro has since grown the
capability to have some functions async and not all.
* x64: Fix an off-by-one in in `i64x2.shr_s`
This commit fixes a mistake from #6372 where one of the immediates to a
`pshufd` instruction was off-by-one in terms of bits. This fixes the
behavior of the wasm `i64x2.shr_s` instruction with constant shift
amounts larger than 32.
* Update release notes
We run in a unique environment with very little memory, so
it's possible if there are many modules that we run out of memory,
callers should be able to handle this gracefully.
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* Add support for `v128` to the typed function API
This commit adds a Rust type `V128` which corresponds to the wasm
`v128` type. This is intended to perhaps one day have accessors for
lanes of various sizes but in the meantime only supports conversion back
and forth between `u128`. The intention of this type is to allow
platforms to perform typed between to functions that take or return
`v128` wasm values.
Previously this was not implemented because it's a bit tricky ABI-wise.
Typed functions work by passing arguments in registers which requires
the calling convention to match in both Cranelift and in Rust. This
should be the case for supported platforms and the default calling
convention, especially now that the wasm calling convention is separate
from the platform calling convention. This does mean, however, that this
feature can only be supported on x86_64 and AArch64. Currently neither
s390x nor RISC-V have a means of supporting the vector calling
convention since the vector types aren't available on stable in Rust
itself. This means that it's now unfortunately possible to write a
Wasmtime embedding that compiles on x86_64 that doesn't compile on s390x
for example, but given how niche this feature is that seems like an ok
tradeoff for now and by the time it's a problem Rust might have native
stable support for vector types on these platforms.
prtest:full
* Fix compile of C API
* Conditionally enable typed v128 tests
* Review comments
* Fix compiler warnings
This commit implements a new behavior for the CLI of the `wasmtime`
executable which will require that options for Wasmtime itself come
before the wasm module being run. Currently they're allowed to come
afterwards, but instead all arguments and flags coming after a module
will be interpreted as arguments for the module itself.
This feature has a bit of a storied history at this point, and the
breadcrumbs are:
* Originally landed in #6737
* Reverted for 12.0.0 in #6830
* Reverted for 13.0.0 in #6944
This PR is intended to be landed as a sibling of #6925, another
independent overhaul of Wasmtime's own options on the CLI, for the
Wasmtime 14.0.0 release. More information about the motivation for this
change, as well as consequences of the fallout, can be found on #6737.
Due to a bug in the qemu emulation of the corresponding instruction,
tests for pseudo min/max support had been disabled. After moving
to qemu 8.0.4, where the bug is fixed, these can be re-enabled.
* add new wasi-test printing unicode on stdout
there's not yet a way to show this test is actually doing what we
intend, I'm using CI to see how this runs on all platforms
Reported in #6824
prtest:full
* cap-std-sync: use std::io::Std{in,out,err}'s {read,write}_vectored directly
instead of using AsFilelike to view these as a File.
This means we also have to use std's mutexes to access these descriptors,
which also seems like a good idea.
We believe will fix unicode output on windows issues:
https://github.com/bytecodealliance/wasmtime/issues/6824
This was removed in #6195 but re-added in #6877, I believe by accident,
so this re-deletes it. I've also edited `.gitmodules` a bit while I was
here to remove it and additionally keep other entries up-to-date with
matching paths.
Afonso Bordado
Alex Crichton
Trevor Elliott
Andrew Brown
Pat Hickey
Nick Fitzgerald
Rainy Sinclair
Ryan Levick
wasmtime-publish
Tyler Rockwood
Jia Yuan Lo
Eduardo de Moura Rodrigues
Adam C. Foltzer
Kevin Menard
Ulrich Weigand