* preview2: make everything but streams/io and poll/poll synchronous
* streams: get rid of as_any method, which is no longer used
* delete legacy sched and pollable concepts
* more code motion and renaming
* make tokio a workspace dep, because we need it directly in wasmtime-wasi
* HostPollable exists
* more fixes
* pollable can trap, and implement clock properly
* HostPollable is now a generator of futures
because we need to be able to poll a pollable many times
* explain various todo!s
* Synchronous version of the wasi-preview2-components tests
* Change with_tokio to accept the future as an argument
* Store futures in the PollOneoff struct instead, to avoid dropping them
* Remove TODO for HostOutputStream impl for WritePipe
* Implement pollable for ReadPipe
* Use a Notify when ReadPipe is ready
* wip
* wip
* Read/write pipe ends with tokio channels
* Empty reader/writer wrappers
* EmptyStream, and warning cleanup
* Wrapped reader/writer structs
* Rework stdio in terms of wrapped read/write
* Add MemoryOutputPipe and update tests
* Remove todo
* rewrite nearly everything
* implement the pipe stuff
* wibble
* fix MemoryOutputPipe just enough to make the tests compile
* Move the table iteration into a helper function
* AsyncFd stream implementation to fix stdin on unix
* Rename Wrapped{Read,Write} streams to Async{Read,Write}Stream
* Move async io wrappers into stream.rs
* Fix the sync tests
* fix test uses of pipes, juggle tokio context for stdin construction
* add some fixmes
* the future i named Never is defined in futures-util as pending
which is a better name
* i believe this is a correct implementation of one global stdin resource
* move unix stdin to its own file
* make most of the mods private
* fix build - we are skipping rust 1.70
due to llvm regressions in s390x and riscv64 which are fixed in 1.71 and
will not be backported
* preview1-in-preview2: use async funcs for io, and the async io interface
prtest:full
* windows stdin support
* done!
* table ext functions: fix tests
* tests: expect poll_oneoff_{files,stdio} to pass on all platforms
* export the bindings under wasmtime_wasi::preview2::bindings
rather than preview2::wasi.
and command moves to wasmtime_wasi::preview2::command as well.
* fix renaming of wasi to bindings in tests
* use block_in_place throughout filesystem
and move block_on and block_in_place to be pub crate at the root
* AsyncFdStream: ensure file is nonblocking
* tests: block_in_place requires multi-threaded runtime
* actually, use fcntl_setfl to make the asyncfd file nonblocking
* fix windows block_on
* docs, remove unnecessary methods
* more docs
* Add a workspace dependency on bytes-1.4
* Remove vectored stream operations
* Rework the read/write stream traits
* Add a size parameter to `read`, and switch to usize for traits
* Pipe through the bool -> stream-status change in wit
* Plumb stream-status through write operations in wit
* write host trait also gives streamstate
* hook new stream host read/write back up to the wit bindgen
* sketchy AsyncReadStream impl
* Fill out implementations for AsyncReadStream and AsyncWriteStream
* some reasonable read tests
* more
* first smoke test for AsyncWriteStream
* bunch of AsyncWriteStream tests
* half-baked idea that the output-stream interface will need a flush mechanism
* adapter: fixes for changes to stream wit
* fix new rust 1.71 warnings
* make stdin work on unix without using AsyncFdStream
inline the tokio docs example of how to impl AsyncRead for an AsyncFd,
except theres some "minor" changes because stdin doesnt impl Read on
&Stdin whereas tcpstream from the example does
* delete AsyncFdStream for now
it turns out to be kinda hard and we can always work on adding it back
in later.
* Implement some memory pipe operations, and move async wrappers to the pipe mod
* Make blocking_write actually block until everything is written
* Remove debug print
* Adapter stdio should use blocking write
Rust guests will panic if the write returns less than the number of
bytes sent with stdio.
* Clean up implementations of {blocking_}write_zeros and skip
* Remove debug macro usage
* Move EmptyStream to pipe, and split it into four variants
Use EmptyInputStream and SinkOutputStream as the defaults for stdin and
stdout/stderr respectively.
* Add a big warning about resource lifetime tracking in pollables
* Start working through changes to the filesystem implementation
* Remove todos in the filesystem implementation
* Avoid lifetime errors by moving blocking operations to File and Dir
* Fix more lifetime issues with `block`
* Finish filling out translation impl
* fix warnings
* we can likely eliminate block_in_place in the stdin implementations
* sync command uses sync filesystem, start of translation layer
* symc filesystem: all the trait boilerplate is in place
just need to finish the from impl boilerplate
* finish type conversion boilerplate
* Revert "half-baked idea that the output-stream interface will need a flush mechanism"
This reverts commit 3eb762e333.
* cargo fmt
* test type fixes
* renames and comments
* refactor stream table internals so we can have a blocking variant...
* preview1 host adapter: stdout/stderr use blocking_write here too
* filesystem streams are blocking now
* fixes
* satisfy cargo doc
* cargo vet: dep upgrades taken care of by imports from mozilla
* unix stdio: eliminate block_in_place
* replace private in_tokio with spawn, since its only used for spawning
* comments
* worker thread stdin implementation can be tested on linux, i guess
and start outlining a test plan
* eliminate tokio boilerplate - no longer using tokios lock
* rename our private block_on to in_tokio
* fill in missing file input skip
* code review: fix MemoryInputPipe. Closed status is always available immediately.
* code review: empty input stream is not essential, closed input stream is a better fi for stdin
* code review: unreachable
* turn worker thread (windows) stdin off
* expect preview2-based poll_oneoff_stdio to fail on windows
* command directory_list test: no need to inherit stdin
* preview1 in preview2: turn off inherit_stdio except for poll_oneoff_stdio
* wasi-preview2-components: apparently inherit_stdio was on everywhere here as well. turn it off
except for poll_oneoff_stdio
* extend timeout for riscv64 i suppose
---------
Co-authored-by: Trevor Elliott <telliott@fastly.com>
* Fix signatures registered with modules-in-components
This commit fixes a minor issue in
`FunctionIndices::link_and_append_code` which previously ended up only
filling out the `wasm_to_native_trampolines` field for the first module
rather than all the modules. Additionally the first module might have
too many entries that encompass all modules instead of just its own
entries. The fix in this commit is to refactor this logic to ensure that
the necessary maps are present for all translations.
While technically a bug that can be surfaced through the embedder API
it's pretty obscure. The given test here panics beforehand but succeeds
afterwards, but this is moreso prep for some future resource-related
work where this map will need persisting into the component metadata
side of things.
* Initial support for resources
Lots of bits and pieces squashed into this commit. Much to be done
still.
* Start supporting destructors
* Get some basic drop tests working
Also add a test which requires host-defined drop to be called which
isn't working.
* Fix rebase issue
* Fix a failing test
* I am zorthax, destroyer of resources
* Remove a branch in compiled code
No need to check for a null funcref when we already know ahead of time
if it's ever going to be null or not.
* Fix the test suite
* Add embedder API to destroy resources
* Add TODO for factc
* Fix a warning and leave a comment
* Integrate resources into `Type`
Plumb around dynamic information about resource types.
* Implement `Val::Own`
* Implement reentrance check for destructors
Implemented both in the raw wasm intrinsic as well as the host.
* Use cast instead of transmute
* Fill out some cranelift-shared comments
* Update codegen for resource.drop shim
The MAY_ENTER flag must always be checked, regardless of whether there's
an actual destructor or not.
* Update wasm-tools crates to latest `main`
* Update resource.drop binary format
* Add some docs
* Implement dynamic tracking for borrow resources
Not actually hooked up anywhere but this should at least be a first stab
at an implementation of the spec.
* Remove git overrides
* Remove no-longer-needed arms in wit-bindgen
* Prepare for mutability in `LiftContext`
* Change `&LiftContext` to `&mut LiftContext`
* Remove `store: &'a StoreOpaque` from `LiftContext`, instead storing
just `memory: &'a [u8]`
* Refactor methods to avoid needing the entire `StoreOpaque`
This'll enable `LiftContext` to store `&'a mut ResourceTable` in an
upcoming commit to refer to the host's resources.
* Lowering a borrow is infallible
* Use `ResourceAny` for both own/borrow
Rename `Val::Own` to `Val::Resource` accordingly.
* Initial implementation of borrowed resources
Lots of juggling of contexts here and there to try and get everything
working but this is hopefully a faithful implementation. Tests not
implemented yet and will come next and additionally likely update
implementation details as issues are weeded out.
* Add a suite of tests for borrowing resources
Code coverage was used to ensure that almost all of the various paths
through the code are taken to ensure all the basic bases are covered.
There's probably still lurking bugs, but this should be a solid enough
base to start from hopefully.
* Fill in an issue for bindgen todo
* Add docs, still more to go
* Fill out more documentation
* Fill out a test TODO
* Update the host `Resource<T>` type
* Add docs everywhere
* Don't require a `Store` for creating the resource or getting the
representation.
The latter point is the main refactoring in this commit. This is done in
preparation for `bindgen!` to use this type where host bindings
generally do not have access to the store.
* Document `ResourceAny`
* Debug assert dtor is non-null
* Review comments on loading libcalls
* Update some comments
* Update a comment
* Fix some typos
* Add a test that host types are the same when guest types differ
* Fix some typos
* Thread things through a bit less
* Undo CompileKey-related changes
* Gate an async function on the async feature
* Fix doc links
* Skip resources tests in miri
They all involve compilation which takes too long and doesn't currently
work
* Change the call to unwrap to except with an error message
* Update crates/c-api/src/linker.rs
* Run rustfmt.
---------
Co-authored-by: Peter Huene <peter@huene.dev>
* Cap the maximum basic block padding in Cranelift
This commit is a fix for a fuzz failure that came in the other day where
a module took more than 2GB of memory to compile, triggering the fuzzer
to fail. Investigating locally it looks like the problem lies in the
basic-block padding added recently. Turning on that option increases the
memory usage of Cranelift from ~70M to ~2.5G. The setting in the test
was to add `1<<14` bytes of padding between all basic blocks, and with a
lot of basic blocks that can add up quite quickly.
The solution implemented here is to implement a per-function limit of
the amount of padding that can be injected. I've set it to 1MB for now
which is hopefully enough to continue to stress the various bits of the
`MachBuffer` while not blowing limits accidentally while fuzzing.
After this commit the fuzz test case in question jumps from 70M to 470M with
basic block padding enabled, which while still large is a bit more
modest and sticks within the limits of the fuzzer.
* Review feedback
* Fix typo
* Tweak some constants
Currently wasmtime's error message for `wasmtime nonexistent.wasm`
doesn't actually refer to `nonexistent.wasm` anywhere, so this commit
fills that in as contextual information on the returned error.
* Remove deny.toml exception for wasm-coredump-builder
This isn't used any more so no need to continue to list this.
* Update Wasmtime's pretty_env_logger dependency
This removes a `deny.toml` exception for that crate, but `openvino-sys`
still depends on `pretty_env_logger 0.4.0` so a new exception is added
for that.
* Update criterion and clap dependencies
This commit started out by updating the `criterion` dependency to remove
an entry in `deny.toml`, but that ended up transitively requiring a
`clap` dependency upgrade from 3.x to 4.x because `criterion` uses
pieces of clap 4.x. Most of this commit is then dedicated to updating
clap 3.x to 4.x which was relatively simple, mostly renaming attributes
here and there.
* Update gimli-related dependencies
I originally wanted to remove the `indexmap` clause in `deny.toml` but
enough dependencies haven't updated from 1.9 to 2.0 that it wasn't
possible. In the meantime though this updates some various dependencies
to bring them to the latest and a few of them now use `indexmap` 2.0.
* Update deps to remove `windows-sys 0.45.0`
This involved updating tokio/mio and then providing new audits for new
crates. The tokio exemption was updated from its old version to the new
version and tokio remains un-audited.
* Update `syn` to 2.x.x
This required a bit of rewriting for the component-macro related bits
but otherwise was pretty straightforward. The `syn` 1.x.x track is still
present in the wasi-crypto tree at this time.
I've additionally added some trusted audits for my own publications of
`wasm-bindgen`
* Update bitflags to 2.x.x
This updates Wasmtime's dependency on the `bitflags` crate to the 2.x.x
track to keep it up-to-date.
* Update the cap-std family of crates
This bumps them all to the next major version to keep up with updates.
I've additionally added trusted entries for publishes of cap-std crates
from Dan.
There's still lingering references to rustix 0.37.x which will need to
get weeded out over time.
* Update memoffset dependency to latest
Avoids having two versions in our crate graph.
* Fix tests
* Update try_from for wiggle flags
* Fix build on AArch64 Linux
* Enable `event` for rustix on Windows too
Currently Cargo's interpretation of prerelease version doesn't quite
match semver's, meaning that hyper, which is taking semver's
interpretation, has broken Wasmtime builds without lock files. The rc.4
version of hyper released is no longer compatible with our
usage which means that `cargo install wasmtime-cli` currently fails. It
passes, however, with `--locked`.
To resolve this the dependency on hyper gets an `=` dependency for now
to force it to be at this version.
* Update wasm-tools dependencies
* Get tests passing after wasm-tools update
Mostly dealing with updates to `wasmparser`'s API.
* Update `cargo vet` for new crates
* Add `equivalent`, `hashbrown`, and `quote` to the list of trusted
authors. We already trust these authors for other crates.
* Pull in some upstream audits for various deps.
* I've audited the `pulldown-cmark` dependency upgrade myself.
* Fix some warnings on nightly Rust
* One is to avoid using `drop(&mut T)`
* Another is to avoid using `()` in FFI functions since apparently
that's not recommended. Working around this in the macro was pretty
tricky, however.
* Fix a typo
* Review feedback
* Attempt versioned exports to facilitate having multiple versions in the same crate
* Modify approach to use `export_name` and `link_name`
* Only apply version to names in assembly and foreign item fns
* Attempt to handle the s390x case
* Fix alignment of backslashes in assembly file
* Pretend I understand the preprocessor
* Version symbols in `crates/runtime/src/helpers.c`
* Stop versioning `__jit_debug_register_code` because gdb relies on it and it is uses `weak` linkage
* Version symbol in `crates/fiber/src/windows.c`
* Consolidate `LitStr` creation in macro
* Add new crate to publish script and supply-chain config
* Fix order in supply chain config
* Set `audit-as-crates-io` to false
* Missing `versioned_link` for Windows
* Version strings used in debug
* Formatting
* Get rid of `versioned_str` and bring back `versioned_suffix`
---------
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
* Update calling conventions for wasm functions slightly
This resolves two issues from recent changes in #6649:
* First the s390x calling convention for wasm functions is changed back
to `WasmtimeSystemV` from `Fast`. This was an accidental omission from
#6649 where the conclusion was that s390x will continue using a
calling convention with little-endian lane order for lane arguments.
The only calling convention that supports this today is
`WasmtimeSystemV`, although the `Tail` calling convention will likely
use it in the future as well.
* Second the apple-aarch64 platform now uses the `Fast` calling
convention instead of `AppleAarch64` calling convention. That
convention was specified in #4195 but local testing shows that is not
necessary in the sense that tests all pass with the `Fast` calling
convention. This means that the prior comment why the `AppleAarch64`
calling convention is required is no longer accurate and in the
absence of a reason not to I went ahead and switched it to `Fast`.
In the near future all wasm functions will unconditionally use the
`Tail` calling convention and at that time the lane order can be
specified on s390x to be little-endian to satisfy all the constraints
here. Additionally any unwinding directives, if necessary for aarch64,
can be specified as needed.
* Fix compile
* Fix signatures registered with modules-in-components
This commit fixes a minor issue in
`FunctionIndices::link_and_append_code` which previously ended up only
filling out the `wasm_to_native_trampolines` field for the first module
rather than all the modules. Additionally the first module might have
too many entries that encompass all modules instead of just its own
entries. The fix in this commit is to refactor this logic to ensure that
the necessary maps are present for all translations.
While technically a bug that can be surfaced through the embedder API
it's pretty obscure. The given test here panics beforehand but succeeds
afterwards, but this is moreso prep for some future resource-related
work where this map will need persisting into the component metadata
side of things.
* Fix build warning
* Store all instantiation arguments in a component
This commit updates how instantiation arguments are rooted within a
component instance. Previously a list of import arguments was "unzipped"
into a list of modules and a list of functions. This is becoming a bit
more cumbersome in an upcoming change for resources where a third style
of import is being added, so instead of this unzipping operation this
commit instead replaces the list of imports with an `Arc`. This way
instantiation is actually a bit cheaper since only one `Arc` needs to be
retained instead of each individual argument imported into a module.
Additionally this refactors the way that exported modules are handled by
avoiding translating everything into one list and instead continuing to
store modules in two separate lists: those in the component and those
imported.
* Add some more words
* handle interface functions correctly in `component::Linker::func_new`
Many months ago, I implemented `func_new`, but only supporting top-level
function imports. If you tried to link a host function under an imported
interface, it would mistakenly treat it as a top-level function and either error
out if it couldn't find a corresponding type definition in the passed
`&Component`; or, if it found a top-level function that happened to have the
same name, it would use that type (which would coincidentally work if the type
happens to match, but lead to a runtime error later on otherwise).
This fixes the issue by looking up the correct component instance when necessary
and getting the type from there.
Note I've made no effort to optimize for performance here. Happy to revisit
that if there's a need.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* print names in `func_new`, not intern indexes
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* use `Vec` instead of linked list in `LinkerInstance`
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
---------
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
* Fix a typo in an `unsafe impl Send/Sync`
This commit fixes a mistake in the `PrePatchedFuncRef` type where it has
an `unsafe impl` to make it send/sync but the target of the impl was
mistakenly `InstancePre<T>`
Note that this doesn't actually have any impact on the send/sync-ness of
`InstancePre<T>` since it's not storing an instance of `T`, so it's
always `Send`/`Sync` anyway. Nevertheless this reduces the scope of
unsafety slightly as was originally intended.
* Remove the `PrePatchedFuncRef` entirely
No longer necessary for `Send`/`Sync`-ness any more
* Update v8 and proc-macro2 dependencies
Gets them both compiling on the latest nightly so we can unpin the Rust
compiler version in OSS-Fuzz.
* Update nightly in CI
* Add WasiCtxBuilder setters for the two clock types
Signed-off-by: Ryan Levick <ryan.levick@fermyon.com>
* Move two clocks to dedicated fields in WasiCtx
* Rename clock traits to Host prefix convention
Signed-off-by: Ryan Levick <ryan.levick@fermyon.com>
* Fix tests
---------
Signed-off-by: Ryan Levick <ryan.levick@fermyon.com>
* Shepherd along type information in lifting/lowering
This commit is a large refactor to the component runtime of Wasmtime to
shepherd along type information when lifting and lowering values.
Previously lifting and lowering would assume type information given
context such as "surely lowering a `u32` must lower into the type
`InterfaceType::U32`" or "lowering a `Val` works as it knows its own
type". This is still true, and this commit isn't changing these
features. The rationale for this commit instead stems from the upcoming
implementation of resources in Wasmtime.
Resources are trickier than all existing types in Wasmtime because what
exactly is the type of a resource depends on who you're asking. For
example the host might have one type called `http::Headers` but a
component could import it as two distinct types:
(component
(import "headers1" (type $h1 (sub resource)))
(import "headers2" (type $h2 (sub resource)))
;; ...
)
in the above component the `$h1` and `$h2` types will each get their own
table at runtime for managing their state. This means that if the host
instantiates the component with `http::Headers` supplied as both types
then the same type on the outside maps to two different types inside.
This means that the lowering of a host-defined type into the component
is now dependent on the "name" that the component has for the type,
basically if the function used `$h1` or `$h2`. This overall means that
the type that the component assigned for a function is significant as
part of lifting and lowering. Hence the rationale for this commit,
threading around this type information.
The major change in this commit is updates to the `Lift` and `Lower`
traits. Previously they took a mishmash of parameters and now they
needed to take more parameters, so I've updated them with:
* `Lift` operations take a `&LiftContext<'_>` and an `InterfaceType` as
contextual information. The context stores the store, the options, and
type information. The `InterfaceType` is the type that's being lifted,
which would indicate which resource table to load from for example.
* `Lower` operations now take a `&mut LowerContext<'_, T>` and an
`InterfaceType`. The `LowerContext` is similar to its lift cousin
where it stores the store, options, and type information.
The different context passed in to `lift` and `load`, for example, is no
longer distinguished and both simply take a `&LiftContext<'_>` which
simplifies things a bit.
This refactoring was pretty far reaching and touches quite a bit of the
component model implementation. This is because basically everything
deals with type information as types can be recursively nested in one
another. I've taken the liberty to make code continue to be
ergonomic/understandable where appropriate so some "shapes" of code are
now different to continue to accommodate readability and
maintainability.
Finally it's worth noting that this should not have any actual function
impact on components running today (or tomorrow). User-facing APIs
haven't changed at all here and it's just the guts that are refactored.
One unfortunate aspect, though, is that this is going to be a small perf
hit on lifting/lowering due to the fact that type information
essentially needs to be "iterated" over during the lifting/lowering
process. This iteration involves index lookups in `&ComponentTypes`
along with assertions that when you lower `Vec<T>` that the type is
`InterfaceType::List(i)`. These assertions should always succeed, and in
theory could become some sort of `unreachable_unchecked` in the future,
but for now it's all left as safe checks/panics for us to optimize at a
later date if necessary.
* Fill out a TODO comment
* Fill out more comments
Fixing typo of wasmtime_wasi_nn -> wasmtime_wasi_crypto that was causing
a build error when building the bench-api with the wasi-crypto feature
enabled.
* Remove Wasmtime ABIs from Cranelift
This commit removes the `Wasmtime*` family of ABIs from Cranelift. These
were originally added to support multi-value in Wasmtime via the
`TypedFunc` API, but they should now no longer be necessary. In general
this is a higher-level Wasmtime concern than something all backends of
Cranelift should have to deal with.
Today with recent refactorings it's possible to remove the reliance on
ABI details for multi-value and instead codify it directly into the
Cranelift IR generated. For example wasm calls are able to have a
"purely internal" ABI which Wasmtime's Rust code doesn't see at all, and
the Rust code only interacts with the native ABI. The native ABI is
redefined to be what the previous Wasmtime ABIs were, which is to return
the first of a 2+ value return through a register (native return value)
and everything else through a return pointer.
* Remove some wasmtime_system_v usage in tests
* Add back WasmtimeSystemV for s390x
* Fix some docs and references in winch
* Fix another doc link
* poll.wit: we can returl a list<bool> now
* adapter: fixes for poll-oneoff returning list<bool>
* wasi preview2: fixes for poll-oneoff returning list<bool>
* adapter: manually import poll-oneoff to avoid pulling in std to allocate return vec
* comment describing the skip functions
* preview2: refactor WasiCtxBuilder impl, and make WasiCtx fields private
The optional-fields WasiCtxBuilder was an intermediate stepping stone
which has outlived its usefulness - there is now only one sensible
default value for each field, so we no longer need to expose the Default
(empty) constructor.
There is no need for the WasiCtx::builder method being an alias for
default, and it creates user confusion.
Finally, the WasiCtx itself is a private implementation detail for use
in this crate only. The WasiCtxBuilder is the only way to customize
its contents. We don't want to let users modify the fields of WasiCtx
after it has been used by a wasm guest, because the guest may have made
assumptions that fields won't change - e.g. the stderr field is fetched
once by the adapter and assumed to always be the same, environment
variables are copied into libc once and assumed to always be the same.
* fix tests
* [wit-bindgen] provide more control over type ownership
This replaces the `duplicate_if_necessary` parameter to
`wasmtime::component::bindgen` with a new `ownership` parameter which provides
finer-grained control over whether and how generated types own their fields.
The default is `Ownership::Owning`, which means types own their fields
regardless of how they are used in functions. These types passed by reference
when used as parameters to guest-exported functions. Note that this also
affects how unnamed types (e.g. `list<list<string>>`) are passed: using a
reference only at the top level (e.g. `&[Vec<String>]` instead of `&[&[&str]]`,
which is more difficult to construct when using non-`'static` data).
The other option is `Ownership::Borrowing`, which includes a
`duplicate_if_necessary` field, providing the same code generation strategy as
was used prior to this change.
If we're happy with this approach, I'll open another PR in the `wit-bindgen`
repo to match.
This also fixes a bug that caused named types to be considered owned and/or
borrowed when they shouldn't have been due to having fields with unnamed types
which were owned and/or borrowed in unrelated interfaces.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* fix test breakage
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
---------
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* Lean on wasmparser's type information when translating components
Before this commit Wasmtime would build up its own representation of
type information independently of wasmparser, effectively duplicating
things such as type scopes and managing indices. This to some degree is
required because Wasmtime's type information is serialized into compiled
images and wasmparser's information isn't easily serialized. With the
advent of resources in components, however, the task of doing this
correctly has become much more difficult.
When component translation was first written it was more difficult to
acquire type information from `wasmparser::Validator`. Nowadays there's
helpful type information exposed every step of the way which makes it
much easier to get at the types of items while we're translating rather
than only at the very end (or not at all). This is one of the
motivations for this commit where now it's possible to avoid duplicating
the work `wasmparser` is doing whereas before it wasn't as easy.
Additionally with resources in the component model they perform namely a
number of involved type substitutions during instantiation of arbitrary
components which isn't implemented in Wasmtime today. Some of this will
be required for Wasmtime to correctly implement resources so instead of
doing all that again I've decided to replace Wasmtime's management of
types with wasmparser's management of types.
The main difference in this commit is that Wasmtime no longer tracks
type information during translation and that conversion into Wasmtime's
type hierarchy now has a different entry point. Previously conversion
would happen based on raw types read from the wasm file (think
index-based things) whereas now translation happens on `wasmparser`'s
parsed and validated hierarchy of types (think ID-based things rather
than index-based things). This makes translation slightly more involved
but overall it's largely performing the same work.
One gotcha with this PR is that core wasm modules using GC types and
typed function references could theoretically have worked previously but
they no longer work. It turns out that `wasmparser` is not correctly
surfacing type information in components for core modules that use GC
types, namely because `wasmparser`'s validated type hierarchy uses the
same core wasm types as what's read raw from the type section. This in
turn means that the index-based format can't be resolved. This is a bug
in `wasmparser` which will need resolving but is a big chunk of work to
take on, so for now the component model will panic on these sorts of
modules (which are disabled by default anyway).
Overall the end-goal of this commit is to ease the implementation of
resources a bit by heavily relying on `wasmparser`'s understanding of
resources, chiefly the functionality of performing type substitutions on
subcomponent instantiations.
* Fix a warning
* Fix factc build
* Refactor compilation of component-related functions
Extract some common functionality out to helper functions to reduce the
boilerplate involved in compiling component-related functions. This is
in preparation for upcoming resource-related work which adds more
functions to compile.
* Fix a typo
* x64: Fix ISA requirement for `pmaddubsw`
This was erroneously listed as SSE2 but this is actually an SSSE3
instruction. This commit updates the ISA requirement as well as related
lowerings to have everything work with an SSE2 baseline as well which is
mostly plumbing to make sure the relaxed-simd dot product instructions
use it when it's available but otherwise fall-back to the deterministic
lowering.
* Add require ssse3 feature
* Fix preexisting wat test
This commit removes a setting for Cranelift which I've found a bit
confusing historically and I think is no longer necessary. The setting
is currently documented as enabling SIMD instructions, but that only
sort of works for the x64 backend and none of the other backends look at
it. Historically this was used to flag to Cranelift that a higher x64
baseline feature set is required for codegen but as of #6625 that's no
longer necessary.
Otherwise it seems more Cranelift-like nowadays to say that vector
instructions generate SIMD instructions where non-vector instructions
probably don't, but may still depending on activated CPU features. In
that sense I'm not sure if a dedicated `enable_simd` setting is still
motivated, so this PR removes it.
This renames some features in the x86 backend such as `use_avx_simd` to
`use_avx` since the `_simd` part is no longer part of the computation
now that `enable_simd` is gone.
All instructions in Cranelift now have lowerings for SSE2 as a baseline,
even if they're not exactly the speediest things in the world. This
enables lowering the baseline required for the SIMD proposal for
WebAssembly to SSE2, the base features set of x86_64. Lots of tests were
updated here to remove explicit `has_foo=false` annotations as they no
longer have any effect.
Additionally fuzzing has been updated to enable disabling `sse3` and
`ssse3` which will help stress-test all previously-added lowerings.
* Add a `set_random` function to `WasiCtxBuilder`.
Add a `set_random` function to `WasiCtxBuilder`, allowing users that
construct a default interface via `WasiCtxBuilder::default()` or
`WasiCtx::builder()` to initialize the `random` state.
Fixes#6576.
* Rename `set_random_for_testing` to `set_secure_random_to_custom_generator`
Remove the `cfg(test)`, and add comments advising users of the potential
hazards.
* rustfmt
* Fix documentation links.
This commit fixes a test that has failed on CI and seems flaky. This
test asserts that stderr/stdout are writable and that a 200ms timeout
does not elapse when calculating this. The implementation, on Windows at
least, of `poll_oneoff` always reports stdout/stderr as "ready for
write" meaning that this test should trivially pass. The flakiness comes
from the timeout parameter where apparently sometimes on CI the
determination of this takes more than 200ms. This means that the timer
subscription may or may not fire depending on the timing of the test,
and currently the test always fails if the timeout subscription fires.
This commit updates the test to store whether a timeout has passed and
only fail if `poll_oneoff` is attempted when the timeout has already
elapsed. This will allow the timeout to fire so long as the two streams
are considered writable at the same time, achieving the desired result
of the test to assert that, without timing out both stdout and stderr
are considered writable.
* Fix inconsistent deadlines for monotonic clock timeouts
Rename `MonotonicClockSubscription`'s `deadline` field to
`absolute_deadline` and change the code that computes the value to
compute an absolute time rather than a relative time, so that it's
interpreted consistently everywhere.
Fixes#6588.
* Polling on stdio is not yet implemented on Windows.
* x64: Add non-SSSE3 lowerings of `pshufb`
Or, more accurately, add lowerings which don't use `pshufb`'s
functionality at all where possible or otherwise fall back to a new
libcall. This particular instruction seemed uniquely difficult to
implement in the backend so I decided to "cop out" and use libcall
instead. The libcall will be used for `popcnt`, `shuffle`, and
`swizzle` instructions when SSSE3 isn't available.
* Implemente SSE2 popcnt with Hacker's Delight
* x64: Implement passing vector arguments in the fastcall convention
Windows says that vector arguments are passed indirectly so handle that
here through the `ABIArg::ImplicitPtrArg` variant. Some additional
handling is added to the general machinst backend.
* Update `gen_load_base_offset` for x64
* Fill out remaining bits of fastcall and vector parameters
* Remove now-unnecessary `Clone` bound
* Update adapter build
* Rename the binary artifact to `wasi_snapshot_preview1.wasm` and update
build scripts to account for this.
* Update documentation to mention difference between reactor/command
builds.
Closes#6569
* More renaming
* Force `execute_across_threads` to use multiple threads
Currently this uses tokio's `spawn_blocking` but that will reuse threads
in its thread pool. Instead spawn a thread and perform a single poll on
that thread to force lots of fresh threads to be used and ideally stress
TLS management further.
* Add a guard against using stale stacks
This commit adds a guard to Wasmtime's async support to double-check
that when a call to `poll` is finished that the currently active TLS
activation pointer does not point to the stack that is being switched
off of. This is attempting to be a bit of a defense-in-depth measure to
prevent stale pointers from sticking around in TLS. This is currently
happening and causing #6493 which can result in unsoundness but
currently is manifesting as a crash.
* Fix a soundness issue with the component model and async
This commit addresses #6493 by fixing a soundness issue with the async
implementation of the component model. This issue has been presence
since the inception of the addition of async support to the component
model and doesn't represent a recent regression. The underlying problem
is that one of the base assumptions of the trap handling code is that
there's only one single activation in TLS that needs to be pushed/popped
when a stack is switched (e.g. a fiber is switched to or from). In the
case of the component model there might be two activations: one for an
invocation of a component function and then a second for an invocation
of a `realloc` function to return results back to wasm (e.g. in the case
an imported function returns a list).
This problem is fixed by changing how TLS is managed in the presence of
fibers. Previously when a fiber was suspended it would pop a single
activation from the top of the stack and save that to get pushed when
the fiber was resumed. This has the benefit of maintaining an entire
linked list of activations for the current thread but has the problem
above where it doesn't handle a fiber with multiple activations on it.
Instead now TLS management is done when a fiber is resumed instead of
suspended. Instead of pushing/popping a single activation the entire
linked list of activations is tracked for a particular fiber and stored
within the fiber itself. In this manner resuming a fiber will push
all activations onto the current thread and suspending a fiber will pop
all activations for the fiber (and store them as a new linked list in
the fiber's state itself).
This end result is that all activations on a fiber should now be managed
correctly, regardless of how many there are. The main downside of this
commit is that fiber suspension and resumption is more complicated, but
the hope there is that fiber suspension typically corresponds with I/O
not being ready or similar so the order of magnitude of TLS operations
isn't too significant compared to the I/O overhead.
Closes#6493
* Review comments
* Fix restoration during panic
* Allow async yield from epoch interruption callback
When an epoch interruption deadline arrives, previously it was possible
to yield to the async executor, or to invoke a callback on the wasm
stack, but not both. This changes the API to allow callbacks to run and
then request yielding to the async executor.
* Fix Wasmtime C API implementation
* Upgrade file-per-thread-logger to v0.2.0
Signed-off-by: Benjamin Bouvier <public@benj.me>
* Update audits too
Signed-off-by: Benjamin Bouvier <public@benj.me>
---------
Signed-off-by: Benjamin Bouvier <public@benj.me>
This change relaxes what kinds of modules can be run when wasi-threads
is enabled via `--wasi-modules experimental-wasi-threads`. Previously,
as reported in #6153, simple modules that made no use of thread spawning
or shared memories were preemptively rejected when the wasi-threads
context was created. This is too restrictive.
Instead, this change does the following:
- it moves the check for whether a module is valid according to the
wasi-threads specification to the point a new thread is spawned; this
resolves#6153
- as noted in #6153, this change also adds a better error message
indicating that wasi-threads expects a shared memory import
- the way this is implemented also improves the module instantiation: by
constructing an `InstancePre` once when the `WasiThreadsCtx` is built,
we might shave off a bit of time from the "spawn a thread" call;
this supercedes a similar effort in #5741
This commit removes the SSE4.1 requirement for the `enable_simd` CLIF
feature. This means that the new baseline required is SSSE3 for the
WebAssembly SIMD proposal. Many existing tests for codegen were all
updated to explicitly enable `has_sse41` and runtests were updated to
test with and without SSE 4.1.
Wasmtime's fuzzing is additionally updated to flip the SSE4.1 feature to
enable fuzz-testing this.
Pat Hickey
Nick Fitzgerald
Alex Crichton
MindSwipe
Marco Concetto Rudilosso
Adam Petro
Joel Dice
wasmtime-publish
Ryan Levick
Stephen Doyle
Dan Gohman
Trevor Elliott
Doug A
Iceber Gu
Jamey Sharp
Benjamin Bouvier
Brendan Burns
Andrew Brown