* Use generated bindings to run components in the CLI
This commit updates the component support in the `wasmtime` CLI
from #6836 to use the generated bindings for the "command" style of
components rather than having that custom-written in the CLI.
* Fix tests
* Update release date of Wasmtime 12.0.0
* Update date to today
---------
Co-authored-by: Wasmtime Publish <wasmtime-publish@users.noreply.github.com>
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
This commit refactors the `wasmtime` CLI executable to be able to
support not only compiling components but additionally executing
components. While I was doing this I've additionally added a new
`--preview2` argument to enable using the new experimental
implementation of preview1 based on preview2 type/structs. This is
off-by-default but is expected to become the default in the future.
Some notable features of this change are:
* The preview1-implemented-with-preview2 module now sports
`add_to_linker_{async,sync}` to replace the previous `add_to_linker`
which always did async.
* Some trait bounds in the preview1-implemented-with-preview2 module are
simplified.
* Some minor changes were made to `wiggle`'s macros to support a "block
on" that isn't the default wiggle dummy executor (as now we actually
do need Tokio)
* Many options related to core wasm `Linker` configuration, such as
`--default-values-unknown-imports`, are not implemented for components
at this time. When used with components these options return an error.
* Construction of WASI contexts has been refactored to pass around fewer
arguments to avoid threading through lots of values for both preview1
and preview2.
* Reading the input to the Wasmtime CLI has been updated to read the
input in the `run` subcommand before handing it off to the `wasmtime`
crate's API to enable the CLI to use the contents of what's loaded to
determine what to do next.
* Our generic `./ci/run-tests.sh` script has been updated to pass the
`--features component-model` flag to ensure that this CLI support is
tested during the normal test suite.
* The CLI support for `wasi-nn` supports components as well as core wasm
modules.
* Wasmtime: Rename `IndexAllocator` to `ModuleAffinityIndexAllocator`
We will have multiple kinds of index allocators soon, so clarify which one this
is.
* Wasmtime: Introduce a simple index allocator
This will be used in future commits refactoring the pooling allocator.
* Wasmtime: refactor the pooling allocator for components
We used to have one index allocator, an index per instance, and give out N
tables and M memories to every instance regardless how many tables and memories
they need.
Now we have an index allocator for memories and another for tables. An instance
isn't associated with a single instance, each of its memories and tables have an
index. We allocate exactly as many tables and memories as the instance actually
needs.
Ultimately, this gives us better component support, where a component instance
might have varying numbers of internal tables and memories.
Additionally, you can now limit the number of tables, memories, and core
instances a single component can allocate from the pooling allocator, even if
there is the capacity for that many available. This is to give embedders tools
to limit individual component instances and prevent them from hogging too much
of the pooling allocator's resources.
* Remove unused file
Messed up from rebasing, this code is actually just inline in the index
allocator module.
* Address review feedback
* Fix benchmarks build
* Fix ignoring test under miri
The `async_functions` module is not even compiled-but-ignored with miri, it is
completely `cfg`ed off. Therefore we ahve to do the same with this test that
imports stuff from that module.
* Fix doc links
* Allow testing utilities to be unused
The exact `cfg`s that unlock the tests that use these are platform and feature
dependent and ends up being like 5 things and super long. Simpler to just allow
unused for when we are testing on other platforms or don't have the compile time
features enabled.
* Debug assert that the pool is empty on drop, per Alex's suggestion
Also fix a couple scenarios where we could leak indices if allocating an index
for a memory/table succeeded but then creating the memory/table itself failed.
* Fix windows compile errors
* Add splat simplify opt for iadd, isub, imul, ineg & iabs
* Add splat simplify opt for smulhi & hmulhi
* Add splat simplify opt for band, bor & bxor
* Add splat simplify opt for a few more ops
namely, popcnt, smin, umin, smax, umax, sadd_sat, uadd_sat, ssub_sat & usub_sat
* Add splat simplify opt for bnot
* Add splat simplify opt for shift and rotate ops
and avg_round as well
* Remove splat simplify opt for certain ops
because they don't support scalars.
* Add lane_type to splat opts for shift/rotate
Some of the wasmtime spec tests were failing without this
* Remove splat simplify opt for avg_round
because it doesn't support scalars.
* stream wit definition eliminates stream-error
In https://github.com/WebAssembly/wasi-io/pull/38 we got review feedback
to eliminate the stream-error in favor of the empty error in wit
`result<a>`.
This means we cant use trappable error anymore, and therefore leads to
all this other unsightly transformation of the streams trait definition
and all its call sites.
We'll fix the wasmtime-wit-bindgen macro to support this case better in
the future, but rn we gotta stay synchronized with upstream
On the upside this showed us that the host stream trait design doesnt
differentiate between a runtime and a trapping error, so lets fix that
next
introduce a StreamRuntimeError, use it in filesystem streams
and fix an incorrect error transformation in the filesystem read impl
fill in fixmes for distinguishing a stream runtime error
delete outdated fixmes: downcast is now guaranteed by child resource tracking
* dont try to detect rustix io error - just call all read/write errors runtime
I don't think we should trap on any of the errors possible here,
reporting them as failures is sufficient
* wmemcheck: update docs.
This PR expands the documentation for the Wasm memchecker (`wmemcheck`)
feature significantly, and also links it from the top-level
documentation hierarchy.
* Add syntax/language annotations to quotation sections to keep mdbook happy.
This change removes the temporary `*.wit` files and bumps the `spec`
directory to the latest [wasi-nn] commit. This is now possible because
the upstream `spec` repository has all of the updated WIT and WITX bits.
The `load_by_name` implementations are left as TODOs for now and will be
included in a subsequent PR.
One other change is a refactoring: we wrap up some types--`Graph` and
`ExecutionContext`--to avoid passing around `Box<dyn ...>`. This
simplifies some of the code but should not change behavior.
Run all tests: prtest:full
[wasi-nn]: https://github.com/WebAssembly/wasi-nn.
* wasi-nn: refactor to allow `preview2` access
This change refactors the `wasmtime-wasi-nn` crate to allow access from
both `preview1` and `preview2` ABIs. Though the `wasi-nn` specification
has included a WIT description for some time, here we use some in-tree
files until https://github.com/WebAssembly/wasi-nn/pull/38 is landed.
The `preview2` code is not exercised anywhere yet: ideally this would be
wired up once component model `resource`s are fully implemented in
Wasmtime.
prtest:full
* wasi-nn: use `preview1` linkage
prtest:full
* review: rename `preview*` to `wit*`
This is based on @pchickey's [comments] on ABI naming.
[comments]: https://bytecodealliance.zulipchat.com/#narrow/stream/266558-wasi-nn/topic/wasi-nn.20.2B.20preview2/near/383368292
* review: update README
* fix: remove broken doc links
* fix: replace typo `wit` with `gen`
* review: use `wit` types everywhere
This removes the crate-specific types in order to use the WIT-generated
types throughout the crate. The main effect of this is that the crate
no longer optionally includes `wasmtime` with the `component-model`
feature--now that is required.
* review: move `BackendKind` conversion into `witx.rs`
* review: remove `<'a>`
* review: use `tracing` crate instead of `eprintln!`
* rename wasi-cli-base to wasi-cli, delete `preview` package, import wasi-cli
wasi-cli import is sum of
https://github.com/WebAssembly/wasi-cli/pull/19 and
https://github.com/WebAssembly/wasi-cli/pull/20
* wasi impl: change bindgen arguments and mod paths from cli_base to cli
* correct name of wasi-cli deps dir to just `deps/cli/`
it turns out this isnt semantically meaningful, since the package name
is in the document itself now, but lets be consistient
* track whether stdio isatty in ctx, and impl the cli/terminal-* interfaces
* rebase fixup
* wasi wits: define the reactor adapter's world
* component adapter: fixes
* test-programs/command-tests: fix renaming cli_base to cli
* component adapter: fix manually-defined export and import names
* test harness fixes
* preview1 component adapter: fill in isatty detection
* implement isatty in preview2-to-preview1 host adapter
* test-programs: cover both when stdio isatty and not
prtest:full
* split isatty test for regular file and stdio, detect host stdio is_terminal
CI environments vary - let the test runner make sure the host process's
stdio is in fact a terminal before asserting that the guest sees it is a
terminal.
* provide an is-terminal impl for all preview2's stdio types
which means making a newtype around Stdout and Stderr instead of using
a type alias there.
and then use the is-terminal impl to fill in the isatty field in the
builder when inheriting. if you need to override it you can always
builder.stdin(stdio::stdin(), your_own_idea_of_isatty)
* finally, rename IsATTY variants to Yes and No
* Fix the reference to IsATTY::No
* more forgotten renamings
---------
Co-authored-by: Trevor Elliott <telliott@fastly.com>
This commit prepares for the introduction of float support to Winch. Initially,
I intended to include this change as part of the original change supporting
floats, but that change is already sizable enough.
This modification simplifies the Assembler and MacroAssembler interfaces, as
well as the interaction and responsibilities between them, by:
* Eliminating the `Operand` abstraction, which didn't offer a substantial
benefit over simply using the MacroAssembler's `RegImm` and `Address`
abstractions as operands where necessary. This approach also reduces the number
of conversions required prior to emission.
* Shifting the instruction dispatch responsibility solely to the MacroAssembler,
rather than having this responsibility shared across both abstractions. This was
always the original intention behind the MacroAssembler. As a result, function
definitions at the Assembler layer become simpler.
This change also introduces richer type information for immediates, which
results in better instruction selection in some cases, and it's also needed to
support floats.
Not all file descriptors can get registered with epoll, for example
files on Linux or `/dev/null` on macOS return errors. In these
situations the fallback of the worker thread implementation is used
instead.
* Docs: Remove unneeded command in WASI-tutorial.md
The `cat /tmp/somewhere.txt` command in the example using --dir=$PWD is not needed. The error message should be after the first command anyway since it should be the one creating the message.
It looks like a copy/paste error since the previous example showed the same `cat` command a successful copy.
* Docs: Update error messages in WASI-tutorial.md.
The error messages appear to have changed since this document was originally written.
* Add support for vector in DataValueExt::int()
Fixes#6827
* Replace `if` with `match`
* Enable interpreter test for simd ineg
Issue #6827
* Format code with `cargo fmt`
and also improve comment
This commit adds a new `Engine::detect_precompiled` API to inspect some
bytes and determine if they look like a precompiled artifact of either a
core wasm module or component. This is something I'll be using soon in
an upcoming refactor of the Wasmtime CLI to support components, but it's
something we've also talked about before which can be useful for systems
storing both precompiled modules and components.
Implementation-wise this looks at the ELF header of the input and
determines if it's got all the right flags that Wasmtime sets for the
various bits and bobs of our object format.
* Use `Offset32` as `i32` in ISLE
This commit updates the x64 and aarch64 backends to use the `i32`
primitive type in ISLE when working with an `Offset32` instead of a
`u32`. This matches the intended representation of `Offset32` as a type
which is signed internally and represents how offsets on instructions
are often negative too.
This does not actually change any end results of compilation and instead
is intended to be "just" an internal refactoring with fewer casts and
more consistent handling of offsets.
* aarch64: Define the `PairAMode` type in ISLE
This commit moves the definition of the `PairAMode` enum into ISLE
instead of its current Rust-defined location. This is in preparation for
the next commit where all AMode calculations will be moved into ISLE.
* aarch64: Fix a copy/paste typo loading vectors
This commit fixes an assertion that can be tripped in the aarch64
backend where a 64-bit load was accidentally flagged as a 128-bit load.
This was found in future work which ended up tripping the assertion a
bit earlier.
* aarch64: Move AMode computation into ISLE
This commit moves the computation of the `AMode` enum for addressing
from Rust into ISLE. This enables deleting a good deal of Rust code in
favor of (hopefully) more readable ISLE code.
This does not mirror the preexisting logic exactly but instead takes a
different approach for generating the `AMode`. Previously the entire
chain of `iadd`s input into an address were unfolded into 32-bit and
64-bit operations and then those were re-combined as possible into an
`AMode` (possibly emitting `add` instructions. Instead now pattern
matching is used to represent this. The net result is that amodes are
emitted slightly differently here and there in a number of updated test
cases.
I've tried to verify in all test cases that the number of instructions
has not increased and the same logical operation is happening. The exact
`AMode` may differ but at least instruction-wise this shouldn't be a
regression. My hope is that if anything needs changing that can be
represented with updates to the rule precedence in ISLE or various other
special cases.
One part I found a little surprising was that the immediate with a
load/store instruction is not actually used much of the time. I naively
thought that the mid-end optimizations would move iadd immediates into
the load/store immediate but that is not the case. This necessitated two
extra ISLE rules to manually peel off immediates and fold them into the
load/store immediate.
* aarch64: Remove `NarrowValueMode`
This is no longer needed after the prior commit
* aarch64: Fix `return_call`'s interaction with pointer authentication
This commit fixes an issue where a `return_call` would not decrypt the
return address when pointer authentication is enabled. The return
address would be encrypted upon entry into the function but would never
get restored later on.
This addresses the issue by changing the encryption keys in use from the
A/B key plus SP to instead using A/B plus the zero key. The reason for
this is that during a normal function call before returning the SP value
is guaranteed to be the same as it was upon entry. For tail calls though
SP may be different due to differing numbers of stack arguments. This
means that the modifier of SP can't be used for the tail convention.
New `APIKey` definitions were added and that now additionally represents
the A/B key plus the modifier. Non-`tail` calling conventions still use
the same keys as before, it's just the `tail` convention that's different.
Closes#6799
* Fix tests
* Fix test expectations
* Allow `sign_return_address` at all times in filetests
This commit removes the option to generate padding between basic blocks
when fuzzing Wasmtime. Over the weekend lots of OOMs were discovered
related to this option and its most recent update in #6736. The new OOMs
appear to be related to:
* If multiple modules are generated then the configured limits in #6736
aren't relevant because they only cap one module.
* Each imported function generates a new trampoline which has its own
set of padding which wasn't previously accounted for.
* Spec tests have a lot of functions and the limits didn't account for
this.
While each of these is probably individually fixable I think it's
probably not worth the whack-a-mole any more. The `cranelift-fuzzgen`
target should cover the relevant cases for padding without the need for
Wasmtime's fuzzing to cover it as well.
* Configure Mach ports vs signals via `Config`
This commit adds a `Config::macos_use_mach_ports` configuration option
to replace the old `posix-signals-on-macos` compile-time Cargo feature.
This'll make Wasmtime a tad larger on macOS but likely negligibly so.
Otherwise this is intended to provide a resolution to #6785 where
embedders will be able to use any build of Wasmtime and configure at
runtime how trap handling should happen.
Functionally this commit additionally registers a `pthread_atfork`
handler to cause any usage of Wasmtime in the child to panic. This
should help head off a known-invalid state in case anyone runs into it
in the future.
* Fix build on non-macOS
* filesystem.wit: upstream main + pch/blocking_is_a_stream_concern + pch/metadata_hash_fixes
upstream main changes the interface to be named `types`, and has
changes to descriptor-stat, directory-entry, adds the metadata hash
methods.
pch/blocking_is_a_stream_concern upstreams the deletion of non-blocking
from descriptor flags.
pch/metadata_hash_fixes adds missing this: descriptor arguments to
the metadata-hash and metadata-hash-at methods.
* copy in preopens.wit and world.wit from wasi-filesystem repo, and delete wasi-cli-base/preopens.wit
* other wits: fix names of filesystem/{types, preopens}
* fix bindgen invocations and import paths for filesystem changes
* component adapter: fix bindings paths for preopens and filesystem
* wip
* wippp
* fill in a reasonable-ish metadata hash impl
* preview 1 adapter: we need async in the dir entries, so collect the whole thing to a vec
this isnt great but i dont have the time or energy to do this better
right now
* component adapter: use metadata hash to fake ino
* wasi-tests: fix warnings
* reactor-tests: fix filesystem paths
* only create a BuildHasher impl once
* consistiency
* component adapter: use descriptor to call filesystem funcs, not the fd
* reactor test: fix filesystem types paths
* metadata hash: use DefaultHasher
* fix missed trappable-error-type merge conflicts
* preview1-in-preview2: dont use to_le in Filestat assignment of ino
wiggle takes care of endianness translation. The to_le in fd_readdir
entries is because that implementation is writing structs directly to
memory (very cursed)
s390x CI caught this bug
* debugging: forward stdio for fd_readdir test so we can figure out CI failure on s390x
prtest:full
* Don't convert filestat ino to little-endian, as it's not written to memory
fd_readdir will write its results to a buffer, and needs to be careful
with endianness as a result. However, fd_filestat_get returns the value
directly, and as such does not need to call `to_le`.
---------
Co-authored-by: Trevor Elliott <telliott@fastly.com>
* Remove the implementation of wasi-crypto
This commit is a follow-up to the discussion on #6732. This removes
Wasmtime's implementation of the wasi-crypto proposal from in-tree along
with its various support in CI, configuration, etc. See the discussion
on #6732 for the full information but at a high level the main reasons
for removing the implementation at this time are:
* There is not currently an active maintainer of the Wasmtime
integration here for wasi-crypto.
* There are known issues with the code quality of the implementation
such as transmutes of guest-owned memory to `&'static mut [u8]` and
known unsafety in dependencies.
* The size and breadth of the dependency tree brings maintenance burden
and overhead to managing Wasmtime's dependency tree.
As mentioned on the issue this commit does not mean that Wasmtime
doesn't want to implement the wasi-crypto proposal. Instead the "tier 3"
status of wasi-crypto needs to be re-attained to be included back
in-tree, which would mean resolving the above issues.
Note that this commit is intentionally just after the 13.0.0 branch
point which means that this is slated for Wasmtime 14 to be released on
September 20.
* Remove some cfgs
* Remove wasi-crypto CI
* WIP: two-tier processing for MachBuffer veneer processing in island emission
* Unconditionally emit veneers for label forward-refs that cross islands.
* Clean up and fix tests
* Review feedback
* Add some more detailed comments.
* Change preview2 builder methods to use `&mut self`
This commit changes the `WasiCtxBuilder` for preview2 to use a builder
pattern more similar to `std::process::Command` where methods take `&mut
self` and return `&mut Self` instead of taking `self` and returning
`Self`. This pattern enables more easily building up configuration over
time throughout code where ownership transfer might otherwise be
awkward.
A small caveat to this is that the ergonomics of this pattern only
really work out well if the final "build" method takes `&mut self` as
well. In this situation it's difficult to try to figure out what's
supposed to happen if this method is called twice, so I left it to panic
for now so we can more easily update it in the future possibly.
* Synchronize preview1/preview2 builders
* Move preview1 builders to `&mut`-style
* Rename methods on preview2 builder to match names on the preview1 builders
* Fix C API
* Fix more tests
* Fix benchmark build
* Fix unused variable
Sample code provided in Readme of `wasmtime` crate specifies three
generic arguments for `get_typed_func` API, while it needs only two.
This fixes the sample code by removing the last generic argument.
* Update preview1 to trap on misaligned pointers
Previously Wasmtime would return `EINVAL` to a guest but the upstream
documentation indicates that misaligned pointers should trap, so this
commit updates the alignment error to get converted into a trap rather
than than being returned to the guest.
* Change OOB errors to traps too
* Update preview2's implementation of preview1
* Handle some more errors
Returns a summary of the resources required to instantiate this module or
component.
Potential uses of the returned information:
* Determining whether your pooling allocator configuration supports
instantiating this module.
* Deciding how many of which `Module` you want to instantiate within a fixed
amount of resources, e.g. determining whether to create 5 instances of module X
or 10 instances of module Y.
Part of #6627.
* Add config method for whether or not to capture coredumps on trap
* Add core dump support to the runtime
This adds the machinery to capture a core dump when a trap occurs
and attach it to the resulting anyhow::Error that gets bubbled up to
the caller. I've created a CoreDumpStack structure in the runtime, which is
currently just a backtrace until we design a way to recover the locals
stack values when a trap occurs. When that CoreDumpStack gets converted to
a wasmtime::WasmCoreDump, we add additional information from the Store such
as globals, memories, and instance information.
A lot of this is mechanistically similar to how backtraces
are captured and attached to errors. Given that they both are attached as
context to anyhow::Errors, setting coredump_on_trap to true will supercede
any setting for wasm_backtrace.
* Address some PR feedback
* Fix docs
* Switch WasmCoreDump to have vec of module names rather than Modules
* Add smoketests for WasmCoreDump
* clean up tests
* Update memories and globals field to be store_
* add debug impl for wasmcoredump
* Remove duplicate imports
* ignore miri for wasm tests
Alex Crichton
wasmtime-publish
Nick Fitzgerald
Gurinder Singh
Eduardo de Moura Rodrigues
Afonso Bordado
Pat Hickey
Chris Fallin
Andrew Brown
ssunkin-fastly
Trevor Elliott
Saúl Cabrera
Timon
Jia Tan
Daniel Vigovszky
Anoop Elias
Rainy Sinclair