* rename wasi-cli-base to wasi-cli, delete `preview` package, import wasi-cli
wasi-cli import is sum of
https://github.com/WebAssembly/wasi-cli/pull/19 and
https://github.com/WebAssembly/wasi-cli/pull/20
* wasi impl: change bindgen arguments and mod paths from cli_base to cli
* correct name of wasi-cli deps dir to just `deps/cli/`
it turns out this isnt semantically meaningful, since the package name
is in the document itself now, but lets be consistient
* track whether stdio isatty in ctx, and impl the cli/terminal-* interfaces
* rebase fixup
* wasi wits: define the reactor adapter's world
* component adapter: fixes
* test-programs/command-tests: fix renaming cli_base to cli
* component adapter: fix manually-defined export and import names
* test harness fixes
* preview1 component adapter: fill in isatty detection
* implement isatty in preview2-to-preview1 host adapter
* test-programs: cover both when stdio isatty and not
prtest:full
* split isatty test for regular file and stdio, detect host stdio is_terminal
CI environments vary - let the test runner make sure the host process's
stdio is in fact a terminal before asserting that the guest sees it is a
terminal.
* provide an is-terminal impl for all preview2's stdio types
which means making a newtype around Stdout and Stderr instead of using
a type alias there.
and then use the is-terminal impl to fill in the isatty field in the
builder when inheriting. if you need to override it you can always
builder.stdin(stdio::stdin(), your_own_idea_of_isatty)
* finally, rename IsATTY variants to Yes and No
* Fix the reference to IsATTY::No
* more forgotten renamings
---------
Co-authored-by: Trevor Elliott <telliott@fastly.com>
This commit prepares for the introduction of float support to Winch. Initially,
I intended to include this change as part of the original change supporting
floats, but that change is already sizable enough.
This modification simplifies the Assembler and MacroAssembler interfaces, as
well as the interaction and responsibilities between them, by:
* Eliminating the `Operand` abstraction, which didn't offer a substantial
benefit over simply using the MacroAssembler's `RegImm` and `Address`
abstractions as operands where necessary. This approach also reduces the number
of conversions required prior to emission.
* Shifting the instruction dispatch responsibility solely to the MacroAssembler,
rather than having this responsibility shared across both abstractions. This was
always the original intention behind the MacroAssembler. As a result, function
definitions at the Assembler layer become simpler.
This change also introduces richer type information for immediates, which
results in better instruction selection in some cases, and it's also needed to
support floats.
Not all file descriptors can get registered with epoll, for example
files on Linux or `/dev/null` on macOS return errors. In these
situations the fallback of the worker thread implementation is used
instead.
* Docs: Remove unneeded command in WASI-tutorial.md
The `cat /tmp/somewhere.txt` command in the example using --dir=$PWD is not needed. The error message should be after the first command anyway since it should be the one creating the message.
It looks like a copy/paste error since the previous example showed the same `cat` command a successful copy.
* Docs: Update error messages in WASI-tutorial.md.
The error messages appear to have changed since this document was originally written.
* Add support for vector in DataValueExt::int()
Fixes#6827
* Replace `if` with `match`
* Enable interpreter test for simd ineg
Issue #6827
* Format code with `cargo fmt`
and also improve comment
This commit adds a new `Engine::detect_precompiled` API to inspect some
bytes and determine if they look like a precompiled artifact of either a
core wasm module or component. This is something I'll be using soon in
an upcoming refactor of the Wasmtime CLI to support components, but it's
something we've also talked about before which can be useful for systems
storing both precompiled modules and components.
Implementation-wise this looks at the ELF header of the input and
determines if it's got all the right flags that Wasmtime sets for the
various bits and bobs of our object format.
* Use `Offset32` as `i32` in ISLE
This commit updates the x64 and aarch64 backends to use the `i32`
primitive type in ISLE when working with an `Offset32` instead of a
`u32`. This matches the intended representation of `Offset32` as a type
which is signed internally and represents how offsets on instructions
are often negative too.
This does not actually change any end results of compilation and instead
is intended to be "just" an internal refactoring with fewer casts and
more consistent handling of offsets.
* aarch64: Define the `PairAMode` type in ISLE
This commit moves the definition of the `PairAMode` enum into ISLE
instead of its current Rust-defined location. This is in preparation for
the next commit where all AMode calculations will be moved into ISLE.
* aarch64: Fix a copy/paste typo loading vectors
This commit fixes an assertion that can be tripped in the aarch64
backend where a 64-bit load was accidentally flagged as a 128-bit load.
This was found in future work which ended up tripping the assertion a
bit earlier.
* aarch64: Move AMode computation into ISLE
This commit moves the computation of the `AMode` enum for addressing
from Rust into ISLE. This enables deleting a good deal of Rust code in
favor of (hopefully) more readable ISLE code.
This does not mirror the preexisting logic exactly but instead takes a
different approach for generating the `AMode`. Previously the entire
chain of `iadd`s input into an address were unfolded into 32-bit and
64-bit operations and then those were re-combined as possible into an
`AMode` (possibly emitting `add` instructions. Instead now pattern
matching is used to represent this. The net result is that amodes are
emitted slightly differently here and there in a number of updated test
cases.
I've tried to verify in all test cases that the number of instructions
has not increased and the same logical operation is happening. The exact
`AMode` may differ but at least instruction-wise this shouldn't be a
regression. My hope is that if anything needs changing that can be
represented with updates to the rule precedence in ISLE or various other
special cases.
One part I found a little surprising was that the immediate with a
load/store instruction is not actually used much of the time. I naively
thought that the mid-end optimizations would move iadd immediates into
the load/store immediate but that is not the case. This necessitated two
extra ISLE rules to manually peel off immediates and fold them into the
load/store immediate.
* aarch64: Remove `NarrowValueMode`
This is no longer needed after the prior commit
* aarch64: Fix `return_call`'s interaction with pointer authentication
This commit fixes an issue where a `return_call` would not decrypt the
return address when pointer authentication is enabled. The return
address would be encrypted upon entry into the function but would never
get restored later on.
This addresses the issue by changing the encryption keys in use from the
A/B key plus SP to instead using A/B plus the zero key. The reason for
this is that during a normal function call before returning the SP value
is guaranteed to be the same as it was upon entry. For tail calls though
SP may be different due to differing numbers of stack arguments. This
means that the modifier of SP can't be used for the tail convention.
New `APIKey` definitions were added and that now additionally represents
the A/B key plus the modifier. Non-`tail` calling conventions still use
the same keys as before, it's just the `tail` convention that's different.
Closes#6799
* Fix tests
* Fix test expectations
* Allow `sign_return_address` at all times in filetests
This commit removes the option to generate padding between basic blocks
when fuzzing Wasmtime. Over the weekend lots of OOMs were discovered
related to this option and its most recent update in #6736. The new OOMs
appear to be related to:
* If multiple modules are generated then the configured limits in #6736
aren't relevant because they only cap one module.
* Each imported function generates a new trampoline which has its own
set of padding which wasn't previously accounted for.
* Spec tests have a lot of functions and the limits didn't account for
this.
While each of these is probably individually fixable I think it's
probably not worth the whack-a-mole any more. The `cranelift-fuzzgen`
target should cover the relevant cases for padding without the need for
Wasmtime's fuzzing to cover it as well.
* Configure Mach ports vs signals via `Config`
This commit adds a `Config::macos_use_mach_ports` configuration option
to replace the old `posix-signals-on-macos` compile-time Cargo feature.
This'll make Wasmtime a tad larger on macOS but likely negligibly so.
Otherwise this is intended to provide a resolution to #6785 where
embedders will be able to use any build of Wasmtime and configure at
runtime how trap handling should happen.
Functionally this commit additionally registers a `pthread_atfork`
handler to cause any usage of Wasmtime in the child to panic. This
should help head off a known-invalid state in case anyone runs into it
in the future.
* Fix build on non-macOS
* filesystem.wit: upstream main + pch/blocking_is_a_stream_concern + pch/metadata_hash_fixes
upstream main changes the interface to be named `types`, and has
changes to descriptor-stat, directory-entry, adds the metadata hash
methods.
pch/blocking_is_a_stream_concern upstreams the deletion of non-blocking
from descriptor flags.
pch/metadata_hash_fixes adds missing this: descriptor arguments to
the metadata-hash and metadata-hash-at methods.
* copy in preopens.wit and world.wit from wasi-filesystem repo, and delete wasi-cli-base/preopens.wit
* other wits: fix names of filesystem/{types, preopens}
* fix bindgen invocations and import paths for filesystem changes
* component adapter: fix bindings paths for preopens and filesystem
* wip
* wippp
* fill in a reasonable-ish metadata hash impl
* preview 1 adapter: we need async in the dir entries, so collect the whole thing to a vec
this isnt great but i dont have the time or energy to do this better
right now
* component adapter: use metadata hash to fake ino
* wasi-tests: fix warnings
* reactor-tests: fix filesystem paths
* only create a BuildHasher impl once
* consistiency
* component adapter: use descriptor to call filesystem funcs, not the fd
* reactor test: fix filesystem types paths
* metadata hash: use DefaultHasher
* fix missed trappable-error-type merge conflicts
* preview1-in-preview2: dont use to_le in Filestat assignment of ino
wiggle takes care of endianness translation. The to_le in fd_readdir
entries is because that implementation is writing structs directly to
memory (very cursed)
s390x CI caught this bug
* debugging: forward stdio for fd_readdir test so we can figure out CI failure on s390x
prtest:full
* Don't convert filestat ino to little-endian, as it's not written to memory
fd_readdir will write its results to a buffer, and needs to be careful
with endianness as a result. However, fd_filestat_get returns the value
directly, and as such does not need to call `to_le`.
---------
Co-authored-by: Trevor Elliott <telliott@fastly.com>
* Remove the implementation of wasi-crypto
This commit is a follow-up to the discussion on #6732. This removes
Wasmtime's implementation of the wasi-crypto proposal from in-tree along
with its various support in CI, configuration, etc. See the discussion
on #6732 for the full information but at a high level the main reasons
for removing the implementation at this time are:
* There is not currently an active maintainer of the Wasmtime
integration here for wasi-crypto.
* There are known issues with the code quality of the implementation
such as transmutes of guest-owned memory to `&'static mut [u8]` and
known unsafety in dependencies.
* The size and breadth of the dependency tree brings maintenance burden
and overhead to managing Wasmtime's dependency tree.
As mentioned on the issue this commit does not mean that Wasmtime
doesn't want to implement the wasi-crypto proposal. Instead the "tier 3"
status of wasi-crypto needs to be re-attained to be included back
in-tree, which would mean resolving the above issues.
Note that this commit is intentionally just after the 13.0.0 branch
point which means that this is slated for Wasmtime 14 to be released on
September 20.
* Remove some cfgs
* Remove wasi-crypto CI
* WIP: two-tier processing for MachBuffer veneer processing in island emission
* Unconditionally emit veneers for label forward-refs that cross islands.
* Clean up and fix tests
* Review feedback
* Add some more detailed comments.
* Change preview2 builder methods to use `&mut self`
This commit changes the `WasiCtxBuilder` for preview2 to use a builder
pattern more similar to `std::process::Command` where methods take `&mut
self` and return `&mut Self` instead of taking `self` and returning
`Self`. This pattern enables more easily building up configuration over
time throughout code where ownership transfer might otherwise be
awkward.
A small caveat to this is that the ergonomics of this pattern only
really work out well if the final "build" method takes `&mut self` as
well. In this situation it's difficult to try to figure out what's
supposed to happen if this method is called twice, so I left it to panic
for now so we can more easily update it in the future possibly.
* Synchronize preview1/preview2 builders
* Move preview1 builders to `&mut`-style
* Rename methods on preview2 builder to match names on the preview1 builders
* Fix C API
* Fix more tests
* Fix benchmark build
* Fix unused variable
Sample code provided in Readme of `wasmtime` crate specifies three
generic arguments for `get_typed_func` API, while it needs only two.
This fixes the sample code by removing the last generic argument.
* Update preview1 to trap on misaligned pointers
Previously Wasmtime would return `EINVAL` to a guest but the upstream
documentation indicates that misaligned pointers should trap, so this
commit updates the alignment error to get converted into a trap rather
than than being returned to the guest.
* Change OOB errors to traps too
* Update preview2's implementation of preview1
* Handle some more errors
Returns a summary of the resources required to instantiate this module or
component.
Potential uses of the returned information:
* Determining whether your pooling allocator configuration supports
instantiating this module.
* Deciding how many of which `Module` you want to instantiate within a fixed
amount of resources, e.g. determining whether to create 5 instances of module X
or 10 instances of module Y.
Part of #6627.
* Add config method for whether or not to capture coredumps on trap
* Add core dump support to the runtime
This adds the machinery to capture a core dump when a trap occurs
and attach it to the resulting anyhow::Error that gets bubbled up to
the caller. I've created a CoreDumpStack structure in the runtime, which is
currently just a backtrace until we design a way to recover the locals
stack values when a trap occurs. When that CoreDumpStack gets converted to
a wasmtime::WasmCoreDump, we add additional information from the Store such
as globals, memories, and instance information.
A lot of this is mechanistically similar to how backtraces
are captured and attached to errors. Given that they both are attached as
context to anyhow::Errors, setting coredump_on_trap to true will supercede
any setting for wasm_backtrace.
* Address some PR feedback
* Fix docs
* Switch WasmCoreDump to have vec of module names rather than Modules
* Add smoketests for WasmCoreDump
* clean up tests
* Update memories and globals field to be store_
* add debug impl for wasmcoredump
* Remove duplicate imports
* ignore miri for wasm tests
* change all TableError cases to trap, instead of return a runtime error
this is going to be required as part of the shift to resources! so lets
do it right now and see how many things break
* delete dead code
* Scaffold out changes to table to track parent and children
* fill in more parent/child todos
* finish plumbing, HostPollable::TableEntry uses Table::push_child
* command-tests: show that wasm that manages stream pollable lifetime wrong traps
* Update crates/test-programs/build.rs
Co-authored-by: Trevor Elliott <telliott@fastly.com>
* doc comments
* doc fixes. OccupiedEntry made pub.
---------
Co-authored-by: Trevor Elliott <telliott@fastly.com>
* Add a helper for writing stdin tests
* Test a use of preview2::stdio::Stdin on unix
* Abstract the stdin test, and apply it to tokio::stdin and worker_thread_stdin
* re-implement worker thread stdin with a watch
* poll_oneoff_stdio should work on windows now
* comments
* test: show that restarting
* reap tasks from AsyncReadStream and AsyncWriteStream when dropped
the tasks will exit on their own if the receiver drops, but this should
terminate them while they are awaiting on read/write.
* rewrite most of the asyncfd based stdin
this has still got at least one serious bug, noted with FIXME.
also has a bunch of changes which can be backed out because they were
chasing red herrings.
the major bug was that, for some inexplicable reason, i didn't actually
set the fd to nonblocking just prior to asyncfd creation. switching from
the fdflags.difference to fdflags.union operator fixed that bug.
* Refactor to avoid `as_mut` and `.0 .0`
* Rework the stdin implementation to hold the stream in a mutex
* Detect when the task backing stdin has exited
* Have the thread managing stdin exit when it encounters EOF
* Refactor the test to restart the runtime as well
* Format
prtest:full
* internal tokio runtime used for sync interfaces is now new_current_thread
* Remove println from test for stdout
* Remove unneeded `vec!`
---------
Co-authored-by: Pat Hickey <phickey@fastly.com>
Ensure that all imports have unique module/name combos to ensure that
the module can be instantiable in JS where separate functions need to be
provided for each import.
* Wasmtime: Add support for Wasm tail calls
This adds the `Config::wasm_tail_call` method and `--wasm-features tail-call`
CLI flag to enable the Wasm tail calls proposal in Wasmtime.
This PR is mostly just plumbing and enabling tests, since all the prerequisite
work (Wasmtime trampoline overhauls and Cranelift tail calls) was completed in
earlier pull requests.
When Wasm tail calls are enabled, Wasm code uses the `tail` calling
convention. The `tail` calling convention is known to cause a 1-7% slow down for
regular code that isn't using tail calls, which is why it isn't used
unconditionally. This involved shepherding `Tunables` through to Wasm signature
construction methods. The eventual plan is for the `tail` calling convention to
be used unconditionally, but not until the performance regression is
addressed. This work is tracked in
https://github.com/bytecodealliance/wasmtime/issues/6759
Additionally while our x86-64, aarch64, and riscv64 backends support tail calls,
the s390x backend does not support them yet. Attempts to use tail calls on s390x
will return errors. Support for s390x is tracked in
https://github.com/bytecodealliance/wasmtime/issues/6530
* Store `Tunables` inside the `Compiler`
Instead of passing as an argument to every `Compiler` method.
* Cranelift: Support "direct" return calls on riscv64
They still use `jalr` instead of `jal` but this allows us to use the `RiscvCall`
reloc, which Wasmtime handles. Before we were using `LoadExternalName` which
produces an `Abs8` reloc, which Wasmtime intentionally does not handle since
that involves patching code at runtime, which makes loading code slower.
* Fix tests that assume tail call support on s390x
It was generating bit-wise and instructions instead of addition instructions.
While we are at it, use field literal short hand for a struct literal in riscv64
instruction emission.
* Remove unused WIT files from Wasmtime
These files aren't actually read by anything currently. They were added
historically and a previous refactoring in #6390 forgot to remove them.
No tests or build process reads them so this deletes them to get them
out of the way.
* Update dependencies on wasm-tools crates.
This commit updates the deps on the wasm-tools family of crates to bring
in a few fixes for WIT/component-related things. Primarily though this
brings in an update to the component model where empty types are now
disallowed.
Some tests using empty types were adjusted to use non-empty types, but
many tests were also simply removed as they existed to test what would
happen with empty types which now no longer needs to be tested.
* Update `stream-error` in preview2
Add a `dummy` field to make it a non-empty structure. It's expected that
this will change to something else more "official" in the future, but
for now this is here to keep everything compiling.
* Update component fuzzing to avoid empty types
Empty types are no longer valid
* Update crates/wasi/wit/deps/io/streams.wit
Co-authored-by: Peter Huene <peter@huene.dev>
---------
Co-authored-by: Peter Huene <peter@huene.dev>
* Remove shared_defs args from all per-isa definitions
They don't and shouldn't define any isa-independent things
* Inline all define_settings functions
* Merge verify_instruction_formats into shared::define()
It isn't modified in between anyway now.
* Enable document private items for cranelift-codegen-meta
The mentioned cargo bug has been fixed
* Remove unused testing_hooks cargo feature
It has been unused ever since we removed the old backend framework.
* Implement component-to-component calls with resources
This fills out support in FACT in Wasmtime to support
component-to-component calls that use resources. This ended up being
relatively simple as it's "just" a matter of moving resources between
tables which at this time bottoms out in calls to the host. These new
trampolines are are relatively easy to add after #6751 which helps keep
this change contained.
Closes#6696
* Review comments
* Remove DisplayFunctionAnnotations
It used to exist for printing the debuginfo value ranges with the clif
ir, but this no longer happens, so it is now useless.
* Remove debug info collection from DummyEnvironment
There are no remaining users of it
* Remove ComparableSourceLoc
It is unused
* Move LabelValueLoc re-export out of the ir module
It encodes target specific information, so shouldn't be in the target
independent ir module.
* Remove RelocDistance dependency from ir::extfunc and ir::globalvalue
* preview2::pipe sink_write_stream test bugfix
I didn't realize this test was racy, but it makes sense because the
worker is consuming these writes in another thread. usually the
foreground thread wins, but we observed the other case once in CI
* fix wasi-preview2-components-sync poll_oneoff_stdio flaky test
I made a typo translating the sync calls to async calls, and in the sync
implementation stream::blocking_write was calling async stream::write,
so it was missing the logic to make the write blocking.
This would in turn cause panics in rust's std::io::stdio::print_to
when writes return less than the length of the input.
thank you to @silentbicycle for autoclave, which we used to reproduce
this bug.
In poking at #6696 I felt that the amount of boilerplate for defining
new kinds of trampolines in the component model was getting a bit
excessive. There was already 6 different types and I was adding two more
and I had to touch just a few too many places to get this done. In the
end I decided to refactor how trampolines are handled in the component
model to make it much easier to add new kinds of trampolines.
To that end the type-specific counts/lists/etc are all gone now in favor
of a single concept of a trampoline. This means components now track
trampolines in-bulk rather than individually by type. For example
compiling trampolines is now a loop over "compile this list of
trampolines" where previously it was N loops for the N types of
trampolines. The `Trampoline` definition is where the enum and dispatch
happens where that contains all possible trampolines that a component
could require.
This ended up being a large refactor to the Cranelift component
integration, but there is not intended to be any functional change from
this refactoring. Additionally all trampolines are now removed from the
global initializers list since there's nothing preventing them from
being initialized earlier on during the instantiation process.
Overall this should drastically reduce the number of locations that need
to have trampoline-specific knowledge to translation, the dataflow
graph, and compilation. Nearly everything else can operate over
everything in bulk and forward between these systems.
ssunkin-fastly
Chris Fallin
Pat Hickey
Trevor Elliott
Saúl Cabrera
Alex Crichton
Timon
Jia Tan
Gurinder Singh
Afonso Bordado
Andrew Brown
Daniel Vigovszky
wasmtime-publish
Anoop Elias
Nick Fitzgerald
Rainy Sinclair
Tom Stuart
bjorn3
guregu