* Refactor the test-programs test suite
This commit is a large refactoring that reorganizes `test-programs` and
how we tests wasms in Wasmtime. Often writing tests requires complicated
interactions with the guest which can't be done via hand-written `*.wat`
syntax and requires a compiler to get engaged. For this purpose Wasmtime
currently has the `crates/test-programs/*` test suite which builds files
from source and then runs the tests. This has been somewhat cumbersome
in the past though and it's not been easy to extend this over time, so
this commit attempts to address this.
The scheme implemented in this PR looks like:
* All wasm test programs live in `crates/test-programs/src/bin/*.rs`.
All of them, no exceptions.
* Wasm tests have shared support located at
`crates/test-programs/src/lib.rs` and its submodules, such as bindings
generation for WASI.
* Wasm tests are built by a new `crates/test-programs/artifacts` crate.
This crate compiles modules and additionally creates components for
all test programs. The crate itself only records the path to these
outputs and a small amount of testing support, but otherwise doesn't
interact with `wasmtime`-the-crate itself.
* All tests in `crates/test-programs/tests/*.rs` have moved. For example
wasi-http tests now live at `crates/wasi-http/tests/*.rs`. Legacy
tests of wasi-common now live at `crates/wasi-common/tests/*.rs`.
Modern tests for preview2 live at `crates/wasi/tests/*.rs`.
* Wasm tests are bucketed based on their filename prefix. For example
`preview1_*` is tested in wasi-common and wasmtime-wasi. The
`preview2_*` prefix is only tested with wasmtime-wasi, however.
* A new `cli_*` prefix is used to execute tests as part of
`tests/all/main.rs`. This is a new submodule in
`tests/all/cli_tests.rs` which executes these components on the
command line. Many old "command" tests were migrated here.
* Helper macros are generated to assert that a test suite is run in its
entirety. This way if a `preview1_*` test is added it's asserted to
get added to both wasi-common and wasmtime-wasi in the various modes
they run tests.
Overall this moved a number of tests around and refactored some edges of
the tests, but this should not lose any tests (except one that wasn't
actually testing anything). Additionally the hope is that it's much
easier to add tests in the future. The process is to add a new file in
`crates/test-programs/src/bin/*.rs` named appropriately. For example a
preview2 executable is `preview2_*` and a CLI tests is `cli_*`. When
building the test suite an error is generated in the appropriate module
then of "please write a test here", and then a test is written in the
same manner as the other tests in the module.
* Remove no-longer-needed fetches
prtest:full
* I'm worried wasi is running low on semicolons
* Add the WASI target in all CI actions
* Add unknown-unknown target on all CI builders too
* Fix building test artifacts under miri
Need to avoid wrappers for these cross-compiled targets
* Break circular dependency for packaging
Don't use the workspace dep for `wasmtime-wasi` since it injects a
version, instead use a `path = '..'` dependency to fool Cargo into
dropping the dependency during the package phase.
* Fix some merge conflicts with tests
* Fix rebase for new tests
* Remove stray comment
* Fix some flaky tests
* Fix network tests in synchronous mode
This commit is an attempt to fix some networking tests in synchronous
mode in our test suite. Currently networking tests don't actually run in
synchronous mode on CI which is why no failures have been surfaced yet,
but the refactoring in #7182 is going to start doing this.
Currently the `udp_sample_application.rs` test blocks infinitely in
synchronous mode for me locally, most of the time. This appears to be an
interaction between how Tokio handles readiness and how we're
entering the event loop. We're effectively entering the Tokio event loop
with a future that's always ready which ends up starving Tokio of
otherwise performing its background work such as updating flags for
readiness of reading/writing.
The fix here is to add a yield at the start of an `in_tokio` block which
is used in synchronous mode. This is a kludge fix but the intention is
to enable Tokio to have a chance to update readiness flags and process
events from epoll/kqueue/etc.
An additional fix to this issue is WebAssembly/wasi-sockets#64 where the
test is waiting on `READABLE` or `WRITABLE`, but in this specific case
it should only wait on `READABLE`. If it waited on just this then that
would also fix this issue. Nevertheless having a `yield_now` is expected
to have little-to-no overhead and otherwise fix this edge case of an
always-ready future.
* Fix passing empty arguments on the CLI
* Add another blocking accept
* Update crates/test-programs/src/bin/api_proxy.rs
Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>
---------
Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>
* Refactor HTTP tests to be less flaky and more robust
This commit refactors the wasi-http tests we have in this repository
with a few changes:
* Each test now uses a dedicated port for the test. The port 0 is bound
and whatever the OS gives is used for the duration of the test. This
prevents tests from possibly interfering with each other.
* Server spawning is abstracted behind a single `Server` type now which
internally has http1/2 constructors.
* Timeouts for server shutdown are removed since they were failing in CI
and are likely best handled for each test individually if necessary.
As a minor cleanup the `tokio` usage in guest programs was removed as it
was boilerplate in this case. This shouldn't affect the runtimes of
tests, however.
* Remove unused import
* Don't panic from worker thread
* Improve failure error message
prtest:full
* Fix some windows-specific issues
* Fix async tests
* Relax test assertion string
* Implement the `wasi:sockets/ip-name-lookup` interface
This commit is an initial implementation of the new `ip-name-lookup`
interface from the `wasi-sockets` proposal. The intention is to get a
sketch of what an implementation would look like for the preview2
release later this year. Notable features of this implementation are:
* Name lookups are disabled by default and must be explicitly enabled in
`WasiCtx`. This seems like a reasonable default for now while the full
set of configuration around this is settled.
* I've added new "typed" methods to `preview2::Table` to avoid the need
for extra helpers when using resources.
* A new `-Sallow-ip-name-lookup` option is added to control this on the
CLI.
* Implementation-wise this uses the blocking resolution in the Rust
standard library, built on `getaddrinfo`. This doesn't invoke
`getaddrinfo` "raw", however, so information such as error details can
be lost in translation. This will probably need to change in the
future.
Closes#7070
* Validate the input domain name.
* Use a crate-level helper for `spawn_blocking`
---------
Co-authored-by: Dave Bakker <github@davebakker.io>
* c-api: Add a feature for async
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* c-api: Add support for async config
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* c-api: Add support for calling async functions
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* c-api: Add ability to yield execution of Wasm in a store
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* c-api: Introduce wasmtime_linker_instantiate_async
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* c-api: Support defining async host functions
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* gitignore: ignore cmake cache for examples
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* examples: Add example of async API in C
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* c-api: Consolidate async functionality into a single place
Put all the async stuff in it's own header and own rust source file
Also remove the wasmtime_async_continuation_new function, users can just
allocate it directly.
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* c-api: Make async function safe
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* c-api: Remove wasmtime_call_future_get_results
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* c-api: Simplify CHostCallFuture
Move the result translation and hostcall_val_storage usage into an async
function
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* c-api: Simplify C continuation implementation
Remove the caller, which means that we don't need another struct for the
future implementation.
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* c-api: Improve async.h documentation
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* c-api: Cleanup from previous changes
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* examples: Fix example
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* c-api: Simplify continuation callback
This gives more duality with calling an async function and also means
that the implementation can pretty much mirror the sync version.
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* c-api: Fix async.h documentation
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* c-api: Fix documentation for async.h
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* c-api: Review feedback
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* examples: Downgrade async.cpp example to C++11
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* c-api: initialize continuation with a panic callback
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* prtest:full
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
---------
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* Rename `Host*` things to avoid name conflicts with bindings.
* Update to the latest resource-enabled wit files.
* Adapting the code to the new bindings.
* Update wasi-http to the resource-enabled wit deps.
* Start adapting the wasi-http code to the new bindings.
* Make `get_directories` always return new owned handles.
* Simplify the `poll_one` implementation.
* Update the wasi-preview1-component-adapter.
FIXME: temporarily disable wasi-http tests.
Add logging to the cli world, since stderr is now a reseource that
can only be claimed once.
* Work around a bug hit by poll-list, fix a bug in poll-one.
* Comment out `test_fd_readwrite_invalid_fd`, which panics now.
* Fix a few FIXMEs.
* Use `.as_ref().trapping_unwrap()` instead of `TrappingUnwrapRef`.
* Use `drop_in_place`.
* Remove `State::with_mut`.
* Remove the `RefCell` around the `State`.
* Update to wit-bindgen 0.12.
* Update wasi-http to use resources for poll and I/O.
This required making incoming-body and outgoing-body resourrces too, to
work with `push_input_stream_child` and `push_output_stream_child`.
* Re-enable disabled tests, remove logging from the worlds.
* Remove the `poll_list` workarounds that are no longer needed.
* Remove logging from the adapter.
That said, there is no replacement yet, so add a FIXME comment.
* Reenable a test that now passes.
* Remove `.descriptors_mut` and use `with_descriptors_mut` instead.
Replace `.descriptors()` and `.descriptors_mut()` with functions
that take closures, which limits their scope, to prevent them from
invalid aliasing.
* Implement dynamic borrow checking for descriptors.
* Add a cargo-vet audit for wasmtime-wmemcheck.
* Update cargo vet for wit-bindgen 0.12.
* Cut down on duplicate sync/async resource types (#1)
* Allow calling `get-directories` more than once (#2)
For now `Clone` the directories into new descriptor slots as needed.
* Start to lift restriction of stdio only once (#3)
* Start to lift restriction of stdio only once
This commit adds new `{Stdin,Stdout}Stream` traits which take over the
job of the stdio streams in `WasiCtxBuilder` and `WasiCtx`. These traits
bake in the ability to create a stream at any time to satisfy the API
of `wasi:cli`. The TTY functionality is folded into them as while I was
at it.
The implementation for stdin is relatively trivial since the stdin
implementation already handles multiple streams reading it. Built-in
impls of the `StdinStream` trait are also provided for helper types in
`preview2::pipe` which resulted in the implementation of
`MemoryInputPipe` being updated to support `Clone` where all clones read
the same original data.
* Get tests building
* Un-ignore now-passing test
* Remove unneeded argument from `WasiCtxBuilder::build`
* Fix tests
* Remove some workarounds
Stdio functions can now be called multiple times.
* If `poll_oneoff` fails part-way through, clean up properly.
Fix the `Drop` implementation for pollables to only drop the pollables
that have been successfully added to the list.
This fixes the poll_oneoff_files failure and removes a FIXME.
---------
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
* Remove usage of `is-terminal` and `atty` crates
This functionality is now folded into the standard library itself.
* Fix syntax
* Fix a unix/windows cfg
* Move the incoming_handler impl into http_impl
* Remove the incoming handler -- we need to use it as a guest export
* Start adding a test-programs test for the server side of wasi-http
* Progress towards running a server test
* Implement incoming-request-method
* Validate outparam value
* Initial incoming handler test
* Implement more of the incoming api
* Finish the incoming api implementations
* Initial cut at `wasmtime serve`
* fix warning
* wasmtime-cli: invoke ServeCommand, and add enough stuff to the linker to run trivial test
* fix warnings
* fix warnings
* argument parsing: allow --addr to specify sockaddr
* rustfmt
* sync wit definitions between wasmtime-wasi and wasmtime-wasi-http
* cargo vet: add an import config and wildcard audit for wasmtime-wmemcheck
* cargo vet: audit signal-hook-registry
* Remove duplicate add_to_linker calls for preview2 interfaces
prtest:full
* Add a method to finish outgoing responses
Co-authored-by: Adam Foltzer <acfoltzer@fastly.com>
Co-authored-by: Pat Hickey <phickey@fastly.com>
* Mark the result of the incoming_{request,response}_consume methods as own
* Explicit versions for http-body and http-body-util
* Explicit `serve` feature for the `wasmtime serve` command
* Move the spawn outside of the future returned by `ProxyHandler::call`
* Review feedback
---------
Co-authored-by: Trevor Elliott <telliott@fastly.com>
Co-authored-by: Adam Foltzer <acfoltzer@fastly.com>
* Add memory protection keys (MPK)
In order to use MPK on an x86_64 Linux system, we need access to the
underlying `pkey_*` system calls (`sys`), control of the x86 PKRU
register (`pkru`), and a way of determining if MPK is even supported
(`is_supported`). These various parts are wrapped in a `ProtectionKey`
abstraction along with a `ProtectionMask` that can be used `allow` the
CPU to access protected regions.
* Integrate MPK into the pooling allocator
This change adds "stripes" to the pooling allocator's `MemoryPool`. Now,
when requesting a slot in which to instantiate, the user (i.e.,
`InstanceAllocationRequest`) will be transparently assigned to one of
the stripes, each of which is associated with a protection key. The user
can also request a specific protection key to use, which will override
the original "find me a slot logic".
This has implications for how instances get allocated: once a store is
assigned a protection key, it will only allocate requests with that key,
limiting how many slots it has access to. E.g., if 15 keys are active,
the store can only ever access 1/15th of the slots.
This change also includes a tri-bool configuration field,
`memory_protection_keys`, which is disabled by default for the time
being.
* Address review comments
This is a rollup of 43 commits addressing review comments of various
kinds: bug fixes, refactorings, documentation improvements, etc. It also
ensures that CI runs all checks. A big thanks to @fitzgen and
@alexcrichton for the review!
prtest:full
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
---------
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
* Bump wasm-tools crates
Two major changes/reasons for this update:
* Primarily pulling in support for semicolons-in-WIT files. Semicolons are
not currently required, though, so I'll follow-up later with actual
semicolons.
* The syntax for parsing `(if ...)` was fixed in `wast`. Previously it
did not require `(then ...)` but this is required by the spec. New
spec tests require this as well. This breaks existing text format
tests which don't use `(then ...)` inside of an `(if ...)`. Most tests
were updated by hand but `embenchen_*` tests were updated by running
through the old parser to produce non-s-expression using code.
* Fix an example `*.wat`
* Update wasm-tools family of crates
Mostly minor updates, but staying up-to-date.
* Update text format syntax
* Update cargo vet entries
* Update more old text syntax
* Stream backpressure v2
Co-authored-by: Pat Hickey <phickey@fastly.com>
Co-authored-by: Trevor Elliott <telliott@fastly.com>
Co-authored-by: Dan Gohman <dev@sunfishcode.online>
Stop testing pseudocode
Restructure when notifications are sent, and make sure to flush the writer
Fix the wasi-http module versions of flush and blocking_flush
Use blocking_write_and_flush for blocking writes in the adapters
Fix a warning in wasi-http
Remove an unused DropPollable
add comment explaining try_write for tcpstream
refactor: separate struct for representing TcpReadStream
by factoring into HostTcpSocket a little bit
tcp read stream: handle stream closing
tcp tests: use blocking_read where its expecting to wait for input
move common test body into wasi-sockets-tests/src/lib.rs
ensure parent socket outlives pollable
input and output streams can be children now
tcp's streams are the sockets children
tcp.wit: document child relationships
tcp tests: fix to drop socket after its child streams
review feedback: propogate worker task panic
style
error source fix
tcp: use preview2::spawn, and propogate worker panics
join handle await always propogates panic
background task handles ewouldblock as well
document choice of constant
* sync wit notes into wasi-http
* improve wit docs for output-stream
* doc: document `HostOutputStream` (#6980)
* doc: document `HostOutputStream`
Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net>
Co-authored-by: Pat Hickey <pat@moreproductive.org>
* fix(wasi): fail when `MemoryOutputStream` buffer is full
Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net>
---------
Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net>
Co-authored-by: Pat Hickey <pat@moreproductive.org>
* rustfmt
prtest:full
* windows and doc fixes
* cli test wasi-http: use blocking-write-and-flush
* Disable some tests, and adjust timeouts when running under qemu
* Try to reproduce the riscv64 failures
* Update riscv to LLVM 17 with beta rust
* Revert "Try to reproduce the riscv64 failures"
This reverts commit 8ac678171f.
* Pin the beta version for riscv64
* Fix a warning on nightly
---------
Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net>
Co-authored-by: Roman Volosatovs <rvolosatovs@users.noreply.github.com>
Co-authored-by: Trevor Elliott <telliott@fastly.com>
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
* Redesign Wasmtime's CLI
This commit follows through on discussion from #6741 to redesign the
flags that the `wasmtime` binary accepts on the CLI. Almost all flags
have been renamed/moved and will require callers to update. The main
motivation here is to cut down on the forest of options in `wasmtime -h`
which are difficult to mentally group together and understand.
The main change implemented here is to move options behind "option
groups" which are intended to be abbreviated with a single letter:
* `-O foo` - an optimization or performance-tuning related option
* `-C foo` - a codegen option affecting the compilation process.
* `-D foo` - a debug-related option
* `-W foo` - a wasm-related option, for example changing wasm semantics
* `-S foo` - a WASI-related option, configuring various proposals for example
Each option group can be explored by passing `help`, for example `-O
help`. This will print all options within the group along with their
help message. Additionally `-O help-long` can be passed to print the
full comment for each option if desired.
Option groups can be specified multiple times on the command line, for
example `-Wrelaxed-simd -Wthreads`. They can also be combined together
with commas as `-Wrelaxed-simd,threads`. Configuration works as a "last
option wins" so `-Ccache,cache=n` would end up with a compilation
cache disabled.
Boolean options can be specified as `-C foo` to enable `foo`, or they
can be specified with `-Cfoo=$val` with any of `y`, `n`, `yes`, `no`,
`true`, or `false`. All other options require a `=foo` value to be
passed and the parsing depends on the type.
This commit additionally applies a few small refactorings to the CLI as
well. For example the help text no longer prints information about wasm
features after printing the option help. This is still available via
`-Whelp` as all wasm features have moved from `--wasm-features` to `-W`.
Additionally flags are no longer conditionally compiled in, but instead
all flags are always supported. A runtime error is returned if support
for a flag is not compiled in. Additionally the "experimental" name of
WASI proposals has been dropped in favor of just the name of the
proposal, for example `--wasi nn` instead of `--wasi-modules
experimental-wasi-nn`. This is intended to mirror how wasm proposals
don't have "experimental" in the name and an opt-in is required
regardless.
A full listing of flags and how they have changed is:
| old cli flag | new cli flag |
|-----------------------------------------------|-------------------------------------------------|
| `-O, --optimize` | removed |
| `--opt-level <LEVEL>` | `-O opt-level=N` |
| `--dynamic-memory-guard-size <SIZE>` | `-O dynamic-memory-guard-size=...` |
| `--static-memory-forced` | `-O static-memory-forced` |
| `--static-memory-guard-size <SIZE>` | `-O static-memory-guard-size=N` |
| `--static-memory-maximum-size <MAXIMUM>` | `-O static-memory-maximum-size=N` |
| `--dynamic-memory-reserved-for-growth <SIZE>` | `-O dynamic-memory-reserved-for-growth=...` |
| `--pooling-allocator` | `-O pooling-allocator` |
| `--disable-memory-init-cow` | `-O memory-init-cow=no` |
| `--compiler <COMPILER>` | `-C compiler=..` |
| `--enable-cranelift-debug-verifier` | `-C cranelift-debug-verifier` |
| `--cranelift-enable <SETTING>` | `-C cranelift-NAME` |
| `--cranelift-set <NAME=VALUE>` | `-C cranelift-NAME=VALUE` |
| `--config <CONFIG_PATH>` | `-C cache-config=..` |
| `--disable-cache` | `-C cache=no` |
| `--disable-parallel-compilation` | `-C parallel-compilation=no` |
| `-g` | `-D debug-info` |
| `--disable-address-map` | `-D address-map=no` |
| `--disable-logging` | `-D logging=no` |
| `--log-to-files` | `-D log-to-files` |
| `--coredump-on-trap <PATH>` | `-D coredump=..` |
| `--wasm-features all` | `-W all-proposals` |
| `--wasm-features -all` | `-W all-proposals=n` |
| `--wasm-features bulk-memory` | `-W bulk-memory` |
| `--wasm-features multi-memory` | `-W multi-memory` |
| `--wasm-features multi-value` | `-W multi-value` |
| `--wasm-features reference-types` | `-W reference-types` |
| `--wasm-features simd` | `-W simd` |
| `--wasm-features tail-call` | `-W tail-call` |
| `--wasm-features threads` | `-W threads` |
| `--wasm-features memory64` | `-W memory64` |
| `--wasm-features copmonent-model` | `-W component-model` |
| `--wasm-features function-references` | `-W function-references` |
| `--relaxed-simd-deterministic` | `-W relaxed-simd-deterministic` |
| `--enable-cranelift-nan-canonicalization` | `-W nan-canonicalization` |
| `--fuel <N>` | `-W fuel=N` |
| `--epoch-interruption` | `-W epoch-interruption` |
| `--allow-unknown-exports` | `-W unknown-exports-allow` |
| `--trap-unknown-imports` | `-W unknown-imports-trap` |
| `--default-values-unknown-imports` | `-W unknown-imports-default` |
| `--max-instances <MAX_INSTANCES>` | `-W max-instances=N` |
| `--max-memories <MAX_MEMORIES>` | `-W max-memories=N` |
| `--max-memory-size <BYTES>` | `-W max-memory-size=N` |
| `--max-table-elements <MAX_TABLE_ELEMENTS>` | `-W max-table-elements=N` |
| `--max-tables <MAX_TABLES>` | `-W max-tables=N` |
| `--max-wasm-stack <MAX_WASM_STACK>` | `-W max-wasm-stack=N` |
| `--trap-on-grow-failure` | `-W trap-on-grow-failure` |
| `--wasm-timeout <TIME>` | `-W timeout=N` |
| `--wmemcheck` | `-W wmemcheck` |
| `--wasi-modules default` | removed |
| `--wasi-modules -default` | removed |
| `--wasi-modules wasi-common` | `-S common` |
| `--wasi-modules -wasi-common` | `-S common=n` |
| `--wasi-modules experimental-wasi-nn` | `-S nn` |
| `--wasi-modules experimental-wasi-threads` | `-S threads` |
| `--wasi-modules experimental-wasi-http` | `-S http` |
| `--listenfd` | `-S listenfd` |
| `--tcplisten <SOCKET ADDRESS>` | `-S tcplisten=...` |
| `--wasi-nn-graph <FORMAT::HOST>` | `-S nn-graph=FORMAT::HOST` |
| `--preview2` | `-S preview2` |
| `--dir <DIRECTORY>` | `--dir ...` |
| `--mapdir <GUEST_DIR::HOST_DIR>` | `--dir a::b` |
* Be more descriptive with help text
* Document `=val` is optional for `-Ccranelift-xxx`
* Fix compile after rebase
* Fix rebase of `--inherit-network`
* Fix wasi-http test
* Fix compile without pooling allocator support
* Update some flags in docs
* Fix bench-api build
* Update flags for gdb/lldb tests
* Fixup optimization flags
prtest:full
* [wasmtime-wasi] fix logic error in `monotonic-clock/subscribe`
When calculating the number of nanoseconds to wait, we should subtract the
current time from the deadline, not vice-versa. This was causing guests to
sleep indefinitely due to integer underflow.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* add `sleep` test to `wasi-tests`
Note that this is annotated `should_panic` when testing preview1 scenarios,
since those won't have preview2 imports.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
---------
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
This only affects tests and the adapter itself, but not in any breaking
way. The tests for wasi-http are reorganized to be commands which is
also required to not have any exports currently since wit-bindgen for
Rust guests doesn't support generating bindings in one crate and
exporting in another.
* feat(wasmtime-cli): add async support flag
Within the wasmtime CLI, the current default behavior
is to only inject the synchronous functions to linkers.
This will add a flag called `--async` that will inject
the asynchronous one instead.
* chore: refactor wasi http crate
* feat(wasmtime-wasi): make in_tokio function public
* feat(wasi-http): define default feature called sync
* Revert "feat(wasmtime-cli): add async support flag"
This reverts commit b743ff2003a2e391972330aa8e8437c6356e580a.
* chore: improve flaky tests for wasi http
* feat(wasi-http): expose sync api for components
* chore: add tests for sync api of wasi http components
* feat(wasmtime-cli): restore support for wasi http module
* chore: revert change to outbound http request invalid test
* chore: have extra tracing to help debugging
* feat(wasi-http): allow modules with sync functions in linker
* fix(wasi-http): missing response body in sync api
* feat: include blocking for io streams
* chore: add tests for wasi http module in cli
* chore: disable preview2 flag in wasi http test
* chore: use preview2 flag in wasi http test
* fix(wasi-http): missing stream output in sync api
* chore: fix tests for wasi http
* chore: add tracing for poll oneoff call
* chore: send exit signal on wasi http test
* chore: swap println to tracing debug
* chore: set http server timeout to 50 secs by default
* chore: add test posting large file
* chore: revert formatting in cargo toml
* chore: fix wasi-http feature and skip failing tests
prtest:full
---------
Co-authored-by: Eduardo Rodrigues <eduardomourar@users.noreply.github.com>
By not activating the `derive` feature on `serde`, the compilation speed
can be improved by a lot. This is because `serde` can then compile in
parallel to `serde_derive`, allowing it to finish compilation possibly
even before `serde_derive`, unblocking all the crates waiting for
`serde` to start compiling much sooner.
As it turns out the main deciding factor for how long the compile time of a
project is, is primarly determined by the depth of dependencies rather
than the width. In other words, a crate's compile times aren't affected
by how many crates it depends on, but rather by the longest chain of
dependencies that it needs to wait on. In many cases `serde` is part of
that long chain, as it is part of a long chain if the `derive` feature
is active:
`proc-macro2` compile build script > `proc-macro2` run build script >
`proc-macro2` > `quote` > `syn` > `serde_derive` > `serde` >
`serde_json` (or any crate that depends on serde)
By decoupling it from `serde_derive`, the chain is shortened and compile
times get much better.
Check this issue for a deeper elaboration:
https://github.com/serde-rs/serde/issues/2584
For `wasmtime` I'm seeing a reduction from 24.75s to 22.45s when
compiling in `release` mode. This is because wasmtime through `gimli`
has a dependency on `indexmap` which can only start compiling when
`serde` is finished, which you want to happen as early as possible so
some of wasmtime's dependencies can start compiling.
To measure the full effect, the dependencies can't by themselves
activate the `derive` feature. I've upstreamed a patch for
`fxprof-processed-profile` which was the only dependency that activated
it for `wasmtime` (not yet published to crates.io). `wasmtime-cli` and
co. may need patches for their dependencies to see a similar
improvement.
* update serde and serde_derive to 1.0.188
Co-authored-by: Christopher Serr <christopher.serr@gmail.com>
* cargo vet: import audits for updates to serde, serde_derive, and syn
---------
Co-authored-by: Christopher Serr <christopher.serr@gmail.com>
* Implement the `tcp` interface of wasi-sockets.
Implement the `tcp`, `tcp-create-socket`, and `network` interfaces of
wasi-sockets.
* Minor cleanups.
* Update to the latest upstream wasi-sockets.
* Address review feedback.
* Handle zero-length reads and writes, and other cleanups.
* Fix compilation on macOS.
* Fix compilation on Windows.
* Update all the copies of wasi-socket wit files.
* Sync up more wit files.
* Fix the errno code for non-blocking `connect` on Windows.
prtest:full
* Tolerate `NOTCONN` errors when cleaning up with `shutdown`.
* Simplify the polling mechanism.
This requires an updated tokio for `Interest::ERROR`.
* Downgrade to tokio 1.29.1 for now.
* Move `tcp_state` out of the `Arc`.
* `accept` doesn't need a write lock.
* Remove `tcp_state`'s `RwLock`.
* wasi-nn: refactor to allow `preview2` access
This change refactors the `wasmtime-wasi-nn` crate to allow access from
both `preview1` and `preview2` ABIs. Though the `wasi-nn` specification
has included a WIT description for some time, here we use some in-tree
files until https://github.com/WebAssembly/wasi-nn/pull/38 is landed.
The `preview2` code is not exercised anywhere yet: ideally this would be
wired up once component model `resource`s are fully implemented in
Wasmtime.
prtest:full
* wasi-nn: use `preview1` linkage
prtest:full
* review: rename `preview*` to `wit*`
This is based on @pchickey's [comments] on ABI naming.
[comments]: https://bytecodealliance.zulipchat.com/#narrow/stream/266558-wasi-nn/topic/wasi-nn.20.2B.20preview2/near/383368292
* review: update README
* fix: remove broken doc links
* fix: replace typo `wit` with `gen`
* review: use `wit` types everywhere
This removes the crate-specific types in order to use the WIT-generated
types throughout the crate. The main effect of this is that the crate
no longer optionally includes `wasmtime` with the `component-model`
feature--now that is required.
* review: move `BackendKind` conversion into `witx.rs`
* review: remove `<'a>`
* review: use `tracing` crate instead of `eprintln!`
* rename wasi-cli-base to wasi-cli, delete `preview` package, import wasi-cli
wasi-cli import is sum of
https://github.com/WebAssembly/wasi-cli/pull/19 and
https://github.com/WebAssembly/wasi-cli/pull/20
* wasi impl: change bindgen arguments and mod paths from cli_base to cli
* correct name of wasi-cli deps dir to just `deps/cli/`
it turns out this isnt semantically meaningful, since the package name
is in the document itself now, but lets be consistient
* track whether stdio isatty in ctx, and impl the cli/terminal-* interfaces
* rebase fixup
* wasi wits: define the reactor adapter's world
* component adapter: fixes
* test-programs/command-tests: fix renaming cli_base to cli
* component adapter: fix manually-defined export and import names
* test harness fixes
* preview1 component adapter: fill in isatty detection
* implement isatty in preview2-to-preview1 host adapter
* test-programs: cover both when stdio isatty and not
prtest:full
* split isatty test for regular file and stdio, detect host stdio is_terminal
CI environments vary - let the test runner make sure the host process's
stdio is in fact a terminal before asserting that the guest sees it is a
terminal.
* provide an is-terminal impl for all preview2's stdio types
which means making a newtype around Stdout and Stderr instead of using
a type alias there.
and then use the is-terminal impl to fill in the isatty field in the
builder when inheriting. if you need to override it you can always
builder.stdin(stdio::stdin(), your_own_idea_of_isatty)
* finally, rename IsATTY variants to Yes and No
* Fix the reference to IsATTY::No
* more forgotten renamings
---------
Co-authored-by: Trevor Elliott <telliott@fastly.com>
* Remove the implementation of wasi-crypto
This commit is a follow-up to the discussion on #6732. This removes
Wasmtime's implementation of the wasi-crypto proposal from in-tree along
with its various support in CI, configuration, etc. See the discussion
on #6732 for the full information but at a high level the main reasons
for removing the implementation at this time are:
* There is not currently an active maintainer of the Wasmtime
integration here for wasi-crypto.
* There are known issues with the code quality of the implementation
such as transmutes of guest-owned memory to `&'static mut [u8]` and
known unsafety in dependencies.
* The size and breadth of the dependency tree brings maintenance burden
and overhead to managing Wasmtime's dependency tree.
As mentioned on the issue this commit does not mean that Wasmtime
doesn't want to implement the wasi-crypto proposal. Instead the "tier 3"
status of wasi-crypto needs to be re-attained to be included back
in-tree, which would mean resolving the above issues.
Note that this commit is intentionally just after the 13.0.0 branch
point which means that this is slated for Wasmtime 14 to be released on
September 20.
* Remove some cfgs
* Remove wasi-crypto CI
* Add config method for whether or not to capture coredumps on trap
* Add core dump support to the runtime
This adds the machinery to capture a core dump when a trap occurs
and attach it to the resulting anyhow::Error that gets bubbled up to
the caller. I've created a CoreDumpStack structure in the runtime, which is
currently just a backtrace until we design a way to recover the locals
stack values when a trap occurs. When that CoreDumpStack gets converted to
a wasmtime::WasmCoreDump, we add additional information from the Store such
as globals, memories, and instance information.
A lot of this is mechanistically similar to how backtraces
are captured and attached to errors. Given that they both are attached as
context to anyhow::Errors, setting coredump_on_trap to true will supercede
any setting for wasm_backtrace.
* Address some PR feedback
* Fix docs
* Switch WasmCoreDump to have vec of module names rather than Modules
* Add smoketests for WasmCoreDump
* clean up tests
* Update memories and globals field to be store_
* add debug impl for wasmcoredump
* Remove duplicate imports
* ignore miri for wasm tests
* Remove unused WIT files from Wasmtime
These files aren't actually read by anything currently. They were added
historically and a previous refactoring in #6390 forgot to remove them.
No tests or build process reads them so this deletes them to get them
out of the way.
* Update dependencies on wasm-tools crates.
This commit updates the deps on the wasm-tools family of crates to bring
in a few fixes for WIT/component-related things. Primarily though this
brings in an update to the component model where empty types are now
disallowed.
Some tests using empty types were adjusted to use non-empty types, but
many tests were also simply removed as they existed to test what would
happen with empty types which now no longer needs to be tested.
* Update `stream-error` in preview2
Add a `dummy` field to make it a non-empty structure. It's expected that
this will change to something else more "official" in the future, but
for now this is here to keep everything compiling.
* Update component fuzzing to avoid empty types
Empty types are no longer valid
* Update crates/wasi/wit/deps/io/streams.wit
Co-authored-by: Peter Huene <peter@huene.dev>
---------
Co-authored-by: Peter Huene <peter@huene.dev>
* preview2: make everything but streams/io and poll/poll synchronous
* streams: get rid of as_any method, which is no longer used
* delete legacy sched and pollable concepts
* more code motion and renaming
* make tokio a workspace dep, because we need it directly in wasmtime-wasi
* HostPollable exists
* more fixes
* pollable can trap, and implement clock properly
* HostPollable is now a generator of futures
because we need to be able to poll a pollable many times
* explain various todo!s
* Synchronous version of the wasi-preview2-components tests
* Change with_tokio to accept the future as an argument
* Store futures in the PollOneoff struct instead, to avoid dropping them
* Remove TODO for HostOutputStream impl for WritePipe
* Implement pollable for ReadPipe
* Use a Notify when ReadPipe is ready
* wip
* wip
* Read/write pipe ends with tokio channels
* Empty reader/writer wrappers
* EmptyStream, and warning cleanup
* Wrapped reader/writer structs
* Rework stdio in terms of wrapped read/write
* Add MemoryOutputPipe and update tests
* Remove todo
* rewrite nearly everything
* implement the pipe stuff
* wibble
* fix MemoryOutputPipe just enough to make the tests compile
* Move the table iteration into a helper function
* AsyncFd stream implementation to fix stdin on unix
* Rename Wrapped{Read,Write} streams to Async{Read,Write}Stream
* Move async io wrappers into stream.rs
* Fix the sync tests
* fix test uses of pipes, juggle tokio context for stdin construction
* add some fixmes
* the future i named Never is defined in futures-util as pending
which is a better name
* i believe this is a correct implementation of one global stdin resource
* move unix stdin to its own file
* make most of the mods private
* fix build - we are skipping rust 1.70
due to llvm regressions in s390x and riscv64 which are fixed in 1.71 and
will not be backported
* preview1-in-preview2: use async funcs for io, and the async io interface
prtest:full
* windows stdin support
* done!
* table ext functions: fix tests
* tests: expect poll_oneoff_{files,stdio} to pass on all platforms
* export the bindings under wasmtime_wasi::preview2::bindings
rather than preview2::wasi.
and command moves to wasmtime_wasi::preview2::command as well.
* fix renaming of wasi to bindings in tests
* use block_in_place throughout filesystem
and move block_on and block_in_place to be pub crate at the root
* AsyncFdStream: ensure file is nonblocking
* tests: block_in_place requires multi-threaded runtime
* actually, use fcntl_setfl to make the asyncfd file nonblocking
* fix windows block_on
* docs, remove unnecessary methods
* more docs
* Add a workspace dependency on bytes-1.4
* Remove vectored stream operations
* Rework the read/write stream traits
* Add a size parameter to `read`, and switch to usize for traits
* Pipe through the bool -> stream-status change in wit
* Plumb stream-status through write operations in wit
* write host trait also gives streamstate
* hook new stream host read/write back up to the wit bindgen
* sketchy AsyncReadStream impl
* Fill out implementations for AsyncReadStream and AsyncWriteStream
* some reasonable read tests
* more
* first smoke test for AsyncWriteStream
* bunch of AsyncWriteStream tests
* half-baked idea that the output-stream interface will need a flush mechanism
* adapter: fixes for changes to stream wit
* fix new rust 1.71 warnings
* make stdin work on unix without using AsyncFdStream
inline the tokio docs example of how to impl AsyncRead for an AsyncFd,
except theres some "minor" changes because stdin doesnt impl Read on
&Stdin whereas tcpstream from the example does
* delete AsyncFdStream for now
it turns out to be kinda hard and we can always work on adding it back
in later.
* Implement some memory pipe operations, and move async wrappers to the pipe mod
* Make blocking_write actually block until everything is written
* Remove debug print
* Adapter stdio should use blocking write
Rust guests will panic if the write returns less than the number of
bytes sent with stdio.
* Clean up implementations of {blocking_}write_zeros and skip
* Remove debug macro usage
* Move EmptyStream to pipe, and split it into four variants
Use EmptyInputStream and SinkOutputStream as the defaults for stdin and
stdout/stderr respectively.
* Add a big warning about resource lifetime tracking in pollables
* Start working through changes to the filesystem implementation
* Remove todos in the filesystem implementation
* Avoid lifetime errors by moving blocking operations to File and Dir
* Fix more lifetime issues with `block`
* Finish filling out translation impl
* fix warnings
* we can likely eliminate block_in_place in the stdin implementations
* sync command uses sync filesystem, start of translation layer
* symc filesystem: all the trait boilerplate is in place
just need to finish the from impl boilerplate
* finish type conversion boilerplate
* Revert "half-baked idea that the output-stream interface will need a flush mechanism"
This reverts commit 3eb762e333.
* cargo fmt
* test type fixes
* renames and comments
* refactor stream table internals so we can have a blocking variant...
* preview1 host adapter: stdout/stderr use blocking_write here too
* filesystem streams are blocking now
* fixes
* satisfy cargo doc
* cargo vet: dep upgrades taken care of by imports from mozilla
* unix stdio: eliminate block_in_place
* replace private in_tokio with spawn, since its only used for spawning
* comments
* worker thread stdin implementation can be tested on linux, i guess
and start outlining a test plan
* eliminate tokio boilerplate - no longer using tokios lock
* rename our private block_on to in_tokio
* fill in missing file input skip
* code review: fix MemoryInputPipe. Closed status is always available immediately.
* code review: empty input stream is not essential, closed input stream is a better fi for stdin
* code review: unreachable
* turn worker thread (windows) stdin off
* expect preview2-based poll_oneoff_stdio to fail on windows
* command directory_list test: no need to inherit stdin
* preview1 in preview2: turn off inherit_stdio except for poll_oneoff_stdio
* wasi-preview2-components: apparently inherit_stdio was on everywhere here as well. turn it off
except for poll_oneoff_stdio
* extend timeout for riscv64 i suppose
---------
Co-authored-by: Trevor Elliott <telliott@fastly.com>
Do this by updating `crossbeam-epoch` and auditing this update of
crossbeam. The newer version of crossbeam additionally updates its
version of `memoffset`.
This commit adds `cargo vet` trust entries for any crate published by
BurntSushi, of which a good number are in our dependency graph. This
additionally updates the `bstr` crate to its latest version and updates
regex-related dependencies from other crates to avoid duplication of
versions.
* Remove deny.toml exception for wasm-coredump-builder
This isn't used any more so no need to continue to list this.
* Update Wasmtime's pretty_env_logger dependency
This removes a `deny.toml` exception for that crate, but `openvino-sys`
still depends on `pretty_env_logger 0.4.0` so a new exception is added
for that.
* Update criterion and clap dependencies
This commit started out by updating the `criterion` dependency to remove
an entry in `deny.toml`, but that ended up transitively requiring a
`clap` dependency upgrade from 3.x to 4.x because `criterion` uses
pieces of clap 4.x. Most of this commit is then dedicated to updating
clap 3.x to 4.x which was relatively simple, mostly renaming attributes
here and there.
* Update gimli-related dependencies
I originally wanted to remove the `indexmap` clause in `deny.toml` but
enough dependencies haven't updated from 1.9 to 2.0 that it wasn't
possible. In the meantime though this updates some various dependencies
to bring them to the latest and a few of them now use `indexmap` 2.0.
* Update deps to remove `windows-sys 0.45.0`
This involved updating tokio/mio and then providing new audits for new
crates. The tokio exemption was updated from its old version to the new
version and tokio remains un-audited.
* Update `syn` to 2.x.x
This required a bit of rewriting for the component-macro related bits
but otherwise was pretty straightforward. The `syn` 1.x.x track is still
present in the wasi-crypto tree at this time.
I've additionally added some trusted audits for my own publications of
`wasm-bindgen`
* Update bitflags to 2.x.x
This updates Wasmtime's dependency on the `bitflags` crate to the 2.x.x
track to keep it up-to-date.
* Update the cap-std family of crates
This bumps them all to the next major version to keep up with updates.
I've additionally added trusted entries for publishes of cap-std crates
from Dan.
There's still lingering references to rustix 0.37.x which will need to
get weeded out over time.
* Update memoffset dependency to latest
Avoids having two versions in our crate graph.
* Fix tests
* Update try_from for wiggle flags
* Fix build on AArch64 Linux
* Enable `event` for rustix on Windows too
* Cranelift: upgrade to regalloc2 0.9.2.
This pulls in bytecodealliance/regalloc2#152, which fixes a bug that is
reachable on RISC-V: when two different register classes have the same
stackslot size, the register allocation result might share a slot
between two different classes, which can result in moves between classes
that will cause a panic. The fix properly separates slots by class.
* cargo-vet update for regalloc2 0.9.2.
* Update wasm-tools dependencies
* Get tests passing after wasm-tools update
Mostly dealing with updates to `wasmparser`'s API.
* Update `cargo vet` for new crates
* Add `equivalent`, `hashbrown`, and `quote` to the list of trusted
authors. We already trust these authors for other crates.
* Pull in some upstream audits for various deps.
* I've audited the `pulldown-cmark` dependency upgrade myself.
* Attempt versioned exports to facilitate having multiple versions in the same crate
* Modify approach to use `export_name` and `link_name`
* Only apply version to names in assembly and foreign item fns
* Attempt to handle the s390x case
* Fix alignment of backslashes in assembly file
* Pretend I understand the preprocessor
* Version symbols in `crates/runtime/src/helpers.c`
* Stop versioning `__jit_debug_register_code` because gdb relies on it and it is uses `weak` linkage
* Version symbol in `crates/fiber/src/windows.c`
* Consolidate `LitStr` creation in macro
* Add new crate to publish script and supply-chain config
* Fix order in supply chain config
* Set `audit-as-crates-io` to false
* Missing `versioned_link` for Windows
* Version strings used in debug
* Formatting
* Get rid of `versioned_str` and bring back `versioned_suffix`
---------
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
* Update v8 and proc-macro2 dependencies
Gets them both compiling on the latest nightly so we can unpin the Rust
compiler version in OSS-Fuzz.
* Update nightly in CI
Alex Crichton
Trevor Elliott
wasmtime-publish
Tyler Rockwood
Dan Gohman
Pat Hickey
Andrew Brown
Joel Dice
Eduardo de Moura Rodrigues
Christopher Serr
Lann
Quentin Gliech
Alphyr
ssunkin-fastly
Rainy Sinclair
Chris Fallin
Adam Petro
Nick Fitzgerald