* wasi-http: Validate Content-Length when present (#7527)
* Modify outgoing-body.finish to return a result
* Validate content length when finishing an outgoing-body
* Track written vs expected, instead of decrementing
* Switch to the new errors
* Move content-length tracking to BodyWriteStream
* Review feedback
* Remove a TODO
* WASI wits: use 0.2.0-rc-2023-11-10 (#7533)
* wit deps: use 0.2.0-rc-2023-11-10 from WebAssembly/wasi-* repos
* local wits: depend on rc-2023-11-10
* wit-bindgen invocation: use rc-2023-11-10
* wasi-http wit subdir: sync with wasi
* wasi-http wit-bindgen: update for 11-10 rc
* wasi-preview1-component-adapter: imports are from 11-10 rc
* cli test component-basic: use wasi rc 11-10
---------
Co-authored-by: Trevor Elliott <telliott@fastly.com>
Co-authored-by: Pat Hickey <phickey@fastly.com>
* Remove timezone interface from wasi-clocks (#7515)
* delete wasi-clocks timezone interface: import wit changes from https://github.com/WebAssembly/wasi-clocks/pull/55
* remove other references to wasi:clocks/timezone in wits
* remove todo! implementation of clocks/timezone and add_to_linker funcs
* Move the `wasi:io/stream/error` resource into `wasi:io/error` (#7521)
* Move the `error` resource into `wasi:io/error`
* error.wit: update doc comments
* downstream fixes to streams.wit doc comments
* fix package name in error.wit
---------
Co-authored-by: Trevor Elliott <telliott@fastly.com>
* wasi-http: Make child fields immutable (#7524)
* Make child fields immutable
* Add `get_fields` and remove `FieldMapMutability`
Clean up the interface to immutable fields by adding a different
accessor.
* Clean up the diff
* wasi-http: Migrate to more descriptive error variant (#7434)
* Migrate to a more specific error-code variant in wasi-http
Co-authored-by: Pat Hickey <phickey@fastly.com>
* Optional fields, and align with upstream pr
* Update for upstream changes to the error-code variant
* Sync with the upstream implementation
* Missed updating an error for riscv64 and s390x
* More debuggable error
prtest:full
* Try to stabilize the test on windows
---------
Co-authored-by: Pat Hickey <phickey@fastly.com>
* Remove a debugging eprintln (#7528)
* Remove no-longer-necessary reactor world (#7516)
The `wasi:cli` WIT package now has a `reactor` world so the adapter can
use that instead of defining its own.
---------
Co-authored-by: Pat Hickey <phickey@fastly.com>
Co-authored-by: Trevor Elliott <telliott@fastly.com>
This commit is in the same spirit as #7461 and is aimed at removing an
extra task when interoperating between WebAssembly and Hyper. The
intention here is to ensure that when data is ready immediately a
context switch between Tokio tasks isn't necessary. I'm still not sure
why this context switch would tank performance so much.
The goal of this commit is to remove the Tokio task associated with
reading bodies. Currently a task is spawned with two channels coming
out: one for frames and one for trailers. Instead the body is now read
directly when data frames are read and trailers are managed similarly.
The interactions here are somewhat nontrivial due to the management of
resources and how the body needs to jump from the body stream object to
the trailers object, but it's overall not the worst thing in the world.
Locally on Linux this takes a "hello world" component which starts out
by reading its HTTP body and then responds with a string from 6.7k rps
locally via `wasmtime serve` to 16.8k rps. This restores the performance
of wasi-http to be in-line with the historical SDK that Fermyon offered,
for example, that's based on a custom HTTP interface. Note that these
performance numbers use #7461 as a baseline.
This commit has a downside, however, that if the incoming body is not
being subscribed to actively by the wasm then no activity will happen on
the host. This is a function of `poll` isn't being actively watched
unless `subscribe`'s result is being actively watched via a wasm call to
`wasi:io/poll`. Manual investigation of Hyper shows that this is
probably ok for now since bodies are primarily channel-based which seems
like the tasks doing the actual I/O are managed elsewhere. In that sense
this shouldn't cause any immediate impact, but Hyper's situation in the
future could change or my analysis could additionally be wrong. Given
the performance benefits here though I'd be tempted to push in favor of
this commit and handle "poll in the background" as necessary in the
future if needed.
This adds a new `send_request` method to `WasiHttpView`, allowing embedders to
override the default implementation with their own if the desire. The default
implementation behaves exactly as before.
I've also added a few new `wasi-http` tests: one to test the above, and two
others to test streaming and concurrency. These tests are ports of the
`test_wasi_http_echo` and `test_wasi_http_hash_all` tests in the
[Spin](https://github.com/fermyon/spin) integration test suite. The component
they instantiate is likewise ported from the Spin
`wasi-http-rust-streaming-outgoing-body` component.
Fixes#7259
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* Refactor the test-programs test suite
This commit is a large refactoring that reorganizes `test-programs` and
how we tests wasms in Wasmtime. Often writing tests requires complicated
interactions with the guest which can't be done via hand-written `*.wat`
syntax and requires a compiler to get engaged. For this purpose Wasmtime
currently has the `crates/test-programs/*` test suite which builds files
from source and then runs the tests. This has been somewhat cumbersome
in the past though and it's not been easy to extend this over time, so
this commit attempts to address this.
The scheme implemented in this PR looks like:
* All wasm test programs live in `crates/test-programs/src/bin/*.rs`.
All of them, no exceptions.
* Wasm tests have shared support located at
`crates/test-programs/src/lib.rs` and its submodules, such as bindings
generation for WASI.
* Wasm tests are built by a new `crates/test-programs/artifacts` crate.
This crate compiles modules and additionally creates components for
all test programs. The crate itself only records the path to these
outputs and a small amount of testing support, but otherwise doesn't
interact with `wasmtime`-the-crate itself.
* All tests in `crates/test-programs/tests/*.rs` have moved. For example
wasi-http tests now live at `crates/wasi-http/tests/*.rs`. Legacy
tests of wasi-common now live at `crates/wasi-common/tests/*.rs`.
Modern tests for preview2 live at `crates/wasi/tests/*.rs`.
* Wasm tests are bucketed based on their filename prefix. For example
`preview1_*` is tested in wasi-common and wasmtime-wasi. The
`preview2_*` prefix is only tested with wasmtime-wasi, however.
* A new `cli_*` prefix is used to execute tests as part of
`tests/all/main.rs`. This is a new submodule in
`tests/all/cli_tests.rs` which executes these components on the
command line. Many old "command" tests were migrated here.
* Helper macros are generated to assert that a test suite is run in its
entirety. This way if a `preview1_*` test is added it's asserted to
get added to both wasi-common and wasmtime-wasi in the various modes
they run tests.
Overall this moved a number of tests around and refactored some edges of
the tests, but this should not lose any tests (except one that wasn't
actually testing anything). Additionally the hope is that it's much
easier to add tests in the future. The process is to add a new file in
`crates/test-programs/src/bin/*.rs` named appropriately. For example a
preview2 executable is `preview2_*` and a CLI tests is `cli_*`. When
building the test suite an error is generated in the appropriate module
then of "please write a test here", and then a test is written in the
same manner as the other tests in the module.
* Remove no-longer-needed fetches
prtest:full
* I'm worried wasi is running low on semicolons
* Add the WASI target in all CI actions
* Add unknown-unknown target on all CI builders too
* Fix building test artifacts under miri
Need to avoid wrappers for these cross-compiled targets
* Break circular dependency for packaging
Don't use the workspace dep for `wasmtime-wasi` since it injects a
version, instead use a `path = '..'` dependency to fool Cargo into
dropping the dependency during the package phase.
* Fix some merge conflicts with tests
* Fix rebase for new tests
* Remove stray comment
* Fix some flaky tests
* Fix network tests in synchronous mode
This commit is an attempt to fix some networking tests in synchronous
mode in our test suite. Currently networking tests don't actually run in
synchronous mode on CI which is why no failures have been surfaced yet,
but the refactoring in #7182 is going to start doing this.
Currently the `udp_sample_application.rs` test blocks infinitely in
synchronous mode for me locally, most of the time. This appears to be an
interaction between how Tokio handles readiness and how we're
entering the event loop. We're effectively entering the Tokio event loop
with a future that's always ready which ends up starving Tokio of
otherwise performing its background work such as updating flags for
readiness of reading/writing.
The fix here is to add a yield at the start of an `in_tokio` block which
is used in synchronous mode. This is a kludge fix but the intention is
to enable Tokio to have a chance to update readiness flags and process
events from epoll/kqueue/etc.
An additional fix to this issue is WebAssembly/wasi-sockets#64 where the
test is waiting on `READABLE` or `WRITABLE`, but in this specific case
it should only wait on `READABLE`. If it waited on just this then that
would also fix this issue. Nevertheless having a `yield_now` is expected
to have little-to-no overhead and otherwise fix this edge case of an
always-ready future.
* Fix passing empty arguments on the CLI
* Add another blocking accept
* Update crates/test-programs/src/bin/api_proxy.rs
Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>
---------
Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>
Trevor Elliott
Alex Crichton
Ryan Levick
Joel Dice