* Update wit-bindgen
This commit updates wit-bindgen to 0.25 and applies some "extra
trickery" to work around the now-default providing of the realloc
symbol.
* Add audits
With cargo-vet the cross-organization trust model is not quite the same
with these two constructs in cargo-vet. Previously Wasmtime/wasm-tools
crates were flagged as `[[wildcard-audits]]` but now being changed to
all using `wasmtime-publish` to publish crates the `[[trusted]]` entries
were added at the recommendation of `cargo vet`. This means that other
organizations could no longer import our own audits since `[[trusted]]`
entries aren't imported, only suggested.
This commit changes all these entries to `wildcard-audits` with an
explanation as to why.
Wasmtime and Cranelift have a few miscellaenous use cases for "just take
this Rust type and make it bytes", for example Wasmtime's serialization
of internal metadata into a compiled module. Previously Wasmtime used
the `bincode` crate for performing these tasks as the format was
generally optimized to be small and fast, not general purpose (e.g.
JSON). The `bincode` crate on crates.io doesn't work on `no_std`,
however, and with the work in #8341 that's an issue now for Wasmtime.
This crate switches instead to the `postcard` crate. This crate is
listed in Serde's documentation as:
> Postcard, a no_std and embedded-systems friendly compact binary
> format.
While I've not personally used it before it checks all the boxes we
relied on `bincode` for and additionally works with `no_std`. After
auditing the crate this commit then switches out Wasmtime's usage of
`bincode` for `postcard` throughout the repository.
This provides a bit of a nicer experience than the default "build your
own test harness" experience by providing things like filters and
parallel execution by default. This helps speed up the `disas` test
suite, for example, which previously had no parallelism.
* add an initial implemenation for onnxruntime backend of wasi-nn
Signed-off-by: David Justice <david@devigned.com>
* vet: audit ONNX dependencies
This change is the result of a long slog through the dependencies of the
`ort` library. The only missing dependency is `compact_str`, which needs
further discussion.
* vet: add ONNX audit entry for compact_str 0.7.1
Signed-off-by: David Justice <david@devigned.com>
* refactor tests to break out onnx and openvino
Signed-off-by: David Justice <david@devigned.com>
* mark wasi-nn onnx example as publish false
Signed-off-by: David Justice <david@devigned.com>
* update the ONNX classification example
* do not use wasi-nn onnx feature if riskv or s390
Signed-off-by: David Justice <david@devigned.com>
* prtest:full fix running WASI-NN ONNX tests across arch os
Signed-off-by: David Justice <david@devigned.com>
---------
Signed-off-by: David Justice <david@devigned.com>
Co-authored-by: Andrew Brown <andrew.brown@intel.com>
* Migrate mach to mach2
* Add audit for mach2 0.4.1 -> 0.4.2
* Use mach2 things instead of custom addons
* Fix build on aarch64-apple-darwin.
* Cast the u32 flags to i32.
* Add back custom definition that doesn't match `mach2`
prtest:full
* Fix x64 unused import
---------
Co-authored-by: Junji Takakura <j.takakura@gmail.com>
Co-authored-by: Chris Fallin <chris@cfallin.org>
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
* Add a "custom" platform configuration for Wasmtime
This commit leverages adds a new "platform" to Wasmtime to be supported
in the `crates/runtime/src/sys` folder. This joins preexisting platforms
such as Unix and Windows. The goal of this platform is to be an opt-in
way to build Wasmtime for targets that don't have a predefined way to
run.
The new "custom" platform requires `--cfg wasmtime_custom_platform` to
be passed to the Rust compiler, for example by using `RUSTFLAGS`. This
new platform bottoms out in a C API that is intended to be small and
Linux-like. The C API is effectively the interface to virtual memory
that Wasmtime requires. This C API is also available as a header file at
`examples/min-platform/embedding/wasmtime-platform.h` (generated by
`cbindgen`).
The main purpose of this is to make it easier to experiment with porting
Wasmtime to new platforms. By decoupling a platform implementation from
Wasmtime itself it should be possible to run these experiments
out-of-tree. An example of this I've been working on is getting
Wasmtime running on bare-metal with a custom kernel. This support
enables defining the platform interface of the custom kernel's syscalls
outside of Wasmtime.
* Exclude wasmtime-platform.h from formatting
* Include build-wasmtime-target-wasm32 in final job
* Don't force any single toolchain
* Add notes to no_std docs
* Add rust-src to CI
* Review comments
* Change APIs to be fallible
* Only compile the min-platform example on Linux
* Fix compile of min-platform example
* Fix another compile error in the example
* Remove the `WASI` submodule
Historically the `WASI` submodule in this repository was used to pull
the sources of the `*.witx` files for preview1. This has never been used
by the `wasmtime-wasi` crate (which has its own copy in the
`crates/wasi/witx` folder) and was only ever used by `wasi-common`. This
submodule also served as a location for the `witx` crate itself.
Neither of these are really needed any more as the `*.witx` files are
unlikely to ever really change again. This commit removes the submodule
entirely, as well as the `path` dependency on `witx`, and copies the
`*.witx` files in the same manner as the `wasmtime-wasi` crate.
* Updated vet entries for witx
* Remove witx special cases in publish script
* Change how the wasi-common package is built
This fixes the build on the latest nightly Rust. I was able to vet
`ahash` itself but `zerocopy` is such a large and full-of-unsafe
dependency I've added an exemption for it. The documentation of it seems
to indicate it's a pretty well thought out crate with lots of care
behind it, so at least at a first glance it did not seem overly
worrisome.
Looks like it's been awhile since our last update. I've updated the
dependency here and additionally vetted the various dependency upgrades.
I notably did not vet `zstd-sys` since that would require vetting all of
zstd's C implementation itself which I am not, nor do I suspect many of
are, equipped to do. In lieu of that I've updated the exemption of
`zstd-sys` to the newer version.
Closes#7869
Like the rest of the `windows-*` crates published by Kenny Kerr, this
change also adds the `windows` crate itself to the trusted list. This is
necessary for use in #7807.
* Update several dependencies to windows-sys 0.52.
Update cap-std, rustix, io-extras, errno, fd-lock, fs-set-times, and
winx to versions which use windows-sys 0.52.
I started out hoping that everything could be updated to
windows-sys 0.52, however tokio and mio have not yet updated to
windows-sys 0.52 due to supporting an older MSRV.
* Link the C++ examples with CoreFoundation.
This is needed due to the iana-time-zone support.
prtest:full
* Fix compilation on non-Apple platfoms.
* Fix compilation on Apple platforms.
* wasi-nn: add test programs
This change adds new test programs for wasi-nn in a way fits in with the
existing WASI test infrastructure. The code is not new, though: this
reuses the wasi-nn `examples`, which are currently used by the
`run-wasi-nn-example.sh` CI script. Eventually the examples will be
removed in favor of these tests.
Because wasi-nn's component model support is still in flight, this
change also skips the generation of components for `nn_`-prefixed tests.
* wasi-nn: add `testing` module
This testing-only module has code (i.e., `check_test!`) to check whether
OpenVINO and some test artifacts are available. The test artifacts are
downloaded and cached if not present, expecting `curl` to be present on
the command line (as discussed in the previous version of this, #6895).
* wasi-nn: run `nn_*` test programs as integration tests
Following the pattern of other WASI crates, this change adds the
necessary infrastructure to run the `nn_*` files in
`crates/test-programs` (built by `test-program-artifacts`). These tests
are only run when two sets of conditions are true:
- statically: we only run these tests where we expect OpenVINO to be
easy to install and run (e.g., the `cfg_attr` parts)
- dynamically: we also only run these tests when the OpenVINO libraries
can be located and the model artifacts can be downloaded
* ci: install OpenVINO for running wasi-nn tests
prtest:full
* vet: certify the `wasi-nn` crate
* ci: remove wasi-nn test script
* mpk: allow checking for MPK without a config instance
It is inconvenient to have to construct a `PoolingAllocationConfig` in
order to check if memory protection keys are available. This removes
the unused `&self` restriction.
* mpk: improve logging of calculated slab layout
When double-checking the slab layout calculations it is quite convenient
to see the total slab size. This helps in correlating with mapped
regions.
* mpk: add an example testing the memory limits
This adds an example that can be run with `cargo run --example mpk`. Not
only does the example demonstrate how to build a pool-allocated engine
that uses MPK, it performs an exponential search to find the maximum
number of slots the system can support, with and without MPK.
* review: document Linux requirement
* review: `env_logger::init`
* review: replace `proc-maps` with manual parsing
* vet: audit `bytesize`
* fix: provide `main` for non-Linux systems
* fix: move `cfg` to avoid unused code
* Update `errno` crate to drop unused dependency
Small change to our dependency graph.
* Update rayon-core to drop some dependency edges
Just a bit of pruning for our dependency graph.
* Update num_cpus dependency to drop a dependency
Minor pruning of our dependency graph.
* Update wasm-tools crates
This commit updates the wasm-tools family of crate for a number of
notable updates:
* bytecodealliance/wasm-tools#1257 - wasmparser's ID-based
infrastructure has been refactored to have more precise types for each
ID rather than one all-purpose `TypeId`.
* bytecodealliance/wasm-tools#1262 - the implementation of
"implementation imports" for the component model which both updates
the binary format in addition to adding more syntactic forms of
imports.
* bytecodealliance/wasm-tools#1260 - a new encoding scheme for component
information for `wit-component` in objects (not used by Wasmtime but
used by bindings generators).
Translation for components needed to be updated to account for the first
change, but otherwise this was a straightforward update.
* Remove a TODO
* Update some crates to align on latest `rustix`
This commit addresses some dependabot warnings showing up on the
Wasmtime repository by updating all dependencies to using the latest
`rustix` release.
* Debug CI
prtest:full
* Downgrade rustix to 0.38.14
Works around bytecodealliance/rustix#901
Some notable changes here are:
* The `wasm-tools` crates have enabled the `relaxed-simd`, `threads`,
and `multi-memory` proposals by default. For now I've left these
disabled-by-default in Wasmtime to get enabled in a future PR.
* The `wast` crate has support for parsing `thread` and `wait`
constructs from the `threads` proposal for WebAssembly. They're left
unimplemented for now and return errors. This will get filled in in a
future update.
* Rename `Host*` things to avoid name conflicts with bindings.
* Update to the latest resource-enabled wit files.
* Adapting the code to the new bindings.
* Update wasi-http to the resource-enabled wit deps.
* Start adapting the wasi-http code to the new bindings.
* Make `get_directories` always return new owned handles.
* Simplify the `poll_one` implementation.
* Update the wasi-preview1-component-adapter.
FIXME: temporarily disable wasi-http tests.
Add logging to the cli world, since stderr is now a reseource that
can only be claimed once.
* Work around a bug hit by poll-list, fix a bug in poll-one.
* Comment out `test_fd_readwrite_invalid_fd`, which panics now.
* Fix a few FIXMEs.
* Use `.as_ref().trapping_unwrap()` instead of `TrappingUnwrapRef`.
* Use `drop_in_place`.
* Remove `State::with_mut`.
* Remove the `RefCell` around the `State`.
* Update to wit-bindgen 0.12.
* Update wasi-http to use resources for poll and I/O.
This required making incoming-body and outgoing-body resourrces too, to
work with `push_input_stream_child` and `push_output_stream_child`.
* Re-enable disabled tests, remove logging from the worlds.
* Remove the `poll_list` workarounds that are no longer needed.
* Remove logging from the adapter.
That said, there is no replacement yet, so add a FIXME comment.
* Reenable a test that now passes.
* Remove `.descriptors_mut` and use `with_descriptors_mut` instead.
Replace `.descriptors()` and `.descriptors_mut()` with functions
that take closures, which limits their scope, to prevent them from
invalid aliasing.
* Implement dynamic borrow checking for descriptors.
* Add a cargo-vet audit for wasmtime-wmemcheck.
* Update cargo vet for wit-bindgen 0.12.
* Cut down on duplicate sync/async resource types (#1)
* Allow calling `get-directories` more than once (#2)
For now `Clone` the directories into new descriptor slots as needed.
* Start to lift restriction of stdio only once (#3)
* Start to lift restriction of stdio only once
This commit adds new `{Stdin,Stdout}Stream` traits which take over the
job of the stdio streams in `WasiCtxBuilder` and `WasiCtx`. These traits
bake in the ability to create a stream at any time to satisfy the API
of `wasi:cli`. The TTY functionality is folded into them as while I was
at it.
The implementation for stdin is relatively trivial since the stdin
implementation already handles multiple streams reading it. Built-in
impls of the `StdinStream` trait are also provided for helper types in
`preview2::pipe` which resulted in the implementation of
`MemoryInputPipe` being updated to support `Clone` where all clones read
the same original data.
* Get tests building
* Un-ignore now-passing test
* Remove unneeded argument from `WasiCtxBuilder::build`
* Fix tests
* Remove some workarounds
Stdio functions can now be called multiple times.
* If `poll_oneoff` fails part-way through, clean up properly.
Fix the `Drop` implementation for pollables to only drop the pollables
that have been successfully added to the list.
This fixes the poll_oneoff_files failure and removes a FIXME.
---------
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
* Move the incoming_handler impl into http_impl
* Remove the incoming handler -- we need to use it as a guest export
* Start adding a test-programs test for the server side of wasi-http
* Progress towards running a server test
* Implement incoming-request-method
* Validate outparam value
* Initial incoming handler test
* Implement more of the incoming api
* Finish the incoming api implementations
* Initial cut at `wasmtime serve`
* fix warning
* wasmtime-cli: invoke ServeCommand, and add enough stuff to the linker to run trivial test
* fix warnings
* fix warnings
* argument parsing: allow --addr to specify sockaddr
* rustfmt
* sync wit definitions between wasmtime-wasi and wasmtime-wasi-http
* cargo vet: add an import config and wildcard audit for wasmtime-wmemcheck
* cargo vet: audit signal-hook-registry
* Remove duplicate add_to_linker calls for preview2 interfaces
prtest:full
* Add a method to finish outgoing responses
Co-authored-by: Adam Foltzer <acfoltzer@fastly.com>
Co-authored-by: Pat Hickey <phickey@fastly.com>
* Mark the result of the incoming_{request,response}_consume methods as own
* Explicit versions for http-body and http-body-util
* Explicit `serve` feature for the `wasmtime serve` command
* Move the spawn outside of the future returned by `ProxyHandler::call`
* Review feedback
---------
Co-authored-by: Trevor Elliott <telliott@fastly.com>
Co-authored-by: Adam Foltzer <acfoltzer@fastly.com>
This only affects tests and the adapter itself, but not in any breaking
way. The tests for wasi-http are reorganized to be commands which is
also required to not have any exports currently since wit-bindgen for
Rust guests doesn't support generating bindings in one crate and
exporting in another.
* Implement the `tcp` interface of wasi-sockets.
Implement the `tcp`, `tcp-create-socket`, and `network` interfaces of
wasi-sockets.
* Minor cleanups.
* Update to the latest upstream wasi-sockets.
* Address review feedback.
* Handle zero-length reads and writes, and other cleanups.
* Fix compilation on macOS.
* Fix compilation on Windows.
* Update all the copies of wasi-socket wit files.
* Sync up more wit files.
* Fix the errno code for non-blocking `connect` on Windows.
prtest:full
* Tolerate `NOTCONN` errors when cleaning up with `shutdown`.
* Simplify the polling mechanism.
This requires an updated tokio for `Interest::ERROR`.
* Downgrade to tokio 1.29.1 for now.
* Move `tcp_state` out of the `Arc`.
* `accept` doesn't need a write lock.
* Remove `tcp_state`'s `RwLock`.
* Remove the implementation of wasi-crypto
This commit is a follow-up to the discussion on #6732. This removes
Wasmtime's implementation of the wasi-crypto proposal from in-tree along
with its various support in CI, configuration, etc. See the discussion
on #6732 for the full information but at a high level the main reasons
for removing the implementation at this time are:
* There is not currently an active maintainer of the Wasmtime
integration here for wasi-crypto.
* There are known issues with the code quality of the implementation
such as transmutes of guest-owned memory to `&'static mut [u8]` and
known unsafety in dependencies.
* The size and breadth of the dependency tree brings maintenance burden
and overhead to managing Wasmtime's dependency tree.
As mentioned on the issue this commit does not mean that Wasmtime
doesn't want to implement the wasi-crypto proposal. Instead the "tier 3"
status of wasi-crypto needs to be re-attained to be included back
in-tree, which would mean resolving the above issues.
Note that this commit is intentionally just after the 13.0.0 branch
point which means that this is slated for Wasmtime 14 to be released on
September 20.
* Remove some cfgs
* Remove wasi-crypto CI
Do this by updating `crossbeam-epoch` and auditing this update of
crossbeam. The newer version of crossbeam additionally updates its
version of `memoffset`.
This commit adds `cargo vet` trust entries for any crate published by
BurntSushi, of which a good number are in our dependency graph. This
additionally updates the `bstr` crate to its latest version and updates
regex-related dependencies from other crates to avoid duplication of
versions.
* Remove deny.toml exception for wasm-coredump-builder
This isn't used any more so no need to continue to list this.
* Update Wasmtime's pretty_env_logger dependency
This removes a `deny.toml` exception for that crate, but `openvino-sys`
still depends on `pretty_env_logger 0.4.0` so a new exception is added
for that.
* Update criterion and clap dependencies
This commit started out by updating the `criterion` dependency to remove
an entry in `deny.toml`, but that ended up transitively requiring a
`clap` dependency upgrade from 3.x to 4.x because `criterion` uses
pieces of clap 4.x. Most of this commit is then dedicated to updating
clap 3.x to 4.x which was relatively simple, mostly renaming attributes
here and there.
* Update gimli-related dependencies
I originally wanted to remove the `indexmap` clause in `deny.toml` but
enough dependencies haven't updated from 1.9 to 2.0 that it wasn't
possible. In the meantime though this updates some various dependencies
to bring them to the latest and a few of them now use `indexmap` 2.0.
* Update deps to remove `windows-sys 0.45.0`
This involved updating tokio/mio and then providing new audits for new
crates. The tokio exemption was updated from its old version to the new
version and tokio remains un-audited.
* Update `syn` to 2.x.x
This required a bit of rewriting for the component-macro related bits
but otherwise was pretty straightforward. The `syn` 1.x.x track is still
present in the wasi-crypto tree at this time.
I've additionally added some trusted audits for my own publications of
`wasm-bindgen`
* Update bitflags to 2.x.x
This updates Wasmtime's dependency on the `bitflags` crate to the 2.x.x
track to keep it up-to-date.
* Update the cap-std family of crates
This bumps them all to the next major version to keep up with updates.
I've additionally added trusted entries for publishes of cap-std crates
from Dan.
There's still lingering references to rustix 0.37.x which will need to
get weeded out over time.
* Update memoffset dependency to latest
Avoids having two versions in our crate graph.
* Fix tests
* Update try_from for wiggle flags
* Fix build on AArch64 Linux
* Enable `event` for rustix on Windows too
* Cranelift: upgrade to regalloc2 0.9.2.
This pulls in bytecodealliance/regalloc2#152, which fixes a bug that is
reachable on RISC-V: when two different register classes have the same
stackslot size, the register allocation result might share a slot
between two different classes, which can result in moves between classes
that will cause a panic. The fix properly separates slots by class.
* cargo-vet update for regalloc2 0.9.2.
* Update wasm-tools dependencies
* Get tests passing after wasm-tools update
Mostly dealing with updates to `wasmparser`'s API.
* Update `cargo vet` for new crates
* Add `equivalent`, `hashbrown`, and `quote` to the list of trusted
authors. We already trust these authors for other crates.
* Pull in some upstream audits for various deps.
* I've audited the `pulldown-cmark` dependency upgrade myself.
We discussed this in today's Wasmtime meeting and the consensus was that
we trust each of these people to have a sufficient standard of care for
anything they release.
This reduces our estimated audit backlog by about 184 kLOC.
For the most part, the trust records I'm adding here are identical to
trust records that Mozilla is using. The fact that they've also decided
these publishers are trustworthy is reassuring additional evidence for
our decision. The exceptions and notable cases are as follows:
I've chosen to not trust three crates by these authors that Mozilla did
not trust. I suspect Mozilla simply doesn't use these crates or has
manually audited them, rather than there being any problem with the
crates themselves. But I've chosen to be conservative about what we
trust.
- autocfg: we only have an exception for an old version, and that
version is only used transitively by wasi-crypto.
- env_logger: Mozilla has audited some versions; we should update, or
add delta audits.
- thread_local: only used by tracing-subscriber which is only used in
dev-dependencies.
I've trusted one crate that Mozilla did not: libm, when published by
Amanieu. We're trusting libc when published by the same author, and libm
is a small extension of the same trust.
Recent versions of the toml crate have been published by epage so I
looked at in this process, but Mozilla only trusts the older versions
which were published by alexcrichton. They've been delta-auditing the
newer versions. I've chosen to follow their lead on this; Alex is a
trusted contributor to Wasmtime anyway.
* Update v8 and proc-macro2 dependencies
Gets them both compiling on the latest nightly so we can unpin the Rust
compiler version in OSS-Fuzz.
* Update nightly in CI
Alex Crichton
Lann
David Justice
Nick Fitzgerald
Philip Craig
Trevor Elliott
Andrew Brown
Dan Gohman
Iceber Gu
Rahul
Pat Hickey
Chris Fallin
Jamey Sharp