* Update some crates to align on latest `rustix`
This commit addresses some dependabot warnings showing up on the
Wasmtime repository by updating all dependencies to using the latest
`rustix` release.
* Debug CI
prtest:full
* Downgrade rustix to 0.38.14
Works around bytecodealliance/rustix#901
* Update to arbitrary 1.3.1
And use workspace dependencies for arbitrary.
* Prune cargo vet's supply-chain files
This is the mechanical changes made by running `cargo vet prune` which was
suggested to me when I ran `cargo vet`.
* Move the incoming_handler impl into http_impl
* Remove the incoming handler -- we need to use it as a guest export
* Start adding a test-programs test for the server side of wasi-http
* Progress towards running a server test
* Implement incoming-request-method
* Validate outparam value
* Initial incoming handler test
* Implement more of the incoming api
* Finish the incoming api implementations
* Initial cut at `wasmtime serve`
* fix warning
* wasmtime-cli: invoke ServeCommand, and add enough stuff to the linker to run trivial test
* fix warnings
* fix warnings
* argument parsing: allow --addr to specify sockaddr
* rustfmt
* sync wit definitions between wasmtime-wasi and wasmtime-wasi-http
* cargo vet: add an import config and wildcard audit for wasmtime-wmemcheck
* cargo vet: audit signal-hook-registry
* Remove duplicate add_to_linker calls for preview2 interfaces
prtest:full
* Add a method to finish outgoing responses
Co-authored-by: Adam Foltzer <acfoltzer@fastly.com>
Co-authored-by: Pat Hickey <phickey@fastly.com>
* Mark the result of the incoming_{request,response}_consume methods as own
* Explicit versions for http-body and http-body-util
* Explicit `serve` feature for the `wasmtime serve` command
* Move the spawn outside of the future returned by `ProxyHandler::call`
* Review feedback
---------
Co-authored-by: Trevor Elliott <telliott@fastly.com>
Co-authored-by: Adam Foltzer <acfoltzer@fastly.com>
* Implement the `tcp` interface of wasi-sockets.
Implement the `tcp`, `tcp-create-socket`, and `network` interfaces of
wasi-sockets.
* Minor cleanups.
* Update to the latest upstream wasi-sockets.
* Address review feedback.
* Handle zero-length reads and writes, and other cleanups.
* Fix compilation on macOS.
* Fix compilation on Windows.
* Update all the copies of wasi-socket wit files.
* Sync up more wit files.
* Fix the errno code for non-blocking `connect` on Windows.
prtest:full
* Tolerate `NOTCONN` errors when cleaning up with `shutdown`.
* Simplify the polling mechanism.
This requires an updated tokio for `Interest::ERROR`.
* Downgrade to tokio 1.29.1 for now.
* Move `tcp_state` out of the `Arc`.
* `accept` doesn't need a write lock.
* Remove `tcp_state`'s `RwLock`.
* Remove the implementation of wasi-crypto
This commit is a follow-up to the discussion on #6732. This removes
Wasmtime's implementation of the wasi-crypto proposal from in-tree along
with its various support in CI, configuration, etc. See the discussion
on #6732 for the full information but at a high level the main reasons
for removing the implementation at this time are:
* There is not currently an active maintainer of the Wasmtime
integration here for wasi-crypto.
* There are known issues with the code quality of the implementation
such as transmutes of guest-owned memory to `&'static mut [u8]` and
known unsafety in dependencies.
* The size and breadth of the dependency tree brings maintenance burden
and overhead to managing Wasmtime's dependency tree.
As mentioned on the issue this commit does not mean that Wasmtime
doesn't want to implement the wasi-crypto proposal. Instead the "tier 3"
status of wasi-crypto needs to be re-attained to be included back
in-tree, which would mean resolving the above issues.
Note that this commit is intentionally just after the 13.0.0 branch
point which means that this is slated for Wasmtime 14 to be released on
September 20.
* Remove some cfgs
* Remove wasi-crypto CI
This commit adds `cargo vet` trust entries for any crate published by
BurntSushi, of which a good number are in our dependency graph. This
additionally updates the `bstr` crate to its latest version and updates
regex-related dependencies from other crates to avoid duplication of
versions.
* Remove deny.toml exception for wasm-coredump-builder
This isn't used any more so no need to continue to list this.
* Update Wasmtime's pretty_env_logger dependency
This removes a `deny.toml` exception for that crate, but `openvino-sys`
still depends on `pretty_env_logger 0.4.0` so a new exception is added
for that.
* Update criterion and clap dependencies
This commit started out by updating the `criterion` dependency to remove
an entry in `deny.toml`, but that ended up transitively requiring a
`clap` dependency upgrade from 3.x to 4.x because `criterion` uses
pieces of clap 4.x. Most of this commit is then dedicated to updating
clap 3.x to 4.x which was relatively simple, mostly renaming attributes
here and there.
* Update gimli-related dependencies
I originally wanted to remove the `indexmap` clause in `deny.toml` but
enough dependencies haven't updated from 1.9 to 2.0 that it wasn't
possible. In the meantime though this updates some various dependencies
to bring them to the latest and a few of them now use `indexmap` 2.0.
* Update deps to remove `windows-sys 0.45.0`
This involved updating tokio/mio and then providing new audits for new
crates. The tokio exemption was updated from its old version to the new
version and tokio remains un-audited.
* Update `syn` to 2.x.x
This required a bit of rewriting for the component-macro related bits
but otherwise was pretty straightforward. The `syn` 1.x.x track is still
present in the wasi-crypto tree at this time.
I've additionally added some trusted audits for my own publications of
`wasm-bindgen`
* Update bitflags to 2.x.x
This updates Wasmtime's dependency on the `bitflags` crate to the 2.x.x
track to keep it up-to-date.
* Update the cap-std family of crates
This bumps them all to the next major version to keep up with updates.
I've additionally added trusted entries for publishes of cap-std crates
from Dan.
There's still lingering references to rustix 0.37.x which will need to
get weeded out over time.
* Update memoffset dependency to latest
Avoids having two versions in our crate graph.
* Fix tests
* Update try_from for wiggle flags
* Fix build on AArch64 Linux
* Enable `event` for rustix on Windows too
We discussed this in today's Wasmtime meeting and the consensus was that
we trust each of these people to have a sufficient standard of care for
anything they release.
This reduces our estimated audit backlog by about 184 kLOC.
For the most part, the trust records I'm adding here are identical to
trust records that Mozilla is using. The fact that they've also decided
these publishers are trustworthy is reassuring additional evidence for
our decision. The exceptions and notable cases are as follows:
I've chosen to not trust three crates by these authors that Mozilla did
not trust. I suspect Mozilla simply doesn't use these crates or has
manually audited them, rather than there being any problem with the
crates themselves. But I've chosen to be conservative about what we
trust.
- autocfg: we only have an exception for an old version, and that
version is only used transitively by wasi-crypto.
- env_logger: Mozilla has audited some versions; we should update, or
add delta audits.
- thread_local: only used by tracing-subscriber which is only used in
dev-dependencies.
I've trusted one crate that Mozilla did not: libm, when published by
Amanieu. We're trusting libc when published by the same author, and libm
is a small extension of the same trust.
Recent versions of the toml crate have been published by epage so I
looked at in this process, but Mozilla only trusts the older versions
which were published by alexcrichton. They've been delta-auditing the
newer versions. I've chosen to follow their lead on this; Alex is a
trusted contributor to Wasmtime anyway.
* Attempt versioned exports to facilitate having multiple versions in the same crate
* Modify approach to use `export_name` and `link_name`
* Only apply version to names in assembly and foreign item fns
* Attempt to handle the s390x case
* Fix alignment of backslashes in assembly file
* Pretend I understand the preprocessor
* Version symbols in `crates/runtime/src/helpers.c`
* Stop versioning `__jit_debug_register_code` because gdb relies on it and it is uses `weak` linkage
* Version symbol in `crates/fiber/src/windows.c`
* Consolidate `LitStr` creation in macro
* Add new crate to publish script and supply-chain config
* Fix order in supply chain config
* Set `audit-as-crates-io` to false
* Missing `versioned_link` for Windows
* Version strings used in debug
* Formatting
* Get rid of `versioned_str` and bring back `versioned_suffix`
---------
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
* Update v8 and proc-macro2 dependencies
Gets them both compiling on the latest nightly so we can unpin the Rust
compiler version in OSS-Fuzz.
* Update nightly in CI
* wasi-tests and wasi-http-tests no longer have their own workspace
* wasi-tests: fix warnings
* rewrite the test-programs build.rs to generate {package}_modules.rs and _components.rs
The style is cribbed from preview2-prototying repo, but I ended up
refactoring it a bit.
* better escaping should help with windows?
* long form cap-std-sync and tokio test suites
* convert wasi-http test
* fixes, comments
* apply cargo fmt to whole workspace
* bump test-programs and wasi-http-tests to all use common dependency versions
wit-bindgen 0.6.0 and wit-component 0.7.4
* add new audits
* cargo vet prune
* package and supply chain updates to fix vulnerabilities
h2 upgraded from 0.3.16 -> 0.3.19 to fix vulnerability
tempfile upgraded from 0.3.3 -> 0.3.5 to eliminate dep on vulnerable
remove_dir_all
* deny: temporarily allow duplicate wasm-encoder, wasmparser, wit-parser
prtest:full
* convert more dependencies to { workspace = true }
Alex asked me to do thsi for wit-component and wit-bindgen, and I found
a few more (cfg-if, tempfile, filecheck, anyhow...
I also reorganized the workspace dependencies section to make the ones
our team maintains more clearly separated from our external
dependencies.
* test-programs build: ensure that the user writes a #[test] for each module, component
* fix build of wasi-tests on windows
* misspelled macos
* mark wasi-tests crate test=false so we dont try building it natively...
* mark wasi-http-tests test=false as well
* try getting the cargo keys right
* just exclude wasi-tests and wasi-http-tests in run-tests.sh
* interesting paths fails on windows
* misspelling so nice i did it twice
* new cargo deny exception: ignore all of wit-bindgen's dependencies
* auto-import wildcard vets
* wasi: add the `wasi-testsuite` tests for wasi-common
As described [here], this uses the `prod/testsuite-base` branch in which
the tests are built as `.wasm` files.
[here]: https://github.com/WebAssembly/wasi-testsuite/#getting-started
* chore: update `walkdir` everywhere to its latest version
This is done in order to use it for `wasi_testsuite` testing.
* vet: extend `walkdir`'s exemption
* test: factor out `get_wasmtime_command`
This will be helpful for `wasi_testsuite` testing.
* test: use all `wasi-testsuite` test cases
This change alters the `wasi_testsuite` test to run all of the available
test cases in [wasi-testsuite]. This involved making the test runner a
bit more robust to the various shapes of JSON specifications in that
project. Unfortunately, the `wasi_testsuite` test fails some of the
cases, so I added a `WASI_COMMON_IGNORE_LIST` to avoid these
temporarily. (This may remind some of the Wasm testsuite ignore lists in
Cranelift; those relied on `build.rs` to create a `#[test]` for each
test case, which I felt is not yet needed here).
It's unclear to me why the tests are failing. It could be because:
- wasi-common has a bug
- wasi-testsuite overspecifies (or incorrectly specifies) a test
- the test runner incorrectly configures Wasmtime's CLI execution.
But this change makes it easier to resolve this. Remove the file from
`WASI_COMMON_IGNORE_LIST` and run `cargo test wasi_testsuite --
--nocapture`. The printed output will show the expected result, the
actual result, and a command to replicate the failure from the command
line.
[wasi-testsuite]: https://github.com/WebAssembly/wasi-testsuite
* review: add "shrinking" comment
This trims down the `[exemptions]` list ever-so-slightly by following
the suggestions of `cargo vet suggest` and updating a few crates across
some minor versions.
This updates to rustix 0.37.13, which contains some features we can use to
implement more features in wasi-common for the wasi-sockets API. This also
pulls in several other updates to avoid having multiple versions of rustix.
This does introduce multiple versions of windows-sys, as the errno and tokio
crates are currently using 0.45 while rustix and other dependencies have
updated to 0.48; PRs updating these are already in flight so this will
hopefully be resolved soon.
It also includes cap-std 1.0.14, which disables the use of `openat2` and
`statx` on Android, fixing a bug where some Android devices crash the
process when those syscalls are executed.
The `ring` crate needed to be exempted: it contains a large quantity of asm and native binary implementations of crypto primitives. It is a major undertaking to certify the safety of those implementations.
ring also pulled in the wasm-bindgen family of crates for its wasm32-unknown-unknown target, which this project will not be using. Because we don't care about that platform, I added exemptions for all of these crates, so we don't have to audit them.
The actual supply chain audits for rusttls, rustls-webpki, sct, and tokio-rustls were unremarkable. I also audited a small diff on wasm-bindgen-shared because it was trivial.
* add cargo-deny exception for duplicate versions of windows-sys
* cargo vetting for all new deps introduced by https://github.com/bytecodealliance/wasmtime/pull/5929
The audits are straightforward. The exemptions, as always, need to be justified:
* core-foundation, core-foundation-sys, security-framework, security-framework-sys: these are large crates which are FFI bindings to Mac OS frameworks. As such they contain tons of unsafe code to make these FFI calls and manage memory. These crates are too big to audit.
* schannel: same as the above, except this is a windows component, which I'm also unfamiliar with.
* openssl, openssl-sys: also large FFI bindings which are impractical to audit.
* futures-macro, futures-task: while not as complex as futures-util, these are beyond my personal understanding of futures to vet practically. I've asked Alex to look at auditing these, and he will after he returns from vacation next week.
* futures-util: 25kloc of code, over 149 instances of the substring "unsafe" (case insensitive), this is impractical to audit in the extreme.
* h2, http, httparse, hyper, mio, tokio: this so-called tokio/hyper family are very large and challenging to audit. Bobby Holley has indicated that he is working to get the AWS engineers who maintain these crates to publish their own audits, which we can then import. We expect to exempt these until those imports are available.
Aside from a few new features (notably automatic registry suggestions), this
release removes the need to import description for criteria that are not
directly used, and adds an explicit version to the cargo-vet instance.
* Update wasm-tools crates
Pulls in a new component binary format which should hopefully be the
last update for awhile.
* Update cargo vet configuration
* Import Wasmtime support from the `wit-bindgen` repo
This commit imports the `wit-bindgen-gen-host-wasmtime-rust` crate from
the `wit-bindgen` repository into the upstream Wasmtime repository. I've
chosen to not import the full history here since the crate is relatively
small and doesn't have a ton of complexity. While the history of the
crate is quite long the current iteration of the crate's history is
relatively short so there's not a ton of import there anyway. The
thinking is that this can now continue to evolve in-tree.
* Refactor `wasmtime-component-macro` a bit
Make room for a `wit_bindgen` macro to slot in.
* Add initial support for a `bindgen` macro
* Add tests for `wasmtime::component::bindgen!`
* Improve error forgetting `async` feature
* Add end-to-end tests for bindgen
* Add an audit of `unicase`
* Add a license to the test-helpers crate
* Add vet entry for `pulldown-cmark`
* Update publish script with new crate
* Try to fix publish script
* Update audits
* Update lock file
The main change here is that io-lifetimes 1.0 switches to use the I/O safety
feature in the standard library rather than providing its own copy.
This also updates to windows-sys 0.42.0 and rustix 0.36.
There were several issues with ISLE's existing error reporting
implementation.
- When using Miette for more readable error reports, it would panic if
errors were reported from multiple files in the same run.
- Miette is pretty heavy-weight for what we're doing, with a lot of
dependencies.
- The `Error::Errors` enum variant led to normalization steps in many
places, to avoid using that variant to represent a single error.
This commit:
- replaces Miette with codespan-reporting
- gets rid of a bunch of cargo-vet exemptions
- replaces the `Error::Errors` variant with a new `Errors` type
- removes source info from `Error` variants so they're easy to construct
- adds source info only when formatting `Errors`
- formats `Errors` with a custom `Debug` impl
- shares common code between ISLE's callers, islec and cranelift-codegen
- includes a source snippet even with fancy-errors disabled
I tried to make this a series of smaller commits but I couldn't find any
good split points; everything was too entangled with everything else.
* Make send and remove wrapper around WasiNnCtx·
This removes the wrapper around WasiNnCtx and no longer requires borrow_mut(). Once send/sync
changes in OpenVINO crate are merged in it will allow·use by frameworks that requires this trait.
* Bump openvino to compatible version.
* BackendExecutionContext should be Send and Sync
* Fix rust format issues.
* Update Cargo.lock for openvino
* Audit changes to openvino crates.
A minor update of a few other crates drops `semver` and `rustc_version`
from `Cargo.lock`. I've audited the deltas in versions for the other
crates here as well and they all look good.
* Update wasm-tools dependencies
This update brings in a number of features such as:
* The component model binary format and AST has been slightly adjusted
in a few locations. Names are dropped from parameters/results now in
the internal representation since they were not used anyway. At this
time the ability to bind a multi-return function has not been exposed.
* The `wasmparser` validator pass will now share allocations with prior
functions, providing what's probably a very minor speedup for Wasmtime
itself.
* The text format for many component-related tests now requires named
parameters.
* Some new relaxed-simd instructions are updated to be ignored.
I hope to have a follow-up to expose the multi-return ability to the
embedding API of components.
* Update audit information for new crates
* Update to cap-std 0.26.
This is primarily to pull in bytecodealliance/cap-std#271, the fix for #4936,
compilation on Rust nightly on Windows.
It also updates to rustix 0.35.10, to pull in bytecodealliance/rustix#403,
the fix for bytecodealliance/rustix#402, compilation on newer versions of
the libc crate, which changed a public function from `unsafe` to safe.
Fixes#4936.
* Update the system-interface audit for 0.23.
* Update the libc supply-chain config version.
* Optimize the WASI `random_get` implementation.
Use `StdRng` instead of the `OsRng` in the default implementation of
`random_get`. This uses a userspace CSPRNG, making `random_get` 3x faster
in simple benchmarks.
* Update cargo-vet audits for cap-std 0.25.3.
* Update all cap-std packages to 0.25.3.
I looked through some of our smaller dependencies to vet them and add an
audit for them. These were the ones that were all "obviously correct" to
me before I ran out of steam reviewing other crates.
* Implement the remaining socket-related WASI functions.
The original WASI specification included `sock_read`, `sock_write`, and
`shutdown`. Now that we have some sockets support, implement these
additional functions, to make it easier for people porting existing code
to WASI.
It's expected that this will all be subsumed by the wasi-sockets
proposal, but for now, this is a relatively small change which should
hopefully unblock people trying to use the current `accept` support.
* Update to system-interface 0.22, which has fixes for Windows.
* Implement strings in adapter modules
This commit is a hefty addition to Wasmtime's support for the component
model. This implements the final remaining type (in the current type
hierarchy) unimplemented in adapter module trampolines: strings. Strings
are the most complicated type to implement in adapter trampolines
because they are highly structured chunks of data in memory (according
to specific encodings). Additionally each lift/lower operation can
choose its own encoding for strings meaning that Wasmtime, the host, may
have to convert between any pairwise ordering of string encodings.
The `CanonicalABI.md` in the component-model repo in general specifies
all the fiddly bits of string encoding so there's not a ton of wiggle
room for Wasmtime to get creative. This PR largely "just" implements
that. The high-level architecture of this implementation is:
* Fused adapters are first identified to determine src/dst string
encodings. This statically fixes what transcoding operation is being
performed.
* The generated adapter will be responsible for managing calls to
`realloc` and performing bounds checks. The adapter itself does not
perform memory copies or validation of string contents, however.
Instead each transcoding operation is modeled as an imported function
into the adapter module. This means that the adapter module
dynamically, during compile time, determines what string transcoders
are needed. Note that an imported transcoder is not only parameterized
over the transcoding operation but additionally which memory is the
source and which is the destination.
* The imported core wasm functions are modeled as a new
`CoreDef::Transcoder` structure. These transcoders end up being small
Cranelift-compiled trampolines. The Cranelift-compiled trampoline will
load the actual base pointer of memory and add it to the relative
pointers passed as function arguments. This trampoline then calls a
transcoder "libcall" which enters Rust-defined functions for actual
transcoding operations.
* Each possible transcoding operation is implemented in Rust with a
unique name and a unique signature depending on the needs of the
transcoder. I've tried to document inline what each transcoder does.
This means that the `Module::translate_string` in adapter modules is by
far the largest translation method. The main reason for this is due to
the management around calling the imported transcoder functions in the
face of validating string pointer/lengths and performing the dance of
`realloc`-vs-transcode at the right time. I've tried to ensure that each
individual case in transcoding is documented well enough to understand
what's going on as well.
Additionally in this PR is a full implementation in the host for the
`latin1+utf16` encoding which means that both lifting and lowering host
strings now works with this encoding.
Currently the implementation of each transcoder function is likely far
from optimal. Where possible I've leaned on the standard library itself
and for latin1-related things I'm leaning on the `encoding_rs` crate. I
initially tried to implement everything with `encoding_rs` but was
unable to uniformly do so easily. For now I settled on trying to get a
known-correct (even in the face of endianness) implementation for all of
these transcoders. If an when performance becomes an issue it should be
possible to implement more optimized versions of each of these
transcoding operations.
Testing this commit has been somewhat difficult and my general plan,
like with the `(list T)` type, is to rely heavily on fuzzing to cover
the various cases here. In this PR though I've added a simple test that
pushes some statically known strings through all the pairs of encodings
between source and destination. I've attempted to pick "interesting"
strings that one way or another stress the various paths in each
transcoding operation to ideally get full branch coverage there.
Additionally a suite of "negative" tests have also been added to ensure
that validity of encoding is actually checked.
* Fix a temporarily commented out case
* Fix wasmtime-runtime tests
* Update deny.toml configuration
* Add `BSD-3-Clause` for the `encoding_rs` crate
* Remove some unused licenses
* Add an exemption for `encoding_rs` for now
* Split up the `translate_string` method
Move out all the closures and package up captured state into smaller
lists of arguments.
* Test out-of-bounds for zero-length strings
* Remove dependency on `more-asserts`
In my recent adventures to do a bit of gardening on our dependencies I
noticed that there's a new major version for the `more-asserts` crate.
Instead of updating to this though I've opted to instead remove the
dependency since I don't think we heavily lean on this crate and
otherwise one-off prints are probably sufficient to avoid the need for
pulling in a whole crate for this.
* Remove exemption for `more-asserts`
Alex Crichton
Pat Hickey
Nick Fitzgerald
Dan Gohman
Jamey Sharp
Adam Petro
Bobby Holley
Andrew Brown
Trevor Elliott
Matthew Tamayo-Rios
Bobby Holley