* Attempt to automatically configure release notes
This commit is an attempt to tackle #7068 by configuring the release
notes in Github Releases with the handwritten release notes from
`RELEASES.md`. The basic idea here is to split the markdown file on
`-----` delimiters and then find the one which matches the version being
released. Once one is found the `body` field of the API call to create
the release is configured.
* Update .github/actions/github-release/main.js
---------
Co-authored-by: Andrew Brown <andrew.brown@intel.com>
* ci: Upgrade QEMU to `8.1.1`
This adds support for RISC-V's Zcb extension that includes some
extra compressed instructions.
It also removes the current cpuinfo patch, that has been released
in 8.1
* wasmtime: Don't assert the exact faulting address for wasm traps
* Update Rust in CI to 1.72.0
* Update CI, tooling, and docs for MSRV
This commit codifies an MSRV policy for Wasmtime at "stable minus two"
meaning that the latest three releases of Rust will be supported. This
is enforced on CI with a full test suite job running on Linux x86_64
with the minimum supported Rust version. The full test suite will use
the latest stable version. A downside of this approach is that new
changes may break MSRV support on non-Linux or non-x86_64 platforms and
we won't know about it, but that's deemed a minor enough risk at this
time.
A minor fix is applied to Wasmtime's `Cargo.toml` to support Rust 1.70.0
instead of requiring Rust 1.71.0
* Fix installation of rust
* Scrape MSRV from Cargo.toml
* Cranelift is the same as Wasmtime's MSRV now, more words too
* Fix a typo
This commit enables the `component-model` feature at compile time for
our CI-originating binaries in the same manner that `all-arch` is
enabled for CI as well.
* Remove the implementation of wasi-crypto
This commit is a follow-up to the discussion on #6732. This removes
Wasmtime's implementation of the wasi-crypto proposal from in-tree along
with its various support in CI, configuration, etc. See the discussion
on #6732 for the full information but at a high level the main reasons
for removing the implementation at this time are:
* There is not currently an active maintainer of the Wasmtime
integration here for wasi-crypto.
* There are known issues with the code quality of the implementation
such as transmutes of guest-owned memory to `&'static mut [u8]` and
known unsafety in dependencies.
* The size and breadth of the dependency tree brings maintenance burden
and overhead to managing Wasmtime's dependency tree.
As mentioned on the issue this commit does not mean that Wasmtime
doesn't want to implement the wasi-crypto proposal. Instead the "tier 3"
status of wasi-crypto needs to be re-attained to be included back
in-tree, which would mean resolving the above issues.
Note that this commit is intentionally just after the 13.0.0 branch
point which means that this is slated for Wasmtime 14 to be released on
September 20.
* Remove some cfgs
* Remove wasi-crypto CI
Released today this updates from 1.69.0 to 1.71.0. Note that 1.70.0 was
skipped due to presumed codegen bugs for s390x and riscv64. These were
previously tested to work with 1.71.0, so it's time to confirm.
* Update v8 and proc-macro2 dependencies
Gets them both compiling on the latest nightly so we can unpin the Rust
compiler version in OSS-Fuzz.
* Update nightly in CI
* Run `cargo vet` on automated version bumps
As shown in the original CI of #6686 the recent changes for `cargo vet`
0.8.0 mean that the previous release process no longer works since it
requires changes to the lock file for `cargo vet`. This updates the
version bumping process to run `cargo vet` as part of the committed
changes which hopefully will get everything to succeed. To ensure that
the same version of `cargo vet` is used in both locations the
installation procedure was extracted into its own separate little action.
* Configure shell to run in
I originally thought this was the cause of PRs bouncing last night but
now that they're landing again I'm less certain that this is the cause.
Nevertheless this seems good to do regardless.
* wasi-common: need FileEntry to track the mode with which a file was opened
* add a test for read, write access modes in path_open
* a directory needs to report full set of rights in fdstat
implementations such as wasi-libc use these as a mask for any rights
they request for a new fd when invoking path_open
* preview2: mask file perms. and we really need to enforce them on stream creation
which will mean editing the wit.
* filesystem.wit: make {read, write, append}-via-stream fallible with an error-code
in order to fail appropriately when a file is not open for reading or
writing.
* filesystem.wit: update comments
* preview2 impls: fix error handling for {read,write,append}_via_stream
* wasi-common: normalize wrong access mode error on windows to badf
* remove kubkob from labeler
I appreciate his work on this stuff in the past but he hasnt been around
for a few years now
* preview 2 filesystem: allow stat on any opened fd
since file perms is now more accurately the file's access mode, a file
open for writing was having problems here.
in reality i dont think it makes sense to restrict this based on the
perms. if a file is opened, you should be able to stat it.
this fixes the host adapter's path_link test, which was broken by prior
changes. additionally, this fix lets the path_open_dirfd_not_dir test
pass.
* fix the directory base & inheriting rights
in order to work with wasi-testsuite, it needs to be possible to
path_open(dirfd, ".", ...) with the same rights reported in the
fdstat of that dirfd. When we report the Rights::all() set, both
FD_READ and FD_WRITE are set in the base rights, which results in
unix rejecting an openat2(dirfd, ".", O_RDWR) with EISDIR.
By not having the FD_READ and FD_WRITE rights present in the base
rights, the open syscall defaults to O_RDONLY, which is the only
access mode allowed for opening directories.
* path_open of a directory with read and write succeeds on windows
* Update Wasmtime for upcoming WIT changes
This PR integrates bytecodealliance/wasm-tools#1027 into Wasmtime. The
main changes here are:
* WIT syntax is updated with WebAssembly/component-model#193
* Generated bindings in the `bindgen!` macro have been updated to
reflect the new structure of WIT.
* The accepted component model binary format has been updated to account
for changes.
This PR disables wasi-http tests and the on-by-default feature because
the WIT syntax has been updated but the submodule containing the WITs
has not been updated yet so there's no way to get that building
temporarily. Once that's updated then this can be reenabled.
* Update wasmtime-wasi crate with new WIT
* Add wit-bindgen override for the updated version
* Officially disable wasi-http tests/building
* Move test-reactor WIT into the main WIT files
Don't store duplicates with the rest of the WASI WIT files we have.
* Remove adapter's copy of WIT files
* Disable default features for wit-bindgen
* Plumb disabling wasi-http tests a bit more
* Fix reactor tests and adapter build
* Remove no-longer-needed feature
* Update adapter verification script
* Back out some wasi-http hacks
* Update vet and some dependency sources
* Move where wit-bindgen comes from
Make it a more "official" location which is also less likely to be
accidentally deleted in the future.
* Don't document wasi-http-tests
* Add adapter build to dependency for the entire CI build
* Build adapter on PR CI when it changes
* Update adapter verification for new stdin/stdout/stderr
* Update reactor build script
* Reduce some duplication with some shared variables
* Rename adapters to `wasi_snapshot_preview1.{reactor,command}.wasm`
* Fix upload pattern
* winch(fuzz): Refactor Winch's fuzzing
This change is a follow-up to the discussion in
https://github.com/bytecodealliance/wasmtime/pull/6281.
The most notable characteristic of this change is that it enables
`winch` by default in the fuzzers. If compilation time is a big enough
concern I can add the cargo feature back. I opted to enable `winch` by
default for several reasons:
* It substantially reduces the `cfg` complexity -- at first I thought
I had covered all the places in which a `cfg` check would be needed,
but then I realized that I missed the Cranelift specific compiler
flags.
* It's the fastest route to enable winch by default in the fuzzers,
which we want to do eventually -- the only change we'd need at that
point would be to get rid of the winch-specific environment variable.
* We can get rid of the winch-specific checks in CI for fuzzing
* Implement Arbitraty for CompilerStrategy
Unconditionally return `Cranelift` for the `Arbitrary` implementation of
`CompilerStrategy`. This ensures that `Cranelift` is used as the
compiler for all the targets unless explicitly requested otherwise. As
of this change, only the differential target overrides the
`CompilerStrategy`
this is causing a link error because pulldown_cmark is available both in
the rust_wasi_markdown_parser deps directory and, now, in the root of
the project as well.
Currently all the jobs in GitHub Actions are configured to cancel the
entire run if any of them fail. This is similar to `fail-fast: true` but
that can't be used because the jobs aren't all part of the same matrix.
The original goal for this was to save CI time on the merge queue where
if something failed it should get out of the merge queue quickly instead
of waiting for all other jobs to succeed and then report the one quick
failure.
This seems to run into an issue, though, where if a Windows builder
fails then running cancellation will instead mark the Windows builder as
cancelled instead of failed. This can be confusing for test failures
because nothing looks obvious and each log needs to be manually pored
over.
To help address this the commit here disables auto-cancellation of the
"test" matrix of jobs, re-enabling `fail-fast: true` at the same time.
This way test if a test job fails it won't actually cancel everything
else so everything else will run to completion, but given that tests are
typically the longest part of the build that's hopefully ok.
* Make Wasmtime compatible with Stacked Borrows in MIRI
The fact that Wasmtime executes correctly under Tree Borrows but not
Stacked Borrows is a bit suspect and given what I've since learned about
the aliasing models I wanted to give it a stab to get things working
with Stacked Borrows. It turns out that this wasn't all that difficult,
but required two underlying changes:
* First the implementation of `Instance::vmctx` is now specially crafted
in an intentional way to preserve the provenance of the returned
pointer. This way all `&Instance` pointers will return a `VMContext`
pointer with the same provenance and acquiring the pointer won't
accidentally invalidate all prior pointers.
* Second the conversion from `VMContext` to `Instance` has been updated
to work with provenance and such. Previously the conversion looked
like `&mut VMContext -> &mut Instance`, but I think this didn't play
well with MIRI because `&mut VMContext` has no provenance over any
data since it's zero-sized. Instead now the conversion is from `*mut
VMContext` to `&mut Instance` where we know that `*mut VMContext` has
provenance over the entire instance allocation. This shuffled a fair
bit around to handle the new closure-based API to prevent escaping
pointers, but otherwise no major change other than the structure and
the types in play.
This commit additionally picks up a dependency on the `sptr` crate which
is a crate for prototyping strict-provenance APIs in Rust. This is I
believe intended to be upstreamed into Rust one day (it's in the
standard library as a Nightly-only API right now) but in the meantime
this is a stable alternative.
* Clean up manual `unsafe impl Send` impls
This commit adds a new wrapper type `SendSyncPtr<T>` which automatically
impls the `Send` and `Sync` traits based on the `T` type contained.
Otherwise it works similarly to `NonNull<T>`. This helps clean up a
number of manual annotations of `unsafe impl {Send,Sync} for ...`
throughout the runtime.
* Remove pointer-to-integer casts with tables
In an effort to enable MIRI's "strict provenance" mode this commit
removes the integer-to-pointer casts in the runtime `Table`
implementation for Wasmtime. Most of the bits were already there to
track all this, so this commit plumbed around the various pointer types
and with the help of the `sptr` crate preserves the provenance of all
related pointers.
* Remove integer-to-pointer casts in CoW management
The `MemoryImageSlot` type stored a `base: usize` field mostly because I
was too lazy to have a `Send`/`Sync` type as a pointer, so this commit
updates it to use `SendSyncPtr<u8>` and then plumbs the pointer-ness
throughout the implementation. This removes all integer-to-pointer casts
and has pointers stores as actual pointers when they're at rest.
* Remove pointer-to-integer casts in "raw" representations
This commit changes the "raw" representation of `Func` and `ExternRef`
to a `*mut c_void` instead of the previous `usize`. This is done to
satisfy MIRI's requirements with strict provenance, properly marking the
intermediate value as a pointer rather than round-tripping through
integers.
* Minor remaining cleanups
* Switch to Stacked Borrows for MIRI on CI
Additionally enable the strict-provenance features to force warnings
emitted today to become errors.
* Fix a typo
* Replace a negative offset with `sub`
* Comment the sentinel value
* Use NonNull::dangling
In #6338 that PR is bouncing on CI due to Windows running out of disk
space. Locally a `./ci/run-tests.sh` compile produces a ~13G build
directory. Testing on CI it looks like Windows builders have ~13G of
free space on them for GitHub Actions. I think something in that PR has
tipped us just over the edge of requiring too much space, although I'm
not sure exactly what.
To address the issue this commit disables DWARF debug information
entirely on all builders on CI. Not only should this save a sliver of
compile time but this reduces a local build directory to ~4.7G, over a
50% reduction. That should keep us fitting in CI for more time to come
hopefully.
* Run some tests in MIRI on CI
This commit is an implementation of getting at least chunks of Wasmtime
to run in MIRI on CI. The full test suite is not possible to run in MIRI
because MIRI cannot run Cranelift-produced code at runtime (aka it
doesn't support JITs). Running MIRI, however, is still quite valuable if
we can manage it because it would have trivially detected
GHSA-ch89-5g45-qwc7, our most recent security advisory. The goal of this
PR is to select a subset of the test suite to execute in CI under MIRI
and boost our confidence in the copious amount of `unsafe` code in
Wasmtime's runtime.
Under MIRI's default settings, which is to use the [Stacked
Borrows][stacked] model, much of the code in `Instance` and `VMContext`
is considered invalid. Under the optional [Tree Borrows][tree] model,
however, this same code is accepted. After some [extremely helpful
discussion][discuss] on the Rust Zulip my current conclusion is that
what we're doing is not fundamentally un-sound but we need to model it
in a different way. This PR, however, uses the Tree Borrows model for
MIRI to get something onto CI sooner rather than later, and I hope to
follow this up with something that passed Stacked Borrows. Additionally
that'll hopefully make this diff smaller and easier to digest.
Given all that, the end result of this PR is to get 131 separate unit
tests executing on CI. These unit tests largely exercise the embedding
API where wasm function compilation is not involved. Some tests compile
wasm functions but don't run them, but compiling wasm through Cranelift
in MIRI is so slow that it doesn't seem worth it at this time. This does
mean that there's a pretty big hole in MIRI's test coverage, but that's
to be expected as we're a JIT compiler after all.
To get tests working in MIRI this PR uses a number of strategies:
* When platform-specific code is involved there's now `#[cfg(miri)]` for
MIRI's version. For example there's a custom-built "mmap" just for
MIRI now. Many of these are simple noops, some are `unimplemented!()`
as they shouldn't be reached, and some are slightly nontrivial
implementations such as mmaps and trap handling (for native-to-native
function calls).
* Many test modules are simply excluded via `#![cfg(not(miri))]` at the
top of the file. This excludes the entire module's worth of tests from
MIRI. Other modules have `#[cfg_attr(miri, ignore)]` annotations to
ignore tests by default on MIRI. The latter form is used in modules
where some tests work and some don't. This means that all future test
additions will need to be effectively annotated whether they work in
MIRI or not. My hope though is that there's enough precedent in the
test suite of what to do to not cause too much burden.
* A number of locations are fixed with respect to MIRI's analysis. For
example `ComponentInstance`, the component equivalent of
`wasmtime_runtime::Instance`, was actually left out from the fix for
the CVE by accident. MIRI dutifully highlighted the issues here and
I've fixed them locally. Some locations fixed for MIRI are changed to
something that looks similar but is subtly different. For example
retaining items in a `Store<T>` is now done with a Wasmtime-specific
`StoreBox<T>` type. This is because, with MIRI's analyses, moving a
`Box<T>` invalidates all pointers derived from this `Box<T>`. We don't
want these semantics, so we effectively have a custom `Box<T>` to suit
our needs in this regard.
* Some default configuration is different under MIRI. For example most
linear memories are dynamic with no guards and no space reserved for
growth. Settings such as parallel compilation are disabled. These are
applied to make MIRI "work by default" in more places ideally. Some
tests which perform N iterations of something perform fewer iterations
on MIRI to not take quite so long.
This PR is not intended to be a one-and-done-we-never-look-at-it-again
kind of thing. Instead this is intended to lay the groundwork to
continuously run MIRI in CI to catch any soundness issues. This feels,
to me, overdue given the amount of `unsafe` code inside of Wasmtime. My
hope is that over time we can figure out how to run Wasm in MIRI but
that may take quite some time. Regardless this will be adding nontrivial
maintenance work to contributors to Wasmtime. MIRI will be run on CI for
merges, MIRI will have test failures when everything else passes,
MIRI's errors will need to be deciphered by those who have probably
never run MIRI before, things like that. Despite all this to me it seems
worth the cost at this time. Just getting this running caught two
possible soundness bugs in the component implementation that could have
had a real-world impact in the future!
[stacked]: https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md
[tree]: https://perso.crans.org/vanille/treebor/
[discuss]: https://rust-lang.zulipchat.com/#narrow/stream/269128-miri/topic/Tree.20vs.20Stacked.20Borrows.20.26.20a.20debugging.20question
* Update alignment comment
Previously an `event` filter was applied to lookup the merge queue's
github run ID but this filter doesn't work after #6288. The filter isn't
strictly necessary, though, so remove it.
Use this to prime caches used by PRs to `main` and additionally the
merge queue used to merge into `main`.
While I'm here additionally update the trigger for merge-queue-based PRs
to use `merge_group:` now that it's been fixed.
Closes#6285
It seems that this fell through given that the incremental cache is
behind a cargo feature. I noticed this while building
`cranelift-codegen` via `cargo build --all-features`.
I decided to add a check in CI to hopefully prevent this in the future,
but I'm happy to remove it / update it if there's a better way or another way.
* winch(fuzz): Initial support for differential fuzzing
This commit introduces initial support for differential fuzzing for Winch. In
order to fuzz winch, this change introduces the `winch` cargo feature. When the
`winch` cargo feature is enabled the differential fuzz target uses `wasmi` as
the differential engine and `wasm-smith` and `single-inst` as the module sources.
The intention behind this change is to have a *local* approach for fuzzing and
verifying programs generated by Winch and to have an initial implementation that
will allow us to eventually enable this change by default. Currently it's not
worth it to enable this change by default given all the filtering that needs to
happen to ensure that the generated modules are supported by Winch.
It's worth noting that the Wasm filtering code will be temporary, until Winch
reaches feature parity in terms of Wasm operators.
* Check build targets with the `winch` feature flag
* Rename fuzz target feature to `fuzz-winch`
* cranelift-native: Move riscv to separate module
* cranelift-native: Read /proc/cpuinfo to parse RISC-V extensions
* ci: Add QEMU cpuinfo emulation patch
This patch emulates the /proc/cpuinfo interface for RISC-V. This
allows us to do feature detection for the RISC-V backend.
It has been queued for QEMU 8.1 so we should remove it as soon as
that is available.
* ci: Enable QEMU RISC-V extensions
* cranelift-native: Cleanup ISA string parsing
Co-Authored-By: Jamey Sharp <jsharp@fastly.com>
* cranelift-native: Rework `/proc/cpuinfo` parsing
Co-Authored-By: Jamey Sharp <jsharp@fastly.com>
---------
Co-authored-by: Jamey Sharp <jsharp@fastly.com>
In #6089, I accidentally left in a change that pegged the
`install-openvino-action` to a commit instead of the latest released
version. This change uses the latest released version.
(Run all CI actions: prtest:full)
* ci: unpin the wasi-nn tasks from an older Ubuntu
Previously, OpenVINO's lack of APT packages for Ubuntu 22.04 (`jammy`)
prevented us from upgrading the GitHub runner to use `ubuntu-latest`. I
updated the `install-openvino-action` to substitute in the `focal`
packages in this case (this is what the OpenVINO team considers the fix)
so this pin should no longer be necessary. Fixes#5408.
(Run all CI actions: prtest:full)
* vet: audit the openvino version bump
Overall, I'm just trying to make these bits of documentation reflect our
process as it stands today. There are some specific changes I want to
draw attention to though.
Asking new contributors to pick a reviewer is a waste of time for two
reasons: Only people with write access to the repository are allowed to
pick reviewers, and new contributors have no idea who would be a good
reviewer for their PR anyway. So I'm deleting all mention of that. We
now auto-assign reviewers instead.
By the time someone is opening a PR, asking them to open an issue just
makes extra work for everyone. They've already picked an approach
without discussing it; we might as well look at what they did. We may
then have to ask them to take a different approach, but at that point,
asking them to open an issue won't save them any effort.
I removed mention of tests from the pull request template. There are
many things we'd like to see in a PR, and we may have to ask for them
during review if the contributor doesn't follow our development process
documentation. But I think the only crucial information for starting a
review is the two questions I'm leaving in the template: why do you want
this, and where can I find more context?
The code of conduct link still had the branch name as `master`, which is
a hint at how long it's been since anyone reviewed it.
* Integrate experimental HTTP into wasmtime.
* Reset Cargo.lock
* Switch to bail!, plumb options partially.
* Implement timeouts.
* Remove generated files & wasm, add Makefile
* Remove generated code textfile
* Update crates/wasi-http/Cargo.toml
Co-authored-by: Eduardo de Moura Rodrigues <16357187+eduardomourar@users.noreply.github.com>
* Update crates/wasi-http/Cargo.toml
Co-authored-by: Eduardo de Moura Rodrigues <16357187+eduardomourar@users.noreply.github.com>
* Extract streams from request/response.
* Fix read for len < buffer length.
* Formatting.
* types impl: swap todos for traps
* streams_impl: idioms, and swap todos for traps
* component impl: idioms, swap all unwraps for traps, swap all todos for traps
* http impl: idiom
* Remove an unnecessary mut.
* Remove an unsupported function.
* Switch to the tokio runtime for the HTTP request.
* Add a rust example.
* Update to latest wit definition
* Remove example code.
* wip: start writing a http test...
* finish writing the outbound request example
havent executed it yet
* better debug output
* wasi-http: some stubs required for rust rewrite of the example
* add wasi_http tests to test-programs
* CI: run the http tests
* Fix some warnings.
* bump new deps to latest releases (#3)
* Add tests for wasi-http to test-programs (#2)
* wip: start writing a http test...
* finish writing the outbound request example
havent executed it yet
* better debug output
* wasi-http: some stubs required for rust rewrite of the example
* add wasi_http tests to test-programs
* CI: run the http tests
* bump new deps to latest releases
h2 0.3.16
http 0.2.9
mio 0.8.6
openssl 0.10.48
openssl-sys 0.9.83
tokio 1.26.0
---------
Co-authored-by: Brendan Burns <bburns@microsoft.com>
* Update crates/test-programs/tests/http_tests/runtime/wasi_http_tests.rs
* Update crates/test-programs/tests/http_tests/runtime/wasi_http_tests.rs
* Update crates/test-programs/tests/http_tests/runtime/wasi_http_tests.rs
* wasi-http: fix cargo.toml file and publish script to work together (#4)
unfortunately, the publish script doesn't use a proper toml parser (in
order to not have any dependencies), so the whitespace has to be the
trivial expected case.
then, add wasi-http to the list of crates to publish.
* Update crates/test-programs/build.rs
* Switch to rustls
* Cleanups.
* Merge switch to rustls.
* Formatting
* Remove libssl install
* Fix tests.
* Rename wasi-http -> wasmtime-wasi-http
* prtest:full
Conditionalize TLS on riscv64gc.
* prtest:full
Fix formatting, also disable tls on s390x
* prtest:full
Add a path parameter to wit-bindgen, remove symlink.
* prtest:full
Fix tests for places where SSL isn't supported.
* Update crates/wasi-http/Cargo.toml
---------
Co-authored-by: Eduardo de Moura Rodrigues <16357187+eduardomourar@users.noreply.github.com>
Co-authored-by: Pat Hickey <phickey@fastly.com>
Co-authored-by: Pat Hickey <pat@moreproductive.org>
Alex Crichton
Afonso Bordado
Eduardo de Moura Rodrigues
ssunkin-fastly
Chris Fallin
Bobby Holley
Pat Hickey
Trevor Elliott
Saúl Cabrera
Nick Fitzgerald
Jamey Sharp
Andrew Brown
Brendan Burns