* mpk: enable MPK if available in CI
After discussing several options for testing MPK in CI, this stopgap
measure at least provides some coverage. Other options include
maintaining a separate MPK-enabled CI runner (configuration is not
transparent, hard to maintain) or running the MPK-enabled tests in a
system-mode QEMU VM (tricky to get right, also hard to maintain). For
now, those of us at the Cranelift meeting agreed this at least gets some
CI testing for the MPK bits, which shouldn't be changing too often.
Since not all GitHub runners have MPK available, we only set the
`WASMTIME_TEST_FORCE_MPK` environment variable when it is. This change
also emits a warning to the GitHub logs in either case for easier
troubleshooting (e.g., to attempt to provide context if MPK logic breaks
and is only found in a later CI run).
prtest:full
* review: create a separate matrix entry
As requested by @alexcrichton, this change limits the MPK testing (and
the associated warnings) to a new matrix target: `Test Linux x86_64 with
MPK`.
* Add `runtime` feature to `wasmtime` crate
This feature can be disabled to build `wasmtime` only for compilation.
This can be useful when cross-compiling, especially on a target that
can't run wasmtime itself (e.g. `wasm32`).
* prtest:full
* don't round pages without runtime feature
* fix async assertions
* move profiling into runtime
* enable runtime for wasmtime-wasi
* enable runtime for c-api
* fix build_artifacts in non-cache case
* fix miri extensions
* enable runtime for wast
* enable runtime for explorer
* support cranelift all-arch on wasm32
* add doc links for `WeakEngine`
* simplify lib runtime cfgs
* move limits and resources to runtime
* move stack to runtime
* move coredump and debug to runtime
* add runtime to coredump and async features
* add wasm32 build job
* combine engine modules
* single compile mod
* remove allow for macro paths
* add comments
Try not passing `-y -u` to `pacman` to avoid full system updates.
Currently full system updates might update the `msys2-runtime` package
before actually updating the package we requested, meaning that this
might not actually update anything given an update. This is what's
currently happening on CI which is breaking due to an update of gcc not
actually updating gcc. I'm mostly reading the invocation in
rust-lang/rust CI and seeing that it doesn't pass `-y -u` and hopeing
that by copying that here things might work.
prtest:full
* windows-sys 0.52
Not locally tested due to C dependencies which I don't have the tooling for.
The intent is to use GH's CI.
* MEMORYMAPPEDVIEW_HANDLE -> MEMORY_MAPPED_VIEW_ADDRESS
* .Value instead of cast
* fmt
* Remove cast to MEMORY_MAPPED_VIEW_ADDRESS with struct creation
* Cast ptr from *mut u8 from *mut void
* Remove CI changes which were supposed to be on a distinct branch
* Update binutils when the target is x86_64-pc-windows-gnu
* Escape paths in CI
* Update .github/workflows/main.yml
Co-authored-by: Chris Denton <christophersdenton@gmail.com>
* Correct type from p2sh to pwsh
* Use | to correct YAML parsing error
* Fix msys2's installation of MinGW
* Finally fix msys2 handling
---------
Co-authored-by: Chris Denton <christophersdenton@gmail.com>
* wasi-nn: add test programs
This change adds new test programs for wasi-nn in a way fits in with the
existing WASI test infrastructure. The code is not new, though: this
reuses the wasi-nn `examples`, which are currently used by the
`run-wasi-nn-example.sh` CI script. Eventually the examples will be
removed in favor of these tests.
Because wasi-nn's component model support is still in flight, this
change also skips the generation of components for `nn_`-prefixed tests.
* wasi-nn: add `testing` module
This testing-only module has code (i.e., `check_test!`) to check whether
OpenVINO and some test artifacts are available. The test artifacts are
downloaded and cached if not present, expecting `curl` to be present on
the command line (as discussed in the previous version of this, #6895).
* wasi-nn: run `nn_*` test programs as integration tests
Following the pattern of other WASI crates, this change adds the
necessary infrastructure to run the `nn_*` files in
`crates/test-programs` (built by `test-program-artifacts`). These tests
are only run when two sets of conditions are true:
- statically: we only run these tests where we expect OpenVINO to be
easy to install and run (e.g., the `cfg_attr` parts)
- dynamically: we also only run these tests when the OpenVINO libraries
can be located and the model artifacts can be downloaded
* ci: install OpenVINO for running wasi-nn tests
prtest:full
* vet: certify the `wasi-nn` crate
* ci: remove wasi-nn test script
* ci: log CPU details when testing
When testing, there are certain CPU-dependent features that influence
Cranelift's codegen (e.g., availability of AVX512 instructions). This
additional CI step logs the current CPU information to aid in
troubleshooting, such as the MPK-related troubleshooting over in #7445.
Also, if we let this run in CI for a while, we may be able to run
queries on the logs to determine how often jobs run on servers with
certain features enabled.
prtest:full
* Add Windows variant of 'lscpu'
* Add MacOS variant of 'lscpu'
* Configure Rust lints at the workspace level
This commit adds necessary configuration knobs to have lints configured
at the workspace level in Wasmtime rather than the crate level. This
uses a feature of Cargo first released with 1.74.0 (last week) of the
`[workspace.lints]` table. This should help create a more consistent set
of lints applied across all crates in our workspace in addition to
possibly running select clippy lints on CI as well.
* Move `unused_extern_crates` to the workspace level
This commit configures a `deny` lint level for the
`unused_extern_crates` lint to the workspace level rather than the
previous configuration at the individual crate level.
* Move `trivial_numeric_casts` to workspace level
* Change workspace lint levels to `warn`
CI will ensure that these don't get checked into the codebase and
otherwise provide fewer speed bumps for in-process development.
* Move `unstable_features` lint to workspace level
* Move `unused_import_braces` lint to workspace level
* Start running Clippy on CI
This commit configures our CI to run `cargo clippy --workspace` for all
merged PRs. Historically this hasn't been all the feasible due to the
amount of configuration required to control the number of warnings on
CI, but with Cargo's new `[lint]` table it's possible to have a
one-liner to silence all lints from Clippy by default. This commit by
default sets the `all` lint in Clippy to `allow` to by-default disable
warnings from Clippy. The goal of this PR is to enable selective access
to Clippy lints for Wasmtime on CI.
* Selectively enable `clippy::cast_sign_loss`
This would have fixed#7558 so try to head off future issues with that
by warning against this situation in a few crates. This lint is still
quite noisy though for Cranelift for example so it's not worthwhile at
this time to enable it for the whole workspace.
* Fix CI error
prtest:full
On PRs it's helpful to see full failures if they arise, but for non-PRs
they're typically in the background so continuing after a job fails
isn't going to help much.
* Attempt to automatically configure release notes
This commit is an attempt to tackle #7068 by configuring the release
notes in Github Releases with the handwritten release notes from
`RELEASES.md`. The basic idea here is to split the markdown file on
`-----` delimiters and then find the one which matches the version being
released. Once one is found the `body` field of the API call to create
the release is configured.
* Update .github/actions/github-release/main.js
Co-authored-by: Andrew Brown <andrew.brown@intel.com>
---------
Co-authored-by: Andrew Brown <andrew.brown@intel.com>
* Update some crates to align on latest `rustix`
This commit addresses some dependabot warnings showing up on the
Wasmtime repository by updating all dependencies to using the latest
`rustix` release.
* Debug CI
prtest:full
* Downgrade rustix to 0.38.14
Works around bytecodealliance/rustix#901
* riscv64: Extend distance trampolines can jump
Use a PIC-friendly set of instructions to enable destination of the
trampoline to be more than 4k away from the tail call site of the
trampoline itself.
* Build "min" artifacts on CI
This commit updates the binary artifacts produced by CI to include "min"
builds where all default features are disabled. Additionally all the
stops are pulled in terms of build flags, nightly versions, etc, to get
a build that is as small as possible without actual source code changes.
This effectively codifies the instructions in #7282 into an easily
downloadable artifact.
No new tarballs are created for github releases but instead tarballs
that previously had a `wasmtime` executable for example now have a
`wasmtime-min` executable. Furthermore the C API which previously had
`libwasmtime.so` for example now has `libwasmtime-min.so`. The intention
is that the minimum-size artifacts are handy for determining a rough
size of Wasmtime but they're not so important that it seems worthwhile
to dedicate entire release entries for.
CI is refactored to support these minimum builds with separate builders.
This means that a single tarball produced as a final result is actually
two separate tarballs merged together, one for the normal build we do
today plus a new "min" tarball produced on the new "min" builders.
Various scripts and CI organization has been adjusted accordingly.
While here I went ahead and enabled `panic=abort` and debuginfo
stripping in our current release artifacts. While this doesn't affect a
whole lot it's less to upload to GitHub Actions all the time.
* Fix Windows unzip
* Move `wasmtime explore` behind a Cargo feature
Enable this Cargo feature by default, but enable building the CLI
without the `explore` subcommand.
* Move the `wast` subcommand behind a Cargo feature
* Move support for `wat` behind a CLI feature
This was already conditional in crates such as `wasmtime` and this makes
it an optional dependency of the CLI as well.
* Move CLI cache support behind a Cargo feature
Additionally refactor `wasmtime-cli-flags` to not unconditionally pull
in caching support by removing its `default` feature and appropriately
enabling it from the CLI.
* Move `rayon` behind an optional feature
* Move `http-body-util` dependency behind `serve` feature
* Add a Cargo feature for compiling out log statements
This sets the static features of `log` and `tracing` to statically
remove all log statements from the binary to cut down on binary size.
* Move logging support behind a Cargo feature
Enables statically removing logging support in addition to the previous
compiling out log statements themselves.
* Move demangling support behind a Cargo feature
* Enable building the CLI without cranelift
Compile out the `compile` subcommand for example.
* Gate all profiling support behind one feature flag
This commit moves vtune/jitdump support behind a single `profiling`
feature flag that additionally includes the guest profiler dependencies
now too.
* Move support for core dumps behind a feature flag
* Move addr2line behind a feature
* Fix rebase
* Document cargo features and a minimal build
* Tidy up the source a bit
* Rename compile-out-logging
* Document disabling logging
* Note the host architecture as well
* Fix tests
* Fix tests
* Mention debuginfo stripping
* Fix CI configuration for checking features
* Fix book tests
* Update lock file after rebase
* Enable coredump feature by default
* Update MSRV to 1.71.0
With the 1.73.0 release today update our MSRV to a one-higher version.
This will cause 1.73.0 to be tested in CI.
* Fix a warning on windows
* Don't store tables as `&'static mut`
Use `SendSyncPtr` instead (a small wrapper around `NonNull`) which
appeases MIRI.
* Refactor the test-programs test suite
This commit is a large refactoring that reorganizes `test-programs` and
how we tests wasms in Wasmtime. Often writing tests requires complicated
interactions with the guest which can't be done via hand-written `*.wat`
syntax and requires a compiler to get engaged. For this purpose Wasmtime
currently has the `crates/test-programs/*` test suite which builds files
from source and then runs the tests. This has been somewhat cumbersome
in the past though and it's not been easy to extend this over time, so
this commit attempts to address this.
The scheme implemented in this PR looks like:
* All wasm test programs live in `crates/test-programs/src/bin/*.rs`.
All of them, no exceptions.
* Wasm tests have shared support located at
`crates/test-programs/src/lib.rs` and its submodules, such as bindings
generation for WASI.
* Wasm tests are built by a new `crates/test-programs/artifacts` crate.
This crate compiles modules and additionally creates components for
all test programs. The crate itself only records the path to these
outputs and a small amount of testing support, but otherwise doesn't
interact with `wasmtime`-the-crate itself.
* All tests in `crates/test-programs/tests/*.rs` have moved. For example
wasi-http tests now live at `crates/wasi-http/tests/*.rs`. Legacy
tests of wasi-common now live at `crates/wasi-common/tests/*.rs`.
Modern tests for preview2 live at `crates/wasi/tests/*.rs`.
* Wasm tests are bucketed based on their filename prefix. For example
`preview1_*` is tested in wasi-common and wasmtime-wasi. The
`preview2_*` prefix is only tested with wasmtime-wasi, however.
* A new `cli_*` prefix is used to execute tests as part of
`tests/all/main.rs`. This is a new submodule in
`tests/all/cli_tests.rs` which executes these components on the
command line. Many old "command" tests were migrated here.
* Helper macros are generated to assert that a test suite is run in its
entirety. This way if a `preview1_*` test is added it's asserted to
get added to both wasi-common and wasmtime-wasi in the various modes
they run tests.
Overall this moved a number of tests around and refactored some edges of
the tests, but this should not lose any tests (except one that wasn't
actually testing anything). Additionally the hope is that it's much
easier to add tests in the future. The process is to add a new file in
`crates/test-programs/src/bin/*.rs` named appropriately. For example a
preview2 executable is `preview2_*` and a CLI tests is `cli_*`. When
building the test suite an error is generated in the appropriate module
then of "please write a test here", and then a test is written in the
same manner as the other tests in the module.
* Remove no-longer-needed fetches
prtest:full
* I'm worried wasi is running low on semicolons
* Add the WASI target in all CI actions
* Add unknown-unknown target on all CI builders too
* Fix building test artifacts under miri
Need to avoid wrappers for these cross-compiled targets
* Break circular dependency for packaging
Don't use the workspace dep for `wasmtime-wasi` since it injects a
version, instead use a `path = '..'` dependency to fool Cargo into
dropping the dependency during the package phase.
* Fix some merge conflicts with tests
* Fix rebase for new tests
* Remove stray comment
* Fix some flaky tests
* Fix network tests in synchronous mode
This commit is an attempt to fix some networking tests in synchronous
mode in our test suite. Currently networking tests don't actually run in
synchronous mode on CI which is why no failures have been surfaced yet,
but the refactoring in #7182 is going to start doing this.
Currently the `udp_sample_application.rs` test blocks infinitely in
synchronous mode for me locally, most of the time. This appears to be an
interaction between how Tokio handles readiness and how we're
entering the event loop. We're effectively entering the Tokio event loop
with a future that's always ready which ends up starving Tokio of
otherwise performing its background work such as updating flags for
readiness of reading/writing.
The fix here is to add a yield at the start of an `in_tokio` block which
is used in synchronous mode. This is a kludge fix but the intention is
to enable Tokio to have a chance to update readiness flags and process
events from epoll/kqueue/etc.
An additional fix to this issue is WebAssembly/wasi-sockets#64 where the
test is waiting on `READABLE` or `WRITABLE`, but in this specific case
it should only wait on `READABLE`. If it waited on just this then that
would also fix this issue. Nevertheless having a `yield_now` is expected
to have little-to-no overhead and otherwise fix this edge case of an
always-ready future.
* Fix passing empty arguments on the CLI
* Add another blocking accept
* Update crates/test-programs/src/bin/api_proxy.rs
Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>
---------
Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>
This commit updates CI to require semicolons in all WIT files and parsed
WIT documents. Some minor updates were required in existing WIT files
and the wasi-nn proposal was additionally updated to its latest version
with semicolons. The wasi-nn update brought some minor changes to the
WIT which required some minor changes here as well.
The task was disabled due to a failing checksum. Since then, the
`install-openvino-action` has improved and now _almost_ has
cross-platform support. This change just restores the status quo in
expectation that #6895 will significantly improve the testing story.
prtest:full
* ci: Upgrade QEMU to `8.1.1`
This adds support for RISC-V's Zcb extension that includes some
extra compressed instructions.
It also removes the current cpuinfo patch, that has been released
in 8.1
* wasmtime: Don't assert the exact faulting address for wasm traps
* Update Rust in CI to 1.72.0
* Update CI, tooling, and docs for MSRV
This commit codifies an MSRV policy for Wasmtime at "stable minus two"
meaning that the latest three releases of Rust will be supported. This
is enforced on CI with a full test suite job running on Linux x86_64
with the minimum supported Rust version. The full test suite will use
the latest stable version. A downside of this approach is that new
changes may break MSRV support on non-Linux or non-x86_64 platforms and
we won't know about it, but that's deemed a minor enough risk at this
time.
A minor fix is applied to Wasmtime's `Cargo.toml` to support Rust 1.70.0
instead of requiring Rust 1.71.0
* Fix installation of rust
* Scrape MSRV from Cargo.toml
* Cranelift is the same as Wasmtime's MSRV now, more words too
* Fix a typo
This commit enables the `component-model` feature at compile time for
our CI-originating binaries in the same manner that `all-arch` is
enabled for CI as well.
* Remove the implementation of wasi-crypto
This commit is a follow-up to the discussion on #6732. This removes
Wasmtime's implementation of the wasi-crypto proposal from in-tree along
with its various support in CI, configuration, etc. See the discussion
on #6732 for the full information but at a high level the main reasons
for removing the implementation at this time are:
* There is not currently an active maintainer of the Wasmtime
integration here for wasi-crypto.
* There are known issues with the code quality of the implementation
such as transmutes of guest-owned memory to `&'static mut [u8]` and
known unsafety in dependencies.
* The size and breadth of the dependency tree brings maintenance burden
and overhead to managing Wasmtime's dependency tree.
As mentioned on the issue this commit does not mean that Wasmtime
doesn't want to implement the wasi-crypto proposal. Instead the "tier 3"
status of wasi-crypto needs to be re-attained to be included back
in-tree, which would mean resolving the above issues.
Note that this commit is intentionally just after the 13.0.0 branch
point which means that this is slated for Wasmtime 14 to be released on
September 20.
* Remove some cfgs
* Remove wasi-crypto CI
Released today this updates from 1.69.0 to 1.71.0. Note that 1.70.0 was
skipped due to presumed codegen bugs for s390x and riscv64. These were
previously tested to work with 1.71.0, so it's time to confirm.
* Update v8 and proc-macro2 dependencies
Gets them both compiling on the latest nightly so we can unpin the Rust
compiler version in OSS-Fuzz.
* Update nightly in CI
* Run `cargo vet` on automated version bumps
As shown in the original CI of #6686 the recent changes for `cargo vet`
0.8.0 mean that the previous release process no longer works since it
requires changes to the lock file for `cargo vet`. This updates the
version bumping process to run `cargo vet` as part of the committed
changes which hopefully will get everything to succeed. To ensure that
the same version of `cargo vet` is used in both locations the
installation procedure was extracted into its own separate little action.
* Configure shell to run in
I originally thought this was the cause of PRs bouncing last night but
now that they're landing again I'm less certain that this is the cause.
Nevertheless this seems good to do regardless.
* wasi-common: need FileEntry to track the mode with which a file was opened
* add a test for read, write access modes in path_open
* a directory needs to report full set of rights in fdstat
implementations such as wasi-libc use these as a mask for any rights
they request for a new fd when invoking path_open
* preview2: mask file perms. and we really need to enforce them on stream creation
which will mean editing the wit.
* filesystem.wit: make {read, write, append}-via-stream fallible with an error-code
in order to fail appropriately when a file is not open for reading or
writing.
* filesystem.wit: update comments
* preview2 impls: fix error handling for {read,write,append}_via_stream
* wasi-common: normalize wrong access mode error on windows to badf
* remove kubkob from labeler
I appreciate his work on this stuff in the past but he hasnt been around
for a few years now
* preview 2 filesystem: allow stat on any opened fd
since file perms is now more accurately the file's access mode, a file
open for writing was having problems here.
in reality i dont think it makes sense to restrict this based on the
perms. if a file is opened, you should be able to stat it.
this fixes the host adapter's path_link test, which was broken by prior
changes. additionally, this fix lets the path_open_dirfd_not_dir test
pass.
* fix the directory base & inheriting rights
in order to work with wasi-testsuite, it needs to be possible to
path_open(dirfd, ".", ...) with the same rights reported in the
fdstat of that dirfd. When we report the Rights::all() set, both
FD_READ and FD_WRITE are set in the base rights, which results in
unix rejecting an openat2(dirfd, ".", O_RDWR) with EISDIR.
By not having the FD_READ and FD_WRITE rights present in the base
rights, the open syscall defaults to O_RDONLY, which is the only
access mode allowed for opening directories.
* path_open of a directory with read and write succeeds on windows
* Update Wasmtime for upcoming WIT changes
This PR integrates bytecodealliance/wasm-tools#1027 into Wasmtime. The
main changes here are:
* WIT syntax is updated with WebAssembly/component-model#193
* Generated bindings in the `bindgen!` macro have been updated to
reflect the new structure of WIT.
* The accepted component model binary format has been updated to account
for changes.
This PR disables wasi-http tests and the on-by-default feature because
the WIT syntax has been updated but the submodule containing the WITs
has not been updated yet so there's no way to get that building
temporarily. Once that's updated then this can be reenabled.
* Update wasmtime-wasi crate with new WIT
* Add wit-bindgen override for the updated version
* Officially disable wasi-http tests/building
* Move test-reactor WIT into the main WIT files
Don't store duplicates with the rest of the WASI WIT files we have.
* Remove adapter's copy of WIT files
* Disable default features for wit-bindgen
* Plumb disabling wasi-http tests a bit more
* Fix reactor tests and adapter build
* Remove no-longer-needed feature
* Update adapter verification script
* Back out some wasi-http hacks
* Update vet and some dependency sources
* Move where wit-bindgen comes from
Make it a more "official" location which is also less likely to be
accidentally deleted in the future.
* Don't document wasi-http-tests
* Add adapter build to dependency for the entire CI build
* Build adapter on PR CI when it changes
* Update adapter verification for new stdin/stdout/stderr
* Update reactor build script
* Reduce some duplication with some shared variables
* Rename adapters to `wasi_snapshot_preview1.{reactor,command}.wasm`
* Fix upload pattern
* winch(fuzz): Refactor Winch's fuzzing
This change is a follow-up to the discussion in
https://github.com/bytecodealliance/wasmtime/pull/6281.
The most notable characteristic of this change is that it enables
`winch` by default in the fuzzers. If compilation time is a big enough
concern I can add the cargo feature back. I opted to enable `winch` by
default for several reasons:
* It substantially reduces the `cfg` complexity -- at first I thought
I had covered all the places in which a `cfg` check would be needed,
but then I realized that I missed the Cranelift specific compiler
flags.
* It's the fastest route to enable winch by default in the fuzzers,
which we want to do eventually -- the only change we'd need at that
point would be to get rid of the winch-specific environment variable.
* We can get rid of the winch-specific checks in CI for fuzzing
* Implement Arbitraty for CompilerStrategy
Unconditionally return `Cranelift` for the `Arbitrary` implementation of
`CompilerStrategy`. This ensures that `Cranelift` is used as the
compiler for all the targets unless explicitly requested otherwise. As
of this change, only the differential target overrides the
`CompilerStrategy`
this is causing a link error because pulldown_cmark is available both in
the rust_wasi_markdown_parser deps directory and, now, in the root of
the project as well.
Andrew Brown
Alex Crichton
Adam Bratschi-Kaye
Luke Parker
Bruce Mitchener
Tyler Rockwood
Afonso Bordado
Eduardo de Moura Rodrigues
ssunkin-fastly
Chris Fallin
Bobby Holley
Pat Hickey
Trevor Elliott
Saúl Cabrera