* wasi-nn: add test programs
This change adds new test programs for wasi-nn in a way fits in with the
existing WASI test infrastructure. The code is not new, though: this
reuses the wasi-nn `examples`, which are currently used by the
`run-wasi-nn-example.sh` CI script. Eventually the examples will be
removed in favor of these tests.
Because wasi-nn's component model support is still in flight, this
change also skips the generation of components for `nn_`-prefixed tests.
* wasi-nn: add `testing` module
This testing-only module has code (i.e., `check_test!`) to check whether
OpenVINO and some test artifacts are available. The test artifacts are
downloaded and cached if not present, expecting `curl` to be present on
the command line (as discussed in the previous version of this, #6895).
* wasi-nn: run `nn_*` test programs as integration tests
Following the pattern of other WASI crates, this change adds the
necessary infrastructure to run the `nn_*` files in
`crates/test-programs` (built by `test-program-artifacts`). These tests
are only run when two sets of conditions are true:
- statically: we only run these tests where we expect OpenVINO to be
easy to install and run (e.g., the `cfg_attr` parts)
- dynamically: we also only run these tests when the OpenVINO libraries
can be located and the model artifacts can be downloaded
* ci: install OpenVINO for running wasi-nn tests
prtest:full
* vet: certify the `wasi-nn` crate
* ci: remove wasi-nn test script
This commit aims to address a discrepancy in Wasmtime where the world
supported by `wasmtime serve` is too large today. This includes
WIT interfaces which are not specified in `wasi:http/proxy` such as
`wasi:filesystem/types`, aka access to a filesystem.
This commit slims down `wasmtime serve` to, by default, only supporting
the `wasi:http/proxy` world. Like with `wasmtime run` various CLI flags
can be passed to enable more interfaces, however:
* `-Scommon` - this enables "common" interfaces such as
`wasi:filesystem`, `wasi:sockets`, and `wasi:cli/*`.
* `-Snn` - this enables wasi-nn
It's expected that more will get extended here over time too.
This change is enabled by a third build of the adapter, a "proxy" mode.
In this mode most functions are cfg'd to return `ERRNO_NOTSUP` to
indicate that the runtime does not support it. Notably this includes the
filesystem, arguments, and environment variables.
This change is tested by updating all `api_proxy*` tests to use this new
adapter which is now required that many previous interfaces are no
longer present by default in the proxy world.
With a custom standard library disabling all features means disabling
some support for feature-detection macros of the native platform. This
meant that `wasmtime compile` output locally wasn't runnable in
`wasmtime-min run` because it couldn't correctly detect that features
were in fact available.
* riscv64: Extend distance trampolines can jump
Use a PIC-friendly set of instructions to enable destination of the
trampoline to be more than 4k away from the tail call site of the
trampoline itself.
* Build "min" artifacts on CI
This commit updates the binary artifacts produced by CI to include "min"
builds where all default features are disabled. Additionally all the
stops are pulled in terms of build flags, nightly versions, etc, to get
a build that is as small as possible without actual source code changes.
This effectively codifies the instructions in #7282 into an easily
downloadable artifact.
No new tarballs are created for github releases but instead tarballs
that previously had a `wasmtime` executable for example now have a
`wasmtime-min` executable. Furthermore the C API which previously had
`libwasmtime.so` for example now has `libwasmtime-min.so`. The intention
is that the minimum-size artifacts are handy for determining a rough
size of Wasmtime but they're not so important that it seems worthwhile
to dedicate entire release entries for.
CI is refactored to support these minimum builds with separate builders.
This means that a single tarball produced as a final result is actually
two separate tarballs merged together, one for the normal build we do
today plus a new "min" tarball produced on the new "min" builders.
Various scripts and CI organization has been adjusted accordingly.
While here I went ahead and enabled `panic=abort` and debuginfo
stripping in our current release artifacts. While this doesn't affect a
whole lot it's less to upload to GitHub Actions all the time.
* Fix Windows unzip
This commit changes the `--dir` argument on the `wasmtime` CLI to be
`HOST::GUEST` rather than `GUEST::HOST`. This matches Docker for example
and is a little more consistent with only `--dir path` where the first
argument is always treated as a host directory.
In terms of breaking-ness the movement from `--mapdir` to `--dir` hasn't
been released with Wasmtime 14 yet so my hope is that this can land on
both `main` and Wasmtime 14.0.0 before it's released to avoid any
breakage other than existing scripts migrating from `--mapdir` to
`--dir`.
* Update MSRV to 1.71.0
With the 1.73.0 release today update our MSRV to a one-higher version.
This will cause 1.73.0 to be tested in CI.
* Fix a warning on windows
* Don't store tables as `&'static mut`
Use `SendSyncPtr` instead (a small wrapper around `NonNull`) which
appeases MIRI.
* Refactor the test-programs test suite
This commit is a large refactoring that reorganizes `test-programs` and
how we tests wasms in Wasmtime. Often writing tests requires complicated
interactions with the guest which can't be done via hand-written `*.wat`
syntax and requires a compiler to get engaged. For this purpose Wasmtime
currently has the `crates/test-programs/*` test suite which builds files
from source and then runs the tests. This has been somewhat cumbersome
in the past though and it's not been easy to extend this over time, so
this commit attempts to address this.
The scheme implemented in this PR looks like:
* All wasm test programs live in `crates/test-programs/src/bin/*.rs`.
All of them, no exceptions.
* Wasm tests have shared support located at
`crates/test-programs/src/lib.rs` and its submodules, such as bindings
generation for WASI.
* Wasm tests are built by a new `crates/test-programs/artifacts` crate.
This crate compiles modules and additionally creates components for
all test programs. The crate itself only records the path to these
outputs and a small amount of testing support, but otherwise doesn't
interact with `wasmtime`-the-crate itself.
* All tests in `crates/test-programs/tests/*.rs` have moved. For example
wasi-http tests now live at `crates/wasi-http/tests/*.rs`. Legacy
tests of wasi-common now live at `crates/wasi-common/tests/*.rs`.
Modern tests for preview2 live at `crates/wasi/tests/*.rs`.
* Wasm tests are bucketed based on their filename prefix. For example
`preview1_*` is tested in wasi-common and wasmtime-wasi. The
`preview2_*` prefix is only tested with wasmtime-wasi, however.
* A new `cli_*` prefix is used to execute tests as part of
`tests/all/main.rs`. This is a new submodule in
`tests/all/cli_tests.rs` which executes these components on the
command line. Many old "command" tests were migrated here.
* Helper macros are generated to assert that a test suite is run in its
entirety. This way if a `preview1_*` test is added it's asserted to
get added to both wasi-common and wasmtime-wasi in the various modes
they run tests.
Overall this moved a number of tests around and refactored some edges of
the tests, but this should not lose any tests (except one that wasn't
actually testing anything). Additionally the hope is that it's much
easier to add tests in the future. The process is to add a new file in
`crates/test-programs/src/bin/*.rs` named appropriately. For example a
preview2 executable is `preview2_*` and a CLI tests is `cli_*`. When
building the test suite an error is generated in the appropriate module
then of "please write a test here", and then a test is written in the
same manner as the other tests in the module.
* Remove no-longer-needed fetches
prtest:full
* I'm worried wasi is running low on semicolons
* Add the WASI target in all CI actions
* Add unknown-unknown target on all CI builders too
* Fix building test artifacts under miri
Need to avoid wrappers for these cross-compiled targets
* Break circular dependency for packaging
Don't use the workspace dep for `wasmtime-wasi` since it injects a
version, instead use a `path = '..'` dependency to fool Cargo into
dropping the dependency during the package phase.
* Fix some merge conflicts with tests
* Fix rebase for new tests
* Remove stray comment
* Fix some flaky tests
* Fix network tests in synchronous mode
This commit is an attempt to fix some networking tests in synchronous
mode in our test suite. Currently networking tests don't actually run in
synchronous mode on CI which is why no failures have been surfaced yet,
but the refactoring in #7182 is going to start doing this.
Currently the `udp_sample_application.rs` test blocks infinitely in
synchronous mode for me locally, most of the time. This appears to be an
interaction between how Tokio handles readiness and how we're
entering the event loop. We're effectively entering the Tokio event loop
with a future that's always ready which ends up starving Tokio of
otherwise performing its background work such as updating flags for
readiness of reading/writing.
The fix here is to add a yield at the start of an `in_tokio` block which
is used in synchronous mode. This is a kludge fix but the intention is
to enable Tokio to have a chance to update readiness flags and process
events from epoll/kqueue/etc.
An additional fix to this issue is WebAssembly/wasi-sockets#64 where the
test is waiting on `READABLE` or `WRITABLE`, but in this specific case
it should only wait on `READABLE`. If it waited on just this then that
would also fix this issue. Nevertheless having a `yield_now` is expected
to have little-to-no overhead and otherwise fix this edge case of an
always-ready future.
* Fix passing empty arguments on the CLI
* Add another blocking accept
* Update crates/test-programs/src/bin/api_proxy.rs
Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>
---------
Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>
* ci: Upgrade QEMU to `8.1.1`
This adds support for RISC-V's Zcb extension that includes some
extra compressed instructions.
It also removes the current cpuinfo patch, that has been released
in 8.1
* wasmtime: Don't assert the exact faulting address for wasm traps
* Stream backpressure v2
Co-authored-by: Pat Hickey <phickey@fastly.com>
Co-authored-by: Trevor Elliott <telliott@fastly.com>
Co-authored-by: Dan Gohman <dev@sunfishcode.online>
Stop testing pseudocode
Restructure when notifications are sent, and make sure to flush the writer
Fix the wasi-http module versions of flush and blocking_flush
Use blocking_write_and_flush for blocking writes in the adapters
Fix a warning in wasi-http
Remove an unused DropPollable
add comment explaining try_write for tcpstream
refactor: separate struct for representing TcpReadStream
by factoring into HostTcpSocket a little bit
tcp read stream: handle stream closing
tcp tests: use blocking_read where its expecting to wait for input
move common test body into wasi-sockets-tests/src/lib.rs
ensure parent socket outlives pollable
input and output streams can be children now
tcp's streams are the sockets children
tcp.wit: document child relationships
tcp tests: fix to drop socket after its child streams
review feedback: propogate worker task panic
style
error source fix
tcp: use preview2::spawn, and propogate worker panics
join handle await always propogates panic
background task handles ewouldblock as well
document choice of constant
* sync wit notes into wasi-http
* improve wit docs for output-stream
* doc: document `HostOutputStream` (#6980)
* doc: document `HostOutputStream`
Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net>
Co-authored-by: Pat Hickey <pat@moreproductive.org>
* fix(wasi): fail when `MemoryOutputStream` buffer is full
Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net>
---------
Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net>
Co-authored-by: Pat Hickey <pat@moreproductive.org>
* rustfmt
prtest:full
* windows and doc fixes
* cli test wasi-http: use blocking-write-and-flush
* Disable some tests, and adjust timeouts when running under qemu
* Try to reproduce the riscv64 failures
* Update riscv to LLVM 17 with beta rust
* Revert "Try to reproduce the riscv64 failures"
This reverts commit 8ac678171f.
* Pin the beta version for riscv64
* Fix a warning on nightly
---------
Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net>
Co-authored-by: Roman Volosatovs <rvolosatovs@users.noreply.github.com>
Co-authored-by: Trevor Elliott <telliott@fastly.com>
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
* Redesign Wasmtime's CLI
This commit follows through on discussion from #6741 to redesign the
flags that the `wasmtime` binary accepts on the CLI. Almost all flags
have been renamed/moved and will require callers to update. The main
motivation here is to cut down on the forest of options in `wasmtime -h`
which are difficult to mentally group together and understand.
The main change implemented here is to move options behind "option
groups" which are intended to be abbreviated with a single letter:
* `-O foo` - an optimization or performance-tuning related option
* `-C foo` - a codegen option affecting the compilation process.
* `-D foo` - a debug-related option
* `-W foo` - a wasm-related option, for example changing wasm semantics
* `-S foo` - a WASI-related option, configuring various proposals for example
Each option group can be explored by passing `help`, for example `-O
help`. This will print all options within the group along with their
help message. Additionally `-O help-long` can be passed to print the
full comment for each option if desired.
Option groups can be specified multiple times on the command line, for
example `-Wrelaxed-simd -Wthreads`. They can also be combined together
with commas as `-Wrelaxed-simd,threads`. Configuration works as a "last
option wins" so `-Ccache,cache=n` would end up with a compilation
cache disabled.
Boolean options can be specified as `-C foo` to enable `foo`, or they
can be specified with `-Cfoo=$val` with any of `y`, `n`, `yes`, `no`,
`true`, or `false`. All other options require a `=foo` value to be
passed and the parsing depends on the type.
This commit additionally applies a few small refactorings to the CLI as
well. For example the help text no longer prints information about wasm
features after printing the option help. This is still available via
`-Whelp` as all wasm features have moved from `--wasm-features` to `-W`.
Additionally flags are no longer conditionally compiled in, but instead
all flags are always supported. A runtime error is returned if support
for a flag is not compiled in. Additionally the "experimental" name of
WASI proposals has been dropped in favor of just the name of the
proposal, for example `--wasi nn` instead of `--wasi-modules
experimental-wasi-nn`. This is intended to mirror how wasm proposals
don't have "experimental" in the name and an opt-in is required
regardless.
A full listing of flags and how they have changed is:
| old cli flag | new cli flag |
|-----------------------------------------------|-------------------------------------------------|
| `-O, --optimize` | removed |
| `--opt-level <LEVEL>` | `-O opt-level=N` |
| `--dynamic-memory-guard-size <SIZE>` | `-O dynamic-memory-guard-size=...` |
| `--static-memory-forced` | `-O static-memory-forced` |
| `--static-memory-guard-size <SIZE>` | `-O static-memory-guard-size=N` |
| `--static-memory-maximum-size <MAXIMUM>` | `-O static-memory-maximum-size=N` |
| `--dynamic-memory-reserved-for-growth <SIZE>` | `-O dynamic-memory-reserved-for-growth=...` |
| `--pooling-allocator` | `-O pooling-allocator` |
| `--disable-memory-init-cow` | `-O memory-init-cow=no` |
| `--compiler <COMPILER>` | `-C compiler=..` |
| `--enable-cranelift-debug-verifier` | `-C cranelift-debug-verifier` |
| `--cranelift-enable <SETTING>` | `-C cranelift-NAME` |
| `--cranelift-set <NAME=VALUE>` | `-C cranelift-NAME=VALUE` |
| `--config <CONFIG_PATH>` | `-C cache-config=..` |
| `--disable-cache` | `-C cache=no` |
| `--disable-parallel-compilation` | `-C parallel-compilation=no` |
| `-g` | `-D debug-info` |
| `--disable-address-map` | `-D address-map=no` |
| `--disable-logging` | `-D logging=no` |
| `--log-to-files` | `-D log-to-files` |
| `--coredump-on-trap <PATH>` | `-D coredump=..` |
| `--wasm-features all` | `-W all-proposals` |
| `--wasm-features -all` | `-W all-proposals=n` |
| `--wasm-features bulk-memory` | `-W bulk-memory` |
| `--wasm-features multi-memory` | `-W multi-memory` |
| `--wasm-features multi-value` | `-W multi-value` |
| `--wasm-features reference-types` | `-W reference-types` |
| `--wasm-features simd` | `-W simd` |
| `--wasm-features tail-call` | `-W tail-call` |
| `--wasm-features threads` | `-W threads` |
| `--wasm-features memory64` | `-W memory64` |
| `--wasm-features copmonent-model` | `-W component-model` |
| `--wasm-features function-references` | `-W function-references` |
| `--relaxed-simd-deterministic` | `-W relaxed-simd-deterministic` |
| `--enable-cranelift-nan-canonicalization` | `-W nan-canonicalization` |
| `--fuel <N>` | `-W fuel=N` |
| `--epoch-interruption` | `-W epoch-interruption` |
| `--allow-unknown-exports` | `-W unknown-exports-allow` |
| `--trap-unknown-imports` | `-W unknown-imports-trap` |
| `--default-values-unknown-imports` | `-W unknown-imports-default` |
| `--max-instances <MAX_INSTANCES>` | `-W max-instances=N` |
| `--max-memories <MAX_MEMORIES>` | `-W max-memories=N` |
| `--max-memory-size <BYTES>` | `-W max-memory-size=N` |
| `--max-table-elements <MAX_TABLE_ELEMENTS>` | `-W max-table-elements=N` |
| `--max-tables <MAX_TABLES>` | `-W max-tables=N` |
| `--max-wasm-stack <MAX_WASM_STACK>` | `-W max-wasm-stack=N` |
| `--trap-on-grow-failure` | `-W trap-on-grow-failure` |
| `--wasm-timeout <TIME>` | `-W timeout=N` |
| `--wmemcheck` | `-W wmemcheck` |
| `--wasi-modules default` | removed |
| `--wasi-modules -default` | removed |
| `--wasi-modules wasi-common` | `-S common` |
| `--wasi-modules -wasi-common` | `-S common=n` |
| `--wasi-modules experimental-wasi-nn` | `-S nn` |
| `--wasi-modules experimental-wasi-threads` | `-S threads` |
| `--wasi-modules experimental-wasi-http` | `-S http` |
| `--listenfd` | `-S listenfd` |
| `--tcplisten <SOCKET ADDRESS>` | `-S tcplisten=...` |
| `--wasi-nn-graph <FORMAT::HOST>` | `-S nn-graph=FORMAT::HOST` |
| `--preview2` | `-S preview2` |
| `--dir <DIRECTORY>` | `--dir ...` |
| `--mapdir <GUEST_DIR::HOST_DIR>` | `--dir a::b` |
* Be more descriptive with help text
* Document `=val` is optional for `-Ccranelift-xxx`
* Fix compile after rebase
* Fix rebase of `--inherit-network`
* Fix wasi-http test
* Fix compile without pooling allocator support
* Update some flags in docs
* Fix bench-api build
* Update flags for gdb/lldb tests
* Fixup optimization flags
prtest:full
* Update Rust in CI to 1.72.0
* Update CI, tooling, and docs for MSRV
This commit codifies an MSRV policy for Wasmtime at "stable minus two"
meaning that the latest three releases of Rust will be supported. This
is enforced on CI with a full test suite job running on Linux x86_64
with the minimum supported Rust version. The full test suite will use
the latest stable version. A downside of this approach is that new
changes may break MSRV support on non-Linux or non-x86_64 platforms and
we won't know about it, but that's deemed a minor enough risk at this
time.
A minor fix is applied to Wasmtime's `Cargo.toml` to support Rust 1.70.0
instead of requiring Rust 1.71.0
* Fix installation of rust
* Scrape MSRV from Cargo.toml
* Cranelift is the same as Wasmtime's MSRV now, more words too
* Fix a typo
* wasi-nn: add [named models]
This change adds a way to retrieve preloaded ML models (i.e., "graphs"
in wasi-nn terms) from a registry. The wasi-nn specification includes a
new function, `load_by_name`, that can be used to access these models
more efficiently than before; previously, a user's only option was to
read/download/etc. all of the bytes of an ML model and pass them to the
`load` function.
[named models]: https://github.com/WebAssembly/wasi-nn/issues/36
In Wasmtime's implementation of wasi-nn, we call the registry that holds
the models a `GraphRegistry`. We include a simplistic `InMemoryRegistry`
for use in the Wasmtime CLI (more on this later) but the idea is that
production use will involve some more complex caching and thus a new
implementation of a registry--a `Box<dyn GraphRegistry>`--passed into
the wasi-nn context. Note that, because we now must be able to `clone` a
graph out of the registry and into the "used graphs" table, the OpenVINO
`BackendGraph` is updated to be easier to copy around.
To allow experimentation with this "preload a named model"
functionality, this change also adds a new Wasmtime CLI flag: `--graph
<encoding>:<host dir>`. Wasmtime CLI users can now preload a model from
a directory; the directory `basename` is used as the model name. Loading
models from a directory is probably not desired in Wasmtime embeddings
so it is cordoned off into a separate `BackendFromDir` extension trait.
* wasi-nn: add "named model" test
Add a new example crate which loads a model by name and performs image
classification. It uses the same MobileNet model as the existing test
but a new version of the Rust bindings. The new crate is built and run
with the new CLI flag in the `ci/run-wasi-nn-example.sh` script.
prtest:full
* review: rename `--graph` to `--wasi-nn-graph`
This commit refactors the `wasmtime` CLI executable to be able to
support not only compiling components but additionally executing
components. While I was doing this I've additionally added a new
`--preview2` argument to enable using the new experimental
implementation of preview1 based on preview2 type/structs. This is
off-by-default but is expected to become the default in the future.
Some notable features of this change are:
* The preview1-implemented-with-preview2 module now sports
`add_to_linker_{async,sync}` to replace the previous `add_to_linker`
which always did async.
* Some trait bounds in the preview1-implemented-with-preview2 module are
simplified.
* Some minor changes were made to `wiggle`'s macros to support a "block
on" that isn't the default wiggle dummy executor (as now we actually
do need Tokio)
* Many options related to core wasm `Linker` configuration, such as
`--default-values-unknown-imports`, are not implemented for components
at this time. When used with components these options return an error.
* Construction of WASI contexts has been refactored to pass around fewer
arguments to avoid threading through lots of values for both preview1
and preview2.
* Reading the input to the Wasmtime CLI has been updated to read the
input in the `run` subcommand before handing it off to the `wasmtime`
crate's API to enable the CLI to use the contents of what's loaded to
determine what to do next.
* Our generic `./ci/run-tests.sh` script has been updated to pass the
`--features component-model` flag to ensure that this CLI support is
tested during the normal test suite.
* The CLI support for `wasi-nn` supports components as well as core wasm
modules.
* Remove the implementation of wasi-crypto
This commit is a follow-up to the discussion on #6732. This removes
Wasmtime's implementation of the wasi-crypto proposal from in-tree along
with its various support in CI, configuration, etc. See the discussion
on #6732 for the full information but at a high level the main reasons
for removing the implementation at this time are:
* There is not currently an active maintainer of the Wasmtime
integration here for wasi-crypto.
* There are known issues with the code quality of the implementation
such as transmutes of guest-owned memory to `&'static mut [u8]` and
known unsafety in dependencies.
* The size and breadth of the dependency tree brings maintenance burden
and overhead to managing Wasmtime's dependency tree.
As mentioned on the issue this commit does not mean that Wasmtime
doesn't want to implement the wasi-crypto proposal. Instead the "tier 3"
status of wasi-crypto needs to be re-attained to be included back
in-tree, which would mean resolving the above issues.
Note that this commit is intentionally just after the 13.0.0 branch
point which means that this is slated for Wasmtime 14 to be released on
September 20.
* Remove some cfgs
* Remove wasi-crypto CI
* Require wasmtime options are first when running modules
Currently the way we've configured argument parsing it's valid to
execute a command such as:
wasmtime run foo.wasm -O
which is the same as:
wasmtime run -O foo.wasm
or otherwise all flags are attempted to be parsed as Wasmtime flags and
an error is generated when they're not wasmtime flags. I've personally
found this a bit confusing in the past and I find myself frequently
executing:
wasmtime run -- foo.wasm -other -arguments
While this works my general impression is that many other "wrapper
commands" don't behave this way and typically don't require `--` to pass
flags to the target executable. This commit reconfigures argument
parsing to consider any argument after the WebAssembly module itself to
be an argument to the wasm program rather than an argument to Wasmtime.
This means that all Wasmtime options must come between the `run` command
and the `foo.wasm` WebAssembly argument.
* Update wasi testsuite runner
* Support `wasmtime -- run`
Additionally use more clap features to avoid manually checking against
subcommands.
* Remove stale comment
* Reorder wasi-nn arguments
* Reorder more flags
* Fix unused import on Windows
* Don't run a stdio test on Windows
* Update gdb/lldb tests
* Don't assert that the write succeeds
prtest:full
* Update adapter build
* Rename the binary artifact to `wasi_snapshot_preview1.wasm` and update
build scripts to account for this.
* Update documentation to mention difference between reactor/command
builds.
Closes#6569
* More renaming
* Add adapter build to dependency for the entire CI build
* Build adapter on PR CI when it changes
* Update adapter verification for new stdin/stdout/stderr
* Update reactor build script
* Reduce some duplication with some shared variables
* Rename adapters to `wasi_snapshot_preview1.{reactor,command}.wasm`
* Fix upload pattern
* wasi-tests and wasi-http-tests no longer have their own workspace
* wasi-tests: fix warnings
* rewrite the test-programs build.rs to generate {package}_modules.rs and _components.rs
The style is cribbed from preview2-prototying repo, but I ended up
refactoring it a bit.
* better escaping should help with windows?
* long form cap-std-sync and tokio test suites
* convert wasi-http test
* fixes, comments
* apply cargo fmt to whole workspace
* bump test-programs and wasi-http-tests to all use common dependency versions
wit-bindgen 0.6.0 and wit-component 0.7.4
* add new audits
* cargo vet prune
* package and supply chain updates to fix vulnerabilities
h2 upgraded from 0.3.16 -> 0.3.19 to fix vulnerability
tempfile upgraded from 0.3.3 -> 0.3.5 to eliminate dep on vulnerable
remove_dir_all
* deny: temporarily allow duplicate wasm-encoder, wasmparser, wit-parser
prtest:full
* convert more dependencies to { workspace = true }
Alex asked me to do thsi for wit-component and wit-bindgen, and I found
a few more (cfg-if, tempfile, filecheck, anyhow...
I also reorganized the workspace dependencies section to make the ones
our team maintains more clearly separated from our external
dependencies.
* test-programs build: ensure that the user writes a #[test] for each module, component
* fix build of wasi-tests on windows
* misspelled macos
* mark wasi-tests crate test=false so we dont try building it natively...
* mark wasi-http-tests test=false as well
* try getting the cargo keys right
* just exclude wasi-tests and wasi-http-tests in run-tests.sh
* interesting paths fails on windows
* misspelling so nice i did it twice
* new cargo deny exception: ignore all of wit-bindgen's dependencies
* auto-import wildcard vets
Since the latest updates to our release process which transitioned to
merge queues it appears that patch release create incorrectly named
tarballs. The version in the tarball is based on the branch name, which
doesn't change for patch releases, so the version needs to come from
`Cargo.toml`. Thankfully there's already a helpful shell script to do
that so use the shell script instead of using the branch name.
* riscv64: Remove unused code
* riscv64: Add vector types
* riscv64: Initial Vector ABI Load/Stores
* riscv64: Vector Loads/Stores
* riscv64: Fix `vsetvli` encoding error
* riscv64: Add SIMD `iadd` runtests
* riscv64: Rename `VecSew`
The SEW name is correct, but only for VType. We also use this type
in loads/stores as the Efective Element Width, so the name isn't
quite correct in that case.
* ci: Add V extension to RISC-V QEMU
* riscv64: Misc Cleanups
* riscv64: Check V extension in `load`/`store` for SIMD
* riscv64: Fix `sumop` doc comment
* cranelift: Fix comment typo
* riscv64: Add convert for VType and VecElementWidth
* riscv64: Remove VecElementWidth converter
* cranelift-native: Move riscv to separate module
* cranelift-native: Read /proc/cpuinfo to parse RISC-V extensions
* ci: Add QEMU cpuinfo emulation patch
This patch emulates the /proc/cpuinfo interface for RISC-V. This
allows us to do feature detection for the RISC-V backend.
It has been queued for QEMU 8.1 so we should remove it as soon as
that is available.
* ci: Enable QEMU RISC-V extensions
* cranelift-native: Cleanup ISA string parsing
Co-Authored-By: Jamey Sharp <jsharp@fastly.com>
* cranelift-native: Rework `/proc/cpuinfo` parsing
Co-Authored-By: Jamey Sharp <jsharp@fastly.com>
---------
Co-authored-by: Jamey Sharp <jsharp@fastly.com>
* ci: unpin the wasi-nn tasks from an older Ubuntu
Previously, OpenVINO's lack of APT packages for Ubuntu 22.04 (`jammy`)
prevented us from upgrading the GitHub runner to use `ubuntu-latest`. I
updated the `install-openvino-action` to substitute in the `focal`
packages in this case (this is what the OpenVINO team considers the fix)
so this pin should no longer be necessary. Fixes#5408.
(Run all CI actions: prtest:full)
* vet: audit the openvino version bump
* Integrate experimental HTTP into wasmtime.
* Reset Cargo.lock
* Switch to bail!, plumb options partially.
* Implement timeouts.
* Remove generated files & wasm, add Makefile
* Remove generated code textfile
* Update crates/wasi-http/Cargo.toml
Co-authored-by: Eduardo de Moura Rodrigues <16357187+eduardomourar@users.noreply.github.com>
* Update crates/wasi-http/Cargo.toml
Co-authored-by: Eduardo de Moura Rodrigues <16357187+eduardomourar@users.noreply.github.com>
* Extract streams from request/response.
* Fix read for len < buffer length.
* Formatting.
* types impl: swap todos for traps
* streams_impl: idioms, and swap todos for traps
* component impl: idioms, swap all unwraps for traps, swap all todos for traps
* http impl: idiom
* Remove an unnecessary mut.
* Remove an unsupported function.
* Switch to the tokio runtime for the HTTP request.
* Add a rust example.
* Update to latest wit definition
* Remove example code.
* wip: start writing a http test...
* finish writing the outbound request example
havent executed it yet
* better debug output
* wasi-http: some stubs required for rust rewrite of the example
* add wasi_http tests to test-programs
* CI: run the http tests
* Fix some warnings.
* bump new deps to latest releases (#3)
* Add tests for wasi-http to test-programs (#2)
* wip: start writing a http test...
* finish writing the outbound request example
havent executed it yet
* better debug output
* wasi-http: some stubs required for rust rewrite of the example
* add wasi_http tests to test-programs
* CI: run the http tests
* bump new deps to latest releases
h2 0.3.16
http 0.2.9
mio 0.8.6
openssl 0.10.48
openssl-sys 0.9.83
tokio 1.26.0
---------
Co-authored-by: Brendan Burns <bburns@microsoft.com>
* Update crates/test-programs/tests/http_tests/runtime/wasi_http_tests.rs
* Update crates/test-programs/tests/http_tests/runtime/wasi_http_tests.rs
* Update crates/test-programs/tests/http_tests/runtime/wasi_http_tests.rs
* wasi-http: fix cargo.toml file and publish script to work together (#4)
unfortunately, the publish script doesn't use a proper toml parser (in
order to not have any dependencies), so the whitespace has to be the
trivial expected case.
then, add wasi-http to the list of crates to publish.
* Update crates/test-programs/build.rs
* Switch to rustls
* Cleanups.
* Merge switch to rustls.
* Formatting
* Remove libssl install
* Fix tests.
* Rename wasi-http -> wasmtime-wasi-http
* prtest:full
Conditionalize TLS on riscv64gc.
* prtest:full
Fix formatting, also disable tls on s390x
* prtest:full
Add a path parameter to wit-bindgen, remove symlink.
* prtest:full
Fix tests for places where SSL isn't supported.
* Update crates/wasi-http/Cargo.toml
---------
Co-authored-by: Eduardo de Moura Rodrigues <16357187+eduardomourar@users.noreply.github.com>
Co-authored-by: Pat Hickey <phickey@fastly.com>
Co-authored-by: Pat Hickey <pat@moreproductive.org>
This refactoring from #5766 accidentally broke the tag name calculation
for the `build-src-tarball.sh` script so this fixes it by copying over
the same logic from `build-tarballs.sh`.
* Test all backends when a runtest is modified
* Check that this triggers all backend tests
* Revert "Check that this triggers all backend tests"
This reverts commit 1d12536d04.
* Don't run LLDB tests on PRs
These take an extra minute or so, so only run them on the full test
suite of a merge instead of on all PRs as well.
* Add a test for the x64 isa files
This guarantees that if cranelift's x64 backend is modified that the
tests will be run on a PR, even if other backends were also modified.
GitHub recently made its merge queue feature available for use in public
repositories owned by organizations meaning that the Wasmtime repository
is a candidate for using this. GitHub's Merge Queue feature is a system
that's similar to Rust's bors integration where PRs are tested before
merging and only passing PRs are merged. This implements the "not rocket
science" rule where the `main` branch of Wasmtime, for example, is
always tested and passes CI. This is in contrast to our current
implementation of CI where PRs are merged when they pass their own CI,
but the code that was tested is not guaranteed to be the state of `main`
when the PR is merged, meaning that we're at risk now of a failing
`main` branch despite all merged PRs being green. While this has
happened with Wasmtime this is not a common occurrence, however.
The main motivation, instead, to use GitHub's Merge Queue feature is
that it will enable Wasmtime to greatly reduce the amount of CI running
on PRs themselves. Currently the full test suite runs on every push to
every PR, meaning that our workers on GitHub Actions are frequently
clogged throughout weekdays and PRs can take quite some time to come
back with a successful run. Through the use of a Merge Queue, however,
we're able to configure only a small handful of checks to run on PRs
while deferring the main body of checks to happening on the
merge-via-the-queue itself. This is hoped to free up capacity on CI and
overall improve CI times for Wasmtime and Cranelift developers.
The implementation of all of this required quite a lot of plumbing and
retooling of our CI. I've been testing this in an [external
repository][testrepo] and I think everything is working now. A list of
changes made in this PR are:
* The `build.yml` workflow is merged back into the `main.yml` workflow
as the original reason to split it out is not longer applicable (it'll
run on all merges). This was also done to fit in the dependency graph
of jobs of one workflow.
* Publication of the `gh-pages` branch, the `dev` tag artifacts, and
release artifacts have been moved to a separate
`publish-artifacts.yml` workflow. This workflow runs on all pushes to
`main` and all tags. This workflow no longer actually preforms any
builds, however, and relies on a merge queue or similar being used for
branches/tags where artifacts are downloaded from the workflow run to
be uploaded. For pushes to `main` this works because a merge queue is
run meaning that by the time the push happens all artifacts are ready.
For release branches this is handled by..
* The `push-tag.yml` workflow is subsumed by the `main.yml` workflow. CI
for a tag being pushed will upload artifacts to a release in GitHub,
meaning that all builds must finish first for the commit. The
`main.yml` workflow at the end now scans commits for the preexisting
magical marker and pushes a tag if necessary.
* CI is currently a flat list of "run all these jobs" and this is now
rearchitected to a "fan out" approach where some jobs run to determine
the next jobs to run which then get "joined" into a finish step. The
purpose for this is somewhat nuanced and this has implications for CI
runtime as well. The Merge Queue feature requires branches to be
protected with "these checks must pass" and then the same checks are
gates both to enter the merge queue as well as pass the merge queue.
The saving grace, however, is that a "skipped" check counts as
passing, meaning checks can be skipped on PRs but run to completion on
the merge queue. A problem with this though is the build matrix used
for tests where PRs want to only run one element of the build matrix
ideally but there's no means on GitHub Actions right now for the
skipped entries to show up as skipped easily (or not that I know of).
This means that the "join" step serves the purpose of being the single
gate for both PR and merge queue CI and there's just more inputs to it
for merge queue CI. The major consequence of this decision is that
GitHub's actions scheduling doesn't work out well here. Jobs are
scheduled in a FIFO order meaning that the job for "ok complete the CI
run" is queued up after everything else has completed, possibly
after lots of other CI requests in the middle for other PRs. The hope
here is that by using a merge queue we can keep CI relatively under
control and this won't affect merge times too much.
* All jobs in the `main.yml` workflow will not automatically cancel the
entire run if they fail. Previously this fail-fast behavior was only
part of the matrix runs (and just for that matrix), but this is
required to make the merge queue expedient. The gate of the merge
queue is the final "join" step which is only executed once all
dependencies have finished. This means, for example, that if rustfmt
fails quickly then the tests which take longer might run for quite
awhile before the join step reports failure, meaning that the PR sits
in the queue for longer than needed being tested when we know it's
already going to fail. By having all jobs cancel the run this means
that failures immediately bail out and mark the whole job as
cancelled.
* A new "determine" CI job was added to determine what CI actually needs
to run. This is a "choke point" which is scheduled at the start of CI
that quickly figures out what else needs to be run. This notably
indicates whether large swaths of ci (the `run-full` flag) like the
build matrix are executed. Additionally this dynamically calculates a
matrix of tests to run based on a new `./ci/build-test-matrix.js`
script. Various inputs are considered for this such as:
1. All pushes, meaning merge queue branches or release-branch merges,
will run full CI.
2. PRs to release branches will run full CI.
3. PRs to `main`, the most common, determine what to run based on
what's modified and what's in the commit message.
Some examples for (3) above are if modifications are made to
`cranelift/codegen/src/isa/*` then that corresponding builder is
executed on CI. If the `crates/c-api` directory is modified then the
CMake-based tests are run on PRs but are otherwise skipped.
Annotations in commit messages such as `prtest:*` can be used to
explicitly request testing.
Before this PR merges to `main` would perform two full runs of CI: one
on the PR itself and one on the merge to `main`. Note that the one as a
merge to `main` was quite frequently cancelled due to a merge happening
later. Additionally before this PR there was always the risk of a bad
merge where what was merged ended up creating a `main` that failed CI to
to a non-code-related merge conflict.
After this PR merges to `main` will perform one full run of CI, the one
as part of the merge queue. PRs themselves will perform one test job
most of the time otherwise. The `main` branch is additionally always
guaranteed to pass tests via the merge queue feature.
For release branches, before this PR merges would perform two full
builds - one for the PR and one for the merge. A third build was then
required for the release tag itself. This is now cut down to two full
builds, one for the PR and one for the merge. The reason for this is
that the merge queue feature currently can't be used for our
wildcard-based `release-*` branch protections. It is now possible,
however, to turn on required CI checks for the `release-*` branch PRs so
we can at least have a "hit the button and forget" strategy for merging
PRs now.
Note that this change to CI is not without its risks. The Merge Queue
feature is still in beta and is quite new for GitHub. One bug that
Trevor and I uncovered is that if a PR is being tested in the merge
queue and a contributor pushes to their PR then the PR isn't removed
from the merge queue but is instead merged when CI is successful, losing
the changes that the contributor pushed (what's merged is what was
tested). We suspect that GitHub will fix this, however.
Additionally though there's the risk that this may increase merge time
for PRs to Wasmtime in practice. The Merge Queue feature has the ability
to "batch" PRs together for a merge but this is only done if concurrent
builds are allowed. This means that if 5 PRs are batched together then 5
separate merges would be created for the stack of 5 PRs. If the CI for
all 5 merged together passes then everything is merged, otherwise a PR
is kicked out. We can't easily do this, however, since a major purpose
for the merge queue for us would be to cut down on usage of CI builders
meaning the max concurrency would be set to 1 meaning that only one PR
at a time will be merged. This means PRs may sit in the queue for awhile
since previously many `main`-based builds are cancelled due to
subsequent merges of other PRs, but now they must all run to 100%
completion.
[testrepo]: https://github.com/bytecodealliance/wasmtime-merge-queue-testing
This commit includes a set of changes that add initial support for `wasi-threads` to Wasmtime:
* feat: remove mutability from the WasiCtx Table
This patch adds interior mutability to the WasiCtx Table and the Table elements.
Major pain points:
* `File` only needs `RwLock<cap_std::fs::File>` to implement
`File::set_fdflags()` on Windows, because of [1]
* Because `File` needs a `RwLock` and `RwLock*Guard` cannot
be hold across an `.await`, The `async` from
`async fn num_ready_bytes(&self)` had to be removed
* Because `File` needs a `RwLock` and `RwLock*Guard` cannot
be dereferenced in `pollable`, the signature of
`fn pollable(&self) -> Option<rustix::fd::BorrowedFd>`
changed to `fn pollable(&self) -> Option<Arc<dyn AsFd + '_>>`
[1] da238e324e/src/fs/fd_flags.rs (L210-L217)
* wasi-threads: add an initial implementation
This change is a first step toward implementing `wasi-threads` in
Wasmtime. We may find that it has some missing pieces, but the core
functionality is there: when `wasi::thread_spawn` is called by a running
WebAssembly module, a function named `wasi_thread_start` is found in the
module's exports and called in a new instance. The shared memory of the
original instance is reused in the new instance.
This new WASI proposal is in its early stages and details are still
being hashed out in the [spec] and [wasi-libc] repositories. Due to its
experimental state, the `wasi-threads` functionality is hidden behind
both a compile-time and runtime flag: one must build with `--features
wasi-threads` but also run the Wasmtime CLI with `--wasm-features
threads` and `--wasi-modules experimental-wasi-threads`. One can
experiment with `wasi-threads` by running:
```console
$ cargo run --features wasi-threads -- \
--wasm-features threads --wasi-modules experimental-wasi-threads \
<a threads-enabled module>
```
Threads-enabled Wasm modules are not yet easy to build. Hopefully this
is resolved soon, but in the meantime see the use of
`THREAD_MODEL=posix` in the [wasi-libc] repository for some clues on
what is necessary. Wiggle complicates things by requiring the Wasm
memory to be exported with a certain name and `wasi-threads` also
expects that memory to be imported; this build-time obstacle can be
overcome with the `--import-memory --export-memory` flags only available
in the latest Clang tree. Due to all of this, the included tests are
written directly in WAT--run these with:
```console
$ cargo test --features wasi-threads -p wasmtime-cli -- cli_tests
```
[spec]: https://github.com/WebAssembly/wasi-threads
[wasi-libc]: https://github.com/WebAssembly/wasi-libc
This change does not protect the WASI implementations themselves from
concurrent access. This is already complete in previous commits or left
for future commits in certain cases (e.g., wasi-nn).
* wasi-threads: factor out process exit logic
As is being discussed [elsewhere], either calling `proc_exit` or
trapping in any thread should halt execution of all threads. The
Wasmtime CLI already has logic for adapting a WebAssembly error code to
a code expected in each OS. This change factors out this logic to a new
function, `maybe_exit_on_error`, for use within the `wasi-threads`
implementation.
This will work reasonably well for CLI users of Wasmtime +
`wasi-threads`, but embedders will want something better in the future:
when a `wasi-threads` threads fails, they may not want their application
to exit. Handling this is tricky, because it will require cancelling the
threads spawned by the `wasi-threads` implementation, something that is
not trivial to do in Rust. With this change, we defer that work until
later in order to provide a working implementation of `wasi-threads` for
experimentation.
[elsewhere]: https://github.com/WebAssembly/wasi-threads/pull/17
* review: work around `fd_fdstat_set_flags`
In order to make progress with wasi-threads, this change temporarily
works around limitations induced by `wasi-common`'s
`fd_fdstat_set_flags` to allow `&mut self` use in the implementation.
Eventual resolution is tracked in
https://github.com/bytecodealliance/wasmtime/issues/5643. This change
makes several related helper functions (e.g., `set_fdflags`) take `&mut
self` as well.
* test: use `wait`/`notify` to improve `threads.wat` test
Previously, the test simply executed in a loop for some hardcoded number
of iterations. This changes uses `wait` and `notify` and atomic
operations to keep track of when the spawned threads are done and join
on the main thread appropriately.
* various fixes and tweaks due to the PR review
---------
Signed-off-by: Harald Hoyer <harald@profian.com>
Co-authored-by: Harald Hoyer <harald@profian.com>
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
This fixes the build issue identified in #5664 at the toolchain level
rather than working around it in our own build. The next step in fixing
this will be to remove the nightly override in the future when the
toolchain becomes stable.
* Add source tarballs to our releases
This commit adds a small script to create a source tarball as part of
the release process. This goes further than requested by #3808 by
vendoring all Rust dependencies as well to be more in line with
"download the source once then build somewhere without a network".
Vendoring the Rust dependencies makes the tarball pretty beefy (67M
compressed, 500M uncompressed). Unfortunately most of this size comes
from vendored crates such as v8, pqcrypto-kyber, winapi, capstone-sys,
plotters, and web-sys. Only `winapi` in this list is actually needed for
`wasmtime`-the-binary and only on Windows as well but for now this is
the state of things related to `cargo vendor`. If this becomes an issue
we could specifically remove the bulky contents of crates in the
`vendor` directory such as `v8` since it's only used for fuzzing.
Closes#3808
* Review feedback
* Review comments
* ci: replace OpenVINO installer action
To test wasi-nn, we currently use an OpenVINO backend. The Wasmtime CI
must install OpenVINO using a custom GitHub action. This CI action has
not been updated in some time and in the meantime OpenVINO (and the
OpenVINO crates) have released several new versions.
https://github.com/abrown/install-openvino-action is an external action
that we plan to keep up to date with the latest releases. This change
replaces the current CI action with that one.
* wasi-nn: upgrade openvino dependency to v0.4.1
This eliminates a `lazy_static` dependency and changes a few parameters
to pass by reference. Importantly, it enables support for the latest
versions of OpenVINO (v2022.*) in wasi-nn.
* ci: update wasi-nn script to source correct env script
* ci: really use the correct path for the env script
Also, clarify which directory OpenVINO is installed in (the symlink may
not be present).
* Refactor binary-compatible-builds for releases
I was poking around this yesterday and noticed a few things that could
be improved for our release builds:
* The centos container for the x86_64 builds contained a bunch of extra
tooling we no longer need such as python3 and a C++ compiler. Along
with custom toolchain things this could all get removed since the C we
include now is quite simple.
* The aarch64 and s390x cross-compiled builds had relatively high glibc
version requirements compared to the x86_64 build. This was because we
don't use a container to build the cross-compiled binaries. I added
containers here along the lines of the x86_64 build to use an older
glibc to build the release binary to lower our version requirement.
This lower the aarch64 version requirement from glibc 2.28 to 2.17.
Additionally the s390x requirement dropped from 2.28 to 2.16.
* To make the containers a bit easier to read/write I added
`Dockerfile`s for them in a new `ci/docker` directory instead of
hardcoding install commands in JS.
This isn't intended to be a really big change or anything for anyone,
but it's intended to keep our Linux-based builds consistent at least as
best we can.
* Remove temporary change
This PR fixes#4066: it modifies the Cranelift `build.rs` workflow to
invoke the ISLE DSL compiler on every compilation, rather than only
when the user specifies a special "rebuild ISLE" feature.
The main benefit of this change is that it vastly simplifies the mental
model required of developers, and removes a bunch of failure modes
we have tried to work around in other ways. There is now just one
"source of truth", the ISLE source itself, in the repository, and so there
is no need to understand a special "rebuild" step and how to handle
merge errors. There is no special process needed to develop the compiler
when modifying the DSL. And there is no "noise" in the git history produced
by constantly-regenerated files.
The two main downsides we discussed in #4066 are:
- Compile time could increase, by adding more to the "meta" step before the main build;
- It becomes less obvious where the source definitions are (everything becomes
more "magic"), which makes exploration and debugging harder.
This PR addresses each of these concerns:
1. To maintain reasonable compile time, it includes work to cut down the
dependencies of the `cranelift-isle` crate to *nothing* (only the Rust stdlib),
in the default build. It does this by putting the error-reporting bits
(`miette` crate) under an optional feature, and the logging (`log` crate) under
a feature-controlled macro, and manually writing an `Error` impl rather than
using `thiserror`. This completely avoids proc macros and the `syn` build slowness.
The user can still get nice errors out of `miette`: this is enabled by specifying
a Cargo feature `--features isle-errors`.
2. To allow the user to optionally inspect the generated source, which nominally
lives in a hard-to-find path inside `target/` now, this PR adds a feature `isle-in-source-tree`
that, as implied by the name, moves the target for ISLE generated source into
the source tree, at `cranelift/codegen/isle_generated_source/`. It seems reasonable
to do this when an explicit feature (opt-in) is specified because this is how ISLE regeneration
currently works as well. To prevent surprises, if the feature is *not* specified, the
build fails if this directory exists.
* Fix the release process's latest step
The automated release of 0.36.0 was attempted last night but it failed
due to a [failure on CI][bad]. This failure comes about because it was
trying to change the release date of 0.35.0 which ended up not modifying
any fails so `git` failed to commit as no files were changed.
The original bug though was that 0.35.0 was being changed instead of
0.36.0. The reason for this is that the script used
`--sort=-committerdate` to determine the latest branch. I forgot,
though, that with backports it's possible for 0.35.0 to have a more
recent commit date than 0.36.0 (as is currently the case). This commit
updates the script to perform a numerical sort outside of git to get the
latest release branch.
Additionally this adds in some `set -ex` commands for the shell which
should help print out commands as they're run and assist in future
debugging.
[bad]: https://github.com/bytecodealliance/wasmtime/runs/6087188708
* Replace sed with rust
* Add m1 to release process
This will create a pre-compiled binary for m1 macs and adds
a link to review embark studios CI for verification.
* remove test for macos arm
Tests will not succeed for macos arm until GitHub provides a an m1 hosted runner.
Co-authored-by: Bailey Hayes <bhayes@singlestore.com>
* Bump to 0.36.0
* Add a two-week delay to Wasmtime's release process
This commit is a proposal to update Wasmtime's release process with a
two-week delay from branching a release until it's actually officially
released. We've had two issues lately that came up which led to this proposal:
* In #3915 it was realized that changes just before the 0.35.0 release
weren't enough for an embedding use case, but the PR didn't meet the
expectations for a full patch release.
* At Fastly we were about to start rolling out a new version of Wasmtime
when over the weekend the fuzz bug #3951 was found. This led to the
desire internally to have a "must have been fuzzed for this long"
period of time for Wasmtime changes which we felt were better
reflected in the release process itself rather than something about
Fastly's own integration with Wasmtime.
This commit updates the automation for releases to unconditionally
create a `release-X.Y.Z` branch on the 5th of every month. The actual
release from this branch is then performed on the 20th of every month,
roughly two weeks later. This should provide a period of time to ensure
that all changes in a release are fuzzed for at least two weeks and
avoid any further surprises. This should also help with any last-minute
changes made just before a release if they need tweaking since
backporting to a not-yet-released branch is much easier.
Overall there are some new properties about Wasmtime with this proposal
as well:
* The `main` branch will always have a section in `RELEASES.md` which is
listed as "Unreleased" for us to fill out.
* The `main` branch will always be a version ahead of the latest
release. For example it will be bump pre-emptively as part of the
release process on the 5th where if `release-2.0.0` was created then
the `main` branch will have 3.0.0 Wasmtime.
* Dates for major versions are automatically updated in the
`RELEASES.md` notes.
The associated documentation for our release process is updated and the
various scripts should all be updated now as well with this commit.
* Add notes on a security patch
* Clarify security fixes shouldn't be previewed early on CI
Updates the OpenVINO backend for wasi-nn to pre-configure all inputs as `NHWC`. This is not a long-term fix but no worse than the status quo, which configures all input tensors to the same layout. This change updates the CI script to use the latest version of OpenVINO. Closes#3948.
Co-authored-by: Andrew Brown <andrew.brown@intel.com>
Alex Crichton
Andrew Brown
Trevor Elliott
Afonso Bordado
Pat Hickey
Eduardo de Moura Rodrigues
Chris Fallin
Brendan Burns
yuyang-ok
Bailey Hayes
Brian Jones