Turns out just having `git` wasn't enough. In the containers things are
running as `root` and `git` doesn't like that so try to convince git
that it should like it.
Precompiled artifacts for macOS show this for `wasmtime --version`
wasmtime-cli 24.0.0 (6fc3d274c 2024-08-20)
whereas for Linux they show
wasmtime-cli 24.0.0
and this is due to `git` not being available in the build environment on
Linux.
* Clean up dist build configuration
* Move updating `$PATH` to the `main.js` script which is the one that
mounts `/rust/bin` so that knowledge isn't spread around.
* Remove some unused env vars in docker containers.
* Forward cargo/rust-specific env vars to the build from outside of
containers to the build itself.
prtest:full
* Change how musl rustflags are configured
* More rustflags changes
* Review feedback
No major changes happen in this release so it's just bumping version
numbers. The goal here is to turn the crank and ensure that nothing else
breaks as part of this release process. There's a surprising number of
versions to update in-repo which we may want to make easier in the
future but for now this just manually updates.
It is planned that this does not actually break anyone in practice.
Older runtimes should support newer versions and Wasmtime after this PR
should continue to support older binaries as well.
Put another way this should not break things and if it does this will
get reverted and fixed. We've done our best to make sure this won't
break things, so let's find out in the real world now.
In #8873, we stopped tracking the wasi-nn's upstream WIT files
temporarily because it was not clear to me at the time how to implement
errors as CM resources. This PR fixes that, resuming tracking in the
`vendor-wit.sh` and implementing what is needed in the wasi-nn crate.
This leaves several threads unresolved, though:
- it looks like the `vendor-wit.sh` has a new mechanism for retrieving
files into `wit/deps`--at some point wasi-nn should migrate to use
that (?)
- it's not clear to me that "errors as resources" is even the best
approach here; I've opened [#75] to consider the possibility of using
"errors as records" instead.
- this PR identifies that the constructor for errors is in fact
unnecessary, prompting an upstream change ([#76]) that should
eventually be implemented here.
[#75]: https://github.com/WebAssembly/wasi-nn/pull/75
[#76]: https://github.com/WebAssembly/wasi-nn/pull/76
prtest:full
Now that we've got CI for Pulley throw in `wasmtime-fiber` as well with
a few extra fixes. This should also help improve the compilation
experience of Wasmtime to 32-bit platforms to hit a more first-class
error about unsupported platforms.
This commit updates `build-test-matrix.js` to calculate PR tests to
never early-return `false`. The current structure for PR tests will
iterate over most matrix entries and filter them out based on whether
they're applicable or not given the commit message and modified files.
The thinking is that an early-return of `false` might be a
false-negative where a later filter enables the test. By only
early-returning `true` this enables any passing predicate to cause the
tests to be run on a PR. Note that this has no effect on merge queues
since those always and unconditionally run everything.
* CI: Check that Pulley builds for `no_std` targets
* CI: Test Pulley on 32-bit targets
This adds CI testing for Pulley on the following 32-bit targets:
* `i686-unknown-linux-gnu`
* `armv7-unknown-linux-gnueabihf`
The previous commit added checks that Pulley builds for `no_std` targets.
Our existing test sharding means that it is already tested on a big-endian
platform (`s390x-unknown-linux-gnu`) as well as on different OSes (Linux,
Windows, and macOS).
So altogether we are now testing a pretty wide range of portability dimensions
for Pulley in CI:
* 32- vs. 64-bit architectures
* `std` vs. `no_std`
* Little- vs. big-endian
* Linux vs. Windows vs. macOS
I think our CI coverage for Pulley is in a pretty good place now!
Co-Authored-By: Alex Crichton <alex@alexcrichton.com>
---------
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
* Add wasi adapter provider template which is materialised in CI
* Add rustfmt component to adapter CI
* Draft an extra publish step for the adapter provider
* Check adapter provider in a separate step with adapter artifacts
* Use artifact downloads in the publish action as well
* Record results from adapter provider step as well
* Refactor to use composite actions
* Add missing shell property
* Fix spelling mistake
* Try using the env context
* Use Ubuntu-16.04 for x86_64-linux binary-compatible-builds
* Revert "Use Ubuntu-16.04 for x86_64-linux binary-compatible-builds"
This reverts commit 5625941dee.
* Use AlmaLinux 8
prtest:full
* wasi-nn: use resources
Recent discussion in the wasi-nn proposal (see [wasi-nn#59], e.g.) has
concluded that the right approach for representing wasi-nn "things"
(tensors, graph, etc.) is with a component model _resource_. This
sweeping change brings Wasmtime's implementation in line with that
decision.
Initially I had structured this PR to remove all of the WITX-based
implementation (#8530). But, after consulting in a Zulip [thread] on
what other WASI proposals aim to do, this PR pivoted to support _both_`
the WITX-based and WIT-based ABIs (e.g., preview1 era versus preview2,
component model era). What is clear is that the WITX-based specification
will remain "frozen in time" while the WIT-based implementation moves
forward.
What that means for this PR is a "split world" paradigm. In many places,
we have to distinguish between the `wit` and `witx` versions of the same
thing. This change isn't the end state yet: it's a big step forward
towards bringing Wasmtime back in line with the WIT spec but, despite my
best efforts, doesn't fully fix all the TODOs left behind over several
years of development. I have, however, taken the liberty to refactor and
fix various parts as I came across them (e.g., the ONNX backend). I plan
to continue working on this in future PRs to figure out a good error
paradigm (the current one is too wordy) and device residence.
[wasi-nn#59]: https://github.com/WebAssembly/wasi-nn/pull/59
[thread]: https://bytecodealliance.zulipchat.com/#narrow/stream/219900-wasi/topic/wasi-nn's.20preview1.20vs.20preview2.20timeline
prtest:full
* vet: audit `ort`-related crate updates
* Simplify `WasiNnView`
With @alexcrichton's help, this change removes the `trait WasiNnView`
and `struct WasiNnImpl` wrapping that the WIT-based implementation used
for accessing the host context. Instead, `WasiNnView` is now a `struct`
containing the mutable references it needs to make things work. This
unwraps one complex layer of abstraction, though it does have the
downside that it complicates CLI code to split borrows of `Host`.
* Temporarily disable WIT check
* Refactor errors to use `trappable_error_type`
This change simplifies the return types of the host implementations of
the WIT-based wasi-nn. There is more work to be done with errors, e.g.,
to catch up with the upstream decision to return errors as resources.
But this is better than the previous mess.
This reduces the size of wasi_snapshot_preview1.command.wasm from 79625
bytes to 75029 bytes for a total win of 4596 bytes. Of this reduction
enabling LTO is responsible for 3103 bytes, while enabling bulk-memory
is responsible for 1493 bytes
* wasi-adapter: Implement provider crate that embeds the adapter binaries
* Upgrade wasi adapters to the latest version
* Update adapter docs
* Recompile asi adapters with 1.78
* Recompile wasi adapters with 1.79
* Add some debugging to adapter build script
* Fix script debugging
* Compute wasi adapter version based on latest adapter commit hash
* Try to bless wasi adapters again
* Try to work around CI auto-merges
* Revert to just using workspace version
* Add the wasi adapter provider to the crate publication list
* Use wasi adapter provider in artifacts test + explicit MSRV in CI
* Explicit adapter crate version
* Small fix
* Remove version info from adapter metadata
* Check but don't install rust toolchain in build script
* Bless after rebase
---------
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
* ci: Add support for RISC-V Zicond extension
* riscv64: Add Zicond Extension and Instructions
* riscv64: Add base rules for czero instruction
* riscv64: Support swapped arguments for `czero` instruction
* riscv64: Add generic select lowering using zicond
* riscv64: Update ISA Tests
* riscv64: Fix incorrect zero swap rule
Additionally add some checks to prevent a stack overflow in the zero
swap rules
* riscv64: Gate select_xreg zero rotation rules to zicond
* riscv64: Add ZiCond select base case for all integer compares
* Refactor how release notes are managed
This commit updates how Wasmtime manages release notes across released
versions of Wasmtime. One of the most onerous parts about releases right
now is writing release notes in all the locations and making sure
they're all up-to-date and in-sync. This is inevitably forgotten in some
cases and various pieces will slip through the cracks. The basic idea of
this PR is to change our release notes to only document the release
branch that they're on. All historical release notes are relegated to
historical branches.
With this change there's no longer any need to backport or forward-port
release notes for any changes. Instead release notes are written once
on one branch and that's it.
The major downside of this change is that there's no easy way to get a
bird's eye view of all release notes for Wasmtime any more. If necessary
that could theoretically be automated in the future (like
https://releases.rs/), but for now this feels like an acceptable
compromise to make releases much easier.
The contents of this PR are to update `RELEASES.md` with back-links to
historical release notes as well as the various pieces of automation we
have about managing release notes.
* Add more notes about the release process
* Fix a typo
* Make it more obvious that patch releases are documented
* Add release binaries for x86_64-musl
This was requested in bytecodealliance/wasmtime-py#237 and shouldn't
cost us too much in terms of CI resources and maintenance overhead.
* Fix combining rustflags
prtest:full
* Refactor installation of C API and features supported
This commit overhauls and refactors the management of the building of
the C API. Instead of this being script-based it's now done entirely
through CMake and CMake is the primary focus for building the C API. For
example building the C API release artifacts is now done through CMake
instead of through a shell script's `cargo build` and manually moving
artifacts.
The benefits that this brings are:
* The C API now properly hides symbols in its header files that weren't
enabled at build time. This is done through a new build-time generated
`conf.h` templated on a `conf.h.in` file in the source tree.
* The C API's project now supports enabling/disabling Cargo features to
have finer-grained support over what's included (plus auto-management
of the header file).
* Building the C API and managing it is now exclusively done through
CMake. For example invoking `doxygen` now lives in CMake, installation
lives there, etc.
The `CMakeLists.txt` file for the C API is overhauled in the process of
doing this. The build now primarily matches on the Rust target being
built rather than the host target. Additionally installation will now
install both the static and shared libraries instead of just one.
Additionally during this refactoring various bits and pieces of
Android-specific code were all removed. Management of the C toolchain
feels best left in scope of the caller (e.g. configuring `CC_*` env vars
and such) rather than here.
prtest:full
* Don't use `option` for optional strings
* Invert release build check
Also adjust some indentation
* Fix more indentation
* Remove no-longer-used variable
* Reduce duplication in feature macro
* Add Android release binaries to CI
This commit is inspired by #6480 and historical asks for Android
binaries. This does the bare minimum necessary to configure C compilers
such that we can produce binaries but I'll admit that I'm no Android
developer myself so I have no idea if these are actually suitable for
use anywhere. Otherwise though this build subsumes the preexisting check
in CI that the build works for Android, so that part is removed too.
This additionally changes how the NDK is managed from before. Previously
a GitHub Action was used to download Java and the NDK and additionally
used the `cargo ndk` subcommand. That's all removed now in favor of
configuring C compilers directly with a pre-installed version of the NDK
which should help reduce the CI dependencies a bit.
* Review comments
* List Android as tier 3 target
* Bump Wasmtime's MSRV to 1.76.0
* Update Rust in CI to 1.78.0, the current stable
* Update nightly tests to the latest nightly
prtest:full
* Fix check-cfg with nightly
* More check-cfg fixes
* Remove an async cfg
This is no longer specified for the root crate.
* Move definition of Wasmtime's nightly into one place
Don't change a bunch of places when this is updated, try to update just
one single location instead.
* riscv64: Add minimal support for Zfa Extension
This commit adds support for the Zfa extension and implements the `fminm`/`fmaxm` instructions and lowerings.
The Zfa extension provides additional floating point related instructions not included in either F/D extensions.
* riscv64: Enable Zfa in CI runner
* wasi-nn: remove Git submodule
To more closely align with the conventions in the `wasmtime-wasi` and
`wasmtime-wasi-http` crates, this change removes the Git submodule that
previously provided the WIT and WITX files for `wasmtime-wasi-nn`. Like
those other crates, the syncing of wasi-nn WIT and WITX files will
happen manually for the time being. This is the first PR towards
upgrading the wasi-nn implementation to match recent spec changes and
better preview2-ABI compatibility.
prtest:full
* ci: auto-vendor the wasi-nn WIT files
Looks like GitHub is changing `macos-latest` to arm64 so change the test
builder that test x64 macos to `macos-13` which is the last builder that
wasn't arm64. Additionally drop `macos-14` from the C API tests since
testing `macos-latest` should be sufficient enough.
* Remove wasm-c-api submodule
This submodule hasn't been updated in ~3 years at this point and we
additionally don't need most of the submodule. Instead add a script to
copy the files we need and verify in CI that the files are up-to-date.
This also makes using the C API a bit nicer where you don't have to have
two `include` directories with a Wasmtime source tree, just one
suffices.
* Don't format wasm.h{,h} vendored files
* Add a script to vendor WIT files
This is a follow-up to #7929 where it was concluded that we want to
retain textual WIT files in this repository. Currently today we have no
automated checks for vendoring and re-vendoring requires hand-editing.
To help incrementally improve this situation this commit adds a script
to do all the vendoring for us. This way CI can verify that everything
is generated with respect to the output of this script.
* Review comments
* add an initial implemenation for onnxruntime backend of wasi-nn
Signed-off-by: David Justice <david@devigned.com>
* vet: audit ONNX dependencies
This change is the result of a long slog through the dependencies of the
`ort` library. The only missing dependency is `compact_str`, which needs
further discussion.
* vet: add ONNX audit entry for compact_str 0.7.1
Signed-off-by: David Justice <david@devigned.com>
* refactor tests to break out onnx and openvino
Signed-off-by: David Justice <david@devigned.com>
* mark wasi-nn onnx example as publish false
Signed-off-by: David Justice <david@devigned.com>
* update the ONNX classification example
* do not use wasi-nn onnx feature if riskv or s390
Signed-off-by: David Justice <david@devigned.com>
* prtest:full fix running WASI-NN ONNX tests across arch os
Signed-off-by: David Justice <david@devigned.com>
---------
Signed-off-by: David Justice <david@devigned.com>
Co-authored-by: Andrew Brown <andrew.brown@intel.com>
* Test with aarch64 macOS on CI
Rebase of #7129 with the `macos-14` string which GitHub indicates should
now work for public repos.
prtest:full
* Fix syntax
* mpk: enable MPK if available in CI
After discussing several options for testing MPK in CI, this stopgap
measure at least provides some coverage. Other options include
maintaining a separate MPK-enabled CI runner (configuration is not
transparent, hard to maintain) or running the MPK-enabled tests in a
system-mode QEMU VM (tricky to get right, also hard to maintain). For
now, those of us at the Cranelift meeting agreed this at least gets some
CI testing for the MPK bits, which shouldn't be changing too often.
Since not all GitHub runners have MPK available, we only set the
`WASMTIME_TEST_FORCE_MPK` environment variable when it is. This change
also emits a warning to the GitHub logs in either case for easier
troubleshooting (e.g., to attempt to provide context if MPK logic breaks
and is only found in a later CI run).
prtest:full
* review: create a separate matrix entry
As requested by @alexcrichton, this change limits the MPK testing (and
the associated warnings) to a new matrix target: `Test Linux x86_64 with
MPK`.
* wasi-nn: add test programs
This change adds new test programs for wasi-nn in a way fits in with the
existing WASI test infrastructure. The code is not new, though: this
reuses the wasi-nn `examples`, which are currently used by the
`run-wasi-nn-example.sh` CI script. Eventually the examples will be
removed in favor of these tests.
Because wasi-nn's component model support is still in flight, this
change also skips the generation of components for `nn_`-prefixed tests.
* wasi-nn: add `testing` module
This testing-only module has code (i.e., `check_test!`) to check whether
OpenVINO and some test artifacts are available. The test artifacts are
downloaded and cached if not present, expecting `curl` to be present on
the command line (as discussed in the previous version of this, #6895).
* wasi-nn: run `nn_*` test programs as integration tests
Following the pattern of other WASI crates, this change adds the
necessary infrastructure to run the `nn_*` files in
`crates/test-programs` (built by `test-program-artifacts`). These tests
are only run when two sets of conditions are true:
- statically: we only run these tests where we expect OpenVINO to be
easy to install and run (e.g., the `cfg_attr` parts)
- dynamically: we also only run these tests when the OpenVINO libraries
can be located and the model artifacts can be downloaded
* ci: install OpenVINO for running wasi-nn tests
prtest:full
* vet: certify the `wasi-nn` crate
* ci: remove wasi-nn test script
This commit aims to address a discrepancy in Wasmtime where the world
supported by `wasmtime serve` is too large today. This includes
WIT interfaces which are not specified in `wasi:http/proxy` such as
`wasi:filesystem/types`, aka access to a filesystem.
This commit slims down `wasmtime serve` to, by default, only supporting
the `wasi:http/proxy` world. Like with `wasmtime run` various CLI flags
can be passed to enable more interfaces, however:
* `-Scommon` - this enables "common" interfaces such as
`wasi:filesystem`, `wasi:sockets`, and `wasi:cli/*`.
* `-Snn` - this enables wasi-nn
It's expected that more will get extended here over time too.
This change is enabled by a third build of the adapter, a "proxy" mode.
In this mode most functions are cfg'd to return `ERRNO_NOTSUP` to
indicate that the runtime does not support it. Notably this includes the
filesystem, arguments, and environment variables.
This change is tested by updating all `api_proxy*` tests to use this new
adapter which is now required that many previous interfaces are no
longer present by default in the proxy world.
With a custom standard library disabling all features means disabling
some support for feature-detection macros of the native platform. This
meant that `wasmtime compile` output locally wasn't runnable in
`wasmtime-min run` because it couldn't correctly detect that features
were in fact available.
* riscv64: Extend distance trampolines can jump
Use a PIC-friendly set of instructions to enable destination of the
trampoline to be more than 4k away from the tail call site of the
trampoline itself.
* Build "min" artifacts on CI
This commit updates the binary artifacts produced by CI to include "min"
builds where all default features are disabled. Additionally all the
stops are pulled in terms of build flags, nightly versions, etc, to get
a build that is as small as possible without actual source code changes.
This effectively codifies the instructions in #7282 into an easily
downloadable artifact.
No new tarballs are created for github releases but instead tarballs
that previously had a `wasmtime` executable for example now have a
`wasmtime-min` executable. Furthermore the C API which previously had
`libwasmtime.so` for example now has `libwasmtime-min.so`. The intention
is that the minimum-size artifacts are handy for determining a rough
size of Wasmtime but they're not so important that it seems worthwhile
to dedicate entire release entries for.
CI is refactored to support these minimum builds with separate builders.
This means that a single tarball produced as a final result is actually
two separate tarballs merged together, one for the normal build we do
today plus a new "min" tarball produced on the new "min" builders.
Various scripts and CI organization has been adjusted accordingly.
While here I went ahead and enabled `panic=abort` and debuginfo
stripping in our current release artifacts. While this doesn't affect a
whole lot it's less to upload to GitHub Actions all the time.
* Fix Windows unzip
This commit changes the `--dir` argument on the `wasmtime` CLI to be
`HOST::GUEST` rather than `GUEST::HOST`. This matches Docker for example
and is a little more consistent with only `--dir path` where the first
argument is always treated as a host directory.
In terms of breaking-ness the movement from `--mapdir` to `--dir` hasn't
been released with Wasmtime 14 yet so my hope is that this can land on
both `main` and Wasmtime 14.0.0 before it's released to avoid any
breakage other than existing scripts migrating from `--mapdir` to
`--dir`.
* Update MSRV to 1.71.0
With the 1.73.0 release today update our MSRV to a one-higher version.
This will cause 1.73.0 to be tested in CI.
* Fix a warning on windows
* Don't store tables as `&'static mut`
Use `SendSyncPtr` instead (a small wrapper around `NonNull`) which
appeases MIRI.
* Refactor the test-programs test suite
This commit is a large refactoring that reorganizes `test-programs` and
how we tests wasms in Wasmtime. Often writing tests requires complicated
interactions with the guest which can't be done via hand-written `*.wat`
syntax and requires a compiler to get engaged. For this purpose Wasmtime
currently has the `crates/test-programs/*` test suite which builds files
from source and then runs the tests. This has been somewhat cumbersome
in the past though and it's not been easy to extend this over time, so
this commit attempts to address this.
The scheme implemented in this PR looks like:
* All wasm test programs live in `crates/test-programs/src/bin/*.rs`.
All of them, no exceptions.
* Wasm tests have shared support located at
`crates/test-programs/src/lib.rs` and its submodules, such as bindings
generation for WASI.
* Wasm tests are built by a new `crates/test-programs/artifacts` crate.
This crate compiles modules and additionally creates components for
all test programs. The crate itself only records the path to these
outputs and a small amount of testing support, but otherwise doesn't
interact with `wasmtime`-the-crate itself.
* All tests in `crates/test-programs/tests/*.rs` have moved. For example
wasi-http tests now live at `crates/wasi-http/tests/*.rs`. Legacy
tests of wasi-common now live at `crates/wasi-common/tests/*.rs`.
Modern tests for preview2 live at `crates/wasi/tests/*.rs`.
* Wasm tests are bucketed based on their filename prefix. For example
`preview1_*` is tested in wasi-common and wasmtime-wasi. The
`preview2_*` prefix is only tested with wasmtime-wasi, however.
* A new `cli_*` prefix is used to execute tests as part of
`tests/all/main.rs`. This is a new submodule in
`tests/all/cli_tests.rs` which executes these components on the
command line. Many old "command" tests were migrated here.
* Helper macros are generated to assert that a test suite is run in its
entirety. This way if a `preview1_*` test is added it's asserted to
get added to both wasi-common and wasmtime-wasi in the various modes
they run tests.
Overall this moved a number of tests around and refactored some edges of
the tests, but this should not lose any tests (except one that wasn't
actually testing anything). Additionally the hope is that it's much
easier to add tests in the future. The process is to add a new file in
`crates/test-programs/src/bin/*.rs` named appropriately. For example a
preview2 executable is `preview2_*` and a CLI tests is `cli_*`. When
building the test suite an error is generated in the appropriate module
then of "please write a test here", and then a test is written in the
same manner as the other tests in the module.
* Remove no-longer-needed fetches
prtest:full
* I'm worried wasi is running low on semicolons
* Add the WASI target in all CI actions
* Add unknown-unknown target on all CI builders too
* Fix building test artifacts under miri
Need to avoid wrappers for these cross-compiled targets
* Break circular dependency for packaging
Don't use the workspace dep for `wasmtime-wasi` since it injects a
version, instead use a `path = '..'` dependency to fool Cargo into
dropping the dependency during the package phase.
* Fix some merge conflicts with tests
* Fix rebase for new tests
* Remove stray comment
* Fix some flaky tests
* Fix network tests in synchronous mode
This commit is an attempt to fix some networking tests in synchronous
mode in our test suite. Currently networking tests don't actually run in
synchronous mode on CI which is why no failures have been surfaced yet,
but the refactoring in #7182 is going to start doing this.
Currently the `udp_sample_application.rs` test blocks infinitely in
synchronous mode for me locally, most of the time. This appears to be an
interaction between how Tokio handles readiness and how we're
entering the event loop. We're effectively entering the Tokio event loop
with a future that's always ready which ends up starving Tokio of
otherwise performing its background work such as updating flags for
readiness of reading/writing.
The fix here is to add a yield at the start of an `in_tokio` block which
is used in synchronous mode. This is a kludge fix but the intention is
to enable Tokio to have a chance to update readiness flags and process
events from epoll/kqueue/etc.
An additional fix to this issue is WebAssembly/wasi-sockets#64 where the
test is waiting on `READABLE` or `WRITABLE`, but in this specific case
it should only wait on `READABLE`. If it waited on just this then that
would also fix this issue. Nevertheless having a `yield_now` is expected
to have little-to-no overhead and otherwise fix this edge case of an
always-ready future.
* Fix passing empty arguments on the CLI
* Add another blocking accept
* Update crates/test-programs/src/bin/api_proxy.rs
Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>
---------
Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>
Alex Crichton
Andrew Brown
Afonso Bordado
Xinzhao Xu
Nick Fitzgerald
Juniper Tyree
Trevor Elliott
bjorn3
FrankReh
David Justice
vuittont60