Implement Wasmtime's new API as designed by RFC 11. This is quite a large commit which has had lots of discussion externally, so for more information it's best to read the RFC thread and the PR thread.
Apparently `powershell Compress-Archive` produces zip files with
backslashes in filesnames which makes them unable to be extracted with
some Unix variants of extraction. For example [this failure][build] and
using macOS's built-in unzip feature it creates filenames with
backslashes in them rather than subdirectories.
[build]: https://github.com/bytecodealliance/wasmtime-go/runs/2680596219?check_suite_focus=true
This commit removes the publish step in GitHub actions, insteading
folding all functionality into the release build steps. This avoids
having a separately scheduled job after all the release build jobs which
ends up getting delayed for quite a long time given the current
scheduling algorithm.
This involves refactoring the tarball assembly scripts and refactoring the
github asset upload script too. Tarball assembly now manages everything
internally and does platform-specific bits where necessary. The upload
script is restructured to be run in parallel (in theory) and hopefully
catches various errors and tries to not stomp over everyone else's work.
The main trickiness here is handling `dev`, which is less critical for
correctness than than tags themselves.
As a small tweak build-wise the QEMU build for cross-compiled builders
is now cached unlike before where it was unconditionally built, shaving
a minute or two off build time.
* wasi-nn: turn it on by default
This change makes the wasi-nn Cargo feature a default feature. Previously, a wasi-nn user would have to build a separate Wasmtime binary (e.g. `cargo build --features wasi-nn ...`) to use wasi-nn and the resulting binary would require OpenVINO shared libraries to be present in the environment in order to run (otherwise it would fail immediately with linking errors). With recent changes to the `openvino` crate, the wasi-nn implementation can defer the loading of the OpenVINO shared libraries until runtime (i.e., when the user Wasm program calls `wasi_ephemeral_nn::load`) and display a user-level error if anything goes wrong (e.g., the OpenVINO libraries are not present on the system). This runtime-linking addition allows the wasi-nn feature to be turned on by default and shipped with upcoming releases of Wasmtime. This change should be transparent for users who do not use wasi-nn: the `openvino` crate is small and the newly-available wasi-nn imports only affect programs in which they are used.
For those interested in reviewing the runtime linking approach added to the `openvino` crate, see https://github.com/intel/openvino-rs/pull/19.
* wasi-nn spec path: don't use canonicalize
* Allow dependencies using the ISC license
The ISC license should be [just as permissive](https://choosealicense.com/licenses/isc) as MIT, e.g., with no additional limitations.
* Add a `--wasi-modules` flag
This flag controls which WASI modules are made available to the Wasm program. This initial commit enables `wasi-common` by default (equivalent to `--wasi-modules=all`) and allows `wasi-nn` and `wasi-crypto` to be added in either individually (e.g., `--wasi-modules=wasi-nn`) or as a group (e.g., `--wasi-modules=all-experimental`).
* wasi-crypto: fix unused dependency
Co-authored-by: Pat Hickey <pat@moreproductive.org>
This PR switches the default backend on x86, for both the
`cranelift-codegen` crate and for Wasmtime, to the new
(`MachInst`-style, `VCode`-based) backend that has been under
development and testing for some time now.
The old backend is still available by default in builds with the
`old-x86-backend` feature, or by requesting `BackendVariant::Legacy`
from the appropriate APIs.
As part of that switch, it adds some more runtime-configurable plumbing
to the testing infrastructure so that tests can be run using the
appropriate backend. `clif-util test` is now capable of parsing a
backend selector option from filetests and instantiating the correct
backend.
CI has been updated so that the old x86 backend continues to run its
tests, just as we used to run the new x64 backend separately.
At some point, we will remove the old x86 backend entirely, once we are
satisfied that the new backend has not caused any unforeseen issues and
we do not need to revert.
* Use stable Rust on CI to test the x64 backend
This commit leverages the newly-released 1.51.0 compiler to test the
new backend on Windows and Linux with a stable compiler instead of a
nightly compiler. This isolates the nightly build to just the nightly
documentation generation and fuzzing, both of which rely on nightly for
the best results right now.
* Use updated stable in book build job
* Run rustfmt for new stable
* Silence new warnings for wasi-nn
* Allow some dead code in the x64 backend
Looks like new rustc is better about emitting some dead-code warnings
* Update rust in peepmatic job
* Fix a test in the pooling allocator
* Remove `package.metdata.docs.rs` temporarily
Needs resolution of https://github.com/rust-lang/cargo/pull/9300 first
* Fix a warning in a wasi-nn example
* Add support for the experimental wasi-crypto APIs
The sole purpose of the implementation is to allow bindings and
application developers to test the proposed APIs.
Rust and AssemblyScript bindings are also available as examples.
Like `wasi-nn`, it is currently disabled by default, and requires
the `wasi-crypto` feature flag to be compiled in.
* Rename the wasi-crypto/spec submodule
* Add a path dependency into the submodule for wasi-crypto
* Tell the publish script to vendor wasi-crypto
Since downloading the wasi-nn artifacts take a bit of time, the example script's first argument serves as a directory to reuse for running this script. This change cleans up temporary directories only when a directory was not specified.
* Add an initial wasi-nn implementation for Wasmtime
This change adds a crate, `wasmtime-wasi-nn`, that uses `wiggle` to expose the current state of the wasi-nn API and `openvino` to implement the exposed functions. It includes an end-to-end test demonstrating how to do classification using wasi-nn:
- `crates/wasi-nn/tests/classification-example` contains Rust code that is compiled to the `wasm32-wasi` target and run with a Wasmtime embedding that exposes the wasi-nn calls
- the example uses Rust bindings for wasi-nn contained in `crates/wasi-nn/tests/wasi-nn-rust-bindings`; this crate contains code generated by `witx-bindgen` and eventually should be its own standalone crate
* Test wasi-nn as a CI step
This change adds:
- a GitHub action for installing OpenVINO
- a script, `ci/run-wasi-nn-example.sh`, to run the classification example
We do a `cargo fetch --locked` for most of our CI builds, but
`run-experimental-x64-ci.sh` was not doing this. As a result, some CI
runs seem to fail depending on which versions of crates they download. A
common failure mode is that two different versions of the `syn` crate
get into the build somehow, resulting in errors in wiggle/witx.
This change simply adds the `--locked` flag to the `cargo test` run for
the new x64 backend.
This commit extends our CI to produce release artifacts for the
x86_64-pc-windows-gnu target. This was originally motivate by the [go
extension] where it looks like the Go toolchain primarily interoperates
with MinGW, not with MSVC natively.
The support here turned out to be quite trivial, largely just adding the
configuration to make the release. I don't think we should necessarily
commit to this being a primary platform for Wasmtime at this time
though. If the support here regresses in the future for a
difficult-to-fix reason I think it would be fine to back out the
platform at least temporarily.
Note that this does not add a full test suite for the MinGW target, only
a release builder. This release builder does run tests, but not with
full debug assertions enabled.
[go extension]: https://github.com/bytecodealliance/wasmtime-go/issues/3Closes#1535
* Add AArch64 tests to CI
This commit enhances our CI with an AArch64 builder. Currently we have
no physical hardware to run on so for now we run all tests in an
emulator. The AArch64 build is cross-compiled from x86_64 from Linux.
Tests all happen in release mode with a recent version of QEMU (recent
version because it's so much faster, and in release mode because debug
mode tests take quite a long time in an emulator).
The goal here was not to get all tests passing on CI, but rather to get
AArch64 running on CI and get it green at the same time. To achieve that
goal many tests are now ignored on aarch64 platforms. Many tests fail
due to unimplemented functionality in the aarch64 backend (#1521), and
all wasmtime tests involving compilation are also disabled due to
panicking attempting to generate generate instruction offset information
for trap symbolication (#1523).
Despite this, though, all Cranelift tests and other wasmtime tests
should be runnin on AArch64 through QEMU with this PR. Additionally
we'll have an AArch64 binary release of Wasmtime for Linux, although it
won't be too useful just yet since it will panic on almost all wasm
modules.
* Review comments
* Remove the wasmtime Python extension from this repo
This commit removes the `crates/misc/py` folder and all associated
doo-dads like CI. This module has been rewritten to use the C API
natively and now lives at
https://github.com/bytecodealliance/wasmtime-py as discussed on #1390
* Remove the old wast and wasm2obj commands.
These are subsumed by the `wasmtime wast` and `wasmtime wasm2obj` commands.
Fixes#827.
* Remove wasm2obj install commands.
* Move all examples to a top-level directory
This commit moves all API examples (Rust and C) to a top-level
`examples` directory. This is intended to make it more discoverable and
conventional as to where examples are located. Additionally all examples
are now available in both Rust and C to see how to execute the example
in the language you're familiar with. The intention is that as more
languages are supported we'd add more languages as examples here too.
Each example is also accompanied by either a `*.wat` file which is
parsed as input, or a Rust project in a `wasm` folder which is compiled
as input.
A simple driver crate was also added to `crates/misc` which executes all
the examples on CI, ensuring the C and Rust examples all execute
successfully.
* Move the C API to a separate crate
This commit moves the C API from `crates/api/src/wasm.rs` to
`crates/capi/src/lib.rs` to be located in a separate crate. There's a
number of reasons for this:
* When a Rust program depends on the `wasmtime` crate, there's no need
to compile in the C API.
* This should improve compile times of the `wasmtime` crate since it's
not producing artifacts which aren't always used.
* The development of the C API can be guaranteed to only use the public
API of the `wasmtime` crate itself.
Some CI pieces are tweaked and this overall shouldn't have much impact
on users, it's intended that it's a cleanup/speedup for developers!
* Disable rustdoc/tests for capi
* Review feedback
* Add back in accidentally deleted comment
* More renamings
* Try to fix dotnet build
A small amount of cleanup for the top-level of this repository, where
the `installer` directory just had one misc file for the Windows
installer which we can put elsewhere inside of `ci`, another misc folder
but one that already has a few files in it.
This commit migrates wasmtime's CI infrastructure from Azure Pipelines
to Github Actions. Using Github Actions has a few benefits over other
offerings:
* Being natively integrated with Github means that there's no degree of
user account configuration or access control management, it's all
inherent via already existing Github permissions.
* Github Actions gives 20 parallel builders instead of Azure's 10 by
default, which is a nice boost to have!
Overall I've found Github Actions to feel a bit cleaner than Azure
Pipelines as well. Subjectively I've found the configuration to be more
readable and more pleasant to work with, although they're both just as
"powerful" I think. Additionally Github Actions has been pretty solid in
my own personal testing for a number of other projects.
The main trickiness with wasmtime's CI is the rolling `dev` release of
the master branch as well as binary releases for tags. Github Actions
doesn't have quite as much built in functionality as Azure Pipelines,
but Github Actions does have a nice feature where you can define the
code for an action locally rather than only using built-in actions.
This migration adds three local actions with some associated JS code to
run the action (currently it looks like it basically requires JS)
* An `install-rust` action papers over the gotchas about installing
Rust, allowing Rust installation to be a one-liner in the configuration.
* A `binary-compatible-builds` action allows easily configuring the
wheels and the binaries to be "more binary compatible" and handles
things like compilation flags on OSX and Windows while handling the
`centos:6` container on Linux.
* The `github-release` action is the logic using the `@actions/github`
JS package to orchestrate the custom way we manage rolling releases,
ensuring that a new release is made for the master branch under `dev`
(deleting the previous tag/release ahead of time) and then also
manages tagged releases by uploading them there.
I'm hoping that most of the inline actions here will largely go away.
For example `install-rust` should be simply `rustup update $toolchain`
once various environment issues are fixed on Github Actions runner
images. Additionally `github-release` will ideally migrate to something
like https://github.com/actions/create-release or similar once it has
enough functionality. I'm also hoping that the maintenance in the
meantime of these actions is pretty low-cost, but if it becomes an issue
we can look into other solutions!
This commit adds initial support for [WebAssembly Interface
Types][proposal] to wasmtime. This is all intended to be quite
experimental, so experimental in fact that even the name of the
[proposal] is still in flux. (this has otherwise been known as "host
bindings" or "webidl bindings" or "wasm bindings").
The goal of this commit is to start adding support the wasmtime set of
crates for WebAssembly Interface Types. A new `wasmtime-interface-types`
crate has been added with very basic support for dynamically invoking
and inspecting the various bindings of a module. This is in turn powered
by the `wasm-webidl-bindings` crate which is shared with the
`wasm-bindgen` CLI tool as a producer of this section.
Currently the only integration in `wasmtime`-the-binary itself is that
when passed the `--invoke` argument the CLI will now attempt to invoke
the target function with arguments as parsed from the command line
itself. For example if you export a function like:
fn render(&str) -> String
Then passing `--invoke render` will require one argument on the command
line, which is the first argument as a string, and the return value is
printed to the console. This differs from today's interpretation of
`--invoke` where it is a failure if the invoked function takes more than
one argument and the return values are currently ignored.
This is intended to also be the basis of embedding wasmtime in other
contexts which also want to consume WebAssembly interface types. A
Python extension is also added to this repository which implements the
`wasmtime` package on PyPI. This Python extension is intended to make it
as easy as `pip3 install wasmtime` to load a WebAssembly file with
WebAssembly Interface Types into Python. Extensions for other languages
is of course possible as well!
One of the major missing pieces from this is handling imported functions
with interface bindings. Currently the embedding support doesn't have
much ability to support handling imports ergonomically, so it's intended
that this will be included in a follow-up patch.
[proposal]: https://github.com/webassembly/webidl-bindings
Co-authored-by: Yury Delendik <ydelendik@mozilla.com>
Currently our Linux binaries aren't quite as portable as they otherwise could
be. There's two primary reasons for this, and one of them is that the binary is
produced in a relatively recent Linux distribution (Ubuntu 16.04) which means
it has a relatively recent requirement in terms of glibc versions. On
OSX/Windows we can set some flags to rely on older libc implementations, but on
Linux we have to actually build against an older version.
This commit switches the container for the build to CentOS 6 instead of the
default Ubuntu 16.04. The main trick here is also finding a C++11-capable
compiler to compile wabt. Turns out though there's a helpful tutorial for this
at https://edwards.sdsu.edu/research/c11-on-centos-6/ and it was as easy as
installing a few packages.
The second portability concern of our Linux binaries is that they link
dynamically to `libstdc++.so` which isn't always installed on target systems,
or even if it is it may be too old or have a different ABI. This is solved by
statically linking to `libstdc++.a` in the build on Azure by doing a bit of
trickery with libraries and what's available.
After these results the glibc requirements drops from 2.18 (released in 2013)
to 2.6 (released in 2007) and avoids the need for users to have libstdc++.so
installed. We may eventually want to investigate fully-static musl binaries,
but finding a musl compiler for C++ is something I'm not that good at, so I
figure this is probably good enough for now.
* Refactor Azure Pipelines config and tweak releases
* Extract out doc/rustfmt jobs into their own separate builders. Helps
avoiding having to skip tons of steps and can get failing results more
quickly.
* Extract out Rust installation logic to a dedicated template.
* Separate out the build/test job matrices, where one matrix runs tests
and another runs a full build
* Refactor release directory structure to follow a convention where
`foo.tar.gz` extracts to a folder called `foo` and follow unix-like
conventions and copy over the license/readme files into the release
tarballs.
* Swap order of build/test
Alex Crichton
Benjamin Bouvier
Andrew Brown
Chris Fallin
bjorn3
Frank Denis
Dan Gohman
Yury Delendik
Luiz Irber
Artur Jamro
Till Schneidereit