* Add wasmtime-c-api-impl to the list of crates to publish
* Enable rustdoc and publishing for c-api crate
* Provide paths to c-api headers as cargo links metadata
* Add a README section about using wasm-c-api in a rust crate
* In C API doc comment, mention use case for crates w/ C bindings
* Enable publishing for wasmtime-c-api-macros (prtest:full)
* Move c-api crates later in the publishing sequence (prtest:full)
Looks like it's been awhile since our last update. I've updated the
dependency here and additionally vetted the various dependency upgrades.
I notably did not vet `zstd-sys` since that would require vetting all of
zstd's C implementation itself which I am not, nor do I suspect many of
are, equipped to do. In lieu of that I've updated the exemption of
`zstd-sys` to the newer version.
Closes#7869
* update Wasmi fuzzing oracle to version 0.31.0
This allows us to enable the bulk-memory, reference-types and tail-call Wasm proposals for the Wasmi fuzzing oracle.
* apply rustfmt
* be more explicit about supported Wasm features
* align Wasmi config to input config
I am not sure if this is how it is intended to be used. Please review and provide feedback.
* remove duplicate threads_enabled
* remove min and max tables
We can do this since Wasmi supports reference-types Wasm proposal.
* add comment about config mutation
* use Wasmi v0.31.1
* be more explicit about supported Wasm features
* add comment about config mutation
* update wasmi_arena to v0.4.1
* Move `jit` crate to `environ`
Move the platform agnostic parts of the crate `wasmtime-jit` to
`wasmtime-environ`. This is the first part of the refactoring discussed
here: https://github.com/bytecodealliance/wasmtime/issues/7652 and a
follow up will move the remaining parts of `wasmtime-jit` so that the
crate can be deleted.
* Move `jit` crate to `wasmtime`
Move the remaining parts of `wasmtime-jit` to the `wasmtime` crate and
remove `wasmtime-jit`. This is part of the refactoring discussed in
https://github.com/bytecodealliance/wasmtime/issues/7652.
* undo toml formatting
* Trigger pipeline: prtest:full
* Remove `jit` directory
* move `ProfilingAgent` out of `profiling` feature
* add links to ELF_NAME_DATA
* Update the wasm-tools family of crates
Brings in support for validating gc instructions, but they're all left
disabled for now.
* Update fuzz test case generation
* More test fixes, remove stray files
* More test fixes
* Rebase
* Update several dependencies to windows-sys 0.52.
Update cap-std, rustix, io-extras, errno, fd-lock, fs-set-times, and
winx to versions which use windows-sys 0.52.
I started out hoping that everything could be updated to
windows-sys 0.52, however tokio and mio have not yet updated to
windows-sys 0.52 due to supporting an older MSRV.
* Link the C++ examples with CoreFoundation.
This is needed due to the iana-time-zone support.
prtest:full
* Fix compilation on non-Apple platfoms.
* Fix compilation on Apple platforms.
* windows-sys 0.52
Not locally tested due to C dependencies which I don't have the tooling for.
The intent is to use GH's CI.
* MEMORYMAPPEDVIEW_HANDLE -> MEMORY_MAPPED_VIEW_ADDRESS
* .Value instead of cast
* fmt
* Remove cast to MEMORY_MAPPED_VIEW_ADDRESS with struct creation
* Cast ptr from *mut u8 from *mut void
* Remove CI changes which were supposed to be on a distinct branch
* Update binutils when the target is x86_64-pc-windows-gnu
* Escape paths in CI
* Update .github/workflows/main.yml
Co-authored-by: Chris Denton <christophersdenton@gmail.com>
* Correct type from p2sh to pwsh
* Use | to correct YAML parsing error
* Fix msys2's installation of MinGW
* Finally fix msys2 handling
---------
Co-authored-by: Chris Denton <christophersdenton@gmail.com>
* wasi-nn: add test programs
This change adds new test programs for wasi-nn in a way fits in with the
existing WASI test infrastructure. The code is not new, though: this
reuses the wasi-nn `examples`, which are currently used by the
`run-wasi-nn-example.sh` CI script. Eventually the examples will be
removed in favor of these tests.
Because wasi-nn's component model support is still in flight, this
change also skips the generation of components for `nn_`-prefixed tests.
* wasi-nn: add `testing` module
This testing-only module has code (i.e., `check_test!`) to check whether
OpenVINO and some test artifacts are available. The test artifacts are
downloaded and cached if not present, expecting `curl` to be present on
the command line (as discussed in the previous version of this, #6895).
* wasi-nn: run `nn_*` test programs as integration tests
Following the pattern of other WASI crates, this change adds the
necessary infrastructure to run the `nn_*` files in
`crates/test-programs` (built by `test-program-artifacts`). These tests
are only run when two sets of conditions are true:
- statically: we only run these tests where we expect OpenVINO to be
easy to install and run (e.g., the `cfg_attr` parts)
- dynamically: we also only run these tests when the OpenVINO libraries
can be located and the model artifacts can be downloaded
* ci: install OpenVINO for running wasi-nn tests
prtest:full
* vet: certify the `wasi-nn` crate
* ci: remove wasi-nn test script
* Gate more functionality behind `debug-builtins`
This commit extends the gating behavior of the preexisting
`debug-builtins` Cargo feature to cover more GDB-related functionality
associated with debugging. This can additionally slim down the set of
exposed symbols from Wasmtime over the default with them included.
* Move timing in Cranelift behind a Cargo feature
This commit adds a `timing` feature to the `cranelift-codegen` crate
which is enabled by default. This feature gates the timing functionality
in Cranelift to enable turning it off if desired. The goal of this
commit is to remove a system dependency on `Instant` for possibly
esoteric environments.
* Consolidate platform-specific definitions in Wasmtime
Prior to this commit Wasmtime did not not have a style or system for
containing platform-specific logic in files. The goal of this commit is
to consolidate all platform-specific functionality into one location to
make it easier to port Wasmtime to new systems. This commit creates a
`sys` module within the `wasmtime-runtime` crate which conditionally
defines all of the platform support that Wasmtime requires, namely
things related to virtual memory management and trap handling. Many of
the previous `unix.rs` files interspersed throughout the tree are now
all located in a single `unix` directory. This additionally helps split
out `miri`-specific functionality by pretending `miri` is its own
platform.
This change additionally goes through `#[cfg]` directives throughout
`wasmtime-runtime`, `wasmtime-jit`, and `wasmtime` itself to place all
of this target-specific functionality within this `sys` directory. The
end state is that there are two new top-level modules in the
`wasmtime-runtime` crate:
* `arch` - this conditionally defines architecture-specific logic such
as the state used by backtraces, libcalls, etc.
* `sys` - this conditionally defines OS or platform-specific logic such
as virtual memory management.
One secondary goal of this commit is to enable the ability to easily
add new platforms to Wasmtime, even if it's in a fork of Wasmtime.
Previously patches might have to touch a good number of locations where
now they'd ideally only have to touch `sys/mod.rs` to declare a new
platform and `sys/$platform/*.rs` to define all the functionality.
* Fix build on Windows
prtest:full
* Fix some build warnings
* Fix miri build
* Include debug-builtins when testing
* Try to fix Windows tests
* Fix warnings on miri
* Fix miri build
* Fix debug-builtins feature
* More feature fixes
* Rename `fd` field
* Fix speeling
* Review comments
* mpk: allow checking for MPK without a config instance
It is inconvenient to have to construct a `PoolingAllocationConfig` in
order to check if memory protection keys are available. This removes
the unused `&self` restriction.
* mpk: improve logging of calculated slab layout
When double-checking the slab layout calculations it is quite convenient
to see the total slab size. This helps in correlating with mapped
regions.
* mpk: add an example testing the memory limits
This adds an example that can be run with `cargo run --example mpk`. Not
only does the example demonstrate how to build a pool-allocated engine
that uses MPK, it performs an exponential search to find the maximum
number of slots the system can support, with and without MPK.
* review: document Linux requirement
* review: `env_logger::init`
* review: replace `proc-maps` with manual parsing
* vet: audit `bytesize`
* fix: provide `main` for non-Linux systems
* fix: move `cfg` to avoid unused code
This commit updates to the latest wasm-tools and `wit-bindgen` to bring
the family of crates forward. This update notably includes Nick's work
on packed indices in the `wasmparser` crate for validation for the
upcoming implementation of GC types. This meant that translation from
`wasmparser` types to Wasmtime types now may work with a "type id"
instead of just a type index which required plumbing not only Wasmtime's
own type information but additionally `wasmparser`'s type information
throughout translation.
This required a fair bit of refactoring to get this working but no
change in functionality is intended, only a different way of doing
everything prior.
* Configure Rust lints at the workspace level
This commit adds necessary configuration knobs to have lints configured
at the workspace level in Wasmtime rather than the crate level. This
uses a feature of Cargo first released with 1.74.0 (last week) of the
`[workspace.lints]` table. This should help create a more consistent set
of lints applied across all crates in our workspace in addition to
possibly running select clippy lints on CI as well.
* Move `unused_extern_crates` to the workspace level
This commit configures a `deny` lint level for the
`unused_extern_crates` lint to the workspace level rather than the
previous configuration at the individual crate level.
* Move `trivial_numeric_casts` to workspace level
* Change workspace lint levels to `warn`
CI will ensure that these don't get checked into the codebase and
otherwise provide fewer speed bumps for in-process development.
* Move `unstable_features` lint to workspace level
* Move `unused_import_braces` lint to workspace level
* Start running Clippy on CI
This commit configures our CI to run `cargo clippy --workspace` for all
merged PRs. Historically this hasn't been all the feasible due to the
amount of configuration required to control the number of warnings on
CI, but with Cargo's new `[lint]` table it's possible to have a
one-liner to silence all lints from Clippy by default. This commit by
default sets the `all` lint in Clippy to `allow` to by-default disable
warnings from Clippy. The goal of this PR is to enable selective access
to Clippy lints for Wasmtime on CI.
* Selectively enable `clippy::cast_sign_loss`
This would have fixed#7558 so try to head off future issues with that
by warning against this situation in a few crates. This lint is still
quite noisy though for Cranelift for example so it's not worthwhile at
this time to enable it for the whole workspace.
* Fix CI error
prtest:full
* Update `errno` crate to drop unused dependency
Small change to our dependency graph.
* Update rayon-core to drop some dependency edges
Just a bit of pruning for our dependency graph.
* Update num_cpus dependency to drop a dependency
Minor pruning of our dependency graph.
Pull in bytecodealliance/wasm-tools#1277 to switch the defaults for some
WIT behaviors:
* Semicolons are now required by default in WIT files
* Wasm-encoded WIT packages now use the "new" format by default.
WIT files without semicolons can still be parsed with
`WIT_REQUIRE_SEMICOLONS=0`. The "old" format for wasm-encoded WIT
packages can be emitted via `WIT_COMPONENT_ENCODING_V2=0`. Note that for
wasm-encoded WIT packages both the old and the new format can be decoded
irregardless of env vars.
* Update wasm-tools crates
This commit updates the wasm-tools family of crate for a number of
notable updates:
* bytecodealliance/wasm-tools#1257 - wasmparser's ID-based
infrastructure has been refactored to have more precise types for each
ID rather than one all-purpose `TypeId`.
* bytecodealliance/wasm-tools#1262 - the implementation of
"implementation imports" for the component model which both updates
the binary format in addition to adding more syntactic forms of
imports.
* bytecodealliance/wasm-tools#1260 - a new encoding scheme for component
information for `wit-component` in objects (not used by Wasmtime but
used by bindings generators).
Translation for components needed to be updated to account for the first
change, but otherwise this was a straightforward update.
* Remove a TODO
* Add compatibility shims for Wasmtime 13 CLI
This commit introduces a compatibility shim for the Wasmtime 13 CLI and
prior. The goal of this commit is to address concerns raised in #7336
and other locations as well. While the new CLI cannot be un-shipped at
this point this PR attempts to ameliorate the situation somewhat through
a few avenues:
* A complete copy of the old CLI parser is now included in `wasmtime` by
default.
* The `WASMTIME_NEW_CLI=0` environment variable can force usage of the
old CLI parser for the `run` and `compile` commands.
* The `WASMTIME_NEW_CLI=1` environment variable can force usage of the
new CLI parser.
* Otherwise both the old and the new CLI parser are executed. Depending
on the result one is selected to be executed, possibly with a warning
printed.
* If both CLI parsers succeed but produce the same result, then no
warning is emitted and execution continues as usual.
* If both CLI parsers succeed but produce different results then a
warning is emitted indicating this. The warning points to #7384 which
has further examples of how to squash the warning. The warning also
mentions the above env var to turn the warning off. In this situation
the old semantics are used at this time instead of the new semantics.
It's intended that eventually this will change in the future.
* If the new CLI succeeds and the old CLI fails then the new semantics
are executed without warning.
* If the old CLI succeeds and the new CLI fails then a warning is issued
and the old semantics are used.
* If both the old and the new CLI fail to parse then the error message
for the new CLI is emitted.
Note that this doesn't add up to a perfect transition. The hope is that
most of the major cases of change at the very least have something
printed. My plan is to land this on `main` and then backport it to the
14.0.0 branch as a 14.0.3 release.
* Wordsmith messages
* Update messages
* More wording updates
* Fix grammar
* More updates
* Update some crates to align on latest `rustix`
This commit addresses some dependabot warnings showing up on the
Wasmtime repository by updating all dependencies to using the latest
`rustix` release.
* Debug CI
prtest:full
* Downgrade rustix to 0.38.14
Works around bytecodealliance/rustix#901
This commit introduces a wrapper crate which is now the new "real" C
API. The purpose of this change is to enable using LTO when building the
C API. Currently LTO is disabled because one of the crate types of the C
API is an "rlib" which means that it can't have LTO performed due to
rustc limitations. The solution here is to remove the "cdylib" and
"staticlib" crate types from the "wasmtime-c-api" crate, rename that
crate to "wasmtime-c-api-impl", and reintroduce 'wasmtime-c-api' as a
new crate which wraps the previous crate and reexports it.
This way LTO can be enabled when just building the artifacts and the use
case from #6765 is still satisfied by having a crate that can be linked
to from Rust. Locally this reduces the size of the C API artifact for me
by nearly 1M.
* Move `wasmtime explore` behind a Cargo feature
Enable this Cargo feature by default, but enable building the CLI
without the `explore` subcommand.
* Move the `wast` subcommand behind a Cargo feature
* Move support for `wat` behind a CLI feature
This was already conditional in crates such as `wasmtime` and this makes
it an optional dependency of the CLI as well.
* Move CLI cache support behind a Cargo feature
Additionally refactor `wasmtime-cli-flags` to not unconditionally pull
in caching support by removing its `default` feature and appropriately
enabling it from the CLI.
* Move `rayon` behind an optional feature
* Move `http-body-util` dependency behind `serve` feature
* Add a Cargo feature for compiling out log statements
This sets the static features of `log` and `tracing` to statically
remove all log statements from the binary to cut down on binary size.
* Move logging support behind a Cargo feature
Enables statically removing logging support in addition to the previous
compiling out log statements themselves.
* Move demangling support behind a Cargo feature
* Enable building the CLI without cranelift
Compile out the `compile` subcommand for example.
* Gate all profiling support behind one feature flag
This commit moves vtune/jitdump support behind a single `profiling`
feature flag that additionally includes the guest profiler dependencies
now too.
* Move support for core dumps behind a feature flag
* Move addr2line behind a feature
* Fix rebase
* Document cargo features and a minimal build
* Tidy up the source a bit
* Rename compile-out-logging
* Document disabling logging
* Note the host architecture as well
* Fix tests
* Fix tests
* Mention debuginfo stripping
* Fix CI configuration for checking features
* Fix book tests
* Update lock file after rebase
* Enable coredump feature by default
This adds a new `send_request` method to `WasiHttpView`, allowing embedders to
override the default implementation with their own if the desire. The default
implementation behaves exactly as before.
I've also added a few new `wasi-http` tests: one to test the above, and two
others to test streaming and concurrency. These tests are ports of the
`test_wasi_http_echo` and `test_wasi_http_hash_all` tests in the
[Spin](https://github.com/fermyon/spin) integration test suite. The component
they instantiate is likewise ported from the Spin
`wasi-http-rust-streaming-outgoing-body` component.
Fixes#7259
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
Some notable changes here are:
* The `wasm-tools` crates have enabled the `relaxed-simd`, `threads`,
and `multi-memory` proposals by default. For now I've left these
disabled-by-default in Wasmtime to get enabled in a future PR.
* The `wast` crate has support for parsing `thread` and `wait`
constructs from the `threads` proposal for WebAssembly. They're left
unimplemented for now and return errors. This will get filled in in a
future update.
Some things I noticed from #7239 which are very much not critical but I
figure might be nice-to-haves:
* Move the logging configuration to the `wasmtime-cli-flags` crate with
the other logging configuration to keep it in one place.
* Remove `pretty_env_logger` since `tracing-subscriber` probably
supplants it.
* Don't explicitly inherit env vars in tests since that happens
automatically with `Command`.
* Update to arbitrary 1.3.1
And use workspace dependencies for arbitrary.
* Prune cargo vet's supply-chain files
This is the mechanical changes made by running `cargo vet prune` which was
suggested to me when I ran `cargo vet`.
* Introduce StackMemory and StackMemoryCreator
This allows custom implementations of stack memory to be plugged into
the async functionality for wasmtime. Currently, stacks are always
mmapped, and this custom allocator allows embedders to use any memory
they would like.
The new APIs are also exposed in the C api.
This has no effect on windows, as our hands are tied with fibers there.
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* Add test for custom host memory
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* fix allocator test
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* Address review comments
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* Fix lint warnings
prtest:full
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* fix windows lint warning
prtest:full
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
---------
Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
* Refactor the test-programs test suite
This commit is a large refactoring that reorganizes `test-programs` and
how we tests wasms in Wasmtime. Often writing tests requires complicated
interactions with the guest which can't be done via hand-written `*.wat`
syntax and requires a compiler to get engaged. For this purpose Wasmtime
currently has the `crates/test-programs/*` test suite which builds files
from source and then runs the tests. This has been somewhat cumbersome
in the past though and it's not been easy to extend this over time, so
this commit attempts to address this.
The scheme implemented in this PR looks like:
* All wasm test programs live in `crates/test-programs/src/bin/*.rs`.
All of them, no exceptions.
* Wasm tests have shared support located at
`crates/test-programs/src/lib.rs` and its submodules, such as bindings
generation for WASI.
* Wasm tests are built by a new `crates/test-programs/artifacts` crate.
This crate compiles modules and additionally creates components for
all test programs. The crate itself only records the path to these
outputs and a small amount of testing support, but otherwise doesn't
interact with `wasmtime`-the-crate itself.
* All tests in `crates/test-programs/tests/*.rs` have moved. For example
wasi-http tests now live at `crates/wasi-http/tests/*.rs`. Legacy
tests of wasi-common now live at `crates/wasi-common/tests/*.rs`.
Modern tests for preview2 live at `crates/wasi/tests/*.rs`.
* Wasm tests are bucketed based on their filename prefix. For example
`preview1_*` is tested in wasi-common and wasmtime-wasi. The
`preview2_*` prefix is only tested with wasmtime-wasi, however.
* A new `cli_*` prefix is used to execute tests as part of
`tests/all/main.rs`. This is a new submodule in
`tests/all/cli_tests.rs` which executes these components on the
command line. Many old "command" tests were migrated here.
* Helper macros are generated to assert that a test suite is run in its
entirety. This way if a `preview1_*` test is added it's asserted to
get added to both wasi-common and wasmtime-wasi in the various modes
they run tests.
Overall this moved a number of tests around and refactored some edges of
the tests, but this should not lose any tests (except one that wasn't
actually testing anything). Additionally the hope is that it's much
easier to add tests in the future. The process is to add a new file in
`crates/test-programs/src/bin/*.rs` named appropriately. For example a
preview2 executable is `preview2_*` and a CLI tests is `cli_*`. When
building the test suite an error is generated in the appropriate module
then of "please write a test here", and then a test is written in the
same manner as the other tests in the module.
* Remove no-longer-needed fetches
prtest:full
* I'm worried wasi is running low on semicolons
* Add the WASI target in all CI actions
* Add unknown-unknown target on all CI builders too
* Fix building test artifacts under miri
Need to avoid wrappers for these cross-compiled targets
* Break circular dependency for packaging
Don't use the workspace dep for `wasmtime-wasi` since it injects a
version, instead use a `path = '..'` dependency to fool Cargo into
dropping the dependency during the package phase.
* Fix some merge conflicts with tests
* Fix rebase for new tests
* Remove stray comment
* Fix some flaky tests
* Fix network tests in synchronous mode
This commit is an attempt to fix some networking tests in synchronous
mode in our test suite. Currently networking tests don't actually run in
synchronous mode on CI which is why no failures have been surfaced yet,
but the refactoring in #7182 is going to start doing this.
Currently the `udp_sample_application.rs` test blocks infinitely in
synchronous mode for me locally, most of the time. This appears to be an
interaction between how Tokio handles readiness and how we're
entering the event loop. We're effectively entering the Tokio event loop
with a future that's always ready which ends up starving Tokio of
otherwise performing its background work such as updating flags for
readiness of reading/writing.
The fix here is to add a yield at the start of an `in_tokio` block which
is used in synchronous mode. This is a kludge fix but the intention is
to enable Tokio to have a chance to update readiness flags and process
events from epoll/kqueue/etc.
An additional fix to this issue is WebAssembly/wasi-sockets#64 where the
test is waiting on `READABLE` or `WRITABLE`, but in this specific case
it should only wait on `READABLE`. If it waited on just this then that
would also fix this issue. Nevertheless having a `yield_now` is expected
to have little-to-no overhead and otherwise fix this edge case of an
always-ready future.
* Fix passing empty arguments on the CLI
* Add another blocking accept
* Update crates/test-programs/src/bin/api_proxy.rs
Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>
---------
Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>
* Refactor HTTP tests to be less flaky and more robust
This commit refactors the wasi-http tests we have in this repository
with a few changes:
* Each test now uses a dedicated port for the test. The port 0 is bound
and whatever the OS gives is used for the duration of the test. This
prevents tests from possibly interfering with each other.
* Server spawning is abstracted behind a single `Server` type now which
internally has http1/2 constructors.
* Timeouts for server shutdown are removed since they were failing in CI
and are likely best handled for each test individually if necessary.
As a minor cleanup the `tokio` usage in guest programs was removed as it
was boilerplate in this case. This shouldn't affect the runtimes of
tests, however.
* Remove unused import
* Don't panic from worker thread
* Improve failure error message
prtest:full
* Fix some windows-specific issues
* Fix async tests
* Relax test assertion string
Fredrik Fornwall
Max Brunsfeld
Alex Crichton
wasmtime-publish
Robin Freyler
Adam Bratschi-Kaye
Tyler Rockwood
Dan Gohman
Iceber Gu
Luke Parker
Andrew Brown
Trevor Elliott
Simonas Kazlauskas
Rahul
Afonso Bordado
Joel Dice
Pat Hickey
Nick Fitzgerald
Tyler Rockwood