|
|
|
|
[package]
|
|
|
|
|
name = "wasmtime-cli"
|
|
|
|
|
version.workspace = true
|
|
|
|
|
authors.workspace = true
|
|
|
|
|
description = "Command-line interface for Wasmtime"
|
|
|
|
|
license = "Apache-2.0 WITH LLVM-exception"
|
|
|
|
|
documentation = "https://bytecodealliance.github.io/wasmtime/cli.html"
|
|
|
|
|
categories = ["wasm"]
|
|
|
|
|
keywords = ["webassembly", "wasm"]
|
|
|
|
|
repository = "https://github.com/bytecodealliance/wasmtime"
|
|
|
|
|
readme = "README.md"
|
|
|
|
|
edition.workspace = true
|
|
|
|
|
default-run = "wasmtime"
|
|
|
|
|
rust-version.workspace = true
|
|
|
|
|
|
|
|
|
|
[lints]
|
|
|
|
|
workspace = true
|
|
|
|
|
|
|
|
|
|
[lib]
|
|
|
|
|
doctest = false
|
|
|
|
|
|
|
|
|
|
[[bin]]
|
|
|
|
|
name = "wasmtime"
|
|
|
|
|
path = "src/bin/wasmtime.rs"
|
|
|
|
|
doc = false
|
|
|
|
|
|
|
|
|
|
[dependencies]
|
|
|
|
|
wasmtime = { workspace = true }
|
|
|
|
|
wasmtime-cache = { workspace = true, optional = true }
|
|
|
|
|
wasmtime-cli-flags = { workspace = true }
|
|
|
|
|
wasmtime-cranelift = { workspace = true, optional = true }
|
|
|
|
|
wasmtime-environ = { workspace = true }
|
|
|
|
|
wasmtime-explorer = { workspace = true, optional = true }
|
|
|
|
|
wasmtime-wast = { workspace = true, optional = true }
|
|
|
|
|
wasi-common = { workspace = true, default-features = true, features = ["exit" ], optional = true }
|
|
|
|
|
wasmtime-wasi = { workspace = true, default-features = true, optional = true }
|
|
|
|
|
wasmtime-wasi-nn = { workspace = true, optional = true }
|
wasi-threads: an initial implementation (#5484)
This commit includes a set of changes that add initial support for `wasi-threads` to Wasmtime:
* feat: remove mutability from the WasiCtx Table
This patch adds interior mutability to the WasiCtx Table and the Table elements.
Major pain points:
* `File` only needs `RwLock<cap_std::fs::File>` to implement
`File::set_fdflags()` on Windows, because of [1]
* Because `File` needs a `RwLock` and `RwLock*Guard` cannot
be hold across an `.await`, The `async` from
`async fn num_ready_bytes(&self)` had to be removed
* Because `File` needs a `RwLock` and `RwLock*Guard` cannot
be dereferenced in `pollable`, the signature of
`fn pollable(&self) -> Option<rustix::fd::BorrowedFd>`
changed to `fn pollable(&self) -> Option<Arc<dyn AsFd + '_>>`
[1] https://github.com/bytecodealliance/system-interface/blob/da238e324e752033f315f09c082ad9ce35d42696/src/fs/fd_flags.rs#L210-L217
* wasi-threads: add an initial implementation
This change is a first step toward implementing `wasi-threads` in
Wasmtime. We may find that it has some missing pieces, but the core
functionality is there: when `wasi::thread_spawn` is called by a running
WebAssembly module, a function named `wasi_thread_start` is found in the
module's exports and called in a new instance. The shared memory of the
original instance is reused in the new instance.
This new WASI proposal is in its early stages and details are still
being hashed out in the [spec] and [wasi-libc] repositories. Due to its
experimental state, the `wasi-threads` functionality is hidden behind
both a compile-time and runtime flag: one must build with `--features
wasi-threads` but also run the Wasmtime CLI with `--wasm-features
threads` and `--wasi-modules experimental-wasi-threads`. One can
experiment with `wasi-threads` by running:
```console
$ cargo run --features wasi-threads -- \
--wasm-features threads --wasi-modules experimental-wasi-threads \
<a threads-enabled module>
```
Threads-enabled Wasm modules are not yet easy to build. Hopefully this
is resolved soon, but in the meantime see the use of
`THREAD_MODEL=posix` in the [wasi-libc] repository for some clues on
what is necessary. Wiggle complicates things by requiring the Wasm
memory to be exported with a certain name and `wasi-threads` also
expects that memory to be imported; this build-time obstacle can be
overcome with the `--import-memory --export-memory` flags only available
in the latest Clang tree. Due to all of this, the included tests are
written directly in WAT--run these with:
```console
$ cargo test --features wasi-threads -p wasmtime-cli -- cli_tests
```
[spec]: https://github.com/WebAssembly/wasi-threads
[wasi-libc]: https://github.com/WebAssembly/wasi-libc
This change does not protect the WASI implementations themselves from
concurrent access. This is already complete in previous commits or left
for future commits in certain cases (e.g., wasi-nn).
* wasi-threads: factor out process exit logic
As is being discussed [elsewhere], either calling `proc_exit` or
trapping in any thread should halt execution of all threads. The
Wasmtime CLI already has logic for adapting a WebAssembly error code to
a code expected in each OS. This change factors out this logic to a new
function, `maybe_exit_on_error`, for use within the `wasi-threads`
implementation.
This will work reasonably well for CLI users of Wasmtime +
`wasi-threads`, but embedders will want something better in the future:
when a `wasi-threads` threads fails, they may not want their application
to exit. Handling this is tricky, because it will require cancelling the
threads spawned by the `wasi-threads` implementation, something that is
not trivial to do in Rust. With this change, we defer that work until
later in order to provide a working implementation of `wasi-threads` for
experimentation.
[elsewhere]: https://github.com/WebAssembly/wasi-threads/pull/17
* review: work around `fd_fdstat_set_flags`
In order to make progress with wasi-threads, this change temporarily
works around limitations induced by `wasi-common`'s
`fd_fdstat_set_flags` to allow `&mut self` use in the implementation.
Eventual resolution is tracked in
https://github.com/bytecodealliance/wasmtime/issues/5643. This change
makes several related helper functions (e.g., `set_fdflags`) take `&mut
self` as well.
* test: use `wait`/`notify` to improve `threads.wat` test
Previously, the test simply executed in a loop for some hardcoded number
of iterations. This changes uses `wait` and `notify` and atomic
operations to keep track of when the spawned threads are done and join
on the main thread appropriately.
* various fixes and tweaks due to the PR review
---------
Signed-off-by: Harald Hoyer <harald@profian.com>
Co-authored-by: Harald Hoyer <harald@profian.com>
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
2 years ago
|
|
|
wasmtime-wasi-threads = { workspace = true, optional = true }
|
Begin implementation of wasi-http (#5929)
* Integrate experimental HTTP into wasmtime.
* Reset Cargo.lock
* Switch to bail!, plumb options partially.
* Implement timeouts.
* Remove generated files & wasm, add Makefile
* Remove generated code textfile
* Update crates/wasi-http/Cargo.toml
Co-authored-by: Eduardo de Moura Rodrigues <16357187+eduardomourar@users.noreply.github.com>
* Update crates/wasi-http/Cargo.toml
Co-authored-by: Eduardo de Moura Rodrigues <16357187+eduardomourar@users.noreply.github.com>
* Extract streams from request/response.
* Fix read for len < buffer length.
* Formatting.
* types impl: swap todos for traps
* streams_impl: idioms, and swap todos for traps
* component impl: idioms, swap all unwraps for traps, swap all todos for traps
* http impl: idiom
* Remove an unnecessary mut.
* Remove an unsupported function.
* Switch to the tokio runtime for the HTTP request.
* Add a rust example.
* Update to latest wit definition
* Remove example code.
* wip: start writing a http test...
* finish writing the outbound request example
havent executed it yet
* better debug output
* wasi-http: some stubs required for rust rewrite of the example
* add wasi_http tests to test-programs
* CI: run the http tests
* Fix some warnings.
* bump new deps to latest releases (#3)
* Add tests for wasi-http to test-programs (#2)
* wip: start writing a http test...
* finish writing the outbound request example
havent executed it yet
* better debug output
* wasi-http: some stubs required for rust rewrite of the example
* add wasi_http tests to test-programs
* CI: run the http tests
* bump new deps to latest releases
h2 0.3.16
http 0.2.9
mio 0.8.6
openssl 0.10.48
openssl-sys 0.9.83
tokio 1.26.0
---------
Co-authored-by: Brendan Burns <bburns@microsoft.com>
* Update crates/test-programs/tests/http_tests/runtime/wasi_http_tests.rs
* Update crates/test-programs/tests/http_tests/runtime/wasi_http_tests.rs
* Update crates/test-programs/tests/http_tests/runtime/wasi_http_tests.rs
* wasi-http: fix cargo.toml file and publish script to work together (#4)
unfortunately, the publish script doesn't use a proper toml parser (in
order to not have any dependencies), so the whitespace has to be the
trivial expected case.
then, add wasi-http to the list of crates to publish.
* Update crates/test-programs/build.rs
* Switch to rustls
* Cleanups.
* Merge switch to rustls.
* Formatting
* Remove libssl install
* Fix tests.
* Rename wasi-http -> wasmtime-wasi-http
* prtest:full
Conditionalize TLS on riscv64gc.
* prtest:full
Fix formatting, also disable tls on s390x
* prtest:full
Add a path parameter to wit-bindgen, remove symlink.
* prtest:full
Fix tests for places where SSL isn't supported.
* Update crates/wasi-http/Cargo.toml
---------
Co-authored-by: Eduardo de Moura Rodrigues <16357187+eduardomourar@users.noreply.github.com>
Co-authored-by: Pat Hickey <phickey@fastly.com>
Co-authored-by: Pat Hickey <pat@moreproductive.org>
2 years ago
|
|
|
wasmtime-wasi-http = { workspace = true, optional = true }
|
|
|
|
|
wasmtime-runtime = { workspace = true, optional = true }
|
|
|
|
|
clap = { workspace = true }
|
|
|
|
|
anyhow = { workspace = true }
|
|
|
|
|
target-lexicon = { workspace = true }
|
|
|
|
|
once_cell = { workspace = true }
|
|
|
|
|
listenfd = { version = "1.0.0", optional = true }
|
|
|
|
|
wat = { workspace = true, optional = true }
|
|
|
|
|
serde = { workspace = true }
|
|
|
|
|
serde_derive = { workspace = true }
|
|
|
|
|
serde_json = { workspace = true }
|
|
|
|
|
wasmparser = { workspace = true }
|
|
|
|
|
tracing = { workspace = true }
|
|
|
|
|
log = { workspace = true }
|
|
|
|
|
humantime = { workspace = true }
|
|
|
|
|
|
|
|
|
|
async-trait = { workspace = true }
|
|
|
|
|
bytes = { workspace = true }
|
|
|
|
|
cfg-if = { workspace = true }
|
|
|
|
|
tokio = { workspace = true, optional = true, features = [ "signal", "macros" ] }
|
|
|
|
|
hyper = { workspace = true, optional = true }
|
|
|
|
|
http = { workspace = true, optional = true }
|
|
|
|
|
http-body-util = { workspace = true, optional = true }
|
|
|
|
|
|
|
|
|
|
[target.'cfg(unix)'.dependencies]
|
|
|
|
|
rustix = { workspace = true, features = ["mm", "param", "process"] }
|
|
|
|
|
|
|
|
|
|
[dev-dependencies]
|
Refactor fuzzing configuration and sometimes disable debug verifier. (#3664)
* fuzz: Refactor Wasmtime's fuzz targets
A recent fuzz bug found is related to timing out when compiling a
module. This timeout, however, is predominately because Cranelift's
debug verifier is enabled and taking up over half the compilation time.
I wanted to fix this by disabling the verifier when input modules might
have a lot of functions, but this was pretty difficult to implement.
Over time we've grown a number of various fuzzers. Most are
`wasm-smith`-based at this point but there's various entry points for
configuring the wasm-smith module, the wasmtime configuration, etc. I've
historically gotten quite lost in trying to change defaults and feeling
like I have to touch a lot of different places. This is the motivation
for this commit, simplifying fuzzer default configuration.
This commit removes the ability to create a default `Config` for
fuzzing, instead only supporting generating a configuration via
`Arbitrary`. This then involved refactoring all targets and fuzzers to
ensure that configuration is generated through `Arbitrary`. This should
actually expand the coverage of some existing fuzz targets since
`Arbitrary for Config` will tweak options that don't affect runtime,
such as memory configuration or jump veneers.
All existing fuzz targets are refactored to use this new method of
configuration. Some fuzz targets were also shuffled around or
reimplemented:
* `compile` - this now directly calls `Module::new` to skip all the
fuzzing infrastructure. This is mostly done because this fuzz target
isn't too interesting and is largely just seeing what happens when
things are thrown at the wall for Wasmtime.
* `instantiate-maybe-invalid` - this fuzz target now skips instantiation
and instead simply goes into `Module::new` like the `compile` target.
The rationale behind this is that most modules won't instantiate
anyway and this fuzz target is primarily fuzzing the compiler. This
skips having to generate arbitrary configuration since
wasm-smith-generated-modules (or valid ones at least) aren't used
here.
* `instantiate` - this fuzz target was removed. In general this fuzz
target isn't too interesting in isolation. Almost everything it deals
with likely won't pass compilation and is covered by the `compile`
fuzz target, and otherwise interesting modules being instantiated can
all theoretically be created by `wasm-smith` anyway.
* `instantiate-wasm-smith` and `instantiate-swarm` - these were both merged
into a new `instantiate` target (replacing the old one from above).
There wasn't really much need to keep these separate since they really
only differed at this point in methods of timeout. Otherwise we much
more heavily use `SwarmConfig` than wasm-smith's built-in options.
The intention is that we should still have basically the same coverage
of fuzzing as before, if not better because configuration is now
possible on some targets. Additionally there is one centralized point of
configuration for fuzzing for wasmtime, `Arbitrary for ModuleConfig`.
This internally creates an arbitrary `SwarmConfig` from `wasm-smith` and
then further tweaks it for Wasmtime's needs, such as enabling various
wasm proposals by default. In the future enabling a wasm proposal on
fuzzing should largely just be modifying this one trait implementation.
* fuzz: Sometimes disable the cranelift debug verifier
This commit disables the cranelift debug verifier if the input wasm
module might be "large" for the definition of "more than 10 functions".
While fuzzing we disable threads (set them to 1) and enable the
cranelift debug verifier. Coupled with a 20-30x slowdown this means that
a module with the maximum number of functions, 100, gives:
60x / 100 functions / 30x slowdown = 20ms
With only 20 milliseconds per function this is even further halved by
the `differential` fuzz target compiling a module twice, which means
that, when compiling with a normal release mode Wasmtime, if any
function takes more than 10ms to compile then it's a candidate for
timing out while fuzzing. Given that the cranelift debug verifier can
more than double compilation time in fuzzing mode this actually means
that the real time budget for function compilation is more like 4ms.
The `wasm-smith` crate can pretty easily generate a large function that
takes 4ms to compile, and then when that function is multiplied 100x in
the `differential` fuzz target we trivially time out the fuzz target.
The hope of this commit is to buy back half our budget by disabling the
debug verifier for modules that may have many functions. Further
refinements can be implemented in the future such as limiting functions
for just the differential target as well.
* Fix the single-function-module fuzz configuration
* Tweak how features work in differential fuzzing
* Disable everything for baseline differential fuzzing
* Enable selectively for each engine afterwards
* Also forcibly enable reference types and bulk memory for spec tests
* Log wasms when compiling
* Add reference types support to v8 fuzzer
* Fix timeouts via fuel
The default store has "infinite" fuel so that needs to be consumed
before fuel is added back in.
* Remove fuzzing-specific tests
These no longer compile and also haven't been added to in a long time.
Most of the time a reduced form of original the fuzz test case is added
when a fuzz bug is fixed.
3 years ago
|
|
|
# depend again on wasmtime to activate its default features for tests
|
|
|
|
|
wasmtime = { workspace = true, features = ['component-model', 'async', 'default', 'winch', 'debug-builtins'] }
|
|
|
|
|
env_logger = { workspace = true }
|
|
|
|
|
log = { workspace = true }
|
|
|
|
|
filecheck = { workspace = true }
|
|
|
|
|
tempfile = { workspace = true }
|
|
|
|
|
wasmtime-runtime = { workspace = true }
|
|
|
|
|
tokio = { workspace = true, features = ["rt", "time", "macros", "rt-multi-thread"] }
|
|
|
|
|
wast = { workspace = true }
|
|
|
|
|
criterion = "0.5.0"
|
|
|
|
|
num_cpus = "1.13.0"
|
Use an mmap-friendly serialization format (#3257)
* Use an mmap-friendly serialization format
This commit reimplements the main serialization format for Wasmtime's
precompiled artifacts. Previously they were generally a binary blob of
`bincode`-encoded metadata prefixed with some versioning information.
The downside of this format, though, is that loading a precompiled
artifact required pushing all information through `bincode`. This is
inefficient when some data, such as trap/address tables, are rarely
accessed.
The new format added in this commit is one which is designed to be
`mmap`-friendly. This means that the relevant parts of the precompiled
artifact are already page-aligned for updating permissions of pieces
here and there. Additionally the artifact is optimized so that if data
is rarely read then we can delay reading it until necessary.
The new artifact format for serialized modules is an ELF file. This is
not a public API guarantee, so it cannot be relied upon. In the meantime
though this is quite useful for exploring precompiled modules with
standard tooling like `objdump`. The ELF file is already constructed as
part of module compilation, and this is the main contents of the
serialized artifact.
THere is some extra information, though, not encoded in each module's
individual ELF file such as type information. This information continues
to be `bincode`-encoded, but it's intended to be much smaller and much
faster to deserialize. This extra information is appended to the end of
the ELF file. This means that the original ELF file is still a valid ELF
file, we just get to have extra bits at the end. More information on the
new format can be found in the module docs of the serialization module
of Wasmtime.
Another refatoring implemented as part of this commit is to deserialize
and store object files directly in `mmap`-backed storage. This avoids
the need to copy bytes after the artifact is loaded into memory for each
compiled module, and in a future commit it opens up the door to avoiding
copying the text section into a `CodeMemory`. For now, though, the main
change is that copies are not necessary when loading from a precompiled
compilation artifact once the artifact is itself in mmap-based memory.
To assist with managing `mmap`-based memory a new `MmapVec` type was
added to `wasmtime_jit` which acts as a form of `Vec<T>` backed by a
`wasmtime_runtime::Mmap`. This type notably supports `drain(..N)` to
slice the buffer into disjoint regions that are all separately owned,
such as having a separately owned window into one artifact for all
object files contained within.
Finally this commit implements a small refactoring in `wasmtime-cache`
to use the standard artifact format for cache entries rather than a
bincode-encoded version. This required some more hooks for
serializing/deserializing but otherwise the crate still performs as
before.
* Review comments
3 years ago
|
|
|
memchr = "2.4"
|
|
|
|
|
async-trait = { workspace = true }
|
|
|
|
|
wat = { workspace = true }
|
|
|
|
|
rayon = "1.5.0"
|
|
|
|
|
wasmtime-wast = { workspace = true, features = ['component-model'] }
|
|
|
|
|
wasmtime-component-util = { workspace = true }
|
|
|
|
|
component-macro-test = { path = "crates/misc/component-macro-test" }
|
|
|
|
|
component-test-util = { workspace = true }
|
|
|
|
|
bstr = "1.6.0"
|
|
|
|
|
libc = { workspace = true }
|
|
|
|
|
serde = { workspace = true }
|
|
|
|
|
serde_json = { workspace = true }
|
|
|
|
|
walkdir = { workspace = true }
|
Refactor the test-programs test suite (#7182)
* Refactor the test-programs test suite
This commit is a large refactoring that reorganizes `test-programs` and
how we tests wasms in Wasmtime. Often writing tests requires complicated
interactions with the guest which can't be done via hand-written `*.wat`
syntax and requires a compiler to get engaged. For this purpose Wasmtime
currently has the `crates/test-programs/*` test suite which builds files
from source and then runs the tests. This has been somewhat cumbersome
in the past though and it's not been easy to extend this over time, so
this commit attempts to address this.
The scheme implemented in this PR looks like:
* All wasm test programs live in `crates/test-programs/src/bin/*.rs`.
All of them, no exceptions.
* Wasm tests have shared support located at
`crates/test-programs/src/lib.rs` and its submodules, such as bindings
generation for WASI.
* Wasm tests are built by a new `crates/test-programs/artifacts` crate.
This crate compiles modules and additionally creates components for
all test programs. The crate itself only records the path to these
outputs and a small amount of testing support, but otherwise doesn't
interact with `wasmtime`-the-crate itself.
* All tests in `crates/test-programs/tests/*.rs` have moved. For example
wasi-http tests now live at `crates/wasi-http/tests/*.rs`. Legacy
tests of wasi-common now live at `crates/wasi-common/tests/*.rs`.
Modern tests for preview2 live at `crates/wasi/tests/*.rs`.
* Wasm tests are bucketed based on their filename prefix. For example
`preview1_*` is tested in wasi-common and wasmtime-wasi. The
`preview2_*` prefix is only tested with wasmtime-wasi, however.
* A new `cli_*` prefix is used to execute tests as part of
`tests/all/main.rs`. This is a new submodule in
`tests/all/cli_tests.rs` which executes these components on the
command line. Many old "command" tests were migrated here.
* Helper macros are generated to assert that a test suite is run in its
entirety. This way if a `preview1_*` test is added it's asserted to
get added to both wasi-common and wasmtime-wasi in the various modes
they run tests.
Overall this moved a number of tests around and refactored some edges of
the tests, but this should not lose any tests (except one that wasn't
actually testing anything). Additionally the hope is that it's much
easier to add tests in the future. The process is to add a new file in
`crates/test-programs/src/bin/*.rs` named appropriately. For example a
preview2 executable is `preview2_*` and a CLI tests is `cli_*`. When
building the test suite an error is generated in the appropriate module
then of "please write a test here", and then a test is written in the
same manner as the other tests in the module.
* Remove no-longer-needed fetches
prtest:full
* I'm worried wasi is running low on semicolons
* Add the WASI target in all CI actions
* Add unknown-unknown target on all CI builders too
* Fix building test artifacts under miri
Need to avoid wrappers for these cross-compiled targets
* Break circular dependency for packaging
Don't use the workspace dep for `wasmtime-wasi` since it injects a
version, instead use a `path = '..'` dependency to fool Cargo into
dropping the dependency during the package phase.
* Fix some merge conflicts with tests
* Fix rebase for new tests
* Remove stray comment
* Fix some flaky tests
* Fix network tests in synchronous mode
This commit is an attempt to fix some networking tests in synchronous
mode in our test suite. Currently networking tests don't actually run in
synchronous mode on CI which is why no failures have been surfaced yet,
but the refactoring in #7182 is going to start doing this.
Currently the `udp_sample_application.rs` test blocks infinitely in
synchronous mode for me locally, most of the time. This appears to be an
interaction between how Tokio handles readiness and how we're
entering the event loop. We're effectively entering the Tokio event loop
with a future that's always ready which ends up starving Tokio of
otherwise performing its background work such as updating flags for
readiness of reading/writing.
The fix here is to add a yield at the start of an `in_tokio` block which
is used in synchronous mode. This is a kludge fix but the intention is
to enable Tokio to have a chance to update readiness flags and process
events from epoll/kqueue/etc.
An additional fix to this issue is WebAssembly/wasi-sockets#64 where the
test is waiting on `READABLE` or `WRITABLE`, but in this specific case
it should only wait on `READABLE`. If it waited on just this then that
would also fix this issue. Nevertheless having a `yield_now` is expected
to have little-to-no overhead and otherwise fix this edge case of an
always-ready future.
* Fix passing empty arguments on the CLI
* Add another blocking accept
* Update crates/test-programs/src/bin/api_proxy.rs
Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>
---------
Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>
1 year ago
|
|
|
test-programs-artifacts = { workspace = true }
|
|
|
|
|
bytesize = "1.3.0"
|
|
|
|
|
wit-component = { workspace = true }
|
|
|
|
|
|
|
|
|
|
[target.'cfg(windows)'.dev-dependencies]
|
|
|
|
|
windows-sys = { workspace = true, features = ["Win32_System_Memory"] }
|
|
|
|
|
|
|
|
|
|
[build-dependencies]
|
|
|
|
|
anyhow = { workspace = true }
|
|
|
|
|
|
|
|
|
|
[profile.release.build-override]
|
|
|
|
|
opt-level = 0
|
|
|
|
|
|
|
|
|
|
[workspace]
|
|
|
|
|
resolver = '2'
|
|
|
|
|
members = [
|
|
|
|
|
"cranelift",
|
|
|
|
|
"cranelift/isle/fuzz",
|
|
|
|
|
"cranelift/isle/islec",
|
|
|
|
|
"cranelift/serde",
|
|
|
|
|
"crates/bench-api",
|
|
|
|
|
"crates/c-api/artifact",
|
|
|
|
|
"crates/environ/fuzz",
|
Refactor the test-programs test suite (#7182)
* Refactor the test-programs test suite
This commit is a large refactoring that reorganizes `test-programs` and
how we tests wasms in Wasmtime. Often writing tests requires complicated
interactions with the guest which can't be done via hand-written `*.wat`
syntax and requires a compiler to get engaged. For this purpose Wasmtime
currently has the `crates/test-programs/*` test suite which builds files
from source and then runs the tests. This has been somewhat cumbersome
in the past though and it's not been easy to extend this over time, so
this commit attempts to address this.
The scheme implemented in this PR looks like:
* All wasm test programs live in `crates/test-programs/src/bin/*.rs`.
All of them, no exceptions.
* Wasm tests have shared support located at
`crates/test-programs/src/lib.rs` and its submodules, such as bindings
generation for WASI.
* Wasm tests are built by a new `crates/test-programs/artifacts` crate.
This crate compiles modules and additionally creates components for
all test programs. The crate itself only records the path to these
outputs and a small amount of testing support, but otherwise doesn't
interact with `wasmtime`-the-crate itself.
* All tests in `crates/test-programs/tests/*.rs` have moved. For example
wasi-http tests now live at `crates/wasi-http/tests/*.rs`. Legacy
tests of wasi-common now live at `crates/wasi-common/tests/*.rs`.
Modern tests for preview2 live at `crates/wasi/tests/*.rs`.
* Wasm tests are bucketed based on their filename prefix. For example
`preview1_*` is tested in wasi-common and wasmtime-wasi. The
`preview2_*` prefix is only tested with wasmtime-wasi, however.
* A new `cli_*` prefix is used to execute tests as part of
`tests/all/main.rs`. This is a new submodule in
`tests/all/cli_tests.rs` which executes these components on the
command line. Many old "command" tests were migrated here.
* Helper macros are generated to assert that a test suite is run in its
entirety. This way if a `preview1_*` test is added it's asserted to
get added to both wasi-common and wasmtime-wasi in the various modes
they run tests.
Overall this moved a number of tests around and refactored some edges of
the tests, but this should not lose any tests (except one that wasn't
actually testing anything). Additionally the hope is that it's much
easier to add tests in the future. The process is to add a new file in
`crates/test-programs/src/bin/*.rs` named appropriately. For example a
preview2 executable is `preview2_*` and a CLI tests is `cli_*`. When
building the test suite an error is generated in the appropriate module
then of "please write a test here", and then a test is written in the
same manner as the other tests in the module.
* Remove no-longer-needed fetches
prtest:full
* I'm worried wasi is running low on semicolons
* Add the WASI target in all CI actions
* Add unknown-unknown target on all CI builders too
* Fix building test artifacts under miri
Need to avoid wrappers for these cross-compiled targets
* Break circular dependency for packaging
Don't use the workspace dep for `wasmtime-wasi` since it injects a
version, instead use a `path = '..'` dependency to fool Cargo into
dropping the dependency during the package phase.
* Fix some merge conflicts with tests
* Fix rebase for new tests
* Remove stray comment
* Fix some flaky tests
* Fix network tests in synchronous mode
This commit is an attempt to fix some networking tests in synchronous
mode in our test suite. Currently networking tests don't actually run in
synchronous mode on CI which is why no failures have been surfaced yet,
but the refactoring in #7182 is going to start doing this.
Currently the `udp_sample_application.rs` test blocks infinitely in
synchronous mode for me locally, most of the time. This appears to be an
interaction between how Tokio handles readiness and how we're
entering the event loop. We're effectively entering the Tokio event loop
with a future that's always ready which ends up starving Tokio of
otherwise performing its background work such as updating flags for
readiness of reading/writing.
The fix here is to add a yield at the start of an `in_tokio` block which
is used in synchronous mode. This is a kludge fix but the intention is
to enable Tokio to have a chance to update readiness flags and process
events from epoll/kqueue/etc.
An additional fix to this issue is WebAssembly/wasi-sockets#64 where the
test is waiting on `READABLE` or `WRITABLE`, but in this specific case
it should only wait on `READABLE`. If it waited on just this then that
would also fix this issue. Nevertheless having a `yield_now` is expected
to have little-to-no overhead and otherwise fix this edge case of an
always-ready future.
* Fix passing empty arguments on the CLI
* Add another blocking accept
* Update crates/test-programs/src/bin/api_proxy.rs
Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>
---------
Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>
1 year ago
|
|
|
"crates/test-programs",
|
|
|
|
|
"crates/wasi-preview1-component-adapter",
|
|
|
|
|
"crates/wasi-preview1-component-adapter/verify",
|
Initial skeleton for Winch (#4907)
* Initial skeleton for Winch
This commit introduces the initial skeleton for Winch, the "baseline"
compiler.
This skeleton contains mostly setup code for the ISA, ABI, registers,
and compilation environment abstractions. It also includes the
calculation of function local slots.
As of this commit, the structure of these abstractions looks like the
following:
+------------------------+
| v
+----------+ +-----+ +-----------+-----+-----------------+
| Compiler | --> | ISA | --> | Registers | ABI | Compilation Env |
+----------+ +-----+ +-----------+-----+-----------------+
| ^
+------------------------------+
* Compilation environment will hold a reference to the function data
* Add basic documentation to the ABI trait
* Enable x86 and arm64 in cranelift-codegen
* Add reg_name function for x64
* Introduce the concept of a MacroAssembler and Assembler
This commit introduces the concept of a MacroAsesembler and
Assembler. The MacroAssembler trait will provide a high enough
interface across architectures so that each ISA implementation can use their own low-level
Assembler implementation to fulfill the interface. Each Assembler will
provide a 1-1 mapping to each ISA instruction.
As of this commit, only a partial debug implementation is provided for
the x64 Assembler.
* Add a newtype over PReg
Adds a newtype `Reg` over regalloc2::PReg; this ensures that Winch
will operate only on the concept of `Reg`. This change is temporary
until we have the necessary machinery to share a common Reg
abstraction via `cranelift_asm`
* Improvements to local calcuation
- Add `LocalSlot::addressed_from_sp`
- Use `u32` for local slot and local sizes calculation
* Add helper methods to ABIArg
Adds helper methods to retrieve register and type information from the argument
* Make locals_size public in frame
* Improve x64 register naming depending on size
* Add new methods to the masm interface
This commit introduces the ability for the MacroAssembler to reserve
stack space, get the address of a given local and perform a stack
store based on the concept of `Operand`s.
There are several motivating factors to introduce the concept of an
Operand:
- Make the translation between Winch and Cranelift easier;
- Make dispatching from the MacroAssembler to the underlying Assembler
- easier by minimizing the amount of functions that we need to define
- in order to satisfy the store/load combinations
This commit also introduces the concept of a memory address, which
essentially describes the addressing modes; as of this commit only one
addressing mode is supported. We'll also need to verify that this
structure will play nicely with arm64.
* Blank masm implementation for arm64
* Implementation of reserve_stack, local_address, store and fp_offset
for x64
* Implement function prologue and argument register spilling
* Add structopt and wat
* Fix debug instruction formatting
* Make TargetISA trait publicly accessible
* Modify the MacroAssembler finalize siganture to return a slice of strings
* Introduce a simple CLI for Winch
To be able to compile Wasm programs with Winch independently. Mostly
meant for testing / debugging
* Fix bug in x64 assembler mov_rm
* Remove unused import
* Move the stack slot calculation to the Frame
This commit moves the calculation of the stack slots to the frame
handler abstraction and also includes the calculation of the limits
for the function defined locals, which will be used to zero the locals
that are not associated to function arguments
* Add i32 and i64 constructors to local slots
* Introduce the concept of DefinedLocalsRange
This commit introduces `DefinedLocalsRange` to track the stack offset
at which the function-defined locals start and end; this is later used
to zero-out that stack region
* Add constructors for int and float registers
* Add a placeholder stack implementation
* Add a regset abstraction to track register availability
Adds a bit set abstraction to track register availability for register
allocation.
The bit set has no specific knowledge about physical registers, it
works on the register's hardware encoding as the source of truth.
Each RegSet is expected to be created with the universe of allocatable
registers per ISA when starting the compilation of a particular function.
* Add an abstraction over register and immediate
This is meant to be used as the source for stores.
* Add a way to zero local slots and an initial skeletion of regalloc
This commit introduces `zero_local_slots` to the MacroAssembler; which
ensures that function defined locals are zeroed out when starting the
function body.
The algorithm divides the defined function locals stack range
into 8 byte slots and stores a zero at each address. This process
relies on register allocation if the amount of slots that need to be
initialized is greater than 1. In such case, the next available
register is requested to the register set and it's used to store a 0,
which is then stored at every local slot
* Update to wasmparser 0.92
* Correctly track if the regset has registers available
* Add a result entry to the ABI signature
This commuit introduces ABIResult as part of the ABISignature;
this struct will track how function results are stored; initially it
will consiste of a single register that will be requested to the
register allocator at the end of the function; potentially causing a spill
* Move zero local slots and add more granular methods to the masm
This commit removes zeroing local slots from the MacroAssembler and
instead adds more granular methods to it (e.g `zero`, `add`).
This allows for better code sharing since most of the work done by the
algorithm for zeroing slots will be the same in all targets, except
for the binary emissions pieces, which is what gets delegated to the masm
* Use wasmparser's visitor API and add initial support for const and add
This commit adds initial support for the I32Const and I32
instructions; this involves adding a minimum for register
allocation. Note that some regalloc pieces are still incomplete, since
for the current set of supported instructions they are not needed.
* Make the ty field public in Local
* Add scratch_reg to the abi
* Add a method to get a particular local from the Frame
* Split the compilation environment abstraction
This commit splits the compilation environment into two more concise
abstractions:
1. CodeGen: the main abstraction for code generation
2. CodeGenContext: abstraction that shares the common pieces for
compilation; these pieces are shared between the code generator and
the register allocator
* Add `push` and `load` to the MacroAssembler
* Remove dead code warnings for unused paths
* Map ISA features to cranelift-codegen ISA features
* Apply formatting
* Fix Cargo.toml after a bad rebase
* Add component-compiler feature
* Use clap instead of structopt
* Add winch to publish.rs script
* Minor formatting
* Add tests to RegSet and fix two bugs when freeing and checking for
register availability
* Add tests to Stack
* Free source register after a non-constant i32 add
* Improve comments
- Remove unneeded comments
- And improve some of the TODO items
* Update default features
* Drop the ABI generic param and pass the word_size information directly
To avoid dealing with dead code warnings this commit passes the word
size information directly, since it's the only piece of information
needed from the ABI by Codegen until now
* Remove dead code
This piece of code will be put back once we start integrating Winch
with Wasmtime
* Remove unused enum variant
This variant doesn't get constructed; it should be added back once a
backend is added and not enabled by default or when Winch gets
integrated into Wasmtime
* Fix unused code in regset tests
* Update spec testsuite
* Switch the visitor pattern for a simpler operator match
This commit removes the usage of wasmparser's visitor pattern and
instead defaults to a simpler operator matching approach. This removes
the complexity of having to define all the visitor trait functions at once.
* Use wasmparser's Visitor trait with a different macro strategy
This commit puts back wasmparser's Visitor trait, with a sigle;
simpler macro, only used for unsupported operators.
* Restructure Winch
This commit restuructures Winch's parts. It divides the initial
approach into three main crates: `winch-codegen`,`wasmtime-winch` and `winch-tools`.
`wasmtime-winch` is reponsible for the Wasmtime-Winch integration.
`winch-codegen` is solely responsible for code generation.
`winch-tools` is CLI tool to compile Wasm programs, mainly for testing purposes.
* Refactor zero local slots
This commit moves the logic of zeroing local slots from the codegen
module into a method with a default implementation in the
MacroAssembler trait: `zero_mem_range`.
The refactored implementation is very similar to the previous
implementation with the only difference
that it doesn't allocates a general-purpose register; it instead uses
the register allocator to retrieve the scratch register and uses this
register to unroll the series of zero stores.
* Tie the codegen creation to the ISA ABI
This commit makes the relationship between the ISA ABI and the codegen
explicit. This allows us to pass down ABI-specific bit and pieces to
the codegeneration. In this case the only concrete piece that we need
is the ABI word size.
* Mark winch as publishable directory
* Revamp winch docs
This commit ensures that all the code comments in Winch are compliant
with the syle used in the rest of Wasmtime's codebase.
It also imptoves, generally the quality of the comments in some modules.
* Panic when using multi-value when the target is aarch64
Similar to x64, this commit ensures that the abi signature of the
current function doesn't use multi-value returns
* Document the usage of directives
* Use endianness instead of endianess in the ISA trait
* Introduce a three-argument form in the MacroAssembler
This commit introduces the usage of three-argument form for the
MacroAssembler interface. This allows for a natural mapping for
architectures like aarch64. In the case of x64, the implementation can
simply restrict the implementation asserting for equality in two of
the arguments of defaulting to a differnt set of instructions.
As of this commit, the implementation of `add` panics if the
destination and the first source arguments are not equal; internally
the x64 assembler implementation will ensure that all the allowed
combinations of `add` are satisfied. The reason for panicking and not
emitting a `mov` followed by an `add` for example is simply because register
allocation happens right before calling `add`, which ensures any
register-to-register moves, if needed.
This implementation will evolve in the future and this panic will be
lifted if needed.
* Improve the documentation for the MacroAssembler.
Documents the usage of three-arg form and the intention around the
high-level interface.
* Format comments in remaining modules
* Clean up Cargo.toml for winch pieces
This commit adds missing fields to each of Winch's Cargo.toml.
* Use `ModuleTranslation::get_types()` to derive the function type
* Assert that start range is always word-size aligned
2 years ago
|
|
|
"crates/winch",
|
|
|
|
|
"examples/fib-debug/wasm",
|
|
|
|
|
"examples/wasi/wasm",
|
|
|
|
|
"examples/tokio/wasm",
|
|
|
|
|
"examples/component/wasm",
|
|
|
|
|
"examples/min-platform",
|
|
|
|
|
"examples/min-platform/embedding",
|
|
|
|
|
"fuzz",
|
Implement AOT compilation for components (#5160)
* Pull `Module` out of `ModuleTextBuilder`
This commit is the first in what will likely be a number towards
preparing for serializing a compiled component to bytes, a precompiled
artifact. To that end my rough plan is to merge all of the compiled
artifacts for a component into one large object file instead of having
lots of separate object files and lots of separate mmaps to manage. To
that end I plan on eventually using `ModuleTextBuilder` to build one
large text section for all core wasm modules and trampolines, meaning
that `ModuleTextBuilder` is no longer specific to one module. I've
extracted out functionality such as function name calculation as well as
relocation resolving (now a closure passed in) in preparation for this.
For now this just keeps tests passing, and the trajectory for this
should become more clear over the following commits.
* Remove component-specific object emission
This commit removes the `ComponentCompiler::emit_obj` function in favor
of `Compiler::emit_obj`, now renamed `append_code`. This involved
significantly refactoring code emission to take a flat list of functions
into `append_code` and the caller is responsible for weaving together
various "families" of functions and un-weaving them afterwards.
* Consolidate ELF parsing in `CodeMemory`
This commit moves the ELF file parsing and section iteration from
`CompiledModule` into `CodeMemory` so one location keeps track of
section ranges and such. This is in preparation for sharing much of this
code with components which needs all the same sections to get tracked
but won't be using `CompiledModule`. A small side benefit from this is
that the section parsing done in `CodeMemory` and `CompiledModule` is no
longer duplicated.
* Remove separately tracked traps in components
Previously components would generate an "always trapping" function
and the metadata around which pc was allowed to trap was handled
manually for components. With recent refactorings the Wasmtime-standard
trap section in object files is now being generated for components as
well which means that can be reused instead of custom-tracking this
metadata. This commit removes the manual tracking for the `always_trap`
functions and plumbs the necessary bits around to make components look
more like modules.
* Remove a now-unnecessary `Arc` in `Module`
Not expected to have any measurable impact on performance, but
complexity-wise this should make it a bit easier to understand the
internals since there's no longer any need to store this somewhere else
than its owner's location.
* Merge compilation artifacts of components
This commit is a large refactoring of the component compilation process
to produce a single artifact instead of multiple binary artifacts. The
core wasm compilation process is refactored as well to share as much
code as necessary with the component compilation process.
This method of representing a compiled component necessitated a few
medium-sized changes internally within Wasmtime:
* A new data structure was created, `CodeObject`, which represents
metadata about a single compiled artifact. This is then stored as an
`Arc` within a component and a module. For `Module` this is always
uniquely owned and represents a shuffling around of data from one
owner to another. For a `Component`, however, this is shared amongst
all loaded modules and the top-level component.
* The "module registry" which is used for symbolicating backtraces and
for trap information has been updated to account for a single region
of loaded code holding possibly multiple modules. This involved adding
a second-level `BTreeMap` for now. This will likely slow down
instantiation slightly but if it poses an issue in the future this
should be able to be represented with a more clever data structure.
This commit additionally solves a number of longstanding issues with
components such as compiling only one host-to-wasm trampoline per
signature instead of possibly once-per-module. Additionally the
`SignatureCollection` registration now happens once-per-component
instead of once-per-module-within-a-component.
* Fix compile errors from prior commits
* Support AOT-compiling components
This commit adds support for AOT-compiled components in the same manner
as `Module`, specifically adding:
* `Engine::precompile_component`
* `Component::serialize`
* `Component::deserialize`
* `Component::deserialize_file`
Internally the support for components looks quite similar to `Module`.
All the prior commits to this made adding the support here
(unsurprisingly) easy. Components are represented as a single object
file as are modules, and the functions for each module are all piled
into the same object file next to each other (as are areas such as data
sections). Support was also added here to quickly differentiate compiled
components vs compiled modules via the `e_flags` field in the ELF
header.
* Prevent serializing exported modules on components
The current representation of a module within a component means that the
implementation of `Module::serialize` will not work if the module is
exported from a component. The reason for this is that `serialize`
doesn't actually do anything and simply returns the underlying mmap as a
list of bytes. The mmap, however, has `.wasmtime.info` describing
component metadata as opposed to this module's metadata. While rewriting
this section could be implemented it's not so easy to do so and is
otherwise seen as not super important of a feature right now anyway.
* Fix windows build
* Fix an unused function warning
* Update crates/environ/src/compilation.rs
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
Co-authored-by: Nick Fitzgerald <fitzgen@gmail.com>
2 years ago
|
|
|
"winch",
|
|
|
|
|
"winch/codegen", "crates/slab",
|
|
|
|
|
]
|
|
|
|
|
exclude = [
|
|
|
|
|
'docs/rust_wasi_markdown_parser',
|
|
|
|
|
]
|
|
|
|
|
|
|
|
|
|
[workspace.package]
|
|
|
|
|
version = "20.0.0"
|
|
|
|
|
authors = ["The Wasmtime Project Developers"]
|
|
|
|
|
edition = "2021"
|
|
|
|
|
# Wasmtime's current policy is that this number can be no larger than the
|
|
|
|
|
# current stable release of Rust minus 2.
|
|
|
|
|
rust-version = "1.74.0"
|
|
|
|
|
|
|
|
|
|
[workspace.lints.rust]
|
|
|
|
|
# Turn on some lints which are otherwise allow-by-default in rustc.
|
|
|
|
|
unused_extern_crates = 'warn'
|
|
|
|
|
trivial_numeric_casts = 'warn'
|
|
|
|
|
unstable_features = 'warn'
|
|
|
|
|
unused_import_braces = 'warn'
|
|
|
|
|
|
|
|
|
|
[workspace.lints.clippy]
|
|
|
|
|
# The default set of lints in Clippy is viewed as "too noisy" right now so
|
|
|
|
|
# they're all turned off by default. Selective lints are then enabled below as
|
|
|
|
|
# necessary.
|
|
|
|
|
all = 'allow'
|
|
|
|
|
|
|
|
|
|
[workspace.dependencies]
|
|
|
|
|
arbitrary = { version = "1.3.1" }
|
|
|
|
|
wasmtime-wmemcheck = { path = "crates/wmemcheck", version = "=20.0.0" }
|
|
|
|
|
wasmtime = { path = "crates/wasmtime", version = "20.0.0", default-features = false }
|
|
|
|
|
wasmtime-c-api-macros = { path = "crates/c-api-macros", version = "=20.0.0" }
|
|
|
|
|
wasmtime-cache = { path = "crates/cache", version = "=20.0.0" }
|
|
|
|
|
wasmtime-cli-flags = { path = "crates/cli-flags", version = "=20.0.0" }
|
|
|
|
|
wasmtime-cranelift = { path = "crates/cranelift", version = "=20.0.0" }
|
|
|
|
|
wasmtime-cranelift-shared = { path = "crates/cranelift-shared", version = "=20.0.0" }
|
|
|
|
|
wasmtime-winch = { path = "crates/winch", version = "=20.0.0" }
|
|
|
|
|
wasmtime-environ = { path = "crates/environ", version = "=20.0.0" }
|
|
|
|
|
wasmtime-explorer = { path = "crates/explorer", version = "=20.0.0" }
|
|
|
|
|
wasmtime-fiber = { path = "crates/fiber", version = "=20.0.0" }
|
|
|
|
|
wasmtime-types = { path = "crates/types", version = "20.0.0" }
|
|
|
|
|
wasmtime-jit-debug = { path = "crates/jit-debug", version = "=20.0.0" }
|
|
|
|
|
wasmtime-runtime = { path = "crates/runtime", version = "=20.0.0" }
|
|
|
|
|
wasmtime-wast = { path = "crates/wast", version = "=20.0.0" }
|
|
|
|
|
wasmtime-wasi = { path = "crates/wasi", version = "20.0.0", default-features = false }
|
|
|
|
|
wasmtime-wasi-http = { path = "crates/wasi-http", version = "=20.0.0", default-features = false }
|
|
|
|
|
wasmtime-wasi-nn = { path = "crates/wasi-nn", version = "20.0.0" }
|
|
|
|
|
wasmtime-wasi-threads = { path = "crates/wasi-threads", version = "20.0.0" }
|
|
|
|
|
wasmtime-component-util = { path = "crates/component-util", version = "=20.0.0" }
|
|
|
|
|
wasmtime-component-macro = { path = "crates/component-macro", version = "=20.0.0" }
|
|
|
|
|
wasmtime-asm-macros = { path = "crates/asm-macros", version = "=20.0.0" }
|
|
|
|
|
wasmtime-versioned-export-macros = { path = "crates/versioned-export-macros", version = "=20.0.0" }
|
|
|
|
|
wasmtime-slab = { path = "crates/slab", version = "=20.0.0" }
|
|
|
|
|
component-test-util = { path = "crates/misc/component-test-util" }
|
|
|
|
|
component-fuzz-util = { path = "crates/misc/component-fuzz-util" }
|
|
|
|
|
wiggle = { path = "crates/wiggle", version = "=20.0.0", default-features = false }
|
|
|
|
|
wiggle-macro = { path = "crates/wiggle/macro", version = "=20.0.0" }
|
|
|
|
|
wiggle-generate = { path = "crates/wiggle/generate", version = "=20.0.0" }
|
|
|
|
|
wasi-common = { path = "crates/wasi-common", version = "=20.0.0", default-features = false }
|
|
|
|
|
wasmtime-fuzzing = { path = "crates/fuzzing" }
|
|
|
|
|
wasmtime-jit-icache-coherence = { path = "crates/jit-icache-coherence", version = "=20.0.0" }
|
|
|
|
|
wasmtime-wit-bindgen = { path = "crates/wit-bindgen", version = "=20.0.0" }
|
Refactor the test-programs test suite (#7182)
* Refactor the test-programs test suite
This commit is a large refactoring that reorganizes `test-programs` and
how we tests wasms in Wasmtime. Often writing tests requires complicated
interactions with the guest which can't be done via hand-written `*.wat`
syntax and requires a compiler to get engaged. For this purpose Wasmtime
currently has the `crates/test-programs/*` test suite which builds files
from source and then runs the tests. This has been somewhat cumbersome
in the past though and it's not been easy to extend this over time, so
this commit attempts to address this.
The scheme implemented in this PR looks like:
* All wasm test programs live in `crates/test-programs/src/bin/*.rs`.
All of them, no exceptions.
* Wasm tests have shared support located at
`crates/test-programs/src/lib.rs` and its submodules, such as bindings
generation for WASI.
* Wasm tests are built by a new `crates/test-programs/artifacts` crate.
This crate compiles modules and additionally creates components for
all test programs. The crate itself only records the path to these
outputs and a small amount of testing support, but otherwise doesn't
interact with `wasmtime`-the-crate itself.
* All tests in `crates/test-programs/tests/*.rs` have moved. For example
wasi-http tests now live at `crates/wasi-http/tests/*.rs`. Legacy
tests of wasi-common now live at `crates/wasi-common/tests/*.rs`.
Modern tests for preview2 live at `crates/wasi/tests/*.rs`.
* Wasm tests are bucketed based on their filename prefix. For example
`preview1_*` is tested in wasi-common and wasmtime-wasi. The
`preview2_*` prefix is only tested with wasmtime-wasi, however.
* A new `cli_*` prefix is used to execute tests as part of
`tests/all/main.rs`. This is a new submodule in
`tests/all/cli_tests.rs` which executes these components on the
command line. Many old "command" tests were migrated here.
* Helper macros are generated to assert that a test suite is run in its
entirety. This way if a `preview1_*` test is added it's asserted to
get added to both wasi-common and wasmtime-wasi in the various modes
they run tests.
Overall this moved a number of tests around and refactored some edges of
the tests, but this should not lose any tests (except one that wasn't
actually testing anything). Additionally the hope is that it's much
easier to add tests in the future. The process is to add a new file in
`crates/test-programs/src/bin/*.rs` named appropriately. For example a
preview2 executable is `preview2_*` and a CLI tests is `cli_*`. When
building the test suite an error is generated in the appropriate module
then of "please write a test here", and then a test is written in the
same manner as the other tests in the module.
* Remove no-longer-needed fetches
prtest:full
* I'm worried wasi is running low on semicolons
* Add the WASI target in all CI actions
* Add unknown-unknown target on all CI builders too
* Fix building test artifacts under miri
Need to avoid wrappers for these cross-compiled targets
* Break circular dependency for packaging
Don't use the workspace dep for `wasmtime-wasi` since it injects a
version, instead use a `path = '..'` dependency to fool Cargo into
dropping the dependency during the package phase.
* Fix some merge conflicts with tests
* Fix rebase for new tests
* Remove stray comment
* Fix some flaky tests
* Fix network tests in synchronous mode
This commit is an attempt to fix some networking tests in synchronous
mode in our test suite. Currently networking tests don't actually run in
synchronous mode on CI which is why no failures have been surfaced yet,
but the refactoring in #7182 is going to start doing this.
Currently the `udp_sample_application.rs` test blocks infinitely in
synchronous mode for me locally, most of the time. This appears to be an
interaction between how Tokio handles readiness and how we're
entering the event loop. We're effectively entering the Tokio event loop
with a future that's always ready which ends up starving Tokio of
otherwise performing its background work such as updating flags for
readiness of reading/writing.
The fix here is to add a yield at the start of an `in_tokio` block which
is used in synchronous mode. This is a kludge fix but the intention is
to enable Tokio to have a chance to update readiness flags and process
events from epoll/kqueue/etc.
An additional fix to this issue is WebAssembly/wasi-sockets#64 where the
test is waiting on `READABLE` or `WRITABLE`, but in this specific case
it should only wait on `READABLE`. If it waited on just this then that
would also fix this issue. Nevertheless having a `yield_now` is expected
to have little-to-no overhead and otherwise fix this edge case of an
always-ready future.
* Fix passing empty arguments on the CLI
* Add another blocking accept
* Update crates/test-programs/src/bin/api_proxy.rs
Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>
---------
Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>
1 year ago
|
|
|
test-programs-artifacts = { path = 'crates/test-programs/artifacts' }
|
|
|
|
|
|
|
|
|
|
cranelift-wasm = { path = "cranelift/wasm", version = "0.107.0" }
|
|
|
|
|
cranelift-codegen = { path = "cranelift/codegen", version = "0.107.0", default-features = false, features = ["std", "unwind"] }
|
|
|
|
|
cranelift-frontend = { path = "cranelift/frontend", version = "0.107.0" }
|
|
|
|
|
cranelift-entity = { path = "cranelift/entity", version = "0.107.0" }
|
|
|
|
|
cranelift-native = { path = "cranelift/native", version = "0.107.0" }
|
|
|
|
|
cranelift-module = { path = "cranelift/module", version = "0.107.0" }
|
|
|
|
|
cranelift-interpreter = { path = "cranelift/interpreter", version = "0.107.0" }
|
|
|
|
|
cranelift-reader = { path = "cranelift/reader", version = "0.107.0" }
|
|
|
|
|
cranelift-filetests = { path = "cranelift/filetests" }
|
|
|
|
|
cranelift-object = { path = "cranelift/object", version = "0.107.0" }
|
|
|
|
|
cranelift-jit = { path = "cranelift/jit", version = "0.107.0" }
|
|
|
|
|
cranelift-fuzzgen = { path = "cranelift/fuzzgen" }
|
|
|
|
|
cranelift-bforest = { path = "cranelift/bforest", version = "0.107.0" }
|
|
|
|
|
cranelift-control = { path = "cranelift/control", version = "0.107.0" }
|
|
|
|
|
cranelift = { path = "cranelift/umbrella", version = "0.107.0" }
|
|
|
|
|
|
|
|
|
|
winch-codegen = { path = "winch/codegen", version = "=0.18.0" }
|
|
|
|
|
winch-filetests = { path = "winch/filetests" }
|
|
|
|
|
winch-test-macros = { path = "winch/test-macros" }
|
Initial skeleton for Winch (#4907)
* Initial skeleton for Winch
This commit introduces the initial skeleton for Winch, the "baseline"
compiler.
This skeleton contains mostly setup code for the ISA, ABI, registers,
and compilation environment abstractions. It also includes the
calculation of function local slots.
As of this commit, the structure of these abstractions looks like the
following:
+------------------------+
| v
+----------+ +-----+ +-----------+-----+-----------------+
| Compiler | --> | ISA | --> | Registers | ABI | Compilation Env |
+----------+ +-----+ +-----------+-----+-----------------+
| ^
+------------------------------+
* Compilation environment will hold a reference to the function data
* Add basic documentation to the ABI trait
* Enable x86 and arm64 in cranelift-codegen
* Add reg_name function for x64
* Introduce the concept of a MacroAssembler and Assembler
This commit introduces the concept of a MacroAsesembler and
Assembler. The MacroAssembler trait will provide a high enough
interface across architectures so that each ISA implementation can use their own low-level
Assembler implementation to fulfill the interface. Each Assembler will
provide a 1-1 mapping to each ISA instruction.
As of this commit, only a partial debug implementation is provided for
the x64 Assembler.
* Add a newtype over PReg
Adds a newtype `Reg` over regalloc2::PReg; this ensures that Winch
will operate only on the concept of `Reg`. This change is temporary
until we have the necessary machinery to share a common Reg
abstraction via `cranelift_asm`
* Improvements to local calcuation
- Add `LocalSlot::addressed_from_sp`
- Use `u32` for local slot and local sizes calculation
* Add helper methods to ABIArg
Adds helper methods to retrieve register and type information from the argument
* Make locals_size public in frame
* Improve x64 register naming depending on size
* Add new methods to the masm interface
This commit introduces the ability for the MacroAssembler to reserve
stack space, get the address of a given local and perform a stack
store based on the concept of `Operand`s.
There are several motivating factors to introduce the concept of an
Operand:
- Make the translation between Winch and Cranelift easier;
- Make dispatching from the MacroAssembler to the underlying Assembler
- easier by minimizing the amount of functions that we need to define
- in order to satisfy the store/load combinations
This commit also introduces the concept of a memory address, which
essentially describes the addressing modes; as of this commit only one
addressing mode is supported. We'll also need to verify that this
structure will play nicely with arm64.
* Blank masm implementation for arm64
* Implementation of reserve_stack, local_address, store and fp_offset
for x64
* Implement function prologue and argument register spilling
* Add structopt and wat
* Fix debug instruction formatting
* Make TargetISA trait publicly accessible
* Modify the MacroAssembler finalize siganture to return a slice of strings
* Introduce a simple CLI for Winch
To be able to compile Wasm programs with Winch independently. Mostly
meant for testing / debugging
* Fix bug in x64 assembler mov_rm
* Remove unused import
* Move the stack slot calculation to the Frame
This commit moves the calculation of the stack slots to the frame
handler abstraction and also includes the calculation of the limits
for the function defined locals, which will be used to zero the locals
that are not associated to function arguments
* Add i32 and i64 constructors to local slots
* Introduce the concept of DefinedLocalsRange
This commit introduces `DefinedLocalsRange` to track the stack offset
at which the function-defined locals start and end; this is later used
to zero-out that stack region
* Add constructors for int and float registers
* Add a placeholder stack implementation
* Add a regset abstraction to track register availability
Adds a bit set abstraction to track register availability for register
allocation.
The bit set has no specific knowledge about physical registers, it
works on the register's hardware encoding as the source of truth.
Each RegSet is expected to be created with the universe of allocatable
registers per ISA when starting the compilation of a particular function.
* Add an abstraction over register and immediate
This is meant to be used as the source for stores.
* Add a way to zero local slots and an initial skeletion of regalloc
This commit introduces `zero_local_slots` to the MacroAssembler; which
ensures that function defined locals are zeroed out when starting the
function body.
The algorithm divides the defined function locals stack range
into 8 byte slots and stores a zero at each address. This process
relies on register allocation if the amount of slots that need to be
initialized is greater than 1. In such case, the next available
register is requested to the register set and it's used to store a 0,
which is then stored at every local slot
* Update to wasmparser 0.92
* Correctly track if the regset has registers available
* Add a result entry to the ABI signature
This commuit introduces ABIResult as part of the ABISignature;
this struct will track how function results are stored; initially it
will consiste of a single register that will be requested to the
register allocator at the end of the function; potentially causing a spill
* Move zero local slots and add more granular methods to the masm
This commit removes zeroing local slots from the MacroAssembler and
instead adds more granular methods to it (e.g `zero`, `add`).
This allows for better code sharing since most of the work done by the
algorithm for zeroing slots will be the same in all targets, except
for the binary emissions pieces, which is what gets delegated to the masm
* Use wasmparser's visitor API and add initial support for const and add
This commit adds initial support for the I32Const and I32
instructions; this involves adding a minimum for register
allocation. Note that some regalloc pieces are still incomplete, since
for the current set of supported instructions they are not needed.
* Make the ty field public in Local
* Add scratch_reg to the abi
* Add a method to get a particular local from the Frame
* Split the compilation environment abstraction
This commit splits the compilation environment into two more concise
abstractions:
1. CodeGen: the main abstraction for code generation
2. CodeGenContext: abstraction that shares the common pieces for
compilation; these pieces are shared between the code generator and
the register allocator
* Add `push` and `load` to the MacroAssembler
* Remove dead code warnings for unused paths
* Map ISA features to cranelift-codegen ISA features
* Apply formatting
* Fix Cargo.toml after a bad rebase
* Add component-compiler feature
* Use clap instead of structopt
* Add winch to publish.rs script
* Minor formatting
* Add tests to RegSet and fix two bugs when freeing and checking for
register availability
* Add tests to Stack
* Free source register after a non-constant i32 add
* Improve comments
- Remove unneeded comments
- And improve some of the TODO items
* Update default features
* Drop the ABI generic param and pass the word_size information directly
To avoid dealing with dead code warnings this commit passes the word
size information directly, since it's the only piece of information
needed from the ABI by Codegen until now
* Remove dead code
This piece of code will be put back once we start integrating Winch
with Wasmtime
* Remove unused enum variant
This variant doesn't get constructed; it should be added back once a
backend is added and not enabled by default or when Winch gets
integrated into Wasmtime
* Fix unused code in regset tests
* Update spec testsuite
* Switch the visitor pattern for a simpler operator match
This commit removes the usage of wasmparser's visitor pattern and
instead defaults to a simpler operator matching approach. This removes
the complexity of having to define all the visitor trait functions at once.
* Use wasmparser's Visitor trait with a different macro strategy
This commit puts back wasmparser's Visitor trait, with a sigle;
simpler macro, only used for unsupported operators.
* Restructure Winch
This commit restuructures Winch's parts. It divides the initial
approach into three main crates: `winch-codegen`,`wasmtime-winch` and `winch-tools`.
`wasmtime-winch` is reponsible for the Wasmtime-Winch integration.
`winch-codegen` is solely responsible for code generation.
`winch-tools` is CLI tool to compile Wasm programs, mainly for testing purposes.
* Refactor zero local slots
This commit moves the logic of zeroing local slots from the codegen
module into a method with a default implementation in the
MacroAssembler trait: `zero_mem_range`.
The refactored implementation is very similar to the previous
implementation with the only difference
that it doesn't allocates a general-purpose register; it instead uses
the register allocator to retrieve the scratch register and uses this
register to unroll the series of zero stores.
* Tie the codegen creation to the ISA ABI
This commit makes the relationship between the ISA ABI and the codegen
explicit. This allows us to pass down ABI-specific bit and pieces to
the codegeneration. In this case the only concrete piece that we need
is the ABI word size.
* Mark winch as publishable directory
* Revamp winch docs
This commit ensures that all the code comments in Winch are compliant
with the syle used in the rest of Wasmtime's codebase.
It also imptoves, generally the quality of the comments in some modules.
* Panic when using multi-value when the target is aarch64
Similar to x64, this commit ensures that the abi signature of the
current function doesn't use multi-value returns
* Document the usage of directives
* Use endianness instead of endianess in the ISA trait
* Introduce a three-argument form in the MacroAssembler
This commit introduces the usage of three-argument form for the
MacroAssembler interface. This allows for a natural mapping for
architectures like aarch64. In the case of x64, the implementation can
simply restrict the implementation asserting for equality in two of
the arguments of defaulting to a differnt set of instructions.
As of this commit, the implementation of `add` panics if the
destination and the first source arguments are not equal; internally
the x64 assembler implementation will ensure that all the allowed
combinations of `add` are satisfied. The reason for panicking and not
emitting a `mov` followed by an `add` for example is simply because register
allocation happens right before calling `add`, which ensures any
register-to-register moves, if needed.
This implementation will evolve in the future and this panic will be
lifted if needed.
* Improve the documentation for the MacroAssembler.
Documents the usage of three-arg form and the intention around the
high-level interface.
* Format comments in remaining modules
* Clean up Cargo.toml for winch pieces
This commit adds missing fields to each of Winch's Cargo.toml.
* Use `ModuleTranslation::get_types()` to derive the function type
* Assert that start range is always word-size aligned
2 years ago
|
|
|
|
|
|
|
|
wasi-preview1-component-adapter = { path = "crates/wasi-preview1-component-adapter" }
|
|
|
|
|
byte-array-literals = { path = "crates/wasi-preview1-component-adapter/byte-array-literals" }
|
|
|
|
|
|
|
|
|
|
# Bytecode Alliance maintained dependencies:
|
|
|
|
|
# ---------------------------
|
|
|
|
|
regalloc2 = "0.9.3"
|
|
|
|
|
|
|
|
|
|
# cap-std family:
|
|
|
|
|
target-lexicon = { version = "0.12.13", default-features = false, features = ["std"] }
|
|
|
|
|
cap-std = "3.0.0"
|
|
|
|
|
cap-rand = { version = "3.0.0", features = ["small_rng"] }
|
|
|
|
|
cap-fs-ext = "3.0.0"
|
|
|
|
|
cap-net-ext = "3.0.0"
|
|
|
|
|
cap-time-ext = "3.0.0"
|
|
|
|
|
cap-tempfile = "3.0.0"
|
|
|
|
|
fs-set-times = "0.20.1"
|
|
|
|
|
system-interface = { version = "0.27.1", features = ["cap_std_impls"] }
|
|
|
|
|
io-lifetimes = { version = "2.0.3", default-features = false }
|
|
|
|
|
io-extras = "0.18.1"
|
|
|
|
|
rustix = "0.38.31"
|
|
|
|
|
# wit-bindgen:
|
|
|
|
|
wit-bindgen = { version = "0.20.0", default-features = false }
|
|
|
|
|
|
|
|
|
|
# wasm-tools family:
|
|
|
|
|
wasmparser = "0.201.0"
|
|
|
|
|
wat = "1.201.0"
|
|
|
|
|
wast = "201.0.0"
|
|
|
|
|
wasmprinter = "0.201.0"
|
|
|
|
|
wasm-encoder = "0.201.0"
|
|
|
|
|
wasm-smith = "0.201.0"
|
|
|
|
|
wasm-mutate = "0.201.0"
|
|
|
|
|
wit-parser = "0.201.0"
|
|
|
|
|
wit-component = "0.201.0"
|
|
|
|
|
|
|
|
|
|
# Non-Bytecode Alliance maintained dependencies:
|
|
|
|
|
# --------------------------
|
|
|
|
|
object = { version = "0.33", default-features = false, features = ['read_core', 'elf', 'std'] }
|
|
|
|
|
gimli = { version = "0.28.0", default-features = false, features = ['read', 'std'] }
|
|
|
|
|
anyhow = "1.0.22"
|
|
|
|
|
windows-sys = "0.52.0"
|
|
|
|
|
env_logger = "0.10"
|
|
|
|
|
log = { version = "0.4.8", default-features = false }
|
|
|
|
|
clap = { version = "4.3.12", default-features = false, features = ["std", "derive"] }
|
|
|
|
|
hashbrown = { version = "0.14", default-features = false }
|
|
|
|
|
capstone = "0.12.0"
|
|
|
|
|
once_cell = "1.12.0"
|
|
|
|
|
smallvec = { version = "1.6.1", features = ["union"] }
|
|
|
|
|
tracing = "0.1.26"
|
|
|
|
|
bitflags = "2.0"
|
|
|
|
|
thiserror = "1.0.43"
|
|
|
|
|
async-trait = "0.1.71"
|
|
|
|
|
heck = "0.4"
|
|
|
|
|
similar = "2.1.0"
|
|
|
|
|
toml = "0.8.10"
|
|
|
|
|
# serde and serde_derive must have the same version
|
|
|
|
|
serde = "1.0.188"
|
|
|
|
|
serde_derive = "1.0.188"
|
|
|
|
|
serde_json = "1.0.80"
|
|
|
|
|
glob = "0.3.0"
|
Improve longevity for fuzzing corpus of wasm modules (#6322)
* Improve longevity for fuzzing corpus of wasm modules
This commit is an improvement to the longevity of Wasmtime's corpus of
fuzz inputs to the `instantiate` fuzzer. Currently the input to this
fuzzers is arbitrary binary data which is a "DNA" of sorts of what to
do. This DNA changes over time as we update the fuzzer and add
configuration options, for example. When this happens though the
meaning of all existing inputs in the corpus changes because they all
have slightly different meanings now. The goal of this commit is to
improve the usefulness of a historical corpus, with respect to the
WebAssembly modules generated, across changes to the DNA.
A custom mutator is now provided for the `instantiate` fuzzer. This
mutator will not only perform libfuzzer's default mutation for the input
but will additionally place an "envelope" around the fuzz input. Namely,
the fuzz input is encoded as a valid WebAssembly module where the actual
input to the fuzzer is a trailing custom section. When the fuzzer runs
over this input it will read the custom section, perform any
configuration generation necessary, and then use the envelope module as
the actual input to the fuzzer instead of whatever was generated from
the fuzz input. This means that when a future update is made to the DNA
of a module the interpretation of the fuzz input section will change but
the module in question will not change. This means that any interesting
shapes of modules with respect to instructions should be preserved over
time in theory.
Some consequences of this strategy, however, are:
* If the DNA changes then it's difficult to produce minor mutations of
the original module. This is because mutations generate a module based
on the new DNA which is likely much different than the preexisting
module. This mainly just means that libFuzzer will have to rediscover
how to mutate up into interesting shapes on DNA changes but it'll
still be able to retain all the existing interesting modules.
Additionally this can be mitigate with the integration of
`wasm-mutate` perhaps into these fuzzers as well.
* Protection is necessary against libFuzzer itself with respect to the
module. The existing fuzzers only expect valid modules to be created,
but libFuzzer can now create mutations which leave the trailing
section in place, meaning the module is no longer valid. One option is
to record a cryptographic hash in the fuzz input section of the
previous module, only using the module if the hashes match. This
approach will not work over time in the face of binary format changes,
however. For example the multi-memory proposal changed binary
encodings a year or so ago meaning that any previous fuzz-generated
cases would no longer be guaranteed to be valid. The strategy settled
by this PR is to pass a flag to the execution function indicating if
the module is "known valid" and gracefully handle error if it isn't
(for example if it's a prior test case).
I'll note that this new strategy of fuzzing is not applied to the
`differential` fuzzer. This could theoretically use the same strategy
but it relies much more strictly on being able to produce a module with
properties like NaN canonicalization, resource limits, fuel to limit
execution, etc. While it may be possible to integrate this with
`differential` in the future I figured it'd be better to start with the
`instantiate` fuzzer and go from there.
* Fix doc build
2 years ago
|
|
|
libfuzzer-sys = "0.4.0"
|
|
|
|
|
walkdir = "2.3.3"
|
|
|
|
|
cfg-if = "1.0"
|
|
|
|
|
tempfile = "3.1.0"
|
|
|
|
|
filecheck = "0.5.0"
|
|
|
|
|
libc = "0.2.60"
|
|
|
|
|
file-per-thread-logger = "0.2.0"
|
|
|
|
|
tokio = { version = "1.26.0", features = [ "rt", "time" ] }
|
|
|
|
|
hyper = "1.0.1"
|
|
|
|
|
http = "1.0.0"
|
|
|
|
|
http-body = "1.0.0"
|
|
|
|
|
http-body-util = "0.1.0"
|
WASI Preview 2: rewrite streams and pollable implementation (#6556)
* preview2: make everything but streams/io and poll/poll synchronous
* streams: get rid of as_any method, which is no longer used
* delete legacy sched and pollable concepts
* more code motion and renaming
* make tokio a workspace dep, because we need it directly in wasmtime-wasi
* HostPollable exists
* more fixes
* pollable can trap, and implement clock properly
* HostPollable is now a generator of futures
because we need to be able to poll a pollable many times
* explain various todo!s
* Synchronous version of the wasi-preview2-components tests
* Change with_tokio to accept the future as an argument
* Store futures in the PollOneoff struct instead, to avoid dropping them
* Remove TODO for HostOutputStream impl for WritePipe
* Implement pollable for ReadPipe
* Use a Notify when ReadPipe is ready
* wip
* wip
* Read/write pipe ends with tokio channels
* Empty reader/writer wrappers
* EmptyStream, and warning cleanup
* Wrapped reader/writer structs
* Rework stdio in terms of wrapped read/write
* Add MemoryOutputPipe and update tests
* Remove todo
* rewrite nearly everything
* implement the pipe stuff
* wibble
* fix MemoryOutputPipe just enough to make the tests compile
* Move the table iteration into a helper function
* AsyncFd stream implementation to fix stdin on unix
* Rename Wrapped{Read,Write} streams to Async{Read,Write}Stream
* Move async io wrappers into stream.rs
* Fix the sync tests
* fix test uses of pipes, juggle tokio context for stdin construction
* add some fixmes
* the future i named Never is defined in futures-util as pending
which is a better name
* i believe this is a correct implementation of one global stdin resource
* move unix stdin to its own file
* make most of the mods private
* fix build - we are skipping rust 1.70
due to llvm regressions in s390x and riscv64 which are fixed in 1.71 and
will not be backported
* preview1-in-preview2: use async funcs for io, and the async io interface
prtest:full
* windows stdin support
* done!
* table ext functions: fix tests
* tests: expect poll_oneoff_{files,stdio} to pass on all platforms
* export the bindings under wasmtime_wasi::preview2::bindings
rather than preview2::wasi.
and command moves to wasmtime_wasi::preview2::command as well.
* fix renaming of wasi to bindings in tests
* use block_in_place throughout filesystem
and move block_on and block_in_place to be pub crate at the root
* AsyncFdStream: ensure file is nonblocking
* tests: block_in_place requires multi-threaded runtime
* actually, use fcntl_setfl to make the asyncfd file nonblocking
* fix windows block_on
* docs, remove unnecessary methods
* more docs
* Add a workspace dependency on bytes-1.4
* Remove vectored stream operations
* Rework the read/write stream traits
* Add a size parameter to `read`, and switch to usize for traits
* Pipe through the bool -> stream-status change in wit
* Plumb stream-status through write operations in wit
* write host trait also gives streamstate
* hook new stream host read/write back up to the wit bindgen
* sketchy AsyncReadStream impl
* Fill out implementations for AsyncReadStream and AsyncWriteStream
* some reasonable read tests
* more
* first smoke test for AsyncWriteStream
* bunch of AsyncWriteStream tests
* half-baked idea that the output-stream interface will need a flush mechanism
* adapter: fixes for changes to stream wit
* fix new rust 1.71 warnings
* make stdin work on unix without using AsyncFdStream
inline the tokio docs example of how to impl AsyncRead for an AsyncFd,
except theres some "minor" changes because stdin doesnt impl Read on
&Stdin whereas tcpstream from the example does
* delete AsyncFdStream for now
it turns out to be kinda hard and we can always work on adding it back
in later.
* Implement some memory pipe operations, and move async wrappers to the pipe mod
* Make blocking_write actually block until everything is written
* Remove debug print
* Adapter stdio should use blocking write
Rust guests will panic if the write returns less than the number of
bytes sent with stdio.
* Clean up implementations of {blocking_}write_zeros and skip
* Remove debug macro usage
* Move EmptyStream to pipe, and split it into four variants
Use EmptyInputStream and SinkOutputStream as the defaults for stdin and
stdout/stderr respectively.
* Add a big warning about resource lifetime tracking in pollables
* Start working through changes to the filesystem implementation
* Remove todos in the filesystem implementation
* Avoid lifetime errors by moving blocking operations to File and Dir
* Fix more lifetime issues with `block`
* Finish filling out translation impl
* fix warnings
* we can likely eliminate block_in_place in the stdin implementations
* sync command uses sync filesystem, start of translation layer
* symc filesystem: all the trait boilerplate is in place
just need to finish the from impl boilerplate
* finish type conversion boilerplate
* Revert "half-baked idea that the output-stream interface will need a flush mechanism"
This reverts commit 3eb762e3330a7228318bfe01296483b52d0fdc16.
* cargo fmt
* test type fixes
* renames and comments
* refactor stream table internals so we can have a blocking variant...
* preview1 host adapter: stdout/stderr use blocking_write here too
* filesystem streams are blocking now
* fixes
* satisfy cargo doc
* cargo vet: dep upgrades taken care of by imports from mozilla
* unix stdio: eliminate block_in_place
* replace private in_tokio with spawn, since its only used for spawning
* comments
* worker thread stdin implementation can be tested on linux, i guess
and start outlining a test plan
* eliminate tokio boilerplate - no longer using tokios lock
* rename our private block_on to in_tokio
* fill in missing file input skip
* code review: fix MemoryInputPipe. Closed status is always available immediately.
* code review: empty input stream is not essential, closed input stream is a better fi for stdin
* code review: unreachable
* turn worker thread (windows) stdin off
* expect preview2-based poll_oneoff_stdio to fail on windows
* command directory_list test: no need to inherit stdin
* preview1 in preview2: turn off inherit_stdio except for poll_oneoff_stdio
* wasi-preview2-components: apparently inherit_stdio was on everywhere here as well. turn it off
except for poll_oneoff_stdio
* extend timeout for riscv64 i suppose
---------
Co-authored-by: Trevor Elliott <telliott@fastly.com>
1 year ago
|
|
|
bytes = "1.4"
|
|
|
|
|
futures = { version = "0.3.27", default-features = false }
|
|
|
|
|
indexmap = "2.0.0"
|
|
|
|
|
pretty_env_logger = "0.5.0"
|
|
|
|
|
syn = "2.0.25"
|
|
|
|
|
test-log = { version = "0.2", default-features = false, features = ["trace"] }
|
|
|
|
|
tracing-subscriber = { version = "0.3.1", default-features = false, features = ['fmt', 'env-filter', 'ansi', 'tracing-log'] }
|
|
|
|
|
url = "2.3.1"
|
|
|
|
|
humantime = "2.0.0"
|
|
|
|
|
|
|
|
|
|
# =============================================================================
|
|
|
|
|
#
|
|
|
|
|
# Features for the Wasmtime CLI executable
|
|
|
|
|
#
|
|
|
|
|
#
|
|
|
|
|
# Note that many of these features are inherited from Wasmtime itself or
|
|
|
|
|
# otherwise configure the `wasmtime` crate's execution. Features are provided as
|
|
|
|
|
# compile-time switches to disable functionality primarily if one is interested
|
|
|
|
|
# in configuring binary size and or exploring the binary size implications of
|
|
|
|
|
# various features. Most features are enabled by default but most embeddings
|
|
|
|
|
# likely won't need all features.
|
|
|
|
|
[features]
|
|
|
|
|
default = [
|
|
|
|
|
# All subcommands are included by default.
|
|
|
|
|
"run",
|
|
|
|
|
"compile",
|
|
|
|
|
"explore",
|
|
|
|
|
"serve",
|
|
|
|
|
"wast",
|
|
|
|
|
"config",
|
|
|
|
|
|
|
|
|
|
# On-by-default WASI features
|
|
|
|
|
"wasi-nn",
|
|
|
|
|
"wasi-threads",
|
|
|
|
|
"wasi-http",
|
|
|
|
|
|
|
|
|
|
# Most features of Wasmtime are enabled by default.
|
|
|
|
|
"wat",
|
|
|
|
|
"parallel-compilation",
|
|
|
|
|
"pooling-allocator",
|
|
|
|
|
"cache",
|
|
|
|
|
"logging",
|
|
|
|
|
"demangle",
|
|
|
|
|
"cranelift",
|
|
|
|
|
"profiling",
|
|
|
|
|
"coredump",
|
|
|
|
|
"addr2line",
|
|
|
|
|
"debug-builtins",
|
|
|
|
|
"component-model",
|
|
|
|
|
|
|
|
|
|
# Enable some nice features of clap by default, but they come at a binary size
|
|
|
|
|
# cost, so allow disabling this through disabling of our own `default`
|
|
|
|
|
# feature.
|
|
|
|
|
"clap/default",
|
|
|
|
|
|
|
|
|
|
# By default include compatibility with the "old" CLI from Wasmtime 13 and
|
|
|
|
|
# prior.
|
|
|
|
|
"old-cli",
|
|
|
|
|
]
|
|
|
|
|
|
|
|
|
|
# ========================================
|
|
|
|
|
# Off-by-default features
|
|
|
|
|
#
|
|
|
|
|
# These features are off-by-default but may optionally be enabled.
|
|
|
|
|
all-arch = ["wasmtime/all-arch"]
|
|
|
|
|
winch = ["wasmtime/winch"]
|
|
|
|
|
wmemcheck = ["wasmtime/wmemcheck"]
|
|
|
|
|
|
|
|
|
|
# This feature, when enabled, will statically compile out all logging statements
|
|
|
|
|
# throughout Wasmtime and its dependencies.
|
|
|
|
|
disable-logging = ["log/max_level_off", "tracing/max_level_off"]
|
|
|
|
|
|
|
|
|
|
# ========================================
|
|
|
|
|
# On-by-default features
|
|
|
|
|
#
|
|
|
|
|
# These features are all included in the `default` set above and this is
|
|
|
|
|
# the internal mapping for what they enable in Wasmtime itself.
|
|
|
|
|
wasi-nn = ["dep:wasmtime-wasi-nn"]
|
wasi-threads: an initial implementation (#5484)
This commit includes a set of changes that add initial support for `wasi-threads` to Wasmtime:
* feat: remove mutability from the WasiCtx Table
This patch adds interior mutability to the WasiCtx Table and the Table elements.
Major pain points:
* `File` only needs `RwLock<cap_std::fs::File>` to implement
`File::set_fdflags()` on Windows, because of [1]
* Because `File` needs a `RwLock` and `RwLock*Guard` cannot
be hold across an `.await`, The `async` from
`async fn num_ready_bytes(&self)` had to be removed
* Because `File` needs a `RwLock` and `RwLock*Guard` cannot
be dereferenced in `pollable`, the signature of
`fn pollable(&self) -> Option<rustix::fd::BorrowedFd>`
changed to `fn pollable(&self) -> Option<Arc<dyn AsFd + '_>>`
[1] https://github.com/bytecodealliance/system-interface/blob/da238e324e752033f315f09c082ad9ce35d42696/src/fs/fd_flags.rs#L210-L217
* wasi-threads: add an initial implementation
This change is a first step toward implementing `wasi-threads` in
Wasmtime. We may find that it has some missing pieces, but the core
functionality is there: when `wasi::thread_spawn` is called by a running
WebAssembly module, a function named `wasi_thread_start` is found in the
module's exports and called in a new instance. The shared memory of the
original instance is reused in the new instance.
This new WASI proposal is in its early stages and details are still
being hashed out in the [spec] and [wasi-libc] repositories. Due to its
experimental state, the `wasi-threads` functionality is hidden behind
both a compile-time and runtime flag: one must build with `--features
wasi-threads` but also run the Wasmtime CLI with `--wasm-features
threads` and `--wasi-modules experimental-wasi-threads`. One can
experiment with `wasi-threads` by running:
```console
$ cargo run --features wasi-threads -- \
--wasm-features threads --wasi-modules experimental-wasi-threads \
<a threads-enabled module>
```
Threads-enabled Wasm modules are not yet easy to build. Hopefully this
is resolved soon, but in the meantime see the use of
`THREAD_MODEL=posix` in the [wasi-libc] repository for some clues on
what is necessary. Wiggle complicates things by requiring the Wasm
memory to be exported with a certain name and `wasi-threads` also
expects that memory to be imported; this build-time obstacle can be
overcome with the `--import-memory --export-memory` flags only available
in the latest Clang tree. Due to all of this, the included tests are
written directly in WAT--run these with:
```console
$ cargo test --features wasi-threads -p wasmtime-cli -- cli_tests
```
[spec]: https://github.com/WebAssembly/wasi-threads
[wasi-libc]: https://github.com/WebAssembly/wasi-libc
This change does not protect the WASI implementations themselves from
concurrent access. This is already complete in previous commits or left
for future commits in certain cases (e.g., wasi-nn).
* wasi-threads: factor out process exit logic
As is being discussed [elsewhere], either calling `proc_exit` or
trapping in any thread should halt execution of all threads. The
Wasmtime CLI already has logic for adapting a WebAssembly error code to
a code expected in each OS. This change factors out this logic to a new
function, `maybe_exit_on_error`, for use within the `wasi-threads`
implementation.
This will work reasonably well for CLI users of Wasmtime +
`wasi-threads`, but embedders will want something better in the future:
when a `wasi-threads` threads fails, they may not want their application
to exit. Handling this is tricky, because it will require cancelling the
threads spawned by the `wasi-threads` implementation, something that is
not trivial to do in Rust. With this change, we defer that work until
later in order to provide a working implementation of `wasi-threads` for
experimentation.
[elsewhere]: https://github.com/WebAssembly/wasi-threads/pull/17
* review: work around `fd_fdstat_set_flags`
In order to make progress with wasi-threads, this change temporarily
works around limitations induced by `wasi-common`'s
`fd_fdstat_set_flags` to allow `&mut self` use in the implementation.
Eventual resolution is tracked in
https://github.com/bytecodealliance/wasmtime/issues/5643. This change
makes several related helper functions (e.g., `set_fdflags`) take `&mut
self` as well.
* test: use `wait`/`notify` to improve `threads.wat` test
Previously, the test simply executed in a loop for some hardcoded number
of iterations. This changes uses `wait` and `notify` and atomic
operations to keep track of when the spawned threads are done and join
on the main thread appropriately.
* various fixes and tweaks due to the PR review
---------
Signed-off-by: Harald Hoyer <harald@profian.com>
Co-authored-by: Harald Hoyer <harald@profian.com>
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
2 years ago
|
|
|
wasi-threads = ["dep:wasmtime-wasi-threads"]
|
|
|
|
|
wasi-http = ["component-model", "dep:wasmtime-wasi-http", "dep:tokio", "dep:hyper"]
|
|
|
|
|
pooling-allocator = ["wasmtime/pooling-allocator", "wasmtime-cli-flags/pooling-allocator"]
|
|
|
|
|
component-model = [
|
|
|
|
|
"wasmtime/component-model",
|
|
|
|
|
"wasmtime-wast?/component-model",
|
|
|
|
|
"wasmtime-cli-flags/component-model"
|
|
|
|
|
]
|
|
|
|
|
wat = ["dep:wat", "wasmtime/wat"]
|
|
|
|
|
cache = ["dep:wasmtime-cache", "wasmtime-cli-flags/cache"]
|
|
|
|
|
parallel-compilation = ["wasmtime-cli-flags/parallel-compilation"]
|
|
|
|
|
logging = ["wasmtime-cli-flags/logging"]
|
|
|
|
|
demangle = ["wasmtime/demangle"]
|
|
|
|
|
cranelift = ["wasmtime-cli-flags/cranelift", "dep:wasmtime-cranelift"]
|
|
|
|
|
profiling = ["wasmtime/profiling"]
|
|
|
|
|
coredump = ["wasmtime-cli-flags/coredump"]
|
|
|
|
|
addr2line = ["wasmtime/addr2line"]
|
|
|
|
|
debug-builtins = ["wasmtime/debug-builtins"]
|
|
|
|
|
|
|
|
|
|
# Enables compatibility shims with Wasmtime 13 and prior's CLI.
|
|
|
|
|
old-cli = []
|
|
|
|
|
|
|
|
|
|
# CLI subcommands for the `wasmtime` executable. See `wasmtime $cmd --help`
|
|
|
|
|
# for more information on each subcommand.
|
|
|
|
|
serve = ["wasi-http", "component-model", "dep:http-body-util", "dep:http"]
|
|
|
|
|
explore = ["dep:wasmtime-explorer"]
|
|
|
|
|
wast = ["dep:wasmtime-wast"]
|
|
|
|
|
config = ["cache"]
|
|
|
|
|
compile = ["cranelift"]
|
|
|
|
|
run = ["dep:wasmtime-wasi", "wasmtime/runtime", "wasmtime-runtime", "dep:listenfd", "dep:wasi-common"]
|
|
|
|
|
|
|
|
|
|
[[test]]
|
|
|
|
|
name = "host_segfault"
|
|
|
|
|
harness = false
|
externref: implement stack map-based garbage collection
For host VM code, we use plain reference counting, where cloning increments
the reference count, and dropping decrements it. We can avoid many of the
on-stack increment/decrement operations that typically plague the
performance of reference counting via Rust's ownership and borrowing system.
Moving a `VMExternRef` avoids mutating its reference count, and borrowing it
either avoids the reference count increment or delays it until if/when the
`VMExternRef` is cloned.
When passing a `VMExternRef` into compiled Wasm code, we don't want to do
reference count mutations for every compiled `local.{get,set}`, nor for
every function call. Therefore, we use a variation of **deferred reference
counting**, where we only mutate reference counts when storing
`VMExternRef`s somewhere that outlives the activation: into a global or
table. Simultaneously, we over-approximate the set of `VMExternRef`s that
are inside Wasm function activations. Periodically, we walk the stack at GC
safe points, and use stack map information to precisely identify the set of
`VMExternRef`s inside Wasm activations. Then we take the difference between
this precise set and our over-approximation, and decrement the reference
count for each of the `VMExternRef`s that are in our over-approximation but
not in the precise set. Finally, the over-approximation is replaced with the
precise set.
The `VMExternRefActivationsTable` implements the over-approximized set of
`VMExternRef`s referenced by Wasm activations. Calling a Wasm function and
passing it a `VMExternRef` moves the `VMExternRef` into the table, and the
compiled Wasm function logically "borrows" the `VMExternRef` from the
table. Similarly, `global.get` and `table.get` operations clone the gotten
`VMExternRef` into the `VMExternRefActivationsTable` and then "borrow" the
reference out of the table.
When a `VMExternRef` is returned to host code from a Wasm function, the host
increments the reference count (because the reference is logically
"borrowed" from the `VMExternRefActivationsTable` and the reference count
from the table will be dropped at the next GC).
For more general information on deferred reference counting, see *An
Examination of Deferred Reference Counting and Cycle Detection* by Quinane:
https://openresearch-repository.anu.edu.au/bitstream/1885/42030/2/hon-thesis.pdf
cc #929
Fixes #1804
4 years ago
|
|
|
|
|
|
|
|
[[example]]
|
|
|
|
|
name = "tokio"
|
|
|
|
|
required-features = ["wasi-common/tokio"]
|
|
|
|
|
|
|
|
|
|
[[bench]]
|
|
|
|
|
name = "instantiation"
|
|
|
|
|
harness = false
|
|
|
|
|
|
|
|
|
|
[[bench]]
|
|
|
|
|
name = "thread_eager_init"
|
|
|
|
|
harness = false
|
|
|
|
|
|
|
|
|
|
[[bench]]
|
|
|
|
|
name = "trap"
|
|
|
|
|
harness = false
|
|
|
|
|
|
|
|
|
|
[[bench]]
|
|
|
|
|
name = "call"
|
|
|
|
|
harness = false
|
|
|
|
|
|
|
|
|
|
[[bench]]
|
|
|
|
|
name = "wasi"
|
|
|
|
|
harness = false
|
Begin implementation of wasi-http (#5929)
* Integrate experimental HTTP into wasmtime.
* Reset Cargo.lock
* Switch to bail!, plumb options partially.
* Implement timeouts.
* Remove generated files & wasm, add Makefile
* Remove generated code textfile
* Update crates/wasi-http/Cargo.toml
Co-authored-by: Eduardo de Moura Rodrigues <16357187+eduardomourar@users.noreply.github.com>
* Update crates/wasi-http/Cargo.toml
Co-authored-by: Eduardo de Moura Rodrigues <16357187+eduardomourar@users.noreply.github.com>
* Extract streams from request/response.
* Fix read for len < buffer length.
* Formatting.
* types impl: swap todos for traps
* streams_impl: idioms, and swap todos for traps
* component impl: idioms, swap all unwraps for traps, swap all todos for traps
* http impl: idiom
* Remove an unnecessary mut.
* Remove an unsupported function.
* Switch to the tokio runtime for the HTTP request.
* Add a rust example.
* Update to latest wit definition
* Remove example code.
* wip: start writing a http test...
* finish writing the outbound request example
havent executed it yet
* better debug output
* wasi-http: some stubs required for rust rewrite of the example
* add wasi_http tests to test-programs
* CI: run the http tests
* Fix some warnings.
* bump new deps to latest releases (#3)
* Add tests for wasi-http to test-programs (#2)
* wip: start writing a http test...
* finish writing the outbound request example
havent executed it yet
* better debug output
* wasi-http: some stubs required for rust rewrite of the example
* add wasi_http tests to test-programs
* CI: run the http tests
* bump new deps to latest releases
h2 0.3.16
http 0.2.9
mio 0.8.6
openssl 0.10.48
openssl-sys 0.9.83
tokio 1.26.0
---------
Co-authored-by: Brendan Burns <bburns@microsoft.com>
* Update crates/test-programs/tests/http_tests/runtime/wasi_http_tests.rs
* Update crates/test-programs/tests/http_tests/runtime/wasi_http_tests.rs
* Update crates/test-programs/tests/http_tests/runtime/wasi_http_tests.rs
* wasi-http: fix cargo.toml file and publish script to work together (#4)
unfortunately, the publish script doesn't use a proper toml parser (in
order to not have any dependencies), so the whitespace has to be the
trivial expected case.
then, add wasi-http to the list of crates to publish.
* Update crates/test-programs/build.rs
* Switch to rustls
* Cleanups.
* Merge switch to rustls.
* Formatting
* Remove libssl install
* Fix tests.
* Rename wasi-http -> wasmtime-wasi-http
* prtest:full
Conditionalize TLS on riscv64gc.
* prtest:full
Fix formatting, also disable tls on s390x
* prtest:full
Add a path parameter to wit-bindgen, remove symlink.
* prtest:full
Fix tests for places where SSL isn't supported.
* Update crates/wasi-http/Cargo.toml
---------
Co-authored-by: Eduardo de Moura Rodrigues <16357187+eduardomourar@users.noreply.github.com>
Co-authored-by: Pat Hickey <phickey@fastly.com>
Co-authored-by: Pat Hickey <pat@moreproductive.org>
2 years ago
|
|
|
|
|
|
|
|
[profile.release.package.wasi-preview1-component-adapter]
|
|
|
|
|
opt-level = 's'
|
|
|
|
|
strip = 'debuginfo'
|
|
|
|
|
|
|
|
|
|
[profile.dev.package.wasi-preview1-component-adapter]
|
|
|
|
|
# Make dev look like a release build since this adapter module won't work with
|
|
|
|
|
# a debug build that uses data segments and such.
|
|
|
|
|
incremental = false
|
|
|
|
|
opt-level = 's'
|
|
|
|
|
# Omit assertions, which include failure messages which require string
|
|
|
|
|
# initializers.
|
|
|
|
|
debug-assertions = false
|
|
|
|
|
# Omit integer overflow checks, which include failure messages which require
|
|
|
|
|
# string initializers.
|
|
|
|
|
overflow-checks = false
|
|
|
|
|
|
|
|
|
|
# Same as `wasi-preview1-component-adapter` above
|
|
|
|
|
[profile.dev.package.wit-bindgen]
|
|
|
|
|
incremental = false
|
|
|
|
|
debug-assertions = false
|
|
|
|
|
overflow-checks = false
|
|
|
|
|
opt-level = 's'
|
