cranelift

Commit Graph

Author	SHA1	Message	Date
Xinzhao Xu	bfc10d967a	wasi-keyvalue: remove the Redis provider (#9062 ) * wasi-keyvalue: remove the Redis provider * fix cargo check * cargo vet prune	3 months ago
Xinzhao Xu	dfc4358d0c	Implement wasi-keyvalue (#8983 ) * Implement wasi-keyvalue * Allow preset data for In-Memory provider, rename allow_hosts to allow_redis_hosts * Add vets --------- Co-authored-by: Alex Crichton <alex@alexcrichton.com>	3 months ago
Nick Fitzgerald	934bf9dbaf	CI: shard testing crates across multiple jobs (#8612 ) * CI: shard testing and checking crates across multiple jobs * prtest:full	6 months ago
Alex Crichton	f4be360648	Refactor the test-programs test suite (#7182 ) * Refactor the test-programs test suite This commit is a large refactoring that reorganizes `test-programs` and how we tests wasms in Wasmtime. Often writing tests requires complicated interactions with the guest which can't be done via hand-written `.wat` syntax and requires a compiler to get engaged. For this purpose Wasmtime currently has the `crates/test-programs/` test suite which builds files from source and then runs the tests. This has been somewhat cumbersome in the past though and it's not been easy to extend this over time, so this commit attempts to address this. The scheme implemented in this PR looks like: * All wasm test programs live in `crates/test-programs/src/bin/.rs`. All of them, no exceptions. Wasm tests have shared support located at `crates/test-programs/src/lib.rs` and its submodules, such as bindings generation for WASI. * Wasm tests are built by a new `crates/test-programs/artifacts` crate. This crate compiles modules and additionally creates components for all test programs. The crate itself only records the path to these outputs and a small amount of testing support, but otherwise doesn't interact with `wasmtime`-the-crate itself. * All tests in `crates/test-programs/tests/.rs` have moved. For example wasi-http tests now live at `crates/wasi-http/tests/.rs`. Legacy tests of wasi-common now live at `crates/wasi-common/tests/.rs`. Modern tests for preview2 live at `crates/wasi/tests/.rs`. * Wasm tests are bucketed based on their filename prefix. For example `preview1_` is tested in wasi-common and wasmtime-wasi. The `preview2_` prefix is only tested with wasmtime-wasi, however. * A new `cli_` prefix is used to execute tests as part of `tests/all/main.rs`. This is a new submodule in `tests/all/cli_tests.rs` which executes these components on the command line. Many old "command" tests were migrated here. Helper macros are generated to assert that a test suite is run in its entirety. This way if a `preview1_` test is added it's asserted to get added to both wasi-common and wasmtime-wasi in the various modes they run tests. Overall this moved a number of tests around and refactored some edges of the tests, but this should not lose any tests (except one that wasn't actually testing anything). Additionally the hope is that it's much easier to add tests in the future. The process is to add a new file in `crates/test-programs/src/bin/.rs` named appropriately. For example a preview2 executable is `preview2_` and a CLI tests is `cli_`. When building the test suite an error is generated in the appropriate module then of "please write a test here", and then a test is written in the same manner as the other tests in the module. * Remove no-longer-needed fetches prtest:full * I'm worried wasi is running low on semicolons * Add the WASI target in all CI actions * Add unknown-unknown target on all CI builders too * Fix building test artifacts under miri Need to avoid wrappers for these cross-compiled targets * Break circular dependency for packaging Don't use the workspace dep for `wasmtime-wasi` since it injects a version, instead use a `path = '..'` dependency to fool Cargo into dropping the dependency during the package phase. * Fix some merge conflicts with tests * Fix rebase for new tests * Remove stray comment * Fix some flaky tests * Fix network tests in synchronous mode This commit is an attempt to fix some networking tests in synchronous mode in our test suite. Currently networking tests don't actually run in synchronous mode on CI which is why no failures have been surfaced yet, but the refactoring in #7182 is going to start doing this. Currently the `udp_sample_application.rs` test blocks infinitely in synchronous mode for me locally, most of the time. This appears to be an interaction between how Tokio handles readiness and how we're entering the event loop. We're effectively entering the Tokio event loop with a future that's always ready which ends up starving Tokio of otherwise performing its background work such as updating flags for readiness of reading/writing. The fix here is to add a yield at the start of an `in_tokio` block which is used in synchronous mode. This is a kludge fix but the intention is to enable Tokio to have a chance to update readiness flags and process events from epoll/kqueue/etc. An additional fix to this issue is WebAssembly/wasi-sockets#64 where the test is waiting on `READABLE` or `WRITABLE`, but in this specific case it should only wait on `READABLE`. If it waited on just this then that would also fix this issue. Nevertheless having a `yield_now` is expected to have little-to-no overhead and otherwise fix this edge case of an always-ready future. * Fix passing empty arguments on the CLI * Add another blocking accept * Update crates/test-programs/src/bin/api_proxy.rs Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com> --------- Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>	1 year ago
Trevor Elliott	9d3419741c	Fix a regression with `wasmtime serve`, and enable the serve feature in tests (#7137 )	1 year ago
Alex Crichton	367bdc8c66	Refactor Wasmtime CLI to support components (#6836 ) This commit refactors the `wasmtime` CLI executable to be able to support not only compiling components but additionally executing components. While I was doing this I've additionally added a new `--preview2` argument to enable using the new experimental implementation of preview1 based on preview2 type/structs. This is off-by-default but is expected to become the default in the future. Some notable features of this change are: * The preview1-implemented-with-preview2 module now sports `add_to_linker_{async,sync}` to replace the previous `add_to_linker` which always did async. * Some trait bounds in the preview1-implemented-with-preview2 module are simplified. * Some minor changes were made to `wiggle`'s macros to support a "block on" that isn't the default wiggle dummy executor (as now we actually do need Tokio) * Many options related to core wasm `Linker` configuration, such as `--default-values-unknown-imports`, are not implemented for components at this time. When used with components these options return an error. * Construction of WASI contexts has been refactored to pass around fewer arguments to avoid threading through lots of values for both preview1 and preview2. * Reading the input to the Wasmtime CLI has been updated to read the input in the `run` subcommand before handing it off to the `wasmtime` crate's API to enable the CLI to use the contents of what's loaded to determine what to do next. * Our generic `./ci/run-tests.sh` script has been updated to pass the `--features component-model` flag to ensure that this CLI support is tested during the normal test suite. * The CLI support for `wasi-nn` supports components as well as core wasm modules.	1 year ago
Eduardo de Moura Rodrigues	e250334b8e	feat: align wasi-http with component linker (#6195 ) * feat: align wasi-http with component linker * feat(wasi-http): allow bidirectional stream * feat(wasi-http): clean up children when dropping resource * chore: update based on feedback * chore: replace wasi http context references * chore: fix logical issue with outgoing body stream * chore: use thread-safe reference-counting pointer * chore: cleanup resources using table * fix(wasi-preview1-component-adapter): sync command extended wit * fix(wasi-preview1-component-adapter): sync command extended wit * fix(wasmtime-wasi): sync wit for http types * chore: refactor using wasmtime-wasi crate fix(wasi-http): misconfiguration in wasmtime linkers chore: keep streams details chore: fix wasi http tests * chore: use pollable from wasmtime-wasi * chore: update wasi http linker for module * chore: update test programs for wasi http * fix(wasi-http): ensure proper errors are surfaced * chore: split wasi http tests into individual files * chore: ensure protocol error is mapped correctly * chore: disable temporarily wasi http in wasmtime cli * chore: comment out wasi http in wasmtime cli * chore(ci): ensure wit definitions in sync * feat(wasi-http): generate async host binding * chore: make wasi http tests async * chore: update ci workflow based on suggestion Co-authored-by: Pat Hickey <pat@moreproductive.org> * feat(wasmtime-wasi): update logging world to latest * feat(wasmtime): update proxy world to latest * feat(wasmtime-wasi): add back command extended world * fix(wasi-http): sync wit definitions * chore: update tests with latest wit definitions * Update src/commands/run.rs * Update src/commands/run.rs * Update src/commands/run.rs * Update src/commands/run.rs * Update src/commands/run.rs * Update src/commands/run.rs * Update src/commands/run.rs * Update src/commands/run.rs * Update src/commands/run.rs * chore: fix formatting * Ignore flaky test * chore: fix compilation error for riscv64 arch * Avoid `cp -T` on macos Adding prtest:full to ensure that we've seen a successful build before queuing. * Don't build the wasi-http test programs for the native target * Debug the wit consistency check * Update streams.wit in wasi-http * Mark the component outbound_request_post test flaky * Disable flaky wasi-http-tests on windows only * Use diff instead of rm/cp/git diff * Disable more tests on windows --------- Co-authored-by: Eduardo Rodrigues <eduardomourar@users.noreply.github.com> Co-authored-by: Pat Hickey <pat@moreproductive.org> Co-authored-by: Trevor Elliott <telliott@fastly.com>	1 year ago
Alex Crichton	2897e409db	Remove the implementation of wasi-crypto (#6816 ) * Remove the implementation of wasi-crypto This commit is a follow-up to the discussion on #6732. This removes Wasmtime's implementation of the wasi-crypto proposal from in-tree along with its various support in CI, configuration, etc. See the discussion on #6732 for the full information but at a high level the main reasons for removing the implementation at this time are: * There is not currently an active maintainer of the Wasmtime integration here for wasi-crypto. * There are known issues with the code quality of the implementation such as transmutes of guest-owned memory to `&'static mut [u8]` and known unsafety in dependencies. * The size and breadth of the dependency tree brings maintenance burden and overhead to managing Wasmtime's dependency tree. As mentioned on the issue this commit does not mean that Wasmtime doesn't want to implement the wasi-crypto proposal. Instead the "tier 3" status of wasi-crypto needs to be re-attained to be included back in-tree, which would mean resolving the above issues. Note that this commit is intentionally just after the 13.0.0 branch point which means that this is slated for Wasmtime 14 to be released on September 20. * Remove some cfgs * Remove wasi-crypto CI	1 year ago
Brendan Burns	fc11f56318	Re-enable wasi-http for recent wit bindgen changes. Renable tests. (#6495 ) * Re-enable wasi-http * Address comments.	1 year ago
Pat Hickey	224866122f	exclude command-tests and reactor-tests wasm crates from run-tests	1 year ago
Pat Hickey	338b535a3a	Refactor test-programs to build modules and components (#6385 ) * wasi-tests and wasi-http-tests no longer have their own workspace * wasi-tests: fix warnings * rewrite the test-programs build.rs to generate {package}_modules.rs and _components.rs The style is cribbed from preview2-prototying repo, but I ended up refactoring it a bit. * better escaping should help with windows? * long form cap-std-sync and tokio test suites * convert wasi-http test * fixes, comments * apply cargo fmt to whole workspace * bump test-programs and wasi-http-tests to all use common dependency versions wit-bindgen 0.6.0 and wit-component 0.7.4 * add new audits * cargo vet prune * package and supply chain updates to fix vulnerabilities h2 upgraded from 0.3.16 -> 0.3.19 to fix vulnerability tempfile upgraded from 0.3.3 -> 0.3.5 to eliminate dep on vulnerable remove_dir_all * deny: temporarily allow duplicate wasm-encoder, wasmparser, wit-parser prtest:full * convert more dependencies to { workspace = true } Alex asked me to do thsi for wit-component and wit-bindgen, and I found a few more (cfg-if, tempfile, filecheck, anyhow... I also reorganized the workspace dependencies section to make the ones our team maintains more clearly separated from our external dependencies. * test-programs build: ensure that the user writes a #[test] for each module, component * fix build of wasi-tests on windows * misspelled macos * mark wasi-tests crate test=false so we dont try building it natively... * mark wasi-http-tests test=false as well * try getting the cargo keys right * just exclude wasi-tests and wasi-http-tests in run-tests.sh * interesting paths fails on windows * misspelling so nice i did it twice * new cargo deny exception: ignore all of wit-bindgen's dependencies * auto-import wildcard vets	1 year ago
Pat Hickey	9c61c7454d	exclude adapter from tests build with manifest instead of run-tests.sh	2 years ago
Pat Hickey	e2e9aba4c8	run-tests: exclude the adapter, which doesnt build for native	2 years ago
Brendan Burns	2d34dbef4b	Begin implementation of wasi-http (#5929 ) * Integrate experimental HTTP into wasmtime. * Reset Cargo.lock * Switch to bail!, plumb options partially. * Implement timeouts. * Remove generated files & wasm, add Makefile * Remove generated code textfile * Update crates/wasi-http/Cargo.toml Co-authored-by: Eduardo de Moura Rodrigues <16357187+eduardomourar@users.noreply.github.com> * Update crates/wasi-http/Cargo.toml Co-authored-by: Eduardo de Moura Rodrigues <16357187+eduardomourar@users.noreply.github.com> * Extract streams from request/response. * Fix read for len < buffer length. * Formatting. * types impl: swap todos for traps * streams_impl: idioms, and swap todos for traps * component impl: idioms, swap all unwraps for traps, swap all todos for traps * http impl: idiom * Remove an unnecessary mut. * Remove an unsupported function. * Switch to the tokio runtime for the HTTP request. * Add a rust example. * Update to latest wit definition * Remove example code. * wip: start writing a http test... * finish writing the outbound request example havent executed it yet * better debug output * wasi-http: some stubs required for rust rewrite of the example * add wasi_http tests to test-programs * CI: run the http tests * Fix some warnings. * bump new deps to latest releases (#3) * Add tests for wasi-http to test-programs (#2) * wip: start writing a http test... * finish writing the outbound request example havent executed it yet * better debug output * wasi-http: some stubs required for rust rewrite of the example * add wasi_http tests to test-programs * CI: run the http tests * bump new deps to latest releases h2 0.3.16 http 0.2.9 mio 0.8.6 openssl 0.10.48 openssl-sys 0.9.83 tokio 1.26.0 --------- Co-authored-by: Brendan Burns <bburns@microsoft.com> * Update crates/test-programs/tests/http_tests/runtime/wasi_http_tests.rs * Update crates/test-programs/tests/http_tests/runtime/wasi_http_tests.rs * Update crates/test-programs/tests/http_tests/runtime/wasi_http_tests.rs * wasi-http: fix cargo.toml file and publish script to work together (#4) unfortunately, the publish script doesn't use a proper toml parser (in order to not have any dependencies), so the whitespace has to be the trivial expected case. then, add wasi-http to the list of crates to publish. * Update crates/test-programs/build.rs * Switch to rustls * Cleanups. * Merge switch to rustls. * Formatting * Remove libssl install * Fix tests. * Rename wasi-http -> wasmtime-wasi-http * prtest:full Conditionalize TLS on riscv64gc. * prtest:full Fix formatting, also disable tls on s390x * prtest:full Add a path parameter to wit-bindgen, remove symlink. * prtest:full Fix tests for places where SSL isn't supported. * Update crates/wasi-http/Cargo.toml --------- Co-authored-by: Eduardo de Moura Rodrigues <16357187+eduardomourar@users.noreply.github.com> Co-authored-by: Pat Hickey <phickey@fastly.com> Co-authored-by: Pat Hickey <pat@moreproductive.org>	2 years ago
Andrew Brown	edfa10d607	wasi-threads: an initial implementation (#5484 ) This commit includes a set of changes that add initial support for `wasi-threads` to Wasmtime: * feat: remove mutability from the WasiCtx Table This patch adds interior mutability to the WasiCtx Table and the Table elements. Major pain points: * `File` only needs `RwLock<cap_std::fs::File>` to implement `File::set_fdflags()` on Windows, because of [1] * Because `File` needs a `RwLock` and `RwLockGuard` cannot be hold across an `.await`, The `async` from `async fn num_ready_bytes(&self)` had to be removed Because `File` needs a `RwLock` and `RwLockGuard` cannot be dereferenced in `pollable`, the signature of `fn pollable(&self) -> Option<rustix::fd::BorrowedFd>` changed to `fn pollable(&self) -> Option<Arc<dyn AsFd + '_>>` [1] `da238e324e/src/fs/fd_flags.rs (L210-L217)` wasi-threads: add an initial implementation This change is a first step toward implementing `wasi-threads` in Wasmtime. We may find that it has some missing pieces, but the core functionality is there: when `wasi::thread_spawn` is called by a running WebAssembly module, a function named `wasi_thread_start` is found in the module's exports and called in a new instance. The shared memory of the original instance is reused in the new instance. This new WASI proposal is in its early stages and details are still being hashed out in the [spec] and [wasi-libc] repositories. Due to its experimental state, the `wasi-threads` functionality is hidden behind both a compile-time and runtime flag: one must build with `--features wasi-threads` but also run the Wasmtime CLI with `--wasm-features threads` and `--wasi-modules experimental-wasi-threads`. One can experiment with `wasi-threads` by running: ```console $ cargo run --features wasi-threads -- \ --wasm-features threads --wasi-modules experimental-wasi-threads \ <a threads-enabled module> ``` Threads-enabled Wasm modules are not yet easy to build. Hopefully this is resolved soon, but in the meantime see the use of `THREAD_MODEL=posix` in the [wasi-libc] repository for some clues on what is necessary. Wiggle complicates things by requiring the Wasm memory to be exported with a certain name and `wasi-threads` also expects that memory to be imported; this build-time obstacle can be overcome with the `--import-memory --export-memory` flags only available in the latest Clang tree. Due to all of this, the included tests are written directly in WAT--run these with: ```console $ cargo test --features wasi-threads -p wasmtime-cli -- cli_tests ``` [spec]: https://github.com/WebAssembly/wasi-threads [wasi-libc]: https://github.com/WebAssembly/wasi-libc This change does not protect the WASI implementations themselves from concurrent access. This is already complete in previous commits or left for future commits in certain cases (e.g., wasi-nn). * wasi-threads: factor out process exit logic As is being discussed [elsewhere], either calling `proc_exit` or trapping in any thread should halt execution of all threads. The Wasmtime CLI already has logic for adapting a WebAssembly error code to a code expected in each OS. This change factors out this logic to a new function, `maybe_exit_on_error`, for use within the `wasi-threads` implementation. This will work reasonably well for CLI users of Wasmtime + `wasi-threads`, but embedders will want something better in the future: when a `wasi-threads` threads fails, they may not want their application to exit. Handling this is tricky, because it will require cancelling the threads spawned by the `wasi-threads` implementation, something that is not trivial to do in Rust. With this change, we defer that work until later in order to provide a working implementation of `wasi-threads` for experimentation. [elsewhere]: https://github.com/WebAssembly/wasi-threads/pull/17 * review: work around `fd_fdstat_set_flags` In order to make progress with wasi-threads, this change temporarily works around limitations induced by `wasi-common`'s `fd_fdstat_set_flags` to allow `&mut self` use in the implementation. Eventual resolution is tracked in https://github.com/bytecodealliance/wasmtime/issues/5643. This change makes several related helper functions (e.g., `set_fdflags`) take `&mut self` as well. * test: use `wait`/`notify` to improve `threads.wat` test Previously, the test simply executed in a loop for some hardcoded number of iterations. This changes uses `wait` and `notify` and atomic operations to keep track of when the spawned threads are done and join on the main thread appropriately. * various fixes and tweaks due to the PR review --------- Signed-off-by: Harald Hoyer <harald@profian.com> Co-authored-by: Harald Hoyer <harald@profian.com> Co-authored-by: Alex Crichton <alex@alexcrichton.com>	2 years ago
Nick Fitzgerald	d2d0a0f36b	Remove Peepmatic!!! Peepmatic was an early attempt at a DSL for peephole optimizations, with the idea that maybe sometime in the future we could user it for instruction selection as well. It didn't really pan out, however: * Peepmatic wasn't quite flexible enough, and adding new operators or snippets of code implemented externally in Rust was a bit of a pain. * The performance was never competitive with the hand-written peephole optimizers. It was very size efficient, but that came at the cost of run-time efficiency. Everything was table-based and interpreted, rather than generating any Rust code. Ultimately, because of these reasons, we never turned Peepmatic on by default. These days, we just landed the ISLE domain-specific language, and it is better suited than Peepmatic for all the things that Peepmatic was originally designed to do. It is more flexible and easy to integrate with external Rust code. It is has better time efficiency, meeting or even beating hand-written code. I think a small part of the reason why ISLE excels in these things is because its design was informed by Peepmatic's failures. I still plan on continuing Peepmatic's mission to make Cranelift's peephole optimizer passes generated from DSL rewrite rules, but using ISLE instead of Peepmatic. Thank you Peepmatic, rest in peace!	3 years ago
Alex Crichton	1ee2af0098	Remove the lightbeam backend (#3390 ) This commit removes the Lightbeam backend from Wasmtime as per [RFC 14]. This backend hasn't received maintenance in quite some time, and as [RFC 14] indicates this doesn't meet the threshold for keeping the code in-tree, so this commit removes it. A fast "baseline" compiler may still be added in the future. The addition of such a backend should be in line with [RFC 14], though, with the principles we now have for stable releases of Wasmtime. I'll close out Lightbeam-related issues once this is merged. [RFC 14]: https://github.com/bytecodealliance/rfcs/pull/14	3 years ago
Benjamin Bouvier	aa2c0cd0ec	CI: put tests into a separate script so they can be run externally (#2956 )	3 years ago

16 Commits (main)