cranelift

Commit Graph

Author	SHA1	Message	Date
Alex Crichton	5adcbb9685	Fix native feature detect in "min" builds (#7338 ) With a custom standard library disabling all features means disabling some support for feature-detection macros of the native platform. This meant that `wasmtime compile` output locally wasn't runnable in `wasmtime-min run` because it couldn't correctly detect that features were in fact available.	1 year ago
Alex Crichton	a80da753e0	Build "min" artifacts on CI (#7315 ) * riscv64: Extend distance trampolines can jump Use a PIC-friendly set of instructions to enable destination of the trampoline to be more than 4k away from the tail call site of the trampoline itself. * Build "min" artifacts on CI This commit updates the binary artifacts produced by CI to include "min" builds where all default features are disabled. Additionally all the stops are pulled in terms of build flags, nightly versions, etc, to get a build that is as small as possible without actual source code changes. This effectively codifies the instructions in #7282 into an easily downloadable artifact. No new tarballs are created for github releases but instead tarballs that previously had a `wasmtime` executable for example now have a `wasmtime-min` executable. Furthermore the C API which previously had `libwasmtime.so` for example now has `libwasmtime-min.so`. The intention is that the minimum-size artifacts are handy for determining a rough size of Wasmtime but they're not so important that it seems worthwhile to dedicate entire release entries for. CI is refactored to support these minimum builds with separate builders. This means that a single tarball produced as a final result is actually two separate tarballs merged together, one for the normal build we do today plus a new "min" tarball produced on the new "min" builders. Various scripts and CI organization has been adjusted accordingly. While here I went ahead and enabled `panic=abort` and debuginfo stripping in our current release artifacts. While this doesn't affect a whole lot it's less to upload to GitHub Actions all the time. * Fix Windows unzip	1 year ago
Alex Crichton	183cb0f2f8	Swap the order of directories in `--dir` (#7301 ) This commit changes the `--dir` argument on the `wasmtime` CLI to be `HOST::GUEST` rather than `GUEST::HOST`. This matches Docker for example and is a little more consistent with only `--dir path` where the first argument is always treated as a host directory. In terms of breaking-ness the movement from `--mapdir` to `--dir` hasn't been released with Wasmtime 14 yet so my hope is that this can land on both `main` and Wasmtime 14.0.0 before it's released to avoid any breakage other than existing scripts migrating from `--mapdir` to `--dir`.	1 year ago
Alex Crichton	41e49627f9	Update MSRV to 1.71.0 (#7206 ) * Update MSRV to 1.71.0 With the 1.73.0 release today update our MSRV to a one-higher version. This will cause 1.73.0 to be tested in CI. * Fix a warning on windows * Don't store tables as `&'static mut` Use `SendSyncPtr` instead (a small wrapper around `NonNull`) which appeases MIRI.	1 year ago
Alex Crichton	f4be360648	Refactor the test-programs test suite (#7182 ) * Refactor the test-programs test suite This commit is a large refactoring that reorganizes `test-programs` and how we tests wasms in Wasmtime. Often writing tests requires complicated interactions with the guest which can't be done via hand-written `.wat` syntax and requires a compiler to get engaged. For this purpose Wasmtime currently has the `crates/test-programs/` test suite which builds files from source and then runs the tests. This has been somewhat cumbersome in the past though and it's not been easy to extend this over time, so this commit attempts to address this. The scheme implemented in this PR looks like: * All wasm test programs live in `crates/test-programs/src/bin/.rs`. All of them, no exceptions. Wasm tests have shared support located at `crates/test-programs/src/lib.rs` and its submodules, such as bindings generation for WASI. * Wasm tests are built by a new `crates/test-programs/artifacts` crate. This crate compiles modules and additionally creates components for all test programs. The crate itself only records the path to these outputs and a small amount of testing support, but otherwise doesn't interact with `wasmtime`-the-crate itself. * All tests in `crates/test-programs/tests/.rs` have moved. For example wasi-http tests now live at `crates/wasi-http/tests/.rs`. Legacy tests of wasi-common now live at `crates/wasi-common/tests/.rs`. Modern tests for preview2 live at `crates/wasi/tests/.rs`. * Wasm tests are bucketed based on their filename prefix. For example `preview1_` is tested in wasi-common and wasmtime-wasi. The `preview2_` prefix is only tested with wasmtime-wasi, however. * A new `cli_` prefix is used to execute tests as part of `tests/all/main.rs`. This is a new submodule in `tests/all/cli_tests.rs` which executes these components on the command line. Many old "command" tests were migrated here. Helper macros are generated to assert that a test suite is run in its entirety. This way if a `preview1_` test is added it's asserted to get added to both wasi-common and wasmtime-wasi in the various modes they run tests. Overall this moved a number of tests around and refactored some edges of the tests, but this should not lose any tests (except one that wasn't actually testing anything). Additionally the hope is that it's much easier to add tests in the future. The process is to add a new file in `crates/test-programs/src/bin/.rs` named appropriately. For example a preview2 executable is `preview2_` and a CLI tests is `cli_`. When building the test suite an error is generated in the appropriate module then of "please write a test here", and then a test is written in the same manner as the other tests in the module. * Remove no-longer-needed fetches prtest:full * I'm worried wasi is running low on semicolons * Add the WASI target in all CI actions * Add unknown-unknown target on all CI builders too * Fix building test artifacts under miri Need to avoid wrappers for these cross-compiled targets * Break circular dependency for packaging Don't use the workspace dep for `wasmtime-wasi` since it injects a version, instead use a `path = '..'` dependency to fool Cargo into dropping the dependency during the package phase. * Fix some merge conflicts with tests * Fix rebase for new tests * Remove stray comment * Fix some flaky tests * Fix network tests in synchronous mode This commit is an attempt to fix some networking tests in synchronous mode in our test suite. Currently networking tests don't actually run in synchronous mode on CI which is why no failures have been surfaced yet, but the refactoring in #7182 is going to start doing this. Currently the `udp_sample_application.rs` test blocks infinitely in synchronous mode for me locally, most of the time. This appears to be an interaction between how Tokio handles readiness and how we're entering the event loop. We're effectively entering the Tokio event loop with a future that's always ready which ends up starving Tokio of otherwise performing its background work such as updating flags for readiness of reading/writing. The fix here is to add a yield at the start of an `in_tokio` block which is used in synchronous mode. This is a kludge fix but the intention is to enable Tokio to have a chance to update readiness flags and process events from epoll/kqueue/etc. An additional fix to this issue is WebAssembly/wasi-sockets#64 where the test is waiting on `READABLE` or `WRITABLE`, but in this specific case it should only wait on `READABLE`. If it waited on just this then that would also fix this issue. Nevertheless having a `yield_now` is expected to have little-to-no overhead and otherwise fix this edge case of an always-ready future. * Fix passing empty arguments on the CLI * Add another blocking accept * Update crates/test-programs/src/bin/api_proxy.rs Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com> --------- Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>	1 year ago
Trevor Elliott	9d3419741c	Fix a regression with `wasmtime serve`, and enable the serve feature in tests (#7137 )	1 year ago
Afonso Bordado	40c1f9b8b4	ci: Upgrade QEMU to `8.1.1` (#7096 ) * ci: Upgrade QEMU to `8.1.1` This adds support for RISC-V's Zcb extension that includes some extra compressed instructions. It also removes the current cpuinfo patch, that has been released in 8.1 * wasmtime: Don't assert the exact faulting address for wasm traps	1 year ago
Pat Hickey	a0469b1141	WASI preview 2 output-streams: new backpressure and flushing design (#6877 ) * Stream backpressure v2 Co-authored-by: Pat Hickey <phickey@fastly.com> Co-authored-by: Trevor Elliott <telliott@fastly.com> Co-authored-by: Dan Gohman <dev@sunfishcode.online> Stop testing pseudocode Restructure when notifications are sent, and make sure to flush the writer Fix the wasi-http module versions of flush and blocking_flush Use blocking_write_and_flush for blocking writes in the adapters Fix a warning in wasi-http Remove an unused DropPollable add comment explaining try_write for tcpstream refactor: separate struct for representing TcpReadStream by factoring into HostTcpSocket a little bit tcp read stream: handle stream closing tcp tests: use blocking_read where its expecting to wait for input move common test body into wasi-sockets-tests/src/lib.rs ensure parent socket outlives pollable input and output streams can be children now tcp's streams are the sockets children tcp.wit: document child relationships tcp tests: fix to drop socket after its child streams review feedback: propogate worker task panic style error source fix tcp: use preview2::spawn, and propogate worker panics join handle await always propogates panic background task handles ewouldblock as well document choice of constant * sync wit notes into wasi-http * improve wit docs for output-stream * doc: document `HostOutputStream` (#6980) * doc: document `HostOutputStream` Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net> Co-authored-by: Pat Hickey <pat@moreproductive.org> * fix(wasi): fail when `MemoryOutputStream` buffer is full Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net> --------- Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net> Co-authored-by: Pat Hickey <pat@moreproductive.org> * rustfmt prtest:full * windows and doc fixes * cli test wasi-http: use blocking-write-and-flush * Disable some tests, and adjust timeouts when running under qemu * Try to reproduce the riscv64 failures * Update riscv to LLVM 17 with beta rust * Revert "Try to reproduce the riscv64 failures" This reverts commit `8ac678171f`. * Pin the beta version for riscv64 * Fix a warning on nightly --------- Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net> Co-authored-by: Roman Volosatovs <rvolosatovs@users.noreply.github.com> Co-authored-by: Trevor Elliott <telliott@fastly.com> Co-authored-by: Alex Crichton <alex@alexcrichton.com>	1 year ago
Alex Crichton	8995750aa4	Redesign Wasmtime's CLI (#6925 ) * Redesign Wasmtime's CLI This commit follows through on discussion from #6741 to redesign the flags that the `wasmtime` binary accepts on the CLI. Almost all flags have been renamed/moved and will require callers to update. The main motivation here is to cut down on the forest of options in `wasmtime -h` which are difficult to mentally group together and understand. The main change implemented here is to move options behind "option groups" which are intended to be abbreviated with a single letter: * `-O foo` - an optimization or performance-tuning related option * `-C foo` - a codegen option affecting the compilation process. * `-D foo` - a debug-related option * `-W foo` - a wasm-related option, for example changing wasm semantics * `-S foo` - a WASI-related option, configuring various proposals for example Each option group can be explored by passing `help`, for example `-O help`. This will print all options within the group along with their help message. Additionally `-O help-long` can be passed to print the full comment for each option if desired. Option groups can be specified multiple times on the command line, for example `-Wrelaxed-simd -Wthreads`. They can also be combined together with commas as `-Wrelaxed-simd,threads`. Configuration works as a "last option wins" so `-Ccache,cache=n` would end up with a compilation cache disabled. Boolean options can be specified as `-C foo` to enable `foo`, or they can be specified with `-Cfoo=$val` with any of `y`, `n`, `yes`, `no`, `true`, or `false`. All other options require a `=foo` value to be passed and the parsing depends on the type. This commit additionally applies a few small refactorings to the CLI as well. For example the help text no longer prints information about wasm features after printing the option help. This is still available via `-Whelp` as all wasm features have moved from `--wasm-features` to `-W`. Additionally flags are no longer conditionally compiled in, but instead all flags are always supported. A runtime error is returned if support for a flag is not compiled in. Additionally the "experimental" name of WASI proposals has been dropped in favor of just the name of the proposal, for example `--wasi nn` instead of `--wasi-modules experimental-wasi-nn`. This is intended to mirror how wasm proposals don't have "experimental" in the name and an opt-in is required regardless. A full listing of flags and how they have changed is: \| old cli flag \| new cli flag \| \|-----------------------------------------------\|-------------------------------------------------\| \| `-O, --optimize` \| removed \| \| `--opt-level <LEVEL>` \| `-O opt-level=N` \| \| `--dynamic-memory-guard-size <SIZE>` \| `-O dynamic-memory-guard-size=...` \| \| `--static-memory-forced` \| `-O static-memory-forced` \| \| `--static-memory-guard-size <SIZE>` \| `-O static-memory-guard-size=N` \| \| `--static-memory-maximum-size <MAXIMUM>` \| `-O static-memory-maximum-size=N` \| \| `--dynamic-memory-reserved-for-growth <SIZE>` \| `-O dynamic-memory-reserved-for-growth=...` \| \| `--pooling-allocator` \| `-O pooling-allocator` \| \| `--disable-memory-init-cow` \| `-O memory-init-cow=no` \| \| `--compiler <COMPILER>` \| `-C compiler=..` \| \| `--enable-cranelift-debug-verifier` \| `-C cranelift-debug-verifier` \| \| `--cranelift-enable <SETTING>` \| `-C cranelift-NAME` \| \| `--cranelift-set <NAME=VALUE>` \| `-C cranelift-NAME=VALUE` \| \| `--config <CONFIG_PATH>` \| `-C cache-config=..` \| \| `--disable-cache` \| `-C cache=no` \| \| `--disable-parallel-compilation` \| `-C parallel-compilation=no` \| \| `-g` \| `-D debug-info` \| \| `--disable-address-map` \| `-D address-map=no` \| \| `--disable-logging` \| `-D logging=no` \| \| `--log-to-files` \| `-D log-to-files` \| \| `--coredump-on-trap <PATH>` \| `-D coredump=..` \| \| `--wasm-features all` \| `-W all-proposals` \| \| `--wasm-features -all` \| `-W all-proposals=n` \| \| `--wasm-features bulk-memory` \| `-W bulk-memory` \| \| `--wasm-features multi-memory` \| `-W multi-memory` \| \| `--wasm-features multi-value` \| `-W multi-value` \| \| `--wasm-features reference-types` \| `-W reference-types` \| \| `--wasm-features simd` \| `-W simd` \| \| `--wasm-features tail-call` \| `-W tail-call` \| \| `--wasm-features threads` \| `-W threads` \| \| `--wasm-features memory64` \| `-W memory64` \| \| `--wasm-features copmonent-model` \| `-W component-model` \| \| `--wasm-features function-references` \| `-W function-references` \| \| `--relaxed-simd-deterministic` \| `-W relaxed-simd-deterministic` \| \| `--enable-cranelift-nan-canonicalization` \| `-W nan-canonicalization` \| \| `--fuel <N>` \| `-W fuel=N` \| \| `--epoch-interruption` \| `-W epoch-interruption` \| \| `--allow-unknown-exports` \| `-W unknown-exports-allow` \| \| `--trap-unknown-imports` \| `-W unknown-imports-trap` \| \| `--default-values-unknown-imports` \| `-W unknown-imports-default` \| \| `--max-instances <MAX_INSTANCES>` \| `-W max-instances=N` \| \| `--max-memories <MAX_MEMORIES>` \| `-W max-memories=N` \| \| `--max-memory-size <BYTES>` \| `-W max-memory-size=N` \| \| `--max-table-elements <MAX_TABLE_ELEMENTS>` \| `-W max-table-elements=N` \| \| `--max-tables <MAX_TABLES>` \| `-W max-tables=N` \| \| `--max-wasm-stack <MAX_WASM_STACK>` \| `-W max-wasm-stack=N` \| \| `--trap-on-grow-failure` \| `-W trap-on-grow-failure` \| \| `--wasm-timeout <TIME>` \| `-W timeout=N` \| \| `--wmemcheck` \| `-W wmemcheck` \| \| `--wasi-modules default` \| removed \| \| `--wasi-modules -default` \| removed \| \| `--wasi-modules wasi-common` \| `-S common` \| \| `--wasi-modules -wasi-common` \| `-S common=n` \| \| `--wasi-modules experimental-wasi-nn` \| `-S nn` \| \| `--wasi-modules experimental-wasi-threads` \| `-S threads` \| \| `--wasi-modules experimental-wasi-http` \| `-S http` \| \| `--listenfd` \| `-S listenfd` \| \| `--tcplisten <SOCKET ADDRESS>` \| `-S tcplisten=...` \| \| `--wasi-nn-graph <FORMAT::HOST>` \| `-S nn-graph=FORMAT::HOST` \| \| `--preview2` \| `-S preview2` \| \| `--dir <DIRECTORY>` \| `--dir ...` \| \| `--mapdir <GUEST_DIR::HOST_DIR>` \| `--dir a::b` \| * Be more descriptive with help text * Document `=val` is optional for `-Ccranelift-xxx` * Fix compile after rebase * Fix rebase of `--inherit-network` * Fix wasi-http test * Fix compile without pooling allocator support * Update some flags in docs * Fix bench-api build * Update flags for gdb/lldb tests * Fixup optimization flags prtest:full	1 year ago
Alex Crichton	a04c4930bf	Update Rust in CI to 1.72.0, clarify Wasmtime's MSRV (#6900 ) * Update Rust in CI to 1.72.0 * Update CI, tooling, and docs for MSRV This commit codifies an MSRV policy for Wasmtime at "stable minus two" meaning that the latest three releases of Rust will be supported. This is enforced on CI with a full test suite job running on Linux x86_64 with the minimum supported Rust version. The full test suite will use the latest stable version. A downside of this approach is that new changes may break MSRV support on non-Linux or non-x86_64 platforms and we won't know about it, but that's deemed a minor enough risk at this time. A minor fix is applied to Wasmtime's `Cargo.toml` to support Rust 1.70.0 instead of requiring Rust 1.71.0 * Fix installation of rust * Scrape MSRV from Cargo.toml * Cranelift is the same as Wasmtime's MSRV now, more words too * Fix a typo	1 year ago
Andrew Brown	770c5d0f3f	wasi-nn: add named models (#6854 ) * wasi-nn: add [named models] This change adds a way to retrieve preloaded ML models (i.e., "graphs" in wasi-nn terms) from a registry. The wasi-nn specification includes a new function, `load_by_name`, that can be used to access these models more efficiently than before; previously, a user's only option was to read/download/etc. all of the bytes of an ML model and pass them to the `load` function. [named models]: https://github.com/WebAssembly/wasi-nn/issues/36 In Wasmtime's implementation of wasi-nn, we call the registry that holds the models a `GraphRegistry`. We include a simplistic `InMemoryRegistry` for use in the Wasmtime CLI (more on this later) but the idea is that production use will involve some more complex caching and thus a new implementation of a registry--a `Box<dyn GraphRegistry>`--passed into the wasi-nn context. Note that, because we now must be able to `clone` a graph out of the registry and into the "used graphs" table, the OpenVINO `BackendGraph` is updated to be easier to copy around. To allow experimentation with this "preload a named model" functionality, this change also adds a new Wasmtime CLI flag: `--graph <encoding>:<host dir>`. Wasmtime CLI users can now preload a model from a directory; the directory `basename` is used as the model name. Loading models from a directory is probably not desired in Wasmtime embeddings so it is cordoned off into a separate `BackendFromDir` extension trait. * wasi-nn: add "named model" test Add a new example crate which loads a model by name and performs image classification. It uses the same MobileNet model as the existing test but a new version of the Rust bindings. The new crate is built and run with the new CLI flag in the `ci/run-wasi-nn-example.sh` script. prtest:full * review: rename `--graph` to `--wasi-nn-graph`	1 year ago
Alex Crichton	367bdc8c66	Refactor Wasmtime CLI to support components (#6836 ) This commit refactors the `wasmtime` CLI executable to be able to support not only compiling components but additionally executing components. While I was doing this I've additionally added a new `--preview2` argument to enable using the new experimental implementation of preview1 based on preview2 type/structs. This is off-by-default but is expected to become the default in the future. Some notable features of this change are: * The preview1-implemented-with-preview2 module now sports `add_to_linker_{async,sync}` to replace the previous `add_to_linker` which always did async. * Some trait bounds in the preview1-implemented-with-preview2 module are simplified. * Some minor changes were made to `wiggle`'s macros to support a "block on" that isn't the default wiggle dummy executor (as now we actually do need Tokio) * Many options related to core wasm `Linker` configuration, such as `--default-values-unknown-imports`, are not implemented for components at this time. When used with components these options return an error. * Construction of WASI contexts has been refactored to pass around fewer arguments to avoid threading through lots of values for both preview1 and preview2. * Reading the input to the Wasmtime CLI has been updated to read the input in the `run` subcommand before handing it off to the `wasmtime` crate's API to enable the CLI to use the contents of what's loaded to determine what to do next. * Our generic `./ci/run-tests.sh` script has been updated to pass the `--features component-model` flag to ensure that this CLI support is tested during the normal test suite. * The CLI support for `wasi-nn` supports components as well as core wasm modules.	1 year ago
Eduardo de Moura Rodrigues	e250334b8e	feat: align wasi-http with component linker (#6195 ) * feat: align wasi-http with component linker * feat(wasi-http): allow bidirectional stream * feat(wasi-http): clean up children when dropping resource * chore: update based on feedback * chore: replace wasi http context references * chore: fix logical issue with outgoing body stream * chore: use thread-safe reference-counting pointer * chore: cleanup resources using table * fix(wasi-preview1-component-adapter): sync command extended wit * fix(wasi-preview1-component-adapter): sync command extended wit * fix(wasmtime-wasi): sync wit for http types * chore: refactor using wasmtime-wasi crate fix(wasi-http): misconfiguration in wasmtime linkers chore: keep streams details chore: fix wasi http tests * chore: use pollable from wasmtime-wasi * chore: update wasi http linker for module * chore: update test programs for wasi http * fix(wasi-http): ensure proper errors are surfaced * chore: split wasi http tests into individual files * chore: ensure protocol error is mapped correctly * chore: disable temporarily wasi http in wasmtime cli * chore: comment out wasi http in wasmtime cli * chore(ci): ensure wit definitions in sync * feat(wasi-http): generate async host binding * chore: make wasi http tests async * chore: update ci workflow based on suggestion Co-authored-by: Pat Hickey <pat@moreproductive.org> * feat(wasmtime-wasi): update logging world to latest * feat(wasmtime): update proxy world to latest * feat(wasmtime-wasi): add back command extended world * fix(wasi-http): sync wit definitions * chore: update tests with latest wit definitions * Update src/commands/run.rs * Update src/commands/run.rs * Update src/commands/run.rs * Update src/commands/run.rs * Update src/commands/run.rs * Update src/commands/run.rs * Update src/commands/run.rs * Update src/commands/run.rs * Update src/commands/run.rs * chore: fix formatting * Ignore flaky test * chore: fix compilation error for riscv64 arch * Avoid `cp -T` on macos Adding prtest:full to ensure that we've seen a successful build before queuing. * Don't build the wasi-http test programs for the native target * Debug the wit consistency check * Update streams.wit in wasi-http * Mark the component outbound_request_post test flaky * Disable flaky wasi-http-tests on windows only * Use diff instead of rm/cp/git diff * Disable more tests on windows --------- Co-authored-by: Eduardo Rodrigues <eduardomourar@users.noreply.github.com> Co-authored-by: Pat Hickey <pat@moreproductive.org> Co-authored-by: Trevor Elliott <telliott@fastly.com>	1 year ago
Chris Fallin	16713422b8	CI: upgrade to qemu 8.0.4. (#6849 ) * CI: upgrade to qemu 8.0.4. This should resolve a bug in qemu triggered by changes in #6820 (see [here](https://github.com/bytecodealliance/wasmtime/pull/6820#issuecomment-1678908361)). * ci: Update QEMU patch for 8.0.2 --------- Co-authored-by: Afonso Bordado <afonsobordado@az8.co>	1 year ago
Alex Crichton	2897e409db	Remove the implementation of wasi-crypto (#6816 ) * Remove the implementation of wasi-crypto This commit is a follow-up to the discussion on #6732. This removes Wasmtime's implementation of the wasi-crypto proposal from in-tree along with its various support in CI, configuration, etc. See the discussion on #6732 for the full information but at a high level the main reasons for removing the implementation at this time are: * There is not currently an active maintainer of the Wasmtime integration here for wasi-crypto. * There are known issues with the code quality of the implementation such as transmutes of guest-owned memory to `&'static mut [u8]` and known unsafety in dependencies. * The size and breadth of the dependency tree brings maintenance burden and overhead to managing Wasmtime's dependency tree. As mentioned on the issue this commit does not mean that Wasmtime doesn't want to implement the wasi-crypto proposal. Instead the "tier 3" status of wasi-crypto needs to be re-attained to be included back in-tree, which would mean resolving the above issues. Note that this commit is intentionally just after the 13.0.0 branch point which means that this is slated for Wasmtime 14 to be released on September 20. * Remove some cfgs * Remove wasi-crypto CI	1 year ago
Alex Crichton	8091556514	Require wasmtime options are first when running modules (#6737 ) * Require wasmtime options are first when running modules Currently the way we've configured argument parsing it's valid to execute a command such as: wasmtime run foo.wasm -O which is the same as: wasmtime run -O foo.wasm or otherwise all flags are attempted to be parsed as Wasmtime flags and an error is generated when they're not wasmtime flags. I've personally found this a bit confusing in the past and I find myself frequently executing: wasmtime run -- foo.wasm -other -arguments While this works my general impression is that many other "wrapper commands" don't behave this way and typically don't require `--` to pass flags to the target executable. This commit reconfigures argument parsing to consider any argument after the WebAssembly module itself to be an argument to the wasm program rather than an argument to Wasmtime. This means that all Wasmtime options must come between the `run` command and the `foo.wasm` WebAssembly argument. * Update wasi testsuite runner * Support `wasmtime -- run` Additionally use more clap features to avoid manually checking against subcommands. * Remove stale comment * Reorder wasi-nn arguments * Reorder more flags * Fix unused import on Windows * Don't run a stdio test on Windows * Update gdb/lldb tests * Don't assert that the write succeeds prtest:full	1 year ago
Alex Crichton	aef1f57d13	Update adapter build (#6573 ) * Update adapter build * Rename the binary artifact to `wasi_snapshot_preview1.wasm` and update build scripts to account for this. * Update documentation to mention difference between reactor/command builds. Closes #6569 * More renaming	1 year ago
Brendan Burns	fc11f56318	Re-enable wasi-http for recent wit bindgen changes. Renable tests. (#6495 ) * Re-enable wasi-http * Address comments.	1 year ago
Alex Crichton	3b90faf4a7	Fix adapter build on CI, rename the adapter artifacts (#6460 ) * Add adapter build to dependency for the entire CI build * Build adapter on PR CI when it changes * Update adapter verification for new stdin/stdout/stderr * Update reactor build script * Reduce some duplication with some shared variables * Rename adapters to `wasi_snapshot_preview1.{reactor,command}.wasm` * Fix upload pattern	1 year ago
Pat Hickey	224866122f	exclude command-tests and reactor-tests wasm crates from run-tests	1 year ago
Pat Hickey	338b535a3a	Refactor test-programs to build modules and components (#6385 ) * wasi-tests and wasi-http-tests no longer have their own workspace * wasi-tests: fix warnings * rewrite the test-programs build.rs to generate {package}_modules.rs and _components.rs The style is cribbed from preview2-prototying repo, but I ended up refactoring it a bit. * better escaping should help with windows? * long form cap-std-sync and tokio test suites * convert wasi-http test * fixes, comments * apply cargo fmt to whole workspace * bump test-programs and wasi-http-tests to all use common dependency versions wit-bindgen 0.6.0 and wit-component 0.7.4 * add new audits * cargo vet prune * package and supply chain updates to fix vulnerabilities h2 upgraded from 0.3.16 -> 0.3.19 to fix vulnerability tempfile upgraded from 0.3.3 -> 0.3.5 to eliminate dep on vulnerable remove_dir_all * deny: temporarily allow duplicate wasm-encoder, wasmparser, wit-parser prtest:full * convert more dependencies to { workspace = true } Alex asked me to do thsi for wit-component and wit-bindgen, and I found a few more (cfg-if, tempfile, filecheck, anyhow... I also reorganized the workspace dependencies section to make the ones our team maintains more clearly separated from our external dependencies. * test-programs build: ensure that the user writes a #[test] for each module, component * fix build of wasi-tests on windows * misspelled macos * mark wasi-tests crate test=false so we dont try building it natively... * mark wasi-http-tests test=false as well * try getting the cargo keys right * just exclude wasi-tests and wasi-http-tests in run-tests.sh * interesting paths fails on windows * misspelling so nice i did it twice * new cargo deny exception: ignore all of wit-bindgen's dependencies * auto-import wildcard vets	1 year ago
Pat Hickey	9c61c7454d	exclude adapter from tests build with manifest instead of run-tests.sh	2 years ago
Pat Hickey	e2e9aba4c8	run-tests: exclude the adapter, which doesnt build for native	2 years ago
Pat Hickey	19858ddc98	change adapter to be part of root workspace	2 years ago
Pat Hickey	424deb98fe	add ci/build-wasi-preview1-component-adapter script	2 years ago
Alex Crichton	8157924f68	Fix name of tarballs for patch releases (#6295 ) Since the latest updates to our release process which transitioned to merge queues it appears that patch release create incorrectly named tarballs. The version in the tarball is based on the branch name, which doesn't change for patch releases, so the version needs to come from `Cargo.toml`. Thankfully there's already a helpful shell script to do that so use the shell script instead of using the branch name.	2 years ago
Afonso Bordado	60e4a00413	riscv64: Initial SIMD Vector Implementation (#6240 ) * riscv64: Remove unused code * riscv64: Add vector types * riscv64: Initial Vector ABI Load/Stores * riscv64: Vector Loads/Stores * riscv64: Fix `vsetvli` encoding error * riscv64: Add SIMD `iadd` runtests * riscv64: Rename `VecSew` The SEW name is correct, but only for VType. We also use this type in loads/stores as the Efective Element Width, so the name isn't quite correct in that case. * ci: Add V extension to RISC-V QEMU * riscv64: Misc Cleanups * riscv64: Check V extension in `load`/`store` for SIMD * riscv64: Fix `sumop` doc comment * cranelift: Fix comment typo * riscv64: Add convert for VType and VecElementWidth * riscv64: Remove VecElementWidth converter	2 years ago
Afonso Bordado	9acb649f17	cranelift-native: Detect RISC-V extensions using `/proc/cpuinfo` (#6192 ) * cranelift-native: Move riscv to separate module * cranelift-native: Read /proc/cpuinfo to parse RISC-V extensions * ci: Add QEMU cpuinfo emulation patch This patch emulates the /proc/cpuinfo interface for RISC-V. This allows us to do feature detection for the RISC-V backend. It has been queued for QEMU 8.1 so we should remove it as soon as that is available. * ci: Enable QEMU RISC-V extensions * cranelift-native: Cleanup ISA string parsing Co-Authored-By: Jamey Sharp <jsharp@fastly.com> * cranelift-native: Rework `/proc/cpuinfo` parsing Co-Authored-By: Jamey Sharp <jsharp@fastly.com> --------- Co-authored-by: Jamey Sharp <jsharp@fastly.com>	2 years ago
Andrew Brown	5ba0d696b7	ci: unpin the wasi-nn tasks from an older Ubuntu (#6089 ) * ci: unpin the wasi-nn tasks from an older Ubuntu Previously, OpenVINO's lack of APT packages for Ubuntu 22.04 (`jammy`) prevented us from upgrading the GitHub runner to use `ubuntu-latest`. I updated the `install-openvino-action` to substitute in the `focal` packages in this case (this is what the OpenVINO team considers the fix) so this pin should no longer be necessary. Fixes #5408. (Run all CI actions: prtest:full) * vet: audit the openvino version bump	2 years ago
Brendan Burns	2d34dbef4b	Begin implementation of wasi-http (#5929 ) * Integrate experimental HTTP into wasmtime. * Reset Cargo.lock * Switch to bail!, plumb options partially. * Implement timeouts. * Remove generated files & wasm, add Makefile * Remove generated code textfile * Update crates/wasi-http/Cargo.toml Co-authored-by: Eduardo de Moura Rodrigues <16357187+eduardomourar@users.noreply.github.com> * Update crates/wasi-http/Cargo.toml Co-authored-by: Eduardo de Moura Rodrigues <16357187+eduardomourar@users.noreply.github.com> * Extract streams from request/response. * Fix read for len < buffer length. * Formatting. * types impl: swap todos for traps * streams_impl: idioms, and swap todos for traps * component impl: idioms, swap all unwraps for traps, swap all todos for traps * http impl: idiom * Remove an unnecessary mut. * Remove an unsupported function. * Switch to the tokio runtime for the HTTP request. * Add a rust example. * Update to latest wit definition * Remove example code. * wip: start writing a http test... * finish writing the outbound request example havent executed it yet * better debug output * wasi-http: some stubs required for rust rewrite of the example * add wasi_http tests to test-programs * CI: run the http tests * Fix some warnings. * bump new deps to latest releases (#3) * Add tests for wasi-http to test-programs (#2) * wip: start writing a http test... * finish writing the outbound request example havent executed it yet * better debug output * wasi-http: some stubs required for rust rewrite of the example * add wasi_http tests to test-programs * CI: run the http tests * bump new deps to latest releases h2 0.3.16 http 0.2.9 mio 0.8.6 openssl 0.10.48 openssl-sys 0.9.83 tokio 1.26.0 --------- Co-authored-by: Brendan Burns <bburns@microsoft.com> * Update crates/test-programs/tests/http_tests/runtime/wasi_http_tests.rs * Update crates/test-programs/tests/http_tests/runtime/wasi_http_tests.rs * Update crates/test-programs/tests/http_tests/runtime/wasi_http_tests.rs * wasi-http: fix cargo.toml file and publish script to work together (#4) unfortunately, the publish script doesn't use a proper toml parser (in order to not have any dependencies), so the whitespace has to be the trivial expected case. then, add wasi-http to the list of crates to publish. * Update crates/test-programs/build.rs * Switch to rustls * Cleanups. * Merge switch to rustls. * Formatting * Remove libssl install * Fix tests. * Rename wasi-http -> wasmtime-wasi-http * prtest:full Conditionalize TLS on riscv64gc. * prtest:full Fix formatting, also disable tls on s390x * prtest:full Add a path parameter to wit-bindgen, remove symlink. * prtest:full Fix tests for places where SSL isn't supported. * Update crates/wasi-http/Cargo.toml --------- Co-authored-by: Eduardo de Moura Rodrigues <16357187+eduardomourar@users.noreply.github.com> Co-authored-by: Pat Hickey <phickey@fastly.com> Co-authored-by: Pat Hickey <pat@moreproductive.org>	2 years ago
Afonso Bordado	dd9804514d	CI: Upgrade QEMU to 7.2.0 (#6102 ) * ci: Update QEMU to 7.2.0 * ci: Drop QEMU madvise patch This seems to have been upstreamed in: https://lists.gnu.org/archive/html/qemu-devel/2022-06/msg03752.html Running the full CI to ensure this works on all arches. prtest:full	2 years ago
Alex Crichton	ad0f169d64	Fix naming of the source tarball (#6068 ) This refactoring from #5766 accidentally broke the tag name calculation for the `build-src-tarball.sh` script so this fixes it by copying over the same logic from `build-tarballs.sh`.	2 years ago
Trevor Elliott	4c88acbb89	Test all backends when a runtest is modified (#5872 ) * Test all backends when a runtest is modified * Check that this triggers all backend tests * Revert "Check that this triggers all backend tests" This reverts commit `1d12536d04`.	2 years ago
Alex Crichton	3ce439ce57	Update PR tests slightly and when what runs where (#5805 ) * Don't run LLDB tests on PRs These take an extra minute or so, so only run them on the full test suite of a merge instead of on all PRs as well. * Add a test for the x64 isa files This guarantees that if cranelift's x64 backend is modified that the tests will be run on a PR, even if other backends were also modified.	2 years ago
Alex Crichton	1efee4abdf	Update CI to use GitHub's Merge Queue (#5766 ) GitHub recently made its merge queue feature available for use in public repositories owned by organizations meaning that the Wasmtime repository is a candidate for using this. GitHub's Merge Queue feature is a system that's similar to Rust's bors integration where PRs are tested before merging and only passing PRs are merged. This implements the "not rocket science" rule where the `main` branch of Wasmtime, for example, is always tested and passes CI. This is in contrast to our current implementation of CI where PRs are merged when they pass their own CI, but the code that was tested is not guaranteed to be the state of `main` when the PR is merged, meaning that we're at risk now of a failing `main` branch despite all merged PRs being green. While this has happened with Wasmtime this is not a common occurrence, however. The main motivation, instead, to use GitHub's Merge Queue feature is that it will enable Wasmtime to greatly reduce the amount of CI running on PRs themselves. Currently the full test suite runs on every push to every PR, meaning that our workers on GitHub Actions are frequently clogged throughout weekdays and PRs can take quite some time to come back with a successful run. Through the use of a Merge Queue, however, we're able to configure only a small handful of checks to run on PRs while deferring the main body of checks to happening on the merge-via-the-queue itself. This is hoped to free up capacity on CI and overall improve CI times for Wasmtime and Cranelift developers. The implementation of all of this required quite a lot of plumbing and retooling of our CI. I've been testing this in an [external repository][testrepo] and I think everything is working now. A list of changes made in this PR are: * The `build.yml` workflow is merged back into the `main.yml` workflow as the original reason to split it out is not longer applicable (it'll run on all merges). This was also done to fit in the dependency graph of jobs of one workflow. * Publication of the `gh-pages` branch, the `dev` tag artifacts, and release artifacts have been moved to a separate `publish-artifacts.yml` workflow. This workflow runs on all pushes to `main` and all tags. This workflow no longer actually preforms any builds, however, and relies on a merge queue or similar being used for branches/tags where artifacts are downloaded from the workflow run to be uploaded. For pushes to `main` this works because a merge queue is run meaning that by the time the push happens all artifacts are ready. For release branches this is handled by.. * The `push-tag.yml` workflow is subsumed by the `main.yml` workflow. CI for a tag being pushed will upload artifacts to a release in GitHub, meaning that all builds must finish first for the commit. The `main.yml` workflow at the end now scans commits for the preexisting magical marker and pushes a tag if necessary. * CI is currently a flat list of "run all these jobs" and this is now rearchitected to a "fan out" approach where some jobs run to determine the next jobs to run which then get "joined" into a finish step. The purpose for this is somewhat nuanced and this has implications for CI runtime as well. The Merge Queue feature requires branches to be protected with "these checks must pass" and then the same checks are gates both to enter the merge queue as well as pass the merge queue. The saving grace, however, is that a "skipped" check counts as passing, meaning checks can be skipped on PRs but run to completion on the merge queue. A problem with this though is the build matrix used for tests where PRs want to only run one element of the build matrix ideally but there's no means on GitHub Actions right now for the skipped entries to show up as skipped easily (or not that I know of). This means that the "join" step serves the purpose of being the single gate for both PR and merge queue CI and there's just more inputs to it for merge queue CI. The major consequence of this decision is that GitHub's actions scheduling doesn't work out well here. Jobs are scheduled in a FIFO order meaning that the job for "ok complete the CI run" is queued up after everything else has completed, possibly after lots of other CI requests in the middle for other PRs. The hope here is that by using a merge queue we can keep CI relatively under control and this won't affect merge times too much. * All jobs in the `main.yml` workflow will not automatically cancel the entire run if they fail. Previously this fail-fast behavior was only part of the matrix runs (and just for that matrix), but this is required to make the merge queue expedient. The gate of the merge queue is the final "join" step which is only executed once all dependencies have finished. This means, for example, that if rustfmt fails quickly then the tests which take longer might run for quite awhile before the join step reports failure, meaning that the PR sits in the queue for longer than needed being tested when we know it's already going to fail. By having all jobs cancel the run this means that failures immediately bail out and mark the whole job as cancelled. * A new "determine" CI job was added to determine what CI actually needs to run. This is a "choke point" which is scheduled at the start of CI that quickly figures out what else needs to be run. This notably indicates whether large swaths of ci (the `run-full` flag) like the build matrix are executed. Additionally this dynamically calculates a matrix of tests to run based on a new `./ci/build-test-matrix.js` script. Various inputs are considered for this such as: 1. All pushes, meaning merge queue branches or release-branch merges, will run full CI. 2. PRs to release branches will run full CI. 3. PRs to `main`, the most common, determine what to run based on what's modified and what's in the commit message. Some examples for (3) above are if modifications are made to `cranelift/codegen/src/isa/` then that corresponding builder is executed on CI. If the `crates/c-api` directory is modified then the CMake-based tests are run on PRs but are otherwise skipped. Annotations in commit messages such as `prtest:` can be used to explicitly request testing. Before this PR merges to `main` would perform two full runs of CI: one on the PR itself and one on the merge to `main`. Note that the one as a merge to `main` was quite frequently cancelled due to a merge happening later. Additionally before this PR there was always the risk of a bad merge where what was merged ended up creating a `main` that failed CI to to a non-code-related merge conflict. After this PR merges to `main` will perform one full run of CI, the one as part of the merge queue. PRs themselves will perform one test job most of the time otherwise. The `main` branch is additionally always guaranteed to pass tests via the merge queue feature. For release branches, before this PR merges would perform two full builds - one for the PR and one for the merge. A third build was then required for the release tag itself. This is now cut down to two full builds, one for the PR and one for the merge. The reason for this is that the merge queue feature currently can't be used for our wildcard-based `release-` branch protections. It is now possible, however, to turn on required CI checks for the `release-` branch PRs so we can at least have a "hit the button and forget" strategy for merging PRs now. Note that this change to CI is not without its risks. The Merge Queue feature is still in beta and is quite new for GitHub. One bug that Trevor and I uncovered is that if a PR is being tested in the merge queue and a contributor pushes to their PR then the PR isn't removed from the merge queue but is instead merged when CI is successful, losing the changes that the contributor pushed (what's merged is what was tested). We suspect that GitHub will fix this, however. Additionally though there's the risk that this may increase merge time for PRs to Wasmtime in practice. The Merge Queue feature has the ability to "batch" PRs together for a merge but this is only done if concurrent builds are allowed. This means that if 5 PRs are batched together then 5 separate merges would be created for the stack of 5 PRs. If the CI for all 5 merged together passes then everything is merged, otherwise a PR is kicked out. We can't easily do this, however, since a major purpose for the merge queue for us would be to cut down on usage of CI builders meaning the max concurrency would be set to 1 meaning that only one PR at a time will be merged. This means PRs may sit in the queue for awhile since previously many `main`-based builds are cancelled due to subsequent merges of other PRs, but now they must all run to 100% completion. [testrepo]: https://github.com/bytecodealliance/wasmtime-merge-queue-testing	2 years ago
Andrew Brown	edfa10d607	wasi-threads: an initial implementation (#5484 ) This commit includes a set of changes that add initial support for `wasi-threads` to Wasmtime: * feat: remove mutability from the WasiCtx Table This patch adds interior mutability to the WasiCtx Table and the Table elements. Major pain points: * `File` only needs `RwLock<cap_std::fs::File>` to implement `File::set_fdflags()` on Windows, because of [1] * Because `File` needs a `RwLock` and `RwLockGuard` cannot be hold across an `.await`, The `async` from `async fn num_ready_bytes(&self)` had to be removed Because `File` needs a `RwLock` and `RwLockGuard` cannot be dereferenced in `pollable`, the signature of `fn pollable(&self) -> Option<rustix::fd::BorrowedFd>` changed to `fn pollable(&self) -> Option<Arc<dyn AsFd + '_>>` [1] `da238e324e/src/fs/fd_flags.rs (L210-L217)` wasi-threads: add an initial implementation This change is a first step toward implementing `wasi-threads` in Wasmtime. We may find that it has some missing pieces, but the core functionality is there: when `wasi::thread_spawn` is called by a running WebAssembly module, a function named `wasi_thread_start` is found in the module's exports and called in a new instance. The shared memory of the original instance is reused in the new instance. This new WASI proposal is in its early stages and details are still being hashed out in the [spec] and [wasi-libc] repositories. Due to its experimental state, the `wasi-threads` functionality is hidden behind both a compile-time and runtime flag: one must build with `--features wasi-threads` but also run the Wasmtime CLI with `--wasm-features threads` and `--wasi-modules experimental-wasi-threads`. One can experiment with `wasi-threads` by running: ```console $ cargo run --features wasi-threads -- \ --wasm-features threads --wasi-modules experimental-wasi-threads \ <a threads-enabled module> ``` Threads-enabled Wasm modules are not yet easy to build. Hopefully this is resolved soon, but in the meantime see the use of `THREAD_MODEL=posix` in the [wasi-libc] repository for some clues on what is necessary. Wiggle complicates things by requiring the Wasm memory to be exported with a certain name and `wasi-threads` also expects that memory to be imported; this build-time obstacle can be overcome with the `--import-memory --export-memory` flags only available in the latest Clang tree. Due to all of this, the included tests are written directly in WAT--run these with: ```console $ cargo test --features wasi-threads -p wasmtime-cli -- cli_tests ``` [spec]: https://github.com/WebAssembly/wasi-threads [wasi-libc]: https://github.com/WebAssembly/wasi-libc This change does not protect the WASI implementations themselves from concurrent access. This is already complete in previous commits or left for future commits in certain cases (e.g., wasi-nn). * wasi-threads: factor out process exit logic As is being discussed [elsewhere], either calling `proc_exit` or trapping in any thread should halt execution of all threads. The Wasmtime CLI already has logic for adapting a WebAssembly error code to a code expected in each OS. This change factors out this logic to a new function, `maybe_exit_on_error`, for use within the `wasi-threads` implementation. This will work reasonably well for CLI users of Wasmtime + `wasi-threads`, but embedders will want something better in the future: when a `wasi-threads` threads fails, they may not want their application to exit. Handling this is tricky, because it will require cancelling the threads spawned by the `wasi-threads` implementation, something that is not trivial to do in Rust. With this change, we defer that work until later in order to provide a working implementation of `wasi-threads` for experimentation. [elsewhere]: https://github.com/WebAssembly/wasi-threads/pull/17 * review: work around `fd_fdstat_set_flags` In order to make progress with wasi-threads, this change temporarily works around limitations induced by `wasi-common`'s `fd_fdstat_set_flags` to allow `&mut self` use in the implementation. Eventual resolution is tracked in https://github.com/bytecodealliance/wasmtime/issues/5643. This change makes several related helper functions (e.g., `set_fdflags`) take `&mut self` as well. * test: use `wait`/`notify` to improve `threads.wat` test Previously, the test simply executed in a loop for some hardcoded number of iterations. This changes uses `wait` and `notify` and atomic operations to keep track of when the spawned threads are done and join on the main thread appropriately. * various fixes and tweaks due to the PR review --------- Signed-off-by: Harald Hoyer <harald@profian.com> Co-authored-by: Harald Hoyer <harald@profian.com> Co-authored-by: Alex Crichton <alex@alexcrichton.com>	2 years ago
Alex Crichton	d61758e2e9	Pin release artifacts Rust toolchain (#5669 ) This fixes the build issue identified in #5664 at the toolchain level rather than working around it in our own build. The next step in fixing this will be to remove the nightly override in the future when the toolchain becomes stable.	2 years ago
Trevor Elliott	cc768f22a2	Debug the build step (#5664 ) Change the permissions on libwasmtime.a before copying it, to avoid errors stemming from new behavior in rustc-1.67.	2 years ago
yuyang-ok	cdecc858b4	add riscv64 backend for cranelift. (#4271 ) Add a RISC-V 64 (`riscv64`, RV64GC) backend. Co-authored-by: yuyang <756445638@qq.com> Co-authored-by: Chris Fallin <chris@cfallin.org> Co-authored-by: Afonso Bordado <afonsobordado@az8.co>	2 years ago
Alex Crichton	69483a2575	Add source tarballs to our releases (#4294 ) * Add source tarballs to our releases This commit adds a small script to create a source tarball as part of the release process. This goes further than requested by #3808 by vendoring all Rust dependencies as well to be more in line with "download the source once then build somewhere without a network". Vendoring the Rust dependencies makes the tarball pretty beefy (67M compressed, 500M uncompressed). Unfortunately most of this size comes from vendored crates such as v8, pqcrypto-kyber, winapi, capstone-sys, plotters, and web-sys. Only `winapi` in this list is actually needed for `wasmtime`-the-binary and only on Windows as well but for now this is the state of things related to `cargo vendor`. If this becomes an issue we could specifically remove the bulky contents of crates in the `vendor` directory such as `v8` since it's only used for fuzzing. Closes #3808 * Review feedback * Review comments	2 years ago
Andrew Brown	558a9273e9	wasi-nn: update openvino crate, use external CI action (#4383 ) * ci: replace OpenVINO installer action To test wasi-nn, we currently use an OpenVINO backend. The Wasmtime CI must install OpenVINO using a custom GitHub action. This CI action has not been updated in some time and in the meantime OpenVINO (and the OpenVINO crates) have released several new versions. https://github.com/abrown/install-openvino-action is an external action that we plan to keep up to date with the latest releases. This change replaces the current CI action with that one. * wasi-nn: upgrade openvino dependency to v0.4.1 This eliminates a `lazy_static` dependency and changes a few parameters to pass by reference. Importantly, it enables support for the latest versions of OpenVINO (v2022.) in wasi-nn. ci: update wasi-nn script to source correct env script * ci: really use the correct path for the env script Also, clarify which directory OpenVINO is installed in (the symlink may not be present).	2 years ago
Alex Crichton	08b7c87793	Refactor binary-compatible-builds for releases (#4171 ) * Refactor binary-compatible-builds for releases I was poking around this yesterday and noticed a few things that could be improved for our release builds: * The centos container for the x86_64 builds contained a bunch of extra tooling we no longer need such as python3 and a C++ compiler. Along with custom toolchain things this could all get removed since the C we include now is quite simple. * The aarch64 and s390x cross-compiled builds had relatively high glibc version requirements compared to the x86_64 build. This was because we don't use a container to build the cross-compiled binaries. I added containers here along the lines of the x86_64 build to use an older glibc to build the release binary to lower our version requirement. This lower the aarch64 version requirement from glibc 2.28 to 2.17. Additionally the s390x requirement dropped from 2.28 to 2.16. * To make the containers a bit easier to read/write I added `Dockerfile`s for them in a new `ci/docker` directory instead of hardcoding install commands in JS. This isn't intended to be a really big change or anything for anyone, but it's intended to keep our Linux-based builds consistent at least as best we can. * Remove temporary change	3 years ago
Chris Fallin	5d671952ee	Cranelift: do not check in generated ISLE code; regenerate on every compile. (#4143 ) This PR fixes #4066: it modifies the Cranelift `build.rs` workflow to invoke the ISLE DSL compiler on every compilation, rather than only when the user specifies a special "rebuild ISLE" feature. The main benefit of this change is that it vastly simplifies the mental model required of developers, and removes a bunch of failure modes we have tried to work around in other ways. There is now just one "source of truth", the ISLE source itself, in the repository, and so there is no need to understand a special "rebuild" step and how to handle merge errors. There is no special process needed to develop the compiler when modifying the DSL. And there is no "noise" in the git history produced by constantly-regenerated files. The two main downsides we discussed in #4066 are: - Compile time could increase, by adding more to the "meta" step before the main build; - It becomes less obvious where the source definitions are (everything becomes more "magic"), which makes exploration and debugging harder. This PR addresses each of these concerns: 1. To maintain reasonable compile time, it includes work to cut down the dependencies of the `cranelift-isle` crate to nothing (only the Rust stdlib), in the default build. It does this by putting the error-reporting bits (`miette` crate) under an optional feature, and the logging (`log` crate) under a feature-controlled macro, and manually writing an `Error` impl rather than using `thiserror`. This completely avoids proc macros and the `syn` build slowness. The user can still get nice errors out of `miette`: this is enabled by specifying a Cargo feature `--features isle-errors`. 2. To allow the user to optionally inspect the generated source, which nominally lives in a hard-to-find path inside `target/` now, this PR adds a feature `isle-in-source-tree` that, as implied by the name, moves the target for ISLE generated source into the source tree, at `cranelift/codegen/isle_generated_source/`. It seems reasonable to do this when an explicit feature (opt-in) is specified because this is how ISLE regeneration currently works as well. To prevent surprises, if the feature is not specified, the build fails if this directory exists.	3 years ago
Alex Crichton	bea0433b54	Fix the release process's latest step (#4055 ) * Fix the release process's latest step The automated release of 0.36.0 was attempted last night but it failed due to a [failure on CI][bad]. This failure comes about because it was trying to change the release date of 0.35.0 which ended up not modifying any fails so `git` failed to commit as no files were changed. The original bug though was that 0.35.0 was being changed instead of 0.36.0. The reason for this is that the script used `--sort=-committerdate` to determine the latest branch. I forgot, though, that with backports it's possible for 0.35.0 to have a more recent commit date than 0.36.0 (as is currently the case). This commit updates the script to perform a numerical sort outside of git to get the latest release branch. Additionally this adds in some `set -ex` commands for the shell which should help print out commands as they're run and assist in future debugging. [bad]: https://github.com/bytecodealliance/wasmtime/runs/6087188708 * Replace sed with rust	3 years ago
Bailey Hayes	76f7cde673	Add m1 to build matrix and release (#3983 ) * Add m1 to release process This will create a pre-compiled binary for m1 macs and adds a link to review embark studios CI for verification. * remove test for macos arm Tests will not succeed for macos arm until GitHub provides a an m1 hosted runner. Co-authored-by: Bailey Hayes <bhayes@singlestore.com>	3 years ago
Alex Crichton	c89dc55108	Add a two-week delay to Wasmtime's release process (#3955 ) * Bump to 0.36.0 * Add a two-week delay to Wasmtime's release process This commit is a proposal to update Wasmtime's release process with a two-week delay from branching a release until it's actually officially released. We've had two issues lately that came up which led to this proposal: * In #3915 it was realized that changes just before the 0.35.0 release weren't enough for an embedding use case, but the PR didn't meet the expectations for a full patch release. * At Fastly we were about to start rolling out a new version of Wasmtime when over the weekend the fuzz bug #3951 was found. This led to the desire internally to have a "must have been fuzzed for this long" period of time for Wasmtime changes which we felt were better reflected in the release process itself rather than something about Fastly's own integration with Wasmtime. This commit updates the automation for releases to unconditionally create a `release-X.Y.Z` branch on the 5th of every month. The actual release from this branch is then performed on the 20th of every month, roughly two weeks later. This should provide a period of time to ensure that all changes in a release are fuzzed for at least two weeks and avoid any further surprises. This should also help with any last-minute changes made just before a release if they need tweaking since backporting to a not-yet-released branch is much easier. Overall there are some new properties about Wasmtime with this proposal as well: * The `main` branch will always have a section in `RELEASES.md` which is listed as "Unreleased" for us to fill out. * The `main` branch will always be a version ahead of the latest release. For example it will be bump pre-emptively as part of the release process on the 5th where if `release-2.0.0` was created then the `main` branch will have 3.0.0 Wasmtime. * Dates for major versions are automatically updated in the `RELEASES.md` notes. The associated documentation for our release process is updated and the various scripts should all be updated now as well with this commit. * Add notes on a security patch * Clarify security fixes shouldn't be previewed early on CI	3 years ago
Brian Jones	65b443ad23	Fix for issue #3948 (#3956 ) Updates the OpenVINO backend for wasi-nn to pre-configure all inputs as `NHWC`. This is not a long-term fix but no worse than the status quo, which configures all input tensors to the same layout. This change updates the CI script to use the latest version of OpenVINO. Closes #3948. Co-authored-by: Andrew Brown <andrew.brown@intel.com>	3 years ago
Chris Fallin	ddd39cdb84	Patch qemu in CI to fix madvise semantics. (#3770 ) We currently skip some tests when running our qemu-based tests for aarch64 and s390x. Qemu has broken madvise(MADV_DONTNEED) semantics -- specifically, it just ignores madvise() [1]. We could continue to whack-a-mole the tests whenever we create new functionality that relies on madvise() semantics, but ideally we'd just have emulation that properly emulates! The earlier discussions on the qemu mailing list [2] had a proposed patch for this, but (i) this patch doesn't seem to apply cleanly anymore (it's 3.5 years old) and (ii) it's pretty complex due to the need to handle qemu's ability to emulate differing page sizes on host and guest. It turns out that we only really need this for CI when host and guest have the same page size (4KiB), so we could just pass the madvise()s through. I wouldn't expect such a patch to ever land upstream in qemu, but it satisfies our needs I think. So this PR modifies our CI setup to patch qemu before building it locally with a little one-off patch. [1] https://github.com/bytecodealliance/wasmtime/pull/2518#issuecomment-747280133 [2] https://lists.gnu.org/archive/html/qemu-devel/2018-08/msg05416.html	3 years ago
Nick Fitzgerald	d2d0a0f36b	Remove Peepmatic!!! Peepmatic was an early attempt at a DSL for peephole optimizations, with the idea that maybe sometime in the future we could user it for instruction selection as well. It didn't really pan out, however: * Peepmatic wasn't quite flexible enough, and adding new operators or snippets of code implemented externally in Rust was a bit of a pain. * The performance was never competitive with the hand-written peephole optimizers. It was very size efficient, but that came at the cost of run-time efficiency. Everything was table-based and interpreted, rather than generating any Rust code. Ultimately, because of these reasons, we never turned Peepmatic on by default. These days, we just landed the ISLE domain-specific language, and it is better suited than Peepmatic for all the things that Peepmatic was originally designed to do. It is more flexible and easy to integrate with external Rust code. It is has better time efficiency, meeting or even beating hand-written code. I think a small part of the reason why ISLE excels in these things is because its design was informed by Peepmatic's failures. I still plan on continuing Peepmatic's mission to make Cranelift's peephole optimizer passes generated from DSL rewrite rules, but using ISLE instead of Peepmatic. Thank you Peepmatic, rest in peace!	3 years ago
Alex Crichton	00feefe9a7	Change the bump-version workflow's schedule (#3512 ) * Change the bump-version workflow's schedule Either I don't understand cron or GitHub doesn't understand cron. It's not clear which. I think that https://github.com/bytecodealliance/wasmtime/pull/3511 may have fallen within our schedule but it was supposed to be on a weekday. Otherwise https://github.com/bytecodealliance/wasmtime/pull/3499 was certainly spurious. This commit moves to a simpler "just do it on the same day each month" and we can manually figure out weekdays and such. Hopefully this should reduce the number of spurious PRs we're getting to bump versions. This also removes the script to force a version bump since I found a button on the GitHub UI to do the same thing. Additionally I've updated the patch-release documentation to use this button. Note that this button takes inputs as well which means we can further automate patch releases to look even more like normal release process, differing only in one part of the argument used to trigger the workflow. * Fix a typo	3 years ago

1 2

94 Commits (6c428469f74a54a051c28a0004681791f57d446b)