The identifier for the `cold` calling convention overlaps with the
`cold` keyword for basic blocks so handle another kind of token when
parsing signatures.
The epoch interruption implementation caches the current deadline in a
register, and avoids reloading that cache until the cached deadline has
passed.
However, the first epoch check happens immediately after the cache has
been populated on function entry, so there's never a reason to reload
the cache at that point. It only needs to be reloaded in loops. So this
commit eliminates the double-check on function entry.
When Cranelift optimizations are enabled, the alias analysis pass
correctly detected that this load was redundant, and the egraph pass
optimized away the `icmp` as well. However, since we don't do
conditional constant propagation, the branch couldn't be optimized away.
On x86 this lowered to a redundant `cmp`/`jae` pair of instructions in
every function entry, which this commit eliminates.
To keep track of what code we're generating for epoch interruptions,
I've also added disas tests with a trivial infinite loop.
This was accidentally broken in #8692. It turns out bitcasts from i128 to i128 are legal, that PR accidentally reverted that use case.
This is now added to a runtest to ensure it works on all platforms.
This commit raises the default setting of `max_memory_size` in the
pooling allocator from 10M to 4G. This won't actually impact the virtual
memory reserved in the pooling allocator because we already reserved 6G
of virtual memory for each linear memory this instead allows access to
all of it by default. This matches the default behavior of Wasmtime for
the non-pooling allocator which is to not artificially limit memory by
default.
The main impact of this setting is that the memory-protection-keys
feature, which is disabled by default, will have no effect by default
unless `max_memory_size` is also configured to something smaller than
4G. The documentation has been updated to this effect.
Closes#8846
I noticed that the wasm_memory64 flag was left out of Config's debug impl,
so rather than add it, I decided to use the `bitflags::Flags::FLAGS`
const to iterate the complete set of flags.
THe downside of this change is that it will print flags which do not
have a setter in Config, e.g. `wasm_component_model_nested_names`.
An alternative to this change is, rather than expanding out the single
`features: WasmFeatures` member into many different debug_struct fields,
the debug impl of WasmFeatures is used.
Here is a sample debug of Config with this change:
Config { debug_info: None, wasm_mutable_global: true, wasm_saturating_float_to_int: true, wasm_sign_extension: true, wasm_reference_types: true, wasm_multi_value: true, wasm_bulk_memory: true, wasm_simd: true, wasm_relaxed_simd: false, wasm_threads: false, wasm_shared_everything_threads: false, wasm_tail_call: false, wasm_floats: true, wasm_multi_memory: false, wasm_exceptions: false, wasm_memory64: false, wasm_extended_const: false, wasm_component_model: false, wasm_function_references: false, wasm_memory_control: false, wasm_gc: false, wasm_custom_page_sizes: false, wasm_component_model_values: false, wasm_component_model_nested_names: false, parallel_compilation: true, compiler_config: CompilerConfig { strategy: Some(Cranelift), target: None, settings: {"opt_level": "speed", "enable_verifier": "true"}, flags: {}, cache_store: None, clif_dir: None, wmemcheck: false }, parse_wasm_debuginfo: false }
This commit removes the `simm32` extractor from lowerings as it's not as
useful as it was when it was first introduced. Nowadays an `Imm64` needs
to be interpreted with the type known as well to understand whether bits
being masked off is significant or not. The old `simm32` extractor only
took `Imm64` meaning that it was unable to do this and wouldn't match
negative numbers. This is because the high 32 bits of `Imm64` were
always zero and `simm64` would take the `i64` value from `Imm64` and try
to convert it to an `i32`.
This commit replaces `simm32`, and uses of it, with a new extractor
`i32_from_iconst`. This matches the preexisting `i64_from_iconst` and is
able to take the type of the value into account and produce a correctly
sign-extended value.
cc #8706
* Add tests for patterns I'm about to optimize
* x64: Optimize vector compare-and-branch
This commit implements lowering optimizations for the `vall_true` and
`vany_true` CLIF instructions when combined with `brif`. This is in the
same manner as `icmp` and `fcmp` combined with `brif` where the result
of the comparison is never materialized into a general purpose register
which helps lower register pressure and remove some instructions.
* x64: Optimize `vconst` with an all-ones pattern
This has a single-instruction lowering which doesn't load from memory so
it's probably cheaper than loading all-ones from memory.
* cranelift-entity: Implement `EntitySet` in terms of `cranelift_bitset::CompoundBitSet`
* Shrink the size of `CompoundBitSet` so we don't perturb vmctx size test expectations
* Update vmctx size test expectations anyways because we shrunk "too much"
* Move `cranelift-bitset` to the front of `CRATES_TO_PUBLISH`
* wasmtime: Add profile markers around host-calls
The output of the guest profiler can be misleading around hostcalls.
Whatever happened to be the last sample before the hostcall appears to
run for the entire time of the hostcall. This change ensures that we can
see the actual call stack at the time of the hostcall, and get a visual
indication of which periods are not spent executing guest code.
* wasmtime-cli needs wasmtime/call-hook, but wasmtime itself doesn't
In general, embedders that wish to use the new functionality likely will
need to enable the wasmtime/call-hook feature in order to get Wasmtime
to notify them of when to call into the profiler. However embedders
could consider other alternatives, such as calling the profiler from
selected hostcall implementations.
* Implement semver compatibility for exports
This commit is an implementation of component model semver compatibility
for export lookups. Previously in #7994 component imports were made
semver-aware to ensure that bumping version numbers would not be a
breaking change. This commit implements the same feature for component
exports. This required some refactoring to move the definition of semver
compat around and the previous refactoring in #8786 enables frontloading
this work to happen before instantiation.
Closes#8395
* Review comments
* Fix tests
This commit improves the experience around using the
`trappable_error_type` configuration by fixing two issues:
* When an error can't be resolved it doesn't result in a
`unwrap()`, instead a first-class error is returned to get reported.
* The name lookup procedure is now consistent with the name lookup that
the `with` key does, notably allowing the version to be optional but
still supporting the version.
This fixes an issue that came up recently where a path with a version
was specified but the old lookup logic ended up requiring that the
version wasn't specified because there was only one package with that
version. This behavior resulted in a panic with a very long
backtrace-based error message which was confusing to parse. By returning
an error the error is much more succinct and by supporting more names
the original intuition will work.
* Introduce the `cranelift-bitset` crate
The eventual goal is to deduplicate bitset types between Cranelift and Wasmtime,
especially their use in stack maps.
* Use the `cranelift-bitset` crate inside both Cranelift and Wasmtime
Mostly for stack maps, also for a variety of other random things where
`cranelift_codegen::bitset::BitSet` was previously used.
* Fix stack maps unit test in cranelift-codegen
* Uncomment `no_std` declaration
* Fix `CompountBitSet::reserve` method
* Fix `CompoundBitSet::insert` method
* Keep track of the max in a `CompoundBitSet`
Makes a bunch of other stuff easier, and will be needed for replacing
`cranelift_entity::EntitySet`'s bitset with this thing anyways.
* Add missing parens
* Fix a bug around insert and reserve
* Implement `with_capacity` in terms of `new` and `reserve`
* Rename `reserve` to `ensure_capacity`
* wasi-adapter: Implement provider crate that embeds the adapter binaries
* Upgrade wasi adapters to the latest version
* Update adapter docs
* Recompile asi adapters with 1.78
* Recompile wasi adapters with 1.79
* Add some debugging to adapter build script
* Fix script debugging
* Compute wasi adapter version based on latest adapter commit hash
* Try to bless wasi adapters again
* Try to work around CI auto-merges
* Revert to just using workspace version
* Add the wasi adapter provider to the crate publication list
* Use wasi adapter provider in artifacts test + explicit MSRV in CI
* Explicit adapter crate version
* Small fix
* Remove version info from adapter metadata
* Check but don't install rust toolchain in build script
* Bless after rebase
---------
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
* Update the wasi_testsuite submodule
This commit updates the wasi_testsuite submodule which we haven't
updated in a little over a year and applies a few small fixes but mostly
ignores new tests.
* Add another ignore#
* Un-nest exports in a component
This commit flattens the representation of exports in a component to
make them more easily indexable without forcing traversal through the
hierarchy of instance imports/exports to get there.
* Guarantee type information on component exports
Don't have it optional in some cases and present in others, instead
ensure there's type information for all component exports immediately
available.
* Refactor how component instance exports are loaded
This commit is a change to Wasmtime's public API for
`wasmtime::component::Instance` that reorganizes how component exports
are loaded. Previously there was a system where `Instance::exports()`
was called that that was sort of "iterated over" in a builder-style
pattern to acquire the actual export desired. This required lifetime
trickery for nested instances and some unfortunate API bloat. The major
downside of this approach is that it requires unconditional string
lookups at runtime for exports and additionally does not serve as a
great place to implement the semver-compatible logic of #8395. The goal
of this refactoring is to pave the way to improving this.
The new APIs for loading exports now look a bit more similar to what's
available for core modules. Notably there's a new
`Component::export_index` method which enables performing a string
lookup and returning an index. This index can in turn be passed to
`Instance::get_*` to skip the string lookup when exports are loaded. The
`Instance::exports` API is then entirely removed and dismantled.
The only piece remaining is the ability to load nested exports which is
done through an `Option` parameter to `Component::export_index`. The
way to load a nested instance is now to first lookup the instance with
`None` as this parameter an then the instance itself is `Some` to look
up an export of that instance. This removes the need for a
recursive-style lifetime-juggling API from wasmtime and in theory helps
simplify the usage of loading exports.
* Update `bindgen!` generated structures for exports
This commit updates the output of `bindgen!` to have a different setup
for exports of worlds to handle the changes from the previous commit.
This introduces new `*Pre` structures which are generated alongside the
existing `Guest` structures for example. The `*Pre` versions contain
`ComponentExportIndex` from the previous commit and serve as a path to
accelerating instantiation because all name lookups are skipped.
* Update test expectations for `bindgen!`-generated output
* Review comments
* Fix doc link
* wasi-nn: remove some unncecessary panics from test programs
* Make `libtest-mimic` a workspace dependency
* wasi-nn: use \`libtest-mimic\` for testing
wasi-nn's testing story is complicated by different levels of support on
different platforms (some backends work on certain architectures, others
only work on certain OSes, etc.). This change migrates the `testing`
module, which was included in `src`, to exist solely under `tests`. It
also dynamically checks whether each test is runnable and then chooses
whether to ignore it with a `libtest-mimic` flag. This ensures we can
see all the tests all the time and whether they are running or not,
which is helpful during development.
* Refactor for more subtle `ignore` behavior
On any development machine, with no prior setup, we should be able to
compile and move past the ignored tests without issue:
```console
$ cargo test -- --quiet
running 4 tests
iiii
```
With the proper setup and enabling the right features, tests that are
able to run should do so (eliding a bunch of test output):
```console
$ cargo test --all-features -- --quiet
running 4 tests
iii.
```
On CI, tests that _should_ pass will fail if they can't run:
```console
$ CI=1 cargo test --all-features -- --quiet
iFF.
```
prtest:full
* Add missing `use`
* fix: share download lock between checks
* fix: typo, winml usedx preloaded model
* fix: revert to previous winml behavior
This test was reusing the ONNX test for some reason.
* fix: fully qualify bail!
* Inherit Linux semantics for `fd_pwrite` with `O_APPEND`
This commit updates the implementation of `fd_pwrite` in WASI to match
Linux semantics for an under-specified corner of WASI. Specifically if
`fd_pwrite` is used the offset specified is ignored if the file is
opened in append mode and the bytes are instead appended.
This commit additionally refactors `fd_write` and `fd_pwrite` to have
basically the same code with only a minor branch internally when the
final write is being performed to help deduplicate more logic.
Closes#8817
* Ignore new tests on macos
prtest:full
* Update ignore to all non-linux
Looks like wasi-libc is testing for the READDIR right in addition to
the READ right in the reported flags. Update write-only files to remove
both the READ and READDIR rights accordingly.
Closes#8816
* Force some more permission checks with 0-length writes
When a 0-length write is performed try to send the write all the way to
the underlying file descriptor to at least check that it's valid to
write.
Closes#8818
* Update crates/test-programs/src/bin/preview1_file_write.rs
Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>
* Allow a second error for Windows as well
---------
Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>
* Disable memory protection keys by default at compile time
This commit gates memory protection keys behind a new Cargo feature
which is disabled by default. Memory protection keys are already
disabled by default on all platforms and are only configured to possibly
work with Linux x64. When enabled, however, it unconditionally adds a
small amount of overhead to WebAssembly entries/exits even if the
feature is disabled at runtime for the same reason that the `call-hook`
feature adds overhead. With `call-hook` being disabled by default
in #8808 it seemed reasonable to additionally gate memory protection
keys to avoid needing to disable features in Wasmtime to get the best
performance wasm<->host calls.
* Enable Wasmtime feature for fuzzing
* Disable `call-hook` crate feature by default
This commit disables the `call-hook` feature for the Wasmtime crate
added in #8795 by default. The rationale is that this has a slight cost
to all embeddings even if the feature isn't used and it's not expected
to be that widely used of a feature, so off-by-default seems like a more
appropriate default.
* Enable all features in doc build
* More doc fixes
After https://github.com/bytecodealliance/wasmtime/pull/8809, the mutator cannot
resume from a trap so we don't need to consider them safepoints, as no
GC-managed references are live after the trap. The one exception being the
`debugtrap` CLIF instruction, which is technically still a resumable trap, but
which exists only for emitting the equivalent of an `int3` breakpoint
instruction for pausing in a debugger to inspect state, and should never be used
for mutator-collector interactions.
These were originally a SpiderMonkey-ism and have been unused ever
since. It was introduced for GC integration, where the runtime could do
something to make Cranelift code hit a trap and pause for a GC and then resume
execution once GC completed. But it is unclear that, as implemented, this is
actually a useful mechanism for doing that (compared to, say, loading from some
Well Known page and the GC protecting that page and catching signals to
interrupt the mutator, or simply branching and doing a libcall). And if someone
has that particular use case in the future (Wasmtime and its GC integration
doesn't need exactly this) then we can design something for what is actually
needed at that time, instead of carrying this cruft forward forever.
* Update the spec test suite submodule
* Ignore the `memory64` tests since Wasmtime doesn't implement the
table64 extension yet.
* Ignore `gc/i31.wast` as it's got new tests which Wasmtime doesn't
currently pass.
* Only ignore table64-related memory64 tests
* Fix ignoring some tests on Winch
* Const-propagate some offsets in `VMOffsets`
Before this commit all offsets to all fields in `VMOffsets` were stored
as fields within `VMOffset` itself. All of the fields at the start of
`VMOffsets`, however, are statically known given the pointer size.
Notably this means that the use of `HostPtr` in the runtime still was
forcing a dynamic lookup of these static offsets.
This commit refactors this to reflect all static offsets based solely on
the pointer size in the `PtrSize` trait, removing all the fields from
`VMOffsets`. All the dynamically sized fields, however, remain in
`VMOffsets`.
* Fix expected error message
This commit skips the safety checks of `AutoAssertNoGc` for the duration
of host calls where the types involved are statically known to not
perform any GCs (e.g. integers and floats). This helps recover some
performance loss from indirect calls made on entry/exit of an
`AutoAssertNoGc` scope when the `gc` feature is enabled in Wasmtime.
* Add `anyhow` stuff to our internal `wasmtime` crate prelude
We use it basically everywhere and it is annoying to have to import.
I also did an audit of existing `use` statements and removed the now-redundant
ones and replaced one-off imports with usage of the prelude, so that the prelude
is available by default in more places.
* Fix `cargo doc`
This commit is an attempt to migrate from
`JamesIves/github-pages-deploy-action`, a third party action, to what
appears to be a more official action under the `github.com/actions`
organization. This was first pioneered in
bytecodealliance/wasm-tools#1597 and it looked like it would work well
over here too.
* Initial migration to `wasmtime_test`
This commit introduces the initial migration to the `wasmtime_test`
macro.
This change starts by migrating all the applicable `func.rs` integration
tests and at the same time it removes all the duplicated integration
tests in `winch.rs`.
Additionally, this change introduces a slight change to how the macro
works. Inspired by https://github.com/bytecodealliance/wasmtime/pull/8622#pullrequestreview-2083046911
it defaults to including all the known configuration combinations and
allows the macro user to opt-out where applicable. This makes the usage
of the macro less verbose.
The intention is to follow-up with subsequent PRs to migrate the all the
applicable tests.
* Remove unused `bail` import
* Add the ability to specify `wasm_features`
This commit adds the ability to specify `wasm_features` when invoking
the macro.
If the feature is off by default, the macro will ensure that the feature
is enabled in the resulting config.
If the feature is not supported by any of the compiler strategies, no
tests will be generated for such strategy.
* Migrate usage of the `memoffset` crate to `core::mem::offset_of!`
* C-string literals such as `c"foo"` are now possible
* Some helpers for fixed-length slices such as `slice::first_chunk_mut`
are starting to stabilize.
This accompanies today's release of Rust 1.79.0. This means that CI will
now use the latest stable of 1.79.0 for testing primarily. Additionally
this updates the pinned nightly commit used for testing to today's
nightly and fixes a few warnings that uncovered. I plan on having a
follow-up that leverages some new APIs and features in 1.77.0.
* Add a compile-time feature for call hooks
This commit moves the `Store::call_hook` API behind a Cargo feature
named `call-hook`. This helps speed up the path from wasm into the host
by avoiding branches at the start and the end of the execution. In a
thread on [Zulip] this is locally leading to significant performance
gains in this particular microbenchmark so having an option to disable
it at the crate layer seems like a reasonable way to thread this needle
for now. This definitely has a downside in that it requires a crate
feature at all, but I'm not sure of a better solution as LLVM can't
dynamically detect that `Store::call_hook` is never invoked and
therefore the branch can be optimized away.
[Zulip]: https://bytecodealliance.zulipchat.com/#narrow/stream/217126-wasmtime/topic/Performance.20regression.20since.20rust.201.2E65/near/444505571
* Fix a feature build
When the pooling allocator is itself disabled then there's no use for
enabling MPK so this commit switches the implementation to all of the
disabled versions of the primitives. This notably makes `ProtectionKey`
an uninhabited `enum` meaning that `Option<ProtectionKey>` in the
`Store` is a zero-sized field and `.is_none()` returns a constant
`true`. This is on the hot path of entering/exiting wasm so can help
speed up things a bit there.
Rather than digging only the embedder's store-specific data out and
passing that along, pass access to the entire Store. This is the same
thing we do for epoch interruption hooks, including the annoyance of
having to take the callback out of the store temporarily while calling
it to avoid having multiple mutable borrows.
* Wasmtime: Implement the custom-page-sizes proposal
This commit adds support for the custom-page-sizes proposal to Wasmtime:
https://github.com/WebAssembly/custom-page-sizes
I've migrated, fixed some bugs within, and extended the `*.wast` tests for this
proposal from the `wasm-tools` repository. I intend to upstream them into the
proposal shortly.
There is a new `wasmtime::Config::wasm_custom_page_sizes_proposal` method to
enable or disable the proposal. It is disabled by default.
Our fuzzing config has been updated to turn this feature on/off as dictated by
the arbitrary input given to us from the fuzzer.
Additionally, there were getting to be so many constructors for
`wasmtime::MemoryType` that I added a builder rather than add yet another
constructor.
In general, we store the `log2(page_size)` rather than the page size
directly. This helps cut down on invalid states and properties we need to
assert.
I've also intentionally written this code such that supporting any power of two
page size (rather than just the exact values `1` and `65536` that are currently
valid) will essentially just involve updating `wasmparser`'s validation and
removing some debug asserts in Wasmtime.
* Update error string expectation
* Remove debug logging
* Use a right shift instead of a division
* fix error message expectation again
* remove page size from VMMemoryDefinition
* fix size of VMMemoryDefinition again
* Only dynamically check for `-1` sentinel for 1-byte page sizes
* Import functions that are used a few times
* Better handle overflows when rounding up to the host page size
Propagate errors instead of returning a value that is not actually a rounded up
version of the input.
Delay rounding up various config sizes until runtime instead of eagerly doing it
at config time (which isn't even guaranteed to work, so we already had to have a
backup plan to round up at runtime, since we might be cross-compiling wasm or
not have the runtime feature enabled).
* Fix some anyhow and nostd errors
* Add missing rounding up to host page size at runtime
* Add validate feature to wasmparser dep
* Add some new rounding in a few places, due to no longer rounding in config methods
* Avoid actually trying to allocate the whole address space in the `massive_64_bit_still_limited` test
The point of the test is to ensure that we hit the limiter, so just cancel the
allocation from the limiter, and otherwise avoid MIRI attempting to allocate a
bunch of memory after we hit the limiter.
* prtest:full
* Revert "Avoid actually trying to allocate the whole address space in the `massive_64_bit_still_limited` test"
This reverts commit ccfa34a78dd3d53e49a6158ca03077d42ce8bcd7.
* miri: don't attempt to allocate more than 4GiB of memory
It seems that rather than returning a null pointer from `std::alloc::alloc`,
miri will sometimes choose to simply crash the whole program.
* remove duplicate prelude import after rebasing
* Fix lost `Waker` instances with async stdio streams
This commit fixes a bug in the `Subscribe` trait implementation for
`AsyncStd{in,out}Stream` structures in the `wasmtime-wasi` crate.
Previously these implementations would create a future for the duration
of a single `poll` but then the future was dropped which could lead to
lost wakeups as the waker is gone after the future is dropped. The fix
was to use a `tokio::sync::Mutex` here instead of a `std::sync::Mutex`
and leave some comments about why contention isn't expected.
Closes#8781
* Reduce sizes used in tests
Replace it with an `enum` of the two possibilities that it can be. This
removes the need to have a trait dispatch indirection in the `vm`
module. Previously this was required as `wasmtime-runtime` was a
separate crate, but now it's no longer required.
Alex Crichton
Jamey Sharp
Afonso Bordado
Pat Hickey
Nick Fitzgerald
Juniper Tyree
Andrew Brown
Saúl Cabrera
Michael McLoughlin
Matthew Fisher