* c-api: Tidy up some `wasmtime_func_t` usage
Try to avoid using a `transmute` in the implementation of the C API and
instead use a `union` like is being done in #8451. Additionally add some
helper functions to the header to work with null funcref values instead
of only documenting the implementation.
* Try to fix doxygen
* add cloning for String attributes
* use into_owned instead of to_vec to avoid a clone if possible.
* runs, but does not substitute
* show vars from c program, start cleanup
* tiday
* resolve conflicts
* remove WASI folder
* Add module_builder
add dwarf_package to state for cache
* move dwarf loading to module_environ.rs
pass the dwarf as binary as low as module_environ.rs
* pass dwarf package rather than add to debug_info
* tidy option/result nested if
* revert some toml and whitespace.
* add features cranelift,winch to module_builder and compute_artifacts
remove some `use`s
* address some feedback
remove unused 'use's
* address some feedback
remove unused 'use's
* move wat feature condition to cover whole method.
* More feedback
Another try at wat feature move
* Another try at wat feature move
* change gimli exemption version
add typed-arena exemption
* add None for c-api
* move `use` to #cfg
* fix another config build
* revert unwanted code deletion
* move inner function closer to use
* revert extra param to Module::new
* workaround object crate bug.
* add missing parameter
* add missing parameter
* Merge remote-tracking branch 'origin/main' into dwarf-att-string
# Conflicts:
# crates/wasmtime/src/engine.rs
# crates/wasmtime/src/runtime/module.rs
# src/common.rs
* remove moduke
* use common gimli version of 28.1
* remove wasm feature, revert gimli version
* remove use of object for wasm dwarf
* remove NativeFile workaround, add feature for dwp loading
* sync winch signature
* revert bench api change
* add dwarf for no cache feature
* put back merge loss of module kind
* remove param from docs
* add dwarf fission lldb test
* simplify and include test source
* clang-format
* address feedback, remove packages
add docs
simplify return type
* remove Default use on ModuleTypesBuilder
* Remove an `unwrap()` and use `if let` instead
* Use `&[u8]` instead of `&Vec<u8>`
* Remove an `unwrap()` and return `None` instead
* Clean up some code in `transform_dwarf`
* Clean up some code in `replace_unit_from_split_dwarf`
* Clean up some code in `split_unit`
* Minor refactorings and documentation in `CodeBuilder`
* Restrict visibility of `dwarf_package_binary`
* Revert supply-chain folder changes
* Fix compile error on nightly
* prtest:full
* prtest:full
* prtest:full
* prtest:full
* prtest:full
* prtest:full
* prtest:full
* prtest:full
* use lldb 15
* prtest:full
* prtest:full
* load dwp when loading wasm bytes with path
* correct source file name
* remove debug
---------
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
Fixes https://github.com/bytecodealliance/wasmtime/issues/8446
The WebAssembly tail call proposal is currently not supported in Winch. This commit returns an error when trying to enable the tail call proposal while using Winch as the compiler.
Even though the issue linked above doesn't make use of any of the tail instructions, the trampolines were generated using Cranelift's tail call calling convention.
* Add a fuzzer for async wasm
This commit revives a very old branch of mine to add a fuzzer for
Wasmtime in async mode. This work was originally blocked on
llvm/llvm-project#53891 and while that's still an issue it now contains
a workaround for that issue. Support for async fuzzing required a good
deal of refactorings and changes, and the highlights are:
* The main part is that new intrinsics,
`__sanitizer_{start,finish}_fiber_switch` are now invoked around the
stack-switching routines of fibers. This only works on Unix and is set
to only compile when ASAN is enabled (otherwise everything is a noop).
This required refactoring of things to get it all in just the right
way for ASAN since it appears that these functions not only need to be
called but more-or-less need to be adjacent to each other in the code.
My guess is that while we're switching ASAN is in a "weird state" and
it's not ready to run arbitrary code.
* Stacks are a problem. The above issue in LLVM outlines how stacks
cannot be deallocated at this time because if the deallocated virtual
memory is later used for the heap then ASAN will have a false positive
about stack overflow. To handle this stacks are specially handled in
asan mode by using a special allocation path that never deallocates
stacks. This logic additionally applies to the pooling allocator which
uses a different stack allocation strategy with ASAN.
With all of the above a new fuzzer is added. This fuzzer generates an
arbitrary module, selects an arbitrary means of async (e.g.
epochs/fuel), and then tries to execute the exports of the module with
various values. In general the fuzzer is looking for crashes/panics as
opposed to correct answers as there's no oracle here. This is also
intended to stress the code used to switch on and off stacks.
* Fix non-async build
* Remove unused import
* Review comments
* Fix compile on MIRI
* Fix Windows build
This commit fixes a panic when a host function defined with `Func::new`
returned GC references and was called in async mode. The logic to
auto-gc before the return values go to wasm asserted that a synchronous
GC was possible but the context this function is called in could be
either async or sync. The fix applied in this commit is to remove the
auto-gc. This means that hosts will need to explicitly GC in these
situations until auto-gc is re-added back to Wasmtime.
cc #8433 as this will make the behavior consistent, but we'll want to
re-add the gc behavior.
`Callee::probestack_min_frame` was always set to the guard-page size.
That is the same as the `guard_size` local in `gen_prologue`, which was
also the only place which used `probestack_min_frame`. So don't even
bother storing it, just compute it from the flags as needed.
Also, remove the long-obsolete `probestack_func_adjusts_sp` setting,
which hasn't been used since at least 2021. We may actually want to do
something like what that setting described, but even if we do, it should
be a choice the backend makes rather than a global setting.
Previously, `wasmtime_types::EngineOrModuleTypeIndex` had a raw `u32` for its
engine-level-canonicalized variant. This change allows it to store a
`VMSharedTypeIndex` directly, giving us some additional static assurance that we
aren't messing up our different index types and lets us remove a ton of
conversions back and forth between raw `u32`s and `VMSharedTypeIndex`.
By default previously all return types were wrapped in
`wasmtime::Result<T>` to enable any import to return a trap to the wasm
guest. This is a fair bit of boilerplate, however, and it's easy to
accidentally turn a normal error into a trap. This is additionally
somewhat of a power-user method as many imports probably don't end up
wanting to trap.
This commit adds a new configuration option to `bindgen!`:
`trappable_imports`, and switches the default to `false`. The previous
behavior can be recovered with `trappable_imports: true`.
* Cranelift: Do not dedupe/GVN bitcasts from reference values
Deduping bitcasts to integers from references can make the references no long
longer live across safepoints, and instead only the bitcasted integer results
would be. Because the reference is no longer live after the safepoint, the
safepoint's stack map would not have an entry for the reference, which could
result in the collector reclaiming an object too early, which is basically a
use-after-free bug. Luckily, we sandbox the GC heap now, so such UAF bugs aren't
memory unsafe, but they could potentially result in denial of service
attacks. Either way, we don't want those bugs!
On the other hand, it is technically fine to dedupe bitcasts *to* reference
types. Doing so extends, rather than shortens, the live range of the GC
reference. This potentially adds it to more stack maps than it otherwise would
have been in, which means it might unnecessarily survive a GC it otherwise
wouldn't have. But that is fine. Shrinking live ranges of GC references, and
removing them from stack maps they otherwise should have been in, is the
problematic transformation.
* Add additional logging and debug asserts for GC stuff
\### The `GcRuntime` and `GcCompiler` Traits
This commit factors out the details of the garbage collector away from the rest
of the runtime and the compiler. It does this by introducing two new traits,
very similar to a subset of [those proposed in the Wasm GC RFC], although not
all equivalent functionality has been added yet because Wasmtime doesn't
support, for example, GC structs yet:
[those proposed in the Wasm GC RFC]: https://github.com/bytecodealliance/rfcs/blob/main/accepted/wasm-gc.md#defining-the-pluggable-gc-interface
1. The `GcRuntime` trait: This trait defines how to create new GC heaps, run
collections within them, and execute the various GC barriers the collector
requires.
Rather than monomorphize all of Wasmtime on this trait, we use it
as a dynamic trait object. This does imply some virtual call overhead and
missing some inlining (and resulting post-inlining) optimization
opportunities. However, it is *much* less disruptive to the existing embedder
API, results in a cleaner embedder API anyways, and we don't believe that VM
runtime/embedder code is on the hot path for working with the GC at this time
anyways (that would be the actual Wasm code, which has inlined GC barriers
and direct calls and all of that). In the future, once we have optimized
enough of the GC that such code is ever hot, we have options we can
investigate at that time to avoid these dynamic virtual calls, like only
enabling one single collector at build time and then creating a static type
alias like `type TheOneGcImpl = ...;` based on the compile time
configuration, and using this type alias in the runtime rather than a dynamic
trait object.
The `GcRuntime` trait additionally defines a method to reset a GC heap, for
use by the pooling allocator. This allows reuse of GC heaps across different
stores. This integration is very rudimentary at the moment, and is missing
all kinds of configuration knobs that we should have before deploying Wasm GC
in production. This commit is large enough as it is already! Ideally, in the
future, I'd like to make it so that GC heaps receive their memory region,
rather than allocate/reserve it themselves, and let each slot in the pooling
allocator's memory pool be *either* a linear memory or a GC heap. This would
unask various capacity planning questions such as "what percent of memory
capacity should we dedicate to linear memories vs GC heaps?". It also seems
like basically all the same configuration knobs we have for linear memories
apply equally to GC heaps (see also the "Indexed Heaps" section below).
2. The `GcCompiler` trait: This trait defines how to emit CLIF that implements
GC barriers for various operations on GC-managed references. The Rust code
calls into this trait dynamically via a trait object, but since it is
customizing the CLIF that is generated for Wasm code, the Wasm code itself is
not making dynamic, indirect calls for GC barriers. The `GcCompiler`
implementation can inline the parts of GC barrier that it believes should be
inline, and leave out-of-line calls to rare slow paths.
All that said, there is still only a single implementation of each of these
traits: the existing deferred reference-counting (DRC) collector. So there is a
bunch of code motion in this commit as the DRC collector was further isolated
from the rest of the runtime and moved to its own submodule. That said, this was
not *purely* code motion (see "Indexed Heaps" below) so it is worth not simply
skipping over the DRC collector's code in review.
\### Indexed Heaps
This commit does bake in a couple assumptions that must be shared across all
collector implementations, such as a shared `VMGcHeader` that all objects
allocated within a GC heap must begin with, but the most notable and
far-reaching of these assumptions is that all collectors will use "indexed
heaps".
What we are calling indexed heaps are basically the three following invariants:
1. All GC heaps will be a single contiguous region of memory, and all GC objects
will be allocated within this region of memory. The collector may ask the
system allocator for additional memory, e.g. to maintain its free lists, but
GC objects themselves will never be allocated via `malloc`.
2. A pointer to a GC-managed object (i.e. a `VMGcRef`) is a 32-bit offset into
the GC heap's contiguous region of memory. We never hold raw pointers to GC
objects (although, of course, we have to compute them and use them
temporarily when actually accessing objects). This means that deref'ing GC
pointers is equivalent to deref'ing linear memory pointers: we need to add a
base and we also check that the GC pointer/index is within the bounds of the
GC heap. Furthermore, compressing 64-bit pointers into 32 bits is a fairly
common technique among high-performance GC
implementations[^compressed-oops][^v8-ptr-compression] so we are in good
company.
3. Anything stored inside the GC heap is untrusted. Even each GC reference that
is an element of an `(array (ref any))` is untrusted, and bounds checked on
access. This means that, for example, we do not store the raw pointer to an
`externref`'s host object inside the GC heap. Instead an `externref` now
stores an ID that can be used to index into a side table in the store that
holds the actual `Box<dyn Any>` host object, and accessing that side table is
always checked.
[^compressed-oops]: See ["Compressed OOPs" in
OpenJDK.](https://wiki.openjdk.org/display/HotSpot/CompressedOops)
[^v8-ptr-compression]: See [V8's pointer
compression](https://v8.dev/blog/pointer-compression).
The good news with regards to all the bounds checking that this scheme implies
is that we can use all the same virtual memory tricks that linear memories use
to omit explicit bounds checks. Additionally, (2) means that the sizes of GC
objects is that much smaller (and therefore that much more cache friendly)
because they are only holding onto 32-bit, rather than 64-bit, references to
other GC objects. (We can, in the future, support GC heaps up to 16GiB in size
without losing 32-bit GC pointers by taking advantage of `VMGcHeader` alignment
and storing aligned indices rather than byte indices, while still leaving the
bottom bit available for tagging as an `i31ref` discriminant. Should we ever
need to support even larger GC heap capacities, we could go to full 64-bit
references, but we would need explicit bounds checks.)
The biggest benefit of indexed heaps is that, because we are (explicitly or
implicitly) bounds checking GC heap accesses, and because we are not otherwise
trusting any data from inside the GC heap, we greatly reduce how badly things
can go wrong in the face of collector bugs and GC heap corruption. We are
essentially sandboxing the GC heap region, the same way that linear memory is a
sandbox. GC bugs could lead to the guest program accessing the wrong GC object,
or getting garbage data from within the GC heap. But only garbage data from
within the GC heap, never outside it. The worse that could happen would be if we
decided not to zero out GC heaps between reuse across stores (which is a valid
trade off to make, since zeroing a GC heap is a defense-in-depth technique
similar to zeroing a Wasm stack and not semantically visible in the absence of
GC bugs) and then a GC bug would allow the current Wasm guest to read old GC
data from the old Wasm guest that previously used this GC heap. But again, it
could never access host data.
Taken altogether, this allows for collector implementations that are nearly free
from `unsafe` code, and unsafety can otherwise be targeted and limited in scope,
such as interactions with JIT code. Most importantly, we do not have to maintain
critical invariants across the whole system -- invariants which can't be nicely
encapsulated or abstracted -- to preserve memory safety. Such holistic
invariants that refuse encapsulation are otherwise generally a huge safety
problem with GC implementations.
\### `VMGcRef` is *NOT* `Clone` or `Copy` Anymore
`VMGcRef` used to be `Clone` and `Copy`. It is not anymore. The motivation here
was to be sure that I was actually calling GC barriers at all the correct
places. I couldn't be sure before. Now, you can still explicitly copy a raw GC
reference without running GC barriers if you need to and understand why that's
okay (aka you are implementing the collector), but that is something you have to
opt into explicitly by calling `unchecked_copy`. The default now is that you
can't just copy the reference, and instead call an explicit `clone` method (not
*the* `Clone` trait, because we need to pass in the GC heap context to run the
GC barriers) and it is hard to forget to do that accidentally. This resulted in
a pretty big amount of churn, but I am wayyyyyy more confident that the correct
GC barriers are called at the correct times now than I was before.
\### `i31ref`
I started this commit by trying to add `i31ref` support. And it grew into the
whole traits interface because I found that I needed to abstract GC barriers
into helpers anyways to avoid running them for `i31ref`s, so I figured that I
might as well add the whole traits interface. In comparison, `i31ref` support is
much easier and smaller than that other part! But it was also difficult to pull
apart from this commit, sorry about that!
---------------------
Overall, I know this is a very large commit. I am super happy to have some
synchronous meetings to walk through this all, give an overview of the
architecture, answer questions directly, etc... to make review easier!
prtest:full
This commit refactors the `wasmtime-runtime` crate to avoid the
`std::panic` module entirely if it's compiled with `panic=abort`. From
an optimization perspective this is not really required since it'll
optimize the same either way with `-Cpanic=abort`, but avoiding
`std::panic` can help make the code a bit more portable. This
refactoring bundles in the `catch_unwind` with the longjmp of the panic
to keep the `#[cfg]` in one location. Callers are then updated as
appropriate.
* Lift all serde deps to the workspace level
Deduplicate some versions mentioned throughout crates in the workspace.
* Lift `bincode` deps to the workspace configuration level
Deduplicate some mentioned versions throughout.
* Lift libc deps up to the workspace root
As with prior commits, deduplicate some versions mentioned.
* Bad relocation type generated
When disabling sse on x86_64 architecture machines and generating float
libcall, incorrect relocation type R_X86_64_8 may be obtained, with the
correct type being R_X86_64_64.
* Gate support for the wasm `threads` proposal behind a Cargo feature
This commit moves support for the `threads` proposal behind a new
on-by-default Cargo feature: `threads`. This is intended to support
building Wasmtime with fewer runtime dependencies such as those required
for the atomic operations on memories.
This additionally adds the `gc` feature in a few missing places too.
* Fix compile of C API without threads
* Fix rustdoc warnings on Nightly
I noticed during a failed doc build of another PR we've got a number of
warnings being emitted, so resolve all those here.
* Fix more warnings
* Fix rebase conflicts
* Add a `compile` feature to `wasmtime-environ`
This commit adds a compile-time feature to remove some dependencies of
the `wasmtime-environ` crate. This compiles out support for compiling
modules/components and makes the crate slimmer in terms of amount of
code compiled along with its dependencies. Much of this should already
have been statically removed by native linkers so this likely won't have
any compile-size impact, but it's a nice-to-have in terms of
organization.
This has a fair bit of shuffling around of code, but apart from
renamings and movement there are no major changes here.
* Fix compile issue
* Gate `ModuleTranslation` and its methods on `compile`
* Fix doc link
* Fix doc link
* Plumb coredump feature to `wasmtime-runtime`
The `wasmtime` crate already has a `coredump` feature but whether or not
it's enabled the `wasmtime-runtime` crate still captures a core dump.
Use this flag in the `wasmtime` crate to plumb support to
`wasmtime-runtime` to skip capture if it's not enabled.
* Fix a typo
This commit fixes a mistake in #8181 which meant that the caching for
components was no longer working. The mistake is fixed in this commit as
well as a new test being added too.
* Exit through Cranelift-generated trampolines for builtins
This commit changes how builtin functions in Wasmtime (think
`memory.grow`) are implemented. These functions are required to exit
through some manner of trampoline to handle runtime requirements for
backtracing right now. Currently this is done via inline assembly for
each architecture (or external assembly for s390x). This is a bit
unfortunate as it's a lot of hand-coding and making sure everything is
right, and it's not easy to update as it's multiple platforms to update.
The change in this commit is to instead use Cranelift-generated
trampolines for this purpose instead. The path for invoking a builtin
function now looks like:
* Wasm code calls a statically known symbol for each builtin.
* The statically known symbol will perform exit trampoline duties (e.g.
pc/fp/etc) and then load a function pointer to the host
implementation.
* The host implementation is invoked and then proceeds as usual.
The main new piece for this PR is that all wasm modules and functions
are compiled in parallel but an output of this compilation phase is what
builtin functions are required. All builtin functions are then unioned
together into one set and then anything required is generated just
afterwards. That means that only one builtin-trampoline per-module is
generated per-builtin.
This work is inspired by #8135 and my own personal desire to have as
much about our ABI details flowing through Cranelift as we can. This in
theory makes it more flexible to deal with future improvements to our
ABI.
prtest:full
* Fix some build issues
* Update winch test expectations
* Update Winch to use new builtin shims.
This commit refactors the Winch compiler to use the new trampolines for
all Wasmtime builtins created in the previous commits. This required a
fair bit of refactoring to handle plumbing through a new kind of
relocation and function call.
Winch's `FuncEnv` now contains a `PrimaryMap` from `UserExternalNameRef`
to `UserExternalName`. This is because there's now more than one kind of
name than just wasm function relocations, so the raw index space of
`UserExternalNameRef` is no longer applicable. This required threading
`FuncEnv` to more locations along with some refactorings to ensure that
lifetimes work out ok.
The `CompiledFunction` no longer stores a trait object of how to map
name refs to names and now directly has a `Primarymap`. This also means
that Winch's return value from its `TargetIsa` is a `CompiledFunction`
as opposed to the previous just-a-`MachBuffer` so it can also package up
all the relocation information. This ends up having `winch-codegen`
depend on `wasmtime-cranelift-shared` as a new dependency.
* Review feedback
* Add a `ModuleBuilder` type to the `wasmtime` crate
This commit is extracted from #8055 and adds a new `ModuleBuilder` type
which is intended to be able to further configure compilation beyond
what the constructors of `Module` already provide. For example in #8055
knobs will be added to process `*.dwp` files for more debuginfo
processing.
Co-authored-by: yowl00 <scott.waye@hubse.com>
* Fix build
* Review feedback
* Rename ModuleBuilder to CodeBuilder
* Fix doc errors
---------
Co-authored-by: yowl00 <scott.waye@hubse.com>
* Don't lookup trap codes twice on traps
Currently whenever a signal or trap is handled in Wasmtime we perform
two lookups of the trap code. One during the trap handling itself to
ensure we can handle the trap, and then a second once the trap is
handled. There's not really any need to do two here, however, as the
result of the first can be carried over to the second.
While I was here refactoring things I also changed how some return
values are encoded, such as `take_jmp_buf_if_trap` now returns a more
self-descriptive enum.
* Fix dead code warning on MIRI
* Update crates/environ/src/trap_encoding.rs
Co-authored-by: bjorn3 <17426603+bjorn3@users.noreply.github.com>
* Fix min-platform build
---------
Co-authored-by: bjorn3 <17426603+bjorn3@users.noreply.github.com>
This commit uses the support from #8162 to skip null function pointer
checks when performing an indirect call. Instead of an explicit check
the segfault from accessing the null function pointer is caught and
annotated with the appropriate trap.
Closes#5291
* Use wasmtime-cranelift compiler for winch trampolines
* Plumb the winch calling convention through Tunables
* Guard setting the winch_callable tunable
* Allow the trampolines field to be unused when component-model is disabled
This commit implements bytecodealliance/wit-bindgen#840 for Wasmtime.
Currently `with` keys to bindgen only support either mapping resources
or interfaces. Now they additionally support mapping entire packages or
entire namespaces. This can help clean up some references to interfaces
through our own bindings to avoid listing all of them.
* Add a component type accessor for `Component`
All the bits are already in place so all that's needed is to touch up
some docs and fixup a minor case uncovered through testing. Otherwise
this enables runtime reflection over the imports and exports of an
uninstantiated component.
* Run test through miri
This commit clears out the dynamic tracking of borrows within a
`ResourceAny` when it's converted to a `Resource<T>`. The `Resource<T>`
type doesn't participate in this borrow tracking in the same way as
`ResourceAny` because it's assumed that hosts are well-behaved in this
regard. This means that the conversion here should respect these
guarantees of `Resource<T>` and auto-clear the borrow tracking.
* Remove type information from dynamic component funcs
This commit removes the `&Component` argument from the
`component::Linker::func_new` API. This is inspired by #8062 where `Val`
holds less type information as well in addition to the realization that
type-checking happens at runtime rather than instantiation time.
This argument was originally added to mirror
`wasmtime::Linker::func_new` which takes a type argument of the core
wasm function that's being defined. Unlike core wasm, though, component
functions already have to carry along their type information as part of
function calls to handle resources correctly. This means that when a
host function is invoked the type is already known of all the parameters
and results. Additionally values are already required to be type-checked
going back into wasm, so there's less of a need to perform an additional
type-check up front.
The main consequence of this commit is that it's a bit more difficult
for embeddings to know what the expected types of results are. No type
information is provided when a host function is defined, not even
function arity. This means that when the host function is invoked it may
not know how many results are expected to be produced and of what type.
Typically though a bindings generator is used somewhere along the way so
that's expected to alleviate this issue.
Finally my hope is to enhance this "dynamic" API in the future with a
bit more information so the type information is more readily accessible
at runtime. For now though hosts will have to "simply know what to do".
* Update crates/wasmtime/src/runtime/component/linker.rs
Co-authored-by: Joel Dice <joel.dice@fermyon.com>
* Fix doc links
* Fix component call benchmarks
---------
Co-authored-by: Joel Dice <joel.dice@fermyon.com>
* Move dtor information from `ResourceAny` into a `Store`
This commit moves the internals of the `ResourceAny` type related to
destructors into the auxiliary data within a `Store`. This avoids the
need to carry around pointers with `ResourceAny` and additionally makes
it easier to work with.
As part of this commit this also updates the behavior of
`ResourceAny::try_from_resource` to no longer need an `InstancePre` and
type information. This was required originally to get
`ResourceAny::resource_drop` working to drop the host resource. In
retrospect I'm not sure if this was the best goal to achieve because
`Resource<T>` already has no destructor support and one of the more
common use cases for the conversion is to simply turn around and give it
back to a component where a component already has enough destructor
information.
In the end this should make both `ResourceAny` and `Resource<T>` pretty
close to "just an index" which is easier to reason about when working
with resources than carrying additional pointers.
* Remove now-unneeded ResourceImportIndex machinery
This commit is a large refactor of the `Val` type as used with
components to remove inherent type information present currently. The
`Val` type is now only an AST of what a component model value looks like
and cannot fully describe the type that it is without further context.
For example enums only store the case that's being used, not the full
set of cases.
The motivation for this commit is to make it simpler to use and
construct `Val`, especially in the face of resources. Some problems
solved here are:
* With resources in play managing type information is not trivial and
can often be surprising. For example if you learn the type of a
function from a component and the instantiate the component twice the
type information is not suitable to use with either function due to
exported resources acquiring unique types on all instantiations.
* Functionally it's much easier to construct values when type
information is not required as it no longer requires probing various
pieces for type information here and there.
* API-wise there's far less for us to maintain as there's no need for a
type-per-variant of component model types. Pieces now fit much more
naturally into a `Val` shape without extra types.
* Functionally when working with `Val` there's now only one typecheck
instead of two. Previously a typecheck was performed first when a
`Val` was created and then again later when it was passed to wasm. Now
the typecheck only happens when passed to wasm.
It's worth pointing out that `Val` as-is is a pretty inefficient
representation of component model values, for example flags are stored
as a list of strings. While semantically correct this is quite
inefficient for most purposes other than "get something working". To
that extent my goal is to, in the future, add traits that enable
building a custom user-defined `Val` (of sorts), but still dynamically.
This should enable embedders to opt-in to a more efficient
representation that relies on contextual knowledge.
* Fix `wasmtime settings` command.
Currently the `settings` command panics because the tunables are not set in the
compiler builder.
This commit creates a default tunables based on either the target triple passed
on the command line or uses the default for the host.
Fixes#8058.
* Additional clean up.
* Define garbage collection rooting APIs
Rooting prevents GC objects from being collected while they are actively being
used.
We have a few sometimes-conflicting goals with our GC rooting APIs:
1. Safety: It should never be possible to get a use-after-free bug because the
user misused the rooting APIs, the collector "mistakenly" determined an
object was unreachable and collected it, and then the user tried to access
the object. This is our highest priority.
2. Moving GC: Our rooting APIs should moving collectors (such as generational
and compacting collectors) where an object might get relocated after a
collection and we need to update the GC root's pointer to the moved
object. This means we either need cooperation and internal mutability from
individual GC roots as well as the ability to enumerate all GC roots on the
native Rust stack, or we need a level of indirection.
3. Performance: Our rooting APIs should generally be as low-overhead as
possible. They definitely shouldn't require synchronization and locking to
create, access, and drop GC roots.
4. Ergonomics: Our rooting APIs should be, if not a pleasure, then at least not
a burden for users. Additionally, the API's types should be `Sync` and `Send`
so that they work well with async Rust.
For example, goals (3) and (4) are in conflict when we think about how to
support (2). Ideally, for ergonomics, a root would automatically unroot itself
when dropped. But in the general case that requires holding a reference to the
store's root set, and that root set needs to be held simultaneously by all GC
roots, and they each need to mutate the set to unroot themselves. That implies
`Rc<RefCell<...>>` or `Arc<Mutex<...>>`! The former makes the store and GC root
types not `Send` and not `Sync`. The latter imposes synchronization and locking
overhead. So we instead make GC roots indirect and require passing in a store
context explicitly to unroot in the general case. This trades worse ergonomics
for better performance and support for moving GC and async Rust.
Okay, with that out of the way, this module provides two flavors of rooting
API. One for the common, scoped lifetime case, and another for the rare case
where we really need a GC root with an arbitrary, non-LIFO/non-scoped lifetime:
1. `RootScope` and `Rooted<T>`: These are used for temporarily rooting GC
objects for the duration of a scope. Upon exiting the scope, they are
automatically unrooted. The internal implementation takes advantage of the
LIFO property inherent in scopes, making creating and dropping `Rooted<T>`s
and `RootScope`s super fast and roughly equivalent to bump allocation.
This type is vaguely similar to V8's [`HandleScope`].
[`HandleScope`]: https://v8.github.io/api/head/classv8_1_1HandleScope.html
Note that `Rooted<T>` can't be statically tied to its context scope via a
lifetime parameter, unfortunately, as that would allow the creation and use
of only one `Rooted<T>` at a time, since the `Rooted<T>` would take a borrow
of the whole context.
This supports the common use case for rooting and provides good ergonomics.
2. `ManuallyRooted<T>`: This is the fully general rooting API used for holding
onto non-LIFO GC roots with arbitrary lifetimes. However, users must manually
unroot them. Failure to manually unroot a `ManuallyRooted<T>` before it is
dropped will result in the GC object (and everything it transitively
references) leaking for the duration of the `Store`'s lifetime.
This type is roughly similar to SpiderMonkey's [`PersistentRooted<T>`],
although they avoid the manual-unrooting with internal mutation and shared
references. (Our constraints mean we can't do those things, as mentioned
explained above.)
[`PersistentRooted<T>`]: http://devdoc.net/web/developer.mozilla.org/en-US/docs/Mozilla/Projects/SpiderMonkey/JSAPI_reference/JS::PersistentRooted.html
At the end of the day, both `Rooted<T>` and `ManuallyRooted<T>` are just tagged
indices into the store's `RootSet`. This indirection allows working with Rust's
borrowing discipline (we use `&mut Store` to represent mutable access to the GC
heap) while still allowing rooted references to be moved around without tying up
the whole store in borrows. Additionally, and crucially, this indirection allows
us to update the *actual* GC pointers in the `RootSet` and support moving GCs
(again, as mentioned above).
* Reorganize GC-related submodules in `wasmtime-runtime`
* Reorganize GC-related submodules in `wasmtime`
* Use `Into<StoreContext[Mut]<'a, T>` for `Externref::data[_mut]` methods
* Run rooting tests under MIRI
* Make `into_abi` take an `AutoAssertNoGc`
* Don't use atomics to update externref ref counts anymore
* Try to make lifetimes/safety more-obviously correct
Remove some transmute methods, assert that `VMExternRef`s are the only valid
`VMGcRef`, etc.
* Update extenref constructor examples
* Make `GcRefImpl::transmute_ref` a non-default trait method
* Make inline fast paths for GC LIFO scopes
* Make `RootSet::unroot_gc_ref` an `unsafe` function
* Move Hash and Eq for Rooted, move to impl methods
* Remove type parameter from `AutoAssertNoGc`
Just wrap a `&mut StoreOpaque` directly.
* Make a bunch of internal `ExternRef` methods that deal with raw `VMGcRef`s take `AutoAssertNoGc` instead of `StoreOpaque`
* Fix compile after rebase
* rustfmt
* revert unrelated egraph changes
* Fix non-gc build
* Mark `AutoAssertNoGc` methods inline
* review feedback
* Temporarily remove externref support from the C API
Until we can add proper GC rooting.
* Remove doxygen reference to temp deleted function
* Remove need to `allow(private_interfaces)`
* Fix call benchmark compilation
* Add commentary on advantages/disadvantages of the pooling allocator
Inspired by discussion on #8034
* Add `memory_init_cow` commentary on disadvantages
* Fix doc link
This commit fixes an issue when inspecting the types of a component
where if a resource type wasn't substituted through an import it would
end up panicking. The fix here is to add a third kind of resource type
which represents an uninstantiated component as opposed to an
instantiated component or host type.
Closes#8003
This commit is an implementation of #7860 for Wasmtime where
`wasmtime::component::Linker` is now "semver aware". This means that it
assumes that hosts are always managing WIT interfaces in a
semver-aare fashion meaning that semver-compatible upgrade exclusively
adds functionality. This neatly fits into the idea of subtyping at the
instance-level where if a binary built against 0.2.0 only requests a
subset of functionality from a runtime that provides 0.2.1, that should
work just fine.
Specifically what this change does is:
* For all names inserted into a `Linker` there might also be a "semver
compatible name" which is registered as well. For example `..@1.0.0`
is also registered as `..@1`.
* Semver-compatible names are only provided for versions without a
prerelease and with either a nonzero major or minor version number.
* When looking up an item in the linker if no exact match is found then
if a semver-compatible-name is available for the lookup key then
that's consulted as well.
This semantically means that if a components imports WASI 0.2.0 then a
runtime which only provides WASI 0.2.1 will be able to instantiate the
component. Furthermore if a component imports WASI 0.2.1 but only
imports the subset of WASI that was available in 0.2.0 then it will be
instantiable in a runtime that only supports 0.2.0.
This implementation is intended to be a crucial part of the evolution to
WASI to make it more seamless to upgrade WASI from both a host and guest
perspective. This no longer requires everyone to upgrade to the same
version all at the same time but instead decouples the upgrade
schedules.
Closes#7860
* Wasmtime: Add a `gc` cargo feature
This controls whether support for `ExternRef` and its associated deferred,
reference-counting garbage collector is enabled at compile time or not. It will
also be used for similarly for Wasmtime's full Wasm GC support as that gets
added.
* Add CI for `gc` Cargo feature
* Cut down on the number of `#[cfg(feature = "gc")]`s outside the implementation of `[VM]ExternRef`
* Fix wasmparser reference types configuration with GC disabled/enabled
* More config fix
* doc cfg
* Make the dummy `VMExternRefActivationsTable` inhabited
* Fix winch tests
* final review bits
* Enable wasmtime's gc cargo feature for the C API
* Enable wasmtime's gc cargo feature from wasmtime-cli-flags
* enable gc cargo feature in a couple other crates
First of all, it is just a nice separation of concerns.
Second of all, as I design the GC rooting APIs for Wasmtime's upcoming Wasm GC
support, I want this same thing and I'd rather not open code it multiple times.
* Enable compiling the Wasmtime CLI to Wasm
While not the most useful thing to do in the world it's kind of neat to
play around with. This builds on the previous work to exclude the
runtime from the `wasmtime` crate so it's now possible to compile the
Wasmtime CLI's `compile` command, and only the `compile` command, to
wasm itself. This means you can run Wasmtime in Wasmtime!
* Fix warning on wasm
* Fix some feature combos
* Update some CI dependencies
* Update to the latest nightly toolchain
* Update mdbook
* Update QEMU for cross-compiled testing
* Update `cargo nextest` for usage with MIRI
prtest:full
* Remove lots of unnecessary imports
* Downgrade qemu as 8.2.1 seems to segfault
* Remove more imports
* Remove unused winch trait method
* Fix warnings about unused trait methods
* More unused imports
* More unused imports
This commit updates the allocation scheme for resources in the component
model to start at 1 instead of 0 when communicating with components.
This is an implementation of WebAssembly/component-model#284.
While this broke a number of tests we have this shouldn't actually break
any components in practice. The broken tests were all overly-precise in
their assertions and error messages and this shouldn't idiomatically
come up in any guest language, so this should not be a practically
breaking change.
This change additionally places an upper limit on the maximum
allocatable index at `1 << 30` which is also specified in the above PR.
By moving the registration count into the `Arc`, that is pulling the `Arc`
outwards from containing just the `WasmFuncType` to the registration count as
well, and turning it into an atomic, we can manipulate the registration count
without a write lock. Once that is done, we have the following:
* `RegisteredType::root` only needs a read lock, not a write lock.
* `RegisteredType::clone`, which used to need a write lock, doesn't need any
locking anymore.
* `RegisteredType::drop` doesn't need any locking most of the time. The
exception is when this is this drop that moves the refcount to zero, in which
case grabbing a write lock is still necessary to remove the type from the
registry.
This avoids locking the type registry to look up function types by
`VMSharedTypeIndex` in calls to `realloc`.
This gives a ~13% speedup to `wasmtime serve`'s requests per second for me
locally.
Alex Crichton
yowl
Saúl Cabrera
Brian
Nick Fitzgerald
Jamey Sharp
Wang-Zhongqi
Ryan Levick
Yaodong Yang
Trevor Elliott
Peter Huene