* Temporarily remove support for interface types
This commit temporarily removes support for interface types from the
`wasmtime` CLI and removes the `wasmtime-interface-types` crate. An
error is now printed for any input wasm modules that have wasm interface
types sections to indicate that support has been removed and references
to two issues are printed as well:
* #677 - tracking work for re-adding interface types support
* #1271 - rationale for removal and links to other discussions
Closes#1271
* Update the python extension
* Move all examples to a top-level directory
This commit moves all API examples (Rust and C) to a top-level
`examples` directory. This is intended to make it more discoverable and
conventional as to where examples are located. Additionally all examples
are now available in both Rust and C to see how to execute the example
in the language you're familiar with. The intention is that as more
languages are supported we'd add more languages as examples here too.
Each example is also accompanied by either a `*.wat` file which is
parsed as input, or a Rust project in a `wasm` folder which is compiled
as input.
A simple driver crate was also added to `crates/misc` which executes all
the examples on CI, ensuring the C and Rust examples all execute
successfully.
* Move filetime module to yanix
I've noticed that we could replace every occurrence of `crate::Result`
in `filetime` mods with `io::Result`, so I thought why not move it
to `yanix` and get rid off a lot of unnecessary code duplication
within `wasi-common`. Now, ideally I'd have our `filetime` modifications
backported to Alex's [`filetime`] crate, but one step at a time
(apologies Alex, I was meant to backport this ages ago, just didn't
find the time yet... :-().
Anyway, this commit does just that; i.e., moves the `filetime` modules
into `yanix` which seems a better fit for this type of code.
[`filetime`]: https://github.com/alexcrichton/filetime
There is one caveat here. On Emscripten, converting between `filetime::Filetime`
and `libc::timespec` appears to be lossy, at least as far as the
types are concerned. Now, `filetime::Filetime`'s seconds field is
`i64` while nanoseconds field is `u32`, while Emscripten's
`libc::timespec` requires both to be `i32` width. This might actually
not be a problem since I don't think it's possible to fill `filetime::Filetime`
struct with values of width wider than `i32` since Emscripten is 32bit
but just to be on the safe side, we do a `TryInto` conversion, log
the error (if any), and return `libc::EOVERFLOW`.
* Run cargo fmt
* Use i64::from instead of as cast
* Yanix now returns io::Error
This commit may seem somewhat controversial at first, but hear me
out first. Currently, Yanix would return a custom error that's a
wrapper around three other error types returned by various entities
inside Rust's `libstd`. In particular, Yanix's error type would wrap
`io::Error`, `num::TryFromIntError` and `ffi::NulError`. It turns
out that there is a natural conversion between the first and the last
and provided by the standard library, i.e., `From<ffi::NulError> for io::Error`
is provided. So at the surface it may seem that only the first two
wrapped error types are worth keeping.
Digging a little bit deeper into `libstd`, `num::TryFromIntError`
is essentially speaking only a marker that the integral conversion
went wrong. The struct implementing this error stores a unit type,
and nothing more. It therefore seems like a waste to wrap this
particular error when we could unify everything under `io::Error`.
And so, whenever we perform an int conversion, I suggest we simply
remap the error to `io::Error::from_raw_os_error(libc::EOVERFLOW)`
since this carries a comparable amount of information.
As a result of completely discarding `yanix::Error` custom error type,
we are invariably simplifying `yanix` itself, but also allowing
`wasi-common` to simplify in several places as well.
* Adapt wasi-common to changes in yanix
* Add Cargo.lock
* Unwrap try_into's where possible
* Remove unnecessary type annotation
* Add a wasmtime-specific `wasmtime_wat2wasm` C API
This commit implements a wasmtime-specific C API for converting the text
format to the binary format. An upstream spec issue exists for adding
this to the C API, but in the meantime we can experiment with our own
version of this API and use it in the C# extension, for example!
Closes#1000
* Reorder arguments
* Use wasm_byte_vec_t for input `*.wat`
* Mark wat input as const
* Return an error message and use `fixed`
* Actually include the error message
* Use `fixed` in `Module.cs` as well
* Improve robustness of cache loading/storing
Today wasmtime incorrectly loads compiled compiled modules from the
global cache when toggling settings such as optimizations. For example
if you execute `wasmtime foo.wasm` that will cache globally an
unoptimized version of the wasm module. If you then execute `wasmtime -O
foo.wasm` it would then reload the unoptimized version from cache, not
realizing the compilation settings were different, and use that instead.
This can lead to very surprising behavior naturally!
This commit updates how the cache is managed in an attempt to make it
much more robust against these sorts of issues. This takes a leaf out of
rustc's playbook and models the cache with a function that looks like:
fn load<T: Hash>(
&self,
data: T,
compute: fn(T) -> CacheEntry,
) -> CacheEntry;
The goal here is that it guarantees that all the `data` necessary to
`compute` the result of the cache entry is hashable and stored into the
hash key entry. This was previously open-coded and manually managed
where items were hashed explicitly, but this construction guarantees
that everything reasonable `compute` could use to compile the module is
stored in `data`, which is itself hashable.
This refactoring then resulted in a few workarounds and a few fixes,
including the original issue:
* The `Module` type was split into `Module` and `ModuleLocal` where only
the latter is hashed. The previous hash function for a `Module` left
out items like the `start_func` and didn't hash items like the imports
of the module. Omitting the `start_func` was fine since compilation
didn't actually use it, but omitting imports seemed uncomfortable
because while compilation didn't use the import values it did use the
*number* of imports, which seems like it should then be put into the
cache key. The `ModuleLocal` type now derives `Hash` to guarantee that
all of its contents affect the hash key.
* The `ModuleTranslationState` from `cranelift-wasm` doesn't implement
`Hash` which means that we have a manual wrapper to work around that.
This will be fixed with an upstream implementation, since this state
affects the generated wasm code. Currently this is just a map of
signatures, which is present in `Module` anyway, so we should be good
for the time being.
* Hashing `dyn TargetIsa` was also added, where previously it was not
fully hashed. Previously only the target name was used as part of the
cache key, but crucially the flags of compilation were omitted (for
example the optimization flags). Unfortunately the trait object itself
is not hashable so we still have to manually write a wrapper to hash
it, but we likely want to add upstream some utilities to hash isa
objects into cranelift itself. For now though we can continue to add
hashed fields as necessary.
Overall the goal here was to use the compiler to expose what we're not
hashing, and then make sure we organize data and write the right code to
ensure everything is hashed, and nothing more.
* Update crates/environ/src/module.rs
Co-Authored-By: Peter Huene <peterhuene@protonmail.com>
* Fix lightbeam
* Fix compilation of tests
* Update the expected structure of the cache
* Revert "Update the expected structure of the cache"
This reverts commit 2b53fee426.
* Separate the cache dir a bit
* Add a test the cache is busted with opt levels
* rustfmt
Co-authored-by: Peter Huene <peterhuene@protonmail.com>
This commit implements an initial WASI C API that can be used to instantiate
and configure a WASI instance from C.
This also implements a `WasiBuilder` for the C# API enabling .NET hosts to bind
to Wasmtime's WASI implementation.
* Cargo.lock: Update, to no longer use multiple versions of autocfg
* Update wasmtime-debug and wasmtime-profiling to current gimli 0.20.0
This also eliminates duplicate versions of gimli and arrayvec, and
eliminates the nodrop dependency entirely.
* Update wasmtime-profiling to goblin 0.1.3 and object 0.17.0
This also eliminates two duplicate versions of goblin, and duplicate
versions of proc-macro2, quote, syn, scroll_derive, and unicode-xid.
* Update wasmtime-profiling to current scroll 0.10.1
This eliminates duplicate versions of scroll.
* Update wasmtime-profiling to current target-lexicon 0.10.0
This eliminates duplicate versions of target-lexicon.
* Update wasmtime-interface-types to current walrus and wasm-webidl-bindings
This also eliminates the oldest of the three duplicate versions of
wasmparser.
* Update wasmtime-wast to current wast 8.0.0
This eliminates one of the duplicate versions of wast.
Patch adds support for the perf jitdump file specification.
With this patch it should be possible to see profile data for code
generated and maped at runtime. Specifically the patch adds support
for the JIT_CODE_LOAD and the JIT_DEBUG_INFO record as described in
the specification. Dumping jitfiles is enabled with the --jitdump
flag. When the -g flag is also used there is an attempt to dump file
and line number information where this option would be most useful
when the WASM file already includes DWARF debug information.
The generation of the jitdump files has been tested on only a few wasm
files. This patch is expected to be useful/serviceable where currently
there is no means for jit profiling, but future patches may benefit
line mapping and add support for additional jitdump record types.
Usage Example:
Record
sudo perf record -k 1 -e instructions:u target/debug/wasmtime -g
--jitdump test.wasm
Combine
sudo perf inject -v -j -i perf.data -o perf.jit.data
Report
sudo perf report -i perf.jit.data -F+period,srcline
* witx tagged unions: updates to wig to use new semantics
* wig: emit a `#variant: ()` union variant for empty variants
* wasi-common: translate to use tagged unions
* update to flattened layout of event struct
* wig: generate layout tests, and delete bindgen ones
the bindgen tests became out-of-date with the latest changes to the
representation of unions, and the re-jiggering of various struct
definitions that went along with it.
* wasi: point at master with tagged union PR merged
* fix event struct repr on windows
* Update cranelift to 0.58.0
* Update `wasmprinter` dep to require 0.2.1
We already had it in the lock file, but this ensures we won't ever go back down.
* Ensure that our error messages match `assert_invalid`'s
The bulk of this work was done in
https://github.com/bytecodealliance/wasmparser/pull/186 but now we can test it
at the `wasmtime` level as well.
Fixes#492
* Stop feeling guilty about not matching `assert_malformed` messages
Remove the "TODO" and stop printing warning messages. These would just be busy
work to implement, and getting all the messages the exact same relies on using
the same structure as the spec interpreter's parser, which means that where you
have a helper function and they don't, then things go wrong, and vice versa. Not
worth it.
Fixes#492
* Enable (but ignore) the reference-types proposal tests
* Match test suite directly, instead of roundabout starts/endswith
* Enable (but ignore) bulk memory operations proposal test suite
* Remove the `jit_function_registry` global state
This commit removes on the final pieces of global state in wasmtime
today, the `jit_function_registry` module. The purpose of this module is
to help translate a native backtrace with native program counters into a
wasm backtrace with module names, function names, and wasm module
indices. To that end this module retained a global map of function
ranges to this metadata information for each compiled function.
It turns out that we already had a `NAMES` global in the `wasmtime`
crate for symbolicating backtrace addresses, so this commit moves that
global into its own file and restructures the internals to account for
program counter ranges as well. The general set of changes here are:
* Remove `jit_function_registry`
* Remove `NAMES`
* Create a new `frame_info` module which has a singleton global
registering compiled module's frame information.
* Update traps to use the `frame_info` module to symbolicate pcs,
directly extracting a `FrameInfo` from the module.
* Register and unregister information on a module level instead of on a
per-function level (at least in terms of locking granluarity).
This commit leaves the new `FRAME_INFO` global variable as the only
remaining "critical" global variable in `wasmtime`, which only exists
due to the API of `Trap` where it doesn't take in any extra context when
capturing a stack trace through which we could hang off frame
information. I'm thinking though that this is ok, and we can always
tweak the API of `Trap` in the future if necessary if we truly need to
accomodate this.
* Remove a lazy_static dep
* Add some comments and restructure
* Remove all global state from the caching system
This commit is a continuation of an effort to remove usages of
`lazy_static!` and similar global state macros which can otherwise be
accomodated with passing objects around. Previously there was a global
cache system initialized per-process, but it was initialized in a bit of
a roundabout way and wasn't actually reachable from the `wasmtime` crate
itself. The changes here remove all global state, refactor many of the
internals in the cache system, and makes configuration possible through
the `wasmtime` crate.
Specifically some changes here are:
* Usage of `lazy_static!` and many `static` items in the cache module
have all been removed.
* Global `cache_config()`, `worker()`, and `init()` functions have all
been removed. Instead a `CacheConfig` is a "root object" which
internally owns its worker and passing around the `CacheConfig` is
required for cache usage.
* The `wasmtime::Config` structure has grown options to load and parse
cache files at runtime. Currently only loading files is supported,
although we can likely eventually support programmatically configuring
APIs as well.
* Usage of the `spin` crate has been removed and the dependency is removed.
* The internal `errors` field of `CacheConfig` is removed, instead
changing all relevant methods to return a `Result<()>` instead of
storing errors internally.
* Tests have all been updated with the new interfaces and APIs.
Functionally no real change is intended here. Usage of the `wasmtime`
CLI, for example, should still enable the cache by default.
* Fix lightbeam compilation
* Reimplement `wasmtime-wasi` on top of `wasmtime`
This commit reimplements the `wasmtime-wasi` crate on top of the
`wasmtime` API crate, instead of being placed on top of the `wasmtime-*`
family of internal crates. The purpose here is to continue to exercise
the API as well as avoid usage of internals wherever possible and
instead use the safe API as much as possible.
The `wasmtime-wasi` crate's API has been updated as part of this PR as
well. The general outline of it is now:
* Each module snapshot has a `WasiCtxBuilder`, `WasiCtx`, and `Wasi`
type.
* The `WasiCtx*` types are reexported from `wasi-common`.
* The `Wasi` type is synthesized by the `wig` crate's procedural macro
* The `Wasi` type exposes one constructor which takes a `Store` and a
`WasiCtx`, and produces a `Wasi`
* Each `Wasi` struct fields for all the exported functions in that wasi
module. They're all public an they all have type `wasmtime::Func`
* The `Wasi` type has a `get_export` method to fetch an struct field by
name.
The intention here is that we can continue to make progress on #727 by
integrating WASI construction into the `Instance::new` experience, but
it requires everything to be part of the same system!
The main oddity required by the `wasmtime-wasi` crate is that it needs
access to the caller's `memory` export, if any. This is currently done
with a bit of a hack and is expected to go away once interface types are
more fully baked in.
* Remove now no-longer-necessary APIs from `wasmtime`
* rustfmt
* Rename to from_abi
* Improve handling of strings for backtraces
Largely avoid storing strings at all in the `wasmtime-*` internal
crates, and instead only store strings in a separate global cache
specific to the `wasmtime` crate itself. This global cache is inserted
and removed from dynamically as modules are created and deallocated, and
the global cache is consulted whenever a `Trap` is created to
symbolicate any wasm frames.
This also avoids the need to thread `module_name` through the jit crates
and back, and additionally removes the need for `ModuleSyncString`.
* Run rustfmt
This commit deletes the old C implementation of the original
`wasi_unstable` module, instead only leaving around our single
`wasmtime-wasi` crate as the implementation for both
`wasi_snapshot_preview1` and `wasi_unstable`.
This hasn't been discussed (AFAIK) up until now, so this is also a
proposal! Some thoughts in favor of this deletion I would have are:
* This has been off-by-default for ages
* We don't build or test any of this on CI
* Published binaries with `wasmtime` do not have this possibility
enabled
* Future refactorings to the `wasmtime-wasi` crate will either need to
work around how the C implementation is different or bring it up to
speed.
This is motivated by the last bullet point where I was working on
getting `wasmtime-wasi` working purely as an implementation detail on
top of the `wasmtime` crate itself, but quickly ran into a case where
the CLI would need to multiplex all sorts of wasi implementations. In
any case I'm curious what others think, is this too soon? Is there
something remaining blocking this? (etc)
* Auto-generate shims for old `wasi_unstable` module
This commit is effectively just doing what #707 already did, but
applying it to the `snapshot_0` module as well. The end result is the
same, where we cut down on all the boilerplate in `snapshot_0` and bring
it in line with the main `wasi_snapshot_preview1` implementation. The
goal here is to make it easier to change the two in tandem since they're
both doing the same thing.
* Migrate `wasi_common::hostcalls` to a macro
This commit migrates the `hostcalls` module to being auto-generated by a
macro rather than duplicating a handwritten signature for each wasi
syscall.
* Auto-generate snapshot_0's `hostcalls` module
Similar to the previous commit, but for `snapshot_0`
* Delete the `wasi-common-cbindgen` crate
This is no longer needed with the hostcalls macro now, we can easily
fold the definition of the cbindgen macro into the same crate.
* Rustfmt
* Fix windows build errors
* Rustfmt
* Remove now no-longer-necessary code
* rustfmt
* Replace the global-exports mechanism with a caller-vmctx mechanism.
This eliminates the global exports mechanism, and instead adds a
caller-vmctx argument to wasm functions so that WASI can obtain the
memory and other things from the caller rather than looking them up in a
global registry.
This replaces #390.
* Fixup some merge conflicts
* Rustfmt
* Ensure VMContext is aligned to 16 bytes
With the removal of `global_exports` it "just so happens" that this
isn't happening naturally any more.
* Fixup some bugs with double vmctx in wasmtime crate
* Trampoline stub needed adjusting
* Use pointer type instead of always using I64 for caller vmctx
* Don't store `ir::Signature` in `Func` since we don't know the pointer
size at creation time.
* Skip the first 2 arguments in IR signatures since that's the two vmctx
parameters.
* Update cranelift to 0.56.0
* Handle more merge conflicts
* Rustfmt
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
* Move compilation into Module from Instance.
* Fix fuzzing
* Use wasmtime::Module in fuzzing crates
Instead of wasmtime_jit.
* Compile eagerly.
* Review fixes.
* Always use the saved name.
* Preserve the former behavior for fuzzing oracle
* Preserve full native stack traces in errors
This commit builds on #759 by performing a few refactorings:
* The `backtrace` crate is updated to 0.3.42 which incorporates the
Windows-specific stack-walking code, so that's no longer needed.
* A full `backtrace::Backtrace` type is held in a trap at all times.
* The trap structures in the `wasmtime-*` internal crates were
refactored a bit to preserve more information and deal with raw
values rather than converting between various types and strings.
* The `wasmtime::Trap` type has been updated with these various changes.
Eventually I think we'll want to likely render full stack traces (and/or
partial wasm ones) into error messages, but for now that's left as-is
and we can always improve it later. I suspect the most relevant thing we
need to do is to implement function name symbolication for wasm
functions first, and then afterwards we can incorporate native function
names!
* Fix some test suite assertions
* Move the C API to a separate crate
This commit moves the C API from `crates/api/src/wasm.rs` to
`crates/capi/src/lib.rs` to be located in a separate crate. There's a
number of reasons for this:
* When a Rust program depends on the `wasmtime` crate, there's no need
to compile in the C API.
* This should improve compile times of the `wasmtime` crate since it's
not producing artifacts which aren't always used.
* The development of the C API can be guaranteed to only use the public
API of the `wasmtime` crate itself.
Some CI pieces are tweaked and this overall shouldn't have much impact
on users, it's intended that it's a cleanup/speedup for developers!
* Disable rustdoc/tests for capi
* Review feedback
* Add back in accidentally deleted comment
* More renamings
* Try to fix dotnet build
Alex Crichton
Jakub Konka
Jakub Konka
Nick Fitzgerald
Peter Huene
Josh Triplett
Johnnie Birch
Yury Delendik
Pat Hickey
Dan Gohman
Jonathan Foote
Sergei Pepyakin
Marcin Mielniczuk
Andrew Brown