This patch updates or removes all references to the Cranelift repository. It affects links in README documents, issues that were transferred to the Wasmtime repository, CI badges, and a small bunch of sundry items.
* wiggle-runtime: add as_raw method for [T]
* add trivial borrow checker back in
* integrate runtime borrow checker with as_raw methods
* handle pointer arith overflow correctly in as_raw, create PtrOverflow error
* runtime: add validation back to GuestType
* generate: impl validate for enums, flags, handles, ints
* oops! make validate its own method on trait GuestTypeTransparent
* fix transparent impls for enum, flag, handle, int
* some structs are transparent. fix tests.
* tests: define byte_slice_strat and friends
* wiggle-tests: i believe my allocator is working now
* some type juggling around memset for ease of use
* make GuestTypeTransparent an unsafe trait
* delete redundant validation of pointer align
* fix doc
* wiggle_test: aha, you cant use sets to track memory areas
* add multi-string test
which exercises the runtime borrow checker against
HostMemory::byte_slice_strat
* oops left debug panic in
* remove redundant (& incorrect, since unchecked) length calc
* redesign validate again, and actually hook to as_raw
* makr all validate impls as inline
this should hopefully allow as_raw's check loop to be unrolled to a
no-op in most cases!
* code review fixes
This commit rewrites the runtime crate to provide safety in the face
of recursive calls to the guest. The basic principle is that
`GuestMemory` is now a trait which dynamically returns the
pointer/length pair. This also has an implicit contract (hence the
`unsafe` trait) that the pointer/length pair point to a valid list of
bytes in host memory "until something is reentrant".
After this changes the various suite of `Guest*` types were rewritten.
`GuestRef` and `GuestRefMut` were both removed since they cannot safely
exist. The `GuestPtrMut` type was removed for simplicity, and the final
`GuestPtr` type subsumes `GuestString` and `GuestArray`. This means
that there's only one guest pointer type, `GuestPtr<'a, T>`, where `'a`
is the borrow into host memory, basically borrowing the `GuestMemory`
trait object itself.
Some core utilities are exposed on `GuestPtr`, but they're all 100%
safe. Unsafety is now entirely contained within a few small locations:
* Implementations of the `GuestType` for primitive types (e.g. `i8`,
`u8`, etc) use `unsafe` to read/write memory. The `unsafe` trait of
`GuestMemory` though should prove that they're safe.
* `GuestPtr<'_, str>` has a method which validates utf-8 contents, and
this requires `unsafe` internally to read all the bytes. This is
guaranteed to be safe however given the contract of `GuestMemory`.
And that's it! Everything else is a bunch of safe combinators all built
up on the various utilities provided by `GuestPtr`. The general idioms
are roughly the same as before, with various tweaks here and there. A
summary of expected idioms are:
* For small values you'd `.read()` or `.write()` very quickly. You'd
pass around the type itself.
* For strings, you'd pass `GuestPtr<'_, str>` down to the point where
it's actually consumed. At that moment you'd either decide to copy it
out (a safe operation) or you'd get a raw view to the string (an
unsafe operation) and assert that you won't call back into wasm while
you're holding that pointer.
* Arrays are similar to strings, passing around `GuestPtr<'_, [T]>`.
Arrays also have a `iter()` method which yields an iterator of
`GuestPtr<'_, T>` for convenience.
Overall there's still a lot of missing documentation on the runtime
crate specifically around the safety of the `GuestMemory` trait as well
as how the utilities/methods are expected to be used. Additionally
there's utilities which aren't currently implemented which would be easy
to implement. For example there's no method to copy out a string or a
slice, although that would be pretty easy to add.
In any case I'm curious to get feedback on this approach and see what
y'all think!
This commit puts context object, i.e., the implementor of the
WASI snapshot, behind a reference `&self` rather than a mutable
reference `&mut self`. As suggested by @alexcrichton, this gives
the implementor the possibility to determine how it handles its
interior mutability.
* Add WASI spec (minus unions)
* Fill in all WASI shims
* Clean up derives and fix noncopy struct write method
This commit does three things:
* it uses the full, current snapshot1 WASI spec as a compilation test
* it fixes noncopy struct write method (which was incorrectly resolved
in certain cases to the inherent method of the `GuestPtrMut` rather
than the interface method `GuestType::write`
* it cleans up derives for structs and unions which should not auto-derive
`PartialEq`, `Eq`, or `Hash` since their members are not guaranteed to
be compatible
this allows us to reuse the code in wiggle-generate elsewhere, because
a proc-macro=true lib can only export a #[proc_macro] and not ordinary
functions.
In lucet, I will depend on wiggle-generate to define a proc macro that
glues wiggle to the specifics of the runtime.
This commit refactors trait system for guest types. Namely, as
discussed offline on zulip, `GuestType` now includes `GuestTypeClone`,
whereas `GuestTypeCopy` has been renamed to `GuestTypeTransparent`.
Essentially, table and memory out of bounds errors are no longer link errors,
but traps after linking. This means that the partail writes / inits are visible.
This adds support for the `table.copy` instruction from the bulk memory
proposal. It also supports multiple tables, which were introduced by the
reference types proposal.
Part of #928
these names were artifacts of some early confusion / bad design i made
in the traits. read and write are much simpler names!
also, change a ptr_mut to ptr where we just read the contents in the
argument marshalling for structs. this has no effect, but it is more
correct.
* Allow returning structs if copy
This commit does three things:
1. enables marshalling of structs as return args from interface funcs
but so far *only* for the case when the struct itself is copy
2. puts bits that use `std::convert::TryInto` in a local scope to avoid
multiple reimports
3. for added clarity, we now print for which `tref` type the marshalling
of results is unimplemented
The first case (1.) is required to make `fd_fdstat_get` WASI interface
func work which returns `Fdstat` struct (which is copy). The second
case (2.) caused me some grief somewhere along the lines when I was
playing with snapshot1. Putting the code that requires it inside a local
scope fixed all the issues
* Add proptests for returing struct if copyable
* Use write_ptr_to_guest to marshal value to guest
* Successfully return non-copy struct
* Optimize generated code via the CLI by default
This commit updates the behavior of the CLI and adds a new flag. It
first enables the `--optimize` flag by default, ensuring that usage of
the `wasmtime` CLI will enable cranelift optimizations by default. Next
it also adds a `--opt-level` flag which is similar to Rust's
`-Copt-level` where it takes a string argument of how to optimize. This
is updates to support 0/1/2/s, where 1 is currently the same as 2 but
added for consistency with other compilers. The default setting is
`--opt-level=2`.
When the `-O` flag is not passed the `--opt-level` flag is used,
otherwise `-O` takes precedent in the sense that it implies
`--opt-level=2` which is the highest optimization level. The thinking is
that these flags will in general select the highest optimization level
specified as the final optimization level.
* Add inline docs
* fix a test
* Implement fmt::Display for enums
`wasi_common` relies on `strerror` to nicely format error messages.
`strerror` is autoimplemented in `wig`. I thought it might be useful
to provide a Rust-idiomatic alternative which boils down to autoimplementing
`fmt::Display` for all enums.
* Implement fmt::Display for flags
* Implement fmt::Display for ints
* atoms in one test unit
* factor out pointers test
* factor structs into separate test unit
* factor out arrays, flags
* finally, separate into strings and ints
* Draft out IntDatatype in wiggle-generate
This commit drafts out basic layout for `IntDatatype` structure in
`wiggle`. As it currently stands, an `Int` type is represented as
a one-element tuple struct much like `FlagDatatype`, however, with
this difference that we do not perform any checks on the input
underlying representation since any value for the prescribed type
is legal.
* Finish drafting IntDatatype support in wiggle
This commit adds necessary marshal stubs to properly pass `IntDatatype`
in and out of interface functions. It also adds a basic proptest.
* Add basic GuestString support to wiggle
This commit adds basic `GuestString` support to `wiggle`. `GuestString`
is a wrapper around `GuestArray<'_, u8>` array type which itself can
be made into either an owned (cloned) Rust `String` or borrowed as
a reference `&str`. In both cases, `GuestString` ensures that the
underlying bytes are valid Unicode code units, throwing a `InvalidUtf8`
error if not.
This commit adds support *only* for passing in strings as arguments
in WASI. Marshalling of the return arg has not yet been implemented.
I'm not even sure it's possible without multi-value return args
feature of Wasm. It's not a major setback especially since the WASI
spec (and this includes even the `ephemeral` snapshot) doesn't
return strings anywhere. They are only ever passed in as arguments
to interface functions.
It should be noted that error returned in case of invalid UTF-8
requires a lot more love as it doesn't include anything besides
flagging an event that the string contained an invalid Unicode code unit.
* Borrow all of string's memory including nul-byte
Borrow all of string's underlying memory including the nul-byte.
This perhaps might not have a tremendous impact on anything, but
since the nul-byte is technically part of the WASI string, we should
include it in the borrow as well.
* Fill in wiggle-generate blanks for strings
* Print to screen passed string in proptest
* Strings are PointerLengthPairs!
* Fix generation of strings in compound types
* Update test with simple string strategy
* Generate better test strings
* Finalise proptest for strings
* Fix formatting
* Update crates/runtime/src/memory/string.rs
Removes unnecessary comment in code
* Apply Pat's suggestion to wrap Utf8Error as error
* merge GuestTypePtr and GuestTypeClone into a single GuestTypeClone<'a> trait
* GuestArray can derive Clone (but not impl GuestTypeClone)
* fix array tests
* Fix GuestTypeClone for flags
Co-authored-by: Jakub Konka <kubkon@jakubkonka.com>
Patch adds support for the perf jitdump file specification.
With this patch it should be possible to see profile data for code
generated and maped at runtime. Specifically the patch adds support
for the JIT_CODE_LOAD and the JIT_DEBUG_INFO record as described in
the specification. Dumping jitfiles is enabled with the --jitdump
flag. When the -g flag is also used there is an attempt to dump file
and line number information where this option would be most useful
when the WASM file already includes DWARF debug information.
The generation of the jitdump files has been tested on only a few wasm
files. This patch is expected to be useful/serviceable where currently
there is no means for jit profiling, but future patches may benefit
line mapping and add support for additional jitdump record types.
Usage Example:
Record
sudo perf record -k 1 -e instructions:u target/debug/wasmtime -g
--jitdump test.wasm
Combine
sudo perf inject -v -j -i perf.data -o perf.jit.data
Report
sudo perf report -i perf.jit.data -F+period,srcline
* Adds support for flags datatype
This commit adds support for `FlagsDatatype`. In WASI, flags are
represented as bitfields/bitflags, therefore, it is important
for the type to have bitwise operations implemented, and some
way of checking whether the current set of flags contains some
other set (i.e., they intersect). Thus, this commit automatically
derives `BitAnd`, etc. for the derived flags datatype. It also
automatically provides an `ALL_FLAGS` value which corresponds to
a bitwise-or of all flag values present and is provided for
convenience.
* Simplify read_from_guest
In preparation for landing `string` support in `wiggle`, I've
revisited `GuestArray` and the deref mechanics, and decided to
scrap the current approach in favour of the previous. Namely, for
`T: GuestTypeCopy` we provide `GuestArray::as_ref` which then can
be derefed directly to `&[T]` since the guest and host types have
matching representation, and for other types (thinking here of
our infamous arrays of pointers), I've instead added a way to
iterate over the pointers. Then, it is up to the `wiggle`'s client
to know whether they can deref whatever the pointer stored in array
is pointing at.
* Add array generation to wiggle-generate crate
This commit:
* adds array generation to `wiggle-generate` crate which implies
that we can now generate arrays from `witx` files
* introduces two test interface functions `foo::reduce_excuses` and
`foo::populate_excuses`, and adds matching prop-tests
* adds an out-of-boundary check to `HostMemory::mem_area_strat` since
now, given we're generating arrays for testing with an arbitrary
but bounded number of elements, it is possible to violate the boundary
* refactors `Region::extend` to a new signature `extend(times: u32)`
which multiplies the current pointer `len` by `times`
* fixes bug in `GuestArray::as_ref` and `GuestArrayMut::as_ref_mut` methods
where we were not validating the first element (we always started the
validation from the second element)
* Fix generation of arrays in witx
This commit fixes how `arrays` are auto-generated from `witx` files.
In particular, the changes include:
* Since by design every `array` in `witx` represents an immutable
slab of memory, we will only ever operate on `GuestArray` in which
case I've gone ahead and removed `GuestArrayMut` so as to unclutter
the code somewhat. If we find ourselves in need for it in the future,
I reckon it will be better to write it from scratch (since the codebase
will inevitably evolve by then considerably) rather than maintaining an
unused implementation.
* I've rewritten `GuestArrayRef<T>` to be a wrapper for `Vec<GuestRef<T>>`.
Also, `GuestArray::as_ref` now borrows each underlying "element" of the
array one-by-one rather than borrowing the entire chunk of memory at once.
This change is motivated by the inability to coerce type parameter `T` in
`GuestArray<T>` in more complicated cases such as arrays of guest pointers
`GuestPtr<T>` to `*const T` for reuse in `std::slice::from_raw_parts` call.
(In general, the size of Wasm32 pointer is 4 bytes, while
```
std::mem::size_of::<T>() == std::mem::size_of::<GuestPtr<S>>() == 16
```
which is problematic; i.e., I can't see how I could properly extract guest
pointers from slices of 4 bytes and at the same time not allocate.)
* I've augmented fuzz tests by (re-)defining two `array` types:
```
(typename $const_excuse_array (array (@witx const_pointer $excuse)))
(typename $excuse_array (array (@witx pointer $excuse)))
```
This should hopefully emulate and test the `iovec` and `ciovec` arrays
present in WASI spec.
* Refactor naming and crates info
This commit:
* changes workspace crates to have a `wiggle_` prefix in names
* rename `memory` module of `wiggle-memory` crate to `runtime`
* fixes authors of all crates
* Rename wiggle memory crate to runtime
* Update cranelift to 0.58.0
* Update `wasmprinter` dep to require 0.2.1
We already had it in the lock file, but this ensures we won't ever go back down.
* Ensure that our error messages match `assert_invalid`'s
The bulk of this work was done in
https://github.com/bytecodealliance/wasmparser/pull/186 but now we can test it
at the `wasmtime` level as well.
Fixes#492
* Stop feeling guilty about not matching `assert_malformed` messages
Remove the "TODO" and stop printing warning messages. These would just be busy
work to implement, and getting all the messages the exact same relies on using
the same structure as the spec interpreter's parser, which means that where you
have a helper function and they don't, then things go wrong, and vice versa. Not
worth it.
Fixes#492
* Enable (but ignore) the reference-types proposal tests
* Match test suite directly, instead of roundabout starts/endswith
* Enable (but ignore) bulk memory operations proposal test suite
* Remove all global state from the caching system
This commit is a continuation of an effort to remove usages of
`lazy_static!` and similar global state macros which can otherwise be
accomodated with passing objects around. Previously there was a global
cache system initialized per-process, but it was initialized in a bit of
a roundabout way and wasn't actually reachable from the `wasmtime` crate
itself. The changes here remove all global state, refactor many of the
internals in the cache system, and makes configuration possible through
the `wasmtime` crate.
Specifically some changes here are:
* Usage of `lazy_static!` and many `static` items in the cache module
have all been removed.
* Global `cache_config()`, `worker()`, and `init()` functions have all
been removed. Instead a `CacheConfig` is a "root object" which
internally owns its worker and passing around the `CacheConfig` is
required for cache usage.
* The `wasmtime::Config` structure has grown options to load and parse
cache files at runtime. Currently only loading files is supported,
although we can likely eventually support programmatically configuring
APIs as well.
* Usage of the `spin` crate has been removed and the dependency is removed.
* The internal `errors` field of `CacheConfig` is removed, instead
changing all relevant methods to return a `Result<()>` instead of
storing errors internally.
* Tests have all been updated with the new interfaces and APIs.
Functionally no real change is intended here. Usage of the `wasmtime`
CLI, for example, should still enable the cache by default.
* Fix lightbeam compilation
* Reimplement `wasmtime-wasi` on top of `wasmtime`
This commit reimplements the `wasmtime-wasi` crate on top of the
`wasmtime` API crate, instead of being placed on top of the `wasmtime-*`
family of internal crates. The purpose here is to continue to exercise
the API as well as avoid usage of internals wherever possible and
instead use the safe API as much as possible.
The `wasmtime-wasi` crate's API has been updated as part of this PR as
well. The general outline of it is now:
* Each module snapshot has a `WasiCtxBuilder`, `WasiCtx`, and `Wasi`
type.
* The `WasiCtx*` types are reexported from `wasi-common`.
* The `Wasi` type is synthesized by the `wig` crate's procedural macro
* The `Wasi` type exposes one constructor which takes a `Store` and a
`WasiCtx`, and produces a `Wasi`
* Each `Wasi` struct fields for all the exported functions in that wasi
module. They're all public an they all have type `wasmtime::Func`
* The `Wasi` type has a `get_export` method to fetch an struct field by
name.
The intention here is that we can continue to make progress on #727 by
integrating WASI construction into the `Instance::new` experience, but
it requires everything to be part of the same system!
The main oddity required by the `wasmtime-wasi` crate is that it needs
access to the caller's `memory` export, if any. This is currently done
with a bit of a hack and is expected to go away once interface types are
more fully baked in.
* Remove now no-longer-necessary APIs from `wasmtime`
* rustfmt
* Rename to from_abi
* generator: take an &mut GuestMemory
rather than pass the owned GuestMemory in, just give exclusive access
to it. Makes testing easier.
* tests: start transforming tests to check abi-level generated code as well
* finish lowering of test funcs
* tests: rename variables to more sensible names
* proptesting: reliably finds that we dont allow stuff to be right against end of memory!
* memory: fix off-by-one calc in GuestMemory::contains(&self, Region)
ty proptest!
also, refactored the Region::overlaps to be the same code but easier to
read.
* generator: better location information in GuestError
* testing: proptest generates memory areas, tests everything
* Add some (incomplete set) basic sanity end-to-end tests
This commit adds some (an incomplete set of) basic sanity end-to-end
tests. It uses `test.witx` to autogenerate types and module interface
functions (aka the syscalls), and tests their implementation. For
the host memory, it uses simplistic `&mut [u8]` where we have full
control of the addressing and contents.
* Add sanity test for baz interface func
This commit adds a sanity test for the `Foo::baz` interface func.
* Upcast start/len for Region to avoid overflow
* Reenable alignment checking for memory
* use an array to implement hostmemory
Co-authored-by: Pat Hickey <pat@moreproductive.org>
Pat Hickey
Till Schneidereit
Yury Delendik
Alex Crichton
Jakub Konka
Jakub Konka
Nick Fitzgerald
Peter Huene
Johnnie Birch