* Update wasmtime and wit-bindgen
This commit updates the `wasmtime` dependency and the
`wit-bindgen-guest-rust` dependencies to namely update the `wit-parser`
and component underlying implementations. This pulls in
bytecodealliance/wasm-tools#867 which is the implementation of `use` for
WIT. This does not currently break apart the one large `wasi.wit` file,
instead just gets tests working.
* Split apart `wasi.wit` into multiple `*.wit` files
This commit refactors the monolithic `wasi.wit` file into multiple
files. I've taken some liberties in naming here so suggestions are
definitely welcome! I've resolved some TODO annotations as well about
`use`-ing types between interfaces. There are two issues remaining,
however:
* The `wasi-command` world still hardcodes `u32` because `world` items
don't support `use` just yet. That isn't a hard limitation of WIT,
however, it's just a temporary limitation of the implementation.
* The `wasi-clocks` interface defines the `wasi-future` type. This is
due to a cyclic dependency where `wasi-future` is defined within
`wasi-poll` which depends on `wasi-clocks` for types. I've left this
to a future refactoring to probably have a `types` interface of some
form somewhere.
* Update patch for wasm-tools
* Switch to wasmtime upstream
* Update CI for build
* Remove patch overrides
* Update names of uploaded files
Add a `byte-array` proc-macro crate for converting strings into byte
arrays that don't use static initializers, and use it to implement
`eprintln`, an `unreachable` that prints the line number, and other
macros.
On Windows, there doesn't appear to be a way to sync a directory, to
ensure that the directory entry for a file is sync'd. So for now, just
silently succeed. I've opened WebAssembly/wasi-filesystem#79 to track
this at the spec level.
* Implement file append functionality.
- In the preview1-to-preview2 polyfill, using `append_via_stream`.
- In the host implementation, using a new system-interface `FileIoExt::append` function.
* Add a basic testcase.
After two separate rounds of memory corruption now happening it's
probably best to place some protections in place to try to detect when
this happens earlier rather than much later after the fact.
* Use `use` imports consistently.
* Rename `flags_from_descriptor_flags` to `descriptor_flags_from_flags`.
* Rename `black_box` to `obscure`.
* Make some comments be doc comments.
* Use a hyphen for compound adjectives.
* Delete an unused variable.
* Update the name of `change-file-permissions-at`.
* Use generated typedefs instead of hard-coding u64.
* Use core::hint::black_box now that it's stabilized.
* implement `wasi-filesystem::readdir` and related functions
This adds a `directory_list` test and provides the required host implementation.
I've also added a file length check to the `file_read` test, just to cover a bit
more of the API.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* fix memory corruption in `fd_readdir` polyfill
We were copying `name.len() * 256` bytes instead of just `name.len()` bytes,
which was overwriting other parts of `State` and causing untold havoc.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* check type of entry in `Table::delete`
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
Add a `result` return type to `command` so that it can indicate success
or failure.
The idea here is that this isn't a full `i32` return value because the
meaning of return values isn't portable across platforms. Also, Typed Main
is a better long-term answer for users that want rich error return
values from commands.
* Add pseudo-streams.
This add a pseudo-stream type to the wasi-poll interface, and adds ways
to obtain streams from command invocation and from files. In the future,
it can support sockets too. With this, `command` takes streams for
stdin/stdout, rather than filesystem descriptors.
Streams support reading and writing, as well as skipping,
repeated-element writing, and splicing from one stream to another. And
there are `subscribe-*` functions to produce pseudo-futures from
pseudo-streams, allowing them to be polled.
This makes the polyfill somewhat more complex, but this is largely due
to the polyfill being tied to the preview1 API.
This replaces the `seek` and `tell` functions, and implemented `fd_seek`
and `fd_tell` in terms of the polyfill's own position.
Also, add a dedicated stderr API for writing to stderr in a way that
tolerates strings that aren't necessarily expected to be newlines. And
add a way to test whether stderr is a terminal.
* Implement the host side of `poll_oneoff`.
This implements pseudo-futures and subscription functions, and adds
polling for streams.
* Implement clock subscriptions.
wasi.wit:
- Remove the "timers" API from wasi-clocks, as it's now redundant with
pseudo-future clock subscriptions.
- Remove `subscribe-wall-clock`. Wall-clock timeouts were implemented by
converting them to monotonic-clock timeouts anyway, so just make that
explicit in the WASI API, and teach the polyfill how to convert
wall-clock timeouts into monotonic-clock timeouts.
- Move `subscribe-monotonic-clock` out of wasi-clocks and into wasi-poll,
as it's closely tied to the pseudo-futures mechanism and the `poll-oneoff`
implementation.
- While here, fix `stream-read` and related functions to return an
end-of-stream/file indicator.
Code changes:
- `default_wall_clock()` and `default_monotonic_clock()` now always
create a new table entry, rather than holding a table index in the
`WasiCtx` which could potentially dangle.
- Add support for monotonic-clock poll subscriptions.
- Say "wall clock" instead of "system clock" when we have a choice.
* Remove the `OFlags::APPEND` flag, which is no longer used.
Per #31, we pass env vars and preopens via `command`, just like CLI arguments.
Later, we plan to add another interface (e.g. `cli-reactor`) for reactors
which need env vars and/or preopens, which will have a single function
`initialize` that takes the same two parameters (but not stdio or CLI arg
parameters).
This also removes unused parts of `wasi-common` (e.g. args, and env vars, and
preopen paths) which have been superceded by the `command` interface.
Our eventual goal is to provide a more explicit interface for environment
variables using WIT star imports such that the guest will import a zero-argument
function (or value, when that is supported) for each variable it needs, allowing
the host to statically verify that it can provide all those variables. However,
star imports are not yet supported in `wit-bindgen`, and once they are, we'll
likely still need a dynamic interface to support existing software.
Note that I've added a `file_read` test which does not actually do anything yet,
since not all the required host functions have been implemented. I plan to
address that soon.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
This required fleshing out the `wasi-clocks` host implementation a bit and
adding a `read_vectored_at` implementation for `ReadPipe`.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* Implement `fd_renumber`.
* Implement `Drop` for `Descriptor` to free backend resources.
Instead of trying to remember to call `close` on file descriptors when
they're replaced or removed, just have the `Drop` impl for `Descriptor`.
* Use the local `unwrap_result` instead of `.unwrap()`
This avoids static memory inits.
* Implement the `fd_readdir` function
This roughly matches the implementation in `wasi-common` today where it
uses the directory entry stream as a form of iterator and the `cookie`
represents the `n`th iteration. Not exactly efficient if the `buf`
provided to the hostcall is too small to hold many of the entries but
there's not a whole lot else that can be done at this time.
This also updates the WIT for `read-dir-entry` to return
`option<dir-entry>` instead of `dir-entry` to indicate EOF.
* Fix host compile
* Add a specific method to close a dir-entry-stream
* Add a cache to avoid quadratic dirent behavior
When a readdir call is truncated due to the output buffer not being
large enough save the final state of the readdir iterator into `State`
to possibly get reused on the next call to `readdir`. Some limits are
put in place such as:
* The next call to readdir must be for the same fd and the required
cookie.
* The dirent that didn't fit must have its full name fit within the path
cache.
* If `fd_close` is called after a readdir it clears the cache so a
future `fd_readdir` doesn't actually resume now-stale state.
There's a fair bit of trickiness here so I've attempted to structure
things as "simply" as possible to hopefully reduce the chance there's an
issue, but this is all untested so there's still likely an off-by-one or
similar bug.
* Fix prints in non-command builds
When the `command` entrypoint isn't invoked, such as for non-command
builds, then prints were not working as they would return `ERRNO_BADF`
which is swallowed by at least Rust right now. This instead sets the
initialization of `State` to reflect how fd 0 is an empty stdin stream,
fd 1 goes to a `log` call, and fd 2 also goes to `log`. During `command`
initialization fds 0 and 1 are overwritten with the provided input and
for non-command builds this is the state permanently for the program.
It's possible we can add further configuration hooks for this in the
future, but this should get at least the initial state of non-command
builds more workable.
* Use BADF for reading log fds
* Update a few aspects of the adapter build
* Use the `wasm32-unknown-unknown` target for the adapter and specify
flags in `.cargo/config.toml` to avoid having to pass the same flags
everywhere. This allows using `wasm32-wasi` for tests to ensure the
flags only apply to the adapter.
* Use `opt-level=s` since speed is not of the utmost concern for this
wasm but since it's likely to be included in many places size is
likely more important.
* Use `strip = 'debuginfo'` for the release build to remove the standard
library's debugging information which isn't necessary.
* Remove `debug = 0` from the `dev` profile to have debugging
information for development.
* Add a small `README.md` describing what's here for now.
* Move `command` support behind a `command` feature
This commit adds a `command` feature to main crate to avoid importing
the `_start` function when the `command` feature is disabled, making
this adapter useful for non-command WASI programs as well.
For now this still emits the `command` export in the final component but
with `use` in `*.wit` files it should be easier to avoid that export.
This adds a `poll_oneoff` implementation to src/lib.rs. It's completely
untested. I was planning on adding a host implementation and using that to test
end-to-end, but that raised some tough questions about how much of the existing
`wasi-common` scheduler(s) should be reused. I've decided to focus on other,
more widely-used parts of WASI first, but I wanted to share the work I've
already done here.
Note that I've moved the clock- and socket-specific functions out of `wasi-poll`
and into `wasi-clocks` and `wasi-tcp`, respectively. The latter is a new
interface which will eventually contain functions and types resembling
@npmccallum's https://github.com/npmccallum/wasi-snapshot-preview2#wasi-tcp
proposal.
Per discussion with @sunfishcode:
- `wasi-tcp` includes an `error` enum type, intended to represent only socket-related errors.
- It also includes a `socket` pseudo-handle type, distinct from `wasi-filesystem`'s `descriptor` type.
These fine-grained types help move us away from the "everything is a file
descriptor" and "all errors are errnos" approaches of Preview 1.
If we decide `poll-oneoff` should be usable with files as well as sockets, we
can add `subscribe-read` and `subscribe-write` functions to `wasi-filesystem`
which accept file `descriptor`s. Likewise for any other pseudo-handle type from
which we'd like to create futures.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
This commit updates the implementation of `cabi_export_realloc` to
allocate from a bump-allocated-region in `State` rather than allocating
a separate page for each argument as previously done. Additionally the
argument data is now stored within `State` as well enabling a full
implementation of the `args_get` and `args_sizes_get` syscalls.
* Simplify global state management in adapter
I was looking recently to implement args-related syscalls but that would
require yet-more globals and yet-more state to be managed. Instead of
adding a slew of new globals for this which are all manually kept in
sync I opted to instead redesign how global state is managed in the
adapter.
The previous multiple `global`s are all removed in favor of just one, as
sort of a "tls slot" of sorts. This one remaining slot points to a
one-time-allocated `State` object which internally stores information
like buffer metadata, fd information, etc. Along the way I've also
simplified syscalls with new methods and `?`-using closures.
* Turn off incremental for dev builds
Helps with CGU splitting and ensuring that appropriate code is produced
even without `--release`.
* Review comments
* Add accessors with specific errors
* Update handling of `*_global_ptr`
* Update internal mutability around path buffer
Use an `UnsafeCell` layering to indicate that mutation may happen
through `&T`.
Alex Crichton
Dan Gohman
Joel Dice
Pat Hickey
Pat Hickey