Browse Source
* Rename FdEntry to Entry
* Add custom FdSet container for managing fd allocs/deallocs
This commit adds a custom `FdSet` container which is intended
for use in `wasi-common` to track WASI fd allocs/deallocs. The
main aim for this container is to abstract away the current
approach of spawning new handles
```rust
fd = fd.checked_add(1).ok_or(...)?;
```
and to make it possible to reuse unused/reclaimed handles
which currently is not done.
The struct offers 3 methods to manage its functionality:
* `FdSet::new` initialises the internal data structures,
and most notably, it preallocates an `FdSet::BATCH_SIZE`
worth of handles in such a way that we always start popping
from the "smallest" handle (think of it as of reversed stack,
I guess; it's not a binary heap since we don't really care
whether internally the handles are sorted in some way, just that
the "largets" handle is at the bottom. Why will become clear
when describing `allocate` method.)
* `FdSet::allocate` pops the next available handle if one is available.
The tricky bit here is that, if we run out of handles, we preallocate
the next `FdSet::BATCH_SIZE` worth of handles starting from the
latest popped handle (i.e., the "largest" handle). This
works only because we make sure to only ever pop and push already
existing handles from the back, and push _new_ handles (from the
preallocation step) from the front. When we ultimately run out
of _all_ available handles, we then return `None` for the client
to handle in some way (e.g., throwing an error such as `WasiError::EMFILE`
or whatnot).
* `FdSet::deallocate` returns the already allocated handle back to
the pool for further reuse.
When figuring out the internals, I've tried to optimise for both
alloc and dealloc performance, and I believe we've got an amortised
`O(1)~*` performance for both (if my maths is right, and it may very
well not be, so please verify!).
In order to keep `FdSet` fairly generic, I've made sure not to hard-code
it for the current type system generated by `wig` (i.e., `wasi::__wasi_fd_t`
representing WASI handle), but rather, any type which wants to be managed
by `FdSet` needs to conform to `Fd` trait. This trait is quite simple as
it only requires a couple of rudimentary traits (although `std:#️⃣:Hash`
is quite a powerful assumption here!), and a custom method
```rust
Fd::next(&self) -> Option<Self>;
```
which is there to encapsulate creating another handle from the given one.
In the current state of the code, that'd be simply `u32::checked_add(1)`.
When `wiggle` makes it way into the `wasi-common`, I'd imagine it being
similar to
```rust
fn next(&self) -> Option<Self> {
self.0.checked_add(1).map(Self::from)
}
```
Anyhow, I'd be happy to learn your thoughts about this design!
* Fix compilation on other targets
* Rename FdSet to FdPool
* Fix FdPool unit tests
* Skip preallocation step in FdPool
* Replace 'replace' calls with direct assignment
* Reuse FdPool from snapshot1 in snapshot0
* Refactor FdPool::allocate
* Remove entry before deallocating the fd
* Refactor the design to accommodate `u32` as underlying type
This commit refactors the design by ensuring that the underlying
type in `FdPool` which we use to track and represent raw file
descriptors is `u32`. As a result, the structure of `FdPool` is
simplified massively as we no longer need to track the claimed
descriptors; in a way, we trust the caller to return the handle
after it's done with it. In case the caller decides to be clever
and return a handle which was not yet legally allocated, we panic.
This should never be a problem in `wasi-common` unless we hit a
bug.
To make all of this work, `Fd` trait is modified to require two
methods: `as_raw(&self) -> u32` and `from_raw(raw_fd: u32) -> Self`
both of which are used to convert to and from the `FdPool`'s underlying
type `u32`.
pull/1349/head
Jakub Konka
5 years ago
committed by
GitHub
GitHub
33 changed files with 419 additions and 319 deletions
@ -0,0 +1,133 @@ |
|||
//! Contains mechanism for managing the WASI file descriptor
|
|||
//! pool. It's intended to be mainly used within the `WasiCtx`
|
|||
//! object(s).
|
|||
|
|||
/// Any type wishing to be treated as a valid WASI file descriptor
|
|||
/// should implement this trait.
|
|||
///
|
|||
/// This trait is required as internally we use `u32` to represent
|
|||
/// and manage raw file descriptors.
|
|||
pub(crate) trait Fd { |
|||
/// Convert to `u32`.
|
|||
fn as_raw(&self) -> u32; |
|||
/// Convert from `u32`.
|
|||
fn from_raw(raw_fd: u32) -> Self; |
|||
} |
|||
|
|||
/// This container tracks and manages all file descriptors that
|
|||
/// were already allocated.
|
|||
/// Internally, we use `u32` to represent the file descriptors;
|
|||
/// however, the caller may supply any type `T` such that it
|
|||
/// implements the `Fd` trait when requesting a new descriptor
|
|||
/// via the `allocate` method, or when returning one back via
|
|||
/// the `deallocate` method.
|
|||
#[derive(Debug)] |
|||
pub(crate) struct FdPool { |
|||
next_alloc: Option<u32>, |
|||
available: Vec<u32>, |
|||
} |
|||
|
|||
impl FdPool { |
|||
pub fn new() -> Self { |
|||
Self { |
|||
next_alloc: Some(0), |
|||
available: Vec::new(), |
|||
} |
|||
} |
|||
|
|||
/// Obtain another valid WASI file descriptor.
|
|||
///
|
|||
/// If we've handed out the maximum possible amount of file
|
|||
/// descriptors (which would be equal to `2^32 + 1` accounting for `0`),
|
|||
/// then this method will return `None` to signal that case.
|
|||
/// Otherwise, a new file descriptor is return as `Some(fd)`.
|
|||
pub fn allocate<T: Fd>(&mut self) -> Option<T> { |
|||
if let Some(fd) = self.available.pop() { |
|||
// Since we've had free, unclaimed handle in the pool,
|
|||
// simply claim it and return.
|
|||
return Some(T::from_raw(fd)); |
|||
} |
|||
// There are no free handles available in the pool, so try
|
|||
// allocating an additional one into the pool. If we've
|
|||
// reached our max number of handles, we will fail with None
|
|||
// instead.
|
|||
let fd = self.next_alloc.take()?; |
|||
// It's OK to not unpack the result of `fd.checked_add()` here which
|
|||
// can fail since we check for `None` in the snippet above.
|
|||
self.next_alloc = fd.checked_add(1); |
|||
Some(T::from_raw(fd)) |
|||
} |
|||
|
|||
/// Return a file descriptor back to the pool.
|
|||
///
|
|||
/// If the caller tries to return a file descriptor that was
|
|||
/// not yet allocated (via spoofing, etc.), this method
|
|||
/// will panic.
|
|||
pub fn deallocate<T: Fd>(&mut self, fd: T) { |
|||
let fd = fd.as_raw(); |
|||
if let Some(next_alloc) = self.next_alloc { |
|||
assert!(fd < next_alloc); |
|||
} |
|||
debug_assert!(!self.available.contains(&fd)); |
|||
self.available.push(fd); |
|||
} |
|||
} |
|||
|
|||
#[cfg(test)] |
|||
mod test { |
|||
use super::FdPool; |
|||
use std::ops::Deref; |
|||
|
|||
#[derive(Debug)] |
|||
struct Fd(u32); |
|||
|
|||
impl super::Fd for Fd { |
|||
fn as_raw(&self) -> u32 { |
|||
self.0 |
|||
} |
|||
fn from_raw(raw_fd: u32) -> Self { |
|||
Self(raw_fd) |
|||
} |
|||
} |
|||
|
|||
impl Deref for Fd { |
|||
type Target = u32; |
|||
fn deref(&self) -> &Self::Target { |
|||
&self.0 |
|||
} |
|||
} |
|||
|
|||
#[test] |
|||
fn basics() { |
|||
let mut fd_pool = FdPool::new(); |
|||
let mut fd: Fd = fd_pool.allocate().expect("success allocating 0"); |
|||
assert_eq!(*fd, 0); |
|||
fd = fd_pool.allocate().expect("success allocating 1"); |
|||
assert_eq!(*fd, 1); |
|||
fd = fd_pool.allocate().expect("success allocating 2"); |
|||
assert_eq!(*fd, 2); |
|||
fd_pool.deallocate(1u32); |
|||
fd_pool.deallocate(0u32); |
|||
fd = fd_pool.allocate().expect("success reallocating 0"); |
|||
assert_eq!(*fd, 0); |
|||
fd = fd_pool.allocate().expect("success reallocating 1"); |
|||
assert_eq!(*fd, 1); |
|||
fd = fd_pool.allocate().expect("success allocating 3"); |
|||
assert_eq!(*fd, 3); |
|||
} |
|||
|
|||
#[test] |
|||
#[should_panic] |
|||
fn deallocate_nonexistent() { |
|||
let mut fd_pool = FdPool::new(); |
|||
fd_pool.deallocate(0u32); |
|||
} |
|||
|
|||
#[test] |
|||
fn max_allocation() { |
|||
let mut fd_pool = FdPool::new(); |
|||
// Spoof reaching the limit of allocs.
|
|||
fd_pool.next_alloc = None; |
|||
assert!(fd_pool.allocate::<Fd>().is_none()); |
|||
} |
|||
} |
|||
@ -1,4 +1,4 @@ |
|||
use crate::old::snapshot_0::fdentry::{Descriptor, OsHandleRef}; |
|||
use crate::old::snapshot_0::entry::{Descriptor, OsHandleRef}; |
|||
use crate::old::snapshot_0::{sys::unix::sys_impl, wasi}; |
|||
use std::fs::File; |
|||
use std::io; |
|||
@ -1,4 +1,4 @@ |
|||
use crate::old::snapshot_0::fdentry::{Descriptor, OsHandleRef}; |
|||
use crate::old::snapshot_0::entry::{Descriptor, OsHandleRef}; |
|||
use crate::old::snapshot_0::wasi; |
|||
use std::fs::File; |
|||
use std::io; |
|||
@ -1,4 +1,4 @@ |
|||
use crate::fdentry::{Descriptor, OsHandleRef}; |
|||
use crate::entry::{Descriptor, OsHandleRef}; |
|||
use crate::{sys::unix::sys_impl, wasi}; |
|||
use std::fs::File; |
|||
use std::io; |
|||
@ -1,4 +1,4 @@ |
|||
use crate::fdentry::{Descriptor, OsHandleRef}; |
|||
use crate::entry::{Descriptor, OsHandleRef}; |
|||
use crate::wasi; |
|||
use std::fs::File; |
|||
use std::io; |
|||
Loading…
Reference in new issue