Use relative `call` instructions between wasm functions (#3275)
* Use relative `call` instructions between wasm functions
This commit is a relatively major change to the way that Wasmtime
generates code for Wasm modules and how functions call each other.
Prior to this commit all function calls between functions, even if they
were defined in the same module, were done indirectly through a
register. To implement this the backend would emit an absolute 8-byte
relocation near all function calls, load that address into a register,
and then call it. While this technique is simple to implement and easy
to get right, it has two primary downsides associated with it:
* Function calls are always indirect which means they are more difficult
to predict, resulting in worse performance.
* Generating a relocation-per-function call requires expensive
relocation resolution at module-load time, which can be a large
contributing factor to how long it takes to load a precompiled module.
To fix these issues, while also somewhat compromising on the previously
simple implementation technique, this commit switches wasm calls within
a module to using the `colocated` flag enabled in Cranelift-speak, which
basically means that a relative call instruction is used with a
relocation that's resolved relative to the pc of the call instruction
itself.
When switching the `colocated` flag to `true` this commit is also then
able to move much of the relocation resolution from `wasmtime_jit::link`
into `wasmtime_cranelift::obj` during object-construction time. This
frontloads all relocation work which means that there's actually no
relocations related to function calls in the final image, solving both
of our points above.
The main gotcha in implementing this technique is that there are
hardware limitations to relative function calls which mean we can't
simply blindly use them. AArch64, for example, can only go +/- 64 MB
from the `bl` instruction to the target, which means that if the
function we're calling is a greater distance away then we would fail to
resolve that relocation. On x86_64 the limits are +/- 2GB which are much
larger, but theoretically still feasible to hit. Consequently the main
increase in implementation complexity is fixing this issue.
This issue is actually already present in Cranelift itself, and is
internally one of the invariants handled by the `MachBuffer` type. When
generating a function relative jumps between basic blocks have similar
restrictions. This commit adds new methods for the `MachBackend` trait
and updates the implementation of `MachBuffer` to account for all these
new branches. Specifically the changes to `MachBuffer` are:
* For AAarch64 the `LabelUse::Branch26` value now supports veneers, and
AArch64 calls use this to resolve relocations.
* The `emit_island` function has been rewritten internally to handle
some cases which previously didn't come up before, such as:
* When emitting an island the deadline is now recalculated, where
previously it was always set to infinitely in the future. This was ok
prior since only a `Branch19` supported veneers and once it was
promoted no veneers were supported, so without multiple layers of
promotion the lack of a new deadline was ok.
* When emitting an island all pending fixups had veneers forced if
their branch target wasn't known yet. This was generally ok for
19-bit fixups since the only kind getting a veneer was a 19-bit
fixup, but with mixed kinds it's a bit odd to force veneers for a
26-bit fixup just because a nearby 19-bit fixup needed a veneer.
Instead fixups are now re-enqueued unless they're known to be
out-of-bounds. This may run the risk of generating more islands for
19-bit branches but it should also reduce the number of islands for
between-function calls.
* Otherwise the internal logic was tweaked to ideally be a bit more
simple, but that's a pretty subjective criteria in compilers...
I've added some simple testing of this for now. A synthetic compiler
option was create to simply add padded 0s between functions and test
cases implement various forms of calls that at least need veneers. A
test is also included for x86_64, but it is unfortunately pretty slow
because it requires generating 2GB of output. I'm hoping for now it's
not too bad, but we can disable the test if it's prohibitive and
otherwise just comment the necessary portions to be sure to run the
ignored test if these parts of the code have changed.
The final end-result of this commit is that for a large module I'm
working with the number of relocations dropped to zero, meaning that
nothing actually needs to be done to the text section when it's loaded
into memory (yay!). I haven't run final benchmarks yet but this is the
last remaining source of significant slowdown when loading modules,
after I land a number of other PRs both active and ones that I only have
locally for now.
* Fix arm32
* Review comments
3 years ago
|
|
|
//! These tests are intended to exercise various relocation-based logic of
|
|
|
|
//! Wasmtime, especially the "jump veneer" insertion in the object-file-assembly
|
|
|
|
//! for when platform-specific relative call instructios can't always reach
|
|
|
|
//! their destination within the platform-specific limits.
|
|
|
|
//!
|
|
|
|
//! Note that the limits of AArch64 are primarily what's being stressed here
|
|
|
|
//! where the jump target for a call is 26-bits. On x86_64 the jump target is
|
|
|
|
//! 32-bits, and right now object files aren't supported larger than 4gb anyway
|
|
|
|
//! so we would need a lot of other support necessary to exercise that.
|
|
|
|
|
|
|
|
use anyhow::Result;
|
|
|
|
use wasmtime::*;
|
|
|
|
|
|
|
|
const MB: usize = 1 << 20;
|
|
|
|
|
|
|
|
fn store_with_padding(padding: usize) -> Result<Store<()>> {
|
|
|
|
let mut config = Config::new();
|
|
|
|
// This is an internal debug-only setting specifically recognized for
|
|
|
|
// basically just this set of tests.
|
|
|
|
unsafe {
|
|
|
|
config.cranelift_flag_set(
|
|
|
|
"wasmtime_linkopt_padding_between_functions",
|
|
|
|
&padding.to_string(),
|
|
|
|
);
|
Use relative `call` instructions between wasm functions (#3275)
* Use relative `call` instructions between wasm functions
This commit is a relatively major change to the way that Wasmtime
generates code for Wasm modules and how functions call each other.
Prior to this commit all function calls between functions, even if they
were defined in the same module, were done indirectly through a
register. To implement this the backend would emit an absolute 8-byte
relocation near all function calls, load that address into a register,
and then call it. While this technique is simple to implement and easy
to get right, it has two primary downsides associated with it:
* Function calls are always indirect which means they are more difficult
to predict, resulting in worse performance.
* Generating a relocation-per-function call requires expensive
relocation resolution at module-load time, which can be a large
contributing factor to how long it takes to load a precompiled module.
To fix these issues, while also somewhat compromising on the previously
simple implementation technique, this commit switches wasm calls within
a module to using the `colocated` flag enabled in Cranelift-speak, which
basically means that a relative call instruction is used with a
relocation that's resolved relative to the pc of the call instruction
itself.
When switching the `colocated` flag to `true` this commit is also then
able to move much of the relocation resolution from `wasmtime_jit::link`
into `wasmtime_cranelift::obj` during object-construction time. This
frontloads all relocation work which means that there's actually no
relocations related to function calls in the final image, solving both
of our points above.
The main gotcha in implementing this technique is that there are
hardware limitations to relative function calls which mean we can't
simply blindly use them. AArch64, for example, can only go +/- 64 MB
from the `bl` instruction to the target, which means that if the
function we're calling is a greater distance away then we would fail to
resolve that relocation. On x86_64 the limits are +/- 2GB which are much
larger, but theoretically still feasible to hit. Consequently the main
increase in implementation complexity is fixing this issue.
This issue is actually already present in Cranelift itself, and is
internally one of the invariants handled by the `MachBuffer` type. When
generating a function relative jumps between basic blocks have similar
restrictions. This commit adds new methods for the `MachBackend` trait
and updates the implementation of `MachBuffer` to account for all these
new branches. Specifically the changes to `MachBuffer` are:
* For AAarch64 the `LabelUse::Branch26` value now supports veneers, and
AArch64 calls use this to resolve relocations.
* The `emit_island` function has been rewritten internally to handle
some cases which previously didn't come up before, such as:
* When emitting an island the deadline is now recalculated, where
previously it was always set to infinitely in the future. This was ok
prior since only a `Branch19` supported veneers and once it was
promoted no veneers were supported, so without multiple layers of
promotion the lack of a new deadline was ok.
* When emitting an island all pending fixups had veneers forced if
their branch target wasn't known yet. This was generally ok for
19-bit fixups since the only kind getting a veneer was a 19-bit
fixup, but with mixed kinds it's a bit odd to force veneers for a
26-bit fixup just because a nearby 19-bit fixup needed a veneer.
Instead fixups are now re-enqueued unless they're known to be
out-of-bounds. This may run the risk of generating more islands for
19-bit branches but it should also reduce the number of islands for
between-function calls.
* Otherwise the internal logic was tweaked to ideally be a bit more
simple, but that's a pretty subjective criteria in compilers...
I've added some simple testing of this for now. A synthetic compiler
option was create to simply add padded 0s between functions and test
cases implement various forms of calls that at least need veneers. A
test is also included for x86_64, but it is unfortunately pretty slow
because it requires generating 2GB of output. I'm hoping for now it's
not too bad, but we can disable the test if it's prohibitive and
otherwise just comment the necessary portions to be sure to run the
ignored test if these parts of the code have changed.
The final end-result of this commit is that for a large module I'm
working with the number of relocations dropped to zero, meaning that
nothing actually needs to be done to the text section when it's loaded
into memory (yay!). I haven't run final benchmarks yet but this is the
last remaining source of significant slowdown when loading modules,
after I land a number of other PRs both active and ones that I only have
locally for now.
* Fix arm32
* Review comments
3 years ago
|
|
|
}
|
|
|
|
let engine = Engine::new(&config)?;
|
|
|
|
Ok(Store::new(&engine, ()))
|
|
|
|
}
|
|
|
|
|
|
|
|
#[test]
|
|
|
|
fn forward_call_works() -> Result<()> {
|
|
|
|
let mut store = store_with_padding(128 * MB)?;
|
|
|
|
let module = Module::new(
|
|
|
|
store.engine(),
|
|
|
|
r#"
|
|
|
|
(module
|
|
|
|
(func (export "foo") (result i32)
|
|
|
|
call 1)
|
|
|
|
(func (result i32)
|
|
|
|
i32.const 4)
|
|
|
|
)
|
|
|
|
"#,
|
|
|
|
)?;
|
|
|
|
|
|
|
|
let i = Instance::new(&mut store, &module, &[])?;
|
|
|
|
let foo = i.get_typed_func::<(), i32, _>(&mut store, "foo")?;
|
|
|
|
assert_eq!(foo.call(&mut store, ())?, 4);
|
|
|
|
Ok(())
|
|
|
|
}
|
|
|
|
|
|
|
|
#[test]
|
|
|
|
fn backwards_call_works() -> Result<()> {
|
|
|
|
let mut store = store_with_padding(128 * MB)?;
|
|
|
|
let module = Module::new(
|
|
|
|
store.engine(),
|
|
|
|
r#"
|
|
|
|
(module
|
|
|
|
(func (result i32)
|
|
|
|
i32.const 4)
|
|
|
|
(func (export "foo") (result i32)
|
|
|
|
call 0)
|
|
|
|
)
|
|
|
|
"#,
|
|
|
|
)?;
|
|
|
|
|
|
|
|
let i = Instance::new(&mut store, &module, &[])?;
|
|
|
|
let foo = i.get_typed_func::<(), i32, _>(&mut store, "foo")?;
|
|
|
|
assert_eq!(foo.call(&mut store, ())?, 4);
|
|
|
|
Ok(())
|
|
|
|
}
|
|
|
|
|
|
|
|
#[test]
|
|
|
|
fn mixed() -> Result<()> {
|
|
|
|
test_many_call_module(store_with_padding(MB)?)
|
|
|
|
}
|
|
|
|
|
|
|
|
#[test]
|
|
|
|
fn mixed_forced() -> Result<()> {
|
|
|
|
let mut config = Config::new();
|
|
|
|
unsafe {
|
|
|
|
config.cranelift_flag_set("wasmtime_linkopt_force_jump_veneer", "true");
|
Use relative `call` instructions between wasm functions (#3275)
* Use relative `call` instructions between wasm functions
This commit is a relatively major change to the way that Wasmtime
generates code for Wasm modules and how functions call each other.
Prior to this commit all function calls between functions, even if they
were defined in the same module, were done indirectly through a
register. To implement this the backend would emit an absolute 8-byte
relocation near all function calls, load that address into a register,
and then call it. While this technique is simple to implement and easy
to get right, it has two primary downsides associated with it:
* Function calls are always indirect which means they are more difficult
to predict, resulting in worse performance.
* Generating a relocation-per-function call requires expensive
relocation resolution at module-load time, which can be a large
contributing factor to how long it takes to load a precompiled module.
To fix these issues, while also somewhat compromising on the previously
simple implementation technique, this commit switches wasm calls within
a module to using the `colocated` flag enabled in Cranelift-speak, which
basically means that a relative call instruction is used with a
relocation that's resolved relative to the pc of the call instruction
itself.
When switching the `colocated` flag to `true` this commit is also then
able to move much of the relocation resolution from `wasmtime_jit::link`
into `wasmtime_cranelift::obj` during object-construction time. This
frontloads all relocation work which means that there's actually no
relocations related to function calls in the final image, solving both
of our points above.
The main gotcha in implementing this technique is that there are
hardware limitations to relative function calls which mean we can't
simply blindly use them. AArch64, for example, can only go +/- 64 MB
from the `bl` instruction to the target, which means that if the
function we're calling is a greater distance away then we would fail to
resolve that relocation. On x86_64 the limits are +/- 2GB which are much
larger, but theoretically still feasible to hit. Consequently the main
increase in implementation complexity is fixing this issue.
This issue is actually already present in Cranelift itself, and is
internally one of the invariants handled by the `MachBuffer` type. When
generating a function relative jumps between basic blocks have similar
restrictions. This commit adds new methods for the `MachBackend` trait
and updates the implementation of `MachBuffer` to account for all these
new branches. Specifically the changes to `MachBuffer` are:
* For AAarch64 the `LabelUse::Branch26` value now supports veneers, and
AArch64 calls use this to resolve relocations.
* The `emit_island` function has been rewritten internally to handle
some cases which previously didn't come up before, such as:
* When emitting an island the deadline is now recalculated, where
previously it was always set to infinitely in the future. This was ok
prior since only a `Branch19` supported veneers and once it was
promoted no veneers were supported, so without multiple layers of
promotion the lack of a new deadline was ok.
* When emitting an island all pending fixups had veneers forced if
their branch target wasn't known yet. This was generally ok for
19-bit fixups since the only kind getting a veneer was a 19-bit
fixup, but with mixed kinds it's a bit odd to force veneers for a
26-bit fixup just because a nearby 19-bit fixup needed a veneer.
Instead fixups are now re-enqueued unless they're known to be
out-of-bounds. This may run the risk of generating more islands for
19-bit branches but it should also reduce the number of islands for
between-function calls.
* Otherwise the internal logic was tweaked to ideally be a bit more
simple, but that's a pretty subjective criteria in compilers...
I've added some simple testing of this for now. A synthetic compiler
option was create to simply add padded 0s between functions and test
cases implement various forms of calls that at least need veneers. A
test is also included for x86_64, but it is unfortunately pretty slow
because it requires generating 2GB of output. I'm hoping for now it's
not too bad, but we can disable the test if it's prohibitive and
otherwise just comment the necessary portions to be sure to run the
ignored test if these parts of the code have changed.
The final end-result of this commit is that for a large module I'm
working with the number of relocations dropped to zero, meaning that
nothing actually needs to be done to the text section when it's loaded
into memory (yay!). I haven't run final benchmarks yet but this is the
last remaining source of significant slowdown when loading modules,
after I land a number of other PRs both active and ones that I only have
locally for now.
* Fix arm32
* Review comments
3 years ago
|
|
|
}
|
|
|
|
let engine = Engine::new(&config)?;
|
|
|
|
test_many_call_module(Store::new(&engine, ()))
|
|
|
|
}
|
|
|
|
|
|
|
|
fn test_many_call_module(mut store: Store<()>) -> Result<()> {
|
|
|
|
const N: i32 = 200;
|
|
|
|
|
|
|
|
let mut wat = String::new();
|
|
|
|
wat.push_str("(module\n");
|
|
|
|
wat.push_str("(func $first (result i32) (i32.const 1))\n");
|
|
|
|
for i in 0..N {
|
|
|
|
wat.push_str(&format!("(func (export \"{}\") (result i32 i32)\n", i));
|
|
|
|
wat.push_str("call $first\n");
|
|
|
|
wat.push_str(&format!("i32.const {}\n", i));
|
|
|
|
wat.push_str("i32.add\n");
|
|
|
|
wat.push_str("call $last\n");
|
|
|
|
wat.push_str(&format!("i32.const {}\n", i));
|
|
|
|
wat.push_str("i32.add)\n");
|
|
|
|
}
|
|
|
|
wat.push_str("(func $last (result i32) (i32.const 2))\n");
|
|
|
|
wat.push_str(")\n");
|
|
|
|
|
|
|
|
let module = Module::new(store.engine(), &wat)?;
|
|
|
|
|
|
|
|
let instance = Instance::new(&mut store, &module, &[])?;
|
|
|
|
|
|
|
|
for i in 0..N {
|
|
|
|
let name = i.to_string();
|
|
|
|
let func = instance.get_typed_func::<(), (i32, i32), _>(&mut store, &name)?;
|
|
|
|
let (a, b) = func.call(&mut store, ())?;
|
|
|
|
assert_eq!(a, i + 1);
|
|
|
|
assert_eq!(b, i + 2);
|
|
|
|
}
|
|
|
|
Ok(())
|
|
|
|
}
|