* Bump MSRV to 1.78.0
This commit updates to use Rust 1.80 in CI as well as updating the
minimum-supported-rust-version to 1.78. Additionally the nightly used in
testing in CI is updated as well.
prtest:full
* Fix compat with latest nightly in tests
* Fix some more nightly warnings
* Ignore new tests on MIRI
* Add wasi adapter provider template which is materialised in CI
* Add rustfmt component to adapter CI
* Draft an extra publish step for the adapter provider
* Check adapter provider in a separate step with adapter artifacts
* Use artifact downloads in the publish action as well
* Record results from adapter provider step as well
* Refactor to use composite actions
* Add missing shell property
* Fix spelling mistake
* Try using the env context
This accompanies today's release of Rust 1.79.0. This means that CI will
now use the latest stable of 1.79.0 for testing primarily. Additionally
this updates the pinned nightly commit used for testing to today's
nightly and fixes a few warnings that uncovered. I plan on having a
follow-up that leverages some new APIs and features in 1.77.0.
* Refactor installation of C API and features supported
This commit overhauls and refactors the management of the building of
the C API. Instead of this being script-based it's now done entirely
through CMake and CMake is the primary focus for building the C API. For
example building the C API release artifacts is now done through CMake
instead of through a shell script's `cargo build` and manually moving
artifacts.
The benefits that this brings are:
* The C API now properly hides symbols in its header files that weren't
enabled at build time. This is done through a new build-time generated
`conf.h` templated on a `conf.h.in` file in the source tree.
* The C API's project now supports enabling/disabling Cargo features to
have finer-grained support over what's included (plus auto-management
of the header file).
* Building the C API and managing it is now exclusively done through
CMake. For example invoking `doxygen` now lives in CMake, installation
lives there, etc.
The `CMakeLists.txt` file for the C API is overhauled in the process of
doing this. The build now primarily matches on the Rust target being
built rather than the host target. Additionally installation will now
install both the static and shared libraries instead of just one.
Additionally during this refactoring various bits and pieces of
Android-specific code were all removed. Management of the C toolchain
feels best left in scope of the caller (e.g. configuring `CC_*` env vars
and such) rather than here.
prtest:full
* Don't use `option` for optional strings
* Invert release build check
Also adjust some indentation
* Fix more indentation
* Remove no-longer-used variable
* Reduce duplication in feature macro
* Add Android release binaries to CI
This commit is inspired by #6480 and historical asks for Android
binaries. This does the bare minimum necessary to configure C compilers
such that we can produce binaries but I'll admit that I'm no Android
developer myself so I have no idea if these are actually suitable for
use anywhere. Otherwise though this build subsumes the preexisting check
in CI that the build works for Android, so that part is removed too.
This additionally changes how the NDK is managed from before. Previously
a GitHub Action was used to download Java and the NDK and additionally
used the `cargo ndk` subcommand. That's all removed now in favor of
configuring C compilers directly with a pre-installed version of the NDK
which should help reduce the CI dependencies a bit.
* Review comments
* List Android as tier 3 target
* Bump Wasmtime's MSRV to 1.76.0
* Update Rust in CI to 1.78.0, the current stable
* Update nightly tests to the latest nightly
prtest:full
* Fix check-cfg with nightly
* More check-cfg fixes
* Remove an async cfg
This is no longer specified for the root crate.
* Move definition of Wasmtime's nightly into one place
Don't change a bunch of places when this is updated, try to update just
one single location instead.
* Use `--locked` on all `cargo install` in CI
Prevents any updates to rustc or crates from accidentally causing issues
by ensuring that the same set of deps is used over time.
* Remove rust/WASI markdown parser example
The documentation referring to this example was removed in #6994 and
that forgot to remove this as well. This example is building without a
lock file which is causing issues in #8368.
This was originally added when Cargo would git clone the index and the
significant size of the index meant we got nontrivial speedups during
the cloning process. Nowadays though Cargo does a much more CI-efficient
method by default where it uses an HTTP index instead. This removes the
original need for caching since the index operations should now be much
faster, probably moreso than saving/restoring the cache.
This additionally removes the caching of registry downloads and git
clones too since in theory the cache isn't all that much faster than
what Cargo is already doing.
* Attempt to automatically configure release notes
This commit is an attempt to tackle #7068 by configuring the release
notes in Github Releases with the handwritten release notes from
`RELEASES.md`. The basic idea here is to split the markdown file on
`-----` delimiters and then find the one which matches the version being
released. Once one is found the `body` field of the API call to create
the release is configured.
* Update .github/actions/github-release/main.js
Co-authored-by: Andrew Brown <andrew.brown@intel.com>
---------
Co-authored-by: Andrew Brown <andrew.brown@intel.com>
* riscv64: Extend distance trampolines can jump
Use a PIC-friendly set of instructions to enable destination of the
trampoline to be more than 4k away from the tail call site of the
trampoline itself.
* Build "min" artifacts on CI
This commit updates the binary artifacts produced by CI to include "min"
builds where all default features are disabled. Additionally all the
stops are pulled in terms of build flags, nightly versions, etc, to get
a build that is as small as possible without actual source code changes.
This effectively codifies the instructions in #7282 into an easily
downloadable artifact.
No new tarballs are created for github releases but instead tarballs
that previously had a `wasmtime` executable for example now have a
`wasmtime-min` executable. Furthermore the C API which previously had
`libwasmtime.so` for example now has `libwasmtime-min.so`. The intention
is that the minimum-size artifacts are handy for determining a rough
size of Wasmtime but they're not so important that it seems worthwhile
to dedicate entire release entries for.
CI is refactored to support these minimum builds with separate builders.
This means that a single tarball produced as a final result is actually
two separate tarballs merged together, one for the normal build we do
today plus a new "min" tarball produced on the new "min" builders.
Various scripts and CI organization has been adjusted accordingly.
While here I went ahead and enabled `panic=abort` and debuginfo
stripping in our current release artifacts. While this doesn't affect a
whole lot it's less to upload to GitHub Actions all the time.
* Fix Windows unzip
* Refactor the test-programs test suite
This commit is a large refactoring that reorganizes `test-programs` and
how we tests wasms in Wasmtime. Often writing tests requires complicated
interactions with the guest which can't be done via hand-written `*.wat`
syntax and requires a compiler to get engaged. For this purpose Wasmtime
currently has the `crates/test-programs/*` test suite which builds files
from source and then runs the tests. This has been somewhat cumbersome
in the past though and it's not been easy to extend this over time, so
this commit attempts to address this.
The scheme implemented in this PR looks like:
* All wasm test programs live in `crates/test-programs/src/bin/*.rs`.
All of them, no exceptions.
* Wasm tests have shared support located at
`crates/test-programs/src/lib.rs` and its submodules, such as bindings
generation for WASI.
* Wasm tests are built by a new `crates/test-programs/artifacts` crate.
This crate compiles modules and additionally creates components for
all test programs. The crate itself only records the path to these
outputs and a small amount of testing support, but otherwise doesn't
interact with `wasmtime`-the-crate itself.
* All tests in `crates/test-programs/tests/*.rs` have moved. For example
wasi-http tests now live at `crates/wasi-http/tests/*.rs`. Legacy
tests of wasi-common now live at `crates/wasi-common/tests/*.rs`.
Modern tests for preview2 live at `crates/wasi/tests/*.rs`.
* Wasm tests are bucketed based on their filename prefix. For example
`preview1_*` is tested in wasi-common and wasmtime-wasi. The
`preview2_*` prefix is only tested with wasmtime-wasi, however.
* A new `cli_*` prefix is used to execute tests as part of
`tests/all/main.rs`. This is a new submodule in
`tests/all/cli_tests.rs` which executes these components on the
command line. Many old "command" tests were migrated here.
* Helper macros are generated to assert that a test suite is run in its
entirety. This way if a `preview1_*` test is added it's asserted to
get added to both wasi-common and wasmtime-wasi in the various modes
they run tests.
Overall this moved a number of tests around and refactored some edges of
the tests, but this should not lose any tests (except one that wasn't
actually testing anything). Additionally the hope is that it's much
easier to add tests in the future. The process is to add a new file in
`crates/test-programs/src/bin/*.rs` named appropriately. For example a
preview2 executable is `preview2_*` and a CLI tests is `cli_*`. When
building the test suite an error is generated in the appropriate module
then of "please write a test here", and then a test is written in the
same manner as the other tests in the module.
* Remove no-longer-needed fetches
prtest:full
* I'm worried wasi is running low on semicolons
* Add the WASI target in all CI actions
* Add unknown-unknown target on all CI builders too
* Fix building test artifacts under miri
Need to avoid wrappers for these cross-compiled targets
* Break circular dependency for packaging
Don't use the workspace dep for `wasmtime-wasi` since it injects a
version, instead use a `path = '..'` dependency to fool Cargo into
dropping the dependency during the package phase.
* Fix some merge conflicts with tests
* Fix rebase for new tests
* Remove stray comment
* Fix some flaky tests
* Fix network tests in synchronous mode
This commit is an attempt to fix some networking tests in synchronous
mode in our test suite. Currently networking tests don't actually run in
synchronous mode on CI which is why no failures have been surfaced yet,
but the refactoring in #7182 is going to start doing this.
Currently the `udp_sample_application.rs` test blocks infinitely in
synchronous mode for me locally, most of the time. This appears to be an
interaction between how Tokio handles readiness and how we're
entering the event loop. We're effectively entering the Tokio event loop
with a future that's always ready which ends up starving Tokio of
otherwise performing its background work such as updating flags for
readiness of reading/writing.
The fix here is to add a yield at the start of an `in_tokio` block which
is used in synchronous mode. This is a kludge fix but the intention is
to enable Tokio to have a chance to update readiness flags and process
events from epoll/kqueue/etc.
An additional fix to this issue is WebAssembly/wasi-sockets#64 where the
test is waiting on `READABLE` or `WRITABLE`, but in this specific case
it should only wait on `READABLE`. If it waited on just this then that
would also fix this issue. Nevertheless having a `yield_now` is expected
to have little-to-no overhead and otherwise fix this edge case of an
always-ready future.
* Fix passing empty arguments on the CLI
* Add another blocking accept
* Update crates/test-programs/src/bin/api_proxy.rs
Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>
---------
Co-authored-by: Trevor Elliott <awesomelyawesome@gmail.com>
This commit updates CI to require semicolons in all WIT files and parsed
WIT documents. Some minor updates were required in existing WIT files
and the wasi-nn proposal was additionally updated to its latest version
with semicolons. The wasi-nn update brought some minor changes to the
WIT which required some minor changes here as well.
* Update Rust in CI to 1.72.0
* Update CI, tooling, and docs for MSRV
This commit codifies an MSRV policy for Wasmtime at "stable minus two"
meaning that the latest three releases of Rust will be supported. This
is enforced on CI with a full test suite job running on Linux x86_64
with the minimum supported Rust version. The full test suite will use
the latest stable version. A downside of this approach is that new
changes may break MSRV support on non-Linux or non-x86_64 platforms and
we won't know about it, but that's deemed a minor enough risk at this
time.
A minor fix is applied to Wasmtime's `Cargo.toml` to support Rust 1.70.0
instead of requiring Rust 1.71.0
* Fix installation of rust
* Scrape MSRV from Cargo.toml
* Cranelift is the same as Wasmtime's MSRV now, more words too
* Fix a typo
Released today this updates from 1.69.0 to 1.71.0. Note that 1.70.0 was
skipped due to presumed codegen bugs for s390x and riscv64. These were
previously tested to work with 1.71.0, so it's time to confirm.
* Run `cargo vet` on automated version bumps
As shown in the original CI of #6686 the recent changes for `cargo vet`
0.8.0 mean that the previous release process no longer works since it
requires changes to the lock file for `cargo vet`. This updates the
version bumping process to run `cargo vet` as part of the committed
changes which hopefully will get everything to succeed. To ensure that
the same version of `cargo vet` is used in both locations the
installation procedure was extracted into its own separate little action.
* Configure shell to run in
In #6338 that PR is bouncing on CI due to Windows running out of disk
space. Locally a `./ci/run-tests.sh` compile produces a ~13G build
directory. Testing on CI it looks like Windows builders have ~13G of
free space on them for GitHub Actions. I think something in that PR has
tipped us just over the edge of requiring too much space, although I'm
not sure exactly what.
To address the issue this commit disables DWARF debug information
entirely on all builders on CI. Not only should this save a sliver of
compile time but this reduces a local build directory to ~4.7G, over a
50% reduction. That should keep us fitting in CI for more time to come
hopefully.
There was a missing rest field access. In addition createTag doesn't actually create a tag, it creates a tag object. A tag object is an object which references a commit or other kind of object and has various kinds of metadata. You need to store it in a reference stored in refs/tags/ to actually show as tag in the git ui. The code to update the tag however creates a lightweight tag (which is a file in refs/tags/ which directly references a commit rather than a tag object) as such do the same when creating the initial dev tag by using createRef with a commit id as object sha.
See also https://git-scm.com/book/en/v2/Git-Internals-Git-References for the difference between a lightweight tag and an annotated tag.
* Speed up index fetches on CI
Use the `sparse` protocol from Rust 1.68.0 which should shave a minute
or two off most steps on CI.
* Update nightly toolchains in CI
prtest:full
* Fix date
GitHub recently made its merge queue feature available for use in public
repositories owned by organizations meaning that the Wasmtime repository
is a candidate for using this. GitHub's Merge Queue feature is a system
that's similar to Rust's bors integration where PRs are tested before
merging and only passing PRs are merged. This implements the "not rocket
science" rule where the `main` branch of Wasmtime, for example, is
always tested and passes CI. This is in contrast to our current
implementation of CI where PRs are merged when they pass their own CI,
but the code that was tested is not guaranteed to be the state of `main`
when the PR is merged, meaning that we're at risk now of a failing
`main` branch despite all merged PRs being green. While this has
happened with Wasmtime this is not a common occurrence, however.
The main motivation, instead, to use GitHub's Merge Queue feature is
that it will enable Wasmtime to greatly reduce the amount of CI running
on PRs themselves. Currently the full test suite runs on every push to
every PR, meaning that our workers on GitHub Actions are frequently
clogged throughout weekdays and PRs can take quite some time to come
back with a successful run. Through the use of a Merge Queue, however,
we're able to configure only a small handful of checks to run on PRs
while deferring the main body of checks to happening on the
merge-via-the-queue itself. This is hoped to free up capacity on CI and
overall improve CI times for Wasmtime and Cranelift developers.
The implementation of all of this required quite a lot of plumbing and
retooling of our CI. I've been testing this in an [external
repository][testrepo] and I think everything is working now. A list of
changes made in this PR are:
* The `build.yml` workflow is merged back into the `main.yml` workflow
as the original reason to split it out is not longer applicable (it'll
run on all merges). This was also done to fit in the dependency graph
of jobs of one workflow.
* Publication of the `gh-pages` branch, the `dev` tag artifacts, and
release artifacts have been moved to a separate
`publish-artifacts.yml` workflow. This workflow runs on all pushes to
`main` and all tags. This workflow no longer actually preforms any
builds, however, and relies on a merge queue or similar being used for
branches/tags where artifacts are downloaded from the workflow run to
be uploaded. For pushes to `main` this works because a merge queue is
run meaning that by the time the push happens all artifacts are ready.
For release branches this is handled by..
* The `push-tag.yml` workflow is subsumed by the `main.yml` workflow. CI
for a tag being pushed will upload artifacts to a release in GitHub,
meaning that all builds must finish first for the commit. The
`main.yml` workflow at the end now scans commits for the preexisting
magical marker and pushes a tag if necessary.
* CI is currently a flat list of "run all these jobs" and this is now
rearchitected to a "fan out" approach where some jobs run to determine
the next jobs to run which then get "joined" into a finish step. The
purpose for this is somewhat nuanced and this has implications for CI
runtime as well. The Merge Queue feature requires branches to be
protected with "these checks must pass" and then the same checks are
gates both to enter the merge queue as well as pass the merge queue.
The saving grace, however, is that a "skipped" check counts as
passing, meaning checks can be skipped on PRs but run to completion on
the merge queue. A problem with this though is the build matrix used
for tests where PRs want to only run one element of the build matrix
ideally but there's no means on GitHub Actions right now for the
skipped entries to show up as skipped easily (or not that I know of).
This means that the "join" step serves the purpose of being the single
gate for both PR and merge queue CI and there's just more inputs to it
for merge queue CI. The major consequence of this decision is that
GitHub's actions scheduling doesn't work out well here. Jobs are
scheduled in a FIFO order meaning that the job for "ok complete the CI
run" is queued up after everything else has completed, possibly
after lots of other CI requests in the middle for other PRs. The hope
here is that by using a merge queue we can keep CI relatively under
control and this won't affect merge times too much.
* All jobs in the `main.yml` workflow will not automatically cancel the
entire run if they fail. Previously this fail-fast behavior was only
part of the matrix runs (and just for that matrix), but this is
required to make the merge queue expedient. The gate of the merge
queue is the final "join" step which is only executed once all
dependencies have finished. This means, for example, that if rustfmt
fails quickly then the tests which take longer might run for quite
awhile before the join step reports failure, meaning that the PR sits
in the queue for longer than needed being tested when we know it's
already going to fail. By having all jobs cancel the run this means
that failures immediately bail out and mark the whole job as
cancelled.
* A new "determine" CI job was added to determine what CI actually needs
to run. This is a "choke point" which is scheduled at the start of CI
that quickly figures out what else needs to be run. This notably
indicates whether large swaths of ci (the `run-full` flag) like the
build matrix are executed. Additionally this dynamically calculates a
matrix of tests to run based on a new `./ci/build-test-matrix.js`
script. Various inputs are considered for this such as:
1. All pushes, meaning merge queue branches or release-branch merges,
will run full CI.
2. PRs to release branches will run full CI.
3. PRs to `main`, the most common, determine what to run based on
what's modified and what's in the commit message.
Some examples for (3) above are if modifications are made to
`cranelift/codegen/src/isa/*` then that corresponding builder is
executed on CI. If the `crates/c-api` directory is modified then the
CMake-based tests are run on PRs but are otherwise skipped.
Annotations in commit messages such as `prtest:*` can be used to
explicitly request testing.
Before this PR merges to `main` would perform two full runs of CI: one
on the PR itself and one on the merge to `main`. Note that the one as a
merge to `main` was quite frequently cancelled due to a merge happening
later. Additionally before this PR there was always the risk of a bad
merge where what was merged ended up creating a `main` that failed CI to
to a non-code-related merge conflict.
After this PR merges to `main` will perform one full run of CI, the one
as part of the merge queue. PRs themselves will perform one test job
most of the time otherwise. The `main` branch is additionally always
guaranteed to pass tests via the merge queue feature.
For release branches, before this PR merges would perform two full
builds - one for the PR and one for the merge. A third build was then
required for the release tag itself. This is now cut down to two full
builds, one for the PR and one for the merge. The reason for this is
that the merge queue feature currently can't be used for our
wildcard-based `release-*` branch protections. It is now possible,
however, to turn on required CI checks for the `release-*` branch PRs so
we can at least have a "hit the button and forget" strategy for merging
PRs now.
Note that this change to CI is not without its risks. The Merge Queue
feature is still in beta and is quite new for GitHub. One bug that
Trevor and I uncovered is that if a PR is being tested in the merge
queue and a contributor pushes to their PR then the PR isn't removed
from the merge queue but is instead merged when CI is successful, losing
the changes that the contributor pushed (what's merged is what was
tested). We suspect that GitHub will fix this, however.
Additionally though there's the risk that this may increase merge time
for PRs to Wasmtime in practice. The Merge Queue feature has the ability
to "batch" PRs together for a merge but this is only done if concurrent
builds are allowed. This means that if 5 PRs are batched together then 5
separate merges would be created for the stack of 5 PRs. If the CI for
all 5 merged together passes then everything is merged, otherwise a PR
is kicked out. We can't easily do this, however, since a major purpose
for the merge queue for us would be to cut down on usage of CI builders
meaning the max concurrency would be set to 1 meaning that only one PR
at a time will be merged. This means PRs may sit in the queue for awhile
since previously many `main`-based builds are cancelled due to
subsequent merges of other PRs, but now they must all run to 100%
completion.
[testrepo]: https://github.com/bytecodealliance/wasmtime-merge-queue-testing
* Remove trailing whitespace in `lower.isle` files
* Legalize the `band_not` instruction into simpler form
This commit legalizes the `band_not` instruction into `band`-of-`bnot`,
or two instructions. This is intended to assist with egraph-based
optimizations where the `band_not` instruction doesn't have to be
specifically included in other bit-operation-patterns.
Lowerings of the `band_not` instruction have been moved to a
specialization of the `band` instruction.
* Legalize `bor_not` into components
Same as prior commit, but for the `bor_not` instruction.
* Legalize bxor_not into bxor-of-bnot
Same as prior commits. I think this also ended up fixing a bug in the
s390x backend where `bxor_not x y` was actually translated as `bnot
(bxor x y)` by accident given the test update changes.
* Simplify not-fused operands for riscv64
Looks like some delegated-to rules have special-cases for "if this
feature is enabled use the fused instruction" so move the clause for
testing the feature up to the lowering phase to help trigger other rules
if the feature isn't enabled. This should make the riscv64 backend more
consistent with how other backends are implemented.
* Remove B{and,or,xor}Not from cost of egraph metrics
These shouldn't ever reach egraphs now that they're legalized away.
* Add an egraph optimization for `x^-1 => ~x`
This adds a simplification node to translate xor-against-minus-1 to a
`bnot` instruction. This helps trigger various other optimizations in
the egraph implementation and also various backend lowering rules for
instructions. This is chiefly useful as wasm doesn't have a `bnot`
equivalent, so it's encoded as `x^-1`.
* Add a wasm test for end-to-end bitwise lowerings
Test that end-to-end various optimizations are being applied for input
wasm modules.
* Specifically don't self-update rustup on CI
I forget why this was here originally, but this is failing on Windows
CI. In general there's no need to update rustup, so leave it as-is.
* Cleanup some aarch64 lowering rules
Previously a 32/64 split was necessary due to the `ALUOp` being different
but that's been refactored away no so there's no longer any need for
duplicate rules.
* Narrow a x64 lowering rule
This previously made more sense when it was `band_not` and rarely used,
but be more specific in the type-filter on this rule that it's only
applicable to SIMD types with lanes.
* Simplify xor-against-minus-1 rule
No need to have the commutative version since constants are already
shuffled right for egraphs
* Optimize band-of-bnot when bnot is on the left
Use some more rules in the egraph algebraic optimizations to
canonicalize band/bor/bxor with a `bnot` operand to put the operand on
the right. That way the lowerings in the backends only have to list the
rule once, with the operand on the right, to optimize both styles of
input.
* Add commutative lowering rules
* Update cranelift/codegen/src/isa/x64/lower.isle
Co-authored-by: Jamey Sharp <jamey@minilop.net>
---------
Co-authored-by: Jamey Sharp <jamey@minilop.net>
Looks like #5091 wasn't enough and some of the APIs needed updating with
changes made in the meantime. I've updated the action here and
additionally made a separate change where the release isn't continually
created and deleted but instead left alone and only the tag is updated.
This should work for the `dev` release and avoids deleting/recreating on
each PR, sending out notifications for new releases.
A local github action we have has been broken for about a month now
meaning that the `dev` tag isn't getting updated or getting new
releases. This appears to be due to the publication of new versions of
these dependencies which are running into issues using one another. I
think I've figured out versions that work and have added a
`package-lock.json` to ensure we keep using the same versions.
* Upgrade our github actions to "node16"
Each github actions run has a lot of warnings about using node12 so this
upgrades our repository to using node16. I'm hoping no other changes are
needed and I suspect other actions we're using are on node12 and will
need further updates, but this should help pin down what's remaining.
* Update `actions/checkout` workflow to `v3`
* Update to `actions/cache@v3`
* Update to `actions/upload-artifact@v3`
* Drop usage of `actions-rs/toolchain`
* Update to `actions/setup-python@v4`
* Update mdbook version
* ci: replace OpenVINO installer action
To test wasi-nn, we currently use an OpenVINO backend. The Wasmtime CI
must install OpenVINO using a custom GitHub action. This CI action has
not been updated in some time and in the meantime OpenVINO (and the
OpenVINO crates) have released several new versions.
https://github.com/abrown/install-openvino-action is an external action
that we plan to keep up to date with the latest releases. This change
replaces the current CI action with that one.
* wasi-nn: upgrade openvino dependency to v0.4.1
This eliminates a `lazy_static` dependency and changes a few parameters
to pass by reference. Importantly, it enables support for the latest
versions of OpenVINO (v2022.*) in wasi-nn.
* ci: update wasi-nn script to source correct env script
* ci: really use the correct path for the env script
Also, clarify which directory OpenVINO is installed in (the symlink may
not be present).
* Refactor binary-compatible-builds for releases
I was poking around this yesterday and noticed a few things that could
be improved for our release builds:
* The centos container for the x86_64 builds contained a bunch of extra
tooling we no longer need such as python3 and a C++ compiler. Along
with custom toolchain things this could all get removed since the C we
include now is quite simple.
* The aarch64 and s390x cross-compiled builds had relatively high glibc
version requirements compared to the x86_64 build. This was because we
don't use a container to build the cross-compiled binaries. I added
containers here along the lines of the x86_64 build to use an older
glibc to build the release binary to lower our version requirement.
This lower the aarch64 version requirement from glibc 2.28 to 2.17.
Additionally the s390x requirement dropped from 2.28 to 2.16.
* To make the containers a bit easier to read/write I added
`Dockerfile`s for them in a new `ci/docker` directory instead of
hardcoding install commands in JS.
This isn't intended to be a really big change or anything for anyone,
but it's intended to keep our Linux-based builds consistent at least as
best we can.
* Remove temporary change
Updates the OpenVINO backend for wasi-nn to pre-configure all inputs as `NHWC`. This is not a long-term fix but no worse than the status quo, which configures all input tensors to the same layout. This change updates the CI script to use the latest version of OpenVINO. Closes#3948.
Co-authored-by: Andrew Brown <andrew.brown@intel.com>
This commit removes the publish step in GitHub actions, insteading
folding all functionality into the release build steps. This avoids
having a separately scheduled job after all the release build jobs which
ends up getting delayed for quite a long time given the current
scheduling algorithm.
This involves refactoring the tarball assembly scripts and refactoring the
github asset upload script too. Tarball assembly now manages everything
internally and does platform-specific bits where necessary. The upload
script is restructured to be run in parallel (in theory) and hopefully
catches various errors and tries to not stomp over everyone else's work.
The main trickiness here is handling `dev`, which is less critical for
correctness than than tags themselves.
As a small tweak build-wise the QEMU build for cross-compiled builders
is now cached unlike before where it was unconditionally built, shaving
a minute or two off build time.
Fixes a few issues that have been cropping up:
* Update `rustup` on Windows to latest to skip over the 1.24.1 installed
on GitHub Actions which can fail to install.
* Remove the no-longer-needed `define-llvm-env` action
* Install generic llvm/lldb packges instead of specific ones that may
migrate in versions over time.
CentOS 6 just went EOL at the end of November 2020; as of today, the
repository seems to have disappeared, so our CI builds are failing. This
PR updates us to CentOS 7, which should be usable until June 30, 2024.
* Add an initial wasi-nn implementation for Wasmtime
This change adds a crate, `wasmtime-wasi-nn`, that uses `wiggle` to expose the current state of the wasi-nn API and `openvino` to implement the exposed functions. It includes an end-to-end test demonstrating how to do classification using wasi-nn:
- `crates/wasi-nn/tests/classification-example` contains Rust code that is compiled to the `wasm32-wasi` target and run with a Wasmtime embedding that exposes the wasi-nn calls
- the example uses Rust bindings for wasi-nn contained in `crates/wasi-nn/tests/wasi-nn-rust-bindings`; this crate contains code generated by `witx-bindgen` and eventually should be its own standalone crate
* Test wasi-nn as a CI step
This change adds:
- a GitHub action for installing OpenVINO
- a script, `ci/run-wasi-nn-example.sh`, to run the classification example
* Move most wasmtime tests into one test suite
This commit moves most wasmtime tests into a single test suite which
gets compiled into one executable instead of having lots of test
executables. The goal here is to reduce disk space on CI, and this
should be achieved by having fewer executables which means fewer copies
of `libwasmtime.rlib` linked across binaries on the system. More
importantly though this means that DWARF debug information should only
be in one executable rather than duplicated across many.
* Share more build caches
Globally set `RUSTFLAGS` to `-Dwarnings` instead of individually so all
build steps share the same value.
* Allow some dead code in cranelift-codegen
Prevents having to fix all warnings for all possible feature
combinations, only the main ones which come up.
* Update some debug file paths
We don't need full debug information but rather line tables
(debuginfo=1) should suffice for backtraces if truly necessary. Note
that this doesn't actually work on stable Rust just yet due to it being
an unrelease feature of Cargo. With the Rust release next week though
this'll work on all of stable/beta/nightly.
This should save us about 3GB of target directory disk space and it may
also be a tiny speed boost. There's no real benefit to using incremental
builds on CI because we're not changing code anyway!
This will delete a same-name of a previous release for all tags, not
just the dev tag. That way if we need to retry a tagged release we'll
delete it and recreate it as usual.
Closes#978
Alex Crichton
Juniper Tyree
bjorn3
Trevor Elliott
Andrew Brown
Brian Jones
Chris Fallin
Pat Hickey
Yury Delendik