This commit parallelizes almost everything that can currently be
parallelized. With that, it also introduces a framework for easily
parallelizing other parts of the compiler.
Code for baremetal targets already compiles slightly faster because it
can parallelize the compilation of supporting assembly files. However,
the speedup is especially noticeable when libraries (compiler-rt,
picolibc) also need to be compiled: they will be compiled in parallel
next to the Go files using all available cores. On my dual core laptop
(4 cores if you count hyperthreading) this cuts compilation time roughly
in half when compiling something for a Cortex-M board after running
`tinygo clean`.
On WebAssembly it is possible to grow the heap with the memory.grow
instruction. This commit implements this feature and with that also
removes the -heap-size flag that was reportedly broken (I haven't
verified that). This should make it easier to use TinyGo for
WebAssembly, where there was no good reason to use a fixed heap size.
This commit has no effect on baremetal targets with optimizations
enabled.
Be able to run `tinygo gdb -target=arduino examples/serial` and debug a
program with the power of a real debugger.
Note that this only works on LLVM 11 because older versions have a bug
in the AVR backend that cause it to produce invalid debug information:
https://reviews.llvm.org/D74213.
This commit allows debugging like the following:
GOARCH=arm tinygo gdb ./testdata/alias.go
This can be very useful to debug issues on a different instruction set
architecture but still on a host system.
I tested the following 7 configurations to make sure it works and I
didn't break anything:
GOOS=amd64
GOOS=386
GOOS=arm
GOOS=arm64
tinygo gdb -target=hifive1-qemu
tinygo gdb -target=cortex-m-qemu
tinygo gdb -target=microbit
For example, for running tests with -target=wasm or
-target=cortex-m-qemu. It looks at the output to determine whether tests
were successful in the absence of a status code.
This patch adds the `tinygo targets` command, which lists usable
targets (targets that can be used in the `-target` flag).
The assumption here is that usable targets can either be flashed or
emulated by TinyGo. There is one exception where it doesn't work yet:
the nintendoswitch target. Right now it requires some manual steps to
build a .nro file which can then be run by yuzu, hence why it doesn't
show up in the list.
This is necessary for an upcoming VS Code extension to support TinyGo,
and may be useful for other people wanting to use proper autocompletion
etc in their IDE.
Test binaries must be run in the source directory of the package to be
tested. This wasn't done, leading to a few "file not found" errors.
This commit implements this. Unfortunately, it does not allow more
packages to be tested as both affected packages (debug/macho and
debug/plan9obj) will still fail with this patch even though the "file
not found" errors are gone.
There were a few problems with the go/packages package. While it is more
or less designed for our purpose, it didn't work quite well as it didn't
provide access to indirectly imported packages (most importantly the
runtime package). This led to a workaround that sometimes broke
`tinygo test`.
This PR contains a number of related changes:
* It uses `go list` directly to retrieve the list of packages/files to
compile, instead of relying on the go/packages package.
* It replaces our custom TestMain replace code with the standard code
for running tests (generated by `go list`).
* It adds a dummy runtime/pprof package and modifies the testing
package, to get tests to run again with the code generated by
`go list`.
For now, this is just an extra flag that can be used to print stack
frame information, but this is intended to provide a way to determine
stack sizes for goroutines at compile time in many cases.
Stack sizes are often somewhere around 350 bytes so are in fact not all
that big usually. Once this can be determined at compile time in many
cases, it is possible to use this information when available and as a
result increase the fallback stack size if the size cannot be determined
at compile time. This should reduce stack overflows while at the same
time reducing RAM consumption in many cases.
Interesting output for testdata/channel.go:
function stack usage (in bytes)
Reset_Handler 332
.Lcommand-line-arguments.fastreceiver 220
.Lcommand-line-arguments.fastsender 192
.Lcommand-line-arguments.iterator 192
.Lcommand-line-arguments.main$1 184
.Lcommand-line-arguments.main$2 200
.Lcommand-line-arguments.main$3 200
.Lcommand-line-arguments.main$4 328
.Lcommand-line-arguments.receive 176
.Lcommand-line-arguments.selectDeadlock 72
.Lcommand-line-arguments.selectNoOp 72
.Lcommand-line-arguments.send 184
.Lcommand-line-arguments.sendComplex 192
.Lcommand-line-arguments.sender 192
.Lruntime.run$1 548
This shows that the stack size (if these numbers are correct) can in
fact be determined automatically in many cases, especially for small
goroutines. One of the great things about Go is lightweight goroutines,
and reducing stack sizes is very important to make goroutines
lightweight on microcontrollers.
This commit replaces the existing ad-hoc package loader with a package
loader that uses the x/tools/go/packages package to find all
to-be-loaded packages.
This commit changes the way that packages are looked up. Instead of
working around the loader package by modifying the GOROOT variable for
specific packages, create a new GOROOT using symlinks. This GOROOT is
cached for the specified configuration (Go version, underlying GOROOT
path, TinyGo path, whether to override the syscall package).
This will also enable go module support in the future.
Windows is a bit harder to support, because it only allows the creation
of symlinks when developer mode is enabled. This is worked around by
using symlinks and if that fails, using directory junctions or hardlinks
instead. This should work in the vast majority of cases. The only case
it doesn't work, is if developer mode is disabled and TinyGo, the Go
toolchain, and the cache directory are not all on the same filesystem.
If this is a problem, it is still possible to improve the code by using
file copies instead.
As a side effect, this also makes diagnostics use a relative file path
only when the file is not in GOROOT or in TINYGOROOT.
This is needed to make it available to more packages, for caching
purposes.
For caching, the version itself may not be enough during development.
But for regular releases, the version provides some protection against
accidentally using a cache entry that is invalid in a newer version.
This is necessary to avoid a circular dependency in the loader (which
soon will need to read the Go version) and because it seems like a
better place anyway.
This commit also adds a bit of version independence, in particular for
external commands. It also adds the LLVM version to the `tinygo version`
command, which might help while debugging.
This is necessary because LLVM defines many options in global variables
that are modified when invoking Clang. In particular, LLVM 10 seems to
have a bug in which it always sets the -pgo-warn-misexpect flag. Setting
it multiple times (over various cc1 invocations) results in an error:
clang (LLVM option parsing): for the --pgo-warn-misexpect option: may only occur zero or one times!
This is fixed by running the Clang invocation in a new `tinygo`
invocation.
Because we've had issues with lld in the past, also run lld in a
separate process so similar issues won't happen with lld in the future.
This is necessary for better CGo support on bare metal. Existing
libraries expect to be able to include parts of libc and expect to be
able to link to those symbols.
Because with this all targets have a working libc, it is now possible to
add tests to check that a libc in fact works basically.
Not all parts of picolibc are included, such as the math or stdio parts.
These should be added later, when needed.
This commit also avoids the need for the custom memcpy/memset/memcmp
symbols that are sometimes emitted by LLVM. The C library will take care
of that.
This refactor makes adding a new library (such as a libc) much easier in
the future as it avoids a lot of duplicate code. Additionally, CI should
become a little bit faster (~15s) as build-builtins now uses the build
cache.
This commit switches integration tests to use the same error reporting
mechanism as the tinygo compiler normally uses. It replaces errors like
this:
main_test.go:139: failed to build: interp: branch on a non-constant
With this:
main.go:693: # math/rand
main.go:695: interp: branch on a non-constant
In this particular case the error isn't much better (it gives the
relevant package, though) but other errors should also include the
source location where they happen.
Ayke van Laethem
Fauchon
Takeshi Yoneda
sago35
deadprogram
cebernardi
Jaden Weiss
Elliott Sales de Andrade
Dmitri Goutnik