Switching to a shared semaphore allows multi-build operations (compiler tests, package tests, etc.) to use the expected degree of parallelism efficiently.
While refactoring the job runner, the time complexity was also reduced from O(n^2) to O(n+m) (where n is the number of jobs, and m is the number of dependencies).
Instead of keeping a slice of jobs to run, let the runJobs function
determine which jobs should be run by investigating all dependencies.
This has two benefits:
- The code is somewhat cleaner, as no 'jobs' slice needs to be
maintained while constructing the dependency graph.
- Eventually, some jobs might not be required by any dependency.
While it's possible to avoid adding them to the slice, the simpler
solution is to build a new slice from the dependencies which will
only include required dependencies by design.
Add a 'result' member to the compileJob struct which is used by the link
job to get all the paths that should be linked together. This is not yet
necessary (the paths are fixed), but soon the paths are only known after
a linker dependency has run.
This avoids external commands from finishing after the TinyGo command
exits. For example, when testing out compiler-rt on AVR, I got the
following error:
$ go install; and tinygo run -target=atmega1284p ./testdata/calls.go
[... Clang error removed ...]
error: failed to build /home/ayke/src/github.com/tinygo-org/tinygo/lib/compiler-rt/lib/builtins/extendsfdf2.c: exit status 1
error: unable to open output file '/tmp/tinygo361380649/build-lib-compiler-rt/ffsdi2.c.o': 'No such file or directory'
1 error generated.
That last error ("unable to open output file") is a spurious error
because the temporary directory has already been removed. This commit
waits until all running jobs have finished before returning, so that
these errors won't happen.
This commit parallelizes almost everything that can currently be
parallelized. With that, it also introduces a framework for easily
parallelizing other parts of the compiler.
Code for baremetal targets already compiles slightly faster because it
can parallelize the compilation of supporting assembly files. However,
the speedup is especially noticeable when libraries (compiler-rt,
picolibc) also need to be compiled: they will be compiled in parallel
next to the Go files using all available cores. On my dual core laptop
(4 cores if you count hyperthreading) this cuts compilation time roughly
in half when compiling something for a Cortex-M board after running
`tinygo clean`.
Damian Gryski
Nia Waldvogel
Ayke van Laethem