go-libp2p/docs/flaky-tests.md

Debugging Flaky Tests

If a test is flaky in CI it's probably because there's some timing issue. The test probably depends on some Go routine making progress in the background and polling to see if the expected outcome is achieved.

This will pretty much always work locally because your local machine is likely pretty capable and there isn't too many concurrent processes running. In CI, we are susceptible to both slower hardware and noisier neighbors. However we can mimic this environment locally with cgroups.

Replicating noisy neighbors

We can limit the amount of CPU time relative to real time a process gets with cgroups. This lets us replicate the environment where many other neighboring processes are vying for CPU time.

  # Compile some test we want to run. We do this outside the cgroup so this is
  # fast
  go test -c ./p2p/host/autorelay

  # Create the group
  sudo cgcreate -g cpu:/cpulimit

  # Limit the time to 10,000 microseconds for every 1s
  sudo cgset -r cpu.cfs_quota_us=10000 cpulimit
  sudo cgset -r cpu.cfs_period_us=1000000 cpulimit

  # Run a shell with in our limited environment
  sudo cgexec -g cpu:cpulimit bash

  # In the shell, run the test
  ./autorelay.test -test.v

Flakiness with coverage profile

Sometimes adding the -coverprofile=module-coverage.txt introduces flaky behavior since it adds another goroutine to the mix. If you're having trouble reproducing a flaky test, try enabling this flag.