Reworked how we handle server/worker delays to fix flaky tests.