mirror of
https://github.com/angular/angular
synced 2026-05-24 09:28:37 +00:00
b52a6264ac
I've had some success asking the Antigravity agent to find flaky tests and propose fixes for them, then just running it in the background and reviewing what it finds. Upstreaming this to the repo so others can use it, since it includes helpful notes like `--runs_per_test` and leveraging random seeds as well as an iteration loop.
I opted not to have the agent do anything with PRs just yet, but if this is useful and we build confidence in it, we can explore that in the future.
(cherry picked from commit 337e6e7d6e)
2.6 KiB
2.6 KiB
| description |
|---|
| Find and fix flaky tests in the repository |
Investigate flaky tests in the repo and propose fixes to improve stability. High-level process:
- Run tests in the repo to look for flakes.
- Consider using Bazel's
--runs_per_testflag to easily find flakes. - Be cognizant of not exhausting all the resources on the current
machine, run a subset of tests at a time such as
bazel test //packages/core/....
- Consider using Bazel's
- Once you find some flakes, focus on one at a time.
- Create a new branch named
flakes/${relevantNameFromTest}. - Reproduce the flake to the best of your ability.
- Consider using
--test_env JASMINE_RANDOM_SEED=1234to replicate the broken test ordering.
- Consider using
- Debug the test to understand the failure mode.
- Consider temporarily disabling / skipping other tests with
xitandfitto narrow down where the flake might be coming from if multiple tests are influencing each other. - Consider temporarily ignoring Firefox tests with
--test_tag_filters -firefoxif the flake does not appear to be browser specific. - Consider using
--test_sharding_strategy disabledto run the test in a single shard. - Try to understand why the test was flaky, not just why it failed. Understanding the inconsistency is important to finding the correct fix.
- Consider temporarily disabling / skipping other tests with
- Attempt a fix and validate with
--runs_per_test.- Iterate on the fix until you have something which appears to work.
- If you find yourself stuck and not making meaningful progress, note down what you've learned/where you're struggling, commit what you have, look for another flake to fix, and continue. At the end, surface to the user what you failed to fix.
- Don't try to make significant changes to Angular's runtime behavior, focus just on making the test pass/fail consistently.
- Commit the change with relevant details in the commit message and
move on to the next test.
- Be sure to include your theory of why the test was flaky and how this fix eliminates or reduces that flakiness.
- Iterate as many times as the user requests you to (default 5 branches if not otherwise specified).
- Once you can't find any flaky tests or have iterated as many times as requested, stop and inform the user what you found and fixed.
Additional notes:
- Multiple fixes including the same/related files can go in the same commit or multiple commits on the same branch.
- Distinct test fixes should go in different branches, make a new one for each investigation.
- You may push these branches to
origin, but do not create PRs for them.