Flaky Test
An automated test that produces inconsistent results, sometimes passing and sometimes failing on the same code without any changes, typically due to timing issues, shared state, or external dependencies.
Common causes of flaky tests include: race conditions and timing-dependent assertions (e.g., waiting for an animation), reliance on external services that are intermittently unavailable, tests that depend on execution order or shared data, non-deterministic data (random values, timestamps), and environment-specific differences (screen resolution, timezone).
Flaky tests are sometimes called 'non-deterministic tests.' Teams address them by quarantining flaky tests into a separate suite, tracking flakiness rates, and fixing root causes. Modern frameworks like Playwright include auto-waiting and retry mechanisms to reduce common flakiness patterns.
Why It Matters for QA Teams
Flaky tests erode trust in the entire test suite. When failures are regularly dismissed as 'just a flaky test,' real bugs slip through unnoticed. Every flaky test makes the pipeline less trustworthy.
Example
An E2E test for the shopping cart intermittently fails because it clicks the 'Add to Cart' button before the JavaScript has finished loading. It passes when the CI server is fast and fails when the server is under load. The fix: replace a hard-coded wait(2000) with Playwright's waitForSelector to wait for the button to be interactive.