Killing Flakiness
A flaky test passes and fails on the same code, with no change in between. It is the single most damaging thing that can happen to a test suite — worse than a test that's simply wrong.
Why worse? Because a flaky suite trains your team to ignore red. Once "just re-run it, it's probably flaky" becomes a reflex, a red build stops meaning anything — and the one time it's a real bug, it sails through. The purpose of a suite is trust, and flakiness is how trust dies.
This lesson uses ShopKart's toggleable bug scenarios to make flakiness happen on demand, then kill it properly.
What flaky looks like: the hard wait
The most common cause of flakiness by far is a hard wait — pausing for a fixed number of seconds and hoping the app is ready.
ShopKart gives us a perfect victim: turn on the slow-inventory bug and the catalog takes about 3.5 seconds to render after login (a deliberate delay you toggle with ?bugs=slow-inventory). Here's the naive test:
// ANTI-PATTERN — do not do this
test('slow catalog loads (flaky)', async ({ page }) => {
await page.goto('/shopkart/login');
await page.locator('[data-test="username"]').fill('student');
await page.locator('[data-test="password"]').fill('crio123');
await page.locator('[data-test="login-button"]').click();
// turn the slow-inventory bug on for this navigation
await page.goto('/shopkart/inventory?bugs=slow-inventory');
await page.waitForTimeout(2000); // "2 seconds should be enough"... it isn't
await expect(page.locator('[data-test="inventory-item"]')).toHaveCount(12);
});This test is a coin flip. The catalog needs ~3.5 seconds; we waited 2. It fails today. So you bump it to waitForTimeout(5000) and now it "works" — but you've made every run of this test 5 seconds slower, and the day the app takes 5.5 seconds under load, it flakes again. Hard waits are always either too short (flaky) or too long (slow). There is no correct number.
> 🎯 Practice: feel the flake. Open [/shopkart/inventory?bugs=slow-inventory](https://resources.criodo.com/shopkart/inventory?bugs=slow-inventory) (log in first), then write the flaky version above and the fixed version below and watch the difference. The starter kit's challenges.spec.ts has this exact exercise stubbed for you.
The fix: web-first assertions that retry
Modern tools don't make you guess. A web-first assertion re-checks its condition automatically until it's true or a timeout is hit. You don't wait for a duration — you wait for a condition.
This is the same test, done right. It passes against the live app, and it's as fast as the app allows — no wasted seconds:
import { test, expect } from '@playwright/test';
import { LoginPage } from '../pages/LoginPage';
import { InventoryPage } from '../pages/InventoryPage';
// With slow-inventory on, the catalog renders ~3.5s late. No sleep: the
// assertions retry until the condition holds or expect.timeout (10s) is hit.
test('slow catalog still loads (auto-retrying)', async ({ page }) => {
const login = new LoginPage(page);
const inventory = new InventoryPage(page);
await login.goto();
await login.login();
await page.goto('/shopkart/inventory?bugs=slow-inventory');
await expect(page).toHaveURL(/inventory/); // retries until the URL is right
await expect(inventory.items).toHaveCount(12); // retries until 12 items exist
});The rule: assert on the condition you actually care about, and let the tool poll. toHaveText, toBeVisible, toHaveCount, toHaveURL all retry. waitForTimeout never should appear in a real test.
> Playwright's actions (click, fill) also auto-wait for the element to be actionable — visible, enabled, stable — before acting. Between auto-waiting actions and auto-retrying assertions, a correctly written Playwright test almost never needs an explicit wait at all. (Selenium users: this is what explicit WebDriverWait + expected conditions buy you — never Thread.sleep.)
The five horsemen of e-commerce flakiness
Hard waits are the most common cause, but not the only one. Here's where flakiness actually comes from in a store, and the fix for each:
| Source | What it looks like | Fix |
|---|---|---|
| Timing | Assert runs before the async cart/price updates | Web-first assertions (above) |
| Test-data pollution | Test B fails because Test A left a coupon/order behind | Data isolation — each test owns its data (Lesson 3) |
| Order dependence | Suite passes in order, fails when parallelized/shuffled | Never let one test depend on another running first |
| Shared account state | Two tests log in as the same user and stomp each other's cart | A user/session per worker, or reset state in setup |
| Animation / network races | Click lands mid-transition; response arrives late | Auto-waiting actions; mock slow third-party calls |
Notice that three of the five are really the same disease from Lesson 3 — shared mutable state. Isolation isn't just tidiness; it's your primary anti-flake defense.
Make failures reproducible
You can't fix what you can't see. When a test does fail, you want a recording, not a guess. That's why our playwright.config.ts from Lesson 3 sets:
use: {
trace: 'retain-on-failure', // full step-by-step trace, kept only when a test fails
screenshot: 'only-on-failure', // screenshot at the moment of failure
}A trace lets you scrub through every action with DOM snapshots and network logs, so you can see exactly what the page looked like when the assertion fired. For a flaky failure that only shows up 1 run in 20 on CI, this is often the only way to catch it in the act.
Retries: a scalpel, not a bandage
Test runners let you auto-retry a failed test. This is genuinely useful — and genuinely dangerous.
Legitimate use: absorbing rare, external non-determinism you don't control — a third-party payment sandbox that occasionally times out, a CI runner with a noisy network. A small retry count keeps a real, unrelated hiccup from failing the build.// playwright.config.ts — retry on CI only, never mask flake locally
export default defineConfig({
retries: process.env.CI ? 2 : 0,
});The discipline: retry only after you've diagnosed the cause and confirmed it's external. A test that needs retries to pass is a bug report you haven't written yet. Track your flaky tests, quarantine them, and fix the cause — don't let the retry count quietly climb.
A debugging checklist for a flaky test
When a test flakes, work down this list:
- Is there a
waitForTimeout/sleep? Replace it with a web-first assertion. (Fixes most cases.) - Does it pass alone but fail in the suite? Data pollution or order dependence — check what state it assumes.
- Does it share an account/session with another test? Give it its own.
- Read the trace. Look at the exact DOM and network state at the moment of failure.
- Only now, if the cause is provably external, add a bounded retry — and write down why.
Key takeaways
- A flaky test passes and fails on identical code; its real damage is destroying trust in the whole suite.
- Hard waits (
waitForTimeout) are the top cause — always too short (flaky) or too long (slow). Replace them with web-first assertions that retry until the condition holds. - Most remaining flakiness is shared mutable state — data pollution, order dependence, shared accounts. Isolation is the cure.
- Turn on traces and screenshots on failure so a rare CI flake is reproducible.
- Retries are for provably external non-determinism only — never to mask a test you made flaky, or a real bug in the app.
Next lesson
You've been driving the browser for every test. Next we drop below the UI: API testing and network interception — faster, stabler tests against the HTTP layer, and mocking the network to control exactly what the page receives.