What to Automate (and What Not To)
You already know a tool. You can make a browser click a button and assert some text. This course is about the harder question that no tool teaches you:
*Of the thousands of things you could test, which ones are worth automating?
Get this wrong and you build a suite that takes 40 minutes to run, breaks every other day, and that your team eventually starts ignoring. Get it right and you build a small suite that catches real bugs and that people trust. This lesson is about getting it right.
Our practice app: the ShopKart Test Range
Every code example in this course runs against the [ShopKart Test Range](https://resources.criodo.com/shopkart) — a Crio-built e-commerce app made specifically for practicing automation. It has the full flow: log in → browse products → add to cart → checkout → confirmation. It's stable and deterministic, so your assertions stay reliable, and every element has a clean data-test attribute to target.
The best part: you can read a lesson, then run the exact same thing yourself. The Test Range gives you three ways to practice:
- The store — log in at [
/shopkart/login](https://resources.criodo.com/shopkart/login) with usernamestudent/ passwordcrio123(trylocked_userto practice a login-failure path). - The Locator Playground — practice selectors in-browser and get an instant robustness score.
- A downloadable Playwright starter kit — a ready-made project pointed at the app, so
npm testgoes green in two minutes and you start writing your own tests.
And critically for a practitioner's course, ShopKart can break on command. A floating control panel (or a ?bugs=… URL) injects real, reproducible bugs — a broken checkout field, a wrong total, a slow-loading catalog — which we'll use to practice flakiness and bug reporting later.
For the deeper test-data lesson we'll also use the full ShopKart dataset — the seeded data behind the store: 12,480 order line items, 1,500 customers, and 600 products. The storefront shows a curated slice; the dataset is the whole thing. More on that in Lesson 3.
> Try it now: open [the ShopKart Test Range](https://resources.criodo.com/shopkart/login), log in with student / crio123, and add the Mi USB-C Cable (₹185.00) to your cart. That's the flow we'll automate first.
The only question that matters: is this test worth it?
Every automated test has a cost that never goes away. You pay it three times:
- To write it — once.
- To run it — every single CI build, forever.
- To maintain it — every time the UI changes and the test breaks.
A test earns its keep only when the value it returns beats that ongoing cost. Value comes from two things: how likely is this path to break, and how much does it hurt when it does.
A useful way to score a candidate test:
value = (probability it catches a real bug) × (cost of that bug reaching a user)
cost = time to write + (time to run × number of runs) + expected maintenance
automate if value > costYou won't literally compute this. But it explains every good instinct:
- Checkout and payment — high breakage risk, catastrophic if broken. Automate first.
- Login — runs before almost everything else, cheap, breaks loudly. Automate.
- A one-off admin report only one internal user opens — low value, high churn. Don't. Test it by hand.
- Exact pixel position of a banner — changes constantly, rarely a real bug. Don't automate as a functional test.
The mistake beginners make is automating what's easy to automate. The mistake experienced engineers avoid is automating what isn't worth it.
The test pyramid, applied to e-commerce
You've probably seen the pyramid. Here's what it actually means for a store like ShopKart.
/\
/ \ E2E / UI (few) — login → cart → checkout → confirmation
/----\
/ \ Integration (some) — cart API returns correct totals + tax
/--------\
/ \ Unit (many) — tax calculator, discount rules, price formatting
/____________\- Unit tests are fast (milliseconds), stable, and precise. If the tax on a ₹185.00 item should be ₹14.80, a unit test on the tax function proves it in isolation — no browser, no login.
- Integration tests check that two pieces talk correctly — e.g. the cart endpoint returns the right subtotal and tax for a basket. (You'll test ShopKart's API directly at this layer in Lesson 5.)
- E2E / UI tests drive the real browser through the whole journey. They're slow and the most fragile, but they're the only ones that prove the actual customer flow works end to end.
The pyramid is wide at the bottom for a reason: push each check to the lowest, cheapest layer that can still catch the bug. Don't write a 30-second browser test to verify tax math you could verify in a 2-millisecond unit test. Save the browser for what only the browser can prove.
> The anti-pattern — the "ice cream cone": a suite that's mostly slow UI tests with almost no unit tests underneath. It's top-heavy, slow, and flaky. If your suite looks like an upside-down pyramid, that's the single biggest thing to fix.
Three suites, three jobs
Not all automated tests run at the same time or for the same reason. Split them by purpose:
| Suite | What's in it | When it runs | Target time |
|---|---|---|---|
| Smoke | The 3–5 tests that prove the app is alive: can I log in? does the catalog load? can I reach checkout? | On every commit / every deploy | Under 1 minute |
| Regression | The full set: every important flow, edge case, and past bug | Nightly, or before release | Minutes, not hours |
| E2E critical path | The complete money-making journey, start to finish | Before every production deploy | A few minutes |
Our very first test — "can the student account log in and see the catalog?" — is a smoke test. If that fails, nothing else matters and you don't even bother running the rest.
Practice: write the plan before the code
Before automating the checkout flow, a practitioner writes a short plan. Here's one for ShopKart's checkout:
FEATURE: Guest checkout (single item)
Critical path (E2E, must automate):
- Log in → add a product → cart shows 1 item → checkout →
fill name + postal code → total is correct → order confirmed
Negative cases (automate, cheap and high-value):
- Checkout with missing first name → error shown
- Checkout with missing last name → error shown
- Checkout with missing postal code → error shown
Push down the pyramid (do NOT automate via the browser):
- Tax math (₹185.00 → ₹14.80) → unit test on the tax function
- Price formatting ("₹185.00") → unit test
Don't automate at all (manual / exploratory):
- Exact color of the "Add to cart" button
- One-time promotional banner copyNotice what this plan does: it names the one critical E2E path, adds a few cheap high-value negatives, explicitly pushes math down to unit tests, and explicitly refuses to automate the low-value stuff. That refusal is a decision, not an oversight — write it down so nobody "fixes" it later.
Key takeaways
- Automate a test only when value beats lifetime cost — probability of catching a real bug × cost of that bug, versus write + run + maintain.
- The pyramid says push every check to the cheapest layer that can still catch the bug. Save the browser for the true end-to-end journey.
- Avoid the ice-cream cone: a suite that's mostly slow, flaky UI tests.
- Split tests into smoke (every commit), regression (nightly), and critical-path E2E (every deploy) — each has a different job and a different time budget.
- Write the automation plan before the code, and explicitly list what you're choosing not* to automate.
Next lesson
We have a plan. Next we'll turn one line of it — a plain requirement like "checkout must reject a missing last name" — into concrete, prioritized, data-driven test cases, and write the first real tests against the Test Range.