New series: what assistive tech really hears on your checkout page. Start with part one →
Shipwright
Home
Testing Jun 10, 2026 2 min read

Visual Regression Tests That Do Not Cry Wolf

Sam Whitfield

Sam Whitfield

Engineering manager and recovering QA lead.…

Share
Cover illustration for the article Visual Regression Tests That Do Not Cry Wolf

Stable screenshots through fixed fonts, frozen time, and masked regions. A test suite is only valuable if the team trusts it. The moment people start re-running failures without reading them, the suite has stopped protecting anyone.

Tests as documentation

Reliable tests control everything outside the code under test: time, randomness, network, and fonts. Once those are fixed, a failure means something changed in the code, which is exactly what we want to know.

checkout.spec.ts
import { test, expect } from '@playwright/test';

test('guest can complete checkout', async ({ page }) => {
  await page.clock.setFixedTime(new Date('2024-05-01T10:00:00Z'));
  await page.goto('/cart');
  await page.getByRole('button', { name: 'Checkout' }).click();
  await page.getByLabel('Email').fill('guest@example.com');
  await page.getByRole('button', { name: 'Place order' }).click();
  await expect(page.getByRole('heading', { name: 'Thank you' })).toBeVisible();
});

Making the suite reliable

We track three numbers for the suite: total duration, flaky failure rate, and time from failure to fix. Coverage is useful for finding untested areas, but it is not a goal on its own.

  • Freeze time and seed randomness in tests.
  • Quarantine flaky tests with an owner and a deadline.
  • Keep end-to-end tests for critical journeys.
  • Read every failure before re-running.

What we measure

End-to-end tests are expensive, so we keep them for the journeys that make money or lose data. Everything else is tested closer to the code, where failures are faster and easier to read.

We will revisit this in six months with fresh numbers. If the results change, we will update the post and note what changed at the end.

If you try this on your own project, start with one page or one service rather than the whole codebase. A small experiment gives you real numbers to argue with, and it is easy to roll back if the idea does not fit.

We have been running this approach for several months now. The initial setup took about a day, and the ongoing cost is close to zero once the team knows where to look.

None of this is new, and that is the point. The techniques that last are usually the boring ones that are documented well and supported by the platform itself.

As always, the examples are simplified to fit on a page. The production version has more error handling and logging, but the shape of the code is the same.

Before you ship this

Retries hide flakiness instead of fixing it. If you enable them, also report every test that needed a retry so the list gets shorter over time.

Premature optimization is the root of all evil.

Donald Knuth, Structured Programming with go to Statements, 1974

Wrapping up

None of this is new, and that is the point. The techniques that last are usually the boring ones that are documented well and supported by the platform itself.

Sam Whitfield

Sam Whitfield

Author

Engineering manager and recovering QA lead. Sam writes about testing, code review, and building teams that ship calmly.

Related Posts

Comments 7

Lena Novak

Is there a repo with the full example? Happy to open a PR with a TypeScript version.

Sam Whitfield Author

The renaming spree is a rite of passage.

Diego Alvarez

We tried something similar in 2023 and rolled it back. Curious whether the tooling has caught up since.

Sofia Rossi

Tried it on a side project over the weekend and it worked first time, which never happens.

Alex Moreno

Came for the snippet, stayed for the explanation.

Mara Okafor Member

We run this in production at a fairly large scale and can confirm the caveats are real.

Ben Carter

How does this hold up on older Safari versions? That is where we always get burned.

Leave a comment

Your email will not be published. Sign in for a faster experience.

Supports **bold**, *italic*, and `code`.