Stable screenshots through fixed fonts, frozen time, and masked regions. A test suite is only valuable if the team trusts it. The moment people start re-running failures without reading them, the suite has stopped protecting anyone.
Tests as documentation
Reliable tests control everything outside the code under test: time, randomness, network, and fonts. Once those are fixed, a failure means something changed in the code, which is exactly what we want to know.
import { test, expect } from '@playwright/test';
test('guest can complete checkout', async ({ page }) => {
await page.clock.setFixedTime(new Date('2024-05-01T10:00:00Z'));
await page.goto('/cart');
await page.getByRole('button', { name: 'Checkout' }).click();
await page.getByLabel('Email').fill('guest@example.com');
await page.getByRole('button', { name: 'Place order' }).click();
await expect(page.getByRole('heading', { name: 'Thank you' })).toBeVisible();
});
Making the suite reliable
We track three numbers for the suite: total duration, flaky failure rate, and time from failure to fix. Coverage is useful for finding untested areas, but it is not a goal on its own.
- Freeze time and seed randomness in tests.
- Quarantine flaky tests with an owner and a deadline.
- Keep end-to-end tests for critical journeys.
- Read every failure before re-running.
What we measure
End-to-end tests are expensive, so we keep them for the journeys that make money or lose data. Everything else is tested closer to the code, where failures are faster and easier to read.
We will revisit this in six months with fresh numbers. If the results change, we will update the post and note what changed at the end.
If you try this on your own project, start with one page or one service rather than the whole codebase. A small experiment gives you real numbers to argue with, and it is easy to roll back if the idea does not fit.
We have been running this approach for several months now. The initial setup took about a day, and the ongoing cost is close to zero once the team knows where to look.
None of this is new, and that is the point. The techniques that last are usually the boring ones that are documented well and supported by the platform itself.
As always, the examples are simplified to fit on a page. The production version has more error handling and logging, but the shape of the code is the same.
Before you ship this
Retries hide flakiness instead of fixing it. If you enable them, also report every test that needed a retry so the list gets shorter over time.
Premature optimization is the root of all evil.
Donald Knuth, Structured Programming with go to Statements, 1974
Wrapping up
None of this is new, and that is the point. The techniques that last are usually the boring ones that are documented well and supported by the platform itself.
Is there a repo with the full example? Happy to open a PR with a TypeScript version.
The renaming spree is a rite of passage.
We tried something similar in 2023 and rolled it back. Curious whether the tooling has caught up since.
Tried it on a side project over the weekend and it worked first time, which never happens.
Came for the snippet, stayed for the explanation.
We run this in production at a fairly large scale and can confirm the caveats are real.
How does this hold up on older Safari versions? That is where we always get burned.