Design system: one line, four hundred screenshots

Part 4 of a five part series on building a design system.

We rebranded the product. From the design system’s point of view it was mostly a change of values in two theme files, because no component in the library knows what colour it is. They know they use surface.brand.

That is the good news, and I wrote about why in an earlier post in this series. Here is the bad news that comes attached to it: if changing two files can restyle your entire product, then changing two files can break your entire product, and unit tests will be entirely green while it happens.

This is the structural problem with a design system that nobody warns you about. You have built a single point of visual failure and handed it to every team in the company.

Why normal testing does not help

Our unit tests are good. They are also, by policy, blind to this class of problem.

The rule we settled on is that unit tests cover interactive behaviour only. Does the dropdown open. Does the form submit. Does the disabled button refuse to fire. They deliberately do not assert on styling, because a test that asserts a padding value is a test that fails every time a designer changes their mind, which trains people to update assertions without reading them.

So styling is not covered by unit tests on purpose. Something else has to cover it, and the only thing that can is a picture.

Level one: the library

The first level is straightforward in principle. Every component in the library has stories covering its variants and states, every story is screenshotted, and every screenshot is compared against a committed baseline. We have about 1,430 of those baselines.

The obvious way to build that corpus is one story per combination, and it collapses immediately. Our Button has three variants, five sizes, several tones, plus pill, full width, busy and disabled, and that is before asking what any of them look like on hover or focus. Enumerated individually that is thousands of near-identical screenshots, which costs CI time to produce and, far worse, produces a review queue no human will ever read honestly.

So the stories are built with a Cartesian helper. You give it a component, the prop values you care about, and a list of pseudo-selectors. It computes the product and renders the whole matrix as a labelled grid:

<Cartesian
  component={Button}
  props={{
    size: ['large', 'medium', 'small', 'x-small'],
    fullWidth: [undefined, true],
  }}
  selectors={[':hover', ':focus-visible', ':disabled']}
/>

One story. One screenshot. Thirty-two rendered states, each labelled with the props that produced it, so a diff points at the exact combination that moved.

The pseudo-selectors need a trick, because a screenshot has no cursor and cannot hover. A wrapper reads the document’s stylesheets, finds the rules whose selector matches the element’s own class name plus the pseudo-state, and copies those declarations onto the element as inline styles. :disabled is easier, since it can just set the prop. The whole technique depends on the CSS existing statically in the stylesheet at render time, which it does because our styles are extracted at build time. A runtime styling library that injects rules lazily would make this considerably harder.

This is also where the Storybook investment pays a second time. The stories were written to document the component, and they become the visual test corpus at no extra cost. A component with thorough stories gets thorough visual coverage automatically, which is a much better incentive than asking people to write screenshot tests.

We use Lost Pixel, self-hosted, with baselines committed to the repository and rendering done in Docker so the output does not depend on whose laptop ran it. There are good hosted options, and I will come back to why we did not take one.

Level two: the problem nobody solves for you

Level one tells you your Button still looks like a Button. It tells you nothing about whether the payments flow still looks right, and that is where the money is.

Uber’s Base Web team wrote the clearest description of this I have found. They make a change to the library, verify it looks fine in their own storybook, and have no idea what it did to the hundreds of internal projects consuming them, short of running each one by hand. Their answer was for each dependent project to maintain a small suite of its own scenarios, re-verified when the library version bumps.

That is a good answer and it is a manual one. Somebody has to remember to write the scenarios, and somebody has to remember to run them.

We went at it from the other side. The question we wanted answered automatically was: given these changed files, which screens could possibly look different?

That is a dependency graph question. So we built a small package that traces the import graph, takes the set of files changed against the main branch, and computes which story files are transitively affected, in the library and in the consuming application separately. It returns two filters.

Day to day that means: change a component, run the visual tests, choose “only relevant files based on VCS changes”. It screenshots the library stories your change could have touched. Then, if the change reached further than the library, it signs off with this:

⚠️ Your changes seem to affect the app. Please run visual tests for the app using this filter:

  transactions-list|payment-review|balances-summary|card-details

Four screens, named, because those are the ones that actually import what you touched. Not the whole app, and not nothing.

That warning is the single most useful thing in our testing setup, and it is about eight lines of code sitting on top of the dependency trace. It converts “I hope this did not break anything” into a specific, checkable list.

The flakiness, which is the actual work

Everything above is the architecture, and the architecture was maybe a fifth of the effort. The rest was making screenshots deterministic, which is unglamorous and is where visual regression testing usually dies.

Our configuration file is about ninety lines. Roughly sixty of them are comments explaining why a line is there. Three are worth telling.

Fonts. A screenshot taken before the web font applies is a different screenshot. We wait on document.fonts.ready before capturing, and we serve the font from local disk in the test environment rather than fetching it, so there is no network round trip to be slow at the wrong moment.

Never wait for network idle. This is the counterintuitive one. The standard advice for screenshot stability is to wait until the network goes quiet. Against a Storybook dev server, the hot module reload socket never goes quiet, so a network idle wait does not stabilise anything. It burns its entire timeout on every single shot and then proceeds anyway. We wait for the font and nothing else, because the font is the only asynchronous dependency these shots have.

The mask that silently did nothing. This is my favourite bug in the whole project.

Several of our components have a scroll indicator: a small gradient that fades in when there is more content below. Its opacity is set by JavaScript after mount, driven by resize and mutation observers, so at the moment a screenshot is taken it might be at 0, or 0.3, or 1. Non-deterministic by construction.

The fix is to hide it during screenshots. So we masked it with a CSS rule targeting its class name:

[class*="scrollableContentIndicator"] { opacity: 0 !important; }

This worked perfectly on my machine and did absolutely nothing in CI.

Our styling compiles class names at build time. In development the build is configured to emit readable identifiers, so the class really is scrollableContentIndicator_a1b2c3. In CI, Storybook is built in production mode, where identifiers are short opaque hashes. The selector matched nothing. It failed silently, because a CSS rule that matches no elements is not an error, and we carried on believing the flakiness was fixed for as long as the shots happened not to catch the gradient mid-fade.

The fix was to stop keying off a build-dependent artefact and mask on a data-scrollable-content-indicator attribute instead, which is the same in every build. The general lesson has outlived the bug: never write a test selector against something your build tool is allowed to rename.

There is a deliberate exception, which I like because it shows the limits of the approach. Four stories exist specifically to prove that gradient renders over overflowing content. Masking it there would mean testing that a feature is invisible. Those four are excluded from the mask by name and captured differently, with a comment explaining why, so that the next person does not “fix” the inconsistency.

When the rebrand hit

The rebrand is where all of this got tested at once, and it surfaced something none of us predicted.

That scroll gradient used to resolve to white over near-white content. The diff it produced when caught mid-fade was real but tiny, comfortably below our difference threshold, so it never failed anything. After the rebrand, the same gradient resolved to a very slightly darker grey. Still almost invisible to a human. Just different enough against the content beneath it to push the diff above the threshold.

A flaky test that had been flaky all along, silently passing, started failing. The rebrand did not cause the bug. It removed the margin that had been hiding it.

Our threshold is 0.0001, which we chose with the arithmetic written down next to it: for an 800 by 600 image, that is 48 differing pixels out of 480,000. Tight enough to catch a one pixel border change, loose enough to survive antialiasing. Getting that number right is guesswork until something like a rebrand tells you where the real noise floor is.

Review fatigue, and why we self-host

The documented reason teams abandon visual regression testing is not cost or setup. It is review fatigue. When a build produces fifty diffs and forty-five of them are noise, people stop looking at diffs, and a test suite nobody reads is worse than no test suite because it produces confidence without evidence.

Everything above is really an attack on that. Trace the dependency graph so a small change produces a small diff set. Fix flakiness aggressively so the diffs that appear are real. Give people a filter so they can re-run just the failures instead of the whole suite.

For a token-wide change like a rebrand, none of that helps, because everything legitimately did change. The approach that works there is different: ship it as an isolated pull request with no logic changes mixed in, say up front what the expected visual delta is, batch-approve the library snapshots that match it, and spend the actual human attention on the consuming application, where composition can break in ways the library cannot predict.

Then there is self-hosting. Chromatic’s pitch against it is that you end up running infrastructure, that your own infrastructure updates cause false positives, and that you will spend your time tuning CI machines instead of building your product. They are right. That ninety-line config file with sixty lines of comments is their argument, and I just spent four paragraphs making it for them.

We do it anyway, for two reasons. Screenshots of a banking interface are data I would rather not hand to a third party. And the pricing curve gets ugly fast: Ant Design hit this in public at around six thousand screenshots per pull request, decided hosted no longer added up, and built their own. We are nowhere near that number and I could already feel it coming.

Smaller library, no data concerns, starting over? I would pay for the hosted option and not think about it again, then put the weeks I saved into the dependency graph work. That is the half nobody sells you.

Next in this series: reading components out of Figma, and why the hard part was never the SVG.