Automating Visual QA — How We Caught a Broken Checkout Before Users Did

Last quarter our team pushed a CSS update that looked fine in staging. Passed code review. Passed unit tests. Made it to production at 4 PM on a Thursday.

By Friday morning, support had three tickets about the checkout button being invisible on mobile Safari. White button on white background — classic.

Nobody caught it because nobody manually tested on every browser after every deploy. And honestly, who would?

What changed after that

We added visual regression checks to the deploy pipeline. The concept is straightforward: take a screenshot of key pages before and after each deploy, diff them pixel by pixel, flag anything that changed more than a threshold.

Sounds simple, but running headless browsers in CI has its own headaches. Chrome eats memory, rendering is inconsistent across environments, and maintaining the infrastructure is one more thing to worry about.

We ended up moving the screenshot part to ScreenshotRun — pass it the URLs, get back screenshots at whatever viewport sizes we need. The comparison logic stays in our pipeline, but the actual rendering happens externally.

The setup (simplified)

# Before deploy - capture baseline
for url in "${PAGES[@]}"; do
  curl -o "before_$(echo $url | md5sum | cut -d' ' -f1).png" \
    "https://api.screenshotrun.com/capture?url=$url&width=375&format=png"
done

# After deploy - capture current state
for url in "${PAGES[@]}"; do
  curl -o "after_$(echo $url | md5sum | cut -d' ' -f1).png" \
    "https://api.screenshotrun.com/capture?url=$url&width=375&format=png"
done

# Compare with ImageMagick
compare before_*.png after_*.png -metric RMSE diff.png 2>&1

Real implementation has more to it — we check desktop and mobile viewports, handle dynamic content by masking certain areas, and post results to Slack. But the core loop is just: capture, capture, compare.

What this actually catches

In the two months since we set it up:

• The checkout button incident (would've been caught in minutes, not hours)
• A font that stopped loading because the CDN URL changed
• A banner component that shifted layout on pages where it wasn't supposed to appear
• Footer links disappearing after a dependency update

None of these would've been caught by unit tests or integration tests. They're visual problems. You need eyes on the page — or something that acts like eyes.

What it doesn't catch

Anything below the fold on long pages, unless you explicitly scroll and capture. Interactive states — hover effects, dropdown menus, form validation messages. For those you still need manual QA or more sophisticated test tooling.

But for "did we break the visible layout of our most important pages" — pixel diffing with automated screenshots handles maybe 80% of visual regressions. Good enough ROI for a setup that took an afternoon.