Visual and accessibility testing

Development These docs follow the development version and change with every merge.

Last updated View as Markdown

CI checks every web page Bookclerk ships for layout changes and accessibility problems: the web app's components, the website (tools/site) and the diagnostics collector's pages (tools/diagnostics-collector). Each web app component has Storybook stories beside its source. CI opens every story and every Worker page in a browser, checks it with axe-core, and compares a screenshot with an approved baseline image.

Nothing here ships in the app or the Workers. Storybook, Playwright, the stories and the Workers' local page servers are development tools, and vite build never imports a story, so ui/dist is unchanged.

What runs

For every story in the static Storybook build, the check:

  1. Opens the story at two viewports: desktop (1280 × 800) and phone (390 × 844).
  2. Renders it in light and in dark, the same html.dark switch the app uses.
  3. Runs axe-core against the story with the WCAG 2.2 A and AA rules. Any violation fails the check.
  4. Takes a screenshot of the story's own box and compares it with the baseline of the same name. Any changed pixel fails the check, apart from Playwright's small per-pixel colour threshold.

The story list comes from the build's index.json, so a new story is checked without any other change. A story tagged skip-visual is left out.

Website and collector pages

ui/visual/workers.spec.ts checks the pages the two Cloudflare Workers serve, in the same run and at the same viewports. Each page is opened in light and in dark (both follow prefers-color-scheme), must answer with its expected status, passes the same axe-core rules, and is compared with a baseline of its whole length, named <worker>--<page>--<theme>.png.

  • Website: tools/site/audit/serve.ts serves the built site through its real Worker under wrangler dev, as the website audit does. The pages are VISUAL_PAGES in tools/site/audit/pages.ts: the home, download, privacy and security pages, the 404 and docs-unavailable pages, and one docs page. That docs page is the fixture tools/site/audit/visual-docs/layout.md, which holds one of each kind of docs block. Real docs pages aren't screenshotted, because their words change far more often than their layout. CI builds the site from a fixed commit and date, and the homepage's catalog sample is fixed, so nothing but the layout changes the images.
  • Collector: tools/diagnostics-collector/visual/serve.ts renders every page template in dashboard/pages.ts from fixed sample data and a fixed clock, with the dashboard's static files. No database, GitHub App or session is involved. The pages are VISUAL_PAGES in tools/diagnostics-collector/visual/pages.ts; the server refuses to start when that list and its renderers disagree.

A new collector page template gets a sample render in visual/serve.ts and an entry in visual/pages.ts. A new website page gets an entry in audit/pages.ts.

Write a story

Put the stories next to the component, in <Component>.stories.tsx:

import type { Meta, StoryObj } from "@storybook/react-vite";
import { StarRating } from "./StarRating";

const meta = {
  title: "Library/StarRating",
  component: StarRating,
  args: { value: 4 },
} satisfies Meta<typeof StarRating>;

export default meta;

/** A story of {@link StarRating}. */
type Story = StoryObj<typeof meta>;

/** A fractional rating, with the last star clipped. */
export const Fractional: Story = { args: { value: 3.6 } };
  • Cover each state that looks different: variants, empty, error, disabled and long text.
  • Give stories fixed data. A story must not read the clock, call the API or load a remote image, or its screenshot changes from run to run.
  • Don't let a story read the page's own address either. Storybook serves every story from /iframe.html, so pass a page such as NotFoundPage the path to show.
  • Render the component as the app lays it out, not a layout made up for the story. A component that reads shared app state (the page's player, say) exports a view that takes that state as props, such as PlayerBarView, and the story renders the view.
  • A whole page uses parameters: { layout: "fullscreen" }, which gives it the viewport's height as the app's #root does.
  • Give a form control a label in the story, as the app does. axe-core fails an input with no accessible name.

Run it locally

From ui/:

npm ci
npm run storybook                 # browse the stories at http://127.0.0.1:6006
npm run storybook:build           # the static build the checks use
npx playwright install chromium   # once per Playwright version
npm run test:visual -- --update-snapshots=all

The last command records screenshots into ui/visual/baselines/ and fails on any axe-core violation. That folder is ignored by git.

The run also starts the Workers' page servers, so install and build them first:

(cd ../tools/site && npm ci && SITE_COMMIT=0000000 SITE_BUILT_ON=2026-01-01 npm run build)
(cd ../tools/diagnostics-collector && npm ci)

Add workers or stories after -- to run only one of the two specs.

Important

Local screenshots are for looking, never for committing as baselines. Fonts, anti-aliasing and the browser build differ between machines. Baselines only come from the CI image.

VISUAL_CHROMIUM_PATH points the checks at another Chromium build, for a host whose preinstalled browser isn't the one Playwright downloads.

The CI job

The UI visual and accessibility job (ui_visual check) runs when a file under ui/, tools/site/, tools/diagnostics-collector/ or assets/brand/ changes, or SECURITY.md (the website's security page), and in the full suite. A change to the host that serves the app doesn't select it, and neither does a change under docs/, because neither changes what the checks render. Like the other jobs outside check and test, it is deferred on a draft. It runs on ready pull requests and in the merge queue.

It runs inside the pinned Playwright image (mcr.microsoft.com/playwright:v<version>-noble, pinned by digest), so every run renders with the same browser, fonts and system libraries. The image's version must match @playwright/test in ui/package.json. When they differ, Playwright can't find its browser and the job fails.

Each run first records a full set of screenshots, then, once a set is approved, compares with it. The job uploads a ui-visual artifact: baselines/ holds the recorded set, record/ and compare/ the HTML reports, with the expected, actual and diff image of each mismatch.

Note

While ui/visual/baselines.lock says none, the job only records. The accessibility checks still fail the run.

Baselines

Baseline images do not live in this repository, because years of changed PNGs would make every clone much larger. They live in a separate public repository, Bookclerk/bookclerk-visual-baselines, in the same way the built docs live in Bookclerk/bookclerk-docs.

Layout

  • Each baseline set is one commit in the baselines repository, with one folder per viewport: desktop/<story-id>--<theme>.png and phone/<story-id>--<theme>.png.
  • ui/visual/baselines.lock in this repository names the commit a Bookclerk commit is checked against (or none). So every Bookclerk commit, including a release tag, pins the exact images it was approved with.
  • pending/pr-<number> holds the set a pull request proposes; approved/<commit> tags keep every approved set.

How CI compares

The ui-visual job reads the lock with scripts/visual_baselines.py lock and first checks that the commit carries an approved/<commit> tag. Only the approval workflow creates those tags, so a lock that names a pending set, copied by hand, fails the job instead of skipping the approval. It then checks the baselines repository out at that commit. The repository is public, so this needs no token and works the same on a pull request from a fork. It sets CI_VISUAL_BASELINES to that checkout, and the check compares after recording.

The Visual baselines workflow compares against the same lock CI used: the one in the pull request merged into its base branch. A pull request that leaves the lock alone is compared with the base branch's approved set, so it only gets a comment when its own change alters a screenshot. Without a merge ref (the pull request conflicts with its base), the head's lock stands in. Like CI, the workflow treats an untagged lock as no set at all, so every image needs approval.

How a pull request changes them

A pull request that changes how a component looks fails ui-visual until its new screenshots are approved. The Visual baselines workflow (.github/workflows/visual-baselines.yml) handles the rest:

  1. When CI finishes, the workflow reads the run's recorded set. If it differs from the approved set CI compared with, it pushes the set to pending/pr-<number> and posts one comment on the pull request. The comment opens with how many screenshots are changed, new and removed, and the approval checkbox. Below that, each category is a collapsed section: changed screenshots show the approved and proposed image, new ones only the proposed image and removed ones only the approved image. Every screenshot is listed; only a set too large for one GitHub comment (65,536 characters) shows the last rows by name alone. The workflow then requests a review from the visual-reviewers team, so its members are notified.
  2. An approver ticks the checkbox. Ticking needs write access, so a visual reviewer without it replies @bookclerk approve-visuals instead, the same form as the other bot commands.
  3. The workflow checks the approver and that the pull request is open. It then tags the set approved/<commit> and commits the new lock to the pull request's branch through the API, so GitHub signs the commit. For a fork, the comment gives the author the lock line to commit.
  4. The next run compares against the new set and passes.

A tick from anyone else is undone with the reason. The approval is of the images the comment shows, not of a commit, so a push after the comment doesn't void it. Later runs compare with the approved set, so screenshots that haven't changed since stay approved. Ones that changed need approval again, in a new comment; the approved comment stays as the record. Until a comment is approved, a run that records different images updates it in place, and a run that records the same images leaves it alone.

Because the pull request carries its own lock, the merge queue checks each pull request against its own approved images. Two pull requests that both approve new images change the same lock line, so the second one to merge conflicts. Merging the base branch into it resolves that: CI then compares with the newly approved set, and the comment shows only that pull request's own changes.

Who can approve

  • Anyone with the admin or maintain role on this repository.
  • Any active member of the visual-reviewers organization team. The team grants nothing else, so it can hold a designer without code access.

Safety

The workflow runs from the default branch through workflow_run and issue_comment, and never checks out or runs pull request code. The recorded set comes from a run of the pull request's own code, so it is untrusted. scripts/visual_baselines.py accepts only PNG files with the expected names, folders and sizes, and only a comment edit that ticks the box and changes nothing else (see Fuzzing).

The ui-visual job holds no secrets. The workflow's App tokens are narrowed per step: Contents on the baselines repository alone for sets and tags, and Pull requests, Contents and organization Members (read) on this repository for the comment, the review request, the lock and the team check.

Setup

Before the first approval, an organization owner:

  1. Creates the public repository Bookclerk/bookclerk-visual-baselines with two rulesets, each with the automation App as its only bypass actor (bypass mode Always):

    • All branches: restrict creations, updates and deletions, and block force pushes. Only the App writes the pending/pr-<number> branches.
    • All tags: the same rules. Only the App creates approved/* tags, and it never moves or deletes one.

    The App never deletes a ref, so these rules cost the workflow nothing.

  2. Installs the automation App on it with Contents read and write.

  3. Grants the App the organization Members (read) permission.

  4. Creates the visual-reviewers team, adds the approvers, and gives the team Read access to this repository. GitHub only accepts a review request for a team that can see the repository; Read grants no code write.

The lock stays none until the first set is approved. Until then every pull request that touches ui/ is compared with an empty set and its comment shows every screenshot as new. Approving one of them and merging it sets the lock on the default branch, and later pull requests are compared with that set.

Keeping the repository small

  • Screenshots cover each story's own box, not the whole viewport, so most images are a few kilobytes. The first sets average about 22 KB per image.
  • Only sets that an approved/* tag points at need to be kept. Pending refs are overwritten on each proposal, so a pull request that never merges leaves at most one pending set behind.

Next steps

Source: docs/contributing/ui/visual-testing.md at main · View as Markdown