Visual and accessibility testing
CI checks every web page Bookclerk ships for layout changes and
accessibility problems: the web app's components, the website
(tools/site) and the diagnostics collector's pages
(tools/diagnostics-collector). Each web app component has Storybook
stories beside its source. CI opens every story and every Worker page in a
browser, checks it with axe-core, and compares a screenshot with an approved
baseline image.
Nothing here ships in the app or the Workers. Storybook, Playwright, the
stories and the Workers' local page servers are development tools, and
vite build never imports a story, so ui/dist is unchanged.
For every story in the static Storybook build, the check:
- Opens the story at two viewports:
desktop(1280 × 800) andphone(390 × 844). - Renders it in light and in dark, the same
html.darkswitch the app uses. - Runs axe-core against the story with the WCAG 2.2 A and AA rules. Any violation fails the check.
- Takes a screenshot of the story's own box and compares it with the baseline of the same name. Any changed pixel fails the check, apart from Playwright's small per-pixel colour threshold.
The story list comes from the build's index.json, so a new story is
checked without any other change. A story tagged skip-visual is left out.
ui/visual/workers.spec.ts checks the pages the two Cloudflare Workers
serve, in the same run and at the same viewports. Each page is opened in
light and in dark (both follow prefers-color-scheme), must answer with its
expected status, passes the same axe-core rules, and is compared with a
baseline of its whole length, named <worker>--<page>--<theme>.png.
- Website:
tools/site/audit/serve.tsserves the built site through its real Worker underwrangler dev, as the website audit does. The pages areVISUAL_PAGESintools/site/audit/pages.ts: the home, download, privacy and security pages, the 404 and docs-unavailable pages, and one docs page. That docs page is the fixturetools/site/audit/visual-docs/layout.md, which holds one of each kind of docs block. Real docs pages aren't screenshotted, because their words change far more often than their layout. CI builds the site from a fixed commit and date, and the homepage's catalog sample is fixed, so nothing but the layout changes the images. - Collector:
tools/diagnostics-collector/visual/serve.tsrenders every page template indashboard/pages.tsfrom fixed sample data and a fixed clock, with the dashboard's static files. No database, GitHub App or session is involved. The pages areVISUAL_PAGESintools/diagnostics-collector/visual/pages.ts; the server refuses to start when that list and its renderers disagree.
A new collector page template gets a sample render in visual/serve.ts
and an entry in visual/pages.ts. A new website page gets an entry in
audit/pages.ts.
Put the stories next to the component, in <Component>.stories.tsx:
import type { Meta, StoryObj } from "@storybook/react-vite";
import { StarRating } from "./StarRating";
const meta = {
title: "Library/StarRating",
component: StarRating,
args: { value: 4 },
} satisfies Meta<typeof StarRating>;
export default meta;
/** A story of {@link StarRating}. */
type Story = StoryObj<typeof meta>;
/** A fractional rating, with the last star clipped. */
export const Fractional: Story = { args: { value: 3.6 } };
- Cover each state that looks different: variants, empty, error, disabled and long text.
- Give stories fixed data. A story must not read the clock, call the API or load a remote image, or its screenshot changes from run to run.
- Don't let a story read the page's own address either. Storybook serves
every story from
/iframe.html, so pass a page such asNotFoundPagethe path to show. - Render the component as the app lays it out, not a layout made up for
the story. A component that reads shared app state (the page's player,
say) exports a view that takes that state as props, such as
PlayerBarView, and the story renders the view. - A whole page uses
parameters: { layout: "fullscreen" }, which gives it the viewport's height as the app's#rootdoes. - Give a form control a label in the story, as the app does. axe-core fails an input with no accessible name.
From ui/:
npm ci
npm run storybook # browse the stories at http://127.0.0.1:6006
npm run storybook:build # the static build the checks use
npx playwright install chromium # once per Playwright version
npm run test:visual -- --update-snapshots=all
The last command records screenshots into ui/visual/baselines/ and fails
on any axe-core violation. That folder is ignored by git.
The run also starts the Workers' page servers, so install and build them first:
(cd ../tools/site && npm ci && SITE_COMMIT=0000000 SITE_BUILT_ON=2026-01-01 npm run build)
(cd ../tools/diagnostics-collector && npm ci)
Add workers or stories after -- to run only one of the two specs.
Important
Local screenshots are for looking, never for committing as baselines. Fonts, anti-aliasing and the browser build differ between machines. Baselines only come from the CI image.
VISUAL_CHROMIUM_PATH points the checks at another Chromium build, for
a host whose preinstalled browser isn't the one Playwright downloads.
The UI visual and accessibility job (ui_visual check) runs when a file
under ui/, tools/site/, tools/diagnostics-collector/ or
assets/brand/ changes, or SECURITY.md (the website's security page), and
in the full suite. A change to the host that serves the app doesn't select
it, and neither does a change under docs/, because neither changes what
the checks render.
Like the other jobs outside check and test, it is
deferred on a draft. It runs on ready pull
requests and in the merge queue.
It runs inside the pinned Playwright image
(mcr.microsoft.com/playwright:v<version>-noble, pinned by digest), so every
run renders with the same browser, fonts and system libraries. The image's
version must match @playwright/test in ui/package.json. When they
differ, Playwright can't find its browser and the job fails.
Each run first records a full set of screenshots, then, once a set is
approved, compares with it. The job uploads a ui-visual artifact:
baselines/ holds the recorded set, record/ and compare/ the HTML
reports, with the expected, actual and diff image of each mismatch.
Note
While ui/visual/baselines.lock says none, the job only records. The
accessibility checks still fail the run.
Baseline images do not live in this repository, because years of changed
PNGs would make every clone much larger. They live in a separate public
repository, Bookclerk/bookclerk-visual-baselines, in the same way the
built docs live in Bookclerk/bookclerk-docs.
- Each baseline set is one commit in the baselines repository, with one
folder per viewport:
desktop/<story-id>--<theme>.pngandphone/<story-id>--<theme>.png. ui/visual/baselines.lockin this repository names the commit a Bookclerk commit is checked against (ornone). So every Bookclerk commit, including a release tag, pins the exact images it was approved with.pending/pr-<number>holds the set a pull request proposes;approved/<commit>tags keep every approved set.
The ui-visual job reads the lock with scripts/visual_baselines.py lock
and first checks that the commit carries an approved/<commit> tag. Only
the approval workflow creates those tags, so a lock that names a pending
set, copied by hand, fails the job instead of skipping the approval. It
then checks the baselines repository out at that commit. The repository is
public, so this needs no token and works the same on a pull request from a
fork. It sets CI_VISUAL_BASELINES to that checkout, and the check
compares after recording.
The Visual baselines workflow compares against the same lock CI used:
the one in the pull request merged into its base branch. A pull request
that leaves the lock alone is compared with the base branch's approved set,
so it only gets a comment when its own change alters a screenshot. Without
a merge ref (the pull request conflicts with its base), the head's lock
stands in. Like CI, the workflow treats an untagged lock as no set at all,
so every image needs approval.
A pull request that changes how a component looks fails ui-visual until
its new screenshots are approved. The Visual baselines workflow
(.github/workflows/visual-baselines.yml) handles the rest:
- When CI finishes, the workflow reads the run's recorded set. If it
differs from the approved set CI compared with, it pushes the set to
pending/pr-<number>and posts one comment on the pull request. The comment opens with how many screenshots are changed, new and removed, and the approval checkbox. Below that, each category is a collapsed section: changed screenshots show the approved and proposed image, new ones only the proposed image and removed ones only the approved image. Every screenshot is listed; only a set too large for one GitHub comment (65,536 characters) shows the last rows by name alone. The workflow then requests a review from thevisual-reviewersteam, so its members are notified. - An approver ticks the checkbox. Ticking needs write access, so a visual
reviewer without it replies
@bookclerk approve-visualsinstead, the same form as the other bot commands. - The workflow checks the approver and that the pull request is open. It
then tags the set
approved/<commit>and commits the new lock to the pull request's branch through the API, so GitHub signs the commit. For a fork, the comment gives the author the lock line to commit. - The next run compares against the new set and passes.
A tick from anyone else is undone with the reason. The approval is of the images the comment shows, not of a commit, so a push after the comment doesn't void it. Later runs compare with the approved set, so screenshots that haven't changed since stay approved. Ones that changed need approval again, in a new comment; the approved comment stays as the record. Until a comment is approved, a run that records different images updates it in place, and a run that records the same images leaves it alone.
Because the pull request carries its own lock, the merge queue checks each pull request against its own approved images. Two pull requests that both approve new images change the same lock line, so the second one to merge conflicts. Merging the base branch into it resolves that: CI then compares with the newly approved set, and the comment shows only that pull request's own changes.
- Anyone with the admin or maintain role on this repository.
- Any active member of the
visual-reviewersorganization team. The team grants nothing else, so it can hold a designer without code access.
The workflow runs from the default branch through workflow_run and
issue_comment, and never checks out or runs pull request code. The
recorded set comes from a run of the pull request's own code, so it is
untrusted. scripts/visual_baselines.py accepts only PNG files with the
expected names, folders and sizes, and only a comment edit that ticks the
box and changes nothing else (see
Fuzzing).
The ui-visual job holds no secrets. The workflow's App tokens are
narrowed per step: Contents on the baselines repository alone for sets and
tags, and Pull requests, Contents and organization Members (read) on this
repository for the comment, the review request, the lock and the team
check.
Before the first approval, an organization owner:
-
Creates the public repository
Bookclerk/bookclerk-visual-baselineswith two rulesets, each with the automation App as its only bypass actor (bypass mode Always):- All branches: restrict creations, updates and deletions, and
block force pushes. Only the App writes the
pending/pr-<number>branches. - All tags: the same rules. Only the App creates
approved/*tags, and it never moves or deletes one.
The App never deletes a ref, so these rules cost the workflow nothing.
- All branches: restrict creations, updates and deletions, and
block force pushes. Only the App writes the
-
Installs the automation App on it with Contents read and write.
-
Grants the App the organization Members (read) permission.
-
Creates the
visual-reviewersteam, adds the approvers, and gives the team Read access to this repository. GitHub only accepts a review request for a team that can see the repository; Read grants no code write.
The lock stays none until the first set is approved. Until then every
pull request that touches ui/ is compared with an empty set and its
comment shows every screenshot as new. Approving one of them and merging
it sets the lock on the default branch, and later pull requests are
compared with that set.
- Screenshots cover each story's own box, not the whole viewport, so most images are a few kilobytes. The first sets average about 22 KB per image.
- Only sets that an
approved/*tag points at need to be kept. Pending refs are overwritten on each proposal, so a pull request that never merges leaves at most one pending set behind.
- Web UI development: run the app and the desktop window.
- Continuous integration: how CI selects and runs the jobs.
Source: docs/contributing/ui/visual-testing.md at main · View as Markdown