Accessibility testing

Only what your pull request changed

A repository adopting Merrykat can have multiple violations on day one. Reporting all of them in every pull request is too noisy, so a violation present on both sides is pre-existing, one only in the head commit of the PR is new, and one only in the baseline is resolved. Pre-existing violations are counted and folded away, and never fail a check.

How “new” is decided

Element identity is sometimes hard. axe identifies an element by a CSS selector and its markup, and both can move for reasons that have nothing to do with accessibility: a regenerated id, a renamed class, an inserted sibling shifting an :nth-child. Matching purely on identity would report the same unlabelled input as one new violation and one resolved violation on any pull request that touched its markup.

So the contract is a count, not an identity: a rule reports new violations only when it is entirely new to the head, or when the number of elements it flags grew. Identity then decides which elements to show, once there is something to show.

It is never gated on a visual difference

The most common accessibility regression changes no pixels at all: a dropped alt, a dropped aria-label, a label that lost its for. Those land on stories that compared as visually unchanged. Our analysis does not wait for a visual difference to justify itself. The Merrykat engine keeps the head capture for a story with new violations even when there is no visual difference.

To help contextualize accessibility issues, new violations get an annotated screenshot with the offending element outlined. This makes it super easy to detect where the fix needs to land.

Runs in one viewport by default

In most cases, a missing alt, an unlabelled control and a contrast ratio do not change with the width of the window. So Merrykat defaults to run accessibility tests in only one viewport. Mark any viewport accessibility: true and you can turn on accessibility in multiple targets.

viewports: [
  { name: 'desktop', width: 1280, height: 800, accessibility: true },
  { name: 'mobile', width: 420, height: 700, accessibility: true },
  { name: 'desktop-dark', width: 1280, height: 800, media: { colorScheme: 'dark' } },
],

Setting accessibility: false on every viewport turns the pass off while leaving it configured. The PR comment always names the viewports it covered when it did not cover them all.

Configuration

export default {
  accessibility: {
    enabled: true,
    tags: ['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa'],
    // Merged over the defaults, so this both turns rules off and turns
    // the disabled page-level rules back on.
    rules: { 'color-contrast': { enabled: false } },
    maxAnnotated: 10,
    timeoutMs: 15000,
  },
};

Rules that are off by default

A story renders into a document Storybook generated, not a page you wrote, so the page-level rules judge Storybook’s scaffolding rather than your component. Six are disabled by default: region, document-title, html-has-lang, landmark-one-main, page-has-heading-one and bypass. region is the worst of them — it wants all content inside a landmark, which no single component ever is, so leaving it on marks essentially every story in every Storybook as failing.

Because rules is merged over that list, a repository whose stories really are whole pages switches them back on the same way anything else is turned off: rules: { region: { enabled: true } }.

Failing the check on it

Add new_a11y_violation to failOn. It is deliberately not a default: the pass itself is on by default, so failing by default too would break pull requests for regressions nobody had been told about yet. Reporting them is useful immediately; blocking on them is a policy you opt into once you have seen what your own baseline looks like.

Turning it off

For one run, without touching the configuration: npx merrykat upload --accessibility false. For one story:

export const DecorativeOnly = {
  parameters: { merrykat: { accessibility: false } },
};

A story can only ever turn the pass off, never on. With the pass disabled for the project, axe is not loaded into the browser at all.

When the analysis did not finish

A skipped analysis produces the same zero a clean one does, so the two are distinguished explicitly: if either side’s pass did not complete, the comment says the result is incomplete and should be treated as unknown rather than as a pass. Nothing about the accessibility pass can fail a run on its own.