Observation Guide

Inspecting the accessibility tree by hand

Chrome DevTools → Elements → Accessibility pane shows the computed node for the selected element: role, name and how the name was computed, plus the ancestor chain. The "Full-page accessibility tree" toggle (top of the pane) is the manual version of experiment 1 and the fastest way to see div soup for what it is.

Three things to look at, in order:

  1. Landmarks. Screen-reader users navigate by region first. No landmarks means no navigation.
  2. Headings and their levels. The second navigation mode. Skipped levels and role=heading without aria-level both break it.
  3. The controls list. Anything interactive that is not in it is invisible to assistive tech — regardless of how it looks or what onclick it has.

Keyboard-only testing, which finds more than any tool

Unplug the mouse for ten minutes on a page you own:

CheckFailure signal
Tab through the whole pagefocus disappears, or lands on nothing visible
Visible focus indicator everywhereoutline: none with no replacement
Enter and Space on every controlworks on <button>, silently fails on div role="button"
Escape closes overlayscustom modals routinely miss this — measured 1/7
Focus returns after closing an overlayfocus lost to <body>; screen reader restarts at the top
Tab cannot escape an open modalbackground reachable behind the overlay

Rows 3–6 are exactly the behaviours experiment 2 measures, and exactly the ones native elements provide for free.

Screen readers, briefly

Automated tools cannot tell you whether an interface is usable, only whether it violates specific rules. Thirty minutes with a real screen reader is worth more than any audit score:

PlatformReaderStart
macOSVoiceOver⌘F5; VO = Ctrl+Option; rotor = VO+U
WindowsNVDA (free)Insert = NVDA key; elements list = NVDA+F7
WindowsJAWSstill dominant in enterprise/government

The exercise: open the rotor/elements list and try to navigate by landmark and heading. On the div soup variant there is nothing to navigate — which is the finding, and it is far more persuasive than a table of counts.

What automation catches, and what it cannot

Automated tools findAutomated tools miss
missing altalt that describes the wrong thing
contrast ratiosfocus order that makes no sense
duplicate ids, invalid ARIAa div that should have been a button
missing form labelserror messages not associated with fields
missing langkeyboard traps in custom widgets

The div soup variant would score well on many automated audits. It has no contrast failures, no missing alt text, no invalid ARIA — and exposes zero controls. This is the concrete demonstration of the specification's requirement to "explain the limits of automation" (fe-24).

Rough division: automation covers perhaps 30–40% of WCAG failures by count, and a much smaller share by user impact. Treat an audit score as a regression guard on things you already fixed, never as evidence of accessibility.

What these measurements do not tell you

  • One browser. Accessibility tree computation differs between Chrome, Firefox and Safari, and screen readers consume the platform API (UIA/AX/AT-SPI), not the browser's internal tree.
  • Tree exposure is not usability. Correct roles with a nonsensical reading order or unhelpful names measures identically to a good interface.
  • No real assistive technology was involved. Verbosity settings, browse vs focus mode, and reader-specific quirks are invisible here.
  • Byte counts are structural, not transfer size — gzip changes the ratios, though not the direction.
  • The dialog test covers seven behaviours. A real modal also has scroll locking, aria-labelling, nested-dialog behaviour, and iOS focus quirks that are not measured.