bi-13 — Analysis

Required invariants

1. A test must fail for the reason the bug exists. A fix at the wrong layer with a passing test proves nothing.
2. Run the test against unpatched source first. A test that passes before your fix is not a test.
3. A failing test keeps running. Chromium forbids a bare [ Skip ] so that flakiness data keeps accruing and an accidental fix is detected.
4. Expectations carry a bug link. An entry without one is an untracked known-broken behaviour.
5. WPT is the cross-browser contract. A Chromium-only test cannot express interop.

The policy worth stealing

"Normally we should not skip a test because it's failing or flaky. We should add failure or flaky expectations instead, so that they will still run on bots, and we can collect data about their flakiness."

Three reasons this beats skipping:

1. A skipped test yields no data; an expected-failure that unexpectedly passes is reported.
2. Flakiness is measurable only if the test runs.
3. Skipped tests rot silently; failing-but-running tests stay honest.

Measured: TestExpectations is 9,418 lines, with platform tags down to specific OS versions (Mac13, Win11-arm64). Version-specific expectations are an admission that behaviour varies by OS version — usually through system libraries for fonts, shaping, and media.

Test-layer selection

Preference within web tests: testharness.js → reftest → pixel baseline. Same ordering as behavioural assertions over snapshots in application testing, and for the same reason: a snapshot asserts "it looks like this," not "it is correct."

Failure modes