bi-13 — Broader Ideas

The policy to steal on Monday

A failing test keeps running. Record the expectation; do not skip.

The instinct on a red test is to disable it. Chromium runs one of the largest suites in the world on the opposite rule, because a skipped test yields no data, cannot be measured for flakiness, and rots silently — while an expected-failure that unexpectedly passes is reported.

Implementing this in your own CI needs three things: an expectations file with bug links, a runner that reports unexpected passes, and a review policy that treats an expectation as debt rather than a resolution.

Virtual test suites, and how to migrate safely

Running the existing suite under a new configuration — a feature flag on, a new algorithm — and recording only the deltas is the right shape for any large behavioural migration. The alternative, forking the suite, doubles maintenance and guarantees drift.

If you are ever rewriting something behind a flag, this is the test strategy.

WPT as leverage

A test written upstream improves every browser's conformance signal, requires no C++, and lands through a lighter process. For an application engineer, converting "browsers disagree and it is annoying" into a durable shared artifact is frequently the highest-leverage open-source contribution available.

It is also a genuinely Principal-level move: you changed the information available to the whole ecosystem, not just your own codebase.

Flakiness is a signal about the system

Ordering assumptions, timing dependence, and shared state are product bugs that happened to surface in CI. "It passed on retry" is not a diagnosis. fw-08 and fw-09 make the same demand: deterministic reproduction is the deliverable.

Next

bi-14 turns a TestExpectations entry into a contribution.