Concepts — Chromium Test Architecture and Web Platform Tests

Phase 6 · Spec areas §22 (test architecture), §23 (WPT). Requires a working build for the runnable parts.

1. Why a Principal Engineer needs this

1. A Chromium contribution without a test will not land. Reviewers will ask for one, at the right layer, in the right style. Knowing which layer is a design judgement, not a formality.

2. WPT is the cross-browser contract, and it is where interop is negotiated. Reading a WPT tells you what all browsers agreed to. Reading Chromium's expected failures tells you where they currently disagree — which is a map of tractable contributions and of real risks to your products.

3. Choosing a test layer is a transferable skill. Unit vs integration vs end-to-end is the same decision you make in application code, with the stakes raised and the vocabulary changed.

2. Mental Model

2.1 The layers

The Blink/non-Blink naming split (_test.cc vs _unittest.cc) is a real search hazard — searching only one form silently halves your results (bi-01 Technique 6).

2.2 Web tests and how they assert

Two flavours worth distinguishing:

• testharness.js tests — JavaScript assertions; the output is a pass/fail list. Prefer these: they state intent, and they are portable across browsers.
• Reference tests ("reftests") — render the test and a reference page and require the pixels to match. Right when the behaviour is visual and cannot be asserted in JS.

There are also pixel/expectation-file tests, which compare against a stored baseline. These are the most brittle: a legitimate rendering change requires rebaselining, and a baseline can encode a bug as "expected." Prefer testharness.js, then reftests, then pixel baselines — in that order, for the same reason you prefer behavioural assertions to snapshots in application code.

2.3 Expected failures are declarative

Chromium does not delete tests it fails. It records them, with bug links, in third_party/blink/web_tests/TestExpectations.

TestExpectations is a map of Chromium's known interop gaps. Every entry is a documented, accepted, currently-wrong behaviour with a bug attached.

For this track that file has two uses: it is the richest source of tractable first contributions (§24 rung 3), and it is a genuine input to product risk assessment — if your app depends on a behaviour listed there, you have a supported reason to expect trouble.

2.4 WPT is bidirectional

WPT lives upstream and is imported into Chromium; tests you write upstream reach every browser, and tests added in Chromium's WPT directory are exported. That means a WPT contribution improves interoperability rather than only Chromium — which is often the highest-leverage first contribution available, and it does not require C++.

2.5 The bug-fix workflow

reproduce  ->  identify the correct layer  ->  minimal failing test
   ->  fix  ->  verify  ->  run surrounding tests

"Identify the correct layer" is the step that separates a good contribution from a rejected one. A rendering bug fixed in Blink with only a unit test on a helper function does not demonstrate the user-visible behaviour is fixed. Conversely, a pixel test for an algorithmic off-by-one is brittle and slow. The test should fail for the reason the bug exists.

3. Practice

Commands and paths: bi-00-roadmap/docs/chromium-build-debug-trace.md §6.

Without a build you can still do most of the reading work here: locate tests, read TestExpectations, read WPT sources, and map spec → test → implementation.

3.5 Deep dive: TestExpectations, read properly

Measured 2026-08-10: 9,418 lines. That is not a bug list, it is an institutional memory.

The format

# tags: [ Android Fuchsia Linux Mac Mac13 Mac14 Mac15 Mac26 Win Win10.20h2 Win11 Webview iOS26-simulator ... ]
# tags: [ Release Debug ]
# results: [ Timeout Crash Pass Failure Skip ]

crbug.com/123456 [ Mac ] fast/forms/some-test.html [ Failure ]

Every entry carries a bug link, an optional platform/config tag set, the test path, and an expected result. So the file is a queryable database: which behaviours are wrong, on which platforms, with what tracking issue.

Note the platform tags include specific OS versions (Mac13, Mac15, Win11-arm64). Expectations are frequently version-specific, which is a direct admission that browser behaviour varies by OS version — usually via system libraries for fonts, text shaping, and media.

The policy that is worth stealing

The file's own header states it plainly:

"Single [ Skip ] expectation is not allowed in this file. Normally we should not skip a test because it's failing or flaky. We should add failure or flaky expectations instead, so that they will still run on bots, and we can collect data about their flakiness and update their expectations accordingly."

Read that twice. The rule is: a failing test keeps running. You record that it fails; you do not stop executing it.

The reasons are worth enumerating because they apply to any large test suite:

1. A skipped test yields no data. A test marked Failure that unexpectedly passes is reported — so the system tells you when someone accidentally fixed it.
2. Flakiness is measurable only if the test runs. Skip destroys the signal you would need to decide whether it is flaky or broken.
3. A skipped test rots silently; a failing-but-running test stays honest about its state.

There are narrow exceptions — NeverFixTests for things genuinely out of scope, SlowTests, and VirtualTestSuites' exclusive_tests for tests only meaningful under a virtual suite — and they are named and separated rather than being ad-hoc Skips.

Carry this into your own org. The instinct on a red test is to skip it. The better default is: mark it expected-to-fail, keep it running, attach a bug. Chromium runs one of the largest test suites in the world on this policy.

How to mine it

# What is currently failing in an area you have studied?
grep -n "html/parsing\|fast/table" third_party/blink/web_tests/TestExpectations | head -30

# Which entries have bug links (i.e. are tracked)?
grep -cE "crbug|issues\.chromium" third_party/blink/web_tests/TestExpectations

# Platform-specific breakage
grep -n "\[ Mac \]" third_party/blink/web_tests/TestExpectations | head

Each hit is a documented, accepted, currently-wrong behaviour with an owner-adjacent bug. That is your bi-14 candidate pool, and it is also real product-risk intelligence: if your application depends on a behaviour listed there, you have a supported reason to expect trouble.

3.6 Deep dive: virtual test suites

VirtualTestSuites lets the same test files run again under different flags — a feature flag on, a different compositing mode, a new algorithm. That is how a large behavioural change is validated against the existing corpus before it ships.

The idea generalises well: rather than forking tests for the new implementation, run the existing tests under the new configuration and record only the deltas. If you are ever migrating a large system behind a flag, this is the shape of the test strategy you want — the alternative (duplicating the suite) doubles maintenance and guarantees drift.

3.7 Deep dive: choosing a test layer, with the failure modes

The decision rule that matters:

The test must fail for the reason the bug exists.

A rendering bug fixed in Blink with only a unit test on a helper does not demonstrate the user-visible behaviour is fixed — and a reviewer will say so. Conversely, a pixel test for an off-by-one in an algorithm is slow, brittle, and will be rebaselined away by someone in six months.

Preference order within web tests

1. testharness.js — JS assertions, portable across browsers, states intent.
2. Reference tests — render test and reference, require a pixel match. Correct when the behaviour is visual and cannot be asserted in JS. Survives unrelated rendering changes, because both sides change together.
3. Pixel/baseline tests — compare against a stored image. Catches everything and breaks constantly; a baseline can silently encode a bug as "expected."

This is the same ordering as behavioural assertions over snapshots in application testing, for the same reason: a snapshot asserts "it looks like this," not "it is correct."

3.8 Deep dive: WPT is bidirectional, and that is the leverage

WPT lives upstream and is imported into Chromium; tests added in Chromium's WPT directory are exported back. So:

• A WPT you write improves every browser's conformance signal, not just Chromium's.
• It requires no C++ — HTML and JavaScript.
• It lands through a lighter process than a Blink change.

For an application engineer this is frequently the highest-leverage open-source contribution available, and it is the right alternative if a docs-only first CL feels too trivial (bi-14 rung 1).

The workflow that turns confusion into a contribution:

you hit a cross-browser inconsistency
  → read the spec; decide what SHOULD happen
  → search WPT for existing coverage
  → if none: write a testharness.js test asserting the spec behaviour
  → run it in multiple engines; record who fails
  → land the test upstream

You have now converted "browsers disagree and it is annoying" into a durable, shared artifact that makes the disagreement visible to everyone. That is a genuinely Principal-level move: you changed the information available to the whole ecosystem, not just your own codebase.

3.9 Deep dive: flakiness as a first-class concern

Chromium treats flakiness as data, not as an annoyance to be suppressed. Expectations can record flaky results; bots collect statistics; tests that become reliably-passing get their expectations tightened.

The lesson for a team you lead:

• A flaky test is a signal about the system, not only about the test. Ordering assumptions, timing dependence, and shared state are real product bugs that happened to surface in CI.
• Suppression without measurement is how suites die. If you disable, you must record and revisit; Chromium's answer is "keep running it and record the expectation."
• Deterministic reproduction is the deliverable, not "it passed on retry." fw-08 and fw-09 make the same demand of race-condition labs: if your reproduction is "click fast," you have observed the bug, not reproduced it.

4. Anti-Patterns

Writing a pixel test when a testharness.js test would do.

Rebaselining to make a test pass. Rebaselining is correct only when the new rendering is correct. Establish that first.

Testing implementation details of a helper instead of the specified behaviour.

Adding a test that passes before your fix. It proves nothing. Run it against unpatched source first — always.

Assuming a failing WPT is a Chromium bug. It may be a spec disagreement, an out-of-date imported test, or a deliberate deviation. Check the expectation's bug link.

5. Trade-offs

Fast unit tests vs realistic browser tests. Browser tests catch integration failures and cost minutes; unit tests are seconds and can pass while the feature is broken.

Reftests vs pixel baselines. Reftests express intent ("these two should look the same") and survive unrelated rendering changes; baselines catch everything and break constantly.

Declarative expected-failures vs deleting tests. Recording failures keeps them visible and fixable, at the cost of a large file that can quietly normalise breakage.

6. Lab

1. Layer identification. For five behaviours you studied in bi-03, bi-04, bi-07, bi-08, bi-09, find the test at each layer that covers it, or establish that none exists.
2. Run each kind once. A Blink unit test, a web test directory, and a WPT.
3. Write a testharness.js test for the parsing lab's Case B. Verify it passes on Chromium and reason about whether it should pass on other engines.
4. Mine TestExpectations. Find three currently-failing WPTs in areas you have studied. For each: read the bug, read the spec, and write one paragraph on whether it looks tractable. Keep this list — it is your bi-14 candidate pool.
5. The break-and-observe loop (§23). Pick a behaviour, find the WPT, find the Blink implementation, deliberately break it, watch the test fail, restore. This proves you have connected spec → test → implementation.

Deliverable: the candidate list from (4), with a tractability judgement each.

7. Failure Lab

1. Write a test that passes without your change. Explain how you would have caught this.
2. Rebaseline a pixel test to accept a wrong rendering. Notice how easy it was — this is the argument for preferring reftests.
3. Fix a bug at the wrong layer: patch a symptom in a caller rather than the cause. Write the review comment you would expect to receive.

8. References

• docs/testing/web_tests.md and the web-test expectations documentation in your checkout.
• third_party/blink/web_tests/TestExpectations.
• web-platform-tests documentation; testharness.js API.
• docs/testing/ generally — browser tests, unit tests, flakiness policy.

9. Principal Engineer Review

1. Given a rendering bug, how do you decide the test layer? Give your decision procedure and a case where it is genuinely ambiguous.

2. TestExpectations records known failures rather than deleting tests. Argue this is superior; then describe how it decays and what you would do about it.

3. A WPT fails in Chromium and passes in two other engines. Enumerate what this could mean, ranked, and how you distinguish them.

4. Pixel tests catch everything and break constantly. Argue for banning them; then defend the cases where nothing else works.

5. You are asked to raise a team's confidence in a rendering-heavy product. Design the test strategy across layers, and say what you deliberately will not test.

6. A contributor submits a correct fix with no test. Write the review comment.

7. WPT contributions improve all browsers. Argue this is the highest-leverage work an application engineer can do; then give the strongest counterargument.

8. Your product depends on a behaviour listed as an expected failure in Chromium. Walk through what you tell your team, and what you change.