Concepts — Chromium Test Architecture and Web Platform Tests

Phase 6 · Spec areas §22 (test architecture), §23 (WPT). Requires a working build for the runnable parts.

0. Orientation

The 60-second version

Chromium's testing system exists to answer a harder question than most test suites: does this change the web? Not "does this function work" — does some page, somewhere, behave differently.

That produces machinery you will not have seen elsewhere, and two pieces of it are worth stealing:

1. TestExpectations — recording known failures rather than disabling tests. A failing test that is expected to fail still runs, still reports, and still tells you when it starts passing. A disabled test tells you nothing, forever. The difference between "we know this fails" and "we stopped looking" is an entire quality culture.

2. Virtual test suites — the same tests, run again under different flags. This is how you ship a behaviour change safely: run the whole suite with the new code path on and off, and compare. It is exactly what a team shipping a compiler optimisation needs (fw-06) and almost nobody builds.

WPT is bidirectional, and that is the leverage

Web Platform Tests are shared across engines. A test you write lands in every browser's suite, which means you can make a correctness claim enforceable across the industry rather than in your own codebase. For a Principal engineer that is an unusually high-leverage move: it converts "we found a bug" into "no engine can regress this again."

Flakiness as a first-class concern

A flaky test is worse than a missing one, because it trains people to ignore failures. Chromium treats flakiness as its own problem with its own tooling, rather than as noise — which is the posture worth importing.

What you should be able to do after this module

1. Why a Principal Engineer needs this

1. A Chromium contribution without a test will not land. Reviewers will ask for one, at the right layer, in the right style. Knowing which layer is a design judgement, not a formality.

2. WPT is the cross-browser contract, and it is where interop is negotiated. Reading a WPT tells you what all browsers agreed to. Reading Chromium's expected failures tells you where they currently disagree — which is a map of tractable contributions and of real risks to your products.

3. Choosing a test layer is a transferable skill. Unit vs integration vs end-to-end is the same decision you make in application code, with the stakes raised and the vocabulary changed.

2. Mental Model

2.1 The layers

The Blink/non-Blink naming split (_test.cc vs _unittest.cc) is a real search hazard — searching only one form silently halves your results (bi-01 Technique 6).

2.2 Web tests and how they assert

Two flavours worth distinguishing:

• testharness.js tests — JavaScript assertions; the output is a pass/fail list. Prefer these: they state intent, and they are portable across browsers.
• Reference tests ("reftests") — render the test and a reference page and require the pixels to match. Right when the behaviour is visual and cannot be asserted in JS.

There are also pixel/expectation-file tests, which compare against a stored baseline. These are the most brittle: a legitimate rendering change requires rebaselining, and a baseline can encode a bug as "expected." Prefer testharness.js, then reftests, then pixel baselines — in that order, for the same reason you prefer behavioural assertions to snapshots in application code.

2.3 Expected failures are declarative

Chromium does not delete tests it fails. It records them, with bug links, in third_party/blink/web_tests/TestExpectations.

TestExpectations is a map of Chromium's known interop gaps. Every entry is a documented, accepted, currently-wrong behaviour with a bug attached.

For this track that file has two uses: it is the richest source of tractable first contributions (§24 rung 3), and it is a genuine input to product risk assessment — if your app depends on a behaviour listed there, you have a supported reason to expect trouble.

2.4 WPT is bidirectional

WPT lives upstream and is imported into Chromium; tests you write upstream reach every browser, and tests added in Chromium's WPT directory are exported. That means a WPT contribution improves interoperability rather than only Chromium — which is often the highest-leverage first contribution available, and it does not require C++.

2.5 The bug-fix workflow

reproduce  ->  identify the correct layer  ->  minimal failing test
   ->  fix  ->  verify  ->  run surrounding tests

"Identify the correct layer" is the step that separates a good contribution from a rejected one. A rendering bug fixed in Blink with only a unit test on a helper function does not demonstrate the user-visible behaviour is fixed. Conversely, a pixel test for an algorithmic off-by-one is brittle and slow. The test should fail for the reason the bug exists.

3. Practice

Commands and paths: bi-00-roadmap/docs/chromium-build-debug-trace.md §6.

Without a build you can still do most of the reading work here: locate tests, read TestExpectations, read WPT sources, and map spec → test → implementation.

3.5 Deep dive: TestExpectations, read properly

Measured 2026-08-10: 9,418 lines. That is not a bug list, it is an institutional memory.

The format

# tags: [ Android Fuchsia Linux Mac Mac13 Mac14 Mac15 Mac26 Win Win10.20h2 Win11 Webview iOS26-simulator ... ]
# tags: [ Release Debug ]
# results: [ Timeout Crash Pass Failure Skip ]

crbug.com/123456 [ Mac ] fast/forms/some-test.html [ Failure ]

Every entry carries a bug link, an optional platform/config tag set, the test path, and an expected result. So the file is a queryable database: which behaviours are wrong, on which platforms, with what tracking issue.

Note the platform tags include specific OS versions (Mac13, Mac15, Win11-arm64). Expectations are frequently version-specific, which is a direct admission that browser behaviour varies by OS version — usually via system libraries for fonts, text shaping, and media.

The policy that is worth stealing

The file's own header states it plainly:

"Single [ Skip ] expectation is not allowed in this file. Normally we should not skip a test because it's failing or flaky. We should add failure or flaky expectations instead, so that they will still run on bots, and we can collect data about their flakiness and update their expectations accordingly."

Read that twice. The rule is: a failing test keeps running. You record that it fails; you do not stop executing it.

The reasons are worth enumerating because they apply to any large test suite:

1. A skipped test yields no data. A test marked Failure that unexpectedly passes is reported — so the system tells you when someone accidentally fixed it.
2. Flakiness is measurable only if the test runs. Skip destroys the signal you would need to decide whether it is flaky or broken.
3. A skipped test rots silently; a failing-but-running test stays honest about its state.

There are narrow exceptions — NeverFixTests for things genuinely out of scope, SlowTests, and VirtualTestSuites' exclusive_tests for tests only meaningful under a virtual suite — and they are named and separated rather than being ad-hoc Skips.

Carry this into your own org. The instinct on a red test is to skip it. The better default is: mark it expected-to-fail, keep it running, attach a bug. Chromium runs one of the largest test suites in the world on this policy.

How to mine it

# What is currently failing in an area you have studied?
grep -n "html/parsing\|fast/table" third_party/blink/web_tests/TestExpectations | head -30

# Which entries have bug links (i.e. are tracked)?
grep -cE "crbug|issues\.chromium" third_party/blink/web_tests/TestExpectations

# Platform-specific breakage
grep -n "\[ Mac \]" third_party/blink/web_tests/TestExpectations | head

Each hit is a documented, accepted, currently-wrong behaviour with an owner-adjacent bug. That is your bi-14 candidate pool, and it is also real product-risk intelligence: if your application depends on a behaviour listed there, you have a supported reason to expect trouble.

3.6 Deep dive: virtual test suites

VirtualTestSuites lets the same test files run again under different flags — a feature flag on, a different compositing mode, a new algorithm. That is how a large behavioural change is validated against the existing corpus before it ships.

The idea generalises well: rather than forking tests for the new implementation, run the existing tests under the new configuration and record only the deltas. If you are ever migrating a large system behind a flag, this is the shape of the test strategy you want — the alternative (duplicating the suite) doubles maintenance and guarantees drift.

3.7 Deep dive: choosing a test layer, with the failure modes

The decision rule that matters:

The test must fail for the reason the bug exists.

A rendering bug fixed in Blink with only a unit test on a helper does not demonstrate the user-visible behaviour is fixed — and a reviewer will say so. Conversely, a pixel test for an off-by-one in an algorithm is slow, brittle, and will be rebaselined away by someone in six months.

Preference order within web tests

1. testharness.js — JS assertions, portable across browsers, states intent.
2. Reference tests — render test and reference, require a pixel match. Correct when the behaviour is visual and cannot be asserted in JS. Survives unrelated rendering changes, because both sides change together.
3. Pixel/baseline tests — compare against a stored image. Catches everything and breaks constantly; a baseline can silently encode a bug as "expected."

This is the same ordering as behavioural assertions over snapshots in application testing, for the same reason: a snapshot asserts "it looks like this," not "it is correct."

3.8 Deep dive: WPT is bidirectional, and that is the leverage

WPT lives upstream and is imported into Chromium; tests added in Chromium's WPT directory are exported back. So:

• A WPT you write improves every browser's conformance signal, not just Chromium's.
• It requires no C++ — HTML and JavaScript.
• It lands through a lighter process than a Blink change.

For an application engineer this is frequently the highest-leverage open-source contribution available, and it is the right alternative if a docs-only first CL feels too trivial (bi-14 rung 1).

The workflow that turns confusion into a contribution:

you hit a cross-browser inconsistency
  → read the spec; decide what SHOULD happen
  → search WPT for existing coverage
  → if none: write a testharness.js test asserting the spec behaviour
  → run it in multiple engines; record who fails
  → land the test upstream

You have now converted "browsers disagree and it is annoying" into a durable, shared artifact that makes the disagreement visible to everyone. That is a genuinely Principal-level move: you changed the information available to the whole ecosystem, not just your own codebase.

3.9 Deep dive: flakiness as a first-class concern

Chromium treats flakiness as data, not as an annoyance to be suppressed. Expectations can record flaky results; bots collect statistics; tests that become reliably-passing get their expectations tightened.

The lesson for a team you lead:

• A flaky test is a signal about the system, not only about the test. Ordering assumptions, timing dependence, and shared state are real product bugs that happened to surface in CI.
• Suppression without measurement is how suites die. If you disable, you must record and revisit; Chromium's answer is "keep running it and record the expectation."
• Deterministic reproduction is the deliverable, not "it passed on retry." fw-08 and fw-09 make the same demand of race-condition labs: if your reproduction is "click fast," you have observed the bug, not reproduced it.

4. Anti-Patterns

Writing a pixel test when a testharness.js test would do.

Rebaselining to make a test pass. Rebaselining is correct only when the new rendering is correct. Establish that first.

Testing implementation details of a helper instead of the specified behaviour.

Adding a test that passes before your fix. It proves nothing. Run it against unpatched source first — always.

Assuming a failing WPT is a Chromium bug. It may be a spec disagreement, an out-of-date imported test, or a deliberate deviation. Check the expectation's bug link.

5. Trade-offs

Fast unit tests vs realistic browser tests. Browser tests catch integration failures and cost minutes; unit tests are seconds and can pass while the feature is broken.

Reftests vs pixel baselines. Reftests express intent ("these two should look the same") and survive unrelated rendering changes; baselines catch everything and break constantly.

Declarative expected-failures vs deleting tests. Recording failures keeps them visible and fixable, at the cost of a large file that can quietly normalise breakage.

6. Lab

1. Layer identification. For five behaviours you studied in bi-03, bi-04, bi-07, bi-08, bi-09, find the test at each layer that covers it, or establish that none exists.
2. Run each kind once. A Blink unit test, a web test directory, and a WPT.
3. Write a testharness.js test for the parsing lab's Case B. Verify it passes on Chromium and reason about whether it should pass on other engines.
4. Mine TestExpectations. Find three currently-failing WPTs in areas you have studied. For each: read the bug, read the spec, and write one paragraph on whether it looks tractable. Keep this list — it is your bi-14 candidate pool.
5. The break-and-observe loop (§23). Pick a behaviour, find the WPT, find the Blink implementation, deliberately break it, watch the test fail, restore. This proves you have connected spec → test → implementation.

Deliverable: the candidate list from (4), with a tractability judgement each.

7. Failure Lab

1. Write a test that passes without your change. Explain how you would have caught this.
2. Rebaseline a pixel test to accept a wrong rendering. Notice how easy it was — this is the argument for preferring reftests.
3. Fix a bug at the wrong layer: patch a symptom in a caller rather than the cause. Write the review comment you would expect to receive.

8. References

• docs/testing/web_tests.md and the web-test expectations documentation in your checkout.
• third_party/blink/web_tests/TestExpectations.
• web-platform-tests documentation; testharness.js API.
• docs/testing/ generally — browser tests, unit tests, flakiness policy.

Core terminology

Mental models

Record, do not disable. An expected failure still runs and still tells you when the behaviour changes — including when it accidentally starts passing. A disabled test is a permanently silent one. This distinction is cheap to adopt and it changes what a red suite means.

Run it both ways and compare. Virtual test suites are the general answer to shipping a behaviour change safely, and the pattern transfers directly: a compiler optimisation (fw-06), a feature flag, a new rendering path. The failures these catch are silent correctness bugs, which no amount of code review reliably finds.

Choose the test layer by failure mode, not by convenience. A unit test cannot tell you the element was unreachable; a browser test cannot tell you which branch was wrong. fw-12 states the same rule for application testing: knowing which question each layer answers is test-strategy design.

A flaky test is worse than a missing one, because it teaches the team that failures are noise. Treating flakiness as its own tracked problem — rather than as an annoyance to retry away — is the posture that keeps a suite meaningful.

WPT converts a local finding into an industry-wide guarantee. That is the highest-leverage testing act available to a Principal engineer, and it costs about the same as writing the test internally.

Interop is a testable property. "Two conforming browsers must produce the same DOM for broken markup" (bi-03) is only true because something enforces it. WPT is that something.

Common misconceptions

1. "Disable the flaky test and move on." Record it as expected instead, so it still runs and still reports when the behaviour changes.

2. "WPT is for browser engineers." Anyone can contribute, and a test you land is enforced against every engine. That is leverage available to application engineers too.

3. "Coverage measures test quality." It measures lines executed. Which layer executed them — and therefore which failure modes are covered — is the interesting part.

4. "A green suite means the change is safe." Not for a behaviour change behind a flag, unless you ran it both ways.

5. "Reftests are screenshot tests." They compare two documents that must render identically, which is far more robust to unrelated rendering changes than a golden image.

6. "Flakes are infrastructure noise." Some are; treating all of them that way is how a suite stops meaning anything.

7. "Fuzzing is a security team activity." It is a substantial part of how the parsing and decoding surface is covered at all.

Interview talking points

• "The two ideas I'd steal from Chromium's testing system are recording known failures instead of disabling tests, and virtual test suites. An expected failure still runs and still tells you when it starts passing; a disabled test is silent forever. That difference is an entire quality culture, and it's nearly free to adopt."
• "Virtual test suites — running the whole suite again under different flags and comparing — are the general answer to shipping a behaviour change safely. It's exactly what a team shipping a compiler optimisation needs, because those fail as silent correctness bugs that code review doesn't catch."
• "I choose test layers by failure mode rather than convenience. A unit test can't tell you the element was unreachable; a browser test can't tell you which branch was wrong. Coverage numbers tell you nothing about which of those you have."
• "WPT is bidirectional, and that's unusual leverage: a test you write is enforced against every engine. It converts 'we found a bug' into 'no engine can regress this again' for roughly the cost of writing the test internally."
• "A flaky test is worse than a missing one because it teaches the team that failures are noise. Chromium treats flakiness as its own tracked problem rather than something to retry away, and that posture is what keeps a large suite meaningful."

9. Principal Engineer Review

1. Given a rendering bug, how do you decide the test layer? Give your decision procedure and a case where it is genuinely ambiguous.

2. TestExpectations records known failures rather than deleting tests. Argue this is superior; then describe how it decays and what you would do about it.

3. A WPT fails in Chromium and passes in two other engines. Enumerate what this could mean, ranked, and how you distinguish them.

4. Pixel tests catch everything and break constantly. Argue for banning them; then defend the cases where nothing else works.

5. You are asked to raise a team's confidence in a rendering-heavy product. Design the test strategy across layers, and say what you deliberately will not test.

6. A contributor submits a correct fix with no test. Write the review comment.

7. WPT contributions improve all browsers. Argue this is the highest-leverage work an application engineer can do; then give the strongest counterargument.

8. Your product depends on a behaviour listed as an expected failure in Chromium. Walk through what you tell your team, and what you change.

Connections to other modules

• bi-03 — "two conforming browsers must produce the same DOM" is only true because WPT enforces it.
• bi-14 — landing a CL means passing these gates; a web-visible change triggers more.
• bi-12 — diagnosing a failing or flaky test uses the instruments from that module.
• fw-06 — virtual test suites are the differential-testing strategy a compiler optimisation requires.
• fw-12 — the same "which question does this layer answer?" rule for application testing.
• fe-04 — automated accessibility checks pass on div soup: the limits of automation, concretely.
• fe-01 / fe-02 — harness bugs that produce clean, plausible, wrong data; why a harness needs its own failing test.