bi-03 — HTML Parsing

Phase 1 · Spec areas §6 (parser internals), §7 (mini parser). Prerequisites: bi-01, bi-02. the parsing lab is the entry point and should be done before reading this module.

Cross-track hook: frontend-principal-engineering.md §3 (HTML platform). This module explains why the platform's error recovery is not "the browser being forgiving."

Why a Principal Engineer needs this

1. HTML parsing is the only major web algorithm that is fully specified down to error recovery, and that fact is load-bearing. Before HTML5, every browser guessed differently at malformed markup and the guesses were the compatibility problem. The spec's insistence on defining what happens for invalid input — not merely valid input — is the single largest interop win in the platform's history. Understanding this changes how you evaluate every future proposal that says "undefined behaviour is fine, nobody does that."

2. It is the best available example of "production complexity you would never have designed." Insertion modes, foster parenting, the adoption agency algorithm, and the <input type=hidden> exception all exist because of content that already shipped. This module is where §46 becomes a real skill rather than a worksheet.

3. Parser behaviour determines what your framework can assume. Every templating system, every SSR pipeline, every sanitiser and every innerHTML call inherits these rules. Bugs where "the same HTML produces a different DOM in SSR vs client" are almost always someone meeting fragment parsing without knowing it exists.

4. It is where blocking behaviour originates. Parser-blocking scripts, preload scanning and script streaming are the mechanics behind most "why is first paint slow" answers.

Mental Model

Two machines, not one

bytes → decoder → TOKENIZER ──tokens──► TREE BUILDER → DOM
                  (state machine)       (insertion modes + stack of open elements)

They are separate, they have separate specs, and — critically — they are coupled in both directions. The tree builder can change the tokenizer's state. <script>, <textarea>, <title>, and <plaintext> all switch the tokenizer into a different content model depending on which element the tree builder just opened. So you cannot tokenize HTML correctly without partially parsing it. That is the first thing that makes HTML different from every parser you have written.

The tree builder is a pushdown automaton with exceptions

Three pieces of state carry everything:

1. Insertion mode — "in body", "in table", "in head", "after body", … The mode decides how a token is interpreted.
2. Stack of open elements — the currently-open ancestry.
3. List of active formatting elements — the machinery that makes <b>a<p>b</b>c produce sensible output. This is the adoption agency algorithm, and it is the most notorious part of the spec.

And one rule that is easy to miss and explains most surprises:

Where a node is inserted is decided separately from which token is being processed. The spec calls this the "appropriate place for inserting a node." Foster parenting is a property of the insertion location, not of the insertion mode.

the parsing lab Case A turns on exactly this: the insertion mode is still "in table" when world is inserted, but the current node is a div, so the redirect does not apply.

Error recovery is the specification, not a fallback

There is no parse-failure path. Every byte sequence yields a DOM. "Parse error" in the spec is a diagnostic annotation, not a control-flow branch — the algorithm continues, and what it does next is precisely defined. This is why:

• you cannot make HTML stricter without breaking the web,
• two conforming browsers must produce the same DOM for broken markup,
• and "just use a real XML parser" was tried, as XHTML, and lost.

Fragment parsing is a different algorithm

element.innerHTML = "..." does not run the document parsing algorithm. It runs the fragment parsing algorithm with a context element, which seeds the insertion mode based on that element. Consequences that bite in practice:

• table.innerHTML = "hello" and a document containing <table>hello</table> can differ.
• SSR-then-hydrate mismatches sometimes originate here rather than in the framework.
• Blink has a dedicated fast path for this case — TryParsingHTMLFragment in html_document_parser_fastpath.h — which handles common simple fragments without the full state machine and bails out to the general algorithm on anything unsupported. The header's own signature exposes the bail-out (failed_because_unsupported_tag), and the behaviour flags there (kStripInitialWhitespaceForBody for DOMParser, kIncludeShadowRoots) are a compact catalogue of the special cases fragment parsing has accumulated.

That fast path is a good early lesson: production parsers contain a fast path for the common case and a slow, fully-correct path underneath. Your mini parser will only have the second, and that is the right choice for learning — but knowing the shape of the real thing tells you what "optimise later" actually looks like.

Scripts, and why the parser is not free to run ahead

A classic <script> blocks parsing: the parser stops, the script runs, and the script may call document.write, which injects into the input stream at the current position. This is why the parser cannot simply be moved off the main thread wholesale.

Blink mitigates rather than removes this, and the two mitigations are different and frequently conflated:

The second exists because it does not tie up the main thread, so it can afford to scan everything. Both are speculative: they read ahead without building a DOM, and are allowed to be wrong (a speculative preload that turns out unnecessary costs bandwidth, not correctness). Speculation that cannot affect correctness is the general pattern for making a blocking algorithm faster without changing its semantics — you will meet it again in style invalidation and in the compositor.

Under the Hood

Do not take these as paths to memorise — they are the answers you should be able to re-derive with bi-01 Technique 1 (spec-phrase search). The parser directory has ~95 files; you need about six of them.

The single most useful structural observation: the tree builder does not mutate the DOM directly. It queues tasks on the construction site, which are flushed in batches. Ask why before reading on — the answer involves custom element reactions, mutation observers, and script re-entrancy, and deriving it yourself is worth more than being told.

Connecting to specification

Blink's parser quotes the HTML Standard heavily, which makes the spec a search index (bi-01 Technique 1). Practise the round trip in both directions:

• spec algorithm name → distinctive phrase → Blink implementation,
• Blink function name → the spec step it implements → the WPT that tests it.

If you can do both directions fluently for the parser, you can do them for any spec-defined subsystem, which is most of core/.

Deep dive: how big is this state machine, actually?

Measured in the checkout (2026-08-10):

html_tokenizer.h    :  76 tokenizer states
html_tree_builder.h :  21 insertion modes

Those two numbers are the module in miniature. Seventy-six states to turn text into tokens. Any parser you have written by hand probably had five. The gap is entirely legacy and content models, and enumerating why is the best §46 exercise available.

Why 76 states

The tokenizer states fall into families, and each family is a reason:

The script-escaping family is pure archaeology. Old pages wrapped inline script in HTML comments so that browsers without script support would not render the source:

<script><!--
  document.write("hi");
// --></script>

Supporting that required the tokenizer to track escaped and double-escaped script data, which is where kScriptDataDoubleEscaped* comes from. Nobody would design this. It is in every conforming browser forever because pages that rely on it still exist.

The lesson to carry: a large fraction of production complexity in long-lived systems is not bad design — it is compatibility with things that already shipped. Learning to distinguish "this is complex because the problem is hard" from "this is complex because 1997 happened" is a Principal-level reading skill, and HTML parsing is the best training ground for it.

The 21 insertion modes, and what each is for

kInitialMode              kBeforeHTMLMode           kBeforeHeadMode
kInHeadMode               kInHeadNoscriptMode       kAfterHeadMode
kTemplateContentsMode     kInBodyMode               kTextMode
kInTableMode              kInTableTextMode          kInCaptionMode
kInColumnGroupMode        kInTableBodyMode          kInRowMode
kInCellMode               kAfterBodyMode            kInFramesetMode
kAfterFramesetMode        kAfterAfterBodyMode       kAfterAfterFramesetMode

Grouped by why they exist:

• Document skeleton (Initial, BeforeHTML, BeforeHead, InHead, AfterHead, AfterBody, AfterAfterBody) — enforcing that a document has the right shape even when the markup does not say so. This is why <p>hi produces a full html/head/body tree.
• Tables (InTable, InTableText, InCaption, InColumnGroup, InTableBody, InRow, InCell) — seven of twenty-one modes are tables. That ratio is the honest measure of how much table markup cost the platform.
• Framesets (InFrameset, AfterFrameset, AfterAfterFrameset) — a deprecated feature that still owns three modes.
• Special content (Text, TemplateContents, InHeadNoscript).

kAfterAfterBodyMode is worth a moment: it is the state after </html>. It exists because content can legally appear there and must go somewhere sensible — which is why whitespace after </html> still lands inside body (case C of the lab).

Deep dive: the adoption agency algorithm

The most notorious part of the spec, and the reason the "list of active formatting elements" exists at all. The problem:

<b>bold <p>both</b> italic-ish</p>

Formatting elements (<b>, <i>, <em>, <strong>, <a>, <font>…) can be left open across block boundaries, producing markup whose intended nesting is impossible as a tree. The DOM has no way to represent "this <b> overlaps that <p>."

The algorithm's job is to reconstruct formatting so the visual result matches author intent, by cloning formatting elements into the new block. That is why you get:

<b>bold </b><p><b>both</b> italic-ish</p>

— the <b> was duplicated, because overlapping ranges must become nested trees.

Things worth knowing:

• The list of active formatting elements is separate from the stack of open elements, and the "Noah's Ark clause" limits it to three identical entries so that pathological markup cannot blow up.
• Blink has a test file named after the algorithm (html_tree_builder_adoption_agency_test.cc). A dedicated test file named after one algorithm is a signal: it means the algorithm is both intricate and repeatedly gotten wrong.
• It is the standard example of "the spec is complex because the DOM is a tree and markup is not."

Do not implement this in your mini parser. Read it, understand why it exists, and record it in the complexity notebook.

Deep dive: speculation, and what it may not do

Two scanners, already tabulated in §2. The deeper point is the discipline they follow:

A speculative pass may be wrong, but it may never be observable.

The preload scanner may request a resource that turns out unnecessary — that costs bandwidth, not correctness. The background scanner may stream-compile a script that never runs — that costs CPU. Neither may create a DOM node, run script, or change parser state.

This is the general shape of every safe speculation in the browser, and once you see it you will recognise it everywhere:

Design principle worth stealing: to speed up a blocking algorithm you cannot restructure, add a side-effect-free pass that runs ahead and is permitted to be wrong.

Deep dive: scripts, and the four ways they block

The parser's relationship with <script> is the reason it cannot move off the main thread.

Two consequences that show up in real products:

• async scripts are an ordering hazard, not just a loading strategy. Two async scripts where one depends on the other is a race that passes locally and fails on a slow network.
• A blocking script in <head> stalls tree construction, which is why "put scripts at the end of body" was the advice before defer existed, and why defer is now the better answer (it preserves order and does not block).

document.write, the reason all this is stuck

document.write inserts text into the input stream at the current position. The parser must therefore be able to be interrupted, have its input mutated, and resume. That single API forecloses a fully off-main-thread parser, and it is why Chromium chose speculation over relocation.

It is also actively hostile to performance on slow connections — a document.write of a <script src> in the middle of a page can serialise loading — which is why browsers have shipped interventions that ignore it in some circumstances. That is a rare and instructive event: the platform breaking spec-defined behaviour deliberately, because the data said users were better off.

Deep dive: encoding, and the parse you cannot undo

Before tokenizing you must know the encoding, and you learn it from the bytes you are decoding.

Order of authority, roughly: BOM → HTTP Content-Type charset → <meta charset> in the first chunk → heuristics/locale default.

The awkward case is <meta charset> appearing after you have already begun decoding. The parser scans a prefix looking for it; if a declaration is found late and contradicts the current assumption, the parse must be restarted. That is why <meta charset> is specified to appear early (within the first 1024 bytes) and why it is worth putting first in <head>.

Note what class of problem this is: an input whose interpretation depends on its own content. It shows up again in text shaping, in MIME sniffing, and in bundler module-format detection.

Deep dive: <template>, and why it is a separate mode

kTemplateContentsMode exists because <template> content must be parsed but inert: no scripts run, no images load, no custom elements upgrade. Its children live in a separate DocumentFragment (.content), not in the document tree.

Consequences you can observe:

• document.querySelector('template img') finds nothing — the img is not in the document.
• Template content survives table parsing rules that would otherwise foster-parent it, because the tree builder tracks templates separately. FindFosterSite checks for a topmost template before it checks for a table — which is exactly the "field you did not derive" the lab's answer key points at.

That ordering is not arbitrary: if a template is open inside a table, content belongs to the template, not foster-parented out of the table. Reading that one if in the right order tells you the whole design.

Deep dive: what the parser hands downstream

Parsing does not end at "a DOM exists." As nodes are created the parser also drives:

• custom element reactions — queued, not immediate (bi-04), so author code never observes a half-built tree;
• style recalculation scheduling — new elements are dirty;
• resource loading — <img>, <link>, <script> discovery;
• DOMContentLoaded — fired when parsing finishes and deferred scripts have run;
• incremental rendering — the parser yields so the page can paint before the document is complete. This is why a long document renders progressively rather than appearing at once, and it is a scheduling decision (bi-11) as much as a parsing one.

That last point is the one to hold: the parser deliberately stops parsing to let the page paint. A parser optimised purely for throughput would produce a worse product.

Anti-Patterns

Assuming innerHTML round-trips. el.innerHTML = el.innerHTML can change the DOM. Serialisation and parsing are not inverses.

Reasoning about <table> markup from intuition. Table parsing is the densest special- case area in the spec. Check, don't reason.

Treating "parse error" as "the browser rejected it." Nothing is rejected. Ever.

Concluding a behaviour is a bug because it is ugly. the parsing lab Case B is legal, specified, and intentional. The bug hypothesis should come after the spec check.

Writing a sanitiser on top of a regex or your own parser. The mismatch between your parser and the browser's is the vulnerability class — mutation XSS is exactly "the sanitiser and the parser disagreed about what this string means." This is the strongest practical argument for why exact parser semantics matter to application engineers.

Trade-offs

Spec fidelity vs speed. Blink keeps a fully-correct implementation and adds fast paths that bail out. The alternative — a fast parser that is subtly wrong — was the pre-HTML5 world.

Main-thread parsing vs correctness. A fully off-main-thread parser is impossible while document.write exists. Chromium chose speculation (side-effect-free, discardable) over relocation. Note the general principle: when you cannot move blocking work, move the predictable part of it and keep the ability to be wrong.

Streaming vs buffering. The parser consumes bytes as they arrive so paint can start early, which is why the tokenizer is resumable mid-token and the input stream is a segmented structure rather than a String. Buffering would be far simpler and would delay first paint.

Lab

the parsing lab — foster parenting (bi-03-html-parsing/docs/execution.md) — prerequisite.

mini-browser M2–M3 — extend foster.js into a real tokenizer + tree builder:

1. Tokenizer as an explicit state machine: data, tag open, tag name, attribute name, attribute value (quoted/unquoted), comment, DOCTYPE. Named character references may be stubbed.
2. Tree builder with a stack of open elements and at least: initial, before html, before head, in head, after head, in body, in table, in table text.
3. Error recovery for: unclosed <p>, unclosed <li>, stray </div>, text in tables.
4. Tokenizer/tree-builder coupling — implement <title>/<textarea> switching the tokenizer's content model. This is the stage that teaches the real lesson.
5. A conceptual <script> pause: stop the parse, run a callback, resume. You do not need a JS engine — you need the control flow.

Then compare to Blink: for each of your states, find its counterpart; for each of Blink's that you omitted, say what markup needs it.

Failure Lab

1. Remove foster parenting. Which real-world markup breaks? Predict, then test against a corpus of your own saved pages.
2. Remove the in-table-text buffering and insert characters immediately. Explain why the batching is load-bearing rather than an optimisation. (This is the deepest question in the parsing lab Step 7.)
3. Make the tokenizer independent of the tree builder — no content-model switching. Find markup that now parses catastrophically wrong. This one is a proof that the two machines cannot be decoupled.
4. Mutation XSS: write a naive sanitiser that strips <script> from a string, then find an input where your sanitiser's parse and the browser's parse disagree. Do this defensively, on your own machine, against your own page — the point is to internalise why parser-aware sanitisation (or Trusted Types) is the only sound approach.

Debugging Exercise

With the local content_shell build:

1. Breakpoint in the tree builder's in-table start-tag handler. Load the parsing lab's a.html. Capture the call stack — how did you get here from the network?
2. Set a conditional breakpoint on the foster-parenting predicate, condition it on the current node being a table, and confirm the parsing lab Case A step 5 by observation rather than by reading.
3. Trace a real page load with the blink category. Find the parser's trace events, and identify: how many chunks the parse was split into, and where scripts blocked it.
4. Find where the construction site's queued tasks are flushed. Set a breakpoint and answer from the stack: what triggers a flush?

Testing & QA Considerations

• Find the WPT directory for HTML parsing and identify the html5lib-style data-driven tests. What format are they in, and why is a data-driven format the right choice here?
• Find a parser-related entry in TestExpectations. What behaviour does Chromium currently get wrong, and is it a known interop gap or a deliberate deviation?
• Write a test in the html5lib format for the parsing lab Case B. Run it against Blink.

Further Reading (primary sources first)

• WHATWG HTML Standard §13 Parsing HTML documents — tokenization and tree construction. Read the "in table" and "in table text" modes in full; they are shorter than they look.
• WHATWG HTML Standard — fragment parsing algorithm, and the "appropriate place for inserting a node" algorithm.
• third_party/blink/renderer/core/html/parser/ — start from html_document_parser.h, and read html_document_parser_fastpath.h for the fragment fast path.
• html5lib test data format (used by WPT and by most non-browser HTML parsers).
• Blink's SpecMapping.md for the spec→directory mapping.

Principal Engineer Review

1. Why does the HTML spec define error recovery at all, given that it makes the parser far more complex? What would the web look like if it had not, and what is the modern equivalent decision you would apply this lesson to?

2. element.innerHTML = element.innerHTML can change the DOM. Explain the mechanism, and give a realistic production bug this causes.

3. Your team's SSR output differs from the client-side DOM for the same component. List the parser-level causes, ranked by likelihood, and the fastest check for each.

4. Argue that the tokenizer and tree builder should be fully decoupled for maintainability. Then defeat your own argument with a concrete markup example.

5. document.write is the main obstacle to off-main-thread parsing. Design a deprecation path for it. What breaks, who complains, and how would you actually measure whether the removal is safe?

6. Blink ships a fragment-parsing fast path that bails out to the general algorithm. What invariant must hold for such a fast path to be safe, and how would you test that the two paths agree? What is the failure mode if they silently diverge?

7. The preload scanner runs on the main thread and only scans the first chunk; the background scanner runs on a worker and scans everything. Reconstruct why these are two mechanisms rather than one. What would it take to merge them?

8. A security engineer proposes sanitising HTML with a regex "as a defence in depth layer." Explain precisely why this can make things worse rather than merely being insufficient.

9. Foster parenting exists to handle markup nobody writes deliberately. Make the case for removing it from the platform, then estimate what evidence you would need to justify the attempt. Who would you have to convince, and what would the rollout look like?

10. You find a Blink parser behaviour that contradicts the spec. Walk through your next steps in order, including how you decide whether the spec is what should change.