Concepts — Debugging and Tracing Chromium

Phase 4 · Spec areas §20 (debugging), §21 (tracing). Requires a working local build — see bi-00-roadmap/docs/chromium-build-debug-trace.md.

1. Why a Principal Engineer needs this

1. Source reading answers "what could happen." Debugging answers "what did happen." The gap between those two is where wrong conclusions live, and in a codebase with generated code, feature flags, and platform branches, the gap is wide.

2. Tracing is the highest-leverage skill in this entire track, because it works without a build, spans processes and threads, and correlates directly back to source via trace event names. It is bi-01's rung 3, and it is the tool most people learn last and should learn first.

3. Reproduce → isolate → fix → verify is the contribution workflow (bi-14). Without debugging fluency you can read Chromium but not change it.

2. Mental Model

2.1 Pick the right instrument

Reach for tracing before the debugger when you have no hypothesis. A breakpoint requires you to already know where to put it; a trace tells you where to put it.

2.2 Trace events are greppable identifiers

Trace events come from TRACE_EVENT macros in the source with literal string names. So:

see an event in Perfetto  ->  grep its exact name  ->  land in the code that ran

This is the bridge between observation and source, and it is the single most useful technique in this module. It also works in reverse: add a TRACE_EVENT to code you are studying, rebuild that target, and watch your own event appear in the timeline.

2.3 Multiprocess debugging

A Blink breakpoint must be set in a renderer, not the browser process you launched. Chromium provides startup-dialog flags that pause a child process until you attach. Combine with --disable-hang-monitor so sitting at a breakpoint does not get your renderer killed.

The three reasons a breakpoint does not hit, in order of frequency:

1. wrong process,
2. the function was inlined (set it on the caller),
3. the path is behind a runtime-enabled feature that is off.

2.4 DCHECKs are your friend, if you built with them

With dcheck_always_on = true, breaking an invariant produces a precise message naming the invariant instead of a confusing misrender ten stages later. For a learning checkout this is the highest-value single build flag.

2.5 Logging

LOG()/VLOG() with --enable-logging=stderr --v=1. Verbosity can be scoped per module, which matters in a codebase where global verbose logging is unreadable. Logging beats a debugger when the interesting behaviour is a sequence across many iterations.

3. Practice

Concrete procedures live in bi-00-roadmap/docs/chromium-build-debug-trace.md §4–§5: lldb setup and the in-tree pretty-printers, attaching to a paused renderer, conditional breakpoints, thread backtrace all, and the tracing categories worth recording.

Do thread backtrace all once, early. It converts bi-02's thread diagram from a claim into something you have seen.

3.5 Deep dive: trace categories are a catalogue, and it is in the tree

base/trace_event/builtin_categories.h (~27 KB, roughly 500 category strings) is the authoritative list. You do not have to guess which categories to enable — read the file.

Measured distribution of TRACE_EVENT categories inside core/ alone:

Two things follow.

devtools.timeline is a trace category. The DevTools Performance panel is a view over the same tracing system you can drive yourself from Perfetto. It is curated and JS-centric; the underlying data is richer. Once you know this, "DevTools didn't show me anything" stops being a dead end — you are looking at a filtered view and can widen it.

The categories you actually want, by question:

Trace events are greppable identifiers — the bidirectional bridge

This is the single most useful technique in this module and it works both ways:

observe an event in Perfetto  →  grep its exact literal name  →  land in the code that ran
add a TRACE_EVENT to code     →  rebuild that target          →  watch your event appear

The first direction turns a timeline into source navigation. The second turns source reading into an experiment. Together they are the reason tracing sits at rung 3 of the bi-01 ladder rather than being a specialist performance tool.

Event shapes worth recognising in a trace:

• Duration events (begin/end pairs) — nested, forming the flame chart.
• Instant events — a point in time.
• Async / flow events — the arrows that cross threads and processes. These are how you follow one frame from the renderer main thread to the compositor to the GPU process, which is the observation that makes bi-10 concrete rather than diagrammatic.
• Counters — values over time (memory, tile counts).

3.6 Deep dive: choosing an instrument, expanded

bi-01's ladder said tracing sits above search and below the debugger. Here is the fuller decision table.

The italicised rule: a debugger perturbs time; tracing perturbs it far less. Any bug whose existence depends on ordering is a tracing problem, and reaching for a breakpoint will make it disappear — which is the most frustrating way to lose an afternoon.

3.7 Deep dive: making a Blink breakpoint actually hit

The three failure modes, with fixes.

1. Wrong process. Blink runs in a renderer. Attaching to the browser process you launched gets you nothing. Use the startup-dialog flags to pause a child process and attach to it, and always pass --disable-hang-monitor so a paused renderer is not killed for unresponsiveness.

2. Inlined. Release builds inline aggressively. Symptoms: the breakpoint "resolves" but never fires, or the stack has fewer frames than the source suggests. Fixes, cheapest first:

• set the breakpoint on the caller instead,
• break on a line rather than a symbol,
• rebuild the one file with lower optimisation,
• fall back to a TRACE_EVENT or LOG, which cannot be optimised away.

3. Behind a feature flag. The code exists and is never reached. Check runtime_enabled_features.json5 (bi-01 Technique 4) and try --enable-blink-features=YourFeature.

Conditional breakpoints that are actually usable

Blink's hot paths run thousands of times per frame, so unconditional breakpoints are useless.

(lldb) breakpoint set -n blink::Element::SetAttribute -c 'name == "class"'
(lldb) breakpoint set -n blink::Document::UpdateStyleAndLayout -i 50   # ignore first 50 hits
(lldb) breakpoint command add 1
> bt 12
> continue
> DONE

That last pattern — breakpoint plus automatic backtrace plus continue — gives you a sampled call stack log without stopping execution. It is the debugger being used as an instrument rather than as a pause button, and it is the right tool for "who calls this, in practice, on a real page."

3.8 Deep dive: thread backtrace all, and reading a renderer's threads

Run it once, early, and write down the names. It converts bi-02's thread diagram from a claim into something you have observed.

What you should be able to identify:

• CrRendererMain — Blink, V8, style, layout, paint, rAF
• the compositor thread — cc impl side, input handling for scroll
• raster / worker threads — a pool
• Chrome_ChildIOThread — Mojo message send/receive; never do work here
• worker and worklet threads if the page uses them

Two diagnostics fall straight out of this:

• If your breakpoint is on the compositor thread but you expected the main thread, your mental model of the subsystem is wrong — stop and fix it before reading more code.
• If the IO thread has a deep application stack in it, that is a bug on its own.

3.9 Deep dive: memory investigation

Different tools than performance, and worth naming so you do not reach for the wrong one.

The bi-04 skill — reading a retainer chain — is the one that transfers. Finding that memory grew is easy; finding what is holding it is the job. In a two-heap system the chain may cross from a JS closure to a C++ node and back, which is precisely why the unified heap exists and why the snapshot can show you the path at all.

3.10 Deep dive: logging, done properly

out/Default/content_shell --enable-logging=stderr --v=1 <url>
out/Default/content_shell --enable-logging=stderr --vmodule=html_document_parser=2,style_engine=1

--vmodule is the important one: per-module verbosity. Global --v=2 in Chromium produces an unreadable firehose; scoping to the two files you care about produces something you can actually read.

In code:

VLOG(1) << "state=" << state;            // level-gated, compiled in
DVLOG(1) << ...                          // debug builds only
LOG(ERROR) << ...                        // always
TRACE_EVENT("blink", "MyThing");         // shows in the timeline, correlates across threads

Prefer a TRACE_EVENT over a LOG when the question is when or how long; prefer VLOG when the question is what value. The reason is simple: a log line is a string in a stream with no timeline, no thread correlation, and no flow arrows.

4. Anti-Patterns

Debugging with --single-process. Collapses the boundaries you are studying and is not maintained to production quality.

--no-sandbox as a default. Changes the security model; never draw behavioural conclusions from it.

Adding printf and rebuilding chrome. Build content_shell, or better, add a trace event.

Believing a stack trace from a symbol_level = 0 build.

Long debugging sessions without writing anything down. In a codebase this size, an un-recorded finding is a finding you will re-derive.

5. Trade-offs

Tracing vs debugging. Tracing perturbs timing least and shows the system; the debugger shows exact state and destroys timing. Timing-dependent bugs are usually tracing problems.

DCHECK on vs off. On: earlier, clearer failures. Off: closer to shipping behaviour and faster. For learning, on.

Symbols vs disk/build time. bi-00-roadmap/docs/chromium-build-debug-trace.md recommends symbol_level = 1 with blink_symbol_level = 2 as the compromise.

6. Lab

1. Thread census. Attach to a renderer, thread backtrace all, and write down every thread name. Map each to bi-02's diagram. Note any you did not expect.
2. Pipeline breakpoints. Set breakpoints at DOM creation, style recalc, layout, and paint. Load a trivial page. Record the order and the call stacks connecting them. This is the rendering pipeline, observed rather than read.
3. Conditional breakpoint. Break in setAttribute only when the attribute is class.
4. Trace-to-source. Record a trace, pick three unfamiliar event names, and find each in the source. Write down what each one measures.
5. Source-to-trace. Add your own TRACE_EVENT to a Blink function you studied in bi-03 or bi-07. Rebuild that target only. Confirm it appears.
6. Break an invariant. Deliberately violate a DCHECK (for example by removing a guard), rebuild, and observe the assertion fire. Restore.

Deliverable: the thread census, the pipeline call-stack chain from (2), and the diff from (5).

7. Failure Lab

1. Build with symbol_level = 0 and try to debug. Experience it once; it explains the flag.
2. Set a Blink breakpoint on the browser process and observe it never hit. Diagnose from first principles.
3. Break in a function that gets inlined. Find the workaround.
4. Attach to a renderer without --disable-hang-monitor and wait 30 seconds. Explain the kill.

8. References

• docs/mac/debugging.md, docs/lldbinit.md in your checkout.
• docs/ tracing documentation; Perfetto UI docs.
• base/trace_event/ — the TRACE_EVENT macros themselves.
• bi-00-roadmap/docs/chromium-build-debug-trace.md — your operational reference.

9. Principal Engineer Review

1. You have a rendering bug that reproduces once in 50 loads. Argue for tracing over a debugger, then describe the trace you would capture and what you would look for.

2. DCHECK is compiled out in release. Give a scenario where this means users hit a bug that would have been caught, and say what Chromium does about that class of risk.

3. A teammate debugs with --single-process because it is convenient. Explain the risk with a concrete example of a conclusion it would make them draw wrongly.

4. Design the minimum instrumentation you would add to a subsystem you own so that a stranger could diagnose it from a trace alone. What are you deliberately not instrumenting?

5. Trace events are string identifiers in source. Argue this coupling is good design; then name its failure mode at scale.

6. You must hand off a hard bug at end of day. Write the note: what you observed, what you ruled out, what you would do next, and how someone reproduces it.

7. When is adding a log statement better engineering than setting a breakpoint? Give two cases.

8. You are asked to reduce a team's mean time-to-diagnose for rendering bugs. What do you change first — tooling, process, or knowledge — and how would you know it worked?