Concepts — Debugging and Tracing Chromium

Phase 4 · Spec areas §20 (debugging), §21 (tracing). Requires a working local build — see bi-00-roadmap/docs/chromium-build-debug-trace.md.

0. Orientation

The 60-second version

Debugging Chromium is mostly instrument selection. The expensive mistake is not choosing the wrong breakpoint — it is reaching for a debugger when a trace would have answered in a tenth of the time, or reaching for a trace when the answer was in Code Search.

The ordering that matters, restated from bi-01 because people keep getting it backwards:

A breakpoint encodes a hypothesis about where. A trace produces the hypothesis.

So when you have a behaviour and no theory, trace first — on stock Chrome, with no build at all. When you have a theory and need exact state and ordering, that is what the debugger is for, and it is the most expensive instrument you own.

What each instrument actually answers

The constraint that runs through all of it

Instrumentation that changes the system produces fiction. Verbose logging in a hot path changes the timings you are measuring. A breakpoint in a frame-production path changes frame production. This is the same rule fw-12 states for application-level devtools, and it is more acute here because the code you are instrumenting is often the timing-sensitive part.

What you should be able to do after this module

1. Why a Principal Engineer needs this

1. Source reading answers "what could happen." Debugging answers "what did happen." The gap between those two is where wrong conclusions live, and in a codebase with generated code, feature flags, and platform branches, the gap is wide.

2. Tracing is the highest-leverage skill in this entire track, because it works without a build, spans processes and threads, and correlates directly back to source via trace event names. It is bi-01's rung 3, and it is the tool most people learn last and should learn first.

3. Reproduce → isolate → fix → verify is the contribution workflow (bi-14). Without debugging fluency you can read Chromium but not change it.

2. Mental Model

2.1 Pick the right instrument

Reach for tracing before the debugger when you have no hypothesis. A breakpoint requires you to already know where to put it; a trace tells you where to put it.

2.2 Trace events are greppable identifiers

Trace events come from TRACE_EVENT macros in the source with literal string names. So:

see an event in Perfetto  ->  grep its exact name  ->  land in the code that ran

This is the bridge between observation and source, and it is the single most useful technique in this module. It also works in reverse: add a TRACE_EVENT to code you are studying, rebuild that target, and watch your own event appear in the timeline.

2.3 Multiprocess debugging

A Blink breakpoint must be set in a renderer, not the browser process you launched. Chromium provides startup-dialog flags that pause a child process until you attach. Combine with --disable-hang-monitor so sitting at a breakpoint does not get your renderer killed.

The three reasons a breakpoint does not hit, in order of frequency:

1. wrong process,
2. the function was inlined (set it on the caller),
3. the path is behind a runtime-enabled feature that is off.

2.4 DCHECKs are your friend, if you built with them

With dcheck_always_on = true, breaking an invariant produces a precise message naming the invariant instead of a confusing misrender ten stages later. For a learning checkout this is the highest-value single build flag.

2.5 Logging

LOG()/VLOG() with --enable-logging=stderr --v=1. Verbosity can be scoped per module, which matters in a codebase where global verbose logging is unreadable. Logging beats a debugger when the interesting behaviour is a sequence across many iterations.

3. Practice

Concrete procedures live in bi-00-roadmap/docs/chromium-build-debug-trace.md §4–§5: lldb setup and the in-tree pretty-printers, attaching to a paused renderer, conditional breakpoints, thread backtrace all, and the tracing categories worth recording.

Do thread backtrace all once, early. It converts bi-02's thread diagram from a claim into something you have seen.

3.5 Deep dive: trace categories are a catalogue, and it is in the tree

base/trace_event/builtin_categories.h (~27 KB, roughly 500 category strings) is the authoritative list. You do not have to guess which categories to enable — read the file.

Measured distribution of TRACE_EVENT categories inside core/ alone:

Two things follow.

devtools.timeline is a trace category. The DevTools Performance panel is a view over the same tracing system you can drive yourself from Perfetto. It is curated and JS-centric; the underlying data is richer. Once you know this, "DevTools didn't show me anything" stops being a dead end — you are looking at a filtered view and can widen it.

The categories you actually want, by question:

Trace events are greppable identifiers — the bidirectional bridge

This is the single most useful technique in this module and it works both ways:

observe an event in Perfetto  →  grep its exact literal name  →  land in the code that ran
add a TRACE_EVENT to code     →  rebuild that target          →  watch your event appear

The first direction turns a timeline into source navigation. The second turns source reading into an experiment. Together they are the reason tracing sits at rung 3 of the bi-01 ladder rather than being a specialist performance tool.

Event shapes worth recognising in a trace:

• Duration events (begin/end pairs) — nested, forming the flame chart.
• Instant events — a point in time.
• Async / flow events — the arrows that cross threads and processes. These are how you follow one frame from the renderer main thread to the compositor to the GPU process, which is the observation that makes bi-10 concrete rather than diagrammatic.
• Counters — values over time (memory, tile counts).

3.6 Deep dive: choosing an instrument, expanded

bi-01's ladder said tracing sits above search and below the debugger. Here is the fuller decision table.

The italicised rule: a debugger perturbs time; tracing perturbs it far less. Any bug whose existence depends on ordering is a tracing problem, and reaching for a breakpoint will make it disappear — which is the most frustrating way to lose an afternoon.

3.7 Deep dive: making a Blink breakpoint actually hit

The three failure modes, with fixes.

1. Wrong process. Blink runs in a renderer. Attaching to the browser process you launched gets you nothing. Use the startup-dialog flags to pause a child process and attach to it, and always pass --disable-hang-monitor so a paused renderer is not killed for unresponsiveness.

2. Inlined. Release builds inline aggressively. Symptoms: the breakpoint "resolves" but never fires, or the stack has fewer frames than the source suggests. Fixes, cheapest first:

• set the breakpoint on the caller instead,
• break on a line rather than a symbol,
• rebuild the one file with lower optimisation,
• fall back to a TRACE_EVENT or LOG, which cannot be optimised away.

3. Behind a feature flag. The code exists and is never reached. Check runtime_enabled_features.json5 (bi-01 Technique 4) and try --enable-blink-features=YourFeature.

Conditional breakpoints that are actually usable

Blink's hot paths run thousands of times per frame, so unconditional breakpoints are useless.

(lldb) breakpoint set -n blink::Element::SetAttribute -c 'name == "class"'
(lldb) breakpoint set -n blink::Document::UpdateStyleAndLayout -i 50   # ignore first 50 hits
(lldb) breakpoint command add 1
> bt 12
> continue
> DONE

That last pattern — breakpoint plus automatic backtrace plus continue — gives you a sampled call stack log without stopping execution. It is the debugger being used as an instrument rather than as a pause button, and it is the right tool for "who calls this, in practice, on a real page."

3.8 Deep dive: thread backtrace all, and reading a renderer's threads

Run it once, early, and write down the names. It converts bi-02's thread diagram from a claim into something you have observed.

What you should be able to identify:

• CrRendererMain — Blink, V8, style, layout, paint, rAF
• the compositor thread — cc impl side, input handling for scroll
• raster / worker threads — a pool
• Chrome_ChildIOThread — Mojo message send/receive; never do work here
• worker and worklet threads if the page uses them

Two diagnostics fall straight out of this:

• If your breakpoint is on the compositor thread but you expected the main thread, your mental model of the subsystem is wrong — stop and fix it before reading more code.
• If the IO thread has a deep application stack in it, that is a bug on its own.

3.9 Deep dive: memory investigation

Different tools than performance, and worth naming so you do not reach for the wrong one.

The bi-04 skill — reading a retainer chain — is the one that transfers. Finding that memory grew is easy; finding what is holding it is the job. In a two-heap system the chain may cross from a JS closure to a C++ node and back, which is precisely why the unified heap exists and why the snapshot can show you the path at all.

3.10 Deep dive: logging, done properly

out/Default/content_shell --enable-logging=stderr --v=1 <url>
out/Default/content_shell --enable-logging=stderr --vmodule=html_document_parser=2,style_engine=1

--vmodule is the important one: per-module verbosity. Global --v=2 in Chromium produces an unreadable firehose; scoping to the two files you care about produces something you can actually read.

In code:

VLOG(1) << "state=" << state;            // level-gated, compiled in
DVLOG(1) << ...                          // debug builds only
LOG(ERROR) << ...                        // always
TRACE_EVENT("blink", "MyThing");         // shows in the timeline, correlates across threads

Prefer a TRACE_EVENT over a LOG when the question is when or how long; prefer VLOG when the question is what value. The reason is simple: a log line is a string in a stream with no timeline, no thread correlation, and no flow arrows.

4. Anti-Patterns

Debugging with --single-process. Collapses the boundaries you are studying and is not maintained to production quality.

--no-sandbox as a default. Changes the security model; never draw behavioural conclusions from it.

Adding printf and rebuilding chrome. Build content_shell, or better, add a trace event.

Believing a stack trace from a symbol_level = 0 build.

Long debugging sessions without writing anything down. In a codebase this size, an un-recorded finding is a finding you will re-derive.

5. Trade-offs

Tracing vs debugging. Tracing perturbs timing least and shows the system; the debugger shows exact state and destroys timing. Timing-dependent bugs are usually tracing problems.

DCHECK on vs off. On: earlier, clearer failures. Off: closer to shipping behaviour and faster. For learning, on.

Symbols vs disk/build time. bi-00-roadmap/docs/chromium-build-debug-trace.md recommends symbol_level = 1 with blink_symbol_level = 2 as the compromise.

6. Lab

1. Thread census. Attach to a renderer, thread backtrace all, and write down every thread name. Map each to bi-02's diagram. Note any you did not expect.
2. Pipeline breakpoints. Set breakpoints at DOM creation, style recalc, layout, and paint. Load a trivial page. Record the order and the call stacks connecting them. This is the rendering pipeline, observed rather than read.
3. Conditional breakpoint. Break in setAttribute only when the attribute is class.
4. Trace-to-source. Record a trace, pick three unfamiliar event names, and find each in the source. Write down what each one measures.
5. Source-to-trace. Add your own TRACE_EVENT to a Blink function you studied in bi-03 or bi-07. Rebuild that target only. Confirm it appears.
6. Break an invariant. Deliberately violate a DCHECK (for example by removing a guard), rebuild, and observe the assertion fire. Restore.

Deliverable: the thread census, the pipeline call-stack chain from (2), and the diff from (5).

7. Failure Lab

1. Build with symbol_level = 0 and try to debug. Experience it once; it explains the flag.
2. Set a Blink breakpoint on the browser process and observe it never hit. Diagnose from first principles.
3. Break in a function that gets inlined. Find the workaround.
4. Attach to a renderer without --disable-hang-monitor and wait 30 seconds. Explain the kill.

8. References

• docs/mac/debugging.md, docs/lldbinit.md in your checkout.
• docs/ tracing documentation; Perfetto UI docs.
• base/trace_event/ — the TRACE_EVENT macros themselves.
• bi-00-roadmap/docs/chromium-build-debug-trace.md — your operational reference.

Core terminology

Mental models

Choose the instrument before you start, and stop at the cheapest one that answers. Most wasted debugging time is spent in a more expensive instrument than the question required. Naming the question first — did it run? vs what was the value? vs in what order? — selects the tool mechanically.

Tracing is the hypothesis generator; the debugger is the hypothesis checker. This ordering is the single most common thing people get wrong, and it is why bi-01 puts tracing at rung 3.

Trace categories are a catalogue, and it is in the tree. You do not have to guess which category to enable — the instrumentation points are greppable, which turns "what should I turn on?" into a search rather than a folklore question.

A breakpoint that does not hit is usually about processes, not about the code. Multi-process, sandboxed, with the interesting work happening in a renderer that starts after you attach. Knowing the flag family that resolves this is the difference between debugging and not.

Read all threads, not one stack. A renderer's behaviour is a relationship between the main thread, compositor, raster pool and IO thread. A single backtrace answers almost nothing about that.

Instrumentation that perturbs is producing fiction. Verbose logging in a hot path, or a breakpoint inside frame production, changes what you are measuring. Prefer tracing — which is designed for low overhead — when timing is part of the question.

FROM_HERE is why traces are attributable. The posted-task machinery carries source locations, which is what makes a trace readable rather than a wall of anonymous slices — an argument for instrumenting your own systems the same way.

Common misconceptions

1. "Debugging starts with a debugger." It starts with choosing an instrument, and the debugger is the most expensive one.

2. "Tracing is for performance work." Tracing answers whether and where code runs, which makes it a correctness tool at least as often.

3. "You need a local build to trace." Tracing works on stock Chrome. That is much of why it outranks a build in the ladder.

4. "My breakpoint isn't hit, so the code doesn't run." More often the process you attached to is not the one running it.

5. "More logging is more information." Past a point it changes the timings and buries the signal.

6. "A backtrace tells you what happened." In a multi-threaded renderer, one stack omits the relationship that is usually the answer.

7. "about:memory will show the leak." It shows allocator-level totals. Which heap you are asking about determines the tool (bi-04, fe-02).

8. "Enabling all trace categories gives the fullest picture." It produces overhead and volume that obscure the thing you were looking for.

Interview talking points

• "Most wasted debugging time is spent in a more expensive instrument than the question needed. I name the question first — did it run, what was the value, in what order — and that selects the tool mechanically."
• "The ordering people get backwards is that tracing sits above search and below the debugger. A breakpoint encodes a hypothesis about where; a trace produces the hypothesis. So with a behaviour and no theory, I trace first — on stock Chrome, with no build at all."
• "A breakpoint that doesn't hit is usually a process problem rather than a code problem. It's a multi-process sandboxed browser and the interesting work is in a renderer that starts after you attached. Knowing the flag family that fixes that is the difference between debugging and guessing."
• "In a renderer I read all threads, not one stack. The behaviour is a relationship between the main thread, the compositor, the raster pool and IO — a single backtrace answers almost nothing about that."
• "Instrumentation that perturbs the system is producing fiction, and it's acute here because the code you're instrumenting is often the timing-sensitive part. That's the same rule I'd apply to any home-grown observability layer: if it changes what it measures, the numbers aren't real."

9. Principal Engineer Review

1. You have a rendering bug that reproduces once in 50 loads. Argue for tracing over a debugger, then describe the trace you would capture and what you would look for.

2. DCHECK is compiled out in release. Give a scenario where this means users hit a bug that would have been caught, and say what Chromium does about that class of risk.

3. A teammate debugs with --single-process because it is convenient. Explain the risk with a concrete example of a conclusion it would make them draw wrongly.

4. Design the minimum instrumentation you would add to a subsystem you own so that a stranger could diagnose it from a trace alone. What are you deliberately not instrumenting?

5. Trace events are string identifiers in source. Argue this coupling is good design; then name its failure mode at scale.

6. You must hand off a hard bug at end of day. Write the note: what you observed, what you ruled out, what you would do next, and how someone reproduces it.

7. When is adding a log statement better engineering than setting a breakpoint? Give two cases.

8. You are asked to reduce a team's mean time-to-diagnose for rendering bugs. What do you change first — tooling, process, or knowledge — and how would you know it worked?

Connections to other modules

• bi-01 — the tool ladder this module is rungs 3 and 5 of.
• bi-02 — why breakpoints are a process problem, and which process to attach to.
• bi-06 — FROM_HERE, SEQUENCE_CHECKER and the annotations that make traces legible.
• bi-10 / bi-11 — the frame and scheduling behaviour you will most often be tracing.
• bi-15 — vertical traces are this module's skills used end to end.
• fe-01 — LoAF versus longtask attribution, and harness bugs that produce clean fake data.
• fe-02 — which heap you are asking about determines the instrument.
• fw-12 — the same observer-effect constraint, at application level.