Profiling Linux Code with perf and Flamegraphs

When your program is slow and you do not know where the time goes, the fastest answer is stack sampling: record what the CPU was executing hundreds of times a second, cluster the call stacks, and look at where the samples concentrate. The perf tool on Linux does the recording, and a flamegraph turns thousands of samples into a picture your eye reads in seconds.

Recording a CPU profile

Start by reproducing the slow behaviour, then attach perf to the running process and sample. The choice of call-graph style matters; dwarf gives accurate stacks for most compiled code with debug symbols, while fp relies on frame pointers.

# sample the live process for 30 seconds at 99 Hz
perf record -p $(pgrep -f myapp) -F 99 --call-graph dwarf -o /tmp/perf.data sleep 30
# inspect the flat profile first
perf report -i /tmp/perf.data --stdio | head -n 40

The flat report shows each function's share of samples. The one at the top, with the layout, is the first lead. But a flat table hides the path that got there. A flamegraph keeps that context.

Flattening to a flamegraph

Flamegraphs stack function frames, width proportional to sample count, called functions above callers. The wide bands near the top are the leaf functions eating the CPU. Use the canonical FlameGraph scripts to render:

perf script -i /tmp/perf.data > /tmp/out.perf
FOLD=~/FlameGraph/stackcollapse-perf.pl
FOLD /tmp/out.perf > /tmp/out.folded
~/FlameGraph/flamegraph.pl /tmp/out.folded > /tmp/flame.svg
# open /tmp/flame.svg in a browser

Read it top-down: the widest top-level band is the code running the most. Reading bottom-up: pick a calling chain you control and find where its width comes from.

Profile at the right grain

Sampling rate and run length change what you see. A short run at low frequency misses rare spikes. A long run averages bursts flat. For a steady CPU problem, sample at 99 Hz for at least thirty seconds. For an intermittent one, trigger the slow path specifically and sample only around it.

Before chasing a function, sanity-check it: is the wide band your code, a framework, or the kernel? A wall of time in memcpy, page fault handling, or a driver syscall is a different problem from a loop in your own logic and a different fix.

When sampling is not enough

Sampling shows where, not why, a function spends time. A function that is wide because it waits on cache misses or locks needs counters, not just stacks. Use the perf stat layer to see hardware events driving the cost:

perf stat -p $(pgrep -f myapp) sleep 10
# look at cache-misses, branch-misses, context switches
perf stat -e cache-misses,cycles,instructions -p $(pgrep -f myapp) sleep 10

A high miss ratio or heavy context switching explains the width and points to the true mechanism. Sampling frames the problem; counters explain it.

Off-CPU time needs a different lens

Sampling a running process only sees the CPU on. If the slowness is time spent waiting, the on-CPU flamegraph will look suspiciously clean while requests crawl. Off-CPU analysis uses tracepoints to capture when a task blocks and why, and it is the tool for lock contention, I/O waits, and socket stalls.

# trace context switches with the block stack and wait reason
perf record -e sched:sched_switch -a sleep 20
perf script --show-mmap-events 2>/dev/null | head -n 30

A task that repeatedly switches out while waiting on a futex or an I/O completion tells you the hot path is budgeting, not computing. Combine the on-CPU profile (where compute goes) with an off-CPU trace (where time waits) and you get the complete picture, the same way profiling and counters each held half the story on their own.

Interpret the width, then question it

Wide bands are leads, not verdicts. Before rewriting a wide function, confirm it is genuinely your code and not a systemic cost: check the backtrace for a lock held by a sibling thread, confirm the call count rather than a single gigantic sample, and reproduce with the length of a realistic request. A flamegraph that misleads usually does so by presenting a wide memcpy or a hot lock as an innocent-looking frame, so always read the caller chain and the counters together.

Takeaway

Stack sampling turns a vague slow server into a specific hot function. Record, fold, render, and read the width; then explain the width with counters before you rewrite anything. You can install perf and practice on a disposable Ubuntu instance from Netbay at netbayhosts.in, up in under a minute.