Production Continuous Profiling with eBPF and Pyroscope on Linux

Metrics inform you that CPU utilization is at 100%. Distributed tracing points you to a specific HTTP endpoint. Logs show that a transaction timed out. However, none of these pillars reveal which exact line of code, memory allocation, or kernel lock contention consumed the CPU cycles.

Continuous profiling is the fourth pillar of observability. By leveraging Linux extended Berkeley Packet Filters (eBPF), profiling agents sample CPU instruction pointers at the kernel boundary (e.g., 99Hz) across every running process. This technique requires zero code changes, operates with less than 1% CPU overhead on production Intel Xeon instances, and generates interactive flamegraphs visualizing the exact call trees burning CPU time.

How eBPF Continuous Profiling Works

Traditional profilers required compiling debug wrappers, launching runtime flags (-prof), or running intrusive JVM/Python agents that could crash production workers. In contrast:

1. **Kernel Timer Probes**: An eBPF program attaches to perf events at 99 samples per second per CPU core.
2. **Stack Unwinding**: The probe traverses user-space and kernel-space call frames. For modern languages (Go, Rust, C++), it parses DWARF unwinding tables or frame pointers (-fno-omit-frame-pointer).
3. **Symbolization**: An agent resolves raw memory addresses into function names and source file line numbers using ELF binary symbol tables.
4. **Flamegraph Aggregation**: Grafana Pyroscope merges samples into interactive flamegraphs.

Installing the Pyroscope Profiling Agent

Install the Pyroscope eBPF agent on Ubuntu 24.04:

# Install kernel headers required for eBPF JIT compilation
sudo apt-get update
sudo apt-get install -y linux-headers-$(uname -r) curl

# Download and install Pyroscope agent
curl -fsSL https://github.com/grafana/pyroscope/releases/download/v1.6.0/pyroscope_1.6.0_linux_amd64.deb -o pyroscope.deb
sudo dpkg -i pyroscope.deb
rm pyroscope.deb

Configure /etc/pyroscope/agent.yaml to profile all local system processes and Docker containers:

# /etc/pyroscope/agent.yaml
target: "http://10.0.0.8:4040" # Pyroscope server endpoint

profiles:
  - name: "cpu"
    interval: "15s"

ebpf:
  enabled: true
  sample_rate: 99 # 99Hz avoids lockstep frequency resonance
  demangle: "rust"
  collect_kernel_profile: true

targets:
  - name: "system-wide"
    process_matching:
      include:
        - ".*"

The sample_rate: 99 setting is deliberate: sampling at 99Hz rather than 100Hz prevents harmonic lockstep synchronization with periodic operating system timer interrupts.

sudo systemctl daemon-reload
sudo systemctl enable --now pyroscope-ebpf
sudo systemctl status pyroscope-ebpf

Expand

Reading and Interpreting Flamegraphs

In Grafana Pyroscope, profiling data is visualized as a flamegraph:

• **X-Axis**: Represents the total percentage of CPU time spent in that function and its children (wider boxes consume more CPU). It does not represent chronological time.
• **Y-Axis**: Represents stack depth from top (leaf functions executing CPU instructions) to bottom (entrypoints like main).
• **Search & Filter**: Filter by package or service name to immediately spot functions consuming unexpected runtime.

# Inspect live profiling metrics emitted by the agent
curl -s http://localhost:7071/metrics | grep pyroscope_ebpf_samples_total

Differential Flamegraph Comparisons

When deploying a new software release, Pyroscope enables side-by-side differential flamegraph comparisons. Any function that introduced a performance regression highlights in bright red, pinpointing algorithmic inefficiencies before they affect users.

Deploy production-grade eBPF profiling infrastructure on Netbay Linux VPS in under 60 seconds with High-Speed SSD in Lucknow DC01 — netbayhosts.in.