Performance Regression Hunting with Before/After Measurements

A regression is a slowdown with no obvious cause, and the instinct is to guess. The reliable way to hunt one is to stop guessing and start comparing: capture a clean before, replicate the slow condition, and bisect the changes between them. When you know the baseline and the symptom window, most regressions fall into a small set of suspects within an hour.

Canonicalise a baseline first

Without a stored baseline, you cannot prove anything changed. Set up a repeatable measurement you can run identically on demand: a fixed request mix, a fixed concurrency, and a fixed machine state. Save the numbers so the future you can compare.

# a repeatable latency probe with a fixed mix
ab -n 10000 -c 100 http://127.0.0.1:8080/ 2>&1 |   grep -E '(Requests per second|Time per request|Failed)'
# store it
ab -n 10000 -c 100 http://127.0.0.1:8080/ > /tmp/baseline.txt 2>&1

The baseline must be reproducible: same tool, same flags, same load. A test that varies its concurrency between runs measures nothing. Link the baseline to the deployment commit so you know exactly what code produced it.

Automate the baseline into every deploy

A baseline that only exists on one server from last month rots quickly. Fold the same probe into your deploy or CI step so every release records a before and after number automatically. Then a regression has a timestamp and a diff the moment it ships, instead of a mystery gated on memory.

# a deploy hook that records a latency sample on every ship
./deploy.sh && ab -n 5000 -c 100 http://127.0.0.1:8080/   > /var/log/perf/$(date +%F-%T).txt 2>&1
# check whether a later sample dropped below an earlier one
grep 'Requests per second' /var/log/perf/*.txt | sort

Keep the samples sorted by time and flag the first one whose throughput falls outside the established error band. With a noisy shared host, compare against the median of the last several samples rather than a single stale number, so a blip from a busy neighbour does not masquerade as a code regression.

The disciplines to record, not just run

Run every probe identically, store the raw output with a timestamp and commit hash, and only blame a change you can link back to a moving metric. A regression hunt without a stored before is not a hunt, it is archaeology. The discipline of saving each measurement is what turns a messy slowdown into a bisectable event.

Confirm before you bisect

Before blaming code, rule out the environment. A slowdown can be a quieter, busier neighbour on the shared host, a full disk, or a background job. Run the same probe on a fresh instance with nothing else installed; if it is fast there, the regression is in your stack or config, not the platform.

# is CPU steal or I/O sneaking in?
top -b -n 1 | grep Cpu
iostat -x 1 3 | tail -n 5
# then profile the suspect process
perf record -p $(pgrep -f app) -F 99 --call-graph dwarf sleep 20

If the new instance reproduces the old behaviour, the environment is not the cause and you bisect code. If it is fast, look at what changed around the memory, disk, or network on the original.

Bisect the changes, not your memory

Once you have a baseline and a confirmed symptom window, the changes between them are a finite list: the commits, config edits, new dependencies, and cron additions shipped in that window. Bisect by reverting the most recent change first and re-running the probe. The regression is almost always the change that moved the metric the moment it shipped.

• Check recent deploys and the diff each one introduced.
• Check dependency and kernel updates that happened in the window.
• Check config, cron, and log-rotation changes that can double-spend CPU or disk.

Each check re-runs the identical probe and records the delta. When one change moves the metric back to baseline, you have the culprit.

The profile that confirms the blame

When you have a suspect commit, explain it with a profile. If the change added an N+1 query, the profile shows time in the database driver. If it changed logging, time appears in the log path. Profile the suspect code path and look for the new cost; a regression without a mechanism will come back.

Takeaway

Capture a reproducible baseline, confirm the host is not the cause, bisect the changes, and blame the first one that returns you to baseline. The discipline is the tool: identical probes, recorded results. To hold a clean baseline environment, deploy an Ubuntu instance on Netbay in under 60 seconds from netbayhosts.in and run your probes there.