Catching regressions without crying wolf#

A performance gate only works if people trust it. The usual way to build one — compare this run to the last run, fail if p95 moved more than 10% — produces false alarms on quiet weeks and misses real slowdowns on busy ones, and within a month the team is clicking through it.

surge stores several reference runs and asks a different question: is this value outside the range this system normally produces?

surge run api.yaml -json summary.json          # after a good run on main
surge baseline save summary.json -name checkout

# later, on a branch
surge run api.yaml -json pr.json
surge compare pr.json -baseline checkout       # exit 1 on a real regression

baseline comparison  checkout · median ±3 MAD over 7 runs

  metric                    current   baseline    change   spread  verdict
  p50                        112.00     104.00     +7.7%    ±3.50  ok
  p95                        284.00     204.00    +39.2%   ±8.20  REGRESSED
  p99                        410.00     389.00     +5.4%   ±22.10  ok
  error_rate                   0.12       0.10    +20.0%    ±0.04  ok
  rps                        498.00     501.00     -0.6%   ±9.40   ok
  p95 (checkout)             590.00     402.00    +46.8%   ±14.00  REGRESSED

  ✗ p95
    39.2% worse than the usual 204.00, which is 9.8 deviations out — larger
    than this system's normal run-to-run spread.

  2 metric(s) moved further than this system's normal run-to-run spread,
  measured over 7 reference runs.

  REGRESSION

Why median and MAD#

The reference point is the median of the stored runs, and the spread is the median absolute deviation, not the mean and standard deviation.

One bad run — a noisy neighbour, a cold cache, a flaky dependency — drags a mean far enough to hide the next real regression, and inflates a standard deviation enough to swallow it. The median ignores it. That is the whole reason: the statistic has to survive the kind of data load tests actually produce.

A metric is called a regression when both are true:

1. it is more than -deviations (default 3) MADs from the baseline median, and
2. it moved more than -min-change (default 5%).

The second condition exists because a very steady baseline makes any movement statistically "significant". A 2% change nobody can feel is not worth failing a build over; surge marks it noise and moves on.

Too few baselines is said out loud#

Below -min-runs (default 3) reference runs there is no way to know what normal variation looks like. surge does not pretend otherwise: it falls back to a plain percentage comparison, marks the result unreliable, prints a warning, and does not fail the build.

baseline comparison  checkout · plain 20% change (only 1 baseline run(s):
                     too few to measure normal variation)

  ! Too few reference runs to tell a regression from normal noise — treat
    this as a hint, not a gate.

A gate that fails on a verdict it cannot stand behind is a gate people turn off.

What is compared#

• Overall p50, p95, p99, error_rate and rps
• p95 for every named request, so a slowdown in one step is caught even when the overall figures look fine

Higher is worse for latency and error rate; lower is worse for throughput, and surge accounts for the direction — a run that gets 40% slower and a run that loses 40% of its throughput are both regressions.

Improvements are reported separately and never fail a build.

In CI#

performance:
  script:
    - surge run perf/checkout.yaml -base-url "$STAGING_URL" -json summary.json
    - surge compare summary.json -baseline checkout      # exits 1 on regression
  artifacts:
    when: always
    paths: [summary.json]

# on the default branch only, after the run passes
baseline:
  rules: [{ if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH }]
  script:
    - surge run perf/checkout.yaml -base-url "$STAGING_URL" -json summary.json
    - surge baseline save summary.json -name checkout -commit "$CI_COMMIT_SHA"
    - surge baseline prune -name checkout -keep 20

Baselines live in .surge/baselines/<name>/ as JSON, one file per run. Commit them, or keep them in a CI cache or an artifact bucket — they are small, and plain enough to diff.

Keeping baselines honest#

A baseline is only comparable to a run made the same way. Record the context with it:

surge baseline save summary.json -name checkout \
  -commit "$CI_COMMIT_SHA" -env staging -label "4 replicas, seeded dataset"

and keep these the same between the baseline and the run you compare:

• the same environment, instance count and instance size
• the same data set and roughly the same data volume
• the same load profile — comparing a 50-VU run against a 200-VU baseline measures the difference in load, not a regression
• the same generator setup; a distributed run and a single-machine run can produce different numbers for reasons that have nothing to do with the code

surge baseline prune -name checkout -keep 20 drops old runs. Prune when the environment changes materially: a baseline from before a database upgrade is measuring a system that no longer exists.

Tuning#

If the gate is too noisy, raise -deviations to 4 and -min-change to 10 before you consider turning it off — and check that the environment is really stable, because a noisy gate usually means a noisy environment.