Running surge in CI#

A performance test in CI answers one of two questions, and they want different setups: is it fast enough (a fixed threshold) or did we make it slower (a regression gate). Start with the first; the second needs baselines, which take a few runs to collect.

The shape#

perf:
  script:
    - surge validate perf/checkout.yaml
    - surge preflight perf/checkout.yaml
    - surge run perf/checkout.yaml
        -json current.json -html report.html -failures failures.csv
  artifacts:
    when: always
    paths: [report.html, current.json, failures.csv]

Three things earn their place:

• preflight before the run. The commonest red pipeline is an environment that was not ready, and preflight says so in one line and one request instead of after the full duration.
• when: always. The report is what explains the failure. A pipeline that keeps it only on green has kept the one artefact nobody needs.
• Exit codes, not parsing. 0 pass · 1 a breached threshold or a real regression · 2 a setup error before any traffic.

GitHub Actions#

- uses: actions/setup-go@v5
  with: { go-version: "1.23" }
- run: go install gitlab.com/agentic-fx/surge/cmd/surge@v0.9.0
- run: surge preflight perf/checkout.yaml
- run: surge run perf/checkout.yaml -json current.json -html report.html
  env:
    SURGE_BASE_URL: ${{ secrets.STAGING_URL }}
- uses: actions/upload-artifact@v4
  if: always()
  with:
    name: perf-report
    path: |
      report.html
      current.json

Keep the environment out of the file#

The scenario is committed; which environment this run points at is not. Use SURGE_BASE_URL (see configuration), so the same file runs against staging from the pipeline and against a local docker compose from a laptop with no edit.

Pin the version#

@latest is right while you are trying surge out and wrong the moment a gate depends on it: a tool that changes under a regression gate makes every comparison suspect. Pin it, and bump it deliberately with a baseline reset if the numbers move.

Size it for the runner, not for production#

A shared CI runner has neighbours. surge watches its own CPU and marks a run untrustworthy above 85%, which is a warning you will see a lot if the job asks for more load than the runner can make. Two ways out:

• keep the CI run small and gate on latency percentiles, which are stable, and error rate, which should be zero;
• run the big one on a dedicated box, on a schedule, and gate CI on the small one.

Do not gate on throughput from a shared runner. It is too noisy to mean anything. And do not gate on p99 from a thirty-second run: a handful of samples in the p99 bucket is one slow request away from anything.

A nightly baseline, a gate on the branch#

perf-gate:                       # every merge request
  script:
    - surge run perf/checkout.yaml -json current.json -html report.html
    - surge compare current.json -baseline checkout
  artifacts: { when: always, paths: [report.html, current.json] }

perf-baseline:                   # main only, after the gate is green
  script:
    - surge run perf/checkout.yaml -json summary.json
    - surge baseline save summary.json -name checkout -commit "$CI_COMMIT_SHA"
    - surge baseline prune -name checkout -keep 20

The store lives in .surge/baselines/<name>/. Commit it, or keep it in a cache that outlives the band. Catching regressions is the whole method.

Seeding and cleaning#

Put the commands in the scenario rather than in the pipeline:

setup:    [ ./scripts/seed-accounts.sh 150 ]
teardown: [ ./scripts/clean.sh ]

Teardown runs whatever happened — a failed run, a breached threshold, a cancelled job — because the runs that leave a mess are the runs that went wrong. A pipeline step after the run does not execute when the run exits 1.

Artefacts worth keeping#