surgesurge

Running across several machines

One surge process pushed 126k req/s in testing, so the usual reason to distribute — using more than one CPU core — does not apply. Distribute when you hit one of the real limits: the generator's network card, its open-file or port limits, or a need to apply load from several regions.

text
                    surge run api.yaml -agents gen1:7777,gen2:7777,gen3:7777
                                        │
                     ┌──────────────────┼──────────────────┐
                     ▼                  ▼                  ▼
              surge agent        surge agent        surge agent
              (1/3 of load)      (1/3 of load)      (1/3 of load)
                     └──────────────────┼──────────────────┘
                                        ▼
                       histograms merged · thresholds evaluated once

The coordinator holds no load of its own. Each agent runs its share, and returns raw histogram buckets rather than computed percentiles — which is why the merged p99 is the true p99 across the fleet, not an average of three p99s. Averaging percentiles is wrong, and it is how most home-grown fan-out scripts get this silently wrong.

Running it

On each generator:

bash
surge agent -listen 0.0.0.0:7777 -token "$SURGE_TOKEN"
surge agent -listen 0.0.0.0:7777 -node eu-west-1 -token "$SURGE_TOKEN"   # named, authenticated

From anywhere that can reach them:

bash
surge run api.yaml -agents gen1:7777,gen2:7777,gen3:7777 -html report.html
text
surge 0.1.0 · checkout-journey · closed model · 60 VUs over 12s · total 12s
cluster: 3 agents, load split evenly
  [0] http://gen1:7777            worker-1 · 12 cpus · up 6s
  [1] http://gen2:7777            worker-2 · 12 cpus · up 6s
  [2] http://gen3:7777            worker-3 · 12 cpus · up 6s

per node
  http://gen1:7777   worker-1   140 requests, 0 errors · 20 VUs over 12s
  http://gen2:7777   worker-2   128 requests, 0 errors · 20 VUs over 12s
  http://gen3:7777   worker-3   140 requests, 0 errors · 20 VUs over 12s

Everything that works locally works distributed: -vus, -rate, -duration, -base-url, -var, thresholds, every report format, and surge discover.

What is shipped to an agent

The scenario YAML, the CLI overrides, and the feeder CSV files. Agents need no shared disk and no copy of your repository — a scenario with feeders: { users: { path: users.csv } } works unchanged.

Only the base name of each file is used, so a scenario cannot make an agent write outside its own temporary directory.

How the load is split

Virtual users and iterations are whole things, so they are split with floor((i+1)·T/n) − floor(i·T/n): the parts always add back up to the total, and a remainder lands on the first nodes instead of disappearing into rounding. 10 VUs across 3 agents is 4+3+3, never 3+3+3.

Arrival rates are continuous and simply divide: 900/s across 3 agents is 300/s each.

Feeders are per agent, not global. With mode: circular and three agents, each agent cycles the whole file independently, so row 1 is used three times per cycle rather than once. When every virtual user must have a distinct row, split the CSV per agent, or use mode: random with a file large enough that collisions do not matter.

Failure is not hidden

A distributed run that quietly drops a node reports load that never happened, and a green report from a test that applied two thirds of the intended traffic is worse than no report.

Security

An agent runs whatever scenario it is sent, and a scenario can carry exec steps. So an agent is not only a machine that will send arbitrary HTTP requests anywhere on command; without the two refusals below it would be a shell on that machine for anyone who found the port.

bash
export SURGE_TOKEN=$(openssl rand -hex 24)
surge agent -listen 0.0.0.0:7777 -token "$SURGE_TOKEN"      # on each generator
surge run api.yaml -agents gen1:7777 -token "$SURGE_TOKEN"

It refuses by default

Both are off-switches, not locks: with a token and -allow-exec, an agent does exactly what it is told. Keep agents on a private network, give them a token, and do not put one on the public internet.

The token

SURGE_TOKEN is read from the environment by both the agent and the coordinator, so it never has to appear on a command line or in a scenario file. It is compared on every API call as X-Surge-Token. It is a shared secret over plain HTTP: put the agents on a network you trust, or terminate TLS in front of them.

Docker

The repository ships a Docker setup that runs a whole cluster locally:

bash
cd deploy/docker
docker compose up -d --scale agent=4 target agent
docker compose run --rm coordinator

The coordinator resolves every agent replica through compose DNS, so scaling the fleet needs no configuration change. It exits with the run's exit code, so CI can gate on it, and reports land in ./reports on the host.

Point it at your own system instead of the demo target:

bash
BASE_URL=https://staging.internal SCENARIO=perf/checkout.yaml \
VUS=500 DURATION=10m docker compose run --rm coordinator

Kubernetes

The same image works as a Deployment of agents plus a Job for the coordinator. Give the agents a headless Service so the Job can resolve every pod:

yaml
apiVersion: v1
kind: Service
metadata:
  name: surge-agents
spec:
  clusterIP: None          # headless: DNS returns every pod address
  selector: { app: surge-agent }
  ports: [{ port: 7777 }]

Then build the agent list from DNS in the Job, exactly as deploy/docker/run-cluster.sh does.

Limits today