Finding capacity automatically#

surge discover answers "how much can this system take?" instead of "what happened at the load I picked?".

You give it an SLO. It raises the arrival rate while the run is in flight, stops when the SLO breaks, bisects to find the edge, then holds the answer to check it is stable rather than a lucky window.

go run ./examples/target &                       # something to shoot at
surge discover examples/capacity.yaml

  offered and achieved are iterations (complete scenario passes) per second;
  requests is the resulting request rate.

  phase         offered     achieved   requests       p95       p99   errors  verdict
  climb          800/s        800/s     3192/s      1.6ms      2.3ms   0.00%  ok
  climb         1600/s       1600/s     6383/s      1.8ms      2.6ms   0.00%  ok
  climb         3200/s       3200/s    12768/s      2.8ms      3.7ms   0.00%  ok
  climb         6400/s       6398/s    25544/s      3.4ms      4.7ms   0.00%  breached  p95 < 3ms (actual 3.4ms)
  bisect        4800/s       4799/s    19157/s      2.6ms      3.6ms   0.00%  ok
  bisect        5600/s       5600/s    22354/s      3.4ms      4.8ms   0.00%  breached
  bisect        5200/s       5200/s    20760/s      3.4ms      4.7ms   0.00%  breached
  verify        4800/s       4800/s    19162/s      3.0ms      4.2ms   0.00%  breached  p95 < 3ms (actual 3.0ms)

  CAPACITY  4320 iterations/s   confidence: medium
  = 17249 req/s, at 4.0 requests per iteration
  SLO first broke at 5200 req/s

  The system met the SLO at 4800 req/s during the search but did not hold it
  through the longer verification, so treat 4320 req/s as the safe number.

How it works#

Climb. Start at start_rate and double each level. Each level is held for step_duration, and the first settle of it is thrown away — the seconds just after a rate change measure the change, not the system at that rate.

Bisect. Once a level breaks the SLO, the answer is between the last healthy level and this one. Split that interval refine times. Bisection stops early once the remaining gap is under 2%, because splitting further measures noise.

Confirm. Hold the answer for confirm_duration — longer than a search step. A level that passes for 20 seconds and fails at 60 was never really capacity; when that happens surge reports a number 10% lower and drops the confidence to medium.

Each level is judged by the same threshold engine the final report uses, so "healthy" means exactly what you wrote in the SLO.

The scenario#

name: capacity-check
base_url: http://localhost:8080

load:
  model: discover
  slo:                      # what "still healthy" means
    - p95 < 500ms
    - error_rate < 1%
  start_rate: 100           # arrivals/second to begin at
  max_rate: 20000           # safety cap, so a search cannot run away
  step_duration: 30s        # how long each level is held
  settle: 8s                # thrown away at the start of each level
  refine: 4                 # bisection rounds after the first breach
  confirm_duration: 2m       # final hold to confirm the answer
  max_vus: 3000             # concurrency cap on the generator side

flow:
  - name: browse
    get: /api/items
    check: { status: 200 }

Or without a file:

surge discover api.yaml \
  -slo "p95 < 500ms, error_rate < 1%" \
  -start-rate 200 -max-rate 50000 \
  -step 30s -confirm 2m -refine 5 \
  -json capacity.json

If the scenario already has thresholds: and no slo:, those are used — they already say what healthy means for this system.

Which number is the answer#

The search drives iterations — complete passes through the flow — because that is what an arrival rate schedules. A flow with four steps at 1,000 iterations per second is applying 4,000 requests per second, and the two numbers are not interchangeable:

  CAPACITY  4320 iterations/s   confidence: medium
  = 17249 req/s, at 4.0 requests per iteration

Tell your team the figure that matches the question they asked. "We handle 4,320 checkouts a second" and "we handle 17,249 requests a second" are both true here, and reporting one as the other is off by the number of steps in the flow.

Reading the verdict#

The confidence field is the part to read first.

That last case matters. If surge cannot offer the rate it is trying to test — every worker busy, max_vus reached — then the level says nothing about the target, and reporting it as capacity would be a lie about the system under test. surge detects it, stops, and tells you to add agents or raise max_vus:

  CAPACITY  6400 req/s   confidence: low

  The load generator ran out of capacity before the target did. The target
  handled at least 6400 req/s; its real ceiling is higher and this run cannot
  say where. Add agents with -agents, or raise max_vus, and run it again.

Searching from several machines#

A capacity search is exactly when one generator runs out first. Run the search against a fleet:

surge agent -listen 0.0.0.0:7777 -token "$SURGE_TOKEN"   # on each generator
surge discover api.yaml -agents gen1:7777,gen2:7777,gen3:7777

Picking the settings#

• step_duration — long enough for the system to show its steady state. 30s is a reasonable default; systems with autoscaling or cold caches need minutes, and a search with 20-second steps will find the wrong number on one.
• confirm_duration — at least twice a step. This is what separates "it survived a burst" from "it can run at this rate".
• start_rate — low enough to pass. If the SLO fails at the first level there is nothing to search.
• max_rate — a real cap. It is a safety belt against pointing a capacity search at production.
• refine — 4 rounds narrow a doubling to about 6%. More rounds cost a step each and quickly measure noise instead of capacity.

Honest limits#

• The answer describes this configuration: this data set, this cache state, this instance count, this dependency behaviour. It is not a property of the code.
• A system with autoscaling will show a higher ceiling than it can hold from cold, because the search gives it time to scale. Use a long confirm_duration, and read the per-second timeline.
• Capacity found against a staging environment predicts production only as well as staging resembles production.
• The search applies real load, in increasing amounts, until something breaks. Point it at production only when you mean to.