Testing any protocol, and running it anywhere#

Two questions, in order: how do I write the test for the thing I have — an API, a WebSocket, gRPC, a database — and how do I run it on my laptop, across a fleet, or in Kubernetes.

• First decision: YAML or Go
• Writing the test
  • HTTP and REST APIs
  • WebSocket
  • gRPC
  • Databases
  • SAP, mainframe
  • Anything else
• Running it
  • Locally
  • Across several machines
  • Kubernetes
  • Docker Compose
  • CI
• What runs where

First decision: YAML or Go#

If your system speaks HTTP, start with YAML; you can move to Go later without changing how anything is run. If it speaks anything else, write Go. Both use the same engine, so the statistics, thresholds, reports, capacity search and regression gate are identical.

Writing the test#

HTTP and REST APIs#

name: checkout
base_url: https://api.example.com
load:
  stages:
    - { duration: 30s, target: 200 }
    - { duration: 5m,  target: 200 }
thresholds: [ "p95 < 400ms", "error_rate < 1%" ]
flow:
  - name: login
    post: /auth/login
    json: { username: "${users.username}", password: "${users.password}" }
    extract:
      token: { json: data.token }
  - name: checkout
    post: /api/checkout
    headers: { Authorization: "Bearer ${token}" }
    check:
      status: [200, 201]
      json: { status: confirmed }

Every field: scenario-reference.md. Already have the flow in a browser? surge import har session.har -o checkout.yaml.

WebSocket#

Go. One connection per virtual user, opened in OnStart so the handshake is not counted in every message:

u.OnStart(func(c *surge.Ctx) error {
	ws, err := c.Dial("/stream", surge.Name("connect"))
	if err != nil {
		return err
	}
	c.Set("ws", ws)
	return nil
})
u.OnStop(func(c *surge.Ctx) {
	if ws, ok := c.Value("ws").(*surge.WS); ok {
		ws.Close()
	}
})
u.Task("quote", 1, func(c *surge.Ctx) error {
	ws := c.Value("ws").(*surge.WS)
	reply, err := ws.Request(`{"op":"quote","sym":"ACME"}`, surge.Name("quote"))
	c.Check("got a price", strings.Contains(reply, "price"))
	return err
})

Request measures the round trip. Send, Receive and Listen are there for conversations that are not request/response — a feed you subscribe to and then read, say.

There is no Ping: a pong is a control frame, processed only while a read is in flight, so pinging an idle connection waits for a reply nobody is listening for. Measure something the server actually answers.

gRPC#

A separate module, so an HTTP-only test does not pull in the gRPC dependencies:

go get gitlab.com/agentic-fx/surge/grpcx

u.OnStart(func(c *surge.Ctx) error {
	conn, err := grpcx.Dial(c, "api.internal:50051", grpcx.Insecure())
	if err != nil {
		return err
	}
	c.Set("conn", conn)
	c.Set("client", pb.NewSearchClient(conn.ClientConn()))
	return nil
})
u.OnStop(func(c *surge.Ctx) { c.Value("conn").(*grpcx.Conn).Close() })

u.Task("search", 1, func(c *surge.Ctx) error {
	client := c.Value("client").(pb.SearchClient)
	_, err := grpcx.Call(c, "Search", func(ctx context.Context) (*pb.Result, error) {
		return client.Search(ctx, &pb.Query{Term: "widget"})
	}, grpcx.Bearer(c.Str("token")))
	return err
})

Calls are recorded with their gRPC status code, so the report says DeadlineExceeded ×412 rather than "error". grpcx.Stream measures a stream message by message. grpcx.Expect(codes.NotFound) keeps an expected status out of the error rate.

Databases#

u.OnStart(func(c *surge.Ctx) error {
	db, err := dbx.Open(c, "pgx", "postgres://load:pw@db/shop")  // your own driver
	if err != nil {
		return err
	}
	c.Set("db", db)
	return nil
})
u.Task("orders", 1, func(c *surge.Ctx) error {
	db := c.Value("db").(*dbx.DB)
	return dbx.Query(c, "orders by customer", db,
		"SELECT id FROM orders WHERE customer_id = $1", c.Str("customer"))
})

Errors are grouped by cause, so you get deadlock ×412 instead of four hundred strings carrying different transaction ids. dbx.Watch reports the pool's counters — including how long callers spent waiting for a connection, which is the answer more often than the query is.

SAP, mainframe#

sapx handles OData, Fiori and S/4HANA, including the CSRF handshake and $batch changesets. tn3270 drives a mainframe: telnet negotiation, EBCDIC, a screen buffer, AID keys. Both are ordinary Go packages used the same way as above. Some desktop-client and remote-session protocols are closed and binary, with no public specification. surge does not pretend to speak those; an exec: step times the tool that does.

Anything else: TCP, MQTT, Kafka, a C library#

Whatever you can call from Go, you can measure. Put it inside c.Time and it joins the same table, thresholds and reports as an HTTP request:

u.Task("publish", 1, func(c *surge.Ctx) error {
	return c.Time("mqtt.publish", func() error {
		return client.Publish("topic/x", 1, false, payload).Error()
	})
})

Runnable examples: examples/custom-protocol (TCP and a fake database) and examples/distributed-go (TCP across a fleet).

The page, not the protocol#

A protocol test answers "how long did the server take". It cannot answer "how long until the user saw something", because the rendering, the JavaScript and the requests a page discovers for itself all happen in a browser that a protocol test does not run. That gap is what Core Web Vitals measure, and what a product owner means by "slow".

import "gitlab.com/agentic-fx/surge/browser"

u.OnStart(func(c *surge.Ctx) error {         // Chrome costs ~1s to start:
    b, err := browser.Launch(c)              // not inside every iteration
    c.Set("browser", b)
    return err
})
u.OnStop(func(c *surge.Ctx) { c.Value("browser").(*browser.Browser).Close() })

u.Task("checkout", 1, func(c *surge.Ctx) error {
    b := c.Value("browser").(*browser.Browser)
    v, err := b.Goto(c, "https://shop.example.com/checkout")
    c.Check("no JavaScript errors", v.JSErrors == 0)
    return err
})

Measured against a page that paints late, shifts after painting and throws:

TTFB 44ms · FCP 72ms · LCP 72ms · CLS 0.214 · load 176ms · 3 requests · 1 JS error

The navigation is recorded as a request named after the page, so it appears in the summary table and p95 < 3s gates it like anything else. The vitals go in as trends — lcp_ms, cls, fcp_ms, ttfb_ms, dom_content_loaded_ms, page_requests — so they reach the JSON, the HTML report and surge compare across releases. Google's own thresholds are recorded as checks (LCP under 2.5s, CLS under 0.1).

This drives real Chrome over the Chrome DevTools Protocol, a published protocol over WebSocket. There is no vendor library to install and nothing beyond a Chrome already on the machine; browser.ExecPath names a different one and browser.Remote attaches to one that is already running, which in a container is far cheaper than one browser per virtual user.

b.WaitFor(c, "window.__app.ready", 10*time.Second) is the single-page-app equivalent of a load event, which such pages do not meaningfully have, and b.Eval reads anything else out of the page.

Browsers are expensive — a few hundred megabytes and a CPU core each. The arrangement that works, and the one commercial tools use, is a handful of browser users for the experience alongside thousands of protocol users for the load. Both land in the same report, and one threshold set covers them.

Runnable: examples/browser.

Protocols surge cannot speak#

Some protocols are closed. SAP GUI speaks DIAG, Citrix speaks ICA/HDX, Oracle Forms has its own: binary formats with no public specification, reverse-engineered over decades, and that work is exactly what a commercial licence pays for. surge does not implement them and will not pretend to.

What it can do is let the tool that does speak them run inside a surge scenario, so the timing, the failures and the thresholds sit in one report with everything else:

flow:
  - name: sap gui login
    exec: ["/opt/vendor/run-script", "--user", "${users.username}"]
    exec_timeout: 60s
    check:
      status: [0]                 # the exit code
      body_contains: "transaction="   # the command's output
  - name: api follow-up
    get: /api/orders

The command is timed and classified like a request:

request                           count   err%      rps      avg      p95
sap gui login                        40  20.00     36.7    107.7    487.4
api follow-up                        32   0.00     29.3      1.2      5.4

status  200 ×32 · 3 ×8
errors  exit 3: DIAG session refused ×8
checks 64/80 passed (80.00%)

So p95(sap gui login) < 2s gates the build, the failures are named rather than counted anonymously, and the vendor tool's result sits beside the API calls it triggered. The same step runs a browser-automation script, another load tool, a CLI, or anything else with an exit code. Its stdout is available to later steps as ${step name.stdout}, and the exit code as ${step name.exit}.

Two things it does, because a load generator runs a command thousands of times rather than once: the whole process group is killed on timeout — a shell dying does not kill the tool it started, and the orphans would accumulate — and output is capped, so a chatty tool cannot exhaust memory one iteration at a time.

A scenario that can exec is as powerful as a shell script. Read one before running it, the way you would read a script.

Running it#

A Go scenario ends with s.RunMain(), which gives the compiled binary the same flags as the CLI:

go build -o loadtest ./perf
./loadtest                                   # as written
./loadtest -vus 200 -duration 5m             # override the plan
./loadtest -threshold "p95 < 500ms" -html report.html

Your own options go on s.Flags() before RunMain, which is how every node in a fleet gets told the same target:

target := s.Flags().String("target", "localhost:50051", "gRPC endpoint")
s.OnTestStart(func() { log.Printf("every node is pointed at %s", *target) })

Locally#

surge run checkout.yaml -analyze -web :8088     # YAML
./loadtest -vus 100 -duration 2m -analyze       # Go

One machine goes a long way — about 131k requests/s from a single process on a 12-core laptop — and surge watches its own CPU, marking the run untrustworthy above 85% rather than letting you publish a number that describes the load generator.

Across several machines (master/worker)#

A YAML scenario ships as text, so the stock agent can run it:

surge agent -listen 0.0.0.0:7777 -token "$SURGE_TOKEN"     # on each generator
surge run checkout.yaml -agents gen1:7777,gen2:7777,gen3:7777

A Go scenario is code — the gRPC stubs, the database driver and the WebSocket conversation live in the binary, and none of that fits in a request body. So the same program runs on every node, and the coordinator sends only the share of the load.

./loadtest -target api.internal:50051 -agent -listen :7777   # on each generator
./loadtest -target api.internal:50051 -agents gen1:7777,gen2:7777

Either way the load is split so the parts add back up, and the raw histograms are merged — the merged p99 is the true p99, not an average of three p99s, which is what hand-rolled fan-out scripts usually report.

Three things stop before any traffic is sent, because a green report from a test that half-ran is worse than no report:

• an agent that is unreachable or already busy;
• an agent on a different surge release;
• for a Go scenario, an agent built from different code. Each binary derives a fingerprint from the scenario it was compiled with, and a mismatch is refused by name:

n2:7777: built with scenario checkout@4c1d90ab77e2, the coordinator is running
checkout@8465608eed61 — different builds, so their numbers must not be merged

Set SURGE_TOKEN (or -token) and the agents will only take work from a coordinator that knows it.

Kubernetes#

There is no operator to install. A run is a Job that exits 0 or 1, which Kubernetes already models. Three manifests in deploy/kubernetes:

agents.yaml + run-job.yaml — a Deployment of agents behind a headless Service, and a Job that resolves every pod and coordinates them. For YAML scenarios.

kubectl create configmap surge-scenario --from-file=scenario.yaml=perf/checkout.yaml
kubectl apply -f agents.yaml -f run-job.yaml
kubectl wait --for=condition=complete job/surge-run --timeout=30m
kubectl logs job/surge-run

go-scenario.yaml — the same arrangement for a Go scenario, with your image in place of surge's: the agents run -agent, the coordinator runs -agents.

parallel-job.yaml — no coordinator at all. A Job with parallelism: 4 runs four independent pods, each writing a snapshot, and a second Job merges them:

surge run scenario.yaml -vus 125 -snapshot /results/part-$JOB_COMPLETION_INDEX.json
surge merge /results/part-*.json -threshold "p95 < 500ms" -html report.html

Use this when you would rather not keep agents running: the pods start, work and exit. The cost is that each starts on its own, so a precise ramp across the fleet is not possible — for that, use the coordinator.

What matters either way is what gets merged. A snapshot holds raw histogram buckets, not percentiles, because the mean of four 99th percentiles is not the 99th percentile of anything. surge merge adds the histograms; collecting four -json summaries and averaging them does not, and surge merge refuses a summary file for that reason.

Docker Compose#

cd deploy/docker
docker compose up -d --scale agent=4 target agent
docker compose run --rm coordinator

CI#

The exit code is the verdict — 0 passed, 1 a threshold failed or a real regression, 2 the run could not be made — so nothing needs parsing:

load-test:
  script:
    - go build -o loadtest ./perf
    - ./loadtest -duration 3m -json summary.json -html report.html
    - surge compare summary.json -baseline checkout
  artifacts:
    paths: [report.html, summary.json]

What runs where#

A Go scenario now does everything the CLI does. The remaining asymmetry runs the other way: surge analyze, surge merge and surge compare are commands that read the files a run writes, so a Go scenario uses them by writing -json and -snapshot and then calling them.

Finding the ceiling of something that is not HTTP#

./loadtest -discover -slo "p95 < 5ms" -start-rate 500 -max-rate 40000

  phase         offered     achieved   requests       p95       p99   errors  verdict
  climb         8000/s      7999/s     8000/s      2.9ms      3.6ms    0.00%  ok
  climb        16000/s     15997/s    15898/s      5.2ms      5.4ms    0.00%  breached  p95 < 5ms
  bisect       14500/s     14504/s    14412/s      4.9ms      5.2ms    0.00%  ok
  verify       14500/s     14498/s    14500/s      4.9ms      5.2ms    0.00%  ok

  CAPACITY  14500 req/s   confidence: high

From Go, the same search is s.SLO("p95 < 5ms") and s.Discover(ctx, surge.CapacityOptions{}).