Repository navigation
benchmarking/sweperf: optionally retry ResumeActor when no worker has room - #3540
Nishanth Kotla (Nishanth29) wants to merge 1 commit into
Conversation
| resumeLatency := res.latency | ||
| u.awake = false | ||
| if !noopResume { | ||
| u.resumeMax = max(u.resumeMax, resumeLatency) |
There was a problem hiding this comment.
resumeMax still takes res.latency, which is only the last (successful) attempt. With a budget set, a cycle that waited 30s for room logs a resume_max_ms of a few hundred ms in the task record, so a goodput wake gate that reads that field never sees the wait, and the 24% -> 99% shows up as good tasks. The task record's retries also leaves out these in-resume retries. Let's take res.total for resumeMax (or add a separate resume_wait_max_ms) and count the capacity retries in the task record.
There was a problem hiding this comment.
Done. When a resume was retried, resume_max_ms is now the client wall time including the waits, so the goodput wake gate (#3550) sees them. The task record also has capacity_retries. With the budget at 0 it's unchanged.
| * `--sweperf-resume-retry-budget`: seconds a cycle keeps retrying a `ResumeActor` the fleet | ||
| had no room for (`ResourceExhausted` or `Unavailable`) with 50ms-to-1s backoff before giving | ||
| the cycle up (default `0`, off, giving the resume one attempt; set to `60` to match | ||
| `--agentsession-resume-retry-budget`). |
There was a problem hiding this comment.
Let's say when to use this. With a budget, refusals become wake latency instead of task failures, so a run near capacity looks healthy unless ResumeWithRetry is read as the wake time. "set to 60 to match agentsession" reads like a recommended sweperf setting; for goodput or host-scale sweeps, 0 (or a budget about the size of the wake SLO) keeps overload visible.
There was a problem hiding this comment.
Done. The docs now say to keep it at 0 for goodput and host-scale sweeps, size it like the wake SLO otherwise, and read ResumeWithRetry as the wake time.
| resumeRetryInitialBackoff = 50 * time.Millisecond | ||
| resumeRetryMaxBackoff = time.Second | ||
| resumeRetryJitterFraction = 0.1 |
There was a problem hiding this comment.
Once the backoff reaches the 1s cap (after ~1.5s), each waiting user sends about one ResumeActor per second until the budget runs out. That is up to ~60 attempts per resume with a 60s budget, with no cap on the attempt count and no global retry budget. By Little's law, the 1000-user run (~7.7 resumes/s, 12s p50 wait) has on the order of 100 users waiting at any moment, so roughly 100 refused ResumeActor/s against 7.7 successful ones. Some of the wait we're measuring may be control-plane load that the retries add themselves. Let's put the measured attempt rate (ResumeActor failures/s) in the description, and consider a higher max backoff (a few seconds) or honoring a server retry-after hint.
There was a problem hiding this comment.
Raised the max backoff to 4s; agentsession stays at 1s. Retried refusals are now their own row, ResumeActorRefused. At 1000 users on one metal node, refused attempts went from 72.13/s to 39.29/s, with tasks OK 98.6% vs 98.7% and OK resumes/s 7.68 vs 7.71. ResumeWithRetry p50/p95 went from 12.19/33.62 s to 13.02/37.64 s. Table in the description. ateapi doesn't send a retry-after hint today, so that could be a follow-up.
b36b826 to
7d3ffc9
Compare
Nishanth Kotla (Nishanth29)
changed the title
… room Add --sweperf-resume-retry-budget (default 0, off). With a budget, a cycle retries ResumeActor on ResourceExhausted or Unavailable with 50ms-to-4s backoff until the budget runs out. Retried refusals are recorded on ResumeActorRefused and only the final attempt on ResumeActor. In the task record, resume_max_ms includes the wait when a resume was retried, and capacity_retries counts the refusals. Signed-off-by: Nishanth Kotla <nishanthkotla@google.com>
7d3ffc9 to
df0be56
Compare
Haowei Cai (Roy) (roycaihw)
left a comment
Haowei Cai (Roy) (roycaihw)
left a comment
There was a problem hiding this comment.
LGTM
In sweperf, a ResumeActor refused because no worker has room (ResourceExhausted or Unavailable) fails the whole task. Near capacity most tasks fail while the node keeps serving resumes at the same rate. agentsession already has a retry budget for this.
--sweperf-resume-retry-budget (default 0, off) retries those errors with 50ms-to-4s backoff until the budget runs out. The cap is above agentsession's 1s because a 1s cap sent 9.4 refused attempts per successful resume. Retried refusals go to a new ResumeActorRefused row, and only the final attempt to ResumeActor. In the task record, resume_max_ms includes the wait when a resume was retried, and capacity_retries counts the refusals. Keep the budget at 0 for goodput and host-scale sweeps.
1000 users, one c3-highcpu-192-metal node, 8 microVM workers, minutes 2-10:
AI use: AI-assisted, I reviewed and tested it.