Repository navigation
Conversation
jiayimeow
reviewed
Nov 3, 2026
| description: "Actor density limit: personal-assistant-light on 1 gVisor worker" | ||
| targetCluster: dev | ||
| file: /app/tests/agentsession.py | ||
| duration: 90m |
Collaborator
There was a problem hiding this comment.
Do we roughly know how long the density test would take?
Contributor
Author
There was a problem hiding this comment.
Each step takes ~7–8 min wall-clock (5m load + ~60s boomer teardown + 70s --atelet-lag-s wait). Starting at u=8 (per your other comment), a full search up to 1024 with 2–3 binary search steps is 7–10 total steps, so roughly 50–75 min per suite (capped by -t 90m, and exiting early once the bracket narrows). Moving from u=1 baseline to u=8 baseline saves ~20m.
…light suites Fixed-user runs show latency at one concurrency point, leaving the per-node actor ceiling to manual trial and error. Add density_runner.py, invoked by runner.py --density-step-duration, to double concurrent users from 1 up to -u and binary-search once wake or suspend P90, wake failure ratio, actor survival, or OOM/eviction counters trip. Add density_pa_light_gvisor and density_pa_light_microvm entries to benchmarking/automation/tests.yaml. Signed-off-by: Noah West <westnoah@google.com>
…ween steps, catch silent failures Signed-off-by: Noah West <westnoah@google.com>
noahlwest
force-pushed
the
density-limit-benchmark
branch
from
November 5, 2026 17:02
3b0d5d5 to
6f14c4c
Compare
Extract _make_args and convert the three run_density_sweep scenarios in test_density_runner.py into a subTest table that passes runner.upload directly and checks density_limit fields. Share a module-level ready helper in test_cluster_facts.py alongside discover and snapshot. No behavior change. Signed-off-by: Noah West <westnoah@google.com>
run_density_sweep evaluated and appended a step before checking step.exit_code, so a crashed runner.py subprocess appeared in density_limit as first_failed_users with failure_reason set to missing wake measurements. status.json also tied stats_generated to sweep_ok, marking runs that aborted after earlier passing steps as having written no stats. Break on a non-zero step.exit_code before calling evaluate_step, and set stats_generated from bool(steps). Signed-off-by: Noah West <westnoah@google.com>
actorMemory defaults to 256Mi in workloads/deploy.sh and --lifecycle-mode defaults to async in common/lifecycle_mode.py. Drop both from density_pa_light_gvisor and density_pa_light_microvm, document the remaining sizing knobs, and note how duration bounds step launches and the orchestrator Job timeout. Signed-off-by: Noah West <westnoah@google.com>
Link locust/density_runner.py in docs/guides/benchmarking.md and spell out the density_limit fields, the user-count suffix on density_step_NNNN, what --no-cluster-facts skips, and how the search stops when the -t budget runs out. Tighten the comments on START_USERS, bisection stopping, actor cleanup timeouts, and --density-step-duration. Signed-off-by: Noah West <westnoah@google.com>
Signed-off-by: Noah West <westnoah@google.com>
Signed-off-by: Noah West <westnoah@google.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
why: Existing locust benchmarks run for fixed user counts, so finding how many concurrent actors a node can hold before wake/suspend latency or reliability breaks down takes manual trial-and-error runs.
This adds
--density-step-durationtorunner.py(and addsdensity_runner.py) to automate that search. It runs an 8-user baseline step, doubles the user count each step up to-u, and binary searches once a step trips wake/suspend P90, wake failure rate, live actor count, or OOM/eviction checks, then writes the finalstats.jsonlat the end of the sweep. It also adds gVisor and microVM single-worker density suites forpersonal-assistant-lightintests.yaml.Part of #3134
AI use: AI-assisted code impl, I reviewed the diffs and ran the density sweeps on test clusters.