Skip to content

fix(integration): turn off EVM rate limiting on the benchmark's RPC nodes - #613

Merged
bdchatham merged 2 commits into
mainfrom
fix/nightly-benchmark-evm-rate-limit
Oct 13, 2026
Merged

bdchatham merged 2 commits into
mainfrom
fix/nightly-benchmark-evm-rate-limit

Conversation

@bdchatham

Copy link
Copy Markdown
Collaborator

Summary

The nightly benchmark has failed every night since 2026-10-08. This PR turns off seid's per-IP EVM rate limit on the benchmark's RPC followers, so the load measures the chain again.

Cause

sei-chain#4403 (merged 2026-10-07) turned on per-IP rate limiting for EVM JSON-RPC by default, at 200 requests a second with a burst of 200. The nightly chain runs seid from main, so it has had the limit since the 2026-10-08 image. The benchmark sends all its load from one sei-load pod, which is one client IP.

  • Steady-state fails. Before it sends, sei-load reads the balance of each of its 500 accounts, 16 at a time. The follower answers 429, and the funder stops the run. Every run from 2026-10-08 to 2026-10-13 logs this line:
    Error: failed to fund accounts: funder: balance 0x…: 429 Too Many Requests: too many requests
    
  • Saturation passes, but its number changed. The transactions included in its window fell from 2,049–3,962 (2026-10-07) to 320–338 (2026-10-08 onward). Its only gate is "more than zero included", so it stays green.

Change

spec.overrides cannot carry the switch, because sei-config does not know the rate-limit keys and rejects any key it does not know. spec.configValues takes any key, and harbor's staging syncers already use it.

  • sdk/sei: NodeSpec.ConfigValues maps to spec.configValues. validateNodeSpec refuses an entry with no file name, no key, or a value with no JSON form. The k8s provider keeps each value's JSON type, so false reaches app.toml as a TOML boolean.
  • test/integration: the harness spec gets rpcConfigValues, and TestNightlyBenchmark sets evm.rate_limiting_enabled = false in app.toml on both followers, for both workloads. The validators take no client RPC, so they keep the default.

Verification

  • go test ./sdk/...: pass. The new cases cover the validation, including a valid state-sync spec with a bad config value, and the render of a boolean and an integer.
  • golangci-lint v2.12.1 on ./sdk/...: 0 issues. With --build-tags integration on ./test/integration/..., the 22 findings are the same on main and on this branch.
  • go vet ./... and go vet -tags integration ./test/integration/: clean.
  • The end-to-end check is the next nightly run after the harness pin moves. Steady-state should fund its accounts, and the saturation included count should go back to the range it had before 2026-10-08.

Rollout

After merge, ecr.yml publishes sei/integration-harness:<sha>. A platform PR then moves the pin in clusters/harbor/nightly/harness/cronjobs.yaml. Harbor's controller already serves spec.configValues.

🤖 Generated with Claude Code

…odes

seid now limits EVM JSON-RPC to 200 requests a second per client IP by
default (sei-chain#4403). The nightly benchmark sends all its load from one
sei-load pod. Since 2026-10-08 the steady-state run fails before it sends,
because the funder's balance reads get 429. The saturation run still
passes its loose inclusion gate, but its included count fell about tenfold.

sei-config does not know the rate-limit keys, so spec.overrides rejects
them. The SDK's NodeSpec gains ConfigValues, which maps to
spec.configValues and takes any key. The benchmark sets
evm.rate_limiting_enabled = false in app.toml on its RPC followers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cursor

cursor Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

PR Summary

Low Risk
SDK validation and k8s rendering for an existing CRD field, plus benchmark-only harness tuning; no auth or production runtime logic changes in this diff.

Overview
Adds NodeSpec.ConfigValues so callers can set typed keys in app.toml / config.toml via the CRD’s spec.configValues, including keys Config / overrides cannot carry because sei-config rejects unknown paths. validateNodeSpec now runs validateConfigValues (CRD-aligned limits, .toml file names, dotted keys, no duplicate file+key, JSON-encodable non-null values). The k8s provider maps each entry through json.Marshal into apiextensionsv1.JSON so booleans and integers keep their type in TOML.

Integration harness: new rpcConfigValues on the scenario spec is passed into RPC NodeSpec creation. The nightly benchmark sets evm.rate_limiting_enabled: false on followers so sei-load from a single IP is not throttled at 429 after seid’s default per-IP EVM JSON-RPC limit.

Reviewed by Cursor Bugbot for commit 0fad381. Bugbot is set up for automated code reviews on this repo. Configure here.

@seidroid seidroid Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adds NodeSpec.ConfigValues to the SDK, mapped to spec.configValues and validated before create, and uses it to set evm.rate_limiting_enabled = false in app.toml on the nightly benchmark's RPC followers. The key exists in sei-chain's evmrpc config and the change is correct; the two notes below are gaps in the new SDK validation, which still fails loudly at the API server. I kept codex's typed-nil finding as a suggestion (the PR tree was read at merge ref f2ab50e, and I did not build or test because Go is not installed here).

Non-blocking

2 findings on the changed lines, as inline comments.

seidroid review · decision approve · session 59d73431a39e4dc0a19bfcaa08c788b4 · turn resp_claude_296b8b4f2bf52906e8c82d0d19492ed1 · item 2aad65cff5835f4b9c2378ef4cac5c05

Findings: 0 blocking | 2 non-blocking | 2 posted inline

Comment thread sdk/sei/sei.go Outdated
Comment thread sdk/sei/sei.go Outdated
validateConfigValues checked only for blank fields and an untyped nil.
A typed nil such as a nil *bool encodes as null without an error, and a
malformed file name, a malformed key or a repeated (file, key) pair passed
here and failed at the API server instead. The check now reads the encoded
value and mirrors the CRD's patterns, lengths, item limit and map keys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bdchatham

Copy link
Copy Markdown
Collaborator Author

@seidroid review

@seidroid seidroid Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the last review, commit 0fad381 changed validateConfigValues: it now rejects any value that encodes to null, including a typed nil, and it applies the CRD's file-name and key patterns and length limits, the 100-entry cap, and the rule against a repeated (fileName, key) pair, each with a test, so both of my earlier findings are addressed and nothing else blocks. The codex reading reported nothing, which I agree with; I read the PR tree at merge ref e87e261 and did not build, because Go is not installed here.

seidroid review · decision approve · session 59d73431a39e4dc0a19bfcaa08c788b4 · turn resp_claude_2c5edcb6279ce482ad7ee3ad8898c5d3 · item ab1ed055c7a754338867e9fdf159f0a8

Findings: 0 blocking | 0 non-blocking | 0 posted inline

@bdchatham
bdchatham merged commit 10db420 into main Oct 13, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant