Repository navigation
Feat/benchmark job node affinity - #52
Conversation
Validate required and preferred scheduling rules, apply them to runner Jobs, and retain affinity snapshots in run status responses. Add API, store, and coordinator tests. Signed-off-by: amh1k <abdulmoizx97@gmail.com>
Add required and preferred rule editing, validate and serialize affinity, and display saved rules in run details. Cover validation, submission, status parsing, and UI behavior with frontend tests. Signed-off-by: amh1k <abdulmoizx97@gmail.com>
Detect continuous scheduler rejection after a 60-second grace period, expose scheduling diagnostics, and clean up failed runs. Add regression tests and document the timeout policy. Signed-off-by: amh1k <abdulmoizx97@gmail.com>
Signed-off-by: amh1k <abdulmoizx97@gmail.com>
spron-in
left a comment
There was a problem hiding this comment.
Main concern: the 60s scheduling failure applies to every run (see jobs.go). Also some test-tooling and typing nits; suggestions inline.
| "dev": "vite", | ||
| "build": "tsc --noEmit && vite build", | ||
| "test:form": "tsc src/benchmarkFormValidation.ts --target ES2020 --module ESNext --moduleResolution Bundler --outDir dist/form-test && node tests/benchmarkForm.test.mjs", | ||
| "test:form": "tsc src/BenchmarkForm.tsx --jsx react-jsx --target ES2020 --module ESNext --moduleResolution Bundler --esModuleInterop --skipLibCheck --outDir dist/form-test && node tests/benchmarkForm.test.mjs && node tests/benchmarkNodeAffinity.test.mjs && node tests/benchmarkNodeAffinityForm.test.mjs", |
There was a problem hiding this comment.
To render components under node:test, this now compiles the whole BenchmarkForm.tsx graph with tsc (--jsx, --esModuleInterop, --skipLibCheck). That's why value imports in src/ now need .js extensions while type imports don't. The mixed style will spread to every component a test touches.
Vite is already the build tool, so Vitest + @testing-library/react (jsdom environment) would run the .tsx directly, with no compile step and no import-extension changes. Proposed split: keep the pure-logic tests on the existing runner and move the component test to Vitest:
"test:form": "tsc src/benchmarkFormValidation.ts --target ES2020 --module ESNext --moduleResolution Bundler --outDir dist/form-test && node tests/benchmarkForm.test.mjs && node tests/benchmarkNodeAffinity.test.mjs",
"test:components": "vitest run",Then revert the .js suffixes in src/, port benchmarkNodeAffinityForm.test.mjs to a *.test.tsx, and add npm run test:components to ci.yml. (Not a one-click suggestion on purpose: applying the script change alone would silently drop the component test.)
| "jsdom": "24.1.3", | ||
| "react": "^18.3.1", | ||
| "react-dom": "^18.3.1", | ||
| "react-test-renderer": "18.3.1", |
There was a problem hiding this comment.
Nit: jsdom and react-test-renderer are pinned exactly while every other devDependency uses ^, and together they add ~800 lines to package-lock.json. react-test-renderer is also deprecated upstream; React recommends @testing-library/react. Ideally both get replaced by vitest + @testing-library/react (see the test:form comment). If the current runner stays, at least align the ranges:
| "jsdom": "24.1.3", | |
| "react": "^18.3.1", | |
| "react-dom": "^18.3.1", | |
| "react-test-renderer": "18.3.1", | |
| "jsdom": "^24.1.3", | |
| "react": "^18.3.1", | |
| "react-dom": "^18.3.1", | |
| "react-test-renderer": "^18.3.1", |
|
Follow-up suggestion: tolerations. Dedicated benchmark node pools are usually tainted (e.g. Not a blocker for this PR, but could we track it in a follow-up issue? Rough scope:
|
Signed-off-by: amh1k <abdulmoizx97@gmail.com>
Yes we definitely implement this. I have opened up issue #54 for this. Will open up a pr shortly for this |
…form state Disable early scheduling failure by default, wire configuration through Helm, and strengthen diagnostic lookup assertions. Signed-off-by: amh1k <abdulmoizx97@gmail.com>
…ng Library Signed-off-by: amh1k <abdulmoizx97@gmail.com>
spron-in
left a comment
There was a problem hiding this comment.
Thanks! The backend validation, the scheduling-failure tracking, and cleanup look solid. Requested changes below. Most of this is removing code; the backend and API contract stay as they are.
Needs to change
- Admin placement in Helm. Add
runner.nodeSelectorandrunner.tolerations, applied to every runner Job. Affinity alone can't reach tainted, dedicated benchmark nodes, which is the usual setup for #40. UsenodeSelectorrather than a Helm-levelaffinity: Kubernetes applies it together with the per-runnodeAffinity, so the two never need merging. - Simplify the per-run editor. Replace the AND/OR, weights, and
Gt/Lteditor with a list ofkey = valuerows, sent as a single required rule usingIn. Keep thenodeAffinityAPI field so a richer editor can come later without an API change. Drop most of the TypeScript copy of the Kubernetes label rules inbenchmarkNodeAffinity.ts; basic checks are enough, and the backend's 400 messages can be shown for the rest. - Default
schedulingGracePeriodto something like2m, not0. With0, a run with an impossible rule stays pending for the full 5-minute timeout and holds one of the 2 run slots. - Hide scheduler messages from users. Messages like
0/5 nodes are available: ...had untolerated taintreveal cluster topology to users who may have no Kubernetes access. Return a generic "no node matches the placement rules" and log the details on the backend.
Minor
- The PR description still says unschedulable runs fail after 60 seconds; the code currently disables early failure by default.
- A blank label value submits
In [""], which only matches nodes whose label value is empty. Treat it as a form error. - Use stable IDs as React keys for groups and conditions instead of the array index.
- Show validation errors only after a field has been touched.
- The affinity is stored twice (
Run.Request.NodeAffinityandRun.NodeAffinity); keep only the prepared copy. - Document that pending time counts against the execution timeout.
- Add runner.nodeSelector and runner.tolerations Helm values, validated at startup and applied to every runner Job, so Jobs can reach tainted, dedicated benchmark nodes. Per-run nodeAffinity is applied on top. - Replace the AND/OR, weighted, Gt/Lt affinity editor with label key=value rows sent as a single required term. The nodeAffinity API is unchanged; the UI keeps only basic checks and relies on backend validation. - Default schedulingGracePeriod to 2m so impossible placement rules don't hold a run slot for the whole execution timeout. - Return a generic message for unschedulable runs and keep the scheduler diagnostic, which reveals cluster topology, in the backend log. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Sergey Pronin <sp@solanica.io>
Signed-off-by: amh1k <abdulmoizx97@gmail.com>
Summary
Add benchmark runner placement controls with administrator-defined Helm settings and optional per-run node affinity.
Closes #40.
Changes
runner.nodeSelectorandrunner.tolerationsHelm values, applied to every runner Job.nodeAffinityAPI contract for advanced scheduling rules.2m, configurable through Helm.Validation