fix(cluster_management): clean up leaked status records

[mkoura]

Fix cluster management bugs that could leave status records behind, or give a test fewer resources than it asked for. The stale records cleanup ignores records of live workers, so a leaked record stayed for the rest of the testrun.

• Clean up records when obtaining a cluster instance fails. When get_cluster_instance raised (timeout, dead clusters check, an error during respin):

  • a "prio" record of a priority test kept all other workers waiting until they timed out
  • a "respin in progress" record of a claimed respin kept all workers away from the cluster instance

  Remove both records, and hand back the claimed respin as "needs respin" so other workers perform it once the instance is idle. When _respin itself raises, mark the cluster instance dead, same as when the cluster fails to start - the failure can be permanent (e.g. FORBID_RESPIN), so the respin is not handed back in that case.

• Call on_test_stop even when the log files check fails, so the resources locked by the test are released.

• Make OneOf filters in lock_resources and use_resources of the same request select distinct resources. Previously the "use" filter could select the resource the "lock" filter had just selected, and the test got one resource instead of two. No current fixture mixes the two, so this was latent.

• Return the directory of cached custom startup scripts instead of the start-cluster file. The fixtures returned the file when the scripts were already generated by another worker, so the respin with such scriptsdir failed and the cluster instance was marked dead. Also fix the epoch length in the epoch_length_start_cluster docstring and lock name (1500, not 1200).

Unit tests were added for the record cleanup and the distinct OneOf picks; all framework_tests pass.

[Copilot]

Copilot review overview

🟢 Approval recommended

The cleanup and allocation fixes are consistent with existing coordination semantics and have focused unit coverage.

Review effort: Balanced
Findings: None

What changed in this PR

Improves cluster-management cleanup and resource allocation reliability.

Changes:

• Cleans priority, respin, and resource records on failure paths.
• Ensures OneOf filters select distinct resources.
• Corrects cached startup-script paths and epoch-length naming.

File	Description
cluster_getter.py	Adds failure cleanup and distinct resource selection.
resources_management.py	Supports excluding previously selected resources.
tests/​conftest.py	Guarantees test-stop cleanup after log-check failures.
tests/​test_configuration.py	Fixes cached script paths and epoch naming.
tests/​test_kes.py	Returns the cached scripts directory.
framework_tests/​test_cluster_getter.py	Tests cleanup and resource-selection behavior.
framework_tests/​test_resources_management.py	Tests previously selected resource exclusion.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.