[Misc] Finish repository owned agent tooling

.x/README.md

@@ -6,6 +6,8 @@ Agents and tools are discovered from their directories; no file or function inve
 
 `x.yml` is the only configuration file and uses the same engine-validated schema in every repository. It contains repository identity, a `profile` mapping for workflow/target/routing settings and optional Python pins. Optional feature fields are omitted when unused, not replaced with repository-specific keys or dummy values.
 
+`x.yml` also declares how the engine publishes this repository's changes (`publication`), which repositories and organization data its tools may reach (`access`) and the paths and bounds of its fix jobs (`job`). The engine enforces central caps on all three. The repository is served automatically while the X Engineering Agent GitHub App is installed on it and this `.x/x.yml` exists on its default branch.
+
 A round and its durable jobs retain the original verified commit and source digest. Candidate edits cannot replace their agents or repository checks. A later upstream commit applies only to a later round. An unavailable enabled package blocks explicitly; it never selects a sibling repository or central implementation as a fallback.
 
 Azure CLI and Extensions started from the same preservation baseline but own independent copies and versions. A shared fix needs separate reviewed commits in both repositories; there is no sibling import, shared mutable cache, or automatic synchronisation.
@@ -23,3 +25,5 @@ Local validation checks the source-only contract; it does not authorize a new ru
 Onboarding: keep identity, workflow/routing settings under `profile`, and optional Python pins in the single `x.yml`. Put agent definitions in `definitions/` and Python tools under `tools/<role>/`. Do not create a separate `profile.yml`. Do not maintain duplicate agent, tool or file lists. Validation checks under `tools/validation/` are private Coordinator tools.
 
 Run the offline source linkage regressions with `python -m pytest -q tests/agent_tools --noconftest -p no:cacheprovider`. Tests stay outside the runtime `.x` snapshot.
+
+Live-test selection, skip explanations and target planning are owned by this package. The engine executes the private planning helpers from the pinned snapshot and retains workflow authorization, credential handling and privileged writes.

.x/definitions/tester.md

@@ -1,6 +1,6 @@
 # Repository scope: Azure/azure-cli
 
-This definition is active only for `Azure/azure-cli`. Its `.x/x.yml` profile determines enabled stages. Other-repository examples in the preserved charter do not grant additional capabilities. Generic helper APIs keep their existing deterministic safeguards.
+This definition is active only for `Azure/azure-cli`. Its `.x/x.yml` profile determines enabled stages. Generic helper APIs keep their existing deterministic safeguards.
 
 # Tester - Run Live Tests via GitHub Actions
 
@@ -25,7 +25,7 @@ from x_engineering_agent.tools.live_tests.workflows import (
 )
 run = dispatch_live_test_workflow(
     pr_number=33150,
-    pr_repo="Azure/azure-cli-extensions",  # or "Azure/azure-cli"
+    pr_repo="Azure/azure-cli",
 )
 print(get_workflow_run(run["id"]))
 PYEOF
@@ -44,12 +44,8 @@ The opening `<<'PYEOF'` MUST be quoted. Closing tag at column 0.
 
 Given a PR in a repository whose profile enables live tests:
 
-Do not run Tester on analysis-only `Azure/terraform-provider-azapi` or
-`Azure/azclips`; neither has a live-test path. Use upstream CI for Azclips.
-For `Azure/azure-powershell`, use the profile's
-`live-test-powershell.yml` workflow to run TestFx `Record` tests scoped to
-changed `<Service>.Test` files. Do not apply Azure CLI's `azdev test --live`
-conventions to PowerShell.
+Use the profile's `live-test.yml` workflow, which runs `azdev test --live`
+for the resolved module or extension.
 
 1. **Run for each Copilot complete inflight PR head SHA** — do not gate on draft state or "new test files". A PR is ready when timeline shows Copilot finished work ("Copilot finished work on behalf of ...").
 
@@ -66,9 +62,7 @@ conventions to PowerShell.
    ```python
    from x_engineering_agent.tools.live_tests.workflows import dispatch_live_test_workflow
    # The dispatcher reads this repository's profile and changed files.
-   # PowerShell uses changed_ps_test_files, resolves a psmodule and selects
-   # live-test-powershell.yml. An empty selection returns a neutral skip.
-   # CLI repositories resolve a module or extension and use live-test.yml.
+   # This package resolves a module or extension and uses live-test.yml.
    run = dispatch_live_test_workflow(
        pr_number=pr["pr_number"], pr_repo=pr["repo"],
    )
@@ -78,15 +72,49 @@ conventions to PowerShell.
 5. **Read state ONCE and never wait:**
    ```python
    from x_engineering_agent.tools.live_tests.workflows import get_workflow_run
-   result = get_workflow_run(run["id"])
+   result = get_workflow_run(
+       run["id"], expected_head_sha=run["pr_head_sha"],
+       pr_number=run["pr_number"], pr_repo=run["pr_repo"],
+   )
+   # With pr_repo, a completed run's live-test-summary.md is written to the
+   # "Live test" section of the PR's single Agent report comment.
    # result["status"] is "queued" | "in_progress" | "completed"
-   # result["conclusion"] is "success" | "failure" | "cancelled" | "timed_out" | None
+   # result["conclusion"] is "success" | "failure" | "unverified" |
+   #   "cancelled" | "timed_out" | None
    ```
+   `success` is reported only after the run's title proves it tested
+   `expected_head_sha` and its results artifact (`results.xml` / `*.trx`)
+   shows at least one executed test and no failures or errors. A green
+   workflow that ran 0 tests, skipped every test, uploaded no results or
+   tested another SHA becomes `unverified` (`verification_reason` says why);
+   never describe it as a pass.
    - `status != "completed"` → return `{"ran": True, "pending": True, "run_url": ...}`. The loop must stop the round here; the next round will re-read.
    - `status == "completed"` → return `{"ran": True, "conclusion": result["conclusion"], "run_url": ...}`.
 
 6. **No skip branch for "no new tests"**
    Always dispatch once per PR head SHA and return pending/completed state.
+   For a source-only PR (no runnable changed test file) the dispatcher
+   derives the changed command module(s)/extension(s)/service(s) from the
+   source paths and dispatches a module-scoped run (`test_scope=module`,
+   `module=<names>`) when the workflow declares the `test_scope` input. If
+   the workflow lacks that input, or more than three modules changed, it
+   returns a skip whose reason names the untested modules; that is not a
+   passing result.
+7. **Never drop the head SHA.** A `422` from `workflow_dispatch` is raised as
+   an error; the dispatcher never retries without `pr_head_sha`.
+
+8. **Missing recordings (`pr["needs_recording"]`).** Foundry validation
+   replays the touched module's `ScenarioTest`s offline from the committed
+   recordings, which jobs ship read-only and may not edit. A new or changed
+   scenario test without a matching recording sets `needs_recording` on the
+   job; the in-flight PR exposes it as `pr["needs_recording"]`. Resolve it
+   here: dispatch with `commit_recordings=True` so the workflow records live,
+   verifies the cassette in playback and commits it to the Agent branch. The
+   recording commit is a new head SHA, so the next round tests that head
+   normally. Until a run for the PR has committed the recording, report the
+   live test as not yet resolving the missing recording; never describe the
+   scenario as covered. PowerShell `SessionRecords` are also read-only, but
+   there is no local PowerShell playback; their `Record` runs stay as above.
 
 ## Where the live tests actually run
 
@@ -98,6 +126,8 @@ conventions to PowerShell.
 - Federated `az login` to BAMI via OIDC (repo secrets `BAMI_TENANT_ID`, `BAMI_SUBSCRIPTION_ID`, `BAMI_CLIENT_ID`)
 - `azdev test {name} --live --series`, uploads `results.xml` + log
 - Fails the run if zero tests were collected (silent skip is treated as failure)
+- A run with no changed tests exits green; the Agent verifies the results
+  artifact before reporting a pass, so such a run is `unverified`
 - Comments the pass/fail block back on the PR
 
 All Azure access is inside the runner. Nothing on the agent side.
@@ -108,14 +138,13 @@ https://github.com/Azure/issue-sentinel/blob/main/.github/workflows/live-test.ym
 ## Tools I Use (from the Agent tools package)
 
 ```python
-from x_engineering_agent.tools.agents.tester.azure_cli import (
-    changed_test_files,
+from x_engineering_agent.tools.live_tests.selection import (
+    changed_test_files,                  # changed_test_files(repo_full, pr_files)
     get_pr_regression_coverage_summary,
-    infer_target,
-    resolve_target,
 )
-from x_engineering_agent.tools.agents.tester.azure_powershell import (
-    changed_ps_test_files,
+from x_engineering_agent.tools.targets.inference import (
+    infer_target_for_repo,
+    resolve_target_for_repo,
 )
 from x_engineering_agent.tools.live_tests.workflows import (
     dispatch_live_test_workflow,  # POST workflow_dispatch + locate the new run
@@ -126,5 +155,5 @@ from x_engineering_agent.tools.live_tests.workflows import (
 ## Boundaries
 
 **I do:** Dispatch the live test workflow once per head SHA, read its state once per round, return the conclusion or "still pending."
-**I don't:** Act on `Azure/azclips`, block the round, provision
+**I don't:** Block the round, provision
 infrastructure, SSH anywhere, write code, comment on the PR or approve PRs.

.x/tests/fixtures/pr_template.md

@@ -0,0 +1,24 @@
+**Related command**
+<!--- Please provide the related command with az {command} if you can, so that we can quickly route to the related person to review. --->
+
+**Description**<!--Mandatory-->
+<!--Why this PR? What is changed? What is the effect? etc. A high-quality description can accelerate the review process.-->
+
+**Testing Guide**
+<!--Example commands with explanations.-->
+
+**History Notes**
+<!--If your PR is not customer-facing, use {Component Name} in the PR title. Otherwise, use [Component Name] to allow our pipeline to add the title as a history note. If you need multiple history notes or would like to overwrite the note from the PR title, please fill in the following templates.-->
+
+[Component Name 1] BREAKING CHANGE: `az command a`: Make some customer-facing breaking change
+[Component Name 2] `az command b`: Add some customer-facing feature
+
+---
+
+This checklist is used to make sure that common guidelines for a pull request are followed.
+
+- [ ] The PR title and description has followed the guideline in [Submitting Pull Requests](https://github.com/Azure/azure-cli/tree/dev/doc/authoring_command_modules#submitting-pull-requests).
+
+- [ ] I adhere to the [Command Guidelines](https://github.com/Azure/azure-cli/blob/dev/doc/command_guidelines.md).
+
+- [ ] I adhere to the [Error Handling Guidelines](https://github.com/Azure/azure-cli/blob/dev/doc/error_handling_guidelines.md).

.x/tests/test_automation.py

@@ -0,0 +1,22 @@
+# --------------------------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License. See License.txt in the project root for
+# license information.
+# --------------------------------------------------------------------------
+
+"""Validate this package's scheduled automation settings against the engine schema."""
+
+from pathlib import Path
+
+import yaml
+
+from x_engineering_agent.automation import settings
+
+REPOSITORY = "Azure/azure-cli"
+PACKAGE = Path(__file__).resolve().parents[1]
+
+
+def test_automation_settings_are_valid():
+    section = yaml.safe_load((PACKAGE / "x.yml").read_text())["automation"]
+    parsed = settings.parse(REPOSITORY, section)
+    assert parsed.pipelines and parsed.common_issues and parsed.review_follow_up and parsed.reopen

.x/tests/test_failures.py

@@ -0,0 +1,69 @@
+# --------------------------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License. See License.txt in the project root for
+# license information.
+# --------------------------------------------------------------------------
+import importlib
+import sys
+import types
+from pathlib import Path
+
+import pytest
+
+PACKAGE = Path(__file__).resolve().parents[1]
+TOOLS = PACKAGE / "tools"
+
+
+def _repository_tools():
+    module = sys.modules.get("repository_tools")
+    if module is None:
+        module = types.ModuleType("repository_tools")
+        module.__path__ = [str(TOOLS)]
+        sys.modules["repository_tools"] = module
+    elif str(TOOLS) not in module.__path__:
+        module.__path__.append(str(TOOLS))
+    return module
+
+
+_repository_tools()
+failures = importlib.import_module("repository_tools.reviewer.azure_cli.failures")
+
+STORAGE_SRC = "src/azure-cli/azure/cli/command_modules/storage/custom.py"
+STORAGE_TEST = "src/azure-cli/azure/cli/command_modules/storage/tests/latest/test_blob.py"
+NETWORK_TEST = "src/azure-cli/azure/cli/command_modules/network/tests/latest/test_nsg.py"
+CORE_SRC = "src/azure-cli-core/azure/cli/core/commands/__init__.py"
+CORE_TEST = "src/azure-cli-core/azure/cli/core/tests/test_help.py"
+BUCKETS = ("pr_relevant", "out_of_scope", "unverified", "uncertain")
+
+
+def _failure(path, test_id="test_x"):
+    return {"file": path, "test_id": test_id}
+
+
+@pytest.mark.parametrize(("failure", "base", "bucket", "reason"), [
+    (_failure(NETWORK_TEST), None, "unverified", "not confirmed on base"),
+    (_failure(NETWORK_TEST), {"ref": "dev", "failed": [_failure("azure-cli/" + NETWORK_TEST)]},
+     "out_of_scope", "also fails on base `dev` (pre-existing)"),
+    (_failure(NETWORK_TEST), {"ref": "dev", "passed": [_failure(NETWORK_TEST)]},
+     "uncertain", "passes on base `dev`"),
+    (_failure(NETWORK_TEST), {"ref": "dev", "failed": [_failure(NETWORK_TEST, "test_other")]},
+     "unverified", "not confirmed on base"),
+    (_failure(STORAGE_TEST), {"ref": "dev", "failed": [_failure(STORAGE_TEST)]},
+     "pr_relevant", "also fails on base `dev`"),
+])
+def test_failures_need_base_evidence_before_being_called_pre_existing(failure, base, bucket, reason):
+    result = failures.classify_test_failures([failure], [STORAGE_SRC], base_results=base)
+    assert [item["file"] for item in result[bucket]] == [failure["file"]]
+    assert reason in result[bucket][0]["reason"]
+    assert sum(len(result[key]) for key in BUCKETS) == 1
+
+
+def test_core_change_puts_core_tests_in_scope_and_other_modules_uncertain():
+    result = failures.classify_test_failures(
+        [_failure(CORE_TEST), _failure(NETWORK_TEST)], [CORE_SRC],
+    )
+    assert [item["file"] for item in result["pr_relevant"]] == [CORE_TEST]
+    assert "same core (`azure-cli-core`)" in result["pr_relevant"][0]["reason"]
+    assert [item["file"] for item in result["uncertain"]] == [NETWORK_TEST]
+    assert "azure-cli-core" in result["uncertain"][0]["reason"]
+    assert result["out_of_scope"] == [] and result["unverified"] == []

.x/tests/test_hooks.py

@@ -0,0 +1,98 @@
+# --------------------------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License. See License.txt in the project root for
+# license information.
+# --------------------------------------------------------------------------
+
+"""Run this package's engine hooks through the X Engineering Agent broker."""
+
+from pathlib import Path
+
+import pytest
+
+from x_engineering_agent.repository import testing
+
+REPOSITORY = "Azure/azure-cli"
+PACKAGE = Path(__file__).resolve().parents[1]
+
+
+@pytest.fixture(autouse=True)
+def pinned(monkeypatch, tmp_path):
+    testing.pin(monkeypatch, tmp_path, testing.local(PACKAGE, REPOSITORY))
+
+
+DIRECTORIES = {
+    ("Azure", "azure-cli", "src/azure-cli/azure/cli/command_modules"): {"vm", "network", "storage"},
+    ("Azure", "azure-cli-extensions", "src"): {"containerapp", "aks-preview"},
+}
+
+
+@pytest.fixture(autouse=True)
+def offline_directories(monkeypatch):
+    from x_engineering_agent.tools.targets import discovery
+
+    def listing(owner, repo, path, branch, token=None):
+        return set(DIRECTORIES[(owner, repo, path)])
+
+    monkeypatch.setattr(discovery, "_list_repo_dirs", listing)
+
+
+def hook(name, *args, **kwargs):
+    return testing.hook(REPOSITORY, name, *args, **kwargs)
+
+
+def test_targets_follow_command_module_layout():
+    target = hook("infer_target", "az vm create fails", ["src/azure-cli/azure/cli/command_modules/vm/custom.py"])
+    assert target["name"] == "vm"
+    assert hook("resolve_target", "vm")["name"] == "vm"
+
+
+def test_titles_use_cli_style():
+    title = hook("pr_title", component="vm", issue_number=1, command="az vm create", summary="Fix up")
+    assert title.startswith("[")
+    assert "vm" in title.lower()
+    assert hook("pr_format_guidance", component="vm", issue_number=1)
+    assert hook("execution_instructions")
+
+
+def test_codegen_guidance_targets_core():
+    assert isinstance(hook("codegen_guidance", "vm"), str)
+
+
+def test_changed_tests_and_failures():
+    files = ["src/azure-cli/azure/cli/command_modules/vm/tests/latest/test_vm_commands.py"]
+    assert hook("changed_test_files", files)
+    failed = hook("extract_failed_tests", "FAILED src/x/tests/latest/test_a.py::Test::test_b - boom")
+    assert failed
+
+
+def test_readiness_requires_validation_and_a_matching_title():
+    pr = {"title": "[Compute] `az vm create`: Fix", "body": "",
+          "head": {"sha": "a" * 40, "repo": {"full_name": "fork/azure-cli"}}}
+    changes = [{"filename": "src/azure-cli/azure/cli/command_modules/vm/custom.py", "status": "modified"}]
+    policy = hook("readiness_policy", pr, changes)
+    assert policy["validation_required"] is True
+
+
+def test_definitions_reference_existing_engine_helpers_and_tools():
+    assert testing.definition_problems(testing.local(PACKAGE, REPOSITORY)) == []
+
+
+MODULE = "src/azure-cli/azure/cli/command_modules/{}/custom.py"
+
+
+@pytest.mark.parametrize(("files", "expected"), [
+    ([MODULE.format("storage")], ("module", ["storage"])),
+    ([MODULE.format(name) for name in ("storage", "network", "vm", "keyvault")],
+     ("module", ["storage", "network", "vm", "keyvault"])),
+    (["src/azure-cli-core/azure/cli/core/commands/__init__.py"], (None, [])),
+    (["doc/sphinx/index.rst"], (None, [])),
+    ([], (None, [])),
+    ([MODULE.format("storage"),
+      "src/azure-cli/azure/cli/command_modules/storage/tests/latest/test_blob.py"],
+     ("module", ["storage"])),
+], ids=["one-module", "many-modules", "core-only", "docs-only", "no-files", "ignores-tests"])
+def test_source_only_changes_resolve_module_live_test_scope(files, expected):
+    kind, names = hook("live_test_source_targets", files)
+    assert (kind, list(names)) == expected
+    assert not hook("live_test_plan", files[:1])["runnable"]