From 9056d4443e7d0758c64f21d6b4e01dc78b906024 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 06:58:02 +0800 Subject: [PATCH 01/17] Assert the three macOS probes a runner has now measured and close the 3.10 segfault item --- .github/workflows/platform-smoke.yml | 11 +++++------ Progress.md | 16 --------------- docs/updates/2026-10.md | 6 ++++++ docs/updates/README.md | 3 ++- test/verify/macos_verify.py | 29 ++++++++++++++-------------- 5 files changed, 28 insertions(+), 37 deletions(-) diff --git a/.github/workflows/platform-smoke.yml b/.github/workflows/platform-smoke.yml index f5837130..5a4d666f 100644 --- a/.github/workflows/platform-smoke.yml +++ b/.github/workflows/platform-smoke.yml @@ -196,12 +196,11 @@ jobs: # holds what was measured, and a capability appearing or disappearing # turns this red and names which one. # - # Three probes are newer than that measurement and are only reported - # (REPORTED in the script; they print what they found and cannot fail - # this step): the click-state read-back, the by-id lookup of a minimised - # window, and the point-based `grab_logical`. The first two re-run tests - # the headless suite already has, where a skip is a pass; here it is - # printed as a skip. That is what pytest is installed for. + # Three of the probes (the click-state read-back, the by-id lookup of a + # minimised window, and the point-based `grab_logical`) were measured on + # this runner on 2026-10-09 and are asserted since. The first two re-run + # tests the headless suite already has, where a skip is a pass; here a + # skip is a failure. That is what pytest is installed for. - run: "python -m pip install --only-binary :all: pytest==9.1.1" - name: Verify the macOS backend against a real window server diff --git a/Progress.md b/Progress.md index 64f170ec..52ba53e0 100644 --- a/Progress.md +++ b/Progress.md @@ -331,7 +331,6 @@ U-20261009-18 的每一個裝置面呼叫都只對假的 ADB／uiautomator2／WD - **三個新的 CI job 還沒在 GitHub 上跑過**（`typing-extras`、`docs`、`free-threaded-scope`）；`free-threaded-scope` 用的是 `3.14t`，setup-python 在 Windows 上怎麼命名那個直譯器沒驗過，紅了就拿掉。 - **free-threaded**：變數範圍已修成只在綁定它的執行緒上有效，並在 3.14.8t（Windows）跑過。`rbac/authorization.py`、`self_healing/locator.py`、`wrapper/device_context.py` 也用 `ContextVar`，在 free-threaded 版上仍會被執行緒繼承，沒有檢查。 -- **macOS smoke 的三個新探針**（click-state 讀回、最小化視窗以 id 查找、`grab_logical` 的點座標）是「只回報、不斷言」；等在 runner 上量到一次，再從 `REPORTED` 移到 `PROBES`。 - **能力探測**：Windows 的鎖定工作站、session 0、低完整性與擷取失敗分支只用假資料判斷；macOS 的三個 preflight 呼叫只對假物件跑過， pyobjc 的 `Quartz` 有沒有 `CGPreflightListenEventAccess` 未知（沒有的話 `recording`／`stop_shortcut` 回 `unknown`）。 - **Windows 的版面鍵名**（`slash` 等 8 個）只在美式配置上實際呼叫過。 @@ -386,18 +385,3 @@ U-20261009-11 之後，Anthropic 的兩條路徑（`anthropic.py`、`anthropic_c - 模型只憑摘要能不能接著做——上限是 3，大約每 3 張截圖就壓縮一次，摘要最多列最新 60 個動作、每個截到 240 字元。 - 真的桌面 PNG 的請求大小離 32 MB 多遠。 - 兩個後端都沒送 `cache_control`，所以現在本來就沒有 prompt cache；要開的話是另一個請求形狀的改動。 - ---- - -## Python 3.10 的 headless segfault：已修，等 CI 確認 - -`TODO` — 修正在 U-20261009-25；要看 ubuntu-22.04 與 macos-14 的 3.10 `pytest-headless` 連續幾次都綠才能結案 - -`test_usb_acl_prompt.py::test_bridge_remember_persists_acl_rule` 在 3.10 間歇 SIGSEGV（2026-09-26 三次、09-30 一次、2026-10-08 四次）。 -查到的機制：每個 `UsbPassthroughPromptDialog` 都因 `TranslatableMixin._tr(self, …)` 把自己存進自己的登錄表而成為參照循環，被丟掉後要等循環回收； -3.10／3.11 的回收會在配置物件時發生，測試掃 `QApplication.topLevelWidgets()` 的當下若剛好回收，清單裡的指標就指向已銷毀的 widget。 -在 Windows 的 3.10.22 上以探針確定性重現（存取違規），修掉循環後不再發生；CI 那個崩潰本身沒在這台機器重現。 -被銷毀的是不是那兩個被丟掉的對話框、為什麼只有 Linux／macOS，是推論。 - ---- - diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 89bc5448..5e7bf199 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -410,3 +410,9 @@ Jeffrey_RPA kept its own `KEY_ALIASES` (40 friendly names) and `_EXTRA_KEY_CODES - **What**: All 41 environment variables are in the three READMEs. The Sphinx build went from 181 warnings to 0 and CI fails on a new one. `typing_contract_verify.py --extras` checks against the real PySide6 / aiortc types with a shrink-only list of 70 modules. New CI jobs `typing-extras`, `docs`, `free-threaded-scope`. Fixed: on a free-threaded build a thread started during a run wrote into that run's variable scope; bindings now hold only on the thread that made them. - **Tests**: `test_sphinx_docs_build.py`, `test_typing_extras_contract.py`, the execution-scope thread tests. - **Open items**: the three new jobs have not run on GitHub yet; other `ContextVar` users on free-threaded builds. `Progress.md` › 雜項. + +## U-20261009-33 · 2026-10-09 · The three newer macOS probes are asserted; the Python 3.10 segfault item is closed · #done #macos #ci + +- **What**: `test/verify/macos_verify.py` now asserts `click-state-readback`, `minimised-window-by-id` and `grab-logical-points`. A macos-14 arm64 runner measured all three as working on 2026-10-09 (3 of 3 and 1 of 1 reused tests passed with none skipped; the point-based grab returned the region asked for), so they moved from `REPORTED` to `PROBES`. The 3.10 headless segfault fixed in U-20261009-25 did not recur in the five `pytest-headless` runs since on ubuntu-22.04 and macos-14 with Python 3.10; its `Progress.md` entry is removed. +- **Tests**: `test_macos_verify_reporting.py` (20). +- **Open items**: the measurement is one runner at display scale 1; a Retina display is still unmeasured (`Progress.md` › macOS). diff --git a/docs/updates/README.md b/docs/updates/README.md index 1772b6fa..bdf51f56 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,7 @@ In the same commit: delete the item from `Progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261009-33 | 2026-10-09 | The three newer macOS probes are asserted; the Python 3.10 segfault item is closed | #done #macos #ci | [2026-10](2026-10.md) | | U-20261009-32 | 2026-10-09 | READMEs list every environment variable, Sphinx builds without warnings, typing is also checked against the real Qt types | #done #docs #ci #typing | [2026-10](2026-10.md) | | U-20261009-31 | 2026-10-09 | Journal artifacts and exact-match secret masking, a Robot structure check, a VLM strategy and verified heals, real capability probes | #done #journal #self-healing #capabilities | [2026-10](2026-10.md) | | U-20261009-30 | 2026-10-09 | Config sync: reused operation ids are refused, old calls are causal, back-off is its own state, assets over HTTP | #done #sync #protocol | [2026-10](2026-10.md) | @@ -381,7 +382,7 @@ In the same commit: delete the item from `Progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 45 | +| [2026-10.md](2026-10.md) | 2026-10 | 46 | | [2026-09-e.md](2026-09-e.md) | 2026-09 | 7 | | [2026-09-d.md](2026-09-d.md) | 2026-09 | 55 | | [2026-09-c.md](2026-09-c.md) | 2026-09 | 38 | diff --git a/test/verify/macos_verify.py b/test/verify/macos_verify.py index f543944f..aa168a5c 100644 --- a/test/verify/macos_verify.py +++ b/test/verify/macos_verify.py @@ -85,6 +85,12 @@ # them is at the application layer — they are menu bar and system UI. So # the count is reported and not asserted. "window-management": True, + # Measured on a macos-14 arm64 runner, 2026-10-09 (Python 3.12): 3 of 3 and + # 1 of 1 reused tests passed with none skipped, and the point-based grab + # returned the region it was asked for on a 1920x1080 display at scale 1. + "click-state-readback": True, + "minimised-window-by-id": True, + "grab-logical-points": True, } #: The real-framework tests the two "reuse" probes run, and how many results @@ -466,22 +472,16 @@ def probe_grab_logical() -> Outcome: ("accessibility-tree", probe_accessibility), ("recorder", probe_recorder), ("window-management", probe_window_management), -] - -#: Probes no runner has measured yet: run and printed in both modes, judged in -#: neither. Each was written on Windows, against the tests or fakes it names. -#: -#: ``click-state-readback`` and ``minimised-window-by-id`` re-run tests that -#: did pass on macos-14 inside the headless suite (2026-10-09, Python 3.10 and -#: 3.14) -- but as a child pytest of *this* job, with a skip counted as "did -#: not work", they have not run anywhere. ``grab-logical-points`` has never -#: executed on a Mac at all. -REPORTED: List[Tuple[str, Callable[[], Outcome]]] = [ ("click-state-readback", probe_click_state), ("minimised-window-by-id", probe_minimised_window), ("grab-logical-points", probe_grab_logical), ] +#: Probes no runner has measured yet: run and printed in both modes, judged in +#: neither. Empty since 2026-10-09, when the three that were here were measured +#: on a macos-14 runner and moved to :data:`PROBES`. A new probe starts here. +REPORTED: List[Tuple[str, Callable[[], Outcome]]] = [] + def main(argv: Optional[List[str]] = None) -> int: parser = argparse.ArgumentParser(description=__doc__) @@ -498,10 +498,11 @@ def main(argv: Optional[List[str]] = None) -> int: outcomes = {name: probe(name, fn) for name, fn in PROBES} print("-" * 72) - print("reported, not asserted — no runner has measured these yet:") + if REPORTED: + print("reported, not asserted — no runner has measured these yet:") reported = {name: probe(name, fn) for name, fn in REPORTED} - - print("-" * 72) + if REPORTED: + print("-" * 72) if options.measure: _print_measurement({**outcomes, **reported}) return 0 From 6284d21d8002dfa02e1bca46126f3ca59a750200 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 07:04:16 +0800 Subject: [PATCH 02/17] Hold every task callback weakly so a running task cannot be what keeps its tab alive A task handle and the relay under it are descendants of the tab. A partial of a bound method, a lambda or a nested function connected to one of them, and the controller's own registry, held the tab's wrapper; the last reference to a parentless tab was then released from inside its grandchild's destructor and the process aborted without a traceback. --- je_auto_control/gui/_slow_op.py | 37 +- je_auto_control/gui/_weak_call.py | 72 ++++ je_auto_control/gui/_worker_thread.py | 24 +- .../gui/remote_desktop/_connect_task.py | 11 +- .../gui/remote_desktop/connection_screen.py | 40 +- .../remote_desktop/webrtc_host_connection.py | 6 +- .../gui/remote_desktop/webrtc_panel_common.py | 21 +- .../webrtc_viewer_connection.py | 29 +- je_auto_control/gui/task_controller.py | 19 +- je_auto_control/gui/usb_browser_tab.py | 5 +- je_auto_control/gui/usb_passthrough_panel.py | 7 +- je_auto_control/gui/vlm_tab.py | 5 +- pyproject.toml | 1 + .../headless/test_gui_weak_callbacks.py | 388 ++++++++++++++++++ 14 files changed, 569 insertions(+), 96 deletions(-) create mode 100644 je_auto_control/gui/_weak_call.py create mode 100644 test/unit_test/headless/test_gui_weak_callbacks.py diff --git a/je_auto_control/gui/_slow_op.py b/je_auto_control/gui/_slow_op.py index fb89722b..bf53d9c1 100644 --- a/je_auto_control/gui/_slow_op.py +++ b/je_auto_control/gui/_slow_op.py @@ -30,22 +30,20 @@ kept the tab's wrapper alive released it from inside the handle's destructor, which destroyed a parentless tab in the middle of destroying its own child and aborted the process. A bound method is therefore kept as a weak reference plus -the leading arguments given in ``args``; pass a method and ``args`` rather than -a ``functools.partial`` or a lambda that closes over the tab. +the leading arguments given in ``args`` (:mod:`je_auto_control.gui._weak_call`); +pass a method and ``args``, never a lambda that closes over the tab. """ import functools -import weakref from typing import Any, Callable, Optional, Tuple from PySide6.QtCore import QObject, Signal +from je_auto_control.gui._weak_call import Callback, WeakCall as _WeakCall, weak_slot from je_auto_control.gui.task_controller import ( CancellationToken, TaskController, TaskUsageError, _widgets_reachable_from, task_controller, ) from je_auto_control.utils.logging.logging_instance import autocontrol_logger -Callback = Optional[Callable[..., object]] - def _call(work: Callable[[], Any], _token: CancellationToken) -> Any: """Worker thread: run the work; a stop has nothing to poll a token for.""" @@ -64,27 +62,6 @@ def _refuse_widgets(work: Callable[[], Any]) -> None: "background work must not hold a widget: read what it needs first and pass plain values") -class _WeakCall: - """``callback(*args, outcome)`` that does not keep a bound method's object alive.""" - - def __init__(self, callback: Callback, args: Tuple[Any, ...] = ()) -> None: - self._args = args - self._strong: Callback = None - self._weak: Optional[weakref.WeakMethod[Callable[..., Any]]] = None - if getattr(callback, "__self__", None) is not None and hasattr(callback, "__func__"): - self._weak = weakref.WeakMethod(callback) # type: ignore[arg-type] - else: - self._strong = callback - - def __call__(self, outcome: object) -> bool: - """Run the callback if there is one and its object still exists; return whether it ran.""" - callback = self._weak() if self._weak is not None else self._strong - if not callable(callback): - return False - callback(*self._args, outcome) - return True - - class SlowOp(QObject): """Runs one blocking call at a time for its parent widget; see the module docstring.""" @@ -186,8 +163,7 @@ def retire(self, stop: Callable[[], Any], *, on_done: Callback = None, self._pending += 1 handle.error.connect(_log_failure) if on_done is not None: - # The partial holds this queue (owned by its parent) and a weak call, never the panel. - handle.result.connect(functools.partial(_run_weak, _WeakCall(on_done, args))) + handle.result.connect(weak_slot(on_done, *args)) # a weak call, never the panel handle.finished.connect(self._one_reported) def _one_reported(self) -> None: @@ -196,11 +172,6 @@ def _one_reported(self) -> None: self.drained.emit() -def _run_weak(call: _WeakCall, outcome: object) -> None: - """GUI thread: deliver one shutdown's result to whoever is still there.""" - call(outcome) - - def stop_each(*stops: Callable[[], Any]) -> None: """Worker thread: call every ``stop`` in order; one that fails does not keep the rest from running.""" for stop in stops: diff --git a/je_auto_control/gui/_weak_call.py b/je_auto_control/gui/_weak_call.py new file mode 100644 index 00000000..0311a9d4 --- /dev/null +++ b/je_auto_control/gui/_weak_call.py @@ -0,0 +1,72 @@ +"""Callbacks for a background task that do not keep the tab alive. + +A task's handle, and the relay under it, are descendants of the tab that +started the work. A connection from one of them to ``functools.partial(self.x, +...)``, to a lambda or to a nested function holds the tab's Python wrapper for +as long as the handle exists -- so the last reference to a parentless tab could +be the one its own grandchild held, and was released from inside that +grandchild's destructor. Qt then destroyed the tab in the middle of destroying +its child and the process aborted, with no traceback. + +:class:`WeakCall` keeps a bound method as a weak reference plus its leading +arguments, and :func:`weak_slot` makes the plain callable a signal is +connected to. A ``functools.partial`` of a bound method is taken apart the same +way, so a helper that receives ``on_done`` from a tab can simply wrap what it +was given. Anything else (a module-level function, a signal's ``emit``, a +method of a child widget) is held as it is: write the callback as a method and +pass its arguments, never as a lambda or a nested function that closes over +the tab -- ``test_gui_weak_callbacks.py`` fails a new one. +""" +import functools +import weakref +from typing import Any, Callable, Dict, Optional, Tuple + +Callback = Optional[Callable[..., object]] + + +def _is_bound_method(callback: object) -> bool: + return getattr(callback, "__self__", None) is not None and hasattr(callback, "__func__") + + +class WeakCall: + """``callback(*args, *outcome)`` that does not keep a bound method's object alive.""" + + def __init__(self, callback: Callback, args: Tuple[Any, ...] = ()) -> None: + keywords: Dict[str, Any] = {} + while isinstance(callback, functools.partial): + args = tuple(callback.args) + tuple(args) + keywords = {**callback.keywords, **keywords} + callback = callback.func + self._args = args + self._keywords = keywords + self._strong: Callback = None + self._weak: Optional[weakref.WeakMethod[Callable[..., Any]]] = None + if _is_bound_method(callback): + self._weak = weakref.WeakMethod(callback) # type: ignore[arg-type] + else: + self._strong = callback + + def __call__(self, *outcome: object) -> bool: + """Run the callback if there is one and its object still exists; return whether it ran.""" + callback = self._weak() if self._weak is not None else self._strong + if not callable(callback): + return False + callback(*self._args, *outcome, **self._keywords) + return True + + +def _run(call: WeakCall, *outcome: object) -> None: + """GUI thread: deliver one outcome to whoever is still there.""" + call(*outcome) + + +def weak_slot(callback: Callback, *args: Any) -> Callable[..., None]: + """Return what to connect a task's signal to so that ``callback(*args, value)`` runs weakly. + + The result is a partial of a module-level function: it holds the + :class:`WeakCall`, never the object ``callback`` is bound to. + """ + return functools.partial(_run, WeakCall(callback, args)) + + +__all__ = ["Callback", "WeakCall", "weak_slot"] diff --git a/je_auto_control/gui/_worker_thread.py b/je_auto_control/gui/_worker_thread.py index 1e2d70a3..3de5908b 100644 --- a/je_auto_control/gui/_worker_thread.py +++ b/je_auto_control/gui/_worker_thread.py @@ -37,6 +37,7 @@ from PySide6.QtCore import QObject, Signal +from je_auto_control.gui._weak_call import WeakCall from je_auto_control.utils.logging.logging_instance import autocontrol_logger #: How long interpreter exit waits, in all, for running workers to stop. @@ -116,7 +117,13 @@ def _reaper() -> _Reaper: class _Relay(QObject): - """GUI-thread receiver for a worker's outcome, owned by the tab.""" + """GUI-thread receiver for a worker's outcome, owned by the tab. + + The callbacks are normally methods of the tab this relay is a child of, + so they are held weakly (:mod:`je_auto_control.gui._weak_call`): held + strongly, the relay could be the last holder of a parentless tab and + destroy it from inside its own destructor. + """ thread_ended = Signal() crashed = Signal(str) @@ -126,9 +133,9 @@ def __init__(self, parent: QObject, on_thread_done: Callable[[], None], on_fail: Optional[Callable[[str], None]]) -> None: super().__init__(parent) - self._on_done = on_done - self._on_thread_done = on_thread_done - self._on_fail = on_fail + self._on_done = WeakCall(on_done) + self._on_thread_done = WeakCall(on_thread_done) + self._on_fail = WeakCall(on_fail) self.thread_ended.connect(self.thread_done) self.crashed.connect(self.fail) @@ -138,8 +145,7 @@ def done(self, value: Any) -> None: def fail(self, message: str) -> None: """Forward the worker's failure (runs on the GUI thread).""" - if self._on_fail is not None: - self._on_fail(message) + self._on_fail(message) def thread_done(self) -> None: """Forward the thread's end (runs on the GUI thread), then go away.""" @@ -204,8 +210,10 @@ def start_worker(owner: QObject, worker: QObject, *, Call from the GUI thread. ``worker`` must have a ``finished`` signal and may have ``failed``. ``on_done`` / ``on_fail`` / ``on_thread_done`` run on the GUI thread, and only while ``owner`` exists: destroying ``owner`` - mid-run drops them and leaves the work to finish on its own. The worker is - deleted once the GUI thread has seen its thread end. + mid-run drops them and leaves the work to finish on its own. A bound + method (or a ``functools.partial`` of one) is held weakly, so pass those + rather than a lambda that closes over ``owner``. The worker is deleted + once the GUI thread has seen its thread end. """ reaper = _reaper() relay = _Relay(owner, on_done, on_thread_done, on_fail) diff --git a/je_auto_control/gui/remote_desktop/_connect_task.py b/je_auto_control/gui/remote_desktop/_connect_task.py index 7dadcbdd..fee143d6 100644 --- a/je_auto_control/gui/remote_desktop/_connect_task.py +++ b/je_auto_control/gui/remote_desktop/_connect_task.py @@ -12,6 +12,7 @@ from PySide6.QtCore import QObject +from je_auto_control.gui._weak_call import weak_slot from je_auto_control.gui.task_controller import CancellationToken, TaskHandle, task_controller from je_auto_control.utils.logging.logging_instance import autocontrol_logger @@ -44,13 +45,15 @@ def connect_viewer(owner: QObject, viewer: Any, *, """Run ``viewer.connect`` off the GUI thread for ``owner``; return the task. ``on_connected(viewer)`` or ``on_failed(error)`` runs on the GUI thread, - and only while ``owner`` exists. Cancelling the task, or destroying - ``owner``, disconnects a viewer that connects anyway. + and only while ``owner`` exists; both are held weakly, so pass methods of + the panel (or a ``functools.partial`` of one), never a closure over it. + Cancelling the task, or destroying ``owner``, disconnects a viewer that + connects anyway. """ handle = task_controller().submit(functools.partial(_connect, viewer, timeout_s), owner=owner, discard=_disconnect_quietly) - handle.result.connect(on_connected) - handle.error.connect(on_failed) + handle.result.connect(weak_slot(on_connected)) + handle.error.connect(weak_slot(on_failed)) return handle diff --git a/je_auto_control/gui/remote_desktop/connection_screen.py b/je_auto_control/gui/remote_desktop/connection_screen.py index c7ab9216..ed211d76 100644 --- a/je_auto_control/gui/remote_desktop/connection_screen.py +++ b/je_auto_control/gui/remote_desktop/connection_screen.py @@ -29,6 +29,7 @@ ) from je_auto_control.gui._i18n_helpers import TranslatableMixin +from je_auto_control.gui._weak_call import WeakCall from je_auto_control.gui.remote_desktop._connect_task import connect_viewer from je_auto_control.gui.remote_desktop.connection_recent import _RecentConnectionsMixin from je_auto_control.gui.remote_desktop._helpers import ( @@ -451,7 +452,10 @@ def _viewer_callbacks(self) -> dict: "on_cursor": self._cursor_moved.emit} def _begin_connect(self, slot: str, build, on_live) -> None: - """Take ``slot``, build the viewer, and connect it off the GUI thread (see ``_connect_task``).""" + """Take ``slot``, build the viewer, and connect it off the GUI thread (see ``_connect_task``). + + ``on_live`` is a method of this screen or a partial of one; it is held weakly. + """ self._cancel_pending_connect() self._take_slot(slot) try: @@ -459,15 +463,17 @@ def _begin_connect(self, slot: str, build, on_live) -> None: except (OSError, RuntimeError, ValueError, AutoControlException) as error: self._on_connect_failed(error) return - - def connected(live_viewer) -> None: - registry.adopt(slot, live_viewer, self._owner, displaced_notifier(self)) - on_live() self._connect_task = connect_viewer( - self, viewer, on_connected=connected, on_failed=self._on_connect_failed) + self, viewer, on_connected=functools.partial(self._on_viewer_live, slot, WeakCall(on_live)), + on_failed=self._on_connect_failed) self._connect_task.finished.connect(self._on_connect_finished) self._refresh_status() + def _on_viewer_live(self, slot: str, on_live: WeakCall, live_viewer) -> None: + """GUI thread: the connect succeeded; register the viewer and show it.""" + registry.adopt(slot, live_viewer, self._owner, displaced_notifier(self)) + on_live() + def _cancel_pending_connect(self) -> None: task, self._connect_task = self._connect_task, None if task is not None: @@ -483,12 +489,17 @@ def _on_connect_failed(self, error: Exception) -> None: QMessageBox.warning(self, _t("rd_quick_connect_btn"), str(error)) def _do_tcp_connect(self, host: str, port: int, token: str) -> None: - def live() -> None: - self._remember_tcp(host, port) - self._open_screen_window(f"{host}:{port}") - self._begin_connect(SLOT_VIEWER, lambda: RemoteDesktopViewer( - host=host, port=port, token=token, **self._viewer_callbacks()), live) + host=host, port=port, token=token, **self._viewer_callbacks()), + functools.partial(self._tcp_live, host, port)) + + def _tcp_live(self, host: str, port: int) -> None: + self._remember_tcp(host, port) + self._open_screen_window(f"{host}:{port}") + + def _ws_live(self, base: str, path: str) -> None: + self._remember_url(f"{base}{path}") + self._open_screen_window(base) def _do_ws_connect(self, target: ConnectTarget, token: str) -> None: host, port, path = target.host or "", target.port or 0, target.path or "/" @@ -496,14 +507,9 @@ def _do_ws_connect(self, target: ConnectTarget, token: str) -> None: # wss:// was dialled as plain ws://: the session went unencrypted to # a host the operator took for TLS, and a real TLS host was unreachable. ssl_context = _build_verifying_client_context() if scheme == "wss" else None - - def live() -> None: - self._remember_url(f"{scheme}://{host}:{port}{path}") - self._open_screen_window(f"{scheme}://{host}:{port}") - self._begin_connect(SLOT_WS_VIEWER, lambda: WebSocketDesktopViewer( host=host, port=port, token=token, path=path, ssl_context=ssl_context, - **self._viewer_callbacks()), live) + **self._viewer_callbacks()), functools.partial(self._ws_live, f"{scheme}://{host}:{port}", path)) def _on_remote_cursor(self, x: int, y: int) -> None: """Cursor update, delivered on the GUI thread by ``_cursor_moved``.""" diff --git a/je_auto_control/gui/remote_desktop/webrtc_host_connection.py b/je_auto_control/gui/remote_desktop/webrtc_host_connection.py index c00d2728..8b73b2aa 100644 --- a/je_auto_control/gui/remote_desktop/webrtc_host_connection.py +++ b/je_auto_control/gui/remote_desktop/webrtc_host_connection.py @@ -21,6 +21,7 @@ HostPublishLoopWorker, generate_host_id, retire_worker, ) from je_auto_control.gui._slow_op import stop_each +from je_auto_control.gui._weak_call import weak_slot from je_auto_control.gui.task_controller import CancellationToken, task_controller from je_auto_control.utils.logging.logging_instance import autocontrol_logger from je_auto_control.utils.remote_desktop import ( @@ -159,8 +160,9 @@ def _produce_offer(self) -> None: # used to freeze the window; the backend takes no timeout or cancel. task = task_controller().submit(functools.partial(_create_offer, host), owner=self, discard=functools.partial(_drop_offer, host)) - task.result.connect(functools.partial(self._show_offer, host)) - task.error.connect(functools.partial(self._show_offer_error, host)) + # Weak: the task is this panel's child and must not be what keeps it alive. + task.result.connect(weak_slot(self._show_offer, host)) + task.error.connect(weak_slot(self._show_offer_error, host)) def _show_offer_error(self, host: MultiViewerHostT, error: Exception) -> None: if self._multi_host is host: # a host stopped meanwhile fails by design diff --git a/je_auto_control/gui/remote_desktop/webrtc_panel_common.py b/je_auto_control/gui/remote_desktop/webrtc_panel_common.py index 772d8a28..77a295fb 100644 --- a/je_auto_control/gui/remote_desktop/webrtc_panel_common.py +++ b/je_auto_control/gui/remote_desktop/webrtc_panel_common.py @@ -1,7 +1,7 @@ """Signals, defaults and config readers shared by the WebRTC host and viewer panels.""" from __future__ import annotations -import functools +import weakref from typing import Any, Callable, Optional, Tuple from PySide6.QtCore import QObject, Signal @@ -11,8 +11,9 @@ from je_auto_control.gui.remote_desktop._webrtc_types import AvFrameT, WebRTCConfigT from je_auto_control.gui._i18n_helpers import TranslatableMixin from je_auto_control.gui._slow_op import StopQueue +from je_auto_control.gui._weak_call import weak_slot from je_auto_control.gui.remote_desktop._helpers import _t -from je_auto_control.gui.task_controller import TaskHandle, task_controller +from je_auto_control.gui.task_controller import task_controller from je_auto_control.utils.logging.logging_instance import autocontrol_logger from je_auto_control.utils.remote_desktop import ( @@ -95,20 +96,24 @@ def start_panel_task(panel: QWidget, attribute: str, work: Callable[..., object] Returns ``False`` (and starts nothing) while the previous task under the same attribute has not ended. The attribute is ``None`` again once the - outcome has been delivered, which is what a test waits for. + outcome has been delivered, which is what a test waits for. The callbacks + are held weakly (a method of the panel, or a ``functools.partial`` of + one): the task is the panel's child and must not be what keeps it alive. """ if getattr(panel, attribute, None) is not None: return False task = task_controller().submit(work, owner=panel) setattr(panel, attribute, task) - task.result.connect(on_result) - task.error.connect(on_error) - task.finished.connect(functools.partial(_clear_panel_task, panel, attribute, task)) + task.result.connect(weak_slot(on_result)) + task.error.connect(weak_slot(on_error)) + task.finished.connect(weak_slot(_clear_panel_task, weakref.ref(panel), attribute, weakref.ref(task))) return True -def _clear_panel_task(panel: QWidget, attribute: str, task: TaskHandle) -> None: - if getattr(panel, attribute, None) is task: +def _clear_panel_task(panel_ref: Callable[[], Optional[QWidget]], attribute: str, + task_ref: Callable[[], Optional[QObject]]) -> None: + panel, task = panel_ref(), task_ref() + if panel is not None and task is not None and getattr(panel, attribute, None) is task: setattr(panel, attribute, None) diff --git a/je_auto_control/gui/remote_desktop/webrtc_viewer_connection.py b/je_auto_control/gui/remote_desktop/webrtc_viewer_connection.py index db70a346..0d55db6c 100644 --- a/je_auto_control/gui/remote_desktop/webrtc_viewer_connection.py +++ b/je_auto_control/gui/remote_desktop/webrtc_viewer_connection.py @@ -22,6 +22,7 @@ retire_worker, ) from je_auto_control.gui._slow_op import stop_each +from je_auto_control.gui._weak_call import WeakCall, weak_slot from je_auto_control.gui.task_controller import CancellationToken, task_controller from je_auto_control.utils.logging.logging_instance import autocontrol_logger from je_auto_control.utils.remote_desktop import ( @@ -86,25 +87,29 @@ def _answer_and_push(self, offer_sdp: str) -> None: def _answer_off_thread(self, offer_sdp: str, expected_dtls: Optional[str], on_answer: Callable[[str], None]) -> None: - """Run ``process_offer`` on a worker; ``on_answer(answer)`` only if this viewer is still current.""" + """Run ``process_offer`` on a worker; ``on_answer(answer)`` only if this viewer is still current. + + ``on_answer`` is a method of this panel or a ``functools.partial`` of + one: it is held weakly, as is the delivery itself, because the task is + this panel's child and must not be what keeps the panel alive. + """ try: viewer = self._require_viewer() except RuntimeError as error: self._show_error(error) return - - def deliver(answer: str) -> None: - if self._viewer is viewer: # not stopped or replaced while the answer was made - on_answer(answer) - - def fail(error: Exception) -> None: - if self._viewer is viewer: # a viewer stopped meanwhile fails by design - self._show_error(error) - task = task_controller().submit( functools.partial(_process_offer, viewer, offer_sdp, expected_dtls), owner=self) - task.result.connect(deliver) - task.error.connect(fail) + task.result.connect(weak_slot(self._deliver_answer, viewer, WeakCall(on_answer))) + task.error.connect(weak_slot(self._answer_failed, viewer)) + + def _deliver_answer(self, viewer: WebRTCDesktopViewerT, on_answer: WeakCall, answer: str) -> None: + if self._viewer is viewer: # not stopped or replaced while the answer was made + on_answer(answer) + + def _answer_failed(self, viewer: WebRTCDesktopViewerT, error: Exception) -> None: + if self._viewer is viewer: # a viewer stopped meanwhile fails by design + self._show_error(error) def _push_answer(self, host_id: str, expected_dtls: Optional[str], offer_sdp: str, answer: str) -> None: # First-time TOFU: stash the DTLS fingerprint we just observed diff --git a/je_auto_control/gui/task_controller.py b/je_auto_control/gui/task_controller.py index 049732ac..aacbf0f6 100644 --- a/je_auto_control/gui/task_controller.py +++ b/je_auto_control/gui/task_controller.py @@ -16,8 +16,12 @@ * **Typed outcomes.** :attr:`TaskHandle.result` carries the return value, :attr:`TaskHandle.error` the exception object (not its text) and :attr:`TaskHandle.progress` whatever the work reports. All three are emitted - on the GUI thread, so a lambda connected to them is safe. -* **Owner death.** The handle is a child of ``owner``. Destroying the owner + on the GUI thread. Connect them to a method of the owner, or through + :func:`je_auto_control.gui._weak_call.weak_slot` when it needs arguments: + the handle is the owner's child, and a lambda or a ``functools.partial`` + that holds the owner would make the handle what keeps the owner alive. +* **Owner death.** The handle is a child of ``owner``, and the controller + itself only holds the owner weakly. Destroying the owner cancels the work; a result that is ready but not delivered yet, or arrives later, goes to ``discard`` (close the session nobody will use) and never to a slot of the dead widget. @@ -28,6 +32,7 @@ import functools import threading import time +import weakref from typing import Any, Callable, Dict, List, Optional, Tuple from PySide6.QtCore import QObject, QTimer, Signal @@ -371,7 +376,8 @@ class TaskController: """ def __init__(self) -> None: - self._active: Dict[TaskHandle, Tuple[QObject, Callable[..., None]]] = {} + # The owner is held weakly: a running task must not keep its tab alive. + self._active: Dict[TaskHandle, Tuple["weakref.ref[QObject]", Callable[..., None]]] = {} def submit(self, work: Work, *, owner: QObject, timeout_s: Optional[float] = None, discard: Optional[Callable[[Any], object]] = None) -> TaskHandle: @@ -400,13 +406,14 @@ def owner_destroyed(*_args: object) -> None: # A plain function, not a slot of the handle: the handle dies with the # owner, and this has to run exactly then. owner.destroyed.connect(owner_destroyed) - self._active[handle] = (owner, owner_destroyed) + self._active[handle] = (weakref.ref(owner), owner_destroyed) handle._start(worker, timeout_s, self._forget) # noqa: SLF001 # reason: the handle's second-phase start return handle def _forget(self, handle: TaskHandle) -> None: """GUI thread: ``handle``'s thread ended while its owner is still alive.""" - owner, owner_destroyed = self._active.pop(handle, (None, None)) + owner_ref, owner_destroyed = self._active.pop(handle, (None, None)) + owner = owner_ref() if owner_ref is not None else None if owner is None: return try: @@ -421,7 +428,7 @@ def cancel_all(self, owner: Optional[QObject] = None) -> int: still owns its tasks. """ handles = [handle for handle, (task_owner, _hook) in self._active.items() - if owner is None or task_owner is owner] + if owner is None or task_owner() is owner] return sum(1 for handle in handles if handle.cancel()) def active_count(self) -> int: diff --git a/je_auto_control/gui/usb_browser_tab.py b/je_auto_control/gui/usb_browser_tab.py index f92b5599..93753fac 100644 --- a/je_auto_control/gui/usb_browser_tab.py +++ b/je_auto_control/gui/usb_browser_tab.py @@ -16,6 +16,7 @@ """ from __future__ import annotations +import functools import urllib.error import urllib.parse from email.message import Message @@ -252,12 +253,12 @@ def _on_open_selected(self) -> None: def _start_local_open(self, vid: str, pid: str, serial: Optional[str]) -> None: - # The lambda below runs on the GUI thread: start_worker relays it. + # on_done runs on the GUI thread (start_worker relays it) and is held weakly. self._open_thread = start_worker( self, _CallWorker(lambda: open_local_descriptor( vendor_id=vid, product_id=pid, serial=serial, )), - on_done=lambda descriptor: self._on_local_opened(vid, pid, descriptor), + on_done=functools.partial(self._on_local_opened, vid, pid), on_fail=self._apply_failure, on_thread_done=self._on_open_done) def _on_open_done(self) -> None: diff --git a/je_auto_control/gui/usb_passthrough_panel.py b/je_auto_control/gui/usb_passthrough_panel.py index 117fe92f..2f3b2801 100644 --- a/je_auto_control/gui/usb_passthrough_panel.py +++ b/je_auto_control/gui/usb_passthrough_panel.py @@ -451,7 +451,7 @@ def _open_selected(self) -> None: ) self._run_async( lambda: _probe_device(client, vid, pid, serial), - lambda descriptor: self._opened(vid, pid, descriptor), + functools.partial(self._opened, vid, pid), self._fail, ) @@ -490,8 +490,9 @@ def _run_async(self, fn: Callable[[], Any], on_fail: Callable[[str], None]) -> None: if self._thread is not None: return - # on_done / on_fail are often lambdas; start_worker runs them on the - # GUI thread, where they may touch widgets. + # start_worker runs on_done / on_fail on the GUI thread, where they may + # touch widgets, and holds them weakly: pass methods (or partials of + # them), not lambdas over this panel. self._thread = start_worker(self, _CallWorker(fn), on_done=on_done, on_fail=on_fail, on_thread_done=self._on_thread_done) diff --git a/je_auto_control/gui/vlm_tab.py b/je_auto_control/gui/vlm_tab.py index 463e9a0f..049ca128 100644 --- a/je_auto_control/gui/vlm_tab.py +++ b/je_auto_control/gui/vlm_tab.py @@ -110,4 +110,7 @@ def _on_click(self) -> None: description, model = inputs self._run_off_gui_thread( lambda: click_by_description(description, model=model), - lambda ok: self._status.setText(_t("vlm_ok") if ok else _t("vlm_not_found"))) + self._show_clicked) + + def _show_clicked(self, ok) -> None: + self._status.setText(_t("vlm_ok") if ok else _t("vlm_not_found")) diff --git a/pyproject.toml b/pyproject.toml index c74e1c8d..a39df028 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -443,6 +443,7 @@ module = [ "je_auto_control.utils.executor.run_control", "je_auto_control.gui._dispose", "je_auto_control.gui._slow_op", + "je_auto_control.gui._weak_call", "je_auto_control.utils.codegen.candidate_diff", "je_auto_control.utils.remote_desktop.input_send_queue", "je_auto_control_pytest", diff --git a/test/unit_test/headless/test_gui_weak_callbacks.py b/test/unit_test/headless/test_gui_weak_callbacks.py new file mode 100644 index 00000000..cbf521c1 --- /dev/null +++ b/test/unit_test/headless/test_gui_weak_callbacks.py @@ -0,0 +1,388 @@ +"""A background task's callbacks do not keep the tab that started it alive (offscreen Qt, fakes only). + +A task handle and the relay under it are descendants of the tab. A callback +held strongly by one of them -- ``functools.partial(self.x, ...)``, a lambda or +a nested function -- made the handle the last holder of a parentless tab, which +was then destroyed from inside the destructor of its own grandchild: an abort +with no traceback. The behaviour tests drop the only reference while the work +is still out and expect the tab to be freed at once; the source rule fails a +new strong connection. +""" +import ast +import functools +import gc +import os +import pathlib +import threading +import weakref + +import pytest + +os.environ.setdefault("QT_QPA_PLATFORM", "offscreen") +pytest.importorskip("PySide6.QtWidgets", exc_type=ImportError) + +from PySide6.QtWidgets import QApplication, QMessageBox, QWidget # noqa: E402 + +from je_auto_control.gui import _weak_call # noqa: E402 +from je_auto_control.gui._weak_call import WeakCall, weak_slot # noqa: E402 +from je_auto_control.gui._worker_thread import CallWorker, start_worker # noqa: E402 +from headless._qt_settle import pump_until # noqa: E402 + +_GUI_DIR = pathlib.Path(_weak_call.__file__).resolve().parent + + +@pytest.fixture(autouse=True) +def qapp(monkeypatch): + app = QApplication.instance() or QApplication([]) + for box in ("warning", "information", "question", "critical"): + monkeypatch.setattr(QMessageBox, box, lambda *args: None) + yield app + + +class _Gate: + """Work that waits on the worker thread until the test lets it go.""" + + def __init__(self, value=None): + self.entered, self.release, self.value = threading.Event(), threading.Event(), value + + def __call__(self, *_args): + self.entered.set() + self.release.wait(30.0) + return self.value + + +def _freed_while_out(start): + """Drop the owner ``start()`` returns while its gate still runs; return whether that freed it.""" + owner, gate = start() + seen = weakref.ref(owner) + del owner + gc.collect() + freed = seen() is None + gate.release.set() + assert gate.entered.wait(10.0) + pump_until(lambda: False, timeout=0.2) # the outcome has nobody to go to, and nothing crashes + return freed + + +# --- the helper -------------------------------------------------------------------------------------------------- + +class _Target: + def __init__(self): + self.seen = [] + + def take(self, *args, **kwargs): + self.seen.append((args, kwargs)) + + +def test_a_bound_method_is_held_weakly_with_its_leading_arguments(): + target = _Target() + call = WeakCall(target.take, ("a",)) + assert call("value") is True and target.seen == [(("a", "value"), {})] + del target + gc.collect() + assert call("late") is False + + +def test_a_partial_of_a_bound_method_is_taken_apart(): + target = _Target() + call = WeakCall(functools.partial(functools.partial(target.take, 1, key="k"), 2), (3,)) + seen = weakref.ref(target) + assert call(4) is True and target.seen == [((1, 2, 3, 4), {"key": "k"})] + del target + gc.collect() + assert seen() is None and call(5) is False + + +def test_a_plain_function_and_none_are_accepted(): + seen = [] + assert WeakCall(seen.append)("x") is True and seen == ["x"] + assert WeakCall(None)("x") is False + slot = weak_slot(seen.append) + slot("y") + assert seen == ["x", "y"] + + +# --- the places that deliver a task's outcome -------------------------------------------------------------------- + +class _Tab(QWidget): + def __init__(self): + super().__init__() + self.outcomes = [] + + def done(self, *args): + self.outcomes.append(args) + + def ended(self): + self.outcomes.append("ended") + + +def test_a_worker_does_not_keep_its_tab_alive(qapp): + def start(): + tab, gate = _Tab(), _Gate() + start_worker(tab, CallWorker(gate), on_done=functools.partial(tab.done, "tag"), + on_fail=tab.done, on_thread_done=tab.ended) + return tab, gate + + assert _freed_while_out(start) + + +def test_a_worker_still_delivers_to_a_tab_that_is_there(qapp): + tab, gate = _Tab(), _Gate("value") + start_worker(tab, CallWorker(gate), on_done=functools.partial(tab.done, "tag"), + on_fail=tab.done, on_thread_done=tab.ended) + gate.release.set() + assert pump_until(lambda: "ended" in tab.outcomes) + assert tab.outcomes == [("tag", "value"), "ended"] + tab.deleteLater() + + +def test_a_panel_task_does_not_keep_its_panel_alive(qapp): + from je_auto_control.gui.remote_desktop.webrtc_panel_common import start_panel_task + + def start(): + panel, gate = _Tab(), _Gate() + panel._file_task = None + assert start_panel_task(panel, "_file_task", gate, functools.partial(panel.done, "path"), panel.done) + return panel, gate + + assert _freed_while_out(start) + + +def test_a_panel_task_clears_its_attribute_and_delivers(qapp): + from je_auto_control.gui.remote_desktop.webrtc_panel_common import start_panel_task + panel, gate = _Tab(), _Gate(3) + panel._file_task = None + assert start_panel_task(panel, "_file_task", gate, functools.partial(panel.done, "path"), panel.done) + assert not start_panel_task(panel, "_file_task", gate, panel.done, panel.done) # one at a time + gate.release.set() + assert pump_until(lambda: panel._file_task is None) + assert panel.outcomes == [("path", 3)] + panel.deleteLater() + + +class _Viewer: + def __init__(self, gate): + self._gate = gate + + def connect(self, timeout=None): + self._gate() + + def disconnect(self): + """Nothing to close.""" + + +def test_a_pending_connect_does_not_keep_its_panel_alive(qapp): + from je_auto_control.gui.remote_desktop._connect_task import connect_viewer + + def start(): + panel, gate = _Tab(), _Gate() + connect_viewer(panel, _Viewer(gate), on_connected=functools.partial(panel.done, "slot"), + on_failed=panel.done) + return panel, gate + + assert _freed_while_out(start) + + +def _host_panel_part(gate): + pytest.importorskip("aiortc", exc_type=ImportError) + from je_auto_control.gui.remote_desktop.webrtc_host_connection import _HostConnectionMixin + + class _FakeHost: + def create_session_offer(self): + return gate() + + def stop_session(self, _session_id): + """Nothing to stop.""" + + class _Part(_HostConnectionMixin): + def __init__(self): + super().__init__() + self._multi_host = _FakeHost() + self.shown = [] + + def _show_offer(self, host, outcome): + self.shown.append((host, outcome)) + + return _Part() + + +def test_an_offer_being_made_does_not_keep_the_host_panel_alive(qapp): + def start(): + gate = _Gate(("session", "offer")) + part = _host_panel_part(gate) + part._produce_offer() + return part, gate + + assert _freed_while_out(start) + + +def test_an_offer_still_reaches_a_host_panel_that_is_there(qapp): + gate = _Gate(("session", "offer")) + part = _host_panel_part(gate) + part._produce_offer() + gate.release.set() + assert pump_until(lambda: bool(part.shown)) + assert part.shown == [(part._multi_host, ("session", "offer"))] + part.deleteLater() + + +def _viewer_panel_part(gate): + pytest.importorskip("aiortc", exc_type=ImportError) + from je_auto_control.gui.remote_desktop.webrtc_viewer_connection import _ViewerConnectionMixin + + class _FakeViewer: + def process_offer(self, _offer, expected_dtls_fingerprint=None): + return gate() + + class _Part(_ViewerConnectionMixin): + def __init__(self): + super().__init__() + self._viewer = _FakeViewer() + self.answers = [] + + def _show_manual_answer(self, answer): + self.answers.append(answer) + + return _Part() + + +def test_an_answer_being_made_does_not_keep_the_viewer_panel_alive(qapp): + def start(): + gate = _Gate("answer") + part = _viewer_panel_part(gate) + part._produce_answer("offer") + return part, gate + + assert _freed_while_out(start) + + +def test_an_answer_reaches_the_viewer_panel_only_while_its_viewer_is_current(qapp): + gate = _Gate("answer") + part = _viewer_panel_part(gate) + part._produce_answer("offer") + gate.release.set() + assert pump_until(lambda: bool(part.answers)) + assert part.answers == ["answer"] + late = _Gate("late") + part._viewer = type(part._viewer)() + part._viewer.process_offer = lambda _offer, expected_dtls_fingerprint=None: late() + part._produce_answer("offer") + part._viewer = None # stopped while the answer was being made + late.release.set() + pump_until(lambda: False, timeout=0.3) + assert part.answers == ["answer"] + part.deleteLater() + + +# --- no new strong connection ------------------------------------------------------------------------------------ + +#: Signals of a task handle. A lambda on a signal of this name is refused whatever emits it. +_TASK_SIGNALS = {"result", "error", "finished", "progress"} +#: Helpers that deliver a task's outcome, and where their callbacks sit (keywords, first positional index). +_DELIVERERS = { + "start_worker": ({"on_done", "on_fail", "on_thread_done"}, None), + "connect_viewer": ({"on_connected", "on_failed"}, None), + "start_panel_task": (set(), 3), + "_run_async": (set(), 1), + "_run_off_gui_thread": (set(), 1), + "run": ({"on_done", "on_error"}, None), + "retire": ({"on_done"}, None), +} + + +def _is_self_partial(node): + """``functools.partial(self.x, ...)`` / ``partial(self.x, ...)``.""" + if not (isinstance(node, ast.Call) and node.args): + return False + name = node.func.attr if isinstance(node.func, ast.Attribute) else getattr(node.func, "id", "") + first = node.args[0] + while isinstance(first, ast.Attribute): + first = first.value + return name == "partial" and isinstance(first, ast.Name) and first.id == "self" + + +def _callee(node): + return node.func.attr if isinstance(node.func, ast.Attribute) else getattr(node.func, "id", "") + + +class _Finder(ast.NodeVisitor): + """Collects strong callbacks handed to a task's signal or to a helper that delivers its outcome.""" + + def __init__(self, name): + self.name, self.found, self._nested = name, [], [set()] + + def visit_FunctionDef(self, node): # noqa: N802 # reason: ast.NodeVisitor's naming + nested = {child.name for child in ast.walk(node) + if isinstance(child, (ast.FunctionDef, ast.AsyncFunctionDef)) and child is not node} + self._nested.append(nested) + self.generic_visit(node) + self._nested.pop() + + def _strong(self, node, in_connect): + if isinstance(node, ast.Lambda): + return "a lambda" + if isinstance(node, ast.Name) and node.id in self._nested[-1]: + return f"the nested function {node.id}" + if in_connect and _is_self_partial(node): + return "a partial of a method of self" + return "" + + def _report(self, node, argument, in_connect): + why = self._strong(argument, in_connect) + if why: + self.found.append(f"{self.name}:{node.lineno}: {why}") + + def visit_Call(self, node): # noqa: N802 # reason: ast.NodeVisitor's naming + func = node.func + if (isinstance(func, ast.Attribute) and func.attr == "connect" and isinstance(func.value, ast.Attribute) + and func.value.attr in _TASK_SIGNALS): + for argument in node.args: + self._report(node, argument, True) + keywords, first = _DELIVERERS.get(_callee(node), (set(), None)) + for keyword in node.keywords: + if keyword.arg in keywords: + self._report(node, keyword.value, False) + if first is not None: + for argument in node.args[first:]: + self._report(node, argument, False) + self.generic_visit(node) + + +def _strong_callbacks(path): + finder = _Finder(path.relative_to(_GUI_DIR).as_posix()) + finder.visit(ast.parse(path.read_text(encoding="utf-8"))) + return finder.found + + +def test_no_task_outcome_goes_to_a_callback_that_holds_the_tab(): + files = sorted(_GUI_DIR.rglob("*.py")) + assert len(files) > 100 + found = [hit for path in files for hit in _strong_callbacks(path)] + assert found == [], ("pass a method (and its arguments) through weak_slot / the helper instead; " + f"see gui/_weak_call.py: {found}") + + +def test_the_rule_sees_each_strong_form_and_accepts_the_weak_one(tmp_path): + sample = tmp_path / "sample_tab.py" + sample.write_text( + "import functools\n" + "class Tab:\n" + " def go(self):\n" + " def deliver(value):\n" + " self.show(value)\n" + " task = submit()\n" + " task.result.connect(deliver)\n" + " task.error.connect(lambda error: self.show(error))\n" + " task.finished.connect(functools.partial(self.show, 1))\n" + " task.result.connect(weak_slot(self.show, 1))\n" + " task.finished.connect(self.show)\n" + " start_worker(self, worker, on_done=lambda value: self.show(value), on_thread_done=self.show)\n" + " start_worker(self, worker, on_done=functools.partial(self.show, 1), on_thread_done=self.show)\n" + " start_panel_task(self, '_task', work, deliver, self.show)\n" + " self._run_async(lambda: fetch(), lambda value: self.show(value), self.show)\n", + encoding="utf-8") + finder = _Finder("sample_tab.py") + finder.visit(ast.parse(sample.read_text(encoding="utf-8"))) + assert [hit.split(":", 1)[1] for hit in finder.found] == [ + "7: the nested function deliver", "8: a lambda", "9: a partial of a method of self", + "12: a lambda", "14: the nested function deliver", "15: a lambda"] From c15a23f532e8ab5712cd307a63578965c30f44cc Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 07:06:33 +0800 Subject: [PATCH 03/17] Register and unregister the WebRTC host's LAN advertiser off the GUI thread Unregistering sends zeroconf's goodbye packets and joins its thread, and registering probes the name first; both ran in the Publish and Stop slots. The stats pollers' stop stays where it is: it only schedules a cancel on the bridge's loop, which is now written down and pinned by a test. --- .../remote_desktop/webrtc_host_connection.py | 68 +++++-- .../gui/remote_desktop/webrtc_host_media.py | 2 + .../gui/remote_desktop/webrtc_viewer_media.py | 1 + .../utils/remote_desktop/webrtc_stats.py | 7 + .../test_webrtc_lan_advertise_stop.py | 178 ++++++++++++++++++ 5 files changed, 238 insertions(+), 18 deletions(-) create mode 100644 test/unit_test/headless/test_webrtc_lan_advertise_stop.py diff --git a/je_auto_control/gui/remote_desktop/webrtc_host_connection.py b/je_auto_control/gui/remote_desktop/webrtc_host_connection.py index 8b73b2aa..e4005ebf 100644 --- a/je_auto_control/gui/remote_desktop/webrtc_host_connection.py +++ b/je_auto_control/gui/remote_desktop/webrtc_host_connection.py @@ -6,6 +6,7 @@ from __future__ import annotations import functools +import threading from typing import TYPE_CHECKING, Optional, Tuple from PySide6.QtCore import QTimer @@ -50,6 +51,32 @@ def _drop_offer(host: MultiViewerHostT, outcome: Tuple[str, str]) -> None: autocontrol_logger.debug("dropping an unused offer: %r", error) +#: One zeroconf registration change at a time: Publish stops the old advertiser and starts the +#: next under the same name, and the name is only free once the goodbye packets have gone out. +_ADVERTISE_LOCK = threading.Lock() + + +def _start_advertiser(host_id: str, signaling_url: str, _token: CancellationToken) -> HostAdvertiser: + """Worker thread: register the host on the LAN (zeroconf probes the name first, about a second).""" + from je_auto_control.utils.remote_desktop.lan_discovery import HostAdvertiser + with _ADVERTISE_LOCK: + return HostAdvertiser(host_id=host_id, signaling_url=signaling_url) + + +def _stop_advertiser(advertiser: HostAdvertiser) -> None: + """Worker thread: unregister (goodbye packets) and close, which joins zeroconf's thread.""" + with _ADVERTISE_LOCK: + try: + advertiser.stop() + except (RuntimeError, OSError) as error: + autocontrol_logger.debug("lan advertise stop: %r", error) + + +def _advertise_failed(error: object) -> None: + """GUI thread: LAN discovery is a convenience; a host that cannot advertise still publishes.""" + autocontrol_logger.debug("lan advertise: %r", error) + + class _HostConnectionMixin(_PanelPart): """Methods of ``_WebRTCHostPanel``; the module docstring says which group.""" @@ -58,6 +85,8 @@ class _HostConnectionMixin(_PanelPart): _publish_loop: Optional[HostPublishLoopWorker] _manual_session_id: Optional[str] _lan_advertiser: Optional[HostAdvertiser] + #: Bumped by every start and stop, so an advertiser that reports late knows it is not wanted. + _advertise_generation = 0 def _on_regen_id(self) -> None: self._host_id_edit.setText(generate_host_id()) @@ -88,31 +117,34 @@ def _on_publish_via_server(self) -> None: self._start_lan_advertise() def _start_lan_advertise(self) -> None: + """Advertise this host on the LAN, off the GUI thread; the old advertiser is stopped first.""" try: - from je_auto_control.utils.remote_desktop.lan_discovery import ( - HostAdvertiser, is_discovery_available, - ) + from je_auto_control.utils.remote_desktop.lan_discovery import is_discovery_available except ImportError: return if not is_discovery_available(): return - try: - if self._lan_advertiser is not None: - self._lan_advertiser.stop() - self._lan_advertiser = HostAdvertiser( - host_id=self._host_id_edit.text().strip(), - signaling_url=self._server_edit.text().strip(), - ) - except (RuntimeError, OSError) as error: - autocontrol_logger.debug("lan advertise: %r", error) + self._stop_lan_advertise() + task = task_controller().submit( + functools.partial(_start_advertiser, self._host_id_edit.text().strip(), + self._server_edit.text().strip()), + owner=self, discard=_stop_advertiser) + task.result.connect(weak_slot(self._advertiser_started, self._advertise_generation)) + task.error.connect(_advertise_failed) + + def _advertiser_started(self, generation: int, advertiser: HostAdvertiser) -> None: + """GUI thread: the registration is up -- unless Stop or another Publish came first.""" + if generation != self._advertise_generation: + self._stops.retire(functools.partial(_stop_advertiser, advertiser)) + return + self._lan_advertiser = advertiser def _stop_lan_advertise(self) -> None: - if self._lan_advertiser is not None: - try: - self._lan_advertiser.stop() - except (RuntimeError, OSError): - pass - self._lan_advertiser = None + """Let go of the advertiser; unregistering sends packets and joins a thread, so off-thread.""" + self._advertise_generation += 1 # a registration still on its way is stopped when it reports + advertiser, self._lan_advertiser = self._lan_advertiser, None + if advertiser is not None: + self._stops.retire(functools.partial(_stop_advertiser, advertiser)) def _on_loop_offer_published(self, session_id: str) -> None: autocontrol_logger.debug("publish loop: offer for %s", session_id) diff --git a/je_auto_control/gui/remote_desktop/webrtc_host_media.py b/je_auto_control/gui/remote_desktop/webrtc_host_media.py index 32e189f6..8cc09b78 100644 --- a/je_auto_control/gui/remote_desktop/webrtc_host_media.py +++ b/je_auto_control/gui/remote_desktop/webrtc_host_media.py @@ -240,6 +240,8 @@ def _reset_host_quality_dot(self) -> None: self._host_quality_dot.setToolTip(_t("rd_webrtc_quality_unknown")) def _stop_adaptive(self) -> None: + # Stays on the GUI thread: StatsPoller.stop() cannot block (see its docstring), + # and the controller is only dropped. if self._adaptive_poller is not None: self._adaptive_poller.stop() self._adaptive_poller = None diff --git a/je_auto_control/gui/remote_desktop/webrtc_viewer_media.py b/je_auto_control/gui/remote_desktop/webrtc_viewer_media.py index 8f94dd41..eb363ceb 100644 --- a/je_auto_control/gui/remote_desktop/webrtc_viewer_media.py +++ b/je_auto_control/gui/remote_desktop/webrtc_viewer_media.py @@ -236,6 +236,7 @@ def _on_viewer_stats_sample(self, snapshot: StatsSnapshot) -> None: self._signals.stats.emit(snapshot) def _stop_stats_polling(self) -> None: + # Stays on the GUI thread: StatsPoller.stop() cannot block (see its docstring). if self._stats_poller is not None: self._stats_poller.stop() self._stats_poller = None diff --git a/je_auto_control/utils/remote_desktop/webrtc_stats.py b/je_auto_control/utils/remote_desktop/webrtc_stats.py index 55f19c13..90e5893d 100644 --- a/je_auto_control/utils/remote_desktop/webrtc_stats.py +++ b/je_auto_control/utils/remote_desktop/webrtc_stats.py @@ -64,6 +64,13 @@ async def _async_start(self) -> None: # NOSONAR self._task = asyncio.ensure_future(self._loop()) def stop(self) -> None: + """Stop polling; safe from any thread, and it never waits. + + It sets a flag and asks the bridge's loop to cancel the poll task + (``call_soon_threadsafe``): no thread is joined, no future is waited + for and no socket is closed, so a GUI slot may call it directly. The + loop finishes a sample that is already in flight on its own time. + """ self._stopped = True if self._task is None: return diff --git a/test/unit_test/headless/test_webrtc_lan_advertise_stop.py b/test/unit_test/headless/test_webrtc_lan_advertise_stop.py new file mode 100644 index 00000000..2fc839ca --- /dev/null +++ b/test/unit_test/headless/test_webrtc_lan_advertise_stop.py @@ -0,0 +1,178 @@ +"""WebRTC host panel: the LAN advertiser is registered and unregistered off the GUI thread. + +``HostAdvertiser.stop()`` sends zeroconf's goodbye packets and joins its +thread, and building one probes the name first; both ran in the Publish / Stop +slots. Offscreen Qt with a fake advertiser whose calls wait until the test lets +them go -- nothing here opens a socket. The stats pollers' ``stop()`` stays on +the GUI thread, and the last test pins why that is safe: it only schedules a +cancel on the asyncio bridge. +""" +import os +import threading +import time + +import pytest + +os.environ.setdefault("QT_QPA_PLATFORM", "offscreen") +pytest.importorskip("PySide6.QtWidgets", exc_type=ImportError) +pytest.importorskip("av") +pytest.importorskip("aiortc") + +from PySide6.QtCore import QEvent # noqa: E402 +from PySide6.QtWidgets import QApplication, QMessageBox # noqa: E402 + +from je_auto_control.gui.remote_desktop import webrtc_panel # noqa: E402 +from je_auto_control.utils.remote_desktop import lan_discovery, webrtc_stats, webrtc_transport # noqa: E402 +from headless._qt_settle import deleting, pump_until, settle_op # noqa: E402 + +_PROMPT_S = 5.0 + + +@pytest.fixture(autouse=True) +def qapp(monkeypatch): + app = QApplication.instance() or QApplication([]) + for box in ("warning", "information", "question", "critical"): + monkeypatch.setattr(QMessageBox, box, lambda *args: None) + yield app + + +@pytest.fixture(autouse=True) +def _panels_deleted(): + with deleting(webrtc_panel._WebRTCHostPanel): + yield + + +class _Advertisers: + """Stands in for ``lan_discovery``: records every build and stop, each of which can be held.""" + + def __init__(self): + self.log, self.lock = [], threading.Lock() + self.hold_stop, self.hold_build = threading.Event(), threading.Event() + self.hold_stop.set() + self.hold_build.set() + self.stop_entered = threading.Event() + + def build(self, *, host_id, signaling_url=None): + self.hold_build.wait(30.0) + advertiser = _Advertiser(self, host_id) + with self.lock: + self.log.append(("build", host_id, threading.current_thread() is threading.main_thread())) + return advertiser + + +class _Advertiser: + def __init__(self, owner, host_id): + self._owner, self.host_id = owner, host_id + + def stop(self): + self._owner.stop_entered.set() + self._owner.hold_stop.wait(30.0) + with self._owner.lock: + self._owner.log.append(("stop", self.host_id, threading.current_thread() is threading.main_thread())) + + +@pytest.fixture() +def advertisers(monkeypatch): + fake = _Advertisers() + monkeypatch.setattr(lan_discovery, "HostAdvertiser", fake.build) + monkeypatch.setattr(lan_discovery, "is_discovery_available", lambda: True) + yield fake + fake.hold_stop.set() + fake.hold_build.set() + + +def _panel(host_id="host-1"): + panel = webrtc_panel._WebRTCHostPanel() + panel._host_id_edit.setText(host_id) + return panel + + +def _timed(call): + started = time.monotonic() + call() + return time.monotonic() - started + + +def test_the_advertiser_is_built_off_the_gui_thread_and_kept(advertisers): + panel = _panel() + advertisers.hold_build.clear() + assert _timed(panel._start_lan_advertise) < _PROMPT_S + assert panel._lan_advertiser is None # still probing the name + advertisers.hold_build.set() + assert pump_until(lambda: panel._lan_advertiser is not None) + assert advertisers.log == [("build", "host-1", False)] + + +def test_stop_lets_go_of_the_advertiser_at_once_and_closes_it_in_the_background(advertisers): + panel = _panel() + panel._start_lan_advertise() + assert pump_until(lambda: panel._lan_advertiser is not None) + advertisers.hold_stop.clear() + assert _timed(panel._stop_lan_advertise) < _PROMPT_S + assert panel._lan_advertiser is None and panel._stops.pending == 1 + assert advertisers.stop_entered.wait(10.0) + panel._stop_lan_advertise() # nothing left to stop + assert panel._stops.pending == 1 + advertisers.hold_stop.set() + assert settle_op(panel._stops) + assert advertisers.log == [("build", "host-1", False), ("stop", "host-1", False)] + + +def test_a_new_advertiser_registers_only_after_the_old_one_has_unregistered(advertisers): + """The same host id is published again: registering it while the old one still holds the name fails.""" + panel = _panel() + panel._start_lan_advertise() + assert pump_until(lambda: panel._lan_advertiser is not None) + first = panel._lan_advertiser + advertisers.hold_stop.clear() + panel._start_lan_advertise() # what Publish does: stop the old one, start the new + assert advertisers.stop_entered.wait(10.0) + assert not pump_until(lambda: len(advertisers.log) > 1, timeout=0.3) + advertisers.hold_stop.set() + assert pump_until(lambda: panel._lan_advertiser not in (None, first)) + assert [entry[0] for entry in advertisers.log] == ["build", "stop", "build"] + + +def test_an_advertiser_that_comes_up_after_stop_is_closed(advertisers): + panel = _panel() + advertisers.hold_build.clear() + panel._start_lan_advertise() + panel._stop_lan_advertise() # Stop while the name is still being probed + advertisers.hold_build.set() + assert pump_until(lambda: [entry[0] for entry in advertisers.log] == ["build", "stop"]) + assert panel._lan_advertiser is None and settle_op(panel._stops) + + +def test_an_advertiser_that_comes_up_for_a_panel_that_is_gone_is_closed(advertisers, qapp): + panel = _panel() + advertisers.hold_build.clear() + panel._start_lan_advertise() + panel.deleteLater() + qapp.sendPostedEvents(None, QEvent.Type.DeferredDelete.value) + advertisers.hold_build.set() + assert pump_until(lambda: [entry[0] for entry in advertisers.log] == ["build", "stop"]) + + +def test_a_stats_poller_stop_only_schedules_a_cancel(monkeypatch): + """Why the pollers' ``stop()`` may stay on the GUI thread: no join, no future to wait for.""" + scheduled = [] + + class _Bridge: + def call_soon(self, callback, *args): + scheduled.append(callback) + + def submit(self, _coroutine): + raise AssertionError("stop() must not wait for the loop") + + class _Task: + def cancel(self): + raise AssertionError("cancelled on the calling thread instead of the loop's") + + monkeypatch.setattr(webrtc_transport, "get_bridge", _Bridge) + poller = webrtc_stats.StatsPoller(None, lambda _snapshot: None) + poller.stop() # never started: nothing to schedule + assert scheduled == [] + task = _Task() + poller._task = task + poller.stop() + assert scheduled == [task.cancel] and poller._task is None From 4e8e75001a8e63ba8d9a361431ccb4b980274dc6 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 07:12:13 +0800 Subject: [PATCH 04/17] Give every panel and tab that holds background work its own dispose() Releasing the Remote Desktop tab stopped timers only: a connect still out went on, the pop-out windows stayed, and a WebRTC host kept sharing the screen with no panel left to stop it. A released computer-use or DAG tab's run kept going. dispose() now cancels a tab's bare workers as well as its controller tasks, and the rule that requires the hook covers tabs that only hold background work. --- je_auto_control/gui/_dispose.py | 12 +- je_auto_control/gui/_worker_thread.py | 34 ++ je_auto_control/gui/assertions_tab.py | 9 + je_auto_control/gui/chatops_tab.py | 9 + je_auto_control/gui/computer_use_tab.py | 9 + je_auto_control/gui/dag_tab.py | 9 + je_auto_control/gui/device_matrix_tab.py | 9 + je_auto_control/gui/llm_planner_tab.py | 9 + je_auto_control/gui/media_checks_tab.py | 9 + je_auto_control/gui/mobile_tab.py | 9 + je_auto_control/gui/ocr_tab.py | 9 + .../gui/remote_desktop/connection_screen.py | 13 + je_auto_control/gui/remote_desktop/tab.py | 13 +- .../gui/remote_desktop/viewer_panel.py | 14 + .../gui/remote_desktop/webrtc_panel.py | 35 ++ .../gui/script_builder/builder_tab.py | 9 + je_auto_control/gui/self_healing_tab.py | 9 + je_auto_control/gui/test_suite_tab.py | 9 + je_auto_control/gui/usb_browser_tab.py | 9 + je_auto_control/gui/vlm_tab.py | 9 + je_auto_control/gui/webrunner_tab.py | 9 + .../headless/test_gui_panel_dispose.py | 377 ++++++++++++++++++ .../headless/test_gui_tab_dispose.py | 40 +- 23 files changed, 664 insertions(+), 9 deletions(-) create mode 100644 test/unit_test/headless/test_gui_panel_dispose.py diff --git a/je_auto_control/gui/_dispose.py b/je_auto_control/gui/_dispose.py index 349b30e6..ea3a6a14 100644 --- a/je_auto_control/gui/_dispose.py +++ b/je_auto_control/gui/_dispose.py @@ -9,6 +9,14 @@ hook, if the tab has one. ``dispose()`` lets go of all of it at once, on the call, so "release" means released when it returns. +Background work is cancelled the way destroying the tab would cancel it a +moment later: a task of the :mod:`~je_auto_control.gui.task_controller` through +its token (a script run stops, a result that still arrives goes to the task's +``discard``), a bare :func:`~je_auto_control.gui._worker_thread.start_worker` +worker through its ``request_stop()`` when it has one, and in both cases the +outcome is no longer delivered. Work that cannot be interrupted runs to its +end unobserved. + A tab's ``dispose()`` is one call to :func:`release_resources` naming its own extra releases. It must be safe to call twice, and the ``destroyed`` hooks the tabs already have stay: a tab deleted without ``dispose()`` (the window @@ -18,6 +26,7 @@ from PySide6.QtCore import QObject, QTimer +from je_auto_control.gui._worker_thread import cancel_workers from je_auto_control.gui.task_controller import task_controller from je_auto_control.utils.logging.logging_instance import autocontrol_logger @@ -33,13 +42,14 @@ def stop_timers(owner: QObject) -> int: def release_resources(owner: QObject, *releases: Callable[[], object]) -> None: - """Stop ``owner``'s timers, cancel its background tasks and run each release. + """Stop ``owner``'s timers, cancel its background tasks and workers, and run each release. A release that raises is logged and does not keep the others from running: a half-disposed tab would still be deleted. """ stop_timers(owner) task_controller().cancel_all(owner) + cancel_workers(owner) for release in releases: try: release() diff --git a/je_auto_control/gui/_worker_thread.py b/je_auto_control/gui/_worker_thread.py index 3de5908b..fbfd0c9f 100644 --- a/je_auto_control/gui/_worker_thread.py +++ b/je_auto_control/gui/_worker_thread.py @@ -33,6 +33,7 @@ import atexit import threading import time +import weakref from typing import Any, Callable, Dict, Optional from PySide6.QtCore import QObject, Signal @@ -74,6 +75,22 @@ class WorkerHandle: def __init__(self, worker: QObject) -> None: self.worker = worker self._thread: Optional[threading.Thread] = None + self._owner: Optional["weakref.ref[QObject]"] = None + self._relay: Optional["weakref.ref[_Relay]"] = None + + def cancel(self) -> None: + """Ask the worker to stop if it can, and deliver neither its result nor its failure. + + A worker with ``request_stop()`` ends at its next checkpoint; one + without runs to its end. ``on_thread_done`` still runs when the thread + ends -- it is the tab's bookkeeping, not an outcome. + """ + request_stop = getattr(self.worker, "request_stop", None) + if callable(request_stop): + request_stop() + relay = self._relay() if self._relay is not None else None + if relay is not None: + relay.drop_outcome() def isRunning(self) -> bool: # noqa: N802 # reason: the QThread spelling its callers use """Whether ``run()`` has not returned yet.""" @@ -143,6 +160,10 @@ def done(self, value: Any) -> None: """Forward the worker's result (runs on the GUI thread).""" self._on_done(value) + def drop_outcome(self) -> None: + """Deliver neither the result nor the failure from now on; the thread's end is still reported.""" + self._on_done = self._on_fail = WeakCall(None) + def fail(self, message: str) -> None: """Forward the worker's failure (runs on the GUI thread).""" self._on_fail(message) @@ -173,6 +194,18 @@ def _stop_running_workers() -> None: atexit.register(_stop_running_workers) +def cancel_workers(owner: QObject) -> int: + """Cancel every worker :func:`start_worker` is still running for ``owner``; return how many. + + What a tab's ``dispose()`` needs: see :meth:`WorkerHandle.cancel`. + """ + handles = [handle for handle in list(_RUNNING) + if handle._owner is not None and handle._owner() is owner] # noqa: SLF001 # reason: own class + for handle in handles: + handle.cancel() + return len(handles) + + def running_threads() -> int: """How many workers the GUI thread has not yet seen finish.""" return len(_RUNNING) @@ -222,6 +255,7 @@ def start_worker(owner: QObject, worker: QObject, *, if failed is not None: failed.connect(relay.fail) handle = WorkerHandle(worker) + handle._owner, handle._relay = weakref.ref(owner), weakref.ref(relay) # noqa: SLF001 # reason: set once _RUNNING[handle] = worker thread = threading.Thread(target=_run, args=(handle, relay, reaper), name=f"gui-worker-{type(worker).__name__}", daemon=True) diff --git a/je_auto_control/gui/assertions_tab.py b/je_auto_control/gui/assertions_tab.py index 4b41ae20..f066d83a 100644 --- a/je_auto_control/gui/assertions_tab.py +++ b/je_auto_control/gui/assertions_tab.py @@ -12,6 +12,7 @@ QVBoxLayout, QWidget, ) +from je_auto_control.gui._dispose import release_resources from je_auto_control.gui._i18n_helpers import TranslatableMixin from je_auto_control.gui._tab_task import TabTask from je_auto_control.gui.language_wrapper.multi_language_wrapper import ( @@ -81,6 +82,14 @@ def _build_layout(self) -> None: root.addWidget(self._result) root.addStretch() + def dispose(self) -> None: + """Release what the tab holds beyond its widgets: the command still running in the background. + + A script run is stopped; other work cannot be interrupted, so it runs to its end and its + result is dropped. Called by ``close_tab(key, release=True)``; safe to call twice. + """ + release_resources(self) + def menu_actions(self) -> list: """Expose tab commands to the window-level Actions menu.""" return [ diff --git a/je_auto_control/gui/chatops_tab.py b/je_auto_control/gui/chatops_tab.py index 15a8b06b..2a8b15dd 100644 --- a/je_auto_control/gui/chatops_tab.py +++ b/je_auto_control/gui/chatops_tab.py @@ -11,6 +11,7 @@ QVBoxLayout, QWidget, ) +from je_auto_control.gui._dispose import release_resources from je_auto_control.gui._i18n_helpers import TranslatableMixin from je_auto_control.gui._tab_task import TabTask, was_stopped from je_auto_control.gui.language_wrapper.multi_language_wrapper import ( @@ -74,6 +75,14 @@ def _build_layout(self) -> None: root.addWidget(self._output, stretch=1) self._apply_translations() + def dispose(self) -> None: + """Release what the tab holds beyond its widgets: the command still running in the background. + + A script run is stopped; other work cannot be interrupted, so it runs to its end and its + result is dropped. Called by ``close_tab(key, release=True)``; safe to call twice. + """ + release_resources(self) + def menu_actions(self) -> list: """Expose tab commands to the window-level Actions menu.""" return [ diff --git a/je_auto_control/gui/computer_use_tab.py b/je_auto_control/gui/computer_use_tab.py index f9109337..4bb49f92 100644 --- a/je_auto_control/gui/computer_use_tab.py +++ b/je_auto_control/gui/computer_use_tab.py @@ -9,6 +9,7 @@ QSpinBox, QTextEdit, QVBoxLayout, QWidget, ) +from je_auto_control.gui._dispose import release_resources from je_auto_control.gui._i18n_helpers import TranslatableMixin from je_auto_control.gui._worker_thread import WorkerHandle, start_worker from je_auto_control.gui.language_wrapper.multi_language_wrapper import ( @@ -109,6 +110,14 @@ def _apply_translations(self) -> None: self._output_label.setText(_t("computer_use_output_label")) self._goal_input.setPlaceholderText(_t("computer_use_goal_placeholder")) + def dispose(self) -> None: + """Release what the tab holds beyond its widgets: the run still out, which is asked to stop. + + The worker ends before its next step and its outcome is dropped. Called by + ``close_tab(key, release=True)`` before the widget is deleted; safe to call twice. + """ + release_resources(self) + def menu_actions(self) -> list: """Expose tab commands to the window-level Actions menu.""" return [ diff --git a/je_auto_control/gui/dag_tab.py b/je_auto_control/gui/dag_tab.py index f1a2f2cf..12c6e32f 100644 --- a/je_auto_control/gui/dag_tab.py +++ b/je_auto_control/gui/dag_tab.py @@ -9,6 +9,7 @@ QTableWidget, QTableWidgetItem, QTextEdit, QVBoxLayout, QWidget, ) +from je_auto_control.gui._dispose import release_resources from je_auto_control.gui._i18n_helpers import TranslatableMixin from je_auto_control.gui._worker_thread import WorkerHandle, start_worker from je_auto_control.gui.language_wrapper.multi_language_wrapper import ( @@ -87,6 +88,14 @@ def _build_layout(self) -> None: root.addWidget(self._table, stretch=3) self._apply_translations() + def dispose(self) -> None: + """Release what the tab holds beyond its widgets: the run still out, which is asked to stop. + + The worker ends once the nodes already running have finished and its outcome is dropped. Called by + ``close_tab(key, release=True)`` before the widget is deleted; safe to call twice. + """ + release_resources(self) + def menu_actions(self) -> list: """Expose tab commands to the window-level Actions menu.""" return [ diff --git a/je_auto_control/gui/device_matrix_tab.py b/je_auto_control/gui/device_matrix_tab.py index e6b5d737..f7a444b6 100644 --- a/je_auto_control/gui/device_matrix_tab.py +++ b/je_auto_control/gui/device_matrix_tab.py @@ -14,6 +14,7 @@ QSpinBox, QTableWidget, QTableWidgetItem, QVBoxLayout, QWidget, ) +from je_auto_control.gui._dispose import release_resources from je_auto_control.gui._i18n_helpers import TranslatableMixin from je_auto_control.gui.task_controller import TaskHandle, task_controller from je_auto_control.gui.language_wrapper.multi_language_wrapper import ( @@ -79,6 +80,14 @@ def _build_layout(self) -> None: root.addWidget(self._table, stretch=1) root.addWidget(self._summary) + def dispose(self) -> None: + """Release what the tab holds beyond its widgets: the command still running in the background. + + A script run is stopped; other work cannot be interrupted, so it runs to its end and its + result is dropped. Called by ``close_tab(key, release=True)``; safe to call twice. + """ + release_resources(self) + def menu_actions(self) -> list: """Expose tab commands to the window-level Actions menu.""" return [ diff --git a/je_auto_control/gui/llm_planner_tab.py b/je_auto_control/gui/llm_planner_tab.py index ff45c639..0485f7ea 100644 --- a/je_auto_control/gui/llm_planner_tab.py +++ b/je_auto_control/gui/llm_planner_tab.py @@ -16,6 +16,7 @@ QTextEdit, QVBoxLayout, QWidget, ) +from je_auto_control.gui._dispose import release_resources from je_auto_control.gui._i18n_helpers import TranslatableMixin from je_auto_control.gui._tab_task import TabTask, was_stopped from je_auto_control.gui._worker_thread import WorkerHandle, start_worker @@ -122,6 +123,14 @@ def _build_layout(self) -> None: root.addWidget(self._status) + def dispose(self) -> None: + """Release what the tab holds beyond its widgets: the command still running in the background. + + A script run is stopped; other work cannot be interrupted, so it runs to its end and its + result is dropped. Called by ``close_tab(key, release=True)``; safe to call twice. + """ + release_resources(self) + def menu_actions(self) -> list: """Expose tab commands to the window-level Actions menu.""" return [ diff --git a/je_auto_control/gui/media_checks_tab.py b/je_auto_control/gui/media_checks_tab.py index 24f2b0c1..cc2f04bc 100644 --- a/je_auto_control/gui/media_checks_tab.py +++ b/je_auto_control/gui/media_checks_tab.py @@ -13,6 +13,7 @@ QVBoxLayout, QWidget, ) +from je_auto_control.gui._dispose import release_resources from je_auto_control.gui._i18n_helpers import TranslatableMixin from je_auto_control.gui._tab_task import TabTask from je_auto_control.gui.language_wrapper.multi_language_wrapper import ( @@ -92,6 +93,14 @@ def _build_layout(self) -> None: root.addWidget(self._result) root.addStretch() + def dispose(self) -> None: + """Release what the tab holds beyond its widgets: the command still running in the background. + + A script run is stopped; other work cannot be interrupted, so it runs to its end and its + result is dropped. Called by ``close_tab(key, release=True)``; safe to call twice. + """ + release_resources(self) + def menu_actions(self) -> list: """Expose tab commands to the window-level Actions menu.""" return [ diff --git a/je_auto_control/gui/mobile_tab.py b/je_auto_control/gui/mobile_tab.py index d6f581f7..06cae4b7 100644 --- a/je_auto_control/gui/mobile_tab.py +++ b/je_auto_control/gui/mobile_tab.py @@ -18,6 +18,7 @@ ) import je_auto_control as ac +from je_auto_control.gui._dispose import release_resources from je_auto_control.gui._i18n_helpers import TranslatableMixin from je_auto_control.gui._tab_task import TabTask from je_auto_control.gui.language_wrapper.multi_language_wrapper import ( @@ -99,6 +100,14 @@ def _build_layout(self) -> None: root.addWidget(self._tr(QLabel(), "mob_result")) root.addWidget(self._result, stretch=1) + def dispose(self) -> None: + """Release what the tab holds beyond its widgets: the command still running in the background. + + A script run is stopped; other work cannot be interrupted, so it runs to its end and its + result is dropped. Called by ``close_tab(key, release=True)``; safe to call twice. + """ + release_resources(self) + def menu_actions(self) -> List[Tuple[str, Callable[[], None]]]: """Expose tab commands to the window-level Actions menu.""" return [ diff --git a/je_auto_control/gui/ocr_tab.py b/je_auto_control/gui/ocr_tab.py index 3650e37f..5caebbaa 100644 --- a/je_auto_control/gui/ocr_tab.py +++ b/je_auto_control/gui/ocr_tab.py @@ -14,6 +14,7 @@ QTextEdit, QVBoxLayout, QWidget, ) +from je_auto_control.gui._dispose import release_resources from je_auto_control.gui._i18n_helpers import TranslatableMixin from je_auto_control.gui.language_wrapper.multi_language_wrapper import ( language_wrapper, @@ -96,6 +97,14 @@ def _build_layout(self) -> None: root.addWidget(self._result, stretch=1) root.addWidget(self._status) + def dispose(self) -> None: + """Release what the tab holds beyond its widgets: the command still running in the background. + + A script run is stopped; other work cannot be interrupted, so it runs to its end and its + result is dropped. Called by ``close_tab(key, release=True)``; safe to call twice. + """ + release_resources(self) + def menu_actions(self) -> list: """Expose tab commands to the window-level Actions menu.""" return [ diff --git a/je_auto_control/gui/remote_desktop/connection_screen.py b/je_auto_control/gui/remote_desktop/connection_screen.py index ed211d76..9c3e5937 100644 --- a/je_auto_control/gui/remote_desktop/connection_screen.py +++ b/je_auto_control/gui/remote_desktop/connection_screen.py @@ -29,6 +29,7 @@ ) from je_auto_control.gui._i18n_helpers import TranslatableMixin +from je_auto_control.gui._dispose import release_resources from je_auto_control.gui._weak_call import WeakCall from je_auto_control.gui.remote_desktop._connect_task import connect_viewer from je_auto_control.gui.remote_desktop.connection_recent import _RecentConnectionsMixin @@ -179,6 +180,18 @@ def __init__(self, parent: Optional[QWidget] = None) -> None: self._refresh_status() self._refresh_timer.start() + def dispose(self) -> None: + """Release what the screen holds beyond its widgets: its timer, a connect or upload still out, its window. + + Called through ``RemoteDesktopTab.dispose()``; safe to call twice. The + host this screen started and the session it opened are deliberately + left running: they belong to the registry, where scripts + (``AC_remote_*``) and the other panels still see them and can stop + them. A connect that has not answered is cancelled, and the viewer it + may still produce is disconnected. + """ + release_resources(self, self._cancel_pending_connect, self._close_screen_window) + def retranslate(self) -> None: TranslatableMixin.retranslate(self) self._apply_placeholders() diff --git a/je_auto_control/gui/remote_desktop/tab.py b/je_auto_control/gui/remote_desktop/tab.py index b62c64cb..5b05a4ae 100644 --- a/je_auto_control/gui/remote_desktop/tab.py +++ b/je_auto_control/gui/remote_desktop/tab.py @@ -161,11 +161,14 @@ def _build_sub_panel_list(self) -> List[Tuple[QWidget, str]]: return sub_panels def dispose(self) -> None: - """Release what the sub-panels hold beyond their widgets: every status, preview and reconnect timer. - - Called by ``close_tab(key, release=True)``; safe to call twice. A host - or a session a panel opened is left running -- it belongs to the - registry, where scripts and the other panels still see it. + """Release what the sub-panels hold beyond their widgets, through each panel's own ``dispose()``. + + Called by ``close_tab(key, release=True)``; safe to call twice. + Timers stop, connects and transfers still out are cancelled and the + pop-out windows close. A TCP / WebSocket host or session a panel + opened is left running -- it belongs to the registry, where scripts + and the other panels still see it. A WebRTC host or session belongs to + its panel alone and is ended with it. """ disposals = [getattr(panel, "dispose", None) for panel in self._sub_panels] release_resources(self, *(dispose for dispose in disposals if callable(dispose))) diff --git a/je_auto_control/gui/remote_desktop/viewer_panel.py b/je_auto_control/gui/remote_desktop/viewer_panel.py index 69b37b09..b74bfc14 100644 --- a/je_auto_control/gui/remote_desktop/viewer_panel.py +++ b/je_auto_control/gui/remote_desktop/viewer_panel.py @@ -13,6 +13,7 @@ from je_auto_control.gui._daemon_thread import DaemonThread from je_auto_control.gui._i18n_helpers import TranslatableMixin +from je_auto_control.gui._dispose import release_resources from je_auto_control.gui.remote_desktop._connect_task import connect_viewer from je_auto_control.gui.remote_desktop._helpers import ( _CollapsibleSection, _StatusBadge, _build_insecure_client_context, @@ -219,6 +220,19 @@ def _wire_signals(self) -> None: # --- connection lifecycle ------------------------------------------ + def dispose(self) -> None: + """Release what the panel holds beyond its widgets: the connect still out, its audio player, its window. + + Called through ``RemoteDesktopTab.dispose()``; safe to call twice. The + session this panel opened is deliberately left connected: it belongs + to the registry, where scripts (``AC_remote_*``) and the other panels + still see it and can disconnect it. Only what nobody else could + reach goes -- a connect that has not answered is cancelled and the + viewer it may still produce is disconnected. + """ + release_resources(self, self._cancel_pending_connect, self._stop_audio_player, + self._close_screen_window) + def _connect(self) -> None: host = self._host_field.text().strip() token = self._token.text().strip() diff --git a/je_auto_control/gui/remote_desktop/webrtc_panel.py b/je_auto_control/gui/remote_desktop/webrtc_panel.py index 70cdcb52..6702a6fc 100644 --- a/je_auto_control/gui/remote_desktop/webrtc_panel.py +++ b/je_auto_control/gui/remote_desktop/webrtc_panel.py @@ -27,6 +27,7 @@ from je_auto_control.gui.remote_desktop._webrtc_types import MultiViewerHostT, SessionRecorderT, WebRTCDesktopViewerT from je_auto_control.gui._i18n_helpers import TranslatableMixin +from je_auto_control.gui._dispose import release_resources from je_auto_control.gui._slow_op import StopQueue from je_auto_control.gui.remote_desktop._helpers import ( _t, @@ -155,6 +156,23 @@ def prefill(self, token: Optional[str] = None, if signaling_url and hasattr(self, "_server_edit"): self._server_edit.setText(signaling_url) + def dispose(self) -> None: + """Release what the panel holds beyond its widgets -- including the host it runs. + + Called through ``RemoteDesktopTab.dispose()``; safe to call twice. + The TCP panels leave their host in the registry, where scripts and + the other panels still reach it. A WebRTC host belongs to this panel + alone: with the panel gone nothing could stop it, and it would go on + sharing the screen unseen. So it is stopped here, in the background as + for Stop, together with the signaling loop, the LAN advertiser, the + offer or file push still out and the tray icon. + """ + release_resources(self, self._stop_host_if_any, self._hide_tray) + + def _hide_tray(self) -> None: + if self._tray is not None: + self._tray.hide() + def _on_tray_open(self) -> None: win = self.window() if win is None: @@ -322,6 +340,23 @@ def _update_availability(self) -> None: widget.setEnabled(False) self._status_label.setText(_t("rd_webrtc_unavailable")) + def dispose(self) -> None: + """Release what the panel holds beyond its widgets -- including the session it is in. + + Called through ``RemoteDesktopTab.dispose()``; safe to call twice. + The TCP panels leave their session in the registry, where scripts and + the other panels still reach it. A WebRTC session belongs to this + panel alone, so it is ended here, in the background as for Stop: + the pending reconnect, the signaling workers, the recording, the + folder sync, the upload still out and the pop-out window go with it. + """ + release_resources(self, self._end_session_for_good) + + def _end_session_for_good(self) -> None: + self._user_initiated_disconnect = True # a late "disconnected" must not schedule a reconnect + self._stop_viewer_if_any() + self._close_screen_window() + def _ensure_screen_window(self) -> RemoteScreenWindow: if self._screen_window is not None: return self._screen_window diff --git a/je_auto_control/gui/script_builder/builder_tab.py b/je_auto_control/gui/script_builder/builder_tab.py index c051d44a..91818698 100644 --- a/je_auto_control/gui/script_builder/builder_tab.py +++ b/je_auto_control/gui/script_builder/builder_tab.py @@ -14,6 +14,7 @@ QTextEdit, QToolButton, QVBoxLayout, QWidget, ) +from je_auto_control.gui._dispose import release_resources from je_auto_control.gui._i18n_helpers import TranslatableMixin from je_auto_control.gui._journal_import import candidate_summary, pick_journal_candidate from je_auto_control.gui._tab_task import TabTask, was_stopped @@ -63,6 +64,14 @@ def __init__(self, parent: Optional[QWidget] = None) -> None: self._build_layout() self._wire_signals() + def dispose(self) -> None: + """Release what the tab holds beyond its widgets: the command still running in the background. + + A script run is stopped; other work cannot be interrupted, so it runs to its end and its + result is dropped. Called by ``close_tab(key, release=True)``; safe to call twice. + """ + release_resources(self) + def retranslate(self) -> None: TranslatableMixin.retranslate(self) if self._add_btn is not None: diff --git a/je_auto_control/gui/self_healing_tab.py b/je_auto_control/gui/self_healing_tab.py index b9aacf69..aded9759 100644 --- a/je_auto_control/gui/self_healing_tab.py +++ b/je_auto_control/gui/self_healing_tab.py @@ -19,6 +19,7 @@ QTableWidget, QTableWidgetItem, QVBoxLayout, QWidget, ) +from je_auto_control.gui._dispose import release_resources from je_auto_control.gui._i18n_helpers import TranslatableMixin from je_auto_control.gui._tab_task import TabTask from je_auto_control.gui.language_wrapper.multi_language_wrapper import ( @@ -113,6 +114,14 @@ def _build_measure_group(self) -> QGroupBox: self._measure_box = group return group + def dispose(self) -> None: + """Release what the tab holds beyond its widgets: the command still running in the background. + + A script run is stopped; other work cannot be interrupted, so it runs to its end and its + result is dropped. Called by ``close_tab(key, release=True)``; safe to call twice. + """ + release_resources(self) + def menu_actions(self) -> list: """Expose tab commands to the window-level Actions menu.""" return [ diff --git a/je_auto_control/gui/test_suite_tab.py b/je_auto_control/gui/test_suite_tab.py index b0e6d4b3..d012b787 100644 --- a/je_auto_control/gui/test_suite_tab.py +++ b/je_auto_control/gui/test_suite_tab.py @@ -15,6 +15,7 @@ QWidget, ) +from je_auto_control.gui._dispose import release_resources from je_auto_control.gui._i18n_helpers import TranslatableMixin from je_auto_control.gui._tab_task import TabTask, was_stopped from je_auto_control.gui.language_wrapper.multi_language_wrapper import ( @@ -79,6 +80,14 @@ def _build_layout(self) -> None: root.addWidget(QLabel(_t("suite_q_label"))) root.addWidget(self._quarantine) + def dispose(self) -> None: + """Release what the tab holds beyond its widgets: the command still running in the background. + + A script run is stopped; other work cannot be interrupted, so it runs to its end and its + result is dropped. Called by ``close_tab(key, release=True)``; safe to call twice. + """ + release_resources(self) + def menu_actions(self) -> list: """Expose tab commands to the window-level Actions menu.""" return [ diff --git a/je_auto_control/gui/usb_browser_tab.py b/je_auto_control/gui/usb_browser_tab.py index 93753fac..399ada29 100644 --- a/je_auto_control/gui/usb_browser_tab.py +++ b/je_auto_control/gui/usb_browser_tab.py @@ -28,6 +28,7 @@ QTableWidget, QTableWidgetItem, QVBoxLayout, QWidget, ) +from je_auto_control.gui._dispose import release_resources from je_auto_control.gui._i18n_helpers import TranslatableMixin from je_auto_control.gui._worker_thread import CallWorker as _CallWorker, WorkerHandle, start_worker from je_auto_control.gui.language_wrapper.multi_language_wrapper import ( @@ -162,6 +163,14 @@ def _build_layout(self) -> None: root.addWidget(self._status_label) root.addWidget(self._table, stretch=1) + def dispose(self) -> None: + """Release what the tab holds beyond its widgets: the device listing or open still out, whose answer is dropped. + + The request itself cannot be interrupted and runs to its end. Called by + ``close_tab(key, release=True)`` before the widget is deleted; safe to call twice. + """ + release_resources(self) + def menu_actions(self) -> list: """Expose tab commands to the window-level Actions menu.""" return [ diff --git a/je_auto_control/gui/vlm_tab.py b/je_auto_control/gui/vlm_tab.py index 049ca128..9248b4b6 100644 --- a/je_auto_control/gui/vlm_tab.py +++ b/je_auto_control/gui/vlm_tab.py @@ -5,6 +5,7 @@ QHBoxLayout, QLabel, QLineEdit, QVBoxLayout, QWidget, ) +from je_auto_control.gui._dispose import release_resources from je_auto_control.gui._i18n_helpers import TranslatableMixin from je_auto_control.gui._worker_thread import CallWorker, WorkerHandle, start_worker from je_auto_control.gui.language_wrapper.multi_language_wrapper import ( @@ -57,6 +58,14 @@ def _build_layout(self) -> None: root.addWidget(self._status) root.addStretch() + def dispose(self) -> None: + """Release what the tab holds beyond its widgets: the model request still out, whose answer is dropped. + + The request itself cannot be interrupted and runs to its end. Called by + ``close_tab(key, release=True)`` before the widget is deleted; safe to call twice. + """ + release_resources(self) + def menu_actions(self) -> list: """Expose tab commands to the window-level Actions menu.""" return [ diff --git a/je_auto_control/gui/webrunner_tab.py b/je_auto_control/gui/webrunner_tab.py index 0bbf975b..02387cbe 100644 --- a/je_auto_control/gui/webrunner_tab.py +++ b/je_auto_control/gui/webrunner_tab.py @@ -17,6 +17,7 @@ QVBoxLayout, QWidget, ) +from je_auto_control.gui._dispose import release_resources from je_auto_control.gui._i18n_helpers import TranslatableMixin from je_auto_control.gui._tab_task import TabTask from je_auto_control.gui.language_wrapper.multi_language_wrapper import ( @@ -78,6 +79,14 @@ def _build_layout(self) -> None: root.addWidget(QLabel(_t("web_output_label"))) root.addWidget(self._output, stretch=2) + def dispose(self) -> None: + """Release what the tab holds beyond its widgets: the command still running in the background. + + A script run is stopped; other work cannot be interrupted, so it runs to its end and its + result is dropped. Called by ``close_tab(key, release=True)``; safe to call twice. + """ + release_resources(self) + def menu_actions(self) -> list: """Expose tab commands to the window-level Actions menu.""" return [ diff --git a/test/unit_test/headless/test_gui_panel_dispose.py b/test/unit_test/headless/test_gui_panel_dispose.py new file mode 100644 index 00000000..1656349d --- /dev/null +++ b/test/unit_test/headless/test_gui_panel_dispose.py @@ -0,0 +1,377 @@ +"""``dispose()`` on the remote-desktop panels and on the tabs that start a worker on demand. + +Offscreen Qt with fake viewers, hosts and backends; nothing opens a socket, +shows a window or reaches a model. ``RemoteDesktopTab.dispose()`` used to stop +timers only: a connect still out went on, the pop-out windows stayed, and a +WebRTC host kept sharing the screen with no panel left to stop it. The tabs +that start a worker only when asked (computer use, DAG, VLM, USB browser) had +no ``dispose()`` at all, so a released tab's agent run kept going. +""" +import os +import threading +import types + +import pytest + +os.environ.setdefault("QT_QPA_PLATFORM", "offscreen") +pytest.importorskip("PySide6.QtWidgets", exc_type=ImportError) + +from PySide6.QtCore import QObject, QTimer, Signal # noqa: E402 +from PySide6.QtWidgets import QApplication, QMessageBox, QWidget # noqa: E402 + +from je_auto_control.gui import _dispose, computer_use_tab, dag_tab, usb_browser_tab, vlm_tab # noqa: E402 +from je_auto_control.gui._worker_thread import cancel_workers, start_worker # noqa: E402 +from je_auto_control.gui.remote_desktop import connection_screen, host_panel, viewer_panel # noqa: E402 +from je_auto_control.gui.remote_desktop.remote_screen_window import RemoteScreenWindow # noqa: E402 +from je_auto_control.utils.remote_desktop.address_book import AddressBook # noqa: E402 +from je_auto_control.utils.remote_desktop.connect_coordinator import parse_target # noqa: E402 +from je_auto_control.utils.remote_desktop.registry import SLOT_HOST, SLOT_VIEWER, registry # noqa: E402 +from headless._qt_settle import deleting, pump_until, settle, settle_op # noqa: E402 + + +class _Viewer: + remote_host_id = None + gate = None + made = [] + + def __init__(self, **kwargs): + self.kwargs, self.connected, self.disconnects = kwargs, False, 0 + _Viewer.made.append(self) + + def connect(self, timeout=5.0): + if _Viewer.gate is not None: + _Viewer.gate.wait(30.0) + self.connected = True + + def disconnect(self, timeout=2.0): + self.connected = False + self.disconnects += 1 + + def set_file_receiver(self, receiver): + self.receiver = receiver + + +class _Host: + port, connected_clients, host_id = 4321, 0, "123456789" + + def __init__(self, **kwargs): + self.kwargs, self.is_running = kwargs, False + + def start(self): + self.is_running = True + + def stop(self, timeout=2.0): + self.is_running = False + + def latest_frame(self): + return None + + +class _Window(RemoteScreenWindow): + """The popup, never shown: these tests put nothing on a screen.""" + + def show(self): + return None + + def raise_(self): + return None + + def activateWindow(self): # noqa: N802 # reason: Qt override + return None + + +@pytest.fixture(autouse=True) +def qapp(monkeypatch, tmp_path): + app = QApplication.instance() or QApplication([]) + for box in ("warning", "information", "question", "critical"): + monkeypatch.setattr(QMessageBox, box, lambda *args: None) + for attr in ("_host", "_viewer", "_ws_host", "_ws_viewer"): + monkeypatch.setattr(registry, attr, None) + monkeypatch.setattr(registry, "_claims", {}) + monkeypatch.setattr(_Viewer, "gate", None) + monkeypatch.setattr(_Viewer, "made", []) + for module in (connection_screen, viewer_panel): + monkeypatch.setattr(module, "RemoteDesktopViewer", _Viewer) + monkeypatch.setattr(module, "WebSocketDesktopViewer", _Viewer) + monkeypatch.setattr(module, "RemoteScreenWindow", _Window) + monkeypatch.setattr(connection_screen, "RemoteDesktopHost", _Host) + monkeypatch.setattr(host_panel, "RemoteDesktopHost", _Host) + monkeypatch.setattr(host_panel, "WebSocketDesktopHost", _Host) + monkeypatch.setattr(viewer_panel, "is_audio_backend_available", lambda: False) + monkeypatch.setattr(host_panel, "is_audio_backend_available", lambda: False) + book = AddressBook(tmp_path / "book.json") # never the operator's own address book + monkeypatch.setattr(connection_screen, "default_address_book", lambda: book) + yield app + + +@pytest.fixture(autouse=True) +def _panels_deleted(): + """Each test deletes the parentless panels it built (see ``_qt_settle.deleting``).""" + with deleting(viewer_panel._ViewerPanel, host_panel._HostPanel, connection_screen.QuickConnectScreen, + computer_use_tab.ComputerUseTab, dag_tab.DagTab, vlm_tab.VLMTab, + usb_browser_tab.UsbBrowserTab, _Owner): + yield + + +def _active_timers(widget): + return sum(1 for timer in widget.findChildren(QTimer) if timer.isActive()) + + +def _viewer_tab(): + panel = viewer_panel._ViewerPanel() + panel._host_field.setText("desk") + panel._port.setValue(5555) + panel._token.setText("tok") + panel._connect() + return panel + + +# --- remote desktop: the panels that keep their session in the registry ------------------------------------------ + +def test_the_viewer_panel_closes_its_window_and_leaves_the_session_in_the_registry(): + panel = _viewer_tab() + assert settle(panel) + viewer = panel._own_viewer() + assert viewer.connected and panel._screen_window is not None + panel.dispose() + assert panel._screen_window is None and _active_timers(panel) == 0 + assert registry.owned(SLOT_VIEWER, panel._owner) is viewer and viewer.connected + panel.dispose() # safe to call twice + assert viewer.connected + + +def test_the_viewer_panel_gives_up_a_connect_that_has_not_answered(): + _Viewer.gate = threading.Event() + panel = _viewer_tab() + assert panel._connect_task is not None + panel.dispose() + assert panel._connect_task is None + _Viewer.gate.set() + # The viewer that connects anyway is disconnected, and never adopted. + assert pump_until(lambda: _Viewer.made[-1].disconnects == 1) + assert registry.owner_of(SLOT_VIEWER) is None and panel._screen_window is None + + +def test_quick_connect_stops_polling_and_leaves_its_host_and_session_running(): + screen = connection_screen.QuickConnectScreen() + screen._host_token.setText("tok") + screen._start_hosting() + screen._dispatch_target(parse_target("desk:5555"), "tok") + assert settle(screen) + host, viewer = registry.host, screen._own_viewer() + assert host.is_running and viewer.connected and screen._screen_window is not None + assert _active_timers(screen) == 1 + screen.dispose() + assert _active_timers(screen) == 0 and screen._screen_window is None + assert registry.owner_of(SLOT_HOST) == screen._owner and host.is_running + assert registry.owned(SLOT_VIEWER, screen._owner) is viewer and viewer.connected + screen.dispose() + + +def test_quick_connect_gives_up_a_connect_that_has_not_answered(): + _Viewer.gate = threading.Event() + screen = connection_screen.QuickConnectScreen() + screen._dispatch_target(parse_target("desk:5555"), "tok") + assert screen._connect_task is not None + screen.dispose() + assert screen._connect_task is None + _Viewer.gate.set() + assert pump_until(lambda: _Viewer.made[-1].disconnects == 1) + assert screen._own_viewer() is None and screen._screen_window is None + + +# --- remote desktop: the WebRTC panels, whose session nobody else could reach ------------------------------------- + +class _Gate: + def __init__(self): + self.entered, self.release, self.threads = threading.Event(), threading.Event(), [] + + def __call__(self, *_args, **_kwargs): + self.threads.append(threading.current_thread()) + self.entered.set() + self.release.wait(30.0) + + +@pytest.fixture() +def webrtc(): + pytest.importorskip("av", exc_type=ImportError) + pytest.importorskip("aiortc", exc_type=ImportError) + from je_auto_control.gui.remote_desktop import webrtc_panel + with deleting(webrtc_panel._WebRTCHostPanel, webrtc_panel._WebRTCViewerPanel): + yield webrtc_panel + + +def test_the_webrtc_host_panel_stops_the_host_only_it_could_stop(webrtc): + panel = webrtc._WebRTCHostPanel() + gate = _Gate() + panel._multi_host = types.SimpleNamespace(stop_all=gate, session_count=lambda: 0) + panel.dispose() + assert panel._multi_host is None + assert gate.entered.wait(10.0) and gate.threads[0] is not threading.main_thread() + panel.dispose() # safe to call twice: no host left + gate.release.set() + assert settle_op(panel._stops) and len(gate.threads) == 1 + + +def test_the_webrtc_viewer_panel_ends_its_session_and_cancels_the_reconnect(webrtc): + panel = webrtc._WebRTCViewerPanel() + gate = _Gate() + panel._viewer = types.SimpleNamespace(stop=gate, authenticated=False, _pc=None) + panel._reconnect_timer.start(60_000) + panel.dispose() + assert panel._viewer is None and not panel._reconnect_timer.isActive() + assert gate.entered.wait(10.0) and gate.threads[0] is not threading.main_thread() + panel._maybe_schedule_auto_reconnect() # a late "disconnected" from the session that was ended + assert not panel._reconnect_timer.isActive() + gate.release.set() + assert settle_op(panel._stops) + panel.dispose() + + +def test_the_remote_desktop_tab_disposes_every_sub_panel(webrtc, monkeypatch): + from je_auto_control.gui.remote_desktop import tab as tab_module + with deleting(tab_module.RemoteDesktopTab): + tab = tab_module.RemoteDesktopTab() + assert len(tab._sub_panels) == 5 + called = [] + for panel in tab._sub_panels: + assert callable(getattr(panel, "dispose", None)), type(panel).__name__ + monkeypatch.setattr(panel, "dispose", lambda name=type(panel).__name__: called.append(name)) + tab.dispose() + assert called == ["QuickConnectScreen", "_HostPanel", "_ViewerPanel", "_WebRTCHostPanel", + "_WebRTCViewerPanel"] + assert _active_timers(tab) == 0 + + +# --- tabs that start a worker only when asked -------------------------------------------------------------------- + +def test_a_released_computer_use_tab_stops_its_run_and_drops_the_outcome(monkeypatch): + seen = [] + + def run(stop_event=None, **_params): + seen.append(stop_event) + assert stop_event.wait(30.0) + return "stopped" + + monkeypatch.setattr(computer_use_tab, "run_computer_use", run) + monkeypatch.setattr(computer_use_tab, "result_to_dict", lambda result: {"succeeded": True, "result": result}) + tab = computer_use_tab.ComputerUseTab() + tab._spawn_worker({"goal": "anything"}) + assert pump_until(lambda: bool(seen)) + before = tab._status.text() + tab.dispose() + assert seen[0].is_set() + assert pump_until(lambda: tab._thread is None) # the thread's end is still booked + assert tab._status.text() == before and tab._output.toPlainText() == "" + tab.dispose() + + +def test_a_released_dag_tab_stops_its_run_and_drops_the_outcome(monkeypatch): + seen = [] + + def run(_definition, max_parallel=1, stop_event=None): + seen.append(stop_event) + assert stop_event.wait(30.0) + return types.SimpleNamespace(succeeded=True, elapsed_s=0.0, nodes={"a": None}) + + monkeypatch.setattr(dag_tab, "run_dag", run) + tab = dag_tab.DagTab() + tab._spawn_worker({"nodes": []}) + assert pump_until(lambda: bool(seen)) + tab.dispose() + assert seen[0].is_set() + assert pump_until(lambda: tab._thread is None) + assert tab._table.rowCount() == 0 + + +def test_a_released_vlm_tab_drops_the_answer_of_the_request_still_out(monkeypatch): + release = threading.Event() + + def locate(_description, model=None): + release.wait(30.0) + return (10, 20) + + monkeypatch.setattr(vlm_tab, "locate_by_description", locate) + tab = vlm_tab.VLMTab() + tab._description.setText("the button") + tab._on_locate() + assert tab._vlm_thread is not None + tab.dispose() + release.set() + assert pump_until(lambda: tab._vlm_thread is None) + assert tab._last_result.text() == "" and tab._status.text() == "" + + +def test_a_released_usb_browser_tab_drops_the_open_still_out(monkeypatch): + release = threading.Event() + + def open_local(**_kwargs): + release.wait(30.0) + return b"\x12\x01" + + monkeypatch.setattr(usb_browser_tab, "open_local_descriptor", open_local) + tab = usb_browser_tab.UsbBrowserTab() + tab._start_local_open("1234", "5678", None) + before = tab._status_label.text() + tab.dispose() + release.set() + assert pump_until(lambda: tab._open_thread is None) + assert tab._status_label.text() == before + + +# --- the helper -------------------------------------------------------------------------------------------------- + +class _Worker(QObject): + finished = Signal(object) + failed = Signal(str) + + def __init__(self, fail=False): + super().__init__() + self.stop, self._fail = threading.Event(), fail + + def request_stop(self): + self.stop.set() + + def run(self): + self.stop.wait(30.0) + if self._fail: + self.failed.emit("broken") + else: + self.finished.emit("value") + + +class _Owner(QWidget): + def __init__(self): + super().__init__() + self.seen = [] + + def done(self, value): + self.seen.append(("done", value)) + + def fail(self, message): + self.seen.append(("fail", message)) + + def ended(self): + self.seen.append("ended") + + +@pytest.mark.parametrize("fail", [False, True]) +def test_cancelling_an_owners_workers_stops_them_and_keeps_only_the_end(fail): + owner, other = _Owner(), _Owner() + worker, bystander = _Worker(fail), _Worker() + start_worker(owner, worker, on_done=owner.done, on_fail=owner.fail, on_thread_done=owner.ended) + start_worker(other, bystander, on_done=other.done, on_fail=other.fail, on_thread_done=other.ended) + assert cancel_workers(owner) == 1 + assert worker.stop.is_set() and not bystander.stop.is_set() + assert pump_until(lambda: owner.seen == ["ended"]) + bystander.stop.set() + assert pump_until(lambda: other.seen == [("done", "value"), "ended"]) + assert cancel_workers(owner) == 0 + + +def test_release_resources_cancels_the_owners_bare_workers(): + owner, worker = _Owner(), _Worker() + start_worker(owner, worker, on_done=owner.done, on_fail=owner.fail, on_thread_done=owner.ended) + _dispose.release_resources(owner) + assert worker.stop.is_set() + assert pump_until(lambda: owner.seen == ["ended"]) diff --git a/test/unit_test/headless/test_gui_tab_dispose.py b/test/unit_test/headless/test_gui_tab_dispose.py index c10f15b3..6eda4749 100644 --- a/test/unit_test/headless/test_gui_tab_dispose.py +++ b/test/unit_test/headless/test_gui_tab_dispose.py @@ -240,8 +240,11 @@ def test_releasing_a_tab_through_the_registry_runs_its_dispose(monkeypatch): # --- every tab that holds something has the hook ----------------------------------------------------------------- -#: A tab class whose source does one of these at any point holds something deletion alone releases late. -_HOLDS = ("QTimer(", ".add_listener(", "hold_default_watcher(", ".attach(") +#: A tab class whose source does one of these at any point holds something deletion alone releases late: +#: a timer, a listener, or background work whose outcome would still be delivered (or which keeps running). +_HOLDS = ("QTimer(", ".add_listener(", "hold_default_watcher(", ".attach(", + "start_worker(", "TabTask(", "task_controller()", "SlowOp(", "StopQueue(", "connect_viewer(", + "start_panel_task(") def _tab_classes(): @@ -274,7 +277,38 @@ def test_every_registered_tab_that_holds_a_timer_or_a_listener_implements_dispos continue if any(marker in source for marker in _HOLDS) and "def dispose(self)" not in source: missing.append(f"{module_name}.{class_name}") - assert missing == [], f"tabs that start a timer or register a listener without dispose(): {missing}" + assert missing == [], f"tabs that hold a timer, a listener or background work without dispose(): {missing}" + + +#: Remote Desktop is built by the main widget, so the registry names no module for it; its sub-panels +#: are the classes that hold things. +_REMOTE_DESKTOP_PANELS = [ + ("je_auto_control.gui.remote_desktop.tab", "RemoteDesktopTab"), + ("je_auto_control.gui.remote_desktop.connection_screen", "QuickConnectScreen"), + ("je_auto_control.gui.remote_desktop.host_panel", "_HostPanel"), + ("je_auto_control.gui.remote_desktop.viewer_panel", "_ViewerPanel"), + ("je_auto_control.gui.remote_desktop.webrtc_panel", "_WebRTCHostPanel"), + ("je_auto_control.gui.remote_desktop.webrtc_panel", "_WebRTCViewerPanel"), +] + + +@pytest.mark.parametrize("module_name, class_name", _REMOTE_DESKTOP_PANELS, + ids=[case[1] for case in _REMOTE_DESKTOP_PANELS]) +def test_every_remote_desktop_panel_implements_dispose(module_name, class_name): + try: + source = _class_source(module_name, class_name) + except ImportError: # the webrtc extra is not installed here + pytest.skip("optional extra not installed") + assert "def dispose(self)" in source + + +def test_the_guard_covers_tabs_that_only_hold_background_work(): + sources = {name: _class_source(module, name) for module, name in _tab_classes() + if name in ("VLMTab", "ComputerUseTab", "DagTab", "UsbBrowserTab", "OCRReaderTab")} + assert len(sources) == 5 + for name, source in sources.items(): + assert any(marker in source for marker in _HOLDS[4:]), name + assert not any(marker in source for marker in _HOLDS[:4]), name def test_the_guard_reads_real_tab_classes(): From 04947a1dd6520f5520b4b795bdf7f69c234e39fa Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 07:15:19 +0800 Subject: [PATCH 05/17] Park the pages of closed tabs for a restyle and let a parked page find its real window A closed tab's page is kept as a hidden child of the tab widget, inside the window's tree, so a theme switch still restyled it in the same slot. And while a page is parked its window() is the parking widget: the WebRTC tray's Open would have shown that empty widget as a window instead of raising the main one. --- .../gui/remote_desktop/webrtc_panel.py | 7 +- je_auto_control/gui/workspace_tabs.py | 73 +++++++++++++++++-- test/unit_test/headless/test_gui_restyle.py | 72 +++++++++++++++++- 3 files changed, 140 insertions(+), 12 deletions(-) diff --git a/je_auto_control/gui/remote_desktop/webrtc_panel.py b/je_auto_control/gui/remote_desktop/webrtc_panel.py index 6702a6fc..5c51ea23 100644 --- a/je_auto_control/gui/remote_desktop/webrtc_panel.py +++ b/je_auto_control/gui/remote_desktop/webrtc_panel.py @@ -50,6 +50,7 @@ HostPublishLoopWorker, ViewerAnswerPushWorker, ViewerSignalingWorker, ) from je_auto_control.gui.task_controller import TaskHandle +from je_auto_control.gui.workspace_tabs import real_window from je_auto_control.utils.logging.logging_instance import autocontrol_logger from je_auto_control.utils.remote_desktop import ( default_address_book, @@ -174,9 +175,9 @@ def _hide_tray(self) -> None: self._tray.hide() def _on_tray_open(self) -> None: - win = self.window() - if win is None: - return + # Not self.window(): for a few event-loop turns after a theme switch + # this page is parked outside the main window's tree. + win = real_window(self) win.showNormal() win.raise_() win.activateWindow() diff --git a/je_auto_control/gui/workspace_tabs.py b/je_auto_control/gui/workspace_tabs.py index 7c4d1c77..7c91e02d 100644 --- a/je_auto_control/gui/workspace_tabs.py +++ b/je_auto_control/gui/workspace_tabs.py @@ -24,6 +24,16 @@ parent -- or at once for a tab that gets selected meanwhile. A parked page is still the tab's page for every method here; only its Qt parent differs, and only until the queue has drained. + +The page of a *closed* tab is kept as a hidden child of the tab widget, which +is inside the window's tree too: those pages are parked with the others and +come back, hidden, at the end of the same queue. + +While a page is parked its ``window()`` is the parking widget, not the main +window. Code in a tab that needs the window it belongs to -- to raise it, or +to place a dialog -- asks :func:`real_window` instead, which sees through the +parking; the parking widget itself refuses to be shown, so a stray +``self.window().show()`` cannot put an empty window on screen. """ from typing import Any, Callable, List, Optional @@ -31,6 +41,31 @@ from PySide6.QtWidgets import QFrame, QScrollArea, QTabWidget, QVBoxLayout, QWidget +#: Dynamic property on the page of a closed tab while it is kept, hidden, under the tab widget. +_CLOSED_PAGE = "_workspace_closed_page" + + +class _Parking(QWidget): + """Where pages wait, out of the window's tree, during a restyle; never visible.""" + + def __init__(self, home: QWidget) -> None: + super().__init__() + self.home = home + + def setVisible(self, visible: bool) -> None: # noqa: N802 # reason: Qt name + """Stay hidden: a parked page's ``window()`` is this widget, and must not open as one.""" + super().setVisible(False) + + +def real_window(widget: QWidget) -> QWidget: + """``widget.window()``, or the window its tab belongs to while its page is parked for a restyle.""" + top = widget.window() + if isinstance(top, _Parking): + home: QWidget = top.home.window() + return home + return top + + class PageHolder(QWidget): """What the tab widget really holds for one tab: the page, scrolled or not.""" @@ -96,6 +131,7 @@ def set_page(self, page: QWidget) -> None: """Put ``page`` in the holder.""" self._page = page self._parked = False + page.setProperty(_CLOSED_PAGE, None) # shown in a tab again, wherever it waited if self._area is None: self.layout().addWidget(page) page.show() @@ -124,7 +160,7 @@ def __init__(self, parent: Optional[QWidget] = None) -> None: super().__init__(parent) # Where the pages of unselected tabs wait during a restyle; made on # first use and dropped once empty. - self._parking: Optional[QWidget] = None + self._parking: Optional[_Parking] = None self._unpark_timer = QTimer(self) self._unpark_timer.setInterval(0) self._unpark_timer.timeout.connect(self._unpark_one) @@ -135,17 +171,20 @@ def __init__(self, parent: Optional[QWidget] = None) -> None: def restyle(self, apply: Callable[[], object]) -> int: """Run ``apply`` -- a style sheet or palette change above this widget -- touching only what shows. - The pages of the tabs that are not selected are parked first and come - back afterwards, one per event-loop turn. Returns how many pages were - parked for this call. + The pages of the tabs that are not selected, and the kept pages of + closed tabs, are parked first and come back afterwards, one per + event-loop turn. Returns how many pages were parked for this call. """ if self._parking is None: - self._parking = QWidget() + self._parking = _Parking(self) current = self._holder(self.currentIndex()) parked = 0 for holder in self._holders(): if holder is not current and holder.park(self._parking): parked += 1 + for page in self._closed_pages(self): + page.setParent(self._parking) + parked += 1 try: apply() finally: @@ -157,15 +196,31 @@ def restyle(self, apply: Callable[[], object]) -> int: def parked_pages(self) -> int: """How many pages are still waiting to be put back after a restyle.""" - return sum(1 for holder in self._holders() if holder.parked) + return sum(1 for holder in self._holders() if holder.parked) + len(self._closed_pages(self._parking)) def finish_restyle(self) -> None: """Put every parked page back now instead of over the next event-loop turns.""" for holder in self._holders(): holder.unpark() + for page in self._closed_pages(self._parking): + self._keep_closed(page) self._unpark_timer.stop() self._drop_parking() + @staticmethod + def _closed_pages(below: Optional[QWidget]) -> List[QWidget]: + """The kept pages of closed tabs that are direct children of ``below``.""" + if below is None: + return [] + return [child for child in below.children() + if isinstance(child, QWidget) and child.property(_CLOSED_PAGE)] + + def _keep_closed(self, page: QWidget) -> None: + """Keep the page of a closed tab: hidden, below this widget, where the next open finds it.""" + page.setProperty(_CLOSED_PAGE, True) + page.setParent(self) + page.hide() + def _holders(self) -> List[PageHolder]: holders = (self._holder(index) for index in range(self.count())) return [holder for holder in holders if holder is not None] @@ -180,6 +235,9 @@ def _unpark_one(self) -> None: for holder in self._holders(): if holder.unpark(): return + for page in self._closed_pages(self._parking): + self._keep_closed(page) + return self._unpark_timer.stop() self._drop_parking() @@ -275,8 +333,7 @@ def removeTab(self, index: int) -> None: # noqa: N802 # reason: Qt name return page = holder.take_page() if page is not None: - page.setParent(self) - page.hide() + self._keep_closed(page) holder.setParent(None) holder.deleteLater() diff --git a/test/unit_test/headless/test_gui_restyle.py b/test/unit_test/headless/test_gui_restyle.py index 9ecb9add..e5a8e845 100644 --- a/test/unit_test/headless/test_gui_restyle.py +++ b/test/unit_test/headless/test_gui_restyle.py @@ -18,7 +18,7 @@ from PySide6.QtWidgets import QApplication, QLabel, QVBoxLayout, QWidget # noqa: E402 from je_auto_control.gui.window_settings import SETTINGS_ENV # noqa: E402 -from je_auto_control.gui.workspace_tabs import WorkspaceTabWidget # noqa: E402 +from je_auto_control.gui.workspace_tabs import WorkspaceTabWidget, real_window # noqa: E402 from headless._qt_settle import pump_until # noqa: E402 ROOT = pathlib.Path(__file__).resolve().parents[3] @@ -106,6 +106,76 @@ def test_closing_a_tab_whose_page_is_parked_keeps_the_page(workspace): assert tabs.widget(1) is pages[1] and _size(pages[1]) == 15 +def test_the_page_of_a_closed_tab_is_not_restyled_in_the_same_slot(workspace): + """A closed tab's page is kept as a hidden child of the tab widget, and was restyled with the window.""" + window, tabs, pages = workspace + tabs.removeTab(1) + assert pages[1].parent() is tabs and _size(pages[1]) == 9 + parked = tabs.restyle(lambda: window.setStyleSheet("* { font-size: 17pt; }")) + assert parked == 2 and tabs.parked_pages() == 2 # the other tab's page and the closed one + assert not _in_window(pages[1], window) and pages[1].isHidden() + assert pump_until(lambda: tabs.parked_pages() == 0) + assert pages[1].parent() is tabs and pages[1].isHidden() and tabs.indexOf(pages[1]) == -1 + assert _size(pages[1]) == 17, "the closed page came back with the old style" + assert pump_until(lambda: tabs._parking is None) + + +def test_reopening_a_closed_tab_during_a_restyle_takes_its_page_out_of_the_queue(workspace): + window, tabs, pages = workspace + tabs.removeTab(1) + tabs.restyle(lambda: window.setStyleSheet("* { font-size: 15pt; }")) + tabs.insertTab(1, pages[1], "two") + tabs.setCurrentIndex(1) + assert _in_window(pages[1], window) and _size(pages[1]) == 15 and not pages[1].isHidden() + assert pump_until(lambda: tabs.parked_pages() == 0) + assert pump_until(lambda: tabs._parking is None) + assert tabs.widget(1) is pages[1] and _in_window(pages[1], window) # not taken back out as "closed" + + +def test_finishing_a_restyle_brings_closed_pages_back_too(workspace): + window, tabs, pages = workspace + tabs.removeTab(2) + tabs.restyle(lambda: window.setStyleSheet("* { font-size: 15pt; }")) + tabs.finish_restyle() + assert tabs.parked_pages() == 0 and pages[2].parent() is tabs and pages[2].isHidden() + assert _size(pages[2]) == 15 + + +def test_a_parked_page_still_finds_the_real_window(workspace): + """``page.window()`` is the parking widget for a few event-loop turns; ``real_window`` sees through it.""" + window, tabs, pages = workspace + tabs.removeTab(2) + assert real_window(pages[1]) is window and real_window(pages[2]) is window + tabs.restyle(lambda: window.setStyleSheet("* { font-size: 15pt; }")) + parking = pages[1].window() + assert parking is not window + assert real_window(pages[1]) is window and real_window(pages[1].label) is window + assert real_window(pages[2]) is window # the closed tab's page + parking.show() # what ``self.window().showNormal()`` in a page would have done + parking.showNormal() + assert not parking.isVisible() + tabs.finish_restyle() + assert real_window(pages[1]) is window + + +def test_the_webrtc_tray_raises_the_main_window_even_while_its_page_is_parked(workspace, monkeypatch): + pytest.importorskip("av", exc_type=ImportError) + pytest.importorskip("aiortc", exc_type=ImportError) + from je_auto_control.gui.remote_desktop import webrtc_panel + window, tabs, _pages = workspace + panel = webrtc_panel._WebRTCHostPanel() + tabs.addTab(panel, "host") + raised = [] + for name in ("showNormal", "raise_", "activateWindow"): + monkeypatch.setattr(window, name, lambda name=name: raised.append(name), raising=False) + tabs.restyle(lambda: window.setStyleSheet("* { font-size: 15pt; }")) + assert panel.window() is not window + panel._on_tray_open() + assert raised == ["showNormal", "raise_", "activateWindow"] + assert not panel.window().isVisible() # the parking widget was not shown as a window + tabs.finish_restyle() + + def test_a_restyle_that_raises_still_brings_the_pages_back(workspace): _window, tabs, pages = workspace From 93e78df6e9aa94568dbf4dba22c57ce376525c2e Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 07:23:24 +0800 Subject: [PATCH 06/17] Deny every device while the USB ACL file fails its integrity check An ACL built with default_policy="allow" kept allowing next to integrity_ok False, and rules read from an earlier intact version stayed in force after a later read failed. decide() now answers deny until the file reads intact again or a change is saved. --- .../usb_passthrough_operator_guide.rst | 8 ++ .../usb_passthrough_operator_guide.rst | 6 ++ je_auto_control/utils/usb/passthrough/acl.py | 30 +++++-- .../headless/test_usb_acl_fail_closed.py | 90 +++++++++++++++++++ 4 files changed, 128 insertions(+), 6 deletions(-) create mode 100644 test/unit_test/headless/test_usb_acl_fail_closed.py diff --git a/docs/source/Eng/doc/operations_layer/usb_passthrough_operator_guide.rst b/docs/source/Eng/doc/operations_layer/usb_passthrough_operator_guide.rst index 568f6f8d..9c38802e 100644 --- a/docs/source/Eng/doc/operations_layer/usb_passthrough_operator_guide.rst +++ b/docs/source/Eng/doc/operations_layer/usb_passthrough_operator_guide.rst @@ -163,6 +163,14 @@ On load a mismatch makes the ACL **fail closed** (default-deny, ``UsbAcl.integrity_ok`` reports ``False``). This stops a process that silently rewrites the JSON from granting itself access. +Failing closed means ``decide()`` answers ``"deny"`` for **every** device: +also when the instance was built with ``default_policy="allow"``, and also +for allow rules it had read from an earlier, intact version of the file (an +allow-by-default ACL used to keep allowing next to ``integrity_ok False``). +``default_policy`` reports ``"deny"`` for as long as that lasts. Decisions +return to normal once the file reads intact again, or once the operator saves +a change, which re-signs it. + The signature used to be a sidecar ``usb_acl.json.sig`` written after the data, so the GUI and a host session -- or two processes -- could read new data against the old signature and fall back to deny-all on a file nobody diff --git a/docs/source/Zh/doc/operations_layer/usb_passthrough_operator_guide.rst b/docs/source/Zh/doc/operations_layer/usb_passthrough_operator_guide.rst index a3af2faf..2482173d 100644 --- a/docs/source/Zh/doc/operations_layer/usb_passthrough_operator_guide.rst +++ b/docs/source/Zh/doc/operations_layer/usb_passthrough_operator_guide.rst @@ -147,6 +147,12 @@ ACL 由 HMAC-SHA256 簽章保護，簽章就存在檔案本身的 ``"signature"` **fail-closed**\ （default-deny，``UsbAcl.integrity_ok`` 回 ``False``\ ）。這擋住 偷偷改寫 JSON 想給自己授權的 process。 +fail-closed 的意思是 ``decide()`` 對**每一個**裝置都回 ``"deny"``：即使這個實例是用 +``default_policy="allow"`` 建立的、即使它先前從完好的檔案讀到過允許規則也一樣 +（以前 allow-by-default 的 ACL 會一邊回報 ``integrity_ok False`` 一邊繼續放行）。 +這段期間 ``default_policy`` 也回報 ``"deny"``。等檔案再次讀到完好的內容，或操作者 +存入一次變更（會重新簽章），才恢復正常判斷。 + 簽章以前是寫在資料之後的 sidecar ``usb_acl.json.sig``，所以 GUI 與 host session （或兩個行程）可能讀到新資料配舊簽章，於是在沒有人動過的檔案上退回全部拒絕。現在 資料與簽章是同一個檔案、以一次 rename 取代。變更（新增／移除／匯入／設定預設值） diff --git a/je_auto_control/utils/usb/passthrough/acl.py b/je_auto_control/utils/usb/passthrough/acl.py index 7592332a..ea011f2a 100644 --- a/je_auto_control/utils/usb/passthrough/acl.py +++ b/je_auto_control/utils/usb/passthrough/acl.py @@ -38,9 +38,12 @@ under ``signature``. It covers the rest of the object in a canonical form (sorted keys, no insignificant whitespace), so re-indenting the file does not break it and changing any value does. On load a mismatch makes the ACL fail -closed (default-deny, ``integrity_ok`` False) so a process that silently -rewrites the JSON cannot grant itself access without also forging the -signature. +closed (``integrity_ok`` False) so a process that silently rewrites the JSON +cannot grant itself access without also forging the signature. Failing closed +means ``decide()`` answers ``"deny"`` for every device -- also when the +instance was built with ``default_policy="allow"``, and also for rules it had +read from an earlier, intact version of the file -- until the file reads +intact again or the operator saves a change (which re-signs it). **One file, one lock.** The signature used to live in a sidecar ``.sig``, written after the data. Two instances, or two processes, could @@ -244,13 +247,17 @@ def path(self) -> Path: @property def integrity_ok(self) -> bool: - """False once a signature mismatch was seen on load.""" + """False from a read that failed the signature check until a good read or a save. + + While it is False :meth:`decide` denies every device. + """ return self._integrity_ok @property def default_policy(self) -> str: + """The policy for a device no rule matches; ``"deny"`` while ``integrity_ok`` is False.""" with self._lock: - return self._state.default + return self._state.default if self._integrity_ok else "deny" def list_rules(self) -> List[AclRule]: with self._lock: @@ -370,8 +377,16 @@ def set_default_policy(self, policy: str, *, persist: bool = True) -> None: def decide(self, *, vendor_id: str, product_id: str, serial: Optional[str]) -> str: - """Return ``"allow"`` / ``"deny"`` / ``"prompt"`` for one OPEN.""" + """Return ``"allow"`` / ``"deny"`` / ``"prompt"`` for one OPEN. + + ``"deny"`` for every device while the file fails its integrity check, + whatever ``default_policy`` the instance was built with and whatever + rules it had read before: an allow-by-default ACL used to keep + allowing next to ``integrity_ok`` False. + """ with self._lock: + if not self._integrity_ok: + return "deny" for rule in self._state.rules: if rule.matches(vendor_id=vendor_id, product_id=product_id, serial=serial): @@ -518,6 +533,9 @@ def _load(self) -> None: "(fail closed, default-deny)", self._path, ) return + # Intact again (restored by the operator, or rewritten by another + # instance): the lockdown a bad read started ends with a good one. + self._integrity_ok = True if payload is None: self._set_aside("unparseable: not a JSON object") return diff --git a/test/unit_test/headless/test_usb_acl_fail_closed.py b/test/unit_test/headless/test_usb_acl_fail_closed.py new file mode 100644 index 00000000..9239d4a4 --- /dev/null +++ b/test/unit_test/headless/test_usb_acl_fail_closed.py @@ -0,0 +1,90 @@ +"""A USB ACL whose file fails its integrity check denies everything. + +``UsbAcl(path, default_policy="allow")`` used to keep the constructor's +default when the file failed its check: ``integrity_ok`` said ``False`` and +``decide()`` still answered ``"allow"``. Temp files only. +""" +import json + +import pytest + +from je_auto_control.utils.usb.passthrough.acl import AclRule, UsbAcl + +_DEVICE = {"vendor_id": "1050", "product_id": "0407", "serial": None} +_OTHER = {"vendor_id": "dead", "product_id": "beef", "serial": None} + + +def _signed_file(tmp_path, *, default="allow", prompt=False): + """An intact ACL holding one rule for ``_DEVICE``; return its path.""" + path = tmp_path / "usb_acl.json" + acl = UsbAcl(path=path, default_policy=default) + acl.add_rule(AclRule(vendor_id="1050", product_id="0407", allow=True, prompt_on_open=prompt)) + acl.set_default_policy(default) + return path + + +def _tamper(path) -> None: + stored = json.loads(path.read_text(encoding="utf-8")) + stored["rules"].append({"vendor_id": "dead", "product_id": "beef", "serial": None, + "label": "", "allow": True, "prompt_on_open": False}) + path.write_text(json.dumps(stored), encoding="utf-8") + + +@pytest.mark.parametrize("default_policy", ["allow", "deny"]) +def test_a_tampered_file_denies_whatever_the_default_policy(tmp_path, default_policy): + path = _signed_file(tmp_path) + _tamper(path) + acl = UsbAcl(path=path, default_policy=default_policy) + assert acl.integrity_ok is False + assert acl.decide(**_OTHER) == "deny" + assert acl.decide(**_DEVICE) == "deny" + assert acl.default_policy == "deny" + + +def test_an_intact_allow_file_still_allows(tmp_path): + acl = UsbAcl(path=_signed_file(tmp_path), default_policy="allow") + assert acl.integrity_ok is True + assert acl.decide(**_OTHER) == "allow" + assert acl.decide(**_DEVICE) == "allow" + assert acl.default_policy == "allow" + + +def test_a_missing_file_keeps_the_constructor_default(tmp_path): + acl = UsbAcl(path=tmp_path / "usb_acl.json", default_policy="allow") + assert acl.integrity_ok is True + assert acl.decide(**_OTHER) == "allow" + + +def test_tampering_seen_on_a_later_read_also_denies_rules_loaded_before(tmp_path): + path = _signed_file(tmp_path, prompt=True) + acl = UsbAcl(path=path, default_policy="allow") + assert acl.decide(**_DEVICE) == "prompt" + _tamper(path) + # Removing a rule that is not there re-reads the file and saves nothing. + assert acl.remove_rule(vendor_id="ffff", product_id="ffff") is False + assert acl.integrity_ok is False + assert acl.decide(**_DEVICE) == "deny" + assert acl.decide(**_OTHER) == "deny" + + +def test_a_save_by_the_operator_ends_the_lockdown(tmp_path): + path = _signed_file(tmp_path) + _tamper(path) + acl = UsbAcl(path=path, default_policy="allow") + assert acl.decide(**_OTHER) == "deny" + acl.set_default_policy("allow") + assert acl.integrity_ok is True + assert acl.decide(**_OTHER) == "allow" + assert UsbAcl(path=path).integrity_ok is True + + +def test_an_intact_file_read_after_a_bad_one_is_trusted_again(tmp_path): + path = _signed_file(tmp_path) + good = path.read_bytes() + _tamper(path) + acl = UsbAcl(path=path, default_policy="allow") + assert acl.integrity_ok is False + path.write_bytes(good) + assert acl.remove_rule(vendor_id="ffff", product_id="ffff") is False # re-reads only + assert acl.integrity_ok is True + assert acl.decide(**_DEVICE) == "allow" From 2e9e354b4d60a6a362ace9ee862a002dc9a0a917 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 07:26:39 +0800 Subject: [PATCH 07/17] Hold acknowledged tombstones so a machine the bucket never listed meets the deletion A tombstone waited only for devices under peers. A machine holding the entry that had never synced added it straight back once the tombstone was collected, with no conflict and no report. An acknowledged tombstone is now kept for thirty days from the commit that first carried it (deleted_at), so such a machine's copy becomes a conflict instead. Age only ever keeps a tombstone longer; acknowledgement by the listed devices stays necessary. --- je_auto_control/utils/config_sync/client.py | 15 +- je_auto_control/utils/config_sync/merge.py | 26 ++- je_auto_control/utils/config_sync/versions.py | 45 +++- .../headless/test_config_sync_convergence.py | 4 +- .../headless/test_config_sync_empty_change.py | 4 +- .../test_config_sync_legacy_origin.py | 7 +- .../test_config_sync_unlisted_device.py | 193 ++++++++++++++++++ test/unit_test/headless/test_sync_adapters.py | 6 +- 8 files changed, 274 insertions(+), 26 deletions(-) create mode 100644 test/unit_test/headless/test_config_sync_unlisted_device.py diff --git a/je_auto_control/utils/config_sync/client.py b/je_auto_control/utils/config_sync/client.py index 9aef72c7..662e4718 100644 --- a/je_auto_control/utils/config_sync/client.py +++ b/je_auto_control/utils/config_sync/client.py @@ -19,6 +19,7 @@ from je_auto_control.utils.config_sync.merge import ( ConflictRecord, apply_operation, awaits_ack, batch_operation_id, merge_buckets, settle, ) +from je_auto_control.utils.config_sync.versions import TOMBSTONE_HOLD_S from je_auto_control.utils.exception.exceptions import AutoControlException if TYPE_CHECKING: @@ -50,12 +51,18 @@ class ConfigSyncClient: Methods are intentionally small and synchronous — the GUI wraps them in QThread workers when wiring up periodic sync. + + ``tombstone_hold_s`` is how long a deletion every listed device has + acknowledged is still kept in the bucket this client commits (default + :data:`~je_auto_control.utils.config_sync.versions.TOMBSTONE_HOLD_S`, + thirty days), so a machine that has not synced yet still meets it. """ def __init__(self, server_url: str, *, user_id: str, secret: Optional[str] = None, timeout_s: float = _DEFAULT_TIMEOUT_S, - device_id: Optional[str] = None) -> None: + device_id: Optional[str] = None, + tombstone_hold_s: float = TOMBSTONE_HOLD_S) -> None: if not server_url: raise ConfigSyncError("server_url is required") if not user_id: @@ -65,6 +72,7 @@ def __init__(self, server_url: str, *, self._secret = secret self._timeout = float(timeout_s) self._device_id = device_id or None + self._tombstone_hold_s = max(0.0, float(tombstone_hold_s)) @property def device_id(self) -> str: @@ -198,7 +206,7 @@ def sync(self, local: ConfigBucket, *, max_attempts: int = DEFAULT_SYNC_ATTEMPTS remote = self.fetch() or ConfigBucket(user_id=self._user_id) self._require_active(remote, device) merged, conflicts = merge_buckets(local, remote) - settle(merged, device, remote.revision + 1, stamp, None) + settle(merged, device, remote.revision + 1, stamp, None, self._tombstone_hold_s) try: self.push(merged, base_revision=remote.revision) except ConfigSyncConflict: @@ -238,7 +246,8 @@ def push_operations(self, operations: Sequence["SyncOperation"], *, device_id: s changed = [apply_operation(merged, operation, conflicts) for operation in operations] if not any(changed) and not awaits_ack(remote, device_id): return SyncResult(bucket=remote, revision=remote.revision, conflicts=conflicts) - settle(merged, device_id, remote.revision + 1, stamp, max_offline_s) + settle(merged, device_id, remote.revision + 1, stamp, max_offline_s, + self._tombstone_hold_s) try: revision = self.push(merged, base_revision=remote.revision, operation_id=batch_operation_id(remote.revision, operations)) diff --git a/je_auto_control/utils/config_sync/merge.py b/je_auto_control/utils/config_sync/merge.py index f9d8088d..b97148b6 100644 --- a/je_auto_control/utils/config_sync/merge.py +++ b/je_auto_control/utils/config_sync/merge.py @@ -20,7 +20,8 @@ TOMBSTONE_RETENTION_S, ConfigBucket, ConfigSyncError, is_tombstone, ) from je_auto_control.utils.config_sync.versions import ( - PeerState, SyncConflict, SyncEntry, SyncOperation, collect_tombstones, merge_entries, + TOMBSTONE_HOLD_S, PeerState, SyncConflict, SyncEntry, SyncOperation, collect_tombstones, + merge_entries, ) @@ -215,13 +216,15 @@ def awaits_ack(bucket: ConfigBucket, device_id: str) -> bool: def settle(bucket: ConfigBucket, device_id: str, revision: int, now: float, - max_offline_s: Optional[float]) -> None: + max_offline_s: Optional[float], hold_s: float = TOMBSTONE_HOLD_S) -> None: """Prepare ``bucket`` to be committed as ``revision`` by ``device_id``. - Stamps new tombstones with the revision that first carries them, records - that this device has merged up to it, retires peers unseen for longer - than ``max_offline_s`` (when given), and drops the tombstones every - remaining peer has acknowledged. + Stamps new tombstones with the revision that first carries them (and + ``now``), records that this device has merged up to it, retires peers + unseen for longer than ``max_offline_s`` (when given), and drops the + tombstones every remaining peer has acknowledged -- once they are also + older than ``hold_s``, so a machine the bucket does not list yet still + meets a recent deletion (see :func:`collect_tombstones`). """ peers = bucket.peer_states() peers[device_id] = PeerState(device_id, acked_revision=revision, last_seen=now) @@ -231,15 +234,16 @@ def settle(bucket: ConfigBucket, device_id: str, revision: int, now: float, for device, peer in peers.items()} bucket.peers = {device: peer.to_dict() for device, peer in peers.items()} for section in list(bucket.sections): - versioned = _stamped(bucket.sync_entries(section), revision) - kept = collect_tombstones(versioned, peers.values()) + versioned = _stamped(bucket.sync_entries(section), revision, now) + kept = collect_tombstones(versioned, peers.values(), now=now, hold_s=hold_s) flat = {key: body for key, body in bucket.sections[section].items() if key not in versioned} bucket.sections[section] = {**flat, **{key: entry.to_dict() for key, entry in kept.items()}} -def _stamped(entries: Mapping[str, SyncEntry], revision: int) -> Dict[str, SyncEntry]: - """``entries`` with each new tombstone marked as first carried by ``revision``.""" - return {key: (replace(entry, deleted_revision=revision) +def _stamped(entries: Mapping[str, SyncEntry], revision: int, + now: float) -> Dict[str, SyncEntry]: + """``entries`` with each new tombstone marked as first carried by ``revision``, at ``now``.""" + return {key: (replace(entry, deleted_revision=revision, deleted_at=float(now)) if entry.deleted and not entry.deleted_revision else entry) for key, entry in entries.items()} diff --git a/je_auto_control/utils/config_sync/versions.py b/je_auto_control/utils/config_sync/versions.py index f15d562e..63b5b754 100644 --- a/je_auto_control/utils/config_sync/versions.py +++ b/je_auto_control/utils/config_sync/versions.py @@ -20,6 +20,15 @@ that includes it; a peer that stayed away too long is *retired* and must do an explicit full resync, so it cannot bring deleted entries back. +That rule only knows the devices the bucket lists. A machine that has an +entry but has never synced is not among them, and once the tombstone is gone +its copy would simply be added again. So an acknowledged tombstone is also +*held* for :data:`TOMBSTONE_HOLD_S` after the revision that first carried it: +a machine that joins inside that time meets the deletion, and its copy +becomes a conflict for a person to settle instead of a silent return. This is +the one place a clock is read, and it can only keep a tombstone longer -- +acknowledgement by every listed device stays necessary whatever the clocks say. + Pure standard library; imports no ``PySide6``. """ from __future__ import annotations @@ -31,6 +40,11 @@ from je_auto_control.utils.config_sync.bucket import ConfigSyncError +#: How long a tombstone every listed device has acknowledged is still kept, +#: counted from the revision that first carried it, for the machines the +#: bucket does not list yet. +TOMBSTONE_HOLD_S = 30 * 24 * 3600.0 + #: Relations :func:`compare_vectors` reports. EQUAL = "equal" BEFORE = "before" @@ -78,6 +92,10 @@ class SyncEntry: entry includes; ``origin`` and ``operation_id`` name the change that produced it. ``siblings`` is non-empty while the entry is an unresolved conflict -- then ``value`` is ``None`` and the candidates are there. + ``deleted_revision`` and ``deleted_at`` are set on a tombstone by the + commit that first carries it: the revision peers must acknowledge, and + that commit's wall-clock time (for :data:`TOMBSTONE_HOLD_S`; ``0`` when + an older client stamped it). """ key: str value: Optional[Mapping[str, Any]] = None @@ -88,6 +106,7 @@ class SyncEntry: modified_at: float = 0.0 deleted_revision: int = 0 siblings: Tuple["SyncEntry", ...] = () + deleted_at: float = 0.0 @property def is_deleted(self) -> bool: @@ -142,6 +161,8 @@ def to_dict(self) -> Dict[str, Any]: } if self.deleted_revision: body["deleted_revision"] = int(self.deleted_revision) + if self.deleted_at: + body["deleted_at"] = float(self.deleted_at) if self.siblings: body["siblings"] = [sibling.to_dict() for sibling in self.siblings] return body @@ -165,6 +186,7 @@ def from_dict(cls, key: str, body: Mapping[str, Any], *, _depth: int = 0) -> "Sy modified_at=_number(key, body.get("last_modified", 0.0)), deleted_revision=int(_number(key, body.get("deleted_revision", 0))), siblings=tuple(cls.from_dict(key, sibling, _depth=1) for sibling in siblings), + deleted_at=_number(key, body.get("deleted_at", 0.0)), ) @@ -267,7 +289,8 @@ def _conflicted(key: str, left: SyncEntry, right: SyncEntry) -> SyncEntry: # There is nothing to choose between, so it is not a conflict. return SyncEntry(key=key, value=only.value, vector=vector, origin=only.origin, operation_id=only.operation_id, deleted=only.deleted, - modified_at=only.modified_at, deleted_revision=only.deleted_revision) + modified_at=only.modified_at, deleted_revision=only.deleted_revision, + deleted_at=only.deleted_at) return SyncEntry( key=key, vector=vector, siblings=tuple(survivors), operation_id="conflict:" + "+".join(_identity(candidate) for candidate in survivors)) @@ -364,25 +387,35 @@ def collectable_revision(peers: Iterable[PeerState]) -> int: return min(acked) if acked else 0 -def collect_tombstones(entries: Mapping[str, SyncEntry], peers: Iterable[PeerState], +def collect_tombstones(entries: Mapping[str, SyncEntry], peers: Iterable[PeerState], *, + now: Optional[float] = None, hold_s: float = TOMBSTONE_HOLD_S, ) -> Dict[str, SyncEntry]: """``entries`` without the tombstones every active peer has acknowledged. A tombstone is dropped only when its ``deleted_revision`` -- the server revision that first carried it -- is at or below what each active peer - has merged. Age is never a reason: a tombstone dropped while a machine - that still holds the entry is merely offline lets that machine bring the - entry back. A tombstone not yet stamped with a revision is always kept. + has merged. Age is never a reason to drop one: a tombstone dropped while + a machine that still holds the entry is merely offline lets that machine + bring the entry back. A tombstone not yet stamped with a revision is + always kept. + + Given ``now``, age is a reason to *keep* one a little longer: an + acknowledged tombstone stamped (``deleted_at``) less than ``hold_s`` ago + stays, for the machines not listed among ``peers`` yet. Without ``now``, + or for a tombstone that carries no stamp, acknowledgement alone decides. """ horizon = collectable_revision(peers) + held_after = None if now is None else float(now) - max(0.0, float(hold_s)) return {key: entry for key, entry in entries.items() if not (entry.deleted and not entry.siblings - and 0 < entry.deleted_revision <= horizon)} + and 0 < entry.deleted_revision <= horizon + and not (held_after is not None and entry.deleted_at > held_after))} __all__ = [ "AFTER", "BEFORE", "CONCURRENT", "CONFLICT", "EQUAL", "KEPT_LEFT", "KEPT_RIGHT", "MergeDecision", "PeerState", "SyncConflict", "SyncEntry", "SyncOperation", + "TOMBSTONE_HOLD_S", "collect_tombstones", "collectable_revision", "compare_vectors", "is_versioned", "join_vectors", "merge_collections", "merge_entries", ] diff --git a/test/unit_test/headless/test_config_sync_convergence.py b/test/unit_test/headless/test_config_sync_convergence.py index 76488789..5c38ec8e 100644 --- a/test/unit_test/headless/test_config_sync_convergence.py +++ b/test/unit_test/headless/test_config_sync_convergence.py @@ -214,7 +214,9 @@ def test_a_deletion_reaches_an_offline_machine_however_long_it_was_away(server): def test_retired_peer_requires_full_sync(server): - client = _client() + # No hold: this is about who a tombstone waits for, not for how long it is + # kept afterwards (test_config_sync_unlisted_device.py covers that). + client = ConfigSyncClient("https://sync.invalid", user_id="alice", tombstone_hold_s=0) entry = SyncEntry.create("hk1", {"combo": "ctrl+a"}, "laptop") client.push_operations([_op(entry)], device_id="laptop") client.push_operations([], device_id="desktop") diff --git a/test/unit_test/headless/test_config_sync_empty_change.py b/test/unit_test/headless/test_config_sync_empty_change.py index a765fa01..e632c4d1 100644 --- a/test/unit_test/headless/test_config_sync_empty_change.py +++ b/test/unit_test/headless/test_config_sync_empty_change.py @@ -103,7 +103,9 @@ def test_joining_a_bucket_that_holds_entries_is_still_recorded(tmp_path, server) def test_a_bucket_emptied_of_entries_is_not_joined_either(server): - client = ConfigSyncClient(_URL, user_id="alice") + # tombstone_hold_s=0: the bucket has to be empty for this, so the deletion + # must not be held for the machines that have not synced yet. + client = ConfigSyncClient(_URL, user_id="alice", tombstone_hold_s=0) entry = SyncEntry.create("a", {"v": 1}, "laptop") client.push_operations([SyncOperation("custom", entry)], device_id="laptop") client.push_operations([SyncOperation("custom", entry.removed("laptop"))], device_id="laptop") diff --git a/test/unit_test/headless/test_config_sync_legacy_origin.py b/test/unit_test/headless/test_config_sync_legacy_origin.py index 9ade4cbd..9d27f535 100644 --- a/test/unit_test/headless/test_config_sync_legacy_origin.py +++ b/test/unit_test/headless/test_config_sync_legacy_origin.py @@ -160,8 +160,13 @@ def test_a_tombstone_waits_for_a_device_that_only_uses_the_old_sync(server): # The desktop syncs its stale copy: the deletion wins instead of the entry returning. desktop, _ = _client("desktop").sync(desktop) assert desktop.values("hotkeys") == {} - # Everyone has acknowledged it now, so the next commit may drop it. + # Everyone has acknowledged it now, but it is held for the machines that + # have not synced yet ... laptop, _ = _client("laptop").sync(laptop) + assert SyncEntry.from_dict("hk1", server.body["sections"]["hotkeys"]["hk1"]).deleted + # ... until the hold has passed; then the next commit drops it. + stamp = server.body["sections"]["hotkeys"]["hk1"]["deleted_at"] + laptop, _ = _client("laptop").sync(laptop, now=stamp + 31 * 24 * 3600.0) assert "hk1" not in server.body["sections"]["hotkeys"] diff --git a/test/unit_test/headless/test_config_sync_unlisted_device.py b/test/unit_test/headless/test_config_sync_unlisted_device.py new file mode 100644 index 00000000..3ddf1016 --- /dev/null +++ b/test/unit_test/headless/test_config_sync_unlisted_device.py @@ -0,0 +1,193 @@ +"""A deletion is still there when a machine the bucket never listed turns up. + +A versioned tombstone is dropped once every device under ``peers`` has +acknowledged it. A machine that holds the entry but has never synced is not +under ``peers``: after the tombstone was gone, its first sync simply added the +entry again -- no conflict, no report. An acknowledged tombstone is now also +held for ``TOMBSTONE_HOLD_S`` from the commit that first carried it, so a +machine that joins inside that time meets the deletion. + +Every time here is passed in (``now=``); no test reads the clock. +""" +from unittest.mock import patch + +import pytest + +from je_auto_control.utils.config_sync import ( + ConfigBucket, ConfigSyncClient, ConfigSyncConflict, PeerState, SyncEntry, SyncOperation, + collect_tombstones, +) +from je_auto_control.utils.config_sync.merge import settle +from je_auto_control.utils.config_sync.versions import TOMBSTONE_HOLD_S + +_DAY = 24 * 3600.0 +_T0 = 1_700_000_000.0 + + +class _Server: + """The revision-checked bucket endpoint, in memory.""" + + def __init__(self): + self.body = None + self.revision = 0 + + def request(self, method, body=None): + if method == "GET": + return self.body + if body["base_revision"] != self.revision: + raise ConfigSyncConflict("behind", self.revision) + self.revision += 1 + self.body = {**body["bucket"], "revision": self.revision} + return {"ok": True, "revision": self.revision} + + def bucket(self): + return ConfigBucket.from_dict(self.body) + + +@pytest.fixture +def server(): + endpoint = _Server() + with patch.object(ConfigSyncClient, "_request", + new=lambda _client, method, body=None: endpoint.request(method, body)): + yield endpoint + + +def _client(device, **options): + return ConfigSyncClient("https://sync.invalid", user_id="alice", device_id=device, **options) + + +def _push(device, operations=(), *, now, **options): + entries = [SyncOperation(section="hotkeys", entry=entry) for entry in operations] + return _client(device, **options).push_operations(entries, device_id=device, now=now) + + +def _delete_on_both_listed_devices(now=_T0, **options): + """laptop and desktop both know ``hk1``; laptop deletes it; both acknowledge.""" + created = SyncEntry.create("hk1", {"combo": "ctrl+a"}, "laptop") + _push("laptop", [created], now=now, **options) + _push("desktop", now=now, **options) # joins: it now holds hk1 too + _push("laptop", [created.removed("laptop")], now=now, **options) + _push("desktop", now=now, **options) # acknowledges the deletion + _push("laptop", now=now, **options) + return created + + +def test_a_machine_that_never_synced_meets_the_deletion_instead_of_undoing_it(server): + _delete_on_both_listed_devices() + assert server.bucket().values("hotkeys") == {} + # The tablet has had hk1 all along and syncs for the first time a day later. + result = _push("tablet", [SyncEntry.create("hk1", {"combo": "ctrl+a"}, "tablet")], + now=_T0 + _DAY) + entry = server.bucket().get_entry("hotkeys", "hk1") + assert entry.in_conflict, "the deletion and the tablet's copy are both kept" + assert sorted(sibling.deleted for sibling in entry.siblings) == [False, True] + assert [conflict.key for conflict in result.conflicts] == ["hk1"] + assert server.bucket().values("hotkeys") == {}, "nothing came back by itself" + + +def test_every_machine_sees_the_same_conflict_and_one_choice_settles_it(server): + _delete_on_both_listed_devices() + _push("tablet", [SyncEntry.create("hk1", {"combo": "ctrl+a"}, "tablet")], now=_T0 + _DAY) + seen = [_push(device, now=_T0 + _DAY).bucket.get_entry("hotkeys", "hk1") + for device in ("laptop", "desktop", "tablet")] + assert seen[0] == seen[1] == seen[2] and seen[0].in_conflict + # A person on the desktop decides the deletion stands. + _push("desktop", [seen[1].resolved(None, "desktop")], now=_T0 + _DAY) + for device in ("laptop", "tablet", "desktop"): + assert _push(device, now=_T0 + _DAY).bucket.values("hotkeys") == {} + assert not server.bucket().conflicts() + + +def test_a_flat_copy_from_an_unlisted_machine_stays_deleted(server): + """``client.sync`` with an entry in the flat shape from before version vectors.""" + _delete_on_both_listed_devices() + local = ConfigBucket(user_id="alice") + local.upsert("hotkeys", "hk1", {"combo": "ctrl+a", "last_modified": _T0 + 5}, + versioned=False) + merged, conflicts = _client("tablet").sync(local, now=_T0 + _DAY) + assert merged.values("hotkeys") == {} and server.bucket().values("hotkeys") == {} + assert [(record.entry_id, record.unresolved) for record in conflicts] == [("hk1", False)] + + +def test_the_tombstone_goes_once_it_is_acknowledged_and_the_hold_has_passed(server): + _delete_on_both_listed_devices() + assert server.bucket().get_entry("hotkeys", "hk1").deleted, "held, though both acknowledged" + # Still inside the hold: a later commit keeps it. + _push("laptop", [SyncEntry.create("hk2", {"combo": "b"}, "laptop")], + now=_T0 + TOMBSTONE_HOLD_S - 60) + assert server.bucket().get_entry("hotkeys", "hk1") is not None + # Past it: the next commit drops it, so the bucket does not grow for ever. + _push("laptop", [SyncEntry.create("hk3", {"combo": "c"}, "laptop")], + now=_T0 + TOMBSTONE_HOLD_S + 60) + assert server.bucket().get_entry("hotkeys", "hk1") is None + assert sorted(server.bucket().values("hotkeys")) == ["hk2", "hk3"] + + +def test_a_held_tombstone_does_not_make_acknowledged_devices_commit_again(server): + _delete_on_both_listed_devices() + revision = server.revision + for device in ("laptop", "desktop", "laptop"): + assert _push(device, now=_T0 + _DAY).pushed is False + assert server.revision == revision + + +def test_no_clock_drops_a_tombstone_a_listed_device_has_not_acknowledged(server): + created = SyncEntry.create("hk1", {"combo": "ctrl+a"}, "laptop") + _push("laptop", [created], now=_T0) + _push("desktop", now=_T0) + _push("laptop", [created.removed("laptop")], now=_T0) + # The desktop has not acknowledged; the laptop's clock jumps ten years. + _push("laptop", [SyncEntry.create("hk2", {"combo": "b"}, "laptop")], now=_T0 + 3650 * _DAY) + assert server.bucket().get_entry("hotkeys", "hk1").deleted + # ... and a clock far behind cannot shorten the wait for the desktop either. + _push("laptop", [SyncEntry.create("hk3", {"combo": "c"}, "laptop")], now=1.0) + assert server.bucket().get_entry("hotkeys", "hk1").deleted + + +def test_a_hold_of_zero_is_the_earlier_rule(server): + _delete_on_both_listed_devices(tombstone_hold_s=0) + assert server.bucket().get_entry("hotkeys", "hk1") is None + # ... and that is the case the hold exists for: the entry simply returns. + _push("tablet", [SyncEntry.create("hk1", {"combo": "ctrl+a"}, "tablet")], + now=_T0 + _DAY, tombstone_hold_s=0) + assert server.bucket().values("hotkeys") == {"hk1": {"combo": "ctrl+a"}} + + +def test_a_tombstone_stamped_by_an_older_client_is_collected_on_acknowledgement(): + """No ``deleted_at``: nothing to measure the hold from, so nothing is held.""" + base = SyncEntry.create("hk1", {"combo": "a"}, "laptop") + body = {**base.removed("laptop").to_dict(), "deleted_revision": 3} + assert "deleted_at" not in body + old = SyncEntry.from_dict("hk1", body) + assert old.deleted_at == pytest.approx(0.0) + peers = [PeerState("laptop", acked_revision=3), PeerState("desktop", acked_revision=4)] + assert collect_tombstones({"hk1": old}, peers, now=_T0) == {} + + +def test_collect_tombstones_without_a_time_goes_by_acknowledgement_alone(): + base = SyncEntry.create("hk1", {"combo": "a"}, "laptop") + stamped = SyncEntry.from_dict("hk1", { + **base.removed("laptop").to_dict(), "deleted_revision": 3, "deleted_at": _T0}) + peers = [PeerState("laptop", acked_revision=3)] + assert collect_tombstones({"hk1": stamped}, peers) == {} + assert collect_tombstones({"hk1": stamped}, peers, now=_T0 + 60) == {"hk1": stamped} + assert collect_tombstones({"hk1": stamped}, peers, now=_T0 + 60, hold_s=0) == {} + assert collect_tombstones({"hk1": stamped}, peers, now=_T0 + TOMBSTONE_HOLD_S) == {} + behind = [PeerState("laptop", acked_revision=2)] + assert collect_tombstones({"hk1": stamped}, behind, now=_T0 + 10 * TOMBSTONE_HOLD_S) == { + "hk1": stamped} + + +def test_the_stamp_is_written_once_and_survives_the_bucket_shape(): + bucket = ConfigBucket(user_id="alice") + base = SyncEntry.create("hk1", {"combo": "a"}, "laptop") + bucket.put_entry("hotkeys", base.removed("laptop")) + settle(bucket, "laptop", 5, _T0, None) + stored = bucket.sections["hotkeys"]["hk1"] + assert stored["deleted_revision"] == 5 and stored["deleted_at"] == pytest.approx(_T0) + again = ConfigBucket.from_dict(bucket.to_dict()) + settle(again, "desktop", 6, _T0 + _DAY, None) + kept = again.get_entry("hotkeys", "hk1") + assert kept.deleted_revision == 5 and kept.deleted_at == pytest.approx(_T0) + # A live entry carries neither field. + assert "deleted_at" not in base.to_dict() and "deleted_revision" not in base.to_dict() diff --git a/test/unit_test/headless/test_sync_adapters.py b/test/unit_test/headless/test_sync_adapters.py index 1ffcd5b3..294b84d0 100644 --- a/test/unit_test/headless/test_sync_adapters.py +++ b/test/unit_test/headless/test_sync_adapters.py @@ -435,6 +435,9 @@ def test_a_cancelled_sync_stops_and_keeps_its_changes(tmp_path, server): def test_a_retired_device_is_told_to_resync_and_adopts_the_server_state(tmp_path, server): laptop, desktop = _pair(tmp_path) + # No hold, so the tombstone is really gone when the retired desktop returns: + # the refusal is then all that keeps the entry from coming back. + laptop.client = ConfigSyncClient(_URL, user_id="alice", tombstone_hold_s=0) laptop_store, desktop_store = {"a": {"v": 1}}, {} laptop.adapters = [_DictAdapter("laptop", laptop_store)] desktop.adapters = [_DictAdapter("desktop", desktop_store)] @@ -513,10 +516,7 @@ def test_a_folder_mirror_does_not_push_back_what_it_received(tmp_path): assert engine.poll_once() == [] and sent == ["mine.txt"] # Edited here afterwards: now it is a local change and goes out. - import os (watch / "report.txt").write_bytes(b"edited locally") - stamp = (watch / "report.txt").stat().st_mtime + 5 - os.utime(watch / "report.txt", (stamp, stamp)) assert engine.poll_once() == ["report.txt"] From 87b8026b2f3cb64e5ac1f8c59a3f5623fe93b917 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 07:26:51 +0800 Subject: [PATCH 08/17] Push a mirrored file edited in the clock tick its state was recorded in The folder mirror called a file changed only when its mtime was later than the recorded one, so an edit landing in the same tick (seen in CI right after a received file was noted) was never pushed. It now compares the recorded time and size, and for a file recorded within two seconds of its own last write also the content hash, until that window has passed. The os.utime workarounds in the tests are gone. --- .../operations_layer/operations_layer_doc.rst | 6 +- .../operations_layer/operations_layer_doc.rst | 3 +- .../utils/remote_desktop/file_sync.py | 95 +++++++++-- .../headless/test_folder_sync_same_tick.py | 152 ++++++++++++++++++ .../headless/test_sync_echo_wiring.py | 7 +- 5 files changed, 242 insertions(+), 21 deletions(-) create mode 100644 test/unit_test/headless/test_folder_sync_same_tick.py diff --git a/docs/source/Eng/doc/operations_layer/operations_layer_doc.rst b/docs/source/Eng/doc/operations_layer/operations_layer_doc.rst index 9b2cc0b7..bc608012 100644 --- a/docs/source/Eng/doc/operations_layer/operations_layer_doc.rst +++ b/docs/source/Eng/doc/operations_layer/operations_layer_doc.rst @@ -43,8 +43,10 @@ Behaviour: - Initial snapshot taken on ``start()`` *without* sending — pre-existing files are treated as already-synced. -- Each tick scans the directory; files with a newer ``mtime`` than the - snapshot are sent. +- Each tick scans the directory; files whose ``mtime`` or size differs from + the snapshot are sent. A file recorded within two seconds of its own last + write is also compared by content, so an edit in the same clock tick is + not missed. - A failing sender is retried on the next tick (the snapshot only records successful sends). - Local deletions stop being tracked but do not call the sender. diff --git a/docs/source/Zh/doc/operations_layer/operations_layer_doc.rst b/docs/source/Zh/doc/operations_layer/operations_layer_doc.rst index b711a577..9b7c43f0 100644 --- a/docs/source/Zh/doc/operations_layer/operations_layer_doc.rst +++ b/docs/source/Zh/doc/operations_layer/operations_layer_doc.rst @@ -40,7 +40,8 @@ Headless:: 行為： - ``start()`` 時建立初始快照但 *不* 傳送 — 既存檔案視為已同步。 -- 每個 tick 掃描資料夾；``mtime`` 較快照新的檔案會被傳送。 +- 每個 tick 掃描資料夾；``mtime`` 或大小與快照不同的檔案會被傳送。在檔案自己最後一次寫入後 + 兩秒內被記錄的檔案還會比對內容，所以同一個時間刻度內的編輯不會漏掉。 - 傳送失敗會在下一個 tick 重試（快照只記錄成功的傳送）。 - 本地刪除會停止追蹤但不會呼叫 sender。 diff --git a/je_auto_control/utils/remote_desktop/file_sync.py b/je_auto_control/utils/remote_desktop/file_sync.py index a93bca6f..3b901e2c 100644 --- a/je_auto_control/utils/remote_desktop/file_sync.py +++ b/je_auto_control/utils/remote_desktop/file_sync.py @@ -30,11 +30,20 @@ place; every live engine whose folder holds it records the content's hash through :meth:`FolderSyncEngine.note_received` and leaves the file alone until it changes locally. + +**A change is anything that differs from what was last recorded.** The engine +remembers each file's modification time and size and pushes a file whose +either one differs -- including a file put back with an *older* time. A time +alone cannot tell two writes in one clock tick apart, so a file recorded +within :data:`RACY_WINDOW_S` of its own last write is also remembered by +content hash and compared by content until that window has passed; an older +file is never read to decide whether it changed. """ from __future__ import annotations import hashlib import threading +import time import weakref from pathlib import Path from typing import Callable, Dict, List, NamedTuple, Optional, Sequence, Tuple, Union @@ -48,16 +57,42 @@ #: Name endings that mark a file as still being written; never mirrored. IN_PROGRESS_SUFFIXES = (".part", ".partial", ".tmp", ".crdownload") +#: A file recorded this soon after its last write is compared by content too: +#: within one tick of the file system's clock (2 s on FAT, far less elsewhere) +#: a second write leaves the modification time as it was. +RACY_WINDOW_S = 2.0 +_RACY_WINDOW_NS = int(RACY_WINDOW_S * 1_000_000_000) + _ENGINES: "weakref.WeakSet[FolderSyncEngine]" = weakref.WeakSet() _ENGINES_LOCK = threading.Lock() +def _wall_ns() -> int: + """The wall clock in nanoseconds, the unit file times are compared in.""" + return time.time_ns() + + class _FileState(NamedTuple): """What one scan saw of a file: its mtime, and (mtime ns, size) to compare.""" mtime: float signature: Tuple[int, int] +class _Known(NamedTuple): + """What the engine last recorded of a file it considers in sync. + + ``digest`` is set only while the file's time cannot tell a later write + apart from the recorded one (see :data:`RACY_WINDOW_S`). + """ + signature: Tuple[int, int] + digest: Optional[str] = None + + +def _is_racy(signature: Tuple[int, int], now_ns: int) -> bool: + """Whether a write made now could still carry this file's recorded time.""" + return signature[0] + _RACY_WINDOW_NS >= now_ns + + def _file_digest(path: Path) -> Optional[str]: """The SHA-256 of a file read in chunks; ``None`` when it cannot be read.""" digest = hashlib.sha256() @@ -94,7 +129,7 @@ def __init__(self, *, watch_dir: Path, self._include_subdirs = bool(include_subdirs) self._wait_until_stable = bool(wait_until_stable) self._ignore_suffixes = tuple(suffix.lower() for suffix in ignore_suffixes) - self._snapshot: Dict[str, float] = {} # rel_path -> mtime + self._snapshot: Dict[str, _Known] = {} # rel_path -> what was last in sync # rel_path -> what the last poll saw of a changed file not pushed yet self._settling: Dict[str, Tuple[int, int]] = {} self._received: Dict[str, str] = {} # rel_path -> sha256 of what the peer sent @@ -189,17 +224,41 @@ def note_received(self, remote_name: str, *, sha256: Optional[str] = None) -> No def _digest(path: Path) -> Optional[str]: return _file_digest(path) - def _is_echo(self, rel: str) -> bool: - """Whether ``rel`` still holds exactly what the peer sent.""" + def _echo_digest(self, rel: str) -> Optional[str]: + """The hash ``rel`` holds if that is still exactly what the peer sent.""" with self._received_lock: noted = self._received.get(rel) if noted is None: - return False + return None if self._digest(self._watch / rel) == noted: - return True + return noted # Edited here since it arrived: it is a local change again. with self._received_lock: self._received.pop(rel, None) + return None + + def _known(self, rel: str, state: _FileState, now_ns: int, + digest: Optional[str] = None) -> _Known: + """The record of ``rel`` as it is now, with its hash while its time is racy.""" + if not _is_racy(state.signature, now_ns): + return _Known(state.signature) + return _Known(state.signature, digest or self._digest(self._watch / rel)) + + def _has_changed(self, rel: str, state: _FileState, now_ns: int) -> bool: + """Whether ``rel`` differs from what was last recorded as in sync.""" + known = self._snapshot.get(rel) + if known is None or known.signature != state.signature: + return True + if known.digest is None: + return False + digest = self._digest(self._watch / rel) + if digest is None: + return False # unreadable right now; looked at again next poll + if digest != known.digest: + return True + if not _is_racy(state.signature, now_ns): + # From here on a write gets a later time: stop reading the file. + self._snapshot[rel] = _Known(state.signature) return False def _has_settled(self, rel: str, state: _FileState) -> bool: @@ -217,17 +276,20 @@ def poll_once(self) -> List[str]: engine themselves can call it directly. """ current = self._scan() + now_ns = _wall_ns() if not self._ready.is_set(): - self._snapshot = {rel: state.mtime for rel, state in current.items()} + self._snapshot = {rel: self._known(rel, state, now_ns) + for rel, state in current.items()} self._ready.set() return [] pushed: List[str] = [] changed = [(rel, state) for rel, state in current.items() - if self._snapshot.get(rel, float("-inf")) < state.mtime] + if self._has_changed(rel, state, now_ns)] for rel, state in changed: - if self._is_echo(rel): - self._snapshot[rel] = state.mtime - elif self._has_settled(rel, state) and self._push(rel, state.mtime): + echoed = self._echo_digest(rel) + if echoed is not None: + self._snapshot[rel] = self._known(rel, state, now_ns, echoed) + elif self._has_settled(rel, state) and self._push(rel, state, now_ns): pushed.append(rel) self._forget(current, {rel for rel, _state in changed}) return pushed @@ -240,19 +302,22 @@ def _forget(self, current: Dict[str, _FileState], waiting: set) -> None: # retry promise made in this engine's docstring. Successful # sends already updated ``_snapshot[rel]``. self._snapshot = { - rel: mtime for rel, mtime in self._snapshot.items() + rel: known for rel, known in self._snapshot.items() if rel in current } self._settling = {rel: seen for rel, seen in self._settling.items() if rel in waiting} - def _push(self, rel: str, mtime: float) -> bool: + def _push(self, rel: str, state: _FileState, now_ns: int) -> bool: + # Hashed before the send: a write that slips in during it is then a + # difference from this record, and goes out on the next poll. + known = self._known(rel, state, now_ns) try: self._sender(str(self._watch / rel), rel) except (RuntimeError, OSError, ValueError) as error: # _settling keeps the file's state, so the retry is the next tick. autocontrol_logger.warning("folder sync push %s: %r", rel, error) return False - self._snapshot[rel] = mtime + self._snapshot[rel] = known self._settling.pop(rel, None) autocontrol_logger.info("folder sync: pushed %s", rel) return True @@ -315,4 +380,6 @@ def note_incoming(final_path: Union[str, Path], return len(covering) -__all__ = ["FolderSyncEngine", "IN_PROGRESS_SUFFIXES", "live_engines", "note_incoming"] +__all__ = [ + "FolderSyncEngine", "IN_PROGRESS_SUFFIXES", "RACY_WINDOW_S", "live_engines", "note_incoming", +] diff --git a/test/unit_test/headless/test_folder_sync_same_tick.py b/test/unit_test/headless/test_folder_sync_same_tick.py new file mode 100644 index 00000000..f718053a --- /dev/null +++ b/test/unit_test/headless/test_folder_sync_same_tick.py @@ -0,0 +1,152 @@ +"""The folder mirror notices an edit made in the clock tick it last looked in. + +``FolderSyncEngine`` called a file changed only when its modification time was +*later* than the one it had recorded. A local edit that landed in the same +tick as the recorded state -- seen in CI right after a received file had been +noted as an echo -- kept the same time and was never pushed. The engine now +compares what it recorded (time and size), and for a file recorded so soon +after its last write that the time cannot tell two writes apart, its content. + +The clock is never raced here: the file's time is pinned with ``os.utime`` +and the engine's idea of "now" is replaced. +""" +import hashlib +import os + +import pytest + +from je_auto_control.utils.remote_desktop import file_sync +from je_auto_control.utils.remote_desktop.file_sync import FolderSyncEngine + +_TICK_NS = 1_700_000_000_000_000_000 +_SECOND_NS = 1_000_000_000 + + +@pytest.fixture +def watch(tmp_path): + folder = tmp_path / "watch" + folder.mkdir() + return folder + + +@pytest.fixture +def clock(monkeypatch): + """The engine's wall clock, as a one-item list the test moves.""" + now = [_TICK_NS] + monkeypatch.setattr(file_sync, "_wall_ns", lambda: now[0]) + return now + + +def _write(path, data, tick_ns=_TICK_NS): + """Write ``data`` and pin the file's time, so two writes share one tick.""" + path.write_bytes(data) + os.utime(path, ns=(tick_ns, tick_ns)) + + +def _engine(watch, sent, **options): + options.setdefault("wait_until_stable", False) + engine = FolderSyncEngine(watch_dir=watch, sender=lambda _path, name: sent.append(name), + **options) + engine.poll_once() # the baseline + return engine + + +def test_an_edit_of_another_size_in_the_same_tick_is_pushed(watch, clock): + sent = [] + engine = _engine(watch, sent) + _write(watch / "a.txt", b"first") + assert engine.poll_once() == ["a.txt"] + _write(watch / "a.txt", b"second, and longer") + assert engine.poll_once() == ["a.txt"] + assert engine.poll_once() == [] and sent == ["a.txt", "a.txt"] + + +def test_an_edit_of_the_same_size_in_the_same_tick_is_pushed(watch, clock): + sent = [] + engine = _engine(watch, sent) + _write(watch / "a.txt", b"first") + assert engine.poll_once() == ["a.txt"] + _write(watch / "a.txt", b"FIRST") + assert engine.poll_once() == ["a.txt"], "same time, same size: only the content differs" + assert engine.poll_once() == [] + + +def test_an_edit_in_the_tick_a_received_file_was_noted_in_is_pushed(watch, clock): + sent = [] + engine = _engine(watch, sent) + incoming = b"from peer" + engine.note_received("r.txt", sha256=hashlib.sha256(incoming).hexdigest()) + _write(watch / "r.txt", incoming) + assert engine.poll_once() == [] and sent == [], "what the peer sent is not pushed back" + _write(watch / "r.txt", b"FROM HERE") # same size, same tick + assert engine.poll_once() == ["r.txt"] + assert engine.poll_once() == [] + + +def test_a_file_already_there_at_the_baseline_and_edited_in_that_tick_is_pushed(watch, clock): + _write(watch / "old.txt", b"before") + sent = [] + engine = _engine(watch, sent) + assert engine.poll_once() == [] + _write(watch / "old.txt", b"BEFORE") + assert engine.poll_once() == ["old.txt"] + + +def test_a_file_put_back_with_an_older_time_is_pushed(watch, clock): + sent = [] + engine = _engine(watch, sent) + _write(watch / "a.txt", b"new copy") + assert engine.poll_once() == ["a.txt"] + _write(watch / "a.txt", b"restored", tick_ns=_TICK_NS - 3600 * _SECOND_NS) + assert engine.poll_once() == ["a.txt"], "an older time is still a different file" + + +def test_a_failed_push_in_the_same_tick_is_still_retried(watch, clock): + attempts = [] + + def flaky(_path, name): + attempts.append(name) + if len(attempts) == 1: + raise OSError("transient") + + engine = FolderSyncEngine(watch_dir=watch, sender=flaky, wait_until_stable=False) + engine.poll_once() + _write(watch / "a.txt", b"data") + assert engine.poll_once() == [] + assert engine.poll_once() == ["a.txt"] and attempts == ["a.txt", "a.txt"] + + +def test_waiting_until_stable_still_holds_a_changing_file_back(watch, clock): + sent = [] + engine = _engine(watch, sent, wait_until_stable=True) + _write(watch / "big.bin", b"half") + assert engine.poll_once() == [], "first sight: it may still be growing" + _write(watch / "big.bin", b"half and the rest") + assert engine.poll_once() == [], "it changed since the last poll" + assert engine.poll_once() == ["big.bin"] + assert engine.poll_once() == [] + + +def test_content_is_read_only_while_the_time_cannot_be_trusted(watch, clock, monkeypatch): + read = [] + real = file_sync._file_digest + + def counting(path): + read.append(path.name) + return real(path) + + monkeypatch.setattr(file_sync, "_file_digest", counting) + sent = [] + engine = _engine(watch, sent) + _write(watch / "a.txt", b"first") + assert engine.poll_once() == ["a.txt"] + assert engine.poll_once() == [] and read, "written this tick: compared by content" + # Long after the write, a later edit cannot share its tick any more. + clock[0] = _TICK_NS + 60 * _SECOND_NS + assert engine.poll_once() == [] + del read[:] + assert engine.poll_once() == [] and read == [], "an old file is compared by time and size" + # A file that was already old when first seen is never read at all. + _write(watch / "old.txt", b"old", tick_ns=_TICK_NS - 3600 * _SECOND_NS) + assert engine.poll_once() == ["old.txt"] + assert engine.poll_once() == [] and read == [] diff --git a/test/unit_test/headless/test_sync_echo_wiring.py b/test/unit_test/headless/test_sync_echo_wiring.py index f542df27..dcdabb29 100644 --- a/test/unit_test/headless/test_sync_echo_wiring.py +++ b/test/unit_test/headless/test_sync_echo_wiring.py @@ -118,11 +118,10 @@ def test_a_file_the_webrtc_receiver_writes_into_a_mirror_is_not_pushed_back(watc receiver.handle_message(json.dumps({"type": "file_end"}), on_done=done.append) assert done and (watch / "from-peer.txt").read_bytes() == b"hello" assert _polls(engine) == [] and sent == [] - # A local edit afterwards is this machine's change again. + # A local edit afterwards is this machine's change again -- even in the + # clock tick the file arrived in: the mirror compares what it recorded, + # not only whether the time moved on. (watch / "from-peer.txt").write_bytes(b"edited here") - # The mirror compares modification times; an edit in the same clock tick as - # the arrival would not read as newer. - os.utime(watch / "from-peer.txt", (time.time() + 5, time.time() + 5)) assert _polls(engine) == ["from-peer.txt"] From f942763b5fa678e431edcb011407d04885303247 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 07:26:51 +0800 Subject: [PATCH 09/17] Refuse an oversized blob before uploading it and collect the blobs nothing names The server answers an oversized PUT from its declared length and closes; a client still sending could see the reset first, so the per-file reason read as a connection error. HttpAssetTransport now reads max_blob_bytes from the listing and refuses locally. Nothing ever deleted a blob: config_sync_collect_blobs removes the ones no entry names (AC_config_sync_collect_blobs, ac_config_sync_collect_blobs), going by the age the listing now reports. A blob write also holds a lock file, so two server processes sharing a folder share one quota. --- je_auto_control/__init__.py | 5 + .../gui/script_builder/command_schema.py | 12 + je_auto_control/utils/config_sync/__init__.py | 14 +- je_auto_control/utils/config_sync/assets.py | 201 ++++++++- je_auto_control/utils/config_sync/blobs.py | 51 ++- je_auto_control/utils/config_sync/session.py | 84 +++- .../utils/executor/action_executor.py | 4 +- .../utils/mcp_server/tools/_factories.py | 18 + .../utils/remote_desktop/signaling_server.py | 3 +- .../test_config_sync_blob_housekeeping.py | 415 ++++++++++++++++++ .../headless/test_config_sync_blobs.py | 23 +- 11 files changed, 795 insertions(+), 35 deletions(-) create mode 100644 test/unit_test/headless/test_config_sync_blob_housekeeping.py diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py index b23733db..260f6290 100644 --- a/je_auto_control/__init__.py +++ b/je_auto_control/__init__.py @@ -288,6 +288,10 @@ HotkeySyncAdapter, HttpAssetTransport, LocatorSyncAdapter, OperationMismatchError, RevisionConflictError, ScriptSyncAdapter, TriggerSyncAdapter, run_sync, ) +# ... and blob housekeeping: the server's blobs that no entry names any more +from je_auto_control.utils.config_sync import ( + collect_unreferenced_blobs, config_sync_collect_blobs, referenced_blob_digests, +) # The /config wire format; named for what it versions (the module calls it WIRE_VERSION) from je_auto_control.utils.config_sync import WIRE_VERSION as CONFIG_SYNC_WIRE_VERSION # Step-through debugger / tracer for action lists @@ -1811,6 +1815,7 @@ def start_autocontrol_gui(*args, **kwargs): "DirectoryAssetTransport", "HotkeySyncAdapter", "HttpAssetTransport", "LocatorSyncAdapter", "OperationMismatchError", "RevisionConflictError", "ScriptSyncAdapter", "TriggerSyncAdapter", "run_sync", + "collect_unreferenced_blobs", "config_sync_collect_blobs", "referenced_blob_digests", "ConfigField", "ConfigSchema", "coerce", "validate_config", "PathNotAllowedError", "PathPolicy", "RefResolver", "SecretRefError", "is_ref", "resolve_ref", "resolve_refs_in", diff --git a/je_auto_control/gui/script_builder/command_schema.py b/je_auto_control/gui/script_builder/command_schema.py index 122684c4..6b1cfb15 100644 --- a/je_auto_control/gui/script_builder/command_schema.py +++ b/je_auto_control/gui/script_builder/command_schema.py @@ -3616,6 +3616,18 @@ def _add_resilience_specs(specs: List[CommandSpec]) -> None: description="Adopt the server's state after this device was retired; " "pending local changes are discarded.", )) + specs.append(CommandSpec( + "AC_config_sync_collect_blobs", "Data", "Config Sync: Collect Unused Blobs", + fields=sync_target + ( + FieldSpec("keep", FieldType.STRING, optional=True, + placeholder="sha256,sha256"), + FieldSpec("min_age_s", FieldType.FLOAT, optional=True, default=86400.0), + FieldSpec("dry_run", FieldType.BOOL, optional=True, default=False), + ) + sync_options, + description="Delete the account's asset blobs on the sync server that no " + "entry names any more. Blobs newer than min_age_s and the " + "digests in 'keep' stay; dry_run only lists what would go.", + )) specs.append(CommandSpec( "AC_detect_drift", "Data", "Data Drift: Detect (PSI + KS)", fields=( diff --git a/je_auto_control/utils/config_sync/__init__.py b/je_auto_control/utils/config_sync/__init__.py index 5f15702b..7fc82d33 100644 --- a/je_auto_control/utils/config_sync/__init__.py +++ b/je_auto_control/utils/config_sync/__init__.py @@ -34,6 +34,8 @@ (:class:`HttpAssetTransport`). * :mod:`.blobs` -- :class:`BlobStore`, what the server keeps behind ``/blobs``: per account, content-addressed, size-capped, with a quota. + :func:`collect_unreferenced_blobs` (and :func:`config_sync_collect_blobs`, + which works out what is still referenced) removes the ones no entry names. * :mod:`.device` -- :func:`default_device_id`, this machine's stable id. * :mod:`.session` -- :func:`run_sync`, the whole cycle behind the GUI tab, the ``AC_config_sync_*`` commands and the MCP tools. @@ -57,8 +59,9 @@ ScriptSyncAdapter, SyncAdapter, TriggerSyncAdapter, ) from je_auto_control.utils.config_sync.assets import ( - AssetManifest, AssetRef, AssetSyncError, AssetSyncResult, AssetTransport, - DirectoryAssetTransport, HttpAssetTransport, publish_assets, sync_assets, + DEFAULT_BLOB_GRACE_S, AssetManifest, AssetRef, AssetSyncError, AssetSyncResult, + AssetTransport, DirectoryAssetTransport, HttpAssetTransport, collect_unreferenced_blobs, + publish_assets, sync_assets, ) from je_auto_control.utils.config_sync.blobs import BlobStore, BlobStoreError from je_auto_control.utils.config_sync.client import ( @@ -71,8 +74,9 @@ DrainReport, OutboxError, SyncOutbox, default_outbox_path, ) from je_auto_control.utils.config_sync.session import ( - SYNCABLE_SECTIONS, SyncRunReport, config_sync_full_resync, config_sync_resolve, - config_sync_run, config_sync_status, default_adapters, default_device_id, resolve_conflict, + SYNCABLE_SECTIONS, SyncRunReport, config_sync_collect_blobs, config_sync_full_resync, + config_sync_resolve, config_sync_run, config_sync_status, default_adapters, + default_device_id, referenced_blob_digests, resolve_conflict, resolve_sections, run_full_resync, run_sync, sync_status, ) from je_auto_control.utils.config_sync.store import ( @@ -93,6 +97,8 @@ "config_sync_status", "default_adapters", "default_device_id", "publish_assets", "resolve_conflict", "run_full_resync", "run_sync", "sync_assets", "sync_status", "SYNCABLE_SECTIONS", "resolve_sections", + "DEFAULT_BLOB_GRACE_S", "collect_unreferenced_blobs", "config_sync_collect_blobs", + "referenced_blob_digests", "ConfigBucket", "ConfigStore", "ConfigStoreError", "ConflictRecord", "ConfigSyncClient", "ConfigSyncConflict", "ConfigSyncError", "DEFAULT_SYNC_ATTEMPTS", "DrainReport", "FullResyncRequired", "MergeDecision", "OperationMismatchError", "OutboxError", "PeerState", diff --git a/je_auto_control/utils/config_sync/assets.py b/je_auto_control/utils/config_sync/assets.py index 93d46222..1573cbb3 100644 --- a/je_auto_control/utils/config_sync/assets.py +++ b/je_auto_control/utils/config_sync/assets.py @@ -12,6 +12,7 @@ from __future__ import annotations import hashlib +import json import threading import urllib.parse from dataclasses import dataclass, field @@ -22,6 +23,10 @@ from je_auto_control.utils.exception.exceptions import AutoControlException _SHA256_HEX_CHARS = 64 +#: Blobs stored more recently than this are left alone by +#: :func:`collect_unreferenced_blobs`: a machine uploads a file just before it +#: commits the entry that refers to it. +DEFAULT_BLOB_GRACE_S = 24 * 3600.0 class AssetSyncError(ConfigSyncError): @@ -151,6 +156,7 @@ def has(self, sha256: str) -> bool: return self._blob(sha256).is_file() +_LISTING = "listing" #: Why the server refused a blob request, by status. _HTTP_REFUSALS = { 400: "the server rejected the blob (bad digest or content)", @@ -175,6 +181,16 @@ class HttpAssetTransport: an account may hold; a file over either is reported per file by :func:`publish_assets` rather than stopping the others. + The per-blob limit is read from the account's listing (``GET + /blobs/{user_id}``) before the first upload and remembered, and a file + over it is refused here without being sent. That matters for more than + the saved upload: the server answers an oversized ``PUT`` from its + declared length and closes the connection, and a client still sending + the body can see the reset instead of the ``413`` -- the failure then + read as a connection error. A server that does not say its limit is + simply sent the file; if that upload is cut off the limit is asked for + once more, so "too large" is still the reason given when it is the reason. + Requests go through :mod:`je_auto_control.utils.http_client`, so the egress policy applies, and redirects are not followed -- the secret is never carried to another host. @@ -190,26 +206,87 @@ def __init__(self, server_url: str, *, user_id: str, secret: Optional[str] = Non self._user_id = user_id self._secret = secret self._timeout = float(timeout_s) + self._max_blob_bytes: Optional[int] = None + self._limit_asked = False - def _url(self, sha256: str) -> str: + def _listing_url(self) -> str: account = urllib.parse.quote(self._user_id, safe="") - return f"{self._server_url}/blobs/{account}/{_checked_digest(sha256)}" + return f"{self._server_url}/blobs/{account}" + + def _url(self, sha256: str) -> str: + return f"{self._listing_url()}/{_checked_digest(sha256)}" - def _request(self, method: str, sha256: str, data: Optional[bytes] = None) -> Dict[str, Any]: + def _request(self, method: str, sha256: Optional[str], + data: Optional[bytes] = None) -> Dict[str, Any]: + """One request about blob ``sha256``, or about the account's listing (``None``).""" from je_auto_control.utils.http_client.http_client import build_call, perform_call headers = {"Content-Type": "application/octet-stream"} if data is not None else {} if self._secret: headers["X-Signaling-Secret"] = self._secret try: - call = build_call(self._url(sha256), method=method, headers=headers, data=data, + url = self._listing_url() if sha256 is None else self._url(sha256) + call = build_call(url, method=method, headers=headers, data=data, timeout=self._timeout) call["follow_redirects"] = False - call["want_bytes"] = method == "GET" + call["want_bytes"] = method == "GET" and sha256 is not None return perform_call(call) except AssetSyncError: raise except (OSError, ValueError, AutoControlException) as error: - raise AssetSyncError(f"asset {method} {sha256} failed: {error}") from error + raise AssetSyncError( + f"asset {method} {sha256 or _LISTING} failed: {error}") from error + + @staticmethod + def _json(response: Mapping[str, Any], what: str) -> Dict[str, Any]: + try: + body = json.loads(response.get("text") or "") + except (json.JSONDecodeError, RecursionError) as error: + raise AssetSyncError(f"asset {what}: the reply was not JSON") from error + if not isinstance(body, dict): + raise AssetSyncError(f"asset {what}: the reply was not a JSON object") + return body + + def usage(self) -> Dict[str, Any]: + """What the account holds on the server, and the server's limits. + + ``{"used", "quota", "count", "max_blob_bytes", "blobs": [{"sha256", + "size", "age_s"}, ...]}`` as the server reports it; a server older + than a field leaves it out. Raises :class:`AssetSyncError` when the + server refuses or does not serve the listing. + """ + response = self._request("GET", None) + status = int(response["status"]) + if status != 200: + raise self._refused("GET", _LISTING, status) + listing = self._json(response, _LISTING) + limit = listing.get("max_blob_bytes") + if isinstance(limit, int) and not isinstance(limit, bool) and limit > 0: + self._max_blob_bytes = limit + return listing + + def max_blob_bytes(self, *, refresh: bool = False) -> Optional[int]: + """The largest blob the server accepts; ``None`` when it does not say. + + Asked once and remembered (``refresh`` asks again). A listing that + cannot be had -- the server is unreachable, or predates it -- is not + an error here: the limit is then unknown. + """ + if refresh or not self._limit_asked: + self._limit_asked = True + try: + self.usage() + except AssetSyncError: + pass + return self._max_blob_bytes + + @staticmethod + def _require_fits(sha256: str, size: int, limit: Optional[int], + cause: Optional[BaseException] = None) -> None: + """Raise the "too large" refusal when ``size`` is over a known ``limit``.""" + if limit is not None and size > limit: + raise AssetSyncError( + f"asset PUT {sha256}: {_HTTP_REFUSALS[413]} " + f"({size} bytes; the limit is {limit})") from cause @staticmethod def _refused(method: str, sha256: str, status: int) -> AssetSyncError: @@ -230,10 +307,22 @@ def fetch(self, sha256: str) -> bytes: return bytes(content) def store(self, sha256: str, data: bytes) -> None: - """Upload ``data`` under ``sha256``; the server checks the hash too.""" - status = int(self._request("PUT", sha256, bytes(data))["status"]) + """Upload ``data`` under ``sha256``; the server checks the hash too. + + A file over the server's per-blob limit is refused without being + sent, and always with the same reason (see the class docstring). + """ + digest = _checked_digest(sha256) + body = bytes(data) + self._require_fits(digest, len(body), self.max_blob_bytes()) + try: + status = int(self._request("PUT", digest, body)["status"]) + except AssetSyncError as error: + # Cut off while sending: was it the size? Ask the limit afresh. + self._require_fits(digest, len(body), self.max_blob_bytes(refresh=True), error) + raise if status not in (200, 201): - raise self._refused("PUT", sha256, status) + raise self._refused("PUT", digest, status) def has(self, sha256: str) -> bool: """Whether the server already holds the blob (so it need not be sent).""" @@ -244,6 +333,16 @@ def has(self, sha256: str) -> bool: return False raise self._refused("HEAD", sha256, status) + def delete(self, sha256: str) -> bool: + """Remove the account's blob from the server; ``False`` when it held none.""" + response = self._request("DELETE", sha256) + status = int(response["status"]) + if status == 404: + return False + if status != 200: + raise self._refused("DELETE", sha256, status) + return self._json(response, f"DELETE {sha256}").get("deleted") is True + @dataclass class AssetSyncResult: @@ -337,8 +436,88 @@ def publish_assets(manifest: AssetManifest, transport: AssetTransport, *, return result +def _checked_grace(min_age_s: Any) -> float: + if isinstance(min_age_s, bool) or not isinstance(min_age_s, (int, float)) \ + or not min_age_s >= 0: + raise AssetSyncError(f"min_age_s must be a number >= 0, got {min_age_s!r}") + return float(min_age_s) + + +def _old_enough(age_s: Any, min_age_s: float) -> bool: + """Whether a listed blob is past the grace period (an unknown age never is).""" + if min_age_s <= 0: + return True + return isinstance(age_s, (int, float)) and not isinstance(age_s, bool) and age_s >= min_age_s + + +def _listed(blob: Any) -> Optional[Tuple[str, int, Any]]: + """``(digest, size, age)`` of one listing row; ``None`` for a row that is not one.""" + if not isinstance(blob, Mapping): + return None + try: + digest = _checked_digest(blob.get("sha256")) + except AssetSyncError: + return None + size = blob.get("size") + if isinstance(size, bool) or not isinstance(size, int): + size = 0 + return digest, size, blob.get("age_s") + + +def collect_unreferenced_blobs(transport: HttpAssetTransport, referenced: Iterable[str], *, + min_age_s: float = DEFAULT_BLOB_GRACE_S, + dry_run: bool = False) -> Dict[str, Any]: + """Delete the account's blobs on the server that ``referenced`` does not name. + + ``referenced`` is every SHA-256 still in use -- for config sync, the + digests the bucket's entries name + (:func:`~je_auto_control.utils.config_sync.session.referenced_blob_digests`; + :func:`~je_auto_control.utils.config_sync.session.config_sync_collect_blobs` + gathers them for you). A blob stored less than ``min_age_s`` ago is kept + whatever it is: a machine uploads a file just before it commits the entry + naming it, and that entry is not in anybody's bucket yet. The age is the + server's own figure, so no two clocks are compared; a server that does + not report ages has nothing collected unless ``min_age_s`` is ``0``. + + Returns ``{"deleted": [sha256...], "freed": bytes, "kept": count, + "recent": [sha256...], "failed": {sha256: why}, "dry_run": bool}``. With + ``dry_run`` nothing is deleted and ``deleted`` lists what would be. + + For :class:`HttpAssetTransport` only: its blobs belong to one account. A + folder behind :class:`DirectoryAssetTransport` may hold other users' + files, which nothing here could tell apart. + """ + grace = _checked_grace(min_age_s) + keep = {_checked_digest(digest) for digest in referenced} + rows = transport.usage().get("blobs") + result: Dict[str, Any] = {"deleted": [], "freed": 0, "kept": 0, "recent": [], + "failed": {}, "dry_run": bool(dry_run)} + for digest, size, age in filter(None, map(_listed, rows if isinstance(rows, list) else ())): + if digest in keep: + result["kept"] += 1 + elif not _old_enough(age, grace): + result["recent"].append(digest) + elif _removed(transport, digest, dry_run, result["failed"]): + result["deleted"].append(digest) + result["freed"] += size + return result + + +def _removed(transport: HttpAssetTransport, digest: str, dry_run: bool, + failed: Dict[str, str]) -> bool: + """Delete one blob (unless ``dry_run``); a failure is recorded, not raised.""" + if dry_run: + return True + try: + transport.delete(digest) + except AssetSyncError as error: + failed[digest] = str(error) + return False + return True + + __all__ = [ "AssetManifest", "AssetRef", "AssetSyncError", "AssetSyncResult", "AssetTransport", - "DirectoryAssetTransport", "HttpAssetTransport", "file_sha256", "publish_assets", - "sync_assets", + "DEFAULT_BLOB_GRACE_S", "DirectoryAssetTransport", "HttpAssetTransport", + "collect_unreferenced_blobs", "file_sha256", "publish_assets", "sync_assets", ] diff --git a/je_auto_control/utils/config_sync/blobs.py b/je_auto_control/utils/config_sync/blobs.py index 0261f803..700de17b 100644 --- a/je_auto_control/utils/config_sync/blobs.py +++ b/je_auto_control/utils/config_sync/blobs.py @@ -12,6 +12,13 @@ stored under the digest of its own bytes -- and is written to a temporary file and renamed, so a reader never sees half of one. +The quota check and the write are one step for every process that shares the +folder: a write holds the lock file ``/store.lock`` (the helper the +shared JSON stores use), so two server processes cannot each admit a blob the +other has not counted. Nothing here deletes a blob by itself; the listing +reports each blob's age so a client can collect the ones no entry refers to +(:func:`~je_auto_control.utils.config_sync.assets.collect_unreferenced_blobs`). + Pure standard library; imports no ``PySide6``. """ from __future__ import annotations @@ -20,6 +27,7 @@ import os import secrets import threading +import time from pathlib import Path from typing import Any, Dict, List, Optional, Union @@ -56,6 +64,10 @@ class BlobCapacityError(BlobStoreError): """A new account's first blob would exceed the store's account limit.""" +class BlobStoreBusyError(BlobStoreError): + """Another process held the store's lock file for too long; nothing was written.""" + + def default_blob_dir() -> Path: """``~/.je_auto_control/config_sync_blobs``, resolved at call time.""" return Path.home() / ".je_auto_control" / "config_sync_blobs" @@ -86,6 +98,8 @@ def __init__(self, root: Union[str, Path, None] = None, *, self._quota_bytes = max(1, int(quota_bytes)) self._max_users = max(1, int(max_users)) # One writer at a time: the quota check and the write are one step. + # This lock is for the threads of one process; the lock file taken in + # put() is for the other processes sharing the folder. self._lock = threading.Lock() @property @@ -139,15 +153,25 @@ def get(self, user_id: str, sha256: Any) -> Optional[bytes]: raise BlobStoreError(f"cannot read blob {path.name}: {error}") from error def usage(self, user_id: str) -> Dict[str, Any]: - """``{"used", "quota", "count", "max_blob_bytes", "blobs"}`` for the account.""" + """``{"used", "quota", "count", "max_blob_bytes", "blobs"}`` for the account. + + Each of ``blobs`` is ``{"sha256", "size", "age_s"}``; ``age_s`` is how + long ago, by this machine's clock, the blob was last stored (storing + content the account already holds counts), so a reader on another + machine needs no clock of its own to tell an old blob from a new one. + """ try: - sizes = {entry.name: entry.stat().st_size + stats = {entry.name: entry.stat() for entry in self._entries(self._account_dir(user_id))} except OSError as error: raise BlobStoreError(f"cannot list blobs: {error}") from error - blobs = [{"sha256": digest, "size": size} for digest, size in sorted(sizes.items())] - return {"used": sum(sizes.values()), "quota": self._quota_bytes, - "count": len(blobs), "max_blob_bytes": self._max_blob_bytes, "blobs": blobs} + now = time.time() + blobs = [{"sha256": digest, "size": stat.st_size, + "age_s": max(0.0, now - stat.st_mtime)} + for digest, stat in sorted(stats.items())] + return {"used": sum(stat.st_size for stat in stats.values()), + "quota": self._quota_bytes, "count": len(blobs), + "max_blob_bytes": self._max_blob_bytes, "blobs": blobs} def put(self, user_id: str, sha256: Any, data: bytes) -> bool: """Keep ``data`` under ``sha256``; whether it was new to the account. @@ -157,7 +181,10 @@ def put(self, user_id: str, sha256: Any, data: bytes) -> bool: :class:`BlobQuotaError` when the account's total would pass its quota and :class:`BlobCapacityError` when a new account would pass the account limit. Storing content the account already holds - succeeds and writes nothing. + succeeds and writes nothing -- it only marks the blob as stored now, + so housekeeping that goes by age leaves it alone. + :class:`BlobStoreBusyError` when another process kept the store's + lock file for too long. """ digest = checked_blob_digest(sha256) if len(data) > self._max_blob_bytes: @@ -165,16 +192,22 @@ def put(self, user_id: str, sha256: Any, data: bytes) -> bool: f"blob is {len(data)} bytes; the limit is {self._max_blob_bytes}") if hashlib.sha256(data).hexdigest() != digest: raise BlobDigestError("content does not match the SHA-256 it was sent under") + from je_auto_control.utils.json_store.json_store import _file_lock folder = self._account_dir(user_id) with self._lock: try: - return self._write(folder, digest, data) + with _file_lock(self.root / "store"): + return self._write(folder, digest, data) + except TimeoutError as error: + raise BlobStoreBusyError( + f"blob store {self.root} is locked by another process") from error except OSError as error: raise BlobStoreError(f"cannot store blob {digest}: {error}") from error def _write(self, folder: Path, digest: str, data: bytes) -> bool: target = folder / digest if target.is_file(): + os.utime(target) # held already; it is as good as stored now return False used = sum(entry.stat().st_size for entry in self._entries(folder)) if used + len(data) > self._quota_bytes: @@ -215,7 +248,7 @@ def delete(self, user_id: str, sha256: Any) -> bool: __all__ = [ - "BlobCapacityError", "BlobDigestError", "BlobQuotaError", "BlobStore", "BlobStoreError", - "BlobTooLargeError", "DEFAULT_BLOB_QUOTA_BYTES", "DEFAULT_MAX_BLOB_BYTES", + "BlobCapacityError", "BlobDigestError", "BlobQuotaError", "BlobStore", "BlobStoreBusyError", + "BlobStoreError", "BlobTooLargeError", "DEFAULT_BLOB_QUOTA_BYTES", "DEFAULT_MAX_BLOB_BYTES", "DEFAULT_MAX_BLOB_USERS", "checked_blob_digest", "default_blob_dir", ] diff --git a/je_auto_control/utils/config_sync/session.py b/je_auto_control/utils/config_sync/session.py index 845a324b..b0635b69 100644 --- a/je_auto_control/utils/config_sync/session.py +++ b/je_auto_control/utils/config_sync/session.py @@ -19,15 +19,15 @@ import threading import time from dataclasses import dataclass, field -from typing import Any, Dict, List, Mapping, Optional, Sequence, Tuple +from typing import Any, Dict, Iterable, List, Mapping, Optional, Sequence, Set, Tuple from je_auto_control.utils.config_sync.adapters import ( AddressBookSyncAdapter, HotkeySyncAdapter, LocatorSyncAdapter, ScriptSyncAdapter, SyncAdapter, TriggerSyncAdapter, ) from je_auto_control.utils.config_sync.assets import ( - AssetManifest, AssetTransport, DirectoryAssetTransport, HttpAssetTransport, - publish_assets, sync_assets, + DEFAULT_BLOB_GRACE_S, AssetManifest, AssetTransport, DirectoryAssetTransport, + HttpAssetTransport, collect_unreferenced_blobs, publish_assets, sync_assets, ) from je_auto_control.utils.config_sync.client import ( ConfigBucket, ConfigSyncClient, ConfigSyncError, FullResyncRequired, SyncResult, @@ -487,6 +487,78 @@ def _asset_transport(client: ConfigSyncClient, secret=options.get("secret") or os.environ.get("AC_SIGNALING_SECRET") or None) +def _digests_in(node: Any, found: Set[str]) -> None: + """Add every ``sha256`` value found anywhere inside ``node`` to ``found``.""" + if isinstance(node, Mapping): + digest = node.get("sha256") + if isinstance(digest, str) and len(digest) == 64: + found.add(digest.lower()) + for value in node.values(): + _digests_in(value, found) + elif isinstance(node, (list, tuple)): + for value in node: + _digests_in(value, found) + + +def referenced_blob_digests(*buckets: Optional[ConfigBucket], + operations: Iterable[SyncOperation] = ()) -> Set[str]: + """Every SHA-256 the entries of ``buckets`` (and ``operations``) name. + + What blob housekeeping must keep: the digest of each script too large to + travel in its entry, in any section, including every candidate of an + entry still in conflict. A deleted entry names nothing, so its blob is + free to go. ``None`` buckets are skipped. + """ + found: Set[str] = set() + for bucket in buckets: + if bucket is not None: + _digests_in(bucket.sections, found) + for operation in operations: + _digests_in(operation.entry.to_dict(), found) + return found + + +def config_sync_collect_blobs(server_url: str, user_id: str, *, keep: Any = None, + min_age_s: float = DEFAULT_BLOB_GRACE_S, + dry_run: bool = False, **options: Any) -> Dict[str, Any]: + """Delete this account's blobs on the sync server that no entry names any more. + + Nothing on the server removes a blob by itself, so the scripts deleted or + replaced over time keep counting against the account's quota until this + is run. What is kept: every digest named by the server's bucket, by this + machine's last merged state and by its changes still waiting to be sent, + plus ``keep`` (a list, or one comma-separated string of SHA-256 digests) + -- name there whatever you published yourself with + :func:`~je_auto_control.utils.config_sync.assets.publish_assets`, which no + bucket entry refers to. A blob stored less than ``min_age_s`` ago + (default one day) is kept as well: another machine may have uploaded it + for an entry it has not committed yet. + + Returns :func:`~je_auto_control.utils.config_sync.assets.collect_unreferenced_blobs`'s + result plus ``"referenced"``, how many digests were protected. With + ``dry_run`` nothing is deleted. Takes the options of + :func:`config_sync_run`; ``secret``, ``timeout_s`` and ``outbox_path`` + matter. A server that cannot be reached is :class:`ConfigSyncError` and + nothing is deleted. A blob removed while a machine still has the file is + uploaded again by that machine's next sync. + """ + chosen = _chosen(options) + secret = chosen.get("secret") or os.environ.get("AC_SIGNALING_SECRET") or None + client = ConfigSyncClient(server_url, user_id=user_id, secret=secret, + timeout_s=float(chosen.get("timeout_s", 5.0))) + outbox = SyncOutbox(chosen.get("outbox_path"), account=user_id, endpoint=client.server_url) + stored = outbox.exists() + referenced = referenced_blob_digests( + client.fetch(), outbox.load_baseline() if stored else None, + operations=outbox.pending() if stored else ()) + referenced.update(str(digest).lower() for digest in _section_names(keep or ())) + transport = HttpAssetTransport(client.server_url, user_id=user_id, secret=secret, + timeout_s=client.timeout_s) + result = collect_unreferenced_blobs(transport, referenced, min_age_s=min_age_s, + dry_run=bool(dry_run)) + return {**result, "referenced": len(referenced)} + + def config_sync_status(server_url: str, user_id: str, outbox_path: Optional[str] = None, **options: Any) -> Dict[str, Any]: """The recorded sync state for this account and server; no network. @@ -519,8 +591,10 @@ def config_sync_resolve(server_url: str, user_id: str, section: str, key: str, __all__ = [ "DEFAULT_SECTIONS", "STATE_BACKING_OFF", "STATE_CANCELLED", "STATE_CONFLICT", "STATE_OFFLINE", "STATE_PENDING", - "STATE_RESYNC_REQUIRED", "STATE_SYNCED", "SyncRunReport", "config_sync_full_resync", + "STATE_RESYNC_REQUIRED", "STATE_SYNCED", "SyncRunReport", "config_sync_collect_blobs", + "config_sync_full_resync", "SYNCABLE_SECTIONS", "config_sync_resolve", "config_sync_run", "config_sync_status", - "default_adapters", "default_device_id", "default_device_id_path", "resolve_conflict", + "default_adapters", "default_device_id", "default_device_id_path", + "referenced_blob_digests", "resolve_conflict", "resolve_sections", "run_full_resync", "run_sync", "sync_status", ] diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py index dd25d61f..0266b01f 100644 --- a/je_auto_control/utils/executor/action_executor.py +++ b/je_auto_control/utils/executor/action_executor.py @@ -40,7 +40,8 @@ stoppable_run, ) from je_auto_control.utils.config_sync.session import ( - config_sync_full_resync, config_sync_resolve, config_sync_run, config_sync_status, + config_sync_collect_blobs, config_sync_full_resync, config_sync_resolve, + config_sync_run, config_sync_status, ) from je_auto_control.utils.llm.planner import ( plan_actions as llm_plan_actions, @@ -7809,6 +7810,7 @@ def __init__(self): "AC_config_sync_status": config_sync_status, "AC_config_sync_resolve": config_sync_resolve, "AC_config_sync_full_resync": config_sync_full_resync, + "AC_config_sync_collect_blobs": config_sync_collect_blobs, "AC_check_compatibility": _check_compatibility, "AC_ts_rate": _ts_rate, "AC_ts_downsample": _ts_downsample, diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py index c44fb62b..b5a716b0 100644 --- a/je_auto_control/utils/mcp_server/tools/_factories.py +++ b/je_auto_control/utils/mcp_server/tools/_factories.py @@ -9322,6 +9322,24 @@ def config_sync_tools() -> List[MCPTool]: handler=h_sync.config_sync_full_resync, annotations=DESTRUCTIVE, ), + MCPTool( + name="ac_config_sync_collect_blobs", + description=("DELETE this account's asset blobs on the sync server that no " + "entry names any more (scripts deleted or replaced keep " + "counting against the quota until this runs). Kept: every " + "SHA-256 named by the server's bucket, this machine's merged " + "state and its unsent changes, the digests in 'keep', and " + "any blob stored less than 'min_age_s' seconds ago (default " + "86400). dry_run=true deletes nothing and lists what would " + "go. Returns {deleted, freed, kept, recent, failed, dry_run, " + "referenced}."), + input_schema=schema({**target, **options, + "keep": {"type": "array", "items": {"type": "string"}}, + "min_age_s": {"type": "number"}, + "dry_run": {"type": "boolean"}}, required), + handler=h_sync.config_sync_collect_blobs, + annotations=DESTRUCTIVE, + ), ] diff --git a/je_auto_control/utils/remote_desktop/signaling_server.py b/je_auto_control/utils/remote_desktop/signaling_server.py index 8b0116ca..3e4ac55a 100644 --- a/je_auto_control/utils/remote_desktop/signaling_server.py +++ b/je_auto_control/utils/remote_desktop/signaling_server.py @@ -32,7 +32,8 @@ GET /blobs/{user_id}/{sha256} the bytes, or 404 HEAD /blobs/{user_id}/{sha256} 200 / 404, no body DELETE /blobs/{user_id}/{sha256} {"deleted": true | false} - GET /blobs/{user_id} {"used", "quota", "count", "blobs": [...]} + GET /blobs/{user_id} {"used", "quota", "count", "max_blob_bytes", + "blobs": [{"sha256", "size", "age_s"}]} under the rules ``/config`` follows: the shared secret, the account named by the path (one account never reads another's blob), and a size cap checked diff --git a/test/unit_test/headless/test_config_sync_blob_housekeeping.py b/test/unit_test/headless/test_config_sync_blob_housekeeping.py new file mode 100644 index 00000000..82ea4e27 --- /dev/null +++ b/test/unit_test/headless/test_config_sync_blob_housekeeping.py @@ -0,0 +1,415 @@ +"""An upload that is too large says so, and blobs nobody refers to can be collected. + +Three gaps around the sync server's ``/blobs``: + +* The server refuses an oversized ``PUT`` from its declared length and closes. + A client still sending the body could see the connection reset first (it did + on the Windows CI runner), so the per-file reason read as a connection error. + ``HttpAssetTransport`` now reads the limit from the listing and refuses + locally. +* Nothing deleted a blob, ever: every script removed or replaced kept counting + against the account's quota. +* The quota was one process's lock; two server processes sharing a folder + could each admit a blob the other had not counted. +""" +import hashlib +import json +import os +import threading +import time +from unittest.mock import patch + +import pytest + +import je_auto_control as ac +from je_auto_control.utils.config_sync import ( + AssetSyncError, BlobStore, ConfigBucket, ConfigSyncClient, ConfigSyncError, + HttpAssetTransport, SyncEntry, SyncOperation, SyncOutbox, collect_unreferenced_blobs, + config_sync_collect_blobs, referenced_blob_digests, +) +from je_auto_control.utils.config_sync.blobs import BlobQuotaError, BlobStoreBusyError + +_PERFORM = "je_auto_control.utils.http_client.http_client.perform_call" +_DAY = 24 * 3600.0 + + +def _sha(data): + return hashlib.sha256(data).hexdigest() + + +def _reply(status, body=None): + return {"status": status, "content": b"", "text": "" if body is None else json.dumps(body)} + + +def _transport(): + return HttpAssetTransport("https://sync.invalid/", user_id="alice", secret="s3cret") + + +class _Wire: + """A scripted ``perform_call``: what the listing says, and what a PUT does.""" + + def __init__(self, listing=None, put=None): + self.listing = listing + self.put = put if put is not None else _reply(201) + self.calls = [] + self.deleted = [] + + def __call__(self, call): + method, url = call["method"], call["url"] + self.calls.append((method, url.rsplit("/", 1)[-1])) + if url.endswith("/blobs/alice"): + return self._listing() + if method == "PUT": + if isinstance(self.put, BaseException): + raise self.put + return self.put + if method == "DELETE": + self.deleted.append(url.rsplit("/", 1)[-1]) + return _reply(200, {"deleted": True}) + return _reply(404) + + def _listing(self): + listing = self.listing.pop(0) if isinstance(self.listing, list) else self.listing + if isinstance(listing, BaseException): + raise listing + return _reply(404) if listing is None else _reply(200, listing) + + def methods(self): + return [method for method, _name in self.calls] + + +# --- an upload over the server's limit --------------------------------------------- + +def test_a_file_over_the_limit_is_refused_without_being_sent(): + wire = _Wire(listing={"max_blob_bytes": 10, "blobs": []}) + data = b"x" * 11 + with patch(_PERFORM, new=wire): + with pytest.raises(AssetSyncError, match="larger") as refused: + _transport().store(_sha(data), data) + assert "11 bytes; the limit is 10" in str(refused.value) + assert wire.methods() == ["GET"], "the body never left this machine" + + +def test_the_limit_is_asked_for_once(): + wire = _Wire(listing={"max_blob_bytes": 10, "blobs": []}) + with patch(_PERFORM, new=wire): + transport = _transport() + for data in (b"a", b"b", b"c"): + transport.store(_sha(data), data) + assert transport.max_blob_bytes() == 10 + assert wire.methods() == ["GET", "PUT", "PUT", "PUT"] + + +def test_a_file_at_the_limit_is_sent(): + wire = _Wire(listing={"max_blob_bytes": 10, "blobs": []}) + with patch(_PERFORM, new=wire): + _transport().store(_sha(b"x" * 10), b"x" * 10) + assert wire.methods() == ["GET", "PUT"] + + +@pytest.mark.parametrize("listing", [ + None, # a server that does not serve the listing + {"used": 0, "blobs": []}, # ... or one that does not say its limit + {"max_blob_bytes": "big"}, {"max_blob_bytes": True}, {"max_blob_bytes": 0}, + OSError("unreachable"), +]) +def test_a_server_that_does_not_say_its_limit_is_simply_sent_the_file(listing): + wire = _Wire(listing=listing, put=_reply(413)) + with patch(_PERFORM, new=wire): + transport = _transport() + assert transport.max_blob_bytes() is None + with pytest.raises(AssetSyncError, match="larger"): + transport.store(_sha(b"x" * 11), b"x" * 11) # the server's own 413 + assert "PUT" in wire.methods() + + +def test_a_reset_during_an_oversized_upload_is_reported_as_too_large(): + """The limit was not known beforehand; the reset alone would say nothing.""" + wire = _Wire(listing=[OSError("timed out"), {"max_blob_bytes": 10, "blobs": []}], + put=ConnectionResetError(10054, "connection reset by peer")) + data = b"x" * 11 + with patch(_PERFORM, new=wire): + with pytest.raises(AssetSyncError, match="larger") as refused: + _transport().store(_sha(data), data) + assert "the limit is 10" in str(refused.value) + assert isinstance(refused.value.__cause__, AssetSyncError), "the reset is kept as the cause" + assert wire.methods() == ["GET", "PUT", "GET"] + + +def test_a_reset_during_an_upload_that_fits_is_still_a_connection_error(): + wire = _Wire(listing={"max_blob_bytes": 10, "blobs": []}, + put=ConnectionResetError(10054, "connection reset by peer")) + with patch(_PERFORM, new=wire): + with pytest.raises(AssetSyncError, match="connection reset") as failed: + _transport().store(_sha(b"x"), b"x") + assert "larger" not in str(failed.value) + + +def test_a_limit_lowered_since_it_was_read_is_noticed_when_the_upload_is_cut_off(): + wire = _Wire(listing=[{"max_blob_bytes": 100}, {"max_blob_bytes": 10}], + put=ConnectionResetError(10054, "reset")) + with patch(_PERFORM, new=wire): + with pytest.raises(AssetSyncError, match="the limit is 10"): + _transport().store(_sha(b"x" * 50), b"x" * 50) + + +def test_a_bad_digest_is_refused_before_anything_is_asked(): + wire = _Wire(listing={"max_blob_bytes": 10}) + with patch(_PERFORM, new=wire): + with pytest.raises(AssetSyncError, match="SHA-256"): + _transport().store("not-a-digest", b"x") + assert wire.calls == [] + + +def test_the_listing_and_delete_calls(): + listing = {"used": 3, "quota": 100, "count": 1, "max_blob_bytes": 10, + "blobs": [{"sha256": _sha(b"abc"), "size": 3, "age_s": 5.0}]} + wire = _Wire(listing=listing) + with patch(_PERFORM, new=wire): + transport = _transport() + assert transport.usage() == listing + assert transport.delete(_sha(b"abc")) is True + assert wire.deleted == [_sha(b"abc")] + with patch(_PERFORM, return_value=_reply(404)): + assert _transport().delete(_sha(b"abc")) is False + with pytest.raises(AssetSyncError, match="does not serve"): + _transport().usage() + with patch(_PERFORM, return_value={"status": 200, "text": ""}): + with pytest.raises(AssetSyncError, match="not JSON"): + _transport().usage() + with patch(_PERFORM, return_value=_reply(401)): + with pytest.raises(AssetSyncError, match="secret"): + _transport().delete(_sha(b"abc")) + + +# --- collecting what nothing refers to ------------------------------------------------ + +def _row(data, age_s=None): + row = {"sha256": _sha(data), "size": len(data)} + if age_s is not None: + row["age_s"] = age_s + return row + + +def test_only_old_unreferenced_blobs_are_deleted(): + wire = _Wire(listing={"blobs": [ + _row(b"in use", 9 * _DAY), _row(b"orphan", 2 * _DAY), _row(b"just uploaded", 60.0), + {"sha256": "not a digest", "size": 1, "age_s": 9 * _DAY}, "junk"]}) + with patch(_PERFORM, new=wire): + result = collect_unreferenced_blobs(_transport(), [_sha(b"in use").upper()]) + assert result == {"deleted": [_sha(b"orphan")], "freed": 6, "kept": 1, + "recent": [_sha(b"just uploaded")], "failed": {}, "dry_run": False} + assert wire.deleted == [_sha(b"orphan")] + + +def test_a_dry_run_deletes_nothing_and_says_what_would_go(): + wire = _Wire(listing={"blobs": [_row(b"orphan", 2 * _DAY)]}) + with patch(_PERFORM, new=wire): + result = collect_unreferenced_blobs(_transport(), [], dry_run=True) + assert result["deleted"] == [_sha(b"orphan")] and result["dry_run"] is True + assert wire.deleted == [] and "DELETE" not in wire.methods() + + +def test_a_server_that_reports_no_ages_has_nothing_collected_unless_asked(): + listing = {"blobs": [_row(b"orphan")]} + with patch(_PERFORM, new=_Wire(listing=listing)) as wire: + assert collect_unreferenced_blobs(_transport(), [])["recent"] == [_sha(b"orphan")] + assert wire.deleted == [] + assert collect_unreferenced_blobs( + _transport(), [], min_age_s=0)["deleted"] == [_sha(b"orphan")] + + +def test_one_blob_that_cannot_be_deleted_does_not_stop_the_others(): + listing = {"blobs": [_row(b"one", _DAY * 2), _row(b"two", _DAY * 2)]} + + def perform(call): + if call["method"] == "GET": + return _reply(200, listing) + return _reply(503) if call["url"].endswith(_sha(b"one")) else _reply(200, {"deleted": True}) + + with patch(_PERFORM, new=perform): + result = collect_unreferenced_blobs(_transport(), []) + assert result["deleted"] == [_sha(b"two")] and list(result["failed"]) == [_sha(b"one")] + + +@pytest.mark.parametrize("bad", [-1, "soon", True, float("nan"), None]) +def test_a_grace_period_that_is_not_one_is_refused(bad): + with patch(_PERFORM, new=_Wire(listing={"blobs": []})) as wire: + with pytest.raises(AssetSyncError, match="min_age_s"): + collect_unreferenced_blobs(_transport(), [], min_age_s=bad) + assert wire.calls == [] + + +def test_a_reference_that_is_not_a_digest_is_refused_before_anything_is_deleted(): + with patch(_PERFORM, new=_Wire(listing={"blobs": [_row(b"x", _DAY * 2)]})) as wire: + with pytest.raises(AssetSyncError): + collect_unreferenced_blobs(_transport(), ["nope"]) + assert wire.calls == [] + + +def test_referenced_digests_cover_every_section_conflict_candidates_and_the_queue(): + large = {"sha256": _sha(b"large script"), "size": 99_999} + bucket = ConfigBucket(user_id="alice") + bucket.put_entry("scripts", SyncEntry.create("a.json", large, "laptop")) + base = SyncEntry.create("b.json", {"sha256": _sha(b"v0"), "size": 1}, "laptop") + left = base.edited({"sha256": _sha(b"left"), "size": 1}, "laptop") + right = base.edited({"sha256": _sha(b"right"), "size": 1}, "desktop") + bucket.put_entry("scripts", ac.merge_entries(left, right).entry) + gone = SyncEntry.create("c.json", {"sha256": _sha(b"gone"), "size": 1}, "laptop") + bucket.put_entry("scripts", gone.removed("laptop")) + bucket.put_entry("custom", SyncEntry.create("k", {"nested": [{"sha256": _sha(b"deep")}]}, + "laptop")) + queued = SyncOperation("scripts", SyncEntry.create( + "d.json", {"sha256": _sha(b"queued"), "size": 1}, "laptop")) + assert referenced_blob_digests(bucket, None, operations=[queued]) == { + _sha(b"large script"), _sha(b"left"), _sha(b"right"), _sha(b"deep"), _sha(b"queued")} + assert referenced_blob_digests() == set() + + +def _server_bucket(*contents): + bucket = ConfigBucket(user_id="alice", revision=3) + for index, data in enumerate(contents): + bucket.put_entry("scripts", SyncEntry.create( + f"s{index}.json", {"sha256": _sha(data), "size": len(data)}, "desktop")) + return bucket + + +def test_config_sync_collect_blobs_keeps_what_the_bucket_this_machine_and_keep_name(tmp_path): + outbox_path = tmp_path / "outbox.sqlite3" + outbox = SyncOutbox(outbox_path, account="alice", endpoint="https://sync.invalid") + outbox.save_baseline(_server_bucket(b"merged here")) + outbox.enqueue(SyncOperation("scripts", SyncEntry.create( + "new.json", {"sha256": _sha(b"not sent yet"), "size": 1}, "laptop"))) + names = (b"on the server", b"merged here", b"not sent yet", b"published by hand", b"orphan") + wire = _Wire(listing={"blobs": [_row(data, 3 * _DAY) for data in names]}) + with patch(_PERFORM, new=wire), patch.object( + ConfigSyncClient, "fetch", return_value=_server_bucket(b"on the server")): + result = config_sync_collect_blobs( + "https://sync.invalid/", "alice", keep=_sha(b"published by hand"), + outbox_path=str(outbox_path), secret="s3cret") + assert result["deleted"] == [_sha(b"orphan")] and result["kept"] == 4 + assert result["referenced"] == 4 and wire.deleted == [_sha(b"orphan")] + + +def test_collecting_for_an_account_that_never_synced_here_creates_no_outbox(tmp_path): + outbox_path = tmp_path / "outbox.sqlite3" + wire = _Wire(listing={"blobs": [_row(b"orphan", 3 * _DAY)]}) + with patch(_PERFORM, new=wire), patch.object(ConfigSyncClient, "fetch", return_value=None): + result = config_sync_collect_blobs("https://sync.invalid", "alice", dry_run=True, + outbox_path=str(outbox_path)) + assert result["deleted"] == [_sha(b"orphan")] and result["dry_run"] is True + assert not outbox_path.exists() and wire.deleted == [] + + +def test_an_unreachable_server_deletes_nothing(tmp_path): + wire = _Wire(listing={"blobs": [_row(b"orphan", 3 * _DAY)]}) + with patch(_PERFORM, new=wire), patch.object( + ConfigSyncClient, "fetch", side_effect=ConfigSyncError("offline")): + with pytest.raises(ConfigSyncError, match="offline"): + config_sync_collect_blobs("https://sync.invalid", "alice", + outbox_path=str(tmp_path / "o.sqlite3")) + assert wire.calls == [] + with pytest.raises(ConfigSyncError, match="unknown config sync option"): + config_sync_collect_blobs("https://sync.invalid", "alice", nonsense=1) + + +# --- the store: ages, and one quota for every process --------------------------------- + +def test_the_listing_reports_each_blobs_age_and_storing_again_renews_it(tmp_path): + store = BlobStore(tmp_path / "blobs") + data = b"template" + store.put("alice", _sha(data), data) + blob = next((tmp_path / "blobs").rglob(_sha(data))) + then = time.time() - 5 * _DAY + os.utime(blob, (then, then)) + (listed,) = store.usage("alice")["blobs"] + assert listed["age_s"] == pytest.approx(5 * _DAY, abs=600) + assert store.put("alice", _sha(data), data) is False, "held already: nothing written" + assert store.usage("alice")["blobs"][0]["age_s"] < 600, "but it counts as stored now" + + +def test_two_stores_on_one_folder_share_the_quota(tmp_path): + """Two server processes, as two stores: each has its own in-process lock.""" + stores = [BlobStore(tmp_path / "blobs", max_blob_bytes=10, quota_bytes=30) for _ in range(2)] + refused = [] + + def fill(store, offset): + for number in range(8): + data = bytes([offset + number]) * 10 + try: + store.put("alice", _sha(data), data) + except BlobQuotaError: + refused.append(number) + + threads = [threading.Thread(target=fill, args=(store, index * 100)) + for index, store in enumerate(stores)] + for thread in threads: + thread.start() + for thread in threads: + thread.join(timeout=60) + assert not any(thread.is_alive() for thread in threads) + usage = stores[0].usage("alice") + assert usage["used"] == 30 and usage["count"] == 3 and len(refused) == 13 + assert not (tmp_path / "blobs" / "store.lock").exists(), "the lock is released" + assert stores[1].usage("alice")["count"] == 3, "the lock file is not listed as a blob" + + +def test_a_store_locked_by_another_process_refuses_instead_of_overshooting(tmp_path, monkeypatch): + from je_auto_control.utils.json_store import json_store + monkeypatch.setattr(json_store, "_LOCK_WAIT_S", 0.2) + store = BlobStore(tmp_path / "blobs") + (tmp_path / "blobs").mkdir() + (tmp_path / "blobs" / "store.lock").write_bytes(b"") # held by "another process" + with pytest.raises(BlobStoreBusyError, match="locked by another process"): + store.put("alice", _sha(b"x"), b"x") + assert store.usage("alice")["count"] == 0 + (tmp_path / "blobs" / "store.lock").unlink() + assert store.put("alice", _sha(b"x"), b"x") is True + + +# --- the delivery surfaces ---------------------------------------------------------------- + +def test_the_command_the_tool_and_the_facade_reach_the_same_function(): + from je_auto_control.utils.executor.action_executor import executor + from je_auto_control.utils.mcp_server.tools import build_default_tool_registry + assert ac.config_sync_collect_blobs is config_sync_collect_blobs + assert ac.collect_unreferenced_blobs is collect_unreferenced_blobs + assert ac.referenced_blob_digests is referenced_blob_digests + for name in ("config_sync_collect_blobs", "collect_unreferenced_blobs", + "referenced_blob_digests"): + assert name in ac.__all__ + assert executor.event_dict["AC_config_sync_collect_blobs"] is config_sync_collect_blobs + tools = {tool.name: tool for tool in build_default_tool_registry()} + tool = tools["ac_config_sync_collect_blobs"] + assert tool.handler is config_sync_collect_blobs + assert {"keep", "min_age_s", "dry_run", "secret"} <= set(tool.input_schema["properties"]) + + +def test_the_command_runs_from_an_action_list(tmp_path): + wire = _Wire(listing={"blobs": [_row(b"orphan", 3 * _DAY)]}) + with patch(_PERFORM, new=wire), patch.object(ConfigSyncClient, "fetch", return_value=None): + record = ac.execute_action([["AC_config_sync_collect_blobs", { + "server_url": "https://sync.invalid", "user_id": "alice", "dry_run": True, + "outbox_path": str(tmp_path / "o.sqlite3")}]]) + (result,) = record.values() + assert result["deleted"] == [_sha(b"orphan")] and result["dry_run"] is True + + +# --- against the real server (needs the [signaling] extra; skipped without it) ------------ + +def test_the_server_listing_carries_the_limit_and_the_ages(tmp_path): + pytest.importorskip("fastapi") + testclient = pytest.importorskip("fastapi.testclient") + from je_auto_control.utils.remote_desktop.signaling_server import create_app + client = testclient.TestClient(create_app( + shared_secret="s3cret", serve_web_viewer=False, max_blob_bytes=64, + config_store_path=tmp_path / "buckets.sqlite3")) + headers = {"X-Signaling-Secret": "s3cret"} + data = b"template" + assert client.put(f"/blobs/alice/{_sha(data)}", content=data, + headers=headers).status_code == 201 + listing = client.get("/blobs/alice", headers=headers).json() + assert listing["max_blob_bytes"] == 64 and listing["count"] == 1 + (row,) = listing["blobs"] + assert row["sha256"] == _sha(data) and row["size"] == len(data) + assert isinstance(row["age_s"], float) and 0.0 <= row["age_s"] < 3600.0 diff --git a/test/unit_test/headless/test_config_sync_blobs.py b/test/unit_test/headless/test_config_sync_blobs.py index 15727966..9a348ade 100644 --- a/test/unit_test/headless/test_config_sync_blobs.py +++ b/test/unit_test/headless/test_config_sync_blobs.py @@ -42,7 +42,9 @@ def test_a_blob_round_trips_and_is_stored_once(tmp_path): assert store.get("alice", _sha(data)) == data and store.has("alice", _sha(data)) usage = store.usage("alice") assert (usage["used"], usage["count"]) == (len(data), 1) - assert usage["blobs"] == [{"sha256": _sha(data), "size": len(data)}] + (listed,) = usage["blobs"] + assert (listed["sha256"], listed["size"]) == (_sha(data), len(data)) + assert 0.0 <= listed["age_s"] < 3600.0 def test_building_a_store_creates_nothing(tmp_path): @@ -231,6 +233,8 @@ def test_the_transport_speaks_the_blob_routes(): data = b"\x00\xff binary" def perform(call): + if call["url"].endswith("/blobs/alice"): + return _reply(200, text=json.dumps({"max_blob_bytes": 1024, "blobs": []})) calls.append(call) return {"PUT": _reply(201), "GET": _reply(200, data), "HEAD": _reply(200)}[call["method"]] @@ -325,10 +329,21 @@ def test_assets_travel_between_two_folders_through_a_real_server(live_server, tm (source / "huge.bin").write_bytes(b"z" * 200_001) again = publish_assets(AssetManifest.from_directory(source), transport) assert list(again.failed) == ["huge.bin"] - # The server answers 413 from the declared length and closes; a client still - # writing the body can see the reset before it reads that answer (Windows). + # The transport learned the limit from the listing and never sent the body, + # so the reason no longer depends on who wins the race between the server's + # 413-and-close and the client still writing (a reset, on Windows). reason = again.failed["huge.bin"] - assert "larger" in reason or "asset PUT" in reason + assert "larger" in reason and "200001 bytes; the limit is 200000" in reason + assert transport.max_blob_bytes() == 200_000 + # Housekeeping: what no entry names any more can be collected. Both blobs + # were stored a moment ago, so the default grace period keeps them. + from je_auto_control.utils.config_sync import collect_unreferenced_blobs + kept = collect_unreferenced_blobs(transport, [_sha(picture)]) + assert kept["deleted"] == [] and kept["recent"] == [_sha(b"small")] + swept = collect_unreferenced_blobs(transport, [_sha(picture)], min_age_s=0) + assert swept["deleted"] == [_sha(b"small")] and swept["freed"] == 5 and not swept["failed"] + assert transport.has(_sha(picture)) is True and transport.has(_sha(b"small")) is False + assert transport.usage()["count"] == 1 def test_config_sync_run_can_use_the_server_for_assets(tmp_path): From 3670aa21bda65c3d203373c7565c481d053c6fbe Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 07:26:51 +0800 Subject: [PATCH 10/17] Give the Config Sync tab a section picker and the assets_server switch The tab could only sync the default sections and had no way to keep large scripts on the sync server. It gains one checkbox per section, the assets_server switch, a locator file field and the blob housekeeping command, all remembered with the other form fields (never the secret). The example now imports everything from the facade, and the config sync document covers this round's changes. --- .../Eng/doc/config_sync/config_sync_doc.rst | 152 ++++++++++++++++-- .../Zh/doc/config_sync/config_sync_doc.rst | 113 +++++++++++-- examples/29_config_sync.py | 4 +- je_auto_control/gui/config_sync_tab.py | 138 +++++++++++++--- .../gui/language_wrapper/english.py | 17 ++ .../gui/language_wrapper/japanese.py | 19 +++ .../language_wrapper/simplified_chinese.py | 16 ++ .../language_wrapper/traditional_chinese.py | 17 ++ .../headless/test_config_sync_tab_gui.py | 118 +++++++++++++- 9 files changed, 544 insertions(+), 50 deletions(-) diff --git a/docs/source/Eng/doc/config_sync/config_sync_doc.rst b/docs/source/Eng/doc/config_sync/config_sync_doc.rst index 1a6e1e88..0f736b16 100644 --- a/docs/source/Eng/doc/config_sync/config_sync_doc.rst +++ b/docs/source/Eng/doc/config_sync/config_sync_doc.rst @@ -15,7 +15,9 @@ ac`` gives ``ac.config_sync_run`` and its three siblings, ``ac.run_sync``, ``ac.HttpAssetTransport``, ``ac.ConfigStore``, ``ac.BlobStore``, the errors (``ac.ConfigSyncError``, ``ac.ConfigSyncConflict``, ``ac.FullResyncRequired``, ``ac.OperationMismatchError``, ``ac.ConfigStoreError``, -``ac.RevisionConflictError``) and ``ac.CONFIG_SYNC_WIRE_VERSION`` (the module's +``ac.RevisionConflictError``), blob housekeeping +(``ac.config_sync_collect_blobs``, ``ac.collect_unreferenced_blobs``, +``ac.referenced_blob_digests``) and ``ac.CONFIG_SYNC_WIRE_VERSION`` (the module's ``WIRE_VERSION``, ``2``). The rest stays importable from ``je_auto_control.utils.config_sync``. @@ -176,9 +178,11 @@ stored versioned with ``{"": n}`` as its vector. ``peers`` gains ``{"": {"acked_revision": , "last_seen": ..., "retired": false}}``. A version-2 client from before this release reads all of it: it already understood both entry shapes and ignores peers it does not -know. Let every machine sync once with this release before deleting entries -on any of them -- a tombstone only waits for devices already listed under -``peers``. +know. A tombstone only *waits* for devices already listed under ``peers``, so +a machine that has not synced yet is covered by the hold described below +instead: a deletion stays in the bucket for thirty days, and such a machine +meets it when it first syncs inside that time. Let every machine sync once +within thirty days of a deletion, or before deleting at all. Deletions and retired devices ----------------------------- @@ -187,6 +191,30 @@ A deletion is a tombstone. A versioned tombstone is **never dropped by age**: ``collect_tombstones`` removes it only when every active device recorded under the bucket's ``peers`` has acknowledged a revision that includes it. ``ConfigSyncClient.push_operations`` maintains those acknowledgements. + +**Held for the machines the bucket does not list yet.** Acknowledgement only +covers the devices under ``peers``. A machine that holds the entry but has +never synced is not one of them, and once the tombstone was gone its first +sync simply added the entry again, with no conflict and no report. So an +acknowledged tombstone is also kept for ``TOMBSTONE_HOLD_S`` (thirty days) +from the commit that first carried it. A machine that joins inside that time +meets the deletion: its versioned copy becomes a **conflict** (the deletion +and the copy, for a person to settle with ``config_sync_resolve``), and a +flat copy stays deleted and is reported in a ``ConflictRecord``. After the +hold the next commit by any device drops the tombstone, so the bucket does +not grow for ever; a machine that first syncs later than that can still bring +the entry back. + +The commit that first carries a tombstone stamps it with ``deleted_at`` (that +device's wall clock) next to ``deleted_revision``. Age is only ever a reason +to *keep* a tombstone longer: a wrong clock can shorten or lengthen the hold, +never the wait for the listed devices. ``ConfigSyncClient(..., +tombstone_hold_s=0)`` restores the earlier rule for the buckets that client +commits, and ``collect_tombstones(entries, peers, now=..., hold_s=...)`` +applies the hold only when ``now`` is given. A tombstone stamped by a release +from before this has no ``deleted_at`` and is dropped on acknowledgement as +before; such a release also drops acknowledged tombstones without holding +them, so the hold is only as good as the oldest client still committing. A device joins ``peers`` with its first push; ``push_operations([])`` from a device the bucket does not list is such a push when the bucket holds entries (their later deletion must wait for it) and writes nothing when it holds @@ -370,7 +398,9 @@ content-addressed blobs per account: * - ``DELETE /blobs/{user_id}/{sha256}`` - ``200 {"deleted": true | false}`` * - ``GET /blobs/{user_id}`` - - ``200 {"used", "quota", "count", "max_blob_bytes", "blobs": [{"sha256", "size"}]}`` + - ``200 {"used", "quota", "count", "max_blob_bytes", "blobs": [{"sha256", "size", + "age_s"}]}``; ``age_s`` is the seconds since the blob was last stored, by the + server's clock (a ``PUT`` of content already held counts) The rules are the ones ``/config`` follows: every request needs the shared secret (``X-Signaling-Secret``, ``401`` otherwise); the account is the one in @@ -402,10 +432,12 @@ The folder is created at the first upload, not at start-up. Storage is ``je_auto_control.utils.config_sync.blobs.BlobStore`` (``put`` / ``get`` / ``has`` / ``delete`` / ``usage``); each account's blobs are in a folder named by a hash of the account id, and a blob is written to a temporary file and -renamed. Nothing removes blobs by itself: ``DELETE`` the ones no script -refers to any more when ``used`` approaches ``quota``. The quota is enforced -by one server process; two processes sharing one folder can each admit a blob -at the same moment and overshoot by that much. +renamed. A write holds the lock file ``/store.lock`` (the helper +the shared JSON stores use), so the quota check and the write are one step +for **every** server process sharing the folder, not only for the threads of +one. A lock another process keeps for more than ten seconds is +``BlobStoreBusyError`` (``503``) and nothing is written; a lock left by a +process that died is taken over after thirty seconds. ``HttpAssetTransport`` goes through ``je_auto_control.utils.http_client`` (the egress policy applies) and does not follow redirects. A refused upload says @@ -415,6 +447,65 @@ without stopping the other files. The ``/blobs`` routes are additions: the ``/config`` wire format is still version 2, and a client that does not use them is unaffected. +**Too large is decided before the upload.** The server refuses an oversized +``PUT`` from its declared length and closes the connection; a client still +sending the body could see the reset instead of the ``413``, and the failure +read as a connection error. The transport therefore reads ``max_blob_bytes`` +from the listing before its first upload (``transport.max_blob_bytes()``, +asked once and remembered) and refuses a larger file itself, without sending +it: ``asset PUT : the file is larger than the server accepts for one +blob ( bytes; the limit is )``. + +.. list-table:: + :header-rows: 1 + + * - Client + - Server + - An oversized file + * - this release + - reports ``max_blob_bytes`` (every release that serves ``/blobs``) + - refused locally with the message above; no body is sent + * - this release + - does not report it, or the listing could not be read + - sent; the server's ``413`` reads "larger than the server accepts". If the + upload is cut off instead, the listing is asked once more and the same + reason is given when the file is over the limit it reports + * - this release + - no ``/blobs`` at all + - the ``PUT`` is answered ``404`` / ``405``: "does not serve /blobs" + * - an earlier release + - any + - as before: ``413``, or a connection error when the reset wins + +**Housekeeping.** Nothing on the server removes a blob by itself, so the +scripts deleted or replaced over time keep counting against the quota. +``config_sync_collect_blobs(server_url, user_id, keep=None, min_age_s=86400, +dry_run=False, **options)`` deletes the account's blobs that nothing names any +more and returns ``{"deleted": [sha256...], "freed": bytes, "kept": n, +"recent": [sha256...], "failed": {sha256: why}, "dry_run": bool, +"referenced": n}``. What it keeps: + +* every SHA-256 named by an entry of the server's bucket (any section, every + candidate of an entry still in conflict), of this machine's last merged + state and of its changes still waiting in the outbox + (``referenced_blob_digests``); +* the digests in ``keep`` (a list or one comma-separated string) -- name + there what you published yourself with ``publish_assets``, which no bucket + entry refers to; +* any blob stored less than ``min_age_s`` ago (default one day): a machine + uploads a file just before it commits the entry naming it. The age is the + server's ``age_s``, so no two clocks are compared. + +``dry_run=True`` deletes nothing and lists what would go. A server that +cannot be reached is ``ConfigSyncError`` and nothing is deleted; a blob that +cannot be deleted is listed under ``failed`` and the rest still go. A blob +removed while some machine still has the file is uploaded again by that +machine's next sync. Against a server from before ``age_s`` nothing is +collected unless ``min_age_s=0``. ``collect_unreferenced_blobs(transport, +referenced, min_age_s=..., dry_run=...)`` is the same sweep for a set of +digests you supply; it takes an ``HttpAssetTransport`` only, because a folder +behind ``DirectoryAssetTransport`` may hold other users' files. + One call, three surfaces ------------------------ @@ -472,10 +563,10 @@ locators_path=...)`` decides, and the report's ``sections`` lists the result. A path **adds** its section to the default three; it does not narrow the sync to it, so ``scripts_dir`` alone also syncs this machine's hotkeys, triggers -and address book. That default is deliberate -- the GUI tab has a scripts -folder and no section picker, and hotkeys and triggers need that folder to -turn their script paths into portable references -- so it is unchanged; name -``sections`` when you want less. ``sections`` is a list or one comma-separated +and address book. That default is deliberate -- hotkeys and triggers need +that folder to turn their script paths into portable references -- so it is +unchanged; name ``sections`` (or untick sections in the GUI tab) when you +want less. ``sections`` is a list or one comma-separated string. An unknown name, a section whose path is missing, and a choice that names nothing (``[]``, ``","``) are each ``ConfigSyncError``; a name given twice is synced once. @@ -498,8 +589,11 @@ twice is synced once. * - ``AC_config_sync_full_resync`` - ``ac_config_sync_full_resync`` - adopt the server's state after being retired; pending changes are discarded and listed + * - ``AC_config_sync_collect_blobs`` + - ``ac_config_sync_collect_blobs`` + - delete the server's blobs no entry names any more (``keep``, ``min_age_s``, ``dry_run``) -All four are Script Builder commands under **Data**, and all four share one +All five are Script Builder commands under **Data**, and all five share one shape: ``(server_url, user_id, ..., **options)`` with the options listed above. ``config_sync_status(server_url, user_id, outbox_path=None, **options)`` reads only ``outbox_path`` (still accepted positionally); the other options @@ -519,8 +613,25 @@ until the next automatic attempt while a retry delay is running), the last merge revision, the number of pending changes, the last successful sync and the last error, and lists every conflict with its candidates. Its commands -- *Sync now*, *Cancel sync*, *Refresh sync status*, *Keep selected candidate*, -*Full resync* -- are in the Actions menu. A sync runs on a worker thread; -cancelling, or closing the tab, releases it. +*Full resync*, *Delete unused blobs on the server* (after a confirmation; it +runs ``config_sync_collect_blobs`` with the defaults) -- are in the Actions +menu. A sync runs on a worker thread; cancelling, or closing the tab, +releases it. + +The inputs are the server URL, the user id, the shared secret, the scripts +folder, the locator repository file and the shared assets folder, then: + +* **Sections to sync** -- one checkbox per section of ``SYNCABLE_SECTIONS``. + With every box ticked the tab passes no ``sections`` and the default above + applies. Unticking one passes the ticked ones as ``sections``; a ticked + section whose path is empty is left out rather than refused, and with + nothing left the tab says so and does not sync. The status lists the + sections the last sync covered. +* **Keep large scripts on the sync server** -- passes ``assets_server=True`` + and greys out the shared assets folder, which it replaces. + +Everything but the secret is remembered between runs in the GUI settings file +(``WindowSettings.load_form`` / ``save_form``, form ``config_sync``). Folder mirror and clipboard: no echo ------------------------------------ @@ -539,6 +650,15 @@ wiring in the caller; when no engine mirrors the folder the file is not even hashed. ``FolderSyncEngine.relative_name(path)`` says whether an engine covers a path (a subfolder only with ``include_subdirs``). +**What counts as a change.** The engine records each file's modification +time and size and pushes a file whose either one differs from the record -- +no longer only a file whose time is *later*, so a file put back with an older +time is pushed too. A time alone cannot tell two writes in one clock tick +apart (an edit made right after a received file was noted went unsent), so a +file recorded within ``RACY_WINDOW_S`` (2 s) of its own last write is also +remembered by SHA-256 and compared by content until that window has passed. +An older file is never read to decide whether it changed. + **Files still being written.** A changed file is pushed only once two polls in a row see the same size and modification time, so a large copy is sent whole one poll later instead of truncated now (a file that never stops diff --git a/docs/source/Zh/doc/config_sync/config_sync_doc.rst b/docs/source/Zh/doc/config_sync/config_sync_doc.rst index 3f57fa8f..c2f15525 100644 --- a/docs/source/Zh/doc/config_sync/config_sync_doc.rst +++ b/docs/source/Zh/doc/config_sync/config_sync_doc.rst @@ -12,7 +12,9 @@ ``ac.SyncAdapter`` 與五個具體 adapter(``ac.ScriptSyncAdapter`` ……)、 ``ac.DirectoryAssetTransport``、``ac.HttpAssetTransport``、``ac.ConfigStore``、``ac.BlobStore``、 各個錯誤(``ac.ConfigSyncError``、``ac.ConfigSyncConflict``、``ac.FullResyncRequired``、 -``ac.OperationMismatchError``、``ac.ConfigStoreError``、``ac.RevisionConflictError``) +``ac.OperationMismatchError``、``ac.ConfigStoreError``、``ac.RevisionConflictError``)、 +blob 清理(``ac.config_sync_collect_blobs``、``ac.collect_unreferenced_blobs``、 +``ac.referenced_blob_digests``) 以及 ``ac.CONFIG_SYNC_WIRE_VERSION``(模組內的 ``WIRE_VERSION``,值為 ``2``)。 其餘的仍可從 ``je_auto_control.utils.config_sync`` 匯入。 @@ -150,8 +152,9 @@ Client vector 為 ``{"<裝置 id>": n}``。``peers`` 多出 ``{"<裝置 id>": {"acked_revision": <提交的 revision>, "last_seen": ..., "retired": false}}``。 這個版本之前的 version-2 client 讀得懂全部內容:它本來就認得兩種 entry 形狀, -不認得的 peer 會忽略。請讓每台機器都先用這個版本同步一次,再在任何一台上刪除 entry —— -tombstone 只會等已列在 ``peers`` 下的裝置。 +不認得的 peer 會忽略。Tombstone 只會 *等* 已列在 ``peers`` 下的裝置,還沒同步過的機器改由下面說明的 +保留期涵蓋:刪除會在 bucket 裡留三十天,這樣的機器在這段時間內第一次同步時會遇到它。 +請讓每台機器在刪除後的三十天內同步一次,或是在刪除之前就先同步過。 刪除與退休裝置 -------------- @@ -159,6 +162,22 @@ tombstone 只會等已列在 ``peers`` 下的裝置。 刪除是一個 tombstone。Versioned tombstone **絕不因時間而丟棄**:``collect_tombstones`` 只在 bucket 的 ``peers`` 記錄的每台有效裝置都已確認包含它的 revision 之後才移除。 ``ConfigSyncClient.push_operations`` 負責維護這些確認。 + +**為 bucket 還沒列出的機器保留。** 確認只涵蓋 ``peers`` 下的裝置。手上有那筆 entry、 +但從沒同步過的機器不在其中;tombstone 一旦被移除,它的第一次同步就會把 entry 直接加回來, +沒有衝突也沒有回報。所以已被確認的 tombstone 還會從第一次帶著它的那次提交起再保留 +``TOMBSTONE_HOLD_S``(三十天)。在這段時間內加入的機器會遇到這筆刪除:它的 versioned 副本變成 +**衝突**(刪除與副本並存,由人用 ``config_sync_resolve`` 決定),扁平副本則維持刪除並回報於 +``ConflictRecord``。保留期過後,任何裝置的下一次提交會移除 tombstone,所以 bucket 不會無限成長; +比這更晚才第一次同步的機器仍然可能把 entry 帶回來。 + +第一次帶著 tombstone 的那次提交,會在 ``deleted_revision`` 旁邊蓋上 ``deleted_at``(該裝置的時鐘)。 +時間只會是 *多留一陣子* 的理由:時鐘不準只會讓保留期變短或變長,不會影響對已列出裝置的等待。 +``ConfigSyncClient(..., tombstone_hold_s=0)`` 可讓該 client 提交的 bucket 回到先前的規則; +``collect_tombstones(entries, peers, now=..., hold_s=...)`` 只有在給了 ``now`` 時才套用保留期。 +這個版本之前蓋章的 tombstone 沒有 ``deleted_at``,和以前一樣在確認後就移除;那樣的版本也不會保留 +已確認的 tombstone,所以保留期的效果取決於仍在提交的最舊 client。 + 裝置在第一次 push 時加入 ``peers``;bucket 還沒列出的裝置呼叫 ``push_operations([])``, 在 bucket 有 entry 時就是這樣的一次 push(之後的刪除必須等它),在 bucket 沒有任何 entry 時 不寫入任何東西 —— 空帳號的第一次同步不會提交 revision。 @@ -314,7 +333,9 @@ Adapter:同步什麼、什麼留在本機 * - ``DELETE /blobs/{user_id}/{sha256}`` - ``200 {"deleted": true | false}`` * - ``GET /blobs/{user_id}`` - - ``200 {"used", "quota", "count", "max_blob_bytes", "blobs": [{"sha256", "size"}]}`` + - ``200 {"used", "quota", "count", "max_blob_bytes", "blobs": [{"sha256", "size", + "age_s"}]}``;``age_s`` 是以 server 的時鐘計算、距離該 blob 上次被存入的秒數 + (``PUT`` 已經持有的內容也算一次) 規則和 ``/config`` 相同:每個請求都要帶共享密鑰(``X-Signaling-Secret``,否則 ``401``); 帳號就是路徑中的那個,所以一個帳號無法讀取、列出或刪除另一個帳號的 blob; @@ -343,14 +364,63 @@ Adapter:同步什麼、什麼留在本機 資料夾在第一次上傳時才建立,不是啟動時。儲存由 ``je_auto_control.utils.config_sync.blobs.BlobStore`` 負責(``put`` / ``get`` / ``has`` / ``delete`` / ``usage``);每個帳號的 blob 放在以帳號 id 的 hash 命名的資料夾,blob 先寫到暫存檔再改名。 -沒有任何東西會自動移除 blob:``used`` 接近 ``quota`` 時,請 ``DELETE`` 已經沒有腳本引用的那些。 -配額由單一 server 行程執行;兩個行程共用同一個資料夾時,可能在同一時刻各放行一個 blob 而超出這麼多。 +寫入時會持有鎖檔 ``/store.lock``(和共用 JSON store 相同的 helper), +所以配額檢查與寫入對 **每一個** 共用該資料夾的 server 行程都是同一步,不只是單一行程內的執行緒。 +鎖被另一個行程持有超過十秒時是 ``BlobStoreBusyError``(``503``),不寫入任何東西; +行程當掉留下的鎖會在三十秒後被接手。 ``HttpAssetTransport`` 經由 ``je_auto_control.utils.http_client``(egress policy 適用),不跟隨轉址。 被拒絕的上傳會說明原因 —— 太大、配額用完、密鑰錯誤、或 server 還沒有 ``/blobs`` —— ``publish_assets`` 會把它逐檔回報在 ``failed``,不會中斷其他檔案。``/blobs`` 路由是新增的: ``/config`` 的傳輸格式仍是 version 2,不使用它們的 client 不受影響。 +**「太大」在上傳之前就決定。** Server 依宣告的長度拒絕過大的 ``PUT`` 並關閉連線; +還在送 body 的 client 可能先看到連線被重設而不是 ``413``,失敗就被讀成連線錯誤。 +所以 transport 會在第一次上傳前從清單讀出 ``max_blob_bytes`` +(``transport.max_blob_bytes()``,只問一次並記住),比它大的檔案由本機直接拒絕、不送出: +``asset PUT : the file is larger than the server accepts for one blob +( bytes; the limit is )``。 + +.. list-table:: + :header-rows: 1 + + * - Client + - Server + - 過大的檔案 + * - 這個版本 + - 回報 ``max_blob_bytes``(所有提供 ``/blobs`` 的版本) + - 在本機以上述訊息拒絕;不送出 body + * - 這個版本 + - 沒有回報,或清單讀不到 + - 照常送出;server 的 ``413`` 會讀成「larger than the server accepts」。若上傳是被切斷的, + 會再問一次清單,檔案超過它回報的上限時給出同樣的原因 + * - 這個版本 + - 完全沒有 ``/blobs`` + - ``PUT`` 得到 ``404`` / ``405``:「does not serve /blobs」 + * - 較早的版本 + - 任何 + - 和以前一樣:``413``,或在重設先到時是連線錯誤 + +**清理。** Server 上沒有任何東西會自動移除 blob,所以被刪除或被取代的腳本會一直佔用配額。 +``config_sync_collect_blobs(server_url, user_id, keep=None, min_age_s=86400, +dry_run=False, **options)`` 刪除該帳號已經沒有任何東西引用的 blob,並回傳 +``{"deleted": [sha256...], "freed": bytes, "kept": n, "recent": [sha256...], +"failed": {sha256: 原因}, "dry_run": bool, "referenced": n}``。會保留的有: + +* server 的 bucket、本機上次合併的狀態、以及本機還在 outbox 等待送出的變更中, + 任何 entry 指名的 SHA-256(任何 section,包含仍在衝突中的每個候選;``referenced_blob_digests``); +* ``keep`` 裡的 digest(list 或一個逗號分隔的字串)—— 自己用 ``publish_assets`` 發佈、 + 沒有任何 bucket entry 引用的內容請列在這裡; +* 存入不到 ``min_age_s`` 的 blob(預設一天):機器會在提交引用它的 entry 之前先上傳檔案。 + 年齡用的是 server 的 ``age_s``,不會比較兩台機器的時鐘。 + +``dry_run=True`` 不刪除任何東西,只列出會被刪的。連不到 server 是 ``ConfigSyncError``, +不會刪除任何東西;刪不掉的 blob 列在 ``failed``,其餘照常處理。某台機器還有檔案時被移除的 blob, +會在那台機器下次同步時重新上傳。面對還沒有 ``age_s`` 的 server,除非 ``min_age_s=0`` 否則什麼都不收。 +``collect_unreferenced_blobs(transport, referenced, min_age_s=..., dry_run=...)`` +是對你自己提供的 digest 集合做同樣的清理;它只接受 ``HttpAssetTransport``,因為 +``DirectoryAssetTransport`` 背後的資料夾可能有其他使用者的檔案。 + 一個呼叫,三個介面 ------------------ @@ -400,9 +470,9 @@ locators_path=...)`` 決定,回報的 ``sections`` 會列出結果。 - 正好這兩個 路徑是把它的 section **加到** 預設的三個上,不是把同步縮小到它,所以只傳 ``scripts_dir`` -也會同步本機的快捷鍵、觸發器與通訊錄。這個預設是刻意的 —— GUI 分頁只有腳本資料夾、沒有 section -選擇器,而且快捷鍵與觸發器需要那個資料夾才能把腳本路徑變成可攜的參照 —— 所以維持不變; -想少同步一些就指名 ``sections``。``sections`` 可以是 list 或一個逗號分隔的字串。 +也會同步本機的快捷鍵、觸發器與通訊錄。這個預設是刻意的 —— 快捷鍵與觸發器需要那個資料夾 +才能把腳本路徑變成可攜的參照 —— 所以維持不變; +想少同步一些就指名 ``sections``(或在 GUI 分頁取消勾選 section)。``sections`` 可以是 list 或一個逗號分隔的字串。 不認得的名稱、缺少路徑的 section、以及什麼都沒指名的選擇(``[]``、``","``)都是 ``ConfigSyncError``;同一個名稱寫兩次只同步一次。 @@ -424,8 +494,11 @@ locators_path=...)`` 決定,回報的 ``sections`` 會列出結果。 * - ``AC_config_sync_full_resync`` - ``ac_config_sync_full_resync`` - 被退休後採用 server 的狀態;待送變更會被捨棄並列出 + * - ``AC_config_sync_collect_blobs`` + - ``ac_config_sync_collect_blobs`` + - 刪除 server 上已沒有 entry 指名的 blob(``keep``、``min_age_s``、``dry_run``) -四個都是 Script Builder **Data** 分類下的指令,而且形狀一致: +五個都是 Script Builder **Data** 分類下的指令,而且形狀一致: ``(server_url, user_id, ..., **options)``,options 即上面列的那些。 ``config_sync_status(server_url, user_id, outbox_path=None, **options)`` 只讀 ``outbox_path`` (仍可用位置參數傳);其他選項會被接受,讓同一個 options dict 可以傳給每一個呼叫, @@ -441,9 +514,21 @@ GUI **設定同步** 分頁(分類 *system*)顯示狀態(退避期間會附上距離下次自動重試的秒數)、最後合併的 revision、待送變更數、 上次成功同步與最後的錯誤,並列出每個衝突及其候選。它的指令 —— *立即同步*、*取消同步*、 -*重新整理同步狀態*、*保留所選候選*、*完整重新同步* —— 在 Actions 選單。 +*重新整理同步狀態*、*保留所選候選*、*完整重新同步*、*刪除伺服器上未使用的 blob* +(會先確認;以預設值執行 ``config_sync_collect_blobs``)—— 在 Actions 選單。 同步在 worker 執行緒上執行;取消或關閉分頁都會釋放它。 +輸入欄位有 server URL、使用者 id、共享密鑰、腳本資料夾、定位器儲存庫檔案與共享資產資料夾,另外還有: + +* **要同步的區段** —— ``SYNCABLE_SECTIONS`` 的每個 section 一個核取方塊。全部勾選時分頁不傳 + ``sections``,套用上面的預設。取消勾選任何一個,就把勾選的那些當成 ``sections`` 傳入; + 勾選了但路徑是空的 section 會被略過而不是拒絕,若一個都不剩,分頁會說明並且不同步。 + 狀態會列出上次同步涵蓋的 section。 +* **把大型腳本存放在同步伺服器上** —— 傳 ``assets_server=True``,並把它取代的共享資產資料夾欄位反灰。 + +除了密鑰之外,其餘都會記在 GUI 設定檔裡(``WindowSettings.load_form`` / ``save_form``, +表單 ``config_sync``),下次開啟時還原。 + 資料夾鏡像與剪貼簿:不回送 -------------------------- @@ -457,6 +542,12 @@ GUI 呼叫端不需要接線;沒有引擎鏡像那個資料夾時,檔案連 hash 都不會算。 ``FolderSyncEngine.relative_name(path)`` 回報引擎是否涵蓋某個路徑(子資料夾需要 ``include_subdirs``)。 +**什麼算是變更。** 引擎記下每個檔案的修改時間與大小,兩者任一和紀錄不同就推送 —— +不再只看時間是否 *較晚*,所以被換回較舊時間的檔案也會推送。光靠時間分不出同一個時間刻度內的兩次寫入 +(剛把收到的檔案記下之後緊接著的本機編輯,以前不會被送出),所以在檔案自己最後一次寫入後 +``RACY_WINDOW_S``(2 秒)內被記錄的檔案,還會以 SHA-256 記下內容,在這段時間過去之前以內容比對。 +較舊的檔案不會為了判斷是否變更而被讀取。 + **還在寫入的檔案。** 變更過的檔案要連續兩次輪詢看到相同的大小與修改時間才會推送, 所以大檔案的複製會晚一次輪詢整個送出,而不是現在就送出截斷的前半段(一直在變的檔案,例如 寫入中的 log,要等它停下來才會推送)。傳 ``wait_until_stable=False`` 可回到先前「一看到就推」的行為。 diff --git a/examples/29_config_sync.py b/examples/29_config_sync.py index 88253742..526c7b64 100644 --- a/examples/29_config_sync.py +++ b/examples/29_config_sync.py @@ -59,8 +59,6 @@ from urllib.parse import unquote import je_auto_control as ac -# The store's own conflict error is not re-exported by the facade. -from je_auto_control.utils.config_sync import RevisionConflictError _USER = "alice" @@ -97,7 +95,7 @@ def do_PUT(self) -> None: # noqa: N802 # reason: http.server's naming self._user(), ac.ConfigBucket.from_dict(envelope["bucket"]), base_revision=envelope["base_revision"], operation_id=envelope["operation_id"]) - except RevisionConflictError as conflict: + except ac.RevisionConflictError as conflict: self._send(409, {"detail": "revision conflict", "revision": conflict.current_revision}) return diff --git a/je_auto_control/gui/config_sync_tab.py b/je_auto_control/gui/config_sync_tab.py index 2d286205..933a0117 100644 --- a/je_auto_control/gui/config_sync_tab.py +++ b/je_auto_control/gui/config_sync_tab.py @@ -10,9 +10,19 @@ Syncing runs on a worker thread with a cancel event; closing the tab sets the event, so a sync waiting on the network does not outlive its view. -The server, user and the two folders are remembered between runs in the GUI -settings file (:mod:`je_auto_control.gui.window_settings`). The shared secret -never is: that file is plain text. +Which sections a sync covers is chosen with one checkbox per section +(:data:`session.SYNCABLE_SECTIONS`). With every box ticked nothing is passed +and the session's default applies (hotkeys, triggers and the address book, +plus scripts / locators when their path is filled in); unticking one passes +the ticked ones as ``sections``. A ticked section whose path is empty is left +out, and the status lists the sections the last sync covered. Large scripts +travel through the shared folder, or -- with its box ticked -- through the +sync server itself (``assets_server``). + +The server, user, the paths, the section choice and the ``assets_server`` +switch are remembered between runs in the GUI settings file +(:mod:`je_auto_control.gui.window_settings`). The shared secret never is: that +file is plain text. """ import functools import math @@ -22,8 +32,8 @@ from PySide6.QtCore import Qt from PySide6.QtWidgets import ( - QFormLayout, QHeaderView, QLabel, QLineEdit, QMessageBox, QTableWidget, - QTableWidgetItem, QVBoxLayout, QWidget, + QCheckBox, QFormLayout, QHBoxLayout, QHeaderView, QLabel, QLineEdit, QMessageBox, + QTableWidget, QTableWidgetItem, QVBoxLayout, QWidget, ) from je_auto_control.gui._dispose import release_resources @@ -34,9 +44,13 @@ from je_auto_control.utils.config_sync import session from je_auto_control.utils.exception.exceptions import AutoControlException -_FIELDS = ("server", "user", "secret", "scripts", "assets") +_FIELDS = ("server", "user", "secret", "scripts", "locators", "assets") #: What is kept between runs. Not "secret": the settings file is plain text. -_REMEMBERED = ("server", "user", "scripts", "assets") +_REMEMBERED = ("server", "user", "scripts", "locators", "assets") +#: The input that supplies each session option a section can need. +_OPTION_INPUT = {"scripts_dir": "scripts", "locators_path": "locators"} +_SECTIONS_KEY = "sections" +_ASSETS_SERVER_KEY = "assets_server" _FORM_NAME = "config_sync" _COLUMNS = ("entry", "choice", "origin", "value") @@ -61,9 +75,17 @@ def __init__(self, parent: Optional[QWidget] = None, self._settings_store = settings if settings is not None else WindowSettings() self._inputs: Dict[str, QLineEdit] = {name: QLineEdit() for name in _FIELDS} self._inputs["secret"].setEchoMode(QLineEdit.Password) + self._sections: Dict[str, QCheckBox] = { + name: QCheckBox() for name in session.SYNCABLE_SECTIONS} + for box in self._sections.values(): + box.setChecked(True) + self._assets_server = QCheckBox() + self._note = "" self._restore_form() for name in _REMEMBERED: self._inputs[name].editingFinished.connect(self._remember_form) + for box in (*self._sections.values(), self._assets_server): + box.toggled.connect(self._on_choice_toggled) self._state = QLabel() self._detail = QLabel() self._detail.setWordWrap(True) @@ -86,7 +108,14 @@ def _build_layout(self) -> None: form = QFormLayout() for name in _FIELDS: form.addRow(self._tr(QLabel(), f"config_sync_{name}_label"), self._inputs[name]) + picker = QHBoxLayout() + for name, box in self._sections.items(): + picker.addWidget(self._tr(box, f"config_sync_section_{name}")) + picker.addStretch(1) + form.addRow(self._tr(QLabel(), "config_sync_sections_label"), picker) + form.addRow(self._tr(self._assets_server, "config_sync_assets_server_label")) root.addLayout(form) + self._inputs["assets"].setEnabled(not self._assets_server.isChecked()) root.addWidget(self._state) root.addWidget(self._detail) root.addWidget(self._tr(QLabel(), "config_sync_conflicts_title")) @@ -120,6 +149,7 @@ def menu_actions(self) -> List[Tuple[str, Callable[[], None]]]: ("config_sync_refresh_btn", self.refresh_status), ("config_sync_resolve_btn", self.resolve_selected), ("config_sync_resync_btn", self.full_resync), + ("config_sync_collect_btn", self.collect_blobs), ] # --- inputs ------------------------------------------------------------ @@ -130,20 +160,62 @@ def _restore_form(self) -> None: for name in _REMEMBERED: if saved.get(name): self._inputs[name].setText(saved[name]) + if _SECTIONS_KEY in saved: + ticked = {name.strip() for name in saved[_SECTIONS_KEY].split(",")} + for name, box in self._sections.items(): + box.setChecked(name in ticked) + self._assets_server.setChecked(saved.get(_ASSETS_SERVER_KEY) == "1") def _remember_form(self) -> None: """Save the remembered fields as they stand (a field lost focus, or a sync is starting).""" - self._settings_store.save_form( - _FORM_NAME, {name: self._inputs[name].text().strip() for name in _REMEMBERED}) + fields = {name: self._inputs[name].text().strip() for name in _REMEMBERED} + fields[_SECTIONS_KEY] = ",".join( + name for name, box in self._sections.items() if box.isChecked()) + fields[_ASSETS_SERVER_KEY] = "1" if self._assets_server.isChecked() else "0" + self._settings_store.save_form(_FORM_NAME, fields) + + def _on_choice_toggled(self, _checked: bool = False) -> None: + """A section box or the assets switch changed: remember it, grey out what it replaces.""" + self._inputs["assets"].setEnabled(not self._assets_server.isChecked()) + self._remember_form() + + def _chosen_sections(self, values: Dict[str, str]) -> Optional[List[str]]: + """The ``sections`` argument: ``None`` (the default) while every box is ticked. - def _settings(self) -> Optional[Dict[str, Any]]: - """The form as session arguments, or ``None`` (with a message) if incomplete.""" + Otherwise the ticked sections, without those whose path is empty. + """ + if all(box.isChecked() for box in self._sections.values()): + return None + chosen = [] + for name, box in self._sections.items(): + needs = session.SYNCABLE_SECTIONS[name] + if box.isChecked() and (needs is None or values[_OPTION_INPUT[needs]]): + chosen.append(name) + return chosen + + def _settings(self, *, sections: bool = True) -> Optional[Dict[str, Any]]: + """The form as session arguments, or ``None`` (with a message) if incomplete. + + ``sections`` false leaves the section choice out, for the commands + that do not sync (reading the status, blob housekeeping). + """ values = {name: field.text().strip() for name, field in self._inputs.items()} if not values["server"] or not values["user"]: self._detail.setText(_t("config_sync_required")) return None - options = {"secret": values["secret"], "scripts_dir": values["scripts"], - "assets_dir": values["assets"]} + options: Dict[str, Any] = { + "secret": values["secret"], "scripts_dir": values["scripts"], + "locators_path": values["locators"]} + if self._assets_server.isChecked(): + options[_ASSETS_SERVER_KEY] = True + else: + options["assets_dir"] = values["assets"] + chosen = self._chosen_sections(values) if sections else None + if chosen is not None: + if not chosen: + self._detail.setText(_t("config_sync_no_sections")) + return None + options[_SECTIONS_KEY] = chosen return {"server_url": values["server"], "user_id": values["user"], "options": {name: value for name, value in options.items() if value}} @@ -177,16 +249,28 @@ def _confirm(self, question: str) -> bool: answer = QMessageBox.question(self, _t("tab_config_sync"), question) return answer == QMessageBox.Yes - def _start(self, call: Callable[[], Dict[str, Any]], cancel: threading.Event) -> None: + def collect_blobs(self) -> None: + """Delete this account's blobs on the server that no synced entry names any more.""" + settings = self._settings(sections=False) + if settings is None or not self._confirm(_t("config_sync_collect_confirm")): + return + secret = settings["options"].get("secret") + self._start(lambda: session.config_sync_collect_blobs( + settings["server_url"], settings["user_id"], secret=secret), + threading.Event(), on_done=self._on_collected) + + def _start(self, call: Callable[[], Dict[str, Any]], cancel: threading.Event, + on_done: Optional[Callable[[Dict[str, Any]], None]] = None) -> None: if self.is_busy(): self._detail.setText(_t("config_sync_busy")) return self._cancel_slot[:] = [cancel] + self._note = "" self._status = {**self._status, "state": "syncing", "error": ""} self._render() self._worker = start_worker( - self, CallWorker(call), on_done=self._on_report, on_fail=self._on_failed, - on_thread_done=self._on_thread_done) + self, CallWorker(call), on_done=on_done or self._on_report, + on_fail=self._on_failed, on_thread_done=self._on_thread_done) def cancel(self) -> None: """Ask the running sync to stop; its changes stay in the outbox.""" @@ -194,7 +278,7 @@ def cancel(self) -> None: def refresh_status(self) -> None: """Re-read the recorded state from the outbox; no network.""" - settings = self._settings() + settings = self._settings(sections=False) if settings is None: return try: @@ -228,6 +312,13 @@ def _on_report(self, report: Dict[str, Any]) -> None: self._status = dict(report) self.refresh_status() + def _on_collected(self, result: Dict[str, Any]) -> None: + self._status = {**self._status, "state": "never"} + self._note = _t("config_sync_collect_done").format( + deleted=len(result.get("deleted") or ()), freed=int(result.get("freed") or 0), + kept=int(result.get("kept") or 0), recent=len(result.get("recent") or ())) + self.refresh_status() + def _on_failed(self, message: str) -> None: self._status = {**self._status, "state": "offline", "error": message} self._render() @@ -246,6 +337,11 @@ def _render(self) -> None: if retry_in > 0 and state in ("backing_off", "offline"): text += f" ({_t('config_sync_retry_in').format(seconds=math.ceil(retry_in))})" self._state.setText(text) + self._detail.setText("\n".join(self._detail_lines(status))) + self._render_conflicts(status.get("conflict_details") or []) + + def _detail_lines(self, status: Dict[str, Any]) -> List[str]: + """The lines under the state: revision, queue, last success, sections, error, note.""" succeeded = float(status.get("last_success") or 0.0) when = time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(succeeded)) if succeeded else "-" lines = [ @@ -253,10 +349,14 @@ def _render(self) -> None: f"{_t('config_sync_pending')}: {int(status.get('pending') or 0)}", f"{_t('config_sync_last_success')}: {when}", ] + if status.get("sections"): + covered = ", ".join(str(name) for name in status["sections"]) + lines.append(f"{_t('config_sync_sections_covered')}: {covered}") if status.get("error"): lines.append(f"{_t('config_sync_error')}: {status['error']}") - self._detail.setText("\n".join(lines)) - self._render_conflicts(status.get("conflict_details") or []) + if self._note: + lines.append(self._note) + return lines def _render_conflicts(self, details: List[Dict[str, Any]]) -> None: rows = [(detail, index, choice) for detail in details diff --git a/je_auto_control/gui/language_wrapper/english.py b/je_auto_control/gui/language_wrapper/english.py index 88196b8a..43c07661 100644 --- a/je_auto_control/gui/language_wrapper/english.py +++ b/je_auto_control/gui/language_wrapper/english.py @@ -1286,6 +1286,23 @@ "config_sync_secret_label": "Shared secret:", "config_sync_scripts_label": "Scripts folder (optional):", "config_sync_assets_label": "Shared assets folder (optional):", + "config_sync_locators_label": "Locator repository file (optional):", + "config_sync_sections_label": "Sections to sync:", + "config_sync_section_hotkeys": "Hotkeys", + "config_sync_section_triggers": "Triggers", + "config_sync_section_address_book": "Address book", + "config_sync_section_scripts": "Scripts (needs the folder)", + "config_sync_section_locators": "Locators (needs the file)", + "config_sync_assets_server_label": "Keep large scripts on the sync server instead of a shared folder", + "config_sync_no_sections": "Tick at least one section to sync (scripts and locators need their path)", + "config_sync_sections_covered": "Sections synced", + "config_sync_collect_btn": "Delete unused blobs on the server", + "config_sync_collect_confirm": + "Delete this account's blobs on the sync server that no synced entry refers to " + "any more? Blobs stored within the last day are kept.", + "config_sync_collect_done": + "Blobs: {deleted} deleted ({freed} bytes freed), {kept} in use, " + "{recent} too recent to judge", "config_sync_conflicts_title": "Conflicts: entries changed on two machines at once", "config_sync_col_entry": "Entry", "config_sync_col_choice": "Choice", diff --git a/je_auto_control/gui/language_wrapper/japanese.py b/je_auto_control/gui/language_wrapper/japanese.py index d20cbffb..82711540 100644 --- a/je_auto_control/gui/language_wrapper/japanese.py +++ b/je_auto_control/gui/language_wrapper/japanese.py @@ -1175,6 +1175,25 @@ "config_sync_secret_label": "共有シークレット:", "config_sync_scripts_label": "スクリプトフォルダー (任意):", "config_sync_assets_label": "共有アセットフォルダー (任意):", + "config_sync_locators_label": "ロケーターリポジトリファイル (任意):", + "config_sync_sections_label": "同期するセクション:", + "config_sync_section_hotkeys": "ホットキー", + "config_sync_section_triggers": "トリガー", + "config_sync_section_address_book": "アドレス帳", + "config_sync_section_scripts": "スクリプト (フォルダーが必要)", + "config_sync_section_locators": "ロケーター (ファイルが必要)", + "config_sync_assets_server_label": "大きなスクリプトを共有フォルダーではなく同期サーバーに保存する", + "config_sync_no_sections": + "同期するセクションを 1 つ以上選択してください (スクリプトとロケーターにはパスが必要" + "です)", + "config_sync_sections_covered": "同期したセクション", + "config_sync_collect_btn": "サーバー上の未使用 blob を削除", + "config_sync_collect_confirm": + "同期サーバー上のこのアカウントの blob のうち、どの同期項目からも参照されていないもの" + "を削除しますか? 直近 1 日以内に保存された blob は残します。", + "config_sync_collect_done": + "Blob: {deleted} 個を削除 ({freed} バイト解放)、使用中 {kept} 個、{recent} 個は新しす" + "ぎるため判定保留", "config_sync_conflicts_title": "競合: 2 台で同時に変更された項目", "config_sync_col_entry": "項目", "config_sync_col_choice": "候補", diff --git a/je_auto_control/gui/language_wrapper/simplified_chinese.py b/je_auto_control/gui/language_wrapper/simplified_chinese.py index 41d9487a..679add91 100644 --- a/je_auto_control/gui/language_wrapper/simplified_chinese.py +++ b/je_auto_control/gui/language_wrapper/simplified_chinese.py @@ -1145,6 +1145,22 @@ "config_sync_secret_label": "共享密钥:", "config_sync_scripts_label": "脚本文件夹(可选):", "config_sync_assets_label": "共享资源文件夹(可选):", + "config_sync_locators_label": "定位器仓库文件(可选):", + "config_sync_sections_label": "要同步的分区:", + "config_sync_section_hotkeys": "快捷键", + "config_sync_section_triggers": "触发器", + "config_sync_section_address_book": "通讯录", + "config_sync_section_scripts": "脚本(需要文件夹)", + "config_sync_section_locators": "定位器(需要文件)", + "config_sync_assets_server_label": "把大型脚本存放在同步服务器上,而不是共享文件夹", + "config_sync_no_sections": "请至少勾选一个要同步的分区(脚本和定位器需要填写路径)", + "config_sync_sections_covered": "已同步的分区", + "config_sync_collect_btn": "删除服务器上未使用的 blob", + "config_sync_collect_confirm": + "删除此账号在同步服务器上、已没有任何同步条目引用的 blob?最近一天内存入的 blob 会保留" + "。", + "config_sync_collect_done": + "Blob:已删除 {deleted} 个(释放 {freed} 字节),使用中 {kept} 个,{recent} 个太新暂不判定", "config_sync_conflicts_title": "冲突:两台机器同时修改的条目", "config_sync_col_entry": "条目", "config_sync_col_choice": "候选", diff --git a/je_auto_control/gui/language_wrapper/traditional_chinese.py b/je_auto_control/gui/language_wrapper/traditional_chinese.py index a05668ad..d150d24e 100644 --- a/je_auto_control/gui/language_wrapper/traditional_chinese.py +++ b/je_auto_control/gui/language_wrapper/traditional_chinese.py @@ -1152,6 +1152,23 @@ "config_sync_secret_label": "共享密鑰:", "config_sync_scripts_label": "腳本資料夾(選填):", "config_sync_assets_label": "共享資產資料夾(選填):", + "config_sync_locators_label": "定位器儲存庫檔案(選填):", + "config_sync_sections_label": "要同步的區段:", + "config_sync_section_hotkeys": "快捷鍵", + "config_sync_section_triggers": "觸發器", + "config_sync_section_address_book": "通訊錄", + "config_sync_section_scripts": "腳本(需要資料夾)", + "config_sync_section_locators": "定位器(需要檔案)", + "config_sync_assets_server_label": "把大型腳本存放在同步伺服器上,而不是共享資料夾", + "config_sync_no_sections": "請至少勾選一個要同步的區段(腳本與定位器需要填寫路徑)", + "config_sync_sections_covered": "已同步的區段", + "config_sync_collect_btn": "刪除伺服器上未使用的 blob", + "config_sync_collect_confirm": + "刪除此帳號在同步伺服器上、已沒有任何同步項目引用的 blob?最近一天內存入的 blob 會保留" + "。", + "config_sync_collect_done": + "Blob:已刪除 {deleted} 個(釋放 {freed} 位元組),使用中 {kept} 個,{recent} 個太新尚不判" + "定", "config_sync_conflicts_title": "衝突:兩台機器同時修改的項目", "config_sync_col_entry": "項目", "config_sync_col_choice": "候選", diff --git a/test/unit_test/headless/test_config_sync_tab_gui.py b/test/unit_test/headless/test_config_sync_tab_gui.py index 408ed1ee..34cece05 100644 --- a/test/unit_test/headless/test_config_sync_tab_gui.py +++ b/test/unit_test/headless/test_config_sync_tab_gui.py @@ -56,7 +56,7 @@ def test_the_tab_is_registered_once_and_exposes_its_commands(tab): actions = tab.menu_actions() assert [key for key, _handler in actions] == [ "config_sync_run_btn", "config_sync_cancel_btn", "config_sync_refresh_btn", - "config_sync_resolve_btn", "config_sync_resync_btn"] + "config_sync_resolve_btn", "config_sync_resync_btn", "config_sync_collect_btn"] assert all(callable(handler) for _key, handler in actions) @@ -69,6 +69,7 @@ def test_every_text_exists_in_all_four_languages(tab): keys = set(re.findall(r'"((?:tab_)?config_sync_[a-z_]+)"', source)) keys |= {f"config_sync_{name}_label" for name in tab_module._FIELDS} # noqa: SLF001 keys |= {f"config_sync_col_{name}" for name in tab_module._COLUMNS} # noqa: SLF001 + keys |= {f"config_sync_section_{name}" for name in tab_module.session.SYNCABLE_SECTIONS} keys |= {f"config_sync_state_{state}" for state in ( "never", "syncing", "synced", "pending", "conflict", "offline", "backing_off", "cancelled", "resync_required")} @@ -190,3 +191,118 @@ def test_a_retry_delay_reads_as_waiting_not_as_offline(tab, monkeypatch): lambda _url, _user: {"state": "synced", "retry_in_s": 0.0}) tab.refresh_status() assert "42" not in tab._state.text() # noqa: SLF001 + + +# --- the section picker and the assets_server switch ------------------------------- + +def _options_of_a_sync(widget, qapp, monkeypatch): + """Run "Sync now" against a fake session; the options it was given (``None`` if not run).""" + seen = [] + monkeypatch.setattr(tab_module.session, "config_sync_run", + lambda _url, _user, cancel=None, **options: seen.append(options) or {}) + widget.sync_now() + _pump(qapp, lambda: widget._worker is None) # noqa: SLF001 + return seen[0] if seen else None + + +def test_every_section_ticked_leaves_the_choice_to_the_session(tab, qapp, monkeypatch): + assert list(tab._sections) == list(tab_module.session.SYNCABLE_SECTIONS) # noqa: SLF001 + assert all(box.isChecked() for box in tab._sections.values()) # noqa: SLF001 + options = _options_of_a_sync(tab, qapp, monkeypatch) + assert "sections" not in options and "assets_server" not in options + + +def test_unticking_a_section_names_the_rest(tab, qapp, monkeypatch, tmp_path): + tab._sections["hotkeys"].setChecked(False) # noqa: SLF001 + options = _options_of_a_sync(tab, qapp, monkeypatch) + assert options["sections"] == ["triggers", "address_book"], "no path: scripts, locators out" + tab._inputs["scripts"].setText(str(tmp_path)) # noqa: SLF001 + tab._inputs["locators"].setText(str(tmp_path / "locators.json")) # noqa: SLF001 + options = _options_of_a_sync(tab, qapp, monkeypatch) + assert options["sections"] == ["triggers", "address_book", "scripts", "locators"] + assert options["scripts_dir"] == str(tmp_path) + assert options["locators_path"] == str(tmp_path / "locators.json") + # What the picker passes is what the session accepts. + assert tab_module.session.resolve_sections( + options["sections"], scripts_dir=options["scripts_dir"], + locators_path=options["locators_path"]) == options["sections"] + + +def test_a_choice_that_leaves_nothing_is_said_not_synced(tab, qapp, monkeypatch): + from je_auto_control.gui.language_wrapper.multi_language_wrapper import language_wrapper + for name in ("hotkeys", "triggers", "address_book"): + tab._sections[name].setChecked(False) # noqa: SLF001 + assert _options_of_a_sync(tab, qapp, monkeypatch) is None + assert tab._detail.text() == language_wrapper.translate( # noqa: SLF001 + "config_sync_no_sections", "") + tab.refresh_status() # reading the status needs no section + assert "7" in tab._detail.text() # noqa: SLF001 + + +def test_the_assets_server_switch_replaces_the_shared_folder(tab, qapp, monkeypatch, tmp_path): + tab._inputs["assets"].setText(str(tmp_path)) # noqa: SLF001 + assert _options_of_a_sync(tab, qapp, monkeypatch)["assets_dir"] == str(tmp_path) + tab._assets_server.setChecked(True) # noqa: SLF001 + assert not tab._inputs["assets"].isEnabled() # noqa: SLF001 + options = _options_of_a_sync(tab, qapp, monkeypatch) + assert options["assets_server"] is True and "assets_dir" not in options + tab._assets_server.setChecked(False) # noqa: SLF001 + assert tab._inputs["assets"].isEnabled() # noqa: SLF001 + + +def test_the_picker_and_the_switch_are_remembered_but_never_the_secret(qapp, tmp_path): + from je_auto_control.gui.window_settings import WindowSettings + ini = tmp_path / "gui.ini" + first = ConfigSyncTab(settings=WindowSettings(ini)) + first._inputs["server"].setText("https://sync.invalid") # noqa: SLF001 + first._inputs["secret"].setText("do-not-store-me") # noqa: SLF001 + first._inputs["locators"].setText("locators.json") # noqa: SLF001 + first._sections["triggers"].setChecked(False) # noqa: SLF001 + first._assets_server.setChecked(True) # noqa: SLF001 + first._remember_form() # noqa: SLF001 + first.deleteLater() + assert "do-not-store-me" not in ini.read_text(encoding="utf-8") + + again = ConfigSyncTab(settings=WindowSettings(ini)) + ticked = {name: box.isChecked() for name, box in again._sections.items()} # noqa: SLF001 + assert ticked == {"hotkeys": True, "triggers": False, "address_book": True, + "scripts": True, "locators": True} + assert again._assets_server.isChecked() # noqa: SLF001 + assert not again._inputs["assets"].isEnabled() # noqa: SLF001 + assert again._inputs["locators"].text() == "locators.json" # noqa: SLF001 + assert again._inputs["secret"].text() == "" # noqa: SLF001 + again.deleteLater() + # Nothing saved yet: every section ticked, the switch off. + fresh = ConfigSyncTab(settings=WindowSettings(tmp_path / "none.ini")) + assert all(box.isChecked() for box in fresh._sections.values()) # noqa: SLF001 + assert not fresh._assets_server.isChecked() # noqa: SLF001 + fresh.deleteLater() + + +def test_the_status_lists_the_sections_the_last_sync_covered(tab, monkeypatch): + monkeypatch.setattr(tab_module.session, "config_sync_status", lambda _url, _user: { + "state": "synced", "sections": ["triggers", "scripts"]}) + tab.refresh_status() + assert "triggers, scripts" in tab._detail.text() # noqa: SLF001 + + +def test_collecting_blobs_needs_confirmation_and_reports_what_it_did(tab, qapp, monkeypatch): + ran = [] + + def collect(server_url, user_id, **options): + ran.append((server_url, user_id, options)) + return {"deleted": ["a" * 64, "b" * 64], "freed": 4096, "kept": 3, "recent": ["c" * 64]} + + monkeypatch.setattr(tab_module.session, "config_sync_collect_blobs", collect) + monkeypatch.setattr(ConfigSyncTab, "_confirm", lambda _self, _question: False) + tab.collect_blobs() + assert ran == [] and not tab.is_busy() + monkeypatch.setattr(ConfigSyncTab, "_confirm", lambda _self, _question: True) + tab._inputs["secret"].setText("s3cret") # noqa: SLF001 + for box in tab._sections.values(): # noqa: SLF001 + box.setChecked(False) # housekeeping does not depend on the section choice + tab.collect_blobs() + assert _pump(qapp, lambda: tab._worker is None and ran) # noqa: SLF001 + assert ran == [("https://sync.invalid", "alice", {"secret": "s3cret"})] + shown = tab._detail.text() # noqa: SLF001 + assert "4096" in shown and "2" in shown and "3" in shown From 61532c8423542af2b8c1b01537d6f204f6d1a8c8 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 07:28:12 +0800 Subject: [PATCH 11/17] Give each thread its own UIAutomation object and move the accessibility tabs off the GUI thread The Windows UIA backend kept one COM object on the instance and relied on COM having been initialised wherever comtypes was first imported. A script started from the GUI now runs on a worker thread, where AC_a11y_* used an object the GUI thread had created, on a thread COM was never initialised on. Each calling thread now initialises COM and creates its own object; no raw element outlives a call, so nothing crosses threads. With that in place the Accessibility tab's three commands and the A11y Audit run on a worker like every other tab. Verified only against fake COM objects that refuse use from another thread: comtypes is not installed on the machine this was written on. --- .../doc/new_features/v223_features_doc.rst | 48 ++- .../Eng/doc/new_features/v7_features_doc.rst | 18 + .../Zh/doc/new_features/v223_features_doc.rst | 34 +- .../Zh/doc/new_features/v7_features_doc.rst | 14 + je_auto_control/gui/a11y_audit_tab.py | 34 +- je_auto_control/gui/accessibility_tab.py | 110 +++-- .../backends/windows_automation.py | 204 +++++++++ .../accessibility/backends/windows_backend.py | 87 +--- pyproject.toml | 1 + .../test_accessibility_windows_threads.py | 386 ++++++++++++++++++ .../test_gui_accessibility_offthread.py | 215 ++++++++++ 11 files changed, 1022 insertions(+), 129 deletions(-) create mode 100644 je_auto_control/utils/accessibility/backends/windows_automation.py create mode 100644 test/unit_test/headless/test_accessibility_windows_threads.py create mode 100644 test/unit_test/headless/test_gui_accessibility_offthread.py diff --git a/docs/source/Eng/doc/new_features/v223_features_doc.rst b/docs/source/Eng/doc/new_features/v223_features_doc.rst index f297db05..46daab1e 100644 --- a/docs/source/Eng/doc/new_features/v223_features_doc.rst +++ b/docs/source/Eng/doc/new_features/v223_features_doc.rst @@ -70,15 +70,41 @@ Report) are closed but never released. Every tab that holds something beyond its widgets implements ``dispose()``: Scheduler, Triggers, Hotkeys, E-mail Triggers, Webhooks, REST API, Presence, Live HUD, Inspector, Profiler, Run History, Admin Console, USB Devices, USB -Sharing, Config Sync and Remote Desktop. It stops the tab's timers, cancels -its background tasks, removes its listener from the presence registry, gives -back its share of the USB watcher, takes its tail off the logger and closes a -loopback it opened -- on the call, not when Qt gets round to deleting the -widget. The backend a tab only shows (the scheduler, a REST server, a remote -desktop host) keeps running. A new tab does the same with one call, +Sharing, Config Sync and Remote Desktop, and every tab that runs a command in +the background (Script Builder, VLM, OCR, LLM Planner, Computer Use, DAG, Self +Healing, Test Suite, Assertions, Device Matrix, Mobile, Media Checks, +WebRunner, ChatOps, USB Browser, Accessibility, A11y Audit). It stops the +tab's timers, cancels its background work, removes its listener from the +presence registry, gives back its share of the USB watcher, takes its tail off +the logger and closes a loopback it opened -- on the call, not when Qt gets +round to deleting the widget. Background work is cancelled the way deleting +the tab would cancel it a moment later: a script run is stopped, a Computer +Use or DAG run is asked to stop, and work that cannot be interrupted (a model +request, a device listing) runs to its end with its answer dropped. The +backend a tab only shows (the scheduler, a REST server, a remote desktop host) +keeps running. A new tab does the same with one call, ``release_resources(self, *releases)`` from ``je_auto_control.gui._dispose``; ``test_gui_tab_dispose.py`` fails for a registered tab that creates a -``QTimer`` or registers a listener without the method. +``QTimer``, registers a listener or starts background work without the +method. + +Remote Desktop's ``dispose()`` goes through each of its five panels. A TCP or +WebSocket host or session a panel opened is left running: it belongs to the +registry, where scripts (``AC_remote_*``) and the other panels still see it +and can stop it; only the panel's own timer, a connect that has not answered +and the pop-out window go. A WebRTC host or session belongs to its panel +alone -- with the panel gone nothing could stop it, and a host would go on +sharing the screen unseen -- so releasing the tab ends it, in the background +as the Stop command does. + +A callback given to a background task is held weakly: pass a method of the +tab (or a ``functools.partial`` of one), not a lambda or a nested function +that closes over the tab. The task's handle is a child of the tab, and a +callback that kept the tab alive made the handle the last holder of a tab +nobody else referenced -- it was then destroyed from inside its own child's +destructor. ``weak_slot`` in ``je_auto_control.gui._weak_call`` makes the slot +for a handle's signal, and ``test_gui_weak_callbacks.py`` fails a new strong +one. The View menu ------------- @@ -102,7 +128,13 @@ out of the window's tree for the change or at once when their tab is selected. With all 50 tabs open a switch held the window for 1.5-1.8 s; it holds it for about 0.3 s, and the rest follows in turns of at most 85 ms (measured with ``benchmarks/gui_workloads.py`` on the -development machine). +development machine). The page of a closed tab, which is kept hidden under the +tab widget, is taken out and put back with the others. + +While a page is out of the tree its ``window()`` is not the main window. Code +in a tab that needs the window it belongs to -- to raise it, or to place a +dialog -- calls ``real_window(widget)`` from +``je_auto_control.gui.workspace_tabs`` instead of ``widget.window()``. The contract test ----------------- diff --git a/docs/source/Eng/doc/new_features/v7_features_doc.rst b/docs/source/Eng/doc/new_features/v7_features_doc.rst index 038147bb..94e182e4 100644 --- a/docs/source/Eng/doc/new_features/v7_features_doc.rst +++ b/docs/source/Eng/doc/new_features/v7_features_doc.rst @@ -93,3 +93,21 @@ actions. On platforms / backends without a control driver yet, the calls raise ``AccessibilityNotAvailableError`` with a clear message rather than silently failing. The backend is swappable, so the logic is unit-tested with an injected fake — no real GUI required. + +Threads +------- + +The Windows backend can be called from any thread. A COM object belongs to the +thread that created it, so each calling thread gets its own UIAutomation +object: the first accessibility call on a thread initialises COM there (the +main thread as a single-threaded apartment, which is what ``comtypes`` does on +import and what a GUI toolkit needs; any other thread joins the multi-threaded +apartment; a thread already in an apartment keeps it) and then creates the +object. No element is carried from one call to the next, so nothing ever +crosses threads. A script started from the GUI runs on a worker thread, and +its ``AC_a11y_*`` commands use that thread's object; the Accessibility and +A11y Audit tabs run their commands on a worker thread for the same reason. + +``comtypes`` was not installed where this was written: the rule is enforced by +tests whose fake COM objects refuse any use from a thread other than the one +that created them, not by a run against the real UIAutomation. diff --git a/docs/source/Zh/doc/new_features/v223_features_doc.rst b/docs/source/Zh/doc/new_features/v223_features_doc.rst index c9d893a3..99e1e5c8 100644 --- a/docs/source/Zh/doc/new_features/v223_features_doc.rst +++ b/docs/source/Zh/doc/new_features/v223_features_doc.rst @@ -57,13 +57,30 @@ Actions 選單 凡是在 widget 之外還握有東西的分頁都實作了 ``dispose()``:排程、觸發器、熱鍵、 E-mail 觸發器、Webhooks、REST API、Presence、Live HUD、Inspector、Profiler、 -Run History、Admin Console、USB Devices、USB Sharing、Config Sync 與遠端桌面。 +Run History、Admin Console、USB Devices、USB Sharing、Config Sync 與遠端桌面, +以及每一個會在背景執行指令的分頁(Script Builder、VLM、OCR、LLM Planner、 +Computer Use、DAG、Self Healing、Test Suite、Assertions、Device Matrix、Mobile、 +Media Checks、WebRunner、ChatOps、USB Browser、Accessibility、A11y Audit)。 它會停掉分頁的計時器、取消背景工作、把監聽從 presence registry 移除、歸還 USB watcher 的持有、把 log tail 從 logger 拿掉,並關閉自己開的 loopback——在呼叫當下 -完成,而不是等 Qt 之後刪除 widget。分頁只是顯示的後端(排程器、REST 伺服器、遠端 -桌面 host)會繼續執行。新的分頁用 ``je_auto_control.gui._dispose`` 的 -``release_resources(self, *releases)`` 一行就能做到;已註冊的分頁若建立 ``QTimer`` -或註冊監聽卻沒有這個方法,``test_gui_tab_dispose.py`` 會失敗。 +完成,而不是等 Qt 之後刪除 widget。背景工作的取消方式,與稍後刪除分頁時會發生的 +相同:腳本會被停止,Computer Use 或 DAG 的執行會被要求停止,無法中斷的工作(模型 +請求、裝置清單)會跑完,但結果被丟棄。分頁只是顯示的後端(排程器、REST 伺服器、 +遠端桌面 host)會繼續執行。新的分頁用 ``je_auto_control.gui._dispose`` 的 +``release_resources(self, *releases)`` 一行就能做到;已註冊的分頁若建立 ``QTimer``、 +註冊監聽或啟動背景工作卻沒有這個方法,``test_gui_tab_dispose.py`` 會失敗。 + +遠端桌面的 ``dispose()`` 會逐一呼叫它的五個面板。面板開啟的 TCP 或 WebSocket host +與連線會繼續執行:它們屬於 registry,腳本(``AC_remote_*``)與其他面板仍看得到、 +也能停止它們;被放掉的只有面板自己的計時器、尚未回應的連線嘗試與彈出視窗。WebRTC +的 host 與連線只屬於它的面板——面板不在了就沒有人能停止它,host 會在沒有人看得到的 +情況下繼續分享畫面——所以放掉分頁時會結束它,和 Stop 指令一樣在背景進行。 + +交給背景工作的回呼是弱參照:請傳分頁的方法(或它的 ``functools.partial``),不要傳 +把分頁包進去的 lambda 或巢狀函式。工作的 handle 是分頁的子物件,若回呼讓分頁活著, +handle 就成了一個沒有其他人參照的分頁的最後持有者——分頁於是在自己子物件的解構子裡 +被銷毀。``je_auto_control.gui._weak_call`` 的 ``weak_slot`` 會做出接到 handle 訊號的 +slot;新增強參照回呼時 ``test_gui_weak_callbacks.py`` 會失敗。 View 選單 --------- @@ -81,7 +98,12 @@ View 選單 會先移出視窗的樹(``WorkspaceTabWidget.restyle(apply)``),之後每一輪事件迴圈放回 一頁,若分頁被選取則立刻放回。50 個分頁全開時,切換主題原本會卡住視窗 1.5–1.8 秒; 現在約 0.3 秒,其餘分成每輪最多 85 毫秒完成(在開發機上以 -``benchmarks/gui_workloads.py`` 量測)。 +``benchmarks/gui_workloads.py`` 量測)。已關閉分頁的內容(隱藏地留在分頁元件底下) +也會和其他頁面一起移出、放回。 + +頁面移出樹的期間,它的 ``window()`` 不是主視窗。分頁裡需要「自己所屬的視窗」的程式 +——要把它叫到前面,或擺放對話框——請改用 ``je_auto_control.gui.workspace_tabs`` 的 +``real_window(widget)``,不要用 ``widget.window()``。 契約測試 -------- diff --git a/docs/source/Zh/doc/new_features/v7_features_doc.rst b/docs/source/Zh/doc/new_features/v7_features_doc.rst index 628aab5a..6bb769b4 100644 --- a/docs/source/Zh/doc/new_features/v7_features_doc.rst +++ b/docs/source/Zh/doc/new_features/v7_features_doc.rst @@ -87,3 +87,17 @@ Windows UIAutomation 後端(透過 ``comtypes``)實作全部四個動作。在 控制驅動的平台/後端上,呼叫會拋出帶清楚訊息的 ``AccessibilityNotAvailableError``,而非默默失敗。後端可抽換,因此邏輯以 注入的 fake 後端做單元測試——不需真實 GUI。 + +執行緒 +------ + +Windows 後端可以從任何執行緒呼叫。COM 物件屬於建立它的執行緒,所以每個呼叫的執行緒 +都有自己的 UIAutomation 物件:某個執行緒第一次呼叫無障礙 API 時,先在該執行緒初始化 +COM(主執行緒是單執行緒 apartment,與 ``comtypes`` 匯入時的行為相同,也是 GUI 工具包 +需要的;其他執行緒加入多執行緒 apartment;已經在某個 apartment 裡的執行緒維持原樣), +再建立物件。沒有任何元素會從一次呼叫留到下一次,所以不會有東西跨執行緒。從 GUI 啟動 +的腳本跑在工作執行緒上,它的 ``AC_a11y_*`` 指令用的是那個執行緒自己的物件; +Accessibility 與 A11y Audit 分頁的指令也因此改在工作執行緒上執行。 + +撰寫時的環境沒有安裝 ``comtypes``:這條規則是由測試把關的——測試裡的假 COM 物件會拒絕 +任何「不是建立它的執行緒」的使用——而不是對真實的 UIAutomation 實際跑過。 diff --git a/je_auto_control/gui/a11y_audit_tab.py b/je_auto_control/gui/a11y_audit_tab.py index 2bf6a662..61344500 100644 --- a/je_auto_control/gui/a11y_audit_tab.py +++ b/je_auto_control/gui/a11y_audit_tab.py @@ -1,8 +1,11 @@ """Accessibility Audit tab: surface a11y / i18n defects from the live tree. Thin wrapper over :func:`je_auto_control.run_audit` and -:func:`je_auto_control.contrast_ratio`. +:func:`je_auto_control.contrast_ratio`. The audit walks the accessibility +tree, so it runs off the GUI thread (:class:`~je_auto_control.gui._tab_task.TabTask`); +the contrast check is arithmetic and stays where it is. """ +import functools from typing import Optional from PySide6.QtCore import Qt @@ -11,7 +14,9 @@ QTableWidget, QTableWidgetItem, QVBoxLayout, QWidget, ) +from je_auto_control.gui._dispose import release_resources from je_auto_control.gui._i18n_helpers import TranslatableMixin +from je_auto_control.gui._tab_task import TabTask from je_auto_control.gui.language_wrapper.multi_language_wrapper import ( language_wrapper, ) @@ -29,6 +34,12 @@ def _ints(raw: str): return [int(p.strip()) for p in raw.split(",") if p.strip()] +def _audit(app: Optional[str]) -> dict: + """Worker thread: run the audit and hand back plain data.""" + report: dict = ac.run_audit(app_name=app).to_dict() + return report + + class A11yAuditTab(TranslatableMixin, QWidget): """Run the accessibility / i18n audit and render the issue list.""" @@ -44,9 +55,19 @@ def __init__(self, parent: Optional[QWidget] = None) -> None: self._table.setEditTriggers(QAbstractItemView.NoEditTriggers) self._table.horizontalHeader().setStretchLastSection(True) self._summary = QLabel() + self._task = TabTask(self) # the audit, off the GUI thread; one at a time + self._task.result.connect(self._render) + self._task.error.connect(self._on_audit_error) self._apply_headers() self._build_layout() + def dispose(self) -> None: + """Release what the tab holds beyond its widgets: the audit still running, whose report is dropped. + + Called by ``close_tab(key, release=True)`` before the widget is deleted; safe to call twice. + """ + release_resources(self) + def retranslate(self) -> None: TranslatableMixin.retranslate(self) self._apply_headers() @@ -79,12 +100,11 @@ def menu_actions(self) -> list: def _on_run(self) -> None: app = self._app.text().strip() or None - try: - report = ac.run_audit(app_name=app) - except (RuntimeError, OSError, ValueError) as error: - self._summary.setText(str(error)) - return - self._render(report.to_dict()) + self._task.start(functools.partial(_audit, app)) # False while one runs: the text is true either way + self._summary.setText(_t("task_running")) + + def _on_audit_error(self, error: object) -> None: + self._summary.setText(str(error)) def _render(self, report: dict) -> None: issues = report["issues"] diff --git a/je_auto_control/gui/accessibility_tab.py b/je_auto_control/gui/accessibility_tab.py index d3cc6dee..c1e5902f 100644 --- a/je_auto_control/gui/accessibility_tab.py +++ b/je_auto_control/gui/accessibility_tab.py @@ -1,5 +1,12 @@ -"""Accessibility tab: browse the OS UI tree and click elements by role/name.""" -from typing import Optional +"""Accessibility tab: browse the OS UI tree and click elements by role/name. + +The three commands run off the GUI thread (:class:`~je_auto_control.gui._tab_task.TabTask`): +a desktop listing is hundreds of cross-process calls, and the tree being read +includes this window, which can only answer while its own thread is free. The +backend gives each thread its own automation object, so the worker is safe. +""" +import functools +from typing import List, Optional from PySide6.QtCore import Qt from PySide6.QtWidgets import ( @@ -8,7 +15,9 @@ QVBoxLayout, QWidget, ) +from je_auto_control.gui._dispose import release_resources from je_auto_control.gui._i18n_helpers import TranslatableMixin +from je_auto_control.gui._tab_task import TabTask from je_auto_control.gui.language_wrapper.multi_language_wrapper import ( language_wrapper, ) @@ -17,16 +26,39 @@ list_accessibility_elements, ) from je_auto_control.utils.accessibility.element import ( - AccessibilityNotAvailableError, + AccessibilityElement, AccessibilityNotAvailableError, ) _COLUMN_COUNT = 5 +_LIST, _FOCUSED, _CLICK = "list", "focused", "click" def _t(key: str) -> str: return language_wrapper.translate(key, key) +def _list_matching(app: Optional[str], window: Optional[str], name_filter: str) -> List[AccessibilityElement]: + """Worker thread: list the elements and keep those whose name contains ``name_filter``.""" + # Scoping to one window is not just a filter: the desktop tree is orders + # of magnitude larger, so this is both faster and less ambiguous than + # listing everything and filtering by name. + elements = list_accessibility_elements(app_name=app, window_title=window) + if name_filter: + elements = [element for element in elements if name_filter in element.name.lower()] + return elements + + +def _focused(app: Optional[str]) -> List[AccessibilityElement]: + """Worker thread: the focused element as a list of none or one.""" + element = focused_accessibility_element(app_name=app) + return [] if element is None else [element] + + +def _click(name: Optional[str], role: Optional[str], app: Optional[str]) -> bool: + """Worker thread: click the element; whether one matched.""" + return bool(click_accessibility_element(name=name, role=role, app_name=app)) + + class AccessibilityTab(TranslatableMixin, QWidget): """Discover GUI elements via UIA / AX and click them headlessly.""" @@ -44,9 +76,24 @@ def __init__(self, parent: Optional[QWidget] = None) -> None: header.setSectionResizeMode(QHeaderView.Interactive) header.setStretchLastSection(True) self._status = QLabel() + self._task = TabTask(self) # one command at a time, off the GUI thread + self._task.result.connect(self._on_result) + self._task.error.connect(self._on_error) self._apply_table_headers() self._build_layout() + def dispose(self) -> None: + """Release what the tab holds beyond its widgets: the command still running, whose answer is dropped. + + Called by ``close_tab(key, release=True)`` before the widget is deleted; safe to call twice. + """ + release_resources(self) + + def _start(self, tag: str, work) -> None: + """Run ``work`` off the GUI thread as the command ``tag``; say so, or that one is still running.""" + self._task.start(work, tag=tag) # False while busy: the status below is true either way + self._status.setText(_t("task_running")) + def retranslate(self) -> None: TranslatableMixin.retranslate(self) self._apply_table_headers() @@ -90,35 +137,34 @@ def menu_actions(self) -> list: def _refresh(self) -> None: app = self._app_filter.text().strip() or None window = self._window_filter.text().strip() or None - try: - # Scoping to one window is not just a filter: the desktop tree is - # orders of magnitude larger, so this is both faster and less - # ambiguous than listing everything and filtering by name. - elements = list_accessibility_elements(app_name=app, - window_title=window) - except AccessibilityNotAvailableError as error: - self._status.setText(str(error)) - self._table.setRowCount(0) - return name_filter = self._name_filter.text().strip().lower() - if name_filter: - elements = [e for e in elements - if name_filter in e.name.lower()] - self._populate(elements) - self._status.setText( - _t("a11y_count_label").replace("{n}", str(len(elements))), - ) + self._start(_LIST, functools.partial(_list_matching, app, window, name_filter)) def _show_focused(self) -> None: app = self._app_filter.text().strip() or None - try: - element = focused_accessibility_element(app_name=app) - except AccessibilityNotAvailableError as error: - self._status.setText(str(error)) + self._start(_FOCUSED, functools.partial(_focused, app)) + + def _on_result(self, value: object) -> None: + """GUI thread: the command ``self._task.tag`` answered.""" + tag = self._task.tag + if tag == _CLICK: + self._status.setText("" if value else _t("a11y_click_not_found")) return - self._populate([] if element is None else [element]) - self._status.setText(_t("a11y_no_focus") if element is None - else _t("a11y_count_label").replace("{n}", "1")) + elements = list(value) if isinstance(value, list) else [] + self._populate(elements) + if tag == _FOCUSED and not elements: + self._status.setText(_t("a11y_no_focus")) + return + self._status.setText(_t("a11y_count_label").replace("{n}", str(len(elements)))) + + def _on_error(self, error: object) -> None: + """GUI thread: the command failed -- typically no accessibility backend on this machine.""" + tag = self._task.tag + self._status.setText(str(error)) + if tag == _LIST: + self._table.setRowCount(0) + elif tag == _CLICK and isinstance(error, AccessibilityNotAvailableError): + QMessageBox.warning(self, _t("a11y_click_selected"), str(error)) def _populate(self, elements) -> None: self._table.setRowCount(len(elements)) @@ -142,12 +188,4 @@ def _click_selected(self) -> None: app = self._table.item(row, 0).text() or None role = self._table.item(row, 1).text() or None name = self._table.item(row, 2).text() or None - try: - ok = click_accessibility_element( - name=name, role=role, app_name=app, - ) - except AccessibilityNotAvailableError as error: - QMessageBox.warning(self, _t("a11y_click_selected"), str(error)) - return - if not ok: - self._status.setText(_t("a11y_click_not_found")) + self._start(_CLICK, functools.partial(_click, name, role, app)) diff --git a/je_auto_control/utils/accessibility/backends/windows_automation.py b/je_auto_control/utils/accessibility/backends/windows_automation.py new file mode 100644 index 00000000..a24d28f2 --- /dev/null +++ b/je_auto_control/utils/accessibility/backends/windows_automation.py @@ -0,0 +1,204 @@ +"""The UIAutomation object, and which thread may use it. + +COM objects have thread affinity. The backend used to create one +``CUIAutomation`` on first use and keep it on the instance, with COM +initialised only on the thread that happened to import ``comtypes``. That held +while every caller was the same thread; it stopped holding when scripts started +from the GUI moved to worker threads, where ``AC_a11y_*`` then used an object +another thread had created, on a thread COM had never been initialised on. + +The rule here is **one automation object per calling thread**: a thread that +asks for it first joins a COM apartment (:func:`enter_apartment`) and then +creates its own object (:class:`PerThread` keeps one value per thread on the +backend instance). That is enough because no raw COM pointer outlives a call +of the backend -- every public method finds its element, acts on it and +answers with plain data -- so nothing is ever handed from one thread to +another, and a focus subscription is added and removed inside one call, on +one thread. + +A single owner thread that every call is marshalled to was the alternative. It +was not taken for two reasons. A caller blocked waiting for the owner cannot +answer UIA's own questions: a desktop listing includes the caller's windows, +and a GUI thread parked on a queue would stall each of those providers until +its timeout. And it would move the one path known to work -- a script calling +from the main thread -- onto a different thread, which no machine without +``comtypes`` and a desktop to try it on could confirm. + +The apartment: the main thread is initialised single-threaded, which is what +``comtypes`` itself does on import and what a GUI toolkit that arrives later +needs; every other thread joins the multi-threaded apartment, which needs no +message pump. A thread that already is in an apartment keeps it. COM is never +uninitialised here: the thread's object is released when the thread ends, and +uninitialising under a live object is the worse mistake. + +Imports no ``PySide6``. +""" +import functools +import sys +import threading +from typing import Any, Callable, Optional + +from je_auto_control.utils.accessibility.backends.windows_reads import UIA_READ_ERRORS +from je_auto_control.utils.accessibility.element import AccessibilityNotAvailableError +from je_auto_control.utils.logging.logging_instance import autocontrol_logger + +COINIT_MULTITHREADED = 0x0 +COINIT_APARTMENTTHREADED = 0x2 +_S_OK = 0 +_S_FALSE = 1 # already initialised on this thread, in the same mode +_RPC_E_CHANGED_MODE = 0x80010106 # already initialised on this thread, in the other mode + +#: What :func:`enter_apartment` answers: the apartment this thread joined, the one it already was +#: in, or that there is no COM on this platform. +APARTMENT_MTA = "mta" +APARTMENT_STA = "sta" +APARTMENT_EXISTING = "existing" +APARTMENT_NONE = "none" + +_entered = threading.local() + + +def _system_co_initialize(flags: int) -> Optional[int]: + """``CoInitializeEx(NULL, flags)`` as an unsigned HRESULT; ``None`` where there is no COM. + + Never raises for a COM answer: "already initialised in the other mode" + is a result the caller has a use for, not an error. + """ + if sys.platform != "win32": + return None + import ctypes + ole32 = ctypes.WinDLL("ole32") # a private handle: argtypes set here do not leak to other users + co_initialize = ole32.CoInitializeEx + co_initialize.argtypes = [ctypes.c_void_p, ctypes.c_ulong] + co_initialize.restype = ctypes.c_long + return int(co_initialize(None, flags)) & 0xFFFFFFFF + + +def enter_apartment(co_initialize: Optional[Callable[[int], Optional[int]]] = None) -> str: + """Initialise COM on the calling thread, once per thread; return which apartment it is in. + + Call it before the first COM call on a thread. ``co_initialize`` stands in + for the system call in tests. + """ + state = getattr(_entered, "state", None) + if state is not None: + return str(state) + on_main = threading.current_thread() is threading.main_thread() + answer = (co_initialize or _system_co_initialize)( + COINIT_APARTMENTTHREADED if on_main else COINIT_MULTITHREADED) + result = -1 if answer is None else answer & 0xFFFFFFFF + if answer is None: + state = APARTMENT_NONE + elif result in (_S_OK, _S_FALSE): + state = APARTMENT_STA if on_main else APARTMENT_MTA + elif result == _RPC_E_CHANGED_MODE: + state = APARTMENT_EXISTING # a GUI toolkit or comtypes got here first: use what is there + else: + raise AccessibilityNotAvailableError( + f"COM could not be initialised on thread {threading.current_thread().name!r}: 0x{result:08x}") + _entered.state = state + autocontrol_logger.debug("UIA: thread %s is in the %s apartment", threading.current_thread().name, state) + return state + + +class PerThread: + """An instance attribute with one value per thread; ``None`` on a thread that has not set it. + + A data descriptor, so ``self._automation = x`` and ``self._automation`` + read like an ordinary attribute -- but a thread only ever sees the object + it stored itself, and the value is dropped when that thread ends. + """ + + def __set_name__(self, owner: type, name: str) -> None: + self._key = f"_per_thread_{name}" + + def _slot(self, instance: Any) -> threading.local: + slot = instance.__dict__.get(self._key) + if slot is None: + slot = instance.__dict__.setdefault(self._key, threading.local()) + return slot # type: ignore[no-any-return] + + def __get__(self, instance: Any, owner: Optional[type] = None) -> Any: + if instance is None: + return self + return getattr(self._slot(instance), "value", None) + + def __set__(self, instance: Any, value: Any) -> None: + self._slot(instance).value = value + + +def _is_available() -> bool: + try: + import comtypes.client # noqa: F401 # reason: probe import + return True + except ImportError: + return False + + +# ``CUIAutomation8`` is the only class that hands out ``IUIAutomation2``, which +# is the only way to bound how long UIA waits for an application's provider. +# It matters: a full-screen game that never answers UIA made a single +# ``ElementFromHandle`` block for **60 seconds** here, poisoning every +# desktop-wide search. With the connection timeout set, the same call is 1.0 s. +_CLSID_CUIAUTOMATION8 = "{e22ad333-b25f-460c-83d0-0581107395c9}" +_CLSID_CUIAUTOMATION = "{ff48dba4-60ef-4201-aa87-54103eef594e}" +# Only the connect step is tightened. A provider that cannot even connect within +# a second is not going to answer; how long a legitimate *query* may take is a +# different question, so ``TransactionTimeout`` keeps its default. +_CONNECTION_TIMEOUT_MS = 1000 + + +def _create_automation(uia_module: Any) -> Any: + """The UIAutomation object, with a bounded provider-connect wait if possible.""" + from comtypes import CoCreateInstance, GUID + interface = getattr(uia_module, "IUIAutomation2", None) + if interface is not None: + try: + automation = CoCreateInstance(GUID(_CLSID_CUIAUTOMATION8), + interface=interface) + automation.ConnectionTimeout = _CONNECTION_TIMEOUT_MS + return automation + except UIA_READ_ERRORS as error: + autocontrol_logger.info( + "UIAutomation2 unavailable, provider waits are unbounded: %r", + error) + return CoCreateInstance(GUID(_CLSID_CUIAUTOMATION), + interface=uia_module.IUIAutomation) + + +@functools.lru_cache(maxsize=256) +def _process_name(process_id: int) -> str: + """Executable name for a pid. + + Cached because a desktop listing asks for the same handful of pids + thousands of times, and each miss is an ``OpenProcess`` / + ``QueryFullProcessImageNameW`` / ``CloseHandle`` round trip. Windows does + recycle pids, so a very long-lived session could in principle read a stale + name here; it only labels ``app_name``, and the cache is bounded. + """ + if process_id <= 0 or sys.platform != "win32": + return "" + try: + import ctypes + from ctypes import wintypes + kernel32 = ctypes.WinDLL("kernel32", use_last_error=True) + process_query_information = 0x0400 | 0x0010 + handle = kernel32.OpenProcess(process_query_information, False, process_id) + if not handle: + return "" + try: + buf = ctypes.create_unicode_buffer(260) + size = wintypes.DWORD(len(buf)) + get_image = kernel32.QueryFullProcessImageNameW + if not get_image(handle, 0, buf, ctypes.byref(size)): + return "" + return buf.value.rsplit("\\", 1)[-1] + finally: + kernel32.CloseHandle(handle) + except OSError: + return "" + + +__all__ = [ + "APARTMENT_EXISTING", "APARTMENT_MTA", "APARTMENT_NONE", "APARTMENT_STA", "PerThread", "enter_apartment", +] diff --git a/je_auto_control/utils/accessibility/backends/windows_backend.py b/je_auto_control/utils/accessibility/backends/windows_backend.py index b05902b6..469d0747 100644 --- a/je_auto_control/utils/accessibility/backends/windows_backend.py +++ b/je_auto_control/utils/accessibility/backends/windows_backend.py @@ -7,9 +7,11 @@ level at a time starting from the root desktop, filtered by app if needed. Only ``is_control_element=True`` nodes are surfaced to avoid millions of decorative text children. + +Threads: each calling thread gets its own automation object, created after COM +is initialised on that thread, and no raw element outlives the call that found +it -- see :mod:`~je_auto_control.utils.accessibility.backends.windows_automation`. """ -import functools -import sys from typing import Any, Dict, List, Optional from je_auto_control.utils.accessibility.backends.base import ( @@ -18,6 +20,10 @@ from je_auto_control.utils.accessibility.element import ( AccessibilityElement, AccessibilityNotAvailableError, element_matches, ) +from je_auto_control.utils.accessibility.backends.windows_automation import ( + _CLSID_CUIAUTOMATION, _CLSID_CUIAUTOMATION8, PerThread, _create_automation, _is_available, + _process_name, enter_apartment, +) from je_auto_control.utils.accessibility.backends.windows_query import ( UIA_ERRORS, _is_null, focused_raw, search_roots, walk_elements, ) @@ -73,54 +79,17 @@ } -def _is_available() -> bool: - try: - import comtypes.client # noqa: F401 # reason: probe import - return True - except ImportError: - return False - - -# ``CUIAutomation8`` is the only class that hands out ``IUIAutomation2``, which -# is the only way to bound how long UIA waits for an application's provider. -# It matters: a full-screen game that never answers UIA made a single -# ``ElementFromHandle`` block for **60 seconds** here, poisoning every -# desktop-wide search. With the connection timeout set, the same call is 1.0 s. -_CLSID_CUIAUTOMATION8 = "{e22ad333-b25f-460c-83d0-0581107395c9}" -_CLSID_CUIAUTOMATION = "{ff48dba4-60ef-4201-aa87-54103eef594e}" -# Only the connect step is tightened. A provider that cannot even connect within -# a second is not going to answer; how long a legitimate *query* may take is a -# different question, so ``TransactionTimeout`` keeps its default. -_CONNECTION_TIMEOUT_MS = 1000 - - -def _create_automation(uia_module): - """The UIAutomation object, with a bounded provider-connect wait if possible.""" - from comtypes import CoCreateInstance, GUID - interface = getattr(uia_module, "IUIAutomation2", None) - if interface is not None: - try: - automation = CoCreateInstance(GUID(_CLSID_CUIAUTOMATION8), - interface=interface) - automation.ConnectionTimeout = _CONNECTION_TIMEOUT_MS - return automation - except _UIA_ERRORS as error: - autocontrol_logger.info( - "UIAutomation2 unavailable, provider waits are unbounded: %r", - error) - return CoCreateInstance(GUID(_CLSID_CUIAUTOMATION), - interface=uia_module.IUIAutomation) - class WindowsAccessibilityBackend(AccessibilityBackend): """UIAutomation-based flat element listing.""" name = "windows-uia" + #: One automation object per calling thread: a COM object is used where it was created. + _automation = PerThread() def __init__(self) -> None: import threading self.available = _is_available() - self._automation: Any = None # The comtypes-generated UIAutomationClient module; `Any` because it # is generated at import time and has no declarations to check. self._uia_module: Any = None @@ -134,6 +103,7 @@ def _ensure_automation(self): "comtypes is required for Windows accessibility; " "install it with: pip install comtypes", ) + enter_apartment() # this thread's COM apartment, before its first COM call import comtypes.client # noqa: F401 try: uia_module = comtypes.client.GetModule("UIAutomationCore.dll") @@ -772,34 +742,7 @@ def _convert_uia(raw, cached: bool = False) -> Optional[AccessibilityElement]: ) -@functools.lru_cache(maxsize=256) -def _process_name(process_id: int) -> str: - """Executable name for a pid. - - Cached because a desktop listing asks for the same handful of pids - thousands of times, and each miss is an ``OpenProcess`` / - ``QueryFullProcessImageNameW`` / ``CloseHandle`` round trip. Windows does - recycle pids, so a very long-lived session could in principle read a stale - name here; it only labels ``app_name``, and the cache is bounded. - """ - if process_id <= 0 or sys.platform != "win32": - return "" - try: - import ctypes - from ctypes import wintypes - kernel32 = ctypes.WinDLL("kernel32", use_last_error=True) - process_query_information = 0x0400 | 0x0010 - handle = kernel32.OpenProcess(process_query_information, False, process_id) - if not handle: - return "" - try: - buf = ctypes.create_unicode_buffer(260) - size = wintypes.DWORD(len(buf)) - get_image = kernel32.QueryFullProcessImageNameW - if not get_image(handle, 0, buf, ctypes.byref(size)): - return "" - return buf.value.rsplit("\\", 1)[-1] - finally: - kernel32.CloseHandle(handle) - except OSError: - return "" +__all__ = [ + "WindowsAccessibilityBackend", "_CLSID_CUIAUTOMATION", "_CLSID_CUIAUTOMATION8", "_convert_uia", + "_create_automation", "_is_available", "_process_name", +] diff --git a/pyproject.toml b/pyproject.toml index a39df028..245f7e5a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -444,6 +444,7 @@ module = [ "je_auto_control.gui._dispose", "je_auto_control.gui._slow_op", "je_auto_control.gui._weak_call", + "je_auto_control.utils.accessibility.backends.windows_automation", "je_auto_control.utils.codegen.candidate_diff", "je_auto_control.utils.remote_desktop.input_send_queue", "je_auto_control_pytest", diff --git a/test/unit_test/headless/test_accessibility_windows_threads.py b/test/unit_test/headless/test_accessibility_windows_threads.py new file mode 100644 index 00000000..c6dee828 --- /dev/null +++ b/test/unit_test/headless/test_accessibility_windows_threads.py @@ -0,0 +1,386 @@ +"""The Windows UIA backend from more than one thread (fake ``comtypes``; no COM, no desktop). + +The backend kept one automation object on the instance and relied on COM +having been initialised wherever ``comtypes`` was first imported. A script +started from the GUI now runs on a worker thread, where ``AC_a11y_*`` used an +object the GUI thread had created, on a thread COM had never been initialised +on. ``comtypes`` is not installed where these tests were written, so they +model the one property that matters: every fake COM object remembers the +thread that created it and refuses -- and records -- any use from another. + +What this cannot show is how the real UIAutomation behaves; see the module +docstring of ``windows_automation``. +""" +import gc +import sys +import threading +import types +import weakref + +import pytest + +from je_auto_control.utils.accessibility import backends +from je_auto_control.utils.accessibility.backends import windows_automation +from je_auto_control.utils.accessibility.backends import windows_backend as backend_module +from je_auto_control.utils.accessibility.backends.windows_automation import ( + APARTMENT_EXISTING, APARTMENT_MTA, APARTMENT_NONE, APARTMENT_STA, COINIT_APARTMENTTHREADED, + COINIT_MULTITHREADED, PerThread, enter_apartment, +) +from je_auto_control.utils.accessibility.element import AccessibilityNotAvailableError +from headless._uia_doubles import Rect, UiaModule + +_VALUE_PATTERN, _INVOKE_PATTERN, _TOGGLE_PATTERN = 10002, 10000, 10015 +_CHANGED_MODE = 0x80010106 +_IS_PASSWORD = 30019 + + +class _World: + """Everything the fakes record: who created what, every call, and every use from the wrong thread.""" + + def __init__(self): + self.lock = threading.Lock() + self.calls, self.violations, self.automations, self.initialised = [], [], [], [] + + def note(self, kind, name, owner): + me = threading.get_ident() + with self.lock: + self.calls.append((kind, name, me)) + if me != owner: + self.violations.append(f"{kind}.{name} used on thread {me}, created on {owner}") + if me != owner: + raise OSError("RPC_E_WRONG_THREAD") # what the backend would swallow as "not found" + + +class _Bound: + """A COM object: it belongs to the thread that created it.""" + + def __init__(self, world): + self._world, self._owner = world, threading.get_ident() + + def __getattribute__(self, name): + if not name.startswith("_"): + world = object.__getattribute__(self, "_world") + world.note(type(self).__name__, name, object.__getattribute__(self, "_owner")) + return object.__getattribute__(self, name) + + +class _Pattern(_Bound): + def __init__(self, world, value="hello"): + super().__init__(world) + self.CurrentValue, self.done = value, [] # noqa: N815 # reason: the UIA name + + def SetValue(self, value): # noqa: N802 # reason: the UIA name + self.done.append(("set", value)) + + def Invoke(self): # noqa: N802 # reason: the UIA name + self.done.append(("invoke",)) + + def Toggle(self): # noqa: N802 # reason: the UIA name + self.done.append(("toggle",)) + + +class _Unknown(_Bound): + def __init__(self, world): + super().__init__(world) + self._pattern = _Pattern(world) + + def QueryInterface(self, _interface): # noqa: N802 # reason: the COM name + return self._pattern + + +class _Element(_Bound): + def __init__(self, world, name, children=()): + super().__init__(world) + self._children = list(children) + for prefix in ("Current", "Cached"): + setattr(self, prefix + "Name", name) + setattr(self, prefix + "ControlType", 50000) + setattr(self, prefix + "BoundingRectangle", Rect(0, 0, 10, 20)) + setattr(self, prefix + "ProcessId", 7) + setattr(self, prefix + "AutomationId", name.lower()) + setattr(self, prefix + "IsEnabled", True) + self.focused = 0 + + def GetCurrentPattern(self, pattern_id): # noqa: N802 # reason: the UIA name + known = (_VALUE_PATTERN, _INVOKE_PATTERN, _TOGGLE_PATTERN) + return _Unknown(self._world) if pattern_id in known else None + + def GetCurrentPropertyValue(self, property_id): # noqa: N802 # reason: the UIA name + return False if property_id == _IS_PASSWORD else None + + def SetFocus(self): # noqa: N802 # reason: the UIA name + self.focused += 1 + + +class _Request(_Bound): + def AddProperty(self, _property_id): # noqa: N802 # reason: the UIA name + return None + + +class _Walker(_Bound): + def GetFirstChildElementBuildCache(self, node, _request): # noqa: N802 # reason: the UIA name + children = node._children + return children[0] if children else None + + def GetNextSiblingElementBuildCache(self, _child, _request): # noqa: N802 # reason: the UIA name + return None + + +class _Automation(_Bound): + def __init__(self, world): + super().__init__(world) + self.ConnectionTimeout = None # noqa: N815 # reason: the UIA name + self.ControlViewWalker = _Walker(world) # noqa: N815 # reason: the UIA name + self._handlers = [] + + def ElementFromHandle(self, _hwnd): # noqa: N802 # reason: the UIA name + return _Element(self._world, "Editor", [_Element(self._world, "Field")]) + + def CreateCacheRequest(self): # noqa: N802 # reason: the UIA name + return _Request(self._world) + + def GetFocusedElement(self): # noqa: N802 # reason: the UIA name + return _Element(self._world, "Field") + + def AddFocusChangedEventHandler(self, _request, handler): # noqa: N802 # reason: the UIA name + self._handlers.append(handler) + + def RemoveFocusChangedEventHandler(self, handler): # noqa: N802 # reason: the UIA name + self._handlers.remove(handler) + + +@pytest.fixture() +def world(monkeypatch): + """A fake ``comtypes`` whose objects are thread-bound, and a recorder in place of ``CoInitializeEx``.""" + state = _World() + + def co_create_instance(_clsid, interface=None): + automation = _Automation(state) + with state.lock: + state.automations.append((threading.get_ident(), automation)) + return automation + + def co_initialize(flags): + with state.lock: + state.initialised.append((threading.get_ident(), flags)) + state.calls.append(("COM", "CoInitializeEx", threading.get_ident())) + return 0 + + comtypes = types.ModuleType("comtypes") + comtypes.CoCreateInstance = co_create_instance + comtypes.GUID = lambda text: text + comtypes.COMObject = type("COMObject", (), {}) + client = types.ModuleType("comtypes.client") + client.GetModule = lambda _name: UiaModule() + comtypes.client = client + monkeypatch.setitem(sys.modules, "comtypes", comtypes) + monkeypatch.setitem(sys.modules, "comtypes.client", client) + monkeypatch.setattr(windows_automation, "_system_co_initialize", co_initialize) + monkeypatch.setattr(windows_automation, "_entered", threading.local()) + monkeypatch.setattr(backend_module, "_process_name", lambda pid: f"app{pid}.exe") + + def search_roots(automation, _window_title): + yield automation.ElementFromHandle(1) + + monkeypatch.setattr(backend_module, "search_roots", search_roots) + return state + + +@pytest.fixture() +def backend(world): + return backend_module.WindowsAccessibilityBackend() + + +def _on_thread(call): + """Run ``call`` on a new thread; return ``(thread id, result)`` or re-raise what it raised.""" + box = {} + + def run(): + box["ident"] = threading.get_ident() + try: + box["result"] = call() + except BaseException as error: # noqa: BLE001 # reason: re-raised below, on the test's thread + box["error"] = error + + thread = threading.Thread(target=run, name="uia-test-worker") + thread.start() + thread.join(30.0) + assert not thread.is_alive() + if "error" in box: + raise box["error"] + return box["ident"], box["result"] + + +# --- the backend ------------------------------------------------------------------------------------------------- + +def test_an_object_made_on_one_thread_is_never_used_on_another(backend, world): + """The reported defect: first use on the GUI thread, then ``AC_a11y_*`` from a script's worker thread.""" + assert [element.name for element in backend.list_elements()] == ["Editor", "Field"] + worker, names = _on_thread(lambda: [element.name for element in backend.list_elements()]) + assert names == ["Editor", "Field"] + assert world.violations == [] + assert [ident for ident, _automation in world.automations] == [threading.get_ident(), worker] + + +def test_the_doubles_catch_what_the_backend_used_to_do(world): + """One object kept on the instance, as before: the second thread's use of it is refused and recorded.""" + + class _Shared(backend_module.WindowsAccessibilityBackend): + _automation = None # an ordinary attribute again + + shared = _Shared() + assert [element.name for element in shared.list_elements()] == ["Editor", "Field"] + with pytest.raises(OSError, match="RPC_E_WRONG_THREAD"): + _on_thread(shared.list_elements) + assert world.violations and "_Automation.ElementFromHandle" in world.violations[0] + assert len(world.automations) == 1 + + +def test_each_thread_initialises_com_before_its_first_com_call(backend, world): + backend.list_elements() + backend.list_elements() # the same thread again: nothing new + first, _ = _on_thread(backend.list_elements) + second, _ = _on_thread(backend.list_elements) + main = threading.get_ident() + assert world.initialised == [(main, COINIT_APARTMENTTHREADED), (first, COINIT_MULTITHREADED), + (second, COINIT_MULTITHREADED)] + for ident in (main, first, second): + on_thread = [(kind, name) for kind, name, thread in world.calls if thread == ident] + assert on_thread[0] == ("COM", "CoInitializeEx"), on_thread[:3] + assert len(world.automations) == 3 and world.violations == [] + + +def test_every_read_and_action_stays_on_the_calling_thread(backend, world): + backend.focused_element() # the main thread has its own object by now + + def use_everything(): + return { + "value": backend.get_value(name="Field"), + "set": backend.set_value("typed", name="Field"), + "invoke": backend.invoke(name="Field"), + "toggle": backend.toggle(name="Field"), + "focus": backend.set_focus(name="Field"), + "focused": backend.focused_element().name, + "state": backend.get_state(name="Field") is not None, + "by_id": backend.get_value(automation_id="field"), + } + + worker, results = _on_thread(use_everything) + assert results == {"value": "hello", "set": True, "invoke": True, "toggle": True, "focus": True, + "focused": "Field", "state": True, "by_id": "hello"} + assert world.violations == [] + assert {thread for _kind, _name, thread in world.calls} == {threading.get_ident(), worker} + + +def test_many_threads_at_once_each_keep_to_their_own_objects(backend, world): + start, results, errors = threading.Barrier(6), [], [] + + def run(): + try: + start.wait(10.0) + for _ in range(5): + results.append((backend.get_value(name="Field"), len(backend.list_elements()))) + except BaseException as error: # noqa: BLE001 # reason: reported by the assertion below + errors.append(error) + + threads = [threading.Thread(target=run) for _ in range(6)] + for thread in threads: + thread.start() + for thread in threads: + thread.join(30.0) + assert errors == [] and results == [("hello", 2)] * 30 + assert world.violations == [] and len(world.automations) == 6 + + +def test_a_focus_subscription_is_added_and_removed_on_the_thread_that_waits(backend, world, monkeypatch): + monkeypatch.setattr(backend, "_make_focus_handler", lambda sink: object()) + worker, result = _on_thread(lambda: backend.wait_for_focus_change(timeout=0.01)) + assert result is None and world.violations == [] + events = [(name, thread) for kind, name, thread in world.calls + if kind == "_Automation" and "FocusChangedEventHandler" in name] + assert events == [("AddFocusChangedEventHandler", worker), ("RemoveFocusChangedEventHandler", worker)] + assert [automation._handlers for ident, automation in world.automations if ident == worker] == [[]] + + +def test_a_threads_automation_object_goes_when_the_thread_ends(backend, world): + _on_thread(backend.list_elements) + seen = weakref.ref(world.automations[0][1]) + del world.automations[:] + del world.calls[:] + gc.collect() + assert seen() is None, "the object of a thread that ended was kept alive" + assert backend._automation is None # and this thread never had one + + +def test_the_executor_command_works_from_a_worker_thread_after_the_main_thread_used_it(backend, world, + monkeypatch): + """``AC_a11y_*`` in a script the GUI started: the path that had never run.""" + from je_auto_control.utils.executor.action_executor import executor + monkeypatch.setattr(backends, "_cached_backend", backend) + listed = executor.event_dict["AC_a11y_list"]() + assert [row["name"] for row in listed] == ["Editor", "Field"] + _worker, record = _on_thread(lambda: executor.execute_action([["AC_a11y_list", {}], ["AC_a11y_focused", {}]])) + assert world.violations == [] + assert len(world.automations) == 2 + answers = list(record.values()) + assert [row["name"] for row in answers[0]] == ["Editor", "Field"] and answers[1]["name"] == "Field" + + +# --- the apartment ----------------------------------------------------------------------------------------------- + +@pytest.fixture() +def fresh_thread_state(monkeypatch): + monkeypatch.setattr(windows_automation, "_entered", threading.local()) + + +def test_the_main_thread_is_single_threaded_and_a_worker_joins_the_mta(fresh_thread_state): + asked = [] + + def co_initialize(flags): + asked.append(flags) + return 0 + + assert enter_apartment(co_initialize) == APARTMENT_STA + assert enter_apartment(co_initialize) == APARTMENT_STA # once per thread + assert _on_thread(lambda: enter_apartment(co_initialize))[1] == APARTMENT_MTA + assert asked == [COINIT_APARTMENTTHREADED, COINIT_MULTITHREADED] + + +def test_a_thread_already_initialised_is_used_as_it_is(fresh_thread_state): + assert enter_apartment(lambda _flags: 1) == APARTMENT_STA # S_FALSE: same mode, already there + assert _on_thread(lambda: enter_apartment(lambda _flags: _CHANGED_MODE))[1] == APARTMENT_EXISTING + # The signed spelling ctypes gives a failed HRESULT. + assert _on_thread(lambda: enter_apartment(lambda _flags: _CHANGED_MODE - (1 << 32)))[1] == APARTMENT_EXISTING + + +def test_a_com_that_will_not_initialise_is_a_typed_error_and_is_asked_again(fresh_thread_state): + answers = [0x8007000E, 0] # E_OUTOFMEMORY, then S_OK + with pytest.raises(AccessibilityNotAvailableError, match="0x8007000e"): + enter_apartment(lambda _flags: answers.pop(0)) + assert enter_apartment(lambda _flags: answers.pop(0)) == APARTMENT_STA + + +def test_where_there_is_no_com_nothing_is_initialised(fresh_thread_state): + assert enter_apartment(lambda _flags: None) == APARTMENT_NONE + + +@pytest.mark.skipif(sys.platform == "win32", reason="on Windows the real call runs") +def test_off_windows_the_system_call_is_not_attempted(fresh_thread_state): + assert windows_automation._system_co_initialize(COINIT_MULTITHREADED) is None + assert enter_apartment() == APARTMENT_NONE + + +# --- the per-thread attribute ------------------------------------------------------------------------------------ + +class _Holder: + slot = PerThread() + + +def test_a_per_thread_attribute_is_one_value_per_thread_and_per_instance(): + first, second = _Holder(), _Holder() + assert first.slot is None + first.slot = "main" + assert _on_thread(lambda: first.slot)[1] is None + assert _on_thread(lambda: (setattr(first, "slot", "worker"), first.slot)[1])[1] == "worker" + assert first.slot == "main" and second.slot is None + assert isinstance(_Holder.slot, PerThread) diff --git a/test/unit_test/headless/test_gui_accessibility_offthread.py b/test/unit_test/headless/test_gui_accessibility_offthread.py new file mode 100644 index 00000000..d6f32335 --- /dev/null +++ b/test/unit_test/headless/test_gui_accessibility_offthread.py @@ -0,0 +1,215 @@ +"""The Accessibility and A11y Audit tabs run their commands off the GUI thread (offscreen Qt, fakes only). + +They were the last tabs still calling a backend from a slot: the Windows UIA +backend kept one COM object for whichever thread came first, so moving the call +was not safe until each thread had its own. Every backend call here is a fake; +nothing reads the real accessibility tree and nothing clicks. +""" +import os +import threading +import time +import types + +import pytest + +os.environ.setdefault("QT_QPA_PLATFORM", "offscreen") +pytest.importorskip("PySide6.QtWidgets", exc_type=ImportError) + +from PySide6.QtWidgets import QApplication, QMessageBox # noqa: E402 + +from je_auto_control.gui import a11y_audit_tab, accessibility_tab # noqa: E402 +from je_auto_control.gui.language_wrapper.multi_language_wrapper import language_wrapper # noqa: E402 +from je_auto_control.utils.accessibility.element import ( # noqa: E402 + AccessibilityElement, AccessibilityNotAvailableError, +) +from headless._qt_settle import deleting, pump_until # noqa: E402 + +_PROMPT_S = 5.0 + + +def _t(key): + return language_wrapper.translate(key, key) + + +@pytest.fixture(autouse=True) +def qapp(monkeypatch): + app = QApplication.instance() or QApplication([]) + messages = [] + for box in ("warning", "information", "question", "critical"): + monkeypatch.setattr(QMessageBox, box, lambda *args: messages.append(args[-1])) + app.messages = messages + yield app + + +@pytest.fixture(autouse=True) +def _tabs_deleted(): + with deleting(accessibility_tab.AccessibilityTab, a11y_audit_tab.A11yAuditTab): + yield + + +class _Backend: + """Stands in for the three accessibility calls; each records its thread and can be held.""" + + def __init__(self, monkeypatch): + self.hold, self.threads, self.calls = threading.Event(), [], [] + self.hold.set() + self.elements = [_element("OK"), _element("Cancel"), _element("ok again")] + self.focused, self.clicked, self.error = _element("Field"), True, None + monkeypatch.setattr(accessibility_tab, "list_accessibility_elements", self._list) + monkeypatch.setattr(accessibility_tab, "focused_accessibility_element", self._focused) + monkeypatch.setattr(accessibility_tab, "click_accessibility_element", self._click) + + def _enter(self, call, kwargs): + self.threads.append(threading.current_thread()) + self.calls.append((call, kwargs)) + self.hold.wait(30.0) + if self.error is not None: + raise self.error + + def _list(self, **kwargs): + self._enter("list", kwargs) + return list(self.elements) + + def _focused(self, **kwargs): + self._enter("focused", kwargs) + return self.focused + + def _click(self, **kwargs): + self._enter("click", kwargs) + return self.clicked + + +def _element(name): + return AccessibilityElement(name=name, role="ControlType_50000", bounds=(1, 2, 30, 40), + app_name="app.exe", process_id=7, native_id=name.lower()) + + +@pytest.fixture() +def backend(monkeypatch): + fake = _Backend(monkeypatch) + yield fake + fake.hold.set() + + +def _timed(call): + started = time.monotonic() + call() + return time.monotonic() - started + + +def _idle(tab): + return pump_until(lambda: not tab._task.running) + + +def test_refresh_lists_on_a_worker_thread_and_fills_the_table_when_it_answers(backend): + tab = accessibility_tab.AccessibilityTab() + tab._app_filter.setText("app.exe") + tab._window_filter.setText("Editor") + tab._name_filter.setText("OK") + backend.hold.clear() + assert _timed(tab._refresh) < _PROMPT_S + assert tab._status.text() == _t("task_running") and tab._table.rowCount() == 0 + tab._refresh() # a second command while one runs starts nothing + backend.hold.set() + assert _idle(tab) + assert backend.calls == [("list", {"app_name": "app.exe", "window_title": "Editor"})] + assert backend.threads[0] is not threading.main_thread() + assert [tab._table.item(row, 2).text() for row in range(tab._table.rowCount())] == ["OK", "ok again"] + assert tab._status.text() == _t("a11y_count_label").replace("{n}", "2") + + +def test_a_listing_that_fails_says_why_and_empties_the_table(backend): + tab = accessibility_tab.AccessibilityTab() + tab._refresh() + assert _idle(tab) and tab._table.rowCount() == 3 + backend.error = AccessibilityNotAvailableError("install comtypes") + tab._refresh() + assert _idle(tab) + assert tab._status.text() == "install comtypes" and tab._table.rowCount() == 0 + + +def test_show_focused_reads_off_thread_and_reports_nothing_focused(backend): + tab = accessibility_tab.AccessibilityTab() + tab._show_focused() + assert _idle(tab) + assert backend.threads[0] is not threading.main_thread() + assert tab._table.rowCount() == 1 and tab._table.item(0, 2).text() == "Field" + assert tab._status.text() == _t("a11y_count_label").replace("{n}", "1") + backend.focused = None + tab._show_focused() + assert _idle(tab) + assert tab._table.rowCount() == 0 and tab._status.text() == _t("a11y_no_focus") + + +def test_click_selected_clicks_off_thread_and_reports_a_miss_or_a_missing_backend(backend, qapp): + tab = accessibility_tab.AccessibilityTab() + tab._click_selected() + assert tab._status.text() == _t("a11y_no_selection") and backend.calls == [] + tab._refresh() + assert _idle(tab) + tab._table.setCurrentCell(1, 0) + tab._click_selected() + assert _idle(tab) + assert backend.calls[-1] == ("click", {"name": "Cancel", "role": "ControlType_50000", "app_name": "app.exe"}) + assert backend.threads[-1] is not threading.main_thread() and tab._status.text() == "" + backend.clicked = False + tab._click_selected() + assert _idle(tab) and tab._status.text() == _t("a11y_click_not_found") + backend.error = AccessibilityNotAvailableError("no backend") + tab._click_selected() + assert _idle(tab) + assert qapp.messages == ["no backend"] + + +def test_a_released_accessibility_tab_drops_the_answer_still_out(backend): + tab = accessibility_tab.AccessibilityTab() + backend.hold.clear() + tab._refresh() + tab.dispose() + backend.hold.set() + assert _idle(tab) + pump_until(lambda: False, timeout=0.2) + assert tab._table.rowCount() == 0 + tab.dispose() + + +def test_the_audit_runs_off_thread_and_renders_its_report(monkeypatch): + hold, threads = threading.Event(), [] + + def run_audit(app_name=None): + threads.append((threading.current_thread(), app_name)) + hold.wait(30.0) + report = {"issues": [{"kind": "missing_label", "severity": "error", "target": "Button", "message": "no name"}], + "error_count": 1, "warning_count": 0} + return types.SimpleNamespace(to_dict=lambda: report) + + monkeypatch.setattr(a11y_audit_tab.ac, "run_audit", run_audit) + tab = a11y_audit_tab.A11yAuditTab() + tab._app.setText("app.exe") + assert _timed(tab._on_run) < _PROMPT_S + assert tab._summary.text() == _t("task_running") + tab._on_run() # ignored while the first runs + hold.set() + assert _idle(tab) + assert len(threads) == 1 and threads[0][0] is not threading.main_thread() and threads[0][1] == "app.exe" + assert tab._table.rowCount() == 1 and tab._table.item(0, 3).text() == "no name" + assert tab._summary.text() == _t("audit_summary").replace("{errors}", "1").replace("{warnings}", "0") + + +def test_an_audit_that_fails_says_why(monkeypatch): + def run_audit(app_name=None): + raise AccessibilityNotAvailableError("no accessibility backend") + + monkeypatch.setattr(a11y_audit_tab.ac, "run_audit", run_audit) + tab = a11y_audit_tab.A11yAuditTab() + tab._on_run() + assert _idle(tab) + assert tab._summary.text() == "no accessibility backend" + + +def test_the_contrast_check_still_answers_at_once(): + tab = a11y_audit_tab.A11yAuditTab() + tab._fg.setText("0, 0, 0") + tab._bg.setText("255, 255, 255") + tab._on_contrast() + assert "21.00" in tab._summary.text() and "PASS" in tab._summary.text() From 6e151018a74ea0cc498d96a3bb5476992d5a6aa2 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 07:41:09 +0800 Subject: [PATCH 12/17] Shrink the list of modules that fail against the real PySide6 types from 70 to 18 Three causes covered most of it: Qt enum members spelled without their enum class, _tr() declared to return QWidget instead of the widget it was given, and mixins passing themselves where Qt wants a QWidget. None of it changes what runs; each enum pair was checked to be the same object before the rename. QImage.fromData keeps its str format with a marked ignore: the stub asks for bytes and the binding raises ValueError for bytes. --- je_auto_control/gui/_auto_click_tab.py | 19 +++---- je_auto_control/gui/_i18n_helpers.py | 8 ++- je_auto_control/gui/_image_detect_tab.py | 10 ++-- je_auto_control/gui/_qt_typed.py | 34 ++++++++++++ je_auto_control/gui/_record_tab.py | 23 ++++---- je_auto_control/gui/_screenshot_tab.py | 7 +-- je_auto_control/gui/_script_tab.py | 7 +-- je_auto_control/gui/_worker_thread.py | 8 +-- je_auto_control/gui/a11y_audit_tab.py | 4 +- je_auto_control/gui/accessibility_tab.py | 15 +++--- je_auto_control/gui/admin_console_tab.py | 6 +-- je_auto_control/gui/config_sync_tab.py | 14 ++--- je_auto_control/gui/dag_tab.py | 2 +- je_auto_control/gui/data_source_tab.py | 2 +- je_auto_control/gui/device_matrix_tab.py | 4 +- je_auto_control/gui/email_triggers_tab.py | 10 ++-- je_auto_control/gui/flakiness_tab.py | 8 +-- je_auto_control/gui/flow_editor/scene.py | 2 +- je_auto_control/gui/flow_editor/tab.py | 6 +-- je_auto_control/gui/hotkeys_tab.py | 3 +- je_auto_control/gui/main_widget.py | 2 +- je_auto_control/gui/main_window.py | 5 +- je_auto_control/gui/mobile_tab.py | 7 +-- je_auto_control/gui/presence_tab.py | 4 +- je_auto_control/gui/profiler_tab.py | 8 +-- je_auto_control/gui/rbac_users_panel.py | 9 ++-- .../gui/remote_desktop/_helpers.py | 6 +-- .../gui/remote_desktop/advanced_group.py | 7 ++- .../gui/remote_desktop/connection_screen.py | 4 +- .../gui/remote_desktop/frame_display.py | 5 +- .../gui/remote_desktop/host_panel.py | 2 +- je_auto_control/gui/remote_desktop/tab.py | 4 +- .../gui/remote_desktop/trusted_group.py | 7 ++- .../gui/remote_desktop/viewer_panel.py | 2 +- .../gui/remote_desktop/webrtc_dialogs.py | 3 +- je_auto_control/gui/rest_api_tab.py | 4 +- je_auto_control/gui/run_history_tab.py | 28 +++++----- je_auto_control/gui/run_history_timeline.py | 10 ++-- je_auto_control/gui/scheduler_tab.py | 3 +- .../gui/script_builder/step_list_view.py | 2 +- je_auto_control/gui/secrets_tab.py | 10 ++-- je_auto_control/gui/self_healing_tab.py | 17 +++--- je_auto_control/gui/test_suite_tab.py | 4 +- je_auto_control/gui/trace_replay_tab.py | 12 ++--- je_auto_control/gui/usb_passthrough_panel.py | 5 +- je_auto_control/gui/webhooks_tab.py | 10 ++-- je_auto_control/gui/webrunner_tab.py | 2 +- je_auto_control/gui/workspace_tabs.py | 7 +-- pyproject.toml | 1 + test/unit_test/headless/test_gui_qt_typed.py | 53 ++++++++++++++++++ test/verify/typing_extras_exempt.txt | 54 +------------------ 51 files changed, 277 insertions(+), 212 deletions(-) create mode 100644 je_auto_control/gui/_qt_typed.py create mode 100644 test/unit_test/headless/test_gui_qt_typed.py diff --git a/je_auto_control/gui/_auto_click_tab.py b/je_auto_control/gui/_auto_click_tab.py index 3335d9aa..34b1ef98 100644 --- a/je_auto_control/gui/_auto_click_tab.py +++ b/je_auto_control/gui/_auto_click_tab.py @@ -1,4 +1,4 @@ -from typing import TYPE_CHECKING, Any, Callable +from typing import TYPE_CHECKING, Any, Callable, Optional from PySide6.QtWidgets import ( QWidget, QLineEdit, QComboBox, QVBoxLayout, QLabel, @@ -6,6 +6,7 @@ QGroupBox, ) +from je_auto_control.gui._qt_typed import as_widget from je_auto_control.gui._tab_task import stop_all_script_runs from je_auto_control.gui._validators import int_validator from je_auto_control.utils.exception.exceptions import AutoControlException @@ -150,7 +151,7 @@ def _build_auto_click_tab(self) -> QWidget: sc_layout.addWidget(self._tr(QLabel(), "mouse_scroll_label")) sc_layout.addWidget(self.scroll_value_input) if special_mouse_keys_table: - self.scroll_dir_combo = QComboBox() + self.scroll_dir_combo: Optional[QComboBox] = QComboBox() self.scroll_dir_combo.addItems(list(special_mouse_keys_table.keys())) sc_layout.addWidget(self.scroll_dir_combo) else: @@ -192,12 +193,12 @@ def _start_auto_click(self): try: interval = int(self.interval_input.text()) except ValueError: - QMessageBox.warning(self, "Warning", "Interval must be a number") + QMessageBox.warning(as_widget(self), "Warning", "Interval must be a number") return # The validator's bottom of 1 is a hint only: "0" set a 0 ms timer, # which clicked on every event-loop pass (18,579 clicks in 0.5 s). if interval < 1: - QMessageBox.warning(self, "Warning", "Interval must be at least 1 ms") + QMessageBox.warning(as_widget(self), "Warning", "Interval must be at least 1 ms") return self.repeat_count = 0 try: @@ -248,7 +249,7 @@ def _do_click(self): # throwing backend must land here — otherwise timer.stop() is # skipped and the auto-click QTimer fires the failing action forever. self.timer.stop() - QMessageBox.warning(self, "Error", str(error)) + QMessageBox.warning(as_widget(self), "Error", str(error)) def _get_mouse_pos(self): try: @@ -267,7 +268,7 @@ def _get_mouse_pos(self): self.cursor_x_input.setText(str(x)) self.cursor_y_input.setText(str(y)) except (AutoControlException, OSError, ValueError, TypeError, RuntimeError) as error: - QMessageBox.warning(self, "Error", str(error)) + QMessageBox.warning(as_widget(self), "Error", str(error)) def _send_hotkey(self): try: @@ -275,7 +276,7 @@ def _send_hotkey(self): if keys: hotkey(keys) except (AutoControlException, OSError, ValueError, TypeError, RuntimeError) as error: - QMessageBox.warning(self, "Error", str(error)) + QMessageBox.warning(as_widget(self), "Error", str(error)) def _send_write(self): try: @@ -283,7 +284,7 @@ def _send_write(self): if text: write(text) except (AutoControlException, OSError, ValueError, TypeError, RuntimeError) as error: - QMessageBox.warning(self, "Error", str(error)) + QMessageBox.warning(as_widget(self), "Error", str(error)) def _send_scroll(self): try: @@ -291,4 +292,4 @@ def _send_scroll(self): direction = self.scroll_dir_combo.currentText() if self.scroll_dir_combo else "scroll_up" mouse_scroll(val, scroll_direction=direction) except (AutoControlException, OSError, ValueError, TypeError, RuntimeError) as error: - QMessageBox.warning(self, "Error", str(error)) + QMessageBox.warning(as_widget(self), "Error", str(error)) diff --git a/je_auto_control/gui/_i18n_helpers.py b/je_auto_control/gui/_i18n_helpers.py index 9681f851..dcdb1162 100644 --- a/je_auto_control/gui/_i18n_helpers.py +++ b/je_auto_control/gui/_i18n_helpers.py @@ -13,7 +13,7 @@ C++ object wherever it happened to be — on Python 3.10/3.11 that can be in the middle of a PySide call that is still walking a list of widgets. """ -from typing import List, Optional, Tuple +from typing import List, Optional, Tuple, TypeVar from PySide6.QtWidgets import ( QAbstractButton, QGroupBox, QLabel, QLineEdit, QTabWidget, QWidget, @@ -24,6 +24,10 @@ ) +#: ``_tr`` hands back the widget it was given, as the class it was given. +_W = TypeVar("_W", bound=QWidget) + + def _default_setter(widget: QWidget) -> str: if isinstance(widget, QGroupBox): return "setTitle" @@ -41,7 +45,7 @@ def _tr_init(self) -> None: self._tr_registry: List[Tuple[Optional[QWidget], str, str]] = [] self._tr_tabs: List[Tuple[QTabWidget, int, str]] = [] - def _tr(self, widget: QWidget, key: str, setter: str = "") -> QWidget: + def _tr(self, widget: _W, key: str, setter: str = "") -> _W: """Set ``widget`` text from ``key`` now and on every retranslate.""" if not hasattr(self, "_tr_registry"): self._tr_init() diff --git a/je_auto_control/gui/_image_detect_tab.py b/je_auto_control/gui/_image_detect_tab.py index 535d4cec..ffd0226b 100644 --- a/je_auto_control/gui/_image_detect_tab.py +++ b/je_auto_control/gui/_image_detect_tab.py @@ -7,6 +7,7 @@ ) from je_auto_control.gui._validators import double_validator +from je_auto_control.gui._qt_typed import as_widget from je_auto_control.gui.language_wrapper.multi_language_wrapper import language_wrapper from je_auto_control.gui.selector import crop_template_to_file from je_auto_control.utils.exception.exceptions import AutoControlException @@ -62,24 +63,25 @@ def _build_image_detect_tab(self) -> QWidget: return tab def _browse_img(self): - path, _ = QFileDialog.getOpenFileName(self, _t("template_image"), "", "Images (*.png *.jpg *.bmp);;All (*)") + path, _ = QFileDialog.getOpenFileName( + as_widget(self), _t("template_image"), "", "Images (*.png *.jpg *.bmp);;All (*)") if path: self.img_path_input.setText(path) def _crop_template(self): save_path, _ = QFileDialog.getSaveFileName( - self, _t("crop_template"), "", "PNG (*.png)" + as_widget(self), _t("crop_template"), "", "PNG (*.png)" ) if not save_path: return try: - region = crop_template_to_file(save_path, self) + region = crop_template_to_file(save_path, as_widget(self)) if region is None: return self.img_path_input.setText(save_path) self.detect_result_text.setText(f"Template saved: {save_path} region={region}") except (AutoControlException, OSError, ValueError, RuntimeError) as error: - QMessageBox.warning(self, "Error", str(error)) + QMessageBox.warning(as_widget(self), "Error", str(error)) def _get_detect_params(self): path = self.img_path_input.text() diff --git a/je_auto_control/gui/_qt_typed.py b/je_auto_control/gui/_qt_typed.py new file mode 100644 index 00000000..352b08a1 --- /dev/null +++ b/je_auto_control/gui/_qt_typed.py @@ -0,0 +1,34 @@ +"""Two typed accessors for what Qt's own signatures cannot express. + +Both exist so the GUI type-checks against the real PySide6 types +(``test/verify/typing_contract_verify.py --extras``) without a cast at every +call site: + +* A tab mixin is a ``QWidget`` only once it is mixed into its host, so + ``QMessageBox.warning(self, ...)`` inside one is a type error although it is + correct at run time. :func:`as_widget` says so in one place. +* ``QTableWidget.item()`` may answer ``None``, but a tab that fills every cell + of a row itself knows the cell is there. :func:`filled_item` returns the + item, and reports a cell that is missing as the bug it would be. +""" +from typing import cast + +from PySide6.QtWidgets import QTableWidget, QTableWidgetItem, QWidget + +from je_auto_control.utils.exception.exceptions import AutoControlException + + +def as_widget(part: object) -> QWidget: + """``part`` as the widget it is once mixed into its host (a dialog's parent, a task's owner).""" + return cast(QWidget, part) + + +def filled_item(table: QTableWidget, row: int, column: int) -> QTableWidgetItem: + """The item of a cell the caller filled itself; raise if the cell has none.""" + item = table.item(row, column) + if item is None: + raise AutoControlException(f"the table has no item at row {row}, column {column}") + return item + + +__all__ = ["as_widget", "filled_item"] diff --git a/je_auto_control/gui/_record_tab.py b/je_auto_control/gui/_record_tab.py index 34240ba2..492804af 100644 --- a/je_auto_control/gui/_record_tab.py +++ b/je_auto_control/gui/_record_tab.py @@ -13,6 +13,7 @@ ) from je_auto_control.gui._tab_task import TabTask, was_stopped +from je_auto_control.gui._qt_typed import as_widget from je_auto_control.gui.language_wrapper.multi_language_wrapper import language_wrapper from je_auto_control.utils.exception.exceptions import AutoControlException from je_auto_control.utils.executor.action_executor import execute_action @@ -62,7 +63,7 @@ def _build_record_tab(self) -> QWidget: self.record_list_text.setReadOnly(True) layout.addWidget(self.record_list_text) tab.setLayout(layout) - self._playback_runs = TabTask(self) + self._playback_runs = TabTask(as_widget(self)) self._playback_runs.error.connect(self._on_playback_error) self._playback_runs.finished.connect(self._on_playback_finished) return tab @@ -82,12 +83,12 @@ def _start_record(self): if not record(): # record() logs its failure and returns False; the status # said "Recording..." while nothing was recorded. - QMessageBox.warning(self, "Error", "Recording could not start; see the log") + QMessageBox.warning(as_widget(self), "Error", "Recording could not start; see the log") return self._record_status_key = "record_recording" self._apply_record_status_label() except (AutoControlException, OSError, ValueError, TypeError, RuntimeError) as error: - QMessageBox.warning(self, "Error", str(error)) + QMessageBox.warning(as_widget(self), "Error", str(error)) def _stop_record(self): try: @@ -96,11 +97,11 @@ def _stop_record(self): self._apply_record_status_label() self.record_list_text.setText(json.dumps(self._record_data, indent=2, ensure_ascii=False)) except (AutoControlException, OSError, ValueError, TypeError, RuntimeError) as error: - QMessageBox.warning(self, "Error", str(error)) + QMessageBox.warning(as_widget(self), "Error", str(error)) def _playback_record(self): if not self._record_data: - QMessageBox.warning(self, "Warning", "No recorded data") + QMessageBox.warning(as_widget(self), "Warning", "No recorded data") return # The worker gets its own copy: loading or recording again while the # playback runs must not change the list under it. @@ -116,7 +117,7 @@ def _stop_playback(self) -> None: def _on_playback_error(self, error: object) -> None: if not was_stopped(error): - QMessageBox.warning(self, "Error", str(error)) + QMessageBox.warning(as_widget(self), "Error", str(error)) def _on_playback_finished(self) -> None: self._record_status_key = "record_idle" @@ -125,19 +126,19 @@ def _on_playback_finished(self) -> None: def _save_record(self): try: if not self._record_data: - QMessageBox.warning(self, "Warning", "No recorded data") + QMessageBox.warning(as_widget(self), "Warning", "No recorded data") return - path, _ = QFileDialog.getSaveFileName(self, _t("save_record"), "", _JSON_FILE_FILTER) + path, _ = QFileDialog.getSaveFileName(as_widget(self), _t("save_record"), "", _JSON_FILE_FILTER) if path: write_action_json(path, self._record_data) except (AutoControlException, OSError, ValueError, TypeError, RuntimeError) as error: - QMessageBox.warning(self, "Error", str(error)) + QMessageBox.warning(as_widget(self), "Error", str(error)) def _load_record(self): try: - path, _ = QFileDialog.getOpenFileName(self, _t("load_record"), "", _JSON_FILE_FILTER) + path, _ = QFileDialog.getOpenFileName(as_widget(self), _t("load_record"), "", _JSON_FILE_FILTER) if path: self._record_data = read_action_json(path) self.record_list_text.setText(json.dumps(self._record_data, indent=2, ensure_ascii=False)) except (AutoControlException, OSError, ValueError, TypeError, RuntimeError) as error: - QMessageBox.warning(self, "Error", str(error)) + QMessageBox.warning(as_widget(self), "Error", str(error)) diff --git a/je_auto_control/gui/_screenshot_tab.py b/je_auto_control/gui/_screenshot_tab.py index a0026d91..4f889644 100644 --- a/je_auto_control/gui/_screenshot_tab.py +++ b/je_auto_control/gui/_screenshot_tab.py @@ -7,6 +7,7 @@ ) from je_auto_control.gui._validators import int_validator +from je_auto_control.gui._qt_typed import as_widget from je_auto_control.gui.language_wrapper.multi_language_wrapper import language_wrapper from je_auto_control.gui.selector import open_region_selector from je_auto_control.utils.exception.exceptions import AutoControlException @@ -92,15 +93,15 @@ def _get_screen_size(self): w, h = screen_size() self.screen_size_label.setText(f"{w} x {h}") except (AutoControlException, OSError, ValueError, TypeError, RuntimeError) as error: - QMessageBox.warning(self, "Error", str(error)) + QMessageBox.warning(as_widget(self), "Error", str(error)) def _browse_ss_path(self): - path, _ = QFileDialog.getSaveFileName(self, _t("save_screenshot"), "", "PNG (*.png);;All (*)") + path, _ = QFileDialog.getSaveFileName(as_widget(self), _t("save_screenshot"), "", "PNG (*.png);;All (*)") if path: self.ss_path_input.setText(path) def _pick_ss_region(self): - region = open_region_selector(self) + region = open_region_selector(as_widget(self)) if region is None: return x, y, w, h = region diff --git a/je_auto_control/gui/_script_tab.py b/je_auto_control/gui/_script_tab.py index f1c45bd6..9996baee 100644 --- a/je_auto_control/gui/_script_tab.py +++ b/je_auto_control/gui/_script_tab.py @@ -16,6 +16,7 @@ ) from je_auto_control.gui._tab_task import TabTask, was_stopped +from je_auto_control.gui._qt_typed import as_widget from je_auto_control.gui.language_wrapper.multi_language_wrapper import language_wrapper from je_auto_control.utils.exception.exceptions import AutoControlException from je_auto_control.utils.executor.action_executor import execute_action, execute_files @@ -87,7 +88,7 @@ def _build_script_tab(self) -> QWidget: self.script_result_text.setReadOnly(True) layout.addWidget(self.script_result_text) tab.setLayout(layout) - self._script_runs = TabTask(self) + self._script_runs = TabTask(as_widget(self)) self._script_runs.result.connect(self._show_script_result) self._script_runs.error.connect(self._show_script_error) return tab @@ -114,7 +115,7 @@ def _stop_script(self) -> None: self.script_result_text.setText(_t("task_stopping")) def _browse_script(self): - path, _ = QFileDialog.getOpenFileName(self, _t("load_script"), "", _JSON_FILE_FILTER) + path, _ = QFileDialog.getOpenFileName(as_widget(self), _t("load_script"), "", _JSON_FILE_FILTER) if path: self.script_path_input.setText(path) try: @@ -129,7 +130,7 @@ def _execute_script(self): self._start_script_run(functools.partial(_run_script_file, path)) def _browse_script_dir(self): - path = QFileDialog.getExistingDirectory(self, _t("execute_dir_label")) + path = QFileDialog.getExistingDirectory(as_widget(self), _t("execute_dir_label")) if path: self.script_dir_input.setText(path) diff --git a/je_auto_control/gui/_worker_thread.py b/je_auto_control/gui/_worker_thread.py index fbfd0c9f..8ae112ce 100644 --- a/je_auto_control/gui/_worker_thread.py +++ b/je_auto_control/gui/_worker_thread.py @@ -73,7 +73,9 @@ class WorkerHandle: """A started worker: running until its thread has returned from ``run()``.""" def __init__(self, worker: QObject) -> None: - self.worker = worker + # Any: a worker is duck-typed -- ``run``, a ``finished`` signal, optionally ``failed`` and + # ``request_stop`` -- and Qt's own type for it says none of that. + self.worker: Any = worker self._thread: Optional[threading.Thread] = None self._owner: Optional["weakref.ref[QObject]"] = None self._relay: Optional["weakref.ref[_Relay]"] = None @@ -250,11 +252,11 @@ def start_worker(owner: QObject, worker: QObject, *, """ reaper = _reaper() relay = _Relay(owner, on_done, on_thread_done, on_fail) - worker.finished.connect(relay.done) + handle = WorkerHandle(worker) + handle.worker.finished.connect(relay.done) failed = getattr(worker, "failed", None) if failed is not None: failed.connect(relay.fail) - handle = WorkerHandle(worker) handle._owner, handle._relay = weakref.ref(owner), weakref.ref(relay) # noqa: SLF001 # reason: set once _RUNNING[handle] = worker thread = threading.Thread(target=_run, args=(handle, relay, reaper), diff --git a/je_auto_control/gui/a11y_audit_tab.py b/je_auto_control/gui/a11y_audit_tab.py index 61344500..e0807abe 100644 --- a/je_auto_control/gui/a11y_audit_tab.py +++ b/je_auto_control/gui/a11y_audit_tab.py @@ -52,7 +52,7 @@ def __init__(self, parent: Optional[QWidget] = None) -> None: self._bg = QLineEdit() self._bg.setPlaceholderText("255, 255, 255") self._table = QTableWidget(0, len(_COLS)) - self._table.setEditTriggers(QAbstractItemView.NoEditTriggers) + self._table.setEditTriggers(QAbstractItemView.EditTrigger.NoEditTriggers) self._table.horizontalHeader().setStretchLastSection(True) self._summary = QLabel() self._task = TabTask(self) # the audit, off the GUI thread; one at a time @@ -114,7 +114,7 @@ def _render(self, report: dict) -> None: issue["message"]) for col, text in enumerate(values): item = QTableWidgetItem(str(text)) - item.setFlags(item.flags() & ~Qt.ItemIsEditable) + item.setFlags(item.flags() & ~Qt.ItemFlag.ItemIsEditable) self._table.setItem(row, col, item) self._summary.setText( _t("audit_summary") diff --git a/je_auto_control/gui/accessibility_tab.py b/je_auto_control/gui/accessibility_tab.py index c1e5902f..5f88705e 100644 --- a/je_auto_control/gui/accessibility_tab.py +++ b/je_auto_control/gui/accessibility_tab.py @@ -18,6 +18,7 @@ from je_auto_control.gui._dispose import release_resources from je_auto_control.gui._i18n_helpers import TranslatableMixin from je_auto_control.gui._tab_task import TabTask +from je_auto_control.gui._qt_typed import filled_item from je_auto_control.gui.language_wrapper.multi_language_wrapper import ( language_wrapper, ) @@ -69,11 +70,11 @@ def __init__(self, parent: Optional[QWidget] = None) -> None: self._window_filter = QLineEdit() self._name_filter = QLineEdit() self._table = QTableWidget(0, _COLUMN_COUNT) - self._table.setEditTriggers(QAbstractItemView.NoEditTriggers) - self._table.setSelectionBehavior(QAbstractItemView.SelectRows) + self._table.setEditTriggers(QAbstractItemView.EditTrigger.NoEditTriggers) + self._table.setSelectionBehavior(QAbstractItemView.SelectionBehavior.SelectRows) self._table.verticalHeader().setVisible(False) header = self._table.horizontalHeader() - header.setSectionResizeMode(QHeaderView.Interactive) + header.setSectionResizeMode(QHeaderView.ResizeMode.Interactive) header.setStretchLastSection(True) self._status = QLabel() self._task = TabTask(self) # one command at a time, off the GUI thread @@ -177,7 +178,7 @@ def _populate(self, elements) -> None: ) for col, text in enumerate(values): item = QTableWidgetItem(text) - item.setFlags(item.flags() & ~Qt.ItemIsEditable) + item.setFlags(item.flags() & ~Qt.ItemFlag.ItemIsEditable) self._table.setItem(row, col, item) def _click_selected(self) -> None: @@ -185,7 +186,7 @@ def _click_selected(self) -> None: if row < 0: self._status.setText(_t("a11y_no_selection")) return - app = self._table.item(row, 0).text() or None - role = self._table.item(row, 1).text() or None - name = self._table.item(row, 2).text() or None + app = filled_item(self._table, row, 0).text() or None + role = filled_item(self._table, row, 1).text() or None + name = filled_item(self._table, row, 2).text() or None self._start(_CLICK, functools.partial(_click, name, role, app)) diff --git a/je_auto_control/gui/admin_console_tab.py b/je_auto_control/gui/admin_console_tab.py index 778d3519..f9614571 100644 --- a/je_auto_control/gui/admin_console_tab.py +++ b/je_auto_control/gui/admin_console_tab.py @@ -78,10 +78,10 @@ def __init__(self, parent: Optional[QWidget] = None) -> None: # proxy and the operator can paste an https://… URL here. self._url_input.setPlaceholderText("http://host:9939") # NOSONAR python:S5332 self._token_input = QLineEdit() - self._token_input.setEchoMode(QLineEdit.Password) + self._token_input.setEchoMode(QLineEdit.EchoMode.Password) self._table = QTableWidget(0, 5) self._table.horizontalHeader().setSectionResizeMode( - QHeaderView.ResizeToContents, + QHeaderView.ResizeMode.ResizeToContents, ) self._actions_input = QTextEdit() self._actions_input.setPlaceholderText('[["AC_get_mouse_position"]]') @@ -270,7 +270,7 @@ def _apply_thumbnails(self, png_by_label: Dict[str, Optional[bytes]]) -> None: def _icon_for(png: Optional[bytes]) -> Optional[QIcon]: if not png: return None - image = QImage.fromData(png, "PNG") + image = QImage.fromData(png, "PNG") # type: ignore[arg-type] # reason: the stub says bytes; PySide6 raises ValueError for bytes if image.isNull(): return None scaled = image.scaled( diff --git a/je_auto_control/gui/config_sync_tab.py b/je_auto_control/gui/config_sync_tab.py index 2d286205..2e3f2e4c 100644 --- a/je_auto_control/gui/config_sync_tab.py +++ b/je_auto_control/gui/config_sync_tab.py @@ -60,7 +60,7 @@ def __init__(self, parent: Optional[QWidget] = None, self._tr_init() self._settings_store = settings if settings is not None else WindowSettings() self._inputs: Dict[str, QLineEdit] = {name: QLineEdit() for name in _FIELDS} - self._inputs["secret"].setEchoMode(QLineEdit.Password) + self._inputs["secret"].setEchoMode(QLineEdit.EchoMode.Password) self._restore_form() for name in _REMEMBERED: self._inputs[name].editingFinished.connect(self._remember_form) @@ -91,8 +91,8 @@ def _build_layout(self) -> None: root.addWidget(self._detail) root.addWidget(self._tr(QLabel(), "config_sync_conflicts_title")) root.addWidget(self._conflicts, stretch=1) - self._conflicts.horizontalHeader().setSectionResizeMode(QHeaderView.Stretch) - self._conflicts.setSelectionBehavior(QTableWidget.SelectRows) + self._conflicts.horizontalHeader().setSectionResizeMode(QHeaderView.ResizeMode.Stretch) + self._conflicts.setSelectionBehavior(QTableWidget.SelectionBehavior.SelectRows) self._apply_translations() def _apply_translations(self) -> None: @@ -175,7 +175,7 @@ def full_resync(self) -> None: def _confirm(self, question: str) -> bool: answer = QMessageBox.question(self, _t("tab_config_sync"), question) - return answer == QMessageBox.Yes + return answer == QMessageBox.StandardButton.Yes def _start(self, call: Callable[[], Dict[str, Any]], cancel: threading.Event) -> None: if self.is_busy(): @@ -213,7 +213,7 @@ def resolve_selected(self) -> None: if item is None: self._detail.setText(_t("config_sync_no_conflict_selected")) return - section, key, choice = item.data(Qt.UserRole) + section, key, choice = item.data(Qt.ItemDataRole.UserRole) try: session.config_sync_resolve(settings["server_url"], settings["user_id"], section, key, choice, **settings["options"]) @@ -268,8 +268,8 @@ def _render_conflicts(self, details: List[Dict[str, Any]]) -> None: str(choice.get("origin", "")), value) for column, text in enumerate(cells): item = QTableWidgetItem(text) - item.setFlags(Qt.ItemIsSelectable | Qt.ItemIsEnabled) - item.setData(Qt.UserRole, (detail["section"], detail["key"], index)) + item.setFlags(Qt.ItemFlag.ItemIsSelectable | Qt.ItemFlag.ItemIsEnabled) + item.setData(Qt.ItemDataRole.UserRole, (detail["section"], detail["key"], index)) self._conflicts.setItem(row, column, item) diff --git a/je_auto_control/gui/dag_tab.py b/je_auto_control/gui/dag_tab.py index 12c6e32f..d252250d 100644 --- a/je_auto_control/gui/dag_tab.py +++ b/je_auto_control/gui/dag_tab.py @@ -196,7 +196,7 @@ def _populate_table(self, result: DagRunResult) -> None: ) for col, text in enumerate(values): item = QTableWidgetItem(str(text)) - item.setFlags(Qt.ItemIsSelectable | Qt.ItemIsEnabled) + item.setFlags(Qt.ItemFlag.ItemIsSelectable | Qt.ItemFlag.ItemIsEnabled) self._table.setItem(row, col, item) self._table.resizeColumnsToContents() diff --git a/je_auto_control/gui/data_source_tab.py b/je_auto_control/gui/data_source_tab.py index 2d70af3a..1a18ba02 100644 --- a/je_auto_control/gui/data_source_tab.py +++ b/je_auto_control/gui/data_source_tab.py @@ -40,7 +40,7 @@ def __init__(self, parent: Optional[QWidget] = None) -> None: self._limit = QSpinBox() self._limit.setRange(0, 100000) self._table = QTableWidget(0, 0) - self._table.setEditTriggers(QAbstractItemView.NoEditTriggers) + self._table.setEditTriggers(QAbstractItemView.EditTrigger.NoEditTriggers) self._status = QLabel() self._build_layout() self._sync_visibility(self._kind.currentText()) diff --git a/je_auto_control/gui/device_matrix_tab.py b/je_auto_control/gui/device_matrix_tab.py index f7a444b6..177eab56 100644 --- a/je_auto_control/gui/device_matrix_tab.py +++ b/je_auto_control/gui/device_matrix_tab.py @@ -50,7 +50,7 @@ def __init__(self, parent: Optional[QWidget] = None) -> None: self._parallel.setRange(1, 64) self._parallel.setValue(4) self._table = QTableWidget(0, len(_COLS)) - self._table.setEditTriggers(QAbstractItemView.NoEditTriggers) + self._table.setEditTriggers(QAbstractItemView.EditTrigger.NoEditTriggers) self._table.horizontalHeader().setStretchLastSection(True) self._summary = QLabel() self._task: Optional[TaskHandle] = None # one run at a time @@ -129,7 +129,7 @@ def _render(self, report: dict) -> None: f"{res['duration_s']:.2f}s", res.get("error") or "") for col, text in enumerate(values): item = QTableWidgetItem(str(text)) - item.setFlags(item.flags() & ~Qt.ItemIsEditable) + item.setFlags(item.flags() & ~Qt.ItemFlag.ItemIsEditable) self._table.setItem(row, col, item) self._summary.setText( _t("dm_summary") diff --git a/je_auto_control/gui/email_triggers_tab.py b/je_auto_control/gui/email_triggers_tab.py index 2cd8d533..b478cde1 100644 --- a/je_auto_control/gui/email_triggers_tab.py +++ b/je_auto_control/gui/email_triggers_tab.py @@ -42,7 +42,7 @@ def __init__(self, parent: Optional[QWidget] = None) -> None: self._user_input = QLineEdit() self._user_input.setPlaceholderText("user@example.com") self._password_input = QLineEdit() - self._password_input.setEchoMode(QLineEdit.Password) + self._password_input.setEchoMode(QLineEdit.EchoMode.Password) self._password_input.setPlaceholderText(_t("eml_password_placeholder")) self._mailbox_input = QLineEdit("INBOX") self._search_input = QLineEdit("UNSEEN") @@ -57,11 +57,11 @@ def __init__(self, parent: Optional[QWidget] = None) -> None: self._status_label = QLabel() self._watcher_op = SlowOp(self) self._table = QTableWidget(0, 7) - self._table.setEditTriggers(QAbstractItemView.NoEditTriggers) - self._table.setSelectionBehavior(QAbstractItemView.SelectRows) + self._table.setEditTriggers(QAbstractItemView.EditTrigger.NoEditTriggers) + self._table.setSelectionBehavior(QAbstractItemView.SelectionBehavior.SelectRows) self._table.verticalHeader().setVisible(False) self._table.horizontalHeader().setSectionResizeMode( - QHeaderView.Interactive, + QHeaderView.ResizeMode.Interactive, ) self._table.horizontalHeader().setStretchLastSection(True) self._apply_table_headers() @@ -253,5 +253,5 @@ def _refresh(self) -> None: ) for col, text in enumerate(values): item = QTableWidgetItem(text) - item.setFlags(item.flags() & ~Qt.ItemIsEditable) + item.setFlags(item.flags() & ~Qt.ItemFlag.ItemIsEditable) self._table.setItem(row, col, item) diff --git a/je_auto_control/gui/flakiness_tab.py b/je_auto_control/gui/flakiness_tab.py index 6536a0c3..9acc7e9b 100644 --- a/je_auto_control/gui/flakiness_tab.py +++ b/je_auto_control/gui/flakiness_tab.py @@ -35,12 +35,12 @@ def __init__(self, parent: Optional[QWidget] = None) -> None: super().__init__(parent) self._tr_init() self._table = QTableWidget(0, len(_COLUMN_KEYS)) - self._table.setEditTriggers(QAbstractItemView.NoEditTriggers) - self._table.setSelectionBehavior(QAbstractItemView.SelectRows) + self._table.setEditTriggers(QAbstractItemView.EditTrigger.NoEditTriggers) + self._table.setSelectionBehavior(QAbstractItemView.SelectionBehavior.SelectRows) self._table.verticalHeader().setVisible(False) self._table.horizontalHeader().setStretchLastSection(True) self._table.horizontalHeader().setSectionResizeMode( - QHeaderView.Interactive, + QHeaderView.ResizeMode.Interactive, ) self._limit = QSpinBox() self._limit.setRange(1, 100000) @@ -112,5 +112,5 @@ def _set_row(self, row: int, entry) -> None: ) for col, text in enumerate(values): item = QTableWidgetItem(text) - item.setFlags(item.flags() & ~Qt.ItemIsEditable) + item.setFlags(item.flags() & ~Qt.ItemFlag.ItemIsEditable) self._table.setItem(row, col, item) diff --git a/je_auto_control/gui/flow_editor/scene.py b/je_auto_control/gui/flow_editor/scene.py index 18baeafc..4e1e1628 100644 --- a/je_auto_control/gui/flow_editor/scene.py +++ b/je_auto_control/gui/flow_editor/scene.py @@ -33,7 +33,7 @@ def __init__(self, position: FlowNodePosition, self._on_clicked = on_clicked self._set_default_brush() self.setPen(QPen(_NODE_BORDER, 1.5)) - self.setFlag(QGraphicsItem.ItemIsSelectable, True) + self.setFlag(QGraphicsItem.GraphicsItemFlag.ItemIsSelectable, True) self.setAcceptHoverEvents(True) self._build_label(position) diff --git a/je_auto_control/gui/flow_editor/tab.py b/je_auto_control/gui/flow_editor/tab.py index 2ddd581c..2e18d819 100644 --- a/je_auto_control/gui/flow_editor/tab.py +++ b/je_auto_control/gui/flow_editor/tab.py @@ -38,7 +38,7 @@ def __init__(self, parent: Optional[QWidget] = None) -> None: self._scene = FlowGraphScene(self) self._view = QGraphicsView(self._scene) self._view.setRenderHints(self._view.renderHints()) - self._view.setDragMode(QGraphicsView.RubberBandDrag) + self._view.setDragMode(QGraphicsView.DragMode.RubberBandDrag) self._steps: List[Step] = [] # The other keys of a wrapped file, written back by Save. self._file_extras: Optional[dict] = None @@ -56,7 +56,7 @@ def _build_layout(self) -> None: # Open/save/zoom/fit commands run from the Actions menu; the tab # keeps only the graph view, the inspector, and the status line. root = QVBoxLayout(self) - splitter = QSplitter(Qt.Horizontal) + splitter = QSplitter(Qt.Orientation.Horizontal) splitter.addWidget(self._view) splitter.addWidget(self._inspector) splitter.setStretchFactor(0, 3) @@ -138,7 +138,7 @@ def _on_fit(self) -> None: rect = self._scene.itemsBoundingRect() if rect.isEmpty(): return - self._view.fitInView(rect, Qt.KeepAspectRatio) + self._view.fitInView(rect, Qt.AspectRatioMode.KeepAspectRatio) def _on_node_selected(self, path: Tuple) -> None: step = _resolve_step(self._steps, path) diff --git a/je_auto_control/gui/hotkeys_tab.py b/je_auto_control/gui/hotkeys_tab.py index 10af8e23..45a77328 100644 --- a/je_auto_control/gui/hotkeys_tab.py +++ b/je_auto_control/gui/hotkeys_tab.py @@ -10,6 +10,7 @@ from je_auto_control.gui._dispose import release_resources from je_auto_control.gui._i18n_helpers import TranslatableMixin from je_auto_control.gui._slow_op import SlowOp +from je_auto_control.gui._qt_typed import filled_item from je_auto_control.gui.language_wrapper.multi_language_wrapper import ( language_wrapper, ) @@ -130,7 +131,7 @@ def _on_remove(self) -> None: row = self._table.currentRow() if row < 0: return - bid = self._table.item(row, 0).text() + bid = filled_item(self._table, row, 0).text() default_hotkey_daemon.unbind(bid) self._refresh() diff --git a/je_auto_control/gui/main_widget.py b/je_auto_control/gui/main_widget.py index ed03d1e0..2af3c61a 100644 --- a/je_auto_control/gui/main_widget.py +++ b/je_auto_control/gui/main_widget.py @@ -167,7 +167,7 @@ def _build_remote_desktop_tab() -> QWidget: f"Underlying error: {import_error!r}", ) message.setWordWrap(True) - message.setTextInteractionFlags(Qt.TextSelectableByMouse) + message.setTextInteractionFlags(Qt.TextInteractionFlag.TextSelectableByMouse) layout.addWidget(message) layout.addStretch() return placeholder diff --git a/je_auto_control/gui/main_window.py b/je_auto_control/gui/main_window.py index 6165ecc5..2430e97a 100644 --- a/je_auto_control/gui/main_window.py +++ b/je_auto_control/gui/main_window.py @@ -80,8 +80,9 @@ def __init__(self, settings: Optional[WindowSettings] = None) -> None: self.setCentralWidget(self.auto_control_gui_widget) self._build_navigation() - self._view_menu: QMenu = None - self._actions_menu: QMenu = None + # Both are assigned by _build_menu_bar, two lines down. + self._view_menu: QMenu + self._actions_menu: QMenu self._tab_actions: list = [] self._build_menu_bar() self.auto_control_gui_widget.tabs_changed.connect(self._rebuild_tabs_menu) diff --git a/je_auto_control/gui/mobile_tab.py b/je_auto_control/gui/mobile_tab.py index 06cae4b7..8c5afd47 100644 --- a/je_auto_control/gui/mobile_tab.py +++ b/je_auto_control/gui/mobile_tab.py @@ -21,6 +21,7 @@ from je_auto_control.gui._dispose import release_resources from je_auto_control.gui._i18n_helpers import TranslatableMixin from je_auto_control.gui._tab_task import TabTask +from je_auto_control.gui._qt_typed import filled_item from je_auto_control.gui.language_wrapper.multi_language_wrapper import ( language_wrapper, ) @@ -65,7 +66,7 @@ def __init__(self, parent: Optional[QWidget] = None) -> None: self._result = QPlainTextEdit() self._result.setReadOnly(True) self._table = QTableWidget(0, len(_COLS)) - self._table.setEditTriggers(QAbstractItemView.NoEditTriggers) + self._table.setEditTriggers(QAbstractItemView.EditTrigger.NoEditTriggers) self._table.horizontalHeader().setStretchLastSection(True) self._note = QLabel() self._note.setWordWrap(True) @@ -139,7 +140,7 @@ def result_text(self) -> str: def capability_rows(self) -> Dict[str, str]: """Capability name to state, as the table shows it.""" - return {self._table.item(row, 0).text(): self._table.item(row, 1).text() + return {filled_item(self._table, row, 0).text(): filled_item(self._table, row, 1).text() for row in range(self._table.rowCount())} def _current_platform(self) -> str: @@ -210,6 +211,6 @@ def _render(self, report: Dict[str, Any]) -> None: for row, (name, capability) in enumerate(capabilities.items()): for col, text in enumerate((name, capability["state"], capability["reason"])): item = QTableWidgetItem(str(text)) - item.setFlags(item.flags() & ~Qt.ItemIsEditable) + item.setFlags(item.flags() & ~Qt.ItemFlag.ItemIsEditable) self._table.setItem(row, col, item) self._result.setPlainText(json.dumps(report, indent=2, ensure_ascii=False)) diff --git a/je_auto_control/gui/presence_tab.py b/je_auto_control/gui/presence_tab.py index 69b63bee..02cb159c 100644 --- a/je_auto_control/gui/presence_tab.py +++ b/je_auto_control/gui/presence_tab.py @@ -80,7 +80,7 @@ def _build_layout(self) -> None: root.addWidget(self._table, stretch=1) root.addWidget(self._status) header = self._table.horizontalHeader() - header.setSectionResizeMode(QHeaderView.Stretch) + header.setSectionResizeMode(QHeaderView.ResizeMode.Stretch) self._apply_translations() def menu_actions(self) -> list: @@ -116,7 +116,7 @@ def _populate_row(self, index: int, row: ViewerPresence) -> None: cursor, row.last_seen_iso) for col, text in enumerate(values): item = QTableWidgetItem(str(text)) - item.setFlags(Qt.ItemIsSelectable | Qt.ItemIsEnabled) + item.setFlags(Qt.ItemFlag.ItemIsSelectable | Qt.ItemFlag.ItemIsEnabled) self._table.setItem(index, col, item) def _on_registry_event(self, _viewer_id: str, diff --git a/je_auto_control/gui/profiler_tab.py b/je_auto_control/gui/profiler_tab.py index dc4c2ef9..42a74686 100644 --- a/je_auto_control/gui/profiler_tab.py +++ b/je_auto_control/gui/profiler_tab.py @@ -38,12 +38,12 @@ def __init__(self, parent: Optional[QWidget] = None) -> None: super().__init__(parent) self._tr_init() self._table = QTableWidget(0, _COLUMN_COUNT) - self._table.setEditTriggers(QAbstractItemView.NoEditTriggers) - self._table.setSelectionBehavior(QAbstractItemView.SelectRows) + self._table.setEditTriggers(QAbstractItemView.EditTrigger.NoEditTriggers) + self._table.setSelectionBehavior(QAbstractItemView.SelectionBehavior.SelectRows) self._table.verticalHeader().setVisible(False) self._apply_table_headers() header = self._table.horizontalHeader() - header.setSectionResizeMode(QHeaderView.Interactive) + header.setSectionResizeMode(QHeaderView.ResizeMode.Interactive) header.setStretchLastSection(True) self._totalbar = QProgressBar() self._totalbar.setRange(0, 100) @@ -134,5 +134,5 @@ def _set_row(self, row: int, stats: ActionStats, share: float) -> None: ) for col, text in enumerate(values): item = QTableWidgetItem(text) - item.setFlags(item.flags() & ~Qt.ItemIsEditable) + item.setFlags(item.flags() & ~Qt.ItemFlag.ItemIsEditable) self._table.setItem(row, col, item) diff --git a/je_auto_control/gui/rbac_users_panel.py b/je_auto_control/gui/rbac_users_panel.py index f3b6b2e6..4abb208c 100644 --- a/je_auto_control/gui/rbac_users_panel.py +++ b/je_auto_control/gui/rbac_users_panel.py @@ -19,6 +19,7 @@ ) from je_auto_control.gui._i18n_helpers import TranslatableMixin +from je_auto_control.gui._qt_typed import filled_item from je_auto_control.gui.language_wrapper.multi_language_wrapper import ( language_wrapper, ) @@ -58,7 +59,7 @@ def __init__(self, parent: Optional[QWidget] = None) -> None: self._token_value.setReadOnly(True) self._status_label = QLabel() self._status_label.setWordWrap(True) - self._status_label.setTextInteractionFlags(Qt.TextSelectableByMouse) + self._status_label.setTextInteractionFlags(Qt.TextInteractionFlag.TextSelectableByMouse) self._build_layout() self._retitle_columns() @@ -144,9 +145,9 @@ def _on_row_selected(self) -> None: row = self._table.currentRow() if row < 0 or self._table.item(row, 0) is None: return - self._id_input.setText(self._table.item(row, 0).text()) - self._name_input.setText(self._table.item(row, 1).text()) - self._role_input.setCurrentText(self._table.item(row, 2).text()) + self._id_input.setText(filled_item(self._table, row, 0).text()) + self._name_input.setText(filled_item(self._table, row, 1).text()) + self._role_input.setCurrentText(filled_item(self._table, row, 2).text()) def _target(self) -> Optional[Tuple[str, str]]: """``(store path, user id)`` of the operation, or ``None`` with the reason shown.""" diff --git a/je_auto_control/gui/remote_desktop/_helpers.py b/je_auto_control/gui/remote_desktop/_helpers.py index 8c46dba2..07ff3cd2 100644 --- a/je_auto_control/gui/remote_desktop/_helpers.py +++ b/je_auto_control/gui/remote_desktop/_helpers.py @@ -2,7 +2,7 @@ import inspect import ssl import weakref -from typing import Callable, Optional +from typing import Any, Callable, Optional from PySide6.QtCore import Qt from PySide6.QtGui import QKeyEvent @@ -74,7 +74,7 @@ def _scroll_amount(angle_delta: int) -> int: return 0 -def wire_remote_input(window: QWidget, send: Callable[[dict], None]) -> None: +def wire_remote_input(window: Any, send: Callable[[dict], None]) -> None: """Forward a remote screen window's mouse and keyboard signals to ``send``. Every viewer that opens a ``RemoteScreenWindow`` wires it here, so one @@ -201,7 +201,7 @@ def set_body_layout(self, layout) -> None: self._body.setLayout(layout) -def displaced_notifier(panel: QWidget) -> Callable[[str, str], None]: +def displaced_notifier(panel: Any) -> Callable[[str, str], None]: """Return the registry ``on_displaced`` callback for ``panel``. The registry calls it on whichever thread replaced the panel's host or diff --git a/je_auto_control/gui/remote_desktop/advanced_group.py b/je_auto_control/gui/remote_desktop/advanced_group.py index ac248ef5..3183ee0d 100644 --- a/je_auto_control/gui/remote_desktop/advanced_group.py +++ b/je_auto_control/gui/remote_desktop/advanced_group.py @@ -12,7 +12,7 @@ """ from __future__ import annotations -from typing import Any, Protocol +from typing import Any, Protocol, TypeVar from PySide6.QtWidgets import ( QComboBox, QGridLayout, QGroupBox, QLabel, QLineEdit, QWidget, @@ -26,6 +26,9 @@ DEFAULT_STUN = "stun:stun.l.google.com:19302" +_W = TypeVar("_W", bound=QWidget) + + class AdvancedGroupHost(Protocol): """A WebRTC panel, seen from the advanced group it hosts. @@ -41,7 +44,7 @@ class AdvancedGroupHost(Protocol): _turn_cred_edit: Any _hw_codec_combo: Any - def _tr(self, widget: QWidget, key: str, setter: str = "") -> QWidget: + def _tr(self, widget: _W, key: str, setter: str = "") -> _W: """Register ``widget`` for live re-translation and return it.""" def _on_hw_codec_changed(self) -> None: diff --git a/je_auto_control/gui/remote_desktop/connection_screen.py b/je_auto_control/gui/remote_desktop/connection_screen.py index 9c3e5937..c69be017 100644 --- a/je_auto_control/gui/remote_desktop/connection_screen.py +++ b/je_auto_control/gui/remote_desktop/connection_screen.py @@ -557,7 +557,7 @@ def _on_frame(self, payload: bytes) -> None: window = self._screen_window if window is None: return - image = QImage.fromData(payload, "JPEG") + image = QImage.fromData(payload, "JPEG") # type: ignore[arg-type] # reason: the stub says bytes; PySide6 raises ValueError for bytes if not image.isNull(): window.set_image(image) @@ -579,7 +579,7 @@ def _open_screen_window(self, title: str) -> None: # If frames arrived before this window existed (race with the # motion-dedup capture path), apply the most recent now. if self._pending_frame is not None: - image = QImage.fromData(self._pending_frame, "JPEG") + image = QImage.fromData(self._pending_frame, "JPEG") # type: ignore[arg-type] # reason: the stub says bytes; PySide6 raises ValueError for bytes if not image.isNull(): self._screen_window.set_image(image) self._screen_window.show() diff --git a/je_auto_control/gui/remote_desktop/frame_display.py b/je_auto_control/gui/remote_desktop/frame_display.py index 2ca3bf96..f133c260 100644 --- a/je_auto_control/gui/remote_desktop/frame_display.py +++ b/je_auto_control/gui/remote_desktop/frame_display.py @@ -82,11 +82,12 @@ def has_image(self) -> bool: def paintEvent(self, _event) -> None: # noqa: N802 Qt override painter = QPainter(self) painter.fillRect(self.rect(), Qt.GlobalColor.black) - if not self.has_image(): + image = self._image + if image is None or not self.has_image(): return target = self._fit_rect() if target.isValid(): - painter.drawImage(target, self._image) + painter.drawImage(target, image) self._paint_remote_cursor(painter, target) def _paint_remote_cursor(self, painter: QPainter, target: QRect) -> None: diff --git a/je_auto_control/gui/remote_desktop/host_panel.py b/je_auto_control/gui/remote_desktop/host_panel.py index 17d48ca3..43926303 100644 --- a/je_auto_control/gui/remote_desktop/host_panel.py +++ b/je_auto_control/gui/remote_desktop/host_panel.py @@ -428,6 +428,6 @@ def _refresh_preview(self) -> None: frame = host.latest_frame() if frame is None: return - image = QImage.fromData(frame, "JPEG") + image = QImage.fromData(frame, "JPEG") # type: ignore[arg-type] # reason: the stub says bytes; PySide6 raises ValueError for bytes if not image.isNull(): self._preview.set_image(image) diff --git a/je_auto_control/gui/remote_desktop/tab.py b/je_auto_control/gui/remote_desktop/tab.py index 5b05a4ae..59c0cd18 100644 --- a/je_auto_control/gui/remote_desktop/tab.py +++ b/je_auto_control/gui/remote_desktop/tab.py @@ -148,8 +148,8 @@ def _build_sub_panel_list(self) -> List[Tuple[QWidget, str]]: ] webrtc_host, webrtc_viewer = _load_webrtc_panels() if webrtc_host is not None and webrtc_viewer is not None: - self._webrtc_host_panel = webrtc_host - self._webrtc_viewer_panel = webrtc_viewer + self._webrtc_host_panel: Optional[QWidget] = webrtc_host + self._webrtc_viewer_panel: Optional[QWidget] = webrtc_viewer sub_panels.append((webrtc_host, "rd_webrtc_host_tab")) sub_panels.append((webrtc_viewer, "rd_webrtc_viewer_tab")) else: diff --git a/je_auto_control/gui/remote_desktop/trusted_group.py b/je_auto_control/gui/remote_desktop/trusted_group.py index c403d2ba..d3e7c3b9 100644 --- a/je_auto_control/gui/remote_desktop/trusted_group.py +++ b/je_auto_control/gui/remote_desktop/trusted_group.py @@ -12,7 +12,7 @@ """ from __future__ import annotations -from typing import Any, Protocol +from typing import Any, Protocol, TypeVar from PySide6.QtWidgets import ( QGroupBox, QHBoxLayout, QPushButton, QVBoxLayout, QWidget, @@ -21,12 +21,15 @@ from je_auto_control.gui.remote_desktop.webrtc_dialogs import TrustedViewersList +_W = TypeVar("_W", bound=QWidget) + + class TrustedGroupHost(Protocol): """A WebRTC host panel, seen from the trusted-viewers group it hosts.""" _trusted_list: Any - def _tr(self, widget: QWidget, key: str, setter: str = "") -> QWidget: + def _tr(self, widget: _W, key: str, setter: str = "") -> _W: """Register ``widget`` for live re-translation and return it.""" def _on_remove_trust(self, viewer_id: str) -> None: diff --git a/je_auto_control/gui/remote_desktop/viewer_panel.py b/je_auto_control/gui/remote_desktop/viewer_panel.py index b74bfc14..4f36203e 100644 --- a/je_auto_control/gui/remote_desktop/viewer_panel.py +++ b/je_auto_control/gui/remote_desktop/viewer_panel.py @@ -429,7 +429,7 @@ def _refresh_status(self) -> None: # --- slot handlers (run on GUI thread) ----------------------------- def _on_frame_main(self, payload: bytes) -> None: - image = QImage.fromData(payload, "JPEG") + image = QImage.fromData(payload, "JPEG") # type: ignore[arg-type] # reason: the stub says bytes; PySide6 raises ValueError for bytes if image.isNull(): return if self._screen_window is None: diff --git a/je_auto_control/gui/remote_desktop/webrtc_dialogs.py b/je_auto_control/gui/remote_desktop/webrtc_dialogs.py index 34014211..547b5f0a 100644 --- a/je_auto_control/gui/remote_desktop/webrtc_dialogs.py +++ b/je_auto_control/gui/remote_desktop/webrtc_dialogs.py @@ -481,7 +481,8 @@ def _on_use(self) -> None: row = self._table.currentRow() if row < 0: return - host_id = self._table.item(row, 0).text() if self._table.item(row, 0) else "" + item = self._table.item(row, 0) + host_id = item.text() if item is not None else "" if host_id and host_id in [s.get("host_id") for s in self._services.values()]: for svc in self._services.values(): if svc.get("host_id") == host_id: diff --git a/je_auto_control/gui/rest_api_tab.py b/je_auto_control/gui/rest_api_tab.py index 53b56692..639d45f8 100644 --- a/je_auto_control/gui/rest_api_tab.py +++ b/je_auto_control/gui/rest_api_tab.py @@ -50,9 +50,9 @@ def __init__(self, parent: Optional[QWidget] = None) -> None: self._audit_check = QCheckBox() self._audit_check.setChecked(True) self._url_value = QLabel("-") - self._url_value.setTextInteractionFlags(Qt.TextSelectableByMouse) + self._url_value.setTextInteractionFlags(Qt.TextInteractionFlag.TextSelectableByMouse) self._token_value = QLabel("-") - self._token_value.setTextInteractionFlags(Qt.TextSelectableByMouse) + self._token_value.setTextInteractionFlags(Qt.TextInteractionFlag.TextSelectableByMouse) self._status_label = QLabel() # Start replaces a running server and Stop joins its thread: both off the GUI thread. self._server_op = SlowOp(self) diff --git a/je_auto_control/gui/run_history_tab.py b/je_auto_control/gui/run_history_tab.py index 7696c3ef..0313d061 100644 --- a/je_auto_control/gui/run_history_tab.py +++ b/je_auto_control/gui/run_history_tab.py @@ -80,27 +80,27 @@ def __init__(self, parent: Optional[QWidget] = None) -> None: self._populate_filter() self._filter.currentIndexChanged.connect(self._refresh) self._table = QTableWidget(0, _COLUMN_COUNT) - self._table.setEditTriggers(QAbstractItemView.NoEditTriggers) - self._table.setSelectionBehavior(QAbstractItemView.SelectRows) + self._table.setEditTriggers(QAbstractItemView.EditTrigger.NoEditTriggers) + self._table.setSelectionBehavior(QAbstractItemView.SelectionBehavior.SelectRows) self._table.verticalHeader().setVisible(False) self._apply_table_headers() header = self._table.horizontalHeader() - header.setSectionResizeMode(QHeaderView.Interactive) + header.setSectionResizeMode(QHeaderView.ResizeMode.Interactive) header.setStretchLastSection(True) self._count_label = QLabel() self._journal_label = QLabel() - self._journal_label.setTextInteractionFlags(Qt.TextSelectableByMouse) + self._journal_label.setTextInteractionFlags(Qt.TextInteractionFlag.TextSelectableByMouse) self._timeline = RunHistoryTimeline() self._timeline.run_clicked.connect(self._on_timeline_clicked) self._thumb_label = QLabel() - self._thumb_label.setAlignment(Qt.AlignCenter) + self._thumb_label.setAlignment(Qt.AlignmentFlag.AlignCenter) self._thumb_label.setMinimumSize(220, 160) - self._thumb_label.setFrameShape(QFrame.StyledPanel) + self._thumb_label.setFrameShape(QFrame.Shape.StyledPanel) self._thumb_label.setScaledContents(False) self._thumb_caption = QLabel() self._thumb_caption.setWordWrap(True) - self._thumb_caption.setAlignment(Qt.AlignTop) - self._thumb_caption.setTextInteractionFlags(Qt.TextSelectableByMouse) + self._thumb_caption.setAlignment(Qt.AlignmentFlag.AlignTop) + self._thumb_caption.setTextInteractionFlags(Qt.TextInteractionFlag.TextSelectableByMouse) self._timer = QTimer(self) self._timer.setInterval(_REFRESH_INTERVAL_MS) self._timer.timeout.connect(self._refresh) @@ -162,7 +162,7 @@ def _build_layout(self) -> None: root.addWidget(timeline_label) root.addWidget(self._timeline) - splitter = QSplitter(Qt.Horizontal) + splitter = QSplitter(Qt.Orientation.Horizontal) splitter.addWidget(self._table) preview = QWidget() preview_layout = QVBoxLayout(preview) @@ -238,9 +238,9 @@ def _refresh_journal_label(self) -> None: def _on_clear(self) -> None: reply = QMessageBox.question( self, _t("rh_clear"), _t("rh_confirm_clear"), - QMessageBox.Yes | QMessageBox.No, QMessageBox.No, + QMessageBox.StandardButton.Yes | QMessageBox.StandardButton.No, QMessageBox.StandardButton.No, ) - if reply == QMessageBox.Yes: + if reply == QMessageBox.StandardButton.Yes: default_history_store.clear() self._refresh() @@ -278,7 +278,7 @@ def _set_row(self, row: int, record) -> None: ) for col, text in enumerate(values): item = QTableWidgetItem(text) - item.setFlags(item.flags() & ~Qt.ItemIsEditable) + item.setFlags(item.flags() & ~Qt.ItemFlag.ItemIsEditable) self._table.setItem(row, col, item) def _selected_record(self) -> Optional[RunRecord]: @@ -316,8 +316,8 @@ def _refresh_preview(self) -> None: self._thumb_label.setText(_t("rh_artifact_missing")) else: self._thumb_label.setPixmap(pixmap.scaled( - self._thumb_label.size(), Qt.KeepAspectRatio, - Qt.SmoothTransformation, + self._thumb_label.size(), Qt.AspectRatioMode.KeepAspectRatio, + Qt.TransformationMode.SmoothTransformation, )) caption = ( f"#{record.id} • {record.source_type}/{record.source_id}\n" diff --git a/je_auto_control/gui/run_history_timeline.py b/je_auto_control/gui/run_history_timeline.py index 485713e2..4730315c 100644 --- a/je_auto_control/gui/run_history_timeline.py +++ b/je_auto_control/gui/run_history_timeline.py @@ -43,7 +43,7 @@ class RunHistoryTimeline(QWidget): def __init__(self, parent: Optional[QWidget] = None) -> None: super().__init__(parent) self.setMinimumHeight(96) - self.setSizePolicy(QSizePolicy.Expanding, QSizePolicy.Fixed) + self.setSizePolicy(QSizePolicy.Policy.Expanding, QSizePolicy.Policy.Fixed) self._records: List[RunRecord] = [] self._bars: List[_Bar] = [] self._range: Tuple[float, float] = (0.0, 0.0) @@ -65,7 +65,7 @@ def paintEvent(self, event) -> None: del event painter = QPainter(self) try: - painter.setRenderHint(QPainter.Antialiasing, True) + painter.setRenderHint(QPainter.RenderHint.Antialiasing, True) painter.fillRect(self.rect(), self.palette().window()) self._bars = self._layout_bars() self._draw_axis(painter) @@ -75,7 +75,7 @@ def paintEvent(self, event) -> None: painter.end() def mousePressEvent(self, event: QMouseEvent) -> None: - if event.button() != Qt.LeftButton: + if event.button() != Qt.MouseButton.LeftButton: super().mousePressEvent(event) return for bar in self._bars: @@ -126,7 +126,7 @@ def _layout_bars(self) -> List[_Bar]: def _draw_axis(self, painter: QPainter) -> None: if not self._records: painter.setPen(self.palette().text().color()) - painter.drawText(self.rect(), Qt.AlignCenter, "no runs yet") + painter.drawText(self.rect(), Qt.AlignmentFlag.AlignCenter, "no runs yet") return font = QFont(painter.font()) font.setPointSize(max(7, font.pointSize() - 1)) @@ -154,6 +154,6 @@ def _draw_bar(self, painter: QPainter, bar: _Bar) -> None: if bar.record.id == self._highlight_id: painter.setPen(QColor(255, 255, 255, 220)) else: - painter.setPen(Qt.NoPen) + painter.setPen(Qt.PenStyle.NoPen) painter.setBrush(colour) painter.drawRoundedRect(bar.rect, 3, 3) diff --git a/je_auto_control/gui/scheduler_tab.py b/je_auto_control/gui/scheduler_tab.py index a7361871..4e60ded7 100644 --- a/je_auto_control/gui/scheduler_tab.py +++ b/je_auto_control/gui/scheduler_tab.py @@ -10,6 +10,7 @@ from je_auto_control.gui._dispose import release_resources from je_auto_control.gui._i18n_helpers import TranslatableMixin from je_auto_control.gui._slow_op import SlowOp +from je_auto_control.gui._qt_typed import filled_item from je_auto_control.gui.language_wrapper.multi_language_wrapper import ( language_wrapper, ) @@ -138,7 +139,7 @@ def _on_remove(self) -> None: row = self._table.currentRow() if row < 0: return - job_id = self._table.item(row, 0).text() + job_id = filled_item(self._table, row, 0).text() default_scheduler.remove_job(job_id) self._refresh_table() diff --git a/je_auto_control/gui/script_builder/step_list_view.py b/je_auto_control/gui/script_builder/step_list_view.py index 617afbd3..92b8a985 100644 --- a/je_auto_control/gui/script_builder/step_list_view.py +++ b/je_auto_control/gui/script_builder/step_list_view.py @@ -176,6 +176,6 @@ def _add_body_node(parent: QTreeWidgetItem, body_key: str) -> QTreeWidgetItem: def _find_body_item(parent: QTreeWidgetItem, body_key: str) -> Optional[QTreeWidgetItem]: for i in range(parent.childCount()): child = parent.child(i) - if child.data(0, ROLE_BODY_KEY) == body_key: + if child is not None and child.data(0, ROLE_BODY_KEY) == body_key: return child return None diff --git a/je_auto_control/gui/secrets_tab.py b/je_auto_control/gui/secrets_tab.py index 71244ab8..d80d430c 100644 --- a/je_auto_control/gui/secrets_tab.py +++ b/je_auto_control/gui/secrets_tab.py @@ -28,9 +28,9 @@ def __init__(self, parent: Optional[QWidget] = None) -> None: self._tr_init() self._status_label = QLabel() self._passphrase = QLineEdit() - self._passphrase.setEchoMode(QLineEdit.Password) + self._passphrase.setEchoMode(QLineEdit.EchoMode.Password) self._list = QListWidget() - self._list.setEditTriggers(QAbstractItemView.NoEditTriggers) + self._list.setEditTriggers(QAbstractItemView.EditTrigger.NoEditTriggers) self._build_layout() self._refresh_status() @@ -139,7 +139,7 @@ def _on_add(self) -> None: return value, ok = QInputDialog.getText( self, _t("secret_add"), _t("secret_value_prompt"), - QLineEdit.Password, + QLineEdit.EchoMode.Password, ) if not ok: return @@ -164,13 +164,13 @@ def _on_remove(self) -> None: def _on_change_passphrase(self) -> None: old, ok = QInputDialog.getText( self, _t("secret_change_passphrase"), - _t("secret_old_passphrase_prompt"), QLineEdit.Password, + _t("secret_old_passphrase_prompt"), QLineEdit.EchoMode.Password, ) if not ok: return new, ok = QInputDialog.getText( self, _t("secret_change_passphrase"), - _t("secret_new_passphrase_prompt"), QLineEdit.Password, + _t("secret_new_passphrase_prompt"), QLineEdit.EchoMode.Password, ) if not ok or not new: return diff --git a/je_auto_control/gui/self_healing_tab.py b/je_auto_control/gui/self_healing_tab.py index aded9759..daa16fb4 100644 --- a/je_auto_control/gui/self_healing_tab.py +++ b/je_auto_control/gui/self_healing_tab.py @@ -66,7 +66,7 @@ def __init__(self, parent: Optional[QWidget] = None) -> None: self._candidate_input = QLineEdit() self._revision_input = QLineEdit() self._compare_table = QTableWidget(0, len(COMPARISON_COLUMNS)) - self._compare_table.setEditTriggers(QTableWidget.NoEditTriggers) + self._compare_table.setEditTriggers(QTableWidget.EditTrigger.NoEditTriggers) self._compare_table.verticalHeader().setVisible(False) self._report_view = QPlainTextEdit() self._report_view.setReadOnly(True) @@ -231,7 +231,7 @@ def _fill_comparison(self, report: dict) -> None: if name == "version" and values["baseline"]: text = _t("self_heal_cmp_baseline").replace("{name}", text) item = QTableWidgetItem(text) - item.setFlags(Qt.ItemIsSelectable | Qt.ItemIsEnabled) + item.setFlags(Qt.ItemFlag.ItemIsSelectable | Qt.ItemFlag.ItemIsEnabled) self._compare_table.setItem(row, col, item) self._compare_table.resizeColumnsToContents() @@ -270,7 +270,7 @@ def _accept(self, revision_id: str) -> dict: reply = QMessageBox.question( self, _t("self_heal_rev_accept"), _t("self_heal_accept_unvalidated"), ) - if reply != QMessageBox.Yes: + if reply != QMessageBox.StandardButton.Yes: return {"accepted": False, "revision_id": revision_id} allow = True return accept_template_revision(revision_id, allow_unvalidated=allow).to_dict() @@ -349,7 +349,7 @@ def _on_clear_log(self) -> None: reply = QMessageBox.question( self, _t("self_heal_clear"), _t("self_heal_clear_confirm"), ) - if reply != QMessageBox.Yes: + if reply != QMessageBox.StandardButton.Yes: return default_heal_log.clear() self.refresh_log() @@ -369,7 +369,7 @@ def refresh_log(self) -> None: ) for col, text in enumerate(values): item = QTableWidgetItem(str(text)) - item.setFlags(Qt.ItemIsSelectable | Qt.ItemIsEnabled) + item.setFlags(Qt.ItemFlag.ItemIsSelectable | Qt.ItemFlag.ItemIsEnabled) self._table.setItem(row, col, item) self._table.resizeColumnsToContents() @@ -405,9 +405,10 @@ def _set_form_labels(group: QGroupBox, keys: Sequence[str]) -> None: if not isinstance(layout, QFormLayout): return for row, key in enumerate(keys): - item = layout.itemAt(row, QFormLayout.LabelRole) - if item is not None and isinstance(item.widget(), QLabel): - item.widget().setText(_t(key) if key else "") + item = layout.itemAt(row, QFormLayout.ItemRole.LabelRole) + label = item.widget() if item is not None else None + if isinstance(label, QLabel): + label.setText(_t(key) if key else "") __all__ = ["SelfHealingTab"] diff --git a/je_auto_control/gui/test_suite_tab.py b/je_auto_control/gui/test_suite_tab.py index d012b787..d5956085 100644 --- a/je_auto_control/gui/test_suite_tab.py +++ b/je_auto_control/gui/test_suite_tab.py @@ -50,7 +50,7 @@ def __init__(self, parent: Optional[QWidget] = None) -> None: '{"name": "Demo", "cases": [{"name": "c1", "actions": []}]}', ) self._table = QTableWidget(0, len(_COLS)) - self._table.setEditTriggers(QAbstractItemView.NoEditTriggers) + self._table.setEditTriggers(QAbstractItemView.EditTrigger.NoEditTriggers) self._table.horizontalHeader().setStretchLastSection(True) self._summary = QLabel() self._quarantine = QListWidget() @@ -149,7 +149,7 @@ def _render_result(self, result) -> None: case.message) for col, text in enumerate(values): item = QTableWidgetItem(text) - item.setFlags(item.flags() & ~Qt.ItemIsEditable) + item.setFlags(item.flags() & ~Qt.ItemFlag.ItemIsEditable) self._table.setItem(row, col, item) self._summary.setText( _t("suite_summary") diff --git a/je_auto_control/gui/trace_replay_tab.py b/je_auto_control/gui/trace_replay_tab.py index ce6040cb..b4d3fcbe 100644 --- a/je_auto_control/gui/trace_replay_tab.py +++ b/je_auto_control/gui/trace_replay_tab.py @@ -41,10 +41,10 @@ def __init__(self, parent: Optional[QWidget] = None) -> None: self._tr_init() self._controller: Optional[TraceReplayController] = None self._frame_label = QLabel() - self._frame_label.setAlignment(Qt.AlignCenter) + self._frame_label.setAlignment(Qt.AlignmentFlag.AlignCenter) self._frame_label.setMinimumSize(320, 240) self._frame_label.setStyleSheet("background-color: #1e1e1e;") - self._slider = QSlider(Qt.Horizontal) + self._slider = QSlider(Qt.Orientation.Horizontal) self._slider.setMinimum(0) self._slider.setMaximum(0) self._slider.valueChanged.connect(self._on_slider_changed) @@ -63,7 +63,7 @@ def _build_layout(self) -> None: # the tab keeps only the slider, frame, table, and status. root = QVBoxLayout(self) root.addWidget(self._slider) - splitter = QSplitter(Qt.Horizontal) + splitter = QSplitter(Qt.Orientation.Horizontal) splitter.addWidget(self._frame_label) splitter.addWidget(self._actions_table) splitter.setStretchFactor(0, 3) @@ -168,8 +168,8 @@ def _render_frame(self, state: ReplayState) -> None: self._frame_label.clear() return scaled = pixmap.scaled( - self._frame_label.size(), Qt.KeepAspectRatio, - Qt.SmoothTransformation, + self._frame_label.size(), Qt.AspectRatioMode.KeepAspectRatio, + Qt.TransformationMode.SmoothTransformation, ) self._frame_label.setPixmap(scaled) @@ -184,7 +184,7 @@ def _render_actions(self, state: ReplayState) -> None: ) for col, text in enumerate(values): item = QTableWidgetItem(text) - item.setFlags(Qt.ItemIsSelectable | Qt.ItemIsEnabled) + item.setFlags(Qt.ItemFlag.ItemIsSelectable | Qt.ItemFlag.ItemIsEnabled) self._actions_table.setItem(row, col, item) self._actions_table.resizeColumnsToContents() diff --git a/je_auto_control/gui/usb_passthrough_panel.py b/je_auto_control/gui/usb_passthrough_panel.py index 2f3b2801..1a8520a2 100644 --- a/je_auto_control/gui/usb_passthrough_panel.py +++ b/je_auto_control/gui/usb_passthrough_panel.py @@ -18,6 +18,7 @@ from __future__ import annotations import functools +from pathlib import Path from typing import Any, Callable, List, Optional from PySide6.QtCore import QTimer @@ -361,7 +362,7 @@ def _export_acl(self) -> None: if not path: return try: - export_acl_to_file(self._acl, path) + export_acl_to_file(self._acl, Path(path)) except OSError as error: self._viewer_status.setText( _t("usb_share_acl_import_failed").format(error=str(error)), @@ -376,7 +377,7 @@ def _import_acl(self) -> None: if not path: return try: - count = import_acl_from_file(self._acl, path) + count = import_acl_from_file(self._acl, Path(path)) except (OSError, ValueError) as error: self._viewer_status.setText( _t("usb_share_acl_import_failed").format(error=str(error)), diff --git a/je_auto_control/gui/webhooks_tab.py b/je_auto_control/gui/webhooks_tab.py index e8fb6e65..1454973e 100644 --- a/je_auto_control/gui/webhooks_tab.py +++ b/je_auto_control/gui/webhooks_tab.py @@ -43,17 +43,17 @@ def __init__(self, parent: Optional[QWidget] = None) -> None: self._path_input.setPlaceholderText("/jobs") self._script_input = QLineEdit() self._token_input = QLineEdit() - self._token_input.setEchoMode(QLineEdit.Password) + self._token_input.setEchoMode(QLineEdit.EchoMode.Password) self._method_checks = { method: QCheckBox(method) for method in _DEFAULT_METHODS } self._method_checks["POST"].setChecked(True) self._table = QTableWidget(0, 6) - self._table.setEditTriggers(QAbstractItemView.NoEditTriggers) - self._table.setSelectionBehavior(QAbstractItemView.SelectRows) + self._table.setEditTriggers(QAbstractItemView.EditTrigger.NoEditTriggers) + self._table.setSelectionBehavior(QAbstractItemView.SelectionBehavior.SelectRows) self._table.verticalHeader().setVisible(False) self._table.horizontalHeader().setSectionResizeMode( - QHeaderView.Interactive, + QHeaderView.ResizeMode.Interactive, ) self._table.horizontalHeader().setStretchLastSection(True) self._apply_table_headers() @@ -225,5 +225,5 @@ def _refresh(self) -> None: ) for col, text in enumerate(values): item = QTableWidgetItem(text) - item.setFlags(item.flags() & ~Qt.ItemIsEditable) + item.setFlags(item.flags() & ~Qt.ItemFlag.ItemIsEditable) self._table.setItem(row, col, item) diff --git a/je_auto_control/gui/webrunner_tab.py b/je_auto_control/gui/webrunner_tab.py index 02387cbe..656f3630 100644 --- a/je_auto_control/gui/webrunner_tab.py +++ b/je_auto_control/gui/webrunner_tab.py @@ -124,7 +124,7 @@ def refresh_availability(self) -> None: key = "web_available" if available else "web_unavailable" self._available_label.setText(_t(key)) self._available_label.setTextInteractionFlags( - Qt.TextSelectableByMouse, + Qt.TextInteractionFlag.TextSelectableByMouse, ) # --- handlers ------------------------------------------------- diff --git a/je_auto_control/gui/workspace_tabs.py b/je_auto_control/gui/workspace_tabs.py index 7c91e02d..2b29dcee 100644 --- a/je_auto_control/gui/workspace_tabs.py +++ b/je_auto_control/gui/workspace_tabs.py @@ -79,6 +79,7 @@ def __init__(self, scrollable: bool = True, pending_key: str = "", parent: Optio layout = QVBoxLayout(self) layout.setContentsMargins(0, 0, 0, 0) layout.setSpacing(0) + self._box = layout # the holder's own layout, typed (``layout()`` may answer None) if scrollable: self._area = QScrollArea(self) self._area.setObjectName("WorkspacePageScroll") @@ -125,7 +126,7 @@ def _detach(self, page: QWidget) -> None: if self._area is not None: self._area.takeWidget() else: - self.layout().removeWidget(page) + self._box.removeWidget(page) def set_page(self, page: QWidget) -> None: """Put ``page`` in the holder.""" @@ -133,7 +134,7 @@ def set_page(self, page: QWidget) -> None: self._parked = False page.setProperty(_CLOSED_PAGE, None) # shown in a tab again, wherever it waited if self._area is None: - self.layout().addWidget(page) + self._box.addWidget(page) page.show() return self._area.setWidget(page) @@ -308,7 +309,7 @@ def widget(self, index: int) -> Optional[QWidget]: raw = super().widget(index) return raw.page if isinstance(raw, PageHolder) else raw - def currentWidget(self) -> Optional[QWidget]: # noqa: N802 # reason: Qt name + def currentWidget(self) -> Optional[QWidget]: # type: ignore[override] # noqa: N802 # reason: Qt name; None for a deferred tab """The page of the selected tab.""" return self.widget(self.currentIndex()) diff --git a/pyproject.toml b/pyproject.toml index 245f7e5a..28840048 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -444,6 +444,7 @@ module = [ "je_auto_control.gui._dispose", "je_auto_control.gui._slow_op", "je_auto_control.gui._weak_call", + "je_auto_control.gui._qt_typed", "je_auto_control.utils.accessibility.backends.windows_automation", "je_auto_control.utils.codegen.candidate_diff", "je_auto_control.utils.remote_desktop.input_send_queue", diff --git a/test/unit_test/headless/test_gui_qt_typed.py b/test/unit_test/headless/test_gui_qt_typed.py new file mode 100644 index 00000000..58ef7ed5 --- /dev/null +++ b/test/unit_test/headless/test_gui_qt_typed.py @@ -0,0 +1,53 @@ +"""The typed accessors the tabs use in place of a cast at every call site (offscreen Qt).""" +import os + +import pytest + +os.environ.setdefault("QT_QPA_PLATFORM", "offscreen") +pytest.importorskip("PySide6.QtWidgets", exc_type=ImportError) + +from PySide6.QtCore import Qt # noqa: E402 +from PySide6.QtWidgets import QAbstractItemView, QApplication, QLabel, QPushButton, QTableWidget, QTableWidgetItem # noqa: E402 + +from je_auto_control.gui._i18n_helpers import TranslatableMixin # noqa: E402 +from je_auto_control.gui._qt_typed import as_widget, filled_item # noqa: E402 +from je_auto_control.utils.exception.exceptions import AutoControlException # noqa: E402 + + +@pytest.fixture(autouse=True) +def qapp(): + return QApplication.instance() or QApplication([]) + + +def test_as_widget_is_the_object_itself(): + label = QLabel("x") + assert as_widget(label) is label + label.deleteLater() + + +def test_a_filled_cell_is_returned_and_a_missing_one_is_a_typed_error(): + table = QTableWidget(1, 2) + item = QTableWidgetItem("job-1") + table.setItem(0, 0, item) + assert filled_item(table, 0, 0) is item and filled_item(table, 0, 0).text() == "job-1" + with pytest.raises(AutoControlException, match="row 0, column 1"): + filled_item(table, 0, 1) + table.deleteLater() + + +def test_tr_hands_back_the_widget_it_was_given(): + """``_tr`` is generic over the widget class; at run time it has always returned its argument.""" + + class _Tab(TranslatableMixin): + """A translatable holder.""" + + tab = _Tab() + button = QPushButton() + assert tab._tr(button, "audit_run") is button and button.text() != "" + button.deleteLater() + + +def test_the_scoped_enum_spellings_are_the_members_the_short_ones_named(): + assert QAbstractItemView.EditTrigger.NoEditTriggers is QAbstractItemView.NoEditTriggers + assert Qt.ItemFlag.ItemIsEditable is Qt.ItemIsEditable + assert Qt.ItemDataRole.UserRole is Qt.UserRole diff --git a/test/verify/typing_extras_exempt.txt b/test/verify/typing_extras_exempt.txt index 487fc8f8..7bb148c1 100644 --- a/test/verify/typing_extras_exempt.txt +++ b/test/verify/typing_extras_exempt.txt @@ -12,74 +12,22 @@ # Measure with the versions the `typing-extras` job of `quality.yml` pins. # # Targets: win32, linux, darwin -# Measured entries: 70 +# Measured entries: 18 -je_auto_control.gui._auto_click_tab -je_auto_control.gui._image_detect_tab -je_auto_control.gui._record_tab -je_auto_control.gui._screenshot_tab -je_auto_control.gui._script_tab -je_auto_control.gui._worker_thread -je_auto_control.gui.a11y_audit_tab -je_auto_control.gui.accessibility_tab -je_auto_control.gui.admin_console_tab -je_auto_control.gui.audit_log_tab -je_auto_control.gui.config_sync_tab -je_auto_control.gui.dag_tab -je_auto_control.gui.data_source_tab -je_auto_control.gui.device_matrix_tab -je_auto_control.gui.email_triggers_tab -je_auto_control.gui.flakiness_tab -je_auto_control.gui.flow_editor.scene -je_auto_control.gui.flow_editor.tab -je_auto_control.gui.hotkeys_tab -je_auto_control.gui.inspector_tab -je_auto_control.gui.main_widget -je_auto_control.gui.main_window -je_auto_control.gui.mobile_tab je_auto_control.gui.navigation -je_auto_control.gui.presence_tab -je_auto_control.gui.profiler_tab -je_auto_control.gui.rbac_users_panel -je_auto_control.gui.remote_desktop._helpers -je_auto_control.gui.remote_desktop.advanced_group -je_auto_control.gui.remote_desktop.connection_screen -je_auto_control.gui.remote_desktop.frame_display -je_auto_control.gui.remote_desktop.host_panel -je_auto_control.gui.remote_desktop.tab -je_auto_control.gui.remote_desktop.trusted_group -je_auto_control.gui.remote_desktop.viewer_panel -je_auto_control.gui.remote_desktop.webrtc_dialogs je_auto_control.gui.remote_desktop.webrtc_host_connection je_auto_control.gui.remote_desktop.webrtc_host_media je_auto_control.gui.remote_desktop.webrtc_host_sessions je_auto_control.gui.remote_desktop.webrtc_host_trust je_auto_control.gui.remote_desktop.webrtc_host_ui -je_auto_control.gui.remote_desktop.webrtc_panel je_auto_control.gui.remote_desktop.webrtc_panel_common je_auto_control.gui.remote_desktop.webrtc_viewer_address_book je_auto_control.gui.remote_desktop.webrtc_viewer_connection je_auto_control.gui.remote_desktop.webrtc_viewer_files je_auto_control.gui.remote_desktop.webrtc_viewer_media je_auto_control.gui.remote_desktop.webrtc_viewer_ui -je_auto_control.gui.rest_api_tab -je_auto_control.gui.run_history_tab -je_auto_control.gui.run_history_timeline -je_auto_control.gui.scheduler_tab -je_auto_control.gui.script_builder.builder_tab je_auto_control.gui.script_builder.step_form_view -je_auto_control.gui.script_builder.step_list_view -je_auto_control.gui.secrets_tab -je_auto_control.gui.self_healing_tab -je_auto_control.gui.test_suite_tab -je_auto_control.gui.trace_replay_tab je_auto_control.gui.triggers_tab -je_auto_control.gui.usb_browser_tab -je_auto_control.gui.usb_passthrough_panel -je_auto_control.gui.usb_passthrough_prompt -je_auto_control.gui.webhooks_tab -je_auto_control.gui.webrunner_tab -je_auto_control.gui.workspace_tabs je_auto_control.utils.remote_desktop.hw_codec je_auto_control.utils.remote_desktop.session_recorder je_auto_control.utils.remote_desktop.webrtc_audio From 91d27f080bdbc87bc4dac2b92745bf7a8ca385bb Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 07:46:51 +0800 Subject: [PATCH 13/17] Let a stop wake the polling waits, and give MCP and ChatOps a way to say so AC_wait_window, AC_wait_text, the smart waits, AC_expect_poll and the other polling waits slept on their own time.sleep, so a stopped run sat out the wait's timeout. They now sleep on run_control.pause, which is a plain sleep outside a stoppable run. A stopped ChatOps command answers "stopped" rather than "failed", and MCP gets ac_stop_execution and ac_list_executions. A run's stop token is bound to the thread that entered the run (utils/thread_bound.py), so a thread started inside a run does not answer to its stop on a free-threaded build either. --- .../Eng/doc/mcp_server/mcp_server_doc.rst | 4 +- .../Eng/doc/new_features/new_features_doc.rst | 42 ++- .../Zh/doc/mcp_server/mcp_server_doc.rst | 4 +- .../Zh/doc/new_features/new_features_doc.rst | 34 +- .../utils/actionability/actionability.py | 3 +- je_auto_control/utils/app_idle/app_idle.py | 3 +- .../utils/assertion/combinators.py | 3 +- je_auto_control/utils/chatops/router.py | 13 + je_auto_control/utils/executor/run_control.py | 29 +- .../utils/expect_poll/expect_poll.py | 3 +- .../utils/file_dialog/file_dialog.py | 4 +- je_auto_control/utils/ime_state/ime_state.py | 3 +- .../utils/input_macro/input_macro.py | 11 +- je_auto_control/utils/key_hold/key_hold.py | 16 +- .../utils/lock_session/lock_session.py | 5 +- .../utils/mcp_server/tools/_factories.py | 29 ++ .../utils/mcp_server/tools/_handlers_runs.py | 12 + je_auto_control/utils/ocr/ocr_engine.py | 3 +- .../utils/rate_limit/rate_limit.py | 3 +- .../utils/retry_budget/retry_budget.py | 4 +- .../utils/self_healing/verification.py | 3 +- je_auto_control/utils/smart_waits/waits.py | 4 +- je_auto_control/utils/state_machine/engine.py | 3 +- je_auto_control/utils/thread_bound.py | 73 ++++ .../wrapper/auto_control_window.py | 3 +- je_auto_control/wrapper/mobile_extensions.py | 5 +- pyproject.toml | 1 + .../headless/test_chatops_stopped_run.py | 54 +++ .../headless/test_mcp_stop_execution.py | 104 ++++++ .../headless/test_stop_aware_waits.py | 328 ++++++++++++++++++ 30 files changed, 770 insertions(+), 36 deletions(-) create mode 100644 je_auto_control/utils/thread_bound.py create mode 100644 test/unit_test/headless/test_chatops_stopped_run.py create mode 100644 test/unit_test/headless/test_mcp_stop_execution.py create mode 100644 test/unit_test/headless/test_stop_aware_waits.py diff --git a/docs/source/Eng/doc/mcp_server/mcp_server_doc.rst b/docs/source/Eng/doc/mcp_server/mcp_server_doc.rst index 1ca08e63..a570cb62 100644 --- a/docs/source/Eng/doc/mcp_server/mcp_server_doc.rst +++ b/docs/source/Eng/doc/mcp_server/mcp_server_doc.rst @@ -64,7 +64,9 @@ Recording / replay Action executor / history ``ac_execute_actions``, ``ac_execute_action_file``, - ``ac_list_action_commands``, ``ac_list_run_history``. + ``ac_list_action_commands``, ``ac_list_run_history``, + ``ac_stop_execution``, ``ac_list_executions`` (stop / list the stoppable + runs; a list becomes stoppable by wrapping it in ``AC_run_stoppable``). Scheduler / triggers / hotkeys ``ac_scheduler_add_job``, ``ac_scheduler_remove_job``, diff --git a/docs/source/Eng/doc/new_features/new_features_doc.rst b/docs/source/Eng/doc/new_features/new_features_doc.rst index d23620ac..3d3bfec8 100644 --- a/docs/source/Eng/doc/new_features/new_features_doc.rst +++ b/docs/source/Eng/doc/new_features/new_features_doc.rst @@ -411,6 +411,16 @@ counts only on the thread that made it. To run something on another thread on behalf of the run, hand it the values (or open an ``execution_scope`` there). Coroutines and tasks on the binding thread share the run. +The same rule holds for the other state that belongs to the calling thread: +the RBAC caller a request is served as (``authorization_scope`` / +``current_authorization``), the device bound by ``use_device``, the labels of +``heal_context`` and the stop token of a stoppable run. A thread started +inside one of them starts without it on every build, so a scheduler or +observer thread first started while an admin's request was being served does +not go on running as that admin on free-threaded Python. Code that hands work +to another thread on purpose passes the value along and binds it there, as +``AC_parallel`` does for the caller and the stop token. + Four more callers have a scope chosen for them rather than the process scope: * ``je_auto_control run --dry-run --var name=value`` seeds the variables into @@ -545,6 +555,14 @@ and stop it from any other entry point of the same process:: ``AC_stop_execution`` without ``run_id`` (and ``stop_execution()``) stops every stoppable run except the caller's own. +Over MCP the same two are the tools ``ac_stop_execution`` (``run_id``, +``reason``; needs ``drive_input`` and is absent from a read-only server) and +``ac_list_executions`` (read-only). There is no separate "run stoppably" tool: +``ac_execute_actions`` and ``ac_execute_action_file`` take any action list, so +the list itself carries the ``AC_run_stoppable`` block shown above. A ChatOps +command whose run is stopped replies `` stopped.`` (with the reason +in parentheses) instead of reporting a failure. + What a stop does: * **Checked** before every action, at every pass of ``AC_loop`` / @@ -552,6 +570,21 @@ What a stop does: (also with an empty body), and inside the waits of ``AC_sleep``, ``AC_wait_image``, ``AC_wait_pixel`` and the ``AC_retry`` back-off, which wake at once. ``AC_parallel`` branches stop with their run. +* **Wakes the polling waits** as well -- they sleep between two probes on + ``run_control.pause``, which is a plain ``time.sleep`` outside a stoppable + run: ``AC_wait_window``, ``AC_wait_text``, ``AC_expect_poll``, + ``AC_assert_eventually``, the smart waits (``AC_wait_screen_stable``, + ``AC_wait_pixel_changes``, ``AC_wait_region_idle``, + ``AC_wait_clipboard_change``, ``AC_wait_image_gone``, ``AC_wait_text_gone``, + ``AC_wait_color``, ``AC_wait_window_title``, ``AC_wait_window_closed``, + ``AC_wait_for_file``, ``AC_wait_for_port``, ``AC_wait_for_process``), + ``AC_wait_actionable``, ``AC_wait_until_app_idle``, ``AC_wait_for_unlock``, + ``AC_wait_for_composition_commit``, ``AC_android_wait_for_app`` / + ``AC_ios_wait_for_app``, the dialog wait of ``AC_handle_file_dialog``, the + ``verify`` poll of ``AC_self_heal_click``, an ``after`` timer of + ``AC_run_state_machine``, and the timed holds of ``AC_hold_key``, + ``AC_input_sequence`` (``wait`` steps) and ``AC_replay_timeline`` (which + release what they hold on the way out). * **Raises** ``ExecutionStopped`` (an ``AutoControlException``) on the run's own thread. It is never recorded-and-continued, whatever ``raise_on_error`` says, and ``AC_try`` / ``AC_retry`` do not catch it: ``catch`` is skipped. @@ -566,9 +599,12 @@ What a stop does: next checkpoint. What it does not do: interrupt a command that is already inside its backend -(one image search, one OCR read, one HTTP request, a wrapper's own -``time.sleep``). The run ends when that command returns. Two active runs -cannot share a ``run_id``. +(one image search, one OCR read, one HTTP request, one shell command, a +wrapper's own ``time.sleep``). The run ends when that command returns. Also +not woken: ``AC_wait_for_focus_change`` (a single native UI Automation wait), +the sub-second settle waits after focusing a window, and the pacing sleeps +inside one gesture (``AC_click_mouse`` with ``interval``, humanized moves and +typing, tween drags). Two active runs cannot share a ``run_id``. GUI: the Script, Script Builder, LLM Planner, Record (playback), Test Suites and ChatOps tabs execute off the GUI thread, one run at a time per tab, and diff --git a/docs/source/Zh/doc/mcp_server/mcp_server_doc.rst b/docs/source/Zh/doc/mcp_server/mcp_server_doc.rst index 57e5c412..060b2830 100644 --- a/docs/source/Zh/doc/mcp_server/mcp_server_doc.rst +++ b/docs/source/Zh/doc/mcp_server/mcp_server_doc.rst @@ -61,7 +61,9 @@ list-changed 通知與 elicitation。 動作執行器 / 歷程 ``ac_execute_actions``、``ac_execute_action_file``、 - ``ac_list_action_commands``、``ac_list_run_history``。 + ``ac_list_action_commands``、``ac_list_run_history``、 + ``ac_stop_execution``、``ac_list_executions``（停止／列出可停止的執行；把清單包進 + ``AC_run_stoppable`` 就是可停止的）。 排程 / 觸發 / 熱鍵 ``ac_scheduler_add_job``、``ac_scheduler_remove_job``、 diff --git a/docs/source/Zh/doc/new_features/new_features_doc.rst b/docs/source/Zh/doc/new_features/new_features_doc.rst index 70eb1f50..4a832c34 100644 --- a/docs/source/Zh/doc/new_features/new_features_doc.rst +++ b/docs/source/Zh/doc/new_features/new_features_doc.rst @@ -382,6 +382,14 @@ free-threaded Python 與 ``-X thread_inherit_context=1``\ （新執行緒一開 執行緒上替這次執行做事，請把值傳過去（或在那裡自己開一個 ``execution_scope``\ ）。在綁定執行緒上的 coroutine 與 task 則共用這次執行。 +其他屬於呼叫執行緒的狀態也是同一條規則：請求所代表的 RBAC 呼叫者 +（``authorization_scope`` / ``current_authorization``）、``use_device`` 綁定的裝置、 +``heal_context`` 的標記，以及可停止執行的停止權杖。在它們裡面啟動的執行緒，在每一種 +直譯器上一開始都不帶這些狀態，所以在 free-threaded Python 上，一條在管理員的請求 +進行中才第一次啟動的排程器或 observer 執行緒，不會一直以那位管理員的身分執行。 +刻意把工作交給另一條執行緒的程式會把值帶過去、在那裡重新綁定，``AC_parallel`` 對 +呼叫者與停止權杖就是這樣做的。 + 另外四種呼叫者的範圍是刻意選定的，不再是行程層級的範圍： * ``je_auto_control run --dry-run --var name=value`` 把變數放進該次 dry run @@ -506,12 +514,31 @@ GUI：**Variables** 分頁 — 即時檢視 ``executor.variables``，可單筆 不帶 ``run_id`` 的 ``AC_stop_execution``（以及 ``stop_execution()``）會停止呼叫者 自己以外的所有可停止執行。 +在 MCP 上，同樣的兩件事是工具 ``ac_stop_execution``\ （參數 ``run_id`` 與 ``reason``\ ，需要 +``drive_input``\ ，唯讀的 server 不提供）與 ``ac_list_executions``\ （唯讀）。沒有另外的 +「以可停止方式執行」工具： ``ac_execute_actions`` 與 ``ac_execute_action_file`` 接受 +任何 action list，所以由清單自己帶上面的 ``AC_run_stoppable`` 區塊。ChatOps 指令的 +執行被停止時，回覆是 `` stopped.``\ （原因放在括號裡），而不是回報失敗。 + 停止時會發生的事： * **檢查點**：每個 action 之前、``AC_loop`` / ``AC_while_image`` / ``AC_while_var`` / ``AC_for_each`` / ``AC_for_each_row`` 的每一輪（body 是空的 也算），以及 ``AC_sleep``、``AC_wait_image``、``AC_wait_pixel`` 與 ``AC_retry`` 退避等待的內部——這些等待會立刻醒來。``AC_parallel`` 的分支跟著所屬的執行一起停。 +* **輪詢式的等待也會被叫醒**——它們在兩次探測之間睡在 ``run_control.pause`` 上， + 在可停止的執行之外那就是單純的 ``time.sleep``：``AC_wait_window``、 + ``AC_wait_text``、``AC_expect_poll``、``AC_assert_eventually``、smart wait + （``AC_wait_screen_stable``、``AC_wait_pixel_changes``、``AC_wait_region_idle``、 + ``AC_wait_clipboard_change``、``AC_wait_image_gone``、``AC_wait_text_gone``、 + ``AC_wait_color``、``AC_wait_window_title``、``AC_wait_window_closed``、 + ``AC_wait_for_file``、``AC_wait_for_port``、``AC_wait_for_process``）、 + ``AC_wait_actionable``、``AC_wait_until_app_idle``、``AC_wait_for_unlock``、 + ``AC_wait_for_composition_commit``、``AC_android_wait_for_app`` / + ``AC_ios_wait_for_app``、``AC_handle_file_dialog`` 等對話框出現的等待、 + ``AC_self_heal_click`` 的 ``verify`` 輪詢、``AC_run_state_machine`` 的 ``after`` + 計時，以及 ``AC_hold_key``、``AC_input_sequence``（``wait`` 步驟）與 + ``AC_replay_timeline`` 的計時按住（離開時會放開它們按住的東西）。 * **拋出** ``ExecutionStopped``（``AutoControlException`` 的子類別），在執行自己的 執行緒上。不論 ``raise_on_error`` 為何，它都不會被「記錄後繼續」，``AC_try`` / ``AC_retry`` 也不會接住它：``catch`` 會被跳過。測試套件會整個停止，而不是把 @@ -524,8 +551,11 @@ GUI：**Variables** 分頁 — 即時檢視 ``executor.variables``，可單筆 * **具黏著性**：吞掉這個例外的指令只是把它延後到下一個檢查點。 不會做的事：中斷已經進到後端的指令（一次影像搜尋、一次 OCR、一次 HTTP 請求、 -wrapper 自己的 ``time.sleep``）。該指令返回後執行才結束。兩個進行中的執行不能 -使用同一個 ``run_id``。 +一次 shell 指令、wrapper 自己的 ``time.sleep``）。該指令返回後執行才結束。同樣不會 +被叫醒的還有：``AC_wait_for_focus_change``（單一次原生 UI Automation 等待）、聚焦 +視窗之後不到一秒的穩定等待，以及單一手勢內部的節奏停頓（帶 ``interval`` 的 +``AC_click_mouse``、擬人化的移動與打字、補間拖曳）。兩個進行中的執行不能使用同一個 +``run_id``。 GUI：Script、Script Builder、LLM Planner、Record（回放）、Test Suites 與 ChatOps 分頁都改在 GUI 執行緒之外執行，每個分頁一次只跑一個，並各有 **停止執行** diff --git a/je_auto_control/utils/actionability/actionability.py b/je_auto_control/utils/actionability/actionability.py index 5d33779e..fa2dd152 100644 --- a/je_auto_control/utils/actionability/actionability.py +++ b/je_auto_control/utils/actionability/actionability.py @@ -16,6 +16,7 @@ from typing import Any, Callable, Dict, List, Optional, Tuple from je_auto_control.utils.exception.exceptions import AutoControlActionException +from je_auto_control.utils.executor.run_control import pause Bbox = Tuple[int, int, int, int] BboxProvider = Callable[[], Optional[Bbox]] @@ -29,7 +30,7 @@ class GateConfig: stable_for_s: float = 0.3 poll_interval_s: float = 0.1 clock: Callable[[], float] = time.monotonic - sleep: Callable[[float], None] = time.sleep + sleep: Callable[[float], None] = pause @dataclass(frozen=True) diff --git a/je_auto_control/utils/app_idle/app_idle.py b/je_auto_control/utils/app_idle/app_idle.py index 384d2793..5159990c 100644 --- a/je_auto_control/utils/app_idle/app_idle.py +++ b/je_auto_control/utils/app_idle/app_idle.py @@ -21,6 +21,7 @@ import time from typing import Any, Callable, Dict, Optional, Sequence +from je_auto_control.utils.executor.run_control import pause from je_auto_control.utils.settle_detector import SettleTracker, settle_point from je_auto_control.utils.timeouts import clamp_poll_interval, deadline_after @@ -47,7 +48,7 @@ def wait_until_app_idle(*, busy_probe: Optional[BusyProbe] = None, quiet_samples: int = 3, timeout_s: float = 10.0, interval_s: float = 0.1, clock: Callable[[], float] = time.monotonic, - sleep: Callable[[float], None] = time.sleep + sleep: Callable[[float], None] = pause ) -> Dict[str, Any]: """Block until the app reads idle for ``quiet_samples`` polls, or timeout. diff --git a/je_auto_control/utils/assertion/combinators.py b/je_auto_control/utils/assertion/combinators.py index 20c3d669..400cb4ff 100644 --- a/je_auto_control/utils/assertion/combinators.py +++ b/je_auto_control/utils/assertion/combinators.py @@ -42,6 +42,7 @@ from je_auto_control.utils.exception.exceptions import ( AutoControlAssertionException, ) +from je_auto_control.utils.executor.run_control import pause from je_auto_control.utils.logging.logging_instance import autocontrol_logger # Map each spec ``kind`` to the headless assertion it dispatches to. @@ -223,7 +224,7 @@ def assert_eventually(spec: Mapping[str, Any], return last if time.monotonic() >= deadline: break - time.sleep(min(poll, max(deadline - time.monotonic(), 0.0))) + pause(min(poll, max(deadline - time.monotonic(), 0.0)), time.sleep) message = ( f"assert_eventually failed after {timeout}s / {attempts} attempts: " f"{last.message}" diff --git a/je_auto_control/utils/chatops/router.py b/je_auto_control/utils/chatops/router.py index f221a2a4..0deb4926 100644 --- a/je_auto_control/utils/chatops/router.py +++ b/je_auto_control/utils/chatops/router.py @@ -15,6 +15,7 @@ from typing import Any, Callable, Dict, List, Optional, Tuple, Type from je_auto_control.utils.exception.exceptions import AutoControlException +from je_auto_control.utils.executor.run_control import ExecutionStopped from je_auto_control.utils.sqlite_support import SQLITE_ERRORS @@ -152,6 +153,9 @@ def _dispatch_argv(self, argv: List[str], return spec.handler(rest, context) except ChatOpsError as error: return CommandResult(text=f"{name}: {error}", succeeded=False) + except ExecutionStopped as error: + # A stop is somebody's decision, not a failure of the script. + return _stopped_reply(name, error) except _HANDLER_ERRORS as error: return CommandResult( text=f"{name} failed: {type(error).__name__}: {error}", @@ -182,6 +186,15 @@ def _cmd_help(self, _argv: List[str], # --- internals ---------------------------------------------------- +def _stopped_reply(name: str, error: ExecutionStopped) -> CommandResult: + """The reply for a command whose run was stopped: `` stopped.`` plus the reason.""" + reason = f" ({error.reason})" if error.reason else "" + return CommandResult( + text=f"{name} stopped.{reason}", succeeded=False, + metadata={"stopped": True, "run_id": error.run_id, "reason": error.reason}, + ) + + def _validate_name(name: str) -> str: if not isinstance(name, str): raise ChatOpsError("command name must be a string") diff --git a/je_auto_control/utils/executor/run_control.py b/je_auto_control/utils/executor/run_control.py index eb1d1e03..140cb530 100644 --- a/je_auto_control/utils/executor/run_control.py +++ b/je_auto_control/utils/executor/run_control.py @@ -8,7 +8,9 @@ * The executor checks it **before every action**, at every loop pass, and the waits of the block commands (``AC_sleep``, ``AC_wait_image``, ``AC_wait_pixel``, the ``AC_retry`` back-off) sleep on it, so they wake the - moment a stop is requested. + moment a stop is requested. So do the polling waits outside the executor + (``AC_wait_window``, ``AC_wait_text``, the smart waits, ``AC_expect_poll`` + and the rest): between two probes they sleep on :func:`pause`. * A requested stop raises :class:`ExecutionStopped` on the run's own thread. It is never recorded-and-continued and ``AC_try`` / ``AC_retry`` do not catch it, but it is an ordinary exception: every ``finally`` on the way out @@ -27,16 +29,21 @@ Without a token nothing here does anything: ``executor.execute_action(...)`` called from plain Python behaves exactly as before. + +A run is the thread that entered it. A thread started inside a run is not +part of it -- on every build, including a free-threaded one, where a new +thread inherits its creator's context variables -- unless it is handed the +token and binds it with :func:`bound_stop_token`, as ``AC_parallel`` does. """ import threading import time import uuid from contextlib import contextmanager -from contextvars import ContextVar from typing import Any, Callable, Dict, Iterator, List, Optional, Tuple from je_auto_control.utils.exception.exceptions import AutoControlException from je_auto_control.utils.logging.logging_instance import autocontrol_logger +from je_auto_control.utils.thread_bound import ThreadBoundVar class ExecutionStopped(AutoControlException): @@ -119,8 +126,12 @@ def take_held(self) -> List[Tuple[str, Any]]: return held -_CURRENT: ContextVar[Optional[StopToken]] = ContextVar("je_auto_control_stop_token", default=None) -_SHIELDED: ContextVar[int] = ContextVar("je_auto_control_stop_shield", default=0) +# Thread-bound: where a new thread inherits its creator's context variables (a +# free-threaded build), a scheduler or observer thread first started inside a +# run would otherwise answer to that run's token for ever -- and a stop is +# sticky. A thread joins a run only through bound_stop_token(). +_CURRENT: ThreadBoundVar[Optional[StopToken]] = ThreadBoundVar("je_auto_control_stop_token", None) +_SHIELDED: ThreadBoundVar[int] = ThreadBoundVar("je_auto_control_stop_shield", 0) _REGISTRY_LOCK = threading.Lock() _ACTIVE: Dict[str, StopToken] = {} @@ -144,14 +155,18 @@ def checkpoint() -> None: raise ExecutionStopped(token.run_id, token.reason) -def pause(seconds: float) -> None: +def pause(seconds: float, sleep: Optional[Callable[[float], Any]] = None) -> None: """Sleep ``seconds``; inside a stoppable run, wake and raise on a stop. - Outside one this is exactly ``time.sleep(seconds)``. + Outside one this is exactly ``time.sleep(seconds)`` -- or ``sleep(seconds)`` + when the caller hands in the sleep it used before it became stop-aware, + which keeps a module's own ``time.sleep`` the seam its tests replace. + This is what every polling wait reachable from an ``AC_*`` command sleeps + on between two probes, so a stop ends the wait instead of its timeout. """ token = _CURRENT.get() if token is None or (_SHIELDED.get() and not token.forced): - time.sleep(seconds) + (sleep if sleep is not None else time.sleep)(seconds) return checkpoint() if token.wait(seconds): diff --git a/je_auto_control/utils/expect_poll/expect_poll.py b/je_auto_control/utils/expect_poll/expect_poll.py index 3bc53ab9..7c2c576b 100644 --- a/je_auto_control/utils/expect_poll/expect_poll.py +++ b/je_auto_control/utils/expect_poll/expect_poll.py @@ -17,6 +17,7 @@ from je_auto_control.utils.exception.exceptions import ( AutoControlActionException, AutoControlAssertionException, ) +from je_auto_control.utils.executor.run_control import pause from je_auto_control.utils.timeouts import deadline_after Matcher = Callable[[Any], bool] @@ -82,7 +83,7 @@ def expect_poll(getter: Callable[[], Any], matcher: Matcher, *, timeout_s: float = 5.0, interval_s: float = 0.25, describe: Callable[[Any], str] = repr, clock: Callable[[], float] = time.monotonic, - sleep: Callable[[float], None] = time.sleep) -> PollResult: + sleep: Callable[[float], None] = pause) -> PollResult: """Poll ``getter`` until ``matcher`` passes or ``timeout_s`` elapses. Returns a :class:`PollResult` with the final value, attempt count and elapsed diff --git a/je_auto_control/utils/file_dialog/file_dialog.py b/je_auto_control/utils/file_dialog/file_dialog.py index b5e3bf05..a4a42c74 100644 --- a/je_auto_control/utils/file_dialog/file_dialog.py +++ b/je_auto_control/utils/file_dialog/file_dialog.py @@ -12,6 +12,8 @@ import time from typing import Dict, Optional +from je_auto_control.utils.executor.run_control import pause + _DEFAULT_TITLES = {"open": "Open", "save": "Save As", "folder": "Select Folder"} @@ -37,7 +39,7 @@ def wait_window(self, title: str, timeout_s: float) -> bool: return backend.bring_to_front(window_id) if time.monotonic() >= deadline: return False - time.sleep(0.2) + pause(0.2, time.sleep) def type_path(self, path: str) -> None: from je_auto_control.wrapper.auto_control_keyboard import write diff --git a/je_auto_control/utils/ime_state/ime_state.py b/je_auto_control/utils/ime_state/ime_state.py index e23ca645..4650d580 100644 --- a/je_auto_control/utils/ime_state/ime_state.py +++ b/je_auto_control/utils/ime_state/ime_state.py @@ -24,6 +24,7 @@ import time from typing import Any, Callable, Dict, Optional +from je_auto_control.utils.executor.run_control import pause from je_auto_control.utils.timeouts import clamp_poll_interval, deadline_after # IMM32 conversion-mode (IME_CMODE_*) bit flags. @@ -81,7 +82,7 @@ def is_composing(*, reader: Optional[ImeReader] = None) -> bool: def wait_for_composition_commit( *, reader: Optional[ImeReader] = None, timeout_s: float = 5.0, interval_s: float = 0.1, clock: Callable[[], float] = time.monotonic, - sleep: Callable[[float], None] = time.sleep) -> bool: + sleep: Callable[[float], None] = pause) -> bool: """Block until the IME is no longer composing; ``True``, or ``False`` on timeout. ``clock`` / ``sleep`` / ``reader`` are injectable for deterministic tests. diff --git a/je_auto_control/utils/input_macro/input_macro.py b/je_auto_control/utils/input_macro/input_macro.py index 5d52847d..cbd8c829 100644 --- a/je_auto_control/utils/input_macro/input_macro.py +++ b/je_auto_control/utils/input_macro/input_macro.py @@ -18,6 +18,8 @@ import time from typing import Any, Callable, Dict, List, Optional, Tuple +from je_auto_control.utils.executor.run_control import pause + def _sink_move(event: Dict[str, Any]) -> None: from je_auto_control.wrapper.auto_control_mouse import set_mouse_position @@ -216,7 +218,7 @@ def replay_timeline(events: List[Dict[str, Any]], *, speed: float = 1.0, propagates. """ dispatch = _HeldInputs(sink or _default_sink) - sleeper = sleep or time.sleep + sleeper = sleep or _stoppable_sleep factor = _playback_factor(speed) played = 0 try: @@ -235,6 +237,11 @@ def replay_timeline(events: List[Dict[str, Any]], *, speed: float = 1.0, return played +def _stoppable_sleep(seconds: float) -> None: + """``time.sleep`` that a stop of the surrounding stoppable run wakes.""" + pause(seconds, time.sleep) + + def _run_steps(steps: List[Dict[str, Any]], dispatch: Callable, sleeper: Callable, log: List[Dict[str, Any]]) -> None: for step in steps: @@ -261,7 +268,7 @@ def run_sequence(steps: List[Dict[str, Any]], *, raises, and keys still held when a step fails are released first. """ dispatch = _HeldInputs(sink or _default_sink) - sleeper = sleep or time.sleep + sleeper = sleep or _stoppable_sleep log: List[Dict[str, Any]] = [] try: _run_steps(steps, dispatch, sleeper, log) diff --git a/je_auto_control/utils/key_hold/key_hold.py b/je_auto_control/utils/key_hold/key_hold.py index 5134524c..4fd58f9e 100644 --- a/je_auto_control/utils/key_hold/key_hold.py +++ b/je_auto_control/utils/key_hold/key_hold.py @@ -12,6 +12,7 @@ import time from typing import Any, Callable, Dict, List, Optional +from je_auto_control.utils.executor import run_control from je_auto_control.utils.logging.logging_instance import autocontrol_logger Sink = Callable[[Dict[str, Any]], None] @@ -80,19 +81,26 @@ def _release_held(dispatch: Sink, held: List[str]) -> None: ) +def _stoppable_sleep(seconds: float) -> None: + """``time.sleep`` that a stop of the surrounding stoppable run wakes.""" + run_control.pause(seconds, time.sleep) + + def hold_key(key: str, duration_s: float, *, rate_hz: Optional[float] = None, sink: Optional[Sink] = None, sleep: Optional[Callable[[float], None]] = None) -> Dict[str, Any]: """Hold or auto-repeat ``key`` for ``duration_s``; return the dispatched plan. - ``wait`` ops go to ``sleep`` (default :func:`time.sleep`); key ops go to - ``sink`` (default: the real keyboard backend). A KeyboardInterrupt during - the hold wait, or a failing release, still releases the key via the finally + ``wait`` ops go to ``sleep`` (default: ``run_control.pause``, a plain + ``time.sleep`` that a stop of the surrounding stoppable run wakes); key ops + go to ``sink`` (default: the real keyboard backend). A stop or a + KeyboardInterrupt during the hold wait, or a failing release, still + releases the key via the finally block (auto-repeat plans only ``type`` keys, so nothing is held there). """ plan = plan_key_hold(key, duration_s, rate_hz=rate_hz) dispatch = sink or _default_sink - pause = sleep or time.sleep + pause = sleep or _stoppable_sleep held: List[str] = [] try: for event in plan: diff --git a/je_auto_control/utils/lock_session/lock_session.py b/je_auto_control/utils/lock_session/lock_session.py index 11c4b36c..a40eb332 100644 --- a/je_auto_control/utils/lock_session/lock_session.py +++ b/je_auto_control/utils/lock_session/lock_session.py @@ -22,6 +22,7 @@ import time from typing import Any, Callable, Dict, List, Optional, Sequence +from je_auto_control.utils.executor.run_control import pause from je_auto_control.utils.session_guard import is_session_locked from je_auto_control.utils.session_guard.session_guard import LockProbe from je_auto_control.utils.timeouts import clamp_poll_interval, deadline_after @@ -120,7 +121,7 @@ def _wait_lock_state(target_locked: bool, *, probe: Optional[LockProbe], def wait_for_unlock(*, probe: Optional[LockProbe] = None, timeout_s: float = 30.0, interval_s: float = 0.5, clock: Callable[[], float] = time.monotonic, - sleep: Callable[[float], None] = time.sleep) -> bool: + sleep: Callable[[float], None] = pause) -> bool: """Block until the session is unlocked; return ``True``, or ``False`` on timeout. Reuses :func:`session_guard.is_session_locked` (Windows default probe). @@ -133,7 +134,7 @@ def wait_for_unlock(*, probe: Optional[LockProbe] = None, def wait_for_lock(*, probe: Optional[LockProbe] = None, timeout_s: float = 30.0, interval_s: float = 0.5, clock: Callable[[], float] = time.monotonic, - sleep: Callable[[float], None] = time.sleep) -> bool: + sleep: Callable[[float], None] = pause) -> bool: """Block until the session is locked; return ``True``, or ``False`` on timeout.""" return _wait_lock_state(True, probe=probe, timeout_s=timeout_s, interval_s=interval_s, clock=clock, sleep=sleep) diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py index c44fb62b..9bda6b65 100644 --- a/je_auto_control/utils/mcp_server/tools/_factories.py +++ b/je_auto_control/utils/mcp_server/tools/_factories.py @@ -552,6 +552,35 @@ def system_tools() -> List[MCPTool]: handler=h_runs.execute_action_file, annotations=DESTRUCTIVE, ), + MCPTool( + name="ac_stop_execution", + description=("Ask a stoppable run to stop; returns {stopped: N} " + "at once and the run ends at its next checkpoint " + "(between actions, or inside a wait). A run is " + "stoppable when its action list is wrapped in an " + "AC_run_stoppable block (give it a run_id) or it " + "was started with a run_id from Python or the GUI. " + "Without run_id every stoppable run is asked. A " + "second request for the same run also interrupts " + "its cleanup."), + input_schema=schema({ + "run_id": {"type": "string", + "description": "The run to stop (see ac_list_executions)."}, + "reason": {"type": "string", + "description": "Stored on the run and shown in its stop error."}, + }), + handler=h_runs.stop_execution, + annotations=SIDE_EFFECT_ONLY, + ), + MCPTool( + name="ac_list_executions", + description=("Return the stoppable runs in progress as " + "[{run_id, started_at, stopping}]; run_id is what " + "ac_stop_execution takes."), + input_schema=schema({}), + handler=h_runs.list_executions, + annotations=READ_ONLY, + ), MCPTool( name="ac_list_action_commands", description="Return every action command name the executor recognises.", diff --git a/je_auto_control/utils/mcp_server/tools/_handlers_runs.py b/je_auto_control/utils/mcp_server/tools/_handlers_runs.py index 907438e2..0762d26d 100644 --- a/je_auto_control/utils/mcp_server/tools/_handlers_runs.py +++ b/je_auto_control/utils/mcp_server/tools/_handlers_runs.py @@ -24,6 +24,18 @@ def execute_action_file(file_path: str) -> Dict[str, str]: return {key: str(value) for key, value in result.items()} +def stop_execution(run_id: Optional[str] = None, reason: str = "") -> Dict[str, int]: + """Ask one stoppable run (or, without ``run_id``, every one) to stop.""" + from je_auto_control.utils.executor.run_control import stop_execution as _stop + return {"stopped": _stop(str(run_id) if run_id else None, str(reason or ""))} + + +def list_executions() -> List[Dict[str, Any]]: + """The stoppable runs in progress.""" + from je_auto_control.utils.executor.run_control import active_executions + return active_executions() + + def list_action_commands() -> List[str]: from je_auto_control.utils.executor.action_executor import executor return sorted(executor.known_commands()) diff --git a/je_auto_control/utils/ocr/ocr_engine.py b/je_auto_control/utils/ocr/ocr_engine.py index 7b94ebb5..88288f8a 100644 --- a/je_auto_control/utils/ocr/ocr_engine.py +++ b/je_auto_control/utils/ocr/ocr_engine.py @@ -15,6 +15,7 @@ from typing import List, Optional, Pattern, Sequence, Tuple, Union from je_auto_control.utils.exception.exceptions import AutoControlActionException +from je_auto_control.utils.executor.run_control import pause from je_auto_control.utils.logging.logging_instance import autocontrol_logger from je_auto_control.utils.monitor_layout.logical_frame import grab_logical from je_auto_control.utils.ocr.backends import ( @@ -202,7 +203,7 @@ def wait_for_text(target: str, except AutoControlActionException: if time.monotonic() >= deadline: break - time.sleep(poll) + pause(poll, time.sleep) raise AutoControlActionException(f"OCR: wait_for_text timeout: {target!r}") diff --git a/je_auto_control/utils/rate_limit/rate_limit.py b/je_auto_control/utils/rate_limit/rate_limit.py index b694942a..53c73db5 100644 --- a/je_auto_control/utils/rate_limit/rate_limit.py +++ b/je_auto_control/utils/rate_limit/rate_limit.py @@ -21,6 +21,7 @@ from typing import Callable, Dict, Optional from je_auto_control.utils.exception.exceptions import AutoControlException +from je_auto_control.utils.executor.run_control import pause def _positive_finite(value: float) -> bool: @@ -90,7 +91,7 @@ def time_until_available(self, n: float = 1.0) -> float: return (n - self._tokens) / self._rate def acquire(self, n: float = 1.0, *, timeout: Optional[float] = None, - sleep: Callable[[float], None] = time.sleep) -> bool: + sleep: Callable[[float], None] = pause) -> bool: """Block until ``n`` tokens are taken or ``timeout`` elapses.""" self._check_request(n) # A NaN deadline is never reached, so the wait never timed out. diff --git a/je_auto_control/utils/retry_budget/retry_budget.py b/je_auto_control/utils/retry_budget/retry_budget.py index 440eee1d..cfbed98f 100644 --- a/je_auto_control/utils/retry_budget/retry_budget.py +++ b/je_auto_control/utils/retry_budget/retry_budget.py @@ -22,6 +22,8 @@ from dataclasses import dataclass from typing import Any, Callable, Dict, List, Optional, Tuple, Type +from je_auto_control.utils.executor.run_control import pause + # A uniform sampler: ``(low, high) -> float`` in ``[low, high)``. Uniform = Callable[[float, float], float] @@ -134,7 +136,7 @@ def run_with_budget(func: Callable[..., Any], budget: RetryBudget, *, args: Tuple[Any, ...] = (), kwargs: Optional[Dict[str, Any]] = None, clock: Callable[[], float] = time.monotonic, - sleep: Callable[[float], None] = time.sleep, + sleep: Callable[[float], None] = pause, uniform: Uniform = _default_uniform) -> Any: """Call ``func`` until it succeeds or the budget is spent; re-raise on giveup. diff --git a/je_auto_control/utils/self_healing/verification.py b/je_auto_control/utils/self_healing/verification.py index 7ddcc889..c2c96000 100644 --- a/je_auto_control/utils/self_healing/verification.py +++ b/je_auto_control/utils/self_healing/verification.py @@ -28,6 +28,7 @@ from je_auto_control.utils.exception.exceptions import ( AutoControlException, ImageNotFoundException, ) +from je_auto_control.utils.executor.run_control import pause VERIFY_IMAGE_GONE = "image_gone" VERIFY_IMAGE_PRESENT = "image_present" @@ -149,7 +150,7 @@ def _poll(probe: Probe, kind: str, timeout_s: float, poll_s: float, def build_verifier(spec: Mapping[str, Any], *, template_path: Optional[str] = None, screen_region: Region = None, clock: Callable[[], float] = time.monotonic, - sleep: Callable[[float], None] = time.sleep, + sleep: Callable[[float], None] = pause, ) -> Callable[[Any], bool]: """Turn a ``verify`` object into the check :func:`self_heal_click` calls. diff --git a/je_auto_control/utils/smart_waits/waits.py b/je_auto_control/utils/smart_waits/waits.py index 17212117..924d8646 100644 --- a/je_auto_control/utils/smart_waits/waits.py +++ b/je_auto_control/utils/smart_waits/waits.py @@ -23,6 +23,8 @@ from dataclasses import asdict, dataclass from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple +from je_auto_control.utils.executor.run_control import pause + _TIMEOUT_POSITIVE = "timeout_s must be positive" _POLL_POSITIVE = "poll_interval_s must be positive" @@ -59,7 +61,7 @@ def _pause(deadline: float, poll_interval_s: float) -> None: Every wait slept its whole interval, so a 0.1 s timeout polled at 1.5 s returned after 1.5 s. """ - time.sleep(max(0.0, min(float(poll_interval_s), deadline - time.monotonic()))) + pause(max(0.0, min(float(poll_interval_s), deadline - time.monotonic())), time.sleep) def _default_sampler(region: Optional[Sequence[int]]) -> Frame: diff --git a/je_auto_control/utils/state_machine/engine.py b/je_auto_control/utils/state_machine/engine.py index 7a26f749..e112e62e 100644 --- a/je_auto_control/utils/state_machine/engine.py +++ b/je_auto_control/utils/state_machine/engine.py @@ -5,6 +5,7 @@ from typing import Any, Callable, Dict, Mapping, Optional from je_auto_control.utils.exception.exceptions import AutoControlException +from je_auto_control.utils.executor.run_control import pause from je_auto_control.utils.rbac.deferred import adopted_scope, capture_owner from je_auto_control.utils.script_vars.execution import run_level_scope @@ -130,7 +131,7 @@ def _pick_transition(self, state: Mapping[str, Any], state_name: str, raise StateMachineError( f"no transition fired in state {state_name!r}", ) - time.sleep(min(wait_s, _TIMER_POLL_S)) + pause(min(wait_s, _TIMER_POLL_S), time.sleep) self._check_timeout(started_at, state_name) def _target(self, trans: Mapping[str, Any], state_name: str) -> str: diff --git a/je_auto_control/utils/thread_bound.py b/je_auto_control/utils/thread_bound.py new file mode 100644 index 00000000..9b4e68a4 --- /dev/null +++ b/je_auto_control/utils/thread_bound.py @@ -0,0 +1,73 @@ +"""A context variable whose value counts only on the thread that set it. + +Several pieces of per-call state are kept in a :class:`contextvars.ContextVar` +and documented as belonging to the calling thread: who a request is served as, +which device a matrix worker drives, which run a heal event belongs to, which +stop token a run answers to. That held because a new thread started with an +empty context -- a property of the interpreter, not of the code. On a +free-threaded build, under ``-X thread_inherit_context=1``, and whenever a +library runs a callable through ``contextvars.copy_context()``, a new thread +starts with a *copy of its creator's* context instead. A scheduler, observer +or hotkey thread first started while a request was being served would then +keep that request's identity (or device, or stop token) for the rest of the +process. + +:class:`ThreadBoundVar` records the thread that set a value and returns it on +that thread only; anywhere else it reads as its default, on every build. Code +that hands work to another thread on purpose captures the value first and sets +it again there, as ``AC_parallel`` does for the caller and the stop token. +The run's variable scope (:mod:`je_auto_control.utils.script_vars.execution`) +follows the same rule with its own copy of this logic. +""" +import threading +from contextvars import ContextVar, Token +from typing import Generic, Optional, Tuple, TypeVar + +_Value = TypeVar("_Value") + +_THREAD = threading.local() + + +def thread_marker() -> object: + """An object that identifies the calling thread and nothing else. + + An object, not ``threading.get_ident()``: an ident is reused once its + thread has ended, and a binding keeps this object alive, so a later thread + cannot be mistaken for the one that made it. + """ + marker = getattr(_THREAD, "marker", None) + if marker is None: + marker = _THREAD.marker = object() + return marker + + +class ThreadBoundVar(Generic[_Value]): + """A ``ContextVar`` that a thread started later does not inherit. + + ``get`` / ``set`` / ``reset`` as on a ``ContextVar``; ``get`` returns + ``default`` on any thread other than the one that called ``set``. + Within one thread it behaves exactly as a ``ContextVar`` does, asyncio + tasks included. + """ + + def __init__(self, name: str, default: _Value) -> None: + self._default = default + self._var: "ContextVar[Optional[Tuple[_Value, object]]]" = ContextVar(name, default=None) + + def get(self) -> _Value: + """The value set on this thread, else the default.""" + binding = self._var.get() + if binding is None or binding[1] is not thread_marker(): + return self._default + return binding[0] + + def set(self, value: _Value) -> "Token[Optional[Tuple[_Value, object]]]": + """Set ``value`` for this thread; hand the token to :meth:`reset`.""" + return self._var.set((value, thread_marker())) + + def reset(self, token: "Token[Optional[Tuple[_Value, object]]]") -> None: + """Restore what was set before the :meth:`set` that returned ``token``.""" + self._var.reset(token) + + +__all__ = ["ThreadBoundVar", "thread_marker"] diff --git a/je_auto_control/wrapper/auto_control_window.py b/je_auto_control/wrapper/auto_control_window.py index b450c50f..beda616c 100644 --- a/je_auto_control/wrapper/auto_control_window.py +++ b/je_auto_control/wrapper/auto_control_window.py @@ -16,6 +16,7 @@ from typing import List, Optional, Tuple, Union from je_auto_control.utils.exception.exceptions import AutoControlActionException +from je_auto_control.utils.executor.run_control import pause from je_auto_control.utils.logging.logging_instance import autocontrol_logger from je_auto_control.wrapper.window_backends import get_backend from je_auto_control.utils.timeouts import clamp_poll_interval, deadline_after @@ -118,7 +119,7 @@ def wait_for_window(title_substring: str, remaining = deadline - time.monotonic() if remaining <= 0: break - time.sleep(min(poll, remaining)) + pause(min(poll, remaining), time.sleep) raise AutoControlActionException( f"wait_for_window timeout: {title_substring!r}" ) diff --git a/je_auto_control/wrapper/mobile_extensions.py b/je_auto_control/wrapper/mobile_extensions.py index 8fa53a80..91855497 100644 --- a/je_auto_control/wrapper/mobile_extensions.py +++ b/je_auto_control/wrapper/mobile_extensions.py @@ -22,6 +22,7 @@ from time import monotonic from typing import Any, Callable, Dict, Optional, Protocol, runtime_checkable +from je_auto_control.utils.executor.run_control import checkpoint from je_auto_control.wrapper.device_context import ( MOBILE_PLATFORMS, AppState, DeviceCapability, DeviceError, DeviceSession, DeviceTimeoutError, @@ -150,7 +151,8 @@ def wait_for_app(session: DeviceSession, app_id: str, *, timeout_s: float = 10.0 state: AppState = AppState.FOREGROUND) -> AppState: """Wait until ``app_id`` reaches ``state``; raises :class:`DeviceTimeoutError` if it does not. - Cancelling the session ends the wait at the next poll. + Cancelling the session, or stopping the stoppable run this is part of, + ends the wait at the next poll. """ wanted = AppState(state) deadline = monotonic() + float(timeout_s) @@ -163,6 +165,7 @@ def wait_for_app(session: DeviceSession, app_id: str, *, timeout_s: float = 10.0 f"{app_id} is {current.value}, not {wanted.value}, after {float(timeout_s):g}s") if session.wait_cancelled(_WAIT_POLL_S): session.check_usable() + checkpoint() def accept_alert(session: DeviceSession) -> str: diff --git a/pyproject.toml b/pyproject.toml index c74e1c8d..3470287b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -441,6 +441,7 @@ module = [ "je_auto_control.gui._tab_task", "je_auto_control.gui.remote_desktop.connection_recent", "je_auto_control.utils.executor.run_control", + "je_auto_control.utils.thread_bound", "je_auto_control.gui._dispose", "je_auto_control.gui._slow_op", "je_auto_control.utils.codegen.candidate_diff", diff --git a/test/unit_test/headless/test_chatops_stopped_run.py b/test/unit_test/headless/test_chatops_stopped_run.py new file mode 100644 index 00000000..90ce2615 --- /dev/null +++ b/test/unit_test/headless/test_chatops_stopped_run.py @@ -0,0 +1,54 @@ +"""A stopped ``/run`` is reported as stopped, not as a failed script. + +The script below stops its own stoppable block (``AC_stop_execution`` with the +block's ``run_id``) and then sleeps: no input, no screen, and it returns at +once because the sleep is woken by the stop. +""" +import json + +from je_auto_control.utils.chatops import CommandRouter +from je_auto_control.utils.chatops.handlers import register_default_commands +from je_auto_control.utils.chatops.router import CommandResult +from je_auto_control.utils.executor.run_control import ExecutionStopped, active_executions + +_SCRIPT = [["AC_run_stoppable", {"run_id": "chat-run", "body": [ + ["AC_stop_execution", {"run_id": "chat-run", "reason": "operator said so"}], + ["AC_sleep", {"seconds": 30}], +]}]] + + +def _router() -> CommandRouter: + return register_default_commands(CommandRouter()) + + +def test_a_stopped_run_answers_stopped(tmp_path): + (tmp_path / "long.json").write_text(json.dumps(_SCRIPT), encoding="utf-8") + result = _router().dispatch("/run long.json", context={"script_root": str(tmp_path)}) + assert isinstance(result, CommandResult) + assert result.text == "run stopped. (operator said so)" + assert "failed" not in result.text and "ExecutionStopped" not in result.text + assert result.succeeded is False + assert result.metadata == {"stopped": True, "run_id": "chat-run", + "reason": "operator said so"} + assert active_executions() == [] + + +def test_a_stop_without_a_reason_has_no_trailing_parentheses(): + router = CommandRouter() + + def stopped(_argv, _context): + raise ExecutionStopped("r1") + + router.register("job", stopped) + result = router.dispatch("/job") + assert result is not None + assert result.text == "job stopped." + assert result.metadata["run_id"] == "r1" + + +def test_a_real_failure_still_reads_as_a_failure(tmp_path): + (tmp_path / "bad.json").write_text(json.dumps([["AC_no_such_command", {}]]), encoding="utf-8") + result = _router().dispatch("/run bad.json", context={"script_root": str(tmp_path)}) + assert result is not None + assert result.succeeded is False + assert "stopped" not in result.text diff --git a/test/unit_test/headless/test_mcp_stop_execution.py b/test/unit_test/headless/test_mcp_stop_execution.py new file mode 100644 index 00000000..b285f0a1 --- /dev/null +++ b/test/unit_test/headless/test_mcp_stop_execution.py @@ -0,0 +1,104 @@ +"""MCP tools for the stoppable runs: ``ac_stop_execution`` and ``ac_list_executions``. + +The run under test is ``ac_execute_actions`` given an ``AC_run_stoppable`` +block whose body is ``AC_sleep``: no input and no screen access. +""" +import threading +from typing import Any, Dict + +from je_auto_control.utils.executor.run_control import ( + ExecutionStopped, active_executions, stoppable_run, +) +from je_auto_control.utils.mcp_server.tools import build_default_tool_registry +from je_auto_control.utils.rbac.policy import capability_for_tool +from je_auto_control.utils.rbac.users import Capability + +_WAIT = 10.0 + + +def _tools(**kwargs: Any) -> Dict[str, Any]: + return {tool.name: tool for tool in build_default_tool_registry(aliases=False, **kwargs)} + + +def test_both_tools_are_registered_with_their_annotations(): + tools = _tools(read_only=False) + stop, listing = tools["ac_stop_execution"], tools["ac_list_executions"] + assert stop.annotations.read_only is False + assert listing.annotations.read_only is True + assert set(stop.input_schema["properties"]) == {"run_id", "reason"} + assert stop.input_schema.get("required", []) == [] + assert listing.input_schema["properties"] == {} + + +def test_stopping_needs_drive_input_and_listing_only_read_screen(): + tools = _tools(read_only=False) + stop, listing = tools["ac_stop_execution"], tools["ac_list_executions"] + assert capability_for_tool(stop.name, stop.annotations.read_only) == Capability.DRIVE_INPUT + assert capability_for_tool(listing.name, listing.annotations.read_only) == Capability.READ_SCREEN + + +def test_a_read_only_server_lists_runs_but_cannot_stop_them(): + tools = _tools(read_only=True) + assert "ac_list_executions" in tools + assert "ac_stop_execution" not in tools + + +def test_list_reports_a_run_and_stop_ends_it(): + tools = _tools(read_only=False) + assert tools["ac_list_executions"].handler() == [] + with stoppable_run("mcp-listed") as token: + listed = tools["ac_list_executions"].handler() + assert [row["run_id"] for row in listed] == ["mcp-listed"] + assert listed[0]["stopping"] is False + assert isinstance(listed[0]["started_at"], float) + assert tools["ac_stop_execution"].handler(run_id="mcp-listed", reason="via mcp") == {"stopped": 1} + assert token.stopped and token.reason == "via mcp" + assert tools["ac_list_executions"].handler()[0]["stopping"] is True + assert tools["ac_stop_execution"].handler(run_id="mcp-listed") == {"stopped": 0} + + +def test_stop_without_a_run_id_stops_every_run(): + tools = _tools(read_only=False) + with stoppable_run("mcp-a") as first: + assert tools["ac_stop_execution"].handler() == {"stopped": 0} # the caller's own run is spared + assert not first.stopped + outcome: Dict[str, Any] = {} + worker = threading.Thread( + target=lambda: outcome.update(reply=tools["ac_stop_execution"].handler()), daemon=True) + worker.start() + worker.join(_WAIT) + assert outcome == {"reply": {"stopped": 1}} + assert first.stopped + + +def test_an_action_list_run_through_mcp_is_stopped_by_the_mcp_tool(): + tools = _tools(read_only=False) + outcome: Dict[str, Any] = {} + + def run() -> None: + try: + outcome["result"] = tools["ac_execute_actions"].handler(actions=[ + ["AC_run_stoppable", {"run_id": "mcp-run", "body": [["AC_sleep", {"seconds": 60}]]}]]) + except ExecutionStopped as error: + outcome["error"] = error + + worker = threading.Thread(target=run, daemon=True) + worker.start() + started = threading.Event() + + def poll() -> None: + while not started.is_set(): + if any(row["run_id"] == "mcp-run" for row in tools["ac_list_executions"].handler()): + started.set() + else: + started.wait(0.01) + + watcher = threading.Thread(target=poll, daemon=True) + watcher.start() + assert started.wait(_WAIT), "the run never appeared in ac_list_executions" + assert tools["ac_stop_execution"].handler(run_id="mcp-run", reason="enough") == {"stopped": 1} + worker.join(_WAIT) + assert not worker.is_alive() + assert isinstance(outcome.get("error"), ExecutionStopped) + assert outcome["error"].reason == "enough" + assert active_executions() == [] diff --git a/test/unit_test/headless/test_stop_aware_waits.py b/test/unit_test/headless/test_stop_aware_waits.py new file mode 100644 index 00000000..1adefa44 --- /dev/null +++ b/test/unit_test/headless/test_stop_aware_waits.py @@ -0,0 +1,328 @@ +"""A stop wakes every polling wait an ``AC_*`` command can reach. + +Each wait here is given a probe that never succeeds, a timeout of a minute and +a poll interval of half a minute, and is run inside a stoppable run on a worker +thread. Before the fix they slept on ``time.sleep`` and returned only at their +own timeout; now the stop ends them within the join below. Every probe is a +fake: nothing reads the screen, a window list or the keyboard. +""" +import threading +import time +from typing import Any, Callable, Dict, List + +import pytest + +from je_auto_control.utils.exception.exceptions import AutoControlActionException +from je_auto_control.utils.executor import run_control +from je_auto_control.utils.executor.action_executor import Executor +from je_auto_control.utils.executor.run_control import ( + ExecutionStopped, StopToken, stoppable_run, +) + +_JOIN_S = 10.0 +_TIMEOUT_S = 60.0 +_POLL_S = 30.0 + + +def _stop_while_waiting(wait: Callable[[threading.Event], Any]) -> BaseException: + """Run ``wait(started)`` in a stoppable run, stop it once it probed; return what it raised.""" + token = StopToken() + started = threading.Event() + outcome: Dict[str, Any] = {} + + def target() -> None: + try: + with stoppable_run(token=token): + outcome["result"] = wait(started) + except BaseException as error: # noqa: BLE001 # reason: handed to the asserting thread + outcome["error"] = error + + thread = threading.Thread(target=target, daemon=True) + began = time.monotonic() + thread.start() + assert started.wait(_JOIN_S), "the wait never probed" + token.stop("test") + thread.join(_JOIN_S) + assert not thread.is_alive(), "the wait slept through the stop" + assert time.monotonic() - began < _POLL_S, "the wait ran to its own poll interval" + assert "error" in outcome, f"the wait returned {outcome.get('result')!r} instead of stopping" + return outcome["error"] + + +def _never(started: threading.Event, value: Any = False) -> Callable[..., Any]: + def probe(*_args: Any, **_kwargs: Any) -> Any: + started.set() + return value + return probe + + +def test_wait_for_window_is_woken(monkeypatch): + from je_auto_control.wrapper import auto_control_window + + def wait(started): + monkeypatch.setattr(auto_control_window, "find_window", _never(started, None)) + return auto_control_window.wait_for_window("nope", timeout=_TIMEOUT_S, poll=_POLL_S) + + assert isinstance(_stop_while_waiting(wait), ExecutionStopped) + + +def test_ac_wait_window_is_woken_through_the_executor(monkeypatch): + from je_auto_control.wrapper import auto_control_window + executor = Executor() + + def wait(started): + monkeypatch.setattr(auto_control_window, "find_window", _never(started, None)) + return executor.execute_action( + [["AC_wait_window", {"title_substring": "nope", "timeout": _TIMEOUT_S, "poll": _POLL_S}]]) + + assert isinstance(_stop_while_waiting(wait), ExecutionStopped) + + +def test_wait_for_text_is_woken(monkeypatch): + from je_auto_control.utils.ocr import ocr_engine + + def wait(started): + def missing(*_args, **_kwargs): + started.set() + raise AutoControlActionException("not on screen") + monkeypatch.setattr(ocr_engine, "locate_text_center", missing) + return ocr_engine.wait_for_text("nope", timeout=_TIMEOUT_S, poll=_POLL_S) + + assert isinstance(_stop_while_waiting(wait), ExecutionStopped) + + +def test_smart_wait_until_gone_is_woken(): + from je_auto_control.utils.smart_waits.waits import wait_until_gone + + def wait(started): + return wait_until_gone(_never(started, True), timeout_s=_TIMEOUT_S, poll_interval_s=_POLL_S) + + assert isinstance(_stop_while_waiting(wait), ExecutionStopped) + + +def test_smart_wait_for_file_is_woken(tmp_path, monkeypatch): + from je_auto_control.utils.smart_waits import waits + real_monotonic = time.monotonic + + def wait(started): + def clock() -> float: + started.set() + return real_monotonic() + monkeypatch.setattr(waits.time, "monotonic", clock) + return waits.wait_until_file(str(tmp_path / "never.txt"), + timeout_s=_TIMEOUT_S, poll_interval_s=_POLL_S) + + assert isinstance(_stop_while_waiting(wait), ExecutionStopped) + + +def test_expect_poll_is_woken(): + from je_auto_control.utils.expect_poll.expect_poll import expect_poll + + def wait(started): + return expect_poll(_never(started, 0), lambda value: value == 1, + timeout_s=_TIMEOUT_S, interval_s=_POLL_S) + + assert isinstance(_stop_while_waiting(wait), ExecutionStopped) + + +def test_assert_eventually_is_woken(monkeypatch): + from je_auto_control.utils.assertion import combinators + + class _Failing: + passed = False + message = "no" + + def wait(started): + monkeypatch.setattr(combinators, "run_assertion_spec", _never(started, _Failing())) + return combinators.assert_eventually({"type": "fake"}, timeout=_TIMEOUT_S, interval=_POLL_S) + + assert isinstance(_stop_while_waiting(wait), ExecutionStopped) + + +def test_wait_until_app_idle_is_woken(): + from je_auto_control.utils.app_idle.app_idle import wait_until_app_idle + + def wait(started): + return wait_until_app_idle(busy_probe=_never(started, True), + timeout_s=_TIMEOUT_S, interval_s=_POLL_S) + + assert isinstance(_stop_while_waiting(wait), ExecutionStopped) + + +def test_wait_for_composition_commit_is_woken(): + from je_auto_control.utils.ime_state.ime_state import wait_for_composition_commit + composing = {"open": True, "conversion": 0, "composition": "ni"} + + def wait(started): + return wait_for_composition_commit(reader=_never(started, composing), + timeout_s=_TIMEOUT_S, interval_s=_POLL_S) + + assert isinstance(_stop_while_waiting(wait), ExecutionStopped) + + +@pytest.mark.parametrize("name, locked", [("wait_for_unlock", True), ("wait_for_lock", False)]) +def test_lock_waits_are_woken(name, locked): + from je_auto_control.utils.lock_session.lock_session import wait_for_lock, wait_for_unlock + waits = {"wait_for_unlock": wait_for_unlock, "wait_for_lock": wait_for_lock} + + def wait(started): + return waits[name](probe=_never(started, locked), + timeout_s=_TIMEOUT_S, interval_s=_POLL_S) + + assert isinstance(_stop_while_waiting(wait), ExecutionStopped) + + +def test_wait_actionable_is_woken(): + from je_auto_control.utils.actionability.actionability import GateConfig, wait_actionable + + def wait(started): + return wait_actionable(_never(started, None), + config=GateConfig(timeout_s=_TIMEOUT_S, poll_interval_s=_POLL_S)) + + assert isinstance(_stop_while_waiting(wait), ExecutionStopped) + + +def test_heal_verifier_is_woken(monkeypatch): + from je_auto_control.utils.self_healing import verification + + def wait(started): + monkeypatch.setattr(verification, "text_on_screen", _never(started, False)) + verify = verification.build_verifier( + {"type": "text_present", "text": "Saved", "timeout_s": _TIMEOUT_S, "poll_s": _POLL_S}) + return verify(None) + + assert isinstance(_stop_while_waiting(wait), ExecutionStopped) + + +def test_state_machine_timer_is_woken(): + from je_auto_control.utils.state_machine.engine import StateMachine + spec = {"initial": "idle", "global_timeout_s": 120, + "states": {"idle": {"on_enter": [["AC_fake_enter", {}]], + "transitions": [{"after": _TIMEOUT_S, "go_to": "done"}]}, + "done": {"final": True}}} + + def wait(started): + return StateMachine(spec, execute_action=lambda _action: started.set()).run() + + assert isinstance(_stop_while_waiting(wait), ExecutionStopped) + + +def test_retry_budget_backoff_is_woken(): + from je_auto_control.utils.retry_budget.retry_budget import RetryBudget, run_with_budget + budget = RetryBudget(max_attempts=3, base_delay_s=_POLL_S, max_delay_s=_POLL_S, + jitter="none", exceptions=(ValueError,)) + + def wait(started): + def failing() -> None: + started.set() + raise ValueError("again") + return run_with_budget(failing, budget) + + assert isinstance(_stop_while_waiting(wait), ExecutionStopped) + + +def test_rate_limit_acquire_is_woken(): + from je_auto_control.utils.rate_limit.rate_limit import TokenBucket + bucket = TokenBucket(rate=1.0 / _POLL_S, capacity=1.0) + assert bucket.try_acquire(1.0) + + def wait(started): + started.set() + return bucket.acquire(1.0, timeout=_TIMEOUT_S) + + assert isinstance(_stop_while_waiting(wait), ExecutionStopped) + + +def test_file_dialog_wait_is_woken(monkeypatch): + from je_auto_control.utils.file_dialog.file_dialog import FileDialogDriver + from je_auto_control.wrapper import window_backends + + class _Backend: + def __init__(self, started: threading.Event) -> None: + self._started = started + + def list_windows(self) -> list: + self._started.set() + return [] + + def wait(started): + monkeypatch.setattr(window_backends, "get_backend", lambda: _Backend(started)) + return FileDialogDriver().wait_window("Open", _TIMEOUT_S) + + assert isinstance(_stop_while_waiting(wait), ExecutionStopped) + + +def test_hold_key_is_woken_and_lets_go_of_the_key(): + from je_auto_control.utils.key_hold.key_hold import hold_key + events: List[Dict[str, Any]] = [] + + def wait(started): + def sink(event: Dict[str, Any]) -> None: + events.append(event) + started.set() + return hold_key("a", _TIMEOUT_S, sink=sink) + + assert isinstance(_stop_while_waiting(wait), ExecutionStopped) + assert [event["op"] for event in events] == ["press", "release"] + + +def test_input_sequence_wait_is_woken_and_releases_what_it_holds(): + from je_auto_control.utils.input_macro.input_macro import run_sequence + events: List[Dict[str, Any]] = [] + + def wait(started): + def sink(event: Dict[str, Any]) -> None: + events.append(event) + started.set() + return run_sequence([{"op": "press", "key": "shift"}, + {"op": "wait", "ms": _TIMEOUT_S * 1000}], sink=sink) + + assert isinstance(_stop_while_waiting(wait), ExecutionStopped) + assert [event["op"] for event in events] == ["press", "release"] + + +def test_wait_for_app_is_woken(): + from je_auto_control.wrapper import mobile_extensions + from je_auto_control.wrapper.device_context import AppState + + class _Session: + def __init__(self, started: threading.Event) -> None: + self._started = started + self.usable_checks = 0 + + def app_state(self, _app_id: str) -> AppState: + self._started.set() + return AppState.BACKGROUND + + def wait_cancelled(self, timeout_s: float) -> bool: + time.sleep(min(timeout_s, 0.05)) + return False + + def check_usable(self) -> None: + self.usable_checks += 1 + + def wait(started): + return mobile_extensions.wait_for_app(_Session(started), "com.example", timeout_s=_TIMEOUT_S) + + assert isinstance(_stop_while_waiting(wait), ExecutionStopped) + + +def test_outside_a_stoppable_run_the_waits_still_sleep_on_time_sleep(monkeypatch): + from je_auto_control.utils.expect_poll.expect_poll import expect_poll + slept: List[float] = [] + monkeypatch.setattr(run_control.time, "sleep", slept.append) + values = iter([0, 0, 1]) + result = expect_poll(lambda: next(values), lambda value: value == 1, + timeout_s=5.0, interval_s=0.01) + assert result.ok and result.attempts == 3 + assert slept == [0.01, 0.01] + assert run_control.current_stop_token() is None + + +def test_a_wait_that_succeeds_inside_a_stoppable_run_returns_normally(): + from je_auto_control.utils.expect_poll.expect_poll import expect_poll + values = iter([0, 1]) + with stoppable_run() as token: + result = expect_poll(lambda: next(values), lambda value: value == 1, + timeout_s=5.0, interval_s=0.01) + assert result.ok and not token.stopped From 945c80ac74e55a8e362fa5ac8871120b40984f9b Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 07:46:51 +0800 Subject: [PATCH 14/17] Keep the RBAC caller, the bound device and the heal stamp on the thread that set them They live in context variables and were per-thread only because a new thread starts with an empty context. On a free-threaded build it starts with a copy of its creator's, so a scheduler or observer thread first started during an admin's request kept running as that admin. The values now count on the binding thread only, as the variable scope already does. --- .github/workflows/quality.yml | 7 +- je_auto_control/utils/rbac/authorization.py | 20 +- je_auto_control/utils/self_healing/locator.py | 13 +- je_auto_control/wrapper/device_context.py | 13 +- .../headless/test_thread_bound_context.py | 285 ++++++++++++++++++ 5 files changed, 323 insertions(+), 15 deletions(-) create mode 100644 test/unit_test/headless/test_thread_bound_context.py diff --git a/.github/workflows/quality.yml b/.github/workflows/quality.yml index 0f01e7d8..68cefe80 100644 --- a/.github/workflows/quality.yml +++ b/.github/workflows/quality.yml @@ -268,6 +268,11 @@ jobs: # on 3.14 (test_execution_scope_threads.py); this runs the same three files # on a real free-threaded interpreter with the GIL off. # + # The same rule covers the other per-thread state kept in context variables + # -- the caller a request is served as, the device bound by use_device, the + # heal-event stamp and a run's stop token (utils/thread_bound.py) -- whose + # tests are the fourth file, test_thread_bound_context.py. + # # The project is not pip-installed: opencv-python ships abi3 wheels only, # which a free-threaded interpreter cannot load, and nothing these tests # import needs it. The packages below are what importing the executor needs, @@ -299,7 +304,7 @@ jobs: env: PYTHON_GIL: "0" PYTHONPATH: ${{ github.workspace }} - run: '"$FT_PYTHON" -X thread_inherit_context=1 -m pytest -q -p no:cacheprovider test/unit_test/headless/test_execution_scope_threads.py test/unit_test/headless/test_execution_scope_isolation.py test/unit_test/headless/test_scope_run_level_callers.py' + run: '"$FT_PYTHON" -X thread_inherit_context=1 -m pytest -q -p no:cacheprovider test/unit_test/headless/test_execution_scope_threads.py test/unit_test/headless/test_execution_scope_isolation.py test/unit_test/headless/test_scope_run_level_callers.py test/unit_test/headless/test_thread_bound_context.py' # Sphinx with warnings as errors. The headless suite reads every page with # the `dummy` builder when Sphinx is installed (test_sphinx_docs_build.py), diff --git a/je_auto_control/utils/rbac/authorization.py b/je_auto_control/utils/rbac/authorization.py index 0fc1994e..36fc09fc 100644 --- a/je_auto_control/utils/rbac/authorization.py +++ b/je_auto_control/utils/rbac/authorization.py @@ -15,13 +15,13 @@ import contextlib import os -from contextvars import ContextVar from dataclasses import dataclass, field from pathlib import Path from typing import Iterator, Optional from je_auto_control.utils.exception.exceptions import AutoControlException from je_auto_control.utils.rbac.users import UserAuthError, UserStore, can +from je_auto_control.utils.thread_bound import ThreadBoundVar #: Names the user store file. Setting it is what switches RBAC on for the #: REST API and the MCP HTTP transport; unset or empty leaves both on their @@ -79,12 +79,20 @@ def resolve_token(store: UserStore, token: str) -> Optional[AuthorizationContext return AuthorizationContext(user_id=record.user_id, role=record.role, store=store) -_CURRENT: ContextVar[Optional[AuthorizationContext]] = ContextVar( - "je_auto_control_authorization", default=None) +# Bound to the thread that entered the scope: on a free-threaded build (and +# under ``-X thread_inherit_context=1``) a new thread starts with a copy of its +# creator's context, and a scheduler or observer thread first started while an +# admin's request was being served would have gone on running as that admin. +_CURRENT: ThreadBoundVar[Optional[AuthorizationContext]] = ThreadBoundVar( + "je_auto_control_authorization", None) def current_authorization() -> Optional[AuthorizationContext]: - """The caller this thread is serving, or ``None`` outside any RBAC scope.""" + """The caller this thread is serving, or ``None`` outside any RBAC scope. + + ``None`` also on a thread that was merely started inside a scope: the + scope counts on the thread that entered it, on every build. + """ return _CURRENT.get() @@ -92,7 +100,9 @@ def current_authorization() -> Optional[AuthorizationContext]: def authorization_scope(context: Optional[AuthorizationContext]) -> Iterator[None]: """Serve the enclosed work as ``context``; ``None`` clears any outer scope. - The scope belongs to the calling thread. Work handed to another thread -- + The scope belongs to the calling thread, also where a new thread inherits + its creator's context variables (a free-threaded build). Work handed to + another thread -- a scheduler job, a trigger, a hotkey -- runs outside it, which is why those registries store the caller and re-enter a scope when the work fires (:mod:`je_auto_control.utils.rbac.deferred`). diff --git a/je_auto_control/utils/self_healing/locator.py b/je_auto_control/utils/self_healing/locator.py index 061ed214..4daa177d 100644 --- a/je_auto_control/utils/self_healing/locator.py +++ b/je_auto_control/utils/self_healing/locator.py @@ -25,7 +25,6 @@ from __future__ import annotations from contextlib import contextmanager -from contextvars import ContextVar from dataclasses import asdict, dataclass, replace from datetime import datetime, timezone from time import monotonic @@ -40,6 +39,7 @@ from je_auto_control.utils.self_healing.heal_log import ( HEAL_EVENT_SCHEMA_VERSION, HealEvent, HealEventLog, default_heal_log, ) +from je_auto_control.utils.thread_bound import ThreadBoundVar METHOD_IMAGE = "image" @@ -50,7 +50,11 @@ #: Keys :func:`heal_context` accepts; each is a string column of the heal log. CONTEXT_KEYS = ("run_id", "step_id", "locator_id", "locator_version", "backend") -_context: ContextVar[Mapping[str, str]] = ContextVar("self_heal_context") +# Thread-bound: where a new thread inherits its creator's context variables (a +# free-threaded build), a thread started inside a heal_context block would +# otherwise stamp its own, unrelated heal events with that run and step. +_NO_CONTEXT: Mapping[str, str] = {} +_context: ThreadBoundVar[Mapping[str, str]] = ThreadBoundVar("self_heal_context", _NO_CONTEXT) @contextmanager @@ -60,11 +64,12 @@ def heal_context(**values: Optional[str]) -> Iterator[None]: Accepts the :data:`CONTEXT_KEYS` (``run_id``, ``step_id``, ``locator_id``, ``locator_version``, ``backend``); ``None`` values are ignored and an unknown key raises ``ValueError``. Nested blocks layer over outer ones. + The stamp applies on the calling thread only, on every build. """ unknown = sorted(set(values) - set(CONTEXT_KEYS)) if unknown: raise ValueError(f"unknown heal context key(s) {unknown}; expected {list(CONTEXT_KEYS)}") - merged = dict(_context.get({})) + merged = dict(_context.get()) merged.update({key: str(value) for key, value in values.items() if value is not None}) token = _context.set(merged) try: @@ -338,7 +343,7 @@ def _as_event(outcome: HealOutcome) -> HealEvent: [int(outcome.coordinates[0]), int(outcome.coordinates[1])] if outcome.coordinates is not None else None ) - context = _context.get({}) + context = _context.get() return HealEvent( timestamp=datetime.now(timezone.utc).isoformat(timespec="seconds"), method=outcome.method, diff --git a/je_auto_control/wrapper/device_context.py b/je_auto_control/wrapper/device_context.py index a84cc986..fbeb07d7 100644 --- a/je_auto_control/wrapper/device_context.py +++ b/je_auto_control/wrapper/device_context.py @@ -18,7 +18,6 @@ """ from __future__ import annotations -import contextvars import threading from contextlib import contextmanager from dataclasses import asdict, dataclass, field @@ -27,6 +26,7 @@ from typing import Any, Callable, Dict, Iterator, Mapping, Optional, Tuple, TypeVar, Union from je_auto_control.utils.exception.exceptions import AutoControlException +from je_auto_control.utils.thread_bound import ThreadBoundVar PLATFORM_ANDROID = "android" PLATFORM_IOS = "ios" @@ -481,16 +481,19 @@ def open_device(context: DeviceContext, **transports: Any) -> DeviceSession: return IOSSession(context, **transports) -_BOUND: "contextvars.ContextVar[Optional[DeviceSession]]" = contextvars.ContextVar( - "je_auto_control_device_session", default=None) +_BOUND: "ThreadBoundVar[Optional[DeviceSession]]" = ThreadBoundVar( + "je_auto_control_device_session", None) @contextmanager def use_device(session: DeviceSession) -> Iterator[DeviceSession]: """Make ``session`` the device that mobile commands without an address target. - The binding lives in a context variable, so it is visible only to the - thread that made it: two device-matrix workers each see their own device. + The binding is visible only to the thread that made it: two device-matrix + workers each see their own device, and a thread started inside the block + does not reach the device by leaving an address out -- also on a + free-threaded build, where a new thread inherits its creator's context + variables. """ token = _BOUND.set(session) try: diff --git a/test/unit_test/headless/test_thread_bound_context.py b/test/unit_test/headless/test_thread_bound_context.py new file mode 100644 index 00000000..ad81990c --- /dev/null +++ b/test/unit_test/headless/test_thread_bound_context.py @@ -0,0 +1,285 @@ +"""Per-thread state stays on the thread that set it, on every build. + +The caller a request is served as, the device a matrix worker drives, the +labels a heal event is stamped with and the stop token of a run are kept in +context variables and documented as belonging to the calling thread. A new +thread starts with an empty context on a default build, but with a copy of its +creator's on a free-threaded build and under ``-X thread_inherit_context=1``: +there a scheduler thread first started while an admin's request was being +served kept running as that admin. + +As in ``test_execution_scope_threads.py``, every test runs twice: on the +interpreter as it is, and with thread creation forced to inherit the creator's +context, which is what those builds do and works on every supported version. +The last test runs this file under the real flag where the interpreter has it. +Nothing touches a device, the screen or the keyboard. +""" +import asyncio +import contextvars +import os +import subprocess +import sys +import threading +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path + +import pytest + +from je_auto_control.utils.executor import run_control +from je_auto_control.utils.executor.run_control import ( + ExecutionStopped, bound_stop_token, checkpoint, current_stop_token, shielded, + stop_execution, stoppable_run, +) +from je_auto_control.utils.rbac.authorization import ( + AuthorizationContext, authorization_scope, current_authorization, +) +from je_auto_control.utils.rbac.deferred import adopted_scope, capture_owner +from je_auto_control.utils.self_healing import locator +from je_auto_control.utils.self_healing.locator import heal_context +from je_auto_control.utils.thread_bound import ThreadBoundVar, thread_marker +from je_auto_control.wrapper.device_context import bound_session, use_device + +REPO_ROOT = Path(__file__).resolve().parents[3] +WAIT = 20 +_CHILD_ENV = "AC_THREAD_BOUND_CHILD" +_ADMIN = AuthorizationContext(user_id="admin-user", role="admin") + + +@pytest.fixture(params=["native", "inheriting"]) +def threads(request, monkeypatch): + """How new threads get their context: the interpreter's way, or copied from the creator.""" + if request.param == "inheriting": + start = threading.Thread.start + + def start_in_creators_context(self): + context, run = contextvars.copy_context(), self.run + self.run = lambda: context.run(run) + start(self) + + monkeypatch.setattr(threading.Thread, "start", start_in_creators_context) + return request.param + + +def _in_thread(function): + """Run ``function`` on a new thread and return what it returned.""" + box = {} + worker = threading.Thread(target=lambda: box.setdefault("value", function())) + worker.start() + worker.join(WAIT) + assert not worker.is_alive() + return box["value"] + + +class _Session: + """Just enough of a ``DeviceSession`` for the binding.""" + + platform = "android" + device_id = "emulator-5554" + + +# --- the helper --------------------------------------------------------------------------------- + +def test_a_value_is_read_back_on_its_thread_and_not_on_another(threads): + var = ThreadBoundVar("test_thread_bound", "default") + token = var.set("mine") + try: + assert var.get() == "mine" + assert _in_thread(var.get) == "default" + with ThreadPoolExecutor(max_workers=1) as pool: + assert pool.submit(var.get).result(WAIT) == "default" + finally: + var.reset(token) + assert var.get() == "default" + + +def test_nested_sets_unwind_in_order(threads): + var = ThreadBoundVar("test_thread_bound_nested", 0) + outer = var.set(1) + inner = var.set(2) + assert var.get() == 2 + var.reset(inner) + assert var.get() == 1 + var.reset(outer) + assert var.get() == 0 + + +def test_asyncio_tasks_on_the_binding_thread_see_the_value(threads): + var = ThreadBoundVar("test_thread_bound_async", "default") + + async def view(): + return var.get() + + async def main(): + return await asyncio.gather(asyncio.create_task(view()), view()) + + token = var.set("mine") + try: + assert asyncio.run(main()) == ["mine", "mine"] + finally: + var.reset(token) + + +def test_each_thread_has_its_own_marker(threads): + assert thread_marker() is thread_marker() + assert _in_thread(thread_marker) is not thread_marker() + + +# --- who a request is served as ----------------------------------------------------------------- + +def test_a_thread_started_in_a_request_has_no_caller(threads): + with authorization_scope(_ADMIN): + assert current_authorization() is _ADMIN + assert _in_thread(current_authorization) is None + assert current_authorization() is _ADMIN, "starting a thread must not disturb the request" + assert current_authorization() is None + + +def test_a_daemon_started_in_an_admins_request_does_not_stay_admin(threads): + request_over, done = threading.Event(), threading.Event() + seen = [] + + def daemon(): + request_over.wait(WAIT) + seen.append(current_authorization()) + # Deferred work with no owner of its own must not pick the admin up either. + with adopted_scope(None): + seen.append(current_authorization()) + seen.append(capture_owner()) + done.set() + + with authorization_scope(_ADMIN): + threading.Thread(target=daemon, daemon=True).start() + request_over.set() + assert done.wait(WAIT) + assert seen == [None, None, None] + + +def test_a_caller_handed_over_on_purpose_is_served_on_the_other_thread(threads): + def branch(): + with authorization_scope(_ADMIN): + return current_authorization() + + assert _in_thread(branch) is _ADMIN + + +# --- which device a worker drives --------------------------------------------------------------- + +def test_a_thread_started_inside_use_device_does_not_reach_the_device(threads): + session = _Session() + with use_device(session): + assert bound_session("android") is session + assert _in_thread(lambda: bound_session("android")) is None + assert bound_session("android", "emulator-5554") is session + assert bound_session("android") is None + + +def test_two_workers_each_see_their_own_device(threads): + first, second = _Session(), _Session() + ready, release = threading.Barrier(2), threading.Event() + seen = {} + + def worker(name, session): + with use_device(session): + ready.wait(WAIT) + release.wait(WAIT) + seen[name] = bound_session("android") + + workers = [threading.Thread(target=worker, args=("a", first)), + threading.Thread(target=worker, args=("b", second))] + for thread in workers: + thread.start() + release.set() + for thread in workers: + thread.join(WAIT) + assert seen == {"a": first, "b": second} + + +# --- what a heal event is stamped with ---------------------------------------------------------- + +def test_a_thread_started_in_a_heal_context_is_not_stamped_with_it(threads): + with heal_context(run_id="run-1", step_id="s1"): + assert dict(locator._context.get()) == {"run_id": "run-1", "step_id": "s1"} + assert dict(_in_thread(locator._context.get)) == {} + with heal_context(locator_id="ok-button"): + assert dict(locator._context.get()) == { + "run_id": "run-1", "step_id": "s1", "locator_id": "ok-button"} + assert dict(locator._context.get()) == {} + + +# --- which stop a run answers to ---------------------------------------------------------------- + +def test_a_thread_started_in_a_stoppable_run_is_not_part_of_it(threads): + with stoppable_run("bound-run") as token: + assert current_stop_token() is token + assert _in_thread(current_stop_token) is None + + +def test_a_daemon_started_in_a_stopped_run_is_not_stopped_for_ever(threads): + run_over, done = threading.Event(), threading.Event() + outcome = [] + + def daemon(): + run_over.wait(WAIT) + try: + checkpoint() + run_control.pause(0) + outcome.append("alive") + except ExecutionStopped: + outcome.append("stopped") + # ...and its own "stop every other run" spares nothing by mistake. + outcome.append(stop_execution()) + done.set() + + with stoppable_run("bound-stopped") as token: + threading.Thread(target=daemon, daemon=True).start() + token.stop("over") + run_over.set() + assert done.wait(WAIT) + assert outcome == ["alive", 0] + + +def test_a_branch_given_the_token_stops_with_the_run(threads): + def branch(token): + with bound_stop_token(token): + try: + checkpoint() + except ExecutionStopped as error: + return error.run_id + return None + + with stoppable_run("bound-branch") as token: + token.stop() + assert _in_thread(lambda: branch(token)) == "bound-branch" + + +def test_a_shield_does_not_follow_a_thread_started_inside_it(threads): + def branch(token): + with bound_stop_token(token): + try: + checkpoint() + except ExecutionStopped: + return "stopped" + return "shielded" + + with stoppable_run("bound-shield") as token: + token.stop() + with shielded(): + checkpoint() # this thread's cleanup may finish + assert _in_thread(lambda: branch(token)) == "stopped" + + +# --- the real flag ------------------------------------------------------------------------------ + +@pytest.mark.skipif(not hasattr(sys.flags, "thread_inherit_context"), + reason="-X thread_inherit_context exists from Python 3.14") +@pytest.mark.skipif(bool(os.environ.get(_CHILD_ENV)), reason="already the child run") +def test_these_tests_pass_with_real_context_inheritance(): + """The interpreter's own inheritance, not the emulation: what a free-threaded build does.""" + env = dict(os.environ, PYTHONPATH=str(REPO_ROOT), **{_CHILD_ENV: "1"}) + done = subprocess.run( # nosec B603 # nosemgrep # reason: this interpreter, this test file, no shell + [sys.executable, "-X", "thread_inherit_context=1", "-m", "pytest", "-q", + "-p", "no:cacheprovider", str(Path(__file__))], + cwd=REPO_ROOT, env=env, capture_output=True, text=True, encoding="utf-8", + errors="replace", timeout=300, check=False) + assert done.returncode == 0, (done.stdout + done.stderr)[-3000:] + assert " passed" in done.stdout and " failed" not in done.stdout From bcffd45c28dddf4713e87ae9075b579124dadbf7 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 07:46:52 +0800 Subject: [PATCH 15/17] Pin why AC_llm_run takes no owner An action runs on the thread that executes its list, where the caller is already in scope: the request's own, or the owner scope a scheduler or trigger opened. An owner argument would only let a list's author name another user. --- .../utils/executor/action_executor.py | 2 +- .../headless/test_rbac_deferred_paths.py | 48 ++++++++++++++++++- 2 files changed, 48 insertions(+), 2 deletions(-) diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py index dd25d61f..d5cd74d2 100644 --- a/je_auto_control/utils/executor/action_executor.py +++ b/je_auto_control/utils/executor/action_executor.py @@ -2000,7 +2000,7 @@ def _llm_run_for_executor(description: str, examples: Optional[list] = None, model: Optional[str] = None, max_tokens: int = 2048) -> Dict[str, Any]: - """Executor adapter: plan and execute against the global executor.""" + """Executor adapter: plan and execute, as the caller this thread already serves (so no ``owner``).""" return llm_run_from_description( description, executor=executor, diff --git a/test/unit_test/headless/test_rbac_deferred_paths.py b/test/unit_test/headless/test_rbac_deferred_paths.py index 23387aee..64e3e376 100644 --- a/test/unit_test/headless/test_rbac_deferred_paths.py +++ b/test/unit_test/headless/test_rbac_deferred_paths.py @@ -28,7 +28,7 @@ USERS_ENV, DeferredOwner, Role, UserStore, authorization_scope, capture_owner, current_authorization, resolve_token, ) -from je_auto_control.utils.rbac.deferred import adopted_scope +from je_auto_control.utils.rbac.deferred import adopted_scope, owner_scope from je_auto_control.utils.state_machine import StateMachine _SIGN = "AC_sign_action_file" @@ -259,6 +259,52 @@ def test_a_plan_without_rbac_runs_as_before(ran): assert ran == [(_PROBE, None), (_SIGN, None)] +# ``AC_llm_run`` takes no ``owner``: an action is executed on the thread that +# runs its list, so the caller is already there -- the request's own scope, or +# the ``owner_scope`` a scheduler / trigger / hotkey opened for the entry's +# owner. An ``owner`` argument in an action list would only let its author +# name somebody else. + +_LLM_RUN = [["AC_llm_run", {"description": "sign it"}]] + + +@pytest.fixture() +def fake_llm(monkeypatch): + monkeypatch.setattr("je_auto_control.utils.llm.planner.get_backend", _FakeBackend) + + +def test_ac_llm_run_inside_a_request_runs_as_that_request(users, ran, fake_llm): + with _as(users, Role.OPERATOR): + execute_action(_LLM_RUN, raise_on_error=False) + assert ran == [(_PROBE, "operator-user")] + ran.clear() + with _as(users, Role.ADMIN): + execute_action(_LLM_RUN) + assert ran == [(_PROBE, "admin-user"), (_SIGN, "admin-user")] + + +def test_ac_llm_run_in_deferred_work_runs_as_the_works_owner(users, ran, fake_llm): + with _as(users, Role.OPERATOR): + owner = capture_owner() + + def fire(): + with owner_scope(owner): # what the scheduler does around a job + return execute_action(_LLM_RUN, raise_on_error=False) + + status, _result = _on_thread(fire) + assert status == "ok" + assert ran == [(_PROBE, "operator-user")] + + +def test_ac_llm_run_rejects_an_owner_argument(ran, fake_llm): + record = execute_action( + [["AC_llm_run", {"description": "sign it", + "owner": {"user_id": "admin-user", "role": "admin"}}]], + raise_on_error=False) + assert ran == [] + assert "owner" in str(list(record.values())[0]) + + # --- the helper -------------------------------------------------------------------------------- def test_adopted_scope_yields_to_a_caller_already_in_scope(users): From c279b4d5c56e32623c3fac2b2285f681e1f51d28 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 08:00:02 +0800 Subject: [PATCH 16/17] Record the fourth sweep: re-measured counts, update entries 34-36, compatibility notes and what is still open --- CHANGELOG.md | 28 +++++++++++++++++++ Progress.md | 61 +++++++++++++++++++++-------------------- README.md | 6 ++-- README/README_zh-CN.md | 6 ++-- README/README_zh-TW.md | 6 ++-- architecture_explore.md | 21 ++++++++------ docs/updates/2026-10.md | 18 ++++++++++++ docs/updates/README.md | 5 +++- 8 files changed, 102 insertions(+), 49 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b34a872b..d7845799 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -238,6 +238,34 @@ it shipped into a `## [x.y.z] - date` section of their own; the tag's ### Changed +- Stop: inside a stoppable run the polling waits (`AC_wait_window`, + `AC_wait_text`, the smart waits, `AC_expect_poll`, `AC_assert_eventually`, + `AC_wait_actionable` and others) end with `ExecutionStopped` as soon as a + stop is requested instead of running to their timeout. `expect_poll`, + `wait_until_app_idle`, `wait_for_unlock`, `build_verifier`, + `run_with_budget` and `TokenBucket.acquire` default their `sleep` to + `run_control.pause`. +- ChatOps: a stopped command replies "`` stopped." rather than a failure + naming `ExecutionStopped`. +- USB ACL: while the file fails its integrity check `UsbAcl.decide()` denies + every device and `default_policy` reads `"deny"`, also for an instance built + with `default_policy="allow"`. +- Config sync: an acknowledged deletion is kept for 30 days + (`tombstone_hold_s=0` restores immediate collection). `HttpAssetTransport` + asks the server's blob limit once and refuses a larger file without sending + it. Blob listings carry `age_s`. +- Folder mirror: a file whose size changes at an unchanged modification time, + or whose modification time goes backwards, is now pushed. +- GUI: releasing the Remote Desktop tab (`close_tab(..., release=True)`) ends + a running WebRTC host or viewer session; TCP and WebSocket sessions keep + running. Releasing any tab stops its background work at once. Callbacks + given to `start_worker`, `connect_viewer` and `start_panel_task` are held + weakly. +- Accessibility (Windows): each calling thread gets its own UI Automation + object, with COM initialised on that thread at its first call. +- Threads: a thread started inside `authorization_scope`, `use_device`, + `heal_context` or a stoppable run no longer sees that state on builds where + threads inherit context (free-threaded Python, `-X thread_inherit_context=1`). - Config sync: `ConfigBucket.upsert` / `remove` without `origin=` now store a versioned entry (`{"value", "vector", "origin", ...}`) instead of the flat value. Read values with `bucket.values(section)`, or pass `versioned=False` diff --git a/Progress.md b/Progress.md index 04f82a25..faa458e5 100644 --- a/Progress.md +++ b/Progress.md @@ -2,7 +2,7 @@ ## 跨平台與 GUI 全面改版 -`WIP` — 計畫 A–H 的程式都已交付（A 在 U-20261009-01…14，B–H 在 U-20261009-15…26，留下的缺口在 U-20261009-27…32）；還沒做到的是**實機驗證**、要維護者決定的幾件事，與下面各節列的少數缺口 +`WIP` — 計畫 A–H 的程式都已交付（A 在 U-20261009-01…14，B–H 在 U-20261009-15…26，留下的缺口在 U-20261009-27…37）；還沒做到的是**實機驗證**、要維護者決定的幾件事，與下面各節列的少數缺口 核准設計：[跨平台自動化與 GUI 改版](docs/superpowers/specs/2026-10-02-platform-gui-modernization-design.md)。 實作計畫：[分階段交付計畫](docs/superpowers/plans/2026-10-02-modernization-index.md)。 @@ -226,7 +226,7 @@ U-20261009-06 與 -04 的 macOS 部分是在 Windows 上對假的 pyobjc／Quart ## 安全與伺服器：留下的缺口 -`TODO` — U-20261009-29、-30 之後還開著的 +`TODO` — U-20261009-29、-30、-34、-35 之後還開著的 **要維護者過目或決定的：** @@ -234,7 +234,6 @@ U-20261009-06 與 -04 的 macOS 部分是在 Windows 上對假的 pyobjc／Quart `ac_list_run_history`、`ac_costs_*`、`ac_trace_export`、`ac_self_heal_log_list`、`ac_usb_acl_list`、`ac_vlm_locate`／`ac_self_heal_locate`（會花 VLM 費用）。 - Script Builder 現在永遠不顯示 `AC_user_add`／`AC_user_rotate_token` 回傳的 token（要從 Users 群組、CLI 或腳本取得）。 - 唯讀模式現在在三種工具模式都強制執行，而外掛工具一律被登記成會變更的：**唯讀的 MCP server 不會執行任何外掛工具**。要不要讓外掛宣告自己唯讀。 -- `UsbAcl(path, default_policy="allow")` 遇到被竄改的檔案會回報 `integrity_ok False`，但 `decide()` 仍回 allow（早於這批修改；倉庫內的呼叫端都用預設的 deny）。 - 設定同步：一台裝置加入「已經有內容」的 bucket 時仍會多提交一個 revision（這個登記是之後的刪除會等它的依據；要拿掉是 `merge.awaits_ack` 的一行，代價是那個保證）。 **還沒做的：** @@ -242,8 +241,9 @@ U-20261009-06 與 -04 的 macOS 部分是在 Windows 上對假的 pyobjc／Quart - **Intel Mac 安裝要編譯 `cryptography`**（下限 50 沒有 `macosx_10_9_x86_64` wheel），沒有在 Intel Mac 上實際編過。 - **簽章**：能執行寫檔指令的人仍能換掉公鑰檔本身；遷移模式開著時 HMAC 簽章會被接受。Windows 上新建的私鑰檔只有目前使用者可讀 （SYSTEM、Administrators、備份代理因此讀不到），沒在 FAT／網路磁碟或提權帳號下試過；POSIX 的權限位元檢查在這台機器上是跳過的。 -- **RBAC**：observer 的 predicate 仍不帶身分（只讀螢幕）；`AC_llm_run` 沒有 `owner` 參數；沒對真的 client（Claude Desktop、VS Code、內建 dashboard）試過。 +- **RBAC**：observer 的 predicate 仍不帶身分（只讀螢幕）；沒對真的 client（Claude Desktop、VS Code、內建 dashboard）試過。 MCP HTTP 的 `list_changed` 廣播遇到停住的 client，會讓註冊的執行緒在每條串流上最多等 30 秒。 + MCP 的 `notifications/cancelled` 只設定該次呼叫的 `cancelled_event`，不會停掉 executor 的執行（要接到 `StopToken`）。 - **MCP 路徑根目錄**管不到的：執行動作清單的工具、自由格式物件裡的路徑、沒標註的外掛工具、`ac_launch_process` 的 `argv` 與 `ac_shell` 的 `command`（刻意不管）。 `path-or-other` 有已知的誤判：開了根目錄時，像 `/help` 這種文字目標或剛好對到根目錄外既有檔案的值會被拒絕。 檔案 symlink 的跳脫在這台機器上建不出來；POSIX 的 `file:` URL、`:` 分隔與 `~` 沒在 Linux／macOS 跑。 @@ -251,15 +251,14 @@ U-20261009-06 與 -04 的 macOS 部分是在 Windows 上對假的 pyobjc／Quart - **USB passthrough**：沒對真的舊版 host、真的 WebRTC 通道或真的 USB 裝置跑過。`usb_acl.json` 改成單一檔案（簽章內嵌）之後，**舊版讀不了新檔、會全部拒絕**； 沒有對真的舊版安裝做過降版測試。 - **設定同步**： - - 沒有做過真的兩台機器同步；CI 新加的 `[signaling]` 安裝行與版本 pin 還沒在任何 runner 上跑過。 - - 過渡期風險：墓碑只等已經列在 `peers` 底下的裝置，所以每台機器都要先用這一版同步一次，之後才能有人刪東西。 + - 沒有做過真的兩台機器同步，也沒有真的多行程 server；伺服器端的測試在本機是略過的（共用環境沒裝 `fastapi`），只在 CI 跑。 + - 墓碑：已確認的刪除現在會保留 30 天（`TOMBSTONE_HOLD_S`），沒被列過的機器帶舊資料來時變成衝突。剩下的風險：超過 30 天才第一次同步的機器 + 仍可能讓項目復活；舊版 client 提交時會丟掉保留中的墓碑並去掉 `deleted_at`，所以保護程度取決於還在提交的最舊 client。沒對真的舊版安裝跑過。 - `ConfigBucket.upsert` 不帶 `origin=` 時現在存的是帶版本的項目，直接讀 `bucket.sections[s][id]["field"]` 的程式會壞（`bucket.values(section)` 兩種形狀都能讀）。 - 沒有自動的剪貼簿轉送，所以 `automatic=True`（`ClipboardEchoGuard.should_send`）在產品裡沒有呼叫端。 - - 上傳超過伺服器上限的 blob 時，伺服器依宣告長度回 413 就關閉連線；還在送 body 的 client 可能先看到連線被重設（Windows 的 CI 上發生過）， - 回報的就是連線錯誤而不是「太大」。`HttpAssetTransport` 沒有先問上限。 - - 資料夾鏡像靠修改時間判斷變更：對方送來的檔案落地後，同一個時間刻度內的本機編輯不會被推送（CI 上的測試因此要把 mtime 往後推）。 - - blob：沒有自動回收（只有 `DELETE` 與用量清單）；配額是單一行程內的鎖，兩個 server 行程共用一個資料夾時可能超出一個 blob。 - - Config Sync 分頁沒有區段選擇與 `assets_server` 開關；`examples/29_config_sync.py` 還是從 `utils.config_sync` 匯入而不是門面。 + - blob：鎖檔超過 30 秒會被視為過期而接管，單次寫入比這更久時仍可能超出配額；用 `publish_assets` 手動上傳的 blob 沒有 bucket 項目引用， + 回收（`AC_config_sync_collect_blobs`）時要自己列在 `keep`；對沒有 `age_s` 的舊 server，除非 `min_age_s=0` 否則什麼都不收。 + - 資料夾鏡像的「同刻度」判斷（`RACY_WINDOW_S` 2 秒內以內容比對）與 `os.utime(ns=)` 只在 NTFS 上跑過。 --- @@ -297,24 +296,27 @@ U-20261009-18 的每一個裝置面呼叫都只對假的 ADB／uiautomator2／WD ## GUI：留下的缺口 -`TODO` — U-20261009-27、-28 之後還開著的 - -- **Accessibility 分頁的 3 個動作與 A11y Audit 仍在 GUI 執行緒上**：Windows 的 UIA 後端（`accessibility/backends/windows_backend.py`）快取一個 COM 自動化物件、 - 沒有做每個執行緒的 COM 初始化；開發環境沒裝 `comtypes`，無法驗證搬到別的執行緒後還能用，所以沒有盲改。要先改後端（擁有 COM apartment 的執行緒，或每執行緒初始化）。 - **同一個原因的新風險**：從 GUI 啟動的腳本現在跑在工作執行緒上，腳本裡的 `AC_a11y_*` 會從新的執行緒呼叫 UIA，這條路徑沒有實際跑過。 -- **stop 叫不醒的等待**：`AC_wait_window`、`AC_wait_text`、smart wait、`AC_expect_poll` 用的是自己的 `time.sleep`，會跑到自己的逾時； - 已經進到後端的單一指令（一次影像搜尋、OCR、HTTP 請求、shell 指令）不會被打斷。非腳本的分頁工作在後端無法取消，只是結果被丟掉。 - ChatOps 的 router 會接住 `AutoControlException`，被停止的 `/run` 回的是「run failed: ExecutionStopped …」而不是「Stopped.」。沒有 stop／list 的 MCP 工具。 -- **WebRTC 停止路徑裡沒搬的**：`StatsPoller.stop()`、`_stop_adaptive`、`_stop_lan_advertise`，沒量過它們會不會卡。 -- **`dispose()`**：`RemoteDesktopTab.dispose()` 只停計時器，連線中的 viewer 或 host 刻意留在 registry；`_ViewerPanel`、`QuickConnectScreen` 與 WebRTC 面板自己沒有 `dispose()`； - 只在需要時才開 worker 的分頁（computer use、DAG、VLM、USB browser）沒有。 -- **切換主題後約 0.4 秒內**，被暫存的分頁其 `window()` 不是主視窗；關閉後留作隱藏子元件的分頁仍是同步重設樣式。基準（`benchmarks/gui_*.py`）CI 仍不設門檻。 -- **一個沒修的隱患**：`webrtc_host_connection._produce_offer` 與 `webrtc_viewer_connection._answer_off_thread` 把工作結果接到對面板的強參照回呼上； - 沒有 parent 的面板（測試裡）會因此在孫物件的解構子裡被銷毀而中止行程。應用程式裡面板都有 parent。`SlowOp` 已改成弱參照，這兩處還沒。 +`TODO` — U-20261009-27、-28、-34、-36 之後還開著的 + +- **Windows UIA 後端的執行緒模型完全沒對真的 COM 驗過**：現在每個呼叫執行緒各有一個自動化物件（`accessibility/backends/windows_automation.py`； + 主執行緒 STA、其他執行緒進 MTA），Accessibility 分頁與 A11y Audit 已移出 GUI 執行緒。開發環境沒裝 `comtypes`，CI 也沒有同時有 `comtypes` 與桌面的 job， + 測試全是假 COM 物件。未驗證的假設：MTA 工作執行緒上的 UIA 呼叫可用；焦點事件 callback 不需要 message pump； + `import comtypes` 第一次發生在已經是 MTA 的執行緒上會丟 `OSError`（憑記憶，沒處理）。`AC_wait_for_focus_change` 的原生 `Queue.get` 仍不會被 stop 叫醒。 +- **stop 還叫不醒的**：已經進到後端的單一指令（一次影像搜尋、OCR、HTTP 請求、shell 指令）、單一手勢內的節奏 sleep、`AC_wait_for_focus_change`； + MCP 自己的輪詢（`ac_wait_for_image`／`ac_wait_for_pixel`）不在可停止的執行裡。非腳本的分頁工作在後端無法取消，只是結果被丟掉。 + ChatOps 的 `/run` 只有在腳本自己含 `AC_run_stoppable` 時才停得掉（刻意沒有整個包起來：那會讓 GUI 的「全部停止」也停掉聊天裡的執行，要維護者決定）。 + 新轉成可停止的等待裡，`AC_handle_file_dialog`、`AC_wait_until_app_idle`、`AC_wait_for_composition_commit`、`AC_wait_for_unlock` 與行動裝置的 `wait_for_app` 只對假探針測過。 +- **WebRTC**：`StatsPoller.start()` 會在 GUI 執行緒上最多等 2 秒（`future.result(timeout=2.0)`，是啟動路徑，沒動）。LAN 廣播的註冊與取消已移出 GUI 執行緒， + 但真的 zeroconf 的時間與名稱衝突行為只對假物件測過。 +- **`DECIDE`：釋放 Remote Desktop 分頁時，WebRTC 的 host 與 viewer session 現在會被結束**（TCP／WebSocket 的仍留在 registry）。理由：WebRTC host 只屬於面板， + 面板釋放後沒有東西能停它，會在看不到的情況下繼續分享畫面。要改回去是拿掉兩個 `dispose()` 裡的 `self._stop_host_if_any` 與 `self._end_session_for_good`。 +- **切換主題後那幾個事件迴圈回合內**，以被暫存的頁面為 parent 的 `QMessageBox` 會相對於隱藏的暫存 widget 定位（只影響位置）。 + 系統匣路徑在 offscreen 下沒有 tray，`_on_tray_open` 是直接呼叫測的。基準（`benchmarks/gui_*.py`）CI 仍不設門檻。 - **把 Python 建的 widget 放進分頁列在結束時崩潰**：已查明觸發條件——分頁列按鈕上接了 Python callable、又有排隊中的 `deleteLater()`、而行程經 `os._exit` 離開； 先把 deferred delete 沖掉就不會。機制（Qt 在行程結束時釋放那個 slot）是推論，沒有 C 堆疊可看。`_tab_close_icon.py` 沒有改回自訂按鈕。 -- **對真實 Qt 型別的型別檢查**：`test/verify/typing_extras_exempt.txt` 列著 70 個模組（66 個在 `gui/`、4 個在 `utils/remote_desktop/`，共 711 個錯誤， - 多半是 mixin 呼叫它所混入的 widget 的方法），只准變少。 +- **對真實 Qt 型別的型別檢查**：`test/verify/typing_extras_exempt.txt` 從 70 個模組降到 18 個（11 個 WebRTC 面板 mixin、`navigation`、`step_form_view`、 + `triggers_tab`，與 4 個對 aiortc／av 型別的 `utils/remote_desktop` 模組），只准變少。`QImage.fromData(x, "JPEG")` 的 5 處保留 `# type: ignore[arg-type]`： + stub 要 bytes，實際的 binding 對 bytes 丟 `ValueError`。 - 遠端桌面的輸入改成佇列送出後，`AC_remote_send_input` 回的 `{"sent": True}` 意思是「已排入」；`SO_SNDTIMEO` 的 POSIX 包法與經 TLS socket 的行為沒在這台機器上驗過。 - 沒在 macOS／Linux 上看過新的關閉鈕、捲動與重設樣式；沒在真的多螢幕桌面試過視窗位置還原；PyBreeze 沒有對這一版跑過；各語系新增的字串（日文、簡體中文）沒有母語者看過。 @@ -327,9 +329,8 @@ U-20261009-18 的每一個裝置面呼叫都只對假的 ADB／uiautomator2／WD - **`DECIDE`：`pytest --cov` 可以放行了嗎**。進入點搬到 `je_auto_control_pytest` 之後重新量過（2026-10-09）：`coverage run -m pytest` 87.35%，`pytest --cov` 87.32%。 `CLAUDE.md` 與 `test_coverage_measurement.py` 仍規定只能用 `coverage run`；要不要放寬由維護者決定。 - **本機的共用 `.venv` 裝著 `je_auto_control_dev 0.0.136`**，它的 `pytest11` 進入點還是舊的重量級路徑；重裝即可，不影響 CI。 -- **三個新的 CI job 還沒在 GitHub 上跑過**（`typing-extras`、`docs`、`free-threaded-scope`）；`free-threaded-scope` 用的是 `3.14t`，setup-python 在 Windows 上怎麼命名那個直譯器沒驗過，紅了就拿掉。 -- **free-threaded**：變數範圍已修成只在綁定它的執行緒上有效，並在 3.14.8t（Windows）跑過。`rbac/authorization.py`、`self_healing/locator.py`、`wrapper/device_context.py` - 也用 `ContextVar`，在 free-threaded 版上仍會被執行緒繼承，沒有檢查。 +- **free-threaded**：呼叫者身分、綁定的裝置、自愈的執行標記與 stop token 都改成只在設定它的執行緒上有效（`utils/thread_bound.py`）。 + 這台機器沒有 free-threaded 直譯器：驗證用的是模擬繼承與真的 `-X thread_inherit_context=1`（3.14.7，GIL 開著）；`free-threaded-scope` job（3.14t）是第一次真的跑。 - **能力探測**：Windows 的鎖定工作站、session 0、低完整性與擷取失敗分支只用假資料判斷；macOS 的三個 preflight 呼叫只對假物件跑過， pyobjc 的 `Quartz` 有沒有 `CGPreflightListenEventAccess` 未知（沒有的話 `recording`／`stop_shortcut` 回 `unknown`）。 - **Windows 的版面鍵名**（`slash` 等 8 個）只在美式配置上實際呼叫過。 diff --git a/README.md b/README.md index aaee6769..ca867327 100644 --- a/README.md +++ b/README.md @@ -18,7 +18,7 @@ from JSON files / CLI / servers, and a **GUI tab**. Nothing is GUI-only. - **One API, seven platforms.** `wrapper/platform_wrapper.py` picks the backend at import time; your script does not change between Windows, macOS, X11, and Wayland. -- **Scriptable without Python.** 851 `AC_*` commands cover the whole feature set, so a +- **Scriptable without Python.** 852 `AC_*` commands cover the whole feature set, so a JSON file can do anything the library can — including loops, branches, try/catch, macros, and variables. - **Headless by default.** `import je_auto_control` never loads Qt. The GUI is an @@ -231,7 +231,7 @@ desktop app; tab commands live in the window's **Actions** menu. | Natural-language planner | `plan_actions`, `run_from_description` | `AC_llm_plan` | LLM Planner | | Computer-use agent | `AgentLoop`, `run_agent` | `AC_run_agent` | Computer Use | | Record & replay | `record`, `stop_record` | `AC_record`, `AC_stop_record` | Record | -| JSON scripting | `execute_action`, `execute_files` | all 851 commands | Script, Script Builder | +| JSON scripting | `execute_action`, `execute_files` | all 852 commands | Script, Script Builder | | Variables & flow control | `execute_action_with_vars` | `AC_set_var`, `AC_loop`, `AC_for_each`, `AC_try`, `AC_retry` | Variables | | Data-driven runs | — | `AC_for_each_row` (CSV / JSON / SQLite / Excel) | Data Sources | | Assertions | `assert_text`, `assert_image` | `AC_assert_text` + 20 more | Assertions | @@ -287,7 +287,7 @@ under a relative name are written below that directory. | Surface | Start it with | Notes | |---|---|---| -| **MCP server** | `je_auto_control_mcp` (stdio) or `AC_start_mcp_http_server` | 754 tools for Claude Desktop / Claude Code / custom tool loops: the full `ac_*` surface plus short model-friendly aliases for common GUI actions. Speaks the stateless MCP 2026-07-28 beside the `initialize`-based revisions. Bearer auth, TLS, audit log, rate limit, plugin hot-reload, CI fake backend. | +| **MCP server** | `je_auto_control_mcp` (stdio) or `AC_start_mcp_http_server` | 757 tools for Claude Desktop / Claude Code / custom tool loops: the full `ac_*` surface plus short model-friendly aliases for common GUI actions. Speaks the stateless MCP 2026-07-28 beside the `initialize`-based revisions. Bearer auth, TLS, audit log, rate limit, plugin hot-reload, CI fake backend. | | **REST API** | `je_auto_control start-rest` | Bearer token, per-IP rate limit + lockout, SQLite audit hook, `/metrics`, `/openapi.json`, `/docs` Swagger UI, `/dashboard`. | | **TCP socket server** | `je_auto_control start-server` | Newline-framed JSON action lists. Binds `127.0.0.1` by default. | | **pytest plugin** | installed automatically | Fixtures plus a Gherkin step library for pytest-bdd / behave. | diff --git a/README/README_zh-CN.md b/README/README_zh-CN.md index 06726586..27752064 100644 --- a/README/README_zh-CN.md +++ b/README/README_zh-CN.md @@ -19,7 +19,7 @@ - **一套 API，七个平台。** `wrapper/platform_wrapper.py` 在导入时挑选后端；同一份脚本在 Windows、macOS、X11 与 Wayland 上都不需要改写。 -- **不写 Python 也能脚本化。** 851 个 `AC_*` 命令覆盖全部功能，因此一个 JSON 文件能做到库 +- **不写 Python 也能脚本化。** 852 个 `AC_*` 命令覆盖全部功能，因此一个 JSON 文件能做到库 能做的任何事——包含循环、分支、try/catch、宏与变量。 - **默认无头运行。** `import je_auto_control` 绝不会加载 Qt。GUI 是可选包，包在同一个无头内核之外。 - **四种定位方式。** 模板匹配、OCR、无障碍树、视觉语言模型——可通过锚点定位器与自愈回退串接组合。 @@ -219,7 +219,7 @@ python -c "import je_auto_control; je_auto_control.start_autocontrol_gui()" | 自然语言规划 | `plan_actions`、`run_from_description` | `AC_llm_plan` | LLM Planner | | Computer-use agent | `AgentLoop`、`run_agent` | `AC_run_agent` | Computer Use | | 录制与回放 | `record`、`stop_record` | `AC_record`、`AC_stop_record` | Record | -| JSON 脚本 | `execute_action`、`execute_files` | 全部 851 个命令 | Script、Script Builder | +| JSON 脚本 | `execute_action`、`execute_files` | 全部 852 个命令 | Script、Script Builder | | 变量与流程控制 | `execute_action_with_vars` | `AC_set_var`、`AC_loop`、`AC_for_each`、`AC_try`、`AC_retry` | Variables | | 数据驱动执行 | — | `AC_for_each_row`（CSV／JSON／SQLite／Excel） | Data Sources | | 断言 | `assert_text`、`assert_image` | `AC_assert_text` 等 21 个 | Assertions | @@ -268,7 +268,7 @@ je_auto_control version | 接口 | 启动方式 | 说明 | |---|---|---| -| **MCP 服务器** | `je_auto_control_mcp`（stdio）或 `AC_start_mcp_http_server` | 754 个工具，供 Claude Desktop／Claude Code／自定义 tool loop 使用：完整的 `ac_*` 工具接口，以及常用 GUI 操作的短名称 alias。除了以 `initialize` 握手的各版协议，也支持无状态的 MCP 2026-07-28。Bearer 认证、TLS、审计日志、限流、插件热重载、CI 假后端。 | +| **MCP 服务器** | `je_auto_control_mcp`（stdio）或 `AC_start_mcp_http_server` | 757 个工具，供 Claude Desktop／Claude Code／自定义 tool loop 使用：完整的 `ac_*` 工具接口，以及常用 GUI 操作的短名称 alias。除了以 `initialize` 握手的各版协议，也支持无状态的 MCP 2026-07-28。Bearer 认证、TLS、审计日志、限流、插件热重载、CI 假后端。 | | **REST API** | `je_auto_control start-rest` | Bearer token、按 IP 限流与锁定、SQLite 审计 hook、`/metrics`、`/openapi.json`、`/docs` Swagger UI、`/dashboard`。 | | **TCP socket 服务器** | `je_auto_control start-server` | 以换行分隔的 JSON 动作列表。默认绑定 `127.0.0.1`。 | | **pytest 插件** | 安装后自动生效 | 提供 fixture 与供 pytest-bdd／behave 使用的 Gherkin step library。 | diff --git a/README/README_zh-TW.md b/README/README_zh-TW.md index cd377a40..b5065e7a 100644 --- a/README/README_zh-TW.md +++ b/README/README_zh-TW.md @@ -18,7 +18,7 @@ - **一套 API，七個平台。** `wrapper/platform_wrapper.py` 在匯入時挑選後端；同一份腳本在 Windows、macOS、X11 與 Wayland 上都不需要改寫。 -- **不寫 Python 也能腳本化。** 851 個 `AC_*` 指令涵蓋全部功能，因此一個 JSON 檔能做到函式庫 +- **不寫 Python 也能腳本化。** 852 個 `AC_*` 指令涵蓋全部功能，因此一個 JSON 檔能做到函式庫 能做的任何事——包含迴圈、分支、try/catch、巨集與變數。 - **預設無頭執行。** `import je_auto_control` 絕不會載入 Qt。GUI 是選用套件，包在同一個無頭核心之外。 - **四種定位方式。** 樣板比對、OCR、無障礙樹、視覺語言模型——可透過錨點定位器與自癒後備串接組合。 @@ -218,7 +218,7 @@ python -c "import je_auto_control; je_auto_control.start_autocontrol_gui()" | 自然語言規劃 | `plan_actions`、`run_from_description` | `AC_llm_plan` | LLM Planner | | Computer-use agent | `AgentLoop`、`run_agent` | `AC_run_agent` | Computer Use | | 錄製與重播 | `record`、`stop_record` | `AC_record`、`AC_stop_record` | Record | -| JSON 腳本 | `execute_action`、`execute_files` | 全部 851 個指令 | Script、Script Builder | +| JSON 腳本 | `execute_action`、`execute_files` | 全部 852 個指令 | Script、Script Builder | | 變數與流程控制 | `execute_action_with_vars` | `AC_set_var`、`AC_loop`、`AC_for_each`、`AC_try`、`AC_retry` | Variables | | 資料驅動執行 | — | `AC_for_each_row`（CSV／JSON／SQLite／Excel） | Data Sources | | 斷言 | `assert_text`、`assert_image` | `AC_assert_text` 等 21 個 | Assertions | @@ -267,7 +267,7 @@ je_auto_control version | 介面 | 啟動方式 | 說明 | |---|---|---| -| **MCP 伺服器** | `je_auto_control_mcp`（stdio）或 `AC_start_mcp_http_server` | 754 個工具，供 Claude Desktop／Claude Code／自訂 tool loop 使用：完整的 `ac_*` 工具介面，以及常用 GUI 操作的短名稱 alias。除了以 `initialize` 握手的各版協定，也支援無狀態的 MCP 2026-07-28。Bearer 驗證、TLS、稽核記錄、限流、外掛熱重載、CI 假後端。 | +| **MCP 伺服器** | `je_auto_control_mcp`（stdio）或 `AC_start_mcp_http_server` | 757 個工具，供 Claude Desktop／Claude Code／自訂 tool loop 使用：完整的 `ac_*` 工具介面，以及常用 GUI 操作的短名稱 alias。除了以 `initialize` 握手的各版協定，也支援無狀態的 MCP 2026-07-28。Bearer 驗證、TLS、稽核記錄、限流、外掛熱重載、CI 假後端。 | | **REST API** | `je_auto_control start-rest` | Bearer token、逐 IP 限流與鎖定、SQLite 稽核 hook、`/metrics`、`/openapi.json`、`/docs` Swagger UI、`/dashboard`。 | | **TCP socket 伺服器** | `je_auto_control start-server` | 以換行分隔的 JSON 動作清單。預設綁 `127.0.0.1`。 | | **pytest 外掛** | 安裝後自動生效 | 提供 fixture 與供 pytest-bdd／behave 使用的 Gherkin step library。 | diff --git a/architecture_explore.md b/architecture_explore.md index fb2ac48f..5f001852 100644 --- a/architecture_explore.md +++ b/architecture_explore.md @@ -6,7 +6,7 @@ > 擷取每個模組的 docstring 與頂層公開名稱；統計數字取自實際檔案，非估算。 > 指令數與公開 API 數以 `executor.known_commands()` 與 `je_auto_control.__all__` 在工作樹上實測取得。 > -> **掃描時間**：2026-10-09　**版本**：`pyproject.toml` version `0.0.230`　**分支**：`feat/progress-sweep-3` +> **掃描時間**：2026-10-09　**版本**：`pyproject.toml` version `0.0.230`　**分支**：`feat/progress-sweep-4` --- @@ -22,10 +22,10 @@ iOS（WebDriverAgent）。核心能力是滑鼠／鍵盤控制、影像辨識、 | Python 模組總數（含周邊子專案） | 1,175 | | 程式碼總行數 | 187,645 | | `je_auto_control/utils/` 子套件數 | 311 | -| `AC_*` 動作指令數（`known_commands()` 實測） | 851 | +| `AC_*` 動作指令數（`known_commands()` 實測） | 852 | | 套件門面 `__all__` 公開名稱數 | 1,244 | | GUI 分頁數（`main_widget` 註冊） | 48 | -| MCP 工具數（`build_default_tool_registry()` 實測） | 754 | +| MCP 工具數（`build_default_tool_registry()` 實測） | 757 | | `test_*.py` 測試檔／測試函式 | 478 / 4,654 | | 範例腳本 | 27 | @@ -92,7 +92,7 @@ USB/IP 協定、Prometheus 指標），以維持這條輕相依基線。 | 模式 | 落點 | 說明 | | --- | --- | --- | | **Strategy** | `wrapper/platform_wrapper.py` | 依 `sys.platform` 只匯入當前 OS 的 `keyboard`／`mouse`／`screen`／`recorder` 實作；Linux 再細分 Wayland／X11，Wayland 後端不可用時自動退回 XWayland 並記警告。新增平台不需要改 wrapper。 | -| **Facade** | `je_auto_control/__init__.py` | 把 1,423 個公開名稱集中再匯出，使用者只 `import je_auto_control`。`api/core.py` 另提供一個小而穩定的版本化門面（7 個名稱）給新整合使用。 | +| **Facade** | `je_auto_control/__init__.py` | 把 1,426 個公開名稱集中再匯出，使用者只 `import je_auto_control`。`api/core.py` 另提供一個小而穩定的版本化門面（7 個名稱）給新整合使用。 | | **Command** | `utils/executor/action_executor.py` | `Executor.event_dict` 是字串 → callable 的分派表；JSON 動作檔即指令序列，因此可錄製、序列化、重播、簽章。 | | **Observer** | `utils/callback/`、`utils/observer/`、`utils/triggers/` | 動作完成後觸發回呼；畫面出現／消失／變化與外部事件（webhook／IMAP／檔案）驅動腳本。 | | **Template Method** | `utils/generate_report/` | HTML／JSON／XML 三個產生器共用「收集紀錄 → 格式化 → 寫檔」骨架，各自實作渲染。 | @@ -154,7 +154,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | 模組 | 行數 | 職責 | | --- | ---: | --- | -| `je_auto_control/__init__.py` | 2,156 | **套件門面**。集中匯入並再匯出 1,423 個公開名稱，以功能區塊註解分段（callback／exception／executor／a11y／vision／clipboard…）。 | +| `je_auto_control/__init__.py` | 2,156 | **套件門面**。集中匯入並再匯出 1,426 個公開名稱，以功能區塊註解分段（callback／exception／executor／a11y／vision／clipboard…）。 | | `je_auto_control/__main__.py` | 98 | 舊版 argparse 進入點：`-e` 執行單檔、`-d` 執行整個目錄、`--execute_str` 執行 JSON 字串、`-c` 建立專案。 | | `je_auto_control/cli.py` | 422 | **主 CLI**（`je_auto_control` console script）。子命令：`run`（含 `--var`／`--dry-run`）、`validate`／`lint`、`list-commands`、`fmt`、`record`、`codegen`、`failure-bundle`、`list-jobs`、`start-server`、`start-rest`、`version`。所有子命令延遲匯入，確保不碰 Qt。 | | `je_auto_control/api/__init__.py` | 22 | 版本化整合進入點。 | @@ -283,7 +283,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `utils/dag/` | 564 | 跨主機 DAG 編排器（圖模型 + runner） | | `utils/decision_table/` | 112 | DMN 風格決策表：規則 + 命中策略，把分支外部化 | | `utils/deterministic/` | 116 | 決定性執行控制：固定亂數種子 + 凍結時鐘 | -| `utils/executor/` | 10,368 | **核心**。`Executor` 指令分派表（851 個 `AC_*`）、參數插值、乾跑、逐步 callback；`flow_control` 提供 34 個區塊指令（迴圈／分支／try／巨集／變數） | +| `utils/executor/` | 10,368 | **核心**。`Executor` 指令分派表（852 個 `AC_*`）、參數插值、乾跑、逐步 callback；`flow_control` 提供 34 個區塊指令（迴圈／分支／try／巨集／變數） | | `utils/action_journal/` | 1,001 | 選用的動作日誌：執行邊界記錄 typed 輸入／結果與 parent id，先遮蔽再寫入 JSONL；`codegen.journal_import` 由它產生候選腳本 | | `utils/flow_debugger/` | 166 | action list 的單步除錯器與追蹤器 | | `utils/input_macro/` | 469 | 定時輸入事件：錄製結果的整形（`timeline`／`InputRecorder`，Windows 與 macOS 共用）、重播與宣告式輸入序列 DSL | @@ -508,7 +508,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `utils/cua_action/` | 204 | 標準化 computer-use 動作結構（Anthropic／OpenAI → `AC_*`） | | `utils/llm/` | 387 | 自然語言 → action list 規劃器 + Anthropic／null 後端 | | `utils/mcp_registry/` | 97 | MCP registry `server.json` 資訊清單產生（可被發現） | -| `utils/mcp_server/` | 21,113 | **無頭 MCP 伺服器**（16K LOC，預設註冊 754 個工具＝735 個 `ac_*` + 19 個別名）：stdio + HTTP 傳輸、工具工廠與處理器、資源、prompt、稽核、限流、外掛熱重載 | +| `utils/mcp_server/` | 21,113 | **無頭 MCP 伺服器**（16K LOC，預設註冊 757 個工具＝738 個 `ac_*` + 19 個別名）：stdio + HTTP 傳輸、工具工廠與處理器、資源、prompt、稽核、限流、外掛熱重載 | | `utils/tool_use_schema/` | 195 | 把 `AC_*` 指令匯出成 Claude／OpenAI 的 tool-use schema | | `utils/trajectory_eval/` | 132 | agent 軌跡評估：依評分規準為一次執行打分 | | `utils/vision/` | 580 | VLM 元素定位器（依描述找元素）+ Anthropic／OpenAI／null 後端 | @@ -532,6 +532,9 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 > `utils/remote_desktop/input_send_queue.py`（viewer 輸入的有界送出佇列）、`utils/codegen/candidate_diff.py`（候選腳本與現有內容的 diff）、 > `utils/mcp_server/{_http_responses,_http_origin}.py`（由 `http_transport.py` 拆出）、`utils/action_signing/_private_file.py`（私鑰檔只給目前使用者）、 > `wrapper/{capability_probes,capability_states}.py`（Windows／macOS 的唯讀能力探測）、`gui/remote_desktop/connection_recent.py`（Quick Connect 的 Recent 清單）。 +> **2026-10-09 第四輪新增的模組**：`utils/thread_bound.py`（只在設定它的執行緒上有效的 context 變數，RBAC 身分、裝置綁定、自愈標記與 stop token 共用）、 +> `utils/accessibility/backends/windows_automation.py`（每個執行緒的 COM apartment 與 UIAutomation 物件）、 +> `gui/_weak_call.py`（弱參照持有的工作回呼）、`gui/_qt_typed.py`（對真實 Qt 型別檢查用的小工具）。 ### 5.4.11 伺服器、網路協定與外部整合 @@ -708,7 +711,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | 檔案 | 行數 | 職責 | | --- | ---: | --- | -| `action_executor.py` | 8,730 | `Executor` 類別與 `event_dict` 分派表（851 個指令），另含數百個把 utils 能力接成指令的 adapter 函式；全域單例 `executor` 與 `add_command_to_executor()` 擴充點。 | +| `action_executor.py` | 8,730 | `Executor` 類別與 `event_dict` 分派表（852 個指令），另含數百個把 utils 能力接成指令的 adapter 函式；全域單例 `executor` 與 `add_command_to_executor()` 擴充點。 | | `flow_control.py` | 693 | 真正的流程控制：`AC_loop`／`AC_for_each`／`AC_while_*`／`AC_if_*`／`AC_try`／`AC_retry`／`AC_parallel`／`AC_define_macro`／`AC_call_macro`／變數指令（`AC_set_var`／`AC_get_var`／`AC_inc_var`）。`LoopBreak`／`LoopContinue` 以例外實作。34 個區塊指令的分派表 `BLOCK_COMMANDS` 也在這裡，含下一列匯入的資料來源指令。 | | `flow_data_commands.py` | 272 | `AC_*_to_var` 資料來源與轉換指令：shell、時鐘、亂數、PDF、TOTP、SQL、檔案、HTTP、OCR，加上 `AC_assert_var`／`AC_assert_db`／`AC_assert_duration`／`AC_transform_var`。都不執行巢狀 action list，所以沒有迴圈／分支語意。 | | `action_schema.py` | 223 | action list 的結構驗證：形狀、參數型別、未知指令拒絕。單一走訪同時支援兩種消費方式：`validate_actions()` 遇到第一個問題就拋、`unknown_command_names()` 收齊全部不認得的名字（REST `/execute` 用它回 400）。 | @@ -716,7 +719,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `mouse_aliases.py` | 39 | 單鍵點擊別名（`AC_click_left` 等），executor 與 callback executor 共用。 | | `flags.py` | 19 | 旗標讀取 `as_bool`：依拼法（`"true"`／`"yes"`／`"on"`／`"1"`）而非 Python 真值判斷，所以 JSON、CLI、MCP 傳來的 `"false"` 是關；所有 adapter 與流程指令共用。 | -#### `utils/mcp_server/`（21,113 行，754 個工具）— 最大子系統 +#### `utils/mcp_server/`（21,113 行，757 個工具）— 最大子系統 | 檔案 | 行數 | 職責 | | --- | ---: | --- | diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md index 5e7bf199..9647fa64 100644 --- a/docs/updates/2026-10.md +++ b/docs/updates/2026-10.md @@ -416,3 +416,21 @@ Jeffrey_RPA kept its own `KEY_ALIASES` (40 friendly names) and `_EXTRA_KEY_CODES - **What**: `test/verify/macos_verify.py` now asserts `click-state-readback`, `minimised-window-by-id` and `grab-logical-points`. A macos-14 arm64 runner measured all three as working on 2026-10-09 (3 of 3 and 1 of 1 reused tests passed with none skipped; the point-based grab returned the region asked for), so they moved from `REPORTED` to `PROBES`. The 3.10 headless segfault fixed in U-20261009-25 did not recur in the five `pytest-headless` runs since on ubuntu-22.04 and macos-14 with Python 3.10; its `Progress.md` entry is removed. - **Tests**: `test_macos_verify_reporting.py` (20). - **Open items**: the measurement is one runner at display scale 1; a Retina display is still unmeasured (`Progress.md` › macOS). + +## U-20261009-34 · 2026-10-09 · A stop wakes the polling waits; MCP can stop and list runs; a USB ACL file that fails its check denies everything; per-thread context · #done #executor #mcp #security #rbac + +- **What**: `run_control.pause` is the shared stop-aware sleep and 30-odd polling waits use it (`AC_wait_window`, `AC_wait_text`, the twelve smart waits, `AC_expect_poll`, `AC_assert_eventually`, `AC_wait_actionable`, the timed holds, which release what they hold). ChatOps answers "stopped" for a stopped command. MCP gains `ac_stop_execution` and `ac_list_executions`. While `UsbAcl.integrity_ok` is false `decide()` denies every device whatever the default policy. `utils/thread_bound.py` keeps the RBAC caller, the bound device, the heal stamp and the stop token on the thread that set them. `AC_llm_run` takes no `owner` by design: an action runs on the thread that already carries the caller, and a JSON `owner` would let a list's author run as someone else; three tests pin it. +- **Tests**: `test_stop_aware_waits.py` (22), `test_chatops_stopped_run.py` (3), `test_mcp_stop_execution.py` (6), `test_usb_acl_fail_closed.py` (7), `test_thread_bound_context.py` (29). +- **Open items**: no free-threaded interpreter here; a call already inside a backend is not interrupted. `Progress.md` › GUI and › 雜項. + +## U-20261009-35 · 2026-10-09 · Config sync: oversize blobs refused before upload, unused blobs collected, a deletion held for 30 days, the mirror sees same-tick edits · #done #sync + +- **What**: `HttpAssetTransport` reads the server's `max_blob_bytes` and refuses a larger file locally with a fixed message. `config_sync_collect_blobs` (`AC_config_sync_collect_blobs`, `ac_config_sync_collect_blobs`) deletes blobs nothing references that are older than a day; the blob quota holds a file lock so it is shared across processes. An acknowledged tombstone stays for `TOMBSTONE_HOLD_S` (30 days), so a machine the bucket never listed meets the deletion as a conflict instead of resurrecting the entry. `FolderSyncEngine` compares modification time and size, and compares by content for files recorded within two seconds of their last write. The Config Sync tab has a section picker, an `assets_server` switch and a locator file field. Example 29 uses the facade. +- **Tests**: `test_folder_sync_same_tick.py` (8), `test_config_sync_blob_housekeeping.py` (34), `test_config_sync_unlisted_device.py` (10), tab tests (+7). +- **Open items**: the residual tombstone and stale-lock cases. `Progress.md` › 安全與伺服器. + +## U-20261009-36 · 2026-10-09 · GUI: task callbacks held weakly, dispose() on every panel and tab with background work, per-thread UI Automation, 52 fewer typing exemptions · #done #gui #accessibility #typing + +- **What**: `gui/_weak_call.py`: the worker relay, the task controller and the panel helpers no longer keep a tab alive (the new test aborted the interpreter on the old code); a structural test refuses a lambda or closure connected to a task signal. `dispose()` on the six remote-desktop classes and 17 tabs. Pages of closed tabs are parked for a restyle and `real_window()` finds the main window from a parked page. The Windows UIA backend gives each calling thread its own automation object (`windows_automation.py`), `windows_backend.py` drops to 748 lines and leaves the exemption list, and the accessibility tabs run off the GUI thread. `typing_extras_exempt.txt` shrinks from 70 modules to 18. +- **Tests**: `test_gui_weak_callbacks.py` (14), `test_webrtc_lan_advertise_stop.py` (6), `test_gui_panel_dispose.py` (14), `test_accessibility_windows_threads.py` (14), `test_gui_accessibility_offthread.py` (8), `test_gui_qt_typed.py` (4). +- **Open items**: the COM behaviour is tested against fakes only (`comtypes` is not installed); releasing the Remote Desktop tab now ends a WebRTC host or session — a decision for the maintainer. `Progress.md` › GUI. diff --git a/docs/updates/README.md b/docs/updates/README.md index bdf51f56..c81d63e4 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,9 @@ In the same commit: delete the item from `Progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261009-36 | 2026-10-09 | GUI: task callbacks held weakly, dispose() on every panel and tab with background work, per-thread UI Automation, 52 fewer typing exemptions | #done #gui #accessibility #typing | [2026-10](2026-10.md) | +| U-20261009-35 | 2026-10-09 | Config sync: oversize blobs refused before upload, unused blobs collected, a deletion held for 30 days, the mirror sees same-tick edits | #done #sync | [2026-10](2026-10.md) | +| U-20261009-34 | 2026-10-09 | A stop wakes the polling waits; MCP can stop and list runs; a USB ACL file that fails its check denies everything; per-thread context | #done #executor #mcp #security #rbac | [2026-10](2026-10.md) | | U-20261009-33 | 2026-10-09 | The three newer macOS probes are asserted; the Python 3.10 segfault item is closed | #done #macos #ci | [2026-10](2026-10.md) | | U-20261009-32 | 2026-10-09 | READMEs list every environment variable, Sphinx builds without warnings, typing is also checked against the real Qt types | #done #docs #ci #typing | [2026-10](2026-10.md) | | U-20261009-31 | 2026-10-09 | Journal artifacts and exact-match secret masking, a Robot structure check, a VLM strategy and verified heals, real capability probes | #done #journal #self-healing #capabilities | [2026-10](2026-10.md) | @@ -382,7 +385,7 @@ In the same commit: delete the item from `Progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| -| [2026-10.md](2026-10.md) | 2026-10 | 46 | +| [2026-10.md](2026-10.md) | 2026-10 | 49 | | [2026-09-e.md](2026-09-e.md) | 2026-09 | 7 | | [2026-09-d.md](2026-09-d.md) | 2026-09 | 55 | | [2026-09-c.md](2026-09-c.md) | 2026-09 | 38 | From 63825c3a83fcffe461612a9a548db9e8ca684b02 Mon Sep 17 00:00:00 2001 From: JeffreyChen Date: Fri, 9 Oct 2026 08:16:01 +0800 Subject: [PATCH 17/17] Wait for the worker to register its release before asserting it ran, and never leave it blocked --- test/unit_test/headless/test_gui_tab_dispose.py | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/test/unit_test/headless/test_gui_tab_dispose.py b/test/unit_test/headless/test_gui_tab_dispose.py index 6eda4749..f21f16ee 100644 --- a/test/unit_test/headless/test_gui_tab_dispose.py +++ b/test/unit_test/headless/test_gui_tab_dispose.py @@ -209,9 +209,13 @@ def work(token): controller = task_controller() before = controller.active_count() controller.submit(work, owner=owner) - _dispose.release_resources(owner) - assert released == [True] - release.set() + try: + _dispose.release_resources(owner) + # The worker may not have reached on_cancel yet; a callback registered + # after the cancel runs at once, on the worker thread. + assert pump_until(lambda: released == [True]) + finally: + release.set() # never leave the worker waiting for the next test assert pump_until(lambda: controller.active_count() == before)