diff --git a/CHANGELOG.md b/CHANGELOG.md index fb6a6cb3e..6eb6d0e2e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,41 @@ it shipped into a `## [x.y.z] - date` section of their own; the tag's ### Added +- **Tesseract setup helpers.** `find_tesseract_cmd()` (`$TESSERACT_CMD`, + `PATH`, then the installers' default folders), `set_tessdata_dir(path)` + (sets `TESSDATA_PREFIX`; `None` clears it), `ocr_languages()` (sorted + codes; `[]` when the engine has no language data, `None` when it cannot be + asked) and `ocr_status()`, an `OCRStatus(ok, reason)` tuple whose reason is + `ready`, `missing_package`, `missing_engine`, `engine_unusable` or + `no_language_data`. `TesseractBackend` gained `cmd`, `version()` and + `languages()`. Commands `AC_ocr_status` / `AC_ocr_languages`, MCP tools + `ac_ocr_status` / `ac_ocr_languages`. +- **`click_mouse(..., clicks=1, interval=0.0)`.** `clicks=2` double-clicks + in one call: the clicks land on one point, `interval` seconds apart. + Defaults keep the single click, and a single click records the same + action as before. `AC_click_mouse`, the `ac_click_mouse` MCP tool and the + Script Builder take both parameters. On macOS the clicks still arrive as + separate single clicks (`Progress.md`). +- **Drag pacing: `tween_drag` / `drag_path` take `step_delay_s` and + `settle_s`** (seconds, default 0). `step_delay_s` rests after each move; + `settle_s` rests on the start before the press, after the press and before + the release, for apps that tell a drag from a click by the pointer's + motion. Both must be finite and non-negative (`ValueError` otherwise, + before anything is sent). `AC_tween_drag`, `AC_drag_path`, their MCP tools + and the Script Builder take both. +- **Windows key names.** The OEM keys (`oem_1` .. `oem_8`, `oem_102`, + `oem_plus`, `oem_comma`, `oem_minus`, `oem_period`, `oem_clear`), + `browser_home`, a lower-case `launch_app2` (`LAUNCH_APP2` stays) and + common aliases: `ctrl` / `lctrl` / `rctrl`, `alt` / `lalt` / `ralt`, + `enter`, `esc`, `win` / `super` / `cmd` / `meta`, `backspace` / `bksp`, + `del`, `ins`, `pgup` / `pageup`, `pgdn` / `pagedown`, `caps` / + `capslock`, `printscreen` / `prtsc` / `prtscr`, `scrolllock`, + `numpad0` .. `numpad9`, and `plus` / `comma` / `minus` / `period` for the + four OEM keys that are the same on every layout. Aliases are listed in + `platform_wrapper.keyboard_key_aliases` (empty on other platforms). +- **`keyboard_key_name(keycode)`** returns a code's canonical name and never + an alias, so a recorder's output does not change when an alias is added; + `None` when the table has no name for the code. - `cua_action.resolve_key_name` / `split_key_combo`, and `compile_postcondition(before=...)`. - `pii_text.luhn_valid` and `normalize_text(strip_format=...)`. @@ -54,6 +89,12 @@ it shipped into a `## [x.y.z] - date` section of their own; the tag's ### Changed +- **Windows `keyboard_keys_table` lists its aliases after the canonical + names**, `ctrl` included (it moved from beside `control` into + `keyboard_key_aliases`; same code). A reverse lookup that takes the + shortest name from the raw table now meets aliases such as `esc` or + `prtsc`; skip the names in `keyboard_key_aliases`, or call + `keyboard_key_name`, to keep getting the canonical ones. - **Golden-image capture (`take_golden` / `compare_to_golden`) reads its region in mouse coordinates**, like every other capture; region goldens taken on a scaled display need re-taking. @@ -295,6 +336,11 @@ it shipped into a `## [x.y.z] - date` section of their own; the tag's ### Fixed +- **An aborted `tween_drag` / `drag_path` lets go where the pointer stopped.** + When a step raised, the button was released at the end of the path, which + completed a drop the drag never reached; it is now released in a `finally` + at the last point the pointer reached, and a failing cleanup release is + logged instead of replacing the original error. - **Emergency stop on Linux / macOS**: the stop key wakes a sleeping or waiting main thread there too (SIGINT is sent to the main thread). - **Image analysis and packaging**: colour-vision simulation uses Machado diff --git a/Progress.md b/Progress.md index df4efa7e4..0718550af 100644 --- a/Progress.md +++ b/Progress.md @@ -175,6 +175,14 @@ pip install --dry-run --only-binary=:all: --platform win_arm64 --python-version --- +## macOS 上 `click_mouse(clicks=2)` 不是雙擊 + +`TODO` — `wrapper/auto_control_mouse.py` 的 `click_mouse` 在同一點連點 `clicks` 次。Windows 與 X11 依兩次點擊的時間差與位移判定雙擊，所以這樣就夠；macOS 的應用程式讀的是事件上的點擊次數欄位（`kCGMouseEventClickState`），`osx/mouse/osx_mouse.py` 的 `mouse_event` 從不設定它（一律是 1），所以 macOS 收到的是兩次單擊。文件（`docs/source/API/wrapper/mouse.rst`、`docs/source/{Eng,Zh}/doc/mouse/mouse_doc.rst`、`click_mouse` 的 docstring）照實寫了這個限制。 + +做法：讓 osx 後端的按下／放開帶點擊次數（第 n 次點擊設成 n），wrapper 在 macOS 分支把序號傳下去；要在 macOS 上實測（`quality.yml` 的 `macos-14` 可以跑 CI，但雙擊是否被應用程式認得要真機看），完成後把文件裡的限制拿掉。 + +--- + ## RBAC 還沒接到 REST API 與 MCP server `DECIDE` — 要不要把 `utils/rbac` 接上兩個伺服器，以及現有單一共用 token 怎麼過渡（維護者拍板） diff --git a/README.md b/README.md index d0b515788..e0737b299 100644 --- a/README.md +++ b/README.md @@ -22,7 +22,7 @@ from JSON files / CLI / servers, and a **GUI tab**. Nothing is GUI-only. - **One API, seven platforms.** `wrapper/platform_wrapper.py` picks the backend at import time; your script does not change between Windows, macOS, X11, and Wayland. -- **Scriptable without Python.** 774 `AC_*` commands cover the whole feature set, so a +- **Scriptable without Python.** 776 `AC_*` commands cover the whole feature set, so a JSON file can do anything the library can — including loops, branches, try/catch, macros, and variables. - **Headless by default.** `import je_auto_control` never loads Qt. The GUI is an @@ -73,7 +73,10 @@ sudo apt-get install cmake libssl-dev ``` OCR, VLM, and LLM backends (`pytesseract`, `easyocr`, `paddleocr`, `anthropic`, -`openai`) are loaded on demand — install whichever you actually use. +`openai`) are loaded on demand — install whichever you actually use. For Tesseract, +`find_tesseract_cmd()` locates the executable (`$TESSERACT_CMD`, then `PATH`, then the +installers' default folders), `set_tessdata_dir()` points it at a language-data folder, +and `ocr_status()` / `ocr_languages()` say what is missing before the first OCR call. **Log file:** the library logs to `~/.je_auto_control/logs/AutoControlGUI.log`, created on the first record (importing alone writes nothing) and shared by every @@ -142,11 +145,11 @@ desktop app; tab commands live in the window's **Actions** menu. | Capability | Python API | `AC_*` command | GUI tab | |---|---|---|---| -| Mouse | `click_mouse`, `set_mouse_position`, `mouse_scroll` | `AC_click_mouse` | Auto Click | -| Keyboard | `write`, `hotkey`, `type_keyboard` | `AC_write`, `AC_hotkey` | Auto Click | +| Mouse | `click_mouse` (`clicks=2` double-clicks), `set_mouse_position`, `mouse_scroll`, `tween_drag` / `drag_path` (paced by `step_delay_s`, `settle_s`) | `AC_click_mouse`, `AC_tween_drag`, `AC_drag_path` | Auto Click | +| Keyboard | `write`, `hotkey`, `type_keyboard`, `keyboard_key_name` (code → canonical name; Windows also takes aliases such as `ctrl`, `esc`, `enter`) | `AC_write`, `AC_hotkey` | Auto Click | | Screen & pixels | `screenshot`, `screen_size`, `get_pixel` | `AC_screenshot` | Screenshot | | Image matching | `locate_image_center`, `locate_and_click` | `AC_locate_and_click` | Image Detect | -| OCR text | `click_text`, `wait_for_text`, `read_text_in_region` | `AC_click_text`, `AC_wait_text` | OCR Reader | +| OCR text | `click_text`, `wait_for_text`, `read_text_in_region`, `ocr_status` | `AC_click_text`, `AC_wait_text`, `AC_ocr_status` | OCR Reader | | Accessibility tree | `find_accessibility_element`, `click_accessibility_element` | `AC_a11y_find`, `AC_a11y_click` | Accessibility | | Vision-model locator | `locate_by_description`, `click_by_description` | `AC_vlm_locate`, `AC_vlm_click` | VLM | | Anchor locator | — | `AC_anchor_click`, `AC_anchor_locate` | — | @@ -154,7 +157,7 @@ desktop app; tab commands live in the window's **Actions** menu. | Natural-language planner | `plan_actions`, `run_from_description` | `AC_llm_plan` | LLM Planner | | Computer-use agent | `AgentLoop`, `run_agent` | `AC_run_agent` | Computer Use | | Record & replay | `record`, `stop_record` | `AC_record`, `AC_stop_record` | Record | -| JSON scripting | `execute_action`, `execute_files` | all 774 commands | Script, Script Builder | +| JSON scripting | `execute_action`, `execute_files` | all 776 commands | Script, Script Builder | | Variables & flow control | `execute_action_with_vars` | `AC_set_var`, `AC_loop`, `AC_for_each`, `AC_try`, `AC_retry` | Variables | | Data-driven runs | — | `AC_for_each_row` (CSV / JSON / SQLite / Excel) | Data Sources | | Assertions | `assert_text`, `assert_image` | `AC_assert_text` + 20 more | Assertions | @@ -206,7 +209,7 @@ still goes on to the end), so a CI step fails with it. The legacy | Surface | Start it with | Notes | |---|---|---| -| **MCP server** | `je_auto_control_mcp` (stdio) or `AC_start_mcp_http_server` | 677 tools for Claude Desktop / Claude Code / custom tool loops. Bearer auth, TLS, audit log, rate limit, plugin hot-reload, CI fake backend. | +| **MCP server** | `je_auto_control_mcp` (stdio) or `AC_start_mcp_http_server` | 679 tools for Claude Desktop / Claude Code / custom tool loops. Bearer auth, TLS, audit log, rate limit, plugin hot-reload, CI fake backend. | | **REST API** | `je_auto_control start-rest` | Bearer token, per-IP rate limit + lockout, SQLite audit hook, `/metrics`, `/openapi.json`, `/docs` Swagger UI, `/dashboard`. | | **TCP socket server** | `je_auto_control start-server` | Newline-framed JSON action lists. Binds `127.0.0.1` by default. | | **pytest plugin** | installed automatically | Fixtures plus a Gherkin step library for pytest-bdd / behave. | diff --git a/README/README_zh-CN.md b/README/README_zh-CN.md index 879b4a6cf..4b41328fa 100644 --- a/README/README_zh-CN.md +++ b/README/README_zh-CN.md @@ -20,7 +20,7 @@ - **一套 API，七个平台。** `wrapper/platform_wrapper.py` 在导入时挑选后端；同一份脚本在 Windows、macOS、X11 与 Wayland 上都不需要改写。 -- **不写 Python 也能脚本化。** 774 个 `AC_*` 命令覆盖全部功能，因此一个 JSON 文件能做到库 +- **不写 Python 也能脚本化。** 776 个 `AC_*` 命令覆盖全部功能，因此一个 JSON 文件能做到库 能做的任何事——包含循环、分支、try/catch、宏与变量。 - **默认无头运行。** `import je_auto_control` 绝不会加载 Qt。GUI 是可选包，包在同一个无头内核之外。 - **四种定位方式。** 模板匹配、OCR、无障碍树、视觉语言模型——可通过锚点定位器与自愈回退串接组合。 @@ -63,7 +63,9 @@ sudo apt-get install cmake libssl-dev ``` OCR、VLM 与 LLM 后端（`pytesseract`、`easyocr`、`paddleocr`、`anthropic`、`openai`） -都是按需加载——只装你实际会用到的。 +都是按需加载——只装你实际会用到的。Tesseract 方面，`find_tesseract_cmd()` 找出可执行文件 +（依次是 `$TESSERACT_CMD`、`PATH`、安装程序的默认文件夹），`set_tessdata_dir()` 把它指向 +语言数据文件夹，`ocr_status()`／`ocr_languages()` 在第一次 OCR 之前就说出缺了什么。 **日志文件：** 库写到 `~/.je_auto_control/logs/AutoControlGUI.log`，第一条记录时 才创建（只 import 不会写任何文件），同一个账户的所有进程共用（追加写入、每行带进程 ID， @@ -130,11 +132,11 @@ python -c "import je_auto_control; je_auto_control.start_autocontrol_gui()" | 能力 | Python API | `AC_*` 命令 | GUI 标签页 | |---|---|---|---| -| 鼠标 | `click_mouse`、`set_mouse_position`、`mouse_scroll` | `AC_click_mouse` | Auto Click | -| 键盘 | `write`、`hotkey`、`type_keyboard` | `AC_write`、`AC_hotkey` | Auto Click | +| 鼠标 | `click_mouse`（`clicks=2` 即双击）、`set_mouse_position`、`mouse_scroll`、`tween_drag`／`drag_path`（以 `step_delay_s`、`settle_s` 控制节奏） | `AC_click_mouse`、`AC_tween_drag`、`AC_drag_path` | Auto Click | +| 键盘 | `write`、`hotkey`、`type_keyboard`、`keyboard_key_name`（键码 → 标准键名；Windows 也接受 `ctrl`、`esc`、`enter` 等别名） | `AC_write`、`AC_hotkey` | Auto Click | | 屏幕与像素 | `screenshot`、`screen_size`、`get_pixel` | `AC_screenshot` | Screenshot | | 图像匹配 | `locate_image_center`、`locate_and_click` | `AC_locate_and_click` | Image Detect | -| OCR 文字 | `click_text`、`wait_for_text`、`read_text_in_region` | `AC_click_text`、`AC_wait_text` | OCR Reader | +| OCR 文字 | `click_text`、`wait_for_text`、`read_text_in_region`、`ocr_status` | `AC_click_text`、`AC_wait_text`、`AC_ocr_status` | OCR Reader | | 无障碍树 | `find_accessibility_element`、`click_accessibility_element` | `AC_a11y_find`、`AC_a11y_click` | Accessibility | | 视觉模型定位 | `locate_by_description`、`click_by_description` | `AC_vlm_locate`、`AC_vlm_click` | VLM | | 锚点定位 | — | `AC_anchor_click`、`AC_anchor_locate` | — | @@ -142,7 +144,7 @@ python -c "import je_auto_control; je_auto_control.start_autocontrol_gui()" | 自然语言规划 | `plan_actions`、`run_from_description` | `AC_llm_plan` | LLM Planner | | Computer-use agent | `AgentLoop`、`run_agent` | `AC_run_agent` | Computer Use | | 录制与回放 | `record`、`stop_record` | `AC_record`、`AC_stop_record` | Record | -| JSON 脚本 | `execute_action`、`execute_files` | 全部 774 个命令 | Script、Script Builder | +| JSON 脚本 | `execute_action`、`execute_files` | 全部 776 个命令 | Script、Script Builder | | 变量与流程控制 | `execute_action_with_vars` | `AC_set_var`、`AC_loop`、`AC_for_each`、`AC_try`、`AC_retry` | Variables | | 数据驱动执行 | — | `AC_for_each_row`（CSV／JSON／SQLite／Excel） | Data Sources | | 断言 | `assert_text`、`assert_image` | `AC_assert_text` 等 21 个 | Assertions | @@ -191,7 +193,7 @@ je_auto_control version | 接口 | 启动方式 | 说明 | |---|---|---| -| **MCP 服务器** | `je_auto_control_mcp`（stdio）或 `AC_start_mcp_http_server` | 677 个工具，供 Claude Desktop／Claude Code／自定义 tool loop 使用。Bearer 认证、TLS、审计日志、限流、插件热重载、CI 假后端。 | +| **MCP 服务器** | `je_auto_control_mcp`（stdio）或 `AC_start_mcp_http_server` | 679 个工具，供 Claude Desktop／Claude Code／自定义 tool loop 使用。Bearer 认证、TLS、审计日志、限流、插件热重载、CI 假后端。 | | **REST API** | `je_auto_control start-rest` | Bearer token、按 IP 限流与锁定、SQLite 审计 hook、`/metrics`、`/openapi.json`、`/docs` Swagger UI、`/dashboard`。 | | **TCP socket 服务器** | `je_auto_control start-server` | 以换行分隔的 JSON 动作列表。默认绑定 `127.0.0.1`。 | | **pytest 插件** | 安装后自动生效 | 提供 fixture 与供 pytest-bdd／behave 使用的 Gherkin step library。 | diff --git a/README/README_zh-TW.md b/README/README_zh-TW.md index 4bd926718..dc14ef8b2 100644 --- a/README/README_zh-TW.md +++ b/README/README_zh-TW.md @@ -20,7 +20,7 @@ - **一套 API，七個平台。** `wrapper/platform_wrapper.py` 在匯入時挑選後端；同一份腳本在 Windows、macOS、X11 與 Wayland 上都不需要改寫。 -- **不寫 Python 也能腳本化。** 774 個 `AC_*` 指令涵蓋全部功能，因此一個 JSON 檔能做到函式庫 +- **不寫 Python 也能腳本化。** 776 個 `AC_*` 指令涵蓋全部功能，因此一個 JSON 檔能做到函式庫 能做的任何事——包含迴圈、分支、try/catch、巨集與變數。 - **預設無頭執行。** `import je_auto_control` 絕不會載入 Qt。GUI 是選用套件，包在同一個無頭核心之外。 - **四種定位方式。** 樣板比對、OCR、無障礙樹、視覺語言模型——可透過錨點定位器與自癒後備串接組合。 @@ -63,7 +63,9 @@ sudo apt-get install cmake libssl-dev ``` OCR、VLM 與 LLM 後端（`pytesseract`、`easyocr`、`paddleocr`、`anthropic`、`openai`） -都是按需載入——只裝你實際會用到的。 +都是按需載入——只裝你實際會用到的。Tesseract 的部分，`find_tesseract_cmd()` 找出執行檔 +（依序是 `$TESSERACT_CMD`、`PATH`、安裝程式的預設資料夾），`set_tessdata_dir()` 把它指到 +語言資料夾，`ocr_status()`／`ocr_languages()` 在第一次 OCR 之前就說出缺了什麼。 **記錄檔：** 函式庫寫到 `~/.je_auto_control/logs/AutoControlGUI.log`，第一筆記錄時 才建立（只 import 不會寫任何檔），同一個帳號的所有行程共用（附加寫入、每行帶行程 ID， @@ -130,11 +132,11 @@ python -c "import je_auto_control; je_auto_control.start_autocontrol_gui()" | 能力 | Python API | `AC_*` 指令 | GUI 分頁 | |---|---|---|---| -| 滑鼠 | `click_mouse`、`set_mouse_position`、`mouse_scroll` | `AC_click_mouse` | Auto Click | -| 鍵盤 | `write`、`hotkey`、`type_keyboard` | `AC_write`、`AC_hotkey` | Auto Click | +| 滑鼠 | `click_mouse`（`clicks=2` 就是雙擊）、`set_mouse_position`、`mouse_scroll`、`tween_drag`／`drag_path`（以 `step_delay_s`、`settle_s` 控制節奏） | `AC_click_mouse`、`AC_tween_drag`、`AC_drag_path` | Auto Click | +| 鍵盤 | `write`、`hotkey`、`type_keyboard`、`keyboard_key_name`（鍵碼 → 標準鍵名；Windows 也收 `ctrl`、`esc`、`enter` 等別名） | `AC_write`、`AC_hotkey` | Auto Click | | 螢幕與像素 | `screenshot`、`screen_size`、`get_pixel` | `AC_screenshot` | Screenshot | | 影像比對 | `locate_image_center`、`locate_and_click` | `AC_locate_and_click` | Image Detect | -| OCR 文字 | `click_text`、`wait_for_text`、`read_text_in_region` | `AC_click_text`、`AC_wait_text` | OCR Reader | +| OCR 文字 | `click_text`、`wait_for_text`、`read_text_in_region`、`ocr_status` | `AC_click_text`、`AC_wait_text`、`AC_ocr_status` | OCR Reader | | 無障礙樹 | `find_accessibility_element`、`click_accessibility_element` | `AC_a11y_find`、`AC_a11y_click` | Accessibility | | 視覺模型定位 | `locate_by_description`、`click_by_description` | `AC_vlm_locate`、`AC_vlm_click` | VLM | | 錨點定位 | — | `AC_anchor_click`、`AC_anchor_locate` | — | @@ -142,7 +144,7 @@ python -c "import je_auto_control; je_auto_control.start_autocontrol_gui()" | 自然語言規劃 | `plan_actions`、`run_from_description` | `AC_llm_plan` | LLM Planner | | Computer-use agent | `AgentLoop`、`run_agent` | `AC_run_agent` | Computer Use | | 錄製與重播 | `record`、`stop_record` | `AC_record`、`AC_stop_record` | Record | -| JSON 腳本 | `execute_action`、`execute_files` | 全部 774 個指令 | Script、Script Builder | +| JSON 腳本 | `execute_action`、`execute_files` | 全部 776 個指令 | Script、Script Builder | | 變數與流程控制 | `execute_action_with_vars` | `AC_set_var`、`AC_loop`、`AC_for_each`、`AC_try`、`AC_retry` | Variables | | 資料驅動執行 | — | `AC_for_each_row`（CSV／JSON／SQLite／Excel） | Data Sources | | 斷言 | `assert_text`、`assert_image` | `AC_assert_text` 等 21 個 | Assertions | @@ -191,7 +193,7 @@ je_auto_control version | 介面 | 啟動方式 | 說明 | |---|---|---| -| **MCP 伺服器** | `je_auto_control_mcp`（stdio）或 `AC_start_mcp_http_server` | 677 個工具，供 Claude Desktop／Claude Code／自訂 tool loop 使用。Bearer 驗證、TLS、稽核記錄、限流、外掛熱重載、CI 假後端。 | +| **MCP 伺服器** | `je_auto_control_mcp`（stdio）或 `AC_start_mcp_http_server` | 679 個工具，供 Claude Desktop／Claude Code／自訂 tool loop 使用。Bearer 驗證、TLS、稽核記錄、限流、外掛熱重載、CI 假後端。 | | **REST API** | `je_auto_control start-rest` | Bearer token、逐 IP 限流與鎖定、SQLite 稽核 hook、`/metrics`、`/openapi.json`、`/docs` Swagger UI、`/dashboard`。 | | **TCP socket 伺服器** | `je_auto_control start-server` | 以換行分隔的 JSON 動作清單。預設綁 `127.0.0.1`。 | | **pytest 外掛** | 安裝後自動生效 | 提供 fixture 與供 pytest-bdd／behave 使用的 Gherkin step library。 | diff --git a/architecture.md b/architecture.md index b97e7e460..cfa9092cb 100644 --- a/architecture.md +++ b/architecture.md @@ -126,16 +126,16 @@ wrapper/auto_control_record.record → OS listener (e.g. windows/record/win32_in | Consumer | How it uses this repo | What it relies on | | --- | --- | --- | -| Jeffrey_RPA | Editable install of **this working tree**: uncommitted changes here reach it immediately. Single facade `JeffreyRPA/_gui_control.py`. | Top-level names (e.g. `click_mouse`, `hotkey`, `write`, `screen_size`, `get_pixel`, `post_click_to_window`) and internal paths `je_auto_control.wrapper.auto_control_window`, `je_auto_control.wrapper.auto_control_keyboard.WRITE_CONTROL_KEYS`, `je_auto_control.utils.monitor_layout` (`logical_virtual_rect`, `enumerate_monitors`), and `wrapper.platform_wrapper.keyboard_keys_table` / `mouse_keys_table` — it validates every key name a user types against the keyboard table and reverse-looks-up recorded virtual keys through it, so a name removed there becomes a rejected hotkey over in that repo. | +| Jeffrey_RPA | Editable install of **this working tree**: uncommitted changes here reach it immediately. Single facade `JeffreyRPA/_gui_control.py`. | Top-level names (e.g. `click_mouse`, `hotkey`, `write`, `screen_size`, `get_pixel`, `post_click_to_window`) and internal paths `je_auto_control.wrapper.auto_control_window`, `je_auto_control.wrapper.auto_control_keyboard.WRITE_CONTROL_KEYS`, `je_auto_control.utils.monitor_layout` (`logical_virtual_rect`, `enumerate_monitors`), and `wrapper.platform_wrapper.keyboard_keys_table` / `mouse_keys_table` — it validates every key name a user types against the keyboard table and reverse-looks-up recorded virtual keys through it, so a name removed there becomes a rejected hotkey over in that repo. Added so it can drop its own copies: the Tesseract setup helpers `find_tesseract_cmd`, `set_tessdata_dir`, `ocr_languages` (`None` when the engine cannot be asked, `[]` when it has no language data) and `ocr_status`; `click_mouse(clicks=, interval=)` for a double-click in one call; `tween_drag` / `drag_path` pacing (`step_delay_s`, `settle_s`); `keyboard_key_name` and `wrapper.platform_wrapper.keyboard_key_aliases` for reverse lookups that never answer with an alias, plus the Windows table's OEM keys, `browser_home`, lower-case `launch_app2` and its aliases (`ctrl`, `esc`, `enter`, `win`, `pgup`, `numpad0`, ...), which match the names and codes of its own `KEY_ALIASES` / `_EXTRA_KEY_CODES`. | | PyBreeze | Subprocess `python -m je_auto_control --execute_str ` / `--execute_file `; on Windows the JSON string arrives double-encoded. | Legacy CLI flags; also embeds `je_auto_control.gui.main_widget.AutoControlGUIWidget` and calls `record` / `stop_record` in-process. | | TestPioneer | Optional extra `gui = ["je_auto_control"]`; `parallel_run` starts `python -m je_auto_control --execute_file `. | `execute_action`, `execute_files`, `RecordingThread`; the `--execute_file` flag. | **Guarded by** `test/unit_test/headless/test_cross_project_contracts.py`: every legacy CLI flag (short and long, run as a real child process, including PyBreeze's double-encoded `--execute_str`), the facade names in the three rows above -(Jeffrey_RPA's list is every `ac.` in `_gui_control.py`), the `auto_control_window` functions Jeffrey_RPA calls, -its three internal imports, the two key tables (shape everywhere, Windows key names on Windows), and -`AutoControlGUIWidget`. The test only knows what this table knows: when a consumer starts relying on something -new, add it to both. +(Jeffrey_RPA's list is every `ac.` in `_gui_control.py`, plus the names and keyword parameters added so it can +drop its own copies), the `auto_control_window` functions Jeffrey_RPA calls, its three internal imports, the two key +tables (shape everywhere, Windows key names on Windows), and `AutoControlGUIWidget`. The test only knows what this +table knows: when a consumer starts relying on something new, add it to both. **Outbound (optional):** `utils/webrunner_bridge/bridge.py` imports WebRunner's *internal* `je_web_runner.utils.executor.action_executor.executor` lazily, for `AC_web_*` commands and `gui/webrunner_tab.py`. diff --git a/architecture_explore.md b/architecture_explore.md index 268c83f8a..6d3db41ce 100644 --- a/architecture_explore.md +++ b/architecture_explore.md @@ -6,7 +6,7 @@ > 擷取每個模組的 docstring 與頂層公開名稱；統計數字取自實際檔案，非估算。 > 指令數與公開 API 數以 `executor.known_commands()` 與 `je_auto_control.__all__` 在工作樹上實測取得。 > -> **掃描時間**：2026-09-22　**版本**：`pyproject.toml` version `0.0.221`　**分支**：`feat/coverage-to-80` +> **掃描時間**：2026-10-01　**版本**：`pyproject.toml` version `0.0.223`　**分支**：`feat/jeffrey-rpa-gui-apis` --- @@ -19,14 +19,14 @@ iOS（WebDriverAgent）。核心能力是滑鼠／鍵盤控制、影像辨識、 | 指標 | 數值 | | --- | ---: | -| Python 模組總數（含周邊子專案） | 1,049 | -| 程式碼總行數 | 148,925 | +| Python 模組總數（含周邊子專案） | 1,050 | +| 程式碼總行數 | 149,489 | | `je_auto_control/utils/` 子套件數 | 310 | -| `AC_*` 動作指令數（`known_commands()` 實測） | 774 | -| 套件門面 `__all__` 公開名稱數 | 1,241 | +| `AC_*` 動作指令數（`known_commands()` 實測） | 776 | +| 套件門面 `__all__` 公開名稱數 | 1,248 | | GUI 分頁數（`main_widget` 註冊） | 48 | -| MCP 工具數（`build_default_tool_registry()` 實測） | 677 | -| `test_*.py` 測試檔／測試函式 | 478 / 4,654 | +| MCP 工具數（`build_default_tool_registry()` 實測） | 679 | +| `test_*.py` 測試檔／測試函式 | 628 / 6,673 | | 範例腳本 | 27 | **技術基線**：Python ≥ 3.10、MIT 授權、必要相依只有 `je_open_cv`／`opencv-python`／`pillow`／`mss`／ @@ -49,7 +49,7 @@ USB/IP 協定、Prometheus 指標），以維持這條輕相依基線。 │ 全部只呼叫下面這一層，不含業務邏輯 ┌───────────────────────────────▼──────────────────────────────────────────┐ │ 執行核心 Execution Core │ -│ utils/executor/action_executor.py ── Executor.event_dict（774 個 AC_*） │ +│ utils/executor/action_executor.py ── Executor.event_dict（776 個 AC_*） │ │ utils/executor/flow_control.py ── 34 個區塊指令（迴圈/分支/try/巨集） │ │ utils/script_vars ── ${var} 插值 │ utils/json ── action 檔 I/O │ └───────────────────────────────┬──────────────────────────────────────────┘ @@ -154,7 +154,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | 模組 | 行數 | 職責 | | --- | ---: | --- | -| `je_auto_control/__init__.py` | 1,970 | **套件門面**。集中匯入並再匯出 1,200 個公開名稱，以功能區塊註解分段（callback／exception／executor／a11y／vision／clipboard…）。 | +| `je_auto_control/__init__.py` | 1,977 | **套件門面**。集中匯入並再匯出 1,200 個公開名稱，以功能區塊註解分段（callback／exception／executor／a11y／vision／clipboard…）。 | | `je_auto_control/__main__.py` | 87 | 舊版 argparse 進入點：`-e` 執行單檔、`-d` 執行整個目錄、`--execute_str` 執行 JSON 字串、`-c` 建立專案。 | | `je_auto_control/cli.py` | 338 | **主 CLI**（`je_auto_control` console script）。子命令：`run`（含 `--var`／`--dry-run`）、`validate`／`lint`、`list-commands`、`fmt`、`record`、`codegen`、`failure-bundle`、`list-jobs`、`start-server`、`start-rest`、`version`。所有子命令延遲匯入，確保不碰 Qt。 | | `je_auto_control/api/__init__.py` | 22 | 版本化整合進入點。 | @@ -170,14 +170,14 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | 模組 | 行數 | 職責 | | --- | ---: | --- | -| `wrapper/platform_wrapper.py` | 116 | **Strategy 樞紐**。依 `sys.platform` 匯入唯一後端並匯出 `keyboard`、`keyboard_check`、`keyboard_keys_table`、`mouse`、`mouse_keys_table`、`special_mouse_keys_table`、`screen`、`recorder`；八個名稱都帶著 `backend_contract` 的型別出去，其中 `keyboard`／`mouse` 因為四個分支綁的是三種互不相容的形狀，先落在私有的 `_keyboard`／`_mouse`（`Any`）上再標合約；載入失敗直接拋 `AutoControlException`（fail fast）。 | +| `wrapper/platform_wrapper.py` | 119 | **Strategy 樞紐**。依 `sys.platform` 匯入唯一後端並匯出 `keyboard`、`keyboard_check`、`keyboard_keys_table`、`mouse`、`mouse_keys_table`、`special_mouse_keys_table`、`screen`、`recorder`；八個名稱都帶著 `backend_contract` 的型別出去，其中 `keyboard`／`mouse` 因為四個分支綁的是三種互不相容的形狀，先落在私有的 `_keyboard`／`_mouse`（`Any`）上再標合約；載入失敗直接拋 `AutoControlException`（fail fast）。 | | `wrapper/backend_contract.py` | 238 | 平台縫的型別合約：`ScreenBackend`／`KeyboardCheckBackend`／`RecorderBackend` 三個跨平台 Protocol，加上 `keyboard`／`mouse` 各自的三份——`Win32*`（SendInput 與 Interception）、`Darwin*`（Quartz）、`X11Unix*`（XTest／uinput／Wayland／BSD），因為這兩個名稱的呼叫形狀真的因平台而異；`KeyboardBackend`／`MouseBackend` 依 `sys.platform` 別名到其中一組，所以呼叫端被檢查的是它真的會走到的簽章。四個 `_platform_*` 組裝模組各自標注自己綁的是什麼，少一個成員就在該後端自己的檔案裡紅掉，而不是在三層之上的呼叫點。 | -| `wrapper/_platform_windows.py` | 318 | Windows 後端組裝：Win32 ctypes 模組 + 虛擬鍵表 + 選用 Interception 驅動。 | -| `wrapper/_platform_osx.py` | 157 | macOS 後端組裝（Quartz 事件 + osx 虛擬鍵表）。 | -| `wrapper/_platform_linux.py` | 275 | X11 後端組裝（python-Xlib + 選用 uinput）。 | -| `wrapper/_platform_wayland.py` | 58 | Wayland 後端組裝（libei／ydotool／grim）。 | -| `wrapper/auto_control_mouse.py` | 491 | 滑鼠 API：位置讀寫、按下／放開／點擊、捲動、座標前處理、送訊息給指定視窗。 | -| `wrapper/auto_control_keyboard.py` | 368 | 鍵盤 API：鍵表查詢、按下／放開／敲擊、`write` 字串、`hotkey` 組合鍵、按鍵狀態偵測。**`type_keyboard` 與 `hotkey` 的放開走 `finally`**（見下）。 | +| `wrapper/_platform_windows.py` | 366 | Windows 後端組裝：Win32 ctypes 模組 + 虛擬鍵表（含 OEM 鍵與 `keyboard_key_aliases` 別名表）+ 選用 Interception 驅動。 | +| `wrapper/_platform_osx.py` | 162 | macOS 後端組裝（Quartz 事件 + osx 虛擬鍵表）。 | +| `wrapper/_platform_linux.py` | 278 | X11 後端組裝（python-Xlib + 選用 uinput）。 | +| `wrapper/_platform_wayland.py` | 63 | Wayland 後端組裝（libei／ydotool／grim）。 | +| `wrapper/auto_control_mouse.py` | 537 | 滑鼠 API：位置讀寫、按下／放開／點擊、捲動、座標前處理、送訊息給指定視窗。 | +| `wrapper/auto_control_keyboard.py` | 396 | 鍵盤 API：鍵表查詢、`keyboard_key_name` 反查標準鍵名（不回別名）、按下／放開／敲擊、`write` 字串、`hotkey` 組合鍵、按鍵狀態偵測。**`type_keyboard` 與 `hotkey` 的放開走 `finally`**（見下）。 | | `wrapper/auto_control_screen.py` | 111 | 螢幕 API：`screen_size`、`screenshot`（可指定區域）、`get_pixel`。 | | `wrapper/auto_control_image.py` | 83 | 影像 API：`locate_all_image`、`locate_image_center`、`locate_and_click`。 | | `wrapper/auto_control_record.py` | 124 | 錄製 API：`record`／`stop_record`／`record_to_json`（支援 stop event 與逾時）。 | @@ -186,12 +186,12 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 ### 5.3 平台後端 -#### Windows（`windows/`，23 檔／1,957 行） +#### Windows（`windows/`，23 檔／1,974 行） | 模組 | 行數 | 職責 | | --- | ---: | --- | | `core/utils/win32_ctype_input.py` | 73 | `SendInput` 的 ctypes 結構定義與送出。 | -| `core/utils/win32_vk.py` | 188 | Windows 虛擬鍵碼對照表。 | +| `core/utils/win32_vk.py` | 205 | Windows 虛擬鍵碼對照表。 | | `core/utils/win32_keypress_check.py` | 22 | `GetAsyncKeyState` 按鍵狀態查詢。 | | `mouse/win32_ctype_mouse_control.py` | 220 | 滑鼠事件產生（含多螢幕絕對座標換算）。 | | `keyboard/win32_ctype_keyboard_control.py` | 98 | 鍵盤事件產生。 | @@ -271,7 +271,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 ### 5.4.1 執行引擎與腳本資產 -> 24 個套件、約 14,247 行。 +> 24 個套件、約 14,266 行。 | 模組 | 行數 | 職責 | | --- | ---: | --- | @@ -282,7 +282,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `utils/dag/` | 492 | 跨主機 DAG 編排器（圖模型 + runner） | | `utils/decision_table/` | 112 | DMN 風格決策表：規則 + 命中策略，把分支外部化 | | `utils/deterministic/` | 116 | 決定性執行控制：固定亂數種子 + 凍結時鐘 | -| `utils/executor/` | 9,412 | **核心**。`Executor` 指令分派表（774 個 `AC_*`）、參數插值、乾跑、逐步 callback；`flow_control` 提供 34 個區塊指令（迴圈／分支／try／巨集／變數） | +| `utils/executor/` | 9,431 | **核心**。`Executor` 指令分派表（776 個 `AC_*`）、參數插值、乾跑、逐步 callback；`flow_control` 提供 34 個區塊指令（迴圈／分支／try／巨集／變數） | | `utils/flow_debugger/` | 155 | action list 的單步除錯器與追蹤器 | | `utils/input_macro/` | 451 | 定時輸入事件：錄製結果的整形（`timeline`／`InputRecorder`，Windows 與 macOS 共用）、重播與宣告式輸入序列 DSL | | `utils/json/` | 99 | action JSON 檔讀寫與正規化格式化（`fmt --check` 的後端） | @@ -341,7 +341,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 ### 5.4.4 輸入模擬與動作品質 -> 22 個套件、約 2,722 行。 +> 22 個套件、約 2,802 行。 | 模組 | 行數 | 職責 | | --- | ---: | --- | @@ -357,7 +357,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `utils/ime_state/` | 146 | 讀取即時 IME 組字／轉換狀態，確保 CJK 輸入安全 | | `utils/key_hold/` | 109 | 按住按鍵一段時間，或以固定頻率自動重複 | | `utils/modifier_state/` | 76 | 跨一組動作按住修飾鍵，並保證安全釋放 | -| `utils/mouse_path/` | 106 | 多路徑點滑鼠手勢（沿折線移動或拖曳） | +| `utils/mouse_path/` | 105 | 多路徑點滑鼠手勢（沿折線移動或拖曳） | | `utils/mouse_relative/` | 59 | 相對位移滑鼠移動 | | `utils/postcondition/` | 146 | 宣告式的動作預期結果規格，對照畫面驗證 | | `utils/step_repair/` | 134 | 失敗／無效動作的修復策略（自我修正迴圈） | @@ -365,7 +365,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `utils/input_reach/` | 111 | 送出去的輸入到不到得了：桌面鎖定查詢（免費）＋ 實際送一個 F13 確認沒有被過濾（有副作用，只給診斷用） | | `utils/keyboard_layout/` | 148 | 向系統問「這個鍵盤配置下每個鍵印出什麼字」（`ToUnicodeEx`），問不到退回 US 對照表 | | `utils/text_unicode/` | 151 | 輸入任意 Unicode（emoji／CJK／重音字）：優先送字元按鍵事件，不支援時退回剪貼簿貼上 | -| `utils/tween_drag/` | 101 | 沿曲線的緩動插值拖曳 | +| `utils/tween_drag/` | 182 | 沿曲線的緩動插值拖曳；`_drag_through` 是它與 `drag_path` 共用的按下／移動／放開（`step_delay_s`、`settle_s` 節奏，失敗時在 `finally` 裡於最後到達點放開） | | `utils/verify_field/` | 112 | 打字後讀回欄位，確認內容確實落地 | ### 5.4.5 影像辨識與畫面分析 @@ -414,7 +414,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 ### 5.4.6 OCR 與文字理解 -> 19 個套件、約 3,336 行。 +> 19 個套件、約 3,563 行。 | 模組 | 行數 | 職責 | | --- | ---: | --- | @@ -427,7 +427,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `utils/guardrail/` | 116 | 針對畫面／OCR 文字的啟發式 prompt-injection 防護 | | `utils/heading_segment/` | 69 | 判定 OCR 行是標題或內文，建出文件大綱 | | `utils/near_dup/` | 108 | 近似重複文字偵測（SimHash／MinHash） | -| `utils/ocr/` | 1,126 | OCR 引擎門面 + 三個後端（Tesseract／EasyOCR／PaddleOCR）、版面結構化與跨詞比對（`text_span`） | +| `utils/ocr/` | 1,353 | OCR 引擎門面 + 三個後端（Tesseract／EasyOCR／PaddleOCR）、版面結構化與跨詞比對（`text_span`）、Tesseract 的尋找／設定／健康檢查（`tesseract_setup`） | | `utils/pii_text/` | 119 | 自由文字中的 PII 偵測與遮蔽（email／電話／SSN／卡號／IP／IBAN） | | `utils/readability/` | 138 | 可讀性評分（Flesch、Flesch-Kincaid、Gunning Fog、SMOG、ARI） | | `utils/reading_flow/` | 119 | 以遞迴 XY-cut 推導欄位感知的閱讀順序 | @@ -493,7 +493,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 ### 5.4.9 AI / Agent / LLM -> 13 個套件、約 21,391 行。 +> 13 個套件、約 21,443 行。 | 模組 | 行數 | 職責 | | --- | ---: | --- | @@ -506,7 +506,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `utils/cua_action/` | 204 | 標準化 computer-use 動作結構（Anthropic／OpenAI → `AC_*`） | | `utils/llm/` | 365 | 自然語言 → action list 規劃器 + Anthropic／null 後端 | | `utils/mcp_registry/` | 97 | MCP registry `server.json` 資訊清單產生（可被發現） | -| `utils/mcp_server/` | 17,675 | **無頭 MCP 伺服器**（16K LOC，預設註冊 677 個工具＝658 個 `ac_*` + 19 個別名）：stdio + HTTP 傳輸、工具工廠與處理器、資源、prompt、稽核、限流、外掛熱重載 | +| `utils/mcp_server/` | 17,727 | **無頭 MCP 伺服器**（16K LOC，預設註冊 679 個工具＝660 個 `ac_*` + 19 個別名）：stdio + HTTP 傳輸、工具工廠與處理器、資源、prompt、稽核、限流、外掛熱重載 | | `utils/tool_use_schema/` | 189 | 把 `AC_*` 指令匯出成 Claude／OpenAI 的 tool-use schema | | `utils/trajectory_eval/` | 113 | agent 軌跡評估：依評分規準為一次執行打分 | | `utils/vision/` | 518 | VLM 元素定位器（依描述找元素）+ Anthropic／OpenAI／null 後端 | @@ -695,26 +695,26 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 上表以子套件為單位；以下把行數最大的幾個子系統展開到檔案層。 -#### `utils/executor/`（9,412 行）— 執行核心 +#### `utils/executor/`（9,431 行）— 執行核心 | 檔案 | 行數 | 職責 | | --- | ---: | --- | -| `action_executor.py` | 8,289 | `Executor` 類別與 `event_dict` 分派表（774 個指令），另含數百個把 utils 能力接成指令的 adapter 函式；全域單例 `executor` 與 `add_command_to_executor()` 擴充點。 | +| `action_executor.py` | 8,308 | `Executor` 類別與 `event_dict` 分派表（776 個指令），另含數百個把 utils 能力接成指令的 adapter 函式；全域單例 `executor` 與 `add_command_to_executor()` 擴充點。 | | `flow_control.py` | 622 | 真正的流程控制：`AC_loop`／`AC_for_each`／`AC_while_*`／`AC_if_*`／`AC_try`／`AC_retry`／`AC_parallel`／`AC_define_macro`／`AC_call_macro`／變數指令（`AC_set_var`／`AC_get_var`／`AC_inc_var`）。`LoopBreak`／`LoopContinue` 以例外實作。34 個區塊指令的分派表 `BLOCK_COMMANDS` 也在這裡，含下一列匯入的資料來源指令。 | | `flow_data_commands.py` | 262 | `AC_*_to_var` 資料來源與轉換指令：shell、時鐘、亂數、PDF、TOTP、SQL、檔案、HTTP、OCR，加上 `AC_assert_var`／`AC_assert_db`／`AC_assert_duration`／`AC_transform_var`。都不執行巢狀 action list，所以沒有迴圈／分支語意。 | | `action_schema.py` | 128 | action list 的結構驗證：形狀、參數型別、未知指令拒絕。單一走訪同時支援兩種消費方式：`validate_actions()` 遇到第一個問題就拋、`unknown_command_names()` 收齊全部不認得的名字（REST `/execute` 用它回 400）。 | | `action_redaction.py` | 72 | 記錄與紀錄鍵用的遮蔽：`AC_secret_*` 的參數（金庫通行碼、機密值）在寫進 log、當成結果紀錄的鍵之前換成 `***`，巢狀在區塊指令裡的也一樣。 | | `mouse_aliases.py` | 39 | 單鍵點擊別名（`AC_click_left` 等），executor 與 callback executor 共用。 | -#### `utils/mcp_server/`（17,675 行，677 個工具）— 最大子系統 +#### `utils/mcp_server/`（17,727 行，679 個工具）— 最大子系統 | 檔案 | 行數 | 職責 | | --- | ---: | --- | -| `tools/_factories.py` | 9,005 | 工具工廠：每個函式回傳一個領域的 `MCPTool` 清單（把 `AC_*` 能力包成 MCP 工具）。 | -| `tools/_handlers.py` | 545 | 把 MCP 工具呼叫橋接到 AutoControl 無頭 API 的 adapter；主題模組拆完之後這裡留的是資料／文字／HTTP 那一類與 WebRunner 橋接。 | +| `tools/_factories.py` | 9,036 | 工具工廠：每個函式回傳一個領域的 `MCPTool` 清單（把 `AC_*` 能力包成 MCP 工具）。 | +| `tools/_handlers.py` | 547 | 把 MCP 工具呼叫橋接到 AutoControl 無頭 API 的 adapter；主題模組拆完之後這裡留的是資料／文字／HTTP 那一類與 WebRunner 橋接。 | | `tools/_handlers_qa.py` | 414 | 同一種 adapter，QA 主題：斷言 DSL、資料驅動、SQL／PDF／郵件／HTTP 步驟、codegen、視覺回歸、狀態機、flaky 偵測與隔離、suite runner、無障礙稽核、裝置矩陣、媒體斷言。從 `_handlers.py` 依主題拆出的第一塊（750 行上限）；兩者互不引用。 | -| `tools/_handlers_input.py` | 212 | 同一種 adapter，輸入主題：滑鼠、鍵盤、虛擬手把（ViGEm）。 | -| `tools/_handlers_screen.py` | 305 | 同一種 adapter，螢幕主題：擷取、像素、影像與文字搜尋、螢幕錄影。 | +| `tools/_handlers_input.py` | 215 | 同一種 adapter，輸入主題：滑鼠、鍵盤、虛擬手把（ViGEm）。 | +| `tools/_handlers_screen.py` | 318 | 同一種 adapter，螢幕主題：擷取、像素、影像與文字搜尋、螢幕錄影。 | | `tools/_handlers_system.py` | 566 | 同一種 adapter，桌面工作階段：視窗、行程與 shell、開檔、閒置與睡眠、音量、鎖定、輸入法狀態、欄位驗證與重試、色彩對比、變更排序、元件分類、剪貼簿。 | | `tools/_handlers_runs.py` | 110 | 同一種 adapter，執行主題：executor、執行歷史、錄製、動作檔。 | | `tools/_handlers_scheduling.py` | 200 | 同一種 adapter，排程主題：排程器、觸發器、熱鍵常駐。 | @@ -729,7 +729,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | `_protocol.py` | 167 | JSON-RPC 線路格式：版本與識別常數、`_MCPError`、決定失敗工具行為的錯誤 tuple、envelope 產生器、工具回傳值轉 `content` 區塊。不碰伺服器狀態。 | | `resources.py` | 307 | MCP resource 提供者。 | | `prompts.py` | 220 | MCP prompt 目錄。 | -| `fake_backend.py` | 184 | CI／無頭測試用的記憶體內假後端。 | +| `fake_backend.py` | 187 | CI／無頭測試用的記憶體內假後端。 | | `plugin_watcher.py` | 168 | 檔案變更時熱重載外掛工具的背景 watcher。 | | `tools/_base.py` | 146 | 工具註冊表的共用型別與輔助。 | | `tools/_validation.py` | 122 | MCP 工具用到的 JSON Schema 子集驗證器。 | @@ -832,7 +832,7 @@ socket server 有 8 MiB 讀取上限與 30 秒 handler timeout。 | --- | --- | | `accessibility/` | `accessibility_api.py`（公開 API）、`element.py`（dataclass）、`tree.py`（遞迴樹傾印）、`recorder.py`（輪詢式事件錄製）、`backends/`：`base.py` 330 行抽象、`windows_backend.py` 801 行（comtypes UIA）、`windows_reads.py` 142 行（pattern／文字範圍／表頭／元素屬性的純讀取與 `UIA_READ_ERRORS`）、`windows_query.py` 176 行（UIA 搜尋起點、可中斷走訪、快取請求、NULL COM 指標判定與 `UIA_ERRORS`）、`windows_state.py` 98 行（控制項狀態讀取與密碼欄位判定）、`macos_backend.py` 125 行（pyobjc AX）、`null_backend.py` fallback | | `agent/` | `agent_loop.py`、`computer_use.py`、`backends/`：`anthropic.py`、`anthropic_computer_use.py`（435 行）、`openai.py`、`base.py` | -| `ocr/` | `ocr_engine.py`（門面）、`structure.py`（版面）、`backends/`：`tesseract_backend.py`、`easyocr_backend.py`、`paddleocr_backend.py`、`base.py` | +| `ocr/` | `ocr_engine.py`（門面）、`structure.py`（版面）、`tesseract_setup.py`（找執行檔、`TESSDATA_PREFIX`、`ocr_status`／`ocr_languages`；`None` 與 `[]` 分開回報）、`backends/`：`tesseract_backend.py`、`easyocr_backend.py`、`paddleocr_backend.py`、`base.py` | | `vision/` | `vlm_api.py`、`backends/`：`anthropic_backend.py`、`openai_backend.py`、`null_backend.py`、`_parse.py`、`base.py` | | `llm/` | `planner.py`、`backends/`：`anthropic_backend.py`、`null_backend.py`、`base.py` | | `hotkey/` | `hotkey_daemon.py`、`backends/`：`windows_backend.py`（RegisterHotKey + 訊息幫浦）、`linux_backend.py`（XGrabKey）、`macos_backend.py`（CGEventTap）、`base.py` | @@ -983,7 +983,7 @@ GUI 是**選用 extra**（`pip install je_auto_control[gui]`，PySide6 + qt-mate | `ci_templates/.gitlab-ci.yml` | — | 供使用者專案複製的 GitLab CI 範本。 | | `docs/` | Sphinx（`API`／`Eng`／`Zh`／`getting_started`） | Read the Docs 文件。 | | `architecture_diagram/` | drawio + png | 既有的架構圖原始檔。 | -| `test/` | `unit_test/headless`（主要）、`unit_test/flow_control`、`integrated_test`、`gui_test`、`manual_test`、`verify`、`test_source` | 478 個 `test_*.py`／4,654 個測試函式。**注意**：`test/unit_test/` 下的 `*_test.py` 是會真的驅動滑鼠鍵盤的手動示範腳本，因此 `pyproject.toml` 把 `python_files` 釘成 `test_*.py`。`unit_test/headless/conftest.py` 有一個 autouse fixture，每個測試結束都沖掉 Qt 排隊中的 `deleteLater()`——不沖會讓殘留的 widget 在後面某個不相干的測試裡被銷毀，曾經整個直譯器 `__fastfail`。`test_doc_counts.py` 守住文件引用的指令／工具／子套件／範例數,`test_doc_line_counts.py` 守住所有行數（`--fix` 可一次重新產生）。 `verify/macos_verify.py` 是在真的 `macos-14` runner 上量測 TCC 到底允許什麼的探針（macOS 是唯一沒有容器可用的支援平台），不被 pytest 收集。 | +| `test/` | `unit_test/headless`（主要）、`unit_test/flow_control`、`integrated_test`、`gui_test`、`manual_test`、`verify`、`test_source` | 628 個 `test_*.py`／6,673 個測試函式。**注意**：`test/unit_test/` 下的 `*_test.py` 是會真的驅動滑鼠鍵盤的手動示範腳本，因此 `pyproject.toml` 把 `python_files` 釘成 `test_*.py`。`unit_test/headless/conftest.py` 有一個 autouse fixture，每個測試結束都沖掉 Qt 排隊中的 `deleteLater()`——不沖會讓殘留的 widget 在後面某個不相干的測試裡被銷毀，曾經整個直譯器 `__fastfail`。`test_doc_counts.py` 守住文件引用的指令／工具／子套件／範例數,`test_doc_line_counts.py` 守住所有行數（`--fix` 可一次重新產生）。 `verify/macos_verify.py` 是在真的 `macos-14` runner 上量測 TCC 到底允許什麼的探針（macOS 是唯一沒有容器可用的支援平台），不被 pytest 收集。 | --- @@ -1054,32 +1054,36 @@ socket 預設綁 `127.0.0.1`；資源一律用 `with`。 其中一支專門釘住「`AutoControlKeyboardException` 不在那五個型別底下」這個**前提**——整個 設計靠它成立，前提哪天變了，這批註解也要跟著改。 +滑鼠拖曳是同一個形狀：`utils/tween_drag/tween_drag.py` 的 `_drag_through`（`tween_drag` 與 `mouse_path.drag_path` +共用）在按下之後用 `held` 旗標加 `finally`，任何一步丟例外都會在**游標最後到達的點**放開——不是終點，那等於替一次 +沒走完的拖曳完成放置——而收尾的放開失敗只記 log、不蓋掉原本的例外。守門：`test_drag_pacing.py`。 + --- ## 8. 附錄：各層規模 | 層／子系統 | 檔案數 | 行數 | | --- | ---: | ---: | -| `gui/` | 91 | 26,821 | -| `utils/mcp_server/` | 31 | 17,675 | +| `gui/` | 91 | 26,845 | +| `utils/mcp_server/` | 31 | 17,727 | | `utils/remote_desktop/` | 56 | 12,561 | -| `utils/executor/` | 7 | 9,412 | +| `utils/executor/` | 7 | 9,431 | | `utils/usb/` | 17 | 4,472 | -| `je_auto_control/`（頂層 3 檔） | 3 | 2,395 | +| `je_auto_control/`（頂層 3 檔） | 3 | 2,402 | | `utils/accessibility/` | 14 | 3,032 | -| `wrapper/` | 19 | 3,615 | -| `windows/` | 23 | 1,957 | +| `wrapper/` | 19 | 3,753 | +| `windows/` | 23 | 1,974 | | `utils/rest_api/` | 8 | 1,793 | | `utils/agent/` | 8 | 1,446 | | `linux_with_x11/` | 19 | 1,236 | | `linux_wayland/` | 17 | 2,870 | | `utils/triggers/` | 4 | 1,241 | -| `utils/ocr/` | 9 | 1,126 | +| `utils/ocr/` | 10 | 1,353 | | `utils/usbip/` | 5 | 945 | | `utils/assertion/` | 3 | 881 | | `osx/` | 17 | 919 | | `autocontrol-lsp/` | 8 | 744 | | `utils/hotkey/` | 7 | 837 | -| 其餘模組（約 286 個 `utils/` 子套件 + `android/`／`ios/`／周邊小工具） | 677 | 52,882 | -| **總計** | **1,043** | **148,860** | +| 其餘模組（約 286 個 `utils/` 子套件 + `android/`／`ios/`／周邊小工具） | 677 | 52,962 | +| **總計** | **1,044** | **149,424** | diff --git a/docs/source/API/special/keyboard_keys.rst b/docs/source/API/special/keyboard_keys.rst index dd7cc6790..16796e83f 100644 --- a/docs/source/API/special/keyboard_keys.rst +++ b/docs/source/API/special/keyboard_keys.rst @@ -14,11 +14,12 @@ in keyboard functions. Windows ======= -Common keys: +Common keys. The first name is the Win32 one a reverse lookup +(``keyboard_key_name``) returns; the others are aliases for the same key: .. list-table:: :header-rows: 1 - :widths: 30 70 + :widths: 40 60 * - Key Name - Description @@ -28,46 +29,55 @@ Common keys: - Number keys * - ``f1`` - ``f24`` - Function keys - * - ``enter``, ``return`` + * - ``return``, ``enter`` - Enter / Return * - ``tab`` - Tab * - ``space`` - Space bar - * - ``back`` + * - ``back``, ``backspace``, ``bksp`` - Backspace - * - ``escape`` + * - ``escape``, ``esc`` - Escape - * - ``lcontrol``, ``rcontrol`` - - Left / Right Control - * - ``lshift``, ``rshift`` - - Left / Right Shift - * - ``lalt``, ``ralt`` - - Left / Right Alt - * - ``lwin``, ``rwin`` - - Left / Right Windows key + * - ``control``, ``ctrl``; ``lcontrol``, ``lctrl``; ``rcontrol``, ``rctrl`` + - Control, left / right Control + * - ``shift``, ``lshift``, ``rshift`` + - Shift, left / right Shift + * - ``menu``, ``alt``; ``lmenu``, ``lalt``; ``rmenu``, ``ralt`` + - Alt, left / right Alt + * - ``lwin``, ``win``, ``super``, ``cmd``, ``meta``; ``rwin`` + - Left / right Windows key * - ``up``, ``down``, ``left``, ``right`` - Arrow keys - * - ``insert``, ``delete`` + * - ``insert``, ``ins``; ``delete``, ``del`` - Insert / Delete * - ``home``, ``end`` - Home / End - * - ``pageup``, ``pagedown`` + * - ``prior``, ``pgup``, ``pageup``; ``next``, ``pgdn``, ``pagedown`` - Page Up / Page Down - * - ``capslock`` + * - ``capital``, ``caps``, ``capslock`` - Caps Lock - * - ``numlock`` - - Num Lock - * - ``print_screen`` + * - ``numlock``; ``scroll``, ``scrolllock`` + - Num Lock / Scroll Lock + * - ``snapshot``, ``printscreen``, ``prtsc``, ``prtscr`` - Print Screen - * - ``numpad0`` - ``numpad9`` + * - ``num0`` - ``num9``, ``numpad0`` - ``numpad9`` - Numpad keys - * - ``add``, ``subtract``, ``multiply``, ``divide`` + * - ``add``, ``subtract``, ``multiply``, ``divide``, ``decimal`` - Numpad operators + * - ``oem_plus``, ``plus``; ``oem_comma``, ``comma``; ``oem_minus``, ``minus``; ``oem_period``, ``period`` + - ``=`` / ``,`` / ``-`` / ``.`` keys, the same key on every layout + * - ``oem_1`` - ``oem_8``, ``oem_102`` + - Punctuation keys whose character depends on the layout (``oem_1`` is + ``;`` on a US layout); they have no character-named alias on purpose + * - ``oem_clear`` + - Clear * - ``apps`` - Application / Menu key - * - ``browser_back``, ``browser_forward`` + * - ``browser_back``, ``browser_forward``, ``browser_home`` - Browser navigation keys + * - ``launch_app1``, ``launch_app2`` + - Launch application keys (``LAUNCH_APP2`` also still works) * - ``volume_mute``, ``volume_up``, ``volume_down`` - Volume control keys @@ -77,8 +87,8 @@ Common keys: .. code-block:: python - from je_auto_control import keys_table - for key_name in sorted(keys_table.keys()): + from je_auto_control import keyboard_keys_table + for key_name in sorted(keyboard_keys_table.keys()): print(key_name) Linux (X11) diff --git a/docs/source/API/wrapper/keyboard.rst b/docs/source/API/wrapper/keyboard.rst index 8850ca4fb..c0ce8f0b1 100644 --- a/docs/source/API/wrapper/keyboard.rst +++ b/docs/source/API/wrapper/keyboard.rst @@ -32,6 +32,26 @@ get_keyboard_keys_table ---- +keyboard_key_name +================= + +.. function:: keyboard_key_name(keycode) + + Returns the canonical name of a key code, for recorders and anything else + that turns a captured virtual key back into a name. A code often has + several names in the table; aliases (``esc`` for ``escape``, listed in + ``platform_wrapper.keyboard_key_aliases``) never answer, so adding one + cannot change the names a recording writes. Among the rest, a name of + lower-case letters, digits and underscores wins, then the shortest, then + the alphabetically first. + + :param int keycode: Platform key code (a Win32 virtual key on Windows). + :returns: The name, e.g. ``"escape"`` for 27 on Windows, or ``None`` when + the table has no name for the code. + :rtype: str or None + +---- + press_keyboard_key ================== diff --git a/docs/source/API/wrapper/mouse.rst b/docs/source/API/wrapper/mouse.rst index 2ec151dc8..a1de4c4a1 100644 --- a/docs/source/API/wrapper/mouse.rst +++ b/docs/source/API/wrapper/mouse.rst @@ -98,14 +98,21 @@ release_mouse click_mouse =========== -.. function:: click_mouse(mouse_keycode, x=None, y=None) +.. function:: click_mouse(mouse_keycode, x=None, y=None, clicks=1, interval=0.0) - Presses and releases a mouse button at the specified position. + Presses and releases a mouse button at the specified position, ``clicks`` + times, ``interval`` seconds apart; ``clicks=2`` double-clicks. Windows and + X11 recognise a double-click from the timing and distance of the clicks, so + keep ``interval`` under the system double-click time. On macOS the clicks + arrive as separate single clicks. :param mouse_keycode: Mouse button name (e.g., ``"mouse_left"``). :type mouse_keycode: int or str :param int x: X position (default: current position). :param int y: Y position (default: current position). + :param int clicks: Number of clicks, at least 1 (default: 1). + :param float interval: Seconds between clicks, at least 0 (default: 0.0). + :raises AutoControlMouseException: on an invalid ``clicks`` or ``interval``. :returns: Tuple of ``(keycode, x, y)``. :rtype: tuple diff --git a/docs/source/Eng/doc/keyboard/keyboard_doc.rst b/docs/source/Eng/doc/keyboard/keyboard_doc.rst index 3f2afb1ea..6e8a99f72 100644 --- a/docs/source/Eng/doc/keyboard/keyboard_doc.rst +++ b/docs/source/Eng/doc/keyboard/keyboard_doc.rst @@ -24,6 +24,19 @@ Retrieve available key names: See :doc:`/API/special/keyboard_keys` for the full list of available keyboard keys per platform. +On Windows the table also accepts common aliases (``ctrl``, ``alt``, ``enter``, +``esc``, ``win``, ``backspace``, ``del``, ``pgup``, ``capslock``, ``prtsc``, +``numpad0`` ...). To turn a captured key code back into a name, use +``keyboard_key_name``; it always answers with the canonical name, never an +alias: + +.. code-block:: python + + from je_auto_control import keyboard_key_name + + keyboard_key_name(27) # "escape" on Windows + keyboard_key_name(0xBB) # "oem_plus" + Press and Release ================= diff --git a/docs/source/Eng/doc/mouse/mouse_doc.rst b/docs/source/Eng/doc/mouse/mouse_doc.rst index 6f1654326..6880c342a 100644 --- a/docs/source/Eng/doc/mouse/mouse_doc.rst +++ b/docs/source/Eng/doc/mouse/mouse_doc.rst @@ -49,6 +49,13 @@ Press and immediately release a mouse button: # Left click at specific coordinates click_mouse("mouse_left", x=500, y=300) + # Double-click: two clicks on the same point, 60 ms apart + click_mouse("mouse_left", x=500, y=300, clicks=2, interval=0.06) + +Windows and X11 recognise a double-click from the timing and distance of the +clicks, so keep ``interval`` under the system double-click time (500 ms by +default on Windows). On macOS the clicks arrive as separate single clicks. + Position ======== diff --git a/docs/source/Eng/doc/new_features/v116_features_doc.rst b/docs/source/Eng/doc/new_features/v116_features_doc.rst index 3b39a672f..f0124825b 100644 --- a/docs/source/Eng/doc/new_features/v116_features_doc.rst +++ b/docs/source/Eng/doc/new_features/v116_features_doc.rst @@ -32,12 +32,15 @@ steps (``easing`` is any name from ``path_easings()`` — ``linear`` / duplicate the shared junction points. ``move_along_path`` emits move events through the path; ``drag_path`` presses at the first waypoint, moves through the whole path, and releases at the last — for multi-stop drags. Both take a ``sink`` -override for headless testing. +override for headless testing. ``drag_path`` also takes ``tween_drag``'s pacing, +``step_delay_s`` (after each move) and ``settle_s`` (before the press, after it and +before the release), and like it releases the button where the pointer stopped if a +step raises. Executor commands ----------------- ``AC_move_along_path`` and ``AC_drag_path`` take ``waypoints`` (a JSON -``[[x, y], ...]`` list) plus ``easing`` / ``per_segment_steps`` (and ``button`` -for the drag). Both are exposed as MCP tools (``ac_move_along_path`` / +``[[x, y], ...]`` list) plus ``easing`` / ``per_segment_steps`` (and ``button``, +``step_delay_s`` and ``settle_s`` for the drag). Both are exposed as MCP tools (``ac_move_along_path`` / ``ac_drag_path``) and as Script Builder commands under **Mouse**. diff --git a/docs/source/Eng/doc/new_features/v29_features_doc.rst b/docs/source/Eng/doc/new_features/v29_features_doc.rst index cc6b319d6..0b467ae42 100644 --- a/docs/source/Eng/doc/new_features/v29_features_doc.rst +++ b/docs/source/Eng/doc/new_features/v29_features_doc.rst @@ -27,3 +27,14 @@ coordinates; ``tween_drag`` presses at the start, moves through the points, and releases at the end. Easings: ``linear`` / ``ease_in_out_quad`` / ``ease_out_cubic`` / ``ease_in_cubic`` (see :func:`easing_names`). Exposed as ``AC_tween_drag`` / ``ac_tween_drag`` (``start`` / ``end`` as ``[x, y]``). + +Pacing, for apps that tell a drag from a click by the pointer's motion +(file managers, drawing programs, games):: + + tween_drag((0, 0), (300, 200), steps=24, step_delay_s=0.012, settle_s=0.08) + +``step_delay_s`` rests after each move; ``settle_s`` rests on the start +before the press, after the press and before the release. Both default to +0 (no pause) and must be finite and non-negative. If a step raises, the +button is released in a ``finally`` at the last point the pointer reached, +not at the end, and the original error propagates. diff --git a/docs/source/Eng/doc/ocr_backends/ocr_backends_doc.rst b/docs/source/Eng/doc/ocr_backends/ocr_backends_doc.rst index 9a0487925..138f5e366 100644 --- a/docs/source/Eng/doc/ocr_backends/ocr_backends_doc.rst +++ b/docs/source/Eng/doc/ocr_backends/ocr_backends_doc.rst @@ -63,14 +63,30 @@ Tesseract setup notes ===================== Windows users need to install the Tesseract binary separately (UB-Mannheim -build is the most common). If ``tesseract.exe`` is not on ``PATH``:: +build is the most common); its installer does not add itself to ``PATH``. +``find_tesseract_cmd()`` looks for the executable in ``$TESSERACT_CMD`` +(when it names an existing file), then ``PATH``, then the installers' +default folders (``Program Files``, ``Program Files (x86)`` and the per-user +``%LOCALAPPDATA%\\Programs`` on Windows; Homebrew, ``/usr/local`` and +MacPorts on macOS; ``/usr/bin`` and ``/usr/local/bin`` elsewhere). It only +looks; hand the result to ``set_tesseract_cmd`` to make OCR use it:: - from je_auto_control import set_tesseract_cmd - set_tesseract_cmd(r"C:\Program Files\Tesseract-OCR\tesseract.exe") + from je_auto_control import find_tesseract_cmd, set_tesseract_cmd + + command = find_tesseract_cmd() + if command: + set_tesseract_cmd(command) Language pack files (``*.traineddata``) live under ``Tesseract-OCR\\tessdata\\`` — copy the ones you need from the -`tessdata GitHub repo `_. +`tessdata GitHub repo `_. That +folder needs administrator rights on Windows. To keep the files somewhere +writable instead, point the engine there; this sets ``TESSDATA_PREFIX`` for +the process, and ``None`` restores the engine's default:: + + from je_auto_control import set_tessdata_dir + + set_tessdata_dir("ocr_tessdata") # returns the resolved path EasyOCR / PaddleOCR setup notes ================================ @@ -106,12 +122,36 @@ OCR commands are exposed through the executor for JSON action files:: {"command": "AC_locate_text", "text": "Sign in", "lang": "eng"} {"command": "AC_click_text", "text": "OK", "backend": "easyocr"} + {"command": "AC_ocr_status"} + {"command": "AC_ocr_languages"} -Both commands accept the same arguments as their Python counterparts. +The commands accept the same arguments as their Python counterparts. +``AC_ocr_status`` returns ``{"ok": ..., "reason": ...}`` and +``AC_ocr_languages`` returns ``{"languages": [...]}``, with ``null`` when the +engine cannot be asked. Diagnostics =========== +To ask whether Tesseract can work here, and which part is missing if not:: + + from je_auto_control import ocr_languages, ocr_status + + ok, reason = ocr_status() + # reason: "ready", "missing_package" (no pytesseract), "missing_engine" + # (the configured command is not found), "engine_unusable" (it does not + # run), or "no_language_data" (it ran and listed no languages) + + languages = ocr_languages() + # ["chi_tra", "eng", ...] installed language codes, sorted + # [] the engine answered: no language data at all + # None the engine could not be asked + +``ocr_status`` checks the command OCR would use and changes nothing. +``ocr_languages`` asks the engine on every call, so language files added +while the process runs are seen. Keep ``None`` and ``[]`` apart: they call +for opposite handling. + To check which backends are reachable in the current environment:: from je_auto_control.utils.ocr.backends import get_backend diff --git a/docs/source/Zh/doc/keyboard/keyboard_doc.rst b/docs/source/Zh/doc/keyboard/keyboard_doc.rst index 0c15091c1..f24a38e77 100644 --- a/docs/source/Zh/doc/keyboard/keyboard_doc.rst +++ b/docs/source/Zh/doc/keyboard/keyboard_doc.rst @@ -24,6 +24,17 @@ AutoControl 提供模擬鍵盤輸入的功能，包括按鍵按下/釋放、輸 完整的鍵盤按鍵列表請參考 :doc:`/API/special/keyboard_keys`。 +Windows 的按鍵表也收常見的別名（``ctrl``、``alt``、``enter``、``esc``、``win``、``backspace``、 +``del``、``pgup``、``capslock``、``prtsc``、``numpad0``……）。要把錄到的鍵碼換回名字，用 +``keyboard_key_name``；它一律回標準名稱，不會回別名： + +.. code-block:: python + + from je_auto_control import keyboard_key_name + + keyboard_key_name(27) # Windows 上是 "escape" + keyboard_key_name(0xBB) # "oem_plus" + 按下與釋放 ========== diff --git a/docs/source/Zh/doc/mouse/mouse_doc.rst b/docs/source/Zh/doc/mouse/mouse_doc.rst index f4a680b0e..af5530a80 100644 --- a/docs/source/Zh/doc/mouse/mouse_doc.rst +++ b/docs/source/Zh/doc/mouse/mouse_doc.rst @@ -48,6 +48,12 @@ AutoControl 提供模擬滑鼠操作的功能，包括點擊、定位、捲動 # 在指定座標左鍵點擊 click_mouse("mouse_left", x=500, y=300) + # 雙擊：在同一點連點兩下，間隔 60 毫秒 + click_mouse("mouse_left", x=500, y=300, clicks=2, interval=0.06) + +Windows 與 X11 依兩次點擊的時間差與位移判定雙擊，所以 ``interval`` 要小於系統的雙擊判定時間 +（Windows 預設 500 毫秒）。macOS 上這些點擊會被當成各自獨立的單擊。 + 游標位置 ======== diff --git a/docs/source/Zh/doc/new_features/v116_features_doc.rst b/docs/source/Zh/doc/new_features/v116_features_doc.rst index bd958ffa5..8c779a5df 100644 --- a/docs/source/Zh/doc/new_features/v116_features_doc.rst +++ b/docs/source/Zh/doc/new_features/v116_features_doc.rst @@ -25,11 +25,13 @@ ``plan_path`` 以 ``per_segment_steps`` 個緩動步驟對每個相鄰點對插值(``easing`` 為 ``path_easings()`` 中任一 名稱——``linear`` / ``ease_in_out_quad`` / ``ease_out_cubic`` / ``ease_in_cubic``),且不重複共用的交接點。 ``move_along_path`` 沿路徑發出移動事件;``drag_path`` 在第一個路徑點按下、移動經過整段路徑、在最後一點放開—— -用於多停靠點拖曳。兩者皆可傳入 ``sink`` 以供無頭測試。 +用於多停靠點拖曳。兩者皆可傳入 ``sink`` 以供無頭測試。``drag_path`` 也接受 ``tween_drag`` 的節奏參數 +``step_delay_s``(每次移動後)與 ``settle_s``(按下前、按下後、放開前),而且和它一樣,任何一步丟出例外時會在 +游標停下的位置放開按鍵。 執行器命令 ---------- ``AC_move_along_path`` 與 ``AC_drag_path`` 接受 ``waypoints``(JSON ``[[x, y], ...]`` 列表)以及 ``easing`` / -``per_segment_steps``(拖曳另含 ``button``)。兩者皆以 MCP 工具(``ac_move_along_path`` / ``ac_drag_path``) +``per_segment_steps``(拖曳另含 ``button``、``step_delay_s`` 與 ``settle_s``)。兩者皆以 MCP 工具(``ac_move_along_path`` / ``ac_drag_path``) 以及 Script Builder 中 **Mouse** 分類下的命令提供。 diff --git a/docs/source/Zh/doc/new_features/v29_features_doc.rst b/docs/source/Zh/doc/new_features/v29_features_doc.rst index 728c8c4ad..40b908f9e 100644 --- a/docs/source/Zh/doc/new_features/v29_features_doc.rst +++ b/docs/source/Zh/doc/new_features/v29_features_doc.rst @@ -26,3 +26,11 @@ ``ease_in_out_quad`` / ``ease_out_cubic`` / ``ease_in_cubic``(見 :func:`easing_names`)。對應 ``AC_tween_drag`` / ``ac_tween_drag`` (``start`` / ``end`` 以 ``[x, y]`` 表示)。 + +節奏控制:有些應用程式(檔案總管、繪圖軟體、遊戲)靠游標的移動判定拖曳,而不是點擊:: + + tween_drag((0, 0), (300, 200), steps=24, step_delay_s=0.012, settle_s=0.08) + +``step_delay_s`` 是每次移動後停多久;``settle_s`` 是按下前先停在起點、按下後、放開前各停多久。 +兩者預設 0(不停),必須是有限的非負數。任何一步丟出例外時,按鍵會在 ``finally`` 裡於游標 +最後到達的位置放開,而不是終點,原本的例外照樣往外丟。 diff --git a/docs/source/Zh/doc/ocr_backends/ocr_backends_doc.rst b/docs/source/Zh/doc/ocr_backends/ocr_backends_doc.rst index df1bae4db..e33752353 100644 --- a/docs/source/Zh/doc/ocr_backends/ocr_backends_doc.rst +++ b/docs/source/Zh/doc/ocr_backends/ocr_backends_doc.rst @@ -86,15 +86,28 @@ AutoControl 提供三個可插拔的 OCR 引擎，共用同一組公開 API。 Tesseract 安裝注意 ================== -Windows 必須另外安裝 Tesseract 執行檔（UB-Mannheim build 最常見）。 -若 ``tesseract.exe`` 不在 ``PATH`` 上:: +Windows 必須另外安裝 Tesseract 執行檔（UB-Mannheim build 最常見），安裝程式不會把自己加進 +``PATH``。``find_tesseract_cmd()`` 依序找 ``$TESSERACT_CMD``（指向存在的檔案時）、``PATH``， +最後是安裝程式的預設資料夾（Windows 的 ``Program Files``、``Program Files (x86)`` 與個人的 +``%LOCALAPPDATA%\\Programs``；macOS 的 Homebrew、``/usr/local`` 與 MacPorts；其他平台的 +``/usr/bin`` 與 ``/usr/local/bin``）。它只負責找，要讓 OCR 用上就把結果交給 +``set_tesseract_cmd``:: - from je_auto_control import set_tesseract_cmd - set_tesseract_cmd(r"C:\Program Files\Tesseract-OCR\tesseract.exe") + from je_auto_control import find_tesseract_cmd, set_tesseract_cmd + + command = find_tesseract_cmd() + if command: + set_tesseract_cmd(command) 語言檔（``*.traineddata``）放在 ``Tesseract-OCR\\tessdata\\`` 底下， 要哪個語言就從 `tessdata GitHub repo -`_ 抓對應檔案複製進去。 +`_ 抓對應檔案複製進去。那個資料夾在 Windows 要 +系統管理員權限才寫得進去；想把語言檔放在可寫的地方，就把引擎指過去。這會設定本行程的 +``TESSDATA_PREFIX``，傳 ``None`` 則恢復引擎預設:: + + from je_auto_control import set_tessdata_dir + + set_tessdata_dir("ocr_tessdata") # 回傳解析後的絕對路徑 EasyOCR / PaddleOCR 安裝注意 ============================ @@ -128,12 +141,32 @@ OCR 也透過 executor 對 JSON 動作檔開放:: {"command": "AC_locate_text", "text": "登入", "lang": "chi_tra"} {"command": "AC_click_text", "text": "確定", "backend": "easyocr"} + {"command": "AC_ocr_status"} + {"command": "AC_ocr_languages"} -兩個指令的參數與 Python 端完全一致。 +這些指令的參數與 Python 端完全一致。``AC_ocr_status`` 回傳 ``{"ok": ..., "reason": ...}``， +``AC_ocr_languages`` 回傳 ``{"languages": [...]}``，問不到引擎時是 ``null``。 診斷 ==== +要問 Tesseract 在這裡能不能用、不能用時缺哪一塊:: + + from je_auto_control import ocr_languages, ocr_status + + ok, reason = ocr_status() + # reason："ready"、"missing_package"（沒有 pytesseract）、"missing_engine" + # （找不到設定的執行檔）、"engine_unusable"（執行檔跑不起來），或 + # "no_language_data"（跑起來了，但一個語言都沒列出來） + + languages = ocr_languages() + # ["chi_tra", "eng", ...] 已安裝的語言代碼，已排序 + # [] 引擎回答了：完全沒有語言資料 + # None 問不到引擎 + +``ocr_status`` 檢查的是 OCR 實際會用的執行檔，不改動任何設定。``ocr_languages`` 每次都重新問 +引擎，所以行程執行中補上的語言檔也看得到。``None`` 與 ``[]`` 要分開處理，兩者該有相反的處置。 + 要確認當前環境有哪些後端可用:: from je_auto_control.utils.ocr.backends import get_backend diff --git a/docs/updates/2026-10.md b/docs/updates/2026-10.md new file mode 100644 index 000000000..ece4fa30c --- /dev/null +++ b/docs/updates/2026-10.md @@ -0,0 +1,58 @@ +# 2026-10 update log + +Index and query commands: [README.md](README.md). New entries go at the end. + +--- + +## U-20261001-03 · 2026-10-01 · Tesseract setup helpers: find the engine, set tessdata, report status and languages · #feature #ocr #contract + +Jeffrey_RPA's `_gui_control.py` imported `pytesseract` itself to list languages and check the engine, and wrote `pytesseract.pytesseract.tesseract_cmd` directly, so that module global had two writers. The library now answers those questions itself and `TesseractBackend.set_cmd` stays the only writer. + +- **New module** `utils/ocr/tesseract_setup.py`, exported from the facade and `utils/ocr`: + - `find_tesseract_cmd() -> Optional[str]`: `$TESSERACT_CMD` when it names a file, then `PATH`, then the installers' default folders (Windows `Program Files`, `Program Files (x86)`, `%LOCALAPPDATA%\Programs`; Homebrew, `/usr/local`, MacPorts; `/usr/bin`, `/usr/local/bin`). Looks only. + - `set_tessdata_dir(path) -> Optional[str]`: realpath, must be a directory, sets `TESSDATA_PREFIX`; `None` removes it. + - `ocr_languages() -> Optional[List[str]]`: sorted codes, `[]` when the engine answered with none, `None` when it could not be asked. Never cached. + - `ocr_status() -> OCRStatus(ok, reason)`: `missing_package`, `missing_engine` (the configured command is neither on `PATH` nor a file), `engine_unusable`, `no_language_data` (only for an answered empty list), else `ready`. Checks the command OCR would use and configures nothing. +- **Backend**: `TesseractBackend.cmd`, `version()` and `languages()`, each raising `OCRBackendNotAvailableError` when the engine cannot run. `version()` also catches the `SystemExit` pytesseract raises for a version string it cannot parse, which would otherwise end the calling process. +- **Surfaces**: `AC_ocr_status` → `{ok, reason}`, `AC_ocr_languages` → `{languages}` (`null` when unknown), the matching `ac_ocr_status` / `ac_ocr_languages` MCP tools (read-only) and two Script Builder entries. 776 commands, 679 MCP tools. +- **Real engine** (this host, read-only probe): `ocr_status()` gave `missing_engine` before configuration; `find_tesseract_cmd()` found `C:\Program Files\Tesseract-OCR\tesseract.exe` (not on `PATH`); after `set_tesseract_cmd` it gave `ready`, `ocr_languages()` listed `eng`, `osd`, and after `set_tessdata_dir` on a folder with three language files it listed those three. +- **Tests**: new `test_ocr_tesseract_setup.py` (27 tests, fake pytesseract): `None` versus `[]`, no caching, each status reason including `SystemExit`, one writer for the command, tessdata validation, every lookup step, the executor, MCP and Script Builder wiring. `test_cross_project_contracts.py` pins the four names for Jeffrey_RPA. +- **Docs**: `docs/source/{Eng,Zh}/doc/ocr_backends/ocr_backends_doc.rst` (setup notes, JSON surface, diagnostics), `README.md` and both translations (install note, OCR row, counts), `architecture.md` §6, `architecture_explore.md` (OCR rows, counts, re-measured lines), `je_auto_control/actions.pyi` regenerated (which also picks up the drifted signatures of `AC_queue_complete`, `AC_queue_fail` and `AC_shell_command`, and re-escapes one docstring quote), `CHANGELOG.md`. +- **Open items**: none. + +## U-20261001-04 · 2026-10-01 · click_mouse(clicks=, interval=): a double-click in one call · #feature #input + +Jeffrey_RPA double-clicked with two `click_mouse` calls and its own sleep. The wrapper now does it in one call. + +- **Signature**: `click_mouse(mouse_keycode, x=None, y=None, clicks=1, interval=0.0)` in `wrapper/auto_control_mouse.py`. The point is resolved once and every click lands on it; `time.sleep(interval)` runs only between clicks and only when `interval` is non-zero. Defaults are the old single click, and a single click records the same `click_mouse` action as before (the `clicks` / `interval` keys appear only when used). +- **Validation** (`_click_plan`, raising `AutoControlMouseException` before any click): `clicks` is an int or integer string, at least 1, never a `bool` or a float (a float would be truncated silently); `interval` is finite and at least 0 (NaN passes a plain `< 0` test). +- **Surfaces**: `AC_click_mouse` is the wrapper itself, so action files get both parameters; the `ac_click_mouse` MCP tool and its handler, the MCP fake backend (one recorded click per click) and the Script Builder fields follow. `je_auto_control/actions.pyi` regenerated. +- **macOS**: apps there read a click count the osx backend never sets, so `clicks=2` arrives as two single clicks. Documented in the docstring and docs, and recorded in `Progress.md`. +- **Tests**: new `test_click_mouse_multi.py` (22 tests, stand-in backend and clock): the unchanged default and its record, the double-click and its single pause, no pause at `interval=0`, macOS argument order on every click, each refused `clicks` / `interval` value clicking nothing, and the executor, MCP handler, fake backend, MCP schema and Script Builder wiring. +- **Docs**: `docs/source/API/wrapper/mouse.rst`, `docs/source/{Eng,Zh}/doc/mouse/mouse_doc.rst`, the Mouse row of `README.md` and both translations, `CHANGELOG.md`; `architecture.md` §6 lists the two keywords for Jeffrey_RPA and `test_cross_project_contracts.py` pins them. +- **Open items**: `Progress.md` "macOS 上 `click_mouse(clicks=2)` 不是雙擊". + +## U-20261001-05 · 2026-10-01 · Drag pacing for tween_drag / drag_path; an aborted drag lets go where it stopped · #feature #bugfix #input + +Jeffrey_RPA kept its own drag loop because the library's drags moved as fast as the events could be sent, which apps that judge a drag by the pointer's motion read as a click, and because an aborted drag released the button at the drop target. + +- **Shared sequence**: `utils/tween_drag/tween_drag.py` `_drag_through(points, button, dispatch, step_delay_s, settle_s)` now holds the press / move / release for both `tween_drag` and `mouse_path.drag_path`, which previously had two copies of the same loop. +- **Pacing**: `step_delay_s` rests after each move; `settle_s` (when non-zero) moves to the start and rests before the press, then rests after the press and before the release. Both default to 0, where the events are exactly press, moves, release with no sleep, as before. NaN, infinities, negatives and non-numbers raise `ValueError` before anything is dispatched (a NaN reaching `time.sleep` would raise with the button held). +- **Release on failure**: after the press, a `held` flag and a `finally` (the pattern of `hotkey` / `type_keyboard` and `_release_still_held`) release the button at the last point the pointer reached when any step raises, `KeyboardInterrupt` included. It used to release at the last point of the path, completing a drop the drag never reached. The cleanup release never raises over the original error (logged instead); a failing final release propagates after one cleanup retry, as `hotkey` does. A failed press releases nothing. +- **Surfaces**: the `AC_tween_drag` / `AC_drag_path` adapters, the `ac_tween_drag` / `ac_drag_path` MCP tools and handlers (`_handlers_executor_bridge.py` kept its line count, as its `Progress.md` ceiling requires) and the Script Builder fields take both. `je_auto_control/actions.pyi` regenerated. +- **Signatures**: `tween_drag(start, end, *, steps=30, easing="ease_in_out_quad", button="mouse_left", sink=None, step_delay_s=0.0, settle_s=0.0)` and `drag_path(waypoints, *, button="mouse_left", easing="linear", per_segment_steps=20, sink=None, step_delay_s=0.0, settle_s=0.0)`. `tween_drag` now has eight parameters, one over the `CLAUDE.md` review limit; all are keyword-only after `start` / `end`, and `wait_for_text` already has eight. +- **Tests**: new `test_drag_pacing.py` (25 tests, recording sink and clock): unchanged defaults for both drags, the pause order for each setting, every refused value sending nothing, release at the last reached point (mid-path, first move, interrupted pause), no release after a failed press, a failing cleanup not hiding the original error, the final-release retry, and the executor, MCP and Script Builder wiring. The existing drag tests pass unchanged. +- **Docs**: `docs/source/{Eng,Zh}/doc/new_features/v29_features_doc.rst` and `v116_features_doc.rst`, the Mouse row of `README.md` and both translations, `architecture.md` §6 (with `test_cross_project_contracts.py` pinning the keywords), `architecture_explore.md` (tween_drag row, the `finally` section, re-measured lines), `CHANGELOG.md` (Added and Fixed). +- **Open items**: none. + +## U-20261001-06 · 2026-10-01 · Windows OEM keys, browser_home, launch_app2 and key aliases; canonical reverse lookup · #feature #input #contract + +Jeffrey_RPA kept its own `KEY_ALIASES` (40 friendly names) and `_EXTRA_KEY_CODES` (16 keys the library could not name: the OEM keys, `browser_home`, and `launch_app2`, which existed only upper-case) and converted names to codes before every call. The Windows table now has all of them, with the same names and codes. + +- **Table** (`wrapper/_platform_windows.py`, constants in `windows/core/utils/win32_vk.py`): `oem_1` .. `oem_8`, `oem_102`, `oem_plus`, `oem_comma`, `oem_minus`, `oem_period`, `oem_clear`, `browser_home` (0xAC) and `launch_app2` (0xB7) beside the old `LAUNCH_APP2`. Codes checked against Microsoft's "Virtual-Key Codes". Layout-dependent OEM keys keep Microsoft's names; a character name such as `semicolon` would be wrong on other layouts. +- **Aliases**: one map, `keyboard_key_aliases` (alias → canonical name; the `"ctrl"` entry moved there from the table literal), merged into the table after the canonical names. Every platform module and `platform_wrapper` publish the map (empty off Windows). 230 names on Windows, 40 of them aliases (was 175 names). +- **Reverse lookup**: new `keyboard_key_name(keycode) -> Optional[str]` in `wrapper/auto_control_keyboard.py`, exported from the facade. Aliases never answer. Among the rest a typable name (`[a-z0-9_]+`) wins, then the shortest, then alphabetical order. For every code the old table already named, it returns exactly what a shortest-lower-case lookup over the pre-alias table returned (`control`, `return`, `escape`, `lwin`, `snapshot`, `num0`, `down`, `a`, ...). The new keys get their new names (`launch_app2`, not `LAUNCH_APP2`). +- **Jeffrey_RPA's current `_vk_to_name`** takes the shortest name from the raw table and then maps it through its own aliases. Simulated over VK 1-254 against both tables, it changes for one key: VK 0x2C writes `prtsc` instead of `printscreen`. Both names are the same key and validate and replay identically. Its modifier keys never reach the table. Calling `keyboard_key_name`, or skipping `keyboard_key_aliases`, removes the difference. Its `test_every_code_the_library_can_name_can_also_be_written` positive control ("the real table still has a key named only in upper case") stops holding once `launch_app2` exists in lower case. That test's own message says to revisit the premise in that case. +- **Tests**: new `test_key_names_and_aliases.py` (68 tests; the table checks run on Windows): Microsoft codes for the 16 added keys, the upper-case spelling kept, every alias resolving through `_resolve_keycode` to its target's code, no alias pointing at another alias or at a layout-dependent key, the pre-alias canonical name for every aliased code, an alias-free answer for every code in the table, the preference order on a synthetic table (runs everywhere), and the facade export. `test_cross_project_contracts.py` pins `keyboard_key_name`, `keyboard_key_aliases` and the names Jeffrey_RPA uses. +- **Docs**: `docs/source/API/special/keyboard_keys.rst` (the Windows list rewritten to names that exist, with aliases, OEM and launch keys; it listed `print_screen`, `lalt` and `numpad0`, which did not exist), `docs/source/API/wrapper/keyboard.rst`, `docs/source/{Eng,Zh}/doc/keyboard/keyboard_doc.rst`, the Keyboard row of `README.md` and both translations, `architecture.md` §6, `architecture_explore.md` (two rows, public-API and test counts re-measured, lines), `CHANGELOG.md` (Added and Changed). +- **Open items**: none. diff --git a/docs/updates/README.md b/docs/updates/README.md index 1e44767a8..490b2e8b3 100644 --- a/docs/updates/README.md +++ b/docs/updates/README.md @@ -58,6 +58,10 @@ In the same commit: delete the item from `Progress.md`, add a `#done` entry here | ID | Date | Title | Tags | Batch | |---|---|---|---|---| +| U-20261001-06 | 2026-10-01 | Windows OEM keys, browser_home, launch_app2 and key aliases; canonical reverse lookup | #feature #input #contract | [2026-10](2026-10.md) | +| U-20261001-05 | 2026-10-01 | Drag pacing for tween_drag / drag_path; an aborted drag lets go where it stopped | #feature #bugfix #input | [2026-10](2026-10.md) | +| U-20261001-04 | 2026-10-01 | click_mouse(clicks=, interval=): a double-click in one call | #feature #input | [2026-10](2026-10.md) | +| U-20261001-03 | 2026-10-01 | Tesseract setup helpers: find the engine, set tessdata, report status and languages | #feature #ocr #contract | [2026-10](2026-10.md) | | U-20260924-57 | 2026-09-24 | Type-check the SBOM's optional packaging import in CI's bare install | #ci #typing | [2026-09](2026-09.md) | | U-20260924-56 | 2026-09-24 | Emergency stop wakes a sleeping main thread on Linux and macOS too | #bugfix #ci | [2026-09](2026-09.md) | | U-20260924-55 | 2026-09-24 | Image analysis and packaging: Machado CVD simulation, square-aware widget classes, repair that does not re-act, schema-valid registry manifests | #bugfix #audit | [2026-09](2026-09.md) | @@ -216,6 +220,7 @@ In the same commit: delete the item from `Progress.md`, add a `#done` entry here | File | Period | Entries | |---|---|---:| +| [2026-10.md](2026-10.md) | 2026-10 | 4 | | [2026-09.md](2026-09.md) | 2026-09 | 127 | | [2026-08-f.md](2026-08-f.md) | 2026-08 | 1 | | [2026-08-e.md](2026-08-e.md) | 2026-08 | 2 | diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py index 762f70567..c0ca126c8 100644 --- a/je_auto_control/__init__.py +++ b/je_auto_control/__init__.py @@ -927,6 +927,10 @@ locate_text_center, read_text_in_region, set_tesseract_cmd, wait_for_text, ) +# Tesseract engine discovery, configuration and health checks +from je_auto_control.utils.ocr.tesseract_setup import ( + OCRStatus, find_tesseract_cmd, ocr_languages, ocr_status, set_tessdata_dir, +) # Group OCR word boxes into lines / runs (engines box one word at a time) from je_auto_control.utils.ocr.text_span import find_spans, group_lines # Whether input this process sends can actually arrive @@ -1233,6 +1237,7 @@ from je_auto_control.wrapper.auto_control_keyboard import check_key_is_press from je_auto_control.wrapper.auto_control_keyboard import get_keyboard_keys_table from je_auto_control.wrapper.auto_control_keyboard import hotkey +from je_auto_control.wrapper.auto_control_keyboard import keyboard_key_name from je_auto_control.wrapper.auto_control_keyboard import keyboard_keys_table from je_auto_control.wrapper.auto_control_keyboard import press_keyboard_key from je_auto_control.wrapper.auto_control_keyboard import release_keyboard_key @@ -1296,7 +1301,7 @@ def start_autocontrol_gui(*args, **kwargs): "HumanizedMotion", "humanized_path", "move_mouse_humanized", "humanized_key_delays", "type_text_humanized", "keyboard_keys_table", "press_keyboard_key", "release_keyboard_key", "type_keyboard", "check_key_is_press", - "write", "hotkey", "start_exe", "get_keyboard_keys_table", + "write", "hotkey", "start_exe", "get_keyboard_keys_table", "keyboard_key_name", "screen_size", "screenshot", "locate_all_image", "locate_image_center", "locate_and_click", "CriticalExit", "AutoControlException", "AutoControlKeyboardException", "AutoControlMouseException", "AutoControlCantFindKeyException", @@ -1315,6 +1320,8 @@ def start_autocontrol_gui(*args, **kwargs): "TextMatch", "find_text_matches", "locate_text_center", "wait_for_text", "click_text", "set_tesseract_cmd", "read_text_in_region", "find_text_regex", "find_spans", "group_lines", + "OCRStatus", "find_tesseract_cmd", "ocr_languages", "ocr_status", + "set_tessdata_dir", "char_table", "foreground_keyboard_layout", "layout_char_table", "vk_to_char", "input_desktop_available", "input_reaches_system", # Recording editor diff --git a/je_auto_control/actions.pyi b/je_auto_control/actions.pyi index f16ede33f..7d722728e 100644 --- a/je_auto_control/actions.pyi +++ b/je_auto_control/actions.pyi @@ -591,8 +591,14 @@ def AC_classify_lock_transitions(states: Any) -> Dict[str, Any]: def AC_classify_widget(features: Any) -> Dict[str, Any]: """Adapter: map geometric features to a widget type (pure).""" -def AC_click_mouse(mouse_keycode: int | str, x: int | None = ..., y: int | None = ...) -> Tuple[Any, int, int]: - """在指定座標按下並放開滑鼠按鍵""" +def AC_click_mouse( + mouse_keycode: int | str, + x: int | None = ..., + y: int | None = ..., + clicks: int = ..., + interval: float = ..., +) -> Tuple[Any, int, int]: + """在指定座標按下並放開滑鼠按鍵；``clicks=2`` 就是雙擊""" def AC_click_text( target: str, @@ -837,7 +843,7 @@ def AC_delta_observation( max_lines: Any = ..., interactive_only: Any = ..., ) -> Dict[str, Any]: - """Adapter: token-budgeted "what changed" delta between two element frames.""" + """Adapter: token-budgeted \"what changed\" delta between two element frames.""" def AC_describe_screen(app_name: str | None = ...) -> Dict[str, Any]: """Adapter: structured 'where am I' description of the live screen.""" @@ -903,7 +909,14 @@ def AC_dominant_hue_regions( def AC_dominant_pair(pixels: Any) -> Dict[str, Any]: """Adapter: split sampled RGB pixels into fg/bg dominant colours (pure).""" -def AC_drag_path(waypoints: Any, button: str = ..., easing: str = ..., per_segment_steps: Any = ...) -> Dict[str, Any]: +def AC_drag_path( + waypoints: Any, + button: str = ..., + easing: str = ..., + per_segment_steps: Any = ..., + step_delay_s: Any = ..., + settle_s: Any = ..., +) -> Dict[str, Any]: """Adapter: press, drag through a polyline of waypoints, release.""" def AC_drop_files(hwnd: Any, paths: Any, point: Any = ...) -> Dict[str, Any]: @@ -1943,6 +1956,9 @@ def AC_observe_start() -> Dict[str, Any]: def AC_observe_stop() -> Dict[str, Any]: """Adapter: stop the background observer thread.""" +def AC_ocr_languages() -> Dict[str, Any]: + """Executor adapter: installed Tesseract languages; ``None`` when unknown.""" + def AC_ocr_read_structure( region: List[int] | None = ..., lang: str = ..., @@ -1950,6 +1966,9 @@ def AC_ocr_read_structure( ) -> Dict[str, Any]: """Executor adapter: structured OCR (rows / tables / form fields).""" +def AC_ocr_status() -> Dict[str, Any]: + """Executor adapter: whether Tesseract OCR can work here, as a dict.""" + def AC_open_path(target: str, verb: str = ...) -> Dict[str, Any]: """Adapter: open a file with its default app / a URL in the browser.""" @@ -2146,8 +2165,14 @@ def AC_quarantine_remove(name: str) -> Dict[str, Any]: def AC_queue_add(db: str, data: Any, reference: str | None = ..., name: str = ...) -> Dict[str, Any]: """Adapter: enqueue a work item (skips live duplicate references).""" -def AC_queue_complete(db: str, item_id: int, output: Any = ..., name: str = ...) -> Dict[str, Any]: - """Adapter: mark a work item successful.""" +def AC_queue_complete( + db: str, + item_id: int, + output: Any = ..., + name: str = ..., + claim: int | None = ..., +) -> Dict[str, Any]: + """Adapter: mark a work item successful (``claim`` from AC_queue_next).""" def AC_queue_fail( db: str, @@ -2156,6 +2181,7 @@ def AC_queue_fail( kind: str = ..., max_retries: int = ..., name: str = ..., + claim: int | None = ..., ) -> Dict[str, Any]: """Adapter: fail a work item (application errors retry, business don't).""" @@ -2656,7 +2682,7 @@ def AC_shard_suite( ) -> Dict[str, Any]: """Adapter: balance flows into duration-aware shards.""" -def AC_shell_command(shell_command: str | List[str]) -> None: +def AC_shell_command(shell_command: str | List[str] | None = ..., *, command: str | List[str] | None = ...) -> None: """Execute shell command with shell=False.""" def AC_sign_action_file(path: str, key: str | None = ...) -> Dict[str, Any]: @@ -2871,6 +2897,8 @@ def AC_tween_drag( steps: int = ..., easing: str = ..., button: str = ..., + step_delay_s: Any = ..., + settle_s: Any = ..., ) -> Dict[str, Any]: """Adapter: drag along an eased path from start to end.""" diff --git a/je_auto_control/gui/script_builder/command_schema.py b/je_auto_control/gui/script_builder/command_schema.py index d03043c43..b945cf2d3 100644 --- a/je_auto_control/gui/script_builder/command_schema.py +++ b/je_auto_control/gui/script_builder/command_schema.py @@ -78,6 +78,10 @@ def _add_mouse_specs(specs: List[CommandSpec]) -> None: default="mouse_left"), FieldSpec("x", FieldType.INT, optional=True), FieldSpec("y", FieldType.INT, optional=True), + FieldSpec("clicks", FieldType.INT, optional=True, default=1, + min_value=1), + FieldSpec("interval", FieldType.FLOAT, optional=True, default=0.0, + min_value=0.0), ), )) specs.append(CommandSpec( @@ -878,6 +882,18 @@ def _add_ocr_specs(specs: List[CommandSpec]) -> None: default=60.0, min_value=0.0, max_value=100.0), ), )) + specs.extend(( + CommandSpec( + "AC_ocr_status", "OCR", "OCR Engine Status", + fields=(), + description="Whether Tesseract OCR can work here: {ok, reason}.", + ), + CommandSpec( + "AC_ocr_languages", "OCR", "OCR Installed Languages", + fields=(), + description="Installed Tesseract languages; null when the engine cannot be asked.", + ), + )) specs.append(CommandSpec( "AC_read_qr", "OCR", "Read QR Codes", fields=( @@ -2013,6 +2029,10 @@ def _add_misc_specs(specs: List[CommandSpec]) -> None: optional=True, default="ease_in_out_quad"), FieldSpec("button", FieldType.ENUM, choices=_MOUSE_BUTTONS, optional=True, default="mouse_left"), + FieldSpec("step_delay_s", FieldType.FLOAT, optional=True, + default=0.0, min_value=0.0), + FieldSpec("settle_s", FieldType.FLOAT, optional=True, + default=0.0, min_value=0.0), ), description="Drag along an eased path; 'start'/'end' [x,y] via JSON " "view.", @@ -2044,6 +2064,10 @@ def _add_misc_specs(specs: List[CommandSpec]) -> None: optional=True, default="linear"), FieldSpec("per_segment_steps", FieldType.INT, optional=True, default=20), + FieldSpec("step_delay_s", FieldType.FLOAT, optional=True, + default=0.0, min_value=0.0), + FieldSpec("settle_s", FieldType.FLOAT, optional=True, + default=0.0, min_value=0.0), ), description="Press, drag through a polyline of waypoints, release.", )) diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py index 92a8db5b0..a108e71c9 100644 --- a/je_auto_control/utils/executor/action_executor.py +++ b/je_auto_control/utils/executor/action_executor.py @@ -1880,6 +1880,19 @@ def _ocr_find_regex_as_dicts(pattern: str, ] +def _ocr_status() -> Dict[str, Any]: + """Executor adapter: whether Tesseract OCR can work here, as a dict.""" + from je_auto_control.utils.ocr.tesseract_setup import ocr_status + ok, reason = ocr_status() + return {"ok": ok, "reason": reason} + + +def _ocr_languages() -> Dict[str, Any]: + """Executor adapter: installed Tesseract languages; ``None`` when unknown.""" + from je_auto_control.utils.ocr.tesseract_setup import ocr_languages + return {"languages": ocr_languages()} + + def _email_trigger_add(host: str, username: str, password: str, script_path: str, port: Optional[int] = None, @@ -3959,11 +3972,13 @@ def _move_along_path(waypoints: Any, easing: str = "linear", def _drag_path(waypoints: Any, button: str = "mouse_left", easing: str = "linear", - per_segment_steps: Any = 20) -> Dict[str, Any]: + per_segment_steps: Any = 20, step_delay_s: Any = 0.0, + settle_s: Any = 0.0) -> Dict[str, Any]: """Adapter: press, drag through a polyline of waypoints, release.""" from je_auto_control.utils.mouse_path import drag_path return drag_path(_waypoints(waypoints), button=button, easing=easing, - per_segment_steps=int(per_segment_steps)) + per_segment_steps=int(per_segment_steps), + step_delay_s=float(step_delay_s), settle_s=float(settle_s)) def _set_field_text(text: str, clear: str = "select_all", paste: Any = False, @@ -6395,12 +6410,14 @@ def _generate_sop(actions: List[Any], title: str = "Automation Procedure", def _tween_drag(start: List[int], end: List[int], steps: int = 30, easing: str = "ease_in_out_quad", - button: str = "mouse_left") -> Dict[str, Any]: + button: str = "mouse_left", step_delay_s: Any = 0.0, + settle_s: Any = 0.0) -> Dict[str, Any]: """Adapter: drag along an eased path from start to end.""" from je_auto_control.utils.tween_drag import tween_drag result = tween_drag((int(start[0]), int(start[1])), (int(end[0]), int(end[1])), steps=int(steps), - easing=easing, button=button) + easing=easing, button=button, + step_delay_s=float(step_delay_s), settle_s=float(settle_s)) return {"points": result["points"]} @@ -7166,6 +7183,8 @@ def __init__(self): "AC_click_text": ocr_click_text, "AC_read_text_in_region": _ocr_read_region_as_dicts, "AC_find_text_regex": _ocr_find_regex_as_dicts, + "AC_ocr_status": _ocr_status, + "AC_ocr_languages": _ocr_languages, # Window management "AC_list_windows": list_windows, diff --git a/je_auto_control/utils/mcp_server/fake_backend.py b/je_auto_control/utils/mcp_server/fake_backend.py index e1bddb331..b016a0c46 100644 --- a/je_auto_control/utils/mcp_server/fake_backend.py +++ b/je_auto_control/utils/mcp_server/fake_backend.py @@ -55,11 +55,14 @@ def _fake_set_mouse_position(x: int, y: int) -> Tuple[int, int]: def _fake_click_mouse(mouse_keycode: Any, x: Any = None, - y: Any = None) -> Tuple[Any, int, int]: + y: Any = None, clicks: int = 1, + interval: float = 0.0) -> Tuple[Any, int, int]: + del interval # the fake never waits cx, cy = _STATE.cursor if x is None or y is None else (int(x), int(y)) with _STATE_LOCK: _STATE.cursor = (cx, cy) - _STATE.mouse_actions.append(("click", mouse_keycode, cx, cy)) + for _ in range(int(clicks)): + _STATE.mouse_actions.append(("click", mouse_keycode, cx, cy)) return mouse_keycode, cx, cy diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py index d108faaeb..52f6683dd 100644 --- a/je_auto_control/utils/mcp_server/tools/_factories.py +++ b/je_auto_control/utils/mcp_server/tools/_factories.py @@ -30,12 +30,16 @@ def mouse_tools() -> List[MCPTool]: name="ac_click_mouse", description=("Click a mouse button at (x, y). " "mouse_keycode: mouse_left, mouse_right, mouse_middle. " - "If x/y are omitted, clicks at the current cursor."), + "If x/y are omitted, clicks at the current cursor. " + "clicks=2 double-clicks; interval is the pause in " + "seconds between clicks."), input_schema=schema({ "mouse_keycode": {"type": "string", "description": "mouse_left | mouse_right | mouse_middle"}, "x": {"type": "integer"}, "y": {"type": "integer"}, + "clicks": {"type": "integer", "minimum": 1}, + "interval": {"type": "number", "minimum": 0}, }), handler=h_input.click_mouse, annotations=DESTRUCTIVE, @@ -257,6 +261,24 @@ def image_and_ocr_tools() -> List[MCPTool]: handler=h_screen.click_text, annotations=DESTRUCTIVE, ), + MCPTool( + name="ac_ocr_status", + description=("Report whether Tesseract OCR can work here: " + "{ok, reason}. reason is ready | missing_package | " + "missing_engine | engine_unusable | no_language_data."), + input_schema=schema({}), + handler=h_screen.ocr_status, + annotations=READ_ONLY, + ), + MCPTool( + name="ac_ocr_languages", + description=("List the installed Tesseract language codes: " + "{languages}. languages is null when the engine " + "cannot be asked and [] when it has no language data."), + input_schema=schema({}), + handler=h_screen.ocr_languages, + annotations=READ_ONLY, + ), ] @@ -3506,13 +3528,18 @@ def tween_drag_tools() -> List[MCPTool]: name="ac_tween_drag", description=("Drag from 'start' [x,y] to 'end' [x,y] along an " "eased path (easing: linear / ease_in_out_quad / " - "ease_out_cubic / ease_in_cubic). Returns {points}."), + "ease_out_cubic / ease_in_cubic). 'step_delay_s' " + "pauses after each move, 'settle_s' before the " + "press, after it and before the release. Returns " + "{points}."), input_schema=schema({ "start": {"type": "array", "items": {"type": "integer"}}, "end": {"type": "array", "items": {"type": "integer"}}, "steps": {"type": "integer"}, "easing": {"type": "string"}, - "button": {"type": "string"}}, + "button": {"type": "string"}, + "step_delay_s": {"type": "number", "minimum": 0}, + "settle_s": {"type": "number", "minimum": 0}}, required=["start", "end"]), handler=h.tween_drag, annotations=DESTRUCTIVE, @@ -5562,14 +5589,18 @@ def mouse_path_tools() -> List[MCPTool]: name="ac_drag_path", description=("Press at the first of 'waypoints' ([[x,y],...]), drag " "through them, release at the last. 'button', 'easing', " - "'per_segment_steps'. Returns {points, path}."), + "'per_segment_steps'; 'step_delay_s' pauses after each " + "move, 'settle_s' before the press, after it and before " + "the release. Returns {points, path}."), input_schema=schema({ "waypoints": {"type": "array", "items": {"type": "array", "items": {"type": "integer"}}}, "button": {"type": "string"}, "easing": {"type": "string"}, - "per_segment_steps": {"type": "integer"}}, + "per_segment_steps": {"type": "integer"}, + "step_delay_s": {"type": "number", "minimum": 0}, + "settle_s": {"type": "number", "minimum": 0}}, required=["waypoints"]), handler=h_exec.drag_path, annotations=DESTRUCTIVE, diff --git a/je_auto_control/utils/mcp_server/tools/_handlers.py b/je_auto_control/utils/mcp_server/tools/_handlers.py index 06232ff6b..2eeb88b4b 100644 --- a/je_auto_control/utils/mcp_server/tools/_handlers.py +++ b/je_auto_control/utils/mcp_server/tools/_handlers.py @@ -250,10 +250,12 @@ def generate_sop(actions, title="Automation Procedure", path=None): def tween_drag(start, end, steps=30, easing="ease_in_out_quad", - button="mouse_left"): + button="mouse_left", step_delay_s=0.0, settle_s=0.0): from je_auto_control.utils.tween_drag import tween_drag as _td return {"points": _td(tuple(start), tuple(end), steps=int(steps), - easing=easing, button=button)["points"]} + easing=easing, button=button, + step_delay_s=float(step_delay_s), + settle_s=float(settle_s))["points"]} def fuzzy_ratio(left, right, ignore_case=True): diff --git a/je_auto_control/utils/mcp_server/tools/_handlers_executor_bridge.py b/je_auto_control/utils/mcp_server/tools/_handlers_executor_bridge.py index 84b84e125..b85746a01 100644 --- a/je_auto_control/utils/mcp_server/tools/_handlers_executor_bridge.py +++ b/je_auto_control/utils/mcp_server/tools/_handlers_executor_bridge.py @@ -514,9 +514,9 @@ def move_along_path(waypoints, easing="linear", per_segment_steps=20): def drag_path(waypoints, button="mouse_left", easing="linear", - per_segment_steps=20): + per_segment_steps=20, step_delay_s=0.0, settle_s=0.0): from je_auto_control.utils.executor.action_executor import _drag_path - return _drag_path(waypoints, button, easing, per_segment_steps) + return _drag_path(waypoints, button, easing, per_segment_steps, step_delay_s, settle_s) def set_field_text(text, clear="select_all", paste=False, modifier="ctrl"): diff --git a/je_auto_control/utils/mcp_server/tools/_handlers_input.py b/je_auto_control/utils/mcp_server/tools/_handlers_input.py index 0a8e6ba45..5cb5cdd47 100644 --- a/je_auto_control/utils/mcp_server/tools/_handlers_input.py +++ b/je_auto_control/utils/mcp_server/tools/_handlers_input.py @@ -11,9 +11,12 @@ def click_mouse(mouse_keycode: str = "mouse_left", x: Optional[int] = None, - y: Optional[int] = None) -> List[Any]: + y: Optional[int] = None, + clicks: int = 1, + interval: float = 0.0) -> List[Any]: + """Click ``clicks`` times at (x, y); return ``[keycode, x, y]``.""" from je_auto_control.wrapper.auto_control_mouse import click_mouse as _click - keycode, click_x, click_y = _click(mouse_keycode, x, y) + keycode, click_x, click_y = _click(mouse_keycode, x, y, clicks=clicks, interval=interval) # Real wrapper resolves the string keycode to an int via the keys table; # the fake backend keeps it as a string. Pass through whatever we got. resolved = int(keycode) if isinstance(keycode, int) else keycode diff --git a/je_auto_control/utils/mcp_server/tools/_handlers_screen.py b/je_auto_control/utils/mcp_server/tools/_handlers_screen.py index 9af2c3b14..022cffe85 100644 --- a/je_auto_control/utils/mcp_server/tools/_handlers_screen.py +++ b/je_auto_control/utils/mcp_server/tools/_handlers_screen.py @@ -303,3 +303,16 @@ def click_text(text: str, cx, cy = _click(text, mouse_keycode=mouse_keycode, region=region, min_confidence=float(min_confidence)) return [int(cx), int(cy)] + + +def ocr_status() -> Dict[str, Any]: + """Whether Tesseract OCR can work here, as ``{ok, reason}``.""" + from je_auto_control.utils.ocr.tesseract_setup import ocr_status as _status + ok, reason = _status() + return {"ok": ok, "reason": reason} + + +def ocr_languages() -> Dict[str, Any]: + """Installed Tesseract languages as ``{languages}``; ``None`` when unknown.""" + from je_auto_control.utils.ocr.tesseract_setup import ocr_languages as _languages + return {"languages": _languages()} diff --git a/je_auto_control/utils/mouse_path/mouse_path.py b/je_auto_control/utils/mouse_path/mouse_path.py index 539211bf1..eff97bd85 100644 --- a/je_auto_control/utils/mouse_path/mouse_path.py +++ b/je_auto_control/utils/mouse_path/mouse_path.py @@ -12,7 +12,7 @@ """ from typing import Any, Callable, Dict, List, Optional, Sequence -from je_auto_control.utils.tween_drag.tween_drag import easing_names, tween_points +from je_auto_control.utils.tween_drag.tween_drag import _drag_through, easing_names, tween_points Point = Sequence[int] Sink = Callable[[Dict[str, Any]], None] @@ -77,21 +77,20 @@ def move_along_path(waypoints: Sequence[Point], *, easing: str = "linear", def drag_path(waypoints: Sequence[Point], *, button: str = "mouse_left", easing: str = "linear", per_segment_steps: int = 20, - sink: Optional[Sink] = None) -> Dict[str, Any]: - """Press at the first waypoint, move through the path, release at the last.""" + sink: Optional[Sink] = None, step_delay_s: float = 0.0, + settle_s: float = 0.0) -> Dict[str, Any]: + """Press at the first waypoint, move through the path, release at the last. + + Pacing and failure handling are ``tween_drag``'s: ``step_delay_s`` rests + after each move, ``settle_s`` before the press, after it and before the + release (default 0: no pause), and a step that raises releases the + button where the pointer stopped. + """ points = plan_path(waypoints, easing=easing, per_segment_steps=per_segment_steps) if not points: return {"points": 0, "path": points} - dispatch = sink or _default_sink - first, last = points[0], points[-1] - dispatch({"op": "press", "button": button, "x": first[0], "y": first[1]}) - try: - for x, y in points: - dispatch({"op": "move", "x": x, "y": y}) - finally: - # A failed move used to leave the button held down. - dispatch({"op": "release", "button": button, "x": last[0], "y": last[1]}) + _drag_through(points, button, sink or _default_sink, step_delay_s, settle_s) return {"points": len(points), "path": points} diff --git a/je_auto_control/utils/ocr/__init__.py b/je_auto_control/utils/ocr/__init__.py index d264b0abe..01605869f 100644 --- a/je_auto_control/utils/ocr/__init__.py +++ b/je_auto_control/utils/ocr/__init__.py @@ -3,9 +3,13 @@ TextMatch, click_text, find_text_matches, locate_text_center, set_tesseract_cmd, wait_for_text, ) +from je_auto_control.utils.ocr.tesseract_setup import ( + OCRStatus, find_tesseract_cmd, ocr_languages, ocr_status, set_tessdata_dir, +) from je_auto_control.utils.ocr.text_span import find_spans, group_lines __all__ = [ - "TextMatch", "click_text", "find_spans", "find_text_matches", "group_lines", - "locate_text_center", "set_tesseract_cmd", "wait_for_text", + "OCRStatus", "TextMatch", "click_text", "find_spans", "find_tesseract_cmd", + "find_text_matches", "group_lines", "locate_text_center", "ocr_languages", + "ocr_status", "set_tessdata_dir", "set_tesseract_cmd", "wait_for_text", ] diff --git a/je_auto_control/utils/ocr/backends/tesseract_backend.py b/je_auto_control/utils/ocr/backends/tesseract_backend.py index 21f95a1b0..63554e289 100644 --- a/je_auto_control/utils/ocr/backends/tesseract_backend.py +++ b/je_auto_control/utils/ocr/backends/tesseract_backend.py @@ -1,6 +1,11 @@ -"""Tesseract OCR backend (preserves the original ``ocr_engine`` behaviour).""" +"""Tesseract OCR backend (preserves the original ``ocr_engine`` behaviour). + +This is the only module that imports :mod:`pytesseract` and the only writer of +its ``tesseract_cmd`` global (:meth:`TesseractBackend.set_cmd`). +""" from __future__ import annotations +from subprocess import SubprocessError # nosec B404 # reason: exception type only from typing import List from je_auto_control.utils.ocr.backends.base import OCRBackendNotAvailableError @@ -46,6 +51,49 @@ def set_cmd(self, path: str) -> None: pt = _load() pt.pytesseract.tesseract_cmd = path + @property + def cmd(self) -> str: + """The engine command OCR runs: a path, or a name looked up on ``PATH``. + + Raises :class:`OCRBackendNotAvailableError` without ``pytesseract``. + """ + return str(_load().pytesseract.tesseract_cmd) + + def version(self) -> str: + """Return the engine's version string by running it. + + Raises :class:`OCRBackendNotAvailableError` when the engine cannot run: + missing, not executable, exiting non-zero, or printing a version + ``pytesseract`` cannot parse. ``pytesseract`` reports that last case + with ``SystemExit``, which is caught here so a broken binary cannot end + the calling process. + """ + pt = _load() + try: + return str(pt.get_tesseract_version()) + except (OSError, RuntimeError, ValueError, SubprocessError) as error: + raise OCRBackendNotAvailableError(f"Tesseract cannot run: {error!r}") from error + except SystemExit as error: # NOSONAR python:S5754 # reason: see docstring + raise OCRBackendNotAvailableError( + f"Tesseract printed a version pytesseract cannot read: {error}") from error + + def languages(self) -> List[str]: + """Return the installed language codes, sorted, by asking the engine. + + Asked afresh on every call (``pytesseract`` caches only when told to), + so language data added while the process runs is seen. An empty list + is a real answer: the engine ran and found no language data. Names + ``pytesseract`` filters out (upper-case script models such as + ``Latin``) are not listed. Raises :class:`OCRBackendNotAvailableError` + when the engine cannot be asked. + """ + pt = _load() + try: + return sorted(pt.get_languages(config="")) + except (OSError, RuntimeError, ValueError) as error: + raise OCRBackendNotAvailableError( + f"Tesseract cannot list its languages: {error!r}") from error + def image_to_matches(self, image, lang: str, min_confidence: float) -> List: # Import inside to avoid circular import at module load. diff --git a/je_auto_control/utils/ocr/tesseract_setup.py b/je_auto_control/utils/ocr/tesseract_setup.py new file mode 100644 index 000000000..1aaea6a4f --- /dev/null +++ b/je_auto_control/utils/ocr/tesseract_setup.py @@ -0,0 +1,175 @@ +"""Tesseract engine discovery, configuration and health checks. + +The OCR calls (``find_text_matches`` and friends) only *run* the engine. These +helpers answer what a host needs to know before the first call -- where the +executable is, which language data is installed, whether the engine runs at +all -- so the host never imports ``pytesseract`` itself or writes its module +globals behind this package's back. The engine command still has exactly one +writer, :meth:`TesseractBackend.set_cmd` (``set_tesseract_cmd``). + +Typical start-up, for an engine that may not be on ``PATH``:: + + command = find_tesseract_cmd() + if command: + set_tesseract_cmd(command) + ok, reason = ocr_status() + +Imports no ``PySide6``. +""" +import os +import shutil +from typing import List, NamedTuple, Optional + +from je_auto_control.utils.exception.exceptions import AutoControlActionException +from je_auto_control.utils.logging.logging_instance import autocontrol_logger +from je_auto_control.utils.ocr.backends.base import OCRBackendNotAvailableError +from je_auto_control.utils.ocr.backends.tesseract_backend import TesseractBackend +from je_auto_control.utils.platform_id import is_macos, is_windows + +#: Environment variable naming the engine executable; read by +#: :func:`find_tesseract_cmd` before ``PATH``. +TESSERACT_CMD_ENV = "TESSERACT_CMD" +#: Environment variable the engine reads for its language-data directory. +TESSDATA_ENV = "TESSDATA_PREFIX" + +#: :func:`ocr_status` reasons. Stable strings, safe to branch on. +OCR_READY = "ready" +OCR_MISSING_PACKAGE = "missing_package" +OCR_MISSING_ENGINE = "missing_engine" +OCR_ENGINE_UNUSABLE = "engine_unusable" +OCR_NO_LANGUAGE_DATA = "no_language_data" + +#: Where the installers put the engine when they do not add it to ``PATH`` +#: (the Windows installer does not by default). +_MACOS_CANDIDATES = ("/opt/homebrew/bin/tesseract", "/usr/local/bin/tesseract", + "/opt/local/bin/tesseract") +_UNIX_CANDIDATES = ("/usr/bin/tesseract", "/usr/local/bin/tesseract") + + +class OCRStatus(NamedTuple): + """Whether OCR can work here, and if not, which part is missing. + + ``reason`` is one of the ``OCR_*`` constants in this module. + """ + ok: bool + reason: str + + +def _windows_candidates() -> List[str]: + roots = [os.environ.get("ProgramFiles") or r"C:\Program Files", + os.environ.get("ProgramFiles(x86)") or r"C:\Program Files (x86)"] + local = os.environ.get("LOCALAPPDATA") + if local: + roots.append(os.path.join(local, "Programs")) + return [os.path.join(root, "Tesseract-OCR", "tesseract.exe") for root in roots] + + +def _install_candidates(platform: str = "") -> List[str]: + """Conventional install locations for ``platform`` (default: this one).""" + if is_windows(platform): + return _windows_candidates() + if is_macos(platform): + return list(_MACOS_CANDIDATES) + return list(_UNIX_CANDIDATES) + + +def find_tesseract_cmd() -> Optional[str]: + """Return the absolute path of the Tesseract executable, or ``None``. + + Looks in order at ``$TESSERACT_CMD`` (when it names an existing file), + ``PATH``, then the installers' conventional locations. Only looks: + pass the result to ``set_tesseract_cmd`` to make OCR use it. + """ + override = os.environ.get(TESSERACT_CMD_ENV, "").strip() + if override: + if os.path.isfile(override): + return os.path.abspath(override) + autocontrol_logger.warning( + "%s names no file (%r); searching PATH instead", TESSERACT_CMD_ENV, override) + found = shutil.which("tesseract") + if found: + return os.path.abspath(found) + for candidate in _install_candidates(): + if os.path.isfile(candidate): + return candidate + return None + + +def set_tessdata_dir(path: str | os.PathLike[str] | None) -> str | None: + """Point the engine at the directory holding ``*.traineddata``; return it resolved. + + Sets ``$TESSDATA_PREFIX`` for this process, which every later engine run + inherits, so it also changes what :func:`ocr_languages` reports. ``None`` + removes the variable and the engine falls back to its built-in location. + Raises ``AutoControlActionException`` when ``path`` is not an existing + directory; an empty directory is accepted (``ocr_status`` then reports + ``no_language_data``). + """ + if path is None: + os.environ.pop(TESSDATA_ENV, None) + return None + resolved = os.path.realpath(os.fspath(path)) + if not os.path.isdir(resolved): + raise AutoControlActionException(f"tessdata directory does not exist: {resolved}") + os.environ[TESSDATA_ENV] = resolved + return resolved + + +def ocr_languages() -> Optional[List[str]]: + """Return the installed Tesseract language codes, sorted, or ``None``. + + ``None`` and ``[]`` mean different things and call for opposite handling: + + * ``[]`` -- the engine answered and has **no language data**, so every OCR + call will fail; + * ``None`` -- the engine **could not be asked** (``pytesseract`` or the + executable is missing, or listing failed), so nothing is known about + the languages. + + Asked afresh on every call, never cached. + """ + try: + return TesseractBackend().languages() + except OCRBackendNotAvailableError as error: + autocontrol_logger.info("ocr_languages: the engine cannot be asked: %r", error) + return None + + +def _command_exists(command: str) -> bool: + return shutil.which(command) is not None or os.path.isfile(command) + + +def ocr_status() -> OCRStatus: + """Report whether Tesseract OCR can work here: ``(ok, reason)``. + + Checks, in order, and stops at the first missing part: the ``pytesseract`` + package (``missing_package``), the configured engine command + (``missing_engine``), running it (``engine_unusable``), and its language + data (``no_language_data``, only when the engine *answered* with an empty + list). Otherwise ``(True, "ready")``. It checks the command OCR would use + and configures nothing; see the module docstring for the start-up recipe. + """ + backend = TesseractBackend() + try: + command = backend.cmd + except OCRBackendNotAvailableError: + return OCRStatus(False, OCR_MISSING_PACKAGE) + if not _command_exists(command): + return OCRStatus(False, OCR_MISSING_ENGINE) + try: + backend.version() + except OCRBackendNotAvailableError as error: + autocontrol_logger.warning("ocr_status: the engine does not run: %r", error) + return OCRStatus(False, OCR_ENGINE_UNUSABLE) + languages = ocr_languages() + if languages is not None and not languages: + return OCRStatus(False, OCR_NO_LANGUAGE_DATA) + return OCRStatus(True, OCR_READY) + + +__all__ = [ + "OCRStatus", "OCR_ENGINE_UNUSABLE", "OCR_MISSING_ENGINE", + "OCR_MISSING_PACKAGE", "OCR_NO_LANGUAGE_DATA", "OCR_READY", + "TESSDATA_ENV", "TESSERACT_CMD_ENV", + "find_tesseract_cmd", "ocr_languages", "ocr_status", "set_tessdata_dir", +] diff --git a/je_auto_control/utils/tween_drag/tween_drag.py b/je_auto_control/utils/tween_drag/tween_drag.py index 099c41de0..411dd91b2 100644 --- a/je_auto_control/utils/tween_drag/tween_drag.py +++ b/je_auto_control/utils/tween_drag/tween_drag.py @@ -9,8 +9,20 @@ The point math is pure and unit-testable; dispatch goes through an injectable ``sink`` so the drag is tested without real input. Imports no ``PySide6``. + +The press / move / release sequence itself is :func:`_drag_through`, shared +with ``mouse_path.drag_path``: optional pacing (``step_delay_s`` after each +move, ``settle_s`` before the press, after it and before the release) and a +release that runs in ``finally``, at the last point the pointer reached, if +any step raises. """ -from typing import Any, Callable, Dict, List, Optional, Tuple +import math +import time +from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple + +from je_auto_control.utils.logging.logging_instance import autocontrol_logger + +Sink = Callable[[Dict[str, Any]], None] def _linear(t: float) -> float: @@ -76,20 +88,89 @@ def _default_sink(event: Dict[str, Any]) -> None: release_mouse(event.get("button", "mouse_left"), x, y) -def tween_drag(start: Tuple[int, int], end: Tuple[int, int], *, - steps: int = 30, easing: str = "ease_in_out_quad", - button: str = "mouse_left", - sink: Optional[Callable[[Dict[str, Any]], None]] = None - ) -> Dict[str, Any]: - """Drag from ``start`` to ``end`` along an eased path; return point count.""" - points = tween_points(start, end, steps, easing) - dispatch = sink or _default_sink +def _pause_seconds(value: Any, name: str) -> float: + """``value`` as a finite, non-negative number of seconds, else ValueError. + + A NaN would pass a plain ``< 0`` test and then make ``time.sleep`` raise + with the button already held, so finiteness is checked up front. + """ + try: + seconds = float(value) + except (TypeError, ValueError) as error: + raise ValueError(f"{name} must be a number of seconds, got {value!r}") from error + if not math.isfinite(seconds) or seconds < 0: + raise ValueError(f"{name} must be a finite number >= 0, got {value!r}") + return seconds + + +def _pause(seconds: float) -> None: + if seconds: + time.sleep(seconds) + + +def _release_quietly(dispatch: Sink, button: str, point: Sequence[int]) -> None: + """Release ``button`` at ``point`` from a ``finally``; never raises. + + An exception here would replace the one that ended the drag, which is + what the caller needs to see, so a failed release is logged instead. + ``Exception`` rather than a list: the default sink raises the + ``AutoControlException`` family, which no builtin type covers. + """ + try: + dispatch({"op": "release", "button": button, "x": point[0], "y": point[1]}) + except Exception as error: # noqa: BLE001 # pylint: disable=broad-except # reason: see docstring + autocontrol_logger.error("failed to release %r after an aborted drag: %r", button, error) + + +def _drag_through(points: List[List[int]], button: str, dispatch: Sink, + step_delay_s: Any = 0.0, settle_s: Any = 0.0) -> None: + """Press at ``points[0]``, move through every point, release at ``points[-1]``. + + ``settle_s`` (when non-zero) first moves to the start and rests there, + then rests after the press and again before the release; ``step_delay_s`` + rests after each move. With both at 0 the events are exactly press, + moves, release. If any step raises, the button is released in + ``finally`` at the last point the pointer reached -- not at the end, + which would complete a drop the drag never got to -- and a failing + cleanup release is logged rather than raised over the original error. + Both pauses are checked before anything is dispatched. + """ + step_delay = _pause_seconds(step_delay_s, "step_delay_s") + settle = _pause_seconds(settle_s, "settle_s") first, last = points[0], points[-1] + if settle: + dispatch({"op": "move", "x": first[0], "y": first[1]}) + time.sleep(settle) dispatch({"op": "press", "button": button, "x": first[0], "y": first[1]}) + reached: Sequence[int] = first + held = True try: + _pause(settle) for x, y in points: dispatch({"op": "move", "x": x, "y": y}) - finally: - # A failed move used to leave the button held down. + reached = (x, y) + _pause(step_delay) + _pause(settle) dispatch({"op": "release", "button": button, "x": last[0], "y": last[1]}) + held = False + finally: + if held: + _release_quietly(dispatch, button, reached) + + +def tween_drag(start: Tuple[int, int], end: Tuple[int, int], *, + steps: int = 30, easing: str = "ease_in_out_quad", + button: str = "mouse_left", + sink: Optional[Sink] = None, + step_delay_s: float = 0.0, settle_s: float = 0.0, + ) -> Dict[str, Any]: + """Drag from ``start`` to ``end`` along an eased path; return point count. + + ``step_delay_s`` rests after each move and ``settle_s`` before the press, + after it and before the release (seconds, default 0: no pause). Apps + that tell a drag from a click by the pointer's motion need both. If a + step raises, the button is released where the pointer stopped. + """ + points = tween_points(start, end, steps, easing) + _drag_through(points, button, sink or _default_sink, step_delay_s, settle_s) return {"points": len(points), "path": points} diff --git a/je_auto_control/windows/core/utils/win32_vk.py b/je_auto_control/windows/core/utils/win32_vk.py index ef7a34e28..33bc4fc3b 100644 --- a/je_auto_control/windows/core/utils/win32_vk.py +++ b/je_auto_control/windows/core/utils/win32_vk.py @@ -175,6 +175,7 @@ WIN32_VK_BROWSER_STOP: int = 0xA9 # Browser Stop key WIN32_VK_BROWSER_SEARCH: int = 0xAA # Browser Search key WIN32_VK_BROWSER_FAVORITES: int = 0xAB # Browser Favorites key +WIN32_VK_BROWSER_HOME: int = 0xAC # Browser Start and Home key WIN32_VK_VOLUME_MUTE: int = 0xAD WIN32_VK_VOLUME_DOWN: int = 0xAE WIN32_VK_VOLUME_UP: int = 0xAF @@ -185,4 +186,20 @@ WIN32_VK_LAUNCH_MAIL: int = 0xB4 WIN32_VK_LAUNCH_MEDIA_SELECT: int = 0xB5 WIN32_VK_LAUNCH_APP1: int = 0xB6 -WIN32_VK_LAUNCH_APP2: int = 0xB7 \ No newline at end of file +WIN32_VK_LAUNCH_APP2: int = 0xB7 +# OEM keys (Microsoft "Virtual-Key Codes"). OEM_PLUS / COMMA / MINUS / PERIOD +# are the same key on every layout; OEM_1 .. OEM_8 and OEM_102 vary by layout. +WIN32_VK_OEM_1: int = 0xBA +WIN32_VK_OEM_PLUS: int = 0xBB +WIN32_VK_OEM_COMMA: int = 0xBC +WIN32_VK_OEM_MINUS: int = 0xBD +WIN32_VK_OEM_PERIOD: int = 0xBE +WIN32_VK_OEM_2: int = 0xBF +WIN32_VK_OEM_3: int = 0xC0 +WIN32_VK_OEM_4: int = 0xDB +WIN32_VK_OEM_5: int = 0xDC +WIN32_VK_OEM_6: int = 0xDD +WIN32_VK_OEM_7: int = 0xDE +WIN32_VK_OEM_8: int = 0xDF +WIN32_VK_OEM_102: int = 0xE2 +WIN32_VK_OEM_CLEAR: int = 0xFE \ No newline at end of file diff --git a/je_auto_control/wrapper/_platform_linux.py b/je_auto_control/wrapper/_platform_linux.py index cf66a07b2..5e9d90a7a 100644 --- a/je_auto_control/wrapper/_platform_linux.py +++ b/je_auto_control/wrapper/_platform_linux.py @@ -1,4 +1,4 @@ -from typing import Tuple +from typing import Dict, Tuple from je_auto_control.linux_with_x11.core.utils.x11_linux_vk import ( x11_linux_key_backspace, x11_linux_key_slash_b, x11_linux_key_tab, @@ -271,5 +271,8 @@ def _select_input_backend() -> Tuple[X11UnixKeyboardBackend, screen: ScreenBackend = x11_linux_screen recorder: RecorderBackend = x11_linux_recorder +#: Aliases of canonical key names; none on this backend (see ``_platform_windows``). +keyboard_key_aliases: Dict[str, str] = {} + if None in [keyboard_keys_table, mouse_keys_table, special_mouse_keys_table, keyboard, mouse, screen, recorder]: raise AutoControlException("Can't init auto control") diff --git a/je_auto_control/wrapper/_platform_osx.py b/je_auto_control/wrapper/_platform_osx.py index 0a44983ce..9da36234b 100644 --- a/je_auto_control/wrapper/_platform_osx.py +++ b/je_auto_control/wrapper/_platform_osx.py @@ -1,3 +1,5 @@ +from typing import Dict + from je_auto_control.osx.core.utils.osx_vk import ( osx_key_a, osx_key_A, osx_key_b, osx_key_B, osx_key_c, osx_key_C, osx_key_d, osx_key_D, osx_key_e, osx_key_E, osx_key_f, osx_key_F, @@ -153,5 +155,8 @@ screen: ScreenBackend = osx_screen recorder: RecorderBackend = osx_recorder +#: Aliases of canonical key names; none on this backend (see ``_platform_windows``). +keyboard_key_aliases: Dict[str, str] = {} + if None in [keyboard_keys_table, mouse_keys_table, keyboard_check, keyboard, mouse, screen, recorder]: raise AutoControlException("Can't init auto control") diff --git a/je_auto_control/wrapper/_platform_wayland.py b/je_auto_control/wrapper/_platform_wayland.py index 9c147eab1..d5cbd5f3c 100644 --- a/je_auto_control/wrapper/_platform_wayland.py +++ b/je_auto_control/wrapper/_platform_wayland.py @@ -9,6 +9,8 @@ ydotool BTN_* hex codes are converted in :mod:`linux_wayland.mouse` so the public mouse_keys_table stays consistent across platforms. """ +from typing import Dict + from je_auto_control.linux_wayland import keyboard as wayland_keyboard from je_auto_control.linux_wayland import listener as wayland_listener from je_auto_control.linux_wayland import mouse as wayland_mouse @@ -46,13 +48,16 @@ recorder: RecorderBackend = wayland_record.wayland_recorder +#: Aliases of canonical key names; none on this backend (see ``_platform_windows``). +keyboard_key_aliases: Dict[str, str] = {} + if None in [keyboard_keys_table, mouse_keys_table, special_mouse_keys_table, keyboard, mouse, screen, recorder]: raise AutoControlException("Can't init auto control (Wayland)") __all__ = [ - "keyboard", "keyboard_check", "keyboard_keys_table", + "keyboard", "keyboard_check", "keyboard_key_aliases", "keyboard_keys_table", "mouse", "mouse_keys_table", "recorder", "screen", "special_mouse_keys_table", ] diff --git a/je_auto_control/wrapper/_platform_windows.py b/je_auto_control/wrapper/_platform_windows.py index c5e1b239c..ea89419ce 100644 --- a/je_auto_control/wrapper/_platform_windows.py +++ b/je_auto_control/wrapper/_platform_windows.py @@ -1,5 +1,5 @@ import os -from typing import Tuple +from typing import Dict, Tuple from je_auto_control.utils.exception.exceptions import AutoControlException from je_auto_control.utils.logging.logging_instance import autocontrol_logger @@ -11,7 +11,7 @@ from je_auto_control.windows.core.utils.win32_vk import ( WIN32_VK_ACCEPT, WIN32_VK_ADD, WIN32_VK_APPS, WIN32_VK_BACK, WIN32_VK_BROWSER_BACK, WIN32_VK_BROWSER_FAVORITES, - WIN32_VK_BROWSER_FORWARD, WIN32_VK_BROWSER_REFRESH, + WIN32_VK_BROWSER_FORWARD, WIN32_VK_BROWSER_HOME, WIN32_VK_BROWSER_REFRESH, WIN32_VK_BROWSER_SEARCH, WIN32_VK_BROWSER_STOP, WIN32_VK_CANCEL, WIN32_VK_CAPITAL, WIN32_VK_CLEAR, WIN32_VK_CONTROL, WIN32_VK_CONVERT, WIN32_VK_DECIMAL, WIN32_VK_DELETE, WIN32_VK_DIVIDE, WIN32_VK_DOWN, @@ -31,6 +31,10 @@ WIN32_VK_MEDIA_PREV_TRACK, WIN32_VK_MEDIA_STOP, WIN32_VK_MODECHANGE, WIN32_VK_MULTIPLY, WIN32_VK_Menu, WIN32_VK_NEXT, WIN32_VK_NONCONVERT, WIN32_VK_NUMLOCK, + WIN32_VK_OEM_1, WIN32_VK_OEM_2, WIN32_VK_OEM_3, WIN32_VK_OEM_4, + WIN32_VK_OEM_5, WIN32_VK_OEM_6, WIN32_VK_OEM_7, WIN32_VK_OEM_8, + WIN32_VK_OEM_102, WIN32_VK_OEM_CLEAR, WIN32_VK_OEM_COMMA, + WIN32_VK_OEM_MINUS, WIN32_VK_OEM_PERIOD, WIN32_VK_OEM_PLUS, WIN32_VK_NUMPAD0, WIN32_VK_NUMPAD1, WIN32_VK_NUMPAD2, WIN32_VK_NUMPAD3, WIN32_VK_NUMPAD4, WIN32_VK_NUMPAD5, WIN32_VK_NUMPAD6, WIN32_VK_NUMPAD7, WIN32_VK_NUMPAD8, WIN32_VK_NUMPAD9, @@ -132,6 +136,7 @@ def _build_mouse_keys_table(mouse_module) -> dict: "browser_back": WIN32_VK_BROWSER_BACK, "browser_favorites": WIN32_VK_BROWSER_FAVORITES, "browser_forward": WIN32_VK_BROWSER_FORWARD, + "browser_home": WIN32_VK_BROWSER_HOME, "browser_refresh": WIN32_VK_BROWSER_REFRESH, "browser_search": WIN32_VK_BROWSER_SEARCH, "browser_stop": WIN32_VK_BROWSER_STOP, @@ -139,9 +144,6 @@ def _build_mouse_keys_table(mouse_module) -> dict: "capital": WIN32_VK_CAPITAL, "clear": WIN32_VK_CLEAR, "control": WIN32_VK_CONTROL, - # Linux and macOS call it "ctrl", and so do the defaults of type_unicode, - # set_field_text and with_modifiers; without the alias they failed here. - "ctrl": WIN32_VK_CONTROL, "convert": WIN32_VK_CONVERT, "decimal": WIN32_VK_DECIMAL, "delete": WIN32_VK_DELETE, @@ -184,7 +186,10 @@ def _build_mouse_keys_table(mouse_module) -> dict: "junja": WIN32_VK_JUNJA, "kana": WIN32_VK_KANA, "launch_app1": WIN32_VK_LAUNCH_APP1, + # The upper-case spelling stays for scripts that use it; a lower-case + # caller (or anything that lower-cases a typed name) needs the other. "LAUNCH_APP2": WIN32_VK_LAUNCH_APP2, + "launch_app2": WIN32_VK_LAUNCH_APP2, "launch_mail": WIN32_VK_LAUNCH_MAIL, "launch_media_select": WIN32_VK_LAUNCH_MEDIA_SELECT, "lbutton": WIN32_VK_LBUTTON, @@ -203,6 +208,22 @@ def _build_mouse_keys_table(mouse_module) -> dict: "menu": WIN32_VK_Menu, "next": WIN32_VK_NEXT, "nonconvert": WIN32_VK_NONCONVERT, + # OEM keys are named after Microsoft's constants on purpose: oem_1 is ";" + # on a US layout and something else elsewhere, so "semicolon" would lie. + "oem_1": WIN32_VK_OEM_1, + "oem_2": WIN32_VK_OEM_2, + "oem_3": WIN32_VK_OEM_3, + "oem_4": WIN32_VK_OEM_4, + "oem_5": WIN32_VK_OEM_5, + "oem_6": WIN32_VK_OEM_6, + "oem_7": WIN32_VK_OEM_7, + "oem_8": WIN32_VK_OEM_8, + "oem_102": WIN32_VK_OEM_102, + "oem_clear": WIN32_VK_OEM_CLEAR, + "oem_comma": WIN32_VK_OEM_COMMA, + "oem_minus": WIN32_VK_OEM_MINUS, + "oem_period": WIN32_VK_OEM_PERIOD, + "oem_plus": WIN32_VK_OEM_PLUS, "numlock": WIN32_VK_NUMLOCK, "num0": WIN32_VK_NUMPAD0, "num1": WIN32_VK_NUMPAD1, @@ -304,6 +325,33 @@ def _build_mouse_keys_table(mouse_module) -> dict: "z": WIN32_keyZ, } +# 常見的別名 → 上表的標準鍵名。別名也查得到同一個鍵碼，但反查（`keyboard_key_name`） +# 一律回標準名，所以加別名不會改變錄製寫出來的鍵名。 +# Common aliases -> the canonical names above. An alias resolves to the same +# code, but a reverse lookup (``keyboard_key_name``) never answers with one, so +# adding an alias cannot change the key names a recorder writes. +keyboard_key_aliases: Dict[str, str] = { + # Linux and macOS call it "ctrl", and so do the defaults of type_unicode, + # set_field_text and with_modifiers; without the alias they failed here. + "ctrl": "control", "lctrl": "lcontrol", "rctrl": "rcontrol", + "alt": "menu", "lalt": "lmenu", "ralt": "rmenu", + "enter": "return", "esc": "escape", + "win": "lwin", "super": "lwin", "cmd": "lwin", "meta": "lwin", + "backspace": "back", "bksp": "back", + "del": "delete", "ins": "insert", + "pgup": "prior", "pageup": "prior", "pgdn": "next", "pagedown": "next", + "caps": "capital", "capslock": "capital", + "printscreen": "snapshot", "prtsc": "snapshot", "prtscr": "snapshot", + "scrolllock": "scroll", + **{f"numpad{digit}": f"num{digit}" for digit in range(10)}, + # Microsoft defines these four as the same key on every layout, so the + # readable name does not lie; oem_1 .. oem_8 get no alias for that reason. + "plus": "oem_plus", "comma": "oem_comma", "minus": "oem_minus", + "period": "oem_period", +} +keyboard_keys_table.update( + {alias: keyboard_keys_table[name] for alias, name in keyboard_key_aliases.items()}) + special_mouse_keys_table = None keyboard, mouse = _select_input_backend() # 必須在挑選後端後才建表，否則 Interception 後端會拿到 SendInput 的旗標。 diff --git a/je_auto_control/wrapper/auto_control_keyboard.py b/je_auto_control/wrapper/auto_control_keyboard.py index 35fbceb2d..51ed5368c 100644 --- a/je_auto_control/wrapper/auto_control_keyboard.py +++ b/je_auto_control/wrapper/auto_control_keyboard.py @@ -8,6 +8,7 @@ and the only form a type checker can prune, and raise on a platform that matches neither. """ +import re import sys import warnings from typing import Optional, Union, Tuple @@ -25,7 +26,9 @@ from je_auto_control.utils.platform_id import is_windows, is_x11_unix from je_auto_control.utils.test_record.record_test_class import record_action_to_list from je_auto_control.utils.text_unicode.text_unicode import unicode_code_units -from je_auto_control.wrapper.platform_wrapper import keyboard, keyboard_keys_table, keyboard_check +from je_auto_control.wrapper.platform_wrapper import ( + keyboard, keyboard_check, keyboard_key_aliases, keyboard_keys_table, +) def get_keyboard_keys_table() -> dict: """ @@ -35,6 +38,31 @@ def get_keyboard_keys_table() -> dict: return keyboard_keys_table +#: A key name a user can type back: lower case, digits, underscore. +_TYPABLE_KEY_NAME = re.compile(r"[a-z0-9_]+") + + +def keyboard_key_name(keycode: int) -> Optional[str]: + """ + 鍵碼反查標準鍵名；查不到回 None + Return the canonical key name for ``keycode``, or ``None`` if it has none. + + A code often has several names in the table: aliases (``esc`` for + ``escape``), a legacy upper-case spelling (``LAUNCH_APP2``), a letter's + capital (``A``), two spellings of one key (``down`` / ``vk_down``). + Aliases never answer (``keyboard_key_aliases`` lists them), so adding one + cannot change the name a recorder writes. Among the rest, a name made of + lower-case letters, digits and underscores wins (it can be typed back), + then the shortest, then the alphabetically first, so every machine with + the same table gives the same answer. + """ + names = [name for name, code in keyboard_keys_table.items() + if code == keycode and name not in keyboard_key_aliases] + if not names: + return None + return min(names, key=lambda name: (_TYPABLE_KEY_NAME.fullmatch(name) is None, len(name), name)) + + def _resolve_keycode(keycode: Union[int, str]) -> int: """ 將字串鍵名轉換成對應的 keycode diff --git a/je_auto_control/wrapper/auto_control_mouse.py b/je_auto_control/wrapper/auto_control_mouse.py index 881a967df..f6f23e15a 100644 --- a/je_auto_control/wrapper/auto_control_mouse.py +++ b/je_auto_control/wrapper/auto_control_mouse.py @@ -29,9 +29,11 @@ belt-and-braces half of the same statement. """ import ctypes +import math import sys +import time import warnings -from typing import Optional, Tuple, Union +from typing import Any, Dict, Optional, Tuple, Union from je_auto_control.utils.exception.exception_tags import ( mouse_click_mouse_error_message, mouse_get_position_error_message, mouse_press_mouse_error_message, @@ -251,30 +253,74 @@ def release_mouse(mouse_keycode: Union[int, str], x: Optional[int] = None, raise -def click_mouse(mouse_keycode: Union[int, str], x: Optional[int] = None, - y: Optional[int] = None) -> Tuple[MouseKeycode, int, int]: +def _click_plan(clicks: object, interval: object) -> Tuple[int, float]: + """``(clicks, interval)`` checked, else raise ``AutoControlMouseException``. + + ``clicks`` takes an int or an integer string (an action file's variable + may arrive as text) but not a float, which would be silently truncated, + nor a ``bool``: ``clicks=True`` is a caller's mistake, not one click. A + NaN interval would pass a plain ``< 0`` test, so finiteness is checked + explicitly. + """ + problem = f"clicks must be an integer >= 1, got {clicks!r}" + if isinstance(clicks, bool): + raise AutoControlMouseException(problem) + try: + count = int(str(clicks).strip()) + except ValueError as error: + raise AutoControlMouseException(problem) from error + if count < 1: + raise AutoControlMouseException(problem) + try: + pause = float(interval) # type: ignore[arg-type] + except (TypeError, ValueError) as error: + raise AutoControlMouseException(f"interval must be a number, got {interval!r}") from error + if not math.isfinite(pause) or pause < 0: + raise AutoControlMouseException(f"interval must be a finite number >= 0, got {interval!r}") + return count, pause + + +def click_mouse(mouse_keycode: int | str, x: int | None = None, + y: int | None = None, clicks: int = 1, + interval: float = 0.0) -> Tuple[MouseKeycode, int, int]: """ - 在指定座標按下並放開滑鼠按鍵 - Click mouse button at given position + 在指定座標按下並放開滑鼠按鍵；``clicks=2`` 就是雙擊 + Click mouse button at given position; ``clicks=2`` double-clicks + + All clicks land on the same point, ``interval`` seconds apart. Windows and + X11 recognise a double-click from the timing and distance of two clicks, + so keep ``interval`` under the system double-click time (500 ms by + default on Windows). macOS apps read a click count this backend does not + set, so there the clicks arrive as separate single clicks. :param mouse_keycode: 滑鼠按鍵代碼 Mouse keycode :param x: X 座標 X position :param y: Y 座標 Y position + :param clicks: 連點次數 Number of clicks (>= 1) + :param interval: 兩次點擊之間的秒數 Seconds between clicks (>= 0) :return: (keycode, x, y) """ - autocontrol_logger.info(f"click_mouse, keycode={mouse_keycode}, x={x}, y={y}") - param = {"keycode": mouse_keycode, "x": x, "y": y} + autocontrol_logger.info( + f"click_mouse, keycode={mouse_keycode}, x={x}, y={y}, clicks={clicks}, interval={interval}") + param: Dict[str, Any] = {"keycode": mouse_keycode, "x": x, "y": y} + if clicks != 1 or interval: + # Only when used, so a single click records exactly what it always did. + param.update(clicks=clicks, interval=interval) try: + count, pause = _click_plan(clicks, interval) keycode, x, y = mouse_preprocess(mouse_keycode, x, y) - # macOS orders its mouse backend as (x, y, button) — same convention as - # press_mouse/release_mouse above. Without this branch the arguments - # bind as x=, y=, button=; the osx button table holds - # strings, so the int never matches any branch and the click is - # silently dropped with no exception. - if sys.platform == "darwin": - mouse.click_mouse(x, y, keycode) - else: - mouse.click_mouse(keycode, x, y) + for index in range(count): + if index and pause: + time.sleep(pause) + # macOS orders its mouse backend as (x, y, button) — same convention as + # press_mouse/release_mouse above. Without this branch the arguments + # bind as x=, y=, button=; the osx button table holds + # strings, so the int never matches any branch and the click is + # silently dropped with no exception. + if sys.platform == "darwin": + mouse.click_mouse(x, y, keycode) + else: + mouse.click_mouse(keycode, x, y) record_action_to_list("click_mouse", param) return keycode, x, y except AutoControlMouseException as error: diff --git a/je_auto_control/wrapper/platform_wrapper.py b/je_auto_control/wrapper/platform_wrapper.py index f4a4c4054..199d9cf57 100644 --- a/je_auto_control/wrapper/platform_wrapper.py +++ b/je_auto_control/wrapper/platform_wrapper.py @@ -15,6 +15,9 @@ # target. keyboard_check: KeyboardCheckBackend keyboard_keys_table: Dict[str, int] +#: Alias -> canonical name for entries of ``keyboard_keys_table`` that are +#: aliases (``esc`` for ``escape``); reverse lookups skip them. +keyboard_key_aliases: Dict[str, str] #: Values are platform-specific button codes: a plain int on X11, Wayland and #: macOS, a tuple of three Win32 event flags on Windows. mouse_keys_table: Dict[str, MouseKeycode] @@ -41,13 +44,13 @@ if is_windows(): from je_auto_control.wrapper._platform_windows import ( # noqa: F401 # reason: facade re-export - keyboard as _keyboard, keyboard_check, keyboard_keys_table, + keyboard as _keyboard, keyboard_check, keyboard_key_aliases, keyboard_keys_table, mouse as _mouse, mouse_keys_table, special_mouse_keys_table, screen, recorder, ) elif is_macos(): from je_auto_control.wrapper._platform_osx import ( # noqa: F401 # reason: facade re-export - keyboard as _keyboard, keyboard_check, keyboard_keys_table, + keyboard as _keyboard, keyboard_check, keyboard_key_aliases, keyboard_keys_table, mouse as _mouse, mouse_keys_table, special_mouse_keys_table, screen, recorder, ) @@ -60,7 +63,7 @@ if _DISPLAY_SERVER == "wayland": try: from je_auto_control.wrapper._platform_wayland import ( # noqa: F401 # reason: facade re-export - keyboard as _keyboard, keyboard_check, keyboard_keys_table, + keyboard as _keyboard, keyboard_check, keyboard_key_aliases, keyboard_keys_table, mouse as _mouse, mouse_keys_table, special_mouse_keys_table, screen, recorder, ) @@ -72,13 +75,13 @@ "this warning.", _wayland_error, ) from je_auto_control.wrapper._platform_linux import ( # noqa: F401 # reason: facade re-export - keyboard as _keyboard, keyboard_check, keyboard_keys_table, + keyboard as _keyboard, keyboard_check, keyboard_key_aliases, keyboard_keys_table, mouse as _mouse, mouse_keys_table, special_mouse_keys_table, screen, recorder, ) else: from je_auto_control.wrapper._platform_linux import ( # noqa: F401 # reason: facade re-export - keyboard as _keyboard, keyboard_check, keyboard_keys_table, + keyboard as _keyboard, keyboard_check, keyboard_key_aliases, keyboard_keys_table, mouse as _mouse, mouse_keys_table, special_mouse_keys_table, screen, recorder, ) @@ -89,7 +92,7 @@ # whose socket and portal are Linux desktop infrastructure; a BSD running # Wayland reaches this through XWayland like any other X11 client. from je_auto_control.wrapper._platform_linux import ( # noqa: F401 # reason: facade re-export - keyboard as _keyboard, keyboard_check, keyboard_keys_table, + keyboard as _keyboard, keyboard_check, keyboard_key_aliases, keyboard_keys_table, mouse as _mouse, mouse_keys_table, special_mouse_keys_table, screen, recorder, ) @@ -110,7 +113,7 @@ __all__ = [ - "keyboard", "keyboard_check", "keyboard_keys_table", + "keyboard", "keyboard_check", "keyboard_key_aliases", "keyboard_keys_table", "mouse", "mouse_keys_table", "special_mouse_keys_table", "screen", "recorder", ] diff --git a/test/unit_test/headless/test_click_mouse_multi.py b/test/unit_test/headless/test_click_mouse_multi.py new file mode 100644 index 000000000..646cfabd7 --- /dev/null +++ b/test/unit_test/headless/test_click_mouse_multi.py @@ -0,0 +1,124 @@ +"""``click_mouse(clicks=, interval=)``: multi-click in one call. + +The backend and the clock are stand-ins, so no real click is sent and nothing +sleeps. No Qt imports. +""" +import math +import sys +import types + +import pytest + +from je_auto_control.utils.exception.exceptions import AutoControlMouseException +from je_auto_control.wrapper import auto_control_mouse + + +class _Backend: + """Records each backend click; mirrors the Windows / X11 signature.""" + + def __init__(self): + self.clicks = [] + + def click_mouse(self, mouse_keycode, x=None, y=None): + self.clicks.append((mouse_keycode, x, y)) + + +@pytest.fixture +def env(monkeypatch): + backend = _Backend() + sleeps = [] + records = [] + monkeypatch.setattr(auto_control_mouse, "mouse", backend) + monkeypatch.setattr(auto_control_mouse, "time", types.SimpleNamespace(sleep=sleeps.append)) + monkeypatch.setattr(auto_control_mouse, "mouse_keys_table", {"mouse_left": 1, "mouse_right": 2}) + monkeypatch.setattr(auto_control_mouse, "record_action_to_list", + lambda name, param, error=None: records.append((name, dict(param or {}), error))) + monkeypatch.setattr(auto_control_mouse, "get_mouse_position", lambda: (7, 8)) + monkeypatch.setattr(sys, "platform", "win32") + return types.SimpleNamespace(backend=backend, sleeps=sleeps, records=records) + + +def test_the_default_is_one_click_with_no_pause_and_the_old_record(env): + assert auto_control_mouse.click_mouse("mouse_left", 10, 20) == (1, 10, 20) + assert env.backend.clicks == [(1, 10, 20)] + assert env.sleeps == [] + assert env.records == [("click_mouse", {"keycode": "mouse_left", "x": 10, "y": 20}, None)] + + +def test_a_double_click_is_two_clicks_on_one_point_with_one_pause(env): + auto_control_mouse.click_mouse("mouse_left", 10, 20, clicks=2, interval=0.06) + assert env.backend.clicks == [(1, 10, 20), (1, 10, 20)] + assert env.sleeps == [0.06] + name, param, error = env.records[-1] + assert (name, error) == ("click_mouse", None) + assert param["clicks"] == 2 + assert param["interval"] == 0.06 + + +def test_no_pause_between_clicks_when_interval_is_zero(env): + auto_control_mouse.click_mouse("mouse_right", clicks=3) + assert env.backend.clicks == [(2, 7, 8)] * 3 # the cursor, resolved once + assert env.sleeps == [] + + +def test_an_integer_string_is_accepted(env): + auto_control_mouse.click_mouse("mouse_left", 1, 2, clicks="2") + assert len(env.backend.clicks) == 2 + + +def test_macos_binds_every_click_xy_first(env, monkeypatch): + calls = [] + monkeypatch.setattr(auto_control_mouse, "mouse", types.SimpleNamespace( + click_mouse=lambda x, y, mouse_button: calls.append((x, y, mouse_button)))) + monkeypatch.setattr(sys, "platform", "darwin") + auto_control_mouse.click_mouse("mouse_left", 5, 6, clicks=2) + assert calls == [(5, 6, 1), (5, 6, 1)] + + +@pytest.mark.parametrize("clicks", [0, -1, True, False, 1.5, 2.0, "x", None]) +def test_an_invalid_click_count_clicks_nothing(env, clicks): + with pytest.raises(AutoControlMouseException): + auto_control_mouse.click_mouse("mouse_left", 1, 2, clicks=clicks) + assert env.backend.clicks == [] + + +@pytest.mark.parametrize("interval", [-0.1, math.nan, math.inf, "soon", None]) +def test_an_invalid_interval_clicks_nothing(env, interval): + with pytest.raises(AutoControlMouseException): + auto_control_mouse.click_mouse("mouse_left", 1, 2, clicks=2, interval=interval) + assert env.backend.clicks == [] + assert env.records[-1][2] is not None # the failure is recorded + + +def test_the_executor_command_forwards_both_parameters(env): + from je_auto_control.utils.executor.action_executor import executor + executor.event_dict["AC_click_mouse"]("mouse_left", 3, 4, clicks=2, interval=0.05) + assert env.backend.clicks == [(1, 3, 4), (1, 3, 4)] + assert env.sleeps == [0.05] + + +def test_the_mcp_handler_forwards_both_parameters(env): + from je_auto_control.utils.mcp_server.tools import _handlers_input + assert _handlers_input.click_mouse("mouse_left", 3, 4, clicks=2, interval=0.05) == [1, 3, 4] + assert len(env.backend.clicks) == 2 + + +def test_the_mcp_fake_backend_records_each_click(): + from je_auto_control.utils.mcp_server import fake_backend + from je_auto_control.utils.mcp_server.tools import _handlers_input + fake_backend.reset_fake_state() + fake_backend.install_fake_backend() + try: + _handlers_input.click_mouse("mouse_left", 3, 4, clicks=2) + assert fake_backend.fake_state().mouse_actions == [("click", "mouse_left", 3, 4)] * 2 + finally: + fake_backend.uninstall_fake_backend() + + +def test_the_mcp_tool_and_script_builder_declare_both_parameters(): + from je_auto_control.utils.mcp_server.tools import build_default_tool_registry + tool = next(t for t in build_default_tool_registry() if t.name == "ac_click_mouse") + assert {"clicks", "interval"} <= set(tool.input_schema["properties"]) + from je_auto_control.gui.script_builder.command_schema import _build_specs + spec = next(s for s in _build_specs() if s.command == "AC_click_mouse") + assert {"clicks", "interval"} <= {field.name for field in spec.fields} diff --git a/test/unit_test/headless/test_cross_project_contracts.py b/test/unit_test/headless/test_cross_project_contracts.py index 3951e1f7e..89e12f963 100644 --- a/test/unit_test/headless/test_cross_project_contracts.py +++ b/test/unit_test/headless/test_cross_project_contracts.py @@ -125,6 +125,9 @@ def test_no_flag_exits_non_zero(tmp_path): "save_window_layout", "screen_size", "set_clipboard", "set_mouse_position", "snap_window", "wait_until_clipboard_changes", "wait_until_port", "wait_until_process", "write", + # Jeffrey_RPA: added so it can drop its own copies of the same logic. + "find_tesseract_cmd", "ocr_languages", "ocr_status", "set_tessdata_dir", + "keyboard_key_name", ]) def test_facade_name_other_repositories_call(name): assert name in je_auto_control.__all__, f"{name} left the facade" @@ -154,6 +157,18 @@ def test_internal_names_jeffrey_rpa_imports(): assert isinstance(WRITE_CONTROL_KEYS, dict) and WRITE_CONTROL_KEYS +@pytest.mark.parametrize("name, keywords", [ + ("click_mouse", {"clicks", "interval"}), + ("tween_drag", {"step_delay_s", "settle_s"}), + ("drag_path", {"step_delay_s", "settle_s"}), +]) +def test_keywords_jeffrey_rpa_passes(name, keywords): + """Keyword parameters added so Jeffrey_RPA can drop its own loops.""" + import inspect + parameters = inspect.signature(getattr(je_auto_control, name)).parameters + assert keywords <= set(parameters), (name, keywords - set(parameters)) + + def test_gui_widget_pybreeze_embeds_is_still_there(): """Read, not imported: importing it needs PySide6, which CI lacks here.""" source = (REPO_ROOT / "je_auto_control" / "gui" / "main_widget.py" @@ -183,6 +198,8 @@ def test_key_tables_jeffrey_rpa_reads(): for name in ("up", "down", "left", "right", "space", "tab", "shift", "a", "z", "0", "9", "f1", "f12"): assert name in keyboard, name + # Reverse lookups skip these, so a recorder never writes an alias. + assert isinstance(platform_wrapper.keyboard_key_aliases, dict) @pytest.mark.skipif(not sys.platform.startswith("win"), @@ -199,3 +216,7 @@ def test_windows_key_names_jeffrey_rpa_sends(): for name in ("return", "escape", "control", "menu", "back", "delete", "home", "end", "insert", "capital", "vk_down"): assert name in platform_wrapper.keyboard_keys_table, name + # Added so it can drop its own alias and extra-key tables. + for name in ("ctrl", "alt", "enter", "esc", "win", "pgup", "numpad0", "plus", + "oem_1", "oem_plus", "oem_102", "oem_clear", "browser_home", "launch_app2"): + assert name in platform_wrapper.keyboard_keys_table, name diff --git a/test/unit_test/headless/test_drag_pacing.py b/test/unit_test/headless/test_drag_pacing.py new file mode 100644 index 000000000..80096fde5 --- /dev/null +++ b/test/unit_test/headless/test_drag_pacing.py @@ -0,0 +1,193 @@ +"""Drag pacing (``step_delay_s`` / ``settle_s``) and the release on failure. + +``tween_drag`` and ``drag_path`` share one press / move / release sequence. +Every event goes to a recording sink and every pause to a recording clock, so +no real input is sent and nothing sleeps. No Qt imports. +""" +import importlib +import math +import types + +import pytest + +from je_auto_control.utils.mouse_path import drag_path +from je_auto_control.utils.tween_drag import tween_drag + +tween_module = importlib.import_module("je_auto_control.utils.tween_drag.tween_drag") +path_module = importlib.import_module("je_auto_control.utils.mouse_path.mouse_path") + + +class _Timeline: + """One ordered record of dispatched events and pauses.""" + + def __init__(self, fail_on=None, error=RuntimeError): + self.entries = [] + self.fail_on = fail_on or (lambda event, index: False) + self.error = error + + def sink(self, event): + index = sum(1 for kind, *_ in self.entries if kind == event["op"]) + self.entries.append((event["op"], event.get("x"), event.get("y"))) + if self.fail_on(event, index): + raise self.error(f"{event['op']} #{index} failed") + + def sleep(self, seconds): + self.entries.append(("sleep", seconds, None)) + + def ops(self): + return [kind for kind, *_ in self.entries] + + +@pytest.fixture +def timeline(monkeypatch): + line = _Timeline() + monkeypatch.setattr(tween_module, "time", types.SimpleNamespace(sleep=line.sleep)) + return line + + +def _linear(start, end, **kwargs): + return tween_drag(start, end, steps=2, easing="linear", **kwargs) + + +# --- defaults: the old event sequence, no pauses ------------------------------- + +def test_defaults_dispatch_press_moves_release_and_never_sleep(timeline): + out = _linear((0, 0), (10, 0), sink=timeline.sink) + assert timeline.entries == [("press", 0, 0), ("move", 0, 0), ("move", 5, 0), + ("move", 10, 0), ("release", 10, 0)] + assert out["points"] == 3 + + +def test_drag_path_defaults_are_unchanged_too(timeline): + drag_path([(0, 0), (4, 0)], per_segment_steps=2, sink=timeline.sink) + assert timeline.ops() == ["press", "move", "move", "move", "release"] + + +# --- pacing ---------------------------------------------------------------------- + +def test_step_delay_pauses_after_every_move(timeline): + _linear((0, 0), (10, 0), sink=timeline.sink, step_delay_s=0.012) + assert timeline.ops() == ["press", "move", "sleep", "move", "sleep", + "move", "sleep", "release"] + assert {seconds for kind, seconds, _ in timeline.entries if kind == "sleep"} == {0.012} + + +def test_settle_rests_on_the_start_and_around_the_press_and_release(timeline): + _linear((0, 0), (10, 0), sink=timeline.sink, settle_s=0.08) + assert timeline.entries == [ + ("move", 0, 0), ("sleep", 0.08, None), # hover on the start + ("press", 0, 0), ("sleep", 0.08, None), # after the press + ("move", 0, 0), ("move", 5, 0), ("move", 10, 0), + ("sleep", 0.08, None), # before the release + ("release", 10, 0)] + + +def test_drag_path_takes_the_same_pacing(timeline): + drag_path([(0, 0), (2, 0), (2, 2)], per_segment_steps=1, sink=timeline.sink, + step_delay_s=0.01, settle_s=0.05) + assert timeline.ops() == ["move", "sleep", "press", "sleep", + "move", "sleep", "move", "sleep", "move", "sleep", + "sleep", "release"] + + +@pytest.mark.parametrize("name", ["step_delay_s", "settle_s"]) +@pytest.mark.parametrize("value", [-0.01, math.nan, math.inf, "soon", None]) +def test_an_invalid_pause_is_refused_before_anything_is_sent(timeline, name, value): + with pytest.raises(ValueError): + _linear((0, 0), (10, 0), sink=timeline.sink, **{name: value}) + with pytest.raises(ValueError): + drag_path([(0, 0), (4, 0)], sink=timeline.sink, **{name: value}) + assert timeline.entries == [] + + +def test_an_empty_path_sends_nothing_even_with_pacing(timeline): + assert drag_path([], sink=timeline.sink, settle_s=1.0)["points"] == 0 + assert timeline.entries == [] + + +# --- the release when a step raises ---------------------------------------------- + +def test_a_failed_move_releases_at_the_last_point_reached(timeline): + timeline.fail_on = lambda event, index: event["op"] == "move" and index == 2 + with pytest.raises(RuntimeError, match="move #2"): + _linear((0, 0), (10, 0), sink=timeline.sink) + # The move to (10, 0) failed: let go at (5, 0), not at the drop target. + assert timeline.entries[-1] == ("release", 5, 0) + assert timeline.ops().count("release") == 1 + + +def test_a_failure_on_the_first_move_releases_where_it_was_pressed(timeline): + timeline.fail_on = lambda event, index: event["op"] == "move" and index == 0 + with pytest.raises(RuntimeError): + drag_path([(3, 4), (9, 9)], per_segment_steps=2, sink=timeline.sink) + assert timeline.entries[-1] == ("release", 3, 4) + + +def test_a_failed_press_releases_nothing(timeline): + timeline.fail_on = lambda event, index: event["op"] == "press" + with pytest.raises(RuntimeError): + _linear((0, 0), (10, 0), sink=timeline.sink) + assert "release" not in timeline.ops() + + +def test_an_interrupted_pause_still_releases(timeline, monkeypatch): + def interrupt(seconds): + timeline.entries.append(("sleep", seconds, None)) + if timeline.ops().count("move") == 2: + raise KeyboardInterrupt + monkeypatch.setattr(tween_module, "time", types.SimpleNamespace(sleep=interrupt)) + with pytest.raises(KeyboardInterrupt): + _linear((0, 0), (10, 0), sink=timeline.sink, step_delay_s=0.01) + assert timeline.entries[-1] == ("release", 5, 0) + + +def test_a_failing_cleanup_release_does_not_hide_the_original_error(timeline): + def fail(event, index): + return (event["op"] == "move" and index == 1) or event["op"] == "release" + timeline.fail_on = fail + with pytest.raises(RuntimeError, match="move #1"): + _linear((0, 0), (10, 0), sink=timeline.sink) + assert timeline.entries[-1] == ("release", 0, 0) + + +def test_a_failing_final_release_is_raised_and_retried_once(timeline): + timeline.fail_on = lambda event, index: event["op"] == "release" and index == 0 + with pytest.raises(RuntimeError, match="release #0"): + _linear((0, 0), (10, 0), sink=timeline.sink) + assert timeline.entries[-2:] == [("release", 10, 0), ("release", 10, 0)] + + +# --- wiring ---------------------------------------------------------------------- + +def test_the_executor_adapters_forward_the_pacing(timeline, monkeypatch): + from je_auto_control.utils.executor import action_executor + monkeypatch.setattr(tween_module, "_default_sink", timeline.sink) + monkeypatch.setattr(path_module, "_default_sink", timeline.sink) + action_executor._tween_drag([0, 0], [10, 0], steps=1, easing="linear", + step_delay_s=0.02, settle_s="0.03") + assert ("sleep", 0.02, None) in timeline.entries + assert ("sleep", 0.03, None) in timeline.entries + timeline.entries.clear() + action_executor._drag_path([[0, 0], [5, 5]], per_segment_steps=1, step_delay_s=0.04) + assert ("sleep", 0.04, None) in timeline.entries + + +def test_the_mcp_handlers_forward_the_pacing(timeline, monkeypatch): + from je_auto_control.utils.mcp_server.tools import _handlers, _handlers_executor_bridge + monkeypatch.setattr(tween_module, "_default_sink", timeline.sink) + monkeypatch.setattr(path_module, "_default_sink", timeline.sink) + assert _handlers.tween_drag([0, 0], [10, 0], steps=1, settle_s=0.05) == {"points": 2} + assert ("sleep", 0.05, None) in timeline.entries + timeline.entries.clear() + _handlers_executor_bridge.drag_path([[0, 0], [5, 5]], per_segment_steps=1, step_delay_s=0.06) + assert ("sleep", 0.06, None) in timeline.entries + + +def test_the_mcp_tools_and_script_builder_declare_the_pacing(): + from je_auto_control.utils.mcp_server.tools import build_default_tool_registry + from je_auto_control.gui.script_builder.command_schema import _build_specs + tools = {tool.name: tool for tool in build_default_tool_registry()} + specs = {spec.command: spec for spec in _build_specs()} + for tool, command in (("ac_tween_drag", "AC_tween_drag"), ("ac_drag_path", "AC_drag_path")): + assert {"step_delay_s", "settle_s"} <= set(tools[tool].input_schema["properties"]) + assert {"step_delay_s", "settle_s"} <= {field.name for field in specs[command].fields} diff --git a/test/unit_test/headless/test_key_names_and_aliases.py b/test/unit_test/headless/test_key_names_and_aliases.py new file mode 100644 index 000000000..5b90a98d0 --- /dev/null +++ b/test/unit_test/headless/test_key_names_and_aliases.py @@ -0,0 +1,151 @@ +"""Key aliases, the added Windows keys, and the canonical reverse lookup. + +``keyboard_key_name`` answers "which name does this virtual key have?" for +recorders. Aliases resolve to the same code but must never be the answer, or +adding one would change the key names a recording writes. Reads tables only; +no key is sent. No Qt imports. +""" +import re +import sys + +import pytest + +import je_auto_control as ac +from je_auto_control.wrapper import auto_control_keyboard +from je_auto_control.wrapper import platform_wrapper + +windows_only = pytest.mark.skipif(not sys.platform.startswith("win"), + reason="the Win32 virtual-key table loads on Windows only") + +#: Microsoft "Virtual-Key Codes (Winuser.h)" for the keys added here. +_ADDED_WINDOWS_KEYS = { + "browser_home": 0xAC, "launch_app2": 0xB7, + "oem_1": 0xBA, "oem_plus": 0xBB, "oem_comma": 0xBC, "oem_minus": 0xBD, + "oem_period": 0xBE, "oem_2": 0xBF, "oem_3": 0xC0, "oem_4": 0xDB, + "oem_5": 0xDC, "oem_6": 0xDD, "oem_7": 0xDE, "oem_8": 0xDF, + "oem_102": 0xE2, "oem_clear": 0xFE, +} + +#: What a recorder must keep writing for keys that gained aliases: the name a +#: shortest-lower-case lookup picked before any alias existed. +_CANONICAL_BEFORE_ALIASES = { + 0x11: "control", 0xA2: "lcontrol", 0xA3: "rcontrol", + 0x12: "menu", 0xA4: "lmenu", 0xA5: "rmenu", + 0x0D: "return", 0x1B: "escape", 0x5B: "lwin", 0x08: "back", + 0x2E: "delete", 0x2D: "insert", 0x21: "prior", 0x22: "next", + 0x14: "capital", 0x2C: "snapshot", 0x91: "scroll", + **{0x60 + digit: f"num{digit}" for digit in range(10)}, + 0x28: "down", 0x41: "a", +} + + +def _shortest_lower_case(table, code, skip=()): + names = [name for name, value in table.items() if value == code and name not in skip] + return min(names, key=lambda name: (name != name.lower(), len(name), name), default=None) + + +# --- the Windows table ------------------------------------------------------- + +@windows_only +@pytest.mark.parametrize("name, code", sorted(_ADDED_WINDOWS_KEYS.items())) +def test_the_added_keys_carry_microsofts_codes(name, code): + assert platform_wrapper.keyboard_keys_table[name] == code + + +@windows_only +def test_the_upper_case_launch_app2_spelling_stays(): + assert platform_wrapper.keyboard_keys_table["LAUNCH_APP2"] == 0xB7 + + +@windows_only +def test_every_alias_resolves_to_its_targets_code(): + table = platform_wrapper.keyboard_keys_table + aliases = platform_wrapper.keyboard_key_aliases + assert len(aliases) >= 40, "the alias map is empty; the checks below would pass vacuously" + for alias, target in aliases.items(): + assert target in table, (alias, target) + assert target not in aliases, (alias, target) + assert table[alias] == table[target], alias + assert auto_control_keyboard._resolve_keycode(alias) == table[target], alias + + +@windows_only +@pytest.mark.parametrize("alias, target", [ + ("ctrl", "control"), ("alt", "menu"), ("enter", "return"), ("esc", "escape"), + ("win", "lwin"), ("backspace", "back"), ("del", "delete"), ("pgup", "prior"), + ("pagedown", "next"), ("capslock", "capital"), ("prtsc", "snapshot"), + ("numpad5", "num5"), ("plus", "oem_plus"), ("period", "oem_period"), +]) +def test_common_aliases(alias, target): + assert platform_wrapper.keyboard_key_aliases[alias] == target + + +@windows_only +def test_every_name_is_typable_except_the_legacy_capitals(): + """Callers lower-case what a user types; every added name survives that.""" + typable = re.compile(r"[a-z0-9_]+") + added = set(_ADDED_WINDOWS_KEYS) | set(platform_wrapper.keyboard_key_aliases) + assert all(typable.fullmatch(name) for name in added) + + +@windows_only +def test_layout_dependent_oem_keys_get_no_layout_specific_alias(): + """``oem_1`` is ";" on a US layout only; a name like "semicolon" would lie.""" + layout_bound = {0xBA, 0xBF, 0xC0, 0xDB, 0xDC, 0xDD, 0xDE, 0xDF, 0xE2} + table = platform_wrapper.keyboard_keys_table + for alias in platform_wrapper.keyboard_key_aliases: + assert table[alias] not in layout_bound, alias + + +# --- reverse lookup: canonical names only ---------------------------------------- + +@windows_only +@pytest.mark.parametrize("code, name", sorted(_CANONICAL_BEFORE_ALIASES.items())) +def test_reverse_lookup_keeps_the_pre_alias_name(code, name): + assert ac.keyboard_key_name(code) == name + + +@windows_only +def test_no_alias_changes_any_reverse_lookup(): + """For every code, the answer is the old lookup over the table minus aliases.""" + table = platform_wrapper.keyboard_keys_table + aliases = platform_wrapper.keyboard_key_aliases + for code in set(table.values()): + assert ac.keyboard_key_name(code) == _shortest_lower_case(table, code, aliases), hex(code) + + +@windows_only +def test_reverse_lookup_names_the_added_keys(): + for name, code in _ADDED_WINDOWS_KEYS.items(): + assert ac.keyboard_key_name(code) == name + assert ac.keyboard_key_name(0xFC) is None # VK_NONAME: no entry + + +def test_the_preference_order_on_a_synthetic_table(monkeypatch): + """Platform-independent: aliases lose, then untypable names, then length, then order.""" + monkeypatch.setattr(auto_control_keyboard, "keyboard_keys_table", { + "esc": 27, "escape": 27, # alias vs canonical + "\b": 8, "backspace": 8, # untypable vs typable + "LAUNCH_APP2": 183, "launch_app2": 183, + "vk_down": 40, "down": 40, # length + "next": 34, "pgdn": 34, # same length: alphabetical + "only_alias": 99, + }) + monkeypatch.setattr(auto_control_keyboard, "keyboard_key_aliases", + {"esc": "escape", "pgdn": "next", "only_alias": "x"}) + name = auto_control_keyboard.keyboard_key_name + assert [name(27), name(8), name(183), name(40), name(34)] == [ + "escape", "backspace", "launch_app2", "down", "next"] + assert name(99) is None + assert name(12345) is None + + +def test_every_platform_publishes_an_alias_map(): + aliases = platform_wrapper.keyboard_key_aliases + assert isinstance(aliases, dict) + assert all(alias in platform_wrapper.keyboard_keys_table for alias in aliases) + + +def test_facade_export(): + assert "keyboard_key_name" in ac.__all__ + assert callable(ac.keyboard_key_name) diff --git a/test/unit_test/headless/test_ocr_tesseract_setup.py b/test/unit_test/headless/test_ocr_tesseract_setup.py new file mode 100644 index 000000000..9cd94286b --- /dev/null +++ b/test/unit_test/headless/test_ocr_tesseract_setup.py @@ -0,0 +1,270 @@ +"""Tesseract discovery, configuration and health checks (``ocr/tesseract_setup``). + +A stand-in ``pytesseract`` replaces the real one, so no engine is run and no +screen is read. No Qt imports. +""" +import os +import subprocess # nosec B404 # reason: only CalledProcessError is used, as a fake engine failure +import sys +import types + +import pytest + +import je_auto_control as ac +from je_auto_control.utils.exception.exceptions import AutoControlActionException +from je_auto_control.utils.ocr import tesseract_setup as setup +from je_auto_control.utils.ocr.backends import tesseract_backend +from je_auto_control.utils.ocr.backends.base import OCRBackendNotAvailableError +from je_auto_control.utils.ocr.backends.tesseract_backend import TesseractBackend + + +class _NotFound(OSError): + """What pytesseract raises when the executable cannot be started.""" + + +class _FakePytesseract: + """The slice of ``pytesseract`` the setup helpers reach.""" + + def __init__(self, languages=("eng",), cmd="tesseract"): + self.pytesseract = types.SimpleNamespace(tesseract_cmd=cmd) + self.languages = list(languages) + self.version_error = None + self.languages_error = None + self.language_calls = [] + + def get_tesseract_version(self): + if self.version_error is not None: + raise self.version_error + return "5.3.0" + + def get_languages(self, config=""): + self.language_calls.append(config) + if self.languages_error is not None: + raise self.languages_error + return list(self.languages) + + +@pytest.fixture +def fake(monkeypatch, tmp_path): + """A fake engine whose command is an existing file.""" + engine = tmp_path / "tesseract.exe" + engine.write_bytes(b"") + module = _FakePytesseract(languages=("jpn", "eng", "chi_tra"), cmd=str(engine)) + monkeypatch.setattr(tesseract_backend, "_pytesseract", module) + return module + + +@pytest.fixture +def no_pytesseract(monkeypatch): + monkeypatch.setattr(tesseract_backend, "_pytesseract", None) + monkeypatch.setitem(sys.modules, "pytesseract", None) # import -> ImportError + + +# --- ocr_languages: None and [] are different answers ----------------------- + +def test_languages_are_sorted_and_asked_with_an_empty_config(fake): + assert setup.ocr_languages() == ["chi_tra", "eng", "jpn"] + assert fake.language_calls == [""] + + +def test_an_engine_with_no_language_data_answers_an_empty_list(fake): + fake.languages = [] + assert setup.ocr_languages() == [] + + +def test_languages_are_none_when_the_package_is_missing(no_pytesseract): + assert setup.ocr_languages() is None + + +def test_languages_are_none_when_the_engine_cannot_be_asked(fake): + fake.languages_error = _NotFound("tesseract is not installed") + assert setup.ocr_languages() is None + + +def test_languages_are_asked_afresh_on_every_call(fake): + """Language data added while the process runs must be seen.""" + fake.languages = [] + assert setup.ocr_languages() == [] + fake.languages = ["eng"] + assert setup.ocr_languages() == ["eng"] + assert len(fake.language_calls) == 2 + + +# --- ocr_status -------------------------------------------------------------- + +def test_status_ready(fake): + status = setup.ocr_status() + assert status == (True, setup.OCR_READY) + ok, reason = status + assert ok is True + assert reason == "ready" + assert status.ok is True + assert status.reason == "ready" + + +def test_status_missing_package(no_pytesseract): + assert setup.ocr_status() == (False, setup.OCR_MISSING_PACKAGE) + + +def test_status_missing_engine(fake, tmp_path): + fake.pytesseract.tesseract_cmd = str(tmp_path / "absent" / "tesseract.exe") + assert setup.ocr_status() == (False, setup.OCR_MISSING_ENGINE) + + +@pytest.mark.parametrize("error", [ + _NotFound("wrong architecture"), + subprocess.CalledProcessError(1, ["tesseract", "--version"]), + UnicodeDecodeError("utf-8", b"\xff", 0, 1, "invalid start byte"), +], ids=["os-error", "non-zero-exit", "undecodable-output"]) +def test_status_engine_unusable(fake, error): + fake.version_error = error + assert setup.ocr_status() == (False, setup.OCR_ENGINE_UNUSABLE) + + +def test_an_unreadable_version_does_not_end_the_process(fake): + """pytesseract raises SystemExit for a version string it cannot parse.""" + fake.version_error = SystemExit('Invalid tesseract version: "garbage"') + assert setup.ocr_status() == (False, setup.OCR_ENGINE_UNUSABLE) + backend = TesseractBackend() + with pytest.raises(OCRBackendNotAvailableError): + backend.version() + + +def test_status_no_language_data_only_when_the_engine_answered_empty(fake): + fake.languages = [] + assert setup.ocr_status() == (False, setup.OCR_NO_LANGUAGE_DATA) + # An engine that runs but cannot list is not "no language data". + fake.languages_error = _NotFound("listing failed") + assert setup.ocr_status() == (True, setup.OCR_READY) + + +def test_status_finds_a_bare_command_on_path(fake, monkeypatch): + fake.pytesseract.tesseract_cmd = "tesseract" + monkeypatch.setattr(setup, "shutil", types.SimpleNamespace( + which=lambda name: r"C:\bin\tesseract.exe" if name == "tesseract" else None)) + assert setup.ocr_status() == (True, setup.OCR_READY) + + +# --- the engine command has one writer --------------------------------------- + +def test_set_tesseract_cmd_is_what_status_checks(fake, tmp_path): + other = tmp_path / "other-tesseract.exe" + other.write_bytes(b"") + ac.set_tesseract_cmd(str(other)) + assert fake.pytesseract.tesseract_cmd == str(other) + assert TesseractBackend().cmd == str(other) + assert setup.ocr_status().ok is True + + +# --- set_tessdata_dir ---------------------------------------------------------- + +def test_set_tessdata_dir_sets_the_resolved_directory(monkeypatch, tmp_path): + monkeypatch.delenv(setup.TESSDATA_ENV, raising=False) + data = tmp_path / "tessdata" + data.mkdir() + resolved = setup.set_tessdata_dir(data) + assert resolved == os.path.realpath(data) + assert os.environ[setup.TESSDATA_ENV] == resolved + + +def test_set_tessdata_dir_none_restores_the_engine_default(monkeypatch, tmp_path): + monkeypatch.setenv(setup.TESSDATA_ENV, str(tmp_path)) + assert setup.set_tessdata_dir(None) is None + assert setup.TESSDATA_ENV not in os.environ + + +def test_set_tessdata_dir_refuses_what_is_not_a_directory(monkeypatch, tmp_path): + monkeypatch.setenv(setup.TESSDATA_ENV, "kept") + a_file = tmp_path / "eng.traineddata" + a_file.write_bytes(b"") + for bad in (tmp_path / "absent", a_file): + with pytest.raises(AutoControlActionException): + setup.set_tessdata_dir(bad) + assert os.environ[setup.TESSDATA_ENV] == "kept" + + +# --- find_tesseract_cmd ---------------------------------------------------------- + +def _no_path(monkeypatch): + monkeypatch.setattr(setup, "shutil", types.SimpleNamespace(which=lambda name: None)) + + +def test_find_prefers_the_environment_override(monkeypatch, tmp_path): + engine = tmp_path / "custom-tesseract.exe" + engine.write_bytes(b"") + monkeypatch.setenv(setup.TESSERACT_CMD_ENV, f" {engine} ") + monkeypatch.setattr(setup, "shutil", types.SimpleNamespace( + which=lambda name: pytest.fail("PATH must not be searched"))) + assert setup.find_tesseract_cmd() == os.path.abspath(engine) + + +def test_find_ignores_an_override_naming_no_file(monkeypatch, tmp_path): + monkeypatch.setenv(setup.TESSERACT_CMD_ENV, str(tmp_path / "absent.exe")) + found = tmp_path / "on-path" / "tesseract.exe" + monkeypatch.setattr(setup, "shutil", types.SimpleNamespace(which=lambda name: str(found))) + assert setup.find_tesseract_cmd() == os.path.abspath(found) + + +def test_find_falls_back_to_the_install_locations(monkeypatch, tmp_path): + monkeypatch.delenv(setup.TESSERACT_CMD_ENV, raising=False) + _no_path(monkeypatch) + installed = tmp_path / "Tesseract-OCR" / "tesseract.exe" + installed.parent.mkdir() + installed.write_bytes(b"") + monkeypatch.setattr(setup, "_install_candidates", + lambda: [str(tmp_path / "missing.exe"), str(installed)]) + assert setup.find_tesseract_cmd() == str(installed) + + +def test_find_returns_none_when_nothing_is_installed(monkeypatch, tmp_path): + monkeypatch.delenv(setup.TESSERACT_CMD_ENV, raising=False) + _no_path(monkeypatch) + monkeypatch.setattr(setup, "_install_candidates", lambda: [str(tmp_path / "missing.exe")]) + assert setup.find_tesseract_cmd() is None + + +def test_install_locations_per_platform(monkeypatch): + monkeypatch.setenv("ProgramFiles", r"D:\Apps") + monkeypatch.delenv("ProgramFiles(x86)", raising=False) + monkeypatch.setenv("LOCALAPPDATA", r"D:\Users\me\AppData\Local") + windows = setup._install_candidates("win32") + assert os.path.join(r"D:\Apps", "Tesseract-OCR", "tesseract.exe") in windows + assert os.path.join(r"C:\Program Files (x86)", "Tesseract-OCR", "tesseract.exe") in windows + assert os.path.join(r"D:\Users\me\AppData\Local", "Programs", "Tesseract-OCR", + "tesseract.exe") in windows + assert "/opt/homebrew/bin/tesseract" in setup._install_candidates("darwin") + assert "/usr/bin/tesseract" in setup._install_candidates("linux") + assert "/usr/bin/tesseract" in setup._install_candidates("freebsd14") + + +# --- wiring ------------------------------------------------------------------------ + +def test_executor_commands_return_json_friendly_dicts(fake): + from je_auto_control.utils.executor.action_executor import executor + assert executor.event_dict["AC_ocr_status"]() == {"ok": True, "reason": "ready"} + assert executor.event_dict["AC_ocr_languages"]() == {"languages": ["chi_tra", "eng", "jpn"]} + fake.languages_error = _NotFound("listing failed") + assert executor.event_dict["AC_ocr_languages"]() == {"languages": None} + + +def test_mcp_tools_and_script_builder_specs(): + from je_auto_control.utils.mcp_server.tools import build_default_tool_registry + tools = {tool.name: tool for tool in build_default_tool_registry()} + assert {"ac_ocr_status", "ac_ocr_languages"} <= set(tools) + assert tools["ac_ocr_status"].annotations.read_only is True + from je_auto_control.gui.script_builder.command_schema import _build_specs + assert {"AC_ocr_status", "AC_ocr_languages"} <= {s.command for s in _build_specs()} + + +def test_mcp_handlers_return_json_friendly_dicts(fake): + from je_auto_control.utils.mcp_server.tools import _handlers_screen + assert _handlers_screen.ocr_status() == {"ok": True, "reason": "ready"} + fake.languages = [] + assert _handlers_screen.ocr_languages() == {"languages": []} + + +def test_facade_exports(): + for name in ("OCRStatus", "find_tesseract_cmd", "ocr_languages", "ocr_status", + "set_tessdata_dir"): + assert hasattr(ac, name), name + assert name in ac.__all__, name