Skip to content

perf(daemon): refresh before each request from native file watcher events - #217

Open
sergej-koscejev wants to merge 5 commits into
mainfrom
perf/188-native-file-watcher
Open

sergej-koscejev wants to merge 5 commits into
mainfrom
perf/188-native-file-watcher

Conversation

@sergej-koscejev

Copy link
Copy Markdown
Member

Summary

Fixes #188

Before each request, the daemon now refreshes project and module folders from the native file watcher (fsnotifier) events. It walks the folders only when it cannot trust the watcher.

 ExternalProjectRefresh.refresh
+  WatchedFolderRefresh.begin                        # outside the EDT
+    wait until the watch roots are set
+    fence: create <system>/mops-file-watcher-fence/<uuid>/fence, wait until the watcher reports it
+    flush the EDT (watcher failure notifications arrive there)
+    trusted = fence reported && no failure && the watch roots and fsnotifier starts are unchanged since the last refresh
-  IFile.refresh(.mps, forced)
+  .mps:            trusted ? refreshFiles(non-forced) : markDirtyAndRefresh
   StoreReloadManager.reloadChangedStorageFiles
-  markDirtyAndRefresh(module folders)
+  module folders:  same; a folder walks anyway when new, outside every recursive watch root, or (Windows) on another drive than the fence
   archives, ReloadManager.flush, class path snapshot   # unchanged
+  log "refreshed external changes using <file watcher events | a full walk because …>: …; watcher wait N ms, refresh N ms"

What it adds:

  • daemon-mps-plugin: HeadlessNativeFileWatcher, a second pluggableFileWatcher with isDisabled() limited to idea.filewatcher.disabled. It counts watch-root changes and fsnotifier starts (myStartAttemptCount, via reflection). After a File Watcher Messages notification (inotify limit, gave up, not executable), it reports itself as untrusted.
  • FileWatcherExecutable: before MPS starts, the daemon copies bin/<os>/fsnotifier into <workspace>/fsnotifier/, sets the executable bit, and sets idea.filewatcher.executable.path. For Linux on aarch64 it uses bin/linux-aarch64. With no binary for the platform, it sets idea.filewatcher.disabled=true. MOPS_DAEMON_JAVA_OPTIONS=-Didea.filewatcher.disabled=true turns the watcher off.
  • FileWatcherFence: the fence from the issue. It hooks FileWatcher.myTestNotifier and watches its own root with addRootToWatch.
  • ProjectRequestGate: only readers queued before a refresh started share that refresh. Before this change, a reader that arrived during a refresh shared it and could miss a change made just before the reader arrived. The fence would not close that gap.

Decisions (the issue left these open):

  • Who copies fsnotifier: the daemon copies it, not the CLI launcher. The daemon owns its workspace, and daemon tests (SharedMpsEnvironment) get the same watcher this way. The alternative is the launcher passing -D…executable.path.
  • Where the fence lives: in the IDEA system directory, registered as its own watch root. It does not rely on <config>/plugins being watched by chance.
  • Startup walk: the "one forced walk at startup" happens on the first request, like today's first refresh, after waiting for the watch roots to be set.
  • Untrusted watcher: after a watcher failure notification, the daemon walks on every refresh until it restarts.

Not done: making the class path snapshot() event-driven (#189).

Evidence

  • Fence test (FileWatcherRefreshSemanticsTest): 20 times, rename a node in a model file, then refresh right away, then read the node. The test runs once for a project model and once for a library model outside the project folder. On macOS and on Linux x86-64 it also asserts that the refresh used watcher events.

    • Before: with the fence wait replaced by return true, both tests fail: the refresh misses the edit.
    • After: both pass on 2024.1.6, 2025.1.4, 2026.1.2 and 263.7307.178 (macOS arm64).
  • Watcher off: LibraryModelRefreshIntegrationTest runs the existing scenario once with the watcher and once with -Didea.filewatcher.disabled=true through the installed CLI. The second run checks that edits are visible, that the log says a full walk because the file watcher is not operational, and that no refresh used watcher events. Both pass on 2026.1.1.

  • Wider runs: the whole daemon suite passes on 2024.1.6 and 2025.1.4 with the watcher on. One TestReportStoreTest failure on 2025.1.4 was a NoClassDefFoundError from a jar I rebuilt during the run; it passed when I ran it again. ProjectRequestGateTest has a new case: a reader that arrives during a refresh waits for its own refresh.

  • Per-request cost: a copy of MPS-extensions with MPS 2024.1.6 (260 modules, 51,757 files, built), macOS arm64. The machine was busy with other builds, so wall times are noisy.

    Per request main / watcher off This PR, watcher on
    Folder refresh (daemon log) 158–354 ms, median about 190 ms 4–10 ms (one at 54 ms)
    Watcher wait (roots settled + fence + EDT flush) — 30–308 ms, median about 80 ms
    mops list --depth 0, wall time, 25 runs, median 651 ms (main) 588 ms
    Same, right after an external model edit, 10 runs median 820 ms (main) median 620 ms

    mops daemon ping takes 350–500 ms of CLI time and does not refresh. The first refresh walks every folder (261 ms) and loads the files the VFS has stored.

Merge Danger

Blast Radius: daemon-wide

Every daemon now starts a fsnotifier process. Whether a request sees external changes now depends on the fence and the trust checks. If a check is wrong, a request misses an external edit and nothing reports it. The test that asserts watcher use runs only on macOS and Linux x86-64. Not exercised at runtime:

  • Linux root re-registration.
  • The inotify-limit notification path.
  • Windows. On Windows the fence path is matched as a string; if fsnotifier reports another case, the fence times out (2 s) and the refresh walks the folders.

The new gate rule can split a reader batch into more refreshes when readers arrive while a refresh is running.

…watcher

# Conflicts:
#	docs/test-running-design.md
@sergej-koscejev
sergej-koscejev added this pull request to the merge queue Nov 10, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

daemon: refresh before each request driven by the native file watcher, with the full walk as fallback

1 participant