macOS: pace the window's frame to the display, and never wait for a drawable on the main thread

README.md

@@ -406,6 +406,60 @@ fixed. Physics history stores the creation pose before the first solver result,
 then snapshots before later fixed steps, keeping rendering between two real
 simulation poses without changing either.
 
+### The windowed frame loop on macOS
+
+The window host used to poll: `ControlFlow::Poll`, with the whole frame in
+`about_to_wait`. Under FIFO presentation that frame spends most of its time in
+`get_current_texture`, which on Metal is `nextDrawable` waiting for the next
+vsync, on the main thread. Measured on a game, the main thread was blocked
+there for 90 percent of the run, and it showed up somewhere nobody looks: with
+Magnet running, every click and the start of every title-bar drag waited about
+half a second, less on a 120 Hz display, while the host's own logs showed
+frames every 16.7 ms and input delivered within a millisecond.
+
+The delay sits in front of the process. Window managers such as Magnet and
+BetterTouchTool (its snap areas) install an event tap that filters mouse-down
+events, and before letting a click through they ask the clicked application
+about its window over Accessibility. That request is a message on the main run
+loop, which a polled loop serves one per turn, and every turn is a frame
+blocked until vsync, so a few dozen requests cost a few dozen frames. The
+press and release then reach the window together, a few milliseconds apart.
+winit#1737 is the same report from other engines and apps, and Godot ships a
+workaround that detects Magnet by its bundle identifier.
+
+The host now paces the windowed frame on macOS instead (`pacing` in the
+winit host). The event loop sleeps in `WaitUntil` until a frame is due, one
+monitor refresh after the last; the frame is requested with `request_redraw`
+and runs in `RedrawRequested`; and a worker thread acquires each drawable as
+soon as the previous frame is presented, so a due frame whose drawable is not
+in hand yet is put off and checked again every millisecond rather than waited
+for. Between frames the run loop is asleep in the kernel, where an
+Accessibility request wakes it at once. Pacing alone was not enough: a timer
+deadline does not stay in phase with vsync, so with every drawable queued the
+acquire still blocked for the phase difference, up to a whole frame, and clicks
+were still held for up to a hundred milliseconds. With the frame gated on a
+free drawable, measured with a diagnostic build that counted presented
+drawables, the share of the run spent blocked fell to nothing and the oldest
+click on arrival to about 16 ms, the tap's own cost.
+
+The alternatives that lost: a `CADisplayLink` driving the frame paced well but
+still blocked in acquire without the gate, and with it was no better than the
+timer while adding a macOS 14 dependency; one frame of latency instead of two
+halved the frame rate, because a single queued drawable is not released before
+the next frame wants it; and counting presented drawables needs a handler on
+the `CAMetalDrawable`, which wgpu does not expose, so it meant patching
+wgpu-hal. Acquiring on a worker uses only wgpu's public API (`Surface` and
+`SurfaceTexture` are `Send` and `Sync`) and answers exactly the question the
+gate asks. One consequence to know: wgpu-hal's Metal acquire reads the hosting
+`NSWindow`'s `occlusionState` to skip occluded windows, and that read now
+happens on the worker. A reconfigure first waits for the acquisition in flight
+and drops what it acquired, since a surface is not configured with a texture
+outstanding. A frame put off for 100 ms runs anyway and waits for its drawable
+as before, so a surface that stops answering cannot stop the game. Other
+platforms keep the polled loop: nothing there has shown the failure, and the
+render benchmark's interactive view keeps it too, since it measures uncapped
+presentation.
+
 ## Platform services this tree does not carry
 
 Three services the SDL implementation shipped are settled differently here, and

docs/gfx/benchmarks.md

@@ -26,6 +26,10 @@ supported. Otherwise they use nonblocking `Mailbox` presentation, which can
 render faster than the display. A surface supporting only synchronized FIFO
 presentation is rejected. The host prints the chosen mode.
 
+The interactive view also keeps the host's polled frame loop on every platform.
+An ordinary window on macOS is paced to the display instead, one frame per
+refresh, so its frame rate is the display's.
+
 The interactive frame rate includes window presentation and excludes initial
 scene setup. It counts submitted frames and uses elapsed wall time, including
 the cost of its statistics. It does not write a timing report. Use the fixed

native/rust/winit-host/src/graphics.rs

@@ -46,6 +46,7 @@ use crate::graph::{
     parse_graph, ClearMode, DepthMode, Graph, Input, TargetAllocator, TargetFormat, TargetStore,
     MAX_OUTPUTS,
 };
+use crate::pacing::{Acquire, Acquirer};
 use crate::packet::{
     parse_frame, Batch, RetainedScene, CAST_FANOUT, INSTANCE_STRIDE, LANE_BLEND, LANE_COUNT,
     LANE_OPAQUE, LIGHT_STRIDE, MAX_LIGHTS, SAMPLER_COUNT, SCENE_FLOATS,
@@ -478,7 +479,10 @@ pub struct Graphics {
     rendering_view: bool,
     asset_revision: u64,
     view_compositor: Option<views::Compositor>,
-    surface: Option<Surface<'static>>,
+    surface: Option<Arc<Surface<'static>>>,
+    /// Acquires the surface's drawables off the main thread, when the window's
+    /// frame loop asked for it. See `crate::pacing`.
+    acquirer: Option<Acquirer<SurfaceSource>>,
     offscreen: Option<wgpu::Texture>,
     capture_requested: bool,
     captured: Option<Result<Capture>>,
@@ -547,6 +551,16 @@ pub struct Graphics {
     failed: Arc<Mutex<Option<String>>>,
 }
 
+/// The window surface as the drawable source of an [`Acquirer`].
+struct SurfaceSource(Arc<Surface<'static>>);
+
+impl Acquire for SurfaceSource {
+    type Output = CurrentSurfaceTexture;
+    fn acquire(&self) -> CurrentSurfaceTexture {
+        self.0.get_current_texture()
+    }
+}
+
 fn benchmark_present_mode(supported: &[wgpu::PresentMode]) -> Result<wgpu::PresentMode> {
     [wgpu::PresentMode::Immediate, wgpu::PresentMode::Mailbox]
         .into_iter()
@@ -557,7 +571,15 @@ fn benchmark_present_mode(supported: &[wgpu::PresentMode]) -> Result<wgpu::Prese
 }
 
 impl Graphics {
-    pub fn new(window: Arc<Window>, display: OwnedDisplayHandle, benchmark: bool) -> Result<Self> {
+    /// Creates the renderer for a window. `threaded_acquire` acquires each
+    /// drawable on a worker thread (see `crate::pacing`), so that
+    /// [`Graphics::drawable_ready`] can answer without waiting for a vsync.
+    pub fn new(
+        window: Arc<Window>,
+        display: OwnedDisplayHandle,
+        benchmark: bool,
+        threaded_acquire: bool,
+    ) -> Result<Self> {
         let size = window.inner_size();
         let instance = Instance::new(InstanceDescriptor {
             display: Some(Box::new(display)),
@@ -615,7 +637,40 @@ impl Graphics {
             );
         }
         let pack = load_pack()?;
-        Self::assemble(Some(surface), device, queue, config, pack)
+        let surface = Arc::new(surface);
+        let mut graphics = Self::assemble(Some(Arc::clone(&surface)), device, queue, config, pack)?;
+        if threaded_acquire {
+            graphics.acquirer = Some(Acquirer::new(Arc::new(SurfaceSource(surface))));
+        }
+        Ok(graphics)
+    }
+
+    /// Reports whether the next frame can acquire its drawable without waiting.
+    /// Always true without a worker thread, and without a surface.
+    pub fn drawable_ready(&mut self) -> bool {
+        self.acquirer.as_mut().is_none_or(Acquirer::is_ready)
+    }
+
+    /// Reconfigures the surface, first dropping any drawable acquired for the
+    /// old configuration.
+    fn configure_surface(&mut self) {
+        if let Some(acquirer) = &mut self.acquirer {
+            acquirer.settle();
+        }
+        if let Some(surface) = &self.surface {
+            surface.configure(&self.device, &self.config);
+        }
+    }
+
+    /// The next drawable: from the worker thread when there is one (waiting
+    /// for it only if the frame loop ran out of patience), else acquired here.
+    fn acquire_frame(&mut self) -> Option<CurrentSurfaceTexture> {
+        if let Some(acquirer) = &mut self.acquirer {
+            return acquirer.take();
+        }
+        self.surface
+            .as_ref()
+            .map(|surface| surface.get_current_texture())
     }
 
     /// Uses the production frame graph on a fixed GPU target without a desktop surface.
@@ -663,7 +718,7 @@ impl Graphics {
     /// Split out so a test can hand in a headless device and the same code
     /// answers.
     fn assemble(
-        surface: Option<Surface<'static>>,
+        surface: Option<Arc<Surface<'static>>>,
         device: Device,
         queue: Queue,
         config: SurfaceConfiguration,
@@ -943,6 +998,7 @@ impl Graphics {
             asset_revision: 0,
             view_compositor: None,
             surface,
+            acquirer: None,
             offscreen,
             capture_requested: false,
             captured: None,
@@ -1011,9 +1067,7 @@ impl Graphics {
         }
         self.config.width = width;
         self.config.height = height;
-        if let Some(surface) = &self.surface {
-            surface.configure(&self.device, &self.config);
-        }
+        self.configure_surface();
     }
 
     /// Makes one image resident, replacing whatever already lived under its id.
@@ -1951,8 +2005,8 @@ impl Graphics {
             let _ = self.image_bind_group(batch.image, batch.sampler);
         }
 
-        let (frame, reconfigure) = if let Some(surface) = &self.surface {
-            match surface.get_current_texture() {
+        let (frame, reconfigure) = if let Some(acquired) = self.acquire_frame() {
+            match acquired {
                 CurrentSurfaceTexture::Success(frame) => (Some(frame), false),
                 CurrentSurfaceTexture::Suboptimal(frame) => (Some(frame), true),
                 CurrentSurfaceTexture::Timeout | CurrentSurfaceTexture::Occluded => {
@@ -1968,9 +2022,7 @@ impl Graphics {
                     // here and drawn next frame, which is one dropped frame rather
                     // than a dead window. The queued writes are flushed as above.
                     self.queue.submit(std::iter::empty());
-                    if let Some(surface) = &self.surface {
-                        surface.configure(&self.device, &self.config);
-                    }
+                    self.configure_surface();
                     return Ok(false);
                 }
                 CurrentSurfaceTexture::Validation => {
@@ -2266,9 +2318,12 @@ impl Graphics {
             self.queue.present(frame);
         }
         if reconfigure {
-            if let Some(surface) = &self.surface {
-                surface.configure(&self.device, &self.config);
-            }
+            self.configure_surface();
+        }
+        // The worker starts on the next drawable now, while this thread goes
+        // back to the event loop.
+        if let Some(acquirer) = &mut self.acquirer {
+            acquirer.request();
         }
         Ok(true)
     }

native/rust/winit-host/src/main.rs

@@ -8,6 +8,7 @@ mod graph;
 mod graphics;
 #[cfg(test)]
 mod lightingtests;
+mod pacing;
 mod packet;
 #[cfg(test)]
 mod pipelinetests;
@@ -74,6 +75,11 @@ struct App {
     shutdown: bool,
     failure: Option<String>,
     frame_parked: bool,
+    /// Paces the windowed frame on macOS (see `pacing`); `None` polls, with the
+    /// frame in `about_to_wait`.
+    pacer: Option<pacing::FramePacer>,
+    /// Frames until the refresh period is read from the monitor again.
+    refresh_check: u32,
     stats_started: Option<Instant>,
     stats_frames: u32,
     pending_captures: Vec<u64>,
@@ -95,6 +101,7 @@ impl App {
         })?;
         bridge.init().context("initialize the Tecs application")?;
         let now = Instant::now();
+        let pacer = paced_frame_loop(&config).then(|| pacing::FramePacer::new(now));
         Ok(Self {
             bridge,
             config,
@@ -111,6 +118,8 @@ impl App {
             shutdown: false,
             failure: None,
             frame_parked: false,
+            pacer,
+            refresh_check: 0,
             stats_started: None,
             stats_frames: 0,
             pending_captures: Vec::new(),
@@ -172,6 +181,7 @@ impl App {
             Arc::clone(&window),
             event_loop.owned_display_handle(),
             self.config.benchmark_view,
+            self.pacer.is_some(),
         )?;
         self.bridge.attach_window(&state)?;
         self.window_id = Some(window.id());
@@ -367,6 +377,44 @@ impl App {
         Ok(())
     }
 
+    /// Runs one paced frame (see `pacing`), from the `RedrawRequested` the pacer
+    /// asked for, or from one AppKit sent on its own (a live resize) when a
+    /// drawable is free. One AppKit sent while every drawable is taken is left
+    /// to the next due frame, which is at most a period away.
+    fn paced_frame(&mut self, event_loop: &ActiveEventLoop) {
+        let Some(pacer) = self.pacer.as_ref() else {
+            return;
+        };
+        if !pacer.redraw_requested() && !self.graphics.as_mut().is_none_or(Graphics::drawable_ready)
+        {
+            return;
+        }
+        if self.refresh_check == 0 {
+            self.refresh_check = 60;
+            let rate = self
+                .window
+                .as_ref()
+                .and_then(|window| window.current_monitor())
+                .and_then(|monitor| monitor.refresh_rate_millihertz());
+            if let Some(pacer) = self.pacer.as_mut() {
+                pacer.set_refresh_millihertz(rate);
+            }
+        }
+        self.refresh_check -= 1;
+        let now = Instant::now();
+        if let Some(pacer) = self.pacer.as_mut() {
+            pacer.frame_started(now);
+        }
+        if let Err(error) = self.tick(event_loop) {
+            self.fail(event_loop, error);
+        }
+        if self.frame_parked {
+            if let Some(pacer) = self.pacer.as_mut() {
+                pacer.retry_soon(Instant::now());
+            }
+        }
+    }
+
     fn tick(&mut self, event_loop: &ActiveEventLoop) -> Result<()> {
         if self.shutdown || event_loop.exiting() {
             return Ok(());
@@ -500,6 +548,10 @@ impl ApplicationHandler for App {
         if self.shutdown || event_loop.exiting() || Some(window_id) != self.window_id {
             return;
         }
+        if matches!(event, WindowEvent::RedrawRequested) && self.pacer.is_some() {
+            self.paced_frame(event_loop);
+            return;
+        }
         let result = match event {
             WindowEvent::CloseRequested => {
                 let timestamp = self.timestamp();
@@ -666,8 +718,32 @@ impl ApplicationHandler for App {
     }
 
     fn about_to_wait(&mut self, event_loop: &ActiveEventLoop) {
-        if let Err(error) = self.tick(event_loop) {
-            self.fail(event_loop, error);
+        if self.pacer.is_none() {
+            if let Err(error) = self.tick(event_loop) {
+                self.fail(event_loop, error);
+            }
+            return;
+        }
+        let ready = self.graphics.as_mut().is_none_or(Graphics::drawable_ready);
+        let Some(pacer) = self.pacer.as_mut() else {
+            return;
+        };
+        let flow = match pacer.step(Instant::now(), ready) {
+            pacing::Step::Redraw => {
+                if let Some(window) = &self.window {
+                    window.request_redraw();
+                    ControlFlow::Wait
+                } else {
+                    // No window to redraw: the frame runs here.
+                    self.paced_frame(event_loop);
+                    ControlFlow::Poll
+                }
+            }
+            pacing::Step::WaitUntil(instant) => ControlFlow::WaitUntil(instant),
+            pacing::Step::Wait => ControlFlow::Wait,
+        };
+        if !event_loop.exiting() {
+            event_loop.set_control_flow(flow);
         }
     }
 
@@ -932,6 +1008,7 @@ fn run() -> Result<()> {
         return run_headless(config);
     }
     let event_loop = EventLoop::new().context("create the winit event loop")?;
+    // A paced loop sets its own control flow in `about_to_wait`.
     event_loop.set_control_flow(ControlFlow::Poll);
     let mut app = App::new(config)?;
     event_loop
@@ -943,6 +1020,17 @@ fn run() -> Result<()> {
     Ok(())
 }
 
+/// Whether the window's frame is paced by `pacing` rather than polled.
+///
+/// On macOS the polled loop spends most of each frame blocked on a drawable on
+/// the main thread, which delays every click by as many frames as a window
+/// manager's event tap makes Accessibility requests (see `pacing`). Elsewhere the
+/// polled loop stays: nothing there has shown the same failure. The render
+/// benchmark's view measures uncapped frame rates and polls everywhere.
+fn paced_frame_loop(config: &Config) -> bool {
+    cfg!(target_os = "macos") && !config.benchmark_view
+}
+
 /// Assembles the material dispatch into a shader pack a release loads.
 ///
 /// `--pack-shaders <materials directory> <output file>`. This is the packaging