Editor: camera cuts, audio silencing, 50x speed, native preview audio & reliable export

AGENTS.md

@@ -88,6 +88,10 @@ src/
     native.ts                   typed Tauri command wrappers + listen helpers + sidecar types
     autoZoom.ts                 derive zoom segments from cursor-sidecar clicks
     effects.ts                  plugin registry + timeline effect segments (color grade/vignette/grain/glow)
+    clips.ts                    clip sequence (output ↔ source time, split/trim/speed)
+    cameraCuts.ts               camera-track cuts with per-cut time shift / hidden gaps
+    audioPieces.ts              audio-track split pieces with per-piece silence
+    captions.ts                 auto-subtitle model + Canvas2D caption renderer
   styles/
     globals.css                 design system (~2100 lines, ported from the handoff)
     tokens.css                  CSS custom properties / design tokens
@@ -104,7 +108,10 @@ src-tauri/
   src/project.rs                .openscreen save/open + Finder double-click handoff
   src/export.rs                 export sessions: frame streaming + ffmpeg encode/mux
   src/camera.rs                 AVFoundation camera: enumeration, TCC auth, sidecar recorder
-  src/audio_tap.rs              SCK PCM taps → sidecar WAVs, segment concat
+  src/audio_tap.rs              SCK PCM taps → sidecar WAVs (format-aware: f32/i16/i24/i32, BE/LE)
+  src/preview_audio.rs          native editor audio playback (cpal, mmap'd sidecar WAVs)
+  src/video_frames.rs           ffmpeg-decoded RGBA frames for export (per-session decoders)
+  src/transcribe.rs             OpenAI key storage + whisper-1 transcription for captions
   src/main.rs                   thin entry; also handles --probe-screen-recording subprocess
   build.rs                      tauri-build
   binaries/                     bundled ffmpeg-{aarch64,x86_64}-apple-darwin (gitignored, fetched)
@@ -129,14 +136,21 @@ scripts/fetch-ffmpeg.sh         downloads the platform ffmpeg binaries (run on p
   - mic: `start_mic_meter`, `stop_mic_meter`
   - picker: `list_displays`, `list_windows`, `open_picker_overlays`, `close_picker_overlays`, `picker_select_display`, `picker_select_window`, `picker_select_area`
   - editor/project: `open_editor_with_artifact`, `open_dev_editor`, `confirm_and_discard_editor`, `present_editor`, `save_project`, `open_project`
-  - export: `export_begin`, `export_frame`, `export_finish`, `export_cancel`, `pick_export_path`, `copy_file_to_clipboard`
+  - export: `export_begin`, `export_frame`, `export_finish`, `export_cancel`, `pick_export_path`, `copy_file_to_clipboard`, `export_video_frame`
+  - preview audio: `preview_audio_load`, `preview_audio_set`, `preview_audio_gains`, `preview_audio_pos`
+  - captions: `openai_key_status`, `openai_key_set`, `openai_key_clear`, `transcribe_audio`
   - wallpaper: `list_macos_wallpapers`, `current_macos_wallpaper`
   - app: `quit_app` (exit unless the editor window is visible)
 
   The SCK stream + recording output live in `RecordingState`; the `cpal` mic stream owns its own thread because `cpal::Stream` is `!Send` on macOS; `CursorTrack` runs a polling thread and serializes the sidecar on stop.
 - **Frontend ↔ Rust bridge** — [`src/lib/native.ts`](src/lib/native.ts). The `native` object wraps every command; listen helpers: `onMicLevel`, `onRecordingArtifact`, `onRecordingStoppedExternally`, `onPickerSelected`, `onPickerState`, `onPickerHover`, `onMenuSaveProject`, `onMenuOpenProject`. Cursor-sidecar TypeScript types (`CursorSidecar`, `CursorSidecarShapeName`, …) live here too. Add new commands here, keep return types narrow.
 - **Editor** — [`src/components/Editor/index.tsx`](src/components/Editor/index.tsx). The largest component. Owns video playback, timeline scrubbing, auto-zoom segments (via `lib/autoZoom.ts` + the cursor sidecar), the cursor overlay, preview-quality settings, and project save/open.
-- **Clip sequence (cutting / speed)** — [`src/lib/clips.ts`](src/lib/clips.ts). `EditorState.clips` is the edited output: an ordered list of source ranges, each with its own `speed` (0.25–16×) and `muted` flag (empty = whole recording). The timeline axis and the editor's `currentTime` are **output** time; zoom/effect segments, camera keyframes, audio-track trims and the cursor sidecar stay in **source** time and are mapped per clip for display (`srcRangePieces`, `outToSrc`, `srcToOut`). Preview walks the clips in a rAF loop (seek + `playbackRate` per clip); export renders output frames and emits one audio spec per clip × track with an ffmpeg `atempo` for the speed. Pre-v4 projects' `trimStart`/`trimEnd` are converted to a single clip on load.
+- **Clip sequence (cutting / speed)** — [`src/lib/clips.ts`](src/lib/clips.ts). `EditorState.clips` is the edited output: an ordered list of source ranges, each with its own `speed` (0.25–50×, presets up to 50×, custom input) and `muted` flag (empty = whole recording). The timeline axis and the editor's `currentTime` are **output** time; zoom/effect segments, camera keyframes, audio-track trims and the cursor sidecar stay in **source** time and are mapped per clip for display (`srcRangePieces`, `outToSrc`, `srcToOut`). Preview walks the clips in a rAF loop (seek + `playbackRate` per clip); export renders output frames and emits one audio spec per clip × track with an ffmpeg `atempo` for the speed. Pre-v4 projects' `trimStart`/`trimEnd` are converted to a single clip on load. Above `NATIVE_RATE_MAX` (16×, WebKit's `playbackRate` cap) the preview seek-steps instead of playing. Splitting is CapCut-style: `S` with nothing selected cuts clips, camera cuts and audio pieces at the playhead (`splitAt`); with a block selected only that block is cut.
+- **Camera cuts** — [`src/lib/cameraCuts.ts`](src/lib/cameraCuts.ts). The camera row can be split independently; each `CameraCut` has a `shift` (s) applied to the camera source time, so a desynced camera can be slid back into sync (drag or ⌥←/→). Gaps between cuts hide the camera in preview and export.
+- **Audio pieces** — [`src/lib/audioPieces.ts`](src/lib/audioPieces.ts). Mic/system tracks can be split into `AudioPiece`s; ⌫ toggles `muted` on the selected piece. Export builds its audio specs from `audibleRanges`, the preview zeroes gains via `isSilencedAt`.
+- **Preview audio is native** — WKWebView's Web Audio output is silent in the bundled app, so the editor drives [`preview_audio.rs`](src-tauri/src/preview_audio.rs) (cpal/CoreAudio) with `preview_audio_set(playing, t, rate)` and per-track gains. Times sent to it are **source** time; compare them against the source duration, not the output duration.
+- **Export frames are decoded natively** — WebKit purges paused `<video>` decoders under memory pressure (frozen frames, crashes on multi-GB files), so the exporter pulls RGBA frames from [`video_frames.rs`](src-tauri/src/video_frames.rs) (`export_video_frame`, a forward-streaming ffmpeg with VideoToolbox, BT.709). Decoders are closed in `cleanup_session`. Before creating a blob URL for preview, `useSameOriginSrc` probes the file size (`Range: bytes=0-0`) and streams large files instead of fetching them whole.
+- **Auto-subtitles** — [`src/lib/captions.ts`](src/lib/captions.ts) + [`transcribe.rs`](src-tauri/src/transcribe.rs). The OpenAI key is stored in a 0600 file and never returned to the webview; audio is chunked and sent to whisper-1. Captions are drawn by one Canvas2D renderer for both the preview overlay and export (`GLCompositor.drawOverlay`).
 - **Auto-zoom** — [`src/lib/autoZoom.ts`](src/lib/autoZoom.ts). Pure logic: derives `ZoomSegment`s from sidecar clicks; the editor renders/edits them.
 - **Plugin-based timeline video effects** — [`src/lib/effects.ts`](src/lib/effects.ts) is the plugin registry: each `EffectPluginDef` (Black & White, Sepia, Vintage Film, Vignette, Warm Glow, Cool Tone, Dreamy Soft, High Contrast, Film Grain, Invert Flash, …) declares only the `PostFxParams` keys it changes at full strength (brightness/contrast/saturation/temperature/hueRotate/sepia/invert/vignette/grain/blur — the param set lives in `lib/compositor.ts`); adding a look is one registry entry, no renderer changes. `EffectSegment`s place a plugin + 0–1 intensity over a time range on the timeline's Effects track (mirrors `ZoomSegment`, own crossfade at segment edges). `resolveEffectParams(tMs, segments)` blends the active segment toward neutral; `renderFrame` (Canvas2D) and `GLCompositor.render()` (WebGL, via an offscreen FBO + fullscreen grading shader) both apply the *same* resolved params as the last compositing step, so preview and export match. The DOM (non-GL) preview fallback applies only the CSS-filter-representable subset (`postFxCssFilter`) directly to the recorded-window element.
 - **Design system** — [`src/styles/globals.css`](src/styles/globals.css) and [`tokens.css`](src/styles/tokens.css) are the single source of truth for colors, spacing, and component classes. **Do not** restyle components inline; reuse existing class names.

src-tauri/Cargo.toml

@@ -18,6 +18,7 @@ crate-type = ["staticlib", "cdylib", "rlib"]
 tauri-build = { version = "2", features = [] }
 
 [dependencies]
+memmap2 = "0.9"
 tauri = { version = "2", features = ["macos-private-api", "protocol-asset"] }
 tauri-plugin-opener = "2"
 tauri-plugin-dialog = "2"

src-tauri/src/audio_tap.rs

@@ -34,6 +34,7 @@ struct PendingBuf {
     channels: u16,
     sample_rate: u32,
     samples: Vec<f32>, // interleaved
+    fmt: SampleFmt,
 }
 
 struct TapState {
@@ -49,10 +50,35 @@ pub struct AudioTap {
     anchor: VideoAnchor,
 }
 
-fn bytes_to_f32(bytes: &[u8]) -> Vec<f32> {
+#[derive(Clone, Copy, Debug)]
+struct SampleFmt {
+    float: bool,
+    big_endian: bool,
+    bits: u32,
+}
+
+/// Decode PCM bytes to f32 in [-1, 1] per the stream's real format; external
+/// mics often deliver integer PCM rather than SCK's usual f32.
+fn bytes_to_f32(bytes: &[u8], f: SampleFmt) -> Vec<f32> {
+    let width = ((f.bits + 7) / 8).max(1) as usize;
     bytes
-        .chunks_exact(4)
-        .map(|c| f32::from_le_bytes([c[0], c[1], c[2], c[3]]))
+        .chunks_exact(width)
+        .map(|c| {
+            let mut b = [0u8; 8];
+            b[..width].copy_from_slice(c);
+            if f.big_endian {
+                b[..width].reverse();
+            }
+            let v = match (f.float, width) {
+                (true, 4) => f32::from_le_bytes([b[0], b[1], b[2], b[3]]),
+                (true, 8) => f64::from_le_bytes(b) as f32,
+                (false, 2) => i16::from_le_bytes([b[0], b[1]]) as f32 / 32_768.0,
+                (false, 3) => (i32::from_le_bytes([0, b[0], b[1], b[2]]) >> 8) as f32 / 8_388_608.0,
+                (false, 4) => i32::from_le_bytes([b[0], b[1], b[2], b[3]]) as f32 / 2_147_483_648.0,
+                _ => 0.0,
+            };
+            if v.is_finite() { v.clamp(-1.0, 1.0) } else { 0.0 }
+        })
         .collect()
 }
 
@@ -65,6 +91,11 @@ fn extract(sample: &CMSampleBuffer) -> Option<PendingBuf> {
         return None;
     }
     let sample_rate = fmt.audio_sample_rate().unwrap_or(48_000.0).round() as u32;
+    let sfmt = SampleFmt {
+        float: fmt.audio_is_float(),
+        big_endian: fmt.audio_is_big_endian(),
+        bits: fmt.audio_bits_per_channel().filter(|b| *b > 0).unwrap_or(32),
+    };
     let pts = {
         let t = sample.presentation_timestamp();
         if t.timescale == 0 {
@@ -79,10 +110,10 @@ fn extract(sample: &CMSampleBuffer) -> Option<PendingBuf> {
     }
     let (channels, samples) = if n == 1 {
         let buf = list.get(0)?;
-        ((buf.number_channels.max(1) as u16), bytes_to_f32(buf.data()))
+        ((buf.number_channels.max(1) as u16), bytes_to_f32(buf.data(), sfmt))
     } else {
         let chans: Vec<Vec<f32>> = (0..n)
-            .filter_map(|i| list.get(i).map(|b| bytes_to_f32(b.data())))
+            .filter_map(|i| list.get(i).map(|b| bytes_to_f32(b.data(), sfmt)))
             .collect();
         if chans.len() != n {
             return None;
@@ -96,7 +127,7 @@ fn extract(sample: &CMSampleBuffer) -> Option<PendingBuf> {
         }
         (n as u16, out)
     };
-    Some(PendingBuf { pts, channels, sample_rate, samples })
+    Some(PendingBuf { pts, channels, sample_rate, samples, fmt: sfmt })
 }
 
 fn write_interleaved(writer: &mut WavWriter, samples: &[f32]) {
@@ -108,6 +139,11 @@ fn write_interleaved(writer: &mut WavWriter, samples: &[f32]) {
 /// Write `buf` as the first audio of the file: create the writer, then align
 /// the stream to `anchor` by dropping early samples or prepending silence.
 fn write_first(state: &mut TapState, path: &Path, anchor: f64, buf: PendingBuf) {
+    eprintln!(
+        "[audio] {} format: {:?}, {} ch, {} Hz",
+        path.file_name().map(|n| n.to_string_lossy()).unwrap_or_default(),
+        buf.fmt, buf.channels, buf.sample_rate
+    );
     let spec = hound::WavSpec {
         channels: buf.channels,
         sample_rate: buf.sample_rate,

src-tauri/src/export.rs

@@ -567,6 +567,7 @@ pub(crate) fn export_finish(
     // only drop the PNG frames. Otherwise remove the whole session dir.
     if out_path.is_empty() {
         let _ = remove_export_pngs(&dir);
+        close_frame_decoders(&session_id);
         EXPORT_SESSIONS.lock().remove(&session_id);
     } else {
         cleanup_session(&session_id);
@@ -586,6 +587,7 @@ pub(crate) fn remove_export_pngs(dir: &Path) {
 }
 
 pub(crate) fn cleanup_session(session_id: &str) {
+    close_frame_decoders(session_id);
     if let Some(s) = EXPORT_SESSIONS.lock().remove(session_id) {
         let _ = fs::remove_dir_all(&s.dir);
     }

src-tauri/src/lib.rs

@@ -64,7 +64,10 @@ mod editor;
 mod export;
 mod permissions;
 mod picker;
+mod preview_audio;
 mod project;
+mod transcribe;
+mod video_frames;
 
 pub use permissions::probe_screen_recording_and_exit;
 
@@ -73,6 +76,9 @@ use editor::*;
 use export::*;
 use permissions::*;
 use picker::*;
+use preview_audio::*;
+use transcribe::*;
+use video_frames::*;
 use project::*;
 
 #[cfg_attr(mobile, tauri::mobile_entry_point)]
@@ -100,6 +106,7 @@ pub fn run() {
         .manage(MeterState::default())
         .manage(PickerTrackerState::default())
         .manage(PendingProjectFile::default())
+        .manage(PreviewAudio::default())
         .setup(|app| {
             // In debug builds (or when OSS_SKIP_PERMS=1) skip the onboarding
             // window — `tauri dev` runs as the dev shell and re-prompts every
@@ -332,10 +339,19 @@ pub fn run() {
             export_frame,
             export_finish,
             export_cancel,
+            export_video_frame,
             copy_file_to_clipboard,
             pick_export_path,
             quit_app,
+            preview_audio_load,
+            preview_audio_set,
+            preview_audio_gains,
+            preview_audio_pos,
             take_pending_project_file,
+            openai_key_status,
+            openai_key_set,
+            openai_key_clear,
+            transcribe_audio,
         ])
         .build(tauri::generate_context!())
         .expect("error while running tauri application")