Add a General MIDI synthesizer: tecs.audio.synth over rustysynth

AGENTS.md

@@ -108,7 +108,7 @@ tecs/
 │   ├── platform/           # Window, events, audio and gamepad backends
 │   ├── physics/            # Rapier contract and binding
 │   ├── input.nupp          # Gameplay input over the platform event stream
-│   ├── audio.nupp          # Clips, voices, groups, limits, the Sound component
+│   ├── audio/              # Clips, voices, groups, limits, the Sound component, General MIDI
 │   ├── sequence.nupp       # Sequencer, with the tween runtime inside it
 │   ├── assets.nupp         # Asset orchestration
 │   ├── mcp.nupp            # Debug server: transport, tools, sandbox

Cargo.toml

@@ -47,6 +47,7 @@ rayon = "1.12"
 reqwest = { version = "0.13.4", default-features = false, features = ["deflate", "gzip", "rustls-no-provider", "socks", "stream"] }
 resvg = { version = "0.47", default-features = false, features = ["text"] }
 regex = "1.12"
+rustysynth = "1.3.7"
 rmcp = { version = "3.0.0", default-features = false, features = ["client", "server", "transport-io", "transport-streamable-http-client-reqwest", "transport-streamable-http-server"] }
 rustls = { version = "0.23", default-features = false, features = ["ring", "std", "tls12"] }
 serde = { version = "1", features = ["derive"] }

README.md

@@ -50,8 +50,9 @@ test suite reach the same world without starting a graphics stack.
 - Camera, layers with positioning modes, images, clips, sprite sheets with
   animation, and distance-field text.
 - Input across keyboard, pointer, gamepad and touch; audio playback, recording
-  and device enumeration; physics over Rapier; sequencing with tweening; asset
-  orchestration; file watching; and an MCP debug server.
+  and device enumeration; General MIDI from a SoundFont; physics over Rapier;
+  sequencing with tweening; asset orchestration; file watching; and an MCP
+  debug server.
 
 - [Tiled TMX maps and TSX tilesets](docs/tiled/index.md), animated tile and image
   layers, object factories, runtime tile edits and authored collision outlines.
@@ -488,6 +489,49 @@ stream did. It holds one second by default, drops the oldest, and reports the
 loss through `Microphone.overruns`. An unbounded allocation driven by a
 realtime callback was not worth carrying across.
 
+## General MIDI is synthesized in-process
+
+`tecs.audio.synth` renders General MIDI from a SoundFont with `rustysynth`
+(MIT, pure Rust, no dependencies) as one voice of the mixer, rather than
+sending MIDI to whatever synthesizer the operating system offers. On a desktop
+the system synthesizer is software too: Apple's DLS synth on macOS, the
+Microsoft GS Wavetable synth on Windows, and on Linux nothing at all unless the
+player installed and started FluidSynth or TiMidity. Sending to it buys no
+quality and gives up three things a game needs. The music sounds the same on
+every machine, because the SoundFont ships with the game instead of being
+whatever bank the platform has. It goes through the mixer, so group gain, mute,
+pause and master gain apply to it like any other sound, and a system synth
+offers none of them. And it is deterministic: events carry a time on the
+synthesizer's own clock, the callback splits its render at those times, and the
+same schedule renders the same samples on every run, which a test can pin and
+a recording can reproduce. An external synth answers on its own clock and its
+own latency, and the mix of the two cannot be captured offline.
+
+Timing is exact to `rustysynth`'s internal block, which is where it applies
+MIDI and runs its envelopes. Tecs sets that block to eight frames, the smallest
+the library takes: every event lands on the first eight-frame boundary at or
+after its time, 0.17 milliseconds at worst at 48 kHz. Measured on FluidR3 GM
+with 48 notes held and the effects on, that costs 3.3 percent of one core
+against 1.9 at the library's default of 64 frames.
+
+Events cross to the output callback through a bounded single-producer,
+single-consumer ring, 4096 events, with as many again held in a preallocated
+schedule for events sent ahead of their time. The callback never waits and
+never allocates; a send that finds no room returns false and the event is
+dropped, because the alternatives are blocking the frame thread or growing a
+buffer the callback reads. A game streaming a song with a lookahead of a frame
+or two never comes near it.
+
+A hardware MIDI output is the extension this leaves room for and does not
+build. The seam is `tecs.platform.audiobackend.SynthPort`: `send` with a time,
+`notesOff`, `reset` and a clock. A second implementation over `midir`
+(CoreMIDI, WinMM, ALSA) would put a sender thread in Rust that holds the
+schedule and writes each message when its time comes, with `now` on the host's
+monotonic clock instead of rendered frames, chosen when the synth is made.
+Nothing above the port changes. It is deferred because nothing needs it yet,
+and what it costs is the determinism above: a hardware synth cannot be rendered
+offline.
+
 ## Device vocabularies follow the library that reports them
 
 Key names, mouse button names and gamepad button and axis names came from SDL

THIRD_PARTY_NOTICES.md

@@ -81,6 +81,12 @@ Open Font License 1.1. The tests and retained UI example read it. Packages ship
 `share/tecs/assets/fonts` with `JetBrainsMono-OFL.txt` and the versioned
 `JetBrainsMono-NOTICE.md`.
 
+## SoundFonts
+
+Tecs ships no SoundFont. `tecs.audio.synth` reads one a game supplies, and a
+General MIDI set is game content under its own licence, so a game that ships
+one ships that licence beside it. FluidR3 GM, for example, is MIT.
+
 ## Scope of the checks
 
 The generated inventory and source manifest make shipped dependency names,

native/rust/tecs-audio/Cargo.toml

@@ -11,4 +11,5 @@ crate-type = ["cdylib", "rlib"]
 
 [dependencies]
 cpal.workspace = true
+rustysynth.workspace = true
 symphonia.workspace = true

native/rust/tecs-audio/examples/synthscale.rs

@@ -0,0 +1,72 @@
+//! Plays a C major scale on the default output through the synthesizer.
+//!
+//! A manual check for a real machine, kept out of the test suite because it
+//! opens the output device, which on macOS can block without bound when the
+//! audio daemon is unreachable. It takes the SoundFont as its argument or from
+//! `TECS_TEST_SOUNDFONT`, and says it skipped when given neither.
+//!
+//! ```text
+//! cargo run -p tecs-audio --example synthscale -- path/to/FluidR3_GM.sf2
+//! ```
+
+use std::time::Duration;
+
+use tecsaudio::command::{self, Command};
+use tecsaudio::engine::Engine;
+use tecsaudio::synth::Settings;
+
+fn main() {
+    let path = std::env::args()
+        .nth(1)
+        .or_else(|| std::env::var("TECS_TEST_SOUNDFONT").ok());
+    let Some(path) = path else {
+        eprintln!("skipped: name a SoundFont as the argument or in TECS_TEST_SOUNDFONT");
+        return;
+    };
+
+    let engine = Engine::open(48_000, 2, 8);
+    if !engine.available() {
+        eprintln!("no output device: {}", engine.last_error());
+    }
+    let font = match engine.load_sound_font(&path) {
+        Ok(font) => font,
+        Err(reason) => {
+            eprintln!("{reason}");
+            std::process::exit(1);
+        }
+    };
+    let handle = 1;
+    let mut port = engine
+        .open_synth(font, handle, &Settings::default())
+        .expect("the synthesizer opens");
+    engine.submit(&[Command {
+        kind: command::PLAY,
+        handle,
+        clip: 0,
+        flags: command::FLAG_SYNTH,
+        gain: 1.0,
+        pitch: 1.0,
+        start: 0.0,
+        fade: 0.0,
+        loop_start: 0.0,
+        x: 0.0,
+        y: 0.0,
+        z: 0.0,
+    }]);
+
+    // Scheduled ahead in one go, a quarter of a second apart, against the
+    // synthesizer clock rather than against when this thread wakes up.
+    let start = port.now() + 0.1;
+    let keys = [60u8, 62, 64, 65, 67, 69, 71, 72];
+    for (step, key) in keys.iter().enumerate() {
+        let at = start + step as f64 * 0.25;
+        port.send(0x90, *key, 100, at);
+        port.send(0x80, *key, 0, at + 0.22);
+    }
+    std::thread::sleep(Duration::from_millis(3000));
+    eprintln!(
+        "played {} notes; the synthesizer clock reads {:.3} s",
+        keys.len(),
+        port.now()
+    );
+}

native/rust/tecs-audio/src/command.rs

@@ -51,6 +51,10 @@ pub const FLAG_STEREO: u32 = 4;
 /// Marks a `PLAY` reading its input from the file rather than held samples.
 pub const FLAG_STREAM: u32 = 8;
 
+/// Marks a `PLAY` whose input is the synthesizer `tecs_audio_synth_open`
+/// parked under the command's handle, rather than a clip.
+pub const FLAG_SYNTH: u32 = 16;
+
 /// Requests one operation.
 ///
 /// One struct covers every operation, because the array crossing the seam has

native/rust/tecs-audio/src/engine.rs

@@ -16,6 +16,7 @@ use crate::device::{self, Output};
 use crate::mixer::Mixer;
 use crate::source::{ResidentSource, Source, MAX_CHANNELS};
 use crate::stream::Feeder;
+use crate::synth::{self, Port, Settings};
 
 /// Measures a clip's duration against the streaming threshold.
 pub const MODE_AUTO: u32 = 0;
@@ -52,6 +53,13 @@ pub struct Loaded {
 pub struct Engine {
     mixer: Arc<Mutex<Mixer>>,
     clips: RwLock<HashMap<u32, Clip>>,
+    /// SoundFonts by the id `load_sound_font` handed out, shared by every
+    /// synthesizer opened on one.
+    fonts: RwLock<HashMap<u32, Arc<rustysynth::SoundFont>>>,
+    next_font: Mutex<u32>,
+    /// Synthesizers opened and not yet started, by the voice handle whose
+    /// `PLAY` will take them.
+    synths: Mutex<HashMap<u32, Box<dyn Source>>>,
     feeder: Feeder,
     output: Option<Output>,
     /// The reason the most recent failing call gave, kept so the binding can
@@ -115,6 +123,9 @@ impl Engine {
             failed,
             mixer,
             clips: RwLock::new(HashMap::new()),
+            fonts: RwLock::new(HashMap::new()),
+            next_font: Mutex::new(0),
+            synths: Mutex::new(HashMap::new()),
             feeder: Feeder::start(),
             output,
             last_error: Mutex::new(reason),
@@ -291,8 +302,67 @@ impl Engine {
         }
     }
 
+    /// Reads a SoundFont and returns the id synthesizers name it by.
+    ///
+    /// @return the id, from one up, or the reason nothing was loaded
+    pub fn load_sound_font(&self, path: &str) -> Result<u32, String> {
+        let font = synth::load_sound_font(path).map_err(|reason| self.fail(reason))?;
+        let id = match self.next_font.lock() {
+            Err(_) => return Err(self.fail("the SoundFont counter is poisoned".to_string())),
+            Ok(mut next) => {
+                *next += 1;
+                *next
+            }
+        };
+        match self.fonts.write() {
+            Err(_) => Err(self.fail("the SoundFont table is poisoned".to_string())),
+            Ok(mut fonts) => {
+                fonts.insert(id, font);
+                Ok(id)
+            }
+        }
+    }
+
+    /// Drops a SoundFont. Synthesizers already opened on it keep their own
+    /// reference and play on.
+    pub fn release_sound_font(&self, font: u32) {
+        if let Ok(mut fonts) = self.fonts.write() {
+            fonts.remove(&font);
+        }
+    }
+
+    /// Creates a synthesizer on a loaded SoundFont and parks its source for the
+    /// `PLAY` carrying `handle` and `FLAG_SYNTH`.
+    ///
+    /// Everything that allocates happens here, on the caller's thread: the
+    /// synthesizer, its voices, its queue and its scratch buffers. The output
+    /// callback only ever renders it.
+    ///
+    /// @return the port events are sent through, or the reason nothing opened
+    pub fn open_synth(&self, font: u32, handle: u32, settings: &Settings) -> Result<Port, String> {
+        let shared = match self.fonts.read() {
+            Err(_) => return Err(self.fail("the SoundFont table is poisoned".to_string())),
+            Ok(fonts) => match fonts.get(&font) {
+                None => return Err(self.fail(format!("no SoundFont is loaded under {font}"))),
+                Some(value) => Arc::clone(value),
+            },
+        };
+        let (port, source) =
+            synth::open(&shared, self.sample_rate, settings).map_err(|reason| self.fail(reason))?;
+        match self.synths.lock() {
+            Err(_) => Err(self.fail("the synthesizer table is poisoned".to_string())),
+            Ok(mut synths) => {
+                synths.insert(handle, Box::new(source));
+                Ok(port)
+            }
+        }
+    }
+
     /// Builds the source a `PLAY` needs, outside the mixer lock.
     fn prepare(&self, command: &Command) -> Option<Box<dyn Source>> {
+        if command.has(command::FLAG_SYNTH) {
+            return self.synths.lock().ok()?.remove(&command.handle);
+        }
         let clips = self.clips.read().ok()?;
         match clips.get(&command.clip)? {
             Clip::Resident {
@@ -405,6 +475,12 @@ impl Engine {
         if let Ok(mut clips) = self.clips.write() {
             clips.clear();
         }
+        if let Ok(mut fonts) = self.fonts.write() {
+            fonts.clear();
+        }
+        if let Ok(mut synths) = self.synths.lock() {
+            synths.clear();
+        }
     }
 }