tecs.audio.fm: an emulated Yamaha FM chip as a mixer voice (ymfm-oxide)

AGENTS.md

@@ -108,14 +108,14 @@ tecs/
 │   ├── platform/           # Window, events, audio and gamepad backends
 │   ├── physics/            # Rapier contract and binding
 │   ├── input.nupp          # Gameplay input over the platform event stream
-│   ├── audio.nupp          # Clips, voices, groups, limits, the Sound component
+│   ├── audio/              # Clips, voices, groups, limits, the Sound component, General MIDI, Yamaha FM
 │   ├── sequence.nupp       # Sequencer, with the tween runtime inside it
 │   ├── assets.nupp         # Asset orchestration
 │   ├── mcp.nupp            # Debug server: transport, tools, sandbox
 │   └── events.nupp, files.nupp, watch.nupp
 ├── native/rust/
 │   ├── winit-host/         # The application loop, the wgpu renderer, the host ABI
-│   ├── tecs-audio/         # cpal output and capture behind a batched contract
+│   ├── tecs-audio/         # cpal output and capture behind a batched contract; General MIDI and Yamaha FM voices
 │   ├── tecs-gamepad/       # gilrs enumeration and observations
 │   ├── physics/            # Rapier storage and stepping
 │   └── build-support/      # Native packaging and binary inspection

Cargo.toml

@@ -47,6 +47,7 @@ rayon = "1.12"
 reqwest = { version = "0.13.4", default-features = false, features = ["deflate", "gzip", "rustls-no-provider", "socks", "stream"] }
 resvg = { version = "0.47", default-features = false, features = ["text"] }
 regex = "1.12"
+rustysynth = "1.3.7"
 rmcp = { version = "3.0.0", default-features = false, features = ["client", "server", "transport-io", "transport-streamable-http-client-reqwest", "transport-streamable-http-server"] }
 rustls = { version = "0.23", default-features = false, features = ["ring", "std", "tls12"] }
 serde = { version = "1", features = ["derive"] }
@@ -70,6 +71,7 @@ urlencoding = "2.1.3"
 walkdir = "2.5"
 wgpu = "30.0.1"
 winit = "0.30.13"
+ymfm-oxide = { git = "https://github.com/albanmgicquel/ymfm-oxide", rev = "4cc9e106c7dbf28c848a9a463bf3ef74709a5ed0" }
 axum = { version = "0.8", default-features = false, features = ["http1", "tokio"] }
 
 [profile.release]

README.md

@@ -50,8 +50,10 @@ test suite reach the same world without starting a graphics stack.
 - Camera, layers with positioning modes, images, clips, sprite sheets with
   animation, and distance-field text.
 - Input across keyboard, pointer, gamepad and touch; audio playback, recording
-  and device enumeration; physics over Rapier; sequencing with tweening; asset
-  orchestration; file watching; and an MCP debug server.
+  and device enumeration; General MIDI from a SoundFont; emulated Yamaha FM
+  chips, from the OPL2 and OPL3 to the YM2612 and the OPL4; physics over
+  Rapier; sequencing with tweening; asset orchestration; file watching; and an
+  MCP debug server.
 
 - [Tiled TMX maps and TSX tilesets](docs/tiled/index.md), animated tile and image
   layers, object factories, runtime tile edits and authored collision outlines.
@@ -488,6 +490,188 @@ stream did. It holds one second by default, drops the oldest, and reports the
 loss through `Microphone.overruns`. An unbounded allocation driven by a
 realtime callback was not worth carrying across.
 
+## General MIDI is synthesized in-process
+
+`tecs.audio.synth` renders General MIDI from a SoundFont with `rustysynth`
+(MIT, pure Rust, no dependencies) as one voice of the mixer, rather than
+sending MIDI to whatever synthesizer the operating system offers. On a desktop
+the system synthesizer is software too: Apple's DLS synth on macOS, the
+Microsoft GS Wavetable synth on Windows, and on Linux nothing at all unless the
+player installed and started FluidSynth or TiMidity. Sending to it buys no
+quality and gives up three things a game needs. The music sounds the same on
+every machine, because the SoundFont ships with the game instead of being
+whatever bank the platform has. It goes through the mixer, so group gain, mute,
+pause and master gain apply to it like any other sound, and a system synth
+offers none of them. And it is deterministic: events carry a time on the
+synthesizer's own clock, the callback splits its render at those times, and the
+same schedule renders the same samples on every run, which a test can pin and
+a recording can reproduce. An external synth answers on its own clock and its
+own latency, and the mix of the two cannot be captured offline.
+
+Timing is exact to `rustysynth`'s internal block, which is where it applies
+MIDI and runs its envelopes. Tecs sets that block to eight frames, the smallest
+the library takes: every event lands on the first eight-frame boundary at or
+after its time, 0.17 milliseconds at worst at 48 kHz. Measured on FluidR3 GM
+with 48 notes held and the effects on, that costs 3.3 percent of one core
+against 1.9 at the library's default of 64 frames.
+
+Events cross to the output callback through a bounded single-producer,
+single-consumer ring, 4096 events, with as many again held in a preallocated
+schedule for events sent ahead of their time. The callback never waits and
+never allocates; a send that finds no room returns false and the event is
+dropped, because the alternatives are blocking the frame thread or growing a
+buffer the callback reads. A game streaming a song with a lookahead of a frame
+or two never comes near it.
+
+A hardware MIDI output is the extension this leaves room for and does not
+build. The seam is `tecs.platform.audiobackend.SynthPort`: `send` with a time,
+`notesOff`, `reset` and a clock. A second implementation over `midir`
+(CoreMIDI, WinMM, ALSA) would put a sender thread in Rust that holds the
+schedule and writes each message when its time comes, with `now` on the host's
+monotonic clock instead of rendered frames, chosen when the synth is made.
+Nothing above the port changes. It is deferred because nothing needs it yet,
+and what it costs is the determinism above: a hardware synth cannot be rendered
+offline.
+
+## FM synthesis is a register-level chip, emulated by a Rust crate
+
+`tecs.audio.fm` is a Yamaha OPL2 (YM3812) or OPL3 (YMF262) a game drives by
+writing its registers, played as one voice of the mixer on the same scheduler
+as the General MIDI synth. It is for the music and effects of the DOS era,
+which were written as streams of register writes to an AdLib or a Sound
+Blaster, and a port of such a game wants to replay what its driver wrote, not
+an approximation of it.
+
+The interface is registers only: `write`, `writes` for a tick's batch, `reset`
+and a clock. A note-and-instrument layer was the alternative, and it lost
+because it is the part that differs between games. Every driver had its own
+instrument format, its own channel allocation and its own idea of a note, and
+an engine layer would have had to pick one or grow into a superset of all of
+them. The chip is the one contract they all shared. A game that wants notes
+writes a driver for its own format above this, in Nupp, and gets exactly the
+sound that format made. Status and timer reads are left out for the same
+reason they are left out of the scheduler: a driver that paced itself on the
+chip's timers paces itself on `now` instead, and a read would need an answer
+from the audio thread inside the call.
+
+Writes land on the first chip sample at or after their time, a grid of one
+rather than the synth's eight, because a register write takes effect on the
+next sample the chip computes and a driver's timing is part of its sound: a
+key-off and key-on a sample apart is a retrigger, and moving it eight samples
+changes the attack. The scheduler names its grid per target to allow that,
+eight frames for the synth and one for the chip, and the ring, the schedule
+and the source are otherwise the synth's, so the queue bound and the clock
+behave the same way. Writes due on one sample all apply before that sample is
+generated, in sending order, so a key-off and a key-on there count as no
+change: a driver that re-strikes a note spaces the two writes one sample apart,
+which is what the hardware's write timing did anyway. `writes` is all or
+nothing, because a tick applied in part plays a note with half its instrument.
+
+A reset carries a time like a write. Writes sent before it and due before it
+still apply; those due at or after it are discarded, so nothing scheduled
+earlier plays over the silence it makes, and writes sent after it apply as
+usual, at its own sample included. The synth's reset, which lands at its next
+block, keeps its old meaning; the chip's needed a time because a driver
+restarting a song schedules the restart rather than taking it whenever the
+next buffer begins.
+
+The chip renders at its own rate and the mixer voice resamples it as it would
+a clip, so the clock counts chip samples and the output rate does not leak
+into a driver's timing. The rate is the input clock over 72 on the OPL2,
+3.579545 MHz, and over 288 on the OPL3, four times that: 49 715.9 Hz, which
+the emulator truncates to 49715. Rounding it to 49716 instead, as an earlier
+draft did, would be closer by under twenty parts per million, which is a
+thirtieth of a cent of pitch and under a tenth of a second an hour of tempo,
+below anything heard; the clock simply has to agree with the emulator, and a test
+holds `rate` to what the emulator reports for both chips.
+
+The emulator is ymfm-oxide (<https://github.com/albanmgicquel/ymfm-oxide>), a
+pure-Rust, `no_std`, dependency-free port of ymfm, Aaron Giles's BSD-3-Clause
+emulator of the Yamaha FM family, used for its YM3812 and YMF262: OPL3
+4-operator pairs, all eight waveforms, the NEW bit, stereo and rhythm mode.
+Linking ymfm's C++ was the alternative, and it lost to the build: every other
+native service here is Cargo only, and a C++ toolchain on every platform,
+Windows included, for one module was not a cost worth carrying. ymfm was the
+source because it is permissively licensed and written to be read, which is
+what makes a port checkable. The candidates under the LGPL or the GPL,
+Nuked-OPL3 and DOSBox's among them, are excluded by the licence rule in
+`THIRD_PARTY_NOTICES.md`, and so is anything developed from them.
+
+The port lives in its own crate rather than in this tree, and it was first
+written here. Kept here, Tecs carried a few thousand lines of BSD-3-Clause
+source with its own notices, its golden renders and a C++ oracle, none of it
+about the engine; as a crate it is one dependency with one notice, its check
+lives with its code, and the other chips ymfm models come with it for whoever
+wants them. The crate holds itself to ymfm sample for sample with golden
+renders of synthetic register logs, made by ymfm through an oracle program,
+and a randomized differential test against it. It is a git dependency pinned
+to a full commit, and `Cargo.lock` records that commit, so a build cannot pick
+up a different emulator without a change here. `tecs-audio` drives it through
+the chips' own ports: a register below 0x100 through the address port at
+offset 0 and the data port at 1, one from 0x100 up through the OPL3's
+second-bank address port at 2 and the data port at 3, which is how the chip
+itself folds a second-bank write back to the first until the NEW bit is set.
+The OPL3's left and right are its outputs A and B; the OPL2 is mono and plays
+its one output on both sides.
+
+## Every chip the emulator models is one voice, named by its part number
+
+`tecs.audio.fm` began as the OPL2 and OPL3 and now makes any of the 24 chips
+ymfm-oxide models, because the register logs a game ships or a player loads
+are not all DOS ones: a Mega Drive log is a YM2612, an arcade one a YM2151 or
+a YM2610, an MSX one a YM2413 beside a Y8950. The alternative was a module per
+family, and it lost because the families differ only in data a table holds:
+where the registers sit among the ports, what the input clock usually was, how
+the outputs reach two speakers and what memory the chip reads. The contract a
+game writes against, registers, a clock and `at`, is the same for all of them,
+so the chip is a name and a clock and nothing else changes. Names are Yamaha
+part numbers in lower case rather than family names, because a family name
+does not say which chip: an OPN is any of eight. `"opl2"` and `"opl3"` stay as
+aliases, so the first callers never notice.
+
+The register convenience stays the primary interface, since every log and every
+driver's source names registers, and it maps a register's high byte to the
+bank's address and data ports: one bank at offsets 0 and 1, the second at 2 and
+3, the OPL4's PCM registers as a third at 4 and 5, and the YM2149's data port
+at 2. The OPQ has no address port and is written by register. Port writes are
+offered beside it for the driver that wrote the bus itself, and a test holds the
+two to the same render.
+
+The chip renders at the emulator's own rate for its clock, which the voice
+resamples. Two consequences needed decisions. The OPN chips with an SSG default
+in ymfm to their highest fidelity, a megahertz for the YM2203, and run here at
+their lowest, which keeps every FM sample and averages the SSG's; and where the
+emulator still renders faster than 64 kHz, as the YM2203, the YM2608 and the
+YMF288 do, consecutive samples are averaged into one frame of the chip's clock.
+That keeps the mixer's linear resampler away from ratios where it aliases, and
+it puts a ceiling on `rate`, the unit of `at`. Outputs fold to stereo by a
+fixed table: a mono chip on both sides, the OPL3's A and B, the OPL4's mixed
+pair, the SSG at two thirds of its channels' sum, which is ymfm's own mix.
+The SSG's square waves swing between zero and a positive level, which a real
+board's output capacitor took out, so a chip with one has a 5 Hz high-pass
+filter on its output; without it a song would start and end with a click and
+carry an offset into the mix.
+
+External memory is images a game hands over, whole: `setMemory` replaces the
+ADPCM-A, ADPCM-B or PCM image from the next frame, and a reset keeps it. The
+images belong to the audio thread's half of the chip. ymfm reaches memory
+through the chip's interface, which the chip owns and not every chip lends
+back, so the interface holds a pointer to images the target owns, read and
+swapped only between samples on the one thread that renders; a lock around
+them was the alternative and lost, being a lock on the audio thread taken on
+every ADPCM fetch to protect nothing a second thread touches. An image crosses
+on a bounded channel of its own, paired with an event in the write queue that
+says when to take it, and the image it displaces goes back to the game thread
+to be freed there. Timers, busy time and the IRQ line are not offered: a log is
+already timed and a driver paces on `now`, and an answer from the emulator
+inside a call would need the audio thread to reply. What that leaves out is
+the OPN's CSM key-on and the ADPCM end-of-sample flags, which only a timer or a
+status read can drive.
+
+An idle chip at its usual clock costs between a fifth of a percent and one
+percent of a core in a release build, measured over a second of rendering; the
+YMF288 is the dearest and the YM2149 the cheapest.
+
 ## Device vocabularies follow the library that reports them
 
 Key names, mouse button names and gamepad button and axis names came from SDL

THIRD_PARTY_NOTICES.md

@@ -81,6 +81,60 @@ Open Font License 1.1. The tests and retained UI example read it. Packages ship
 `share/tecs/assets/fonts` with `JetBrainsMono-OFL.txt` and the versioned
 `JetBrainsMono-NOTICE.md`.
 
+## SoundFonts
+
+Tecs ships no SoundFont. `tecs.audio.synth` reads one a game supplies, and a
+General MIDI set is game content under its own licence, so a game that ships
+one ships that licence beside it. FluidR3 GM, for example, is MIT.
+
+## Linked BSD-licensed code
+
+### ymfm-oxide
+
+`tecs.audio.fm` emulates every chip it offers with ymfm-oxide
+(<https://github.com/albanmgicquel/ymfm-oxide>), a pure-Rust port by Alban
+Gicquel of ymfm, Aaron Giles's emulator of the Yamaha FM chips
+(<https://github.com/aaronsgiles/ymfm>). It is a Cargo dependency pinned to one
+commit, compiled into `tecsaudio`, so every package carries it and
+`cargo-licenses.txt` records it as BSD-3-Clause. The licence asks for its
+notice to travel with a binary, which is this one; Tecs keeps none of its
+source. The OPLL family plays ymfm's own copyright-free stand-ins for the
+instrument ROMs, made by David Viens and Hubert Lamontagne and carried in the
+crate; no Yamaha ROM ships with Tecs.
+
+```text
+BSD 3-Clause License
+
+Copyright (c) 2021, Aaron Giles
+Copyright (c) 2026, Alban Gicquel
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+```
+
 ## Scope of the checks
 
 The generated inventory and source manifest make shipped dependency names,

native/rust/tecs-audio/Cargo.toml

@@ -11,4 +11,6 @@ crate-type = ["cdylib", "rlib"]
 
 [dependencies]
 cpal.workspace = true
+rustysynth.workspace = true
 symphonia.workspace = true
+ymfm-oxide.workspace = true