Skip to content

Add a General MIDI synthesizer: tecs.audio.synth over rustysynth - #8

Open
albanmgicquel wants to merge 2 commits into
tecs-dev:mainfrom
albanmgicquel:audio-synth
Open

albanmgicquel wants to merge 2 commits into
tecs-dev:mainfrom
albanmgicquel:audio-synth

Conversation

@albanmgicquel

Copy link
Copy Markdown
Collaborator

Adds tecs.audio.synth: General MIDI from a SoundFont (SF2), rendered in-process by rustysynth (MIT, no dependencies) as one voice of the existing mixer, so group gain/mute/pause, master gain and maxVoices apply unchanged.

const synth = require("tecs.audio.synth")
const font, err = synth.loadSoundFont(audio, path)
const s, err = synth.new(audio, font, {group = "music", gain = 1.0, polyphony = 64, effects = true})
s:send(status, data1, data2, at?)       -- plus noteOn / noteOff / controlChange / programChange / pitchBend
s:allNotesOff(channel?, immediate?); s:reset(); s:now(); s:setGain(g); s:destroy()
  • Scheduling: at is a time on the audio clock (now(), frames rendered over the rate). Events land on rustysynth's 8-frame block grid at or after at (never early, at most ~0.15 ms late at 48 kHz), so a given schedule renders identically every time; a nil or past at means the next boundary, in send order. Block 8 measured at 3.3% of one core with FluidR3, 48 notes held, effects on (1.9% at rustysynth's default 64).
  • Real-time: a bounded lock-free SPSC ring (4096) feeds a preallocated schedule (4096); no allocation or blocking in the callback. A full queue refuses the send: the send calls return false (and the drop is counted) rather than blocking or growing.
  • Headless: with no output, or the silent backend, the synth gets a silent port (every call succeeds, now() stays 0); a test backend can supply openSynth to log events.
  • Extension point: a hardware MIDI output (e.g. midir: CoreMIDI, WinMM, ALSA) would be a second SynthPort with the same send/at API; documented, not implemented.
  • Also: tecs.audio moves to audio/init.nupp (to host the child module; require("tecs.audio") is unchanged); the backend contract gains optional synth fields; a README design record explains why General MIDI is synthesized in-process (desktop OS GM synths are software anyway; one sound on every platform; mixer integration; determinism). No SoundFont ships; THIRD_PARTY_NOTICES says a game ships its own font's licence.

Tests: Rust scheduling/queue/reset/clock tests with a recording synth; SoundFont-gated rendering tests (TECS_TEST_SOUNDFONT): a note scheduled at frame 4800 renders the same samples as one at 0 shifted by 4800, identical renders for the same schedule, gain, immediate cut vs release, reset discarding a scheduled note, pause freezing the clock; a Nupp suite (audiosynthtest) over a logging backend. cargo test -p tecs-audio, clippy -D warnings, cargo fmt --check, nupp check --strict, nupp task format-check and nupp test pass. Used in a game with FluidR3_GM, music verified by ear.

Not run here: nupp task docs-check (this toolchain lacks the lunamark rock) and the packaging step. Known: rustysynth's SF2 parser is not hardened against a crafted file (the docs say to load only fonts the game ships); in one early device run the clock appeared to stop at 2.24 s with no stream failure flagged, not reproduced in later runs.

rustysynth (MIT) renders a SoundFont in the output callback as the input of a
voice started by a PLAY carrying the new SYNTH flag. Events cross from the game
thread through a bounded lock-free SPSC ring (4096) into a preallocated
schedule (4096); a send that finds no room is refused and counted. The render
splits at rustysynth's 8-frame block boundaries and applies each event at the
first boundary at or after its frame, so the output is a deterministic
function of the schedule. The clock is frames rendered over the rate.

FFI: tecs_audio_load_soundfont, tecs_audio_release_soundfont,
tecs_audio_synth_open/send/notes_off/reset/now/close. Tests cover the grid,
ordering, reset discarding, queue bounds, and with TECS_TEST_SOUNDFONT a real
SoundFont: sample-exact landing, determinism, voice gain, cuts and pause.
loadSoundFont(audio, path) and new(audio, font, {group, gain, polyphony,
effects}) with send, noteOn, noteOff, controlChange, programChange,
pitchBend, allNotesOff, reset, now, setGain and destroy. Sending methods
return whether the event was queued. The synth takes its voice slot the way
play does, through an internal per-mixer lender, so the voice ceiling, group
gain, mute and pause apply unchanged. The backend contract gains SynthPort and
optional loadSoundFont, releaseSoundFont and openSynth; a backend without them
and a mixer without an output get a silent port.

tecs.audio moves to audio/init.nupp for its child. README records why
synthesis is in-process and where a hardware MIDI output would plug in.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant