0.1.16 | BingBong live voice: return audio

Adds the uplink direction: each player sends their own microphone back to
whoever is puppeting BingBong, gated on proximity to him and attenuated by the
same curve his own voice is heard through. Rides the WebSocket that is already
open, so no new connection, auth or TLS surface, and still nothing over Photon.

Capture postfixes MicWrapper.Read rather than MicrophoneRelay, which copies
only the first 256 samples of each frame. Audio is downmixed, low-passed and
resampled to 16kHz in the mod, then passed through an adaptive noise gate: a
fixed threshold is unusable here because a quiet voice measures below a noisy
room, so the gate learns the background level instead.

F8 mutes the outgoing microphone for the session. uplink_denied and
uplink_roster from the relay are now handled, and the stub relay gained the
return direction plus --deny-uplink to exercise the refusal path.

The real relay and web app do not implement this direction yet, so it currently
only works against the stub.
This commit is contained in:
2026-09-06 14:12:45 +02:00
parent f4d723b23b
commit daa675af79
12 changed files with 1052 additions and 12 deletions

View File

@@ -106,3 +106,18 @@
- Incoming audio is resampled to the output rate by the mod rather than relying on Unity's mixer, so a device running at 44.1kHz or 48kHz sounds identical. A jitter buffer absorbs network timing, rendering what it has and padding the rest on a shortfall rather than muting and refilling, which pumped audibly when it happened often.
- Both Harmony patches are wrapped so that a failure can only cost the voice feature: the patch body is a try/catch around an isolated method, and repeated failures disable it for the session rather than breaking the item.
- `tools/stub_relay.py` is a development stub of the relay, enough to test the client end to end before the real backend exists. It speaks the same wire format and generates steady, swept and speech-shaped test signals, so no microphone or browser is needed.
# v0.1.16 | BingBong live voice: return audio
- Talking near BingBong now sends your own microphone back to whoever is puppeting him, so it is a conversation rather than a broadcast. Each player only ever transmits their own microphone, over the WebSocket that was already open — no second connection, and still nothing over PEAK's Photon servers.
- The uplink is gated on being within the same range BingBong's voice carries and attenuated by the same curve, so holding him up and talking works and walking away fades you out. It only transmits while someone is actually connected on the other end; an idle lobby sends nothing at all.
- Capture taps `MicWrapper.Read` rather than the obvious `MicrophoneRelay`, which copies only the first 256 samples of each frame before handing them on. Vanilla only ever wants a loudness number from it so the truncation is invisible in game, but streaming it gives a buzz at the frame rate instead of speech.
- The microphone is downmixed, low-passed and resampled from the device's rate to 16kHz by the mod. Decimating 48kHz to 16kHz without filtering first folds everything above 8kHz back into the speech band, which sounds like a second voice made of hiss.
- The noise gate learns the room instead of using a fixed level. A fixed threshold cannot work here: measured on real hardware, a quiet voice sits below what a noisy room measures, so any single number either cuts speech or passes the room. It tracks the background level, opens a configurable factor above it, and holds for 300ms so it does not close between words.
- The gate opens on entering range and closes only past a margin beyond it, with a minimum time in each state, so standing on the boundary cannot chatter it on and off.
- New `UplinkMuteKey` (default: F8) mutes the outgoing microphone for the rest of the session, since reaching for the config file is too slow to be the answer to "stop sending". It is not persisted — a mute you forgot you set is worse than one that clears on the next launch.
- The mod now understands the relay's `uplink_denied` and `uplink_roster` messages: a relay with the uplink switched off is believed and not asked again on that connection, and the log names who else is sending room audio, which is what tells "the relay is dropping my audio" apart from "nobody is listening".
- Uplink frames carry a different magic byte from the downlink ones, and the relay only ever forwards them to the speaker. A mis-routed frame therefore cannot replay a room's own audio back into it, which would be a feedback loop rather than a bug.
- The counters that let the receiver detect a dropped frame advance only for frames actually sent, so a gated pause leaves a contiguous timeline. Start and stop messages tell the receiver to rejoin the live edge instead of filling the pause with silence.
- `tools/stub_relay.py` gained the return direction: it forwards uplink audio to the speaker, stamps the stream id itself so a modified client cannot claim another player's, captures each stream to a WAV, and reports what fraction of real time actually arrived. `--deny-uplink` exercises the refusal path.
- The relay and the web app that would play this back are not built yet, so the return direction currently only works against that stub.