Jordan/PEAK-JordanMod
1
0
Fork 0
mirror of https://github.com/TheJordanDev/PEAK-JordanMod.git synced 2026-10-16 23:56:44 +02:00
Code Issues Packages Projects Releases Wiki Activity

0.1.15 | BingBong live voice (in-game client)

Browse Source 
Speak through BingBong with a live microphone, streamed from a relay
rather than over PEAK's Photon servers. This is the in-game half only:
the relay and the web app that captures the microphone are not built
yet, so the feature does nothing on its own until they exist.

Every BingBong gets a voice source as it spawns, positioned on the plush
and inheriting the 3D setup of his own voice lines. His mouth animates to
the live audio on every client, and hands back to the vanilla animation
while one of his own lines plays. Those lines take priority: the live
voice ducks for their duration and resumes at the live position.

Playback uses OnAudioFilterRead rather than a streaming AudioClip. A clip
created with stream:true fills Unity's decode buffer ahead of playback,
and that read-ahead cannot be satisfied from a live stream because the
audio has not arrived yet -- the buffer drained to empty on a cycle no
matter how deep it was made, which was audible as constant clicking. The
filter callback runs in lockstep with playback, so demand can never
outrun real time.

Includes tools/stub_relay.py, a development stub speaking the same wire
format with steady, swept and speech-shaped test signals, so the client
can be tested end to end without the real backend, a microphone or a
browser.
This commit is contained in:
TheJordan
2026-09-05 16:18:04 +02:00
parent 9fc95c2fc5
commit f4d723b23b
9 changed files with 1664 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
CHANGELOG.md
View File

@@ -94,3 +94,15 @@
- The audio bank sync no longer drives `UnityWebRequest` from a background thread, which is main-thread only, and no longer busy-waits on it while blocking a second thread-pool thread. - The audio bank sync no longer drives `UnityWebRequest` from a background thread, which is main-thread only, and no longer busy-waits on it while blocking a second thread-pool thread.
- The song display no longer scans every loaded object looking for its font on every frame. It falls back to the default font immediately and retries for the nicer one on an interval, stopping once found. - The song display no longer scans every loaded object looking for its font on every frame. It falls back to the default font immediately and retries for the nicer one on an interval, stopping once found.
- The audio bank URL is normalised before use: a missing scheme is filled in (http for localhost, https otherwise) and a trailing slash trimmed. Without a scheme every request failed outright with "Invalid BugleSoundAPIURL in config", since the URL could not be parsed. - The audio bank URL is normalised before use: a missing scheme is filled in (http for localhost, https otherwise) and a trailing slash trimmed. Without a scheme every request failed outright with "Invalid BugleSoundAPIURL in config", since the URL could not be parsed.
# v0.1.15 | BingBong live voice (in-game client)
- Speak through BingBong with a live microphone. Audio is streamed from a relay rather than over PEAK's Photon servers, so it adds no load there at all. This release is the **in-game half only** — the relay and the web app that captures the microphone are not built yet, so the feature does nothing on its own until they exist.
- Every BingBong in the world gets a voice source as it spawns, positioned on the plush and inheriting the 3D setup of his own voice lines, so the live voice carries and attenuates like he does and obeys the game's volume sliders. Several BingBongs each speak from their own position.
- His mouth animates to the live audio, on every client rather than just the speaker's, and hands control back to the vanilla animation while one of his own voice lines plays.
- Canned voice lines take priority: the live voice ducks for the length of the line and then resumes at the live position, so the two never talk over each other. Detected from the RPC body, so the duck stays in step across the lobby without any networking of our own.
- New "BingBong Voice" config section, all of it applied live. The relay URL is derived from `BugleSoundAPIURL` when left empty, since that is where the relay will live, and can be overridden to point anywhere — a bare host or `host:port` is completed automatically. Range defaults to 50m, further than his 30m voice lines carry.
- Playback goes through `OnAudioFilterRead` rather than a streaming `AudioClip`. A clip created with `stream: true` fills Unity's decode buffer *ahead* of playback, and that read-ahead cannot be satisfied from a live stream because the audio has not arrived yet — the buffer drained to empty on a cycle no matter how deep it was, which was audible as constant clicking. The filter callback runs in lockstep with playback instead, so demand can never outrun real time.
- Incoming audio is resampled to the output rate by the mod rather than relying on Unity's mixer, so a device running at 44.1kHz or 48kHz sounds identical. A jitter buffer absorbs network timing, rendering what it has and padding the rest on a shortfall rather than muting and refilling, which pumped audibly when it happened often.
- Both Harmony patches are wrapped so that a failure can only cost the voice feature: the patch body is a try/catch around an isolated method, and repeated failures disable it for the session rather than breaking the item.
- `tools/stub_relay.py` is a development stub of the relay, enough to test the client end to end before the real backend exists. It speaks the same wire format and generates steady, swept and speech-shaped test signals, so no microphone or browser is needed.

92
src/JordanMod/ConfigHandler.cs
View File

@@ -29,6 +29,19 @@ public static class ConfigHandler
public static ConfigEntry<string> FavoriteSongsList { get; private set; } = null!; public static ConfigEntry<string> FavoriteSongsList { get; private set; } = null!;
public static ConfigEntry<KeyCode> FavoriteSongToggleKey { get; private set; } = null!; public static ConfigEntry<KeyCode> FavoriteSongToggleKey { get; private set; } = null!;
// BingBong Voice settings
public static ConfigEntry<bool> BingBongVoiceEnabled { get; private set; } = null!;
public static ConfigEntry<string> BingBongVoiceRelayURL { get; private set; } = null!;
public static ConfigEntry<float> BingBongVoiceVolume { get; private set; } = null!;
public static ConfigEntry<float> BingBongVoiceDuckVolume { get; private set; } = null!;
public static ConfigEntry<float> BingBongVoiceDuckLead { get; private set; } = null!;
public static ConfigEntry<float> BingBongVoiceJitterMs { get; private set; } = null!;
public static ConfigEntry<float> BingBongVoiceMaxDistance { get; private set; } = null!;
public static ConfigEntry<bool> BingBongVoiceAutoConnect { get; private set; } = null!;
public static ConfigEntry<bool> BingBongVoiceAnnounceSpeaker { get; private set; } = null!;
public static ConfigEntry<KeyCode> BingBongVoiceDebugKey { get; private set; } = null!;
public static ConfigEntry<bool> BingBongVoiceDebugSource { get; private set; } = null!;
public static void Initialize(ConfigFile configFile) public static void Initialize(ConfigFile configFile)
{ {
Config = configFile; Config = configFile;
@@ -107,6 +120,85 @@ public static class ConfigHandler
new ConfigDescription("Key to toggle favorite status of current song") new ConfigDescription("Key to toggle favorite status of current song")
); );
// BingBong Voice settings
BingBongVoiceEnabled = Config.Bind(
"BingBong Voice",
"Enabled",
true,
new ConfigDescription("Let BingBong speak with a live voice streamed from the relay. When off, no patches are applied at all.")
);
BingBongVoiceRelayURL = Config.Bind(
"BingBong Voice",
"RelayURL",
"",
new ConfigDescription("Override for the voice relay WebSocket URL. Normally leave this EMPTY: it is then derived from BugleSoundAPIURL, keeping that URL's path, so https://host/api becomes wss://host/api/voice/ws. Only set it to point at a different relay, and include the full path - aiming at the website instead of the API gives an HTML page and a 1002 close.")
);
BingBongVoiceVolume = Config.Bind(
"BingBong Voice",
"Volume",
1.0f,
new ConfigDescription("Live voice volume", new AcceptableValueRange<float>(0f, 1f))
);
BingBongVoiceDuckVolume = Config.Bind(
"BingBong Voice",
"DuckVolume",
0.0f,
new ConfigDescription("Live voice volume while one of BingBong's own voice lines is playing", new AcceptableValueRange<float>(0f, 1f))
);
BingBongVoiceDuckLead = Config.Bind(
"BingBong Voice",
"DuckLeadSeconds",
1.0f,
new ConfigDescription("How long before a voice line's audio actually starts to begin ducking. AskRoutine waits about a second before it plays anything.", new AcceptableValueRange<float>(0f, 2f))
);
BingBongVoiceJitterMs = Config.Bind(
"BingBong Voice",
"JitterBufferMs",
80f,
new ConfigDescription("Audio buffered before playback starts. Lower is more responsive, higher survives worse connections.", new AcceptableValueRange<float>(40f, 400f))
);
BingBongVoiceMaxDistance = Config.Bind(
"BingBong Voice",
"MaxDistance",
50f,
new ConfigDescription("How far the live voice carries, in metres. BingBong's own voice lines only reach 30m, which is short for a conversation.", new AcceptableValueRange<float>(5f, 500f))
);
BingBongVoiceAutoConnect = Config.Bind(
"BingBong Voice",
"AutoConnect",
true,
new ConfigDescription("Connect to the voice relay automatically on joining a lobby, and disconnect on leaving. Turn off to connect only with the keybind.")
);
BingBongVoiceAnnounceSpeaker = Config.Bind(
"BingBong Voice",
"AnnounceSpeaker",
true,
new ConfigDescription("Show a message when someone starts speaking through BingBong. Being puppeted silently by someone who is not even in your lobby is a bit much without it.")
);
BingBongVoiceDebugKey = Config.Bind(
"BingBong Voice",
"DebugToggleKey",
KeyCode.F7,
new ConfigDescription("Connect to / disconnect from the voice relay")
);
BingBongVoiceDebugSource = Config.Bind(
"BingBong Voice",
"DebugSource",
false,
new ConfigDescription("Also play the stream through a non-positional source, for testing without a BingBong in the scene. Plays on top of any real BingBong, so leave this off normally.")
);
} }
} }

2
src/JordanMod/JordanMod.csproj
View File

@@ -8,7 +8,7 @@
<!-- This is the display name of your mod. Example: BepInEx Template --> <!-- This is the display name of your mod. Example: BepInEx Template -->
<AssemblyTitle>JordanMod</AssemblyTitle> <AssemblyTitle>JordanMod</AssemblyTitle>
<!-- This is the version number of your mod. --> <!-- This is the version number of your mod. -->
<Version>0.1.14</Version> <Version>0.1.15</Version>
</PropertyGroup> </PropertyGroup>
<PropertyGroup> <PropertyGroup>

295
src/JordanMod/modules/bingbong_voice/BingBongVoiceClient.cs Normal file
View File

@@ -0,0 +1,295 @@
using System;
using System.Security.Authentication;
using System.Threading;
using Newtonsoft.Json.Linq;
using WebSocketSharp;
namespace JordanMod.Modules.BingBongVoice;
/// One decoded stream. Swapped wholesale when the sender changes sample rate, so consumers
/// only ever hold a consistent (ring, rate) pair.
public sealed class VoiceStream
{
public VoiceRingBuffer Ring { get; }
public int SampleRate => Ring.SampleRate;
public VoiceStream(int sampleRate) => Ring = new VoiceRingBuffer(sampleRate);
}
/// <summary>
/// Holds the websocket to the voice relay and turns its binary frames into samples in a ring.
///
/// Everything here runs on websocket-sharp's own receive thread. It must never touch a Unity
/// API: hop through <see cref="Plugin.RunOnMainThread"/> for that, and never per packet — at
/// 50 frames a second that queue would balloon during any frame hitch.
/// </summary>
public sealed class BingBongVoiceClient
{
// Wire format, kept in step with the plan and the stub relay:
// 0 u8 magic | 1 u8 version | 2 u8 codec | 3 u8 rate code | 4 u32 seq | 8 u32 sampleTimestamp
private const int HeaderSize = 12;
private const byte Magic = 0xB1;
private const byte Version = 1;
private const byte CodecPcm16 = 0;
private static readonly int[] RateTable = [16000, 8000, 24000, 48000];
// A gap larger than this is treated as a restart rather than something to paper over.
private const int MaxGapSamples = 48000;
private WebSocket? _socket;
private string? _hello;
private VoiceStream? _stream;
private Thread? _reconnectThread;
private volatile bool _shouldRun;
private volatile bool _speakerActive;
private long _expectedTimestamp = -1;
private int _badFrames;
private volatile int _lastCloseCode;
/// The live stream, or null before the first frame arrives. Read from the audio thread.
public VoiceStream? Stream => Volatile.Read(ref _stream);
/// True between speaker_joined and speaker_left control frames.
public bool SpeakerActive => _speakerActive;
/// Who is speaking, when the relay tells us. Null if nobody is, or if it did not say.
public string? SpeakerName { get; private set; }
/// Raised on the main thread when the speaker changes, so the UI can announce it.
public event Action<string?>? OnSpeakerChanged;
public bool IsConnected => _socket is { ReadyState: WebSocketState.Open };
public string Url { get; private set; } = "";
/// <param name="hello">
/// Optional JSON control frame sent as soon as the socket opens. This is how a lobby
/// registers itself with the relay: the channel comes from the token, and this supplies who
/// is in it, so an admin can see the live sessions and pick one to speak into.
/// </param>
public void Connect(string url, string? hello = null)
{
_hello = hello;
if (_shouldRun) return;
if (string.IsNullOrWhiteSpace(url))
{
Debug.LogWarning("[BingBongVoice] No relay URL configured, not connecting.");
return;
}
// A malformed URL will never start working, so fail here rather than letting the
// reconnect loop retry it every minute for the rest of the session.
if (!Uri.TryCreate(url, UriKind.Absolute, out Uri? parsed) || (parsed.Scheme != "ws" && parsed.Scheme != "wss"))
{
Debug.LogError($"[BingBongVoice] '{url}' is not a usable WebSocket URL, not connecting.");
return;
}
Url = url;
_shouldRun = true;
_reconnectThread = new Thread(ConnectLoop) { IsBackground = true, Name = "BingBongVoice" };
_reconnectThread.Start();
}
public void Disconnect()
{
_shouldRun = false;
_speakerActive = false;
WebSocket? socket = _socket;
_socket = null;
if (socket == null) return;
try { socket.Close(CloseStatusCode.Normal, "disconnecting"); }
catch (Exception e) { Debug.LogWarning($"[BingBongVoice] Error closing socket: {e.Message}"); }
}
// Reconnect with exponential backoff. Runs on its own thread for the whole session.
private void ConnectLoop()
{
int attempt = 0;
System.Random jitter = new();
while (_shouldRun)
{
try
{
OpenSocket();
// Poll rather than block: websocket-sharp gives us no "wait until closed".
while (_shouldRun && _socket is { ReadyState: WebSocketState.Open }) Thread.Sleep(200);
if (!_shouldRun) return;
// A rejected request goes straight to the ceiling rather than climbing to it.
attempt = _lastCloseCode == 1008 ? 6 : attempt + 1;
}
catch (Exception e)
{
attempt++;
// First failure and then every tenth, so an unreachable relay can't flood the log.
if (attempt == 1 || attempt % 10 == 0)
Debug.LogWarning($"[BingBongVoice] Connection attempt {attempt} failed: {e.Message}");
}
if (!_shouldRun) return;
int delayMs = (int)(Math.Min(60000, 2000 * Math.Pow(2, Math.Min(attempt - 1, 5))) * (0.8 + jitter.NextDouble() * 0.4));
for (int slept = 0; slept < delayMs && _shouldRun; slept += 100) Thread.Sleep(100);
}
}
private void OpenSocket()
{
WebSocket socket = new(Url);
// websocket-sharp writes its own "Fatal|WebSocket.connect|..." lines straight to the
// console, including a run of mojibake, every single reconnect attempt. We report
// connection state ourselves, so silence its logger entirely.
socket.Log.Output = (_, _) => { };
// This build of websocket-sharp defaults to a legacy protocol set, so wss:// to any
// modern TLS terminator fails outright without this.
if (Url.StartsWith("wss://", StringComparison.OrdinalIgnoreCase))
{
socket.SslConfiguration.EnabledSslProtocols =
SslProtocols.Tls12 | SslProtocols.Tls11 | SslProtocols.Tls;
}
socket.OnMessage += OnMessage;
socket.OnOpen += (_, _) =>
{
_expectedTimestamp = -1;
_badFrames = 0;
Debug.Log("[BingBongVoice] Connected to relay.");
// Register this lobby member so the relay's session directory knows who is listening.
if (string.IsNullOrEmpty(_hello)) return;
try { socket.Send(_hello); }
catch (Exception ex) { Debug.LogWarning($"[BingBongVoice] Could not announce presence: {ex.Message}"); }
};
socket.OnClose += (_, e) =>
{
_speakerActive = false;
_lastCloseCode = e.Code;
// 1008 means the request itself is wrong -- bad token shape, uppercase channel, an
// Origin header we should not be sending, or the relay being disabled. Retrying
// quickly cannot fix any of those, so say so plainly and let the loop back right off.
if (e.Code == 1008)
{
Debug.LogError($"[BingBongVoice] Relay rejected the connection ({e.Reason}). This will not fix itself; check the relay URL and token.");
}
else if (e.Code == 1002)
{
// The handshake got a valid HTTP response that was not an upgrade -- almost always
// the URL pointing at the web frontend rather than the API. A single-page app
// answers 200 with HTML for any path, so it looks alive and never works.
Debug.LogError($"[BingBongVoice] '{Url}' answered with something that is not a WebSocket. Point it at the API rather than the website: the endpoint usually sits under /api (e.g. https://host/api -> wss://host/api/voice/ws). Clearing RelayURL derives it from BugleSoundAPIURL.");
}
else
{
Debug.Log($"[BingBongVoice] Relay closed ({e.Code} {e.Reason}).");
}
};
socket.OnError += (_, e) => Debug.LogWarning($"[BingBongVoice] Socket error: {e.Message}");
_socket = socket;
socket.Connect();
}
// websocket-sharp's receive thread. No Unity APIs, no allocation beyond what we're handed.
private void OnMessage(object? sender, MessageEventArgs e)
{
try
{
if (!e.IsBinary)
{
HandleControl(e.Data);
return;
}
byte[] d = e.RawData;
if (d.Length < HeaderSize) return;
if (d[0] != Magic || d[1] != Version)
{
if (++_badFrames <= 3) Debug.LogWarning($"[BingBongVoice] Bad frame header (magic {d[0]:X2}, version {d[1]}).");
return;
}
if (d[2] != CodecPcm16)
{
if (++_badFrames <= 3) Debug.LogWarning($"[BingBongVoice] Unsupported codec {d[2]}, frame dropped.");
return;
}
int rate = RateTable[d[3] & 3];
uint timestamp = (uint)(d[8] | (d[9] << 8) | (d[10] << 16) | (d[11] << 24));
VoiceStream stream = EnsureStream(rate);
// The sender's timestamp tells us exactly how much audio went missing, so a dropped
// frame costs its own duration instead of silently shifting everything after it.
if (_expectedTimestamp >= 0 && timestamp != _expectedTimestamp)
{
long gap = (long)timestamp - _expectedTimestamp;
if (gap > 0 && gap <= MaxGapSamples) stream.Ring.WriteSilence((int)gap);
// A negative or huge gap means the sender restarted; just carry on from here.
}
int payload = d.Length - HeaderSize;
stream.Ring.WritePcm16(d, HeaderSize, payload);
_expectedTimestamp = timestamp + (payload >> 1);
}
catch (Exception ex)
{
if (++_badFrames <= 3) Debug.LogError($"[BingBongVoice] Frame handling failed: {ex}");
}
}
private VoiceStream EnsureStream(int rate)
{
VoiceStream? current = Volatile.Read(ref _stream);
if (current != null && current.SampleRate == rate) return current;
VoiceStream replacement = new(rate);
Volatile.Write(ref _stream, replacement);
_expectedTimestamp = -1;
Debug.Log($"[BingBongVoice] Stream rate is {rate} Hz.");
return replacement;
}
private void HandleControl(string json)
{
// Parsed leniently and fielded by substring if that fails: control messages are allowed to
// gain fields, and a malformed payload must never take down the receive thread.
string? speaker = null;
try
{
speaker = JObject.Parse(json)["speaker"]?.ToString();
}
catch
{
// not JSON, or no such field -- the substring checks below still work
}
if (json.Contains("speaker_joined"))
{
_speakerActive = true;
SpeakerName = string.IsNullOrWhiteSpace(speaker) ? null : speaker;
Debug.Log($"[BingBongVoice] A speaker joined{(SpeakerName != null ? $": {SpeakerName}" : "")}.");
Plugin.RunOnMainThread(() => OnSpeakerChanged?.Invoke(SpeakerName));
}
else if (json.Contains("speaker_left") || json.Contains("preempted"))
{
_speakerActive = false;
SpeakerName = null;
_expectedTimestamp = -1;
Debug.Log("[BingBongVoice] The speaker left.");
Plugin.RunOnMainThread(() => OnSpeakerChanged?.Invoke(null));
}
else if (json.Contains("rejected"))
{
Debug.LogWarning($"[BingBongVoice] Relay rejected this listener: {json}");
}
}
}

266
src/JordanMod/modules/bingbong_voice/BingBongVoiceModule.cs Normal file
View File

@@ -0,0 +1,266 @@
using System;
using System.Security.Cryptography;
using System.Text;
using JordanMod.Modules.BetterBugle;
using Newtonsoft.Json;
using Photon.Pun;
using UnityEngine;
namespace JordanMod.Modules.BingBongVoice;
[Module(Enabled = true)]
class BingBongVoiceModule : Module
{
public static BingBongVoiceModule? Instance { get; private set; }
public override string ModuleName => "BingBong Voice Module";
public BingBongVoiceClient? Client { get; private set; }
private GameObject? _debugObject;
private const string ChannelSalt = "jordanmod-bingbong-voice-v1";
public override Type[] GetPatches()
{
// Config is bound before modules load (Plugin.Awake: ConfigHandler.Initialize ->
// SetupModules -> SetupPatches), so turning the feature off here means the Harmony
// patches are never applied at all and the mod has zero footprint.
if (!ConfigHandler.BingBongVoiceEnabled.Value) return [];
return [typeof(BingBongVoicePatch)];
}
public override void Initialize()
{
if (Instance != null) return;
Instance = this;
// The relay lives on the audio bank, so with no override configured every player derives
// the same URL and connects on their own -- nobody has to press anything.
PhotonNetworkEventListener listener = PhotonNetworkEventListener.Instance!;
if (listener != null)
{
listener.RegisterOnJoinedRoom(OnJoinedRoom);
listener.RegisterOnLeftRoom(Disconnect);
listener.RegisterOnDisconnected(_ => Disconnect());
}
else
{
Debug.LogWarning("[BingBongVoice] No network event listener, auto-connect is unavailable.");
}
base.Initialize();
}
private void OnJoinedRoom()
{
if (!ConfigHandler.BingBongVoiceEnabled.Value) return;
if (!ConfigHandler.BingBongVoiceAutoConnect.Value) return;
Connect();
}
public override void Update()
{
if (!ConfigHandler.BingBongVoiceEnabled.Value) return;
if (Input.GetKeyDown(ConfigHandler.BingBongVoiceDebugKey.Value)) ToggleStream();
}
/// <summary>
/// Where the relay lives. The relay is served by the audio bank, so an unset override just
/// derives the WebSocket URL from BugleSoundAPIURL; setting it points somewhere else
/// entirely (a local stub, a separate host).
/// </summary>
public static string ResolveRelayUrl()
{
string overrideUrl = (ConfigHandler.BingBongVoiceRelayURL.Value ?? "").Trim();
string source;
string url;
if (overrideUrl.Length > 0)
{
// An override names the endpoint. Only fill in /voice/ws when it has no path at all,
// so "localhost:8787" works but "wss://host/custom/path" is left alone.
source = overrideUrl;
url = NormalizeOverride(overrideUrl);
}
else
{
// The audio bank URL names a service root, which may itself sit under a path
// (https://audiobank.thejordan.dev/api), so the endpoint always goes on the end.
source = (ConfigHandler.BugleSoundAPIURL.Value ?? "").Trim();
if (source.Length == 0)
{
Debug.LogWarning("[BingBongVoice] Neither RelayURL nor BugleSoundAPIURL is set, nowhere to connect.");
return "";
}
url = DeriveFromApiBase(source);
}
if (url.Length == 0)
{
Debug.LogError($"[BingBongVoice] Could not make a usable WebSocket URL out of '{source}'. Expected something like 'ws://localhost:8787/voice/ws' or just 'localhost:8787'.");
return "";
}
// A URL that already carries a token is left exactly as given -- that is how you point at
// a stub relay, or connect from outside a lobby.
if (url.IndexOf("token=", StringComparison.OrdinalIgnoreCase) < 0)
{
string channel = ResolveChannel();
if (channel.Length == 0)
{
Debug.LogWarning("[BingBongVoice] Not in a lobby, so there is no channel to join. Put a full URL including ?token= in RelayURL to connect from outside one.");
return "";
}
// The relay's published contract is "listen:<channel>". It also accepts a bare channel
// id, but following the documented form is what stays correct if it ever tightens.
// The channel must be lowercase hex -- uppercase is rejected with a 1008.
url += (url.IndexOf('?') >= 0 ? "&" : "?") + "token=" + Uri.EscapeDataString("listen:" + channel);
}
return url;
}
/// Announced to the relay on connect so admins can see which lobbies are live and who is in
/// them. Deliberately carries the player name but NOT the room code: the name is what makes a
/// session recognisable in the web app, whereas the code is what someone would need to join
/// the game, so there is no reason to put it on the wire.
private static string BuildHello()
{
string nickname = PhotonNetwork.LocalPlayer?.NickName ?? "";
if (string.IsNullOrWhiteSpace(nickname)) nickname = "Unknown";
return JsonConvert.SerializeObject(new { type = "hello", player = nickname });
}
/// <summary>
/// Scopes the voice channel to the Photon lobby, so each game session gets its own and
/// players only hear whoever is speaking into theirs.
///
/// Hashed rather than sent as-is: the room code is what someone would need to join the game,
/// and it would otherwise sit in the URL and in every server access log. The salt is baked
/// into the mod so it is not a secret -- it only stops a channel id from being trivially
/// reversible into a joinable room code. Once the relay's own join endpoint exists it should
/// issue channel ids server-side instead, with a salt that really is private.
/// </summary>
public static string ResolveChannel()
{
string room = PhotonNetwork.CurrentRoom?.Name ?? "";
if (string.IsNullOrWhiteSpace(room)) return "";
using SHA256 sha = SHA256.Create();
byte[] hash = sha.ComputeHash(Encoding.UTF8.GetBytes(ChannelSalt + room.Trim().ToUpperInvariant()));
return BitConverter.ToString(hash).Replace("-", "").ToLowerInvariant().Substring(0, 32);
}
private static string NormalizeOverride(string raw)
{
string url = ToWebSocketScheme(raw);
// Split the query/fragment off first so appending a path can't mangle them.
int split = url.IndexOfAny(['?', '#']);
string tail = split >= 0 ? url.Substring(split) : "";
string head = split >= 0 ? url.Substring(0, split) : url;
if (!IsWebSocketUrl(head, out Uri? parsed)) return "";
// A bare host means the caller gave us the root, not the endpoint.
if (parsed.AbsolutePath.Length <= 1) head = head.TrimEnd('/') + "/voice/ws";
return head + tail;
}
private static string DeriveFromApiBase(string apiBase)
{
string url = ToWebSocketScheme(apiBase);
// The base is a service root, so anything after it is not ours to keep.
int split = url.IndexOfAny(['?', '#']);
if (split >= 0) url = url.Substring(0, split);
url = url.TrimEnd('/') + "/voice/ws";
return IsWebSocketUrl(url, out _) ? url : "";
}
private static bool IsWebSocketUrl(string url, out Uri parsed)
{
parsed = null!;
if (!Uri.TryCreate(url, UriKind.Absolute, out Uri? candidate)) return false;
if (candidate.Scheme != "ws" && candidate.Scheme != "wss") return false;
parsed = candidate;
return true;
}
private static string ToWebSocketScheme(string url)
{
if (url.StartsWith("ws://", StringComparison.OrdinalIgnoreCase)) return url;
if (url.StartsWith("wss://", StringComparison.OrdinalIgnoreCase)) return url;
if (url.StartsWith("https://", StringComparison.OrdinalIgnoreCase)) return "wss://" + url.Substring(8);
if (url.StartsWith("http://", StringComparison.OrdinalIgnoreCase)) return "ws://" + url.Substring(7);
// No scheme at all: same rule as everywhere else, loopback gets the insecure variant.
return Helper.EnsureScheme(url, "wss", "ws");
}
// Every BingBong attaches its own source through BingBongVoicePatch; the optional debug
// source is the stage 1 harness, for testing with no BingBong in the scene.
private void ToggleStream()
{
if (Client != null) Disconnect();
else Connect();
}
private void Connect()
{
if (Client != null) return;
string url = ResolveRelayUrl();
if (url.Length == 0) return;
Debug.Log($"[BingBongVoice] Connecting for lobby '{PhotonNetwork.CurrentRoom?.Name}' to {url}");
Client = new BingBongVoiceClient();
Client.OnSpeakerChanged += AnnounceSpeaker;
Client.Connect(url, BuildHello());
if (!ConfigHandler.BingBongVoiceDebugSource.Value) return;
_debugObject = new GameObject("BingBongVoiceDebug");
UnityEngine.Object.DontDestroyOnLoad(_debugObject);
BingBongVoiceSource.Attach(_debugObject, null);
}
// Reuses the Bugle's actionbar because it is the only on-screen text this mod has. Raised on
// the main thread by the client, so touching Unity here is safe.
private static void AnnounceSpeaker(string? speaker)
{
if (!ConfigHandler.BingBongVoiceAnnounceSpeaker.Value) return;
BetterBugleUI.Instance?.ShowActionbar(speaker != null
? $"🎙 BingBong is being puppeted by {speaker}"
: "BingBong is himself again");
}
private void Disconnect()
{
if (Client == null) return;
Client.OnSpeakerChanged -= AnnounceSpeaker;
Debug.Log("[BingBongVoice] Disconnecting from relay.");
Client.Disconnect();
Client = null;
if (_debugObject != null) UnityEngine.Object.Destroy(_debugObject);
_debugObject = null;
}
public override void Destroy()
{
PhotonNetworkEventListener listener = PhotonNetworkEventListener.Instance!;
if (listener != null)
{
listener.JoinedRoom -= OnJoinedRoom;
listener.LeftRoom -= Disconnect;
}
Client?.Disconnect();
Client = null;
if (_debugObject != null) UnityEngine.Object.Destroy(_debugObject);
_debugObject = null;
base.Destroy();
}
}

448
src/JordanMod/modules/bingbong_voice/BingBongVoiceSource.cs Normal file
View File

@@ -0,0 +1,448 @@
using System;
using System.Runtime.CompilerServices;
using UnityEngine;
namespace JordanMod.Modules.BingBongVoice;
/// <summary>
/// Plays the live voice stream through one AudioSource. One of these per BingBong (and one
/// non-positional debug instance), each with its own read cursor into the shared ring.
///
/// Output goes through <c>OnAudioFilterRead</c> rather than a procedural streaming AudioClip.
/// A clip created with <c>stream: true</c> fills Unity's own decode buffer ahead of playback --
/// that read-ahead is by design, and it cannot be satisfied from a live stream because the audio
/// simply has not arrived yet, so the ring drained to empty no matter how deep the jitter buffer
/// was made. OnAudioFilterRead is invoked exactly once per DSP block, in lockstep with playback,
/// so demand can never outrun real time. The AudioSource still needs *something* playing for the
/// filter to run, hence the silent carrier clip.
/// </summary>
public sealed class BingBongVoiceSource : MonoBehaviour
{
private enum State { Buffering, Playing }
private AudioSource? _source;
private AudioClip? _carrier;
private VoiceStream? _stream;
private int _outputRate;
private double _read; // fractional position into the ring, audio thread only
private State _state = State.Buffering;
private float _gain; // smoothed toward _targetGain, audio thread only
private volatile float _targetGain;
private volatile float _amplitude; // post-gain, what you actually hear
private volatile float _rawAmplitude; // pre-gain MEAN |sample|, matching BingBongMouth's units
private volatile bool _dead;
private int _underruns;
private float _lastReportTime;
private int _strikes;
private volatile int _depthSamples;
private volatile int _minDepth = int.MaxValue;
private volatile int _streamRate = 1;
private volatile int _blockFrames; // frames per DSP block, as actually delivered
private static bool _reportedBlock; // once per session, not once per BingBong
private static int _liveSources; // how many voice sources currently exist
private static int _playingSources; // how many are actually enabled and audible
private float _gainStep = 0.0008f;
private long _lastWritePos;
private float _lastRateTime;
private float _appliedRange = -1f;
private BingBongMouth? _mouth;
private int _mouthHash;
private float _mouthValue;
private int _mouthStrikes;
private bool _mouthDisabled;
/// Set by the duck patch. Live voice is attenuated until this time.
public float DuckUntil { get; set; }
public float Amplitude => _amplitude;
private const string ChildName = "BingBongVoice";
/// <summary>
/// Always lives on its own child GameObject. OnAudioFilterRead hooks the DSP chain of the
/// AudioSource sharing its GameObject, and BingBong already carries the vanilla voice-line
/// source on the item root -- putting ours alongside it meant our filter ran on *their*
/// source, so the live stream was only audible while a canned line played and cut out
/// whenever that source stopped. A child of our own keeps the two chains separate, and
/// still follows the item since it is parented to it.
/// </summary>
public static BingBongVoiceSource Attach(GameObject target, AudioSource? template)
{
Transform existing = target.transform.Find(ChildName);
if (existing != null && existing.TryGetComponent(out BingBongVoiceSource found)) return found;
GameObject holder = new(ChildName);
holder.transform.SetParent(target.transform, worldPositionStays: false);
BingBongVoiceSource voice = holder.AddComponent<BingBongVoiceSource>();
voice.Configure(template);
return voice;
}
/// How many voice sources exist right now. Should be one per BingBong in the scene; if it
/// climbs with every grab then dead instances are still playing and everything is audible at
/// once, which reads as "no spatialisation" just as convincingly as a broken panner does.
public static int LiveSources => _liveSources;
/// Enabled instances. PEAK keeps several BingBong objects around (held and prop variants), so
/// what matters is not how many exist but how many are audible at once -- more than one would
/// sum to several times the volume and smear the apparent position.
public static int PlayingSources => _playingSources;
/// Lets the attach patch hand us BingBong's mouth animator. Kept separate from Attach so a
/// change to BingBongMouth can only break the animation, never the audio.
public void BindMouth(BingBongMouth mouth)
{
_mouth = mouth;
string param = string.IsNullOrEmpty(mouth.animValue) ? "Mouth Blend" : mouth.animValue;
_mouthHash = Animator.StringToHash(param);
Debug.Log($"[BingBongVoice] Mouth bound: parameter '{param}', maxMouthOpen={mouth.maxMouthOpen}.");
}
public static bool IsAttached(GameObject target) => Find(target) != null;
public static BingBongVoiceSource? Find(GameObject target)
{
Transform existing = target.transform.Find(ChildName);
return existing != null && existing.TryGetComponent(out BingBongVoiceSource found) ? found : null;
}
private void Configure(AudioSource? template)
{
_outputRate = AudioSettings.outputSampleRate;
_gainStep = 1f / (_outputRate * 0.015f); // ~15ms attack whatever the device runs at
// Carrier of constant 1.0, not silence, and the filter MULTIPLIES rather than overwrites.
// Unity applies 3D panning and distance attenuation as a per-channel gain; if that runs
// before custom filters then the buffer handed to us already carries that gain, and
// overwriting it threw the spatialisation away (the voice was audible at full volume from
// anywhere). Multiplying a unit carrier reapplies it. If the panner instead runs after
// the filter, the incoming value is simply 1.0 and multiplying changes nothing -- so this
// is correct either way round, without having to determine which order Unity uses.
_carrier = AudioClip.Create($"BingBongVoiceCarrier_{GetInstanceID()}", 4096, 1, _outputRate, false);
float[] unit = new float[4096];
for (int i = 0; i < unit.Length; i++) unit[i] = 1f;
_carrier.SetData(unit, 0);
_source = gameObject.AddComponent<AudioSource>();
_source.clip = _carrier;
_source.loop = true;
_source.playOnAwake = false;
if (template != null)
{
// Copy the vanilla voice-line source's 3D setup rather than guessing, so the live
// voice attenuates exactly like BingBong's own lines. Routing through its mixer
// group also means the game's volume sliders apply for free.
_source.spatialBlend = template.spatialBlend;
ApplyRange(ConfigHandler.BingBongVoiceMaxDistance.Value);
_source.rolloffMode = template.rolloffMode;
_source.dopplerLevel = template.dopplerLevel;
_source.spread = template.spread;
_source.outputAudioMixerGroup = template.outputAudioMixerGroup;
}
else
{
_source.spatialBlend = 0f; // debug instance: non-positional, always audible
}
_source.Play();
_liveSources++;
AudioSettings.OnAudioConfigurationChanged += OnAudioConfigChanged;
}
/// <summary>
/// Range needs both distances, not just maxDistance. Under Logarithmic rolloff the falloff is
/// governed by minDistance -- volume goes roughly as minDistance/distance -- and maxDistance
/// only marks where attenuation stops changing. Vanilla's minDistance of 1 means the sound is
/// at 1/30 volume by 30m, so raising maxDistance alone did nothing at all. Scaling both
/// together stretches the same curve over the range asked for: full volume within a tenth of
/// it, and still audible out at the limit.
/// </summary>
private void ApplyRange(float range)
{
if (_source == null) return;
_source.maxDistance = range;
_source.minDistance = Mathf.Max(1f, range / 10f);
_appliedRange = range;
}
private void OnAudioConfigChanged(bool deviceChanged)
{
// Owning the resample ratio ourselves makes this a one-line update instead of a rebuild.
_outputRate = AudioSettings.outputSampleRate;
_gainStep = 1f / (_outputRate * 0.015f);
_state = State.Buffering;
}
private void Update()
{
_stream = BingBongVoiceModule.Instance?.Client?.Stream;
_targetGain = Time.time < DuckUntil
? ConfigHandler.BingBongVoiceDuckVolume.Value
: ConfigHandler.BingBongVoiceVolume.Value;
// Picked up live, like the volumes are, so the range can be tuned without re-spawning
// every BingBong in the scene.
float range = ConfigHandler.BingBongVoiceMaxDistance.Value;
if (!Mathf.Approximately(_appliedRange, range)) ApplyRange(range);
if (_mouth != null && !_mouthDisabled)
{
try
{
DriveMouth();
}
catch (Exception e)
{
Debug.LogError($"[BingBongVoice] Mouth animation failed: {e}");
if (++_mouthStrikes >= 3)
{
_mouthDisabled = true;
Debug.LogError("[BingBongVoice] Mouth animation disabled for this session.");
}
}
}
if (!_reportedBlock && _blockFrames > 0)
{
_reportedBlock = true;
AudioSettings.GetDSPBufferSize(out int dspLength, out int dspCount);
Debug.Log($"[BingBongVoice] DSP block is {_blockFrames} frames ({_blockFrames * 1000 / Math.Max(1, _outputRate)}ms at {_outputRate}Hz); buffer {dspLength}x{dspCount}.");
}
if (Time.time - _lastReportTime > 5f && _underruns > 0)
{
_lastReportTime = Time.time;
int depthMs = _depthSamples * 1000 / Math.Max(1, _streamRate);
int minMs = _minDepth == int.MaxValue ? -1 : _minDepth * 1000 / Math.Max(1, _streamRate);
int targetMs = TargetSamples(_streamRate) * 1000 / Math.Max(1, _streamRate);
long writeNow = _stream?.Ring.WritePosition ?? 0;
float elapsed = Time.time - _lastRateTime;
float producedRatio = elapsed > 0.01f && _lastWritePos > 0
? (writeNow - _lastWritePos) / (elapsed * Math.Max(1, _streamRate))
: 1f;
_lastWritePos = writeNow;
_lastRateTime = Time.time;
string hint = producedRatio < 0.95f
? $" - SENDER is only producing {producedRatio:P0} of real time"
: depthMs <= 1 ? " - CONSUMER is draining faster than real time" : "";
Debug.Log($"[BingBongVoice] buffer health: {_underruns} underruns, depth {depthMs}ms (min {minMs}ms) of {targetMs}ms{hint}");
_underruns = 0;
_minDepth = int.MaxValue;
}
}
/// <summary>
/// Drives BingBong's mouth from the audio we are actually playing, so it works on every
/// client rather than only the speaker's.
///
/// BingBongMouth.Update only writes the animator parameter while canPlay is true, so holding
/// canPlay false is enough to take over cleanly -- no fighting over script execution order.
/// During a duck we stop forcing it and stop writing, which lets OnAsk -> CreateCurveMap set
/// canPlay itself and animate the canned line off its baked curve exactly as shipped.
/// </summary>
[MethodImpl(MethodImplOptions.NoInlining)]
private void DriveMouth()
{
BingBongMouth mouth = _mouth!;
if (mouth == null || mouth.animator == null) return;
if (Time.time < DuckUntil)
{
// Vanilla owns the mouth for the duration of its own voice line.
_mouthValue = 0f;
return;
}
if (_stream != null)
{
mouth.canPlay = false;
// Vanilla's units exactly: BingBongMouth bakes a curve of MEAN |sample| per window and
// feeds the animator curveMap.Evaluate(t) * maxMouthOpen, so maxMouthOpen is the factor
// that lifts those small means into the animator's range. Passing an already-normalised
// 0-1 value and then multiplying by it saturated the parameter, which held the jaw wide
// open for anything above near-silence. Matching the scale means it behaves like a
// canned line does, whatever maxMouthOpen happens to be.
float target = _rawAmplitude * mouth.maxMouthOpen;
_mouthValue = Mathf.Lerp(_mouthValue, target, Time.deltaTime * 25f);
mouth.animator.SetFloat(_mouthHash, _mouthValue);
}
else if (_mouthValue > 0.001f)
{
// Disconnected: close it rather than leaving the jaw hanging where it stopped.
_mouthValue = Mathf.Lerp(_mouthValue, 0f, Time.deltaTime * 10f);
mouth.animator.SetFloat(_mouthHash, _mouthValue);
}
}
// How much audio to hold before playing. One DSP block is consumed on every call, so the
// floor is (target - block); keeping two blocks plus the configured margin leaves that floor
// comfortably above the point where a read cannot be satisfied.
private int TargetSamples(int streamRate)
{
int blockStream = (int)(_blockFrames * (double)streamRate / Math.Max(1, _outputRate)) + 1;
return blockStream * 2 + (int)(streamRate * ConfigHandler.BingBongVoiceJitterMs.Value / 1000f);
}
// ---- audio thread from here down -------------------------------------------------
// Allocation-free, lock-free, no Unity API calls, no logging. An exception escaping onto
// the DSP thread is not recoverable, so everything is wrapped and falls back to silence.
private void OnAudioFilterRead(float[] data, int channels)
{
try
{
Fill(data, Math.Max(1, channels));
}
catch
{
Array.Clear(data, 0, data.Length);
_amplitude = 0f;
if (++_strikes >= 3) _dead = true;
}
}
private void Fill(float[] data, int channels)
{
int frames = data.Length / channels;
_blockFrames = frames;
VoiceStream? stream = _stream;
if (_dead || stream == null)
{
Silence(data);
return;
}
VoiceRingBuffer ring = stream.Ring;
long write = ring.WritePosition;
double ratio = (double)ring.SampleRate / _outputRate;
int targetSamples = TargetSamples(ring.SampleRate);
if (_state == State.Buffering)
{
if (write - (long)_read < targetSamples || write < targetSamples)
{
Silence(data);
return;
}
_read = write - targetSamples;
_state = State.Playing;
}
// Drifted too far behind (sender burst, or we were paused). Rejoin the live edge rather
// than playing further and further into the past.
if (write - (long)_read > targetSamples * 5)
{
_read = write - targetSamples;
}
double needed = frames * ratio;
long available = write - (long)_read;
_depthSamples = (int)Math.Min(int.MaxValue, available);
if (_depthSamples < _minDepth) _minDepth = _depthSamples;
_streamRate = ring.SampleRate;
int usableFrames = frames;
if (available < needed + 2)
{
usableFrames = (int)Math.Max(0, (available - 2) / ratio);
if (usableFrames > frames) usableFrames = frames;
_underruns++;
// Only rebuild the whole buffer when there was essentially nothing there. A brief
// shortfall should cost a few samples, not a fade plus a full buffer of silence.
if (usableFrames < frames / 4) _state = State.Buffering;
}
Render(data, ring, channels, usableFrames, ratio, fadeOut: usableFrames < frames);
// Pad any frames we could not fill.
for (int f = usableFrames; f < frames; f++)
for (int c = 0; c < channels; c++)
data[f * channels + c] = 0f;
}
private void Render(float[] data, VoiceRingBuffer ring, int channels, int frames, double ratio, bool fadeOut)
{
float peak = 0f;
float rawSum = 0f;
float fadeStep = fadeOut && frames > 0 ? _gain / frames : 0f;
for (int f = 0; f < frames; f++)
{
long i0 = (long)_read;
float frac = (float)(_read - i0);
float sample = ring[i0] * (1f - frac) + ring[i0 + 1] * frac;
if (fadeOut) _gain = Math.Max(0f, _gain - fadeStep);
else _gain += (_targetGain - _gain) * _gainStep;
float outSample = sample * _gain;
int baseIndex = f * channels;
for (int c = 0; c < channels; c++) data[baseIndex + c] *= outSample;
float abs = outSample < 0f ? -outSample : outSample;
if (abs > peak) peak = abs;
rawSum += sample < 0f ? -sample : sample;
_read += ratio;
}
_amplitude = peak;
_rawAmplitude = frames > 0 ? rawSum / frames : 0f;
}
private void Silence(float[] data)
{
Array.Clear(data, 0, data.Length);
_gain = 0f;
_amplitude = 0f;
_rawAmplitude = 0f;
}
// ----------------------------------------------------------------------------------
private void OnDestroy()
{
// The audio thread can and will run the filter after Destroy happens on the main thread,
// so flag it dead first: that check is what makes the teardown safe.
_dead = true;
_liveSources--;
AudioSettings.OnAudioConfigurationChanged -= OnAudioConfigChanged;
if (_source != null)
{
_source.Stop();
_source.clip = null;
}
if (_carrier != null) Destroy(_carrier);
}
private void OnDisable()
{
_playingSources--;
if (_source != null) _source.Stop();
_state = State.Buffering;
}
private void OnEnable()
{
_playingSources++;
// Rejoin live rather than trying to resume where we left off.
_state = State.Buffering;
if (_source != null && _carrier != null) _source.Play();
}
}

71
src/JordanMod/modules/bingbong_voice/VoiceRingBuffer.cs Normal file
View File

@@ -0,0 +1,71 @@
using System;
using System.Threading;
namespace JordanMod.Modules.BingBongVoice;
/// <summary>
/// Ring of mono float samples at the stream's sample rate. Single producer (the websocket
/// receive thread), multiple consumers (one <see cref="BingBongVoiceSource"/> per BingBong,
/// each reading from the audio thread with its own cursor).
///
/// Lock-free by construction: the producer publishes <see cref="WritePosition"/> only after the
/// samples are in place, and the ring is far deeper than the jitter target, so a consumer can
/// never observe a slot that is being overwritten under it. Consumers hold absolute positions
/// and resync if they ever fall further behind than the ring is long.
/// </summary>
public sealed class VoiceRingBuffer
{
private readonly float[] _buf;
private readonly int _mask;
private long _write;
public int SampleRate { get; }
public int Capacity => _buf.Length;
/// Total samples ever written. Monotonic; never reset, so consumer cursors stay meaningful.
public long WritePosition => Volatile.Read(ref _write);
public VoiceRingBuffer(int sampleRate, float seconds = 4f)
{
SampleRate = sampleRate;
int wanted = Math.Max(1024, (int)(sampleRate * seconds));
int size = 1;
while (size < wanted) size <<= 1; // power of two so the wrap is a mask, not a modulo
_buf = new float[size];
_mask = size - 1;
}
/// Producer thread only. Converts interleaved little-endian PCM16 straight out of the
/// websocket's byte[] with no intermediate allocation.
public void WritePcm16(byte[] src, int offset, int byteCount)
{
int samples = byteCount >> 1;
if (samples <= 0) return;
long w = _write; // safe unsynchronised read: this thread is the only writer
for (int i = 0; i < samples; i++)
{
int b = offset + (i << 1);
short s = (short)(src[b] | (src[b + 1] << 8));
_buf[(int)((w + i) & _mask)] = s * (1f / 32768f);
}
Volatile.Write(ref _write, w + samples); // publish only once the data is really there
}
/// Producer thread only. Used to reproduce an exact gap the sender told us about, so a
/// dropped frame costs its own duration rather than shifting everything after it.
public void WriteSilence(int samples)
{
if (samples <= 0) return;
if (samples > _buf.Length) samples = _buf.Length;
long w = _write;
for (int i = 0; i < samples; i++) _buf[(int)((w + i) & _mask)] = 0f;
Volatile.Write(ref _write, w + samples);
}
/// Consumer read. The caller is responsible for staying within
/// <see cref="Capacity"/> of <see cref="WritePosition"/>.
public float this[long index] => _buf[(int)(index & _mask)];
}

135
src/JordanMod/patches/BingBongVoicePatch.cs Normal file
View File

@@ -0,0 +1,135 @@
using System;
using System.Runtime.CompilerServices;
using HarmonyLib;
using UnityEngine;
namespace JordanMod.Modules.BingBongVoice;
/// <summary>
/// Gives every BingBong a live-voice source as it spawns.
///
/// Structured defensively on purpose. A postfix that throws propagates out of the method it
/// patched, and because Mono resolves a method's call targets when it JITs that method, a game
/// API that changed shape throws before any guard inside can run -- that is how a spawner
/// postfix once took down every SingleItemSpawner in the level (see 0.1.13). So the patch body
/// is nothing but a try/catch around a separate non-inlined method, and three failures disable
/// it for the session. A broken voice feature must never cost anyone their BingBong.
/// </summary>
public class BingBongVoicePatch
{
private static int _strikes;
private static bool _disabled;
private static int _attachCount;
private static int _duckStrikes;
private static bool _duckDisabled;
[HarmonyPatch(typeof(Item), "Start")]
[HarmonyPostfix]
static void ItemStartPostfix(Item __instance)
{
if (_disabled) return;
try
{
AttachIfBingBong(__instance);
}
catch (Exception e)
{
Fail(e);
}
}
// The only method here that touches game APIs, kept separate and un-inlined so that a
// missing member throws inside the caller's try rather than while JITing the patch itself.
[MethodImpl(MethodImplOptions.NoInlining)]
private static void AttachIfBingBong(Item item)
{
if (item == null || item.UIData == null) return;
if (!item.itemTags.HasFlag(Item.ItemTags.BingBong)) return;
if (BingBongVoiceSource.IsAttached(item.gameObject)) return;
// Copy the vanilla voice-line source's 3D setup so the live voice carries and attenuates
// exactly like BingBong's own lines, and inherits its mixer group.
AudioSource? template = item.TryGetComponent(out Action_AskBingBong ask) ? ask.source : null;
BingBongVoiceSource voice = BingBongVoiceSource.Attach(item.gameObject, template);
// The mouth animator lives on a child of the plush, not the item root.
BingBongMouth mouth = item.GetComponentInChildren<BingBongMouth>(includeInactive: true);
if (mouth != null) voice.BindMouth(mouth);
// Log the copied values: if the live voice turns out not to attenuate with distance,
// this distinguishes "the copy failed" from "Unity spatialises before custom filters".
// Counted because this should happen once per BingBong that exists, ever. If the number
// climbs every time the item is grabbed or thrown then Item.Start is re-running on a
// fresh instance, and the per-attach cost (AudioClip.Create, AddComponent) is the hitch.
_attachCount++;
if (template != null)
Debug.Log($"[BingBongVoice] Attached to a BingBong (#{_attachCount}, {BingBongVoiceSource.LiveSources} live, {BingBongVoiceSource.PlayingSources} audible); spatialBlend={template.spatialBlend}, range={ConfigHandler.BingBongVoiceMaxDistance.Value}m, rolloff={template.rolloffMode}, mixer={(template.outputAudioMixerGroup != null ? template.outputAudioMixerGroup.name : "none")}.");
else
Debug.LogWarning("[BingBongVoice] Attached to a BingBong, but it has no Action_AskBingBong to copy 3D settings from - the voice will not be positional.");
}
private static void Fail(Exception e)
{
Debug.LogError($"[BingBongVoice] Attach patch failed: {e}");
if (++_strikes < 3) return;
_disabled = true;
Debug.LogError("[BingBongVoice] Attach patch disabled for this session after repeated failures.");
}
/// <summary>
/// Ducks the live voice while one of BingBong's own lines plays, so the two never talk over
/// each other. This is the RPC body, so it runs on every client and the duck stays in step
/// across the lobby without any networking of our own.
/// </summary>
[HarmonyPatch(typeof(Action_AskBingBong), nameof(Action_AskBingBong.Ask))]
[HarmonyPostfix]
static void AskPostfix(Action_AskBingBong __instance, int index, bool spamming)
{
if (_duckDisabled) return;
try
{
DuckForVoiceLine(__instance, index, spamming);
}
catch (Exception e)
{
Debug.LogError($"[BingBongVoice] Duck patch failed: {e}");
if (++_duckStrikes < 3) return;
_duckDisabled = true;
Debug.LogError("[BingBongVoice] Duck patch disabled for this session after repeated failures.");
}
}
[MethodImpl(MethodImplOptions.NoInlining)]
private static void DuckForVoiceLine(Action_AskBingBong ask, int index, bool spamming)
{
// Mirror vanilla's own early-outs, so we never duck for audio that is not going to play.
// Ask() does nothing at all unless the item is held, and AskRoutine bails before playing
// anything when it decides you are spamming the interaction.
if (ask.item == null || ask.item.holderCharacter == null) return;
if (spamming) return;
// RunAction picks the index locally and RPCs the raw number, while ReplaceBingBongModule
// appends a response per loaded voice line -- so two players with different audio bank
// state disagree on the array length. Vanilla throws in that case; we must not.
Action_AskBingBong.BingBongResponse[] responses = ask.responses;
if (responses == null || index < 0 || index >= responses.Length) return;
Action_AskBingBong.BingBongResponse response = responses[index];
if (response?.sfx == null || response.sfx.clips == null || response.sfx.clips.Length == 0) return;
AudioClip clip = response.sfx.clips[0];
if (clip == null) return;
BingBongVoiceSource? voice = BingBongVoiceSource.Find(ask.item.gameObject);
if (voice == null) return;
// AskRoutine burns 0.5s in its update loop, stops the source, waits another 0.5s and only
// then plays -- so audio starts about a second after this RPC lands. Ducking from now
// rather than from the actual start reads as BingBong drawing breath, and removes a whole
// class of "the duck began a few frames late and you heard a syllable" bugs.
float duration = ConfigHandler.BingBongVoiceDuckLead.Value + clip.length + 0.2f;
voice.DuckUntil = Math.Max(voice.DuckUntil, Time.time + duration);
}
}

344
tools/stub_relay.py Normal file
View File

@@ -0,0 +1,344 @@
"""Development stub relay for BingBong Live Voice.
Speaks the exact wire format the real relay (an endpoint on the AudioBank FastAPI service)
will use, so the mod can be built and tested before that backend exists. Deliberately minimal:
no auth, no session directory, no persistence. It is a test fixture, not a reference
implementation — build the real thing from the plan, not from this.
Requires `websockets` (pip install websockets).
python tools/stub_relay.py serve # relay on ws://localhost:8787/voice/ws
python tools/stub_relay.py tone # stream a 1 kHz sine into channel "test"
python tools/stub_relay.py tone --sweep # sweep 200-2000 Hz (resampling bugs are obvious)
python tools/stub_relay.py tone --speech # syllables and word gaps (for mouth animation)
python tools/stub_relay.py listen # dump received frames, to verify fan-out
Point the mod at it with, in BepInEx config:
[BingBong Voice]
RelayURL = ws://localhost:8787/voice/ws?token=listen:test
Leaving RelayURL empty instead derives the URL from BugleSoundAPIURL, which is where the
relay lives in production.
Connect as: ws://localhost:8787/voice/ws?token=listen:test
ws://localhost:8787/voice/ws?token=speak:test
Wire format — 12-byte little-endian header, then PCM16LE mono payload:
0 u8 magic 0xB1
1 u8 version 1
2 u8 codec 0=PCM16LE 1=mu-law
3 u8 rate code 0=16k 1=8k 2=24k 3=48k
4 u32 seq
8 u32 sampleTimestamp (samples since stream start)
Binary frame = audio. Text frame = JSON control.
"""
import argparse
import asyncio
import json
import math
import struct
import sys
from collections import defaultdict
from urllib.parse import urlparse, parse_qs
from websockets.asyncio.server import serve
from websockets.asyncio.client import connect
HOST = "localhost"
PORT = 8787
PATH = "/voice/ws"
MAGIC = 0xB1
VERSION = 1
CODEC_PCM16 = 0
RATE_CODES = {16000: 0, 8000: 1, 24000: 2, 48000: 3}
FRAME_MS = 20
HEADER = struct.Struct("<BBBBII")
def pack_header(seq: int, ts: int, rate: int = 16000, codec: int = CODEC_PCM16) -> bytes:
return HEADER.pack(MAGIC, VERSION, codec, RATE_CODES[rate], seq & 0xFFFFFFFF, ts & 0xFFFFFFFF)
# ---------------------------------------------------------------- relay
channels: dict[str, dict] = defaultdict(lambda: {"speaker": None, "listeners": set()})
names: dict = {}
def roster(channel: str) -> str:
"""Who the relay believes is in this lobby -- the stub's stand-in for /voice/sessions."""
return ", ".join(sorted(names.get(w, "?") for w in channels[channel]["listeners"])) or "(nobody)"
def print_sessions():
if not channels:
print("[relay] no active sessions")
return
for chan, ch in channels.items():
if not ch["listeners"] and ch["speaker"] is None:
continue
speaking = "speaking" if ch["speaker"] is not None else "idle"
print(f"[relay] {chan} {len(ch['listeners'])} listening [{speaking}] {roster(chan)}")
def parse_token(raw_path: str):
"""token=<role>:<channel>. Returns (role, channel) or (None, None)."""
q = parse_qs(urlparse(raw_path).query)
token = (q.get("token") or [""])[0]
if not token:
return None, None
# "<role>:<channel>" is this stub's own convention, kept because it makes a speaker and a
# listener easy to start from a shell. The deployed relay takes a bare channel id, which is
# what the mod sends, so accept that too and default it to listening.
if ":" in token:
role, _, channel = token.partition(":")
if role not in ("listen", "speak") or not channel:
return None, None
return role, channel
return "listen", token
async def handler(ws):
role, channel = parse_token(ws.request.path)
if role is None:
await ws.close(1008, "bad token")
return
ch = channels[channel]
if role == "speak":
old = ch["speaker"]
if old is not None:
try:
await old.send(json.dumps({"type": "preempted"}))
await old.close(1000, "preempted")
except Exception:
pass
ch["speaker"] = ws
print(f"[relay] speaker joined '{channel}' ({len(ch['listeners'])} listening)")
await broadcast_control(channel, {"type": "speaker_joined"})
try:
async for msg in ws:
if isinstance(msg, bytes):
await fanout(channel, msg)
except Exception:
pass
finally:
if ch["speaker"] is ws:
ch["speaker"] = None
print(f"[relay] speaker left '{channel}'")
await broadcast_control(channel, {"type": "speaker_left"})
return
ch["listeners"].add(ws)
names[ws] = "?"
print(f"[relay] listener joined '{channel}' -> {len(ch['listeners'])} listening")
try:
# Listeners send a hello frame naming the player. That is what the real relay turns into
# its session directory, so the stub records it too and prints the roster.
async for msg in ws:
if not isinstance(msg, str):
continue
try:
payload = json.loads(msg)
except ValueError:
continue
if payload.get("type") == "hello":
names[ws] = str(payload.get("player", "?"))
print(f"[relay] '{names[ws]}' registered in '{channel}' -> roster: {roster(channel)}")
except Exception:
pass
finally:
ch["listeners"].discard(ws)
who = names.pop(ws, "?")
print(f"[relay] '{who}' left '{channel}' -> {len(ch['listeners'])} listening")
async def fanout(channel: str, payload: bytes):
listeners = list(channels[channel]["listeners"])
if not listeners:
return
# Real relay uses a bounded per-listener queue and drops oldest; the stub just fires and
# forgets, which is fine locally and keeps this file short.
await asyncio.gather(*(safe_send(w, payload) for w in listeners), return_exceptions=True)
async def broadcast_control(channel: str, obj: dict):
msg = json.dumps(obj)
await asyncio.gather(
*(safe_send(w, msg) for w in list(channels[channel]["listeners"])),
return_exceptions=True,
)
async def safe_send(ws, payload):
try:
await ws.send(payload)
except Exception:
pass
async def cmd_serve(args):
async with serve(handler, HOST, args.port, max_size=None):
print(f"[relay] listening on ws://{HOST}:{args.port}{PATH}")
print(f"[relay] listen: ws://{HOST}:{args.port}{PATH}?token=listen:test")
print(f"[relay] speak : ws://{HOST}:{args.port}{PATH}?token=speak:test")
print("[relay] session roster printed every 15s")
while True:
await asyncio.sleep(15)
print_sessions()
# ---------------------------------------------------------------- tone generator
try:
import numpy as _np
except ImportError:
_np = None
def generate_frame(phase, ts, n, rate, args):
"""Returns (new_phase, pcm16_bytes) for one frame, phase-continuous across frames."""
if _np is not None:
idx = _np.arange(n, dtype=_np.float64)
t = (ts + idx) / rate
if args.speech:
# Speech-shaped: a voice-ish carrier under a syllable envelope with word gaps. The
# sweep and steady tones hold a constant amplitude, so they pin a loudness-driven
# mouth wide open -- this is the signal to test mouth animation against.
freq = 150.0 + 60.0 * _np.sin(2 * _np.pi * 1.7 * t)
syllables = (0.5 + 0.5 * _np.sin(2 * _np.pi * 4.0 * t)) ** 2
words = (_np.sin(2 * _np.pi * 0.35 * t) > -0.25).astype(_np.float64)
amp = 0.45 * syllables * words
elif args.sweep:
# 200-2000 Hz over 4 s: any resampling error shows up as an obvious warble
freq = 200.0 + 1800.0 * (0.5 - 0.5 * _np.cos(2 * _np.pi * t / 4.0))
amp = _np.full(n, 0.35)
else:
freq = _np.full(n, float(args.freq))
amp = _np.full(n, 0.35)
ph = phase + _np.cumsum(2 * _np.pi * freq / rate)
samples = (_np.sin(ph) * amp * 32767).astype("<i2")
return float(ph[-1] % (2 * _np.pi)), samples.tobytes()
# Fallback without numpy. Fine for a steady tone; sweep may not hold real time.
out = []
for i in range(n):
t = (ts + i) / rate
freq = args.freq
amp = 0.35
if args.speech:
freq = 150.0 + 60.0 * math.sin(2 * math.pi * 1.7 * t)
syllables = (0.5 + 0.5 * math.sin(2 * math.pi * 4.0 * t)) ** 2
words = 1.0 if math.sin(2 * math.pi * 0.35 * t) > -0.25 else 0.0
amp = 0.45 * syllables * words
elif args.sweep:
freq = 200.0 + 1800.0 * (0.5 - 0.5 * math.cos(2 * math.pi * t / 4.0))
phase = (phase + 2 * math.pi * freq / rate) % (2 * math.pi)
out.append(int(math.sin(phase) * amp * 32767))
return phase, struct.pack(f"<{n}h", *out)
async def cmd_tone(args):
url = f"ws://{HOST}:{args.port}{PATH}?token=speak:{args.channel}"
rate = args.rate
n = int(rate * FRAME_MS / 1000)
print(f"[tone] {url} {rate} Hz, {n} samples/frame ({FRAME_MS} ms)")
loop = asyncio.get_running_loop()
async with connect(url, max_size=None) as ws:
seq = 0
ts = 0
phase = 0.0
# Deadline-based pacing. A plain `sleep(FRAME_MS)` per iteration sleeps for *at least*
# that long and then adds the generate+send time on top, so the sender silently drifts
# behind real time and starves the receiver's jitter buffer. Track an absolute deadline
# and sleep only for the remainder instead.
next_deadline = loop.time()
late_frames = 0
while True:
# Vectorised: generating 320 samples a frame in a Python loop costs more than the
# 20 ms frame budget once sweep mode adds a second transcendental per sample, and the
# sender then silently falls behind real time and starves the receiver.
phase, payload = generate_frame(phase, ts, n, rate, args)
await ws.send(pack_header(seq, ts, rate) + payload)
seq += 1
ts += n
next_deadline += FRAME_MS / 1000
delay = next_deadline - loop.time()
if delay > 0:
await asyncio.sleep(delay)
else:
# Generation itself overran the frame budget; we can't catch up by sleeping.
late_frames += 1
next_deadline = loop.time()
if seq % 50 == 0:
print(f"\r[tone] sent {seq} frames ({ts / rate:6.1f}s), {late_frames} late", end="", flush=True)
# ---------------------------------------------------------------- listener (verification)
async def cmd_listen(args):
url = f"ws://{HOST}:{args.port}{PATH}?token=listen:{args.channel}"
print(f"[listen] {url}")
async with connect(url, max_size=None) as ws:
count = 0
last_seq = None
gaps = 0
async for msg in ws:
if isinstance(msg, str):
print(f"\n[listen] control: {msg}")
continue
magic, ver, codec, rc, seq, ts = HEADER.unpack_from(msg, 0)
if magic != MAGIC or ver != VERSION:
print(f"\n[listen] BAD HEADER magic={magic:#x} ver={ver}")
continue
if last_seq is not None and seq != last_seq + 1:
gaps += 1
print(f"\n[listen] SEQ GAP {last_seq} -> {seq}")
last_seq = seq
count += 1
if count % 50 == 0:
print(
f"\r[listen] {count} frames, {len(msg) - 12} B payload, codec={codec} "
f"rate={rc} ts={ts} gaps={gaps}",
end="",
flush=True,
)
def main():
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("--port", type=int, default=PORT)
sub = p.add_subparsers(dest="cmd", required=True)
sub.add_parser("serve")
t = sub.add_parser("tone")
t.add_argument("--channel", default="test")
t.add_argument("--freq", type=float, default=1000.0)
t.add_argument("--rate", type=int, default=16000, choices=sorted(RATE_CODES))
t.add_argument("--sweep", action="store_true", help="200-2000 Hz sweep; makes resampling errors obvious")
t.add_argument("--speech", action="store_true", help="speech-shaped envelope; use this to test mouth animation")
l = sub.add_parser("listen")
l.add_argument("--channel", default="test")
args = p.parse_args()
fn = {"serve": cmd_serve, "tone": cmd_tone, "listen": cmd_listen}[args.cmd]
try:
asyncio.run(fn(args))
except KeyboardInterrupt:
print("\nbye")
if __name__ == "__main__":
sys.exit(main())