0.1.16 | BingBong live voice: return audio

Adds the uplink direction: each player sends their own microphone back to
whoever is puppeting BingBong, gated on proximity to him and attenuated by the
same curve his own voice is heard through. Rides the WebSocket that is already
open, so no new connection, auth or TLS surface, and still nothing over Photon.

Capture postfixes MicWrapper.Read rather than MicrophoneRelay, which copies
only the first 256 samples of each frame. Audio is downmixed, low-passed and
resampled to 16kHz in the mod, then passed through an adaptive noise gate: a
fixed threshold is unusable here because a quiet voice measures below a noisy
room, so the gate learns the background level instead.

F8 mutes the outgoing microphone for the session. uplink_denied and
uplink_roster from the relay are now handled, and the stub relay gained the
return direction plus --deny-uplink to exercise the refusal path.

The real relay and web app do not implement this direction yet, so it currently
only works against the stub.
This commit is contained in:
2026-09-06 14:12:45 +02:00
parent f4d723b23b
commit daa675af79
12 changed files with 1052 additions and 12 deletions

3
.gitignore vendored
View File

@@ -226,3 +226,6 @@ artifacts/**/*.zip
*.txt *.txt
artifacts/** artifacts/**
Config.Build.user.props Config.Build.user.props
# Per-stream captures written by tools/stub_relay.py's uplink mode
uplink_*.wav

View File

@@ -106,3 +106,18 @@
- Incoming audio is resampled to the output rate by the mod rather than relying on Unity's mixer, so a device running at 44.1kHz or 48kHz sounds identical. A jitter buffer absorbs network timing, rendering what it has and padding the rest on a shortfall rather than muting and refilling, which pumped audibly when it happened often. - Incoming audio is resampled to the output rate by the mod rather than relying on Unity's mixer, so a device running at 44.1kHz or 48kHz sounds identical. A jitter buffer absorbs network timing, rendering what it has and padding the rest on a shortfall rather than muting and refilling, which pumped audibly when it happened often.
- Both Harmony patches are wrapped so that a failure can only cost the voice feature: the patch body is a try/catch around an isolated method, and repeated failures disable it for the session rather than breaking the item. - Both Harmony patches are wrapped so that a failure can only cost the voice feature: the patch body is a try/catch around an isolated method, and repeated failures disable it for the session rather than breaking the item.
- `tools/stub_relay.py` is a development stub of the relay, enough to test the client end to end before the real backend exists. It speaks the same wire format and generates steady, swept and speech-shaped test signals, so no microphone or browser is needed. - `tools/stub_relay.py` is a development stub of the relay, enough to test the client end to end before the real backend exists. It speaks the same wire format and generates steady, swept and speech-shaped test signals, so no microphone or browser is needed.
# v0.1.16 | BingBong live voice: return audio
- Talking near BingBong now sends your own microphone back to whoever is puppeting him, so it is a conversation rather than a broadcast. Each player only ever transmits their own microphone, over the WebSocket that was already open — no second connection, and still nothing over PEAK's Photon servers.
- The uplink is gated on being within the same range BingBong's voice carries and attenuated by the same curve, so holding him up and talking works and walking away fades you out. It only transmits while someone is actually connected on the other end; an idle lobby sends nothing at all.
- Capture taps `MicWrapper.Read` rather than the obvious `MicrophoneRelay`, which copies only the first 256 samples of each frame before handing them on. Vanilla only ever wants a loudness number from it so the truncation is invisible in game, but streaming it gives a buzz at the frame rate instead of speech.
- The microphone is downmixed, low-passed and resampled from the device's rate to 16kHz by the mod. Decimating 48kHz to 16kHz without filtering first folds everything above 8kHz back into the speech band, which sounds like a second voice made of hiss.
- The noise gate learns the room instead of using a fixed level. A fixed threshold cannot work here: measured on real hardware, a quiet voice sits below what a noisy room measures, so any single number either cuts speech or passes the room. It tracks the background level, opens a configurable factor above it, and holds for 300ms so it does not close between words.
- The gate opens on entering range and closes only past a margin beyond it, with a minimum time in each state, so standing on the boundary cannot chatter it on and off.
- New `UplinkMuteKey` (default: F8) mutes the outgoing microphone for the rest of the session, since reaching for the config file is too slow to be the answer to "stop sending". It is not persisted — a mute you forgot you set is worse than one that clears on the next launch.
- The mod now understands the relay's `uplink_denied` and `uplink_roster` messages: a relay with the uplink switched off is believed and not asked again on that connection, and the log names who else is sending room audio, which is what tells "the relay is dropping my audio" apart from "nobody is listening".
- Uplink frames carry a different magic byte from the downlink ones, and the relay only ever forwards them to the speaker. A mis-routed frame therefore cannot replay a room's own audio back into it, which would be a feedback loop rather than a bug.
- The counters that let the receiver detect a dropped frame advance only for frames actually sent, so a gated pause leaves a contiguous timeline. Start and stop messages tell the receiver to rejoin the live edge instead of filling the pause with silence.
- `tools/stub_relay.py` gained the return direction: it forwards uplink audio to the speaker, stamps the stream id itself so a modified client cannot claim another player's, captures each stream to a WAV, and reports what fraction of real time actually arrived. `--deny-uplink` exercises the refusal path.
- The relay and the web app that would play this back are not built yet, so the return direction currently only works against that stub.

View File

@@ -39,6 +39,13 @@ public static class ConfigHandler
public static ConfigEntry<float> BingBongVoiceMaxDistance { get; private set; } = null!; public static ConfigEntry<float> BingBongVoiceMaxDistance { get; private set; } = null!;
public static ConfigEntry<bool> BingBongVoiceAutoConnect { get; private set; } = null!; public static ConfigEntry<bool> BingBongVoiceAutoConnect { get; private set; } = null!;
public static ConfigEntry<bool> BingBongVoiceAnnounceSpeaker { get; private set; } = null!; public static ConfigEntry<bool> BingBongVoiceAnnounceSpeaker { get; private set; } = null!;
public static ConfigEntry<bool> BingBongVoiceUplinkEnabled { get; private set; } = null!;
public static ConfigEntry<KeyCode> BingBongVoiceUplinkMuteKey { get; private set; } = null!;
public static ConfigEntry<float> BingBongVoiceUplinkGain { get; private set; } = null!;
public static ConfigEntry<float> BingBongVoiceUplinkMinGain { get; private set; } = null!;
public static ConfigEntry<float> BingBongVoiceUplinkGateSensitivity { get; private set; } = null!;
public static ConfigEntry<float> BingBongVoiceUplinkNoiseGateDb { get; private set; } = null!;
public static ConfigEntry<bool> BingBongVoiceUplinkLoopback { get; private set; } = null!;
public static ConfigEntry<KeyCode> BingBongVoiceDebugKey { get; private set; } = null!; public static ConfigEntry<KeyCode> BingBongVoiceDebugKey { get; private set; } = null!;
public static ConfigEntry<bool> BingBongVoiceDebugSource { get; private set; } = null!; public static ConfigEntry<bool> BingBongVoiceDebugSource { get; private set; } = null!;
@@ -167,7 +174,7 @@ public static class ConfigHandler
BingBongVoiceMaxDistance = Config.Bind( BingBongVoiceMaxDistance = Config.Bind(
"BingBong Voice", "BingBong Voice",
"MaxDistance", "MaxDistance",
50f, 150f,
new ConfigDescription("How far the live voice carries, in metres. BingBong's own voice lines only reach 30m, which is short for a conversation.", new AcceptableValueRange<float>(5f, 500f)) new ConfigDescription("How far the live voice carries, in metres. BingBong's own voice lines only reach 30m, which is short for a conversation.", new AcceptableValueRange<float>(5f, 500f))
); );
@@ -185,6 +192,55 @@ public static class ConfigHandler
new ConfigDescription("Show a message when someone starts speaking through BingBong. Being puppeted silently by someone who is not even in your lobby is a bit much without it.") new ConfigDescription("Show a message when someone starts speaking through BingBong. Being puppeted silently by someone who is not even in your lobby is a bit much without it.")
); );
BingBongVoiceUplinkEnabled = Config.Bind(
"BingBong Voice",
"UplinkEnabled",
true,
new ConfigDescription("Send your own microphone to whoever is puppeting BingBong, when you are near him and only while someone is actually connected. Turning this off means no microphone patch is applied at all.")
);
BingBongVoiceUplinkMuteKey = Config.Bind(
"BingBong Voice",
"UplinkMuteKey",
KeyCode.F8,
new ConfigDescription("Mutes and unmutes your outgoing microphone for the rest of the session. Faster than the config when you need it to stop now.")
);
BingBongVoiceUplinkGain = Config.Bind(
"BingBong Voice",
"UplinkGain",
3.0f,
new ConfigDescription("Trim on your outgoing mic. The tap sits upstream of Photon's automatic gain control, so the raw signal is quieter than what teammates hear and usually needs lifting.", new AcceptableValueRange<float>(0f, 8f))
);
BingBongVoiceUplinkMinGain = Config.Bind(
"BingBong Voice",
"UplinkMinGain",
0.02f,
new ConfigDescription("Stop transmitting once distance has attenuated you below this. Keeps the edge of the range quiet rather than faintly open.", new AcceptableValueRange<float>(0f, 0.5f))
);
BingBongVoiceUplinkGateSensitivity = Config.Bind(
"BingBong Voice",
"UplinkGateSensitivity",
3.0f,
new ConfigDescription("How far above the measured background noise your voice must be before it is transmitted. The gate learns your room rather than using a fixed level, because a quiet voice can measure lower than a noisy room. Raise it if background noise gets through, lower it if quiet speech is cut off.", new AcceptableValueRange<float>(1.5f, 10f))
);
BingBongVoiceUplinkNoiseGateDb = Config.Bind(
"BingBong Voice",
"UplinkFloorDb",
-60f,
new ConfigDescription("Absolute floor: nothing quieter than this is ever transmitted, whatever the adaptive gate decides. A safety net, not the main control.", new AcceptableValueRange<float>(-80f, -20f))
);
BingBongVoiceUplinkLoopback = Config.Bind(
"BingBong Voice",
"UplinkLoopback",
false,
new ConfigDescription("Debug: play your own processed microphone back out of BingBong instead of sending it anywhere. Lets the capture path be heard with no relay and no second player. Deliberately closes a feedback loop, so use headphones.")
);
BingBongVoiceDebugKey = Config.Bind( BingBongVoiceDebugKey = Config.Bind(
"BingBong Voice", "BingBong Voice",
"DebugToggleKey", "DebugToggleKey",

View File

@@ -8,7 +8,7 @@
<!-- This is the display name of your mod. Example: BepInEx Template --> <!-- This is the display name of your mod. Example: BepInEx Template -->
<AssemblyTitle>JordanMod</AssemblyTitle> <AssemblyTitle>JordanMod</AssemblyTitle>
<!-- This is the version number of your mod. --> <!-- This is the version number of your mod. -->
<Version>0.1.15</Version> <Version>0.1.16</Version>
</PropertyGroup> </PropertyGroup>
<PropertyGroup> <PropertyGroup>

View File

@@ -42,9 +42,11 @@ public sealed class BingBongVoiceClient
private Thread? _reconnectThread; private Thread? _reconnectThread;
private volatile bool _shouldRun; private volatile bool _shouldRun;
private volatile bool _speakerActive; private volatile bool _speakerActive;
private volatile bool _uplinkDenied;
private long _expectedTimestamp = -1; private long _expectedTimestamp = -1;
private int _badFrames; private int _badFrames;
private volatile int _lastCloseCode; private volatile int _lastCloseCode;
private int _sendFailures;
/// The live stream, or null before the first frame arrives. Read from the audio thread. /// The live stream, or null before the first frame arrives. Read from the audio thread.
public VoiceStream? Stream => Volatile.Read(ref _stream); public VoiceStream? Stream => Volatile.Read(ref _stream);
@@ -52,6 +54,11 @@ public sealed class BingBongVoiceClient
/// True between speaker_joined and speaker_left control frames. /// True between speaker_joined and speaker_left control frames.
public bool SpeakerActive => _speakerActive; public bool SpeakerActive => _speakerActive;
/// Set when the relay says it will not take our microphone -- uplink switched off on its side,
/// or too many senders on this channel. Sticky until the next connect, because the answer will
/// not change on this socket and retrying for the rest of the session would just burn bandwidth.
public bool UplinkDenied => _uplinkDenied;
/// Who is speaking, when the relay tells us. Null if nobody is, or if it did not say. /// Who is speaking, when the relay tells us. Null if nobody is, or if it did not say.
public string? SpeakerName { get; private set; } public string? SpeakerName { get; private set; }
@@ -161,6 +168,7 @@ public sealed class BingBongVoiceClient
{ {
_expectedTimestamp = -1; _expectedTimestamp = -1;
_badFrames = 0; _badFrames = 0;
_uplinkDenied = false;
Debug.Log("[BingBongVoice] Connected to relay."); Debug.Log("[BingBongVoice] Connected to relay.");
// Register this lobby member so the relay's session directory knows who is listening. // Register this lobby member so the relay's session directory knows who is listening.
@@ -246,6 +254,36 @@ public sealed class BingBongVoiceClient
} }
} }
/// <summary>
/// Sends one frame on the socket we already hold. Returns false rather than throwing when the
/// socket is not open, so the uplink can count failures without ever driving reconnection --
/// that belongs to ConnectLoop, and a second driver would fight it.
/// </summary>
public bool TrySend(byte[] frame)
{
WebSocket? socket = _socket;
if (socket == null || socket.ReadyState != WebSocketState.Open) return false;
try
{
socket.Send(frame);
return true;
}
catch (Exception e)
{
if (++_sendFailures <= 3) Debug.LogWarning($"[BingBongVoice] Uplink send failed: {e.Message}");
return false;
}
}
/// Control frames for the uplink. Text, like every other control message.
public bool TrySendControl(string json)
{
WebSocket? socket = _socket;
if (socket == null || socket.ReadyState != WebSocketState.Open) return false;
try { socket.Send(json); return true; }
catch { return false; }
}
private VoiceStream EnsureStream(int rate) private VoiceStream EnsureStream(int rate)
{ {
VoiceStream? current = Volatile.Read(ref _stream); VoiceStream? current = Volatile.Read(ref _stream);
@@ -263,9 +301,11 @@ public sealed class BingBongVoiceClient
// Parsed leniently and fielded by substring if that fails: control messages are allowed to // Parsed leniently and fielded by substring if that fails: control messages are allowed to
// gain fields, and a malformed payload must never take down the receive thread. // gain fields, and a malformed payload must never take down the receive thread.
string? speaker = null; string? speaker = null;
JObject? parsed = null;
try try
{ {
speaker = JObject.Parse(json)["speaker"]?.ToString(); parsed = JObject.Parse(json);
speaker = parsed["speaker"]?.ToString();
} }
catch catch
{ {
@@ -287,6 +327,22 @@ public sealed class BingBongVoiceClient
Debug.Log("[BingBongVoice] The speaker left."); Debug.Log("[BingBongVoice] The speaker left.");
Plugin.RunOnMainThread(() => OnSpeakerChanged?.Invoke(null)); Plugin.RunOnMainThread(() => OnSpeakerChanged?.Invoke(null));
} }
else if (json.Contains("uplink_denied"))
{
// The relay has told us it is not taking microphones. Nothing about that changes while
// this socket is up, so stop asking rather than sending 50 frames a second into a drop.
_uplinkDenied = true;
Debug.LogWarning($"[BingBongVoice] The relay will not take our microphone ({parsed?["reason"]?.ToString() ?? json}). Not sending until the next connect.");
}
else if (json.Contains("uplink_roster"))
{
// Purely informational, and the one thing that distinguishes "the relay is dropping my
// audio" from "nobody is listening" when the uplink looks like it is working locally.
string players = parsed?["players"] is JArray list && list.Count > 0
? string.Join(", ", list)
: "nobody";
Debug.Log($"[BingBongVoice] Sending room audio: {players}.");
}
else if (json.Contains("rejected")) else if (json.Contains("rejected"))
{ {
Debug.LogWarning($"[BingBongVoice] Relay rejected this listener: {json}"); Debug.LogWarning($"[BingBongVoice] Relay rejected this listener: {json}");

View File

@@ -4,6 +4,7 @@ using System.Text;
using JordanMod.Modules.BetterBugle; using JordanMod.Modules.BetterBugle;
using Newtonsoft.Json; using Newtonsoft.Json;
using Photon.Pun; using Photon.Pun;
using Photon.Voice.Unity;
using UnityEngine; using UnityEngine;
namespace JordanMod.Modules.BingBongVoice; namespace JordanMod.Modules.BingBongVoice;
@@ -19,6 +20,17 @@ class BingBongVoiceModule : Module
private GameObject? _debugObject; private GameObject? _debugObject;
/// Captures the local microphone. Null when the uplink is off or we are not in a lobby, which
/// is what makes the mic tap a no-op rather than something that needs its own guard.
public MicUplinkSource? Uplink { get; private set; }
/// Stage 1 debug: the captured mic played back through BingBong instead of being sent. Lets
/// the whole capture path be heard with no relay and no second player.
public VoiceStream? LoopbackStream { get; private set; }
private MicUplinkSender? _sender;
private bool _uplinkTransmitting;
private const string ChannelSalt = "jordanmod-bingbong-voice-v1"; private const string ChannelSalt = "jordanmod-bingbong-voice-v1";
public override Type[] GetPatches() public override Type[] GetPatches()
@@ -27,7 +39,12 @@ class BingBongVoiceModule : Module
// SetupModules -> SetupPatches), so turning the feature off here means the Harmony // SetupModules -> SetupPatches), so turning the feature off here means the Harmony
// patches are never applied at all and the mod has zero footprint. // patches are never applied at all and the mod has zero footprint.
if (!ConfigHandler.BingBongVoiceEnabled.Value) return []; if (!ConfigHandler.BingBongVoiceEnabled.Value) return [];
return [typeof(BingBongVoicePatch)];
// The mic tap sits on Photon's per-frame audio path, so an opted-out player should carry
// no patch at all rather than one that early-returns.
return ConfigHandler.BingBongVoiceUplinkEnabled.Value
? [typeof(BingBongVoicePatch), typeof(BingBongMicTapPatch)]
: [typeof(BingBongVoicePatch)];
} }
public override void Initialize() public override void Initialize()
@@ -35,6 +52,9 @@ class BingBongVoiceModule : Module
if (Instance != null) return; if (Instance != null) return;
Instance = this; Instance = this;
// Recorded here so the mic callback's thread can be compared against it.
BingBongMicTapPatch.NoteMainThread();
// The relay lives on the audio bank, so with no override configured every player derives // The relay lives on the audio bank, so with no override configured every player derives
// the same URL and connects on their own -- nobody has to press anything. // the same URL and connects on their own -- nobody has to press anything.
PhotonNetworkEventListener listener = PhotonNetworkEventListener.Instance!; PhotonNetworkEventListener listener = PhotonNetworkEventListener.Instance!;
@@ -63,6 +83,107 @@ class BingBongVoiceModule : Module
{ {
if (!ConfigHandler.BingBongVoiceEnabled.Value) return; if (!ConfigHandler.BingBongVoiceEnabled.Value) return;
if (Input.GetKeyDown(ConfigHandler.BingBongVoiceDebugKey.Value)) ToggleStream(); if (Input.GetKeyDown(ConfigHandler.BingBongVoiceDebugKey.Value)) ToggleStream();
if (Input.GetKeyDown(ConfigHandler.BingBongVoiceUplinkMuteKey.Value)) ToggleUplinkMute();
EnsureUplink();
UpdateUplinkProximity();
}
// Hysteresis on the gate only. The gain is continuous so it never stutters; what stutters is
// the open/closed decision, and each cycle costs the receiver a buffer reset.
private bool _uplinkInRange;
private float _uplinkStateChangedAt;
private float _lastUplinkReport;
private bool _uplinkMuted;
/// Session-scoped mute for the outgoing microphone, on a key because reaching for the config is
/// too slow to be the answer to "stop sending, now". Deliberately not persisted: a mute you
/// forgot you set is worse than one that clears when you next launch.
private void ToggleUplinkMute()
{
_uplinkMuted = !_uplinkMuted;
Debug.Log($"[BingBongVoice] Uplink microphone {(_uplinkMuted ? "MUTED" : "unmuted")}.");
BetterBugleUI.Instance?.ShowActionbar(_uplinkMuted
? "🎙 BingBong microphone muted"
: "🎙 BingBong microphone live");
}
/// <summary>
/// Decides how loudly, if at all, the local microphone should be transmitted: nearest BingBong,
/// attenuated by the same curve his own voice is heard through.
/// </summary>
private void UpdateUplinkProximity()
{
MicUplinkSource? uplink = Uplink;
if (uplink == null) return;
Character? local = Character.localCharacter;
if (local == null)
{
uplink.SetGate(false);
return;
}
float range = ConfigHandler.BingBongVoiceMaxDistance.Value;
float nearest = float.MaxValue;
foreach (BingBongVoiceSource source in BingBongVoiceSource.Active)
{
if (source == null || !source.IsPositional) continue;
float d = Vector3.Distance(local.Head, source.transform.position);
if (d < nearest) nearest = d;
}
// Open on entering the range, close only past a margin beyond it, so walking the boundary
// cannot chatter the gate. The dwell is a minimum time in the CURRENT state before it may
// flip again -- not a check against the change just made, which would undo it every frame.
bool want = _uplinkInRange ? nearest <= range * 1.15f : nearest < range;
if (want != _uplinkInRange)
{
float minDwell = _uplinkInRange ? 1.5f : 0.5f;
if (Time.time - _uplinkStateChangedAt >= minDwell)
{
_uplinkInRange = want;
_uplinkStateChangedAt = Time.time;
}
}
float gain = 0f;
if (_uplinkInRange && nearest < float.MaxValue)
{
// The curve, then a taper over the last fifth so the gate closes on real silence
// instead of a cliff. Unity's own rolloff stops attenuating past maxDistance, which
// copied literally would leave everyone in the level faintly transmitting forever.
gain = BingBongVoiceSource.RangeGain(nearest, range)
* Mathf.SmoothStep(1f, 0f, Mathf.InverseLerp(range * 0.8f, range, nearest))
* ConfigHandler.BingBongVoiceUplinkGain.Value;
}
bool audible = gain > ConfigHandler.BingBongVoiceUplinkMinGain.Value;
// Only while someone is actually puppeting BingBong. An idle lobby sends nothing, and
// "my mic goes out only while someone is on the other end" is a rule a player can hold
// in their head. Loopback ignores it, having no speaker to wait for.
bool speakerPresent = ConfigHandler.BingBongVoiceUplinkLoopback.Value || (Client?.SpeakerActive ?? false);
bool allowed = !_uplinkMuted && !(Client?.UplinkDenied ?? false);
bool transmit = _uplinkInRange && audible && speakerPresent && allowed;
uplink.SetTargetGain(gain);
uplink.SetGate(transmit);
// The receiver's timeline freezes with ours while gated, so it needs telling to rejoin the
// live edge rather than treating the pause as a gap to fill with silence.
if (transmit != _uplinkTransmitting && !ConfigHandler.BingBongVoiceUplinkLoopback.Value)
{
_uplinkTransmitting = transmit;
Client?.TrySendControl(transmit ? "{\"type\":\"uplink_start\"}" : "{\"type\":\"uplink_stop\"}");
Debug.Log($"[BingBongVoice] Uplink {(transmit ? "started" : "stopped")} sending.");
}
if (Time.time - _lastUplinkReport < 5f) return;
_lastUplinkReport = Time.time;
Debug.Log($"[BingBongVoice] Uplink: nearest BingBong {(nearest < float.MaxValue ? nearest.ToString("F1") + "m" : "none")}, " +
$"gain {gain:F2}, gate {(_uplinkInRange && audible ? "open" : "closed")}{(_uplinkMuted ? ", MUTED" : "")}, " +
$"level {uplink.Level:F4} (floor {uplink.NoiseFloor:F5}), {uplink.FramesProduced} captured" +
$"{(_sender != null ? $", {_sender.Sent} sent, {_sender.Dropped} dropped" : "")}. {DescribeMic(local)}");
} }
/// <summary> /// <summary>
@@ -129,7 +250,12 @@ class BingBongVoiceModule : Module
{ {
string nickname = PhotonNetwork.LocalPlayer?.NickName ?? ""; string nickname = PhotonNetwork.LocalPlayer?.NickName ?? "";
if (string.IsNullOrWhiteSpace(nickname)) nickname = "Unknown"; if (string.IsNullOrWhiteSpace(nickname)) nickname = "Unknown";
return JsonConvert.SerializeObject(new { type = "hello", player = nickname });
// Whether we might ever send a microphone, so the relay can allocate a stream id and get the
// roster right before the first audio frame rather than a beat after it.
bool uplink = ConfigHandler.BingBongVoiceUplinkEnabled.Value
&& !ConfigHandler.BingBongVoiceUplinkLoopback.Value;
return JsonConvert.SerializeObject(new { type = "hello", player = nickname, uplink });
} }
/// <summary> /// <summary>
@@ -218,7 +344,6 @@ class BingBongVoiceModule : Module
Debug.Log($"[BingBongVoice] Connecting for lobby '{PhotonNetwork.CurrentRoom?.Name}' to {url}"); Debug.Log($"[BingBongVoice] Connecting for lobby '{PhotonNetwork.CurrentRoom?.Name}' to {url}");
Client = new BingBongVoiceClient(); Client = new BingBongVoiceClient();
Client.OnSpeakerChanged += AnnounceSpeaker;
Client.Connect(url, BuildHello()); Client.Connect(url, BuildHello());
if (!ConfigHandler.BingBongVoiceDebugSource.Value) return; if (!ConfigHandler.BingBongVoiceDebugSource.Value) return;
@@ -237,10 +362,99 @@ class BingBongVoiceModule : Module
: "BingBong is himself again"); : "BingBong is himself again");
} }
/// <summary>
/// Photon's own view of the microphone. When our captured level is zero the question is always
/// whether the game is feeding the device at all -- push-to-talk released, no device selected,
/// or the recorder stopped -- rather than anything in the capture path.
/// </summary>
private static string DescribeMic(Character local)
{
try
{
CharacterVoiceHandler? voice = local.refs?.voice;
if (voice == null) return "(no voice handler)";
Recorder? recorder = voice.GetComponent<Recorder>();
if (recorder == null) return "(no recorder)";
string device = recorder.MicrophoneDevice != null ? recorder.MicrophoneDevice.ToString() : "none";
return $"mic: transmitting={recorder.IsCurrentlyTransmitting}, enabled={recorder.TransmitEnabled}, device={device}";
}
catch (Exception e)
{
return $"(mic state unreadable: {e.GetType().Name})";
}
}
/// <summary>
/// Keeps the capture path in step with the config, every frame. Deliberately independent of the
/// relay connection: loopback has to work with no network at all, and tying capture to a
/// successful connect meant an unreachable relay silently disabled it. Also picks up a config
/// change live, rather than only at the moment a lobby is joined.
/// </summary>
private void EnsureUplink()
{
if (!ConfigHandler.BingBongVoiceUplinkEnabled.Value)
{
if (Uplink != null) StopUplink();
return;
}
bool wantLoopback = ConfigHandler.BingBongVoiceUplinkLoopback.Value;
bool haveLoopback = LoopbackStream != null;
bool senderMatchesClient = wantLoopback || (_sender != null) == (Client != null);
if (Uplink != null && wantLoopback == haveLoopback && senderMatchesClient) return;
StopUplink();
StartUplink();
}
private void StartUplink()
{
if (!ConfigHandler.BingBongVoiceUplinkEnabled.Value) return;
if (Uplink != null) return;
MicUplinkSource source = new();
if (ConfigHandler.BingBongVoiceUplinkLoopback.Value)
{
LoopbackStream = new VoiceStream(MicUplinkSource.WireRate);
source.EmitSilenceWhenGated = true; // keep playback continuous; see the field's note
source.OnFrame = frame => LoopbackStream.Ring.WritePcm16Samples(frame, frame.Length);
Debug.LogWarning("[BingBongVoice] Uplink LOOPBACK is on: your microphone plays back out of BingBong and is not sent anywhere. Wear headphones.");
}
else if (Client != null)
{
MicUplinkSender sender = new(Client);
sender.Start();
source.OnFrame = sender.Enqueue;
_sender = sender;
}
Uplink = source;
Debug.Log($"[BingBongVoice] Uplink capture started{(ConfigHandler.BingBongVoiceUplinkLoopback.Value ? " in LOOPBACK mode" : "")}.");
}
private void StopUplink()
{
if (_uplinkTransmitting)
{
Client?.TrySendControl("{\"type\":\"uplink_stop\"}");
_uplinkTransmitting = false;
}
if (Uplink != null) Uplink.OnFrame = null;
Uplink = null;
LoopbackStream = null;
_sender?.Stop();
_sender = null;
}
private void Disconnect() private void Disconnect()
{ {
StopUplink();
if (Client == null) return; if (Client == null) return;
Client.OnSpeakerChanged -= AnnounceSpeaker;
Debug.Log("[BingBongVoice] Disconnecting from relay."); Debug.Log("[BingBongVoice] Disconnecting from relay.");
Client.Disconnect(); Client.Disconnect();
Client = null; Client = null;

View File

@@ -44,6 +44,7 @@ public sealed class BingBongVoiceSource : MonoBehaviour
private static bool _reportedBlock; // once per session, not once per BingBong private static bool _reportedBlock; // once per session, not once per BingBong
private static int _liveSources; // how many voice sources currently exist private static int _liveSources; // how many voice sources currently exist
private static int _playingSources; // how many are actually enabled and audible private static int _playingSources; // how many are actually enabled and audible
private static readonly System.Collections.Generic.List<BingBongVoiceSource> _active = new();
private float _gainStep = 0.0008f; private float _gainStep = 0.0008f;
private long _lastWritePos; private long _lastWritePos;
private float _lastRateTime; private float _lastRateTime;
@@ -58,6 +59,25 @@ public sealed class BingBongVoiceSource : MonoBehaviour
/// Set by the duck patch. Live voice is attenuated until this time. /// Set by the duck patch. Live voice is attenuated until this time.
public float DuckUntil { get; set; } public float DuckUntil { get; set; }
/// False for the non-positional debug instance, which sits on a DontDestroyOnLoad object at the
/// origin and would otherwise look like a BingBong standing next to you.
public bool IsPositional { get; private set; }
/// Enabled, positional sources. Main thread only.
public static System.Collections.Generic.IReadOnlyList<BingBongVoiceSource> Active => _active;
public static float MinDistanceFor(float range) => Mathf.Max(1f, range / 10f);
/// <summary>
/// Unity's Logarithmic rolloff expressed arithmetically, so the uplink can attenuate a player's
/// microphone by exactly the curve BingBong's own output is heard through.
/// </summary>
public static float RangeGain(float distance, float range)
{
float min = MinDistanceFor(range);
return Mathf.Clamp01(min / Mathf.Max(min, distance));
}
public float Amplitude => _amplitude; public float Amplitude => _amplitude;
private const string ChildName = "BingBongVoice"; private const string ChildName = "BingBongVoice";
@@ -150,6 +170,7 @@ public sealed class BingBongVoiceSource : MonoBehaviour
{ {
_source.spatialBlend = 0f; // debug instance: non-positional, always audible _source.spatialBlend = 0f; // debug instance: non-positional, always audible
} }
IsPositional = template != null;
_source.Play(); _source.Play();
_liveSources++; _liveSources++;
@@ -182,7 +203,8 @@ public sealed class BingBongVoiceSource : MonoBehaviour
private void Update() private void Update()
{ {
_stream = BingBongVoiceModule.Instance?.Client?.Stream; BingBongVoiceModule? module = BingBongVoiceModule.Instance;
_stream = module?.LoopbackStream ?? module?.Client?.Stream;
_targetGain = Time.time < DuckUntil _targetGain = Time.time < DuckUntil
? ConfigHandler.BingBongVoiceDuckVolume.Value ? ConfigHandler.BingBongVoiceDuckVolume.Value
@@ -420,6 +442,7 @@ public sealed class BingBongVoiceSource : MonoBehaviour
// The audio thread can and will run the filter after Destroy happens on the main thread, // The audio thread can and will run the filter after Destroy happens on the main thread,
// so flag it dead first: that check is what makes the teardown safe. // so flag it dead first: that check is what makes the teardown safe.
_dead = true; _dead = true;
_active.Remove(this);
_liveSources--; _liveSources--;
AudioSettings.OnAudioConfigurationChanged -= OnAudioConfigChanged; AudioSettings.OnAudioConfigurationChanged -= OnAudioConfigChanged;
@@ -433,6 +456,7 @@ public sealed class BingBongVoiceSource : MonoBehaviour
private void OnDisable() private void OnDisable()
{ {
_active.Remove(this);
_playingSources--; _playingSources--;
if (_source != null) _source.Stop(); if (_source != null) _source.Stop();
_state = State.Buffering; _state = State.Buffering;
@@ -440,6 +464,7 @@ public sealed class BingBongVoiceSource : MonoBehaviour
private void OnEnable() private void OnEnable()
{ {
if (!_active.Contains(this)) _active.Add(this);
_playingSources++; _playingSources++;
// Rejoin live rather than trying to resume where we left off. // Rejoin live rather than trying to resume where we left off.
_state = State.Buffering; _state = State.Buffering;

View File

@@ -0,0 +1,151 @@
using System;
using System.Threading;
namespace JordanMod.Modules.BingBongVoice;
/// <summary>
/// Frames captured microphone audio and puts it on the socket the client already holds.
///
/// Sends from a dedicated thread with the blocking <c>Send</c> rather than <c>SendAsync</c>:
/// websocket-sharp's async path posts to the thread pool, so ordering across queued sends is not
/// guaranteed, and reordered voice sounds far worse than dropped voice. One thread and a blocking
/// send gives strict ordering for free.
///
/// There is deliberately no pacer. The microphone is the clock -- Photon hands us exactly the
/// audio the hardware produced -- so anything that paced against a wall clock on top of that would
/// only fight it and guarantee drift in one direction.
/// </summary>
public sealed class MicUplinkSender
{
private const byte Magic = 0xB2; // 0xB1 is downlink; a distinct magic makes a relay
private const byte Version = 1; // mis-route unable to replay room audio into the room
private const byte CodecPcm16 = 0;
private const byte RateCode16k = 0;
private const int HeaderSize = 12;
private const int PayloadBytes = MicUplinkSource.FrameSamples * 2;
private const int FrameBytes = HeaderSize + PayloadBytes;
private const int RingSlots = 32; // ~640 ms; far more than a healthy socket ever needs
private readonly byte[][] _slots = new byte[RingSlots][];
private readonly BingBongVoiceClient _client;
private readonly SemaphoreSlim _signal = new(0);
private long _write;
private long _read;
private uint _seq;
private uint _timestamp;
private Thread? _thread;
private volatile bool _running;
private int _dropped;
private int _sent;
private int _consecutiveFailures;
private volatile bool _stopped;
public int Dropped => _dropped;
public int Sent => _sent;
public MicUplinkSender(BingBongVoiceClient client)
{
_client = client;
for (int i = 0; i < RingSlots; i++) _slots[i] = new byte[FrameBytes];
}
public void Start()
{
if (_running) return;
_running = true;
_thread = new Thread(SendLoop) { IsBackground = true, Name = "BingBongUplink" };
_thread.Start();
}
public void Stop()
{
_running = false;
_signal.Release();
}
/// Producer side: called from the mic path. Writes the frame into a pre-allocated slot, so
/// nothing is allocated per frame and the caller never blocks.
public void Enqueue(short[] samples)
{
if (_stopped || !_running) return;
// Ring full means the socket is not draining. Drop the OLDEST and keep the newest: on a
// live stream, stale audio is worth less than current audio, and blocking the producer
// would stall Photon's audio path.
if (_write - _read >= RingSlots)
{
Interlocked.Increment(ref _dropped);
Interlocked.Increment(ref _read);
}
byte[] slot = _slots[(int)(_write % RingSlots)];
slot[0] = Magic;
slot[1] = Version;
slot[2] = CodecPcm16;
slot[3] = RateCode16k; // the relay overwrites the high nibble with our stream id
WriteUInt32(slot, 4, _seq);
WriteUInt32(slot, 8, _timestamp);
for (int i = 0; i < MicUplinkSource.FrameSamples; i++)
{
short s = samples[i];
int b = HeaderSize + (i << 1);
slot[b] = (byte)(s & 0xFF);
slot[b + 1] = (byte)((s >> 8) & 0xFF);
}
// Both counters advance only for frames we actually emit, so a gated pause leaves the
// timeline contiguous and the receiver must not insert silence for it. The control frames
// are what tell it to resync instead.
_seq++;
_timestamp += MicUplinkSource.FrameSamples;
Interlocked.Increment(ref _write);
_signal.Release();
}
private void SendLoop()
{
while (_running)
{
try
{
_signal.Wait(200);
while (_read < Volatile.Read(ref _write))
{
byte[] slot = _slots[(int)(_read % RingSlots)];
if (_client.TrySend(slot))
{
Interlocked.Increment(ref _sent);
_consecutiveFailures = 0;
}
else if (++_consecutiveFailures >= 200)
{
// The socket has been refusing for four seconds. Stop rather than spin;
// reconnection is the client's job and a second driver would fight it.
_stopped = true;
Debug.LogWarning("[BingBongVoice] Uplink stopped: the socket has been refusing frames. It will resume on the next connect.");
return;
}
Interlocked.Increment(ref _read);
}
}
catch (Exception e)
{
Debug.LogError($"[BingBongVoice] Uplink send loop failed: {e}");
return;
}
}
}
private static void WriteUInt32(byte[] buffer, int offset, uint value)
{
buffer[offset] = (byte)(value & 0xFF);
buffer[offset + 1] = (byte)((value >> 8) & 0xFF);
buffer[offset + 2] = (byte)((value >> 16) & 0xFF);
buffer[offset + 3] = (byte)((value >> 24) & 0xFF);
}
}

View File

@@ -0,0 +1,228 @@
using System;
using System.Threading;
using UnityEngine;
namespace JordanMod.Modules.BingBongVoice;
/// <summary>
/// Turns the local microphone into 20 ms frames of 16 kHz mono PCM16, ready to go on the wire.
///
/// Fed by <see cref="BingBongMicTapPatch"/> from Photon's mic read, measured at 48000 Hz mono in
/// 960-sample frames arriving 50 times a second on Unity's main thread -- so the microphone itself
/// is the clock and there is deliberately no pacer here. A deadline pacer on top of a hardware
/// clock would only fight it and guarantee drift in one direction.
///
/// Allocation-free after the first callback: this runs on the audio path of every frame.
/// </summary>
public sealed class MicUplinkSource
{
public const int WireRate = 16000;
public const int FrameSamples = 320; // 20 ms at 16 kHz
private readonly short[] _frame = new short[FrameSamples];
private int _frameFill;
// Scratch copy of the incoming buffer. We must never filter in place: that array belongs to
// Photon and is encoded and sent to teammates immediately after this postfix returns, so
// writing to it would corrupt everyone else's audio.
private float[]? _work;
// Resampler state, carried across callbacks. _tail is the previous buffer's last sample and is
// addressed as index -1, so the interpolation spans the seam instead of skipping it.
private double _read;
private float _tail;
// Five cascaded one-pole low-passes. Decimating 48k to 16k without one folds everything above
// 8 kHz down into the speech band, which sounds harsh and cannot be undone afterwards. Two
// poles measured only -14 dB of rejection at 12 kHz; five at 7 kHz gives -25 dB while still
// passing 95% at 1 kHz, where speech energy actually is.
private const int FilterPoles = 5;
private readonly float[] _lp = new float[FilterPoles];
private float _lpCoefficient = 1f;
private int _rate;
private int _channels = 1;
private int _sourceId = -1;
private float _gain;
private readonly float _gainStep = 1f / (WireRate * 0.015f); // ~15 ms attack
private volatile float _targetGain;
private volatile bool _gateOpen;
private volatile float _lastFrameLevel;
private volatile float _noiseFloorPublished;
private long _framesProduced;
private float _noiseFloor;
private int _holdFrames;
/// Set by the proximity logic on the main thread. 0 mutes; the ramp keeps it click-free.
public void SetTargetGain(float gain) => _targetGain = Mathf.Clamp(gain, 0f, 8f);
/// Whether the uplink may transmit at all (range, push-to-talk, speaker present).
public void SetGate(bool open) => _gateOpen = open;
/// Mean |sample| of the last completed frame, for the level indicator.
public float Level => _lastFrameLevel;
public long FramesProduced => Interlocked.Read(ref _framesProduced);
/// Raised for each completed 20 ms frame that passes the gate. Assigned once, never per frame.
public Action<short[]>? OnFrame;
/// <summary>
/// Emit gated frames as silence rather than dropping them, keeping the output continuous.
///
/// Off for the real uplink: the whole point of the gate is not to spend bandwidth on silence,
/// and the receiver is told to freeze its timeline instead. On for loopback, where the frames
/// feed a jitter-buffered player with no control channel -- dropping 70% of them starves it
/// into permanent rebuffering, which sounds like the capture path is broken when it is fine.
/// </summary>
public bool EmitSilenceWhenGated;
private readonly short[] _silence = new short[FrameSamples];
public void OnMicFrame(float[] buffer, int rate, int channels, int sourceId)
{
if (buffer == null || buffer.Length == 0 || rate <= 0) return;
// The mic device can change mid-session (CharacterVoiceHandler reassigns the recorder's
// device when the setting changes), rebuilding MicWrapper with a possibly different rate.
// Starting the resampler over is cheaper and safer than trying to splice.
if (rate != _rate || channels != _channels || sourceId != _sourceId)
{
Reset(rate, channels, sourceId);
Debug.Log($"[BingBongVoice] Uplink capture at {rate}Hz x{channels}, resampling {(double)rate / WireRate:F2}:1 down to {WireRate}Hz.");
}
int count = _channels > 1 ? buffer.Length / _channels : buffer.Length;
if (count < 2) return;
if (_work == null || _work.Length < count) _work = new float[count];
// Downmix and anti-alias in one pass into our own buffer. The filter has to see every
// input sample, not just the ones the cursor lands on, or it is not an anti-alias filter.
bool filter = _lpCoefficient < 1f;
for (int i = 0; i < count; i++)
{
float x;
if (_channels > 1)
{
float sum = 0f;
int b = i * _channels;
for (int c = 0; c < _channels; c++) sum += buffer[b + c];
x = sum / _channels;
}
else x = buffer[i];
if (filter)
{
for (int p = 0; p < FilterPoles; p++)
{
_lp[p] += (x - _lp[p]) * _lpCoefficient;
x = _lp[p];
}
}
_work[i] = x;
}
Resample(_work, count);
}
private void Reset(int rate, int channels, int sourceId)
{
_rate = rate;
_channels = Math.Max(1, channels);
_sourceId = sourceId;
_read = 0;
_tail = 0f;
Array.Clear(_lp, 0, _lp.Length);
_frameFill = 0;
_gain = 0f;
_noiseFloor = 0f;
_holdFrames = 0;
// One-pole cutoff at ~6.5 kHz, safely under the 8 kHz Nyquist of the 16 kHz wire rate.
_lpCoefficient = rate > WireRate ? 1f - Mathf.Exp(-2f * Mathf.PI * 7000f / rate) : 1f;
}
private void Resample(float[] work, int count)
{
double ratio = (double)_rate / WireRate;
double pos = _read; // may be negative; -1 addresses the carried tail sample
while (true)
{
int i0 = (int)Math.Floor(pos);
if (i0 + 1 >= count) break;
float a = i0 < 0 ? _tail : work[i0];
float b = work[i0 + 1];
float frac = (float)(pos - i0);
_gain += (_targetGain - _gain) * _gainStep;
Push((a * (1f - frac) + b * frac) * _gain);
pos += ratio;
}
_tail = work[count - 1];
_read = pos - count; // carries the fractional remainder to the next buffer
if (_read < -1) _read = -1;
}
/// <summary>
/// Adaptive gate. A fixed threshold cannot work here: measured on real hardware, a silent but
/// noisy room read louder (mean 0.0088) than the same person actually speaking in a quiet one
/// (0.0018), so no single level separates speech from silence across setups. Instead track the
/// background and require speech to stand above it.
///
/// The floor follows quiet quickly and loud very slowly, so it settles on the room rather than
/// on the voice. A hold keeps the gate open through the gaps between words.
/// </summary>
private bool PassesNoiseGate(float level)
{
if (_noiseFloor <= 0f) _noiseFloor = level;
else if (level < _noiseFloor) _noiseFloor += (level - _noiseFloor) * 0.25f; // drop fast
else _noiseFloor += (level - _noiseFloor) * 0.0008f; // rise slowly
_noiseFloorPublished = _noiseFloor;
// Never transmit below the absolute floor, whatever the adaptive part decides.
if (level < Mathf.Pow(10f, ConfigHandler.BingBongVoiceUplinkNoiseGateDb.Value / 20f)) return false;
float open = _noiseFloor * ConfigHandler.BingBongVoiceUplinkGateSensitivity.Value;
float close = open * 0.6f; // hysteresis, so a steady voice cannot chatter the gate
if (level >= open) _holdFrames = 15; // ~300 ms, bridges the gaps between words
else if (level < close && _holdFrames > 0) _holdFrames--;
return _holdFrames > 0;
}
/// The learned background level, for diagnostics.
public float NoiseFloor => _noiseFloorPublished;
private void Push(float sample)
{
if (sample > 1f) sample = 1f;
else if (sample < -1f) sample = -1f;
_frame[_frameFill++] = (short)(sample * 32767f);
if (_frameFill < FrameSamples) return;
_frameFill = 0;
float sum = 0f;
for (int i = 0; i < FrameSamples; i++)
{
int v = _frame[i];
sum += v < 0 ? -v : v;
}
_lastFrameLevel = sum / FrameSamples / 32768f;
Interlocked.Increment(ref _framesProduced);
// Gated after framing, so the level meter keeps reading while muted and the frame counter
// stays a measure of capture rather than of transmission.
bool pass = _gateOpen && PassesNoiseGate(_lastFrameLevel);
if (pass) OnFrame?.Invoke(_frame);
else if (EmitSilenceWhenGated) OnFrame?.Invoke(_silence);
}
}

View File

@@ -53,6 +53,16 @@ public sealed class VoiceRingBuffer
Volatile.Write(ref _write, w + samples); // publish only once the data is really there Volatile.Write(ref _write, w + samples); // publish only once the data is really there
} }
/// Producer thread only. Same as WritePcm16 but from samples we produced ourselves, so the
/// loopback path does not have to encode to bytes and immediately decode again.
public void WritePcm16Samples(short[] src, int count)
{
if (count <= 0) return;
long w = _write;
for (int i = 0; i < count; i++) _buf[(int)((w + i) & _mask)] = src[i] * (1f / 32768f);
Volatile.Write(ref _write, w + count);
}
/// Producer thread only. Used to reproduce an exact gap the sender told us about, so a /// Producer thread only. Used to reproduce an exact gap the sender told us about, so a
/// dropped frame costs its own duration rather than shifting everything after it. /// dropped frame costs its own duration rather than shifting everything after it.
public void WriteSilence(int samples) public void WriteSilence(int samples)

View File

@@ -0,0 +1,156 @@
using System;
using System.Runtime.CompilerServices;
using System.Threading;
using HarmonyLib;
using Photon.Voice.Unity;
namespace JordanMod.Modules.BingBongVoice;
/// <summary>
/// Taps the local microphone for the voice uplink.
///
/// Feeds <see cref="MicUplinkSource"/>, and on the first five seconds of a session also logs a
/// one-off measurement of the capture conditions. That measurement came first and confirmed, on
/// real hardware, the assumptions the design rests on — 48000 Hz mono, 960-sample frames, 50 a
/// second, on Unity's main thread. It is kept because a mic device change or a Photon update could
/// invalidate any of them, and a wrong premise here is silent rather than loud.
///
/// What it settles:
/// - Which thread the mic callback arrives on (statically it looked like Unity's main thread via
/// VoiceConnection.Update, but LoadBalancingTransport.Service also reaches VoiceClient.Service
/// and that path could not be traced).
/// - The device's real sample rate and channel count, which drive resampling and downmixing.
/// MicWrapper is authoritative here; Recorder.SamplingRate is the *encoder* rate and can differ.
/// - The frame length, and specifically whether it exceeds 256. Photon's MicrophoneRelay.SendMic
/// copies only the first 256 samples before invoking its listeners, which is why that API — the
/// obvious one to use — cannot carry continuous audio. If frames really are longer than 256,
/// that finding is confirmed and patching Read here is the right call.
/// - Signal levels, to calibrate the noise gate. The tap sits upstream of Photon's WebRtcAudioDsp
/// (AEC/NS/AGC attach as post-processors), so this is the raw microphone.
///
/// Structured like BingBongVoicePatch: the patch body is only a try/catch around a NoInlining
/// method, with three strikes then permanent disable. That matters more here than anywhere else in
/// the mod — this sits on Photon's per-frame audio path, so an exception escaping would break all
/// voice chat, not just this feature.
/// </summary>
public class BingBongMicTapPatch
{
private static int _strikes;
private static bool _disabled;
private static int _mainThreadId = -1;
private static int _observedThreadId = -1;
private static int _calls;
private static int _minLength = int.MaxValue;
private static int _maxLength;
private static int _rate;
private static int _channels;
private static float _peak;
private static double _sumAbs;
private static long _samples;
private static float _firstCallTime;
private static bool _reported;
private static bool _waitedForMic;
/// Called from the module on the main thread, so the callback's thread can be compared to it.
public static void NoteMainThread() => _mainThreadId = Thread.CurrentThread.ManagedThreadId;
[HarmonyPatch(typeof(MicWrapper), nameof(MicWrapper.Read))]
[HarmonyPostfix]
static void ReadPostfix(MicWrapper __instance, float[] __0, bool __result)
{
if (_disabled || !__result) return;
try
{
if (!_reported) Measure(__instance, __0);
Feed(__instance, __0);
}
catch (Exception e)
{
Fail(e);
}
}
// The only method here that touches Photon or Unity types, kept separate and un-inlined so a
// member that changed shape throws inside the caller's try rather than while JITing the patch.
[MethodImpl(MethodImplOptions.NoInlining)]
private static void Measure(MicWrapper mic, float[] buffer)
{
if (buffer == null || buffer.Length == 0) return;
if (_calls == 0)
{
_observedThreadId = Thread.CurrentThread.ManagedThreadId;
_firstCallTime = UnityEngine.Time.realtimeSinceStartup;
_rate = mic.SamplingRate;
_channels = mic.Channels;
}
_calls++;
if (buffer.Length < _minLength) _minLength = buffer.Length;
if (buffer.Length > _maxLength) _maxLength = buffer.Length;
// Sample sparsely: this runs on an audio-adjacent hot path and a full scan every frame
// would be a cost we are not here to pay.
for (int i = 0; i < buffer.Length; i += 7)
{
float abs = buffer[i] < 0f ? -buffer[i] : buffer[i];
if (abs > _peak) _peak = abs;
_sumAbs += abs;
_samples++;
}
if (_calls < 250) return; // ~5 seconds at 50 frames/sec
// Photon binds the real capture device a few seconds after joining -- until then it reports
// "[Default] (-128)" and hands out silence. Measuring that window produces an all-zero
// report that looks exactly like a broken microphone, so start over instead.
if (_peak <= 0f)
{
_calls = 0;
_sumAbs = 0;
_samples = 0;
_minLength = int.MaxValue;
_maxLength = 0;
if (!_waitedForMic)
{
_waitedForMic = true;
Debug.Log("[BingBongVoice] Microphone is not producing yet (Photon has not bound the device); will measure again shortly.");
}
return;
}
_reported = true;
float elapsed = UnityEngine.Time.realtimeSinceStartup - _firstCallTime;
float mean = _samples > 0 ? (float)(_sumAbs / _samples) : 0f;
bool onMainThread = _observedThreadId == _mainThreadId;
bool longerThan256 = _maxLength > 256;
Debug.Log(
$"[BingBongVoice] MIC TAP (stage 0): {_calls} frames in {elapsed:F1}s ({_calls / Math.Max(0.01f, elapsed):F0}/s), " +
$"rate={_rate}Hz channels={_channels}, frame length min={_minLength} max={_maxLength}, " +
$"peak={_peak:F3} mean|s|={mean:F4}, thread={_observedThreadId} (main={_mainThreadId}, {(onMainThread ? "MAIN" : "NOT main")}).");
Debug.Log(longerThan256
? $"[BingBongVoice] Frames are {_maxLength} samples, longer than the 256 MicrophoneRelay forwards - confirmed, that API cannot carry audio and patching MicWrapper.Read is correct."
: $"[BingBongVoice] Frames are only {_maxLength} samples, so MicrophoneRelay would NOT have truncated them. Revisit the capture design before building on this patch.");
}
// The only path that reaches the uplink. Separate and un-inlined for the same reason Measure
// is: a Photon member that changed shape must throw inside the caller's try.
[MethodImpl(MethodImplOptions.NoInlining)]
private static void Feed(MicWrapper mic, float[] buffer)
{
MicUplinkSource? source = BingBongVoiceModule.Instance?.Uplink;
source?.OnMicFrame(buffer, mic.SamplingRate, mic.Channels, mic.GetHashCode());
}
private static void Fail(Exception e)
{
Debug.LogError($"[BingBongVoice] Mic tap failed: {e}");
if (++_strikes < 3) return;
_disabled = true;
Debug.LogError("[BingBongVoice] Mic tap disabled for this session after repeated failures.");
}
}

View File

@@ -12,6 +12,12 @@ Requires `websockets` (pip install websockets).
python tools/stub_relay.py tone --sweep # sweep 200-2000 Hz (resampling bugs are obvious) python tools/stub_relay.py tone --sweep # sweep 200-2000 Hz (resampling bugs are obvious)
python tools/stub_relay.py tone --speech # syllables and word gaps (for mouth animation) python tools/stub_relay.py tone --speech # syllables and word gaps (for mouth animation)
python tools/stub_relay.py listen # dump received frames, to verify fan-out python tools/stub_relay.py listen # dump received frames, to verify fan-out
python tools/stub_relay.py uplink # act as the speaker; capture the game's mic to WAV
python tools/stub_relay.py uplink --speech # ... and send audio back at the same time
Only ONE speaker may hold a channel (newest wins), so `tone` and `uplink` cannot both run --
use `uplink --tone/--sweep/--speech` to do both directions on a single connection, which is
also what the real web app does.
Point the mod at it with, in BepInEx config: Point the mod at it with, in BepInEx config:
[BingBong Voice] [BingBong Voice]
@@ -49,6 +55,7 @@ PORT = 8787
PATH = "/voice/ws" PATH = "/voice/ws"
MAGIC = 0xB1 MAGIC = 0xB1
MAGIC_UPLINK = 0xB2 # game -> relay -> speaker; a distinct magic makes a loop impossible
VERSION = 1 VERSION = 1
CODEC_PCM16 = 0 CODEC_PCM16 = 0
RATE_CODES = {16000: 0, 8000: 1, 24000: 2, 48000: 3} RATE_CODES = {16000: 0, 8000: 1, 24000: 2, 48000: 3}
@@ -65,6 +72,9 @@ def pack_header(seq: int, ts: int, rate: int = 16000, codec: int = CODEC_PCM16)
channels: dict[str, dict] = defaultdict(lambda: {"speaker": None, "listeners": set()}) channels: dict[str, dict] = defaultdict(lambda: {"speaker": None, "listeners": set()})
names: dict = {} names: dict = {}
stream_ids: dict = {}
uplinkers: dict = {} # ws -> bool, from the hello frame
deny_uplink = False # --deny-uplink, to exercise the client's uplink_denied path
def roster(channel: str) -> str: def roster(channel: str) -> str:
@@ -72,6 +82,12 @@ def roster(channel: str) -> str:
return ", ".join(sorted(names.get(w, "?") for w in channels[channel]["listeners"])) or "(nobody)" return ", ".join(sorted(names.get(w, "?") for w in channels[channel]["listeners"])) or "(nobody)"
def uplink_roster(channel: str) -> list:
"""Who in this lobby says they may send a microphone. The real relay publishes this so the
game can tell "the relay is dropping my audio" apart from "nobody is listening"."""
return sorted(names.get(w, "?") for w in channels[channel]["listeners"] if uplinkers.get(w))
def print_sessions(): def print_sessions():
if not channels: if not channels:
print("[relay] no active sessions") print("[relay] no active sessions")
@@ -134,11 +150,26 @@ async def handler(ws):
ch["listeners"].add(ws) ch["listeners"].add(ws)
names[ws] = "?" names[ws] = "?"
stream_id = next((i for i in range(16) if i not in stream_ids.get(channel, {}).values()), 0)
stream_ids.setdefault(channel, {})[ws] = stream_id
print(f"[relay] listener joined '{channel}' -> {len(ch['listeners'])} listening") print(f"[relay] listener joined '{channel}' -> {len(ch['listeners'])} listening")
try: try:
# Listeners send a hello frame naming the player. That is what the real relay turns into # Listeners send a hello frame naming the player. That is what the real relay turns into
# its session directory, so the stub records it too and prints the roster. # its session directory, so the stub records it too and prints the roster.
async for msg in ws: async for msg in ws:
if isinstance(msg, bytes):
# Uplink audio. Validate, stamp our own stream id so a client cannot claim
# another's, and forward to the speaker ONLY -- never back to listeners, which is
# half of what makes an audio loop structurally impossible.
if deny_uplink:
continue
if len(msg) != 12 + 640 or msg[0] != MAGIC_UPLINK or msg[1] != VERSION:
continue
frame = bytearray(msg)
frame[3] = (frame[3] & 0x0F) | (stream_id << 4)
if ch["speaker"] is not None:
await safe_send(ch["speaker"], bytes(frame))
continue
if not isinstance(msg, str): if not isinstance(msg, str):
continue continue
try: try:
@@ -147,13 +178,29 @@ async def handler(ws):
continue continue
if payload.get("type") == "hello": if payload.get("type") == "hello":
names[ws] = str(payload.get("player", "?")) names[ws] = str(payload.get("player", "?"))
uplinkers[ws] = bool(payload.get("uplink"))
print(f"[relay] '{names[ws]}' registered in '{channel}' -> roster: {roster(channel)}") print(f"[relay] '{names[ws]}' registered in '{channel}' -> roster: {roster(channel)}")
if uplinkers[ws] and deny_uplink:
await safe_send(ws, json.dumps({"type": "uplink_denied", "reason": "disabled"}))
print(f"[relay] denied uplink to '{names[ws]}' (--deny-uplink)")
elif uplinkers[ws]:
await broadcast_control(channel, {"type": "uplink_roster", "players": uplink_roster(channel)})
elif payload.get("type") in ("uplink_start", "uplink_stop"):
# Enriched with who it came from, which is what the web app needs to label streams.
await safe_send(ch["speaker"], json.dumps({
"type": payload["type"], "id": stream_id, "player": names.get(ws, "?"),
})) if ch["speaker"] is not None else None
print(f"[relay] '{names.get(ws, '?')}' {payload['type']} (stream {stream_id})")
except Exception: except Exception:
pass pass
finally: finally:
ch["listeners"].discard(ws) ch["listeners"].discard(ws)
stream_ids.get(channel, {}).pop(ws, None)
was_uplinker = uplinkers.pop(ws, False)
who = names.pop(ws, "?") who = names.pop(ws, "?")
print(f"[relay] '{who}' left '{channel}' -> {len(ch['listeners'])} listening") print(f"[relay] '{who}' left '{channel}' -> {len(ch['listeners'])} listening")
if was_uplinker:
await broadcast_control(channel, {"type": "uplink_roster", "players": uplink_roster(channel)})
async def fanout(channel: str, payload: bytes): async def fanout(channel: str, payload: bytes):
@@ -181,6 +228,10 @@ async def safe_send(ws, payload):
async def cmd_serve(args): async def cmd_serve(args):
global deny_uplink
deny_uplink = getattr(args, "deny_uplink", False)
if deny_uplink:
print("[relay] --deny-uplink: refusing every microphone with uplink_denied")
async with serve(handler, HOST, args.port, max_size=None): async with serve(handler, HOST, args.port, max_size=None):
print(f"[relay] listening on ws://{HOST}:{args.port}{PATH}") print(f"[relay] listening on ws://{HOST}:{args.port}{PATH}")
print(f"[relay] listen: ws://{HOST}:{args.port}{PATH}?token=listen:test") print(f"[relay] listen: ws://{HOST}:{args.port}{PATH}?token=listen:test")
@@ -244,12 +295,19 @@ def generate_frame(phase, ts, n, rate, args):
async def cmd_tone(args): async def cmd_tone(args):
url = f"ws://{HOST}:{args.port}{PATH}?token=speak:{args.channel}" url = f"ws://{HOST}:{args.port}{PATH}?token=speak:{args.channel}"
print(f"[tone] {url}")
async with connect(url, max_size=None) as ws:
await stream_tone(ws, args)
async def stream_tone(ws, args):
"""Sends a test tone downstream on an already-open speaker socket."""
rate = args.rate rate = args.rate
n = int(rate * FRAME_MS / 1000) n = int(rate * FRAME_MS / 1000)
print(f"[tone] {url} {rate} Hz, {n} samples/frame ({FRAME_MS} ms)") print(f"[tone] {rate} Hz, {n} samples/frame ({FRAME_MS} ms)")
loop = asyncio.get_running_loop() loop = asyncio.get_running_loop()
async with connect(url, max_size=None) as ws: if True:
seq = 0 seq = 0
ts = 0 ts = 0
phase = 0.0 phase = 0.0
@@ -315,12 +373,73 @@ async def cmd_listen(args):
) )
async def cmd_uplink(args):
"""Acts as the web-app speaker: receives uplink audio and writes it out per stream.
The real-time ratio is the number that matters. Below ~0.98 with the gate open means the
sender is not keeping up and the browser would hear clicking; while gated it is expected to
sit low, because silence is deliberately not transmitted.
"""
import wave
url = f"ws://{HOST}:{args.port}{PATH}?token=speak:{args.channel}"
print(f"[uplink] {url}")
print("[uplink] writing uplink_<id>.wav, Ctrl-C to stop")
if args.tone or args.sweep or args.speech:
print("[uplink] also sending a test tone downstream on this same socket")
writers: dict = {}
counts: dict = {}
first_time = None
total_samples = 0
async with connect(url, max_size=None) as ws:
sender = None
if args.tone or args.sweep or args.speech:
args.rate = 16000
args.freq = getattr(args, "freq", 1000.0)
sender = asyncio.create_task(stream_tone(ws, args))
try:
async for msg in ws:
if isinstance(msg, str):
print("\n" + f"[uplink] control: {msg}")
continue
if len(msg) < 12 or msg[0] != MAGIC_UPLINK:
continue
sid = msg[3] >> 4
if sid not in writers:
w = wave.open(f"uplink_{sid}.wav", "wb")
w.setnchannels(1)
w.setsampwidth(2)
w.setframerate(16000)
writers[sid] = w
counts[sid] = 0
print("\n" + f"[uplink] stream {sid} opened -> uplink_{sid}.wav")
writers[sid].writeframes(msg[12:])
counts[sid] += 1
total_samples += (len(msg) - 12) // 2
if first_time is None:
first_time = asyncio.get_running_loop().time()
elapsed = asyncio.get_running_loop().time() - first_time
if sum(counts.values()) % 50 == 0 and elapsed > 0:
ratio = (total_samples / 16000) / elapsed
verdict = "OK" if ratio > 0.98 else "(gated or starving)"
print("\r" + f"[uplink] {sum(counts.values())} frames, "
f"real-time ratio {ratio:.3f} {verdict} ", end="", flush=True)
finally:
if sender is not None:
sender.cancel()
for w in writers.values():
w.close()
print("\n" + "[uplink] wav files closed")
def main(): def main():
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("--port", type=int, default=PORT) p.add_argument("--port", type=int, default=PORT)
sub = p.add_subparsers(dest="cmd", required=True) sub = p.add_subparsers(dest="cmd", required=True)
sub.add_parser("serve") sv = sub.add_parser("serve")
sv.add_argument("--deny-uplink", action="store_true", help="refuse every uplink, to test the client's uplink_denied handling")
t = sub.add_parser("tone") t = sub.add_parser("tone")
t.add_argument("--channel", default="test") t.add_argument("--channel", default="test")
@@ -332,8 +451,15 @@ def main():
l = sub.add_parser("listen") l = sub.add_parser("listen")
l.add_argument("--channel", default="test") l.add_argument("--channel", default="test")
u = sub.add_parser("uplink", help="act as the speaker and capture what the game sends up")
u.add_argument("--channel", default="test")
u.add_argument("--tone", action="store_true", help="also send a steady tone downstream, on the same socket")
u.add_argument("--sweep", action="store_true", help="also send a sweep downstream")
u.add_argument("--speech", action="store_true", help="also send speech-shaped audio downstream")
u.add_argument("--freq", type=float, default=1000.0)
args = p.parse_args() args = p.parse_args()
fn = {"serve": cmd_serve, "tone": cmd_tone, "listen": cmd_listen}[args.cmd] fn = {"serve": cmd_serve, "tone": cmd_tone, "listen": cmd_listen, "uplink": cmd_uplink}[args.cmd]
try: try:
asyncio.run(fn(args)) asyncio.run(fn(args))
except KeyboardInterrupt: except KeyboardInterrupt: