mirror of
https://github.com/TheJordanDev/PEAK-JordanMod.git
synced 2026-10-17 08:05:25 +02:00
Compare commits
5 Commits
0d2350553a
...
daa675af79
| Author | SHA1 | Date | |
|---|---|---|---|
| daa675af79 | |||
| f4d723b23b | |||
| 9fc95c2fc5 | |||
| b0cffe1362 | |||
| 3cd47afb25 |
3
.gitignore
vendored
3
.gitignore
vendored
@@ -226,3 +226,6 @@ artifacts/**/*.zip
|
||||
*.txt
|
||||
artifacts/**
|
||||
Config.Build.user.props
|
||||
|
||||
# Per-stream captures written by tools/stub_relay.py's uplink mode
|
||||
uplink_*.wav
|
||||
|
||||
30
CHANGELOG.md
30
CHANGELOG.md
@@ -95,11 +95,29 @@
|
||||
- The song display no longer scans every loaded object looking for its font on every frame. It falls back to the default font immediately and retries for the nicer one on an interval, stopping once found.
|
||||
- The audio bank URL is normalised before use: a missing scheme is filled in (http for localhost, https otherwise) and a trailing slash trimmed. Without a scheme every request failed outright with "Invalid BugleSoundAPIURL in config", since the URL could not be parsed.
|
||||
|
||||
# v0.1.15 | BingBong live voice (work in progress)
|
||||
# v0.1.15 | BingBong live voice (in-game client)
|
||||
|
||||
- Groundwork for speaking through BingBong with a live microphone, streamed from a relay rather than over PEAK's Photon servers. This release contains the in-game client only: it connects to a relay over a WebSocket, decodes the incoming audio and plays it. The relay itself and the web app that captures the microphone are not built yet, so the feature does nothing on its own.
|
||||
- New "BingBong Voice" config section. The relay URL is derived from `BugleSoundAPIURL` when left empty, since that is where the relay will live, and can be overridden to point anywhere (a bare host or `host:port` is accepted and completed automatically).
|
||||
- Speak through BingBong with a live microphone. Audio is streamed from a relay rather than over PEAK's Photon servers, so it adds no load there at all. This release is the **in-game half only** — the relay and the web app that captures the microphone are not built yet, so the feature does nothing on its own until they exist.
|
||||
- Every BingBong in the world gets a voice source as it spawns, positioned on the plush and inheriting the 3D setup of his own voice lines, so the live voice carries and attenuates like he does and obeys the game's volume sliders. Several BingBongs each speak from their own position.
|
||||
- His mouth animates to the live audio, on every client rather than just the speaker's, and hands control back to the vanilla animation while one of his own voice lines plays.
|
||||
- Canned voice lines take priority: the live voice ducks for the length of the line and then resumes at the live position, so the two never talk over each other. Detected from the RPC body, so the duck stays in step across the lobby without any networking of our own.
|
||||
- New "BingBong Voice" config section, all of it applied live. The relay URL is derived from `BugleSoundAPIURL` when left empty, since that is where the relay will live, and can be overridden to point anywhere — a bare host or `host:port` is completed automatically. Range defaults to 50m, further than his 30m voice lines carry.
|
||||
- Playback goes through `OnAudioFilterRead` rather than a streaming `AudioClip`. A clip created with `stream: true` fills Unity's decode buffer *ahead* of playback, and that read-ahead cannot be satisfied from a live stream because the audio has not arrived yet — the buffer drained to empty on a cycle no matter how deep it was, which was audible as constant clicking. The filter callback runs in lockstep with playback instead, so demand can never outrun real time.
|
||||
- Incoming audio is resampled to the output rate by the mod rather than relying on Unity's mixer, so a device running at 44.1kHz or 48kHz sounds identical. Verified at every rate combination.
|
||||
- A jitter buffer absorbs network timing, with a shortfall rendering what it has and padding the rest rather than muting and refilling, which pumped audibly when it happened often.
|
||||
- `tools/stub_relay.py` is a development stub of the relay, enough to test the client end to end before the real backend exists. It speaks the same wire format and can generate a test tone, so no microphone or browser is needed.
|
||||
- Incoming audio is resampled to the output rate by the mod rather than relying on Unity's mixer, so a device running at 44.1kHz or 48kHz sounds identical. A jitter buffer absorbs network timing, rendering what it has and padding the rest on a shortfall rather than muting and refilling, which pumped audibly when it happened often.
|
||||
- Both Harmony patches are wrapped so that a failure can only cost the voice feature: the patch body is a try/catch around an isolated method, and repeated failures disable it for the session rather than breaking the item.
|
||||
- `tools/stub_relay.py` is a development stub of the relay, enough to test the client end to end before the real backend exists. It speaks the same wire format and generates steady, swept and speech-shaped test signals, so no microphone or browser is needed.
|
||||
|
||||
# v0.1.16 | BingBong live voice: return audio
|
||||
|
||||
- Talking near BingBong now sends your own microphone back to whoever is puppeting him, so it is a conversation rather than a broadcast. Each player only ever transmits their own microphone, over the WebSocket that was already open — no second connection, and still nothing over PEAK's Photon servers.
|
||||
- The uplink is gated on being within the same range BingBong's voice carries and attenuated by the same curve, so holding him up and talking works and walking away fades you out. It only transmits while someone is actually connected on the other end; an idle lobby sends nothing at all.
|
||||
- Capture taps `MicWrapper.Read` rather than the obvious `MicrophoneRelay`, which copies only the first 256 samples of each frame before handing them on. Vanilla only ever wants a loudness number from it so the truncation is invisible in game, but streaming it gives a buzz at the frame rate instead of speech.
|
||||
- The microphone is downmixed, low-passed and resampled from the device's rate to 16kHz by the mod. Decimating 48kHz to 16kHz without filtering first folds everything above 8kHz back into the speech band, which sounds like a second voice made of hiss.
|
||||
- The noise gate learns the room instead of using a fixed level. A fixed threshold cannot work here: measured on real hardware, a quiet voice sits below what a noisy room measures, so any single number either cuts speech or passes the room. It tracks the background level, opens a configurable factor above it, and holds for 300ms so it does not close between words.
|
||||
- The gate opens on entering range and closes only past a margin beyond it, with a minimum time in each state, so standing on the boundary cannot chatter it on and off.
|
||||
- New `UplinkMuteKey` (default: F8) mutes the outgoing microphone for the rest of the session, since reaching for the config file is too slow to be the answer to "stop sending". It is not persisted — a mute you forgot you set is worse than one that clears on the next launch.
|
||||
- The mod now understands the relay's `uplink_denied` and `uplink_roster` messages: a relay with the uplink switched off is believed and not asked again on that connection, and the log names who else is sending room audio, which is what tells "the relay is dropping my audio" apart from "nobody is listening".
|
||||
- Uplink frames carry a different magic byte from the downlink ones, and the relay only ever forwards them to the speaker. A mis-routed frame therefore cannot replay a room's own audio back into it, which would be a feedback loop rather than a bug.
|
||||
- The counters that let the receiver detect a dropped frame advance only for frames actually sent, so a gated pause leaves a contiguous timeline. Start and stop messages tell the receiver to rejoin the live edge instead of filling the pause with silence.
|
||||
- `tools/stub_relay.py` gained the return direction: it forwards uplink audio to the speaker, stamps the stream id itself so a modified client cannot claim another player's, captures each stream to a WAV, and reports what fraction of real time actually arrived. `--deny-uplink` exercises the refusal path.
|
||||
- The relay and the web app that would play this back are not built yet, so the return direction currently only works against that stub.
|
||||
|
||||
@@ -32,12 +32,22 @@ public static class ConfigHandler
|
||||
// BingBong Voice settings
|
||||
public static ConfigEntry<bool> BingBongVoiceEnabled { get; private set; } = null!;
|
||||
public static ConfigEntry<string> BingBongVoiceRelayURL { get; private set; } = null!;
|
||||
public static ConfigEntry<string> BingBongVoiceDevToken { get; private set; } = null!;
|
||||
public static ConfigEntry<float> BingBongVoiceVolume { get; private set; } = null!;
|
||||
public static ConfigEntry<float> BingBongVoiceDuckVolume { get; private set; } = null!;
|
||||
public static ConfigEntry<float> BingBongVoiceDuckLead { get; private set; } = null!;
|
||||
public static ConfigEntry<float> BingBongVoiceJitterMs { get; private set; } = null!;
|
||||
public static ConfigEntry<float> BingBongVoiceMaxDistance { get; private set; } = null!;
|
||||
public static ConfigEntry<bool> BingBongVoiceAutoConnect { get; private set; } = null!;
|
||||
public static ConfigEntry<bool> BingBongVoiceAnnounceSpeaker { get; private set; } = null!;
|
||||
public static ConfigEntry<bool> BingBongVoiceUplinkEnabled { get; private set; } = null!;
|
||||
public static ConfigEntry<KeyCode> BingBongVoiceUplinkMuteKey { get; private set; } = null!;
|
||||
public static ConfigEntry<float> BingBongVoiceUplinkGain { get; private set; } = null!;
|
||||
public static ConfigEntry<float> BingBongVoiceUplinkMinGain { get; private set; } = null!;
|
||||
public static ConfigEntry<float> BingBongVoiceUplinkGateSensitivity { get; private set; } = null!;
|
||||
public static ConfigEntry<float> BingBongVoiceUplinkNoiseGateDb { get; private set; } = null!;
|
||||
public static ConfigEntry<bool> BingBongVoiceUplinkLoopback { get; private set; } = null!;
|
||||
public static ConfigEntry<KeyCode> BingBongVoiceDebugKey { get; private set; } = null!;
|
||||
public static ConfigEntry<bool> BingBongVoiceDebugSource { get; private set; } = null!;
|
||||
|
||||
public static void Initialize(ConfigFile configFile)
|
||||
{
|
||||
@@ -129,15 +139,9 @@ public static class ConfigHandler
|
||||
"BingBong Voice",
|
||||
"RelayURL",
|
||||
"",
|
||||
new ConfigDescription("Override for the voice relay WebSocket URL. Leave empty to derive it from BugleSoundAPIURL, which is where the relay lives by default (https://host -> wss://host/voice/ws).")
|
||||
new ConfigDescription("Override for the voice relay WebSocket URL. Normally leave this EMPTY: it is then derived from BugleSoundAPIURL, keeping that URL's path, so https://host/api becomes wss://host/api/voice/ws. Only set it to point at a different relay, and include the full path - aiming at the website instead of the API gives an HTML page and a 1002 close.")
|
||||
);
|
||||
|
||||
BingBongVoiceDevToken = Config.Bind(
|
||||
"BingBong Voice",
|
||||
"DevToken",
|
||||
"listen:test",
|
||||
new ConfigDescription("Temporary token appended to the relay URL when it carries none. The real join flow will fetch this from the audio bank instead.")
|
||||
);
|
||||
|
||||
BingBongVoiceVolume = Config.Bind(
|
||||
"BingBong Voice",
|
||||
@@ -167,11 +171,88 @@ public static class ConfigHandler
|
||||
new ConfigDescription("Audio buffered before playback starts. Lower is more responsive, higher survives worse connections.", new AcceptableValueRange<float>(40f, 400f))
|
||||
);
|
||||
|
||||
BingBongVoiceMaxDistance = Config.Bind(
|
||||
"BingBong Voice",
|
||||
"MaxDistance",
|
||||
150f,
|
||||
new ConfigDescription("How far the live voice carries, in metres. BingBong's own voice lines only reach 30m, which is short for a conversation.", new AcceptableValueRange<float>(5f, 500f))
|
||||
);
|
||||
|
||||
BingBongVoiceAutoConnect = Config.Bind(
|
||||
"BingBong Voice",
|
||||
"AutoConnect",
|
||||
true,
|
||||
new ConfigDescription("Connect to the voice relay automatically on joining a lobby, and disconnect on leaving. Turn off to connect only with the keybind.")
|
||||
);
|
||||
|
||||
BingBongVoiceAnnounceSpeaker = Config.Bind(
|
||||
"BingBong Voice",
|
||||
"AnnounceSpeaker",
|
||||
true,
|
||||
new ConfigDescription("Show a message when someone starts speaking through BingBong. Being puppeted silently by someone who is not even in your lobby is a bit much without it.")
|
||||
);
|
||||
|
||||
BingBongVoiceUplinkEnabled = Config.Bind(
|
||||
"BingBong Voice",
|
||||
"UplinkEnabled",
|
||||
true,
|
||||
new ConfigDescription("Send your own microphone to whoever is puppeting BingBong, when you are near him and only while someone is actually connected. Turning this off means no microphone patch is applied at all.")
|
||||
);
|
||||
|
||||
BingBongVoiceUplinkMuteKey = Config.Bind(
|
||||
"BingBong Voice",
|
||||
"UplinkMuteKey",
|
||||
KeyCode.F8,
|
||||
new ConfigDescription("Mutes and unmutes your outgoing microphone for the rest of the session. Faster than the config when you need it to stop now.")
|
||||
);
|
||||
|
||||
BingBongVoiceUplinkGain = Config.Bind(
|
||||
"BingBong Voice",
|
||||
"UplinkGain",
|
||||
3.0f,
|
||||
new ConfigDescription("Trim on your outgoing mic. The tap sits upstream of Photon's automatic gain control, so the raw signal is quieter than what teammates hear and usually needs lifting.", new AcceptableValueRange<float>(0f, 8f))
|
||||
);
|
||||
|
||||
BingBongVoiceUplinkMinGain = Config.Bind(
|
||||
"BingBong Voice",
|
||||
"UplinkMinGain",
|
||||
0.02f,
|
||||
new ConfigDescription("Stop transmitting once distance has attenuated you below this. Keeps the edge of the range quiet rather than faintly open.", new AcceptableValueRange<float>(0f, 0.5f))
|
||||
);
|
||||
|
||||
BingBongVoiceUplinkGateSensitivity = Config.Bind(
|
||||
"BingBong Voice",
|
||||
"UplinkGateSensitivity",
|
||||
3.0f,
|
||||
new ConfigDescription("How far above the measured background noise your voice must be before it is transmitted. The gate learns your room rather than using a fixed level, because a quiet voice can measure lower than a noisy room. Raise it if background noise gets through, lower it if quiet speech is cut off.", new AcceptableValueRange<float>(1.5f, 10f))
|
||||
);
|
||||
|
||||
BingBongVoiceUplinkNoiseGateDb = Config.Bind(
|
||||
"BingBong Voice",
|
||||
"UplinkFloorDb",
|
||||
-60f,
|
||||
new ConfigDescription("Absolute floor: nothing quieter than this is ever transmitted, whatever the adaptive gate decides. A safety net, not the main control.", new AcceptableValueRange<float>(-80f, -20f))
|
||||
);
|
||||
|
||||
BingBongVoiceUplinkLoopback = Config.Bind(
|
||||
"BingBong Voice",
|
||||
"UplinkLoopback",
|
||||
false,
|
||||
new ConfigDescription("Debug: play your own processed microphone back out of BingBong instead of sending it anywhere. Lets the capture path be heard with no relay and no second player. Deliberately closes a feedback loop, so use headphones.")
|
||||
);
|
||||
|
||||
BingBongVoiceDebugKey = Config.Bind(
|
||||
"BingBong Voice",
|
||||
"DebugToggleKey",
|
||||
KeyCode.F7,
|
||||
new ConfigDescription("Connect/disconnect the relay and play it through a non-positional debug source, for testing without a BingBong")
|
||||
new ConfigDescription("Connect to / disconnect from the voice relay")
|
||||
);
|
||||
|
||||
BingBongVoiceDebugSource = Config.Bind(
|
||||
"BingBong Voice",
|
||||
"DebugSource",
|
||||
false,
|
||||
new ConfigDescription("Also play the stream through a non-positional source, for testing without a BingBong in the scene. Plays on top of any real BingBong, so leave this off normally.")
|
||||
);
|
||||
|
||||
}
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
<!-- This is the display name of your mod. Example: BepInEx Template -->
|
||||
<AssemblyTitle>JordanMod</AssemblyTitle>
|
||||
<!-- This is the version number of your mod. -->
|
||||
<Version>0.1.15</Version>
|
||||
<Version>0.1.16</Version>
|
||||
</PropertyGroup>
|
||||
|
||||
<PropertyGroup>
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
using System;
|
||||
using System.Security.Authentication;
|
||||
using System.Threading;
|
||||
using Newtonsoft.Json.Linq;
|
||||
using WebSocketSharp;
|
||||
|
||||
namespace JordanMod.Modules.BingBongVoice;
|
||||
@@ -36,12 +37,16 @@ public sealed class BingBongVoiceClient
|
||||
private const int MaxGapSamples = 48000;
|
||||
|
||||
private WebSocket? _socket;
|
||||
private string? _hello;
|
||||
private VoiceStream? _stream;
|
||||
private Thread? _reconnectThread;
|
||||
private volatile bool _shouldRun;
|
||||
private volatile bool _speakerActive;
|
||||
private volatile bool _uplinkDenied;
|
||||
private long _expectedTimestamp = -1;
|
||||
private int _badFrames;
|
||||
private volatile int _lastCloseCode;
|
||||
private int _sendFailures;
|
||||
|
||||
/// The live stream, or null before the first frame arrives. Read from the audio thread.
|
||||
public VoiceStream? Stream => Volatile.Read(ref _stream);
|
||||
@@ -49,12 +54,29 @@ public sealed class BingBongVoiceClient
|
||||
/// True between speaker_joined and speaker_left control frames.
|
||||
public bool SpeakerActive => _speakerActive;
|
||||
|
||||
/// Set when the relay says it will not take our microphone -- uplink switched off on its side,
|
||||
/// or too many senders on this channel. Sticky until the next connect, because the answer will
|
||||
/// not change on this socket and retrying for the rest of the session would just burn bandwidth.
|
||||
public bool UplinkDenied => _uplinkDenied;
|
||||
|
||||
/// Who is speaking, when the relay tells us. Null if nobody is, or if it did not say.
|
||||
public string? SpeakerName { get; private set; }
|
||||
|
||||
/// Raised on the main thread when the speaker changes, so the UI can announce it.
|
||||
public event Action<string?>? OnSpeakerChanged;
|
||||
|
||||
public bool IsConnected => _socket is { ReadyState: WebSocketState.Open };
|
||||
|
||||
public string Url { get; private set; } = "";
|
||||
|
||||
public void Connect(string url)
|
||||
/// <param name="hello">
|
||||
/// Optional JSON control frame sent as soon as the socket opens. This is how a lobby
|
||||
/// registers itself with the relay: the channel comes from the token, and this supplies who
|
||||
/// is in it, so an admin can see the live sessions and pick one to speak into.
|
||||
/// </param>
|
||||
public void Connect(string url, string? hello = null)
|
||||
{
|
||||
_hello = hello;
|
||||
if (_shouldRun) return;
|
||||
if (string.IsNullOrWhiteSpace(url))
|
||||
{
|
||||
@@ -105,7 +127,9 @@ public sealed class BingBongVoiceClient
|
||||
while (_shouldRun && _socket is { ReadyState: WebSocketState.Open }) Thread.Sleep(200);
|
||||
|
||||
if (!_shouldRun) return;
|
||||
attempt++;
|
||||
|
||||
// A rejected request goes straight to the ceiling rather than climbing to it.
|
||||
attempt = _lastCloseCode == 1008 ? 6 : attempt + 1;
|
||||
}
|
||||
catch (Exception e)
|
||||
{
|
||||
@@ -144,12 +168,37 @@ public sealed class BingBongVoiceClient
|
||||
{
|
||||
_expectedTimestamp = -1;
|
||||
_badFrames = 0;
|
||||
_uplinkDenied = false;
|
||||
Debug.Log("[BingBongVoice] Connected to relay.");
|
||||
|
||||
// Register this lobby member so the relay's session directory knows who is listening.
|
||||
if (string.IsNullOrEmpty(_hello)) return;
|
||||
try { socket.Send(_hello); }
|
||||
catch (Exception ex) { Debug.LogWarning($"[BingBongVoice] Could not announce presence: {ex.Message}"); }
|
||||
};
|
||||
socket.OnClose += (_, e) =>
|
||||
{
|
||||
_speakerActive = false;
|
||||
Debug.Log($"[BingBongVoice] Relay closed ({e.Code} {e.Reason}).");
|
||||
_lastCloseCode = e.Code;
|
||||
|
||||
// 1008 means the request itself is wrong -- bad token shape, uppercase channel, an
|
||||
// Origin header we should not be sending, or the relay being disabled. Retrying
|
||||
// quickly cannot fix any of those, so say so plainly and let the loop back right off.
|
||||
if (e.Code == 1008)
|
||||
{
|
||||
Debug.LogError($"[BingBongVoice] Relay rejected the connection ({e.Reason}). This will not fix itself; check the relay URL and token.");
|
||||
}
|
||||
else if (e.Code == 1002)
|
||||
{
|
||||
// The handshake got a valid HTTP response that was not an upgrade -- almost always
|
||||
// the URL pointing at the web frontend rather than the API. A single-page app
|
||||
// answers 200 with HTML for any path, so it looks alive and never works.
|
||||
Debug.LogError($"[BingBongVoice] '{Url}' answered with something that is not a WebSocket. Point it at the API rather than the website: the endpoint usually sits under /api (e.g. https://host/api -> wss://host/api/voice/ws). Clearing RelayURL derives it from BugleSoundAPIURL.");
|
||||
}
|
||||
else
|
||||
{
|
||||
Debug.Log($"[BingBongVoice] Relay closed ({e.Code} {e.Reason}).");
|
||||
}
|
||||
};
|
||||
socket.OnError += (_, e) => Debug.LogWarning($"[BingBongVoice] Socket error: {e.Message}");
|
||||
|
||||
@@ -205,6 +254,36 @@ public sealed class BingBongVoiceClient
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Sends one frame on the socket we already hold. Returns false rather than throwing when the
|
||||
/// socket is not open, so the uplink can count failures without ever driving reconnection --
|
||||
/// that belongs to ConnectLoop, and a second driver would fight it.
|
||||
/// </summary>
|
||||
public bool TrySend(byte[] frame)
|
||||
{
|
||||
WebSocket? socket = _socket;
|
||||
if (socket == null || socket.ReadyState != WebSocketState.Open) return false;
|
||||
try
|
||||
{
|
||||
socket.Send(frame);
|
||||
return true;
|
||||
}
|
||||
catch (Exception e)
|
||||
{
|
||||
if (++_sendFailures <= 3) Debug.LogWarning($"[BingBongVoice] Uplink send failed: {e.Message}");
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/// Control frames for the uplink. Text, like every other control message.
|
||||
public bool TrySendControl(string json)
|
||||
{
|
||||
WebSocket? socket = _socket;
|
||||
if (socket == null || socket.ReadyState != WebSocketState.Open) return false;
|
||||
try { socket.Send(json); return true; }
|
||||
catch { return false; }
|
||||
}
|
||||
|
||||
private VoiceStream EnsureStream(int rate)
|
||||
{
|
||||
VoiceStream? current = Volatile.Read(ref _stream);
|
||||
@@ -219,18 +298,54 @@ public sealed class BingBongVoiceClient
|
||||
|
||||
private void HandleControl(string json)
|
||||
{
|
||||
// Deliberately not parsing JSON properly: the only fields that matter are these two
|
||||
// markers, and a substring check can't throw on a malformed payload.
|
||||
// Parsed leniently and fielded by substring if that fails: control messages are allowed to
|
||||
// gain fields, and a malformed payload must never take down the receive thread.
|
||||
string? speaker = null;
|
||||
JObject? parsed = null;
|
||||
try
|
||||
{
|
||||
parsed = JObject.Parse(json);
|
||||
speaker = parsed["speaker"]?.ToString();
|
||||
}
|
||||
catch
|
||||
{
|
||||
// not JSON, or no such field -- the substring checks below still work
|
||||
}
|
||||
|
||||
if (json.Contains("speaker_joined"))
|
||||
{
|
||||
_speakerActive = true;
|
||||
Debug.Log("[BingBongVoice] A speaker joined.");
|
||||
SpeakerName = string.IsNullOrWhiteSpace(speaker) ? null : speaker;
|
||||
Debug.Log($"[BingBongVoice] A speaker joined{(SpeakerName != null ? $": {SpeakerName}" : "")}.");
|
||||
Plugin.RunOnMainThread(() => OnSpeakerChanged?.Invoke(SpeakerName));
|
||||
}
|
||||
else if (json.Contains("speaker_left") || json.Contains("preempted"))
|
||||
{
|
||||
_speakerActive = false;
|
||||
SpeakerName = null;
|
||||
_expectedTimestamp = -1;
|
||||
Debug.Log("[BingBongVoice] The speaker left.");
|
||||
Plugin.RunOnMainThread(() => OnSpeakerChanged?.Invoke(null));
|
||||
}
|
||||
else if (json.Contains("uplink_denied"))
|
||||
{
|
||||
// The relay has told us it is not taking microphones. Nothing about that changes while
|
||||
// this socket is up, so stop asking rather than sending 50 frames a second into a drop.
|
||||
_uplinkDenied = true;
|
||||
Debug.LogWarning($"[BingBongVoice] The relay will not take our microphone ({parsed?["reason"]?.ToString() ?? json}). Not sending until the next connect.");
|
||||
}
|
||||
else if (json.Contains("uplink_roster"))
|
||||
{
|
||||
// Purely informational, and the one thing that distinguishes "the relay is dropping my
|
||||
// audio" from "nobody is listening" when the uplink looks like it is working locally.
|
||||
string players = parsed?["players"] is JArray list && list.Count > 0
|
||||
? string.Join(", ", list)
|
||||
: "nobody";
|
||||
Debug.Log($"[BingBongVoice] Sending room audio: {players}.");
|
||||
}
|
||||
else if (json.Contains("rejected"))
|
||||
{
|
||||
Debug.LogWarning($"[BingBongVoice] Relay rejected this listener: {json}");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,4 +1,10 @@
|
||||
using System;
|
||||
using System.Security.Cryptography;
|
||||
using System.Text;
|
||||
using JordanMod.Modules.BetterBugle;
|
||||
using Newtonsoft.Json;
|
||||
using Photon.Pun;
|
||||
using Photon.Voice.Unity;
|
||||
using UnityEngine;
|
||||
|
||||
namespace JordanMod.Modules.BingBongVoice;
|
||||
@@ -14,26 +20,170 @@ class BingBongVoiceModule : Module
|
||||
|
||||
private GameObject? _debugObject;
|
||||
|
||||
/// Captures the local microphone. Null when the uplink is off or we are not in a lobby, which
|
||||
/// is what makes the mic tap a no-op rather than something that needs its own guard.
|
||||
public MicUplinkSource? Uplink { get; private set; }
|
||||
|
||||
/// Stage 1 debug: the captured mic played back through BingBong instead of being sent. Lets
|
||||
/// the whole capture path be heard with no relay and no second player.
|
||||
public VoiceStream? LoopbackStream { get; private set; }
|
||||
|
||||
private MicUplinkSender? _sender;
|
||||
private bool _uplinkTransmitting;
|
||||
|
||||
private const string ChannelSalt = "jordanmod-bingbong-voice-v1";
|
||||
|
||||
public override Type[] GetPatches()
|
||||
{
|
||||
// Config is bound before modules load (Plugin.Awake: ConfigHandler.Initialize ->
|
||||
// SetupModules -> SetupPatches), so turning the feature off here means the Harmony
|
||||
// patches are never applied at all and the mod has zero footprint.
|
||||
if (!ConfigHandler.BingBongVoiceEnabled.Value) return [];
|
||||
return [];
|
||||
|
||||
// The mic tap sits on Photon's per-frame audio path, so an opted-out player should carry
|
||||
// no patch at all rather than one that early-returns.
|
||||
return ConfigHandler.BingBongVoiceUplinkEnabled.Value
|
||||
? [typeof(BingBongVoicePatch), typeof(BingBongMicTapPatch)]
|
||||
: [typeof(BingBongVoicePatch)];
|
||||
}
|
||||
|
||||
public override void Initialize()
|
||||
{
|
||||
if (Instance != null) return;
|
||||
Instance = this;
|
||||
|
||||
// Recorded here so the mic callback's thread can be compared against it.
|
||||
BingBongMicTapPatch.NoteMainThread();
|
||||
|
||||
// The relay lives on the audio bank, so with no override configured every player derives
|
||||
// the same URL and connects on their own -- nobody has to press anything.
|
||||
PhotonNetworkEventListener listener = PhotonNetworkEventListener.Instance!;
|
||||
if (listener != null)
|
||||
{
|
||||
listener.RegisterOnJoinedRoom(OnJoinedRoom);
|
||||
listener.RegisterOnLeftRoom(Disconnect);
|
||||
listener.RegisterOnDisconnected(_ => Disconnect());
|
||||
}
|
||||
else
|
||||
{
|
||||
Debug.LogWarning("[BingBongVoice] No network event listener, auto-connect is unavailable.");
|
||||
}
|
||||
|
||||
base.Initialize();
|
||||
}
|
||||
|
||||
private void OnJoinedRoom()
|
||||
{
|
||||
if (!ConfigHandler.BingBongVoiceEnabled.Value) return;
|
||||
if (!ConfigHandler.BingBongVoiceAutoConnect.Value) return;
|
||||
Connect();
|
||||
}
|
||||
|
||||
public override void Update()
|
||||
{
|
||||
if (!ConfigHandler.BingBongVoiceEnabled.Value) return;
|
||||
if (Input.GetKeyDown(ConfigHandler.BingBongVoiceDebugKey.Value)) ToggleDebugStream();
|
||||
if (Input.GetKeyDown(ConfigHandler.BingBongVoiceDebugKey.Value)) ToggleStream();
|
||||
if (Input.GetKeyDown(ConfigHandler.BingBongVoiceUplinkMuteKey.Value)) ToggleUplinkMute();
|
||||
EnsureUplink();
|
||||
UpdateUplinkProximity();
|
||||
}
|
||||
|
||||
// Hysteresis on the gate only. The gain is continuous so it never stutters; what stutters is
|
||||
// the open/closed decision, and each cycle costs the receiver a buffer reset.
|
||||
private bool _uplinkInRange;
|
||||
private float _uplinkStateChangedAt;
|
||||
private float _lastUplinkReport;
|
||||
private bool _uplinkMuted;
|
||||
|
||||
/// Session-scoped mute for the outgoing microphone, on a key because reaching for the config is
|
||||
/// too slow to be the answer to "stop sending, now". Deliberately not persisted: a mute you
|
||||
/// forgot you set is worse than one that clears when you next launch.
|
||||
private void ToggleUplinkMute()
|
||||
{
|
||||
_uplinkMuted = !_uplinkMuted;
|
||||
Debug.Log($"[BingBongVoice] Uplink microphone {(_uplinkMuted ? "MUTED" : "unmuted")}.");
|
||||
BetterBugleUI.Instance?.ShowActionbar(_uplinkMuted
|
||||
? "🎙 BingBong microphone muted"
|
||||
: "🎙 BingBong microphone live");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Decides how loudly, if at all, the local microphone should be transmitted: nearest BingBong,
|
||||
/// attenuated by the same curve his own voice is heard through.
|
||||
/// </summary>
|
||||
private void UpdateUplinkProximity()
|
||||
{
|
||||
MicUplinkSource? uplink = Uplink;
|
||||
if (uplink == null) return;
|
||||
|
||||
Character? local = Character.localCharacter;
|
||||
if (local == null)
|
||||
{
|
||||
uplink.SetGate(false);
|
||||
return;
|
||||
}
|
||||
|
||||
float range = ConfigHandler.BingBongVoiceMaxDistance.Value;
|
||||
float nearest = float.MaxValue;
|
||||
foreach (BingBongVoiceSource source in BingBongVoiceSource.Active)
|
||||
{
|
||||
if (source == null || !source.IsPositional) continue;
|
||||
float d = Vector3.Distance(local.Head, source.transform.position);
|
||||
if (d < nearest) nearest = d;
|
||||
}
|
||||
|
||||
// Open on entering the range, close only past a margin beyond it, so walking the boundary
|
||||
// cannot chatter the gate. The dwell is a minimum time in the CURRENT state before it may
|
||||
// flip again -- not a check against the change just made, which would undo it every frame.
|
||||
bool want = _uplinkInRange ? nearest <= range * 1.15f : nearest < range;
|
||||
if (want != _uplinkInRange)
|
||||
{
|
||||
float minDwell = _uplinkInRange ? 1.5f : 0.5f;
|
||||
if (Time.time - _uplinkStateChangedAt >= minDwell)
|
||||
{
|
||||
_uplinkInRange = want;
|
||||
_uplinkStateChangedAt = Time.time;
|
||||
}
|
||||
}
|
||||
|
||||
float gain = 0f;
|
||||
if (_uplinkInRange && nearest < float.MaxValue)
|
||||
{
|
||||
// The curve, then a taper over the last fifth so the gate closes on real silence
|
||||
// instead of a cliff. Unity's own rolloff stops attenuating past maxDistance, which
|
||||
// copied literally would leave everyone in the level faintly transmitting forever.
|
||||
gain = BingBongVoiceSource.RangeGain(nearest, range)
|
||||
* Mathf.SmoothStep(1f, 0f, Mathf.InverseLerp(range * 0.8f, range, nearest))
|
||||
* ConfigHandler.BingBongVoiceUplinkGain.Value;
|
||||
}
|
||||
|
||||
bool audible = gain > ConfigHandler.BingBongVoiceUplinkMinGain.Value;
|
||||
|
||||
// Only while someone is actually puppeting BingBong. An idle lobby sends nothing, and
|
||||
// "my mic goes out only while someone is on the other end" is a rule a player can hold
|
||||
// in their head. Loopback ignores it, having no speaker to wait for.
|
||||
bool speakerPresent = ConfigHandler.BingBongVoiceUplinkLoopback.Value || (Client?.SpeakerActive ?? false);
|
||||
bool allowed = !_uplinkMuted && !(Client?.UplinkDenied ?? false);
|
||||
bool transmit = _uplinkInRange && audible && speakerPresent && allowed;
|
||||
|
||||
uplink.SetTargetGain(gain);
|
||||
uplink.SetGate(transmit);
|
||||
|
||||
// The receiver's timeline freezes with ours while gated, so it needs telling to rejoin the
|
||||
// live edge rather than treating the pause as a gap to fill with silence.
|
||||
if (transmit != _uplinkTransmitting && !ConfigHandler.BingBongVoiceUplinkLoopback.Value)
|
||||
{
|
||||
_uplinkTransmitting = transmit;
|
||||
Client?.TrySendControl(transmit ? "{\"type\":\"uplink_start\"}" : "{\"type\":\"uplink_stop\"}");
|
||||
Debug.Log($"[BingBongVoice] Uplink {(transmit ? "started" : "stopped")} sending.");
|
||||
}
|
||||
|
||||
if (Time.time - _lastUplinkReport < 5f) return;
|
||||
_lastUplinkReport = Time.time;
|
||||
Debug.Log($"[BingBongVoice] Uplink: nearest BingBong {(nearest < float.MaxValue ? nearest.ToString("F1") + "m" : "none")}, " +
|
||||
$"gain {gain:F2}, gate {(_uplinkInRange && audible ? "open" : "closed")}{(_uplinkMuted ? ", MUTED" : "")}, " +
|
||||
$"level {uplink.Level:F4} (floor {uplink.NoiseFloor:F5}), {uplink.FramesProduced} captured" +
|
||||
$"{(_sender != null ? $", {_sender.Sent} sent, {_sender.Dropped} dropped" : "")}. {DescribeMic(local)}");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -73,18 +223,61 @@ class BingBongVoiceModule : Module
|
||||
return "";
|
||||
}
|
||||
|
||||
// Until the real /voice/join flow lands, the token comes from config rather than a
|
||||
// lookup. A URL that already carries one is left alone.
|
||||
// A URL that already carries a token is left exactly as given -- that is how you point at
|
||||
// a stub relay, or connect from outside a lobby.
|
||||
if (url.IndexOf("token=", StringComparison.OrdinalIgnoreCase) < 0)
|
||||
{
|
||||
string token = (ConfigHandler.BingBongVoiceDevToken.Value ?? "").Trim();
|
||||
if (token.Length > 0)
|
||||
url += (url.IndexOf('?') >= 0 ? "&" : "?") + "token=" + Uri.EscapeDataString(token);
|
||||
string channel = ResolveChannel();
|
||||
if (channel.Length == 0)
|
||||
{
|
||||
Debug.LogWarning("[BingBongVoice] Not in a lobby, so there is no channel to join. Put a full URL including ?token= in RelayURL to connect from outside one.");
|
||||
return "";
|
||||
}
|
||||
// The relay's published contract is "listen:<channel>". It also accepts a bare channel
|
||||
// id, but following the documented form is what stays correct if it ever tightens.
|
||||
// The channel must be lowercase hex -- uppercase is rejected with a 1008.
|
||||
url += (url.IndexOf('?') >= 0 ? "&" : "?") + "token=" + Uri.EscapeDataString("listen:" + channel);
|
||||
}
|
||||
|
||||
return url;
|
||||
}
|
||||
|
||||
/// Announced to the relay on connect so admins can see which lobbies are live and who is in
|
||||
/// them. Deliberately carries the player name but NOT the room code: the name is what makes a
|
||||
/// session recognisable in the web app, whereas the code is what someone would need to join
|
||||
/// the game, so there is no reason to put it on the wire.
|
||||
private static string BuildHello()
|
||||
{
|
||||
string nickname = PhotonNetwork.LocalPlayer?.NickName ?? "";
|
||||
if (string.IsNullOrWhiteSpace(nickname)) nickname = "Unknown";
|
||||
|
||||
// Whether we might ever send a microphone, so the relay can allocate a stream id and get the
|
||||
// roster right before the first audio frame rather than a beat after it.
|
||||
bool uplink = ConfigHandler.BingBongVoiceUplinkEnabled.Value
|
||||
&& !ConfigHandler.BingBongVoiceUplinkLoopback.Value;
|
||||
return JsonConvert.SerializeObject(new { type = "hello", player = nickname, uplink });
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Scopes the voice channel to the Photon lobby, so each game session gets its own and
|
||||
/// players only hear whoever is speaking into theirs.
|
||||
///
|
||||
/// Hashed rather than sent as-is: the room code is what someone would need to join the game,
|
||||
/// and it would otherwise sit in the URL and in every server access log. The salt is baked
|
||||
/// into the mod so it is not a secret -- it only stops a channel id from being trivially
|
||||
/// reversible into a joinable room code. Once the relay's own join endpoint exists it should
|
||||
/// issue channel ids server-side instead, with a salt that really is private.
|
||||
/// </summary>
|
||||
public static string ResolveChannel()
|
||||
{
|
||||
string room = PhotonNetwork.CurrentRoom?.Name ?? "";
|
||||
if (string.IsNullOrWhiteSpace(room)) return "";
|
||||
|
||||
using SHA256 sha = SHA256.Create();
|
||||
byte[] hash = sha.ComputeHash(Encoding.UTF8.GetBytes(ChannelSalt + room.Trim().ToUpperInvariant()));
|
||||
return BitConverter.ToString(hash).Replace("-", "").ToLowerInvariant().Substring(0, 32);
|
||||
}
|
||||
|
||||
private static string NormalizeOverride(string raw)
|
||||
{
|
||||
string url = ToWebSocketScheme(raw);
|
||||
@@ -134,34 +327,150 @@ class BingBongVoiceModule : Module
|
||||
return Helper.EnsureScheme(url, "wss", "ws");
|
||||
}
|
||||
|
||||
// Stage 1 harness: connect and play through a non-positional source, so the whole network
|
||||
// and audio path can be tested with no BingBong, no lobby and no browser.
|
||||
private void ToggleDebugStream()
|
||||
// Every BingBong attaches its own source through BingBongVoicePatch; the optional debug
|
||||
// source is the stage 1 harness, for testing with no BingBong in the scene.
|
||||
private void ToggleStream()
|
||||
{
|
||||
if (Client != null)
|
||||
{
|
||||
Debug.Log("[BingBongVoice] Stopping debug stream.");
|
||||
Client.Disconnect();
|
||||
Client = null;
|
||||
if (_debugObject != null) UnityEngine.Object.Destroy(_debugObject);
|
||||
_debugObject = null;
|
||||
return;
|
||||
}
|
||||
if (Client != null) Disconnect();
|
||||
else Connect();
|
||||
}
|
||||
|
||||
private void Connect()
|
||||
{
|
||||
if (Client != null) return;
|
||||
|
||||
string url = ResolveRelayUrl();
|
||||
if (url.Length == 0) return;
|
||||
|
||||
Debug.Log($"[BingBongVoice] Starting debug stream against {url}");
|
||||
Debug.Log($"[BingBongVoice] Connecting for lobby '{PhotonNetwork.CurrentRoom?.Name}' to {url}");
|
||||
Client = new BingBongVoiceClient();
|
||||
Client.Connect(url);
|
||||
Client.Connect(url, BuildHello());
|
||||
|
||||
if (!ConfigHandler.BingBongVoiceDebugSource.Value) return;
|
||||
_debugObject = new GameObject("BingBongVoiceDebug");
|
||||
UnityEngine.Object.DontDestroyOnLoad(_debugObject);
|
||||
BingBongVoiceSource.Attach(_debugObject, null);
|
||||
}
|
||||
|
||||
// Reuses the Bugle's actionbar because it is the only on-screen text this mod has. Raised on
|
||||
// the main thread by the client, so touching Unity here is safe.
|
||||
private static void AnnounceSpeaker(string? speaker)
|
||||
{
|
||||
if (!ConfigHandler.BingBongVoiceAnnounceSpeaker.Value) return;
|
||||
BetterBugleUI.Instance?.ShowActionbar(speaker != null
|
||||
? $"🎙 BingBong is being puppeted by {speaker}"
|
||||
: "BingBong is himself again");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Photon's own view of the microphone. When our captured level is zero the question is always
|
||||
/// whether the game is feeding the device at all -- push-to-talk released, no device selected,
|
||||
/// or the recorder stopped -- rather than anything in the capture path.
|
||||
/// </summary>
|
||||
private static string DescribeMic(Character local)
|
||||
{
|
||||
try
|
||||
{
|
||||
CharacterVoiceHandler? voice = local.refs?.voice;
|
||||
if (voice == null) return "(no voice handler)";
|
||||
|
||||
Recorder? recorder = voice.GetComponent<Recorder>();
|
||||
if (recorder == null) return "(no recorder)";
|
||||
|
||||
string device = recorder.MicrophoneDevice != null ? recorder.MicrophoneDevice.ToString() : "none";
|
||||
return $"mic: transmitting={recorder.IsCurrentlyTransmitting}, enabled={recorder.TransmitEnabled}, device={device}";
|
||||
}
|
||||
catch (Exception e)
|
||||
{
|
||||
return $"(mic state unreadable: {e.GetType().Name})";
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Keeps the capture path in step with the config, every frame. Deliberately independent of the
|
||||
/// relay connection: loopback has to work with no network at all, and tying capture to a
|
||||
/// successful connect meant an unreachable relay silently disabled it. Also picks up a config
|
||||
/// change live, rather than only at the moment a lobby is joined.
|
||||
/// </summary>
|
||||
private void EnsureUplink()
|
||||
{
|
||||
if (!ConfigHandler.BingBongVoiceUplinkEnabled.Value)
|
||||
{
|
||||
if (Uplink != null) StopUplink();
|
||||
return;
|
||||
}
|
||||
|
||||
bool wantLoopback = ConfigHandler.BingBongVoiceUplinkLoopback.Value;
|
||||
bool haveLoopback = LoopbackStream != null;
|
||||
bool senderMatchesClient = wantLoopback || (_sender != null) == (Client != null);
|
||||
if (Uplink != null && wantLoopback == haveLoopback && senderMatchesClient) return;
|
||||
|
||||
StopUplink();
|
||||
StartUplink();
|
||||
}
|
||||
|
||||
private void StartUplink()
|
||||
{
|
||||
if (!ConfigHandler.BingBongVoiceUplinkEnabled.Value) return;
|
||||
if (Uplink != null) return;
|
||||
|
||||
MicUplinkSource source = new();
|
||||
|
||||
if (ConfigHandler.BingBongVoiceUplinkLoopback.Value)
|
||||
{
|
||||
LoopbackStream = new VoiceStream(MicUplinkSource.WireRate);
|
||||
source.EmitSilenceWhenGated = true; // keep playback continuous; see the field's note
|
||||
source.OnFrame = frame => LoopbackStream.Ring.WritePcm16Samples(frame, frame.Length);
|
||||
Debug.LogWarning("[BingBongVoice] Uplink LOOPBACK is on: your microphone plays back out of BingBong and is not sent anywhere. Wear headphones.");
|
||||
}
|
||||
else if (Client != null)
|
||||
{
|
||||
MicUplinkSender sender = new(Client);
|
||||
sender.Start();
|
||||
source.OnFrame = sender.Enqueue;
|
||||
_sender = sender;
|
||||
}
|
||||
|
||||
Uplink = source;
|
||||
Debug.Log($"[BingBongVoice] Uplink capture started{(ConfigHandler.BingBongVoiceUplinkLoopback.Value ? " in LOOPBACK mode" : "")}.");
|
||||
}
|
||||
|
||||
private void StopUplink()
|
||||
{
|
||||
if (_uplinkTransmitting)
|
||||
{
|
||||
Client?.TrySendControl("{\"type\":\"uplink_stop\"}");
|
||||
_uplinkTransmitting = false;
|
||||
}
|
||||
|
||||
if (Uplink != null) Uplink.OnFrame = null;
|
||||
Uplink = null;
|
||||
LoopbackStream = null;
|
||||
|
||||
_sender?.Stop();
|
||||
_sender = null;
|
||||
}
|
||||
|
||||
private void Disconnect()
|
||||
{
|
||||
StopUplink();
|
||||
if (Client == null) return;
|
||||
Debug.Log("[BingBongVoice] Disconnecting from relay.");
|
||||
Client.Disconnect();
|
||||
Client = null;
|
||||
if (_debugObject != null) UnityEngine.Object.Destroy(_debugObject);
|
||||
_debugObject = null;
|
||||
}
|
||||
|
||||
public override void Destroy()
|
||||
{
|
||||
PhotonNetworkEventListener listener = PhotonNetworkEventListener.Instance!;
|
||||
if (listener != null)
|
||||
{
|
||||
listener.JoinedRoom -= OnJoinedRoom;
|
||||
listener.LeftRoom -= Disconnect;
|
||||
}
|
||||
|
||||
Client?.Disconnect();
|
||||
Client = null;
|
||||
if (_debugObject != null) UnityEngine.Object.Destroy(_debugObject);
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
using System;
|
||||
using System.Runtime.CompilerServices;
|
||||
using UnityEngine;
|
||||
|
||||
namespace JordanMod.Modules.BingBongVoice;
|
||||
@@ -29,7 +30,8 @@ public sealed class BingBongVoiceSource : MonoBehaviour
|
||||
private float _gain; // smoothed toward _targetGain, audio thread only
|
||||
|
||||
private volatile float _targetGain;
|
||||
private volatile float _amplitude;
|
||||
private volatile float _amplitude; // post-gain, what you actually hear
|
||||
private volatile float _rawAmplitude; // pre-gain MEAN |sample|, matching BingBongMouth's units
|
||||
private volatile bool _dead;
|
||||
|
||||
private int _underruns;
|
||||
@@ -40,32 +42,111 @@ public sealed class BingBongVoiceSource : MonoBehaviour
|
||||
private volatile int _streamRate = 1;
|
||||
private volatile int _blockFrames; // frames per DSP block, as actually delivered
|
||||
private static bool _reportedBlock; // once per session, not once per BingBong
|
||||
private static int _liveSources; // how many voice sources currently exist
|
||||
private static int _playingSources; // how many are actually enabled and audible
|
||||
private static readonly System.Collections.Generic.List<BingBongVoiceSource> _active = new();
|
||||
private float _gainStep = 0.0008f;
|
||||
private long _lastWritePos;
|
||||
private float _lastRateTime;
|
||||
private float _appliedRange = -1f;
|
||||
|
||||
private BingBongMouth? _mouth;
|
||||
private int _mouthHash;
|
||||
private float _mouthValue;
|
||||
private int _mouthStrikes;
|
||||
private bool _mouthDisabled;
|
||||
|
||||
/// Set by the duck patch. Live voice is attenuated until this time.
|
||||
public float DuckUntil { get; set; }
|
||||
|
||||
/// False for the non-positional debug instance, which sits on a DontDestroyOnLoad object at the
|
||||
/// origin and would otherwise look like a BingBong standing next to you.
|
||||
public bool IsPositional { get; private set; }
|
||||
|
||||
/// Enabled, positional sources. Main thread only.
|
||||
public static System.Collections.Generic.IReadOnlyList<BingBongVoiceSource> Active => _active;
|
||||
|
||||
public static float MinDistanceFor(float range) => Mathf.Max(1f, range / 10f);
|
||||
|
||||
/// <summary>
|
||||
/// Unity's Logarithmic rolloff expressed arithmetically, so the uplink can attenuate a player's
|
||||
/// microphone by exactly the curve BingBong's own output is heard through.
|
||||
/// </summary>
|
||||
public static float RangeGain(float distance, float range)
|
||||
{
|
||||
float min = MinDistanceFor(range);
|
||||
return Mathf.Clamp01(min / Mathf.Max(min, distance));
|
||||
}
|
||||
|
||||
public float Amplitude => _amplitude;
|
||||
|
||||
private const string ChildName = "BingBongVoice";
|
||||
|
||||
/// <summary>
|
||||
/// Always lives on its own child GameObject. OnAudioFilterRead hooks the DSP chain of the
|
||||
/// AudioSource sharing its GameObject, and BingBong already carries the vanilla voice-line
|
||||
/// source on the item root -- putting ours alongside it meant our filter ran on *their*
|
||||
/// source, so the live stream was only audible while a canned line played and cut out
|
||||
/// whenever that source stopped. A child of our own keeps the two chains separate, and
|
||||
/// still follows the item since it is parented to it.
|
||||
/// </summary>
|
||||
public static BingBongVoiceSource Attach(GameObject target, AudioSource? template)
|
||||
{
|
||||
if (target.TryGetComponent(out BingBongVoiceSource existing)) return existing;
|
||||
Transform existing = target.transform.Find(ChildName);
|
||||
if (existing != null && existing.TryGetComponent(out BingBongVoiceSource found)) return found;
|
||||
|
||||
BingBongVoiceSource voice = target.AddComponent<BingBongVoiceSource>();
|
||||
GameObject holder = new(ChildName);
|
||||
holder.transform.SetParent(target.transform, worldPositionStays: false);
|
||||
|
||||
BingBongVoiceSource voice = holder.AddComponent<BingBongVoiceSource>();
|
||||
voice.Configure(template);
|
||||
return voice;
|
||||
}
|
||||
|
||||
/// How many voice sources exist right now. Should be one per BingBong in the scene; if it
|
||||
/// climbs with every grab then dead instances are still playing and everything is audible at
|
||||
/// once, which reads as "no spatialisation" just as convincingly as a broken panner does.
|
||||
public static int LiveSources => _liveSources;
|
||||
|
||||
/// Enabled instances. PEAK keeps several BingBong objects around (held and prop variants), so
|
||||
/// what matters is not how many exist but how many are audible at once -- more than one would
|
||||
/// sum to several times the volume and smear the apparent position.
|
||||
public static int PlayingSources => _playingSources;
|
||||
|
||||
/// Lets the attach patch hand us BingBong's mouth animator. Kept separate from Attach so a
|
||||
/// change to BingBongMouth can only break the animation, never the audio.
|
||||
public void BindMouth(BingBongMouth mouth)
|
||||
{
|
||||
_mouth = mouth;
|
||||
string param = string.IsNullOrEmpty(mouth.animValue) ? "Mouth Blend" : mouth.animValue;
|
||||
_mouthHash = Animator.StringToHash(param);
|
||||
Debug.Log($"[BingBongVoice] Mouth bound: parameter '{param}', maxMouthOpen={mouth.maxMouthOpen}.");
|
||||
}
|
||||
|
||||
public static bool IsAttached(GameObject target) => Find(target) != null;
|
||||
|
||||
public static BingBongVoiceSource? Find(GameObject target)
|
||||
{
|
||||
Transform existing = target.transform.Find(ChildName);
|
||||
return existing != null && existing.TryGetComponent(out BingBongVoiceSource found) ? found : null;
|
||||
}
|
||||
|
||||
private void Configure(AudioSource? template)
|
||||
{
|
||||
_outputRate = AudioSettings.outputSampleRate;
|
||||
_gainStep = 1f / (_outputRate * 0.015f); // ~15ms attack whatever the device runs at
|
||||
|
||||
// Silent carrier: the filter only runs while the source is playing something. Its
|
||||
// contents are irrelevant, OnAudioFilterRead overwrites the buffer wholesale.
|
||||
// Carrier of constant 1.0, not silence, and the filter MULTIPLIES rather than overwrites.
|
||||
// Unity applies 3D panning and distance attenuation as a per-channel gain; if that runs
|
||||
// before custom filters then the buffer handed to us already carries that gain, and
|
||||
// overwriting it threw the spatialisation away (the voice was audible at full volume from
|
||||
// anywhere). Multiplying a unit carrier reapplies it. If the panner instead runs after
|
||||
// the filter, the incoming value is simply 1.0 and multiplying changes nothing -- so this
|
||||
// is correct either way round, without having to determine which order Unity uses.
|
||||
_carrier = AudioClip.Create($"BingBongVoiceCarrier_{GetInstanceID()}", 4096, 1, _outputRate, false);
|
||||
float[] unit = new float[4096];
|
||||
for (int i = 0; i < unit.Length; i++) unit[i] = 1f;
|
||||
_carrier.SetData(unit, 0);
|
||||
|
||||
_source = gameObject.AddComponent<AudioSource>();
|
||||
_source.clip = _carrier;
|
||||
@@ -78,8 +159,8 @@ public sealed class BingBongVoiceSource : MonoBehaviour
|
||||
// voice attenuates exactly like BingBong's own lines. Routing through its mixer
|
||||
// group also means the game's volume sliders apply for free.
|
||||
_source.spatialBlend = template.spatialBlend;
|
||||
_source.minDistance = template.minDistance;
|
||||
_source.maxDistance = template.maxDistance;
|
||||
|
||||
ApplyRange(ConfigHandler.BingBongVoiceMaxDistance.Value);
|
||||
_source.rolloffMode = template.rolloffMode;
|
||||
_source.dopplerLevel = template.dopplerLevel;
|
||||
_source.spread = template.spread;
|
||||
@@ -89,11 +170,29 @@ public sealed class BingBongVoiceSource : MonoBehaviour
|
||||
{
|
||||
_source.spatialBlend = 0f; // debug instance: non-positional, always audible
|
||||
}
|
||||
IsPositional = template != null;
|
||||
|
||||
_source.Play();
|
||||
_liveSources++;
|
||||
AudioSettings.OnAudioConfigurationChanged += OnAudioConfigChanged;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Range needs both distances, not just maxDistance. Under Logarithmic rolloff the falloff is
|
||||
/// governed by minDistance -- volume goes roughly as minDistance/distance -- and maxDistance
|
||||
/// only marks where attenuation stops changing. Vanilla's minDistance of 1 means the sound is
|
||||
/// at 1/30 volume by 30m, so raising maxDistance alone did nothing at all. Scaling both
|
||||
/// together stretches the same curve over the range asked for: full volume within a tenth of
|
||||
/// it, and still audible out at the limit.
|
||||
/// </summary>
|
||||
private void ApplyRange(float range)
|
||||
{
|
||||
if (_source == null) return;
|
||||
_source.maxDistance = range;
|
||||
_source.minDistance = Mathf.Max(1f, range / 10f);
|
||||
_appliedRange = range;
|
||||
}
|
||||
|
||||
private void OnAudioConfigChanged(bool deviceChanged)
|
||||
{
|
||||
// Owning the resample ratio ourselves makes this a one-line update instead of a rebuild.
|
||||
@@ -104,12 +203,35 @@ public sealed class BingBongVoiceSource : MonoBehaviour
|
||||
|
||||
private void Update()
|
||||
{
|
||||
_stream = BingBongVoiceModule.Instance?.Client?.Stream;
|
||||
BingBongVoiceModule? module = BingBongVoiceModule.Instance;
|
||||
_stream = module?.LoopbackStream ?? module?.Client?.Stream;
|
||||
|
||||
_targetGain = Time.time < DuckUntil
|
||||
? ConfigHandler.BingBongVoiceDuckVolume.Value
|
||||
: ConfigHandler.BingBongVoiceVolume.Value;
|
||||
|
||||
// Picked up live, like the volumes are, so the range can be tuned without re-spawning
|
||||
// every BingBong in the scene.
|
||||
float range = ConfigHandler.BingBongVoiceMaxDistance.Value;
|
||||
if (!Mathf.Approximately(_appliedRange, range)) ApplyRange(range);
|
||||
|
||||
if (_mouth != null && !_mouthDisabled)
|
||||
{
|
||||
try
|
||||
{
|
||||
DriveMouth();
|
||||
}
|
||||
catch (Exception e)
|
||||
{
|
||||
Debug.LogError($"[BingBongVoice] Mouth animation failed: {e}");
|
||||
if (++_mouthStrikes >= 3)
|
||||
{
|
||||
_mouthDisabled = true;
|
||||
Debug.LogError("[BingBongVoice] Mouth animation disabled for this session.");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (!_reportedBlock && _blockFrames > 0)
|
||||
{
|
||||
_reportedBlock = true;
|
||||
@@ -142,6 +264,50 @@ public sealed class BingBongVoiceSource : MonoBehaviour
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Drives BingBong's mouth from the audio we are actually playing, so it works on every
|
||||
/// client rather than only the speaker's.
|
||||
///
|
||||
/// BingBongMouth.Update only writes the animator parameter while canPlay is true, so holding
|
||||
/// canPlay false is enough to take over cleanly -- no fighting over script execution order.
|
||||
/// During a duck we stop forcing it and stop writing, which lets OnAsk -> CreateCurveMap set
|
||||
/// canPlay itself and animate the canned line off its baked curve exactly as shipped.
|
||||
/// </summary>
|
||||
[MethodImpl(MethodImplOptions.NoInlining)]
|
||||
private void DriveMouth()
|
||||
{
|
||||
BingBongMouth mouth = _mouth!;
|
||||
if (mouth == null || mouth.animator == null) return;
|
||||
|
||||
if (Time.time < DuckUntil)
|
||||
{
|
||||
// Vanilla owns the mouth for the duration of its own voice line.
|
||||
_mouthValue = 0f;
|
||||
return;
|
||||
}
|
||||
|
||||
if (_stream != null)
|
||||
{
|
||||
mouth.canPlay = false;
|
||||
|
||||
// Vanilla's units exactly: BingBongMouth bakes a curve of MEAN |sample| per window and
|
||||
// feeds the animator curveMap.Evaluate(t) * maxMouthOpen, so maxMouthOpen is the factor
|
||||
// that lifts those small means into the animator's range. Passing an already-normalised
|
||||
// 0-1 value and then multiplying by it saturated the parameter, which held the jaw wide
|
||||
// open for anything above near-silence. Matching the scale means it behaves like a
|
||||
// canned line does, whatever maxMouthOpen happens to be.
|
||||
float target = _rawAmplitude * mouth.maxMouthOpen;
|
||||
_mouthValue = Mathf.Lerp(_mouthValue, target, Time.deltaTime * 25f);
|
||||
mouth.animator.SetFloat(_mouthHash, _mouthValue);
|
||||
}
|
||||
else if (_mouthValue > 0.001f)
|
||||
{
|
||||
// Disconnected: close it rather than leaving the jaw hanging where it stopped.
|
||||
_mouthValue = Mathf.Lerp(_mouthValue, 0f, Time.deltaTime * 10f);
|
||||
mouth.animator.SetFloat(_mouthHash, _mouthValue);
|
||||
}
|
||||
}
|
||||
|
||||
// How much audio to hold before playing. One DSP block is consumed on every call, so the
|
||||
// floor is (target - block); keeping two blocks plus the configured margin leaves that floor
|
||||
// comfortably above the point where a read cannot be satisfied.
|
||||
@@ -233,6 +399,7 @@ public sealed class BingBongVoiceSource : MonoBehaviour
|
||||
private void Render(float[] data, VoiceRingBuffer ring, int channels, int frames, double ratio, bool fadeOut)
|
||||
{
|
||||
float peak = 0f;
|
||||
float rawSum = 0f;
|
||||
float fadeStep = fadeOut && frames > 0 ? _gain / frames : 0f;
|
||||
|
||||
for (int f = 0; f < frames; f++)
|
||||
@@ -246,15 +413,18 @@ public sealed class BingBongVoiceSource : MonoBehaviour
|
||||
|
||||
float outSample = sample * _gain;
|
||||
int baseIndex = f * channels;
|
||||
for (int c = 0; c < channels; c++) data[baseIndex + c] = outSample;
|
||||
for (int c = 0; c < channels; c++) data[baseIndex + c] *= outSample;
|
||||
|
||||
float abs = outSample < 0f ? -outSample : outSample;
|
||||
if (abs > peak) peak = abs;
|
||||
|
||||
rawSum += sample < 0f ? -sample : sample;
|
||||
|
||||
_read += ratio;
|
||||
}
|
||||
|
||||
_amplitude = peak;
|
||||
_rawAmplitude = frames > 0 ? rawSum / frames : 0f;
|
||||
}
|
||||
|
||||
private void Silence(float[] data)
|
||||
@@ -262,6 +432,7 @@ public sealed class BingBongVoiceSource : MonoBehaviour
|
||||
Array.Clear(data, 0, data.Length);
|
||||
_gain = 0f;
|
||||
_amplitude = 0f;
|
||||
_rawAmplitude = 0f;
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------------------
|
||||
@@ -271,6 +442,8 @@ public sealed class BingBongVoiceSource : MonoBehaviour
|
||||
// The audio thread can and will run the filter after Destroy happens on the main thread,
|
||||
// so flag it dead first: that check is what makes the teardown safe.
|
||||
_dead = true;
|
||||
_active.Remove(this);
|
||||
_liveSources--;
|
||||
AudioSettings.OnAudioConfigurationChanged -= OnAudioConfigChanged;
|
||||
|
||||
if (_source != null)
|
||||
@@ -283,12 +456,16 @@ public sealed class BingBongVoiceSource : MonoBehaviour
|
||||
|
||||
private void OnDisable()
|
||||
{
|
||||
_active.Remove(this);
|
||||
_playingSources--;
|
||||
if (_source != null) _source.Stop();
|
||||
_state = State.Buffering;
|
||||
}
|
||||
|
||||
private void OnEnable()
|
||||
{
|
||||
if (!_active.Contains(this)) _active.Add(this);
|
||||
_playingSources++;
|
||||
// Rejoin live rather than trying to resume where we left off.
|
||||
_state = State.Buffering;
|
||||
if (_source != null && _carrier != null) _source.Play();
|
||||
|
||||
151
src/JordanMod/modules/bingbong_voice/MicUplinkSender.cs
Normal file
151
src/JordanMod/modules/bingbong_voice/MicUplinkSender.cs
Normal file
@@ -0,0 +1,151 @@
|
||||
using System;
|
||||
using System.Threading;
|
||||
|
||||
namespace JordanMod.Modules.BingBongVoice;
|
||||
|
||||
/// <summary>
|
||||
/// Frames captured microphone audio and puts it on the socket the client already holds.
|
||||
///
|
||||
/// Sends from a dedicated thread with the blocking <c>Send</c> rather than <c>SendAsync</c>:
|
||||
/// websocket-sharp's async path posts to the thread pool, so ordering across queued sends is not
|
||||
/// guaranteed, and reordered voice sounds far worse than dropped voice. One thread and a blocking
|
||||
/// send gives strict ordering for free.
|
||||
///
|
||||
/// There is deliberately no pacer. The microphone is the clock -- Photon hands us exactly the
|
||||
/// audio the hardware produced -- so anything that paced against a wall clock on top of that would
|
||||
/// only fight it and guarantee drift in one direction.
|
||||
/// </summary>
|
||||
public sealed class MicUplinkSender
|
||||
{
|
||||
private const byte Magic = 0xB2; // 0xB1 is downlink; a distinct magic makes a relay
|
||||
private const byte Version = 1; // mis-route unable to replay room audio into the room
|
||||
private const byte CodecPcm16 = 0;
|
||||
private const byte RateCode16k = 0;
|
||||
private const int HeaderSize = 12;
|
||||
private const int PayloadBytes = MicUplinkSource.FrameSamples * 2;
|
||||
private const int FrameBytes = HeaderSize + PayloadBytes;
|
||||
private const int RingSlots = 32; // ~640 ms; far more than a healthy socket ever needs
|
||||
|
||||
private readonly byte[][] _slots = new byte[RingSlots][];
|
||||
private readonly BingBongVoiceClient _client;
|
||||
private readonly SemaphoreSlim _signal = new(0);
|
||||
|
||||
private long _write;
|
||||
private long _read;
|
||||
private uint _seq;
|
||||
private uint _timestamp;
|
||||
|
||||
private Thread? _thread;
|
||||
private volatile bool _running;
|
||||
private int _dropped;
|
||||
private int _sent;
|
||||
private int _consecutiveFailures;
|
||||
private volatile bool _stopped;
|
||||
|
||||
public int Dropped => _dropped;
|
||||
public int Sent => _sent;
|
||||
|
||||
public MicUplinkSender(BingBongVoiceClient client)
|
||||
{
|
||||
_client = client;
|
||||
for (int i = 0; i < RingSlots; i++) _slots[i] = new byte[FrameBytes];
|
||||
}
|
||||
|
||||
public void Start()
|
||||
{
|
||||
if (_running) return;
|
||||
_running = true;
|
||||
_thread = new Thread(SendLoop) { IsBackground = true, Name = "BingBongUplink" };
|
||||
_thread.Start();
|
||||
}
|
||||
|
||||
public void Stop()
|
||||
{
|
||||
_running = false;
|
||||
_signal.Release();
|
||||
}
|
||||
|
||||
/// Producer side: called from the mic path. Writes the frame into a pre-allocated slot, so
|
||||
/// nothing is allocated per frame and the caller never blocks.
|
||||
public void Enqueue(short[] samples)
|
||||
{
|
||||
if (_stopped || !_running) return;
|
||||
|
||||
// Ring full means the socket is not draining. Drop the OLDEST and keep the newest: on a
|
||||
// live stream, stale audio is worth less than current audio, and blocking the producer
|
||||
// would stall Photon's audio path.
|
||||
if (_write - _read >= RingSlots)
|
||||
{
|
||||
Interlocked.Increment(ref _dropped);
|
||||
Interlocked.Increment(ref _read);
|
||||
}
|
||||
|
||||
byte[] slot = _slots[(int)(_write % RingSlots)];
|
||||
|
||||
slot[0] = Magic;
|
||||
slot[1] = Version;
|
||||
slot[2] = CodecPcm16;
|
||||
slot[3] = RateCode16k; // the relay overwrites the high nibble with our stream id
|
||||
WriteUInt32(slot, 4, _seq);
|
||||
WriteUInt32(slot, 8, _timestamp);
|
||||
|
||||
for (int i = 0; i < MicUplinkSource.FrameSamples; i++)
|
||||
{
|
||||
short s = samples[i];
|
||||
int b = HeaderSize + (i << 1);
|
||||
slot[b] = (byte)(s & 0xFF);
|
||||
slot[b + 1] = (byte)((s >> 8) & 0xFF);
|
||||
}
|
||||
|
||||
// Both counters advance only for frames we actually emit, so a gated pause leaves the
|
||||
// timeline contiguous and the receiver must not insert silence for it. The control frames
|
||||
// are what tell it to resync instead.
|
||||
_seq++;
|
||||
_timestamp += MicUplinkSource.FrameSamples;
|
||||
|
||||
Interlocked.Increment(ref _write);
|
||||
_signal.Release();
|
||||
}
|
||||
|
||||
private void SendLoop()
|
||||
{
|
||||
while (_running)
|
||||
{
|
||||
try
|
||||
{
|
||||
_signal.Wait(200);
|
||||
while (_read < Volatile.Read(ref _write))
|
||||
{
|
||||
byte[] slot = _slots[(int)(_read % RingSlots)];
|
||||
if (_client.TrySend(slot))
|
||||
{
|
||||
Interlocked.Increment(ref _sent);
|
||||
_consecutiveFailures = 0;
|
||||
}
|
||||
else if (++_consecutiveFailures >= 200)
|
||||
{
|
||||
// The socket has been refusing for four seconds. Stop rather than spin;
|
||||
// reconnection is the client's job and a second driver would fight it.
|
||||
_stopped = true;
|
||||
Debug.LogWarning("[BingBongVoice] Uplink stopped: the socket has been refusing frames. It will resume on the next connect.");
|
||||
return;
|
||||
}
|
||||
Interlocked.Increment(ref _read);
|
||||
}
|
||||
}
|
||||
catch (Exception e)
|
||||
{
|
||||
Debug.LogError($"[BingBongVoice] Uplink send loop failed: {e}");
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private static void WriteUInt32(byte[] buffer, int offset, uint value)
|
||||
{
|
||||
buffer[offset] = (byte)(value & 0xFF);
|
||||
buffer[offset + 1] = (byte)((value >> 8) & 0xFF);
|
||||
buffer[offset + 2] = (byte)((value >> 16) & 0xFF);
|
||||
buffer[offset + 3] = (byte)((value >> 24) & 0xFF);
|
||||
}
|
||||
}
|
||||
228
src/JordanMod/modules/bingbong_voice/MicUplinkSource.cs
Normal file
228
src/JordanMod/modules/bingbong_voice/MicUplinkSource.cs
Normal file
@@ -0,0 +1,228 @@
|
||||
using System;
|
||||
using System.Threading;
|
||||
using UnityEngine;
|
||||
|
||||
namespace JordanMod.Modules.BingBongVoice;
|
||||
|
||||
/// <summary>
|
||||
/// Turns the local microphone into 20 ms frames of 16 kHz mono PCM16, ready to go on the wire.
|
||||
///
|
||||
/// Fed by <see cref="BingBongMicTapPatch"/> from Photon's mic read, measured at 48000 Hz mono in
|
||||
/// 960-sample frames arriving 50 times a second on Unity's main thread -- so the microphone itself
|
||||
/// is the clock and there is deliberately no pacer here. A deadline pacer on top of a hardware
|
||||
/// clock would only fight it and guarantee drift in one direction.
|
||||
///
|
||||
/// Allocation-free after the first callback: this runs on the audio path of every frame.
|
||||
/// </summary>
|
||||
public sealed class MicUplinkSource
|
||||
{
|
||||
public const int WireRate = 16000;
|
||||
public const int FrameSamples = 320; // 20 ms at 16 kHz
|
||||
|
||||
private readonly short[] _frame = new short[FrameSamples];
|
||||
private int _frameFill;
|
||||
|
||||
// Scratch copy of the incoming buffer. We must never filter in place: that array belongs to
|
||||
// Photon and is encoded and sent to teammates immediately after this postfix returns, so
|
||||
// writing to it would corrupt everyone else's audio.
|
||||
private float[]? _work;
|
||||
|
||||
// Resampler state, carried across callbacks. _tail is the previous buffer's last sample and is
|
||||
// addressed as index -1, so the interpolation spans the seam instead of skipping it.
|
||||
private double _read;
|
||||
private float _tail;
|
||||
|
||||
// Five cascaded one-pole low-passes. Decimating 48k to 16k without one folds everything above
|
||||
// 8 kHz down into the speech band, which sounds harsh and cannot be undone afterwards. Two
|
||||
// poles measured only -14 dB of rejection at 12 kHz; five at 7 kHz gives -25 dB while still
|
||||
// passing 95% at 1 kHz, where speech energy actually is.
|
||||
private const int FilterPoles = 5;
|
||||
private readonly float[] _lp = new float[FilterPoles];
|
||||
private float _lpCoefficient = 1f;
|
||||
|
||||
private int _rate;
|
||||
private int _channels = 1;
|
||||
private int _sourceId = -1;
|
||||
|
||||
private float _gain;
|
||||
private readonly float _gainStep = 1f / (WireRate * 0.015f); // ~15 ms attack
|
||||
|
||||
private volatile float _targetGain;
|
||||
private volatile bool _gateOpen;
|
||||
private volatile float _lastFrameLevel;
|
||||
private volatile float _noiseFloorPublished;
|
||||
private long _framesProduced;
|
||||
|
||||
private float _noiseFloor;
|
||||
private int _holdFrames;
|
||||
|
||||
/// Set by the proximity logic on the main thread. 0 mutes; the ramp keeps it click-free.
|
||||
public void SetTargetGain(float gain) => _targetGain = Mathf.Clamp(gain, 0f, 8f);
|
||||
|
||||
/// Whether the uplink may transmit at all (range, push-to-talk, speaker present).
|
||||
public void SetGate(bool open) => _gateOpen = open;
|
||||
|
||||
/// Mean |sample| of the last completed frame, for the level indicator.
|
||||
public float Level => _lastFrameLevel;
|
||||
|
||||
public long FramesProduced => Interlocked.Read(ref _framesProduced);
|
||||
|
||||
/// Raised for each completed 20 ms frame that passes the gate. Assigned once, never per frame.
|
||||
public Action<short[]>? OnFrame;
|
||||
|
||||
/// <summary>
|
||||
/// Emit gated frames as silence rather than dropping them, keeping the output continuous.
|
||||
///
|
||||
/// Off for the real uplink: the whole point of the gate is not to spend bandwidth on silence,
|
||||
/// and the receiver is told to freeze its timeline instead. On for loopback, where the frames
|
||||
/// feed a jitter-buffered player with no control channel -- dropping 70% of them starves it
|
||||
/// into permanent rebuffering, which sounds like the capture path is broken when it is fine.
|
||||
/// </summary>
|
||||
public bool EmitSilenceWhenGated;
|
||||
|
||||
private readonly short[] _silence = new short[FrameSamples];
|
||||
|
||||
public void OnMicFrame(float[] buffer, int rate, int channels, int sourceId)
|
||||
{
|
||||
if (buffer == null || buffer.Length == 0 || rate <= 0) return;
|
||||
|
||||
// The mic device can change mid-session (CharacterVoiceHandler reassigns the recorder's
|
||||
// device when the setting changes), rebuilding MicWrapper with a possibly different rate.
|
||||
// Starting the resampler over is cheaper and safer than trying to splice.
|
||||
if (rate != _rate || channels != _channels || sourceId != _sourceId)
|
||||
{
|
||||
Reset(rate, channels, sourceId);
|
||||
Debug.Log($"[BingBongVoice] Uplink capture at {rate}Hz x{channels}, resampling {(double)rate / WireRate:F2}:1 down to {WireRate}Hz.");
|
||||
}
|
||||
|
||||
int count = _channels > 1 ? buffer.Length / _channels : buffer.Length;
|
||||
if (count < 2) return;
|
||||
if (_work == null || _work.Length < count) _work = new float[count];
|
||||
|
||||
// Downmix and anti-alias in one pass into our own buffer. The filter has to see every
|
||||
// input sample, not just the ones the cursor lands on, or it is not an anti-alias filter.
|
||||
bool filter = _lpCoefficient < 1f;
|
||||
for (int i = 0; i < count; i++)
|
||||
{
|
||||
float x;
|
||||
if (_channels > 1)
|
||||
{
|
||||
float sum = 0f;
|
||||
int b = i * _channels;
|
||||
for (int c = 0; c < _channels; c++) sum += buffer[b + c];
|
||||
x = sum / _channels;
|
||||
}
|
||||
else x = buffer[i];
|
||||
|
||||
if (filter)
|
||||
{
|
||||
for (int p = 0; p < FilterPoles; p++)
|
||||
{
|
||||
_lp[p] += (x - _lp[p]) * _lpCoefficient;
|
||||
x = _lp[p];
|
||||
}
|
||||
}
|
||||
_work[i] = x;
|
||||
}
|
||||
|
||||
Resample(_work, count);
|
||||
}
|
||||
|
||||
private void Reset(int rate, int channels, int sourceId)
|
||||
{
|
||||
_rate = rate;
|
||||
_channels = Math.Max(1, channels);
|
||||
_sourceId = sourceId;
|
||||
_read = 0;
|
||||
_tail = 0f;
|
||||
Array.Clear(_lp, 0, _lp.Length);
|
||||
_frameFill = 0;
|
||||
_gain = 0f;
|
||||
_noiseFloor = 0f;
|
||||
_holdFrames = 0;
|
||||
|
||||
// One-pole cutoff at ~6.5 kHz, safely under the 8 kHz Nyquist of the 16 kHz wire rate.
|
||||
_lpCoefficient = rate > WireRate ? 1f - Mathf.Exp(-2f * Mathf.PI * 7000f / rate) : 1f;
|
||||
}
|
||||
|
||||
private void Resample(float[] work, int count)
|
||||
{
|
||||
double ratio = (double)_rate / WireRate;
|
||||
double pos = _read; // may be negative; -1 addresses the carried tail sample
|
||||
|
||||
while (true)
|
||||
{
|
||||
int i0 = (int)Math.Floor(pos);
|
||||
if (i0 + 1 >= count) break;
|
||||
|
||||
float a = i0 < 0 ? _tail : work[i0];
|
||||
float b = work[i0 + 1];
|
||||
float frac = (float)(pos - i0);
|
||||
|
||||
_gain += (_targetGain - _gain) * _gainStep;
|
||||
Push((a * (1f - frac) + b * frac) * _gain);
|
||||
|
||||
pos += ratio;
|
||||
}
|
||||
|
||||
_tail = work[count - 1];
|
||||
_read = pos - count; // carries the fractional remainder to the next buffer
|
||||
if (_read < -1) _read = -1;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Adaptive gate. A fixed threshold cannot work here: measured on real hardware, a silent but
|
||||
/// noisy room read louder (mean 0.0088) than the same person actually speaking in a quiet one
|
||||
/// (0.0018), so no single level separates speech from silence across setups. Instead track the
|
||||
/// background and require speech to stand above it.
|
||||
///
|
||||
/// The floor follows quiet quickly and loud very slowly, so it settles on the room rather than
|
||||
/// on the voice. A hold keeps the gate open through the gaps between words.
|
||||
/// </summary>
|
||||
private bool PassesNoiseGate(float level)
|
||||
{
|
||||
if (_noiseFloor <= 0f) _noiseFloor = level;
|
||||
else if (level < _noiseFloor) _noiseFloor += (level - _noiseFloor) * 0.25f; // drop fast
|
||||
else _noiseFloor += (level - _noiseFloor) * 0.0008f; // rise slowly
|
||||
_noiseFloorPublished = _noiseFloor;
|
||||
|
||||
// Never transmit below the absolute floor, whatever the adaptive part decides.
|
||||
if (level < Mathf.Pow(10f, ConfigHandler.BingBongVoiceUplinkNoiseGateDb.Value / 20f)) return false;
|
||||
|
||||
float open = _noiseFloor * ConfigHandler.BingBongVoiceUplinkGateSensitivity.Value;
|
||||
float close = open * 0.6f; // hysteresis, so a steady voice cannot chatter the gate
|
||||
|
||||
if (level >= open) _holdFrames = 15; // ~300 ms, bridges the gaps between words
|
||||
else if (level < close && _holdFrames > 0) _holdFrames--;
|
||||
|
||||
return _holdFrames > 0;
|
||||
}
|
||||
|
||||
/// The learned background level, for diagnostics.
|
||||
public float NoiseFloor => _noiseFloorPublished;
|
||||
|
||||
private void Push(float sample)
|
||||
{
|
||||
if (sample > 1f) sample = 1f;
|
||||
else if (sample < -1f) sample = -1f;
|
||||
|
||||
_frame[_frameFill++] = (short)(sample * 32767f);
|
||||
if (_frameFill < FrameSamples) return;
|
||||
_frameFill = 0;
|
||||
|
||||
float sum = 0f;
|
||||
for (int i = 0; i < FrameSamples; i++)
|
||||
{
|
||||
int v = _frame[i];
|
||||
sum += v < 0 ? -v : v;
|
||||
}
|
||||
_lastFrameLevel = sum / FrameSamples / 32768f;
|
||||
Interlocked.Increment(ref _framesProduced);
|
||||
|
||||
// Gated after framing, so the level meter keeps reading while muted and the frame counter
|
||||
// stays a measure of capture rather than of transmission.
|
||||
bool pass = _gateOpen && PassesNoiseGate(_lastFrameLevel);
|
||||
if (pass) OnFrame?.Invoke(_frame);
|
||||
else if (EmitSilenceWhenGated) OnFrame?.Invoke(_silence);
|
||||
}
|
||||
}
|
||||
@@ -53,6 +53,16 @@ public sealed class VoiceRingBuffer
|
||||
Volatile.Write(ref _write, w + samples); // publish only once the data is really there
|
||||
}
|
||||
|
||||
/// Producer thread only. Same as WritePcm16 but from samples we produced ourselves, so the
|
||||
/// loopback path does not have to encode to bytes and immediately decode again.
|
||||
public void WritePcm16Samples(short[] src, int count)
|
||||
{
|
||||
if (count <= 0) return;
|
||||
long w = _write;
|
||||
for (int i = 0; i < count; i++) _buf[(int)((w + i) & _mask)] = src[i] * (1f / 32768f);
|
||||
Volatile.Write(ref _write, w + count);
|
||||
}
|
||||
|
||||
/// Producer thread only. Used to reproduce an exact gap the sender told us about, so a
|
||||
/// dropped frame costs its own duration rather than shifting everything after it.
|
||||
public void WriteSilence(int samples)
|
||||
|
||||
156
src/JordanMod/patches/BingBongMicTapPatch.cs
Normal file
156
src/JordanMod/patches/BingBongMicTapPatch.cs
Normal file
@@ -0,0 +1,156 @@
|
||||
using System;
|
||||
using System.Runtime.CompilerServices;
|
||||
using System.Threading;
|
||||
using HarmonyLib;
|
||||
using Photon.Voice.Unity;
|
||||
|
||||
namespace JordanMod.Modules.BingBongVoice;
|
||||
|
||||
/// <summary>
|
||||
/// Taps the local microphone for the voice uplink.
|
||||
///
|
||||
/// Feeds <see cref="MicUplinkSource"/>, and on the first five seconds of a session also logs a
|
||||
/// one-off measurement of the capture conditions. That measurement came first and confirmed, on
|
||||
/// real hardware, the assumptions the design rests on — 48000 Hz mono, 960-sample frames, 50 a
|
||||
/// second, on Unity's main thread. It is kept because a mic device change or a Photon update could
|
||||
/// invalidate any of them, and a wrong premise here is silent rather than loud.
|
||||
///
|
||||
/// What it settles:
|
||||
/// - Which thread the mic callback arrives on (statically it looked like Unity's main thread via
|
||||
/// VoiceConnection.Update, but LoadBalancingTransport.Service also reaches VoiceClient.Service
|
||||
/// and that path could not be traced).
|
||||
/// - The device's real sample rate and channel count, which drive resampling and downmixing.
|
||||
/// MicWrapper is authoritative here; Recorder.SamplingRate is the *encoder* rate and can differ.
|
||||
/// - The frame length, and specifically whether it exceeds 256. Photon's MicrophoneRelay.SendMic
|
||||
/// copies only the first 256 samples before invoking its listeners, which is why that API — the
|
||||
/// obvious one to use — cannot carry continuous audio. If frames really are longer than 256,
|
||||
/// that finding is confirmed and patching Read here is the right call.
|
||||
/// - Signal levels, to calibrate the noise gate. The tap sits upstream of Photon's WebRtcAudioDsp
|
||||
/// (AEC/NS/AGC attach as post-processors), so this is the raw microphone.
|
||||
///
|
||||
/// Structured like BingBongVoicePatch: the patch body is only a try/catch around a NoInlining
|
||||
/// method, with three strikes then permanent disable. That matters more here than anywhere else in
|
||||
/// the mod — this sits on Photon's per-frame audio path, so an exception escaping would break all
|
||||
/// voice chat, not just this feature.
|
||||
/// </summary>
|
||||
public class BingBongMicTapPatch
|
||||
{
|
||||
private static int _strikes;
|
||||
private static bool _disabled;
|
||||
|
||||
private static int _mainThreadId = -1;
|
||||
private static int _observedThreadId = -1;
|
||||
|
||||
private static int _calls;
|
||||
private static int _minLength = int.MaxValue;
|
||||
private static int _maxLength;
|
||||
private static int _rate;
|
||||
private static int _channels;
|
||||
private static float _peak;
|
||||
private static double _sumAbs;
|
||||
private static long _samples;
|
||||
private static float _firstCallTime;
|
||||
private static bool _reported;
|
||||
private static bool _waitedForMic;
|
||||
|
||||
/// Called from the module on the main thread, so the callback's thread can be compared to it.
|
||||
public static void NoteMainThread() => _mainThreadId = Thread.CurrentThread.ManagedThreadId;
|
||||
|
||||
[HarmonyPatch(typeof(MicWrapper), nameof(MicWrapper.Read))]
|
||||
[HarmonyPostfix]
|
||||
static void ReadPostfix(MicWrapper __instance, float[] __0, bool __result)
|
||||
{
|
||||
if (_disabled || !__result) return;
|
||||
try
|
||||
{
|
||||
if (!_reported) Measure(__instance, __0);
|
||||
Feed(__instance, __0);
|
||||
}
|
||||
catch (Exception e)
|
||||
{
|
||||
Fail(e);
|
||||
}
|
||||
}
|
||||
|
||||
// The only method here that touches Photon or Unity types, kept separate and un-inlined so a
|
||||
// member that changed shape throws inside the caller's try rather than while JITing the patch.
|
||||
[MethodImpl(MethodImplOptions.NoInlining)]
|
||||
private static void Measure(MicWrapper mic, float[] buffer)
|
||||
{
|
||||
if (buffer == null || buffer.Length == 0) return;
|
||||
|
||||
if (_calls == 0)
|
||||
{
|
||||
_observedThreadId = Thread.CurrentThread.ManagedThreadId;
|
||||
_firstCallTime = UnityEngine.Time.realtimeSinceStartup;
|
||||
_rate = mic.SamplingRate;
|
||||
_channels = mic.Channels;
|
||||
}
|
||||
|
||||
_calls++;
|
||||
if (buffer.Length < _minLength) _minLength = buffer.Length;
|
||||
if (buffer.Length > _maxLength) _maxLength = buffer.Length;
|
||||
|
||||
// Sample sparsely: this runs on an audio-adjacent hot path and a full scan every frame
|
||||
// would be a cost we are not here to pay.
|
||||
for (int i = 0; i < buffer.Length; i += 7)
|
||||
{
|
||||
float abs = buffer[i] < 0f ? -buffer[i] : buffer[i];
|
||||
if (abs > _peak) _peak = abs;
|
||||
_sumAbs += abs;
|
||||
_samples++;
|
||||
}
|
||||
|
||||
if (_calls < 250) return; // ~5 seconds at 50 frames/sec
|
||||
|
||||
// Photon binds the real capture device a few seconds after joining -- until then it reports
|
||||
// "[Default] (-128)" and hands out silence. Measuring that window produces an all-zero
|
||||
// report that looks exactly like a broken microphone, so start over instead.
|
||||
if (_peak <= 0f)
|
||||
{
|
||||
_calls = 0;
|
||||
_sumAbs = 0;
|
||||
_samples = 0;
|
||||
_minLength = int.MaxValue;
|
||||
_maxLength = 0;
|
||||
if (!_waitedForMic)
|
||||
{
|
||||
_waitedForMic = true;
|
||||
Debug.Log("[BingBongVoice] Microphone is not producing yet (Photon has not bound the device); will measure again shortly.");
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
_reported = true;
|
||||
float elapsed = UnityEngine.Time.realtimeSinceStartup - _firstCallTime;
|
||||
float mean = _samples > 0 ? (float)(_sumAbs / _samples) : 0f;
|
||||
bool onMainThread = _observedThreadId == _mainThreadId;
|
||||
bool longerThan256 = _maxLength > 256;
|
||||
|
||||
Debug.Log(
|
||||
$"[BingBongVoice] MIC TAP (stage 0): {_calls} frames in {elapsed:F1}s ({_calls / Math.Max(0.01f, elapsed):F0}/s), " +
|
||||
$"rate={_rate}Hz channels={_channels}, frame length min={_minLength} max={_maxLength}, " +
|
||||
$"peak={_peak:F3} mean|s|={mean:F4}, thread={_observedThreadId} (main={_mainThreadId}, {(onMainThread ? "MAIN" : "NOT main")}).");
|
||||
|
||||
Debug.Log(longerThan256
|
||||
? $"[BingBongVoice] Frames are {_maxLength} samples, longer than the 256 MicrophoneRelay forwards - confirmed, that API cannot carry audio and patching MicWrapper.Read is correct."
|
||||
: $"[BingBongVoice] Frames are only {_maxLength} samples, so MicrophoneRelay would NOT have truncated them. Revisit the capture design before building on this patch.");
|
||||
}
|
||||
|
||||
// The only path that reaches the uplink. Separate and un-inlined for the same reason Measure
|
||||
// is: a Photon member that changed shape must throw inside the caller's try.
|
||||
[MethodImpl(MethodImplOptions.NoInlining)]
|
||||
private static void Feed(MicWrapper mic, float[] buffer)
|
||||
{
|
||||
MicUplinkSource? source = BingBongVoiceModule.Instance?.Uplink;
|
||||
source?.OnMicFrame(buffer, mic.SamplingRate, mic.Channels, mic.GetHashCode());
|
||||
}
|
||||
|
||||
private static void Fail(Exception e)
|
||||
{
|
||||
Debug.LogError($"[BingBongVoice] Mic tap failed: {e}");
|
||||
if (++_strikes < 3) return;
|
||||
_disabled = true;
|
||||
Debug.LogError("[BingBongVoice] Mic tap disabled for this session after repeated failures.");
|
||||
}
|
||||
}
|
||||
135
src/JordanMod/patches/BingBongVoicePatch.cs
Normal file
135
src/JordanMod/patches/BingBongVoicePatch.cs
Normal file
@@ -0,0 +1,135 @@
|
||||
using System;
|
||||
using System.Runtime.CompilerServices;
|
||||
using HarmonyLib;
|
||||
using UnityEngine;
|
||||
|
||||
namespace JordanMod.Modules.BingBongVoice;
|
||||
|
||||
/// <summary>
|
||||
/// Gives every BingBong a live-voice source as it spawns.
|
||||
///
|
||||
/// Structured defensively on purpose. A postfix that throws propagates out of the method it
|
||||
/// patched, and because Mono resolves a method's call targets when it JITs that method, a game
|
||||
/// API that changed shape throws before any guard inside can run -- that is how a spawner
|
||||
/// postfix once took down every SingleItemSpawner in the level (see 0.1.13). So the patch body
|
||||
/// is nothing but a try/catch around a separate non-inlined method, and three failures disable
|
||||
/// it for the session. A broken voice feature must never cost anyone their BingBong.
|
||||
/// </summary>
|
||||
public class BingBongVoicePatch
|
||||
{
|
||||
private static int _strikes;
|
||||
private static bool _disabled;
|
||||
private static int _attachCount;
|
||||
|
||||
private static int _duckStrikes;
|
||||
private static bool _duckDisabled;
|
||||
|
||||
[HarmonyPatch(typeof(Item), "Start")]
|
||||
[HarmonyPostfix]
|
||||
static void ItemStartPostfix(Item __instance)
|
||||
{
|
||||
if (_disabled) return;
|
||||
try
|
||||
{
|
||||
AttachIfBingBong(__instance);
|
||||
}
|
||||
catch (Exception e)
|
||||
{
|
||||
Fail(e);
|
||||
}
|
||||
}
|
||||
|
||||
// The only method here that touches game APIs, kept separate and un-inlined so that a
|
||||
// missing member throws inside the caller's try rather than while JITing the patch itself.
|
||||
[MethodImpl(MethodImplOptions.NoInlining)]
|
||||
private static void AttachIfBingBong(Item item)
|
||||
{
|
||||
if (item == null || item.UIData == null) return;
|
||||
if (!item.itemTags.HasFlag(Item.ItemTags.BingBong)) return;
|
||||
if (BingBongVoiceSource.IsAttached(item.gameObject)) return;
|
||||
|
||||
// Copy the vanilla voice-line source's 3D setup so the live voice carries and attenuates
|
||||
// exactly like BingBong's own lines, and inherits its mixer group.
|
||||
AudioSource? template = item.TryGetComponent(out Action_AskBingBong ask) ? ask.source : null;
|
||||
|
||||
BingBongVoiceSource voice = BingBongVoiceSource.Attach(item.gameObject, template);
|
||||
|
||||
// The mouth animator lives on a child of the plush, not the item root.
|
||||
BingBongMouth mouth = item.GetComponentInChildren<BingBongMouth>(includeInactive: true);
|
||||
if (mouth != null) voice.BindMouth(mouth);
|
||||
|
||||
// Log the copied values: if the live voice turns out not to attenuate with distance,
|
||||
// this distinguishes "the copy failed" from "Unity spatialises before custom filters".
|
||||
// Counted because this should happen once per BingBong that exists, ever. If the number
|
||||
// climbs every time the item is grabbed or thrown then Item.Start is re-running on a
|
||||
// fresh instance, and the per-attach cost (AudioClip.Create, AddComponent) is the hitch.
|
||||
_attachCount++;
|
||||
if (template != null)
|
||||
Debug.Log($"[BingBongVoice] Attached to a BingBong (#{_attachCount}, {BingBongVoiceSource.LiveSources} live, {BingBongVoiceSource.PlayingSources} audible); spatialBlend={template.spatialBlend}, range={ConfigHandler.BingBongVoiceMaxDistance.Value}m, rolloff={template.rolloffMode}, mixer={(template.outputAudioMixerGroup != null ? template.outputAudioMixerGroup.name : "none")}.");
|
||||
else
|
||||
Debug.LogWarning("[BingBongVoice] Attached to a BingBong, but it has no Action_AskBingBong to copy 3D settings from - the voice will not be positional.");
|
||||
}
|
||||
|
||||
private static void Fail(Exception e)
|
||||
{
|
||||
Debug.LogError($"[BingBongVoice] Attach patch failed: {e}");
|
||||
if (++_strikes < 3) return;
|
||||
_disabled = true;
|
||||
Debug.LogError("[BingBongVoice] Attach patch disabled for this session after repeated failures.");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Ducks the live voice while one of BingBong's own lines plays, so the two never talk over
|
||||
/// each other. This is the RPC body, so it runs on every client and the duck stays in step
|
||||
/// across the lobby without any networking of our own.
|
||||
/// </summary>
|
||||
[HarmonyPatch(typeof(Action_AskBingBong), nameof(Action_AskBingBong.Ask))]
|
||||
[HarmonyPostfix]
|
||||
static void AskPostfix(Action_AskBingBong __instance, int index, bool spamming)
|
||||
{
|
||||
if (_duckDisabled) return;
|
||||
try
|
||||
{
|
||||
DuckForVoiceLine(__instance, index, spamming);
|
||||
}
|
||||
catch (Exception e)
|
||||
{
|
||||
Debug.LogError($"[BingBongVoice] Duck patch failed: {e}");
|
||||
if (++_duckStrikes < 3) return;
|
||||
_duckDisabled = true;
|
||||
Debug.LogError("[BingBongVoice] Duck patch disabled for this session after repeated failures.");
|
||||
}
|
||||
}
|
||||
|
||||
[MethodImpl(MethodImplOptions.NoInlining)]
|
||||
private static void DuckForVoiceLine(Action_AskBingBong ask, int index, bool spamming)
|
||||
{
|
||||
// Mirror vanilla's own early-outs, so we never duck for audio that is not going to play.
|
||||
// Ask() does nothing at all unless the item is held, and AskRoutine bails before playing
|
||||
// anything when it decides you are spamming the interaction.
|
||||
if (ask.item == null || ask.item.holderCharacter == null) return;
|
||||
if (spamming) return;
|
||||
|
||||
// RunAction picks the index locally and RPCs the raw number, while ReplaceBingBongModule
|
||||
// appends a response per loaded voice line -- so two players with different audio bank
|
||||
// state disagree on the array length. Vanilla throws in that case; we must not.
|
||||
Action_AskBingBong.BingBongResponse[] responses = ask.responses;
|
||||
if (responses == null || index < 0 || index >= responses.Length) return;
|
||||
|
||||
Action_AskBingBong.BingBongResponse response = responses[index];
|
||||
if (response?.sfx == null || response.sfx.clips == null || response.sfx.clips.Length == 0) return;
|
||||
|
||||
AudioClip clip = response.sfx.clips[0];
|
||||
if (clip == null) return;
|
||||
|
||||
BingBongVoiceSource? voice = BingBongVoiceSource.Find(ask.item.gameObject);
|
||||
if (voice == null) return;
|
||||
|
||||
// AskRoutine burns 0.5s in its update loop, stops the source, waits another 0.5s and only
|
||||
// then plays -- so audio starts about a second after this RPC lands. Ducking from now
|
||||
// rather than from the actual start reads as BingBong drawing breath, and removes a whole
|
||||
// class of "the duck began a few frames late and you heard a syllable" bugs.
|
||||
float duration = ConfigHandler.BingBongVoiceDuckLead.Value + clip.length + 0.2f;
|
||||
voice.DuckUntil = Math.Max(voice.DuckUntil, Time.time + duration);
|
||||
}
|
||||
}
|
||||
@@ -10,7 +10,14 @@ Requires `websockets` (pip install websockets).
|
||||
python tools/stub_relay.py serve # relay on ws://localhost:8787/voice/ws
|
||||
python tools/stub_relay.py tone # stream a 1 kHz sine into channel "test"
|
||||
python tools/stub_relay.py tone --sweep # sweep 200-2000 Hz (resampling bugs are obvious)
|
||||
python tools/stub_relay.py tone --speech # syllables and word gaps (for mouth animation)
|
||||
python tools/stub_relay.py listen # dump received frames, to verify fan-out
|
||||
python tools/stub_relay.py uplink # act as the speaker; capture the game's mic to WAV
|
||||
python tools/stub_relay.py uplink --speech # ... and send audio back at the same time
|
||||
|
||||
Only ONE speaker may hold a channel (newest wins), so `tone` and `uplink` cannot both run --
|
||||
use `uplink --tone/--sweep/--speech` to do both directions on a single connection, which is
|
||||
also what the real web app does.
|
||||
|
||||
Point the mod at it with, in BepInEx config:
|
||||
[BingBong Voice]
|
||||
@@ -48,6 +55,7 @@ PORT = 8787
|
||||
PATH = "/voice/ws"
|
||||
|
||||
MAGIC = 0xB1
|
||||
MAGIC_UPLINK = 0xB2 # game -> relay -> speaker; a distinct magic makes a loop impossible
|
||||
VERSION = 1
|
||||
CODEC_PCM16 = 0
|
||||
RATE_CODES = {16000: 0, 8000: 1, 24000: 2, 48000: 3}
|
||||
@@ -63,18 +71,49 @@ def pack_header(seq: int, ts: int, rate: int = 16000, codec: int = CODEC_PCM16)
|
||||
# ---------------------------------------------------------------- relay
|
||||
|
||||
channels: dict[str, dict] = defaultdict(lambda: {"speaker": None, "listeners": set()})
|
||||
names: dict = {}
|
||||
stream_ids: dict = {}
|
||||
uplinkers: dict = {} # ws -> bool, from the hello frame
|
||||
deny_uplink = False # --deny-uplink, to exercise the client's uplink_denied path
|
||||
|
||||
|
||||
def roster(channel: str) -> str:
|
||||
"""Who the relay believes is in this lobby -- the stub's stand-in for /voice/sessions."""
|
||||
return ", ".join(sorted(names.get(w, "?") for w in channels[channel]["listeners"])) or "(nobody)"
|
||||
|
||||
|
||||
def uplink_roster(channel: str) -> list:
|
||||
"""Who in this lobby says they may send a microphone. The real relay publishes this so the
|
||||
game can tell "the relay is dropping my audio" apart from "nobody is listening"."""
|
||||
return sorted(names.get(w, "?") for w in channels[channel]["listeners"] if uplinkers.get(w))
|
||||
|
||||
|
||||
def print_sessions():
|
||||
if not channels:
|
||||
print("[relay] no active sessions")
|
||||
return
|
||||
for chan, ch in channels.items():
|
||||
if not ch["listeners"] and ch["speaker"] is None:
|
||||
continue
|
||||
speaking = "speaking" if ch["speaker"] is not None else "idle"
|
||||
print(f"[relay] {chan} {len(ch['listeners'])} listening [{speaking}] {roster(chan)}")
|
||||
|
||||
|
||||
def parse_token(raw_path: str):
|
||||
"""token=<role>:<channel>. Returns (role, channel) or (None, None)."""
|
||||
q = parse_qs(urlparse(raw_path).query)
|
||||
token = (q.get("token") or [""])[0]
|
||||
if ":" not in token:
|
||||
if not token:
|
||||
return None, None
|
||||
role, _, channel = token.partition(":")
|
||||
if role not in ("listen", "speak") or not channel:
|
||||
return None, None
|
||||
return role, channel
|
||||
# "<role>:<channel>" is this stub's own convention, kept because it makes a speaker and a
|
||||
# listener easy to start from a shell. The deployed relay takes a bare channel id, which is
|
||||
# what the mod sends, so accept that too and default it to listening.
|
||||
if ":" in token:
|
||||
role, _, channel = token.partition(":")
|
||||
if role not in ("listen", "speak") or not channel:
|
||||
return None, None
|
||||
return role, channel
|
||||
return "listen", token
|
||||
|
||||
|
||||
async def handler(ws):
|
||||
@@ -110,12 +149,58 @@ async def handler(ws):
|
||||
return
|
||||
|
||||
ch["listeners"].add(ws)
|
||||
names[ws] = "?"
|
||||
stream_id = next((i for i in range(16) if i not in stream_ids.get(channel, {}).values()), 0)
|
||||
stream_ids.setdefault(channel, {})[ws] = stream_id
|
||||
print(f"[relay] listener joined '{channel}' -> {len(ch['listeners'])} listening")
|
||||
try:
|
||||
await ws.wait_closed()
|
||||
# Listeners send a hello frame naming the player. That is what the real relay turns into
|
||||
# its session directory, so the stub records it too and prints the roster.
|
||||
async for msg in ws:
|
||||
if isinstance(msg, bytes):
|
||||
# Uplink audio. Validate, stamp our own stream id so a client cannot claim
|
||||
# another's, and forward to the speaker ONLY -- never back to listeners, which is
|
||||
# half of what makes an audio loop structurally impossible.
|
||||
if deny_uplink:
|
||||
continue
|
||||
if len(msg) != 12 + 640 or msg[0] != MAGIC_UPLINK or msg[1] != VERSION:
|
||||
continue
|
||||
frame = bytearray(msg)
|
||||
frame[3] = (frame[3] & 0x0F) | (stream_id << 4)
|
||||
if ch["speaker"] is not None:
|
||||
await safe_send(ch["speaker"], bytes(frame))
|
||||
continue
|
||||
if not isinstance(msg, str):
|
||||
continue
|
||||
try:
|
||||
payload = json.loads(msg)
|
||||
except ValueError:
|
||||
continue
|
||||
if payload.get("type") == "hello":
|
||||
names[ws] = str(payload.get("player", "?"))
|
||||
uplinkers[ws] = bool(payload.get("uplink"))
|
||||
print(f"[relay] '{names[ws]}' registered in '{channel}' -> roster: {roster(channel)}")
|
||||
if uplinkers[ws] and deny_uplink:
|
||||
await safe_send(ws, json.dumps({"type": "uplink_denied", "reason": "disabled"}))
|
||||
print(f"[relay] denied uplink to '{names[ws]}' (--deny-uplink)")
|
||||
elif uplinkers[ws]:
|
||||
await broadcast_control(channel, {"type": "uplink_roster", "players": uplink_roster(channel)})
|
||||
elif payload.get("type") in ("uplink_start", "uplink_stop"):
|
||||
# Enriched with who it came from, which is what the web app needs to label streams.
|
||||
await safe_send(ch["speaker"], json.dumps({
|
||||
"type": payload["type"], "id": stream_id, "player": names.get(ws, "?"),
|
||||
})) if ch["speaker"] is not None else None
|
||||
print(f"[relay] '{names.get(ws, '?')}' {payload['type']} (stream {stream_id})")
|
||||
except Exception:
|
||||
pass
|
||||
finally:
|
||||
ch["listeners"].discard(ws)
|
||||
print(f"[relay] listener left '{channel}' -> {len(ch['listeners'])} listening")
|
||||
stream_ids.get(channel, {}).pop(ws, None)
|
||||
was_uplinker = uplinkers.pop(ws, False)
|
||||
who = names.pop(ws, "?")
|
||||
print(f"[relay] '{who}' left '{channel}' -> {len(ch['listeners'])} listening")
|
||||
if was_uplinker:
|
||||
await broadcast_control(channel, {"type": "uplink_roster", "players": uplink_roster(channel)})
|
||||
|
||||
|
||||
async def fanout(channel: str, payload: bytes):
|
||||
@@ -143,11 +228,18 @@ async def safe_send(ws, payload):
|
||||
|
||||
|
||||
async def cmd_serve(args):
|
||||
global deny_uplink
|
||||
deny_uplink = getattr(args, "deny_uplink", False)
|
||||
if deny_uplink:
|
||||
print("[relay] --deny-uplink: refusing every microphone with uplink_denied")
|
||||
async with serve(handler, HOST, args.port, max_size=None):
|
||||
print(f"[relay] listening on ws://{HOST}:{args.port}{PATH}")
|
||||
print(f"[relay] listen: ws://{HOST}:{args.port}{PATH}?token=listen:test")
|
||||
print(f"[relay] speak : ws://{HOST}:{args.port}{PATH}?token=speak:test")
|
||||
await asyncio.Future()
|
||||
print("[relay] session roster printed every 15s")
|
||||
while True:
|
||||
await asyncio.sleep(15)
|
||||
print_sessions()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- tone generator
|
||||
@@ -163,36 +255,59 @@ def generate_frame(phase, ts, n, rate, args):
|
||||
"""Returns (new_phase, pcm16_bytes) for one frame, phase-continuous across frames."""
|
||||
if _np is not None:
|
||||
idx = _np.arange(n, dtype=_np.float64)
|
||||
if args.sweep:
|
||||
t = (ts + idx) / rate
|
||||
if args.speech:
|
||||
# Speech-shaped: a voice-ish carrier under a syllable envelope with word gaps. The
|
||||
# sweep and steady tones hold a constant amplitude, so they pin a loudness-driven
|
||||
# mouth wide open -- this is the signal to test mouth animation against.
|
||||
freq = 150.0 + 60.0 * _np.sin(2 * _np.pi * 1.7 * t)
|
||||
syllables = (0.5 + 0.5 * _np.sin(2 * _np.pi * 4.0 * t)) ** 2
|
||||
words = (_np.sin(2 * _np.pi * 0.35 * t) > -0.25).astype(_np.float64)
|
||||
amp = 0.45 * syllables * words
|
||||
elif args.sweep:
|
||||
# 200-2000 Hz over 4 s: any resampling error shows up as an obvious warble
|
||||
t = (ts + idx) / rate
|
||||
freq = 200.0 + 1800.0 * (0.5 - 0.5 * _np.cos(2 * _np.pi * t / 4.0))
|
||||
amp = _np.full(n, 0.35)
|
||||
else:
|
||||
freq = _np.full(n, float(args.freq))
|
||||
amp = _np.full(n, 0.35)
|
||||
ph = phase + _np.cumsum(2 * _np.pi * freq / rate)
|
||||
samples = (_np.sin(ph) * 0.35 * 32767).astype("<i2")
|
||||
samples = (_np.sin(ph) * amp * 32767).astype("<i2")
|
||||
return float(ph[-1] % (2 * _np.pi)), samples.tobytes()
|
||||
|
||||
# Fallback without numpy. Fine for a steady tone; sweep may not hold real time.
|
||||
out = []
|
||||
for i in range(n):
|
||||
t = (ts + i) / rate
|
||||
freq = args.freq
|
||||
if args.sweep:
|
||||
t = (ts + i) / rate
|
||||
amp = 0.35
|
||||
if args.speech:
|
||||
freq = 150.0 + 60.0 * math.sin(2 * math.pi * 1.7 * t)
|
||||
syllables = (0.5 + 0.5 * math.sin(2 * math.pi * 4.0 * t)) ** 2
|
||||
words = 1.0 if math.sin(2 * math.pi * 0.35 * t) > -0.25 else 0.0
|
||||
amp = 0.45 * syllables * words
|
||||
elif args.sweep:
|
||||
freq = 200.0 + 1800.0 * (0.5 - 0.5 * math.cos(2 * math.pi * t / 4.0))
|
||||
phase = (phase + 2 * math.pi * freq / rate) % (2 * math.pi)
|
||||
out.append(int(math.sin(phase) * 0.35 * 32767))
|
||||
out.append(int(math.sin(phase) * amp * 32767))
|
||||
return phase, struct.pack(f"<{n}h", *out)
|
||||
|
||||
|
||||
async def cmd_tone(args):
|
||||
url = f"ws://{HOST}:{args.port}{PATH}?token=speak:{args.channel}"
|
||||
print(f"[tone] {url}")
|
||||
async with connect(url, max_size=None) as ws:
|
||||
await stream_tone(ws, args)
|
||||
|
||||
|
||||
async def stream_tone(ws, args):
|
||||
"""Sends a test tone downstream on an already-open speaker socket."""
|
||||
rate = args.rate
|
||||
n = int(rate * FRAME_MS / 1000)
|
||||
print(f"[tone] {url} {rate} Hz, {n} samples/frame ({FRAME_MS} ms)")
|
||||
print(f"[tone] {rate} Hz, {n} samples/frame ({FRAME_MS} ms)")
|
||||
|
||||
loop = asyncio.get_running_loop()
|
||||
async with connect(url, max_size=None) as ws:
|
||||
if True:
|
||||
seq = 0
|
||||
ts = 0
|
||||
phase = 0.0
|
||||
@@ -258,24 +373,93 @@ async def cmd_listen(args):
|
||||
)
|
||||
|
||||
|
||||
async def cmd_uplink(args):
|
||||
"""Acts as the web-app speaker: receives uplink audio and writes it out per stream.
|
||||
|
||||
The real-time ratio is the number that matters. Below ~0.98 with the gate open means the
|
||||
sender is not keeping up and the browser would hear clicking; while gated it is expected to
|
||||
sit low, because silence is deliberately not transmitted.
|
||||
"""
|
||||
import wave
|
||||
url = f"ws://{HOST}:{args.port}{PATH}?token=speak:{args.channel}"
|
||||
print(f"[uplink] {url}")
|
||||
print("[uplink] writing uplink_<id>.wav, Ctrl-C to stop")
|
||||
if args.tone or args.sweep or args.speech:
|
||||
print("[uplink] also sending a test tone downstream on this same socket")
|
||||
|
||||
writers: dict = {}
|
||||
counts: dict = {}
|
||||
first_time = None
|
||||
total_samples = 0
|
||||
|
||||
async with connect(url, max_size=None) as ws:
|
||||
sender = None
|
||||
if args.tone or args.sweep or args.speech:
|
||||
args.rate = 16000
|
||||
args.freq = getattr(args, "freq", 1000.0)
|
||||
sender = asyncio.create_task(stream_tone(ws, args))
|
||||
try:
|
||||
async for msg in ws:
|
||||
if isinstance(msg, str):
|
||||
print("\n" + f"[uplink] control: {msg}")
|
||||
continue
|
||||
if len(msg) < 12 or msg[0] != MAGIC_UPLINK:
|
||||
continue
|
||||
sid = msg[3] >> 4
|
||||
if sid not in writers:
|
||||
w = wave.open(f"uplink_{sid}.wav", "wb")
|
||||
w.setnchannels(1)
|
||||
w.setsampwidth(2)
|
||||
w.setframerate(16000)
|
||||
writers[sid] = w
|
||||
counts[sid] = 0
|
||||
print("\n" + f"[uplink] stream {sid} opened -> uplink_{sid}.wav")
|
||||
writers[sid].writeframes(msg[12:])
|
||||
counts[sid] += 1
|
||||
total_samples += (len(msg) - 12) // 2
|
||||
if first_time is None:
|
||||
first_time = asyncio.get_running_loop().time()
|
||||
elapsed = asyncio.get_running_loop().time() - first_time
|
||||
if sum(counts.values()) % 50 == 0 and elapsed > 0:
|
||||
ratio = (total_samples / 16000) / elapsed
|
||||
verdict = "OK" if ratio > 0.98 else "(gated or starving)"
|
||||
print("\r" + f"[uplink] {sum(counts.values())} frames, "
|
||||
f"real-time ratio {ratio:.3f} {verdict} ", end="", flush=True)
|
||||
finally:
|
||||
if sender is not None:
|
||||
sender.cancel()
|
||||
for w in writers.values():
|
||||
w.close()
|
||||
print("\n" + "[uplink] wav files closed")
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
p.add_argument("--port", type=int, default=PORT)
|
||||
sub = p.add_subparsers(dest="cmd", required=True)
|
||||
|
||||
sub.add_parser("serve")
|
||||
sv = sub.add_parser("serve")
|
||||
sv.add_argument("--deny-uplink", action="store_true", help="refuse every uplink, to test the client's uplink_denied handling")
|
||||
|
||||
t = sub.add_parser("tone")
|
||||
t.add_argument("--channel", default="test")
|
||||
t.add_argument("--freq", type=float, default=1000.0)
|
||||
t.add_argument("--rate", type=int, default=16000, choices=sorted(RATE_CODES))
|
||||
t.add_argument("--sweep", action="store_true")
|
||||
t.add_argument("--sweep", action="store_true", help="200-2000 Hz sweep; makes resampling errors obvious")
|
||||
t.add_argument("--speech", action="store_true", help="speech-shaped envelope; use this to test mouth animation")
|
||||
|
||||
l = sub.add_parser("listen")
|
||||
l.add_argument("--channel", default="test")
|
||||
|
||||
u = sub.add_parser("uplink", help="act as the speaker and capture what the game sends up")
|
||||
u.add_argument("--channel", default="test")
|
||||
u.add_argument("--tone", action="store_true", help="also send a steady tone downstream, on the same socket")
|
||||
u.add_argument("--sweep", action="store_true", help="also send a sweep downstream")
|
||||
u.add_argument("--speech", action="store_true", help="also send speech-shaped audio downstream")
|
||||
u.add_argument("--freq", type=float, default=1000.0)
|
||||
|
||||
args = p.parse_args()
|
||||
fn = {"serve": cmd_serve, "tone": cmd_tone, "listen": cmd_listen}[args.cmd]
|
||||
fn = {"serve": cmd_serve, "tone": cmd_tone, "listen": cmd_listen, "uplink": cmd_uplink}[args.cmd]
|
||||
try:
|
||||
asyncio.run(fn(args))
|
||||
except KeyboardInterrupt:
|
||||
|
||||
Reference in New Issue
Block a user