Compare commits

..

6 Commits

Author SHA1 Message Date
daa675af79 0.1.16 | BingBong live voice: return audio
Adds the uplink direction: each player sends their own microphone back to
whoever is puppeting BingBong, gated on proximity to him and attenuated by the
same curve his own voice is heard through. Rides the WebSocket that is already
open, so no new connection, auth or TLS surface, and still nothing over Photon.

Capture postfixes MicWrapper.Read rather than MicrophoneRelay, which copies
only the first 256 samples of each frame. Audio is downmixed, low-passed and
resampled to 16kHz in the mod, then passed through an adaptive noise gate: a
fixed threshold is unusable here because a quiet voice measures below a noisy
room, so the gate learns the background level instead.

F8 mutes the outgoing microphone for the session. uplink_denied and
uplink_roster from the relay are now handled, and the stub relay gained the
return direction plus --deny-uplink to exercise the refusal path.

The real relay and web app do not implement this direction yet, so it currently
only works against the stub.
2026-09-06 14:12:45 +02:00
f4d723b23b 0.1.15 | BingBong live voice (in-game client)
Speak through BingBong with a live microphone, streamed from a relay
rather than over PEAK's Photon servers. This is the in-game half only:
the relay and the web app that captures the microphone are not built
yet, so the feature does nothing on its own until they exist.

Every BingBong gets a voice source as it spawns, positioned on the plush
and inheriting the 3D setup of his own voice lines. His mouth animates to
the live audio on every client, and hands back to the vanilla animation
while one of his own lines plays. Those lines take priority: the live
voice ducks for their duration and resumes at the live position.

Playback uses OnAudioFilterRead rather than a streaming AudioClip. A clip
created with stream:true fills Unity's decode buffer ahead of playback,
and that read-ahead cannot be satisfied from a live stream because the
audio has not arrived yet -- the buffer drained to empty on a cycle no
matter how deep it was made, which was audible as constant clicking. The
filter callback runs in lockstep with playback, so demand can never
outrun real time.

Includes tools/stub_relay.py, a development stub speaking the same wire
format with steady, swept and speech-shaped test signals, so the client
can be tested end to end without the real backend, a microphone or a
browser.
2026-09-05 18:07:35 +02:00
9fc95c2fc5 0.1.14 | Audio fixes & optimisations 2026-09-05 16:15:01 +02:00
b0cffe1362 0.1.13 | Fix for PEAK update
Player.EmptySlot and PhotonNetwork.InstantiateItemRoom each gained an
optional parameter, which compiles fine but left the previous build
calling signatures that no longer exist. The InstantiateItemRoom throw
escaped the BagsForEveryone postfix and took down every
SingleItemSpawner in the level, which is why backpacks and BingBong
stopped spawning and campfires no longer advanced the biome.

Also hardens that postfix so a future API change costs the extra bags
and nothing else, registers them with the spawner's SpawnedItemTracker,
and stops the Open Backpack keybind from opening the wheel while the
backpack is held in hand.
2026-09-04 00:10:07 +02:00
3cd47afb25 0.1.12 | Added: Open Passport keybind
Also removes a leftover debug keybind on the BingBong module that was
bound to P and would have fired alongside the new passport keybind.
2026-09-04 00:09:52 +02:00
d2c75cfb53 0.1.11 | Fix for last update 2026-08-21 21:41:47 +02:00
26 changed files with 3298 additions and 190 deletions

3
.gitignore vendored
View File

@@ -226,3 +226,6 @@ artifacts/**/*.zip
*.txt
artifacts/**
Config.Build.user.props
# Per-stream captures written by tools/stub_relay.py's uplink mode
uplink_*.wav

View File

@@ -50,4 +50,74 @@
# v0.1.10 | Bonkable items
- Added Bonkable component to items when thrown.
- Added Bonkable component to items when thrown.
# v0.1.11 | Fix for last update
- Fixed Easy Backpack: PEAK's backpack rework (jetpacks/fannypacks/rocketpacks) removed `BackpackSlot.hasBackpack` and changed `GUIManager.OpenBackpackWheel`'s signature, which broke the build. Updated to use `BackpackSlot.backpackType` and pass slot count/type, and brought the `BackpackWheel.Update()` override back in sync with the new vanilla logic (missing null guard, jetpack fuel slice handling).
- Fixed Stashed Bugle not giving the Bugle: it was passing the item's display name to `SpawnItemInHand`, which actually expects the prefab's Resources name. Now resolves the real prefab name through the ItemDatabase first.
- Fixed AudioSyncWorker getting permanently stuck after a failed sync (e.g. missing API URL): the sync ran on a background thread and called a Unity API (`StartCoroutine`) that's main-thread only, throwing silently and leaving the loading flag stuck forever. Added a main-thread dispatch queue so background work can safely hop back to the main thread.
- Fixed a leftover debug log using the Error level for a successful download, making normal syncs look like they were failing.
- Fixed the in-game right-click "reload songs" prompt on the Bugle: it was calling the wrong method and didn't actually reload anything. It now properly unloads and reloads from local files (distinct from the Sync keybind, which pulls from the audio bank API).
- Fixed the right-click confirmation always showing "No answer, not refreshing songs" even after successfully confirming, due to a timing check that ran before the confirmation window instead of after.
- Fixed an error (NullReferenceException) every time a thrown item bonked something: `Bonkable` added at runtime has no serialized prefab data, so its sound-effect array was `null` instead of empty. It now defaults to an empty array so bonks still ragdoll/knock back the target, just silently.
# v0.1.12 | Added customization keybinds
- Open the passport when pressing the "Open Passport" keybind (default: P)
- Removed a leftover debug keybind on the BingBong module that was also bound to P and would have fired alongside the new passport keybind.
# v0.1.13 | Fix for PEAK update
- Rebuilt against the September PEAK update. Two game methods gained an optional parameter, which compiles fine but leaves an older build calling a signature that no longer exists, so both threw a MissingMethodException at runtime:
- `Player.EmptySlot(slot)` became `EmptySlot(slot, bool andBroadcast = true)`. This killed the whole "Toggle Bugle" keybind (default: V) — both giving the Bugle and swapping it for the Megaphone.
- `PhotonNetwork.InstantiateItemRoom(name, position, rotation)` gained a fourth `bool warnIfNotHost = true`. This one threw inside the Bags for Everyone postfix on `SingleItemSpawner.TrySpawnItems`, and because a throwing postfix propagates out of the method it patched, it took down every `SingleItemSpawner` in the level rather than just the backpack one. Backpacks and BingBong stopped spawning, and lighting a campfire no longer advanced the biome: the spawn loop runs inside `MapHandler`'s transition coroutine, so the throw killed that coroutine and the campfire items, day/night blend, biome title and fog reveal all silently never happened.
- Removed the manual `RPCRemoveItemFromSlot` call the Bugle keybind made right after `EmptySlot`. `EmptySlot` already forwards the removal to the host itself, so the extra call only logged "Only Master Client can remove items!" on every press for anyone who wasn't hosting.
- The "Open Backpack" keybind (default: B) no longer opens the wheel while the backpack is held in your hands. The wheel pulls items out of the backpack's on-back visuals, which the game deactivates while the backpack is in hand, so anything taken out was spawned into a disabled rig and disappeared. Hold-interacting the held backpack still opens it the normal way.
- Hardened the Bags for Everyone patch. Its master-client and spawner-name checks now run in the postfix itself instead of inside the method that spawns the bags, and the spawn is wrapped in a try/catch. Because Mono resolves a method's call targets when it JITs that method, a changed game API used to throw before any of those checks could run, taking down every `SingleItemSpawner` in the level — which is what stopped campfires from advancing the biome in 0.1.12. A failure now costs the extra bags and nothing else.
- The extra backpacks are registered with the spawner's `SpawnedItemTracker`, so a quicksave records them instead of restoring only the vanilla one, and the patch no longer adds a fresh batch on top every time a run is loaded from a save.
# v0.1.14 | Audio fixes & optimisations
- Songs are now kept compressed in memory and decoded during playback instead of being held as raw PCM for the whole session. A 127-file, 110-minute bank measured ~82 MB on disk but around 0.9-1.8 GB once decompressed; it now costs roughly its size on disk, for a little CPU while a bugle is actually sounding.
- Fixed the audio bank sync never removing anything. The removal step ran on the sync's background thread and called `FindObjectsByType`/`Object.Destroy`, which are main-thread only, so it threw on the first song it tried to remove and aborted the whole sync — downloads included. Songs dropped from the bank had simply been reloading from disk on every launch ever since.
- Syncing now makes the folder match the bank exactly: anything the bank doesn't list is pulled out, including files added by hand and stale copies of a song left behind under a different extension. Nothing is deleted — files are moved to a `_removed` subfolder of the bank directory, which the loader ignores. BingBong voice lines are exempt and never touched.
- A sync that can't reach the bank, or gets an empty list back, now cancels instead of treating "no entries" as "nothing belongs here" and clearing out the whole folder.
- Fixed the game freezing the first time a sound is played in a session. `DownloadHandlerAudioClip` hands back a `DecompressOnLoad` clip whose PCM data is only decoded on first use, so that decode was landing inside the first `AudioSource.Play()` on the main thread. The clips are now decoded during loading instead, where the cost is expected and spread across frames.
- Fixed an empty sounds folder permanently breaking the audio system. With no songs loaded, picking a starting song indexed an empty list and threw, and because that ran from the load-complete callback it left the loading flag stuck on, silently killing every later sync and refresh for the rest of the session. The song index is also clamped now, since a refresh that removes songs could leave it past the end of the list.
- Fixed a Bugle picked up off the ground being completely silent. The patch that installs the custom sound only ran for items already in hand, but `ItemState.Ground` is the default and `Item.Start` doesn't set state, so anything spawned into the world never got it — while the vanilla Bugle sound was being muted regardless. The vanilla sound is now only muted where the replacement actually took over.
- Fixed BingBong subtitles never resolving: the localisation key was registered in lowercase, but `LocalizedText.GetText` upper-cases ids before looking them up. The entry also had a single string where one per language is required, which threw for every non-English player.
- Fixed `SFX_Instance` being created with `new`. It derives from `ScriptableObject`, which has to go through `ScriptableObject.CreateInstance` or it skips Unity's native-side initialisation.
- BingBong responses are now built once and shared instead of being rebuilt every time an item is enabled, which was allocating a `ScriptableObject` per response each time and never releasing the previous ones.
- Refreshing songs no longer scans the scene for audio sources once per song — a bank of N songs meant N full scene scans — and no longer forces a blocking garbage collection.
- File hashing is computed on demand on the sync thread rather than for every clip while loading, where it was reading each file end to end and running SHA256 on the main thread.
- The audio bank sync no longer drives `UnityWebRequest` from a background thread, which is main-thread only, and no longer busy-waits on it while blocking a second thread-pool thread.
- The song display no longer scans every loaded object looking for its font on every frame. It falls back to the default font immediately and retries for the nicer one on an interval, stopping once found.
- The audio bank URL is normalised before use: a missing scheme is filled in (http for localhost, https otherwise) and a trailing slash trimmed. Without a scheme every request failed outright with "Invalid BugleSoundAPIURL in config", since the URL could not be parsed.
# v0.1.15 | BingBong live voice (in-game client)
- Speak through BingBong with a live microphone. Audio is streamed from a relay rather than over PEAK's Photon servers, so it adds no load there at all. This release is the **in-game half only** — the relay and the web app that captures the microphone are not built yet, so the feature does nothing on its own until they exist.
- Every BingBong in the world gets a voice source as it spawns, positioned on the plush and inheriting the 3D setup of his own voice lines, so the live voice carries and attenuates like he does and obeys the game's volume sliders. Several BingBongs each speak from their own position.
- His mouth animates to the live audio, on every client rather than just the speaker's, and hands control back to the vanilla animation while one of his own voice lines plays.
- Canned voice lines take priority: the live voice ducks for the length of the line and then resumes at the live position, so the two never talk over each other. Detected from the RPC body, so the duck stays in step across the lobby without any networking of our own.
- New "BingBong Voice" config section, all of it applied live. The relay URL is derived from `BugleSoundAPIURL` when left empty, since that is where the relay will live, and can be overridden to point anywhere — a bare host or `host:port` is completed automatically. Range defaults to 50m, further than his 30m voice lines carry.
- Playback goes through `OnAudioFilterRead` rather than a streaming `AudioClip`. A clip created with `stream: true` fills Unity's decode buffer *ahead* of playback, and that read-ahead cannot be satisfied from a live stream because the audio has not arrived yet — the buffer drained to empty on a cycle no matter how deep it was, which was audible as constant clicking. The filter callback runs in lockstep with playback instead, so demand can never outrun real time.
- Incoming audio is resampled to the output rate by the mod rather than relying on Unity's mixer, so a device running at 44.1kHz or 48kHz sounds identical. A jitter buffer absorbs network timing, rendering what it has and padding the rest on a shortfall rather than muting and refilling, which pumped audibly when it happened often.
- Both Harmony patches are wrapped so that a failure can only cost the voice feature: the patch body is a try/catch around an isolated method, and repeated failures disable it for the session rather than breaking the item.
- `tools/stub_relay.py` is a development stub of the relay, enough to test the client end to end before the real backend exists. It speaks the same wire format and generates steady, swept and speech-shaped test signals, so no microphone or browser is needed.
# v0.1.16 | BingBong live voice: return audio
- Talking near BingBong now sends your own microphone back to whoever is puppeting him, so it is a conversation rather than a broadcast. Each player only ever transmits their own microphone, over the WebSocket that was already open — no second connection, and still nothing over PEAK's Photon servers.
- The uplink is gated on being within the same range BingBong's voice carries and attenuated by the same curve, so holding him up and talking works and walking away fades you out. It only transmits while someone is actually connected on the other end; an idle lobby sends nothing at all.
- Capture taps `MicWrapper.Read` rather than the obvious `MicrophoneRelay`, which copies only the first 256 samples of each frame before handing them on. Vanilla only ever wants a loudness number from it so the truncation is invisible in game, but streaming it gives a buzz at the frame rate instead of speech.
- The microphone is downmixed, low-passed and resampled from the device's rate to 16kHz by the mod. Decimating 48kHz to 16kHz without filtering first folds everything above 8kHz back into the speech band, which sounds like a second voice made of hiss.
- The noise gate learns the room instead of using a fixed level. A fixed threshold cannot work here: measured on real hardware, a quiet voice sits below what a noisy room measures, so any single number either cuts speech or passes the room. It tracks the background level, opens a configurable factor above it, and holds for 300ms so it does not close between words.
- The gate opens on entering range and closes only past a margin beyond it, with a minimum time in each state, so standing on the boundary cannot chatter it on and off.
- New `UplinkMuteKey` (default: F8) mutes the outgoing microphone for the rest of the session, since reaching for the config file is too slow to be the answer to "stop sending". It is not persisted — a mute you forgot you set is worse than one that clears on the next launch.
- The mod now understands the relay's `uplink_denied` and `uplink_roster` messages: a relay with the uplink switched off is believed and not asked again on that connection, and the log names who else is sending room audio, which is what tells "the relay is dropping my audio" apart from "nobody is listening".
- Uplink frames carry a different magic byte from the downlink ones, and the relay only ever forwards them to the speaker. A mis-routed frame therefore cannot replay a room's own audio back into it, which would be a feedback loop rather than a bug.
- The counters that let the receiver detect a dropped frame advance only for frames actually sent, so a gated pause leaves a contiguous timeline. Start and stop messages tell the receiver to rejoin the live edge instead of filling the pause with silence.
- `tools/stub_relay.py` gained the return direction: it forwards uplink audio to the speaker, stamps the stream id itself so a modified client cannot claim another player's, captures each stream to a WAV, and reports what fraction of real time actually arrived. `--deny-uplink` exercises the refusal path.
- The relay and the web app that would play this back are not built yet, so the return direction currently only works against that stub.

1
build.bat Normal file
View File

@@ -0,0 +1 @@
dotnet build -c Release -target:PackTS -v d

View File

@@ -13,6 +13,9 @@ public static class ConfigHandler
// Better Airport
public static ConfigEntry<float> ConveyorSpeedModifier { get; private set; } = null!;
// Open Passport settings
public static ConfigEntry<KeyCode> OpenPassport { get; private set; } = null!;
// Stashed Bugle settings
public static ConfigEntry<KeyCode> ToggleBugle { get; private set; } = null!;
@@ -26,6 +29,26 @@ public static class ConfigHandler
public static ConfigEntry<string> FavoriteSongsList { get; private set; } = null!;
public static ConfigEntry<KeyCode> FavoriteSongToggleKey { get; private set; } = null!;
// BingBong Voice settings
public static ConfigEntry<bool> BingBongVoiceEnabled { get; private set; } = null!;
public static ConfigEntry<string> BingBongVoiceRelayURL { get; private set; } = null!;
public static ConfigEntry<float> BingBongVoiceVolume { get; private set; } = null!;
public static ConfigEntry<float> BingBongVoiceDuckVolume { get; private set; } = null!;
public static ConfigEntry<float> BingBongVoiceDuckLead { get; private set; } = null!;
public static ConfigEntry<float> BingBongVoiceJitterMs { get; private set; } = null!;
public static ConfigEntry<float> BingBongVoiceMaxDistance { get; private set; } = null!;
public static ConfigEntry<bool> BingBongVoiceAutoConnect { get; private set; } = null!;
public static ConfigEntry<bool> BingBongVoiceAnnounceSpeaker { get; private set; } = null!;
public static ConfigEntry<bool> BingBongVoiceUplinkEnabled { get; private set; } = null!;
public static ConfigEntry<KeyCode> BingBongVoiceUplinkMuteKey { get; private set; } = null!;
public static ConfigEntry<float> BingBongVoiceUplinkGain { get; private set; } = null!;
public static ConfigEntry<float> BingBongVoiceUplinkMinGain { get; private set; } = null!;
public static ConfigEntry<float> BingBongVoiceUplinkGateSensitivity { get; private set; } = null!;
public static ConfigEntry<float> BingBongVoiceUplinkNoiseGateDb { get; private set; } = null!;
public static ConfigEntry<bool> BingBongVoiceUplinkLoopback { get; private set; } = null!;
public static ConfigEntry<KeyCode> BingBongVoiceDebugKey { get; private set; } = null!;
public static ConfigEntry<bool> BingBongVoiceDebugSource { get; private set; } = null!;
public static void Initialize(ConfigFile configFile)
{
Config = configFile;
@@ -49,6 +72,14 @@ public static class ConfigHandler
)
);
// Open Passport settings
OpenPassport = Config.Bind(
"Key Bindings",
"OpenPassport",
KeyCode.P,
new ConfigDescription("Open Passport UI")
);
// Stashed Bugle settings
ToggleBugle = Config.Bind(
"Control",
@@ -96,6 +127,134 @@ public static class ConfigHandler
new ConfigDescription("Key to toggle favorite status of current song")
);
// BingBong Voice settings
BingBongVoiceEnabled = Config.Bind(
"BingBong Voice",
"Enabled",
true,
new ConfigDescription("Let BingBong speak with a live voice streamed from the relay. When off, no patches are applied at all.")
);
BingBongVoiceRelayURL = Config.Bind(
"BingBong Voice",
"RelayURL",
"",
new ConfigDescription("Override for the voice relay WebSocket URL. Normally leave this EMPTY: it is then derived from BugleSoundAPIURL, keeping that URL's path, so https://host/api becomes wss://host/api/voice/ws. Only set it to point at a different relay, and include the full path - aiming at the website instead of the API gives an HTML page and a 1002 close.")
);
BingBongVoiceVolume = Config.Bind(
"BingBong Voice",
"Volume",
1.0f,
new ConfigDescription("Live voice volume", new AcceptableValueRange<float>(0f, 1f))
);
BingBongVoiceDuckVolume = Config.Bind(
"BingBong Voice",
"DuckVolume",
0.0f,
new ConfigDescription("Live voice volume while one of BingBong's own voice lines is playing", new AcceptableValueRange<float>(0f, 1f))
);
BingBongVoiceDuckLead = Config.Bind(
"BingBong Voice",
"DuckLeadSeconds",
1.0f,
new ConfigDescription("How long before a voice line's audio actually starts to begin ducking. AskRoutine waits about a second before it plays anything.", new AcceptableValueRange<float>(0f, 2f))
);
BingBongVoiceJitterMs = Config.Bind(
"BingBong Voice",
"JitterBufferMs",
80f,
new ConfigDescription("Audio buffered before playback starts. Lower is more responsive, higher survives worse connections.", new AcceptableValueRange<float>(40f, 400f))
);
BingBongVoiceMaxDistance = Config.Bind(
"BingBong Voice",
"MaxDistance",
150f,
new ConfigDescription("How far the live voice carries, in metres. BingBong's own voice lines only reach 30m, which is short for a conversation.", new AcceptableValueRange<float>(5f, 500f))
);
BingBongVoiceAutoConnect = Config.Bind(
"BingBong Voice",
"AutoConnect",
true,
new ConfigDescription("Connect to the voice relay automatically on joining a lobby, and disconnect on leaving. Turn off to connect only with the keybind.")
);
BingBongVoiceAnnounceSpeaker = Config.Bind(
"BingBong Voice",
"AnnounceSpeaker",
true,
new ConfigDescription("Show a message when someone starts speaking through BingBong. Being puppeted silently by someone who is not even in your lobby is a bit much without it.")
);
BingBongVoiceUplinkEnabled = Config.Bind(
"BingBong Voice",
"UplinkEnabled",
true,
new ConfigDescription("Send your own microphone to whoever is puppeting BingBong, when you are near him and only while someone is actually connected. Turning this off means no microphone patch is applied at all.")
);
BingBongVoiceUplinkMuteKey = Config.Bind(
"BingBong Voice",
"UplinkMuteKey",
KeyCode.F8,
new ConfigDescription("Mutes and unmutes your outgoing microphone for the rest of the session. Faster than the config when you need it to stop now.")
);
BingBongVoiceUplinkGain = Config.Bind(
"BingBong Voice",
"UplinkGain",
3.0f,
new ConfigDescription("Trim on your outgoing mic. The tap sits upstream of Photon's automatic gain control, so the raw signal is quieter than what teammates hear and usually needs lifting.", new AcceptableValueRange<float>(0f, 8f))
);
BingBongVoiceUplinkMinGain = Config.Bind(
"BingBong Voice",
"UplinkMinGain",
0.02f,
new ConfigDescription("Stop transmitting once distance has attenuated you below this. Keeps the edge of the range quiet rather than faintly open.", new AcceptableValueRange<float>(0f, 0.5f))
);
BingBongVoiceUplinkGateSensitivity = Config.Bind(
"BingBong Voice",
"UplinkGateSensitivity",
3.0f,
new ConfigDescription("How far above the measured background noise your voice must be before it is transmitted. The gate learns your room rather than using a fixed level, because a quiet voice can measure lower than a noisy room. Raise it if background noise gets through, lower it if quiet speech is cut off.", new AcceptableValueRange<float>(1.5f, 10f))
);
BingBongVoiceUplinkNoiseGateDb = Config.Bind(
"BingBong Voice",
"UplinkFloorDb",
-60f,
new ConfigDescription("Absolute floor: nothing quieter than this is ever transmitted, whatever the adaptive gate decides. A safety net, not the main control.", new AcceptableValueRange<float>(-80f, -20f))
);
BingBongVoiceUplinkLoopback = Config.Bind(
"BingBong Voice",
"UplinkLoopback",
false,
new ConfigDescription("Debug: play your own processed microphone back out of BingBong instead of sending it anywhere. Lets the capture path be heard with no relay and no second player. Deliberately closes a feedback loop, so use headphones.")
);
BingBongVoiceDebugKey = Config.Bind(
"BingBong Voice",
"DebugToggleKey",
KeyCode.F7,
new ConfigDescription("Connect to / disconnect from the voice relay")
);
BingBongVoiceDebugSource = Config.Bind(
"BingBong Voice",
"DebugSource",
false,
new ConfigDescription("Also play the stream through a non-positional source, for testing without a BingBong in the scene. Plays on top of any real BingBong, so leave this off normally.")
);
}
}

View File

@@ -8,7 +8,7 @@
<!-- This is the display name of your mod. Example: BepInEx Template -->
<AssemblyTitle>JordanMod</AssemblyTitle>
<!-- This is the version number of your mod. -->
<Version>0.1.10</Version>
<Version>0.1.16</Version>
</PropertyGroup>
<PropertyGroup>

View File

@@ -16,6 +16,12 @@ public partial class Plugin : BaseUnityPlugin
private static Harmony? _harmony;
private ModuleManager? _moduleManager;
private static readonly System.Collections.Concurrent.ConcurrentQueue<Action> _mainThreadQueue = new();
// Unity APIs (e.g. StartCoroutine) can only be called from the main thread;
// background work (Task.Run, etc.) must hop back through here to touch them.
public static void RunOnMainThread(Action action) => _mainThreadQueue.Enqueue(action);
private static readonly List<Type> _globalPatches = [
typeof(Patches.PassportPatch),
];
@@ -84,6 +90,11 @@ public partial class Plugin : BaseUnityPlugin
private void Update()
{
while (_mainThreadQueue.TryDequeue(out Action? action))
{
try { action(); }
catch (Exception e) { Log.LogError($"Main thread action failed: {e}"); }
}
_moduleManager?.Update();
}

View File

@@ -1,3 +1,4 @@
using System;
using pworld.Scripts.Extensions;
namespace JordanMod.Events;
@@ -7,12 +8,16 @@ public class GlobalEventListener
public static void Initialize()
{
GlobalEvents.OnItemThrown += OnItemThrown;
// GlobalEvents.OnItemThrown += OnItemThrown;
}
private static void OnItemThrown(Item item)
{
item.gameObject.GetOrAddComponent<Bonkable>();
// Bonkable bonkable = item.gameObject.GetOrAddComponent<Bonkable>();
// // A Bonkable added at runtime has no serialized prefab data, so its `bonk` SFX array
// // defaults to null (not empty) and Bonkable.Bonk() NREs on bonk.Length. Give it an
// // empty array so a bonk still ragdolls/knocks back the target, just without a sound.
// bonkable.bonk ??= Array.Empty<SFX_Instance>();
}
}

View File

@@ -113,8 +113,10 @@ class BetterBugleModule : Module
List<string> scrollActionLocalizations = new(LocalizedText.LANGUAGE_COUNT);
for (int i = 0; i < LocalizedText.LANGUAGE_COUNT; i++) scrollActionLocalizations.Add("Change Song");
LocalizedText.mainTable.Add("SONG_LIST", secondaryActionLocalizations);
LocalizedText.mainTable.Add("CHANGE_SONG", scrollActionLocalizations);
// Indexer, not Add: Add throws if the key is already present, which happens if the game
// rebuilds its table (e.g. on a language change) and this runs again.
LocalizedText.mainTable["SONG_LIST"] = secondaryActionLocalizations;
LocalizedText.mainTable["CHANGE_SONG"] = scrollActionLocalizations;
}
private void OnAllAudioClipsLoaded()
@@ -128,8 +130,22 @@ class BetterBugleModule : Module
if (Song.Songs.ContainsKey(songKeyName) && !Song.FavoriteSongs.Contains(songKeyName))
Song.FavoriteSongs.Add(songKeyName);
if (!Song.Songs.ContainsKey(AudioSyncWorker.CurrentSongName))
AudioSyncWorker.CurrentSongName = Song.GetSongNames_Alphabetically()[AudioSyncWorker.CurrentSongIndex];
// Indexing this unguarded threw ArgumentOutOfRangeException with an empty bank (a fresh
// install, or a sounds folder that lost its files), and because this runs from
// OnAudioLoadComplete it used to leave IsLoading stuck true and kill sync/refresh for
// the rest of the session. CurrentSongIndex can also outrun the list after a refresh
// removes songs, so clamp rather than assume it is still in range.
List<string> songNames = Song.GetSongNames_Alphabetically();
if (songNames.Count == 0)
{
AudioSyncWorker.CurrentSongIndex = 0;
AudioSyncWorker.CurrentSongName = "None";
}
else if (!Song.Songs.ContainsKey(AudioSyncWorker.CurrentSongName))
{
AudioSyncWorker.CurrentSongIndex = Mathf.Clamp(AudioSyncWorker.CurrentSongIndex, 0, songNames.Count - 1);
AudioSyncWorker.CurrentSongName = songNames[AudioSyncWorker.CurrentSongIndex];
}
}
}
@@ -265,6 +281,10 @@ public class BetterBugleUI : MonoBehaviour
private bool fontLoaded = false;
private const float FontSearchInterval = 5f;
private float lastFontSearchTime = -999f;
private int offsetX = 0;
private int offsetY = 70;
@@ -294,28 +314,41 @@ public class BetterBugleUI : MonoBehaviour
private void OnGUI()
{
if (!fontLoaded)
{
Font[] array = Resources.FindObjectsOfTypeAll<Font>();
foreach (Font val in array)
{
if (val.name == "Tetsubin Gothic")
{
customStyle = new GUIStyle(GUI.skin.label);
customStyle.font = val;
customStyle.fontSize = fontSize;
customStyle.alignment = TextAnchor.LowerCenter;
customStyle.normal.textColor = Color.white;
fontLoaded = true;
break;
}
}
}
EnsureStyle();
if (customStyle == null) return;
RenderSoundDisplay();
RenderPlayingDisplay();
}
// This used to run Resources.FindObjectsOfTypeAll<Font>() on every OnGUI call until the font
// turned up -- a full scan of every loaded object, several times per frame, forever if the
// font was never present in the scene. Now we build a usable style immediately with the
// default font and only re-scan for the nicer one on an interval, stopping once found.
private void EnsureStyle()
{
if (fontLoaded) return;
if (customStyle != null && Time.unscaledTime - lastFontSearchTime < FontSearchInterval) return;
lastFontSearchTime = Time.unscaledTime;
Font? found = null;
foreach (Font val in Resources.FindObjectsOfTypeAll<Font>())
{
if (val.name == "Tetsubin Gothic") { found = val; break; }
}
customStyle = new GUIStyle(GUI.skin.label)
{
fontSize = fontSize,
alignment = TextAnchor.LowerCenter,
normal = { textColor = Color.white }
};
if (found != null)
{
customStyle.font = found;
fontLoaded = true;
}
}
private void RenderSoundDisplay()
{
if (customStyle == null) return;

View File

@@ -0,0 +1,351 @@
using System;
using System.Security.Authentication;
using System.Threading;
using Newtonsoft.Json.Linq;
using WebSocketSharp;
namespace JordanMod.Modules.BingBongVoice;
/// One decoded stream. Swapped wholesale when the sender changes sample rate, so consumers
/// only ever hold a consistent (ring, rate) pair.
public sealed class VoiceStream
{
public VoiceRingBuffer Ring { get; }
public int SampleRate => Ring.SampleRate;
public VoiceStream(int sampleRate) => Ring = new VoiceRingBuffer(sampleRate);
}
/// <summary>
/// Holds the websocket to the voice relay and turns its binary frames into samples in a ring.
///
/// Everything here runs on websocket-sharp's own receive thread. It must never touch a Unity
/// API: hop through <see cref="Plugin.RunOnMainThread"/> for that, and never per packet — at
/// 50 frames a second that queue would balloon during any frame hitch.
/// </summary>
public sealed class BingBongVoiceClient
{
// Wire format, kept in step with the plan and the stub relay:
// 0 u8 magic | 1 u8 version | 2 u8 codec | 3 u8 rate code | 4 u32 seq | 8 u32 sampleTimestamp
private const int HeaderSize = 12;
private const byte Magic = 0xB1;
private const byte Version = 1;
private const byte CodecPcm16 = 0;
private static readonly int[] RateTable = [16000, 8000, 24000, 48000];
// A gap larger than this is treated as a restart rather than something to paper over.
private const int MaxGapSamples = 48000;
private WebSocket? _socket;
private string? _hello;
private VoiceStream? _stream;
private Thread? _reconnectThread;
private volatile bool _shouldRun;
private volatile bool _speakerActive;
private volatile bool _uplinkDenied;
private long _expectedTimestamp = -1;
private int _badFrames;
private volatile int _lastCloseCode;
private int _sendFailures;
/// The live stream, or null before the first frame arrives. Read from the audio thread.
public VoiceStream? Stream => Volatile.Read(ref _stream);
/// True between speaker_joined and speaker_left control frames.
public bool SpeakerActive => _speakerActive;
/// Set when the relay says it will not take our microphone -- uplink switched off on its side,
/// or too many senders on this channel. Sticky until the next connect, because the answer will
/// not change on this socket and retrying for the rest of the session would just burn bandwidth.
public bool UplinkDenied => _uplinkDenied;
/// Who is speaking, when the relay tells us. Null if nobody is, or if it did not say.
public string? SpeakerName { get; private set; }
/// Raised on the main thread when the speaker changes, so the UI can announce it.
public event Action<string?>? OnSpeakerChanged;
public bool IsConnected => _socket is { ReadyState: WebSocketState.Open };
public string Url { get; private set; } = "";
/// <param name="hello">
/// Optional JSON control frame sent as soon as the socket opens. This is how a lobby
/// registers itself with the relay: the channel comes from the token, and this supplies who
/// is in it, so an admin can see the live sessions and pick one to speak into.
/// </param>
public void Connect(string url, string? hello = null)
{
_hello = hello;
if (_shouldRun) return;
if (string.IsNullOrWhiteSpace(url))
{
Debug.LogWarning("[BingBongVoice] No relay URL configured, not connecting.");
return;
}
// A malformed URL will never start working, so fail here rather than letting the
// reconnect loop retry it every minute for the rest of the session.
if (!Uri.TryCreate(url, UriKind.Absolute, out Uri? parsed) || (parsed.Scheme != "ws" && parsed.Scheme != "wss"))
{
Debug.LogError($"[BingBongVoice] '{url}' is not a usable WebSocket URL, not connecting.");
return;
}
Url = url;
_shouldRun = true;
_reconnectThread = new Thread(ConnectLoop) { IsBackground = true, Name = "BingBongVoice" };
_reconnectThread.Start();
}
public void Disconnect()
{
_shouldRun = false;
_speakerActive = false;
WebSocket? socket = _socket;
_socket = null;
if (socket == null) return;
try { socket.Close(CloseStatusCode.Normal, "disconnecting"); }
catch (Exception e) { Debug.LogWarning($"[BingBongVoice] Error closing socket: {e.Message}"); }
}
// Reconnect with exponential backoff. Runs on its own thread for the whole session.
private void ConnectLoop()
{
int attempt = 0;
System.Random jitter = new();
while (_shouldRun)
{
try
{
OpenSocket();
// Poll rather than block: websocket-sharp gives us no "wait until closed".
while (_shouldRun && _socket is { ReadyState: WebSocketState.Open }) Thread.Sleep(200);
if (!_shouldRun) return;
// A rejected request goes straight to the ceiling rather than climbing to it.
attempt = _lastCloseCode == 1008 ? 6 : attempt + 1;
}
catch (Exception e)
{
attempt++;
// First failure and then every tenth, so an unreachable relay can't flood the log.
if (attempt == 1 || attempt % 10 == 0)
Debug.LogWarning($"[BingBongVoice] Connection attempt {attempt} failed: {e.Message}");
}
if (!_shouldRun) return;
int delayMs = (int)(Math.Min(60000, 2000 * Math.Pow(2, Math.Min(attempt - 1, 5))) * (0.8 + jitter.NextDouble() * 0.4));
for (int slept = 0; slept < delayMs && _shouldRun; slept += 100) Thread.Sleep(100);
}
}
private void OpenSocket()
{
WebSocket socket = new(Url);
// websocket-sharp writes its own "Fatal|WebSocket.connect|..." lines straight to the
// console, including a run of mojibake, every single reconnect attempt. We report
// connection state ourselves, so silence its logger entirely.
socket.Log.Output = (_, _) => { };
// This build of websocket-sharp defaults to a legacy protocol set, so wss:// to any
// modern TLS terminator fails outright without this.
if (Url.StartsWith("wss://", StringComparison.OrdinalIgnoreCase))
{
socket.SslConfiguration.EnabledSslProtocols =
SslProtocols.Tls12 | SslProtocols.Tls11 | SslProtocols.Tls;
}
socket.OnMessage += OnMessage;
socket.OnOpen += (_, _) =>
{
_expectedTimestamp = -1;
_badFrames = 0;
_uplinkDenied = false;
Debug.Log("[BingBongVoice] Connected to relay.");
// Register this lobby member so the relay's session directory knows who is listening.
if (string.IsNullOrEmpty(_hello)) return;
try { socket.Send(_hello); }
catch (Exception ex) { Debug.LogWarning($"[BingBongVoice] Could not announce presence: {ex.Message}"); }
};
socket.OnClose += (_, e) =>
{
_speakerActive = false;
_lastCloseCode = e.Code;
// 1008 means the request itself is wrong -- bad token shape, uppercase channel, an
// Origin header we should not be sending, or the relay being disabled. Retrying
// quickly cannot fix any of those, so say so plainly and let the loop back right off.
if (e.Code == 1008)
{
Debug.LogError($"[BingBongVoice] Relay rejected the connection ({e.Reason}). This will not fix itself; check the relay URL and token.");
}
else if (e.Code == 1002)
{
// The handshake got a valid HTTP response that was not an upgrade -- almost always
// the URL pointing at the web frontend rather than the API. A single-page app
// answers 200 with HTML for any path, so it looks alive and never works.
Debug.LogError($"[BingBongVoice] '{Url}' answered with something that is not a WebSocket. Point it at the API rather than the website: the endpoint usually sits under /api (e.g. https://host/api -> wss://host/api/voice/ws). Clearing RelayURL derives it from BugleSoundAPIURL.");
}
else
{
Debug.Log($"[BingBongVoice] Relay closed ({e.Code} {e.Reason}).");
}
};
socket.OnError += (_, e) => Debug.LogWarning($"[BingBongVoice] Socket error: {e.Message}");
_socket = socket;
socket.Connect();
}
// websocket-sharp's receive thread. No Unity APIs, no allocation beyond what we're handed.
private void OnMessage(object? sender, MessageEventArgs e)
{
try
{
if (!e.IsBinary)
{
HandleControl(e.Data);
return;
}
byte[] d = e.RawData;
if (d.Length < HeaderSize) return;
if (d[0] != Magic || d[1] != Version)
{
if (++_badFrames <= 3) Debug.LogWarning($"[BingBongVoice] Bad frame header (magic {d[0]:X2}, version {d[1]}).");
return;
}
if (d[2] != CodecPcm16)
{
if (++_badFrames <= 3) Debug.LogWarning($"[BingBongVoice] Unsupported codec {d[2]}, frame dropped.");
return;
}
int rate = RateTable[d[3] & 3];
uint timestamp = (uint)(d[8] | (d[9] << 8) | (d[10] << 16) | (d[11] << 24));
VoiceStream stream = EnsureStream(rate);
// The sender's timestamp tells us exactly how much audio went missing, so a dropped
// frame costs its own duration instead of silently shifting everything after it.
if (_expectedTimestamp >= 0 && timestamp != _expectedTimestamp)
{
long gap = (long)timestamp - _expectedTimestamp;
if (gap > 0 && gap <= MaxGapSamples) stream.Ring.WriteSilence((int)gap);
// A negative or huge gap means the sender restarted; just carry on from here.
}
int payload = d.Length - HeaderSize;
stream.Ring.WritePcm16(d, HeaderSize, payload);
_expectedTimestamp = timestamp + (payload >> 1);
}
catch (Exception ex)
{
if (++_badFrames <= 3) Debug.LogError($"[BingBongVoice] Frame handling failed: {ex}");
}
}
/// <summary>
/// Sends one frame on the socket we already hold. Returns false rather than throwing when the
/// socket is not open, so the uplink can count failures without ever driving reconnection --
/// that belongs to ConnectLoop, and a second driver would fight it.
/// </summary>
public bool TrySend(byte[] frame)
{
WebSocket? socket = _socket;
if (socket == null || socket.ReadyState != WebSocketState.Open) return false;
try
{
socket.Send(frame);
return true;
}
catch (Exception e)
{
if (++_sendFailures <= 3) Debug.LogWarning($"[BingBongVoice] Uplink send failed: {e.Message}");
return false;
}
}
/// Control frames for the uplink. Text, like every other control message.
public bool TrySendControl(string json)
{
WebSocket? socket = _socket;
if (socket == null || socket.ReadyState != WebSocketState.Open) return false;
try { socket.Send(json); return true; }
catch { return false; }
}
private VoiceStream EnsureStream(int rate)
{
VoiceStream? current = Volatile.Read(ref _stream);
if (current != null && current.SampleRate == rate) return current;
VoiceStream replacement = new(rate);
Volatile.Write(ref _stream, replacement);
_expectedTimestamp = -1;
Debug.Log($"[BingBongVoice] Stream rate is {rate} Hz.");
return replacement;
}
private void HandleControl(string json)
{
// Parsed leniently and fielded by substring if that fails: control messages are allowed to
// gain fields, and a malformed payload must never take down the receive thread.
string? speaker = null;
JObject? parsed = null;
try
{
parsed = JObject.Parse(json);
speaker = parsed["speaker"]?.ToString();
}
catch
{
// not JSON, or no such field -- the substring checks below still work
}
if (json.Contains("speaker_joined"))
{
_speakerActive = true;
SpeakerName = string.IsNullOrWhiteSpace(speaker) ? null : speaker;
Debug.Log($"[BingBongVoice] A speaker joined{(SpeakerName != null ? $": {SpeakerName}" : "")}.");
Plugin.RunOnMainThread(() => OnSpeakerChanged?.Invoke(SpeakerName));
}
else if (json.Contains("speaker_left") || json.Contains("preempted"))
{
_speakerActive = false;
SpeakerName = null;
_expectedTimestamp = -1;
Debug.Log("[BingBongVoice] The speaker left.");
Plugin.RunOnMainThread(() => OnSpeakerChanged?.Invoke(null));
}
else if (json.Contains("uplink_denied"))
{
// The relay has told us it is not taking microphones. Nothing about that changes while
// this socket is up, so stop asking rather than sending 50 frames a second into a drop.
_uplinkDenied = true;
Debug.LogWarning($"[BingBongVoice] The relay will not take our microphone ({parsed?["reason"]?.ToString() ?? json}). Not sending until the next connect.");
}
else if (json.Contains("uplink_roster"))
{
// Purely informational, and the one thing that distinguishes "the relay is dropping my
// audio" from "nobody is listening" when the uplink looks like it is working locally.
string players = parsed?["players"] is JArray list && list.Count > 0
? string.Join(", ", list)
: "nobody";
Debug.Log($"[BingBongVoice] Sending room audio: {players}.");
}
else if (json.Contains("rejected"))
{
Debug.LogWarning($"[BingBongVoice] Relay rejected this listener: {json}");
}
}
}

View File

@@ -0,0 +1,480 @@
using System;
using System.Security.Cryptography;
using System.Text;
using JordanMod.Modules.BetterBugle;
using Newtonsoft.Json;
using Photon.Pun;
using Photon.Voice.Unity;
using UnityEngine;
namespace JordanMod.Modules.BingBongVoice;
[Module(Enabled = true)]
class BingBongVoiceModule : Module
{
public static BingBongVoiceModule? Instance { get; private set; }
public override string ModuleName => "BingBong Voice Module";
public BingBongVoiceClient? Client { get; private set; }
private GameObject? _debugObject;
/// Captures the local microphone. Null when the uplink is off or we are not in a lobby, which
/// is what makes the mic tap a no-op rather than something that needs its own guard.
public MicUplinkSource? Uplink { get; private set; }
/// Stage 1 debug: the captured mic played back through BingBong instead of being sent. Lets
/// the whole capture path be heard with no relay and no second player.
public VoiceStream? LoopbackStream { get; private set; }
private MicUplinkSender? _sender;
private bool _uplinkTransmitting;
private const string ChannelSalt = "jordanmod-bingbong-voice-v1";
public override Type[] GetPatches()
{
// Config is bound before modules load (Plugin.Awake: ConfigHandler.Initialize ->
// SetupModules -> SetupPatches), so turning the feature off here means the Harmony
// patches are never applied at all and the mod has zero footprint.
if (!ConfigHandler.BingBongVoiceEnabled.Value) return [];
// The mic tap sits on Photon's per-frame audio path, so an opted-out player should carry
// no patch at all rather than one that early-returns.
return ConfigHandler.BingBongVoiceUplinkEnabled.Value
? [typeof(BingBongVoicePatch), typeof(BingBongMicTapPatch)]
: [typeof(BingBongVoicePatch)];
}
public override void Initialize()
{
if (Instance != null) return;
Instance = this;
// Recorded here so the mic callback's thread can be compared against it.
BingBongMicTapPatch.NoteMainThread();
// The relay lives on the audio bank, so with no override configured every player derives
// the same URL and connects on their own -- nobody has to press anything.
PhotonNetworkEventListener listener = PhotonNetworkEventListener.Instance!;
if (listener != null)
{
listener.RegisterOnJoinedRoom(OnJoinedRoom);
listener.RegisterOnLeftRoom(Disconnect);
listener.RegisterOnDisconnected(_ => Disconnect());
}
else
{
Debug.LogWarning("[BingBongVoice] No network event listener, auto-connect is unavailable.");
}
base.Initialize();
}
private void OnJoinedRoom()
{
if (!ConfigHandler.BingBongVoiceEnabled.Value) return;
if (!ConfigHandler.BingBongVoiceAutoConnect.Value) return;
Connect();
}
public override void Update()
{
if (!ConfigHandler.BingBongVoiceEnabled.Value) return;
if (Input.GetKeyDown(ConfigHandler.BingBongVoiceDebugKey.Value)) ToggleStream();
if (Input.GetKeyDown(ConfigHandler.BingBongVoiceUplinkMuteKey.Value)) ToggleUplinkMute();
EnsureUplink();
UpdateUplinkProximity();
}
// Hysteresis on the gate only. The gain is continuous so it never stutters; what stutters is
// the open/closed decision, and each cycle costs the receiver a buffer reset.
private bool _uplinkInRange;
private float _uplinkStateChangedAt;
private float _lastUplinkReport;
private bool _uplinkMuted;
/// Session-scoped mute for the outgoing microphone, on a key because reaching for the config is
/// too slow to be the answer to "stop sending, now". Deliberately not persisted: a mute you
/// forgot you set is worse than one that clears when you next launch.
private void ToggleUplinkMute()
{
_uplinkMuted = !_uplinkMuted;
Debug.Log($"[BingBongVoice] Uplink microphone {(_uplinkMuted ? "MUTED" : "unmuted")}.");
BetterBugleUI.Instance?.ShowActionbar(_uplinkMuted
? "🎙 BingBong microphone muted"
: "🎙 BingBong microphone live");
}
/// <summary>
/// Decides how loudly, if at all, the local microphone should be transmitted: nearest BingBong,
/// attenuated by the same curve his own voice is heard through.
/// </summary>
private void UpdateUplinkProximity()
{
MicUplinkSource? uplink = Uplink;
if (uplink == null) return;
Character? local = Character.localCharacter;
if (local == null)
{
uplink.SetGate(false);
return;
}
float range = ConfigHandler.BingBongVoiceMaxDistance.Value;
float nearest = float.MaxValue;
foreach (BingBongVoiceSource source in BingBongVoiceSource.Active)
{
if (source == null || !source.IsPositional) continue;
float d = Vector3.Distance(local.Head, source.transform.position);
if (d < nearest) nearest = d;
}
// Open on entering the range, close only past a margin beyond it, so walking the boundary
// cannot chatter the gate. The dwell is a minimum time in the CURRENT state before it may
// flip again -- not a check against the change just made, which would undo it every frame.
bool want = _uplinkInRange ? nearest <= range * 1.15f : nearest < range;
if (want != _uplinkInRange)
{
float minDwell = _uplinkInRange ? 1.5f : 0.5f;
if (Time.time - _uplinkStateChangedAt >= minDwell)
{
_uplinkInRange = want;
_uplinkStateChangedAt = Time.time;
}
}
float gain = 0f;
if (_uplinkInRange && nearest < float.MaxValue)
{
// The curve, then a taper over the last fifth so the gate closes on real silence
// instead of a cliff. Unity's own rolloff stops attenuating past maxDistance, which
// copied literally would leave everyone in the level faintly transmitting forever.
gain = BingBongVoiceSource.RangeGain(nearest, range)
* Mathf.SmoothStep(1f, 0f, Mathf.InverseLerp(range * 0.8f, range, nearest))
* ConfigHandler.BingBongVoiceUplinkGain.Value;
}
bool audible = gain > ConfigHandler.BingBongVoiceUplinkMinGain.Value;
// Only while someone is actually puppeting BingBong. An idle lobby sends nothing, and
// "my mic goes out only while someone is on the other end" is a rule a player can hold
// in their head. Loopback ignores it, having no speaker to wait for.
bool speakerPresent = ConfigHandler.BingBongVoiceUplinkLoopback.Value || (Client?.SpeakerActive ?? false);
bool allowed = !_uplinkMuted && !(Client?.UplinkDenied ?? false);
bool transmit = _uplinkInRange && audible && speakerPresent && allowed;
uplink.SetTargetGain(gain);
uplink.SetGate(transmit);
// The receiver's timeline freezes with ours while gated, so it needs telling to rejoin the
// live edge rather than treating the pause as a gap to fill with silence.
if (transmit != _uplinkTransmitting && !ConfigHandler.BingBongVoiceUplinkLoopback.Value)
{
_uplinkTransmitting = transmit;
Client?.TrySendControl(transmit ? "{\"type\":\"uplink_start\"}" : "{\"type\":\"uplink_stop\"}");
Debug.Log($"[BingBongVoice] Uplink {(transmit ? "started" : "stopped")} sending.");
}
if (Time.time - _lastUplinkReport < 5f) return;
_lastUplinkReport = Time.time;
Debug.Log($"[BingBongVoice] Uplink: nearest BingBong {(nearest < float.MaxValue ? nearest.ToString("F1") + "m" : "none")}, " +
$"gain {gain:F2}, gate {(_uplinkInRange && audible ? "open" : "closed")}{(_uplinkMuted ? ", MUTED" : "")}, " +
$"level {uplink.Level:F4} (floor {uplink.NoiseFloor:F5}), {uplink.FramesProduced} captured" +
$"{(_sender != null ? $", {_sender.Sent} sent, {_sender.Dropped} dropped" : "")}. {DescribeMic(local)}");
}
/// <summary>
/// Where the relay lives. The relay is served by the audio bank, so an unset override just
/// derives the WebSocket URL from BugleSoundAPIURL; setting it points somewhere else
/// entirely (a local stub, a separate host).
/// </summary>
public static string ResolveRelayUrl()
{
string overrideUrl = (ConfigHandler.BingBongVoiceRelayURL.Value ?? "").Trim();
string source;
string url;
if (overrideUrl.Length > 0)
{
// An override names the endpoint. Only fill in /voice/ws when it has no path at all,
// so "localhost:8787" works but "wss://host/custom/path" is left alone.
source = overrideUrl;
url = NormalizeOverride(overrideUrl);
}
else
{
// The audio bank URL names a service root, which may itself sit under a path
// (https://audiobank.thejordan.dev/api), so the endpoint always goes on the end.
source = (ConfigHandler.BugleSoundAPIURL.Value ?? "").Trim();
if (source.Length == 0)
{
Debug.LogWarning("[BingBongVoice] Neither RelayURL nor BugleSoundAPIURL is set, nowhere to connect.");
return "";
}
url = DeriveFromApiBase(source);
}
if (url.Length == 0)
{
Debug.LogError($"[BingBongVoice] Could not make a usable WebSocket URL out of '{source}'. Expected something like 'ws://localhost:8787/voice/ws' or just 'localhost:8787'.");
return "";
}
// A URL that already carries a token is left exactly as given -- that is how you point at
// a stub relay, or connect from outside a lobby.
if (url.IndexOf("token=", StringComparison.OrdinalIgnoreCase) < 0)
{
string channel = ResolveChannel();
if (channel.Length == 0)
{
Debug.LogWarning("[BingBongVoice] Not in a lobby, so there is no channel to join. Put a full URL including ?token= in RelayURL to connect from outside one.");
return "";
}
// The relay's published contract is "listen:<channel>". It also accepts a bare channel
// id, but following the documented form is what stays correct if it ever tightens.
// The channel must be lowercase hex -- uppercase is rejected with a 1008.
url += (url.IndexOf('?') >= 0 ? "&" : "?") + "token=" + Uri.EscapeDataString("listen:" + channel);
}
return url;
}
/// Announced to the relay on connect so admins can see which lobbies are live and who is in
/// them. Deliberately carries the player name but NOT the room code: the name is what makes a
/// session recognisable in the web app, whereas the code is what someone would need to join
/// the game, so there is no reason to put it on the wire.
private static string BuildHello()
{
string nickname = PhotonNetwork.LocalPlayer?.NickName ?? "";
if (string.IsNullOrWhiteSpace(nickname)) nickname = "Unknown";
// Whether we might ever send a microphone, so the relay can allocate a stream id and get the
// roster right before the first audio frame rather than a beat after it.
bool uplink = ConfigHandler.BingBongVoiceUplinkEnabled.Value
&& !ConfigHandler.BingBongVoiceUplinkLoopback.Value;
return JsonConvert.SerializeObject(new { type = "hello", player = nickname, uplink });
}
/// <summary>
/// Scopes the voice channel to the Photon lobby, so each game session gets its own and
/// players only hear whoever is speaking into theirs.
///
/// Hashed rather than sent as-is: the room code is what someone would need to join the game,
/// and it would otherwise sit in the URL and in every server access log. The salt is baked
/// into the mod so it is not a secret -- it only stops a channel id from being trivially
/// reversible into a joinable room code. Once the relay's own join endpoint exists it should
/// issue channel ids server-side instead, with a salt that really is private.
/// </summary>
public static string ResolveChannel()
{
string room = PhotonNetwork.CurrentRoom?.Name ?? "";
if (string.IsNullOrWhiteSpace(room)) return "";
using SHA256 sha = SHA256.Create();
byte[] hash = sha.ComputeHash(Encoding.UTF8.GetBytes(ChannelSalt + room.Trim().ToUpperInvariant()));
return BitConverter.ToString(hash).Replace("-", "").ToLowerInvariant().Substring(0, 32);
}
private static string NormalizeOverride(string raw)
{
string url = ToWebSocketScheme(raw);
// Split the query/fragment off first so appending a path can't mangle them.
int split = url.IndexOfAny(['?', '#']);
string tail = split >= 0 ? url.Substring(split) : "";
string head = split >= 0 ? url.Substring(0, split) : url;
if (!IsWebSocketUrl(head, out Uri? parsed)) return "";
// A bare host means the caller gave us the root, not the endpoint.
if (parsed.AbsolutePath.Length <= 1) head = head.TrimEnd('/') + "/voice/ws";
return head + tail;
}
private static string DeriveFromApiBase(string apiBase)
{
string url = ToWebSocketScheme(apiBase);
// The base is a service root, so anything after it is not ours to keep.
int split = url.IndexOfAny(['?', '#']);
if (split >= 0) url = url.Substring(0, split);
url = url.TrimEnd('/') + "/voice/ws";
return IsWebSocketUrl(url, out _) ? url : "";
}
private static bool IsWebSocketUrl(string url, out Uri parsed)
{
parsed = null!;
if (!Uri.TryCreate(url, UriKind.Absolute, out Uri? candidate)) return false;
if (candidate.Scheme != "ws" && candidate.Scheme != "wss") return false;
parsed = candidate;
return true;
}
private static string ToWebSocketScheme(string url)
{
if (url.StartsWith("ws://", StringComparison.OrdinalIgnoreCase)) return url;
if (url.StartsWith("wss://", StringComparison.OrdinalIgnoreCase)) return url;
if (url.StartsWith("https://", StringComparison.OrdinalIgnoreCase)) return "wss://" + url.Substring(8);
if (url.StartsWith("http://", StringComparison.OrdinalIgnoreCase)) return "ws://" + url.Substring(7);
// No scheme at all: same rule as everywhere else, loopback gets the insecure variant.
return Helper.EnsureScheme(url, "wss", "ws");
}
// Every BingBong attaches its own source through BingBongVoicePatch; the optional debug
// source is the stage 1 harness, for testing with no BingBong in the scene.
private void ToggleStream()
{
if (Client != null) Disconnect();
else Connect();
}
private void Connect()
{
if (Client != null) return;
string url = ResolveRelayUrl();
if (url.Length == 0) return;
Debug.Log($"[BingBongVoice] Connecting for lobby '{PhotonNetwork.CurrentRoom?.Name}' to {url}");
Client = new BingBongVoiceClient();
Client.Connect(url, BuildHello());
if (!ConfigHandler.BingBongVoiceDebugSource.Value) return;
_debugObject = new GameObject("BingBongVoiceDebug");
UnityEngine.Object.DontDestroyOnLoad(_debugObject);
BingBongVoiceSource.Attach(_debugObject, null);
}
// Reuses the Bugle's actionbar because it is the only on-screen text this mod has. Raised on
// the main thread by the client, so touching Unity here is safe.
private static void AnnounceSpeaker(string? speaker)
{
if (!ConfigHandler.BingBongVoiceAnnounceSpeaker.Value) return;
BetterBugleUI.Instance?.ShowActionbar(speaker != null
? $"🎙 BingBong is being puppeted by {speaker}"
: "BingBong is himself again");
}
/// <summary>
/// Photon's own view of the microphone. When our captured level is zero the question is always
/// whether the game is feeding the device at all -- push-to-talk released, no device selected,
/// or the recorder stopped -- rather than anything in the capture path.
/// </summary>
private static string DescribeMic(Character local)
{
try
{
CharacterVoiceHandler? voice = local.refs?.voice;
if (voice == null) return "(no voice handler)";
Recorder? recorder = voice.GetComponent<Recorder>();
if (recorder == null) return "(no recorder)";
string device = recorder.MicrophoneDevice != null ? recorder.MicrophoneDevice.ToString() : "none";
return $"mic: transmitting={recorder.IsCurrentlyTransmitting}, enabled={recorder.TransmitEnabled}, device={device}";
}
catch (Exception e)
{
return $"(mic state unreadable: {e.GetType().Name})";
}
}
/// <summary>
/// Keeps the capture path in step with the config, every frame. Deliberately independent of the
/// relay connection: loopback has to work with no network at all, and tying capture to a
/// successful connect meant an unreachable relay silently disabled it. Also picks up a config
/// change live, rather than only at the moment a lobby is joined.
/// </summary>
private void EnsureUplink()
{
if (!ConfigHandler.BingBongVoiceUplinkEnabled.Value)
{
if (Uplink != null) StopUplink();
return;
}
bool wantLoopback = ConfigHandler.BingBongVoiceUplinkLoopback.Value;
bool haveLoopback = LoopbackStream != null;
bool senderMatchesClient = wantLoopback || (_sender != null) == (Client != null);
if (Uplink != null && wantLoopback == haveLoopback && senderMatchesClient) return;
StopUplink();
StartUplink();
}
private void StartUplink()
{
if (!ConfigHandler.BingBongVoiceUplinkEnabled.Value) return;
if (Uplink != null) return;
MicUplinkSource source = new();
if (ConfigHandler.BingBongVoiceUplinkLoopback.Value)
{
LoopbackStream = new VoiceStream(MicUplinkSource.WireRate);
source.EmitSilenceWhenGated = true; // keep playback continuous; see the field's note
source.OnFrame = frame => LoopbackStream.Ring.WritePcm16Samples(frame, frame.Length);
Debug.LogWarning("[BingBongVoice] Uplink LOOPBACK is on: your microphone plays back out of BingBong and is not sent anywhere. Wear headphones.");
}
else if (Client != null)
{
MicUplinkSender sender = new(Client);
sender.Start();
source.OnFrame = sender.Enqueue;
_sender = sender;
}
Uplink = source;
Debug.Log($"[BingBongVoice] Uplink capture started{(ConfigHandler.BingBongVoiceUplinkLoopback.Value ? " in LOOPBACK mode" : "")}.");
}
private void StopUplink()
{
if (_uplinkTransmitting)
{
Client?.TrySendControl("{\"type\":\"uplink_stop\"}");
_uplinkTransmitting = false;
}
if (Uplink != null) Uplink.OnFrame = null;
Uplink = null;
LoopbackStream = null;
_sender?.Stop();
_sender = null;
}
private void Disconnect()
{
StopUplink();
if (Client == null) return;
Debug.Log("[BingBongVoice] Disconnecting from relay.");
Client.Disconnect();
Client = null;
if (_debugObject != null) UnityEngine.Object.Destroy(_debugObject);
_debugObject = null;
}
public override void Destroy()
{
PhotonNetworkEventListener listener = PhotonNetworkEventListener.Instance!;
if (listener != null)
{
listener.JoinedRoom -= OnJoinedRoom;
listener.LeftRoom -= Disconnect;
}
Client?.Disconnect();
Client = null;
if (_debugObject != null) UnityEngine.Object.Destroy(_debugObject);
_debugObject = null;
base.Destroy();
}
}

View File

@@ -0,0 +1,473 @@
using System;
using System.Runtime.CompilerServices;
using UnityEngine;
namespace JordanMod.Modules.BingBongVoice;
/// <summary>
/// Plays the live voice stream through one AudioSource. One of these per BingBong (and one
/// non-positional debug instance), each with its own read cursor into the shared ring.
///
/// Output goes through <c>OnAudioFilterRead</c> rather than a procedural streaming AudioClip.
/// A clip created with <c>stream: true</c> fills Unity's own decode buffer ahead of playback --
/// that read-ahead is by design, and it cannot be satisfied from a live stream because the audio
/// simply has not arrived yet, so the ring drained to empty no matter how deep the jitter buffer
/// was made. OnAudioFilterRead is invoked exactly once per DSP block, in lockstep with playback,
/// so demand can never outrun real time. The AudioSource still needs *something* playing for the
/// filter to run, hence the silent carrier clip.
/// </summary>
public sealed class BingBongVoiceSource : MonoBehaviour
{
private enum State { Buffering, Playing }
private AudioSource? _source;
private AudioClip? _carrier;
private VoiceStream? _stream;
private int _outputRate;
private double _read; // fractional position into the ring, audio thread only
private State _state = State.Buffering;
private float _gain; // smoothed toward _targetGain, audio thread only
private volatile float _targetGain;
private volatile float _amplitude; // post-gain, what you actually hear
private volatile float _rawAmplitude; // pre-gain MEAN |sample|, matching BingBongMouth's units
private volatile bool _dead;
private int _underruns;
private float _lastReportTime;
private int _strikes;
private volatile int _depthSamples;
private volatile int _minDepth = int.MaxValue;
private volatile int _streamRate = 1;
private volatile int _blockFrames; // frames per DSP block, as actually delivered
private static bool _reportedBlock; // once per session, not once per BingBong
private static int _liveSources; // how many voice sources currently exist
private static int _playingSources; // how many are actually enabled and audible
private static readonly System.Collections.Generic.List<BingBongVoiceSource> _active = new();
private float _gainStep = 0.0008f;
private long _lastWritePos;
private float _lastRateTime;
private float _appliedRange = -1f;
private BingBongMouth? _mouth;
private int _mouthHash;
private float _mouthValue;
private int _mouthStrikes;
private bool _mouthDisabled;
/// Set by the duck patch. Live voice is attenuated until this time.
public float DuckUntil { get; set; }
/// False for the non-positional debug instance, which sits on a DontDestroyOnLoad object at the
/// origin and would otherwise look like a BingBong standing next to you.
public bool IsPositional { get; private set; }
/// Enabled, positional sources. Main thread only.
public static System.Collections.Generic.IReadOnlyList<BingBongVoiceSource> Active => _active;
public static float MinDistanceFor(float range) => Mathf.Max(1f, range / 10f);
/// <summary>
/// Unity's Logarithmic rolloff expressed arithmetically, so the uplink can attenuate a player's
/// microphone by exactly the curve BingBong's own output is heard through.
/// </summary>
public static float RangeGain(float distance, float range)
{
float min = MinDistanceFor(range);
return Mathf.Clamp01(min / Mathf.Max(min, distance));
}
public float Amplitude => _amplitude;
private const string ChildName = "BingBongVoice";
/// <summary>
/// Always lives on its own child GameObject. OnAudioFilterRead hooks the DSP chain of the
/// AudioSource sharing its GameObject, and BingBong already carries the vanilla voice-line
/// source on the item root -- putting ours alongside it meant our filter ran on *their*
/// source, so the live stream was only audible while a canned line played and cut out
/// whenever that source stopped. A child of our own keeps the two chains separate, and
/// still follows the item since it is parented to it.
/// </summary>
public static BingBongVoiceSource Attach(GameObject target, AudioSource? template)
{
Transform existing = target.transform.Find(ChildName);
if (existing != null && existing.TryGetComponent(out BingBongVoiceSource found)) return found;
GameObject holder = new(ChildName);
holder.transform.SetParent(target.transform, worldPositionStays: false);
BingBongVoiceSource voice = holder.AddComponent<BingBongVoiceSource>();
voice.Configure(template);
return voice;
}
/// How many voice sources exist right now. Should be one per BingBong in the scene; if it
/// climbs with every grab then dead instances are still playing and everything is audible at
/// once, which reads as "no spatialisation" just as convincingly as a broken panner does.
public static int LiveSources => _liveSources;
/// Enabled instances. PEAK keeps several BingBong objects around (held and prop variants), so
/// what matters is not how many exist but how many are audible at once -- more than one would
/// sum to several times the volume and smear the apparent position.
public static int PlayingSources => _playingSources;
/// Lets the attach patch hand us BingBong's mouth animator. Kept separate from Attach so a
/// change to BingBongMouth can only break the animation, never the audio.
public void BindMouth(BingBongMouth mouth)
{
_mouth = mouth;
string param = string.IsNullOrEmpty(mouth.animValue) ? "Mouth Blend" : mouth.animValue;
_mouthHash = Animator.StringToHash(param);
Debug.Log($"[BingBongVoice] Mouth bound: parameter '{param}', maxMouthOpen={mouth.maxMouthOpen}.");
}
public static bool IsAttached(GameObject target) => Find(target) != null;
public static BingBongVoiceSource? Find(GameObject target)
{
Transform existing = target.transform.Find(ChildName);
return existing != null && existing.TryGetComponent(out BingBongVoiceSource found) ? found : null;
}
private void Configure(AudioSource? template)
{
_outputRate = AudioSettings.outputSampleRate;
_gainStep = 1f / (_outputRate * 0.015f); // ~15ms attack whatever the device runs at
// Carrier of constant 1.0, not silence, and the filter MULTIPLIES rather than overwrites.
// Unity applies 3D panning and distance attenuation as a per-channel gain; if that runs
// before custom filters then the buffer handed to us already carries that gain, and
// overwriting it threw the spatialisation away (the voice was audible at full volume from
// anywhere). Multiplying a unit carrier reapplies it. If the panner instead runs after
// the filter, the incoming value is simply 1.0 and multiplying changes nothing -- so this
// is correct either way round, without having to determine which order Unity uses.
_carrier = AudioClip.Create($"BingBongVoiceCarrier_{GetInstanceID()}", 4096, 1, _outputRate, false);
float[] unit = new float[4096];
for (int i = 0; i < unit.Length; i++) unit[i] = 1f;
_carrier.SetData(unit, 0);
_source = gameObject.AddComponent<AudioSource>();
_source.clip = _carrier;
_source.loop = true;
_source.playOnAwake = false;
if (template != null)
{
// Copy the vanilla voice-line source's 3D setup rather than guessing, so the live
// voice attenuates exactly like BingBong's own lines. Routing through its mixer
// group also means the game's volume sliders apply for free.
_source.spatialBlend = template.spatialBlend;
ApplyRange(ConfigHandler.BingBongVoiceMaxDistance.Value);
_source.rolloffMode = template.rolloffMode;
_source.dopplerLevel = template.dopplerLevel;
_source.spread = template.spread;
_source.outputAudioMixerGroup = template.outputAudioMixerGroup;
}
else
{
_source.spatialBlend = 0f; // debug instance: non-positional, always audible
}
IsPositional = template != null;
_source.Play();
_liveSources++;
AudioSettings.OnAudioConfigurationChanged += OnAudioConfigChanged;
}
/// <summary>
/// Range needs both distances, not just maxDistance. Under Logarithmic rolloff the falloff is
/// governed by minDistance -- volume goes roughly as minDistance/distance -- and maxDistance
/// only marks where attenuation stops changing. Vanilla's minDistance of 1 means the sound is
/// at 1/30 volume by 30m, so raising maxDistance alone did nothing at all. Scaling both
/// together stretches the same curve over the range asked for: full volume within a tenth of
/// it, and still audible out at the limit.
/// </summary>
private void ApplyRange(float range)
{
if (_source == null) return;
_source.maxDistance = range;
_source.minDistance = Mathf.Max(1f, range / 10f);
_appliedRange = range;
}
private void OnAudioConfigChanged(bool deviceChanged)
{
// Owning the resample ratio ourselves makes this a one-line update instead of a rebuild.
_outputRate = AudioSettings.outputSampleRate;
_gainStep = 1f / (_outputRate * 0.015f);
_state = State.Buffering;
}
private void Update()
{
BingBongVoiceModule? module = BingBongVoiceModule.Instance;
_stream = module?.LoopbackStream ?? module?.Client?.Stream;
_targetGain = Time.time < DuckUntil
? ConfigHandler.BingBongVoiceDuckVolume.Value
: ConfigHandler.BingBongVoiceVolume.Value;
// Picked up live, like the volumes are, so the range can be tuned without re-spawning
// every BingBong in the scene.
float range = ConfigHandler.BingBongVoiceMaxDistance.Value;
if (!Mathf.Approximately(_appliedRange, range)) ApplyRange(range);
if (_mouth != null && !_mouthDisabled)
{
try
{
DriveMouth();
}
catch (Exception e)
{
Debug.LogError($"[BingBongVoice] Mouth animation failed: {e}");
if (++_mouthStrikes >= 3)
{
_mouthDisabled = true;
Debug.LogError("[BingBongVoice] Mouth animation disabled for this session.");
}
}
}
if (!_reportedBlock && _blockFrames > 0)
{
_reportedBlock = true;
AudioSettings.GetDSPBufferSize(out int dspLength, out int dspCount);
Debug.Log($"[BingBongVoice] DSP block is {_blockFrames} frames ({_blockFrames * 1000 / Math.Max(1, _outputRate)}ms at {_outputRate}Hz); buffer {dspLength}x{dspCount}.");
}
if (Time.time - _lastReportTime > 5f && _underruns > 0)
{
_lastReportTime = Time.time;
int depthMs = _depthSamples * 1000 / Math.Max(1, _streamRate);
int minMs = _minDepth == int.MaxValue ? -1 : _minDepth * 1000 / Math.Max(1, _streamRate);
int targetMs = TargetSamples(_streamRate) * 1000 / Math.Max(1, _streamRate);
long writeNow = _stream?.Ring.WritePosition ?? 0;
float elapsed = Time.time - _lastRateTime;
float producedRatio = elapsed > 0.01f && _lastWritePos > 0
? (writeNow - _lastWritePos) / (elapsed * Math.Max(1, _streamRate))
: 1f;
_lastWritePos = writeNow;
_lastRateTime = Time.time;
string hint = producedRatio < 0.95f
? $" - SENDER is only producing {producedRatio:P0} of real time"
: depthMs <= 1 ? " - CONSUMER is draining faster than real time" : "";
Debug.Log($"[BingBongVoice] buffer health: {_underruns} underruns, depth {depthMs}ms (min {minMs}ms) of {targetMs}ms{hint}");
_underruns = 0;
_minDepth = int.MaxValue;
}
}
/// <summary>
/// Drives BingBong's mouth from the audio we are actually playing, so it works on every
/// client rather than only the speaker's.
///
/// BingBongMouth.Update only writes the animator parameter while canPlay is true, so holding
/// canPlay false is enough to take over cleanly -- no fighting over script execution order.
/// During a duck we stop forcing it and stop writing, which lets OnAsk -> CreateCurveMap set
/// canPlay itself and animate the canned line off its baked curve exactly as shipped.
/// </summary>
[MethodImpl(MethodImplOptions.NoInlining)]
private void DriveMouth()
{
BingBongMouth mouth = _mouth!;
if (mouth == null || mouth.animator == null) return;
if (Time.time < DuckUntil)
{
// Vanilla owns the mouth for the duration of its own voice line.
_mouthValue = 0f;
return;
}
if (_stream != null)
{
mouth.canPlay = false;
// Vanilla's units exactly: BingBongMouth bakes a curve of MEAN |sample| per window and
// feeds the animator curveMap.Evaluate(t) * maxMouthOpen, so maxMouthOpen is the factor
// that lifts those small means into the animator's range. Passing an already-normalised
// 0-1 value and then multiplying by it saturated the parameter, which held the jaw wide
// open for anything above near-silence. Matching the scale means it behaves like a
// canned line does, whatever maxMouthOpen happens to be.
float target = _rawAmplitude * mouth.maxMouthOpen;
_mouthValue = Mathf.Lerp(_mouthValue, target, Time.deltaTime * 25f);
mouth.animator.SetFloat(_mouthHash, _mouthValue);
}
else if (_mouthValue > 0.001f)
{
// Disconnected: close it rather than leaving the jaw hanging where it stopped.
_mouthValue = Mathf.Lerp(_mouthValue, 0f, Time.deltaTime * 10f);
mouth.animator.SetFloat(_mouthHash, _mouthValue);
}
}
// How much audio to hold before playing. One DSP block is consumed on every call, so the
// floor is (target - block); keeping two blocks plus the configured margin leaves that floor
// comfortably above the point where a read cannot be satisfied.
private int TargetSamples(int streamRate)
{
int blockStream = (int)(_blockFrames * (double)streamRate / Math.Max(1, _outputRate)) + 1;
return blockStream * 2 + (int)(streamRate * ConfigHandler.BingBongVoiceJitterMs.Value / 1000f);
}
// ---- audio thread from here down -------------------------------------------------
// Allocation-free, lock-free, no Unity API calls, no logging. An exception escaping onto
// the DSP thread is not recoverable, so everything is wrapped and falls back to silence.
private void OnAudioFilterRead(float[] data, int channels)
{
try
{
Fill(data, Math.Max(1, channels));
}
catch
{
Array.Clear(data, 0, data.Length);
_amplitude = 0f;
if (++_strikes >= 3) _dead = true;
}
}
private void Fill(float[] data, int channels)
{
int frames = data.Length / channels;
_blockFrames = frames;
VoiceStream? stream = _stream;
if (_dead || stream == null)
{
Silence(data);
return;
}
VoiceRingBuffer ring = stream.Ring;
long write = ring.WritePosition;
double ratio = (double)ring.SampleRate / _outputRate;
int targetSamples = TargetSamples(ring.SampleRate);
if (_state == State.Buffering)
{
if (write - (long)_read < targetSamples || write < targetSamples)
{
Silence(data);
return;
}
_read = write - targetSamples;
_state = State.Playing;
}
// Drifted too far behind (sender burst, or we were paused). Rejoin the live edge rather
// than playing further and further into the past.
if (write - (long)_read > targetSamples * 5)
{
_read = write - targetSamples;
}
double needed = frames * ratio;
long available = write - (long)_read;
_depthSamples = (int)Math.Min(int.MaxValue, available);
if (_depthSamples < _minDepth) _minDepth = _depthSamples;
_streamRate = ring.SampleRate;
int usableFrames = frames;
if (available < needed + 2)
{
usableFrames = (int)Math.Max(0, (available - 2) / ratio);
if (usableFrames > frames) usableFrames = frames;
_underruns++;
// Only rebuild the whole buffer when there was essentially nothing there. A brief
// shortfall should cost a few samples, not a fade plus a full buffer of silence.
if (usableFrames < frames / 4) _state = State.Buffering;
}
Render(data, ring, channels, usableFrames, ratio, fadeOut: usableFrames < frames);
// Pad any frames we could not fill.
for (int f = usableFrames; f < frames; f++)
for (int c = 0; c < channels; c++)
data[f * channels + c] = 0f;
}
private void Render(float[] data, VoiceRingBuffer ring, int channels, int frames, double ratio, bool fadeOut)
{
float peak = 0f;
float rawSum = 0f;
float fadeStep = fadeOut && frames > 0 ? _gain / frames : 0f;
for (int f = 0; f < frames; f++)
{
long i0 = (long)_read;
float frac = (float)(_read - i0);
float sample = ring[i0] * (1f - frac) + ring[i0 + 1] * frac;
if (fadeOut) _gain = Math.Max(0f, _gain - fadeStep);
else _gain += (_targetGain - _gain) * _gainStep;
float outSample = sample * _gain;
int baseIndex = f * channels;
for (int c = 0; c < channels; c++) data[baseIndex + c] *= outSample;
float abs = outSample < 0f ? -outSample : outSample;
if (abs > peak) peak = abs;
rawSum += sample < 0f ? -sample : sample;
_read += ratio;
}
_amplitude = peak;
_rawAmplitude = frames > 0 ? rawSum / frames : 0f;
}
private void Silence(float[] data)
{
Array.Clear(data, 0, data.Length);
_gain = 0f;
_amplitude = 0f;
_rawAmplitude = 0f;
}
// ----------------------------------------------------------------------------------
private void OnDestroy()
{
// The audio thread can and will run the filter after Destroy happens on the main thread,
// so flag it dead first: that check is what makes the teardown safe.
_dead = true;
_active.Remove(this);
_liveSources--;
AudioSettings.OnAudioConfigurationChanged -= OnAudioConfigChanged;
if (_source != null)
{
_source.Stop();
_source.clip = null;
}
if (_carrier != null) Destroy(_carrier);
}
private void OnDisable()
{
_active.Remove(this);
_playingSources--;
if (_source != null) _source.Stop();
_state = State.Buffering;
}
private void OnEnable()
{
if (!_active.Contains(this)) _active.Add(this);
_playingSources++;
// Rejoin live rather than trying to resume where we left off.
_state = State.Buffering;
if (_source != null && _carrier != null) _source.Play();
}
}

View File

@@ -0,0 +1,151 @@
using System;
using System.Threading;
namespace JordanMod.Modules.BingBongVoice;
/// <summary>
/// Frames captured microphone audio and puts it on the socket the client already holds.
///
/// Sends from a dedicated thread with the blocking <c>Send</c> rather than <c>SendAsync</c>:
/// websocket-sharp's async path posts to the thread pool, so ordering across queued sends is not
/// guaranteed, and reordered voice sounds far worse than dropped voice. One thread and a blocking
/// send gives strict ordering for free.
///
/// There is deliberately no pacer. The microphone is the clock -- Photon hands us exactly the
/// audio the hardware produced -- so anything that paced against a wall clock on top of that would
/// only fight it and guarantee drift in one direction.
/// </summary>
public sealed class MicUplinkSender
{
private const byte Magic = 0xB2; // 0xB1 is downlink; a distinct magic makes a relay
private const byte Version = 1; // mis-route unable to replay room audio into the room
private const byte CodecPcm16 = 0;
private const byte RateCode16k = 0;
private const int HeaderSize = 12;
private const int PayloadBytes = MicUplinkSource.FrameSamples * 2;
private const int FrameBytes = HeaderSize + PayloadBytes;
private const int RingSlots = 32; // ~640 ms; far more than a healthy socket ever needs
private readonly byte[][] _slots = new byte[RingSlots][];
private readonly BingBongVoiceClient _client;
private readonly SemaphoreSlim _signal = new(0);
private long _write;
private long _read;
private uint _seq;
private uint _timestamp;
private Thread? _thread;
private volatile bool _running;
private int _dropped;
private int _sent;
private int _consecutiveFailures;
private volatile bool _stopped;
public int Dropped => _dropped;
public int Sent => _sent;
public MicUplinkSender(BingBongVoiceClient client)
{
_client = client;
for (int i = 0; i < RingSlots; i++) _slots[i] = new byte[FrameBytes];
}
public void Start()
{
if (_running) return;
_running = true;
_thread = new Thread(SendLoop) { IsBackground = true, Name = "BingBongUplink" };
_thread.Start();
}
public void Stop()
{
_running = false;
_signal.Release();
}
/// Producer side: called from the mic path. Writes the frame into a pre-allocated slot, so
/// nothing is allocated per frame and the caller never blocks.
public void Enqueue(short[] samples)
{
if (_stopped || !_running) return;
// Ring full means the socket is not draining. Drop the OLDEST and keep the newest: on a
// live stream, stale audio is worth less than current audio, and blocking the producer
// would stall Photon's audio path.
if (_write - _read >= RingSlots)
{
Interlocked.Increment(ref _dropped);
Interlocked.Increment(ref _read);
}
byte[] slot = _slots[(int)(_write % RingSlots)];
slot[0] = Magic;
slot[1] = Version;
slot[2] = CodecPcm16;
slot[3] = RateCode16k; // the relay overwrites the high nibble with our stream id
WriteUInt32(slot, 4, _seq);
WriteUInt32(slot, 8, _timestamp);
for (int i = 0; i < MicUplinkSource.FrameSamples; i++)
{
short s = samples[i];
int b = HeaderSize + (i << 1);
slot[b] = (byte)(s & 0xFF);
slot[b + 1] = (byte)((s >> 8) & 0xFF);
}
// Both counters advance only for frames we actually emit, so a gated pause leaves the
// timeline contiguous and the receiver must not insert silence for it. The control frames
// are what tell it to resync instead.
_seq++;
_timestamp += MicUplinkSource.FrameSamples;
Interlocked.Increment(ref _write);
_signal.Release();
}
private void SendLoop()
{
while (_running)
{
try
{
_signal.Wait(200);
while (_read < Volatile.Read(ref _write))
{
byte[] slot = _slots[(int)(_read % RingSlots)];
if (_client.TrySend(slot))
{
Interlocked.Increment(ref _sent);
_consecutiveFailures = 0;
}
else if (++_consecutiveFailures >= 200)
{
// The socket has been refusing for four seconds. Stop rather than spin;
// reconnection is the client's job and a second driver would fight it.
_stopped = true;
Debug.LogWarning("[BingBongVoice] Uplink stopped: the socket has been refusing frames. It will resume on the next connect.");
return;
}
Interlocked.Increment(ref _read);
}
}
catch (Exception e)
{
Debug.LogError($"[BingBongVoice] Uplink send loop failed: {e}");
return;
}
}
}
private static void WriteUInt32(byte[] buffer, int offset, uint value)
{
buffer[offset] = (byte)(value & 0xFF);
buffer[offset + 1] = (byte)((value >> 8) & 0xFF);
buffer[offset + 2] = (byte)((value >> 16) & 0xFF);
buffer[offset + 3] = (byte)((value >> 24) & 0xFF);
}
}

View File

@@ -0,0 +1,228 @@
using System;
using System.Threading;
using UnityEngine;
namespace JordanMod.Modules.BingBongVoice;
/// <summary>
/// Turns the local microphone into 20 ms frames of 16 kHz mono PCM16, ready to go on the wire.
///
/// Fed by <see cref="BingBongMicTapPatch"/> from Photon's mic read, measured at 48000 Hz mono in
/// 960-sample frames arriving 50 times a second on Unity's main thread -- so the microphone itself
/// is the clock and there is deliberately no pacer here. A deadline pacer on top of a hardware
/// clock would only fight it and guarantee drift in one direction.
///
/// Allocation-free after the first callback: this runs on the audio path of every frame.
/// </summary>
public sealed class MicUplinkSource
{
public const int WireRate = 16000;
public const int FrameSamples = 320; // 20 ms at 16 kHz
private readonly short[] _frame = new short[FrameSamples];
private int _frameFill;
// Scratch copy of the incoming buffer. We must never filter in place: that array belongs to
// Photon and is encoded and sent to teammates immediately after this postfix returns, so
// writing to it would corrupt everyone else's audio.
private float[]? _work;
// Resampler state, carried across callbacks. _tail is the previous buffer's last sample and is
// addressed as index -1, so the interpolation spans the seam instead of skipping it.
private double _read;
private float _tail;
// Five cascaded one-pole low-passes. Decimating 48k to 16k without one folds everything above
// 8 kHz down into the speech band, which sounds harsh and cannot be undone afterwards. Two
// poles measured only -14 dB of rejection at 12 kHz; five at 7 kHz gives -25 dB while still
// passing 95% at 1 kHz, where speech energy actually is.
private const int FilterPoles = 5;
private readonly float[] _lp = new float[FilterPoles];
private float _lpCoefficient = 1f;
private int _rate;
private int _channels = 1;
private int _sourceId = -1;
private float _gain;
private readonly float _gainStep = 1f / (WireRate * 0.015f); // ~15 ms attack
private volatile float _targetGain;
private volatile bool _gateOpen;
private volatile float _lastFrameLevel;
private volatile float _noiseFloorPublished;
private long _framesProduced;
private float _noiseFloor;
private int _holdFrames;
/// Set by the proximity logic on the main thread. 0 mutes; the ramp keeps it click-free.
public void SetTargetGain(float gain) => _targetGain = Mathf.Clamp(gain, 0f, 8f);
/// Whether the uplink may transmit at all (range, push-to-talk, speaker present).
public void SetGate(bool open) => _gateOpen = open;
/// Mean |sample| of the last completed frame, for the level indicator.
public float Level => _lastFrameLevel;
public long FramesProduced => Interlocked.Read(ref _framesProduced);
/// Raised for each completed 20 ms frame that passes the gate. Assigned once, never per frame.
public Action<short[]>? OnFrame;
/// <summary>
/// Emit gated frames as silence rather than dropping them, keeping the output continuous.
///
/// Off for the real uplink: the whole point of the gate is not to spend bandwidth on silence,
/// and the receiver is told to freeze its timeline instead. On for loopback, where the frames
/// feed a jitter-buffered player with no control channel -- dropping 70% of them starves it
/// into permanent rebuffering, which sounds like the capture path is broken when it is fine.
/// </summary>
public bool EmitSilenceWhenGated;
private readonly short[] _silence = new short[FrameSamples];
public void OnMicFrame(float[] buffer, int rate, int channels, int sourceId)
{
if (buffer == null || buffer.Length == 0 || rate <= 0) return;
// The mic device can change mid-session (CharacterVoiceHandler reassigns the recorder's
// device when the setting changes), rebuilding MicWrapper with a possibly different rate.
// Starting the resampler over is cheaper and safer than trying to splice.
if (rate != _rate || channels != _channels || sourceId != _sourceId)
{
Reset(rate, channels, sourceId);
Debug.Log($"[BingBongVoice] Uplink capture at {rate}Hz x{channels}, resampling {(double)rate / WireRate:F2}:1 down to {WireRate}Hz.");
}
int count = _channels > 1 ? buffer.Length / _channels : buffer.Length;
if (count < 2) return;
if (_work == null || _work.Length < count) _work = new float[count];
// Downmix and anti-alias in one pass into our own buffer. The filter has to see every
// input sample, not just the ones the cursor lands on, or it is not an anti-alias filter.
bool filter = _lpCoefficient < 1f;
for (int i = 0; i < count; i++)
{
float x;
if (_channels > 1)
{
float sum = 0f;
int b = i * _channels;
for (int c = 0; c < _channels; c++) sum += buffer[b + c];
x = sum / _channels;
}
else x = buffer[i];
if (filter)
{
for (int p = 0; p < FilterPoles; p++)
{
_lp[p] += (x - _lp[p]) * _lpCoefficient;
x = _lp[p];
}
}
_work[i] = x;
}
Resample(_work, count);
}
private void Reset(int rate, int channels, int sourceId)
{
_rate = rate;
_channels = Math.Max(1, channels);
_sourceId = sourceId;
_read = 0;
_tail = 0f;
Array.Clear(_lp, 0, _lp.Length);
_frameFill = 0;
_gain = 0f;
_noiseFloor = 0f;
_holdFrames = 0;
// One-pole cutoff at ~6.5 kHz, safely under the 8 kHz Nyquist of the 16 kHz wire rate.
_lpCoefficient = rate > WireRate ? 1f - Mathf.Exp(-2f * Mathf.PI * 7000f / rate) : 1f;
}
private void Resample(float[] work, int count)
{
double ratio = (double)_rate / WireRate;
double pos = _read; // may be negative; -1 addresses the carried tail sample
while (true)
{
int i0 = (int)Math.Floor(pos);
if (i0 + 1 >= count) break;
float a = i0 < 0 ? _tail : work[i0];
float b = work[i0 + 1];
float frac = (float)(pos - i0);
_gain += (_targetGain - _gain) * _gainStep;
Push((a * (1f - frac) + b * frac) * _gain);
pos += ratio;
}
_tail = work[count - 1];
_read = pos - count; // carries the fractional remainder to the next buffer
if (_read < -1) _read = -1;
}
/// <summary>
/// Adaptive gate. A fixed threshold cannot work here: measured on real hardware, a silent but
/// noisy room read louder (mean 0.0088) than the same person actually speaking in a quiet one
/// (0.0018), so no single level separates speech from silence across setups. Instead track the
/// background and require speech to stand above it.
///
/// The floor follows quiet quickly and loud very slowly, so it settles on the room rather than
/// on the voice. A hold keeps the gate open through the gaps between words.
/// </summary>
private bool PassesNoiseGate(float level)
{
if (_noiseFloor <= 0f) _noiseFloor = level;
else if (level < _noiseFloor) _noiseFloor += (level - _noiseFloor) * 0.25f; // drop fast
else _noiseFloor += (level - _noiseFloor) * 0.0008f; // rise slowly
_noiseFloorPublished = _noiseFloor;
// Never transmit below the absolute floor, whatever the adaptive part decides.
if (level < Mathf.Pow(10f, ConfigHandler.BingBongVoiceUplinkNoiseGateDb.Value / 20f)) return false;
float open = _noiseFloor * ConfigHandler.BingBongVoiceUplinkGateSensitivity.Value;
float close = open * 0.6f; // hysteresis, so a steady voice cannot chatter the gate
if (level >= open) _holdFrames = 15; // ~300 ms, bridges the gaps between words
else if (level < close && _holdFrames > 0) _holdFrames--;
return _holdFrames > 0;
}
/// The learned background level, for diagnostics.
public float NoiseFloor => _noiseFloorPublished;
private void Push(float sample)
{
if (sample > 1f) sample = 1f;
else if (sample < -1f) sample = -1f;
_frame[_frameFill++] = (short)(sample * 32767f);
if (_frameFill < FrameSamples) return;
_frameFill = 0;
float sum = 0f;
for (int i = 0; i < FrameSamples; i++)
{
int v = _frame[i];
sum += v < 0 ? -v : v;
}
_lastFrameLevel = sum / FrameSamples / 32768f;
Interlocked.Increment(ref _framesProduced);
// Gated after framing, so the level meter keeps reading while muted and the frame counter
// stays a measure of capture rather than of transmission.
bool pass = _gateOpen && PassesNoiseGate(_lastFrameLevel);
if (pass) OnFrame?.Invoke(_frame);
else if (EmitSilenceWhenGated) OnFrame?.Invoke(_silence);
}
}

View File

@@ -0,0 +1,81 @@
using System;
using System.Threading;
namespace JordanMod.Modules.BingBongVoice;
/// <summary>
/// Ring of mono float samples at the stream's sample rate. Single producer (the websocket
/// receive thread), multiple consumers (one <see cref="BingBongVoiceSource"/> per BingBong,
/// each reading from the audio thread with its own cursor).
///
/// Lock-free by construction: the producer publishes <see cref="WritePosition"/> only after the
/// samples are in place, and the ring is far deeper than the jitter target, so a consumer can
/// never observe a slot that is being overwritten under it. Consumers hold absolute positions
/// and resync if they ever fall further behind than the ring is long.
/// </summary>
public sealed class VoiceRingBuffer
{
private readonly float[] _buf;
private readonly int _mask;
private long _write;
public int SampleRate { get; }
public int Capacity => _buf.Length;
/// Total samples ever written. Monotonic; never reset, so consumer cursors stay meaningful.
public long WritePosition => Volatile.Read(ref _write);
public VoiceRingBuffer(int sampleRate, float seconds = 4f)
{
SampleRate = sampleRate;
int wanted = Math.Max(1024, (int)(sampleRate * seconds));
int size = 1;
while (size < wanted) size <<= 1; // power of two so the wrap is a mask, not a modulo
_buf = new float[size];
_mask = size - 1;
}
/// Producer thread only. Converts interleaved little-endian PCM16 straight out of the
/// websocket's byte[] with no intermediate allocation.
public void WritePcm16(byte[] src, int offset, int byteCount)
{
int samples = byteCount >> 1;
if (samples <= 0) return;
long w = _write; // safe unsynchronised read: this thread is the only writer
for (int i = 0; i < samples; i++)
{
int b = offset + (i << 1);
short s = (short)(src[b] | (src[b + 1] << 8));
_buf[(int)((w + i) & _mask)] = s * (1f / 32768f);
}
Volatile.Write(ref _write, w + samples); // publish only once the data is really there
}
/// Producer thread only. Same as WritePcm16 but from samples we produced ourselves, so the
/// loopback path does not have to encode to bytes and immediately decode again.
public void WritePcm16Samples(short[] src, int count)
{
if (count <= 0) return;
long w = _write;
for (int i = 0; i < count; i++) _buf[(int)((w + i) & _mask)] = src[i] * (1f / 32768f);
Volatile.Write(ref _write, w + count);
}
/// Producer thread only. Used to reproduce an exact gap the sender told us about, so a
/// dropped frame costs its own duration rather than shifting everything after it.
public void WriteSilence(int samples)
{
if (samples <= 0) return;
if (samples > _buf.Length) samples = _buf.Length;
long w = _write;
for (int i = 0; i < samples; i++) _buf[(int)((w + i) & _mask)] = 0f;
Volatile.Write(ref _write, w + samples);
}
/// Consumer read. The caller is responsible for staying within
/// <see cref="Capacity"/> of <see cref="WritePosition"/>.
public float this[long index] => _buf[(int)(index & _mask)];
}

View File

@@ -1,5 +1,6 @@
using System;
using UnityEngine;
using Zorro.Core;
namespace JordanMod.Modules.EasyBackpack;
@@ -38,19 +39,42 @@ class EasyBackpackModule : Module
Character localCharacter = Character.localCharacter;
if (localCharacter == null) return;
bool hasBackpack = localCharacter.player.backpackSlot.hasBackpack;
Character carriedCharacter = localCharacter.data.carriedPlayer;
bool carriedHasBackpack = carriedCharacter != null && carriedCharacter.player.backpackSlot.hasBackpack;
BackpackReference backpackRefs;
if (hasBackpack) backpackRefs = BackpackReference.GetFromEquippedBackpack(localCharacter);
else if (carriedHasBackpack) backpackRefs = BackpackReference.GetFromEquippedBackpack(carriedCharacter);
Character targetCharacter;
BackpackSlot targetSlot;
if (localCharacter.player.backpackSlot.backpackType != BackpackSlot.BackpackType.None)
{
targetCharacter = localCharacter;
targetSlot = localCharacter.player.backpackSlot;
}
else if (carriedCharacter != null && carriedCharacter.player.backpackSlot.backpackType != BackpackSlot.BackpackType.None)
{
targetCharacter = carriedCharacter;
targetSlot = carriedCharacter.player.backpackSlot;
}
else return;
GUIManager.instance.OpenBackpackWheel(backpackRefs);
// The wheel we open is the one for the backpack worn on the target's back, and its
// on-back visuals are what BackpackWheel.Choose() pulls items out of. While the
// backpack is held in hand (its slot, ID 3, is the equipped one) CharacterBackpackHandler
// deactivates those visuals, so anything taken out of the wheel is spawned into a
// disabled rig and vanishes. Refuse to open in that state; the game's own
// hold-interact on the held backpack still works and opens the correct wheel.
if (IsBackpackInHands(targetCharacter)) return;
BackpackReference backpackRefs = BackpackReference.GetFromEquippedBackpack(targetCharacter);
int slotCount = (targetSlot.prefab as Backpack)?.slotCount ?? 4;
GUIManager.instance.OpenBackpackWheel(backpackRefs, slotCount, targetSlot.backpackType);
_isBackpackOpen = true;
}
private static bool IsBackpackInHands(Character character)
{
Optionable<byte> selectedSlot = character.refs.items.currentSelectedSlot;
return selectedSlot.IsSome && selectedSlot.Value == character.player.backpackSlot.itemSlotID;
}
private void CloseBackpack()
{
if (!Application.isFocused) return;

View File

@@ -0,0 +1,28 @@
using System;
using UnityEngine;
namespace JordanMod.Modules.OpenPassport;
[Module(Enabled = true)]
class OpenPassportModule : Module
{
public override string ModuleName => "Open Passport Module";
public override Type[] GetPatches() { return []; }
public override void Update()
{
if (!Input.GetKeyDown(ConfigHandler.OpenPassport.Value)) return;
// PassportManager.instance is a plain static that survives a scene unload, so it can
// be a destroyed object here; the Unity == operator is what catches that. Its
// Initialize() also reads Character.localCharacter.characterName, so don't open it
// before we have a character.
PassportManager passportManager = PassportManager.instance;
if (passportManager == null) return;
if (Character.localCharacter == null) return;
if (passportManager.isOpen) return;
passportManager.ToggleOpen();
}
}

View File

@@ -12,6 +12,8 @@ class ReplaceBingBongModule : Module
{
public override string ModuleName => "Replace BingBong Module";
public const string BingBongSubtitleID = "IDK_FUNNY";
public static bool HasReplacedSounds = false;
public static BingBongResponseData[] OriginalResponsesData = [];
@@ -23,31 +25,64 @@ class ReplaceBingBongModule : Module
public override void Initialize()
{
base.Initialize();
LocalizedText.mainTable.Add("idk_funny", ["Test subtitle!"]);
// LocalizedText.GetText upper-cases the id before looking it up, so a lowercase key never
// resolved at all. It also indexes the list by CURRENT_LANGUAGE, so a single-entry list
// threw IndexOutOfRange (caught and logged by GetText) for every non-English player.
List<string> subtitleLocalizations = new(LocalizedText.LANGUAGE_COUNT);
for (int i = 0; i < LocalizedText.LANGUAGE_COUNT; i++) subtitleLocalizations.Add("Test subtitle!");
LocalizedText.mainTable[BingBongSubtitleID] = subtitleLocalizations;
AudioSyncWorker.OnAudioLoadComplete += OnAudioLoadComplete;
}
public override void Update()
{
base.Update();
if (Input.GetKeyDown(KeyCode.P))
{
Helper.FindItemByName("BingBong_Prop Variant", out Item? item);
if (item == null) return;
Debug.Log($"Found item: {item.name} in scene {item.gameObject.scene.name}");
}
}
private static void OnAudioLoadComplete()
{
if (!HasReplacedSounds) return;
Action_AskBingBong[] allBingBongActions = UnityEngine.Object.FindObjectsByType<Action_AskBingBong>(FindObjectsSortMode.None);
foreach (Action_AskBingBong askBingBong in allBingBongActions) {
ReplaceBingBongResponses(askBingBong);
RebuildResponsesForAll();
}
// Build the new set and hand it to every live instance before destroying the old one, so no
// Action_AskBingBong is ever left pointing at a destroyed SFX_Instance.
private static void RebuildResponsesForAll()
{
Action_AskBingBong.BingBongResponse[]? previous = _cachedResponses;
_cachedResponses = BuildResponses();
foreach (Action_AskBingBong askBingBong in UnityEngine.Object.FindObjectsByType<Action_AskBingBong>(FindObjectsSortMode.None))
{
askBingBong.responses = _cachedResponses;
}
if (previous == null) return;
foreach (Action_AskBingBong.BingBongResponse response in previous)
{
if (response.sfx != null) UnityEngine.Object.Destroy(response.sfx);
}
}
// SFX_Instance derives from ScriptableObject, which cannot be constructed with `new` --
// doing so skips Unity's native-side init, so the object misbehaves and its overloaded
// == null check reports oddly. CreateInstance is the only correct way to make one.
public static SFX_Instance CreateSFX(string name, AudioClip[] clips)
{
SFX_Instance instance = ScriptableObject.CreateInstance<SFX_Instance>();
instance.name = name;
instance.clips = clips;
return instance;
}
// Every Action_AskBingBong ends up with the same set -- it is derived purely from static
// state (OriginalResponsesData plus the loaded voice lines). This used to be rebuilt on every
// OnEnable, allocating a fresh SFX_Instance per response each time and never destroying the
// previous ones, so BingBong items cycling in and out of scope leaked steadily.
public static void ReplaceBingBongResponses(Action_AskBingBong askBingBong)
{
_cachedResponses ??= BuildResponses();
askBingBong.responses = _cachedResponses;
}
private static Action_AskBingBong.BingBongResponse[]? _cachedResponses;
private static Action_AskBingBong.BingBongResponse[] BuildResponses()
{
Action_AskBingBong.BingBongResponse[] currentResponses = new Action_AskBingBong.BingBongResponse[OriginalResponsesData.Length];
for (int index = 0; index < OriginalResponsesData.Length; index++)
@@ -55,8 +90,6 @@ class ReplaceBingBongModule : Module
currentResponses[index] = OriginalResponsesData[index].ToBingBongResponse();
}
askBingBong.responses = [];
Dictionary<string, SFX_Instance> sfxDict = new();
for (int i = 0; i < currentResponses.Length; i++)
{
@@ -79,14 +112,10 @@ class ReplaceBingBongModule : Module
bool isNew = !sfxDict.ContainsKey(voice.Name);
if (isNew)
{
SFX_Instance sFX_Instance = new()
{
clips = [clip]
};
Action_AskBingBong.BingBongResponse newResponse = new()
{
sfx = sFX_Instance,
subtitleID = "idk_funny",
sfx = CreateSFX(voice.Name, [clip]),
subtitleID = BingBongSubtitleID,
mouthCurve = null,
mouthCurveTime = 1f
};
@@ -98,11 +127,7 @@ class ReplaceBingBongModule : Module
}
}
askBingBong.responses = new Action_AskBingBong.BingBongResponse[currentResponses.Length];
for (int i = 0; i < currentResponses.Length; i++)
{
askBingBong.responses[i] = currentResponses[i];
}
return currentResponses;
}
@@ -122,11 +147,7 @@ public class BingBongResponseData
{
return new Action_AskBingBong.BingBongResponse
{
sfx = new SFX_Instance
{
name = SfxName,
clips = (AudioClip[])Clips.Clone()
},
sfx = ReplaceBingBongModule.CreateSFX(SfxName, (AudioClip[])Clips.Clone()),
subtitleID = SubtitleID,
mouthCurve = MouthCurve,
mouthCurveTime = MouthCurveTime

View File

@@ -31,17 +31,12 @@ class StashedBugleModule : Module
{
if (heldItem.UIData.itemName == _bugleItemName)
{
localCharacter.refs.items.DestroyHeldItemRpc();
localCharacter.player.EmptySlot(localCharacter.refs.items.currentSelectedSlot);
localCharacter.player.RPCRemoveItemFromSlot(localCharacter.refs.items.currentSelectedSlot.Value);
localCharacter.refs.items.SpawnItemInHand(_megaphoneItemName);
ClearHeldSlot(localCharacter);
SpawnItemByDisplayName(localCharacter, _megaphoneItemName);
}
else if (heldItem.UIData.itemName == _megaphoneItemName)
{
localCharacter.refs.items.DestroyHeldItemRpc();
localCharacter.player.EmptySlot(localCharacter.refs.items.currentSelectedSlot);
localCharacter.player.RPCRemoveItemFromSlot(localCharacter.refs.items.currentSelectedSlot.Value);
ClearHeldSlot(localCharacter);
}
}
else if (heldItem == null)
@@ -54,8 +49,32 @@ class StashedBugleModule : Module
withBugleSlot = itemSlot;
break;
}
if (withBugleSlot == null) localCharacter.refs.items.SpawnItemInHand(_bugleItemName);
if (withBugleSlot == null) SpawnItemByDisplayName(localCharacter, _bugleItemName);
else localCharacter.refs.items.EquipSlot(Optionable<byte>.Some(withBugleSlot.itemSlotID));
}
}
// Player.EmptySlot already tells the host to clear the slot (it RPCs
// RPCRemoveItemFromSlot to the master client when we aren't the host), so calling
// RPCRemoveItemFromSlot ourselves on top of it only logged "Only Master Client can
// remove items!" on every non-host press.
private static void ClearHeldSlot(Character character)
{
character.refs.items.DestroyHeldItemRpc();
character.player.EmptySlot(character.refs.items.currentSelectedSlot);
}
// SpawnItemInHand expects the item's Resources prefab name, which is not
// guaranteed to match its UIData.itemName display string, so resolve it
// via the ItemDatabase instead of assuming the two are identical.
private static void SpawnItemByDisplayName(Character character, string displayName)
{
Item? match = Helper.ItemDatabase.Objects.Find(item => item.UIData != null && item.UIData.itemName == displayName);
if (match == null)
{
Debug.LogWarning($"[StashedBugle] Could not find an item with UIData.itemName '{displayName}' in ItemDatabase.");
return;
}
character.refs.items.SpawnItemInHand(match.name);
}
}

View File

@@ -1,4 +1,7 @@
using System;
using System.Collections.Generic;
using HarmonyLib;
using Peak;
using Photon.Pun;
using UnityEngine;
@@ -7,34 +10,48 @@ namespace JordanMod.Modules.BagsForEveryone;
public class BagsForEveryonePatch
{
[HarmonyPatch(typeof(SingleItemSpawner), "TrySpawnItems")]
[HarmonyPostfix]
static void Postfix(SingleItemSpawner __instance)
{
ForEachPlayer(__instance);
}
private const string BackpackSpawnerName = "Backpack_Spawner";
private const string FirstBiomeName = "Biome_1";
private static void ForEachPlayer(SingleItemSpawner spawner)
[HarmonyPatch(typeof(SingleItemSpawner), "TrySpawnItems")]
[HarmonyPostfix]
static void Postfix(SingleItemSpawner __instance, List<PhotonView> __result)
{
if (!Helper.IsMasterClient()) return;
if (__instance.transform.name != BackpackSpawnerName) return;
bool isBackpackSPawner = spawner.transform.name == "Backpack_Spawner";
if (!isBackpackSPawner) return;
bool isFirstSpawner = spawner.transform.IsChildOf(GameObject.Find("Biome_1").transform);
if (!isFirstSpawner) return;
GameObject firstBiome = GameObject.Find(FirstBiomeName);
if (firstBiome == null || !__instance.transform.IsChildOf(firstBiome.transform)) return;
if (__result != null && __result.Count > 1) return;
int playerCount = PhotonNetwork.PlayerList.Length - 1;
if (playerCount <= 0) return;
for (int i = 1; i <= playerCount; i++)
int extraBags = PhotonNetwork.PlayerList.Length - 1;
if (extraBags <= 0) return;
try
{
Vector3 spawnPosition = spawner.transform.position + Vector3.up * 0.1f + Vector3.right * i;
PhotonView component = PhotonNetwork.InstantiateItemRoom(spawner.prefab.name, spawnPosition, spawner.transform.rotation).GetComponent<PhotonView>();
if (spawner.isKinematic)
{
component.GetComponent<PhotonView>().RPC("SetKinematicRPC", RpcTarget.AllBuffered, true, component.transform.position, component.transform.rotation);
}
SpawnExtraBags(__instance, extraBags);
}
catch (Exception e)
{
Plugin.Log.LogError($"Failed to spawn extra backpacks, continuing without them: {e}");
}
}
}
private static void SpawnExtraBags(SingleItemSpawner spawner, int count)
{
List<PhotonView> spawned = new(count);
for (int i = 1; i <= count; i++)
{
Vector3 spawnPosition = spawner.transform.position + Vector3.up * 0.1f + Vector3.right * i;
PhotonView view = PhotonNetwork.InstantiateItemRoom(spawner.prefab.name, spawnPosition, spawner.transform.rotation).GetComponent<PhotonView>();
if (spawner.isKinematic)
{
view.RPC("SetKinematicRPC", RpcTarget.AllBuffered, true, view.transform.position, view.transform.rotation);
}
spawned.Add(view);
}
if (spawner.HasSpawnTracking(out SpawnedItemTracker tracker)) tracker.TrackSpawnedItems(spawned);
}
}

View File

@@ -10,16 +10,23 @@ namespace JordanMod.Modules.BetterBugle;
public class BetterBuglePatch
{
private static readonly List<string> SupportedItemNames = ["Bugle", "Bugle_Magic", "Megaphone"];
[HarmonyPatch(typeof(Item), "Start")]
[HarmonyPostfix]
static void ItemStartPostfix(Item __instance)
{
if (__instance.itemState != ItemState.Held) return;
if (__instance.UIData == null) return;
List<string> supportedItemNames = ["Bugle", "Bugle_Magic", "Megaphone"];
if (!SupportedItemNames.Contains(__instance.UIData.itemName)) return;
if (!supportedItemNames.Contains(__instance.UIData.itemName)) return;
// This used to bail unless itemState was already Held. ItemState.Ground is the enum's
// zero value and Item.Start doesn't set state, so anything spawned into the world never
// got a BetterBugleSFX -- and since BugleSFXUpdatePostfix silences the vanilla BugleSFX,
// a Bugle picked up off the ground was completely silent. It only ever worked because
// SpawnItemInHand calls Interact() on the same frame it instantiates, landing
// SetState(Held) before Start runs.
if (__instance.TryGetComponent<BetterBugleSFX>(out _)) return;
{
Action secondaryAction = OnRightClick;
Action<float> scrollAction = OnScroll;
@@ -45,7 +52,7 @@ public class BetterBuglePatch
// BetterBugleUI.Instance?.ShowActionbar("No songs available.");
// return;
// }
if (AudioSyncWorker.IsLoading) return;
if (AudioSyncWorker.IsLoading || AudioSyncWorker.IsSyncing) return;
if (!BetterBugleModule.HadConfirmation)
{
BetterBugleUI.Instance?.ShowActionbar("Are you sure you want to refresh songs ? Right-click again to reload.");
@@ -57,15 +64,18 @@ public class BetterBuglePatch
{
BetterBugleModule.HadConfirmation = false; // Reset confirmation state
BetterBugleUI.Instance?.ShowActionbar("Refreshing songs...");
AudioSyncService.GetAudioClips();
// Refresh = unload everything and reload from local files only (no network).
// For downloading from the audio bank API, use the Sync keybind instead.
AudioSyncService.ClearAudioClips();
AudioSyncWorker.GetAudioClips();
}
}
private static IEnumerator ResetConfirmation()
{
if (!BetterBugleModule.HadConfirmation) yield break;
yield return new WaitForSeconds(2f);
if (!BetterBugleModule.HadConfirmation) yield break; // already confirmed (or reset) in the meantime
BetterBugleUI.Instance?.ShowActionbar("No answer, not refreshing songs.");
BetterBugleModule.HadConfirmation = false;
}
@@ -117,6 +127,9 @@ public class BetterBuglePatch
[HarmonyPostfix]
static void BugleSFXUpdatePostfix(BugleSFX __instance)
{
// Only silence the vanilla bugle where our replacement actually took over. Muting
// unconditionally meant any bugle we failed to attach to played nothing at all.
if (!__instance.TryGetComponent<BetterBugleSFX>(out _)) return;
if (__instance.volume > 0f) __instance.volume = 0;
}

View File

@@ -0,0 +1,156 @@
using System;
using System.Runtime.CompilerServices;
using System.Threading;
using HarmonyLib;
using Photon.Voice.Unity;
namespace JordanMod.Modules.BingBongVoice;
/// <summary>
/// Taps the local microphone for the voice uplink.
///
/// Feeds <see cref="MicUplinkSource"/>, and on the first five seconds of a session also logs a
/// one-off measurement of the capture conditions. That measurement came first and confirmed, on
/// real hardware, the assumptions the design rests on — 48000 Hz mono, 960-sample frames, 50 a
/// second, on Unity's main thread. It is kept because a mic device change or a Photon update could
/// invalidate any of them, and a wrong premise here is silent rather than loud.
///
/// What it settles:
/// - Which thread the mic callback arrives on (statically it looked like Unity's main thread via
/// VoiceConnection.Update, but LoadBalancingTransport.Service also reaches VoiceClient.Service
/// and that path could not be traced).
/// - The device's real sample rate and channel count, which drive resampling and downmixing.
/// MicWrapper is authoritative here; Recorder.SamplingRate is the *encoder* rate and can differ.
/// - The frame length, and specifically whether it exceeds 256. Photon's MicrophoneRelay.SendMic
/// copies only the first 256 samples before invoking its listeners, which is why that API — the
/// obvious one to use — cannot carry continuous audio. If frames really are longer than 256,
/// that finding is confirmed and patching Read here is the right call.
/// - Signal levels, to calibrate the noise gate. The tap sits upstream of Photon's WebRtcAudioDsp
/// (AEC/NS/AGC attach as post-processors), so this is the raw microphone.
///
/// Structured like BingBongVoicePatch: the patch body is only a try/catch around a NoInlining
/// method, with three strikes then permanent disable. That matters more here than anywhere else in
/// the mod — this sits on Photon's per-frame audio path, so an exception escaping would break all
/// voice chat, not just this feature.
/// </summary>
public class BingBongMicTapPatch
{
private static int _strikes;
private static bool _disabled;
private static int _mainThreadId = -1;
private static int _observedThreadId = -1;
private static int _calls;
private static int _minLength = int.MaxValue;
private static int _maxLength;
private static int _rate;
private static int _channels;
private static float _peak;
private static double _sumAbs;
private static long _samples;
private static float _firstCallTime;
private static bool _reported;
private static bool _waitedForMic;
/// Called from the module on the main thread, so the callback's thread can be compared to it.
public static void NoteMainThread() => _mainThreadId = Thread.CurrentThread.ManagedThreadId;
[HarmonyPatch(typeof(MicWrapper), nameof(MicWrapper.Read))]
[HarmonyPostfix]
static void ReadPostfix(MicWrapper __instance, float[] __0, bool __result)
{
if (_disabled || !__result) return;
try
{
if (!_reported) Measure(__instance, __0);
Feed(__instance, __0);
}
catch (Exception e)
{
Fail(e);
}
}
// The only method here that touches Photon or Unity types, kept separate and un-inlined so a
// member that changed shape throws inside the caller's try rather than while JITing the patch.
[MethodImpl(MethodImplOptions.NoInlining)]
private static void Measure(MicWrapper mic, float[] buffer)
{
if (buffer == null || buffer.Length == 0) return;
if (_calls == 0)
{
_observedThreadId = Thread.CurrentThread.ManagedThreadId;
_firstCallTime = UnityEngine.Time.realtimeSinceStartup;
_rate = mic.SamplingRate;
_channels = mic.Channels;
}
_calls++;
if (buffer.Length < _minLength) _minLength = buffer.Length;
if (buffer.Length > _maxLength) _maxLength = buffer.Length;
// Sample sparsely: this runs on an audio-adjacent hot path and a full scan every frame
// would be a cost we are not here to pay.
for (int i = 0; i < buffer.Length; i += 7)
{
float abs = buffer[i] < 0f ? -buffer[i] : buffer[i];
if (abs > _peak) _peak = abs;
_sumAbs += abs;
_samples++;
}
if (_calls < 250) return; // ~5 seconds at 50 frames/sec
// Photon binds the real capture device a few seconds after joining -- until then it reports
// "[Default] (-128)" and hands out silence. Measuring that window produces an all-zero
// report that looks exactly like a broken microphone, so start over instead.
if (_peak <= 0f)
{
_calls = 0;
_sumAbs = 0;
_samples = 0;
_minLength = int.MaxValue;
_maxLength = 0;
if (!_waitedForMic)
{
_waitedForMic = true;
Debug.Log("[BingBongVoice] Microphone is not producing yet (Photon has not bound the device); will measure again shortly.");
}
return;
}
_reported = true;
float elapsed = UnityEngine.Time.realtimeSinceStartup - _firstCallTime;
float mean = _samples > 0 ? (float)(_sumAbs / _samples) : 0f;
bool onMainThread = _observedThreadId == _mainThreadId;
bool longerThan256 = _maxLength > 256;
Debug.Log(
$"[BingBongVoice] MIC TAP (stage 0): {_calls} frames in {elapsed:F1}s ({_calls / Math.Max(0.01f, elapsed):F0}/s), " +
$"rate={_rate}Hz channels={_channels}, frame length min={_minLength} max={_maxLength}, " +
$"peak={_peak:F3} mean|s|={mean:F4}, thread={_observedThreadId} (main={_mainThreadId}, {(onMainThread ? "MAIN" : "NOT main")}).");
Debug.Log(longerThan256
? $"[BingBongVoice] Frames are {_maxLength} samples, longer than the 256 MicrophoneRelay forwards - confirmed, that API cannot carry audio and patching MicWrapper.Read is correct."
: $"[BingBongVoice] Frames are only {_maxLength} samples, so MicrophoneRelay would NOT have truncated them. Revisit the capture design before building on this patch.");
}
// The only path that reaches the uplink. Separate and un-inlined for the same reason Measure
// is: a Photon member that changed shape must throw inside the caller's try.
[MethodImpl(MethodImplOptions.NoInlining)]
private static void Feed(MicWrapper mic, float[] buffer)
{
MicUplinkSource? source = BingBongVoiceModule.Instance?.Uplink;
source?.OnMicFrame(buffer, mic.SamplingRate, mic.Channels, mic.GetHashCode());
}
private static void Fail(Exception e)
{
Debug.LogError($"[BingBongVoice] Mic tap failed: {e}");
if (++_strikes < 3) return;
_disabled = true;
Debug.LogError("[BingBongVoice] Mic tap disabled for this session after repeated failures.");
}
}

View File

@@ -0,0 +1,135 @@
using System;
using System.Runtime.CompilerServices;
using HarmonyLib;
using UnityEngine;
namespace JordanMod.Modules.BingBongVoice;
/// <summary>
/// Gives every BingBong a live-voice source as it spawns.
///
/// Structured defensively on purpose. A postfix that throws propagates out of the method it
/// patched, and because Mono resolves a method's call targets when it JITs that method, a game
/// API that changed shape throws before any guard inside can run -- that is how a spawner
/// postfix once took down every SingleItemSpawner in the level (see 0.1.13). So the patch body
/// is nothing but a try/catch around a separate non-inlined method, and three failures disable
/// it for the session. A broken voice feature must never cost anyone their BingBong.
/// </summary>
public class BingBongVoicePatch
{
private static int _strikes;
private static bool _disabled;
private static int _attachCount;
private static int _duckStrikes;
private static bool _duckDisabled;
[HarmonyPatch(typeof(Item), "Start")]
[HarmonyPostfix]
static void ItemStartPostfix(Item __instance)
{
if (_disabled) return;
try
{
AttachIfBingBong(__instance);
}
catch (Exception e)
{
Fail(e);
}
}
// The only method here that touches game APIs, kept separate and un-inlined so that a
// missing member throws inside the caller's try rather than while JITing the patch itself.
[MethodImpl(MethodImplOptions.NoInlining)]
private static void AttachIfBingBong(Item item)
{
if (item == null || item.UIData == null) return;
if (!item.itemTags.HasFlag(Item.ItemTags.BingBong)) return;
if (BingBongVoiceSource.IsAttached(item.gameObject)) return;
// Copy the vanilla voice-line source's 3D setup so the live voice carries and attenuates
// exactly like BingBong's own lines, and inherits its mixer group.
AudioSource? template = item.TryGetComponent(out Action_AskBingBong ask) ? ask.source : null;
BingBongVoiceSource voice = BingBongVoiceSource.Attach(item.gameObject, template);
// The mouth animator lives on a child of the plush, not the item root.
BingBongMouth mouth = item.GetComponentInChildren<BingBongMouth>(includeInactive: true);
if (mouth != null) voice.BindMouth(mouth);
// Log the copied values: if the live voice turns out not to attenuate with distance,
// this distinguishes "the copy failed" from "Unity spatialises before custom filters".
// Counted because this should happen once per BingBong that exists, ever. If the number
// climbs every time the item is grabbed or thrown then Item.Start is re-running on a
// fresh instance, and the per-attach cost (AudioClip.Create, AddComponent) is the hitch.
_attachCount++;
if (template != null)
Debug.Log($"[BingBongVoice] Attached to a BingBong (#{_attachCount}, {BingBongVoiceSource.LiveSources} live, {BingBongVoiceSource.PlayingSources} audible); spatialBlend={template.spatialBlend}, range={ConfigHandler.BingBongVoiceMaxDistance.Value}m, rolloff={template.rolloffMode}, mixer={(template.outputAudioMixerGroup != null ? template.outputAudioMixerGroup.name : "none")}.");
else
Debug.LogWarning("[BingBongVoice] Attached to a BingBong, but it has no Action_AskBingBong to copy 3D settings from - the voice will not be positional.");
}
private static void Fail(Exception e)
{
Debug.LogError($"[BingBongVoice] Attach patch failed: {e}");
if (++_strikes < 3) return;
_disabled = true;
Debug.LogError("[BingBongVoice] Attach patch disabled for this session after repeated failures.");
}
/// <summary>
/// Ducks the live voice while one of BingBong's own lines plays, so the two never talk over
/// each other. This is the RPC body, so it runs on every client and the duck stays in step
/// across the lobby without any networking of our own.
/// </summary>
[HarmonyPatch(typeof(Action_AskBingBong), nameof(Action_AskBingBong.Ask))]
[HarmonyPostfix]
static void AskPostfix(Action_AskBingBong __instance, int index, bool spamming)
{
if (_duckDisabled) return;
try
{
DuckForVoiceLine(__instance, index, spamming);
}
catch (Exception e)
{
Debug.LogError($"[BingBongVoice] Duck patch failed: {e}");
if (++_duckStrikes < 3) return;
_duckDisabled = true;
Debug.LogError("[BingBongVoice] Duck patch disabled for this session after repeated failures.");
}
}
[MethodImpl(MethodImplOptions.NoInlining)]
private static void DuckForVoiceLine(Action_AskBingBong ask, int index, bool spamming)
{
// Mirror vanilla's own early-outs, so we never duck for audio that is not going to play.
// Ask() does nothing at all unless the item is held, and AskRoutine bails before playing
// anything when it decides you are spamming the interaction.
if (ask.item == null || ask.item.holderCharacter == null) return;
if (spamming) return;
// RunAction picks the index locally and RPCs the raw number, while ReplaceBingBongModule
// appends a response per loaded voice line -- so two players with different audio bank
// state disagree on the array length. Vanilla throws in that case; we must not.
Action_AskBingBong.BingBongResponse[] responses = ask.responses;
if (responses == null || index < 0 || index >= responses.Length) return;
Action_AskBingBong.BingBongResponse response = responses[index];
if (response?.sfx == null || response.sfx.clips == null || response.sfx.clips.Length == 0) return;
AudioClip clip = response.sfx.clips[0];
if (clip == null) return;
BingBongVoiceSource? voice = BingBongVoiceSource.Find(ask.item.gameObject);
if (voice == null) return;
// AskRoutine burns 0.5s in its update loop, stops the source, waits another 0.5s and only
// then plays -- so audio starts about a second after this RPC lands. Ducking from now
// rather than from the actual start reads as BingBong drawing breath, and removes a whole
// class of "the duck began a few frames late and you heard a syllable" bugs.
float duration = ConfigHandler.BingBongVoiceDuckLead.Value + clip.length + 0.2f;
voice.DuckUntil = Math.Max(voice.DuckUntil, Time.time + duration);
}
}

View File

@@ -9,6 +9,12 @@ class EasyBackpackPatch
[HarmonyPrefix]
static bool BackpackWheelUpdatePrefix(BackpackWheel __instance)
{
if (!__instance.backpack.exists || __instance.backpack.locationTransform == null)
{
GUIManager.instance.CloseBackpackWheel();
return false;
}
bool isBackPackOpen = EasyBackpackModule.Instance?._isBackpackOpen ?? false;
if (!Character.localCharacter.input.interactIsPressed && !isBackPackOpen)
@@ -18,13 +24,17 @@ class EasyBackpackPatch
return false;
}
if (__instance.backpack.locationTransform != null && Vector3.Distance(__instance.backpack.locationTransform.position, Character.localCharacter.Center) > 6f)
if (Vector3.Distance(__instance.backpack.locationTransform.position, Character.localCharacter.Center) > 6f)
{
GUIManager.instance.CloseBackpackWheel();
return false;
}
if (__instance.chosenSlice.IsSome && !__instance.chosenSlice.Value.isBackpackWear && !__instance.slices[__instance.chosenSlice.Value.slotID + 1].image.enabled)
if (__instance.chosenSlice.IsSome && __instance.chosenSlice.Value.isJetpackFuelSlice)
{
__instance.currentlyHeldItem.transform.position = Vector3.Lerp(__instance.currentlyHeldItem.transform.position, __instance.jetpackSlice.transform.GetChild(0).GetChild(0).position, Time.deltaTime * 20f);
}
else if (__instance.chosenSlice.IsSome && !__instance.chosenSlice.Value.isBackpackWear && !__instance.slices[__instance.chosenSlice.Value.slotID + 1].image.enabled)
{
__instance.currentlyHeldItem.transform.position = Vector3.Lerp(__instance.currentlyHeldItem.transform.position, __instance.slices[__instance.chosenSlice.Value.slotID + 1].transform.GetChild(0).GetChild(0).position, Time.deltaTime * 20f);
}
@@ -32,7 +42,7 @@ class EasyBackpackPatch
{
__instance.currentlyHeldItem.transform.localPosition = Vector3.Lerp(__instance.currentlyHeldItem.transform.localPosition, Vector3.zero, Time.deltaTime * 20f);
}
return false;
return false;
}
}

View File

@@ -3,6 +3,7 @@ using System.Collections;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net;
using System.Security.Cryptography;
using System.Threading.Tasks;
using JordanMod.Modules.BetterBugle;
@@ -14,18 +15,33 @@ namespace JordanMod.Utils;
class AudioSyncService
{
public static string API_BASE_URL => ConfigHandler.BugleSoundAPIURL.Value;
// Normalised once here rather than at each call site: the configured value is hand-typed, so
// it may be missing a scheme or carry a trailing slash, either of which broke the Uri parse.
public static string API_BASE_URL => Helper.EnsureScheme(ConfigHandler.BugleSoundAPIURL.Value, "https", "http").TrimEnd('/');
public async static Task<bool> DownloadAPIAudio(APIAudioFormat apiAudio, string SoundsDirectory, Song? existingSong = null)
public static bool DownloadAPIAudio(APIAudioFormat apiAudio, string SoundsDirectory, Song? existingSong = null)
{
bool success = true;
try
{
// Same audio under a different name in the bank: retire the local copy.
if (existingSong != null && apiAudio.Filename != existingSong.Name)
{
File.Delete(Path.Combine(SoundsDirectory, $"{existingSong.Name}.{existingSong.Extension}"));
string renamedFrom = Path.Combine(SoundsDirectory, $"{existingSong.Name}.{existingSong.Extension}");
if (File.Exists(renamedFrom)) AudioSyncWorker.TryQuarantine(renamedFrom);
}
await apiAudio.DownloadToFolder(SoundsDirectory);
// Same name under a different extension -- a leftover .mp3 when the bank now serves
// .ogg, say. Both would be queued by the loader and fight over the same song name, so
// retire every variant that isn't the one we're about to write.
foreach (string ext in AudioSyncWorker.AudioTypes.Keys)
{
if (string.Equals(ext, apiAudio.Extension, StringComparison.OrdinalIgnoreCase)) continue;
string stale = Path.Combine(SoundsDirectory, $"{apiAudio.Filename}.{ext}");
if (File.Exists(stale)) AudioSyncWorker.TryQuarantine(stale);
}
apiAudio.DownloadToFolder(SoundsDirectory);
}
catch (Exception ex)
{
@@ -46,7 +62,7 @@ class AudioSyncService
return audioClips;
}
using var client = new System.Net.WebClient();
using var client = new WebClient();
try
{
string json = client.DownloadString(uri);
@@ -63,15 +79,19 @@ class AudioSyncService
public static void ClearAudioClips()
{
// One scene scan shared by every song. Song.Dispose() used to run its own
// FindObjectsByType<AudioSource>() call, so clearing a bank of N songs meant N full
// scans of the scene -- the bulk of the freeze on the Bugle's right-click refresh.
AudioSource[] audioSources = UnityEngine.Object.FindObjectsByType<AudioSource>(FindObjectsSortMode.None);
foreach (Song song in Song.Sounds.Values.ToList())
{
song.Dispose();
song.Dispose(audioSources);
}
Song.Sounds.Clear();
Song.SoundsByHash.Clear();
Song.Songs.Clear();
Song.BB_VoiceLines.Clear();
GC.Collect();
// No GC.Collect() here on purpose: the audio memory is native and is already released by
// Object.Destroy on each clip, so a blocking full collection only added a stall.
}
public class APIAudioFormat
@@ -100,7 +120,10 @@ class AudioSyncService
[JsonProperty("owner")]
public string Owner { get; set; } = string.Empty;
public async Task DownloadToFolder(string folderPath)
// Plain WebClient rather than UnityWebRequest: this runs on a background thread, and
// UnityWebRequest is main-thread only. The old version also spun on `await Task.Yield()`
// waiting for an operation that a pool thread never drives, burning a core.
public void DownloadToFolder(string folderPath)
{
if (string.IsNullOrEmpty(Filename) || string.IsNullOrEmpty(Extension))
{
@@ -115,17 +138,11 @@ class AudioSyncService
if (File.Exists(filePath)) File.Delete(filePath);
string url = $"{API_BASE_URL}/audio/{Id}/download?hash={Hash}";
Debug.LogError($"Downloading audio from URL: {url}");
Debug.Log($"Downloading audio from URL: {url}");
using UnityWebRequest www = UnityWebRequest.Get(url);
var operation = www.SendWebRequest();
while (!operation.isDone) await Task.Yield();
if (www.result != UnityWebRequest.Result.Success)
{
Debug.LogError($"Failed to download API audio: {www.error}");
return;
}
File.WriteAllBytes(filePath, www.downloadHandler.data);
using var client = new WebClient();
byte[] data = client.DownloadData(url);
File.WriteAllBytes(filePath, data);
}
}
@@ -142,6 +159,11 @@ class AudioSyncWorker
}
public static readonly string SoundsDirectory = Path.Combine(BepInEx.Paths.BepInExRootPath, "bugleSounds");
// Files the sync pulls out of the bank folder are moved here rather than deleted. It's a
// subfolder of the bank itself, which is safe because the loader's Directory.GetFiles calls
// are non-recursive and will never see it.
public static readonly string QuarantineDirectory = Path.Combine(SoundsDirectory, "_removed");
public static readonly Dictionary<string, AudioType> AudioTypes = new()
{
{ "wav", AudioType.WAV },
@@ -153,6 +175,11 @@ class AudioSyncWorker
public static bool IsLoading = false;
public static bool IsSyncing = false;
// How many clips arrived from the download handler without their audio data decoded. Any
// number above zero means the first play of each would otherwise have stalled the main
// thread; reported once per load so the log can confirm it.
public static int ClipsNeedingPreload = 0;
public static int CurrentSongIndex = 0;
public static string CurrentSongName = "None";
@@ -169,14 +196,66 @@ class AudioSyncWorker
public static void TrySyncAndLoadAudioClips()
{
if (IsLoading || IsSyncing) return;
Task.Run(() =>
Task.Run(SyncAndLoadAudioClips);
}
// Moves a file out of the bank folder instead of deleting it, so a mistaken sync is always
// recoverable. Pure file I/O, safe to call from the sync thread.
internal static bool TryQuarantine(string filePath)
{
try
{
SyncAndLoadAudioClipsCoroutine().GetAwaiter().GetResult();
});
Directory.CreateDirectory(QuarantineDirectory);
string target = Path.Combine(QuarantineDirectory, Path.GetFileName(filePath));
if (File.Exists(target))
{
string stem = Path.GetFileNameWithoutExtension(filePath);
string ext = Path.GetExtension(filePath);
target = Path.Combine(QuarantineDirectory, $"{stem}_{DateTime.Now:yyyyMMdd_HHmmss}{ext}");
if (File.Exists(target)) File.Delete(target);
}
File.Move(filePath, target);
Debug.Log($"Moved '{Path.GetFileName(filePath)}' out of the bank folder into _removed");
return true;
}
catch (Exception ex)
{
Debug.LogError($"Could not move '{filePath}' out of the sounds folder: {ex.Message}");
return false;
}
}
// Everything in the folder that the bank doesn't list gets pulled out, so every player ends
// up with an identical set. BingBong voice lines are exempt.
private static int QuarantineFilesNotInBank(HashSet<string> bankNames)
{
Dictionary<string, Song> loaded = new(Song.Sounds);
int moved = 0;
foreach (string ext in AudioTypes.Keys)
{
foreach (string file in Directory.GetFiles(SoundsDirectory, $"*.{ext}"))
{
string name = Path.GetFileNameWithoutExtension(file);
if (Song.IsBingBongVoiceLine(name)) continue;
if (bankNames.Contains(name)) continue;
if (!TryQuarantine(file)) continue;
moved++;
if (loaded.TryGetValue(name, out Song? song)) Plugin.RunOnMainThread(song.Dispose);
}
}
return moved;
}
// ShowActionbar reads Time.time and touches a MonoBehaviour, so it can't be called straight
// from the sync thread.
private static void ShowActionbarOnMainThread(string message)
{
Plugin.RunOnMainThread(() => BetterBugleUI.Instance?.ShowActionbar(message));
}
private static IEnumerator LoadAllAudioClipsCoroutine(string directoryPath, string[]? forceReload = null)
{
ClipsNeedingPreload = 0;
List<(string filePath, string ext, string name)> filesToLoad = new();
foreach (var ext in AudioTypes.Keys)
@@ -212,13 +291,32 @@ class AudioSyncWorker
loadedCount += loadCoroutines.Count;
BetterBugleUI.Instance?.ShowActionbar($"Loading audio clips... {loadedCount}/{filesToLoad.Count}");
}
OnAudioLoadComplete?.Invoke();
if (ClipsNeedingPreload > 0)
Debug.Log($"Pre-decoded {ClipsNeedingPreload}/{filesToLoad.Count} clips that would have stalled the main thread on first play.");
// Clear the flag before notifying: a subscriber that throws must not leave IsLoading
// stuck true, which would wedge every sync and refresh for the rest of the session.
IsLoading = false;
try
{
OnAudioLoadComplete?.Invoke();
}
catch (Exception ex)
{
Debug.LogError($"An audio load handler threw: {ex}");
}
}
private static IEnumerator LoadAudioClipCoroutine(string filePath, string ext, string name, bool forceReload = false)
{
using UnityWebRequest www = UnityWebRequestMultimedia.GetAudioClip($"file://{filePath}", AudioTypes[ext]);
// Keep the clip compressed in memory and let Unity decode it during playback. The default
// (DecompressOnLoad) holds raw PCM for the whole session -- roughly 10 MB per minute of
// 44.1kHz stereo, so a ~80 MB bank becomes ~1 GB resident. Compressed costs a little CPU
// per playing voice, which is nothing when one bugle is sounding, and avoids both the
// memory and the big upfront decode. Must be set before SendWebRequest.
if (www.downloadHandler is DownloadHandlerAudioClip audioHandler) audioHandler.compressed = true;
yield return www.SendWebRequest();
if (www.result != UnityWebRequest.Result.Success) yield break;
@@ -236,63 +334,104 @@ class AudioSyncWorker
AudioClip audioClip = DownloadHandlerAudioClip.GetContent(www);
if (audioClip == null) yield break;
// GetContent hands back a DecompressOnLoad clip whose PCM data is decoded lazily, on
// first use. Left alone that decode happens inside the first AudioSource.Play(), on the
// main thread, freezing the game the first time anyone plays a song. Force it here
// instead: we're already on the loading path, where a stall is expected and we can yield
// through it. If the clip already arrived loaded this is a no-op.
if (audioClip.loadState != AudioDataLoadState.Loaded)
{
ClipsNeedingPreload++;
// loadInBackground is serialized on the asset and read-only at runtime, so this
// decodes synchronously. That's the point: the cost lands here rather than mid-game,
// and the batching in LoadAllAudioClipsCoroutine already yields between clips.
audioClip.LoadAudioData();
while (audioClip.loadState == AudioDataLoadState.Loading) yield return null;
}
Song song = new(name, ext, filePath, audioClip);
song.Register();
}
private static async Task SyncAndLoadAudioClipsCoroutine()
// Runs on a thread pool thread via Task.Run. Everything in here must stay off Unity APIs;
// hop back through Plugin.RunOnMainThread for anything that touches the engine.
private static void SyncAndLoadAudioClips()
{
if (IsLoading || IsSyncing) return;
IsSyncing = true;
Dictionary<AudioSyncService.APIAudioFormat, Song?> toDownload = new();
string[] existingSongNames = [.. Song.Sounds.Keys];
AudioSyncService.APIAudioFormat[] existingAPIFormats = [.. AudioSyncService.GetAudioClips()];
string[] apiExistingNames = [.. existingAPIFormats.Select(apiAudio => apiAudio.Filename)];
var songsToRemove = existingSongNames.Except(apiExistingNames).ToArray();
foreach (var songName in songsToRemove)
try
{
if (Song.Sounds.TryGetValue(songName, out var songToDispose))
Dictionary<AudioSyncService.APIAudioFormat, Song?> toDownload = new();
AudioSyncService.APIAudioFormat[] existingAPIFormats = [.. AudioSyncService.GetAudioClips()];
// A failed fetch (bad URL, offline, server error) also returns an empty list, and an
// empty bank would mean "nothing belongs here" -- quarantining the entire folder. Only
// a bank that actually listed something is allowed to drive removals.
if (existingAPIFormats.Length == 0)
{
songToDispose.Dispose();
songToDispose.DeleteFile();
Debug.LogWarning("Audio bank returned no entries; cancelling sync rather than emptying the folder.");
ShowActionbarOnMainThread("Audio bank unreachable or empty, sync cancelled.");
IsSyncing = false;
return;
}
HashSet<string> bankNames = new(existingAPIFormats.Select(apiAudio => apiAudio.Filename), StringComparer.OrdinalIgnoreCase);
int quarantined = QuarantineFilesNotInBank(bankNames);
// Hashing reads the whole file off disk, so build the lookup here on the background
// thread rather than eagerly during loading where it stalled the main thread once
// per clip.
Dictionary<string, Song> soundsByHash = new();
foreach (Song song in Song.Sounds.Values.ToList())
{
// Skip anything just quarantined: its file has moved, so hashing would only fail
// and log noise. Its Dispose is already queued on the main thread.
if (!File.Exists(song.FilePath)) continue;
string hash = song.Hash;
if (!string.IsNullOrEmpty(hash)) soundsByHash[hash] = song;
}
foreach (AudioSyncService.APIAudioFormat apiAudio in existingAPIFormats)
{
Song? existingSong = soundsByHash.GetValueOrDefault(apiAudio.Hash);
if (existingSong == null || existingSong.Hash != apiAudio.Hash)
{
toDownload.Add(apiAudio, existingSong);
}
}
string quarantineNote = quarantined > 0 ? $", {quarantined} moved to _removed" : "";
ShowActionbarOnMainThread($"Syncing audio bank... {toDownload.Count} changed/new files found{quarantineNote}.");
string[] filesToOverload = [];
foreach (AudioSyncService.APIAudioFormat apiAudio in toDownload.Keys)
{
bool success = AudioSyncService.DownloadAPIAudio(apiAudio, SoundsDirectory, toDownload[apiAudio]);
if (success)
{
Debug.Log($"Successfully downloaded audio: {apiAudio.Filename}.{apiAudio.Extension}, adding to forceload");
filesToOverload = [.. filesToOverload, $"{apiAudio.Filename}.{apiAudio.Extension}"];
}
}
IsSyncing = false;
IsLoading = true;
// StartCoroutine is a Unity API - must run on the main thread, not on this Task.Run background thread.
Plugin.RunOnMainThread(() => Plugin.Instance.StartCoroutine(LoadAllAudioClipsCoroutine(SoundsDirectory, filesToOverload)));
}
foreach (AudioSyncService.APIAudioFormat apiAudio in existingAPIFormats)
catch (Exception ex)
{
Song? existingSong = Song.SoundsByHash.GetValueOrDefault(apiAudio.Hash);
if (existingSong == null || existingSong.Hash != apiAudio.Hash)
{
toDownload.Add(apiAudio, existingSong);
}
Debug.LogError($"Audio sync failed: {ex}");
IsSyncing = false;
IsLoading = false;
}
BetterBugleUI.Instance?.ShowActionbar($"Syncing audio bank... {toDownload.Count} changed/new files found.");
string[] filesToOverload = [];
foreach (AudioSyncService.APIAudioFormat apiAudio in toDownload.Keys)
{
bool success = await AudioSyncService.DownloadAPIAudio(apiAudio, SoundsDirectory, toDownload[apiAudio]);
if (success)
{
Debug.Log($"Successfully downloaded audio: {apiAudio.Filename}.{apiAudio.Extension}, adding to forceload");
filesToOverload = [.. filesToOverload, $"{apiAudio.Filename}.{apiAudio.Extension}"];
}
}
IsSyncing = false;
IsLoading = true;
Plugin.Instance.StartCoroutine(LoadAllAudioClipsCoroutine(SoundsDirectory, filesToOverload));
}
}
public class Song : IDisposable
{
public static readonly Dictionary<string, Song> Sounds = new();
public static readonly Dictionary<string, Song> SoundsByHash = new();
public static readonly Dictionary<string, Song> Songs = new();
public static readonly Dictionary<string, Song> BB_VoiceLines = new();
@@ -306,6 +445,8 @@ public class Song : IDisposable
.ThenBy(name => name)];
}
public static bool IsBingBongVoiceLine(string name) => name.StartsWith(BingBongPrefix, StringComparison.Ordinal);
public static void UpdateRealIndices()
{
var sortedNames = Songs.Keys.OrderBy(name => name).ToList();
@@ -318,65 +459,74 @@ public class Song : IDisposable
}
}
// Voice lines are personal, not part of the shared bank, so the sync never touches them.
public const string BingBongPrefix = "SFX_VO_BingBong_";
public string Name { get; set; }
public string Extension { get; set; }
public string FilePath { get; set; }
public AudioClip AudioClip { get; }
public string Hash { get; }
public int RealIndex { get; set; }
private string? _hash;
// Computed on demand rather than in the constructor: hashing reads the entire file and runs
// SHA256 over it, and the constructor is called from the load coroutine on the main thread.
// The only consumer is the sync comparison, which already runs on a background thread.
public string Hash => _hash ??= GenerateHash(FilePath);
public Song(string name, string extension, string filePath, AudioClip audioClip)
{
Name = name;
Extension = extension;
FilePath = filePath;
AudioClip = audioClip;
Hash = GenerateHash(filePath);
}
public void Register()
{
Sounds[Name] = this;
SoundsByHash[Hash] = this;
if (Name.StartsWith("SFX_VO_BingBong_")) BB_VoiceLines[Name] = this;
if (IsBingBongVoiceLine(Name)) BB_VoiceLines[Name] = this;
else Songs[Name] = this;
}
public void Dispose()
public void Dispose() => Dispose(null);
// knownAudioSources lets a bulk clear pass in a single scene scan instead of paying for one
// per song.
public void Dispose(AudioSource[]? knownAudioSources)
{
if (AudioClip == null) return;
var audioSources = UnityEngine.Object.FindObjectsByType<AudioSource>(FindObjectsSortMode.None);
var audioSources = knownAudioSources ?? UnityEngine.Object.FindObjectsByType<AudioSource>(FindObjectsSortMode.None);
foreach (var audioSource in audioSources)
{
if (audioSource.clip == AudioClip)
if (audioSource != null && audioSource.clip == AudioClip)
{
audioSource.Stop();
audioSource.clip = null;
}
}
Sounds.Remove(Name);
SoundsByHash.Remove(Hash);
if (Name.StartsWith("SFX_VO_BingBong_")) BB_VoiceLines.Remove(Name);
if (IsBingBongVoiceLine(Name)) BB_VoiceLines.Remove(Name);
else Songs.Remove(Name);
UnityEngine.Object.Destroy(AudioClip);
}
public void DeleteFile()
{
if (AudioClip == null) return;
var filePath = Path.Combine(AudioSyncWorker.SoundsDirectory, $"{Name}.{Extension}");
if (File.Exists(filePath))
{
File.Delete(filePath);
Debug.Log($"Deleted local file: {filePath}");
}
}
public string GenerateHash(string filePath)
{
using var hasher = SHA256.Create();
var fileBytes = File.ReadAllBytes(filePath);
var hashBytes = hasher.ComputeHash(fileBytes);
return BitConverter.ToString(hashBytes).Replace("-", "").ToLowerInvariant();
try
{
using var hasher = SHA256.Create();
var fileBytes = File.ReadAllBytes(filePath);
var hashBytes = hasher.ComputeHash(fileBytes);
return BitConverter.ToString(hashBytes).Replace("-", "").ToLowerInvariant();
}
catch (Exception ex)
{
// A missing or locked file just means "no match", which makes the sync re-download it.
Debug.LogWarning($"Could not hash '{filePath}': {ex.Message}");
return string.Empty;
}
}
}
}

View File

@@ -16,6 +16,25 @@ public static class Helper
return item;
}
/// <summary>
/// Config URLs get typed by hand, so a missing scheme is common. Adds one when there isn't
/// already a scheme, picking the insecure variant for loopback since a local dev server
/// almost never has TLS in front of it. Call as EnsureScheme(url, "https", "http") or
/// EnsureScheme(url, "wss", "ws").
/// </summary>
public static string EnsureScheme(string? url, string secureScheme, string insecureScheme)
{
string trimmed = (url ?? "").Trim();
if (trimmed.Length == 0) return "";
if (trimmed.Contains("://")) return trimmed;
bool loopback = trimmed.StartsWith("localhost", System.StringComparison.OrdinalIgnoreCase)
|| trimmed.StartsWith("127.0.0.1", System.StringComparison.Ordinal)
|| trimmed.StartsWith("[::1]", System.StringComparison.Ordinal);
return (loopback ? insecureScheme : secureScheme) + "://" + trimmed;
}
public static bool IsOnIsland()
{
return SceneManager.GetActiveScene().name.ToLower().StartsWith("level_") || SceneManager.GetActiveScene().name == "WilIsland";

470
tools/stub_relay.py Normal file
View File

@@ -0,0 +1,470 @@
"""Development stub relay for BingBong Live Voice.
Speaks the exact wire format the real relay (an endpoint on the AudioBank FastAPI service)
will use, so the mod can be built and tested before that backend exists. Deliberately minimal:
no auth, no session directory, no persistence. It is a test fixture, not a reference
implementation — build the real thing from the plan, not from this.
Requires `websockets` (pip install websockets).
python tools/stub_relay.py serve # relay on ws://localhost:8787/voice/ws
python tools/stub_relay.py tone # stream a 1 kHz sine into channel "test"
python tools/stub_relay.py tone --sweep # sweep 200-2000 Hz (resampling bugs are obvious)
python tools/stub_relay.py tone --speech # syllables and word gaps (for mouth animation)
python tools/stub_relay.py listen # dump received frames, to verify fan-out
python tools/stub_relay.py uplink # act as the speaker; capture the game's mic to WAV
python tools/stub_relay.py uplink --speech # ... and send audio back at the same time
Only ONE speaker may hold a channel (newest wins), so `tone` and `uplink` cannot both run --
use `uplink --tone/--sweep/--speech` to do both directions on a single connection, which is
also what the real web app does.
Point the mod at it with, in BepInEx config:
[BingBong Voice]
RelayURL = ws://localhost:8787/voice/ws?token=listen:test
Leaving RelayURL empty instead derives the URL from BugleSoundAPIURL, which is where the
relay lives in production.
Connect as: ws://localhost:8787/voice/ws?token=listen:test
ws://localhost:8787/voice/ws?token=speak:test
Wire format — 12-byte little-endian header, then PCM16LE mono payload:
0 u8 magic 0xB1
1 u8 version 1
2 u8 codec 0=PCM16LE 1=mu-law
3 u8 rate code 0=16k 1=8k 2=24k 3=48k
4 u32 seq
8 u32 sampleTimestamp (samples since stream start)
Binary frame = audio. Text frame = JSON control.
"""
import argparse
import asyncio
import json
import math
import struct
import sys
from collections import defaultdict
from urllib.parse import urlparse, parse_qs
from websockets.asyncio.server import serve
from websockets.asyncio.client import connect
HOST = "localhost"
PORT = 8787
PATH = "/voice/ws"
MAGIC = 0xB1
MAGIC_UPLINK = 0xB2 # game -> relay -> speaker; a distinct magic makes a loop impossible
VERSION = 1
CODEC_PCM16 = 0
RATE_CODES = {16000: 0, 8000: 1, 24000: 2, 48000: 3}
FRAME_MS = 20
HEADER = struct.Struct("<BBBBII")
def pack_header(seq: int, ts: int, rate: int = 16000, codec: int = CODEC_PCM16) -> bytes:
return HEADER.pack(MAGIC, VERSION, codec, RATE_CODES[rate], seq & 0xFFFFFFFF, ts & 0xFFFFFFFF)
# ---------------------------------------------------------------- relay
channels: dict[str, dict] = defaultdict(lambda: {"speaker": None, "listeners": set()})
names: dict = {}
stream_ids: dict = {}
uplinkers: dict = {} # ws -> bool, from the hello frame
deny_uplink = False # --deny-uplink, to exercise the client's uplink_denied path
def roster(channel: str) -> str:
"""Who the relay believes is in this lobby -- the stub's stand-in for /voice/sessions."""
return ", ".join(sorted(names.get(w, "?") for w in channels[channel]["listeners"])) or "(nobody)"
def uplink_roster(channel: str) -> list:
"""Who in this lobby says they may send a microphone. The real relay publishes this so the
game can tell "the relay is dropping my audio" apart from "nobody is listening"."""
return sorted(names.get(w, "?") for w in channels[channel]["listeners"] if uplinkers.get(w))
def print_sessions():
if not channels:
print("[relay] no active sessions")
return
for chan, ch in channels.items():
if not ch["listeners"] and ch["speaker"] is None:
continue
speaking = "speaking" if ch["speaker"] is not None else "idle"
print(f"[relay] {chan} {len(ch['listeners'])} listening [{speaking}] {roster(chan)}")
def parse_token(raw_path: str):
"""token=<role>:<channel>. Returns (role, channel) or (None, None)."""
q = parse_qs(urlparse(raw_path).query)
token = (q.get("token") or [""])[0]
if not token:
return None, None
# "<role>:<channel>" is this stub's own convention, kept because it makes a speaker and a
# listener easy to start from a shell. The deployed relay takes a bare channel id, which is
# what the mod sends, so accept that too and default it to listening.
if ":" in token:
role, _, channel = token.partition(":")
if role not in ("listen", "speak") or not channel:
return None, None
return role, channel
return "listen", token
async def handler(ws):
role, channel = parse_token(ws.request.path)
if role is None:
await ws.close(1008, "bad token")
return
ch = channels[channel]
if role == "speak":
old = ch["speaker"]
if old is not None:
try:
await old.send(json.dumps({"type": "preempted"}))
await old.close(1000, "preempted")
except Exception:
pass
ch["speaker"] = ws
print(f"[relay] speaker joined '{channel}' ({len(ch['listeners'])} listening)")
await broadcast_control(channel, {"type": "speaker_joined"})
try:
async for msg in ws:
if isinstance(msg, bytes):
await fanout(channel, msg)
except Exception:
pass
finally:
if ch["speaker"] is ws:
ch["speaker"] = None
print(f"[relay] speaker left '{channel}'")
await broadcast_control(channel, {"type": "speaker_left"})
return
ch["listeners"].add(ws)
names[ws] = "?"
stream_id = next((i for i in range(16) if i not in stream_ids.get(channel, {}).values()), 0)
stream_ids.setdefault(channel, {})[ws] = stream_id
print(f"[relay] listener joined '{channel}' -> {len(ch['listeners'])} listening")
try:
# Listeners send a hello frame naming the player. That is what the real relay turns into
# its session directory, so the stub records it too and prints the roster.
async for msg in ws:
if isinstance(msg, bytes):
# Uplink audio. Validate, stamp our own stream id so a client cannot claim
# another's, and forward to the speaker ONLY -- never back to listeners, which is
# half of what makes an audio loop structurally impossible.
if deny_uplink:
continue
if len(msg) != 12 + 640 or msg[0] != MAGIC_UPLINK or msg[1] != VERSION:
continue
frame = bytearray(msg)
frame[3] = (frame[3] & 0x0F) | (stream_id << 4)
if ch["speaker"] is not None:
await safe_send(ch["speaker"], bytes(frame))
continue
if not isinstance(msg, str):
continue
try:
payload = json.loads(msg)
except ValueError:
continue
if payload.get("type") == "hello":
names[ws] = str(payload.get("player", "?"))
uplinkers[ws] = bool(payload.get("uplink"))
print(f"[relay] '{names[ws]}' registered in '{channel}' -> roster: {roster(channel)}")
if uplinkers[ws] and deny_uplink:
await safe_send(ws, json.dumps({"type": "uplink_denied", "reason": "disabled"}))
print(f"[relay] denied uplink to '{names[ws]}' (--deny-uplink)")
elif uplinkers[ws]:
await broadcast_control(channel, {"type": "uplink_roster", "players": uplink_roster(channel)})
elif payload.get("type") in ("uplink_start", "uplink_stop"):
# Enriched with who it came from, which is what the web app needs to label streams.
await safe_send(ch["speaker"], json.dumps({
"type": payload["type"], "id": stream_id, "player": names.get(ws, "?"),
})) if ch["speaker"] is not None else None
print(f"[relay] '{names.get(ws, '?')}' {payload['type']} (stream {stream_id})")
except Exception:
pass
finally:
ch["listeners"].discard(ws)
stream_ids.get(channel, {}).pop(ws, None)
was_uplinker = uplinkers.pop(ws, False)
who = names.pop(ws, "?")
print(f"[relay] '{who}' left '{channel}' -> {len(ch['listeners'])} listening")
if was_uplinker:
await broadcast_control(channel, {"type": "uplink_roster", "players": uplink_roster(channel)})
async def fanout(channel: str, payload: bytes):
listeners = list(channels[channel]["listeners"])
if not listeners:
return
# Real relay uses a bounded per-listener queue and drops oldest; the stub just fires and
# forgets, which is fine locally and keeps this file short.
await asyncio.gather(*(safe_send(w, payload) for w in listeners), return_exceptions=True)
async def broadcast_control(channel: str, obj: dict):
msg = json.dumps(obj)
await asyncio.gather(
*(safe_send(w, msg) for w in list(channels[channel]["listeners"])),
return_exceptions=True,
)
async def safe_send(ws, payload):
try:
await ws.send(payload)
except Exception:
pass
async def cmd_serve(args):
global deny_uplink
deny_uplink = getattr(args, "deny_uplink", False)
if deny_uplink:
print("[relay] --deny-uplink: refusing every microphone with uplink_denied")
async with serve(handler, HOST, args.port, max_size=None):
print(f"[relay] listening on ws://{HOST}:{args.port}{PATH}")
print(f"[relay] listen: ws://{HOST}:{args.port}{PATH}?token=listen:test")
print(f"[relay] speak : ws://{HOST}:{args.port}{PATH}?token=speak:test")
print("[relay] session roster printed every 15s")
while True:
await asyncio.sleep(15)
print_sessions()
# ---------------------------------------------------------------- tone generator
try:
import numpy as _np
except ImportError:
_np = None
def generate_frame(phase, ts, n, rate, args):
"""Returns (new_phase, pcm16_bytes) for one frame, phase-continuous across frames."""
if _np is not None:
idx = _np.arange(n, dtype=_np.float64)
t = (ts + idx) / rate
if args.speech:
# Speech-shaped: a voice-ish carrier under a syllable envelope with word gaps. The
# sweep and steady tones hold a constant amplitude, so they pin a loudness-driven
# mouth wide open -- this is the signal to test mouth animation against.
freq = 150.0 + 60.0 * _np.sin(2 * _np.pi * 1.7 * t)
syllables = (0.5 + 0.5 * _np.sin(2 * _np.pi * 4.0 * t)) ** 2
words = (_np.sin(2 * _np.pi * 0.35 * t) > -0.25).astype(_np.float64)
amp = 0.45 * syllables * words
elif args.sweep:
# 200-2000 Hz over 4 s: any resampling error shows up as an obvious warble
freq = 200.0 + 1800.0 * (0.5 - 0.5 * _np.cos(2 * _np.pi * t / 4.0))
amp = _np.full(n, 0.35)
else:
freq = _np.full(n, float(args.freq))
amp = _np.full(n, 0.35)
ph = phase + _np.cumsum(2 * _np.pi * freq / rate)
samples = (_np.sin(ph) * amp * 32767).astype("<i2")
return float(ph[-1] % (2 * _np.pi)), samples.tobytes()
# Fallback without numpy. Fine for a steady tone; sweep may not hold real time.
out = []
for i in range(n):
t = (ts + i) / rate
freq = args.freq
amp = 0.35
if args.speech:
freq = 150.0 + 60.0 * math.sin(2 * math.pi * 1.7 * t)
syllables = (0.5 + 0.5 * math.sin(2 * math.pi * 4.0 * t)) ** 2
words = 1.0 if math.sin(2 * math.pi * 0.35 * t) > -0.25 else 0.0
amp = 0.45 * syllables * words
elif args.sweep:
freq = 200.0 + 1800.0 * (0.5 - 0.5 * math.cos(2 * math.pi * t / 4.0))
phase = (phase + 2 * math.pi * freq / rate) % (2 * math.pi)
out.append(int(math.sin(phase) * amp * 32767))
return phase, struct.pack(f"<{n}h", *out)
async def cmd_tone(args):
url = f"ws://{HOST}:{args.port}{PATH}?token=speak:{args.channel}"
print(f"[tone] {url}")
async with connect(url, max_size=None) as ws:
await stream_tone(ws, args)
async def stream_tone(ws, args):
"""Sends a test tone downstream on an already-open speaker socket."""
rate = args.rate
n = int(rate * FRAME_MS / 1000)
print(f"[tone] {rate} Hz, {n} samples/frame ({FRAME_MS} ms)")
loop = asyncio.get_running_loop()
if True:
seq = 0
ts = 0
phase = 0.0
# Deadline-based pacing. A plain `sleep(FRAME_MS)` per iteration sleeps for *at least*
# that long and then adds the generate+send time on top, so the sender silently drifts
# behind real time and starves the receiver's jitter buffer. Track an absolute deadline
# and sleep only for the remainder instead.
next_deadline = loop.time()
late_frames = 0
while True:
# Vectorised: generating 320 samples a frame in a Python loop costs more than the
# 20 ms frame budget once sweep mode adds a second transcendental per sample, and the
# sender then silently falls behind real time and starves the receiver.
phase, payload = generate_frame(phase, ts, n, rate, args)
await ws.send(pack_header(seq, ts, rate) + payload)
seq += 1
ts += n
next_deadline += FRAME_MS / 1000
delay = next_deadline - loop.time()
if delay > 0:
await asyncio.sleep(delay)
else:
# Generation itself overran the frame budget; we can't catch up by sleeping.
late_frames += 1
next_deadline = loop.time()
if seq % 50 == 0:
print(f"\r[tone] sent {seq} frames ({ts / rate:6.1f}s), {late_frames} late", end="", flush=True)
# ---------------------------------------------------------------- listener (verification)
async def cmd_listen(args):
url = f"ws://{HOST}:{args.port}{PATH}?token=listen:{args.channel}"
print(f"[listen] {url}")
async with connect(url, max_size=None) as ws:
count = 0
last_seq = None
gaps = 0
async for msg in ws:
if isinstance(msg, str):
print(f"\n[listen] control: {msg}")
continue
magic, ver, codec, rc, seq, ts = HEADER.unpack_from(msg, 0)
if magic != MAGIC or ver != VERSION:
print(f"\n[listen] BAD HEADER magic={magic:#x} ver={ver}")
continue
if last_seq is not None and seq != last_seq + 1:
gaps += 1
print(f"\n[listen] SEQ GAP {last_seq} -> {seq}")
last_seq = seq
count += 1
if count % 50 == 0:
print(
f"\r[listen] {count} frames, {len(msg) - 12} B payload, codec={codec} "
f"rate={rc} ts={ts} gaps={gaps}",
end="",
flush=True,
)
async def cmd_uplink(args):
"""Acts as the web-app speaker: receives uplink audio and writes it out per stream.
The real-time ratio is the number that matters. Below ~0.98 with the gate open means the
sender is not keeping up and the browser would hear clicking; while gated it is expected to
sit low, because silence is deliberately not transmitted.
"""
import wave
url = f"ws://{HOST}:{args.port}{PATH}?token=speak:{args.channel}"
print(f"[uplink] {url}")
print("[uplink] writing uplink_<id>.wav, Ctrl-C to stop")
if args.tone or args.sweep or args.speech:
print("[uplink] also sending a test tone downstream on this same socket")
writers: dict = {}
counts: dict = {}
first_time = None
total_samples = 0
async with connect(url, max_size=None) as ws:
sender = None
if args.tone or args.sweep or args.speech:
args.rate = 16000
args.freq = getattr(args, "freq", 1000.0)
sender = asyncio.create_task(stream_tone(ws, args))
try:
async for msg in ws:
if isinstance(msg, str):
print("\n" + f"[uplink] control: {msg}")
continue
if len(msg) < 12 or msg[0] != MAGIC_UPLINK:
continue
sid = msg[3] >> 4
if sid not in writers:
w = wave.open(f"uplink_{sid}.wav", "wb")
w.setnchannels(1)
w.setsampwidth(2)
w.setframerate(16000)
writers[sid] = w
counts[sid] = 0
print("\n" + f"[uplink] stream {sid} opened -> uplink_{sid}.wav")
writers[sid].writeframes(msg[12:])
counts[sid] += 1
total_samples += (len(msg) - 12) // 2
if first_time is None:
first_time = asyncio.get_running_loop().time()
elapsed = asyncio.get_running_loop().time() - first_time
if sum(counts.values()) % 50 == 0 and elapsed > 0:
ratio = (total_samples / 16000) / elapsed
verdict = "OK" if ratio > 0.98 else "(gated or starving)"
print("\r" + f"[uplink] {sum(counts.values())} frames, "
f"real-time ratio {ratio:.3f} {verdict} ", end="", flush=True)
finally:
if sender is not None:
sender.cancel()
for w in writers.values():
w.close()
print("\n" + "[uplink] wav files closed")
def main():
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("--port", type=int, default=PORT)
sub = p.add_subparsers(dest="cmd", required=True)
sv = sub.add_parser("serve")
sv.add_argument("--deny-uplink", action="store_true", help="refuse every uplink, to test the client's uplink_denied handling")
t = sub.add_parser("tone")
t.add_argument("--channel", default="test")
t.add_argument("--freq", type=float, default=1000.0)
t.add_argument("--rate", type=int, default=16000, choices=sorted(RATE_CODES))
t.add_argument("--sweep", action="store_true", help="200-2000 Hz sweep; makes resampling errors obvious")
t.add_argument("--speech", action="store_true", help="speech-shaped envelope; use this to test mouth animation")
l = sub.add_parser("listen")
l.add_argument("--channel", default="test")
u = sub.add_parser("uplink", help="act as the speaker and capture what the game sends up")
u.add_argument("--channel", default="test")
u.add_argument("--tone", action="store_true", help="also send a steady tone downstream, on the same socket")
u.add_argument("--sweep", action="store_true", help="also send a sweep downstream")
u.add_argument("--speech", action="store_true", help="also send speech-shaped audio downstream")
u.add_argument("--freq", type=float, default=1000.0)
args = p.parse_args()
fn = {"serve": cmd_serve, "tone": cmd_tone, "listen": cmd_listen, "uplink": cmd_uplink}[args.cmd]
try:
asyncio.run(fn(args))
except KeyboardInterrupt:
print("\nbye")
if __name__ == "__main__":
sys.exit(main())