[raghavm243512]

This PR has many changes and specific choices:

Overall design has 2 parts:

During assistant turn

User accepts STT streaming in text, on each text delta make a decision to interrupt. While the decision is being made, keep STT stream open to accept additional context, like a person generally can do

These decision can either be to keep listening, backchannel, or interrupt

• keep listening means do nothing
• backchannel means send a pre-rendered phrase from a phrase cache, this is intended to focus on longer sequences and as a form of acknowledgment. Minimal purpose or moving parts. Importantly, back channels are not considered as part of the audit log or transcript, but are tracked separately. This is because while audio plays, there is no actual turn logic or turn switching that should be associated with the backchannel. It is a test of the assistant VAD, nothing more.
• Interrupt will take the latest transcript (remember how I mentioned keeping the stream open) and send it to an LLM and at the same time will play an opened phrase from a different phrase cache than the backchannel cache. Then it starts talking as if it was actually the user's turn, and fires all regular user turn events. All parts including the opener are tracked in conversation history

After Assistant Turn (aka during User Turn)

Take committed transcript, generate audio and TTS like normal. Optionally, the LLM can decide it doesn't need to say anything yet. This specifically prevents speaking during pre_tool speech from the model (and the LLM can decide intelligently, using the content like "1 moment while I check" as a signal to not say anything)

Important qualities of decisions and their consequences

Both back channel and interrupt have specific cooldowns, and minimum talking amounts that must occur before they are able to be decided upon or made. Nobody says 'mhm' after someone says 'hello' to them, so some simple guards to induce realism and save compute are in place.

Things that changed (not new, but changed from base)

Using OpenAI STT as default as it is much faster, and with the open STT channel missed_words (new diagnostic metric) coming down to roughly 3-4 vs 6 for ElevenLabs

Metric/tracking

No metrics have received changes. For interruption, this is largely something that gets handled already with behavioral fidelity, the [interrupted] tags and stuff we had before handle this behavior. The audit log sees all interruptions but no back_channels

Nothing is consuming the saved back channel information yet. This is something to look at when a model breaks down as a turn taking problem

There are additional fields in result.json and a file for user_simulator_events.jsonl that is also not being consumed

Misc pieces

Some small updates to other scripts. All multilingual parts are included to keep the phrase caches up to date. Additional environment variables to control cooldown times, thresholds, are added.

If the gap between the opener and the content on an interruption is too long (set to 1 second but can adjust) then the run is failed. This should not trigger often, but I see an occasional long one which would break realism)

Pending things

• Still must tune prompt and choice descriptions to get good interruption and back_channel use.
• Possibly support back_channels with programmatic rules rather than or in addition to decisions (reduce complexity of model's choice)
• Reduce failure rate through prompt tuning for decisions
• Consume back channel and potentially more of the tracked barge in information in some meaningful way
• If we have to deal with assistant back channels (future problem), need to not consume those in STT -> decisions