Repository navigation
Document and spec live captions for Teams meetings - #12
Merged
Merged
Conversation
Stage 2 needs the Teams module's own tools to reach agents, and a SharePoint token whose audience is the user's own tenant, which no manifest built once for everyone can spell out. The registry now lists the read tools of active modules as <module>_<tool>, with an optional account the core adds and resolves. The core checks the arguments against the declared schema, the read permission before the call and again after it for the same account, drops items of conversations the agent may not read, and takes only a JSON object of at most 1 MiB back. Write tools, bad names, names taken twice or by a core tool, schemas the core cannot check and schemas declaring account are not listed; status says why. Every MCP session hears tools/list_changed when the list changes, from a scan or from the host stopping a module after crashes. A WAM scope may now be https://*.<domain>/.default, checked at load like a host pattern; it covers one lowercase DNS label on the module's host list and nothing else. Literal scopes compare as before. The scope check now runs before the tests' fake WAM too, and the approval window says what the SharePoint pattern hands over.
Review of fc631e3 (Codex, GLM): a module file that declared, say, `read` + `messages` made every read_messages call end as that module's unknown or disabled tool, even while the module waited for approval. A core tool's name now never routes to a module. Codex also found that the declaration checked was read before the account's module handle was taken: a replacement approved in between could run a tool that writes in the new version under the read permission. The call now goes ahead only if the module it reaches declares the very same tool.
Since main gained the window languages (#8), every Polish literal of the daemon needs its English beside it. The approval's text for a WAM scope pattern was Polish only; it now follows the window's language, with a test of both.
D1: local transcription is wanted, in its own spec. D2: no extra gate, open_mess is for the owner's own use, so transcripts reach agents under the account's read permission like chat messages. D3: a transcript fetch must never count as a view in the recording owner's analytics, so step 0 gets a gate that checks the counters on the user's own recording with a control, and the download path ships only if it passes. Reviewed by Codex and GLM in round 11 (both approve).
Agents can now see the Teams meetings of the user's calendar with the meeting chat each one has, which read_messages already reads. The module asks the middle tier's calendarView, as the official client does, with the authsvc token and the skype token it now keeps from the exchange; a skype token about to expire, or a 401, gets a new exchange. The live check of step 0 (docs, "Live results" 10) showed one answer for a dense month, times always with an offset, the enum values the module now maps, and no cancelled or private marker in the window. Only Teams meetings the user has not declined are returned, ids as opaque tokens, subject and organizer under untrusted; a continuation key or a field in another shape is an incompatible answer, never a silent first page. meetings.rs also holds the WebVTT parser and the paging of cues for the transcript tool that follows.
Review of 5802273 (GLM, two P3): list_meetings treated a `from` or `to` that is not text as left out and answered for the default window, and a NOTE block with `-->` in its text failed the whole transcript. A value of another type is now invalid, and NOTE, STYLE and REGION blocks are skipped before the timing line is looked for.
The owner approved reading the transcript of one recurring meeting of
their own organization, although the recording is a colleague's. The
probe made the requests of spec 4.2 steps 1-6 once, printing counts,
key names and character classes only.
What it settled: MCPS paginationInfo is {hasNextPage} and the calendar
event's iCalUID picks the occurrence; TranscriptV2 metadata times are
.NET ticks, not ISO times; the ODSP transcript id equals
metadata.transcriptId; streamContent?is=1 is WebVTT with a byte order
mark and voice tags, and the module's own parse_vtt and page handle
the real file. The allTime view counters stayed at 0 right after the
download, which is not yet a D3 result: they are read again a day
later, and the check on the owner's own recording still stands.
Review of 800ab63 (GLM P1) pointed out that the download ran on a colleague's recording before the D3 gate, while the doc called it step 0. It was the owner's explicit decision for that one meeting; the doc now says so, says what the owner accepted and that the D3 gate on the owner's own recording still has to pass before the tool ships. Also stop item 9 from saying that a download in the owner's organization is still to be captured (GLM P3).
The live download in the owner's organization (docs "Live results" 12) showed that TranscriptV2 times are .NET ticks, not ISO times, and that MCPS pages carry only hasNextPage with no continuation field. The spec now converts ticks with a format check and treats hasNextPage true as an incomplete list instead of paging blindly. It also records the owner's one-meeting exception to the D3 order, which the commit review of 800ab63 found missing from the spec; the D3 gate itself is unchanged. Review round 12 found that a VTT file over 100 000 cues was cut silently, so the last page would claim the transcript ended. The parser now returns an error instead, with a test. Round 13: both reviewers approve.
The owner wants an agent to summarize tomorrow's meeting from its transcript, before the D3 view-count check is settled. A view of a file lands in the analytics of whoever owns the place the file lies in, so the tool reads only transcripts whose location is under the user's own OneDrive site (from personalFileSite) and refuses anything else before asking for any SharePoint token. metadata.ownerId is not used: what it names is not confirmed (spec review R14). The path is spec 4.2 as checked live: MCPS contents with the MCPS token, TranscriptV2 times as .NET ticks, hasNextPage true never picked from, the origin only one of the organization's two, the ODSP list read before and after the WebVTT so a changing transcript is an error, and pages pinned to one version. SharePoint refusals are answers about the transcript, not a lost sign-in. The manifest asks for the MCPS scope and the SharePoint pattern, so the module needs approval again. Tests run the built module against a fake Teams with fixtures of the live shapes and synthetic values. Spec rounds R14-R16; both reviewers approve R16.
The owner decided on 2026-10-30 that teams_get_transcript should work for every meeting he attends, not only for recordings in his own OneDrive, and accepted that a download may count as one view in the recording owner's analytics (spec section 10, D3). Tomorrow's meeting is likely organized by someone else, so its transcript sits in the organizer's OneDrive. The refusal for transcripts outside the user's personal site is gone, and so is the path check behind it; the session now keeps only the tenant label it needs to derive the organization's two SharePoint origins. The other checks stay: only GETs, no read receipts, the owner tenant must be the user's, and no SharePoint token is asked for any other host. The manifest's SharePoint scope pattern already covers every OneDrive on the tenant's -my host, so the scopes and hosts are unchanged. The refusal test now proves that a colleague's OneDrive and a team site are read with a token for their own origin; the tests for other organizations and foreign hosts stay. Spec rounds R17 and R18.
c85deae dropped the lowercasing of the personalSiteUrl path when the path stopped being compared with transcript locations, so a site path written as /Personal/ would fail the shape check and break teams_get_transcript. SharePoint paths ignore letter case; the check does again, with a test. Found by GLM in the review of c85deae.
The first live teams_get_transcript run returned speaker names with raw references such as `ó`: the VTT parser knew only a few named entities. It now reuses the message text decoder, which handles named and numeric (decimal and hex) references and leaves invalid or out-of-range ones as text. That decoder gains `‎` and `‏`, which the VTT parser decoded before.
The owner asked whether open_mess could get live captions by joining a meeting itself, as the user's second endpoint without audio or video. The installed client shows that caption text comes from a server-side bot over the call's data channel, a media modality: signaling and Trouter carry only the bot's roster entry and commands, and the signaling-only "subscribe" join gets no data channel. So it needs a negotiated media session (SDP, ICE, DTLS, SCTP), and the user's own client would show the meeting as running on another device. Static analysis of the package only, marked as not verified live; no code changes (the tree is the one d28cfff passed CI with).
Review of 57a056a (GLM, two P3): section 10 used markers outside the file's legend and its open questions were missing from "Needs live capture", which is what live checks are planned from; the date and the status line were off by the new section. Docs only.
Review of c46fca7 (GLM, P3): two UNCONFIRMED facts of section 10.4 - an organizer option for captions alone, and whether the service keeps anything of captions without transcription - had no entry in "Needs live capture". Docs only.
The owner wants agents to read a Teams meeting's live captions while the user is in it. Captions reach a participant only on the call's data channel (docs 10), so open_mess would join as the user's second endpoint with no audio or video: a new host import for one data channel through TURN relays on the module's host list to address ranges the manifest names, module write tools through pending actions, Trouter in the Teams module, and leave guaranteed on stop, update, replacement and crash. Reviewed in eleven rounds (the eleventh with the owner's consent): GLM approved R9-R11; Codex rejected R11 over one race, fixed here without a further round at the owner's call. Steps 0a and 0b (static and live checks) gate the code and send the spec back to review if the protocol differs; owner decisions O1-O3 are open.
Review of add1643 (Codex, P2): after adopt put a replacement Teams module in place, an agent could join a meeting through it before end-sessions had ended the old version's session, leaving one account with two sessions. A rescan now takes the same order as an update: block module writes, let running ones finish, adopt, end the old sessions, drop the old handles, unblock. Spec only.
Review of 0b2b0ff (Codex P2, GLM P3): a second rescan started while the first waited on end-sessions could lift the first one's write block and let a join through the new module version while the old session still ran. Rescans now hold one lock for their whole run, and the spec adds a test of the block during a rescan. Spec only.
The owner accepts that, while a live-captions session they approved is in a meeting, their official Teams client shows them in the meeting on another device. AGENTS.md names this as the one exception to "Never disturb the official clients"; nothing else of the official client may change. O2 and O3 stay open.
Review of 61091e3 (Codex, P2): with write=autonomous an agent could join a meeting for captions without the owner approving that session, while the O1 exception in AGENTS.md covers only sessions the owner approves. Module write tools now always wait for approval in the window; autonomous acts like confirm for them. Spec only.
open_mess reads live captions only when someone in the meeting already has them on, and never adds the caption bot or sends it any command. With nobody's captions on, a session ends with no_captions after 30 s; captions turned off mid-session show as captions_on false and end the session after 5 minutes. The bot's command URL and the tokens it would need drop out of the design. O2 stays open until step 0b.
Review of 3836639 (GLM, three P3): the timer list in 5.3 lacked the new waits for captions to be on, 4.4 did not say that open_mess never turns captions on, and the manifest note still named the bot as a token audience. Spec only.
Static analysis of the installed client for step 0a of the live captions spec: the conversation-service join (default epconv URL, body keys, SDP offer in callInvitation.mediaContent, callback links over Trouter), the answer and roster callbacks with the caption bot's processing modes, leave, a presence check in the user's call store that needs no join, TURN credentials from trap/tokens and the relay hosts, no-trickle ICE, and the 11-byte framing of the data channel. The steps of the spec hold; the open points go to step 0b. Docs only.
With the owner in a meeting and their OK for reads only, the call store answered with the active call but holds no meeting thread, so it can tell that the user is in a call, not in which meeting; the relay credentials endpoint answered with the shape static analysis found. No join and no Trouter connection. Docs only.
The captions probe and the live captions module both depend on how Trouter delivers the call callbacks and on what the join body and its callbacks look like. AGENTS.md wants those facts written down before code depends on them. Static analysis of the shipped client shows that v4c (one WebSocket, no allocate step) is the default and v4a is the fallback, gives the Socket.IO 0.9 frames, the downstream request and response envelope, gzip callbacks, the endpoint capability bits, that callModalities has no data modality, the MS-SDP transform, the callAcceptance acknowledgement and the join response keys. Section 5 and "Needs live capture" 7 now point there, and "For the spec" lists what step 0b has to settle. No network request was made for this.
4 tasks done
#11 landed on main squashed, so this branch and main no longer share its commits and docs/protocols/teams.md conflicts. The tree is this branch's own tree unchanged; main brings nothing new beyond what the branch already had.
SychPL
added a commit
that referenced
this pull request
Nov 5, 2026
#12 landed on main squashed, so this branch and main no longer share its commits. The tree is this branch's own tree unchanged; main brings nothing new beyond what the branch already had.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #11, which is stacked on #10: this PR's base is
feat/teams-meetings, so its diff shows only its own commits. Merge #10 and #11 first; after their squash merges this branch is rebased ontomainand retargeted.Change
This PR adds documentation and the spec for live captions in Teams meetings (a slice of stage 3). It contains no code.
docs/protocols/teams.md, section 10: how the official client gets live captions. The facts come from static analysis of the current client, then from step 0a and the read-only part of step 0b:main-channeldata channel (data id 3recognitionResults, 11-byte framing),The questions that need a live capture are listed under "Needs live capture", items 17-19.
.specs/2026-10-29-live-captions.md: reading captions through a second endpoint that carries no media. The spec covers:end-sessionsruns at shutdown and before an update.AGENTS.md: the owner's O1 exception. While a live captions session the user approved is in a meeting, the official client shows the user in the meeting on another device. Nothing else of the official client changes.The spec is not final. Here it ends at R11: GLM approved, and a Codex P2 was fixed without another round. The PoC session for live captions continues the spec on its own branch (rounds R12+ and the live results of step 0b). Implementation waits for that work and for decision O2.
Spec
This PR is the spec itself:
.specs/2026-10-29-live-captions.md, rounds R1-R11.Tests
Not applicable: docs and spec only.
cargo xtask ci(CARGO_BUILD_JOBS=3) is green on this branch: fmt, clippy-D warnings, the wasm module build and tests all green, 593 passed, 0 failed.Commit reviews
Most commits are cherry-picks with the same patch as reviewed commits of the working branch. Each
.processedline points at the original.fa3cd8b(c46fca7, teams.md conflict resolved by keeping the meetings branch's item 16 text): Codex APPROVE; GLM APPROVE with the same P3 as onc46fca7(two 10.4 facts not in "Needs live capture"), fixed in the next commitbca34f8.eaac48e(ca5a45e, conflict resolved: item 11 placed before item 12, read-only range back to 1-4 and 7-12): Codex APPROVE, GLM APPROVE.1b01a66(add1643): a replacement could take a join before the old module'send-sessions. Fixed in7b37b1a.7b37b1a(0b2b0ff): parallel rescans lift the block early. Fixed in0abcce6.be87b17(61091e3): an autonomous join without a per-session approval, against O1. Fixed in3f926f7.Coexistence (platform modules only)
No code here. The reads behind "Live results" 11 were GETs on the owner's account, with no join and no Trouter connection.
Checklist
unwrap/expect/panic!/ unchecked indexing in non-test code (no code)