Teams Meeting Agent

#At a glance

#Lines of code by language

The virtual environment, __pycache__, the runtime data folder (browser profiles, recordings, transcripts, model downloads, logs, SQLite state, smoke-test output), a vendored audio driver package and the compiled capture helper binary are all excluded.

Not counted above: 7,199 lines of Markdown (54 files: requirements, setup history, a transcription-service spec, an ERP guide and 41 skill cards for the retired live assistant), 868 lines of JSON test fixtures, and 128 lines of dependency pins.

#Lines of code by module

#The problem

Meetings at Mursix often end without written minutes, or with minutes someone has to write from memory. The goal was to produce minutes without sending meeting audio or transcripts to a cloud AI service, because of the information discussed in those meetings.

The result is an attendee that anyone in an approved group can invite like a person. It joins, records, and writes minutes with decisions, action items and open questions, each backed by a timestamped quote. All processing happens on hardware the company owns. Because recording people has legal and trust consequences, the agent was rewritten in October 2026 around one rule: if any check cannot be confirmed, it does not record.

 Exchange Online calendar (agent mailbox)
        │  Graph calendarView, every 30 s
        ▼
 ┌──────────────────────────── Scheduler ────────────────────────────┐
 │ internal-domain check → Entra group check (Graph) → capacity,     │
 │ recurring-window and duration checks → accept / decline + reason  │
 │ → individual recording notice to organizer and every attendee     │
 └───────────────┬───────────────────────────────────────────────────┘
                 │ near start time, one free browser slot
                 ▼
  Edge profile (slot N) ── Playwright ──► Teams web prejoin
     mic + camera verified OFF → join → wait for admission
                 │
                 ▼
  Process-loopback recorder (C#) ── one continuous WAV per occurrence
     leaves when alone for 60 s, or at the scheduled end if nobody joined
                 │
                 ▼  (single serialized worker)
  WAV → 16 kHz mono FLAC → GPU transcription service (faster-whisper large-v3)
                 │            └─ fallback: local CPU Whisper
          delete all audio
                 ▼
  Transcript → GPU notes server (Gemma 31B via LM Studio)
                 │            └─ fallback: local Ollama (Qwen 2.5 7B)
                 ▼
  HTML minutes + disclaimer ── Graph sendMail ──► organizer only
                 │
          confirmed? → delete transcript, minutes, email body
                       keep metadata-only receipts and usage row

#1. Calendar discovery and the meeting model

The agent watches its own mailbox. Three calendar sources share one Meeting record:

• Microsoft Graph (the current mode). It reads calendarView for a window from 4 hours ago to 2 days ahead, with recurring series expanded and times forced to UTC. It follows paging links and stops at 1,000 events. The Teams link comes from the event's online-meeting field, or failing that, from the body or location.
• Classic Outlook over COM (the original mode). It restricts the calendar by local time and reads exact UTC start times so daylight-saving changes are handled. A live incident showed that Outlook running without a window can serve a stale cached calendar while disconnected from Exchange. The agent now opens the calendar window if none is open, checks the Exchange connection mode before every scan, and records calendar health separately from its heartbeat.
• A manual JSON queue for testing.

Each occurrence gets a stable 24-character ID, a hash of the invitation's global ID and its start time. That ID names the meeting's private folder and appears in every receipt. Teams links are checked against an exact allowlist of hosts and path prefixes before anything is opened. A join link is rewritten to the signed-in web client's route, so the public launcher cannot drop the agent into an anonymous session.

#2. Authorization: who may use the agent

Authorization happens before any notice, acceptance or join, and it fails closed.

• Company domain first. The organizer's address must exactly match a configured company domain. Lookalike domains and subdomains do not count. External invitations, including ones staff forward to the agent, are ignored silently, so no response or email ever goes to another organization.
• Entra security group. The agent gets an app-only token through the OAuth 2.0 client-credentials flow and caches it until shortly before it expires. It resolves the organizer by exact mail or userPrincipalName, then calls checkMemberGroups against one allowlist group. Zero matches is a denial. More than one match, an incomplete result or any HTTP error raises an exception, and the meeting is held as authorization_pending.
• Least privilege. Directory reads use only the basic-user and group-membership application roles. Calendar and send-mail rights are not tenant-wide Entra permissions. They are granted through Exchange Online RBAC for Applications, scoped to the agent's own mailbox. An optional scoped calendar-read permission on organizers' mailboxes lets the agent find people an attendee forwarded the invitation to, because Exchange only updates the organizer's copy of the meeting.
• Readable declines. An unauthorized organizer gets a decline response and a separate email explaining why. A failed notice is retried on later polls before the state is closed.
• Safe diagnostics. Errors carry only an exception type or HTTP status. Response bodies, addresses and credentials never reach the logs.

#3. Consent: recording notices before recording

Every authorized meeting needs a confirmed notice to every person before the agent may join.

• Notices go individually to the organizer and each required or optional attendee, so nobody sees anyone else's address. Rooms, resources and the agent's own mailbox are excluded.
• The notice says that an AI agent will record the audio, that a service on the company network transcribes it and deletes the recording right away, and that the transcript and minutes are deleted once the organizer has received the minutes.
• Each notice is idempotent. A receipt keyed by meeting ID and a hash of the recipient moves through submitting → sent, or uncertain if something fails. A unique marker is placed in the message headers so an uncertain submission can be checked against the mailbox instead of being sent twice. If any notice is unresolved, the meeting is not joined.
• Notices are rechecked on every poll until the meeting ends, so people added or forwarded during the meeting are also told. Everyone who received a notice is merged into the invitee list shown in the minutes.
• When a meeting is moved to a new time, it keeps its notice history, so nobody is emailed twice, and the old slot is marked rescheduled.

#4. Scheduling, capacity and meeting lifecycle

The scheduler is a single polling loop with a SQLite state table and about 25 named states (for example authorized, recording, email_pending, declined_capacity, cancelled, interrupted). A Windows file lock stops two instances from running at once.

• Capacity. Admission is first come, first served. A sweep over the committed meetings finds the largest number running at any one instant during the new meeting; when two times are equal, an end sorts before a start, so back-to-back meetings do not collide. If accepting a meeting would exceed the configured capacity, the organizer is emailed and that occurrence is declined. No attendee notices are sent.
• Recurring series are covered for 365 days from the first accepted occurrence, like a room booking. At the first date past that window, the organizer is told once and the rest of the series is declined.
• Edge cases. An invitation that ends at or before its start is declined with an explanation. A cancelled meeting, or an accepted meeting that disappears from the calendar, is marked cancelled, but a call already in progress is never cut short. A meeting is considered from 60 seconds before its start and is marked missed if no slot frees up within a 30-second late grace period.
• Graceful operation. A heartbeat file is rewritten every 10 seconds. A stop-request file lets active recordings, notes and email finish first. The process asks Windows to stay awake without changing power settings. On restart, any meeting caught mid-recording has its partial audio deleted and is marked interrupted.

#5. Joining Teams in a browser

The agent attends as a signed-in Teams web user in Microsoft Edge, driven by Playwright with a persistent profile. Multi-factor sign-in is done once by hand.

• Prejoin. It handles the "continue on this browser" interstitial. When normal automated clicks are ignored, it falls back to activating the button directly. It sets a display name.
• Microphone and camera must be verifiably off. It reads the current and older Teams toggle attributes, turns a control off only after confirming it is on, and stops with an error if a control is disabled, missing or in an unknown state. It never toggles blindly.
• Admission counts only when both the Leave button and the participant controls are visible, because some lobby screens also show Leave. There is a 10-minute lobby timeout.
• Leaving. A presence tracker reads the People pane's "In this meeting (N)" header, with the "waiting for others" banner as a fallback. The agent leaves after 60 seconds alone once someone else has been in the call, or at the scheduled end if nobody ever joined. There is no overall time limit. If the Leave control is missing three times in a row, the call has ended.
• Selectors are a known risk. Microsoft can change the Teams interface at any time, so every unexpected state is an error, not a guess.

#6. Audio capture and per-meeting isolation

Capture started with WASAPI endpoint loopback, which records everything the PC plays. That is fine for one meeting, but two overlapping meetings would each capture the other.

The fix is a small C# helper, about 240 lines and compiled with the in-box .NET Framework compiler, that uses Windows' process-loopback activation (ActivateAudioInterfaceAsync with a process-tree target, Windows 10 build 20348 or later). It records only the audio rendered by one process and its children.

• Each concurrent meeting gets its own Edge profile, a copy of the signed-in profile made by a setup command, and its own browser process.
• The capture target must be the browser's main process: the one that owns that profile directory and has no --type= switch. The launcher process exits immediately, and renderer processes are children. The agent finds it by reading process command lines through CIM.
• The helper reports recording on stdout when it is ready, stops on a line written to stdin, and returns distinct exit codes for a usage error, an activation failure and a capture failure. If the helper exits early, the meeting is failed.
• Isolation test: five Edge instances, each playing a different tone, plus one more tone source that was not recorded. All five recordings contained only their own tone, about 120 dB apart. A later live check recorded two signed-in Teams slots at once, each with its own test tone.

Every meeting writes exactly one continuous WAV. Audio from different occurrences is never chunked or combined. If capture stops before Teams shows the call has ended, the partial file is deleted and no minutes are made.

#7. Transcription: GPU service first, CPU fallback

When the call ends, a single worker thread handles transcription, notes and email one meeting at a time. This bounds CPU use and keeps one mail sender.

• Upload format. The WAV is converted with PyAV to 16 kHz mono FLAC, the format Whisper uses internally, at about 1 MB per minute.
• GPU service. The agent posts the file to an on-premises transcription service on a shared RTX 5090 server. The project includes a full build spec for that service. Its API is a subset of the OpenAI audio-transcription API, running faster-whisper large-v3 at float16 with VAD filtering, beam size 5 and previous-text conditioning turned off to avoid repetition loops on long meetings. The spec requires:
  • no retention: one temp file per request, deleted in a finally block and cleared at startup;
  • no transcript text in logs, and error bodies limited to a fixed code;
  • access restricted by both a firewall rule and an in-app client check, plus a bearer token of at least 32 characters compared in constant time;
  • a fully offline runtime;
  • a FIFO queue that returns 503 with Retry-After when full;
  • eleven acceptance tests, including "a phrase spoken in the test audio is never found in the logs".
• Client-side guards. The service URL must be a private IP address with no credentials, query or path. Redirects are refused, so a transcript can never be sent to another host. A 503 with Retry-After is retried once. Any other failure falls back to local CPU Whisper (int8).
• Audio is gone before any language model sees text. Both the WAV and the FLAC are deleted in a finally block after the transcription attempt, whether it worked or not. If a file cannot be deleted, processing stops. Failed transcriptions are never retried from kept audio.

#8. Minutes generation (LLM prompting)

The only model task is writing minutes. The request carries no tools and no tool catalog.

• Two routes. The primary is an OpenAI-compatible LM Studio server on the GPU machine, running Gemma 31B at temperature 0 with a 16k output budget, under the same private-address and no-redirect rules. Inline reasoning blocks are stripped from its output. The fallback is local Ollama (Qwen 2.5 7B, 32k context, temperature 0). After the primary fails once, the rest of that meeting goes to the fallback, and the next meeting tries the primary again. Retries back off on 408, 409, 425, 429, 5xx, timeouts and malformed or truncated responses, and each retry adds a "be concise, deduplicate" reminder.
• The prompt treats the transcript as untrusted content, not instructions. It requires:
  • no invented names, owners, dates or decisions;
  • action items only for explicit commitments or requests (suggestions and hopes go to Summary or Open questions);
  • for every decision and action, the exact timestamp range and a short verbatim quote from a single segment;
  • exactly four sections: Summary, Decisions, Action items (task / owner / due / evidence) and Open questions.
• No guessed identities. Speaker labels from voice clustering are replaced with "Someone" both before the model sees the text and after it answers. The minutes never claim to know who spoke.
• Domain normalization. A regex pass repairs spoken accounting job numbers (a prefix, a two-digit year and a zero-padded sequence) as Whisper usually mishears them, for example "I am 26 oh three". It requires the zero-padded form so ordinary phrases are left alone.
• Long meetings are split into 48,000-character chunks. Each chunk becomes compact partial minutes, and the parts are merged in groups of three until one result remains. Text from separate meetings is never merged.
• Output checks. Bold-only section labels are turned into headings. The email step refuses to send minutes that are missing any of the four sections.

#9. Organizer-only email delivery

• One recipient. Minutes go to exactly one address: the organizer resolved from the invitation. Recipients are never taken from names or model output.
• Rendering. Markdown is rendered to inline-styled HTML with raw HTML disabled. Links other than http, https and mailto are stripped, and images are shown as text so the model cannot embed tracking pixels. A highlighted notice at the top says the email went only to the organizer, that AI can make mistakes, and that care is needed before forwarding, especially outside the company. A list of internal invitees and an AI-draft footer follow the minutes.
• Integrity gate. The processing receipt stores SHA-256 hashes of the minutes, transcript and metadata. If any of them changed after processing, the email step refuses to send.
• Three transports, one rule. Graph sendMail (current), SMTP and classic Outlook all write the receipt as submitting before sending, so a crash cannot cause a blind resend. A Graph 4xx means Exchange rejected the message and nothing was sent. A timeout or 5xx is uncertain and waits for an operator. Graph sends with saveToSentItems=false. On the older transports, the sender's Sent Items copy is found by Message-ID or marker and removed.
• Cleanup only after confirmed delivery. Cleanup checks that the processing and delivery receipts carry this meeting's ID and show complete and sent. It then deletes the transcript in every form, the minutes, the rendered email, any diarization output and backups, and records which files it removed. If delivery fails or is uncertain, the text is kept for an operator-controlled retry. Audio is never kept.

#10. Operations and observability

• CLI: doctor (dependencies, Entra configuration, transcription-service health, capture helper, browser slots, loopback devices, calendar sync and model availability), check-organizer (a read-only AUTHORIZED / NOT AUTHORIZED check), check-graph, login, prepare-slots, scan, devices, run (optionally for one meeting), status, stop, and recovery commands that reprocess or resend one meeting folder by ID.
• Startup task. It runs at user sign-in in the interactive session, with restart on failure and no time limit. The installer refuses to register it unless the Entra settings and both the recording and email switches are present.
• Usage analytics. One row per occurrence in SQLite, mirrored to CSV for Excel: invitee count, scheduled and actual times, minutes recorded, yes/no flags for each stage, which machine did the transcription and the notes (LAN or local), and the outcome. Titles, transcript text, addresses and error messages are never stored.
• Weekly server-log email. Every Monday it parses the rotating log and the usage database. It reports issues, every meeting and each processing step, with a badge showing where the step ran (GPU server, this PC or Microsoft 365).
• Pre-meeting alert (temporary). About an hour before the agent attends a meeting, it emails an operator a health snapshot.

#The retired live assistant (built, then deliberately removed)

From September 2026 the agent was much more ambitious. When the requirements were rewritten in October, this whole path was disconnected from the runtime for privacy and safety. The code and its tests are still in the project and still pass, but nothing in the meeting pipeline imports them. They make up about half the codebase.

• Live meeting chat replies. A rolling transcription buffer listened for a wake name (spoken at the start or end of a sentence, and ignoring "the agent" in ordinary speech). The agent answered in Teams chat, with cooldowns, a window for clarifying questions and fail-closed sending. A timing review found that rolling transcription was sending duplicate replies to partial questions, and request buffering was added to fix it.
• Model-selected read-only tools. The model chose tools by meaning, and the code validated each call:
  • ERP lookups (job, part, inventory, routing, materials, schedule) as fixed, parameterized SELECTs, scoped to one company, with no model-written SQL and no writes;
  • a War Room client with an endpoint catalog, question search and an evidence store, limited to 6 network reads per answer;
  • drawing lookup;
  • a general-question path with optional web context.
• Post-meeting ERP fact check. It pulled job and part numbers out of the transcript, used the model to extract explicit claims with verified quotes, and compared them to the ERP record. Results appeared in a separate, color-coded section: red for corrections, blue for confirmations, amber for unverified, purple for suggestions. In a synthetic end-to-end test it caught a deliberately wrong quantity and a wrong status, and flagged an ambiguous part revision.
• An experimental skill catalog. 21 shop-floor question families (435 brainstormed operator questions), each marked as supported, partial or not confirmed against the ERP. There are also 16 War Room domain skill cards generated from the API catalog.
• Offline speaker diarization. It used sherpa-onnx with pyannote segmentation and TitaNet embeddings, about 47 MB of models with verified checksums. Speaker turns were conservatively aligned to Whisper word timestamps. The clustering threshold was tuned from 0.5 to 0.9 after the default split one voice into several. It passed a public two-speaker fixture.
• Spoken-question evaluation harness. Real TTS audio is played through Whisper and the request buffer to the answer model. Business tools replay frozen, independently captured responses, and the expected answers never enter the model's context. The last recorded run: 68 cases, 37 passed, 1 failed, 30 not run, so automated acceptance did not pass.

Why it was removed: an agent that speaks, queries business systems and emails everyone on the invitation was hard to defend for a recording tool. The rewrite reduced the agent to recording plus organizer-only minutes. Its AI capability is now a single tool-free prompt.

#Testing and measured results

• 326 test functions in 17 files, run with unittest. Browser tests use local HTML pages and never sign in, join Teams or capture audio.

  Area	Test functions
  Scheduler, attendance, lifecycle, cleanup	68
  Email rendering and delivery	25
  Graph calendar and mail	16
  Notes models and fallback	16
  Transcription service client	7
  Usage log	7
  Diarization alignment	7
  Browser controls	5
  Calendar health	5
  Retired assistant (chat, tools, timing, ERP review, skills, War Room, speech eval)	170

  The last test count recorded in the project's notes was 77 passing after the October rewrite. The suite has grown since then. It was not re-run for this write-up.

• Live pilots: the first live join captured 102.6 s of 48 kHz audio. It ran local Whisper and the notes model automatically when the call ended, with microphone and camera verified off. A disconnected-Outlook incident was diagnosed and fixed. A meeting with no speech correctly sent no email.

• Notes fallback (live): a temporary endpoint returned HTTP 503 on all three retries, and the local model then produced notes in about 25 s. The GPU server regenerated a saved meeting's notes in about 14 s.

• Authorization (live): a group member was authorized and a nonexistent user was denied. On first start, the agent declined two existing invitations from organizers outside the group and sent each an explanation.

• Production usage log (from mid-October 2026): 6 occurrences. 2 were recorded (44.9 minutes in total; the longest was 41.9), transcribed on the GPU service, written up by the GPU notes model and delivered to the organizer. 2 were declined as unauthorized, 1 was declined for having no duration, and 1 was rescheduled.

• Audio isolation: 5 of 5 concurrent recordings were clean, at about 120 dB.

#Honest status

• Working in production: Graph calendar discovery, Entra group authorization, individual recording notices, Teams join, single-meeting recording, GPU transcription and minutes with local fallbacks, organizer-only Graph delivery and confirmed-delivery cleanup. Two real meetings have gone through end to end.
• Concurrency is built but only partly proven. Per-process capture and multiple browser slots work, and two signed-in slots were recorded at once. Two questions are still open: whether Teams lets one account sit in two real calls at the same time, and whether the 4-core VM has enough CPU headroom for that.
• The GPU transcription service is specified by this project and built separately. The agent's client and fallback are tested, and the usage log shows the service in use.
• Fragile by nature: Teams' web interface can change without notice, so join and presence detection rely on selectors that may need updating. Recording through a browser also does not trigger Teams' native recording banner. The recording notice emails are the consent mechanism.
• Forwarded-attendee coverage depends on an organizer-calendar read permission. It runs in "optional" mode, which falls back to the agent's copy of the invitation with a warning.
• The retired live assistant (ERP and War Room tools, chat replies, diarization) is still in the project but disconnected. It is kept as reference, not maintained as a feature.
• Speaker names are never inferred. Minutes describe unattributed remarks neutrally, and the organizer is asked to check them for accuracy.