Stony Brook's CONCORD Lets AI Assistants Share Conversations Without Recording Bystanders
CONCORD reframes always-listening AI as a coordination problem between two agents that each hear only their owner, then negotiate the missing context safely.
Takeaways
- CONCORD is a privacy-first framework where each AI assistant records only its owner, then asks other assistants for missing context.
- Uses ECAPA-TDNN speaker verification at 0.08 real-time factor to filter out non-owner speech on-device.
- Achieves 91.4% recall on information gap detection and 97% true negative rate on privacy-sensitive disclosure decisions.
- A three-tier relationship classifier (intimate, social, professional) gates what one assistant can reveal to another.
- Beats Chain-of-Thought, Self-Reflexion, and Tree-of-Thought baselines across GPT-4o, Claude 3.5, and Gemini 1.5 in the paper.
- Limited to two-speaker dialogue today; no public code or dataset release yet from the Stony Brook and Microsoft Research team.
Always-listening voice assistants have an uncomfortable design flaw. To be helpful, they have to record everyone in the room, including people who never agreed to be recorded. A new framework called CONCORD, from researchers at Stony Brook University and Microsoft Research, tries to escape that trade-off by making each assistant deaf to everyone except its own owner, then having the assistants quietly talk to each other to fill in what they missed.
The paper introduces an asynchronous assistant-to-assistant (A2A) protocol where two AI agents, each running privacy-preserving speaker verification, reconstruct a shared conversation through negotiated queries rather than raw audio capture. Across a multi-domain dialogue dataset, CONCORD achieves 91.4% recall in gap detection, 96% relationship classification accuracy, and a 97% true negative rate on privacy-sensitive disclosure decisions.
The bystander problem nobody solved
The industry has been sprinting toward proactive assistants that anticipate what you need before you ask. LlamaPIE, DailyLLM, and a wave of MCP-connected agents all assume the microphone can just stay on. Model Context Protocol implementations primarily emphasize connectivity and tool invocation, without explicitly modeling interpersonal relationships, interaction stakes, or socially grounded access constraints. In practice that means every conversation partner, coworker, doctor, or stranger in a coffee shop gets captured whether they consent or not.
The naive fix, having the assistant guess the missing half of the conversation from the owner's side, creates a different problem. Inference-based reconstruction may produce incorrect assumptions, such as guessing the answer to "What time is the meeting?" without verified access to scheduling details. That is exactly the hallucination trap CONCORD is designed to avoid.
Listen to one, ask the other
CONCORD's pipeline starts with a hard privacy gate. The team implements speaker verification using the ECAPA-TDNN architecture at the voice recording end, chosen for its lightweight footprint and strong performance. Deployed on Android with TFLite optimizations, the system achieves a Real-Time Factor of 0.08, allowing it to run in the background with minimal computational overhead. On the VoxConverse test set it hits a 0.8% false positive rate at 99.2% true positive rate, with the threshold explicitly tuned to prefer missing the owner over accidentally recording a bystander.