Spike Jonze released Her in 2013. Thirteen years later, the most advanced voice AI systems on the planet still do not reliably demonstrate the ability to distinguish between "that sounds great" said with relief, and "that sounds great" said through clenched teeth.  Samantha felt surreal because she not only heard but listened and adjusted her model of Theodore in real time, across hundreds of hours, without him ever having to explain himself.

The dominant voice AI architecture is the cascade: ASR → LLM → TTS. Speech comes in, gets transcribed, gets reasoned about, gets spoken back. Modular. Debuggable. Each component is improvable independently. Each component swappable independently, and utterance-level model routing becomes tractable when the text boundary gives you a clean intervention point. This architecture won for good reasons. But it has a structural flaw that no amount of component improvement can fix: it discards or flattens much of the paralinguistic information at the ASR ->  text boundary. 

When a customer says “that sounds great,” the cascade passes along a short string of text. What it loses: whether those words carried genuine enthusiasm or resigned acceptance. Whether the speaker’s voice tightened slightly, suggesting doubt they haven’t articulated. It reasons over an impoverished representation of what was actually communicated.

The alternative is end-to-end models that process audio tokens directly and promises to preserve this information. OpenAI’s voice mode and Kyutai’s Moshi are public examples of this direction. But end-to-end models introduce their own problems. Reasoning quality degrades in audio-token-space versus text-token-space. Debuggability disappears. Safety and alignment become harder without an intermediate text layer to inspect.

A February 2026 paper, “The Cascade Equivalence Hypothesis,” tested four speech LLMs (Qwen2-Audio, Ultravox, Phi-4-MM, Gemini) and found that the E2E promise hasn’t been delivered. The problem is two-layered: cascades structurally discard paralinguistic signals at transcription; the information is gone before reasoning begins. But public results do not show that the current E2E models have solved this. In many cases, behavior is consistent with the model relying heavily on text-like internal representations while making limited use of retained acoustic information. Under noise, they perform worse than cascades, not just equivalently.  Architectural access to paralinguistic signals may exist, but effective utilization has not yet been clearly demonstrated in public evaluations. What’s happening inside proprietary systems at OpenAI, Google DeepMind, or other major labs is unknown.

The field divides speech tokens into “acoustic tokens” (optimized for waveform reconstruction) and “semantic tokens” (optimized for linguistic meaning). A comprehensive late-2025 review revealed that many so-called “semantic tokens” appear to be more phonetic than semantic; capturing sound patterns more readily than communicative meaning. We don’t have the right vocabulary to describe what “understanding” audio means at the token level.

Recent benchmarks quantify how badly current models fail at paralinguistic understanding. LISTEN (October 2025) found that when text and audio emotion cues conflict — neutral words in an angry tone — model accuracy collapses to 37–43%, near random. For purely paralinguistic scenarios with no lexical cues: 12.5%, barely above chance. EMIS confirmed: when speech models are explicitly prompted to ignore word meaning and focus on acoustic emotion, they still predict based on word-level labels over 80% of the time. These results suggest that current models still struggle to separate what was said from how it was said.

The most practically advanced work is narrow: AssemblyAI’s Universal-Streaming model trains on a special end-of-turn token, replacing crude silence-threshold approaches. It’s real progress. But knowing when someone stopped talking is orders of magnitude easier than knowing what they meant while talking.

Look at where the investment went in 2024 and 2025. ElevenLabs v3, Fish Speech S2, StepFun EditX, Cartesia Sonic-3: a cluster of major TTS releases compressed into months of each other.

The incentive structure is legible. TTS produces a demo. You play a clip, it sounds human, and the investor writes a check. Quality benchmarks are clean, and the finding that a standard LLaMA backbone plus rigorous RL training beat exotic architectures made the playbook copyable. VC incentives compound the effect: TTS revenue is demonstrable, while ‘improved paralinguistic understanding’ is a harder sell.

But this is not only herd behavior. The input side is robust streaming ASR, noise handling, and full semantic understanding is genuinely harder. Foundational models like Whisper were designed for batch transcription, and bending them toward real-time use requires non-trivial engineering. Getting a robust streaming ASR that works across production acoustic diversity is unsolved at scale.

The semantic understanding layer above transcription is harder still. TTS has a natural training signal: does it sound right? ASR understanding has no clean equivalent. You are either proxying through emotion labels — with a noisy ceiling — or through behavior prediction. The labeling bottleneck is a philosophical obstacle, not just an engineering one.

To understand where the real knowledge lives, we need to look at how the leading voice AI companies have approached speech and be honest about what their architectures actually are.

What all of these companies share is hard-won knowledge about the paralinguistic layer on the output side. Nobody has solved it on the input side. Hume AI came closest commercially, training production-grade emotion recognition models to modulate response generation — but their framework reduces to categorical emotion labels rather than a communicative state model, and with the core engineering team reportedly now at DeepMind, it is unclear whether that earlier momentum remains concentrated there.

StepFun’s EditX work demonstrates that RL can teach a model to represent paralinguistic signals with enough fidelity to reproduce them. The same representational infrastructure, pointed at incoming audio, is the closest the field has come to the input understanding layer this article describes.

What doesn’t exist and should: an intermediate representation between raw audio and text that preserves the signals humans actually use to understand each other. This layer would capture confidence trajectories (how certainty shifts within a single utterance), emotional dynamics (state changes during conversation, not just point-in-time labels), turn-taking signals (pause as invitation vs. thought), cultural communication patterns, and cognitive load indicators (speech rate, filler frequency, self-corrections, breathing patterns).

Emotion classification is the paralinguistic equivalent of keyword spotting. Labeling an utterance “happy” or “angry” tells you almost nothing about what happens next. The best specialized SER models achieve 66–78% accuracy; even 100% wouldn’t solve the problem, because knowing someone sounds angry doesn’t tell you why, what they’ll do, or what they need right now.

The real gap is pragmatic inference. What’s needed is a communicative state model: epistemic state (what the speaker believes right now, how certainty shifts), pragmatic intent (what they’re trying to accomplish by saying this in this way), the unsaid (the pause before an answer, the breath suggesting they almost said something else), and model of the listener (are they adjusting because they think I don’t understand?). Emotion classification alone does not get you there.

Humans often don’t understand their own intent. A customer doesn’t always know whether they’re going to buy. Intent is often constructed after the fact.

This helps explain why emotion recognition systems often plateaus at 66–78% accuracy on benchmark settings. The labels are inherently noisy because the thing being labeled isn’t objectively accessible. Audio alone is fundamentally insufficient. If faces are insufficient without context, audio-only clips are even more impoverished. Real communication is vastly ambiguous.

What actually works? Interpretive models that are useful without being “true.” A skilled therapist doesn’t discover a patient’s objective intent — they construct a working model that changes behavior. AI should do the same. Not a single label. Not pure behavior prediction. Ranked working hypotheses.

The fix isn’t better emotion labels. It’s a different architecture entirely.

First, outcome-grounded representation learning. Stop training on subjective emotion labels. Train on observable outcomes: the customer renews, the deal closes, the student passes.

Second, multi-hypothesis state modeling. A single label is always wrong. The architecture needs to output a weighted distribution across interpretive frameworks — ranked working hypotheses, just like a human listener.

Third, a parallel integration architecture. ASR handles the words. A parallel path handles the communicative state. Both get passed to the LLM. You don’t evaluate this on classification accuracy. You evaluate it on intervention: did the system’s model lead to a better response?

Sales: surfaces “the buyer’s interest dropped when you mentioned timeline.” Healthcare: contextualizes vocal biomarkers within conversational dynamics. Customer experience: identifies when frustration shifted to resignation, the moment you’ve lost them. Education: detects confusion from prosodic signals before a student articulates it.

We’ve built AI that can hear. We’ve built AI that can think. We’ve built AI that can speak. What we haven’t built is AI that can listen. The companies that build the layers bridging perception to inference — preserving signal through the translation instead of flattening or discarding it — will create the most durable value in voice AI.

Roam is the virtual workspace our team relies on to stay connected across time zones. It makes collaboration feel natural with shared spaces, private rooms, and built-in AI tools.

Roam’s focus on human-centered collaboration is why they’re our Premier Partner, supporting our mission to connect the builders and leaders shaping the future of AI.

The AI Collective is a community of volunteers, made for volunteers. All proceeds directly fund future initiatives that benefit this community.

Thank you to the thousands of volunteers around the world who make this work possible. We truly could not do this without you.