Prof. Dagmar Schuller of audEERING: “The Voice Is the Mirror of the Soul”

Prof. Schuller, you’ve been working in the field of AI-based audio analysis for over a decade. What has changed the most in terms of how this topic is perceived—both among your clients and in society at large?

Dagmar Schuller: A lot has happened over the past ten years. The catalyst was certainly that generative AI became accessible to many people overnight, even though research had been underway for a long time. The biggest change is that, for the first time, people are truly recognizing the potential it holds and are focusing much more on practical applications. What hasn’t changed is the European reluctance: people are afraid of many things they don’t understand. Some expect too much from AI, others too little. But the bottom line is clear: AI is here to stay. Those who view it from an opportunity-oriented perspective have an advantage.

The typical response when it comes to AI and audio is: transcription, text-to-speech, and voice assistants. You do something different—you don’t analyze what is being said, but how it is said. How do you explain that to someone in the media industry?

Schuller: The easiest way to explain this is with an example. Imagine you ask three people, “Do you want to buy a car in the next twelve months?” All three say “yes.” In terms of content—based purely on speech recognition—you have the same word three times. But in reality, it may be that only the first person is truly convinced and wants to buy a car, the second is still undecided, and the third says “yes” ironically but actually means “no way.” A traditional transcript doesn’t tell you that. The voice does. Socrates, Plato, and the Greek philosophers all agreed on this: The voice is the mirror of the soul. That sounds poetic, but it’s surprisingly accurate.

Music Selection and Emotion: Technology in Practice

What are some specific use cases for the media industry—radio, podcasts, and streaming?

Schuller: There are so many. One exciting approach, for example, would be interactive music selection on the radio based on “crowd emotion”—that is, how is the majority of callers feeling right now, and which playlist matches that mood? We could also measure how listeners react to certain formats or information. This is particularly interesting at a time when fake news is a hot topic: There’s hardly any fake content on the radio, but you could analyze listeners’ reactions to specific moments in a conversation and assess how credible something seems. And, of course, there’s the area of personalized advertising: displaying a different ad based on the listener’s emotional state.

You analyzed the Trump-Harris debate. What did you see there?

Schuller: That was very revealing. We looked at where the candidates’ voices fell on the emotional spectrum and created a sort of heat map for emotional dimensions. For much of the debate, Trump’s voice had a very strong negative valence and high negative activation—in other words, it was emotionally charged in the negative range. That’s actually unusual for him. And it’s interesting because people often perceive such pitch levels subconsciously, even when they aren’t listening intently. They trigger reactions that people might not even be able to consciously identify.

Such abilities can also be misused. How do you deal with that?

Schuller: From the very beginning, we’ve been very clear about which customer segments we don’t want to enter. We’ve always rejected inquiries from the security sector—such as whether this technology can be used in interrogation situations. Not least because AI never achieves 100 percent accuracy. Anyone who tells you otherwise is lying. And in situations with enormous consequences, you cannot rely on full automation. Humans must remain in the dominant decision-making role. That is very important to us.

If we imagine a Bavarian radio station—a classical music station with 20 hosts broadcasting live—where exactly would audEERING technology fit in?

Schuller: There are some exciting possibilities. First, you could give the host an intelligent co-host that reacts fully automatically based on the conversation and also interacts with the listeners. Second, you could gauge the mood: How is a particular region feeling right now, and what music should you play to match that mood? And third—which I find particularly fascinating—you can detect anomalies in recorded interviews: Is that still the real voice, or was a deepfake used here? This is becoming increasingly important for journalism.

El Dorado: A Vibrant Media Ecosystem in Bavaria

What are the limits of technology? What can’t a machine do?

Schuller: The machine can’t recognize gut feelings. Or a guilty conscience. It recognizes what it has been taught and replicates how humans would react. In certain areas, it’s better than a human because it has more data: detecting intoxication, certain symptoms of illness. But anyone who knows what the system has been trained to do can also outsmart it. And while that becomes more difficult with better data—it’s never impossible.

audEERING started as a spin-off from the Technical University of Munich. What, specifically, does Bavaria offer you as a location?

Schuller: Bavaria is Germany’s El Dorado for startups, and it’s no coincidence that Munich ranks among the top three European locations—after London and Paris, and sometimes even between the two. The ecosystem centered around TUM, LMU, Bayern Innovativ, and Munich Innovation is extremely dynamic. What has helped us in particular is the talent pool, because many people are very eager to settle here. And then there’s the quality of life and the concentration of major tech companies. The only downside: finding an apartment is expensive and difficult.

In five years, how will AI-powered audio analysis have changed things that hardly anyone is aware of today?

Schuller: AI-powered audio analysis will be an essential component of the next generation of large language models. What we’re seeing right now is a trend toward multisensory processing. Audio, video, and text are converging, and the systems are becoming significantly more context-sensitive. And then there’s Physical AI: how we interact with our physical environment. Audio, as a natural input, will play a central role in this. Another trend on the horizon: the conversion of audio to text and back will increasingly become a thing of the past. We’ll see audio-to-audio; OpenAI recently released new models that already perform very well end-to-end. That’s the next wave.