Speech recognition has improved dramatically over the past few years. Modern models can transcribe spoken language with impressive accuracy.
But for language-learning applications, a transcript is only the beginning.
Suppose a learner says:
"I have been working here for five years."
Speech recognition tells us what was said.
It cannot tell us:
- Was the pronunciation correct?
- Was the speech fluent?
- Did it sound natural?
Answering those questions requires speech assessment.
Rather than producing only text, speech assessment evaluates how someone speaks and provides feedback that helps learners improve.
In this article, we'll explore what speech assessment systems evaluate, how they work, and how recent advances are changing language-learning applications.
Speech Recognition and Speech Assessment
Speech recognition and speech assessment solve different problems.
Speech Recognition
"What was spoken?"
Speech Assessment
"How well was it spoken?"
A speech recognition model may produce a perfect transcript, but it cannot determine whether the pronunciation, fluency, or intonation was good.
Speech assessment builds on top of speech recognition by evaluating the quality of spoken language rather than simply recognizing the words.
What Speech Assessment Evaluates
Modern systems evaluate spoken language across multiple dimensions instead of relying on a single score.
Pronunciation
Correctness of sounds, words, and phonemes.
Fluency
Smoothness of speech, including pauses, hesitations, and pacing.
Prosody
Stress, rhythm, and intonation that make speech sound natural.
Language Understanding
Grammar, vocabulary, coherence, and response quality (typically evaluated after transcription).
Together, these form a more complete view of spoken communication.
How Speech Assessment Works
Speech assessment is not a single model. It is a combination of specialized components, each focusing on different aspects of speech.
Each component contributes a different type of signal.
Pronunciation
Measures whether sounds are articulated correctly.
Modern systems can evaluate pronunciation at multiple levels:
- sentence
- word
- syllable
- phoneme
This allows fine-grained detection of pronunciation issues instead of a single aggregated score.
Fluency
Measures speech flow rather than correctness.
It captures:
- pauses
- hesitation
- repetition
- speaking rhythm
A grammatically correct sentence can still sound unnatural if fluency is poor.
Prosody
Measures how speech is delivered.
This includes:
- stress
- rhythm
- intonation
Prosody is often what distinguishes “correct speech” from “natural speech”.
Language Understanding
After transcription, language models evaluate:
- grammar
- vocabulary
- coherence
- completeness
This layer focuses on communication quality rather than acoustic properties.
Exploring Speech Assessment Approaches
Cloud Speech Assessment Services
If you want to experiment with AI speech assessment without building pronunciation models yourself, cloud services are a practical place to start.
One of the most mature offerings is Microsoft Azure Speech Pronunciation Assessment. It provides a ready-to-use API that evaluates spoken language and returns detailed assessment results in real time.
Before diving into the documentation, Microsoft provides an interactive online demo where you can record your voice and immediately see the assessment results. It's a quick way to understand what the service is capable of.
- Interactive demo: https://ai.azure.com/explore/aiservices/speech/pronunciationassessment
- Official announcement: https://techcommunity.microsoft.com/blog/azure-ai-foundry-blog/speech-pronunciation-assessment-is-generally-available
Rather than producing only a transcript, the service evaluates several aspects of speech, including:
- Pronunciation Accuracy – how closely the pronunciation matches the expected sounds.
- Fluency – the smoothness and pacing of speech.
- Prosody – rhythm, stress, and intonation.
- Completeness – whether the expected words were spoken (primarily for scripted reading scenarios).
One feature I particularly like is its multi-level analysis. Instead of returning only an overall score, the assessment can drill down from the entire sentence to individual words, syllables, and even phonemes.
This level of granularity makes it much easier to identify where pronunciation problems occur. For example, an application can highlight a specific word or phoneme that needs improvement instead of simply telling the learner that their pronunciation score is low.
Azure also supports two common assessment scenarios:
- Scripted reading, where the expected text is known in advance. This enables detailed pronunciation analysis, completeness checking, and miscue detection.
- Unscripted (spontaneous) speech, where learners speak freely without a reference script. In this mode, Azure first performs speech recognition before evaluating pronunciation and fluency.
From our experience at Zamo, Azure Pronunciation Assessment has been a solid choice for building English learning applications. The service provides high-quality pronunciation scoring out of the box, allowing us to focus on the learning experience rather than implementing complex speech assessment algorithms ourselves.
That said, it is important to understand its scope.
Azure excels at evaluating how speech was delivered—pronunciation, fluency, and prosody—but it does not attempt to judge the quality of the learner's ideas or communication. Questions such as "Was this a good interview answer?" or "Was the grammar appropriate?" require additional language understanding, which is where Large Language Models can complement pronunciation assessment.
This combination of specialized speech assessment and LLM-based evaluation is becoming a common architecture in modern language-learning applications.
TextPA: Combining Speech Models and LLMs
Large Language Models are excellent at understanding and generating natural language, but they are not designed to evaluate raw speech signals.
Instead of feeding audio directly into an LLM, TextPA (Text-based Pronunciation Assessment) introduces an intermediate representation that converts pronunciation information into text. This allows each model to focus on what it does best: speech models analyze audio, while the LLM interprets the results and generates feedback.
Below is architect reference from A Zero-Shot Pronunciation Assessment Using Textual Descriptions and LLMs
Conceptually, the architecture can be divided into three stages.
Stage 1 – Extract pronunciation cues
The process begins with the learner's speech.
Rather than relying on a single speech model, TextPA combines several pre-trained models to extract complementary information from the audio.
- An Automatic Speech Recognition (ASR) model produces the transcript.
- A phonetic aligner identifies the recognized phoneme sequence and timing information, including pauses between words.
- A phoneme recognition model predicts the pronunciation in the International Phonetic Alphabet (IPA) representation.
Each model contributes a different view of the same speech sample.
For example, after processing the sentence:
"Maybe we should get some cards."
the extracted information might look like:
Transcript
Maybe we should get some cards.Recognized phonemes (CMU)
M EH M B IY W IY SH UH D (0.12s pause) G EH T S AH M K AH TRecognized phonemes (IPA)
/meɪbi wi ʃʊd gɛt sʌm kɑt/Notice that the system has not attempted to score pronunciation yet. It is simply collecting different textual descriptions of the speech.
Stage 2 – Build a textual representation
Instead of sending raw audio to the LLM, TextPA converts all extracted information into a structured prompt.
Conceptually, the prompt might look like this:
Transcript:
"Maybe we should get some cards."
Recognized CMU phonemes:
...
Recognized IPA phonemes:
...
Task:
- Evaluate pronunciation accuracy and fluency.
- Explain the reasoning behind your assessment.This is the key idea behind TextPA.
Rather than asking the LLM to understand audio, it provides the LLM with structured linguistic information that it already knows how to reason about.
Stage 3 – Generate assessment and feedback
The LLM analyzes this structured representation and produces pronunciation scores together with natural-language explanations.
For example:
Accuracy: 4 / 5
Fluency: 3 / 5
Reasoning
- Pronunciation is generally accurate.
- The pause before "get" slightly reduces fluency.
- The final consonant in "cards" is weakly articulated, making the ending less distinct.This hybrid architecture is what makes TextPA particularly interesting.
Instead of trying to replace traditional pronunciation assessment with an LLM, it combines specialized speech models with language models in a complementary way.
- Speech models analyze pronunciation, phonemes, and timing.
- The LLM interprets those signals, assigns scores, and generates detailed explanations.
- IPA matching provides an additional objective measure to refine pronunciation accuracy.
By separating these responsibilities, TextPA is able to produce feedback that is both technically grounded and easy for learners to understand.
Applying Speech Assessment in Koffi Talk
At Zamo, one of the products we have been building is Koffi Talk, an AI-powered English speaking practice platform. Rather than practicing through multiple-choice questions or text conversations, learners improve their speaking skills by having natural conversations with an AI tutor.
Because the goal is to help learners become more confident English speakers, speech assessment is an important part of the learning experience. Simply converting speech into text is not enough—we also need to evaluate how the learner speaks and provide feedback that helps them improve over time.
While designing Koffi Talk, we researched and evaluated various speech assessment technologies. This article highlights only two examples—Microsoft Azure Speech Pronunciation Assessment as a production-ready service and TextPA as an emerging research approach—but they represent only a small part of the broader landscape. Depending on your requirements, there are many other commercial services, open-source projects, and research techniques worth exploring.
For Koffi Talk, our priorities were clear: we needed a solution that was mature, reliable, and straightforward to integrate into a production system while still providing detailed pronunciation feedback.
After evaluating several options, we chose Azure Speech Pronunciation Assessment as the foundation for our pronunciation evaluation. It provides detailed assessment across multiple levels—from sentences down to individual phonemes—along with pronunciation, fluency, and prosody scores. This allowed us to focus our engineering effort on building the learning experience instead of implementing and maintaining complex pronunciation assessment models.
That decision reflects the requirements of our product rather than a universal recommendation.
Every application has different priorities. Some teams may prioritize rapid integration and operational simplicity. Others may need greater control over the assessment pipeline, support for on-premises deployment, lower operational costs at scale, or the flexibility to experiment with custom models and research-driven approaches.
The important takeaway is not that one solution is universally better than another. Instead, understanding the available approaches—and the trade-offs they involve—makes it much easier to select the technology that best fits your product, users, and long-term goals.