🎯

Pronunciation Scoring Model

Speech pronunciation quality scoring system built on OpenAI Whisper large-v3, taking student audio and reference text as input and outputting a 0-100 pronunciation score.

PyTorchTransformersWhisper large-v3ONNX Runtime

Core Work

Designed Cross-Attention + Attention Pooling scoring architecture: frozen Whisper encoder extracts audio temporal features (1500×1280), Cross-Attention achieves text-audio timestep alignment

Attention Pooling learns weighted pooling, solving the temporal information loss problem of mean pooling

Trained with MSE + PCC joint loss function, combined with Early Stopping and ReduceLROnPlateau strategies, on 150K labeled samples (~9.5M trainable parameters)

Implemented feature pre-extraction + fast training pipeline: offline Whisper encoder feature extraction saved as NPY (memmap writing to avoid memory overflow), achieving seconds/epoch during training for rapid hyperparameter tuning

Completed ONNX export and INT8 quantization deployment: GPU fp16 inference 110ms, ONNX CPU INT8 inference 4.2s, cosine similarity 0.989

Tech Stack

PyTorch / Transformers / ONNX Runtime / Whisper large-v3

🏗️ Architecture diagram coming soon...

📈 Performance metrics coming soon...