Multimodal framework jointly improves ASR and Dialect Identification in Indian languages

Researchers propose a multimodal framework that simultaneously enhances Automatic Speech Recognition (ASR) and Dialect Identification (DID) for low-resource Indian languages. The method uses a Bottleneck Encoder to extract dialectal features from Conformer-based speech representations and a RoBERTa encoder to process ASR-generated CTC embeddings.

• A gating mechanism merges the extracted features, followed by an attention encoder to refine representations.
• Learned embeddings are concatenated with Conformer outputs to enhance ASR features.
• Evaluated on eight Indian languages with thirty-three dialects, the method achieves an average DID accuracy of 81.63%.
• The system reaches an average Character Error Rate (CER) of 4.65% and Word Error Rate (WER) of 17.73%.

These results highlight the effectiveness of joint ASR-DID modeling for improving performance across diverse dialectal variations.