This study investigates factors that could affect inter-examiner reliability in the pronunciation assessment component of speaking tests. We hypothesise that the rating of pronunciation is susceptible to variation in assessment due to the type and amount of exposure examiners have to non-native English accents.
In this study we conducted an inter-rater variability analysis on the English pronunciation ratings of three representative test candidate interlanguages: Chinese, Korean and Indian English. Pronunciation was rated
by 99 examiners across five geographically dispersed test centres where examiners variously reported either prolonged exposure, or no prolonged exposure to the interlanguage of the candidates. The examiners rated the three speaking test candidates with a significant level of inter-rater variation. Pronunciation was rated significantly higher when the candidate’s interlanguage phonology was familiar, and lower when it was unfamiliar. Moreover, a strong association between familiarity and the pronunciation rating was found.
We attribute this to psychoacoustic processes, namely, the perceptual magnet effect, and the resulting sociolinguistic phenomenon at the level of communicative interaction. This phenomenon we have termed interlanguage phonology accommodation. We found that interlanguage phonology accommodation is associated with inter-rater variation and should therefore be a major consideration in the design of speaking tests and rater training.