Holistic assessments of oral language proficiency are often made in relation to performance in conversational language proficiency interviews, one such example of which is the IELTS Oral Interview. This study seeks to explore the rating practices of trained and accredited IELTS raters when judging candidates' performance in IELTS interviews. In particular, it aims to address questions such as:
- How do raters cope with the task of having to base an assessment of ability on a single performance?
- What is the relationship of linguistic and non-linguistic aspects of the performance?
- How is the interlocutor's performance dealt with in the assessment of the candidate's ability?
- Do raters focus on criteria other than those specifically mentioned in the descriptors?
- How salient are the stated criteria?
- Does the same performance elicit judgements of the same kind from different raters?
This study adds to a small but growing body of qualitative research into the judgements made in assessments of second language speaking proficiency. Using data (taped IELTS interviews) collected in an earlier study (Brown and Hill, 1998), eight IELTS raters each rated four interviews selected from a set of eight using the IELTS bandscales. For each interview they provided a verbal protocol where they first summarised the reasons for the score they had awarded and then reviewed the tape in order to identify those features of the rating procedure which influenced their scoring. This methodology is known as stimulated verbal recall (di Pardo, 1994). In these, the raters were asked talk about the judging process and to identify the salient decision-making points of the interview.
The raters were all accredited and practicing IELTS interviewers. The candidates were all overseas students drawn from a pre-university (Foundation) course. At the time of the interviews they were preparing to take IELTS prior to submitting applications for tertiary study in Australia.
The protocols were transcribed and coded. Findings are discussed and implications are drawn regarding the validity of this test format.