17th EALTA Conference: Variations on Common Standards
Tuesday 1 June 2021
Online Event
Variations in Setting Cut Scores: How Comparable are Cut Scores Across Media, Methods, Time and Instrument Length?
The purpose of European Association for Language Testing and Assessment (EALTA) is to promote the understanding of theoretical principles of language testing and assessment, and the improvement and sharing of testing and assessment practices throughout Europe. From 1 – 6 June, they are holding the 17th EALTA conference which is titled Variations on Common Standards.
On 5 June at 14:00, NFER’s Research Director Charalambos Kollias will be presenting on ‘Variations in Setting Cut Scores: How Comparable are Cut Scores Across Media, Methods, Time and Instrument Length?’
Setting standards, an integral part of assessment, is a complex, time consuming, and expensive endeavor. This has resulted in several standard setting practitioners exploring possible other avenues to reduce the associated high costs with and/or the time needed for conducting a standard setting workshop. Some practitioners have investigated setting cut scores in virtual environments, while others have explored setting cut scores on a subset of items.
Charalambos will speak on his paper, which reports on 5 different panels setting a Common European Framework (CEFR) cut score on an English language test. The first panel convened face-to-face to set a cut score using the Modified Angoff method on the full range of items. Several years later, four different panels were convened to set a cut score on only a subset of the items in two virtual environments (audio or video) using the Yes/No Angoff method.
As all cut scores were analysed through Many-facet Rasch Measurement Theory (MFRM), cut scores were directly comparable across. This paper will present the findings of variations in setting cut scores across media, methods, time, and subsets of items and will discuss the implications and directions for future research.