Pretest effect research: literature reviews and interview strand 2

Client: QCA | NFER Contact: Liz Twist

The development process in national curriculum testing is sophisticated and thorough, using advanced methods for analysing and linking tests in order to produce instruments that function well and maintain the standard from test to test. Over several years, one area in the process where a level of uncertainty has remained concerns what has become known as the pretest effect - evidence of some differences in pupils’ performance on a ‘live’ test compared to that on a pretest which have an impact on test scores. Whilst strategies have been developed to try to identify and accommodate these differences at the level setting phase, this has been on a test-by-test basis. The project provided the opportunity to look across a range of subjects, both key stage 2 and key stage 3, and the work of different agencies, both here and abroad, to try to identify what might be influencing this apparent effect and what may need to be considered in order to mitigate it.

Research questions

• What are the factors that influence the pretest effect?
• Are certain item types more prone to the pretest effect?
• Which types of test equating methodologies are most prone to the pretest effect?
• Are pretest effects stable over time for different cohorts?
• How do changes to test specifications affect the pretest effect?

Impact and outcomes

A final report to the sponsor provided a summary of the literature review, including the search strategy and rationale. Outcomes of interviews with international test developers and psychometricians were also provided.
A better understanding of the pretest effect and how it is addressed outside the national curriculum test context would enable the sponsor to respond more critically to the phenomenon as it appears in national curriculum tests.

Research design and methods

The literature review concentrated on studies examining:

• test development procedures, and in particular the use of embedded items in pretesting
• differences in test administration and concomitant ‘item parameter drift’
• social factors associated with item parameter drift such as motivation and test preparation
• differential item functioning and test equating methodologies.

Interviews with international test developers and pyschometricians:

• identified key features of tests developed including the assessment aim and frequency of different item types used
• established whether pretest effect is recognised as an issue in the test development process
• where pretest effect was observed, explored the impact of the pretest effect on the level setting process and perceptions of causes of the pretest effect
• if test development processes existed where pretest effect is not an issue, established key features of the process
• gained an insight into the wider political influences that impact on the test development process.

Audience(s)

The research is of use to personnel at Qualifications and Curriculum Authority as well as the wider assessment/measurement community.

Time scale: July 2007 - December 2007

NFER Project Code: PTF