Validation Study of the PISA 2000, PISA 2003 and TIMSS-2003 International Studies of Pupil Attainment
04 July 2006
Research report available to download from DFE
Research brief available to download from DFE
In this study, commissioned by the Department for Education & Skills (DfES), a panel of expert raters examined test items from two key international studies, the Programme for International Student Assessment (PISA, http://www.pisa.oecd.org), and the Trends in International Mathematics and Science Study (TIMSS, http://timss.bc.edu). Test items from TIMSS and PISA were compared with questions drawn from Key Stage 3 and GCSE papers, to assess whether they differed sufficiently to explain any differences in the performance in the two studies of students in England. The focus of the study was the degree to which test items were familiar to, and appropriate for, students in England; not their level of difficulty.
Key Findings
- The science items used in TIMSS and PISA were found to have generally similar ratings for familiarity and for appropriateness. In mathematics, however, the TIMSS items were rated as more appropriate and likely to be more familiar in context and item format than the PISA ones.
- PISA, TIMSS, key stage 3 tests and GCSE all used the same range of item types - multiple choice, short response and more extended response - although the balance between these types varied.
- The most significant difference found between PISA and TIMSS was the amount of reading required in PISA, which was also greater than that required in the key stage 3 tests or GCSE examinations in England. Often this high reading demand was found to be coupled with a relatively low level of mathematics or science demand. This combination would be expected given PISA's focus on the assessment of mathematical and scientific literacy.
- The PISA approach, assessing mathematical or scientific literacy, leads directly to assessment in context. This makes the largest difference in mathematics, where TIMSS, KS3 and GCSE all present some out-of-context questions, while PISA does not.Students in England are, however, familiar with both in- and out-of-context mathematics, and the study indicates that PISA's heavy use of contextualised items neither favours or disadvantages them.