Does Progress 8 need refining to make it fairer?

The latest GCSE and equivalent attainment results, which were published yesterday by the Department for Education (DfE), come at a time when there is some questioning about the inherent fairness of Progress 8 towards schools with higher proportions of disadvantaged and SEN pupils. In this blog, we review the history of the measure and set out some of the different points of view.

Introduction

Progress 8 is a form of value added measure which aims to capture the progress that pupils in a school make from the end of primary school to the end of Key Stage 4. It was introduced as the principal headline performance measure for all secondary schools in 2015/16 [1].

Prior to the introduction of Progress 8, the main headline accountability measure in use was the proportion of pupils achieving five or more A* to C grade passes, including English and maths. These types of measures are commonly described as threshold measures as they measure what proportion of pupils achieve a certain level. There were a number of concerns about using this measure to hold schools to account, in particular that it did not take any account of pupils’ prior attainment when they started at a school. NFER's research into accountability systems in different countries highlights the way in which threshold measures can distort school behaviour, encouraging them to focus on children just below the threshold, at the expense of those performing less well (or higher attainers). Similar concerns have been expressed in this country, that some schools were just focusing on pupils on the C/D grade borderline rather than everyone in the cohort.

Prior attainment is a key factor in explaining how well children do at the end of school: Education Endowment Fund analysis shows that a pre-test score (KS2) explains around half of the variation in GCSE scores (depending on the subject).

Progress 8 was designed to address these concerns. As well as measuring progress from Key Stage 2, it was designed to encourage schools to offer a broader, balanced curriculum with a focus on an academic core at Key Stage 4; to encourage schools to focus on all of their pupils as every increase in a grade will count towards their score; and to measure performance across a broader curriculum of 8 qualifications.

Should Progress 8 also take account of other factors?

Many commentators agree that Progress 8 has been a big improvement compared to the previous threshold measure as it takes account of or “controls” for prior attainment. There are, nonetheless, some concerns that Progress 8 does not take account of differences in pupil demographics and socioeconomic factors which can vary substantially between schools.

Why does this matter? Well research shows that pupils from disadvantaged backgrounds achieve around half a grade lower in each subject compared to pupils with similar prior attainment. This has a disproportionate effect on schools with higher proportions of disadvantaged pupils on their rolls. Research published yesterday by Leckie and Goldstein suggests that over a third of “underperforming” secondary schools would no longer fall into the category if progress measures were re-weighted to account for pupils’ backgrounds.

There is a counter perspective to this. One of the objectives of this and previous Governments is to achieve higher levels of social mobility. The education system is one of the key levers that the Government have to take forward this objective. They want to ensure that the right incentives are in place to encourage schools to close the attainment gaps that exist between disadvantaged and non-disadvantaged pupils, between pupils with SEN and without SEN, between boys and girls, between different ethnic groups, etc. Controlling for demographic and socio-economic factors could arguably weaken these incentives for schools to help their pupils achieve their full potential.

So what is the answer?

If there was a perfect accountability measure, we would no doubt be using it now! 

There is no perfect solution, but most people agree that Progress 8 is definitely much better than the old threshold measure that was used before as taking account of prior attainment is a much better basis on which to draw conclusions about the extent to which schools have contributed to pupils’ achievements at the end of secondary school.

That said, demographic and socioeconomic factors do vary significantly between schools. Wilkinson, Bryson and Stokes suggest these factors explain between 15 to 17 per cent of the variation in GCSE scores. In addition, some of the large and persistent attainment gaps we see between, say, disadvantaged pupils and their more affluent peers may be due to other factors (e.g. home environment factors and different levels of support from other public services) which are outside of a school’s control. It therefore seems tough on schools with less affluent intakes not to take these factors into account in some way – and perhaps does not stretch “higher” performing schools in affluent areas sufficiently.

Various solutions have been proposed, including developing an adjusted Progress 8 measure which takes account of demographic and socioeconomic characteristics or the NAHT proposal to compare Progress 8 across similar school groups. We believe that this is the time to take a fresh look at this, to see whether Progress 8 can be refined so that all schools feel they are being more fairly judged in future.

[1] Secondary schools were given the opportunity to opt into the new accountability measures in 2014/15. Only about 10% did.