Teacher Choices Feasibility Trials
05 October 2023
‘A Winning Start’ evaluation reports on the EEF website
‘The Story Time’ evaluation report on the EEF website
In 2019, EEF commissioned NFER to undertake Teacher Choices feasibility trials in response to a need for research that directly applies to teachers’ everyday practices. These trials investigate everyday classroom decisions, assess the feasibility of implementing the choices and explore the robust and efficient ways to evaluate the trials. Research questions in these trials address feasibility of school recruitment, adequacy of choice guidance, selection of outcome measures and expected trial outcomes. The trials aim to offer recommendations for future implementation and scale-up.
Teacher Choices trial: A Winning Start (first run) evaluation report
This was the first Teacher Choices trial that compared retrieval quizzes and discussions for science lesson starters. The COVID-19 pandemic prevented full trial implementation. In Spring 2020, 981 Year 8 students in 40 science classes across 13 schools took part in this crossover randomised controlled trial. Science teachers were randomly assigned to use quiz or discussion starters in each half-term, and they switched the science topic and the lesson starter approach for the second half-term.
Findings indicate:
- In the majority of lessons, teachers delivered allocated lesson starters in the allotted time, though only half the teachers delivered the allocated starters within 10 minutes in all their lessons. The implementation guidance received positive feedback for its clarity and detail and teachers felt well-supported by it. They often prepared lesson starters alone, especially quizzes, indicating ease in implementation. Quiz starters were more prevalent in practice and favoured for their adaptability and effectiveness. Teachers expressed positivity towards both quiz and discussion starters, with a preference for quizzes.
- End-of-topic tests often lacked internal consistency, indicating they assessed many different sub-domains. The comparison of classes with differing student attainment grouping within schools contributed to substantial heterogeneity in the meta-analysis. This confounded the effects of the lesson starters, and the variation in outcomes was almost certainly due to attainment groups within schools.
- The trial was well-received, but improvements in data collection and trial design were recommended for future trials.
Teacher Choices Trial: A Winning Start (second run) evaluation report
This was a second and a complete implementation of "A Winning Start" trial. It assessed the feasibility of running and analysing a crossover trial. The trial involved 1,074 Year 8 students in 48 teacher-class units across 15 schools. Science teachers were randomly assigned to quiz or discussion starters in each half-term, and switched the science topic and the lesson starter approach for the second half-term.
Findings indicate:
- Most teachers adhered to assigned lesson starters, though only half delivered the allocated starters within 10 minutes in all their lessons. There were also examples of overlap between the two types of lesson starters in terms of focus – with quizzes and discussions that included both recap and introduction of new content.
- While most teachers understood and complied with trial requirements, providing data for topic tests was challenging. Moreover, the teacher-developed topic tests lacked reliability and validity, leading partially to heterogeneity in the meta-analysis.
- The analysis found no significant difference between lesson starters, even with reliable tests. Attainment grouping practices confounded the lesson starter effects, contributing to an overall null effect. The setting practices introduce extreme class-level variance, making it challenging to address through repeated measures in a crossover analysis.
Teacher Choices: The Story Time Trial
This was a second Teacher Choices trial. It assessed feasibility of a Teacher Choices trial in primary schools and examined reading aloud approaches with 7,783 Year 4 and 5 pupils across 91 schools.
Two reading aloud approaches were compared in this trial. One approach involved reading aloud to the class without stopping (GO! approach) and the other involved reading aloud with the teacher stopping at different points in the text to ask questions (STOP! approach). The class teachers read aloud to the whole class for 15 minutes a day over a period of three weeks in November 2021. The same book – The Iron Woman – was used in both GO! and STOP! approaches. The evaluation involved a two-armed randomised controlled trial using cluster randomisation at school-level.
Findings indicate:
- The STOP! and GO! reading aloud approaches were reasonably distinctive to each other in the trial, with evidence of occasional contamination between the two approaches.
- It is possible to develop a reliable and valid text-specific assessment when developed by assessment experts. Given the theory-building nature of Teacher Choices trials and the primary analysis findings of this trial, it is suggested that they are more suited to proximal outcomes.
- Pupils in the STOP! group scored higher on the text-aligned reading and listening comprehension score compared to GO! pupils, on average. This effect size was 0.208, and the confidence interval does not cross zero. This is also reflected in teachers’ perceptions of the greater effectiveness of the STOP! approach for aiding pupils’ comprehension of the text. The STOP! approach also aligns more closely with teachers’ usual practice and pedagogical beliefs.