Evaluation of the implementation and impact of Diplomas
The introduction of Diplomas for 14-19 year olds, at three levels and across 14 or more ‘lines’ of learning, represents a major innovation in educational opportunity for young people in England.
The Diploma consists of:
- ‘generic learning’ (functional skills in English, mathematics and information and communication technology (ICT), a project, and development of personal, learning and thinking skills)
- ‘principal learning’ (sector-related knowledge and underpinning skills needed to progress in relevant sectors)
- ‘additional/specialist’ learning (a selection of options can be chosen from a range of qualifications).
Diplomas also include learning in the workplace (a minimum of ten days’ work experience) and learning through realistic work environments to enable the development of practical skills and work-related application of learning.
In March 2007, the DCSF announced that up to 40,000 Diploma places would be available from September 2008 through the first 149 consortia (groups of schools, colleges, training providers and employers) approved to offer Diploma courses in the first five ‘lines’ of learning (construction and the built environment, engineering, information technology (IT), creative and media, and health, society and development in phase 1). A further five lines will be available from September 2009 (phase 2) and four others in 2010 (phase 3). In October 2007 the Secretary of State announced that a further three ‘broad subject’ lines are to be added from 2011. In January 2008, the DCSF commissioned the NFER and the University of Exeter to conduct the national evaluation of the implementation and impact of Diplomas over the period 2008-2013.
The purpose of the evaluation is to provide policy makers and practitioners with systematic and robust evidence which will enable them to make informed judgements about the outcomes of the Diplomas for different stakeholders and to make improvements to design and delivery, if appropriate. The evaluation has two main aims:
- to review the implementation and delivery of the Diplomas
- to assess the impact of the Diplomas on young people’s participation, attainment and progression.
Impact and outcomes
The evaluation will inform the implementation of the Diplomas across the three phases and beyond. It will gather evidence to help the DCSF and its partners understand the impact of Diplomas on young people, educators, employers and higher education institutions (HEIs). The evaluation will offer numerous dissemination opportunities such as briefings for policy-makers and reports/articles for practitioners. The evaluation team will produce three reports in each year: an interim report in May, an annual report in October,and a thematic report at an appropriate time.
Research design and methods
The research design provides a complementary mixed-method approach to address the complex range of issues and aims, and has three main strands: surveys of stakeholders, in-depth case-study visits and analysis of external datasets. Further details about each strand are presented below.
In each phase of implementation, surveys of a range of stakeholders will be undertaken in a sample of 30 consortia.
Surveys of learners
Prior to each phase of implementation, ‘take-up’ surveys will be completed by learners in year 9 and year 11 in the selected consortia in order to explore reasons for take-up and non-take up of Diplomas and whether this changes over time (spring 2008 and 2010).
Longitudinal questionnaire surveys will also be carried out with all students who have chosen to pursue a Diploma (starting when they are in either year 10 or year 12 or equivalent) in the sampled consortia in each phase of implementation (spring 2009-2013). In addition, longitudinal surveys of a comparison group of students in the same institutions will be undertaken. These surveys will explore learners’ experiences of their Diploma journey and the impact of Diplomas on their choices and future destinations.
Surveys of teaching staff (spring 2010, 2011, 2012)
The research study will include cross-sectional questionnaire surveys of teachers and tutors employed by schools, colleges and training providers, who are engaged in delivering the Diplomas in each phase. The sample will comprise up to ten teachers for each school, and up to five tutors for each college and training provider in the sampled consortia. The surveys will explore their experiences of delivering Diplomas and the factors they think facilitate successful implementation.
Survey of employers (autumn 2010 and 2012)
A baseline telephone survey of 140 employers not actively involved in Diploma delivery will be undertaken in autumn 2010 in order to explore their levels of awareness of Diplomas and the extent to which they are planning to include Diplomas as part of their recruitment policy. A follow-up survey of the same individuals will be carried out in autumn 2012, to investigate any change in perceptions over time.
Surveys of parents (summer 2009, 2010, 2011)
Parents’ personal characteristics, their experiences, aspirations, and expectations are likely to have an impact on their child’s life choices and life chances. Therefore, in order to understand the choices young people have made in relation to Diplomas, a telephone survey will be carried out with parents of Diploma and non-Diploma students in the sample consortia at each phase.
Survey of HEIs
In order to explore how Diploma implementation is meeting its goal of securing progression to higher education (HE) at all levels of achievement, the study will survey senior managers and admissions tutors in 18 HEIs at two time points in each phase of implementation, prior to the first cohort achieving level 3 awards (2008/09 and 2009/10) and after the first awards are made and Diploma applicants have progressed to higher education (2010/11 to 2012/13). These surveys will explore institutions’ admissions policies in relation to Diploma applicants, the size of anticipated Diploma intake, views on national progress of Diplomas and their experiences of having admitted Diploma-holders to undergraduate programmes.
2. Case Studies
In order to explore in detail the implementation of the Diplomas, a longitudinal programme of qualitative case studies will be undertaken in 15 consortia in each of the three phases of implementation (45 in total). Within each of the consortia selected, visits to four institutions will take place - to two schools and two colleges or training providers - giving a total of 60 institutions at each of the three phases. Interviews will be conducted with the strategic and operational staff of schools, colleges and training providers, learners and partners (employers and HEIs).
Four stages of research activity will be replicated with each of the 15 new consortia included at each phase. The four stages will be as follows.
A baseline visit prior to the introduction of each phase of implementation (spring 2008, 2009 and 2010): this will involve interviews with key stakeholders and young people. The focus will be on how prepared consortia are prior to implementation, and on the exploration of the influences on young people’s choice to take up a Diploma or not.
Year 1 case-study visit: this will take place in the first year of each phase of implementation (January to March 2009, 2010 and 2011) to explore early implementation issues and initial experiences and perceptions of the Diplomas amongst key stakeholders and Diploma students.
Year 2 case-study follow up visit (January to March 2010, 2011 and 2012): will explore the lessons learned by key stakeholders, changes in perceptions about Diplomas, and the perceived impact on young people.
Follow-up telephone survey of young people post-16 (autumn 2010, 2011 and 12): will track a sample of young people post-16 (that is, those who started a Diploma in year 10 in September 2008) to explore the Diploma journey and the impact the Diploma has had on their choice of post-16 destinations.
3. Statistical Analysis of External Datasets
In order to explore the outcomes and impact of the Diplomas on a larger scale than would be possible through surveys or qualitative data collection, datasets such as the Diploma Aggregation Service, the National Pupil Dataset, the Individual Learner Record, Higher Education Statistics Agency data, the Ofsted database, UCAS data and the National Employer Skills Survey will be analysed.
The main audience will be policy makers at DCSF, Qualifications and Curriculum Authority, the Sector Skills Development Agency, the General Teaching Council and the Local Government Association. The study will also be of interest to a wide variety of audiences such pre- and post-16 practitioners, employers, HEIs, information, advice and guidance providers, and parents.