Evaluation of the East Midlands Development Agency Science, Technology, Engineering and Mathematics (STEM) Programme
There are a number of overarching questions which it will be important for the evaluation to explore. These are detailed below.
Programme design and implementation
- Is the design of the emda STEM programme and its combination of activities likely to achieve the desired regional impacts in terms of:
- increased attainment, participation and progression within STEM subjects
- increased job-related skills and interest and involvement in STEM careers
- increased STEM literacy amongst the adult population?
- How far does the programme build on rather than duplicate existing STEM activities?
- Do projects tackle the main challenges in relation to making STEM subjects more interesting and inspiring (for example, through enquiry based methods and enrichment activities and on-going subject specific CPD for teachers)?
- Are activities of the necessary scale and do they have sufficient reach and coverage?
- How will project activities achieve the necessary critical mass to make a difference?
- How will seemingly one-off activities lead to sustained change?
Learning and good practice
- What learning and good practice is being demonstrated and is this being disseminated effectively?
- Is learning and best practice being drawn on from other regions?
Outcomes and impacts
- What are the ‘hard’ outcomes and impacts of the programme and individual projects (for example, in relation to participation and attainment within STEM subjects and progression into STEM subjects within higher education (HE) or into STEM careers)?
- Is the programme ‘closing the gap’ between different local authorities and schools within the region?
- What are the ‘soft’ outcomes and impacts of the programme and individual projects (for example, in terms of changes in perceptions, attitudes and intentions amongst pupils and changes in culture and practice amongst teachers)?
- How can we separate the impacts of emda’s STEM programme from those of other initiatives being delivered within the region, for example via the SETPOINTs, Education Business Links Consortia (EBLOCs), HEFCE, Aimhigher and individual schools and employers?
- Taking into account additionality, deadweight and multipliers, what are the net outcomes and impacts of the programme?
- What is the programme’s strategic added value?
Impact and outcomes
Disseminating learning and good practice will be central to the evaluation approach. We will facilitate learning workshops within each phase of the evaluation to allow the exchange of learning and best practice in relation to both delivery and evaluation.
The research will culminate in a detailed end of evaluation report which will set out recommendations for the on-going development of STEM provision in the region and for the longer-term evaluation of emda STEM activities.
Research design and methods
The overall aims for Phase One will be to:
- review the delivery of the programme to date
- test perceptions of STEM in the region thus gaining a baseline position
- make recommendations for the future development of the programme.
The overall aims for Phases Two and Three will be to:
- assess the progress and success of the programme as it progresses, with a particular focus on the assessment of net outcomes and impacts
- inform the on-going refinement and development of the STEM programme and STEM provision beyond the current period of emda funding.
The evaluation will utilise a mixed methods approach to appeal to the range of consultees and will be flexible in terms of using the most effective method for the target group. Data will be collected through a combination of interviews, longitudinal surveys and case study work with those involved in the programme. This is likely to include employers, young people, teachers, programme and project managers.
In addition, the evaluation will seek to draw on rather than duplicate internal evaluation data being collected at programme and project level, for example in relation to increases in attainment and in post-16/18 participation in STEM subjects and careers.
The evaluation findings will be of interest to those seeking to promote the uptake of STEM subjects/employment at all levels and in all sectors, for example primary and secondary teachers, further education (FE) and HE institutions, employers.