By Adam Runacres, Impact and Policy Manager and Lucy Davies, Programme Manager, The Access Project
This paper is part of our NEON Summer Symposium 2024 series.
Abstract
This paper will focus on using practitioner assessment to improve mentoring and drive student outcomes in widening participation programmes. The Access Project works with school partners to deliver in-person mentoring to students from widening participation backgrounds from Year 10 to Year 13. The aim of the mentoring is to help students fulfil their ambitions of attending a top university, and Access Project mentors are embedded within schools and meet with students at fixed points to drive specific outcomes. These outcomes derive from our Theory of Change, specified in our outcomes framework and are incorporated into programme materials and mentoring training. They address specific barriers to access and risks to equality of opportunity, including but not limited to revision and study skills, exploring academic interests, the importance of super-curricular experiences, the application process, responding to offers and applying for financial support. Mentors assess students on meeting-specific outcomes on a 1 to 4 scale following every one-to-one meeting, allowing them to both reactively mentor and track student progress.
The paper will explore how this practitioner assessment generates cross-regional and cross-programme insights, contributing to programme design, internal evaluations and reporting and how it facilitates mentors’ own enhancement of their mentoring practice, ensuring the programme is bespoke for each student. It will also outline some of the methodological and practical challenges of using practitioner assessment as part of evaluation and reporting as well as share reflections from Access Project mentors about striking the balance between consistency and flexibility when mentoring young people in their own local contexts. Although practitioner assessment of student outcomes is common across education, particularly in relation to attainment, it is rarer in the context of widening participation. The paper argues not only for the power of expert mentoring but also for the importance of practitioner assessment in tailoring mentoring practice, deepening impact and providing real-time feedback for what works. We hope that this spurs a broader conversation about the use of data in WP to enhance practitioner skills alongside programme performance.
Introduction: Mentoring and widening participation
Providing mentoring to prospective applicants is a popular approach in widening participation, often encompassed within what the government’s ‘What Works’ Centre, Transforming Access and Student Outcomes (TASO), has called ‘Multi-intervention outreach and mentoring’ (MIOM) programmes. Reviews have drawn out some key findings to show both how impactful mentoring can be and what effective mentoring looks like (Robinson and Salvestrini 2020; TASO 2023a, b).
There is broader evidence from within and outside the UK that mentoring has a positive impact on attainment as well as the social, emotional and behavioural development of young people (DuBois et al. 2002, 2011; Children’s Commissioner & LKMco. 2018; EEF 2021; Gaffney et al. 2022). In the context of access to higher education, Robinson and Salvestrini (2020) conclude that mentoring programmes boost confidence and motivation, raise aspirations and “support students from low participation groups to progress to the most prestigious institutions” (25).
When it comes to evaluation, isolating and measuring the causal impact of mentoring is challenging for a few reasons.
- Mentoring is often encompassed within multi-intervention programmes, making it challenging to deduce whether any observed impact is due to the mentoring, the other components, a combination, or something else entirely.
- Mentoring is a broad intervention touching on academic, interpersonal and other types of skills and knowledges, so selecting the right outcomes to measure is a challenge.
- Unlike grades or well-being, there are no databases or large-scale studies of student outcomes that facilitates benchmarking or comparison across control groups for mentoring despite its prevalence even within widening participation.
- Finally, mentees, particularly in a school-based or youth programme, are likely to receive some form of mentoring from a variety of different people in contexts within and outside school.
These challenges decrease the likelihood that an evaluation will accurately be able to establish a counterfactual and demonstrate causal linkages for mentoring interventions. This doesn’t mean that we shouldn’t establish M&E frameworks and data collection practices for mentoring programmes at all. If anything, these challenges mean that evaluation, data, design, and delivery staff must work collaboratively to find ways for data to have importance beyond reporting impact.
The Access Project’s mentoring model
At The Access Project, mentoring is at the heart of our programmes. Our model is rooted in the power of informed, purposeful and close mentor-mentee relationships, featuring a) high levels of rapport between mentor and mentee and b) high clarity of purpose (Clutterbuck & Megginson 2004). Mentors tailor information, advice and guidance, applying their expertise in university access through a structured programme of support to help each individual student succeed.
Our theory of change articulates how one-to-one mentoring sessions aim to improve students’ knowledge of the university application process, independence to pursue their academic interests, motivation to apply and attend a top university and their academic self-efficacy. This drives applications and offers from top universities, leading to our programme outcome: “placement at a top third university”.
From the theory of change, we generate an outcomes framework which assigns specific aims to each activity. These aims inform the design of the materials delivered to students and our data collection practices.
- Aims linked to workshops drive the design of the information provided, activities used to consolidate knowledge and actions set.
- Aims linked to one-to-one meetings drive the design of the discussion guide, the objectives for the meeting and the proposed actions to complete. The one-to-one aims often mirror the workshop aims so mentors can check and consolidate any knowledge from the workshop when they meet with the student.
During the meeting, the mentor uses a discussion guide to prompt the student to discuss the relevant material from the workshop and work through the complex decisions. This can include ‘how to access a selective university’ in Autumn term of Year 12, exploring access schemes, or ‘the personal statement’ in Summer term of Year 12, exploring the statement and its various components.
By meeting the student at these junctures on the journey to university, the mentor is able to assess to what extent they are meeting the outcomes on the framework. For example, a key outcome which is repeated across the programme is ‘Can articulate an ambition to attend a top university’. This falls under ‘Motivation to attend university’. In speaking with the student, the mentor deduces to what extent the student can do this, and using a guidance sheet, mark the student on a 1 to 4 scale based on their answer. That mark is then uploaded to a central database and stored on a student record. This form of practitioner assessment is fairly unique in widening participation and creates a large amount of data on student outcomes across the entirety of the four-year mentoring programme (Y10-Y13).
This can serve a range of purpose, but here we will focus on two.
- It informs mentoring practice
- Reactive mentoring: help the student to meet the upper end of the aim during the meeting
- Proactive mentoring: preparing ahead of the next meeting or providing additional support or supplementary materials in between
- It facilitates data analysis within and across programmes
In what follows, we’ll detail some of the advantages and limitations of using practitioner assessment in these ways, reflecting on key learnings from an internal analysis of the practice and looking ahead to potential future directions
Mentoring practice and stakeholder relationships
Scoring students on specific outcomes enhances mentoring practice and informs mentoring training and quality assurance. During a one-to-one meeting with a student, a mentor may use a suggested question to gauge their progress on a particular outcome. For example, “can articulate an ambition to attend a top university” (Motivation) may be deduced by asking, “do you want to attend university and why”. The student’s answer indicates to what extent they meet this outcome, and this can facilitate reactive mentoring during the meeting to help the student more completely fulfil the outcome (i.e. achieve a higher score). Then, the mentor may remind them of material in the preceding workshop or help them to talk through their options or any other decisions they are considering. By being outcome-driven and having descriptors to benchmark student progress, mentors can reactively support students and immediately impact their journey through the intervention. Where the outcome repeats, the mentor will hopefully see that the student progresses when a similar question is asked in the following meeting.
Practitioner assessment also facilitates proactive mentoring. When uploading the scores to the database after meeting with students, mentors can note specific areas to support students between and before the next activity. This can be through resources, signposting to opportunities, providing ad hoc support, speaking with teachers or communicating with The Access Project tutors. Ahead of each meeting, mentors can also check previous scores to ensure they target the correct areas. This also applies when mentors change roles, making handovers to new mentors easier. In addition, outcomes are a key feature of mentoring quality assurance and mentoring training at The Access Project, and managers emphasise how mentors should refer to short-term outcomes as part of developing their mentoring practice.
Finally, The Access Project works in partnership with schools to deliver our interventions, and consistently and repeatedly communicating impact is an ongoing challenge. Having practitioner assessed outcomes is an advantage in this area, allowing mentors and managers to effectively communicate with teachers and school leaders about where students or cohorts are excelling and where they might need more support. There are significant limitations to this relating to the repeatability of outcomes and the consistency of assessment (outlined below), but short-term communication of student progress via practitioner assessment is a powerful tool to both improve the intervention, communicate impact and strengthen relationships with stakeholders.
Evaluation and improving outcomes
Mentor assessment of student outcomes after each meeting generates data on how students are at points on the programme. Once each mentor uploads the short-term outcome score to the central database, this automatically generates a cohort average and a score across each outcomes ‘bucket’ or group (Knowledge, Motivation, Independence and Academic Self-Efficacy) which the mentor uses to see where the individual student may require support. These groupings also facilitate comparative analysis across different groups of students which we can form based on the range of geo-demographic criteria gathered at enrolment (gender, ethnicity, parental background in HE, Free School Meals/Pupil Premium, postcode related measures, etc). The data can also be compared across schools and regions.
This short-term outcome data has been invaluable in identifying specific areas for improvement. For example, in recent years, the data has indicated that students aren’t meeting outcomes related to budgeting, student finance, goal-setting and revision. This evidence directly informs internal programme design processes through which the programme quality improves. In theory, this data can also facilitate tracking over time, especially when the outcomes are repeated across multiple meetings. The best example of this are the short-term outcomes relating to academic self-efficacy, which mentors assess after every meeting with a student and which students themselves assess at the start of each year and in an annual survey. This allows us to see whether they have improved across the four years, and recent analysis has revealed that academic self-efficacy outcomes improved steadily from Year 10 to Year 13 with a slight dip in the trend line at the start of Year 12. One possible explanation for this could be the step up to Key Stage 5 from Key Stage 4. Another key finding from recent analysis shows that students with higher academic self-efficacy have a higher probability of placing at a top university. There are other outcomes which are repeated across the programme, including ‘Can articulate an ambition to attend university’ or ‘Understands the benefits of attending university’.
However, there are some significant limitations to consider. Firstly, practitioner assessment of student outcomes is not equivalent to student assessment of their own progress on those outcomes. Therefore, is it best suited within impact or process evaluation? Secondly, practitioner assessment relies on the judgement of the practitioner during a mentoring conversation, which will never be consistent across meetings. Although each outcome is accompanied by a guidance sheet and mentors receive training, there will be variation in how they judge student responses (if they even ask the same question or the question at all). This is particularly true for assessing outcomes over time when mentors rarely stay in the same role for four years. Finally, mentors and mentees may not meet at regular intervals each term or each half-term due to capacity and competing priorities across year groups.
These limitations emphasise the need for practitioner assessment to inform both mentoring practice and evaluation. In addition, they inform the basis for ongoing reviews of the outcomes framework, mentoring training and data collection processes.
Conclusion
So where does this leave practitioner assessment of mentoring outcomes? One potential consideration has been to supplement practitioner assessment with student self-assessment on the same outcomes. This can give a clearer assessment of how students are experiencing the impact of the programme as they go through it as formative evaluation. It can also highlight discrepancies in mentor and mentee assessment and allow for quality checks and interventions. However, there is always a risk of feedback fatigue with students, so striking that balance would be key.
As with any good evaluation tool, practitioner assessment must evolve and adapt to the circumstances. At The Access Project, this is why our internal evaluations are so comprehensive, driving the quality, inclusivity, and consistency of our programmes to deliver for the young people who participate. Equally, our mentors and their practice must be cultivated and supported, and to this end our approach does go some way. Our hope is that the diversity of data use, cross-functional collaboration and sector-wide transparency continues so that we all learn together how to effectively removing barriers to accessing quality education.
References
Children’s Commissioner & LKMco. 2018. Forging futures through mentoring: A risk worth pursuing?. Available at: https://cfey.org/wp-content/uploads/2018/04/Forging-futures-through-mentoring.pdf
DuBois, D. L. et al. 2011. How effective are mentoring programs for youth? A systematic assessment of the evidence. Psychological Science in the Public Interest, 12(2), 57–91. https://doi.org/10.1177/1529100611414806
DuBois, D.L. et al. 2002. Effectiveness of Mentoring Programs for Youth: A Meta-Analytic Review. American Journal of Community Psychology, 30(2): 157-197. https://doi.org/10.1023/A:1014628810714.
EEF. 2021. Mentoring. Education Endowment Foundation. Available at: https://educationendowmentfoundation.org.uk/education-evidence/teaching-learning-toolkit/mentoring.
EEF. 2021. Social and Emotional Learning. Education Endowment Foundation. Available at: https://educationendowmentfoundation.org.uk/education-evidence/teaching-learning-toolkit/social-and-emotional-learning.
Gaffney et al. 2022. Mentoring Programmes: Toolkit Technical Report. Youth Endowment Fund. Available at: https://youthendowmentfund.org.uk/wp-content/uploads/2022/10/Mentoring-Technical-Report_Final.pdf.
Robinson, D. & Salvestrini, V. (2020). The impact of interventions for widening access to education. A review of the evidence. Education Policy Institute for the Centre for Transforming Access and Student Outcomes in Higher Education (TASO). Available at: https://epi.org.uk/wp-content/uploads/2020/03/Widening_participation-review_EPI-TASO_2020-1.pdf
TASO. 2023a. Evaluating multi-intervention outreach and mentoring programmes. TASO. Available at https://s33320.pcdn.co/wp-content/uploads/TASO-Summary-Report-%E2%80%93-Multi-intervention-outreach-and-mentoring.pdf .
TASO. 2023b. Understanding online mentoring delivered as part of multi-intervention outreach programmes. TASO. Available at: https://s33320.pcdn.co/wp-content/uploads/TASO-Report-%E2%80%93-Understanding-online-mentoring-delivered-as-part-of-multi-intervention-outreach-programmes.pdf.