In our previous blog post, we explained why we decided that this was the year to update our Theory of Change outcomes. We wanted them to incorporate the latest research, be more accessible and understandable for young people, and therefore link directly to our ultimate mission of helping them make confident and informed choices about their education and career pathways.
In this blog, our Impact Manager, Hannah, will walk you through how we’re measuring our new outcomes and how we’ve integrated youth voice into these decisions.
Measuring pathways knowledge and self-knowledge
In the process of reviewing our outcomes, we discovered that human capital (valuable skills and knowledge) emerged as a crucial factor for confident and informed decision making, both from external research and our own impact data. To more accurately measure this, we’ve decided to focus on two specific aspects of human capital: pathways knowledge and self-knowledge.
Pathways knowledge involves exploring various career options, personalising knowledge to connect choices with future possibilities, and gaining awareness of how the labour market works.
Self-knowledge refers to reflecting on your own strengths and gaining a clear understanding of current priorities and future aspirations.
By measuring these outcomes we’ll be able to delve deeper into the development of essential skills and knowledge that empower individuals to navigate their journey towards confident decision making.
We’re aware that our interpretations of pathways knowledge and self-knowledge might not be widely acknowledged across the sector, so we were careful to use these specific definitions when looking for ways to measure them. While interviews and focus groups help us gather valuable insights about our impact, the bulk of our data is quantitative, collected through baseline and exit surveys, so we began looking for existing tools to help us measure distance travelled.
Measurement tools
We searched for potential measurement tools, eliminating those that weren’t specific to the context we work in i.e. secondary to early career young people from less advantaged backgrounds in the UK education system. This was a key learning from our previous Theory of Change outcomes, where lots of the tools had been built to assess different ultimate outcomes or different demographics. We then presented the most relevant options to our Youth Board, impact team, cross-organisational working group, and our board-led impact subcommittee.
After much debate, we decided to use questions from the Careers and Enterprise Company’s Future Skills Questionnaire as the basis for pathways knowledge and self-knowledge.
Although the Future Skills Questionnaire was designed to be used as a single survey to assess and address the future skills of young people in the context of career development and decision making, we only selected a few questions that directly support our definitions of pathways and self-knowledge.
We made this decision as we wanted to ensure we were incorporating our own learning from our Theory of Change review. Part of the reason for the outcomes review was to address internal challenges we have with data collection, such as mentee survey response drop-off and the collection of too much data that has hindered the practical use of our findings.
Measuring confidence in career decision making
Of all our new outcomes, career decision making self-efficacy stands out as the one widely acknowledged in our sector, with established surveys to measure it. Rather than use the term ‘career decision making self-efficacy’, we’ve opted to call it ‘confidence in career decision making’ externally so it’s easy to understand and accessible for everyone we work with, including our mentors and mentees. While we considered using the Career Decision Self-Efficacy (CDSE) scale, which is already validated and widely recognised, we found that it was developed in a different context.
The scale was first introduced in the early 1980s as part of the broader field of career psychology, aiming to understand and quantify the level of confidence individuals have in their ability to make effective career-related decisions. In addition to being an older, more general metric, it was also developed internationally and for an older audience.
Given this mismatch, we opted to explore alternative options that would better suit our mentees’ needs. Our investigation led us to the Student Career Readiness Index (SCRI) scale, developed by the International Centre for Guidance Studies (ICEGS).
This scale was specifically designed to assess career readiness among secondary school students, offering a more aligned and fitting approach to measure our mentees’ career decision-making self-efficacy. Again, whilst the SCRI was designed as a comprehensive measure, we opted to include only three of its questions in our survey.
We did this because we wanted to keep the survey a manageable length, maintaining a proportional focus on our intervention. By selecting these specific SCRI questions, we strike a balance between capturing insights and preventing survey fatigue among our mentees. While this may not be seen as best practice in more academic circles, we wanted to ensure our impact management system was specific to us, accessible to mentees, and resulted in actionable findings as possible.
Engaging young voices: amplifying impact through collaboration
We wanted to integrate youth voice as much as possible in the process of renewing our Theory of Change outcomes. When researching potential outcomes, we reviewed a large volume of mentees’ qualitative data to ensure we were capturing their thoughts in a proportionate way (you can read more about this in the first blog).
We wanted to make sure we were also capturing the opinions of young people when we developed our new surveys. Although the SCRI and Future Skills Questionnaire questions have been tested extensively with young people during the development phase, we wanted to make sure the user experience of our surveys was high-quality so the people we work with could easily understand the purpose and intention of our surveys.
To gather a wide range of opinions on our survey, we reached out to our network of mentees and also shared our survey on social media platforms. We wanted to hear from as many young people as possible about their experience with the survey. To incentivise responses, we offered participants the opportunity to win a voucher.
We ended up with 20 responses from a range of ages (14 to 26) that gave us practical advice for layout, wording changes, and structure.
Offering the opportunity to provide feedback on our survey to anyone, regardless of whether they’ve been a mentor or mentee with us, allowed us to involve a wider range of participants. While incentivising young people to answer the survey resulted in over 500 invalid responses, and some laborious data cleansing, we received 20 responses that have directly influenced further development of our survey.
In summary
We hope you enjoyed learning more about our new measurement tools. We’ve had positive feedback from survey respondents and our Youth Board so far. In the next and final blog in this impact series, we’ll explain how we’ll use, analyse, and benchmark this data moving forward.
If you want to learn more about impact at Brightside, or read our latest impact report, here’s the page you need.
