Rapid AI-Powered RoI Evaluation
A framework & power-prompt to analyse the likely return on investment of training *before* you build it
Hello friends!
This week, I ran a hackathon exploring the wicked problem of calculating the impact and return on on investment of workplace training.
The big question we asked was: how can we measure the likely return on investment of a training for the organisation before we invest in building it?
One of the methods we explored was “Scrap Learning” evaluation. Here’s the TLDR.
What is Scrap Learning?
Scrap learning is a term used in the context of employee training to describe the amount of learning content that will not be applied on the job. Essentially, Scrap Learning represents wasted training time and money.
Scrap Learning is now a pretty standard measure of training effectiveness and serves as a leading indicator of application and value that can be monetised, i.e. Return on Investment (RoI).
The Scrap Learning rate is calculated by asking learners what percentage of the training material they plan to apply back on the job and then subtracting that from 100%.
So, for example, if target learners indicate that they will use 60% of the training they received, the Scrap Learning rate for that training is 40% (100%-60%). If the training costs the organisation $10,000 to produce and deliver, $4,000 of that cost is considered waste.
L&D teams that consistently track Scrap Learning can use this data to:
track and increase the impact of training on employee performance;
track and improve the RoI of training by minimising waste.
Why is Scrap Learning Evaluation So Rare?
The next question we asked is: if it’s so powerful, why don’t we see more Scrap Learning evaluation happening in the workplace?
The answer is that it’s time consuming and challenging to execute well.
Those organisations who do run scrap Learning evaluation run them manually through mass learner surveying & focus group interviews. The challenge here, of course, is data gathering.
The average response rate to post-training surveys in workplace L&D can vary significantly based on the type of survey and the target population. Generally speaking, response rates range from around 32% to 52%, making robust analysis challenging.
The next question we asked, of course, is can AI help?
AI-Powered RoI Analysis
We the started to explore various AI use cases to see if it’s possible to design and run Scrap Learning evaluation without the need for time consuming and low-response-rate surveying.
After running various experiments, we agreed that with the right inputs generic AI models (specifically, we used ChatGPT4, 4o & Claude 3.5) can enable us to run rapid Scrap Learning analyses which use training outlines and learner profile data to:
reliably predict Scrap Learning scores;
reliably predict the likely RoI of a training;
produce recommendations for how to optimise the RoI of a training before it’s built.
TLDR: there’s still a lot for us to test and learn, but initial testing suggests that with the right prompting, inputs and validation (AI needs you!), generic AI tools can be a powerful partner in our quest to optimise for impact on both employee and business outcomes.
Try it for yourself! Copy and paste the prompt below and edit the parts in italics to fit your context.
Context:
You are an expert instructional designer who specialises in analysis. Your task is to evaluate a proposed training program using the scrap learning method, in order to ensure that the training content will be effectively applied by learners on the job, therefore helping to ensure impact on performance.
I will give you a description of the scrap learning method, some information about my target learners and a training outline. You will use this data to run a Scrap learning analysis and identify what % and what parts of the training content is likely and unlikely to applied by learners. You will also provide feedback and suggestions on how to improve the outline in order to ensure that everything can be applied in practice and there is therefore zero redundancy training time and resources.
Instructions:
You must use the provided learner profile and training outline to evaluate the potential for scrap learning in the proposed training program. Generate feedback and recommendations to optimise the training design based on the scrap learning method.
Your evaluation and output must include:
1. Analysis of the training content's alignment with the learners' job roles, skills, and preferences. For each, describe how well it is aligned and explain your scoring.
2. A scrap learning score and summary. For each score explain your process and scoring.
3. Identification of areas where training content might have low applicability or relevance to the defined learners.
4. Detailed suggestions for adjusting the training content, methods, and materials to reduce scrap learning and improve applicability and impact with a rationale.
5. Simulated learner feedback to illustrate how learners might perceive and apply the training. You must make a comment on both the pedagogical and practical strengths and weaknesses from the learner’s perspective.
6. For each part of the training, a predicted return on investment for the org based on the scrap learning score.
7. A revised training outline which minimises scrap learning, based on your analysis. You must include info on how each adjustment reduces scrap learning. For each part of the revised training, a predicted return on investment for the org based on the scrap learning score.
Details:
Scrap learning is a measure of training effectiveness that identifies the portion of training content that learners do not apply on the job. It represents wasted training time and resources. The Scrap Learning rate is calculated by asking learners what percentage of the course material they plan to apply back on the job and then subtracting that from 100%.
For example, if learners indicate they will use 60% of the content, the Scrap Learning rate is 40%. A lower scrap learning score indicates a higher likelihood that the training will be applied and have an impact on employee performance. The goal is to design training that is relevant, practical, and aligned with learners' needs to minimise scrap learning.Inputs:
1. Learner Profile:
- Job Roles and Responsibilities: Sales representatives responsible for managing client relationships, identifying new business opportunities, and achieving sales targets.
- Skills and Competencies: Current skills include basic sales techniques and product knowledge. Gaps include advanced negotiation skills, CRM software proficiency, and data analysis.
- Experience Levels: Range from 1-10 years of experience in sales.
- Learning Preferences: Prefer interactive modules and hands-on activities over lectures.
- Motivations and Goals: Motivated by career advancement and achieving higher sales performance.
- Barriers to Learning: Potential barriers include high workload and limited access to advanced CRM tools.
2. Training Outline:
- Training Objectives: Improve advanced negotiation skills, increase CRM software proficiency, and enhance data analysis capabilities.
- Content Outline:
1. Advanced Negotiation Techniques
2. CRM Software Training
3. Data Analysis for Sales
- Training Methods and Materials: Interactive modules, hands-on CRM software sessions, and data analysis workshops. Supplementary videos and handouts provided.
- Assessment Methods: Quizzes after each module, practical exercises with CRM software, and a final project involving data analysis.
- Support and Follow-Up: Access to mentors for questions, follow-up sessions to reinforce learning, and additional online resources for practice.
Outputs:
Table 1: Analysis
1. Analysis of the training content's alignment with the learners' job roles, skills, and preferences. For each, provide a scrap learning score and explain your scoring.
2. A scrap learning score and summary. For each score explain your process and scoring. Highlight areas where training content might have low applicability or relevance to the defined learners.
3. Simulated learner feedback on each element of the training to illustrate how learners might perceive and apply the training.
4. For each part of the training, a predicted return on investment for the org based on the scrap learning score.
Table 2: Recommendations
1. A revised training outline which minimises scrap learning, based on your analysis. You must include info on how each adjustment reduces scrap learning.
2. For each area, a detailed description of how to adjust the training content, methods, and materials to reduce scrap learning and improve applicability and impact with a rationale.
3. For each area, a column for the current and projected scrap learning scores and a hypothesis on the projected scrap learning score with a rationale.
4. For each part of the training, a predicted return on investment for the org based on the updated scrap learning score.
Table 3: 0% Scrap Learning
A table which describes what further analysis and/or training design changes would be necessary to reduce the scrap learning score to 0%. For each recommendation you make, you must provide a description and justification for your recommendation. For each part of the training, you must provide a predicted return on investment for the org based on the updated 0% scrap learning score.
All outputs must be presented in a table format.
Happy experimenting!
Phil 👋
PS: If you want to get hands on and experiment with AI supported by me, you can apply for a place on my AI-Powered Learning Design Bootcamp.