AI for Impact Evaluation
How to use AI to measure the impact & return on investment of workplace training
This week, I led a workshop with learning and development (L&D) folks at Air BnB. Among other things, we explored the question: what are the biggest, most wicked problems faced by L&D teams in 2024?
We concluded that perhaps the biggest and most fundamental problem that we face as a profession is measuring the impact of the training we design and - related to this - showing the level of return on investment in our work.
TLDR: while we typically measure learners’ reaction to learning experiences (think: surveys and happy sheets), we lack the tools and skills required to analyse, measure and report on what is ultimately the purpose of our work: impact on our learners’ capability and performance.
The big question we then explored is: how, if at all, might AI impact our ability to analyse and report on learner performance and return on investment (RoI) on training?
In this week’s blog post I share my thoughts on this question, in the form of some case studies which shows how AI might be used to measure impact and RoI.
Bonus: I’ve also included some structured prompts so you can start to experiment with AI-powered impact evaluation yourself using real or dummy data. You can download a PDF version of the prompts here.
Let’s go!
_____
Using AI to Measure Impact on Employee Performance
Case Study 1: Sales Team Training Evaluation
Problem:
ABC Corp is a global software company with a strong focus on customer-centric selling which employs 5,000 sales professionals worldwide. ABC Corp had implemented a new consultative selling training program for its global sales team to enhance objection handling, needs analysis, and closing techniques. However, the company struggled to measure the effectiveness of the program in a data-driven way.
It was challenging to gauge whether the training improved sales representatives' objection handling and closing techniques, leading to inconsistent conversion rates across regions.
Solution: ABC Corp employed an AI analysis tool to evaluate the impact of the consultative selling training by analysing pre- and post-training sales call recordings. The tool focused on key conversational markers, including objection handling, needs analysis, and closing techniques, to measure training adoption.
Theory of Change: An AI analysis tool can objectively assess the adoption of new selling techniques in customer conversations. By identifying gaps in technique implementation, the tool can provide actionable insights to refine training programs and improve conversion rates.
KPIs:
Increase in objection handling success rates.
Increase in conversion rates.
Improved consistency in selling techniques across regions.
AI Prompt > Sales Team Performance
Context: You are a skilled AI analysis tool focused on evaluating sales team performance.Instruction: I will give you transcripts from pre- and post-training sales call recordings. You will analyse the recordings to identify if and how well the employee has achieved the key performance indicators provided. Using the data you will, you will produce a report and feedback for the employee, as detailed below.
Details:
1. Analyse each call recording for objection handling, needs analysis, and closing techniques.
2. Compare pre- and post-training recordings to identify improvements or gaps.
3. Calculate changes in objection handling success rates and conversion rates.
Input:
- [Paste transcripts of call recordings from pre- and post-training periods]
- [List Key Performance Indicators, e.g. objection handling, needs analysis, closing techniques, empathy, ability to meet a resolution].
Output 1: A report summarising improvements or gaps in objection handling, needs analysis, and closing techniques, along with calculated changes in success rates and conversion rates.
Output 2: Succinct, clear and actionable feedback to give to the employee in question. The feedback must be mapped to the Key Performance Markers provided.
Case Study 2: Diversity & Inclusion Workshop Impact
Problem:
GlobalHealth is a leading healthcare services provider with over 50,000 employees. GlobalHealth implemented a comprehensive Diversity and Inclusion (D&I) training program to foster an inclusive culture.
The organisation wanted to measure how well the training influenced daily workplace interactions. Assessing whether the training effectively impacted communication and behaviour towards inclusivity was difficult with traditional evaluation methods.
Solution: GlobalHealth used an AI sentiment analysis to monitor internal communication in Slack channels and meeting transcripts. AI was used to analyse the tone and inclusivity of language across multiple channels before and after the training.
Theory of Change: AI-powered sentiment analysis can reveal shifts in employee communication towards inclusivity, providing actionable insights into the D&I training program's effectiveness and impact.
AI Prompt > Diversity & Inclusion Behaviours
Context: You are a world leading expert in sentiment analysis who specialises in evaluating levels of diversity and inclusion in large organisation.Instruction: I will give you pre- and post-training transcripts from various internal communications channels. You will analyse the transcripts and use them to identify key performance indicators for our D&I work using the rubric provided. Using the data you will, you will produce a report and a hypothesis as detailed below.
Details:
1. Analyse each transcript for tone and inclusivity of language and behaviour.
2. Compare pre- and post-training data to identify shifts in communication behaviour related to D&I.
3. Calculate changes in the use of inclusive language and positive sentiment.
Input:
- [Add meeting and Slack transcripts from pre- and post-training periods].
- [List Key Performance Indicators for D&I, e.g. inclusive tone, inclusive language, considerations of diversity].
Output: A report summarising shifts in tone and inclusivity of language, along with calculated changes in positive sentiment and the use of inclusive language. A hypothesis on the type and rate of change in levels of diversity and inclusion across the org and specific departments and teams pre- and post-training, based on the data.
Using AI to Demonstrate Return on Investment (RoI)
Case Study 1: Performance and Productivity Metrics
Problem:
LogiTech, a global logistics company, employs over 15,000 staff across 30 countries. Its operations are focused on efficient transportation and warehousing. LogiTech introduced a new process optimisation training program to enhance productivity among warehouse and transport teams.
However, assessing its impact on performance required deeper analysis. Traditional methods could not effectively measure how the new training program impacted employee productivity, leading to uncertainty around RoI.
Solution: LogiTech used an AI data analysis tool to evaluate key productivity metrics like on-time delivery rates, picking accuracy, and average order processing time before and after the training. The tool identified the training's impact on these productivity metrics and calculated the cost savings.
Theory of Change: AI can objectively measure improvements in productivity after training, offering clear insights into its RoI by comparing cost savings against training costs.
AI Prompt > Performance & Productivity
Context: You are an expert data analyst who specialises in evaluating and analysing trends in rates of employee productivity.Instruction: I will give you a set of employee metric, one gathered before a training and one gathered after. You will analyse the metrics to identify any changes in employee performance before and after the training. Using the data, you will produce a report and summary as detailed below.
Details:
1. Analyse each productivity metric for on-time delivery rates, picking accuracy, and average order processing time.
2. Compare pre- and post-training metrics to identify improvements or gaps.
3. Calculate changes in productivity and the corresponding cost savings.
4. Provide a summary of how confident you are that the training is the causal factor in changed to employee performance.Input:
- [Add productivity metrics from pre- and post-training periods].
- [List Key Performance Indicators, e.g. on-time delivery rates, picking accuracy, average order processing time.]
Output: A report summarising productivity improvements, cost savings, and Return on Investment. A summary of how confident you are that the training is the causal factor in changed to employee performance, along with a rationale for your level of confidence.
Case Study 2: Attrition & Retention Analysis
Problem: MedSys is a leading healthcare software provider with over 10,000 employees globally. The company was struggling with employee retention and attrition, so MedSys rolled out a comprehensive professional development training program for mid-career employees. The company wanted to assess whether the program improved retention rates.
Solution: MedSys used AI to compare historical retention and attrition data patterns before and after training. The tool assessed trends in voluntary turnover and calculated the cost savings from improved retention.
Theory of Change: AI can objectively reveal patterns in employee attrition, offering valuable insights into the training program's ROI by quantifying cost savings from improved retention.
AI Prompt > Attrition & Retention RoI
Context: You are an expert data analyst who specialises in assessing employee attrition and retention rates in the workplace.
Instruction: I will give you pre- and post-training attrition data. You will analyse the data to identify changes in attrition and retention rates before and after the training.Details:
1. Analyse each data set for voluntary attrition rates and hiring costs.
2. Compare pre- and post-training data to identify improvements or gaps.
3. Calculate changes in attrition rates and the corresponding cost savings.
4. Provide a summary of how confident you are that the training is the causal factor in changed to employee attrition and retention.Input:
- [Add attrition and retention data from pre- and post-training periods].
- [List Key Performance Indicators, e.g. voluntary attrition rates, hiring and onboarding costs].
Output: A report summarising changes in attrition and retention rates, cost savings, and calculated RoI. A summary of how confident you are that the training is the causal factor in changed to employee performance, along with a rationale for your level of confidence.
That’s all folks! I’d love to hear how your experiments go - feel free to give me a shout out on LinkedIn if you try this out and get some interesting results!
You can download a PDF version of the prompts here.
Happy experimenting!
Phil 👋
PS: If you want to be part of a thriving community of AI + learning design folks, join me and my amazing community over on LinkedIn.
PPS: If want to dive into the world of AI & learning with me and a community of innovators from the world of education, check out my 3 week online bootcamp.
PPPS: If want to get access to a monthly digest of the most important research and experimentation in the world of AI & education, check out my Learning Futures Digest.