Can a Single Prompt Reliably Predict Your Learners' Needs?
A practical experiment for you to try at home
Hey folks! 👋
In August, I shared my reflections on a study by Hewitt et al. (2024), which found that GPT-4 can predict human responses with a correlation coefficient of r = 0.85 (i.e. with the right level of context, GPT4 is pretty good at predicting human responses and behaviours).
And this got me thinking: can we use AI to act as a reliable proxy for our learners?
Imagine the possibilities if we could use AI to simulate learner responses during the instructional design process. What if you didn’t need to conduct endless interviews, waiting for learner feedback that sometimes never arrives? What if you could have instant conversations with virtual learners and rapidly generate reliable data to inform your design?
According to the research, this might well be possible. But just how reliable is AI in predicting learner needs and behaviours? Can it really replace the insights we get from speaking with real humans? And - critically - if we don’t know our learners, how do we know if the insights that AI generates are reliable or not?
This is one of the questions that we explore in my bootcamp partly via a hands-on activity. This week I thought I’d share the experiment with you so that you can put it to the test for yourself!
Let’s go! 🚀
AI-Powered Needs Analysis - a Prompt & Experiment
Step 1: Create a Target Persona
First, you need to create a learner persona.
Before you put your faith in AI to simulate responses from hundreds of learners, it’s a good idea to see how it performs when you’re the learner.
So, to test GPT-4’s reliability, create a persona for a learner that you know really well—you! You know your own strengths, challenges, and learning preferences better than anyone, so you’ll be able to judge whether the AI’s predictions feel accurate or not.
Here’s an example persona that I based on me & what I’d like to learn. Use this as a template to create a persona for yourself, complete with name, age and all of the other categories specified here:
Example Learner Persona Outline:
Name: User 101
Age: 45
Course: basics of Python for instructional designers
Role: Instructional Designer, Learning & Development
Learning Needs: Understand and apply the basics of Python in day-to-day work
Technical Skills: No prior experience with Python
Learning Goals: Master the basics of Python to expand understanding and use of AI in instructional design
Challenges: Limited study time due to client work and team management
Motivations: Stay ahead in the field, integrate AI into instructional design, gain technical expertise in AI
Time Commitment: Max. 5 hours total, with one hour per week (flexibility required due to job)
Step 2: Prompt GPT-4
Next, using GPT4 (e.g. ChatGPT 4o) use the following structured prompt, including your persona data, to run your needs analysis.
This prompt will guide the AI in performing a thorough needs analysis by simulating both the interviewer (instructional designer) and the learner (in this case me - User 101).
Customise the prompt below to suit your specific persona and course context.
Prompt:
You are an expert instructional designer who specialises in learner needs analysis. I will give you a list of information to generate and a learner persona. Your task is to use your expertise in learner needs analysis to conduct a deep needs analysis for the learner persona and the specified course.
You must act as both the interviewer and the respondent. Your task is to both ask and respond to a series of probing questions in order to assess:
User 101’s prior knowledge and skills in the topic
User 101’s Zone of Proximal Development (ZPD), including a list of concepts and skills that are in and out of scope, and an explanation of why they are in or out of scope
User 101’s preferred mode of delivery, including preferred instructional strategy and preference for online asynchronous, blended, or in-person
The relevant objectives for this course, given their goals, ZPD, and available learning time
You must ask probing questions. For each of the four areas, you must ask at least two follow-up questions in order to get an in-depth understanding of the learner and their needs.
Afterward, you must create a final report about the interview, detailing what you learned from the needs analysis and covering the following key takeaways:
User 101’s prior knowledge and skills in the topic, including a rationale for how you came to this conclusion
User 101’s Zone of Proximal Development (ZPD), including a list of concepts and skills that are in and out of scope, and a rationale for how you came to this conclusion
User 101’s preferred mode of delivery, including the preferred instructional strategy and learning format, with a rationale for how you came to this conclusion
The recommended learning objectives for User 101’s course, including a rationale for how you arrived at each objective and how it will impact User 101’s day-to-day job.*
Learner Persona:
Name: User 101
Age: 45
Course: basics of Python for instructional designers
Role: Instructional Designer, Learning & Development
Learning Needs: Understand and apply the basics of Python in day-to-day work
Technical Skills: No prior experience with Python
Learning Goals: Master the basics of Python to expand understanding and use of AI in instructional design
Challenges: Limited study time due to client work and team management
Motivations: Stay ahead in the field, integrate AI into instructional design, gain technical expertise in AI
Time Commitment: Max. 5 hours total, with one hour per week (flexibility required due to job)
Hit enter and see what AI produces!
Step 3: Evaluate AI’s Performance
Now that the AI has generated an needs analysis, it’s time to assess how well it performed.
To evaluate the reliability of AI’s responses, we need to analyse about four key things:
Accuracy: Did the AI accurately assess your prior knowledge and learning constraints?
Relevance: Were the AI’s suggestions for modes of delivery and learning support practical for your situation?
Scope: Did the AI identify which skills and concepts were within your ZPD and which were out of scope?
Realism: Did the AI set realistic and relevant learning objectives based on your time constraints and goals?
If the AI’s feedback matches your real experiences and goals, then it’s a good sign that GPT-4 can be a useful tool for running virtual needs analyses. If it missed key points, this may highlight areas where AI still needs to improve or where more context should be provided when using AI in needs analysis “for real”.
Download the rubric below and use it help you to score and assess the reliability of AI’s output:
Scoring Guidelines:
28-32 points (Excellent): The AI performed exceptionally well, providing detailed, accurate, and actionable insights. You can likely trust AI as a reliable tool for simulating learner needs.
21-27 points (Good): The AI provided generally accurate and useful responses but may have missed some nuances. It could be a helpful tool, but supplement with human input where necessary.
14-20 points (Fair): The AI’s responses were somewhat helpful but lacked depth and accuracy in key areas. Use with caution, and rely more on human-led insights for key decisions.
7-13 points (Poor): The AI failed to provide reliable or useful insights. It may not be suitable for your needs analysis process at this time.
Final Thoughts: Is AI Ready to Replace Human Learners in Needs Analyses?
According to the research, GPT-4 can reliably predict human responses, making it a powerful tool for simulating learner feedback. But testing it on yourself is a great way to see how well it really performs when predicting learning needs and behaviours.
If GPT-4 can reliably reflect your own thoughts and needs as a learner, it’s more likely to be reliable when you use it to simulate other learner personas. If it makes errors or misses things, it’s likely to do the same with you use it to simulate other learner personas.
But remember, AI isn’t perfect: you’ll always need to validate its insights with real learner data and your own judgment.
Give this exercise a try, and let me know how it goes! Did GPT-4 predict your learning needs accurately? You can share your thoughts in the comments or on LinkedIn.
Happy experimenting!
Phil 👋
P.S. Want to take your AI learning design skills to the next level? Check out my AI Learning Design Bootcamp for hands-on practice in integrating AI into your instructional design process!