AI Tutors Double Rates of Learning in Less Learning Time
Inside Harvard's new groundbreaking study
A new study from Harvard - currently still under peer review - found that when students were given access to an AI tutor designed using pedagogical principles, it not only doubled their learning gains but did so in less time than traditional methods. The results offer compelling evidence that AI, when thoughtfully implemented using strict pedagogical principles, could transform how we design, deliver and experience education.
In this week’s post, we'll explore the study's rigorous methodology, its fascinating results, and the broader implications for the future of education.
Let's dig in! 🚀
The Research Project: Inside the Study
Conducted at Harvard University in Fall 2023, this research involved 194 undergraduate students in Physical Sciences 2 (PS2), an introductory physics course. The study employed a sophisticated crossover design where each student experienced both learning conditions - AI tutoring and active classroom learning - across two different topics: surface tension and fluid flow.
The Setup
Group A - the AI Tutoring Group: Students used "PS2 Pal," powered by GPT-4 which provided:
Identical worksheet materials to the classroom
Pre-recorded video introductions
Structured question-by-question guidance
Ability to ask questions and confirm answers
Self-paced learning environment
Group B - the Active Learning Classroom Group: Students participated in:
75-minute in-person sessions
Peer group discussions
Direct instructor guidance
Course staff support
To ensure robust results, the researchers implemented multiple controls:
Two different instructors taught the lessons to control for instructor effect
Tests were designed by separate team members to prevent bias
Two versions (A/B) of tests were used to control for specific question effects
Student groups were matched for demographics and prior physics knowledge
90% of participants hadn't studied these specific topics before
The Results: Three Key Findings
Enhanced Learning Gains
The data showed significant improvements in learning outcomes:
Students using the AI tutor achieved more than twice the learning gains compared to those in the active learning classroom
Statistical analysis showed this was highly significant (z = -5.6, p < 10⁻⁸)
Effect size ranged from 0.73 to 1.3 with quantile regression
Increased Engagement and Motivation
The AI group showed higher levels of engagement than those who were taught in the classroom:
Engagement ratings: 4.1/5 (AI) vs 3.6/5 (classroom)
Motivation ratings: 3.4/5 (AI) vs 3.1/5 (classroom)
Greater Learning Efficiency
The AI approach also proved more time-efficient, with learners in this group learning more than their peer in less time:
Median AI session duration: 49 minutes (vs 60-minute classroom period)
70% of AI students completed in under 60 minutes
30% chose to spend more time for deeper understanding
Inside the AI Tutor
The success of "PS2 Pal" wasn't accidental. The system was carefully designed around established pedagogical principles principles. Specifically, the system prompt which powered the tutor was engineered to:
Keep responses brief and focused
Provide one step at a time to manage cognitive load
Encourage students to attempt solutions before giving answers
Maintain a supportive, friendly tone
Promote active learning and growth mindset
Looking Forward: Questions & Possibilities
While the results are very interesting and promising, several very important questions remain for future research.
Here are a few big questions which I noted when I read the paper:
Sample and Generalisability: The results are impressive, but the sample is very small and niche. How would these results translate beyond Harvard's high-achieving student population? How can AI tutoring adapt to different learner profiles?
Long Term Learning Quality: How effectively do these learning gains persist over time? The study measured immediate gains, but long-term retention needs investigation.
Higher-Order Thinking Skills: While the study focused on developing foundational knowledge, how might we use AI tutoring to higher order skills like critical thinking abilities, creative problem-solving and seep analytical skills? Can AI foster the kind of complex reasoning traditionally developed through human interaction?
Lack of Ambitiousness in AI-Ed Research? This research demonstrates AI's effectiveness within a very traditional educational paradigm - what educators often call the "sage on the stage" model. Whether human or AI, we're still relying on a tutor-led, knowledge transfer system where an expert:
Presents information
Guides learning
Tests recall and understanding
Provides feedback
Measures success through assessment
For me, this raises a crucial question: are we being ambitious enough in how we imagine and test the impact of AI on education, teaching & learning? Are we at risk of digitising and optimising a single, dominant model of education which is optimal only in specific contexts with specific outcomes for a very specific type of learner?
Beyond the AI Tutor Paradigm
Research has shown that several alternative instructional strategies can be equally - and sometimes significantly more - powerful and inclusive than the tutor-led instruction model. Here are just three examples:
1.Self-Organised Learning Environments (SOLE)
Students direct their own learning journey
Learning emerges from curiosity and exploration
Knowledge is constructed through discovery
The teacher becomes a facilitator and coach, rather than an instructor
How might we use AI to:
Create safe exploration environments
Provide resources on demand
Facilitate discovery without directing it
Offer guidance only when requested
2.Collaborative Problem-Solving
Learning through peer interaction and group discovery
Knowledge building through shared experience
Development of critical social and communication skills
Natural emergence of leadership and teamwork abilities
How might we use AI to:
Connect learners with shared interests
Facilitate group problem-solving
Provide real-time translation for global collaboration
Help identify and bridge knowledge gaps within teams
3.Project-Based Learning (Solo and Collaborative)
Learning through doing
Real-world application of concepts
Integration of multiple subjects and skills
Development of self-direction and initiative
Enhanced motivation through meaningful outcomes
How might we use AI to:
Generate project ideas based on learning objectives
Connect projects to real-world applications
Provide just-in-time resources and tools
Help learners reflect on and document their learning journey
All of these approaches have two tings in common:
They deliver models of education which are proven to be more inclusive - i.e. to have benefits for more learners.
They de-center (rather than re-center) the tutor.
Perhaps we need to start asking not just how AI can make 1:1 tutoring better and more scaleable, but how it might enable, enhance and scale new and alternative approaches to teaching and learning.
Conclusion
This Harvard study provides robust evidence that AI tutoring, when thoughtfully designed, can significantly enhance learning outcomes. The combination of doubled learning gains, increased engagement, and reduced time to competency suggests we're seeing just the beginning of AI's potential in education and that its potential is significant.
If this data is anything to go by, and if we - as humans - are open and willing to acting on it, it’s possible AI will have a significant and for some deeply positive impact on how we design and deliver learning experiences.
That said, as we look forward, the question shouldn’t just be, “how AI can enhance current educational methods?”, but also “how it might AI transform the very nature of learning itself?”. With continued research and careful implementation, we could be moving toward an era of education that's more effective but also more accessible than ever before.
Happy innovating!
Phil 👋
PS: If you want to keep on top of all of the latest research in AI and education, check out my two monthly newsletters, the Learning Research Digest and Learning Futures Digest.