Bonus New Year Post: Can ChatGPT Grade Essays Like a Professor?
Results from some rapid experimentation
One of the many research questions on my list of experiments in AI is: can ChatGPT grade & give feedback like a professor?
Here’s what I found when I did some initial rapid experimentation.
Step 1: Upload the Intel’
AI is only as smart as we make it. In order to know what great looks like, you need to tell ChatGPT what to look for.
This might be more simple than you think.
To get started, I Googled “mark scheme examples” and grabbed the one that appeared at the top of my search list - an MPhil marking criteria from the Music Dept. at the University of Cambridge.
I then asked ChatGPT if it could give me a grade based on the criteria I had provided. She was keen to oblige…
So, I went ahead and copied and pasted the mark scheme text into ChatGPT.
Step 2: Upload the Content & Ask for Feedback
Next, I grabbed an example of a music essay from Google and pasted it into ChatGPT. This is where I hit a problem: the essay was too long.
So, I uploaded a sample of the essay, including the abstract, two paragraphs and the conclusion. This was enough data to generate a response.
Whether or not this is a fair grade is for a Music Professor to decide.
What I know for sure is that:
the grade awarded maps directly back to the mark scheme I uploaded;
when asked to expand on the feedback, ChatGPT was able to make recommendations for improving the grade based on the mark scheme;
ChatGPT was able to differentiate between great, good and terrible essays based on the criteria uploaded;
ChatGPT was able to differentiate within grades using the uploaded criteria, e.g. some essays were given a B, some a B+;
ChatGPT can spot original thought through comparing what students write to what it knows already exists.
Key take away: it might be easier than we think to scale expert assessment & feedback. Have a go and see what sorts of results you get!
Happy Designing!
Phil 👋
PS: Want to learn more about all things Learning Science? Check out my website, where you’ll find free guides, subscriptions and other info.