Like many students around the world, Eithne, 14, in Chorley, United Kingdom, was struggling to keep up in math at school after more than a year of COVID-19 related disruptions. In June 2021, her parents signed her up for a summer program offered by Eedi, an online math tutoring service.
“Just dealing with lockdown, she hadn’t had enough of a really good background,” said her mother, Arianna. “She missed most of the Year 7 Maths, then Year 8. So, we thought, ‘Let’s give it a go, let’s see where she needs a bit of help.’”
Newly enrolled students on Eedi are asked to take a dynamic quiz of 10 multiple choice diagnostic questions that the service uses to learn where students struggle most in math. This information allows the service to place students on a learning pathway to overcome those specific obstacles, or misconceptions.
“We ask them a question based roughly on their age group and then we say, ‘Well, what’s the next best question to ask them based on their previous answer?’” explained Iris Hulls, the head of operations at Eedi. “We learn as much about them as possible to predict either growth or comfort topics for them.”
The dynamic quiz is powered by AI developed by researchers at the Microsoft Research Lab in Cambridge, United Kingdom, who specialize in machine learning algorithms that help people make decisions.
The AI uses each answer to predict the probability the student will correctly answer each of thousands of other possible next questions and then weighs those probabilities to decide what question to ask next to pinpoint knowledge gaps.
The information gleaned from the quiz is akin to what a teacher might learn from a one-on-one conversation with a student, explained Cheng Zhang, a Microsoft principal researcher at the lab who led the development of the machine learning model that powers Eedi’s dynamic quiz.
“If the student doesn’t know 3 times 7, we may want to ask 1 plus 1,” Zhang said. “We want to adapt the quiz based on the previous answer.”
Once students’ misconceptions are identified, the Eedi platform slots students onto a learning pathway that helps them overcome their misconceptions and do better in math at school.
Eithne was slotted onto a pathway that included a review of topics covered in Year 8 and prepared her for success in Year 9, including geometry.
“It’s very good for finding your weaknesses and your strengths and being able to understand why you’re maybe not as good in this one area,” Eithne said. “You’re able to realize, ‘I’ve been doing this wrong for ages.’”
Eithne, 14, in Chorley, United Kingdom, gained confidence in math through lessons on Eedi, an online tutoring service that uses AI developed by Microsoft. Photo by Jonathan Banks.
Good questions, good data
The success of Microsoft’s next-best-question model hinges on the data used to train it, noted Zhang. In Eedi’s case, these are thousands of vetted, high-quality diagnostic questions developed specifically to help teachers identify student misconceptions about math topics.
“Our technology is just an enhancer that makes this high-quality data give more insights,” Zhang said.
Diagnostic questions are well-thought-through multiple choice questions that have one correct answer and three wrong answers, with each wrong answer designed to reveal a specific misconception.
“Maths lends itself quite well to this kind of multiple-choice assessment because more often than not there’s a right answer and these wrong answers; it’s much less subjective than some of the humanities subjects,” said Craig Barton, an Eedi co-founder and the company’s director of education.
Barton latched on to the power of diagnostic questions when, as a math teacher, he attended a training course on formative assessments and learned that well-formulated wrong answers can provide insight to why a student is struggling.
“In the past, it was always kids got things right, which is fine, or they got things wrong and then I had to start doing detective work to figure out where they were going wrong,” he said. “That’s okay if you work one-to-one, but if you’ve got 30 kids in a class, that’s potentially quite time consuming.”
Good diagnostic questions, Barton said, must be clear and unambiguous, check for one thing, be answerable in 20 seconds, link each wrong answer to a misconception and ensure that a student is unable to answer it correctly while having a key misconception.
“This notion that the kids can’t get it right whilst having a key misconception is the hardest one to factor in, but it’s probably the most important,” he said.
For example, consider the question: “Which of the following is a multiple of 6? – A: 20, B: 62, C: 24, or D: 26.”
According to Barton, on the surface this is a decent question. That’s because students could think a “multiple” means the “6” is the first number (B) or last number (D), or the student could have difficulty with their multiplication tables and select A. The correct answer is C: 24.
“But the major flaw in this question is if you don’t know the difference between a factor and a multiple, you could get this question right, whereas experience will tell us that the biggest misconception students have with multiples is they mix them up with factors,” he said.
A better question to ask, then, is, “Which of these is a multiple of 15? – A: 1, B: 5, C: 60 or D: 55.” That’s because the possible answers include factors and multiples. The correct answer is C: 60. A student who confuses factors with multiples might instead pick A: 1 or B: 5, and a student who needs work on multiplication might pick D: 55.
“When you write these things, you’ve really got to think, ‘What are all the different ways kids can go wrong and how am I going to capture those in three wrong answers?’” Barton explained.
In this diagnostic question, the correct answer is “B:4.” Students who answer “A:20” took the first step to find the mean, totaling the numbers. “C:3” represents confusion between the concepts of median and mean. “D:2” is a mix up of the concepts mode and mean.
Teacher tools to online tutor
After the workshop, Barton went home and wrote about 50 diagnostic questions and tested them out on students in his class. They worked.
Barton is also a math book author and podcaster with thousands of followers on social media. He used his influence to spread the word on diagnostic questions and collaborated with Eedi co-founder Simon Woodhead to build an online database with thousands of diagnostic questions for teachers to access for their lesson planning.
“Then I thought, ‘Wait a minute, we could do something a bit better than this,’” Barton said. “’Imagine if the kids could answer the questions online and we could capture that data and then, before you know it, we’ve got insights into specific areas where students struggle.’”
The website exploded in popularity and attracted investors as well as the attention of Hulls, who along with colleagues was exploring options to use data to scale and make the benefits of math tutoring accessible to more families. The team formed Eedi. An advisor introduced them to Zhang and her team’s research on the next-best-question algorithm, which aims to accelerate decision making by gathering and analyzing relevant personal information.
At the time, the Microsoft researchers were working on healthcare scenarios, using AI to help doctors more efficiently make decisions about what tests to order to diagnose patient ailments.
For example, if a patient walks into an emergency room with a hurt arm, the doctor will ask a series of questions leading up to an X-ray, such as “How did you hurt your arm?” and, “Can you move your fingers?” instead of, “Do you have a cold?” because the answer will reveal relevant information for this patient’s treatment. The next-best-question algorithm automates this information gathering process.
The advisor thought the model would work well with Eedi’s dataset of diagnostic questions, automating the collection of information a tutor could glean from a one-on-one conversation with a student.
“We were aware that we had collected a lot of data. We wanted to do smarter stuff with our data; we wanted to be able to predict what misconceptions students might have before they even answer questions,” said Woodhead, who is Eedi’s chief data scientist.
The Eedi team worked with the Microsoft researchers to train the model on their diagnostic questions to efficiently pinpoint where students need the most support in math.
The model works without collecting any personal identifying information from the students, Woodhead noted.
“It doesn’t need to know a name. It doesn’t need to know an email address. It’s looking at patterns,” he said.
From this information, the system can pinpoint the best lessons for students to take on Eedi. Without that guidance, students tend to rely on strategies they’re already using at school, which isn’t the right starting point for the majority of students who are looking for a private tutor, according to Hulls.
“It really helps direct the children and their families at home to know where to start,” she said.
It’s a great idea that there might be personalized learning pathways or lessons for students. Not all students learn at the same pace, or in the same way.
Cause and effect
Eedi’s internal data shows that the tutoring service resolves 95{29fe85292aceb8cf4c6c5bf484e3bcf0e26120073821381a5855b08e43d3ac09} of the student misconceptions, noted Hulls, and students who use the platform overwhelmingly express greater confidence in math.
After the summer program on the Eedi platform, Eithne entered her Year 9 a step ahead of her classmates.
“I was like, ‘I can do this,’” she said. “I can actually explain to the people around me how to do the problems.”
Eithne has continued to use the Eedi platform, often turning to the service when she gets stuck on her homework. She enjoys a rewards program that incentivizes students to keep up with their lessons and finds the platform’s explanation of misconceptions helpful in learning key concepts.
The Eedi team is now working with Microsoft researchers to implement a next-generation machine learning model that builds on the deep end-to-end causal inference algorithm to recommend personalized learning pathways for each individual student.
“People think all students need to learn the Venn diagram first and then geometry. But maybe that is not the best for every single student. Every student learns differently,” Zhang said. “Maybe for one student the order should be switched, and for another student we need to revisit some other topic.”
This next-generation algorithm is at the forefront of causal machine learning, a field of research that incorporates the notion of cause and effect, of causality, into tools that help people make decisions.
“It’s a great idea that there might be personalized learning pathways or lessons for students,” said Arianna, Eithne’s mother. “Not all students learn at the same pace or in the same way.”
Related:
John Roach writes about Microsoft research and innovation. Follow him on Twitter.
Top image: Eithne, 14, in Chorley, United Kingdom, does a lesson on the online math tutoring service Eedi. The service uses AI developed by Microsoft. Photo by Jonathan Banks.
