AI Tutoring with Khanmigo in a Two-Year School Experiment (edworkingpapers.com)

A two-year study conducted in 18 Tennessee middle schools has shed light on the impact of Khan Academy's AI tutor, Khanmigo, on student learning in math. The trial, which randomly assigned students to use Khanmigo during remedial sessions, found that those who engaged with the AI tutor achieved an increase of approximately 1.3 national percentile ranks per term. This improvement, translating to about 0.06 to 0.08 standard deviations per year, is comparable to outcomes from traditional Khan Academy practices without AI support. However, the study also revealed a crucial barrier: student engagement. Although 96% of participants accessed Khanmigo at least once, most interacted infrequently and primarily responded with basic answers rather than engaging in meaningful dialogue. This suggests that the AI's potential as a tutor remains underutilized if students do not actively engage with it. For the AI/ML community, this experiment underscores the importance of not just technology deployment but also fostering a robust engagement strategy to realize the full benefits of AI in education. Enhanced usage and interaction could be the key to unlocking the transformative potential of AI tutoring systems.