Investigating the Impact of Codio Coach

{"authors":[{"id":190942385185,"isPublished":true,"name":"Mohit Chandarana","type":"foreignid"},{"id":192024886915,"isPublished":true,"name":"Sindhu Ramachandra","type":"foreignid"},{"id":190937676652,"isPublished":true,"name":"Joshua Ball","type":"foreignid"},{"id":190937676646,"isPublished":true,"name":"Maura Lyons","type":"foreignid"},{"id":192024886917,"isPublished":true,"name":"Phillip Snalune","type":"foreignid"}],"background_color_hero":{"createdAt":1748516984940,"createdByUserId":50962823,"id":2,"isHubspotDefined":false,"label":"Secondary","labelTranslations":{},"name":"secondary","order":1,"type":"option","updatedAt":1748516984940,"updatedByUserId":50962823},"category":{"createdAt":1748850948616,"createdByUserId":50962823,"id":1,"isHubspotDefined":false,"label":"Research paper","labelTranslations":{},"name":"research_paper","order":0,"type":"option","updatedAt":1748850948616,"updatedByUserId":50962823},"content":"<h3>Abstract</h3>\n<div>Recent research has demonstrated significant advancements in the applications of Large Language Models (LLMs) in educational environments, particularly in delivering immediate, personalized student feedback. This study examines the impact of Codio Coach, a specialized AI learning assistant integrated into the Codio platform, on student engagement and performance in asynchronous MOOC-style computer science courses.. It utilizes Large Language Models (LLMs) to provide support without supplying direct answers. It consists of three modules: Summarizer, which simplifies assignment instructions; Error Explanation, which clarifies programming error messages; and Hints, which provides Socratic-style hints by posing questions or suggestions to guide students toward solutions. Analysis revealed an immediate and sustained uptake in assistant usage, with \"Explain this error\" being the most frequent interaction (56.3%), confirming engagement and highlighting student need for error comprehension support. Assignments where Coach was enabled showed improved student performance, with a 12% increase in Mean Grade and a 15% increase in Median Grade. Furthermore, an impressively low error event rate (0.12%) observed in these AI-assisted courses suggests early signs that such tools may contribute to more effective programming environments. These findings provide valuable evidence for the efficacy of tailored AI learning assistants in enhancing student engagement and performance in CS education. We recommend educators guide students in leveraging custom, context-specific assistants to improve learning and develop critical AI application skills.</div>","description":"How does a specialized AI learning assistant impact students' learning?","download_file":{"id":191995010941,"url":"https://2227229.fs1.hubspotusercontent-na1.net/hubfs/2227229/Final_Draft_Investigating_the_Impact_of_Codio_Coach.pdf","type":"file"},"hs_child_table_id":0,"hs_created_at":1751312637256,"hs_created_by_user_id":57037473,"hs_deleted_at":0,"hs_id":191988741478,"hs_initial_published_at":1751312931860,"hs_is_edited":false,"hs_name":"Investigating the Impact of Codio Coach","hs_path":"impact-of-codio-coach","hs_published_at":1751381117133,"hs_updated_at":1751381115534,"hs_updated_by_user_id":57037473,"image":{"url":"https://2227229.fs1.hubspotusercontent-na1.net/hubfs/2227229/paper%20icon.webp","altText":"paper icon","fileId":190840343051,"height":189,"type":"image","width":268},"name":"Investigating the Impact of Codio Coach: A Specialized AI Learning Assistant on Computing Student Engagement and Performance","topic":[{"createdAt":1748955554949,"createdByUserId":7438622,"id":1,"isHubspotDefined":false,"label":"AI","labelTranslations":{},"name":"ai","order":0,"type":"option","updatedAt":1748955554949,"updatedByUserId":7438622}],"url":"impact-of-codio-coach"}

Codio research / Research paper

Investigating the Impact of Codio Coach: A Specialized AI Learning Assistant on Computing Student Engagement and Performance

Mohit Chandarana

30 Jun 2025

•

2 min read

Sindhu Ramachandra

Joshua Ball

Maura Lyons

Phillip Snalune

Abstract

Recent research has demonstrated significant advancements in the applications of Large Language Models (LLMs) in educational environments, particularly in delivering immediate, personalized student feedback. This study examines the impact of Codio Coach, a specialized AI learning assistant integrated into the Codio platform, on student engagement and performance in asynchronous MOOC-style computer science courses.. It utilizes Large Language Models
(LLMs) to provide support without supplying direct answers. It consists of three modules: Summarizer, which simplifies assignment instructions; Error Explanation, which clarifies programming error messages; and Hints, which provides Socratic-style hints by posing questions or suggestions to guide students toward solutions.

Analysis revealed an immediate and sustained uptake in assistant usage, with "Explain this error" being the most frequent interaction (56.3%), confirming engagement and highlighting student need for error comprehension support. Assignments where Coach was enabled showed improved student performance, with a 12% increase in Mean Grade and a 15% increase in Median Grade. Furthermore, an impressively low error event rate (0.12%) observed in these AI-assisted courses suggests early signs that such tools may contribute to more effective programming environments.

These findings provide valuable evidence for the efficacy of tailored AI learning assistants in enhancing student engagement and performance in CS education. We recommend educators guide students in leveraging custom, context-specific assistants to improve learning and develop critical AI application skills.

Download the full paper

[Drive Innovation] in Your Learning Environment

Get a Demo Try for Free

Investigating the Impact of Codio Coach: A Specialized AI Learning Assistant on Computing Student Engagement and Performance

Abstract

Download the full paper

Related Articles