Student Use of AI Assistance in Software Design Tasks

Abstract

Advanced artificial intelligence (AI) code completion tools like GitHub Copilot have been found to enhance software developer satisfaction, productivity, and flow. Recent studies have focused on the performance of these tools in simple programming tasks. However, the helpfulness of these tools in more complex activities, such as software design, remains largely unexplored. This research project aims to assess the impact of GitHub Copilot on software students' learning experience, productivity, performance, and satisfaction in completing software design tasks. The research invited 77 participants from the SENG 371: Software Evolution course at the University of Victoria to complete four software design pattern exercises where each participant used GitHub Copilot for a randomly assigned half of the questions. Exercise solution submissions, a survey, and a semi-structured group discussion were employed for data collection. As the researchers are not permitted to access the data from the experiment until after the finalization of the SENG 371 course marks in Spring 2023, the pending work on the project must wait until such time. The study seeks to contribute to research into the use of advanced code completion tools and could help educators make better-informed decisions about integrating such tools into software design education.