Kahlert School of Computing Ph.D. student Sara Nurollahian and Assistant Prof. Eliane Wiese have received the best paper award for the Software Engineering Education and Training Track at the International Conference on Software Engineering (ICSE 2023) for their paper titled Improving Assessment of Programming Pattern Knowledge through Code Editing and Revision that was presented at the conference earlier in May 2023.
When programmers write code, it’s not enough for a program to run, or even to work, as expected. Even if a computer is capable of parsing and properly executing the code, humans need to be able to read and understand it as well in order to maintain, test, and debug it over the course of its life. Given the complex nature of modern code, how can computer science students learn to think about, design, and write quality code that is highly technical and highly readable? Addressing this larger question is the motivation behind the research in the award-winning paper. Advised by Prof. Eliane Wiese, Nurollahian examined how student coders demonstrate understanding of their code through different activities. “ICSE is the premier conference on software engineering and it’s exciting to have Sara present her work in their education track, which reaches a broad range of Computer Science education researchers around the world,” Prof. Eliane Wiese said.
In their research, Nurollahian and Wiese used a survey to measure student understanding of two important code quality issues: avoiding repetitions and returning boolean expressions rather than literals in simple methods. These types of quality issues are similar to simplifying math expressions. For example, it is difficult to think of the two integer values for x and y for the equation 4x + 8y + 3 = 39, but it is a lot easier for this equivalent one: x + 2y = 9.
A common approach to research in this area is to measure student’ understanding by examining the code they write. However, students may write elegant code without understanding the principles behind it or write poor-quality code even if they know better. Instead of looking only at code writing, the study additionally asked students to revise their own code and edit some code written by others. For code writing, students were given the signature of a method along with a description of its desired behavior and asked to fill in the body. For editing, students were given low-quality code blocks written by others and asked to improve them. Similarly, for revising, students who wrote low-quality code in the writing task were shown their own code and asked to improve it. However, for revising, students were given a series of hints, starting with a simple prompt to revise (without indicating what was wrong or how to fix it), and ending with an example for a similar code block.
“These three tasks allowed us to examine the revising and editing performance of students who wrote low-quality code, considering that students whose code is low in quality due to knowledge gaps will not be able to perform well in revising and editing without receiving extra support, ” Nurollahian said. Students who wrote poor-quality code needed different levels of hints to revise successfully, allowing the researchers to differentiate between knowledge gaps and other factors, such as motivation. For example, over half of the students who returned boolean literals in the writing task were able to revise successfully when prompted to improve (without getting information on what to fix or how), suggesting that they simply needed more motivation. However, the majority of the students who wrote code that contained repetitions were not able to revise successfully, even after seeing hints on what to change and an example of implementing the change in a similar code block, indicating that they needed greater knowledge in this area. The editing tasks additionally showed that code writing can give an incomplete picture of student understanding. While students who wrote well initially were generally able to edit more successfully, there were exceptions in both directions. For returning boolean literals (vs. expressions), between one quarter and one third of students who wrote poorly were successful in editing someone else’s code, while a similar percentage of students who initially wrote well made unsuccessful edits. For code repetitions, fewer students overall wrote or edited successfully, but the general pattern still held.
These findings highlight the benefit of using a variety of tasks to measure student understanding of code quality. The work also underscores the importance of further research to examine the quality issues that are challenging for students. As Nurollahian explained, “The distinction between simple and complex quality issues can help instructors allocate their time and effort to the more challenging ones. Additionally, distinguishing between students who have deep conceptual gaps and those whose low-quality code is due to other reasons, such as lack of motivation, can help appropriately tailor instruction, feedback, and learning interventions.”