Measuring the Effectiveness of Error Messages Designed for Novice Programmers
Guillaume Marceau, Kathi Fisler, Shriram Krishnamurthi
Technical Symposium on Computer Science Education (SIGCSE)
January 2011
Abstract
Good error messages are critical for novice programmers. Recognizing this, the DrRacket programming environment provides a series of pedagogically-inspired language subsets with error messages customized to each subset. We apply human-factors research methods to explore the effectiveness of these messages. Unlike existing work in this area, we study messages at a fine-grained level by analyzing the edits students make in response to various classes of errors. We present a rubric (which is not language specific) to evaluate student responses, apply it to a course worth of student lab work, and describe what we have learned about using the rubric effectively. We also discuss some concrete observations on the effectiveness of these messages. *Best paper award.*
More than it had been done before, we were able to quantify the great extent with which poor error messages undermine computer programming education. Plus, we were able to provide a rubric to measure any improvements future efforts would bring, thus leaving on a up-note.