Good error messages are critical for novice programmers. Recognizing this, the DrRacket programming environment provides a series of pedagogically-inspired language subsets with error messages customized to each subset. We apply human-factors research methods to explore the effectiveness of these messages. Unlike existing work in this area, we study messages at a finegrained level by analyzing the edits students make in response to various classes of errors. We present a rubric (which is not language specific) to evaluate student responses, apply it to a courseworth of student lab work, and describe what we have learned about using the rubric effectively. We also discuss some concrete observations on the effectiveness of these messages. best paper award
More than it had been done before, we were able to quantify the great extent with which poor error messages undermine computer programming education. Plus, we were able to provide a rubric to measure any improvements future efforts would bring, thus leaving on a up-note.