E. Effects of individual attributes
We examined each of the features individually to understand the decision characteristics of attributes and the strength of relationship with usefulness. Due to space restriction we report some interesting observation, but omit detail correlation outcomes.
If a comment triggered changes within one line distance to comment-highlighted lines, it was highly likely to be useful (precision: 88% and recall: 78%). If the author did not participate in a comment-thread, it was more likely to be useful (88%) than those threads where the author did (49%). A manual investigation of comments showed that absence of the author in a comment-thread very often indicated an implicit acknowledgment by the author and a useful comment. On the other hand, author participation indicated either a useful comment with explicit acknowledgment (e.g., ‘done’, ‘fixed’, and ‘nice catch’) from the author or a not useful question / false positive, which the author responded to.
Comment-threads with only one comment or participant were more likely to be useful (88%), than those with more than one comment or participant (51%). The explanation is similar as the explanation for author participation, as participation of several engineers indicated a discussion which might or might not be useful. No discussion often indicated implicit agreement. Our keyword based classification also showed promising results. Table I shows keywords, which belonged to at least 15 comments in our oracle and are at least twice more likely to be in a particular class of comments (i.e. either useful or not useful) than the other. We found that comments