A. Reviewer characteristics
Prior studies on software inspection found wide variation in the effectiveness of different inspectors (i.e., the person examining the code), even when they are using the same technique on the same artifact [27]. Similarly Rigby et al. suggested using experienced members or co-developers as reviewers [29]. Since those studies suggest that reviewer characteristics can have an influence on review usefulness, we studied the following three aspects of reviewer characteristics: 1) experience with the artifacts in the review, 2) experience in the organization, and 3) being in the same team as the change author. We also examined one aspect of the project the reviews belongs to: how the effectiveness of comments in an entire project changes over time.
1) Do reviewers that have prior experience with a software artifact give more useful comments?: We investigated two aspects of experience: first, experience in changing, and second experience in reviewing an artifact. We used a source code file as the level of granularity for experience. We compared the density of useful comments for developers who had previously made changes to the files in a review to the density of useful comments made by developers who had not made changes. The developers who had made prior changes to files in a change under review had a higher proportion of useful comments in four out of the five projects (all but Exchange which shows marginal increases), but we did not see a difference in effectiveness based on the number of times that a developer had worked on a file. That is, comments from developers who had changed a file ten times had the same usefulness density as from developers how had only changed a file once. In detail, experience in changing a file at least once increases the density of useful comments from 66% to 74% for Azure, from 60%