2) Do reviewers from the same team give more useful comments?: We hypothesize that a reviewer may give more useful comments to member of his or her own team because they are familiar with that person, their abilities, and are more invested in the quality of the code that the team ships. A team in this sense is a group of usually four to ten developers all working under the same manager or developer lead (i.e., each project in our analysis comprises many teams). We found that roughly three quarters (76%) of review comments come from reviewers on the same team as the author. Although crossteam reviewers were less frequent, we found that reviewers from different teams gave slightly more useful comments than reviewers from the same team in all the five projects. As Table III shows however, the magnitude of the difference is quite small (under 1.5% for all but one project) and is statistically significant only because of the large sample used (over one million comments in total). Based on this, we conclude that there is no noticeable difference in comment usefulness density between reviewers who are on the same team or on different teams than the author.
3) How do comment usefulness densities vary over time?: Porter et al. found in their study on software inspection that effectiveness and defect discovery rates vary over different time periods [28]. We investigated whether reviewers are becoming more efficient in making comments on the same project over time by looking at the the comment usefulnesss density of the entire project for different periods of time. We found that for four out of the five projects the density of useful comments increases over time and we suspect this can be attributed to both increased experience with the project, similar to our findings in Section VI-A1 and also refinement of the code reviewing process (more training of developers, better tracking of code review data, etc.). Figure 7 (a) shows the temporal trend for a snapshot of time for the Office project. Even though the long term trend shows an increase density of useful comments, for three out of the five projects we noticed peaks and valleys in the density of useful comments for limited periods. For example, Figure 7:(b) shows a temporary drop for the Bing project. Examining trends of usefulness density can help managers determine whether or not code review practices are improving. For example, during times where the usefulness density drops, managers can be alerted and can easily “drill-down” to the specific reviews, reviewers, changes, or components, that are contributing to the drop and can take corrective action.