A. Developer Interviews
We chose interview research because interviewing is a frequently used exploratory technique used to gain insight into the opinions and thoughts of the participants – which cannot be obtained by quantitative measures, such as data mining [14]. We conducted semi-structured individual interviews with seven developers, selected from four different Microsoft projects based on their varying level of development / code-review experiences. Semi-structured interviews allowed us to combine open-ended questions to elicit information on their perception on code reviews and comments with specific questions revealing the perceived usefulness of individual code review comments. Each interview lasted 30 minutes and was structured into three phases. First (≈ 5 minutes), we asked the interviewee about their job role, their experience, and what role comments play during code reviewing. In the second phase (≈ 20 minutes), we showed the interviewee 20 ∼ 25 randomly selected review comments from recently submitted code reviews for which the interviewee was the author. We asked the interviewee to rate the usefulness of each comment and categorize it based on a classification scheme adopted from a prior study to identify the types of issues detected during code reviews [15] (see 3 for the list of categories). We used the ratings from the author because they are the only ones with the “ground truth” of the usefulness of the comment (i.e. the author is the best judge of whether the comment was helpful to him or her and/or the change). For each of these comments, we asked the interviewees to