A. Manual Classification
Based on our insights from the developer interviews, we manually classified 844 review comments from five projects across Microsoft: Azure, Bing, Exchange, Office, and Visual Studio.
We classified each of the comments into one of the two categories: Not Useful, and Useful. Although our exploratory interviews included three classifications, authors in those interviews indicated that Somewhat Useful comments still were valuable enough to improve their code. For example, as we observed disagreements between authors related to how they classify nit-picking comments (Section IV-B), we discussed with them how to rate nit-picking comments. Since nit-picking comments are useful for long-term project maintenance and often led to changes in the code, we agreed to rate them as ‘Useful’. Similar discussions happened for the other types of somewhat useful comments, thus we consider comments that would fall into the Somewhat Useful category as described in Section IV-B to be useful.