C. Quality Assurance
We browsed the official website of each paper in PS to
confirm that it had stated a mandatory process of peer review
for paper acceptance. We also verified that each paper indeed
satisfied all the specified inclusion and exclusion criteria by
manually reading its full text. For instance, we found that
although ACM SIGOPS Operating System Review did not
specify a paper review process, its paper entitled “Cloud9: A
Software Testing Service” [S5] was a published version of
[8] which had been peer-reviewed earlier in a workshop with
no proceedings. Hence, we decided to include [S5] in PS. All
other papers in PS were also verified to have been peerreviewed
and met all the inclusion and exclusion criteria.