Assessing the relevance of a paper to the CST-interface is
a challenging task. To ensure impartiality, the task was
performed independently by more than one researcher in this
case study. Our chosen keywords and their variants certainly
influenced the set of articles generated for analysis. Our
focus is software testing for cloud computing. We have
extended the concept of testing to “analysis” in formulating
the keyword string. We did not include “software” in the
keyword because to our knowledge, a paper may not
mention “software” or “program” in its abstract. This
omission might have introduced ambiguity. To mitigate this
risk, we read the included papers in detail to ensure their true
relevance to software testing. We have not loosened our
criterion on the term “cloud”, because we could not find
what other popular terms have been used with a similar
meaning as “cloud” in the publication dataset. We once
considered the terms “utility” or “grid”, but they usually
refer to non-cloud articles (such as papers on the testing of
grid-based applications) that are related to some classical
topics rather than the emerging topic which we are interested
in. Finally, we only included papers that had been peerreviewed.
As such, some work like technical reports, white
papers, or patents may have been excluded. This represents a
limitation of our current case study.