Abstract One of the main goals of an applied research field such as software engineering
is the transfer and widespread use of research results in industry. To impact industry,
researchers developing technologies in academia need to provide tangible evidence of the
advantages of using them. This can be done trough step-wise validation, enabling
researchers to gradually test and evaluate technologies to finally try them in real settings
with real users and applications. The evidence obtained, together with detailed information
on how the validation was conducted, offers rich decision support material for industry
practitioners seeking to adopt new technologies and researchers looking for an empirical
basis on which to build new or refined technologies. This paper presents model for
evaluating the rigor and industrial relevance of technology evaluations in software
engineering. The model is applied and validated in a comprehensive systematic literature
review of evaluations of requirements engineering technologies published in software
engineering journals. The aim is to show the applicability of the model and to characterize
how evaluations are carried out and reported to evaluate the state-of-research. The review
shows that the model can be applied to characterize evaluations in requirements
engineering. The findings from applying the model also show that the majority of
technology evaluations in requirements engineering lack both industrial relevance and rigor.
In addition, the research field does not show any improvements in terms of industrial
relevance over time.
Keywords Systematic review. Requirements engineering . Technology evaluation