The G study revealed very good relative and absolute coefficients for all level of observers.
Given the robust nature of this type of analysis, it suggests that the ORTAS instrument could be used to evaluate crew performance of students by a single observer with little variance due to error.
With 2 observers taking ratings, the instrument has very good reliability, and this number would be acceptable for future evaluations of this sort involving interprofessional student team training.
As in this study, however, observer training on the instrument is necessary to calibrate ratings.
The advantage of the ORTAS as a formative and/or summative evaluative instrument is that it allows for both individual and overall team-based rating of performances.
Additionally, it is designed to allow self, peer, and observer evaluation, permitting the type of comparisons between different rater viewpoints as was done in this study.