lntra-observer and Inter-observer Agreement
Intra- and inter-observer agreements were first examined to investigate response consistency. Intra-observer agreement examines whether or not the responses of assessments agree with the repeated counterparts from the same observers, whereas inter-observer agreement investigates whether or not an observer’s responses agree with the majority, i.e., the mean scores of all observers’ responses.
In this study, the measure of root mean square (RMS) was used to determine the level of both intra- and inter-observer agreements, as given in equation 1. RMS determines how well two data sets agree with each other and is a commonly used index