Additional considerations
When you perform attribute agreement studies, randomization is importance for many reasons, including avoiding bias due to memory recall.
The Automotive Industry Action Group (AIAG) guidelines suggests: “Values of kappa greater than 0.75 indicate good to excellent agreement (with a maximum kappa = 1); values less than 0.40 indicate poor agreement”. Many companies have their own guidelines for acceptable of kappa.