Ranking of participants based on average performances in each test problem showed that participants with 3 or more years of experience working or doing research in desalination were consistently ranked the highest, while partici- pants with 1e2 years of experience were ranked similar/slightly lower compared to participants with no knowledge at all.
By analyzing the data collected based on participants’ responses, it was found that several behavioral characteristics were correlated to participants’ perfor- mance in the experimental design tasks. One of these behavioral characteristics is a common phenomenon of participants consistently going away from the target design. This phenomenon was observed when a participant failed to notice the coupling effects between two input parameters and changed them in the incorrect order or direction, making the design worse, and resulting in many more iterations needed to solve the problem. Participants with strong domain knowledge were generally able to avoid this behavior compared to participants with some knowledge or no knowledge. Debriefing interviews suggested participants with some domain knowledge often rely on incorrect or partial understanding of the technical principles.