After students finished working with the ALP (maximum allowed working time was 3 h), their performances were measured with a multiple-choice test consisting of seven questions concerning the content of the programme, focusing on factual knowledge about the content (Cronbach's α=0.57). Such tests are very often used at, for instance, the Open University. A second effect measurement was based on the reflection reports: the quality of the reports was judged based on the number of correct content statements that they contained, controlling for the total number of words that students used to write the report. The content statements were based on the relevant content information available in the ALP for solving the problem. Examples of content statements were ‘bus drivers experience a high workload’ and ‘the manager is completely unsympathetic’. The reports were scored by two independent raters with and inter-rater reliability of 0.95.
Intrinsic motivation was measured with a 5-point scale consisting of 6 items (example of item: ‘Learning in this environment is fun’; Cronbach's α=0.81; n=33). Measuring intrinsic motivation with 5- or 7-point scales has often been done and many researchers report high validity and reliability (e.g. Ryan et al. 1990; Boekaerts & Minnaert 2003). Items were partly copied, translated into Dutch and adapted to the specific test situation. The six-item scale that measured the self-reported tendency towards explorative behaviour in the environment (e.g. ‘This environment incites explorative learning’; Cronbach's Alpha=.92; n=33) was newly developed.
The programme Watch allowed accurate and reliable track analysis. Due to the specific design of the learning environment, each action on the computer (mouse click) was represented by a new page in the loggings that Watch automatically created. This made it possible to count the total number of ‘pages’ or items that students visited in the electronic learning environment. Furthermore, the learning environment was sorted according to pages that were explorative and pages that were not explorative. Explorative pages were defined as pages that students were not explicitly directed to by external sources. The three steps mentioned in the previous section (interview, work floor research and report) defined the pages that were not explorative, because these steps explicitly stimulated students to visit certain pages. For example, if students (electronically) interviewed the manager of the bus company, they were not exploring, but if students looked for information in the archive of the consultancy firm, they were. By using this definition, the explorative pages in the programme could be designated, which made it possible to count the number of explorative pages a student visited. The explorative pages that were counted were visits to the senior advisor or to the secretary, and the number of pages that were read in the archive or in emails. About a quarter of the pages that could be visited by the participants was labelled as explorative. Participants were not aware of these labels. The variable proportion exploration was calculated by dividing the number of explorative pages a participant had visited by the total number of pages that this participant had visited. This variable was an indicator of the tendency towards explorative behaviour, corrected for the total number of pages visited. A high score on proportion exploration means that this participant has a relatively high tendency towards explorative behaviour.