Abstract:
Background: our ability to measure what we do and how we think is poor. Our reliance on MCQs and OSCEs may be misplaced. Our assumption that smart people do good things is unfounded. We need better ways to explore activity and reasoning.
Approach: we are currently exploring various approaches in assessment and activity metrics. Using Script Concordance Testing, Situational Judgment Testing and Signal Detection Theory to assess skills in clinical reasoning, we use OpenLabyrinth as an exam emulator. High fidelity simulation is highly valued for effective, metrics-based feedback but is too resource intensive for widespread use. Increasingly, we use OpenLabyrinth as a broad-based, educational research toolset to explore task performance, cognitive load, and operational reliability. Semi-automated item generation will help relieve stressed teachers from creating large banks of assessment items. Big data and learning analytics show great promise – the Experience API affords the ability to connect related systems: the LMS, LRS, OpenLabyrinth, WordPress for social learning.
Results: so far, much turbulence which hinders smooth progress. But we have some striking examples where simple metrics are most illuminating on topics such as preceptors’ failure to fail; maladaptive study patterns in struggling learners; and interesting patterns of workflow in routine clinical tasks. The challenge of designing and implementing computer-based learning environments is to incorporate into them teaching features: the capacity to recognize and capture relevant events from observing the learner activity, the ability to understand the learning needs and then to provide the adequate feedback in the most adapted form. This challenge is even more demanding when the knowledge at stake cannot be fully accessible with explicit representations but has to be materialised by means of simulations and shared within an apprenticeship approach. What means "relevant event", "learning needs" and "adequate feedback" are epistemological questions which cannot be answered only in technical terms. This talk will present concepts from the theory of didactical situations and a modelling approach knowledge centered which can ground the search for responses. Concrete examples from mathematics and surgery will be used to provide an intuition of the generic value of the proposed framework.
