Results, analysis and discussion

Item writing

80% of items were self-generated, stemming from PMS learning environments. Students chose not to scour the internet for item ideas, and found it difficult to develop questions from the Hong Kong IDEAL Database. A student typically takes a half day session to create a complete item of high enough quality to be used in a test. The time taken depends on a variety of factors such as the quality of training in item writing, the prior knowledge of the individual, the number of revisions required of the item by the review process etc.

Items were created to reflect all 5 themes of the course (Basic science, Human science, Clinical and communication skills, Public health, and Personal and professional development). Additionally each item was mapped onto a matrix of common clinical presentations and themes of applied medical knowledge.

There was a strong consensus that item writing was of huge value to their knowledge and understanding of the curriculum, it consolidated their learning on the program so far and they felt that they were being 'being paid to learn'.

A key finding of the study so far is the importance of training in item writing. For the first iteration the training needs were underestimated and the quality of the initial items was sometimes poor. Common mistakes were to create an item where the vignette was redundant- thereby testing basic factual recall rather than clinical reasoning of applied medical knowledge, the items sometimes contained technical item flaws, and the response contingent feedback was sometimes unfocussed.

The items created improved significantly after each iteration, as the training needs of the students became clearer, and the item writing training was redesigned to target the key areas of need. Students required guidance to help focus the questions on clinically relevant material and avoid less important topics. They required guidance on writing authentic vignettes and incorporating appropriate distracters. A 'good item' checklist was developed, against which to evaluate the quality of items.

A key development from the initial iterations of this project is that extensive training is required to write high quality feed-forward from each item. A lot of time is now invested in developing the students' teaching skills that are required to write high quality feed forward. Use of English has to be precise, direct and focused if it is to be 'satisfying' to the learner. Common mistakes were to stray from the key intended learning outcome, the illogical progression of ideas, and the use of jargon. The writing of high quality feed-forward is predicated on an understanding of the intended learning outcome and high quality teaching skills. It is this high level cognitive activity that we hope to explore in future iterations of the project.

Students worked together efficiently, and the task fostered an effective learning environment. Inevitably there were tensions in the team that arose from the process of collaborative creative work and the students developed and practiced their communication and team skills.

مواضيع ذات صلة

Is it sufficient to award just a mark?

Value of feedback

Analysis of responses Overall improvement in performance

Overall results

Results Initial intentions

Practical skills (subject specific)

Key/transferable skills (generic)

Cognitive/Intellectual skills (generic)

Structure of the reflective assignment Second part

Structure of the reflective assignment First part

Methodology Structure of the reflective assignment

Methodology Essay assignments