Pusan National University Researchers Assess the Impact of Repeated Item Development Faculty Training on Item Difficulty Prediction

Repeated item development training helps faculty better predict and adjust item difficulty in medical education assessments

BUSAN, South Korea, July 2, 2024 /PRNewswire/ -- Evaluation is essential to learning, as it determines whether students have grasped the taught concepts. The item difficulty index (the ratio of correct responses to total responses) plays a key role in this process, providing a measure of test difficulty. Educators' ability to create items and predict their difficulty indices significantly impacts evaluation. Educators often overestimate item difficulty, underscoring the need for better predictive skills. However, research on faculty development programs to enhance this ability is limited. 

To address this gap, a team of researchers at Pusan National University led by Professor Sang Lee, Vice President for Medical Affairs, Professor of Medical Education, and Professor of Family Medicine, conducted a study, the findings of which were published online in BMC Medical Education on May 30, 2024. The study investigated whether repeated item development training for medical school faculty improved their ability to predict and adjust the difficulty level of multiple-choice questions (MCQs). 

Explaining the background of their study, Prof. Lee elucidates, "Just like the final stroke completes and perfects a painting, education is perfected through evaluation. Medical school faculty members cannot create high-quality items without proper training. Item development training is essential, and this study demonstrates that the effectiveness of such training increases with repetition."

Item development workshops were conducted with 62 participants, first in 2016 and later in 2018. The estimated accuracies of item difficulty predictions were compared. Before the workshop, the teachers developed newly drafted items which were then reviewed. An item development committee trained the faculty members by offering continuous feedback and helping them revise the newly drafted items according to the national exam standards, with an ideal difficulty range and an application-based focus. Furthermore, the difficulty indices predicted by the participants were compared with fourth-year medical student evaluation analyses.

The study found that before the training, significant agreement between the predicted and actual item difficulty indices was observed for only cardiology. After training, significant agreement was observed for cardiology, neurology, internal medicine, and preventative medicine. These findings suggest systematic and effective training can improve the quality of MCQ assessments in medical education.

Repeated training sessions significantly enhanced faculty members' ability to predict and adjust item difficulty levels accurately, leading to effective assessments and better educational outcomes. Despite the benefits, sustaining them might be challenging due to the workshop's three-day duration and busy faculty schedules. However, trained educators will be better equipped to create and adjust items, improving assessment practices across all academic fields. In conclusion, continuous faculty development programs are essential to ensure the creation of appropriate items aligned with evaluation purposes.

Talking about the potential applications of their study, Prof. Lee shares, "Repeated item development training not only helps adjust the difficulty level but also enhances the construction of the items, increases their discriminating power, and properly addresses the issue of validity." He further adds, "Soon there will be an era of item development using AI. For that, studies like ours are important for providing necessary information about existing items and students' answer data, which will help in developing an AI-powered automated item development program."

Reference

Title of original paper: The impact of repeated item development training on the prediction of medical faculty members' item difficulty index

Journal: BMC Medical Education

DOI: https://doi.org/10.1186/s12909-024-05577-x

About the institute
Website: https://www.pusan.ac.kr/eng/Main.do

Contact:
Jae-Eun Lee
82 51 510 7928
[email protected]

SOURCE Pusan National University