 | |||||||||||||||||||||||||||||||||||||
 | |||||||||||||||||||||||||||||||||||||
|
| | |||||||||||||||||||||||||||||||||||
 | |||||||||||||||||||||||||||||||||||||
 | |||||||||||||||||||||||||||||||||||||
An Ability-Based Assessment Program at the Medical School
Purpose
In recent years, dissatisfaction with conventional methods of certifying student competence, especially in medical education has grown. Faculty find that methods using primarily written and oral examinations assess only limited medical skills. They in sist that students be helped to develop a core of behavior, in addition to a core of knowledge, in order to solve both common and unique clinical problems.
Innovative Features
Faculty of the Medical School at the University of Wisconsin decided that it no longer sufficed to infer ability to perform from a student s knowledge base alone. Accordingly, they proceeded to build an ability-based assessment system that would assess both knowing and doing. This outcome-based project expands on work at other medical schools by incorporating the cognitive and non-cognitive abilities needed by medical school graduates.
Project staff asked the residency program directors at the University Hospital and Clinics to identify nine generic abilities that they felt a medical graduate should possess to be admitted into residency training. These abilities included appraisal, analysis, assessing own and peer performance, self-directed learning, handling stress, completing tasks, communication (including listening, speaking, writing and reading), consideration of professional ethics in decision-making, and interpersonal skills. Then, approximately 80 faculty from the basic and clinical sciences agreed upon explicit criteria to be used in judging medical abilities of increasing levels of complexity throughout the curriculum.
Thus, not only did this major medical school put aside multiple-choice testing of students in favor of the more difficult task of directly assessing their abilities, but it created measures to systematically develop, reinforce, practice and assess these abilities in multiple contexts. This innovation required faculty to develop clinical vignettes (incidents) using standardized patients (actors) that would allow student responses to be observed. These vignettes provided both new learning experiences for medical students and an opportunity to evaluate teaching that had not existed before.
Three clinical departments--internal medicine, primary care, and ob/gyn--in addition to faculty in anesthesiology, psychiatry, ophthalmology and emergency medicine worked to design the ability-based assessment exercises. The departments of Internal Medicine and Family Practice assessed students' ability to perform at the conclusion of each clinical rotation. At least 50 faculty developed the clinical vignettes, while another 100 repeatedly assessed student performance in them.
One hundred forty-three medical students were assessed each year over a three-year period. In addition, pilot studies of 106 oncology students and 26 pediatric residents assessed their written and oral communication skills.
Evaluation
Students as well as faculty deemed the project worthwhile, an integral part of teaching and learning rather than a disruption of the process. The new assessment method produced a unique performance database about student achievement that complements the knowledge acquisition data regularly collected on students.
The performance-based assessment instrument--the Objective Structured Clinical Examination ( OSCE )--required students in study groups first to complete pre-curriculum surveys and view three OSCE videos of resident/patient interview scenarios. Then, after the students finished the curriculum, they completed the post-curriculum survey and viewed the OSCE scenarios again. The control group completed the pre- and post-components without benefit of the program. The OSCE students consistently outscored the control students on the knowledge and patient interview questions in the new curriculum, but the differences between the two were not statistically significant.
Since the OSCE stresses what a student can do, it was possible to discriminate among competent and less competent students on appraisal, analysis, communication and interpersonal abilities. In the evaluation of students oral skills in interviewing patients with neoplastic diseases, rarely measured in any medical courses, inter-rater reliability among examiners was high and allowed finer differentiations in students ability levels than had been possible in the past.
The correlation between standardized patient ratings and faculty assessor ratings was found to be significant, as was the interrelation of separate abilities, e.g., between interpersonal skills and analysis skills during physical examinations. Even as relevant new competency areas were being judged for the first time, performance assessment pointed out the need for improvement--which traditional methods had failed to detect--in certain other areas.
Lessons Learned
Charting new territory can be challenging and exciting, but it can also be discouraging: ability-based assessment has not been readily adopted in programs of medical education, where persons without an M.D. find it difficult to exert influence. Most m edical educators have not yet realized that unless new pedagogies (e.g., problem-based learning) are accompanied by assessment methods focusing on new outcomes, the competencies of students will not change. Slowly, more medical schools are using them, but certainly not the majority.
To succeed in this kind of endeavor, the director of this project advises developing support groups, both within and without the institution. It may be advisable to form a consortium of medical schools to develop and implement ability-based assessment.
Despite these obstacles, the project director concluded that feedback in relation to explicit assessment criteria aids student learning. And the faculty learned that assessing student performance gives them new insights into reforming the curriculum and their teaching.
Project Continuation
The project has been incorporated into the Medical School s educational plan, with its own evaluation design to monitor effectiveness. Work remains to be done on several related issues:
- The school as a whole needs to assess the generic abilities that it expects of its graduates, rather than leaving the task up to individual departments. Performance assessment exercises need to be developed for the basic science years of medical education and for additional clinical areas.
- These generic abilities should become the primary focus of all of the other educational activities in the medical school--the content taught, the teaching methods used, and the assessment of student competence.
Available Information
In addition to a final report to FIPSE, the project has produced at least four books, three articles and several handbooks and videotapes on ability-based assessment.
The project director will respond to written inquiries:
Howard L. Stone
Medical Sciences Center
University of Wisconsin-Madison
1300 University Avenue
Madison, WI 53706
[Miami University] [Table of Contents] [Mathematical Association of America]
|
||||||||||||
|
|
|||||||||||
| |
||||||||||||
