Indian Journal of Medical Sciences Medknow Publications on behalf of Indian Journal of Medical Sciences Trust ISSN: 0019-5359 EISSN: 1998-3654 Vol. 60, Num. 12, 2006, pp. 514-519

Indian Journal of Medical Sciences, Vol. 60, No. 12, December, 2006, pp. 514-519

Original Contributions

Resident psychiatrists as assessors for lectures in continued medical education in psychiatry

Melamed Yuval, Ophir Gil, Nechama Yael, Abramovitzh Ruth, Notzer Netta, Apter Alan

Lev-Hasharon Mental Health Center, Netanya and Dept. of Psychiatry, Sackler Faculty of Medicine, Tel-Aviv University, Tel-Aviv
Correspondence Address:Lev Hasharon Mental Health Center, PO Box - 90000, Netanya - 42100, ymelamed@post.tau.ac.il

Code Number: ms06076

Abstract

OBJECTIVES: We examined the quality of instruction in a continued medical education course and the correspondence between the residents and lecturers evaluations of the program.
MATERIALS AND METHODS : Resident psychiatrists and instructors completed structured evaluation forms immediately following each lecture in a psychiatry course for one academic year.
RESULTS : Residents' and lecturers' evaluations of goal achievement, but not general ratings of lecture quality correlated positively. Instructors' enthusiasm, clarity and appropriateness of subject matter and encouragement of independent thinking, but not audio-visual aids significantly correlated with resident's positive evaluations.
CONCLUSIONS : Ongoing evaluation of classroom lectures by resident psychiatrists may provide valuable feedback to instructors and impact the quality of medical education. The "classic" role of the teacher organization and enthusiasm are the most important components of quality education.

Keywords: Continued medical education, psychiatry, quality assessment

Introduction

Training resident doctors and medical students is a core medical faculty duty. Medical schools invest great effort in constant examination of educational needs and in improving teaching methods and facilities.[1],[2]

Medical schools should help their physicians improve their teaching skills[3] by improving the clinical learning environment.[4] Clinician-educators' teaching abilities should be considered when assessing academic performance for advancement.[5]

Medical students' career choice of psychiatry has declined world over in the last decade.[6],[7] Appropriate instruction may attract more students to the profession. It is equally important to include more psychiatry classes in the general medical curriculum.[8]

Clinicians are often involved in teaching medical students, lecturing peers in staff meetings or journal clubs or in presenting research findings in medical conferences or other academic settings. Although instruction is a major task for academic physicians,[9] the development of teaching skills is rarely part of medical training. Some faculty members consider teaching a burden that detracts from their primary goal of academic advancement through research.[10]

Feedback from medical learners can contribute significantly to the quality of teaching[11],[12] and medical residents, as learners are often actively involved in evaluating courses and lectures.[13] Peer review is an additional valuable tool for maintaining high standards of medical instruction.[14] The problem is generally in the method of critique, which is usually done at the end of a course of study, retrospectively and is therefore not always reliable.[15] Receiving feedback[16] and comparing evaluations by senior lecturers and resident physicians[17] may have additional benefits. Residents are better equipped than medical students, to provide feedback as learners, based on experience and owing to their broader knowledge.[18] The assessment also contributes to the growth and professional understanding of the person performing the evaluation.[19]

Our study focused on a program for Continued Medical Education for resident psychiatrists.

In our Department of Psychiatry, in cooperation with the Department of Medical Education, we examined the psychiatric residents' and instructors' evaluations of 4 hours of weekly psychiatry lectures [2 consecutive lectures] regarding the quality of the lectures for the duration of one academic year.

In this study, the authors sought to evaluate the quality of the lectures in the Continued Medical Education program for residents in psychiatry, using feedback attained through structured evaluation forms developed expressly for this survey and completed by the residents (learners) and comparing it with the feedback from the instructors.

In order to enhance reliability, evaluations were done at the end of each lecture and compared to the instructor's evaluation of his own lecture.

Materials and methods

Ethics committee approval was not required for this survey, since completion of questionnaires was voluntary, with the intent of providing actual feedback to the lecturers. The chairman of the IRB confirmed that the university does not require ethical verification of teaching evaluation protocols.

Study duration and assessments

The survey was conducted for 33 weeks, one academic year. There were 66 lectures, including 20 in neurology, which were not evaluated since the instructors were from a different department. Of the 46 psychiatry lectures, 44 were evaluated. The remaining two lectures were by the author (Y.M.), who was not included in the survey. The lectures were evaluated immediately following presentation, on an ongoing basis. Thirty four instructors participated in the program; some gave more than one lecture.

Instruments

Structured 3-part evaluation form

Part one included 7 questions regarding quality of punctuality, lesson plan, class participation, student behavior, enthusiasm of instructor and appropriateness of class material. We evaluated the use of audio-visual aids (e.g. slides, overhead projectors, video clips, power point presentations, posters) and assessed whether they were used appropriately and whether or not there were technical difficulties. Part two included 6 questions for the residents' overall evaluation of presentation using a 5 point likert scale [1=minimum, 5=maximum] and included questions on goal achievement, presentation, stimulation of independent thought and overall evaluation of the lecture. Part three, which was separate, included 3 questions for the instructors' self-evaluation of the lecture, using the same likert scale, with the same two questions regarding goal achievement and overall evaluation of the lesson and evaluation of objective factors that may influence the quality of the lesson.

The questionnaire was developed with the cooperation of the psychiatry and the medical education departments.

Senior lecturers in the faculty of medicine evaluated the validity of the questionnaire for face value.

Evaluation process

Three resident psychiatrists who participated in the course were trained to independently complete the structured evaluation forms. The study supervisor analyzed the results of the evaluations following the first five lectures, until the observed agreement reached more than 60%, [kappa>0.61].

Reliability was determined by internal consistency of the overall evaluation score [part 2]: Cronbach alpha = 0.79. One resident and the lecturer then filled out the questionnaires independently, following each lecture. Raters did not work in the departments headed by the lecturers and lecturers were blinded to the ratings of the resident physicians.

Results

Residents noted that only 52% of the lectures began on time and 75% of the classes ended as scheduled, shortening class time for about a quarter of the lessons. 68% of the lectures used audio-visual aids, 30% of which had technical difficulties [Table - 1]. Residents rated instructors "very enthusiastic" in 74% of the lectures, demonstrating good knowledge in 82% of the lectures and found subject matter appropriate to goals of the lectures 76% of the time.

Residents and instructors did not agree on the overall evaluation of the lectures. 21% of the residents vs 15% of the instructors rated lectures as "very good"; 34% of the residents and 61% of the instructors rated lectures "good", 36% of the residents and 24% of the instructors rated lectures "mediocre" and 9% of the residents gave "poor" ratings in [Table - 2].

Statistical analysis

Chi[2] was used to determine the significance of the evaluations.

Positive significant correlations were found between the residents' overall evaluation of the lecture and: (a) the enthusiasm of the lecturer (χ ²=0.5046, P < 0.05); (b) the appropriateness of the subject matter (χ ² =0.6093, P < 0.05); (c) clarity of goals (χ ²=0.6835 P < 0.05); (d) encouragement of independent thinking (χ ²=0.6072, P < 0.05).

Positive significant correlations were also found between the residents' overall evaluation of the lecture and the instructors' feeling that they achieved the goals of the lesson (χ ²=0.3557, P < 0.05).

No correlation was found between the overall evaluation of the lecture and the type of audio-visual aids used or the possible malfunction of technical aids during the lecture.

Discussion

The authors evaluated the lectures in the Continued Medical Education program for residents in psychiatry, using structured evaluation forms completed by knowledgeable resident psychiatrists (learners) and compared results with feedback from the instructors.

Evaluators were interested in the product but free of competition and academic pressures. Feedback was provided to the instructors after each lecture. The instructors also evaluated their own presentations and their self-evaluations were compared with the residents' feedback. The structured questionnaire did not allow for open evaluation or comments.

Positive correlations between the overall evaluation and the content of the lesson, the instructor's knowledge, goals of the lesson, enthusiasm of the instructor and organization of material, emphasize the importance of these components to the learner. Though there was a mutual positive effect of feedback among some components, this correlation was not attributed to a halo effect, since the scores for each of the independent items revealed notable differences.

General evaluation was not influenced by tardiness of the instructor; utilization of outdated teaching aids (the blackboard) or malfunction of audio-visual props. Though it is difficult to imagine a lecture without powerpoint or multi-media, dependency on audio-visual equipment may be disastrous if there is a technical breakdown. Basic teaching skills are thus the most valuable aids and should be cultivated.

Most lecturers expressed satisfaction with the feedback and willingly evaluated their own performance. Some remarked that they had not been aware of the points raised that would improve their lectures.

Though we found a significant correlation between the residents' overall evaluation of the lectures and the instructors' satisfaction with achievement of the goals of the lectures, no other clear correlations were found between the residents' and instructors' ratings. This discrepancy may suggest an inherent gap in perspective between the learners and instructors. Residents preparing for specialization exams may evaluate lectures based on whether or not the material required by the syllabus was adequately covered in the lecture, while self-evaluation may have gone beyond the perspective of the amount of information provided. Residents serving as reviewers of their instructors are unique because they are the target audience.[11],[15] High quality feedback is valuable and the attainment of clinical competence and professional growth can be facilitated by reliable feedback.[19]

The residents concluded that successful lectures are based largely on organization and the knowledge and enthusiasm of the instructor. Most clinical departments have specific supervision in the subjects that are taught[20] and many universities now include courses on teaching skills in the curriculum for their staff.[21] Since medical instruction is integral to the role of the senior physician, teaching skills should be included in the medical school curriculum. The uniqueness of our study is that the evaluation was performed by both the learners and the instructors, using a structured feedback mechanism designed for use in the setting of continued medical education. This method is applicable in most clinical learning environments and may enhance the quality of medical learning.

Limitations and future directions

The evaluation questionnaire included only pre-selected items that, based on the literature and personal experience, the authors felt were important in the evaluation of the quality of continued medical education lectures. The questionnaire did not allow for additional comments or suggestions which would have contributed to improving the lectures. Future surveys should allow for additional comments by the assessors.

To increase compliance with completion of evaluation forms, the pool of assessors was intentionally small. This study did not examine whether presentation of feedback to the lecturers contributed to improved performance in the same academic year. The assessment of feedback related improvement warrants further study. In future studies, increasing the pool of assessors should be considered and training the assessors in methodology for evaluating lectures may help them improve their own teaching skills. In addition, future studies should examine whether the assessors' evaluations are representative of all of the residents, how the feedback contributes to the quality of the lectures and whether the degree of concordance between the lecturer and residents improves.

References

Copyright 2006 - Indian Journal of Medical Sciences

The following images related to this document are available:

Photo images