Journal of Postgraduate Medicine Medknow Publications and Staff Society of Seth GS Medical College and KEM Hospital, Mumbai, India ISSN: 0022-3859 EISSN: 0972-2823 Vol. 56, Num. 4, 2010, pp. 297-300

Journal of Postgraduate Medicine, Vol. 56, No. 4, October-December, 2010, pp. 297-300

Education Forum

Evaluation of clinical skills for first-year surgical residents using orientation programme and objective structured clinical evaluation as a tool of assessment

JS Pandya, SM Bhagwat, SL Kini

Department of General Surgery, T N Medical College and B Y L Nair Ch. Hospital, Mumbai, India

Correspondence Address:J S Pandya, Department of General Surgery, T N Medical College and B Y L Nair Ch. Hospital, Mumbai, India, smruti63@hotmail.com

Date of Submission: 26-Apr-2010
Date of Decision: 22-May-2010
Date of Acceptance: 01-Jul-2010

Code Number: jp10087

PMID: 20935404

DOI: 10.4103/0022-3859.70950

Abstract

Background: Postgraduate specialities require a combination of knowledge and clinical skills. The internship year is less structured. Clinical and practical skills that are picked up during training are not well regulated and the impact is not assessed. In this study, we assessed knowledge and skills using objective structured clinical examination (OSCE).
Aim: To evaluate the clinical skills of new first-year surgical residents using orientation programme and OSCE as a tool for assessment.
Settings and Design: Observational study.
Materials and Methods: Twenty new first-year surgical residents (10 each in 2008 and 2009) participated in a detailed structured orientation programme conducted over a period of 7 days. Clinically important topics and skills expected at this level (e.g., suturing, wound care etc.) were covered. The programme was preceded by an OSCE to test pre-programme knowledge (the "pre-test"). The questions were validated by senior department staff. A post-programme OSCE (the "post-test") helped to evaluate the change in clinical skill level brought about by the orientation programme.
Statistical Analysis: Wilcoxson matched-pairs signed-ranks test.
Results: Passing performance was achieved by all participants in both pre- and post-tests. Following the orientation programme, significant improvement was seen in tasks testing the psychomotor and cognitive domains. (P = 0.0001 and P = 0.0401, respectively). Overall reliability of the OSCE was found to be 0.7026 (Cronbach's coefficient alpha).
Conclusions: This study highlighted the lacunae in current internship training, especially for skill-based tasks. There is a need for universal inclusion of structured orientation programmes in the training of first-year residents. OSCE is a reliable, valid and effective method for the assessment of clinical skills.

Keywords: Clinical skills assessment, objective structured clinical evaluation, residents′ training programme

Introduction

All postgraduate specialities in the field of medicine, especially those in surgery, require a combination of knowledge and clinical skills. Existing methods of evaluation like common entrance tests (CETs) at the postgraduate and subspecialty levels test only theoretical knowledge. Formal medical school teaching adheres to the Medical Council of India (MCI) guidelines, but the internship year is less structured. Inconsistencies in quality of training during internship ensure that incoming surgical trainees frequently are underconfident and poorly prepared to assume the different responsibilities that comprise patient care. ^[1] Clinical and practical skills are supposed to be picked up "on-the-job." While many institutions provide entry-level orientation programmes, these are neither well regulated nor is their impact always objectively assessed. The traditional clinical examination has been shown to have serious limitations in terms of its validity and reliability. ^[2] In this study, we attempted to assess and compare the practical knowledge and skills of entering first-year surgical residents before and after an orientation programme (the teaching-learning tool), using Objective Structured Clinical Examination (OSCE) as the assessment tool .The OSCE was designed and validated by senior surgical faculty. In a similar study from the UK, face and content validity were assessed by a panel of clinicians and from feedback from OSCE participants. ^[3]

Materials and Methods

Approval from the Hospital Ethics Committee was sought and obtained before carrying out this observational study and written informed consent was obtained from the participants.

The participants were 20 new first-year surgical residents registered for MS degree with Maharashtra University of Health Sciences. Two consecutive batches of 10 residents each entering in May 2008 and May 2009 were included. They received a pre-orientation assessment OSCE (the "pre-test") to judge baseline competence followed by a lecture and demonstration-based orientation programme, which was then followed by a post-orientation assessment OSCE (the "post-test") to assess the impact of the programme. The entire activity spanned 10 days and was held early during the first month of their first post.

The OSCE was designed by senior surgical faculty with valuable input from senior residents as to what exactly they expected of their new juniors. The topics chosen were representative of the type and level of clinical and practical skills required by a first-year resident. These included demonstration of clinical signs, performance of ward procedures, radiological identification and common theory topics. Case-note writing skills, basic administrative duties, consent-taking and breaking bad news were also tested. Before administration of the tests, the stations were demonstrated to senior surgical faculty and validated by them. Both pre-and post-test OSCE consisted of 10 stations each [Table - 1] and [Table - 2]. All three domains, i.e. cognitive, psychomotor and affective, were tested separately. A combination of actual patients, simple models for simulation, case scenarios, brief viva voce and questions needing written answers was used. Each station was scored out of five marks using preset checklists. Each point on the checklist was scored according to a binary, i.e. "yes/no," scale and marks were given accordingly. Surgical staff and residents acted as observers to score the stations. Candidates were required to score 25 out of a total of 50 marks (50%) to pass.

After the pre-test, a lecture and demonstration-based orientation programme designed by senior faculty was conducted by them over 7 days. Important clinical, practical and administrative "must-know" topics [Table - 3] were taught in an informal small-group setting with plenty of opportunities for hands-on practice by the residents.

The post-test was held immediately on completion of the programme. Feedback questionnaires from the participants were completed after the orientation programme and after each assessment.

The results were tabulated and analyzed using standard statistical software (SPSS, v. 17).

Results

The results of both batches of new residents were considered together as one batch of 20 to facilitate analysis.

Passing performance (50%) was achieved by all 20 residents in both pre-test and post-test.

Analysis of both pre- and post-test results showed that the new residents fared better overall on questions testing the cognitive domain and less well on those testing the psychomotor and affective domains. This finding was expected and was attributed to both a lack of sufficient practical experience as well as a lack of useful training during the internship year.

Pre- and post-test results were then grouped into three groups [Table - 4],[Table - 5],[Table - 6]: cognitive domain station scores (four stations, 20 marks, "knowledge"), psychomotor domain station scores (five stations, 25 marks, "skills") and affective domain station scores (one station, 5 marks, "emotional content"). Pre- and post-test scores for each group were compared to determine whether the post-test results were better and, if so, whether the difference was of significance. The participants′ feedback was also scrutinized to get an idea of the impact of the orientation programme, the response to the use of OSCE as an assessment tool and to gather suggestions for improvement.

In the post-test, overall performance by all participants improved. The Wilcoxson matched-pairs signed-ranks test was found to be appropriate for analyzing the data. The difference between the pre-and post-test scores for the stations testing the psychomotor domain was found to be significant (two-tailed, P = 0.0001). The difference between pre-and post-test scores for the stations testing the cognitive domain was also found to be significant (two-tailed, P= 0.0401). However, there was only minimal improvement in the post-test scores for the affective domain station, and the difference between the pre- and post-test scores was not significant (P = 0.8203).

Overall reliability of the OSCE was found to be 0.7026 (Cronbach′s coefficient alpha), showing good reliability.

Data gathered from the feedback forms indicated that the residents felt more informed and empowered after the orientation programme. The OSCE format of assessment was appreciated as being comprehensive and fair.

Discussion

Orientation programmes for incoming residents are offered in many forms: as a set of lectures in one day, as provision of "resident′s manuals", as practical demonstrations, as computer simulations or as simulated clinical "calls." ^[4],[5] In some countries, formal courses like Basic Life Support, Advanced Cardiac Life Support and Basic Surgical Skills courses are mandatory for new residents. ^[4] Surgical residents in India complete their course in 3 years. Most of their time is spent on patient care and there is little left for formal training. Postgraduate departments have to constantly come up with innovative ways to impart knowledge-based confidence to the new juniors. In this study, the orientation programme was administered in a lecture-demonstration format suitable to the department′s resources.

The results of this small study indicated that provision of basic clinical and practical instruction at the beginning of a surgical residency resulted in an immediate improvement in performance. To that end, the orientation programme was successful as a teaching-learning tool. Maintaining development and improvement of skills would however require such programmes to be held on an ongoing basis in addition to the formal postgraduate curriculum.

The results of the pre-test highlighted the lacunae in internship training. Questions answered consistently incorrectly served to point out common misconceptions about ward work, e.g. the use of undiluted hydrogen peroxide solution on healthy wounds.

Similarly, consistent mistakes in the post-test indicated which procedures needed close supervision and which residents needed extra help.

There were some limitations however. Although a statistically significant difference was shown in the means of nine of the 10 stations tested, it would still be incorrect to attribute the improvement in results entirely to the orientation programme. The sample size was small, the residents might have understood the OSCE format better the second time around and might have worked on their skills during the orientation week as deficiencies had been pointed out in the pre-test. In order to remove these biases, our follow-up study would include larger numbers, multiple batches, longer and more periodic testing and comparison between senior and junior residents. In particular, more stations would be included to test the affective domain (communication skills and professionalism).

The OSCE format is fast becoming the accepted method for clinical assessment all over the world. ^[2],[6],[7] It is proven to be a reliable, valid and effective assessment tool due to its clear advantages of objectivity, fairness and reproducibility. This was our experience as well. In this study, the residents participated in both pre- and post-test OSCEs with enthusiasm and viewed the stations as learning opportunities. They felt that they had been examined fairly and that many topics had been tested in a short time. The most cited positive feedback was that the entire programme addressed the juniors′ common problem areas in a friendly yet objective way; thus, increasing their confidence. The most cited suggestion was that more such programmes held more frequently and covering more topics should be held.

A more proactive stance by postgraduate teachers in India toward the initial and ongoing formal training of surgical residents is suggested. Creative use of methods like problem-based learning and moulages (dynamic case scenarios) can supplement more traditional seminars and case presentations. "Mentoring" of the new residents by senior residents and/or faculty members like the postgraduate guide could be an informal adjunct to training. Regular orientation programmes and yearly evaluation using OSCE are also suggested.

Conclusions

From this study, we conclude that structured orientation programmes are desired by and objectively benefit new residents. Evaluation of new residents′ clinical skills can be successfully performed using OSCE. The OSCE format of assessment was found to be reliable, valid and effective. It can be easily adapted as a teaching tool and assessment tool. Interesting and creative ways to encourage learning by residents need not be resource-dependent. All contribute toward the ultimate goal of improved patient care.

References

Copyright 2010 - Journal of Postgraduate Medicine

The following images related to this document are available:

Photo images