 |
| About Bioline | All Journals | Testimonials | Membership | News |
|
||||||
|
||||||
Indian Journal of Medical Sciences, Vol. 62, No. 7, July, 2008, pp. 267-274 ORIGINAL CONTRIBUTION The internal consistency of medical students' scores in their physiopathology and clinical courses Esmaeili Aryan, Haghdoost Ali-Akbar Dept. Educational Development Center, Kerman University of Medical Sciences, Jomhoori Islami Blvd, 7618747653, Kerman Code Number: ms08048 Abstract Background: To quantify the internal consistency of medical students' scores.Aim: We assessed the associations between medical students' scores in physiopathology and clinical courses and compared these scores with their scores in their comprehensive exams. Settings and Design: We collected medical students' scores in their courses and also in their comprehensive exam in six consecutive years. Materials and Methods: We assessed the associations between students' scores and their personal characteristics, and the consistency between theoretical and practical courses. Statistical Analysis: We used Pearson correlation coefficient and linear regression. In addition, we computed difficulty and discrimination indices of students' scores in their courses by comparing these scores with comprehensive clinical exam (CCE). Results: Generally, females and younger students were more successful. CCE were predicted by students' scores and their characteristics relatively accurate (the adjusted R2 of the model was 0.59). Students' scores in the pathology and in thesis had the maximum and minimum discrimination indices, while the difficulties of these two courses were in reverse order. The strongest association was observed between theoretical and practical scores in internal medicine while the associations between theoretical and practical scores in the other courses were not strong although all of them were statistically significant. Conclusions: Using this approach to explore the students' score, might highlight the weak points of the current educational system. For example we found that the students' score in thesis had the minimum accuracy; although students obtained very high score in this course. Hence, for better comparison of the accuracy students' scores in colleges around the world, we recommend similar quantitative approach Keywords: Difficulty index, discrimination index, Iran, medical education, validity Introduction In Iran, medical students study basic science in five semesters and participate in a comprehensive exam (Basic Science Comprehensive Exam: BSCE). After that, they study preclinical courses on physiopathology of main body systems and also pharmacology, pathology for three semesters. In the next step, they start theoretical and practical clinical courses for two years. A successful completion of these periods makes students eligible to participate in another national comprehensive clinical exam (CCE). In this exam, students are evaluated for all of pre-clinical and clinical courses. An acceptable internal consistency between students′ scores in preclinical and clinical courses and also significant associations between scores in these courses and in comprehensive exams may indicate the validity of exams indirectly. Particularly, this method of validity assessment is more appropriate in course-based educational curriculums [1] such as the model of medical curriculum in Iran. There are a great deal of studies which assessed the relationship of some of variables such as students′ scores in high school, [2],[3] premedical summer programs [4] and admission tests [5],[6] and even their personal characteristics with students′ scores in their courses. [2],[3],[7] Most of these studies used those scores and characteristics as predictors of students′ achievement. Nonetheless, this paper mainly explored the validity of students′ scores in their courses using backward approach by comparing students′ scores in CCE with their scores in their courses, using the concepts of difficulty and discrimination indices. This approach is not a common approach and we believe that the concepts of difficulty and discrimination indices could be applied to check the validity of whole exams. According to the above explanation, we quantified the internal consistency of medical students′ scores in the physiopathology and clinical courses to assess the internal validity of their scores. In addition, we computed difficulty and discrimination indices of students′ scores in their courses by comparing these scores with CCE score. Additionally, we assessed the age and gender effects on the academic achievement. Materials and Methods Medical students in Kerman University of Medical Sciences (KUMS) were classified into separate cohorts based on the entry year between 1995 and 2000. Then, their physiopathology and clinical course scores were obtained from the registry of KUMS in paper forms. These forms also contained the students′ BSCE and CCE scores, sex and date of birth. However, due to legal restrictions, the forms were anonymous and we could not link their data to other personal records. The data were double entered and the validity of the data entry process was assessed. Six academic achievement indicators (AAIs) were computed as follows:
The scoring system in KUMS is on a scale of 0 to 20; however, the comprehensive exams are scored on a scale of 200 points. For easier comparison, BSCE and CCE scores were converted to one on a scale of 20 points. The associations between the AAIs and also between AAIs and the students′ scores in their courses were assessed by computing Pearson correlation coefficients. In addition, 27% of students with the top and lowest scores in the CCE were labeled successful and unsuccessful groups; then the discrimination and difficulty indices of all courses were computed using the Whitney and Sabers formula for essay tests. [8] The computed difficulty index implies how difficult the course was for students, while the computed discrimination index quantifies the accuracy of the students′ scores in a course in discriminating the top and lowest groups. The analysis was done using the SPSS software version 11.5; the significant level was 0.05. Results From 1995 to 2000, 481 medical students started their studies at KUMS (39.7% male). The minimum and maximum annual number of enrolled students was 45 (in 2000) and 99 (in 1997 and 1998), respectively. Females were more successful in their studies based in all AAIs ( P < 0.001), except in CCE and BSCE scores. In CCE and BSCE, males′ scores were slightly greater than females′ scores, but the differences were not statistically significant [Table - 1]. Negative associations were observed between the entrance age and the academic achievement [Table - 1]. Students were classified based on their entrance age into three groups: 1) under 19 years of age; most of who successfully started their academic studies right after high school, 2) 19 and 20 years of age; who started their academic studies with a one or two year gap, and 3) over 21 years of age. The trend of all achievement indicators showed that the success rate decreased with age ( P < 0.001). The greatest correlation coefficient observed between age and BSCE (r=-0.2). There were strong correlations between students′ scores in all of the AAIs in both genders. The strongest association was observed between the students′ scores in the physio-pathology and clinical courses (in males: r=0.811, in female: r=0.802) [Table - 2]. Enrolling all possible predictors of students′ scores in CCE, a linear regression model was generated [Table - 3]. The adjusted R 2 of the model was 0.59 which implied an acceptable accuracy. Adjusting for the effects of the other variables, age and students′ scores in their theoretical courses were not significant. While, the adjusted mean difference between males′ and females′ score was -0.522 ( P -value< 0.001). In addition, the results showed that by one unit increase in student scores in BSCE, physio-pathology and practical courses, their score in CCE was increased 0.36, 0.25, and 0.46 units respectively. In the next step, we compared the students′ scores in theoretical and practical exams, classified by courses [Figure - 1]. These results showed a wide variation between coefficients, the strongest correlation coefficient was observed in internal medicine (r=0.65) and the weakest coefficient in neurology (r=0.24). Nonetheless, all of these coefficients were highly significant ( P -values < 0.0001). Based on the results of [Table - 4], students got the greatest and the lowest scores in thesis (19.01) and in pathology of diseases (14.28) respectively. In fact, the score of students in thesis were considerably greater than the other courses with the minimum standard deviation (SD=0.86) which means that students′ scores in thesis were much closer than their scores in the other courses. In contrast, the mean scores of students in BSCE and CCE were much lower that their scores in the other courses (12.88 and 11.72 respectively). On the other hand, based on the computed discrimination indices, thesis score had the minimum discrimination index (0.17, P -value=0.32); in the other words, the mean difference of top and weak students′ scores in thesis was only 0.17. While, the maximum discrimination index was observed for the pathology of diseases (2.95, P -value = < 0.001). After that, the mean score in all physio-pathology courses, in internal medicine and in pharmacology had the maximum discrimination indices [Table - 4]. Discussion The results showed that the associations between AAIs were relatively strong. Nonetheless, the consistency of students′ scores in theoretical and practical courses in some subjects such as neurology, cardiology and infection diseases were much less than that in internal medicine. Generally, students′ scores in thesis were much greater than the other courses, but it had the minimum discrimination index. In contrast, although students got the lowest scores in the pathology of diseases, it had the maximum discrimination index which implies that the score of this course could discriminate top and lowest group of students much better than other scores. Nonetheless, in all of the courses, the discrimination indices were not considerable. Moreover, the results of multivariate analysis showed that the students′ scores in their theoretical courses did not predict their scores in CCE. Generally, younger students and females were more successful. There was a strong negative association between entrance age and AAIs, which has been reported in many studies. [7] In Iran, female students, particularly single ones, have fewer responsibilities in the family and they are mostly dependent on financial support from their families. In addition, they socialize less, and therefore have much more time to dedicate to their studies. Although these factors are culture dependent, there is evidence that shows females were more successful in some other countries as well. [7] It should be added that male students were slightly more successful in the comprehensive exams, which may imply that their long term achievement is at least in the same level as females. We applied the concept of the discrimination index commonly used in the analyses of question appropriateness to assess the appropriateness of exams. The discrimination index is an indicator that shows how perfectly a question can discriminate successful and unsuccessful respondents. For this purpose, you define successful and unsuccessful respondents based on their scores in an exam; then, you check the proportion of successful and unsuccessful respondents who provide correct responses to every question. The discrimination index for each question is the difference between proportions of correct responses in successful and unsuccessful respondents. With an exactly similar logic, we defined successful and unsuccessful students based on their scores in the CCE, and compared their scores in courses. Based on the above logic, we can imply than the students′ scores in their thesis had the minimum power to discriminate successful and unsuccessful students. Surprisingly students got the best scores in their thesis. Therefore, we can imply that the scores of thesis had the minimum accuracy. On the other hand, the students′ scores in comprehensive exams were considerably lower than their scores in their courses which might partly be explained because CCE is a national exam and its standards are different. Although it is reported in other studies as well, [9],[10] we may think more about the validity of local exams. It is one of the basic concepts in exams that the questions should focus on the topics that students must learn based on the teaching curriculum. [11],[12] Nonetheless, it is not hard to believe that examiner will focus on those topics that were taught if they had played the role of teacher as well. Therefore, in the best scenario we can suggest independent professionals evaluate students based on their course plan. The correlations between students′ scores in practical and theoretical courses were not as strong as we might expect. On an average, the correlation coefficients were around 0.4. These low associations also imply that there were some problems in either the teaching methods or in exams. The strongest association was found between students′ scores in theoretical and practical exams of internal medicine. In addition, the discrimination index of internal medicine was among the best ones. These findings may show an acceptable validity of the exams in internal medicine courses. Nonetheless, we may remember that internal medicine is the most important course for medical students [13] and students pay more attention to its contents and study internal medicine much deeper than the other courses. This study only reviewed the internal consistency of medical students′ scores only in Kerman University of Medical Sciences. The pedagogic shift from traditional approach to a need-based approach requires a fundamental change of the roles and commitments of educators, planners and policymakers. [14] We could not find similar analysis on the scores of students in other colleges to compare our findings. Therefore, we encourage researchers around the world to explore students′ scores with similar methodology. For sure, comparison between the internal consistencies of students′ scores in different colleges can expand our knowledge about the effects of different teaching curriculum on the learning of students. Conclusion We found that the internal consistencies between students′ scores in their courses were not generally strong and in a few courses such as thesis, the validity of their scores was not acceptable. The weak associations between students′ scores in practical and theoretical courses was only acceptable in internal medicine course and it seems that other departments in Kerman University of Medical Sciences should review their teaching curriculums and their exams to find the sources of these weak associations. However, the strong associations between the averages of scores in comprehensive exams and also in the physio-pathology, theoretical and practical courses may imply that the averages of scores are much more accurate than the scores of individual courses.References
Copyright 2008 - Indian Journal of Medical Sciences The following images related to this document are available:Photo images[ms08048t4.jpg] [ms08048t3.jpg] [ms08048t1.jpg] [ms08048f1.jpg] [ms08048t2.jpg] |
| |||||||||
