African Health Sciences Makerere University Medical School ISSN: 1680-6905 EISSN: 1729-0503 Vol. 10, Num. 2, 2010, pp. 204-207

African Health Sciences, Vol. 10, No. 2, July-December, 2010, pp. 204-207

Practice Point

Reporting statistics in clinical trials published in Indian journals: A survey

Jaykaran¹, ND Kantharia¹, Y Preeti¹, P Bharddwaj², J Goyal³

¹ Department of Pharmacology, Govt. Medical College, New Civil Hospital, Lucknow, India
² Department of Community Medicine, Era Medical College, Lucknow, India
³ Department of Pediatrics, Govt. Medical College, New Civil Hospital, Surat 395001, India

Correspondence Address:J Goyal, Assistant professor, Department of Pharmacology, Govt. Medical College, New Civil Hospital, Surat 395001 India, drjaykaran@yahoo.co.in

Code Number: hs10047

Abstract

Keywords: Clinical trials, statistics, type 1 error

Introduction

The randomized controlled clinical trial is considered as the gold standard for an evaluation of effectiveness of any new drug or intervention. The methodologies of clinical trials are improved over the years. Statistics used in analysis of data of clinical trials also improved because of the availability of the software for analysis of data. By using the software more complex statistics can be performed like analysis of several endpoints, repeat measurement of endpoints over the course of treatment, subgroup analysis and comparison of more than two treatments, but this leads to various types of statistical errors like danger of excessive use of the statistical testing and inflation of type 1 error. Also if all findings are not expressed in article or abstract then it may lead to exaggeration or underplaying of actual treatment difference. So our aim in this study is to discuss few issues associated with the reporting of statistics of clinical trials published in three Indian journals in year 2008-2009. Very few studies have bee carried out to address these issues and most of them are not for Indian journals, ^[1] .

Methods

We evaluated 46 clinical trials which were published in three Indian medical journals in 2007 and 2008. Out of these 46, 15 from Indian pediatrics, 24 were from Indian journal of pharmacology and 7 from journal of post graduate medicine. All authors evaluated each clinical trial for various parameters related to design, analysis and reporting. Discrepancies between the authors were resolved by consensus. 10 clinical trials reevaluated by first author to check the reliability and no substantial difference found. Our survey included only the comparative trials.

Statistical analysis

Descriptive statistics used for the measurement of frequency of events. Exact frequency reported with proportion and 95% confidence interval around the proportion. Comparison between the frequencies of reporting confidence interval between three journals done by chi square test with Yates correction with the help of EPI 6 software.

Results

We observed that the median number of patients of the trial was 65. In 7 trials sample size was less than 30 and in 8 trials it was more than 100. Range of sample size was 16 to 206.

[Table - 1] shows the number of endpoints mentioned in each clinical trial report, as well as the number for which statistical test used to see the difference between groups. The median number of end points was 4. 25 (0.54, 95%CI 0.40 to 0.67) trials reported number of end points 5 or less than 5. 9 or more than 9 end points were observed in 8 (0.17, 95% CI 0.09 to 0.30) trials. Median end points reported in the clinical trials of three journals were similar (4 for Indian journal of pharmacology, 4 in Indian pediatrics and 5 in journal of post graduate medicine). Significant test used for most of the end points. Median end points for which significant test used was 4. Most of the trials contain the end points of both qualitative and quantitative type; only 1 (0.021, 95%CI 0.003 to 0.113) trial has survival analysis as end point. Primary end points were mentioned in 23 (0.50, 95%CI 0.36 to 0.63) trials. Multiple end points were not adjusted in any trial. In the studies where primary end points were mentioned, in none of the study statistically significant secondary endpoints were emphasized over the primary endpoint.

In trials related to quantitative end points, end points are usually measured before the start of intervention and many times after the start of intervention. In our study 21 (0.45, 95%CI 0.32 to 0.59) of trials reported repeated measurements. Among these 18 (0.85, 95%CI 0.65 to 0.95) trials reported significant test at every measurement. There was no technique used to decrease type I error generated because of repeated measurement over the time in any trial.

21 (0.45, 95%CI 0.32 to 0.59) trials were having two treatment groups. 18 (0.85, 95% CI 0.65 to 0.95) trials were having three or more than three treatment groups. Among these 18 trials, 15 (0.83, 95%CI 0.60 to 0.94) were from Indian journal of pharmacology. 7 (0.15, 95%CI 0.07 to 0.28) trials were having two period crossover designs. In the trials where three or more than three groups were recruited ANOVA or its non parametric equivalent (Friedman one-way ANOVA) was used to analyze data (17 (0.94, 95%CI 0.74 to 0.99) trials.

We found that median number of significance test was 15 [Table - 2]. It includes all significance tests including the subgroup analysis, multiple end points and repeated measurement. More than 20 significant tests were used in 5 (0.10, 95%CI 0.04 to 0.23) trials. Range of number of significance tests was 2 to 53.

In this study we found that subgroup analysis was done in 9 (0.19, 95%CI 0.10 to 0.33) of studies. 3 (0.33, 95%CI 0.12 to 0.64) of these has more than one prognostic factor in the subgroup analysis.

Confidence interval was calculated in 11 (0.23, 95%CI 0.13 to 0.37) trials. it was more frequently calculated in journal of post graduate medicine (6 out of 7) as compared to Indian journal of pharmacology (1 out of 24) and Indian pediatrics (4 out of 15).(X ² = 18.9, df- 2,P = 0.000118).

We observed that baseline comparison between study groups were done in 41(0.89, 95%CI 0.76 to 0.95) articles. In all trials the comparison was done by a statistical test.

Exact sample size was calculated in 18 (0.85, 95%CI 0.65 to 0.95) trials (3 in journal of post graduate medicine, 7 in Indian pediatrics and 8 in Indian journal of pharmacology). In all these 18 trials, the minimum difference to be detected and the statistical power needed to find such a difference were reported.

Discussion

We observed that multiple end points were evaluated in all of the trials. This enhances the chance of type I error, ^[2] . One method of addressing this problem is by considering most important end point as primary end point during the design of trial and validity of hypothesis checked by using statistical test on this end point only. Other end points should be considered as secondary end points and should be analyzed as exploratory, ^[1] . In this study 23 (50%) trials used this method. In some condition it is very difficult to identify single end point in advance. In that condition some other statistical methods can be used to adjust the inflation of type 1 error like Bonferroni correction method and composite end point method, ^[1],[3],[4] . In a study done by Ton J et al (2006) it was found that among the 16 randomized controlled trials with positive results published in British medical journal in 2004, only 8 trials remain positive after Bonferroni correction ^[3] .

We also observed repeated measurement of end points in many of the trials (45 %). It may also leads to type 1 error. During the design of trial a strategy should be planned to deal with the repeated measurement. There are various strategies available for this like comparing average effect of treatment over the time or one or two time point can be fixed in advance for comparison between the treatment or time period to attend specific threshold value can be compared, ^[1],[3],[5] . In our study we found that in no trial these adjustments were done so there is profound chance of having type 1 error in these trials.

Same problem also seen in the case of more treatment groups that will also leads to type 1 error. ANOVA and Friedman one-way ANOVA are used to decrease the inflation of type I error. This method was used in 17 trials. This is an encouraging finding. There are some other statistical methods available which can be used for adjustment of P values, ^[2],[3] .

We found that median number of statistical test per trial was 15. As compared to other similar studies done on western medical journals, the number is more, ^[1] . Actual number of statistical tests may be more as authors usually report the significant tests only. The best scenario of validity of statistical test is, when only one predefined end point is used in analysis. More statistical tests lead to type I error, ^[6] . So a primary end point should be defined before starting the clinical trial and secondary end points should be used for exploration of hypothesis ^[2],[3] .

In our study subgroup analysis was done in 9 (19%) of trials done which is less compared to similar studies published in western medical journals, ^[7] . It is common practice to explore a subgroup of patients if overall result is not significant ^[7] . If author decides to do the subgroup analysis then it should be designed before the start of trial. In our study we found that planned subgroup analysis was not reported in any study. Subgroup analysis can increase type I error. However, a preferred alternative to subgroup analysis is to combine the factors into a single predictive model (an equation, such as a regression analysis), rather than to analyze each subgroup separately. The researchers thus test for interactions between the variable and the endpoint to avoid subgroup analysis. We found that this method was not used in any of the 9 trials in which subgroup analysis was reported. Subgroup analyses always include fewer patients than does the overall analysis, they carry a greater risk of making a type II error-falsely concluding that there is no difference. Guideline should be followed to report the subgroup analysis of clinical trials, ^[7] .

In this study we observed that in most of the trial P value was the only method of reporting the results. P value is often misinterpreted, ^[8] and even if it is interpreted rightly it has some limitations, ^[9] . Not writing the exactP value enhanced the problem further, ^[6] . For the main results absolute difference between the groups with 95% confidence interval should be reported with or without P value, ^[6] .

We found that in many of the trials baseline comparison was done with the help of statistical test. In a properly randomized trial each participant has equal chance of assigment to any of the study groups so any difference in the prognostic factor is because of the chance not due to bias. It shows statistical imbalance. The result of the trials should be adjusted for this statistical imbalance by regression methods, ^[1],[6] .

It was observed that exact sample size was calculated in only 39%.In clinical trials sample should be big enough to have a high chance of detecting, as statistically significant, a worthwhile effect if it exists, and thus to be reasonably sure that no benefit exists if it is not found in trial. for sample size calculation in hypothesis testing researcher must know the effect size, standard deviation, significant level and power of study, ^[10] . Effect size and standard deviation of new agents can be calculated by pilot study.

In a review by Alexander M et al in 2007 our findings are confirmed, ^[11] . We evaluate clinical trials on the basis of few statistical problems reported in clinical trials of western medical journals; this is the limitation of our study as more extensive criteria could have been used. We believe that very few studies are published regarding reporting of the statistics in Indian journals and this study is one of them.

So we believe that clinical trials published in three representative Indian journals are not devoid of statistical problems and most important problem is type I error. Though there are various methods available to decrease these of errors, they are not used during the reporting of these trials. It leads to exaggeration of results. Reader should take care during the formation of opinion on the basis of these trials.

What is already known?

Reporting of statistics of clinical trials published in western journals shows that because of various statistical issues, results of clinical trials should be interpreted cautiously.

What this Study Adds?

Similar findings are observed in clinical trials published in Indian journals.

References

Copyright 2010 - African Health Sciences

The following images related to this document are available:

Photo images