Poor methodology in meta-analysis of vitamins

Dr Steve Hickey, Dr Len Noriega and Dr Hilary

Scientific papers often reveal more than is apparent from the reported results. A recent review of clinical trials by Bjelakovic et al. claimed to show that certain antioxidant vitamins increased the risk of death.¹ Superficially, this study appears to have a degree of scientific rigour because of a detailed and extensive use of statistics. However, the statistics were inappropriately applied to poorly selected data, thus the conclusions are invalid. Researchers need to remember the fundamentals of the scientific method to avoid introducing experimenter bias. In this case, experimenter bias was compounded by a basic misuse of statistical testing.

 

Bjelakovic’s review was a meta-analysis of 16,111 scientific papers. Meta-analysis is a statistical technique which summarizes the results of several studies, giving a greater weighting to higher quality studies. The problem with Bjelakovic’s review relates to how the studies were chosen for inclusion in the analysis. Of the initial studies, 14,910 (93%) were discarded, with only a brief explanation of the exclusion criteria. Studies were dismissed because they were cancer studies, duplicates, or because they were deemed ‘not relevant’. However, studies of precancerous lesions² and skin cancer³ were included in the group designated as having a low risk of bias. Following the initial selection, 1201 research papers, covering 815 clinical trials, were described as being “reviewed.” It might be more accurate to say these papers were subjected to additional selection procedures: 747 (92%) of the 815 were rejected, for example, because no subject died during the experiment. The remaining 68 studies were included in the analysis.

Bjelakovic’s review states that this decision, to exclude 9 out of 10 studies (i.e. 747 from 815), depended on the judgement of three of the authors. This is a clear indication of potential selection bias, as the reviewers had access to the experimental results in addition to the experimental procedures.

Selection of trials for meta-analysis should be almost mechanical, based on rigorous objective criteria with critical justification. The large number of studies by Bjelakovic himself raises concerns in respect of objectivity, as the probability of trials being selected for inclusion in a meta-analysis can be influenced by knowledge of their results, leading to inclusion bias.⁴

Two of the researchers in the Bjelakovic meta-analysis further segmented the data into two groups, according to the perceived quality of the experimental procedures. However, once again the selection method did not exclude experimenter bias, as the researchers may have been influenced by the results of the studies. The complete selected data set of 68 trials reportedly showed no effect of vitamins on mortality. Notably, the group selected for low risk of bias showed an increased risk of mortality with supplements (RR 1.05). A reduced risk of mortality was found in the other group (RR 0.91). These results are consistent with experimenter bias, based on knowledge of outcomes in the selection. Selection of data is a powerful technique. To take an analogy, imagine we were to survey passenger-carrying vehicles in central London. Unwanted traffic includes bicycles, milk floats and delivery vans, so we exclude vehicles with less than four wheels, without side windows, and quiet ones. Small vehicles have a high risk of bias, since they can be hidden behind other traffic, so we reject any vehicle less than 20 feet long. Dark vehicles are hard to see at dawn or dusk, light coloured ones do not show up well against the local stone, and blue, green or yellow ones are hard to see against the panels of a nearby building site, so all are eliminated from the study. After excluding the groups with a high risk of bias, we count the vehicles and register their type. The survey concludes that all road passengers in London travel by red bus!

 

A critical failing of the Bjelakovic paper is the absence of detail on the number of statistical tests performed on the data. For example, at least two groups of tests reported concerned vitamin A. Vitamin A was tested singly and in combination with other supplements. Both sets of tests showed no significant effects. Then it was multiply retested: as a single or combined supplement, or taken with selenium, and again after exclusion of high bias risk. In this second group of tests, vitamin A reportedly increased mortality. The fact that this many tests were carried out on just one of the supplements investigated suggests the results of the study are unreliable. Conventionally, a single statistical test has a 1 in 20 probability of being significant by chance alone. With 100 such tests, we would therefore expect five ‘significant’ results, just by chance. The equation for computing the probability of a positive result, p, at significance level α, in n tests is:

p= (1-(1-α)ⁿ)

With a large data set and repeated testing of factors and subsets, several significant results could be attributable to chance alone.⁵ In this case, the paper gives no indication of the number of statistical tests employed, or justification for the probability values provided.

 

Bjelakovic’s meta-analysis has little biological meaning, because of the large number of ill-defined substances that have been grouped together. The meta-analysis includes a diverse range of doses of the individual supplements, with no concern for the expected physiological effects. In one of the included trials, a single dose of vitamin A was followed up over a period of three months.⁶ Bjelakovic also analysed studies of ‘vitamin E,’ an almost meaningless term in terms of nutrition or pharmacology. Vitamin E refers to a number of fat-soluble antioxidants, including four natural forms each of tocopherols and tocotrienols. Additional synthetic forms of tocopherol are widely used for vitamin E studies. Thus, it is not clear to which actual nutrient Bjelakovic’s ‘vitamin E’ results would apply. Moreover, one of the vitamin E studies selected by Bjelakovic, as having a ‘low risk of bias’, has previously been cited by Hickey and Roberts as a prime example of bias in vitamin studies.⁷ Only studies with recorded deaths were included by Bjelakovic: this was presumably considered necessary in a study of death rates. However, this selection has the potential to increase bias, as it clearly excludes studies where supplements could not be associated with increased mortality. A secondary effect of this selection technique is that the included population tended to be sick, rather than healthy. Although most included studies were on the sick, they used nutritional rather than pharmacological doses. For example, doses of vitamin C ranged from 60 to 2000 mg; these are too small to be helpful against serious illnesses.⁷ Furthermore, trials on nutritional supplements in disease do not necessarily apply to healthy members of the population.

 

The paper by Bjelakovic was reported widely by the media but was not subjected to scientific criticism. Media reports gave the impression that scientific evidence suggests vitamins may be harmful. In fact, no evidence has been provided to this effect. The statistics provided were insufficient to support a claim that vitamin supplements will increase mortality. Moreover, the results cannot validly be generalized to a relatively healthy general population.