The evaluation of physical treatments
Clive E. Adams
The strengths and weakness of the single trial
Strengths
New treatments, or variations of older therapies, rarely represent a revolutionary departure from what has gone before. As progress is usually made in modest steps, evaluation in prospective randomized trials is needed. These studies, comparing a new treatment with a relevant control, may be able to highlight and quantify relatively subtle but important differences in outcome.
Randomization controls for selection biases. If undertaken carefully, it should ensure that both known and unknown confounding variables, such as age, sex, and additional medications, are evenly distributed between groups. Any differences in outcome should then be due to the treatment, or the intention to give the treatment (see below). In 1991, the World Health Organization stated that the randomized controlled trial, if ethical and feasible, is the most objective means of evaluating mental health interventions.(1)
Certainly, large well-conducted trials, with participants, interventions, and outcomes recognizable to those working in health services, are potent guides to clinical practice. Nevertheless, even when such trials exist, it is important to view them alongside all other comparable evidence. Should the large study affirm the findings of smaller trials the clinician can proceed with confidence. If there is a discrepancy then debate will be generated, which should clarify important issues relating to the participants, interventions, or outcomes measured or to the methods by which the trial was conducted.(2)
Power
As numbers within a study increase so does the precision of results, enabling important but subtle differences to be detected, if they do indeed exist. Should a new treatment be considerably better than its predecessors few people would have to be randomized in order to demonstrate clearly the advantage of the innovative approach. As the advantage expected of new treatments is usually modest, reasonably large studies are often needed.
The power calculation is an important prerequisite for any randomized trial. For example, if clinical observation suggests that a new treatment can help 20 per cent more people avoid admission than the standard care, this can form the basis for a power calculation for a trial. Using a simple formula(3) the trialist can work out how many people would have to be randomized in order to have a known probability of highlighting such a difference, should one really exist. In this case, about 150 people would have to be allocated to each arm of the trial to be reasonably confident of detecting a true 20 per cent difference (α=0.05, β=0.8). Most mental health trials are far too small to show up anything but very gross differences between treatments. For example, the average number of participants in schizophrenia trials is about 100 with only a slow increase over time.(4)
A single small trial should not greatly influence the clinician, but the combined results of several studies may begin to have the power to inform practice.
Biases
Randomization attempts to control for the biases that would influence treatment allocation (selection biases). Blinding at outcome attempts to control for biases that would result from participants or raters knowing which treatment had been allocated to whom (observer bias). Inadequate randomization leads to an overestimate of effect in the region of 31 to 40 per cent, and poor blinding at outcome to that of about 17 per cent.(5) Further overestimate results from the use of unpublished or modified scales, commonly seen in mental health studies, and a financial or academic investment in the therapy by the trialists.(6,7)
There are many threats to the validity of a single trial. Viewing all relevant studies, each of which was subject to different degrees of bias, should give a more balanced picture. Of course, the reader of a review should be vigilant for the systematic bias, across all trials, that may consistently sway results one particular direction.
Generalizability
Even if a study is adequately powered and undertaken with due regard for bias, a single trial may be difficult to apply to everyday practice.
(a) Participants
Most studies involve unusual participants. Frequently those eligible for trials have to give informed consent, their problems are well defined and do not involve multiple pathologies, and they are expected to tolerate the demands of a study.
(b) Interventions
Applying the results of a single study is made even more difficult because study interventions are often impractical. For example, drug trials may use rigid dose regimens impossible to apply to routine care. Psychosocial therapies tested within a trial are often of such high quality that they bear little resemblance to what an overstretched clinical service can provide.
(c) Outcomes
Measurement of outcome may also limit the value of a single trial. In a survey of 2000 schizophrenia trials, 640 different scales were used to record outcomes such as mental state, behaviour, global impression, and adverse effects.(4) Specific subspecialties within psychiatry may have an even greater propensity to create scales for trials.(8) Even within poorly powered trials, these sensitive tools may be able to detect real differences between treatments that may be statistically if not clinically significant. However, few clinicians use such scales in everyday practice and interpreting results becomes a matter of conjecture.
Trials that involve carefully defined groups of participants receiving meticulously controlled treatments and having outcomes measured on sensitive scales are called ‘explanatory’ studies.(9) Such trials dominate the literature, although calls for more pragmatic or ‘real world’ methodology are increasing(4,10) and there are now examples of this broader approach.(11,12,13,14) Currently, generalizing from the results of a single trial to day-to-day practice is inadvisable. If, however, several explanatory studies, all undertaken with constrained, but different, methodologies are giving a similar result, the clinician can feel a little more comfortable when acting on their findings.
The rogue result
Even the well-conducted generalizable trial can produce a rogue result. Currently, the acceptable level of chance is one in 20. A statistically significant result, often denoted as p<0.05, suggests that the finding, if the experiment was to be replicated, should occur 19 out of 20 times. One time in 20, however, a different result will appear simply because of chance. This can lead to an interesting paradox. A single trial may not provide the best evidence of how to manage people, even in the locality that the study was undertaken. The play of chance may result in an erroneous result and unless that trial is viewed in the company of all relevant evidence, clinicians will be mislead.
Time
It is inadvisable to act on the results of a single trial because of issues of power, biases, generalizability, and the possibility of chance erroneous results. There is, however, also the issue of time. Clinicians may often prefer to read the results of a single review rather than spending time assimilating information from several similar trials. Most practitioners have very little time to keep up with research relevant to their practice. Clinicians admit to having half an hour of reading time per week,(15) and much of that may not be retained.(16) Reading reviews is time efficient.
Reviews
There are two main approaches to the reviewing process—the traditional and the systematic.
The traditional review
The traditional review is often undertaken by a person well respected in the relevant field who uses knowledge and acumen, supplemented by research, to produce a synopsis of the literature. This approach still dominates the current texts, journals, and lecture tours. For example, in 1987 in four major North American medical journals, 86 per cent of review articles depended on qualitative synthesis and contained no ‘methods’ section whatsoever.(17) In only 6 per cent of the reviews was quantification attempted in order to support opinion and the situation did not improve much across the next decade.(18) Therefore, the clinician is left in a situation where it is difficult not to operate under a double standard. On the one hand, a large relevant trial providing objective evaluation would be desirable, but frequently, a traditional subjective review is all that is available.
Systematic reviews
The form of a systematic review encourages the introduction of basic epidemiological principles and quantification into the process of reviewing. Gene Glass, an educational psychologist, was the first to add the results of similar studies in the hope of quantifying the effects of a treatment.(19) Glass defined ‘meta-analysis’ as ‘the statistical analysis of a large collection of analyses results from individual studies for the purpose of integrating the findings’.(20) Unsurprisingly, in the sensitive area of the psychotherapies, their first and flawed attempts in the new discipline generated controversy.(21) Critics were quick to point out that drawing conclusions from summation of very different types of therapies, undertaken by practitioners of varied experience, was likely to be inadvisable. These pioneers, who even years later are still being criticized for adding ‘apples and oranges’,(22) are nevertheless owed a great debt by the rest of medicine. After all, it depends on the question being asked. It is fine to mix apples and oranges, if your question is about fruit.(23)
Systematic reviews attempt to minimize bias in the identification, extraction, and summation of relevant data by applying good survey methods to the process of literature reviewing. An analogy may help. In a community survey of the prevalence of mental disorders, a researcher stands on the doorstep of the hospital and suggests that 5 per cent of the population suffers from serious mental illness. By chance, the final estimate may even be correct, but the work could not be seen as methodologically rigorous. The researcher should have written a study protocol, clearly defined an
unbiased sample of individuals to interview, and specified a priori the analyses to be undertaken. A systematic review should do this for a survey of a ‘population’ of relevant literature. Within such a review, the objectives, criteria for selection of relevant studies, search strategy, methods of study selection, data extraction, and assimilation are all made explicit.
unbiased sample of individuals to interview, and specified a priori the analyses to be undertaken. A systematic review should do this for a survey of a ‘population’ of relevant literature. Within such a review, the objectives, criteria for selection of relevant studies, search strategy, methods of study selection, data extraction, and assimilation are all made explicit.
The advantages of the systematic approach
As is suggested above, a systematic review may, by adding the results of similar studies together, at least begin to address the issue of the underpowered study, single trials of biased methodology, poor generalizability, and idiosyncratic results. Although often longer than most traditional reviews, the systematic review is still a timeefficient way to appraise research. Additional advantages are both intuitive and practical.
(a) Objectivity
Medicine remains a scientific discipline and the attempt at objective quantification must be an integral part of this approach. However, the systematic review and meta-analysis should never become a source of clinical tyranny. Individual clinicians will always have to use wisdom and judgement in their day-to-day decision-making, but to exclude objective appraisal from this process is foolhardy.
(b) Clinical empowerment
Systematic reviews can provide clear information to clinicians, policy makers, and recipients of care, and so help inform the decisionmaking process. For example, a systematic review of family therapy suggests that this educational, psychosocial package can help those with schizophrenia avoid or postpone relapse.(24) This finding is very much in line with the suggestions of traditional reviews.(25) However, the systematic review is able to illustrate how seven families have to undergo regular therapy, for up to a year, in order for a single relapse to be postponed. Such data, of course, mean different things to different people. Clinicians may find this an acceptable degree of effort, whereas managers of services, or even families of those with schizophrenia, may not. Although the findings may not decrease controversy, at least debate can be informed.
The quantification of trial data can sometimes provide information quite at odds with the advice of traditional reviewers. The best example comes from outside mental health care. In 1992, Antman et al. undertook a meta-analysis of randomized trials evaluating the care of those with acute myocardial infarction.(26) As the reviewers added trial data, they found that by 1973 enough studies existed to show clearly that thrombolytic treatment saved lives. Subsequent trials added precision to the result but did not change the finding. Antman and colleagues also showed how traditional reviews continued to fail to mention thrombolytic therapy up to 15 years after the summated trial data could have shown its value.(26) These traditional reviews recommended treatments for myocardial infarction that were positively harmful. Examples have emerged from mental health. Sometimes, traditional reviews make bold claims or recommendations which are not supported by the evidence in quantitative, systematic reviews. For example, some claims made for cognitive therapy for schizophrenia(27) go well beyond objectively summarized evidence.(28, 29) Conversely, even when evidence is of high quality and readily accessible, traditional reviewers can be blind. For example, both the strengths and the limitations of new generation antipsychotic drugs have been evident for years(30,31,32) but traditional reviews and texts have encouraged uncritical enthusiasm for their use(33, 34) and even guilt for failing to prescribe.(35)
(c) Gaps in research
Often a systematic review will highlight unsuspected gaps in research. The trial-base of much routine practice is not strong, and systematic reviews can help shape questions to be tested in well-planned and conducted trials.(36) Certainly, some research funders are now requiring that a systematic review be undertaken before a randomized trial is funded. This also avoids wasting resources on questions that have already been answered.
Related posts:
Stay updated, free articles. Join our Telegram channel
Full access? Get Clinical Tree
