David Sackett, an internist and epidemiologist, historically defined an evidence-based medicine approach as consisting of five steps:

Although Sackett’s classic definition of an evidence-based approach is invaluable, this chapter focuses on the unique context of doing pediatric mental health. Child psychiatry is not internal medicine. It is typically practiced in outpatient multiprofessional teams where psychosocial interventions have a much greater role than in internal medicine. Readers seeking textbooks on basic principles of EBP might consult multiple guidebooks, journal articles, and the Internet (1,2,3,4,5).

This chapter begins with how various professional groups use the phrase “evidence-based.” It then proceeds to make a case for how processes based on evidence-based practice (EBP) are useful and significantly different than usual practice. It continues with common objections to EBP, but offers a rebuttal for each and the evidence supporting EBP. The second part of this chapter presents selected elements of EBP within the context of pediatric mental health: tips on searching the most relevant databases, diagnostic approaches consistent with EBP, choosing a treatment, developing local data, and ideas about developing evidence-based systems.

Varied authors and groups have used “evidence-based” (EB) as a modifying phrase. Evidence-based practice, or EBP (6,7), is the term used here, but there is also evidence-based medicine, or EBM (5), evidence-based services, or EBS (8), and evidence-based treatments, or EBTs (9). The phrase “empirically supported treatments,” or ESTs, is often used interchangeably with EBT (10). To oversimplify, there are three groups. The first group, using the term EBM, is often associated with medication issues and child psychiatrists. The second group, using the term EBT, is associated with psychologists, psychosocial treatments and the American Psychological Association (APA). The third group, EBS, is associated with systems striving to better use empirical results to improve outcomes. In this chapter, when we refer to EBP, we’re including all three groups. These multiple terms share a common interest in the evidence of empirical data, both about our patients and our own functioning as well as off the shelf in the literature; combined, these three terms contain a powerful flood of ideas.

There is considerable overlap between EBM, EBT, and EBS, but each retains its distinct flavor. EBM authors often focus on changing individual practitioner behavior. For example, EBM tries to interest practitioners in researching “answerable questions” regarding individual patients. Often examples used involve medications. Many well known EBM leaders like David Sackett are epidemiologists or internists. On the other hand, EBT authors tend to focus more on studying specific manualized psychosocial treatments. EBT authors are often university-affiliated psychologists who study the effectiveness of a psychosocial intervention for a specific disorder. They also study how a psychosocial intervention “travels” when it is “exported” to sites other than where it was developed. Finally, evidence-based services (EBS) is a term used by clinicians in delivery systems trying to improve outcomes by better
use of empirical evidence (11). In EBS a consensus-building group of providers, administrators, and consumers agrees on a menu of effective interventions including both EBTs and medications. These three empirical approaches are summarized in Table 2.1.2.1. The term evidence-based practices, or EBPs, used here is an umbrella term for processes based on all three groups while valuing patient preference and clinical expertise as well. EBP welcomes the use of clinical expertise, for example, in formulating the context of symptoms (12). And, while including both medication and psychosocial interventions, EBP is neutral in choosing between them, an advantage on a multidisciplinary team.

Significant boundaries between child psychology and child psychiatry have shaped these three streams of empiricism. Child psychologists and child psychiatrists typically belong to different professional organizations, attend different conventions, publish in different journals, and occupy different niches in clinical organizations such as state clinics or hospitals. These different worlds naturally evolve different ways of thinking, sometimes referred to as cognitive boundaries (13). In fact, the prevailing paradigm in each discipline may be so different that each discipline has distinct cognitive assumptions and may advance different claims to knowledge. At their worst, boundaries can be sufficiently extreme that there is no common ground for productive dialogue (14).

EBP is a helpful antidote to the tendency of practitioners to identify with a particular discipline or treatment. For example, a practitioner may think of himself as primarily a psychopharmacologist, or as a family therapist, a play therapist, a behavior therapist, or as an expert in delivering a specific manualized therapy. EBP as conceived here, on the other hand, is not attached to a specific treatment modality or profession. Instead, EBP chooses those feasible treatments proven in the most valid studies to deliver the most rapid, complete, and long lasting improvement in functioning and symptoms with the least harm. A commitment to EBP therefore significantly changes the identity of practitioners: A commitment to finding and using both published and “local” evidence becomes a central value.

In a multidisciplinary team, the processes of making a diagnosis, choosing a treatment, and assessing its results are all significantly different in a team committed to EBP than in a team proceeding “as usual.” Table 2.1.2.2 lists a chain of clinical processes fundamental to EBP, and highlights differences between the EBP approach and usual practice in defining the clinical population, in choosing an intervention, and in evaluating its effects relative to a comparison or control group. This order is the familiar PICO format derived from epidemiology: Population, Intervention (or Exposure), Control (or Comparison), and Outcome. Of course, a solo practitioner can also use many of these processes. For example, both the well known PubMed site as well as the Internet site for the Journal of the American Academy of Child and Adolescent Psychiatry offer extensive resources for searching answerable questions. A major issue for practitioners, however, is how to offer ESTs that often require extensive training. Parent Management Training for Oppositional Defiant Disorder, for example (15), or Cognitive-Behavioral Treatment or Interpersonal Therapy for depression (16) all require training. Yet a practitioner may choose to obtain training in those ESTs which will be most useful in the practice she has developed. Informal subspecialization among community therapists is also an option.

At EBP lectures and informally in conversations with colleagues, certain objections to EBP seem inevitable. Here are some common ones; each is followed by a rebuttal.

A variant of this complaint is There aren’t RCTs for everything. This objection fails to note that the EBP practitioner is committed to the judicious use of the most valid and relevant evidence available, not just RCTs. EBP does use a hierarchy of evidence based on validity of research design, and RCTs are high up in that hierarchy. But if only case reports exist, then the EBP approach is to use them as evidence. However, the rapid growth of information about the treatment of child psychiatric disorders in the past decade makes this increasingly unnecessary.

How closely are therapeutic relationship variables and outcomes associated in youth psychotherapy? How is that association moderated by age, presenting problem (emotional or behavioral), treatment type, and structure (behavioral or not, manualized or not), and whether the treatment was received from an ongoing clinical service or a “one-time” research team? And do therapeutic relationship variables function as the change mechanism itself? Or are they “merely” a necessary factor to promote attendance at sessions? Understanding these
issues is made more difficult because effective therapy improves clients’ perceptions of the relationship. The client’s perception of progress in the therapy therefore inflates ratings of the alliance if the alliance is measured late in the course of therapy. Furthermore, understanding the impact of variables measuring the therapeutic relationship in youth psychotherapy is made more complex by the variety of psychotherapies, varying developmental levels, varying presenting problems, and no unifying agreement about how to measure relationship variables (17). In the adult psychotherapy literature, two meta-analytic reviews of the relationship between alliance and outcome have demonstrated a significant albeit modest effect (18).

In the youth literature, a meta-analysis of 23 studies with a median length of treatment of 19 sessions using numerous scales to measure the alliance/therapeutic relationship showed a small effect (estimated weighted effect size of .21) (17). There was no significant moderator effect for children vs. adolescents, behavioral vs. nonbehavioral, manualized vs. nonmanualized treatments, but the relationship was more important in youth with externalizing problems than those with internalizing problems.

And finally, because some of the studies measured the alliance late in treatment, high relationship ratings may reflect at least in part positive perceptions of the relationship based on a positive treatment response achieved by means other than the relationship. Other studies have shown that behaviorally oriented psychotherapists may be more inclined to see a positive relationship as essential to ensure attendance, whereas psychodynamic ones may see a positive relationship as a change mechanism in its own right (19).

In community-based child therapy, a strong therapeutic relationship supported continuing to attend sessions (20). In addition, the parent–therapist alliance was associated with reductions in internalizing psychopathology, while a close child–therapist alliance assessed during treatment was associated with a reduction in anxiety symptoms (21,22). In summary, the therapeutic alliance matters in promoting attendance, in dealing with externalizing, and internalizing disorders, as well in community based child therapy. Nevertheless, these results suggest that the alliance by itself is inadequate to produce the larger effects clinicians and patients seek.

At least some therapists object to EBP because they note it examines outcomes for groups of youths and thereby loses the individuality of each youth. According to this argument, EBP conclusions based on groups of youth fail to respond to each youth’s uniqueness since no two youth share exactly the same genetic makeup, cultural heritage, social circumstances, developmental history, and family background. Yet it is a caricature of EBP interventions that they are applied indiscriminately without any interest in the individual. Consider an EBP psychosocial intervention as having a hard core but a soft exterior (23): Whereas the outside can be modified and individualized to make it easy to swallow, the hard inner core contains the essential components which create change.

In addition, newer generations of ESTs are sensitive to the criticism that a manualized treatment needs to be individualized as well as lively and engaging. Consider a recent study of collaborative problem solving in moody children with ODD (24). The design of this study allows for therapists providing the intervention to determine session content on the basis of their assessment of the clinical needs of the child and family. Other authors have also called for blending creativity and flexibility into a manualized treatment to allow individual variation within a defined intervention (25).

Finally, the first paradigm of the therapist as artist, working with the unique patient to create a unique solution, and the second paradigm of the therapist as adherent to the results
of the best science, may be growing toward each other in recent years. For example, some recent EBM articles have softened their position towards the value of experience and clinical judgement (26), while the American Psychological Association has hardened its position about the importance of randomized trials (27). Both perspectives can be valued by a practitioner sensitive to multiple inputs: patient preference, his own experience, the nuances of each child, and the most valid available evidence.

Two studies have shown that, in real-world practice settings, it is difficult for child mental health interventions to show an effect compared to a control group (28,29). In the first, a randomized study by Bickman and colleagues (1999), one arm of the study received an increase in resources. Interviews were conducted for two years following collection of baseline data. Results in measures of symptoms and functioning showed that, while the arm with a considerable increase in resources did have improved access to care and in fact actually received more care, these access differences did not translate into improved clinical outcomes. In addition, children who did not receive any services improved at the same rate as treated children. This sobering conclusion is supported by Weisz and Jensen’s review of the effectiveness of both medication and psychosocial intervention in the context of the real world of caring for clinically referred youth. The authors note that evidence, where available, on the effectiveness of such treatment is minimal when compared to the large body of evidence on efficacy. It seems reasonable to conclude that such null results suggest the need for change.

In contrast to the sobering conclusions of these null results, committed efforts to build an evidence-based system of care in Hawaii’s Child and Adolescent Mental Health Division (CAMHD) appear to have improved outcomes. This story begins in 1994 when federal courts charged the state with establishing a system of care to provide effective mental health and special education services for children and youth as required by federal law (11). The initial system responses to the court’s decree included planning efforts and increases in service capacity, allowing more youth to access a wider variety of services, as well as increased quality monitoring and more interagency coordination. The statewide quality monitoring included basic quantitative feedback; this feedback demonstrated more youth being served by more services at a higher cost. Since stakeholders wanted assurances of efficiency, the focus turned to asking whether the increase in resources had led to improved symptomatic and functional outcomes with empirical results as the arbiter. In addition, CAMHD’s leaders wanted the system to develop in a way that frontline decisions about patient care were based on the best available evidence. They focused, therefore, on linking the best and most relevant evidence to clinical decisions (11).

To accomplish this linking, the Hawaii Department of Health organized a task force on empirical services in October, 1999 (10); the only requirement for membership was, and remains today, regular attendance and willingness to read and review studies. Its membership has included clinicians from several disciplines, university faculty, parents, administrators, and CAMHD employees. This task force is charged with conducting ongoing multidisciplinary evaluation of psychosocial interventions for common disorders using methodology developed in the Clinical Division of the American Psychological Association. Additional topics can be reviewed as well, such as the efficacy of seclusion procedures. Each search uses a structured methodology; results are evaluated with a five-level system ranking the efficacy and effectiveness of each intervention. (Effectiveness is based on the performance of the intervention under naturalistic, or real-world, conditions.) The task force begins with a literature-based approach but its diverse constituency tempers the results to fit local conditions.

The result of the task force’s work has been a “menu” of recommended treatments distributed to clinicians on a single sheet of blue paper, creating the nickname “blue menu,” summarizing recommended psychosocial treatments. It is also posted on the Internet. A one-page review of the task force’s conclusions regarding psychotropic medications’ efficacy and effectiveness for the children of Hawaii is also distributed and posted on the Internet. Both are updated biennially.

CAMHD also tracks its own results at case, clinic, and system levels wide as well with the Child and Adolescent Functional Assessment Scale, or CAFAS (30), the Child Behavior Check List, Teacher Report Form, and Youth Self-Report (31). Following individual cases allows clinical staff to identify whether or not a youth is improving. Documented ongoing progress leads to the recommendation of continuing the present treatment. If a youth is not improving, the clinicians can reexamine whether there is a problem in treatment selection; if so, a more favorable intervention is sought (11).

Quarterly outcomes based on parent, teacher, and clinician reports for Hawaiian youths treated improved significantly during the years 2001 through 2004, years when the system was actively moving toward evidence-based practice (32). The slope of mean improvement in functioning as rated by the CAFAS showed a 146% increase; the slope of mean improvement in CBCL showed a 271% increase, and the TRF a 50% increase over the course of the three-year period (32). The proportion of youth showing a pattern of improvement during the service episode based on CBCL data rose from 54.7% to 68.2%, based on TRF data from 50.7 to 58.6, and based on the CAFAS from 66.5 to 69.0%.

These results are consistent with the hypothesis that implementing evidence-based services significantly impacts both functional outcome and symptomatic outcome. Furthermore, the results are large and clinically significant. Whether the changes in functioning improvement and symptom reduction stabilize here or continue to improve requires continued study. Although this study did not control for such potentially confounding variables as diagnostic mix, gender, or ethnicity impacting the results, it nevertheless shows consistent and large results across three separate informant groups— parent, teacher, and clinician. If EBP seems, therefore, well worth the effort, let’s turn now to its core ideas.

The number needed to treat, or NNT, for any given intervention in a defined population, is the number of patients we need to treat with the intervention in order to prevent one additional bad outcome (1). This is calculated as follows:

NNT = 1/(Proportion of subjects in control group with bad

The denominator in the NNT equation is called the absolute risk reduction (ARR). Hence, a brief version of the formula is:

NNT = 1/ARR

As an example of NNTs, consider the Treatment for Adolescents with Depression Study (TADS). The “bad outcome” chosen was a failure to score either much improved or very much improved when assessed by an independent rater using the Clinical Global Impression (CGI) scale. Using this definition of bad outcome, 39.4% of subjects in the fluoxetine cell had a bad outcome versus 65.2% of subjects in the placebo cell. Thus:

NNT = 1 (.652–.394) = 1/.258 = 3.87

Therefore the NNT reported for fluoxetine alone was 4, with 95% CI 3 to 8 when response is defined as a Clinical Global Impression score of much improved or very much improved at the end of treatment. The calculated NNT for combined treatment with CBT and fluoxetine using the identical definition of response was 3, with 95% CI 2 to 4 (33). The value of adding CBT to fluoxetine is evident not only in the improved NNT but also in the much narrower confidence intervals. Table 2.1.2.3 calculates NNTs in child and adolescent psychiatry for a variety of other disorders as well. Note that, in general, these NNTs hold up well in comparison with many standard interventions in medicine: Sackett (2000) notes that the NNT for preventing diabetic neuropathy with 6.5 years of intensive insulin treatment is 15, and that the risk of preventing a death over five weeks using streptokinase infusion in patients with acute myocardial infarction is 19. The NNTs in this table are a reflection of the progress made in recent decades in child and adolescent psychiatry.

By convention, NNT is always a whole number, rounded off to an integer. Note that NNT is impacted by several factors in addition to the effect of the intervention itself. First, how recovery is defined affects NNT; choosing a cutoff that makes it “easy” to achieve recovery produces a lower NNT. Second, how many in the control group spontaneously recover affects NNT; the more spontaneous recoveries in the control group, the more difficult it is to achieve a low NNT. Third, when the data are “sliced” to compare the intervention and comparison group may affect the NNT; time periods when the control and intervention groups diverge the most, such as longer followup times, produce lower NNTs. Ideally, NNTs are presented with 95% confidence intervals.

Note that classically, the NNT is calculated relative to a placebo group. However, some studies are purposefully designed without a placebo group for ethical or other reasons. Studies in Table 2.1.2.3 without a placebo arm are the MTA study, the Brent 1997 study, and the Kazdin 1992 study. Consider, for example, the MTA study, and the calculated NNT of 3 in the medication arm. This means it would be necessary to treat about 3 children with the carefully crafted medication management strategy in order to get 1 to reach the defined threshold for improvement who would not have reached that threshold if he had been treated in the community treatment arm. These studies, therefore, set an upper bound on the true NNT: the NNT relative to a true placebo, the formal definition of NNT. Therefore this column is indicated NNT ≤. The only exception to the upper bound rule is the study of PMT and PSST where no change is assumed as a control in the calculation, based on the natural history of the disorder.

The number needed to harm (NNH) is a useful measure of the frequency of undesired consequences from a treatment, and is calculated the same as the NNT, but based on the proportion of patients with the undesired consequence in the intervention group compared to the comparison group. For example, in the TADS study, 3.7% of the fluoxetine alone group had either an elevated mood, mania, or hypomania during the study, versus l.78% of the placebo group (33). To put this number in context, however, it is important to note that adolescents with known bipolar disorder were excluded, and so were adolescents hospitalized for dangerousness to self or others within three months of consent or were deemed “high risk” for suicide. (The NNH regards a mood “switch” for fluoxetine in this sample is therefore 53.) In the same study, the adverse events of irritable or depressed mood including a worsening of depression or irritability or hypersensitivity or anger occurred in 4.6% of the fluoxetine group and. 9% of the placebo group, generating an NNH of 22. One or more adverse events occurred in 18.3% of youth in the fluoxetine arm and 8.0% in the placebo group, for an NNH of ten for at least one adverse event.

Both NNT and NNH are ideally stated with 95% confidence intervals. In practice, few parents wish to hear technical discussions of sample size and confidence intervals. Nevertheless, the issue of confidence intervals can be included with verbal statements like “this has been studied so well we are quite certain about this” if the confidence intervals are narrow. Alternatively, for large confidence intervals, the clinician may state something like “the studies so far have not been definitive, so there’s a big range of possible answers once this gets studied in more detail.”