Effect of Stimulant Medication on Children with Attention Deficit Disorder: A “Review of Reviews”

The University of California, Irvine ADD Center recently conducted a synthesis of the literature on the use of stimulants with children with attention deficit disorder (ADD), using a unique “review of reviews” methodology. In this article, we compare three reviews from each of three review types (traditional, meta-analytic, general audience) and illustrate how coding variables can highlight sources of divergence. In general, divergent conclusions stemmed from variations in goal rather than from variations in the sources selected to review. Across quantitative reviews, the average effect size for symptomatic improvement (.83) was twice that for benefits on IQ and achievement measures (.35). A summary of what should and should not be expected of the use of stimulants with ADD children, derived from the literature synthesis, is provided.

o An enormous scientific literature exists on the use of stimulant medication to treat children with attention deficit disorder (ADD).We provide an overview of the efforts of the University of California, Irvine (UCI), ADD Center to perform a research synthesis to locate this literature, describe it, and distill the panorama of findings and recommendations down to a manageable set of generalizations.Because of the number of excellent reviews on the effects of stimulants on chi1-154 dren with ADD, the UCI ADD Center decided that yet another review would have limited impact.The state of the literature called for a "review of reviews" of original work published between 1937 and 1993, instead of the usual review of published articles reporting original research.
We first describe our strategy ofreviewing existing review articles.Next, we compare three reviews selected from each of three different categories: (a) traditional narration descriptive OctoberlNovember 1993 reviews, (b) meta-analyses, and (c) general public reviews.Then we present conclusions that can be drawn from more recent reviews of research on stimulant medication and discuss limitations in the literature and current critical issues.(For our final report, see Swanson, 1993.)METHODOLOGY Our methodology was complex and is described in detail elsewhere (Swanson, 1993).Briefly, we adapted integrative review methodology described by Cooper (1989), and we used a liberal definition of a "review" paper.We combined a list of reviews located by "invisible college" (expert) recommendations and reviews located by an "ancestral" search of their citation lists with a list of reviews located by a simultaneous parallel computer search of four electronic databases in medical (Medline), educational (ERIC), psychological (PsycInfo), and government (GPO) publication arenas.
The electronic search identified 183 reviews.The expert search identified 245 reviews.Of the combined 428 retrievals, only 87 reviews were identified by both search strategies, resulting in 341 unique reviews with a search-strategy overlap of26% (87/341).The electronic search failed to identify 158 reviews located by the primary approach (reviews and citations derived from the "invisible college" of experts).This represented 46% of the dual-search total and 64% of the expert-search total.The primary approach failed to identify 96 reviews that were retrieved by the electronic search, and this represented 28% of the dual-search total and 52% of the electronicsearch total.Thus, the dual-search strategy was important for defining a broad review literature to be reviewed.Either of the single-method strategies would have missed a significant percentage of the 341 sources.Psychologists and physicians then evaluated each of the review documents on a list of variables, including several derived from Cooper's (1988) taxonomy of reviews.As we demonstrate here, differences in the taxonomy variables of Focus, Goal, Perspective, Coverage, Organization, and Audience often were associated with differences in the reviews' conclusions.

TRADITIONAL REVIEWS
Three influential reviews using traditional narrative-descriptive methodology were published in the late 1970s: Whalen and Henker (1976), Bark-Exceptional Children ley (1977), and Adelman and Compas (1977).The traditional reviews show a surprising consensus about the actual effects of stimulant medication on children with ADD.First, they agreed that in a majority (about 75%) of cases, treatment with stimulant medication produces immediate and dramatic positive changes in parent and teacher perceptions and improvements in performance on tests requiring concentration and attention.Second, they acknowledged that placebo and expectancy effects, as well as pharmacological effects, contribute to the perceived positive effects of stimulants on children.Third, the reviews agreed that the short-termperceivedpositive change cannot be predictedby premedicationphysiological or psychological profiles of the children being treated Fourth, they agreed that the documented effects of stimulant medication on longterm adjustment (academic achievement or prosocial behavior) are negligible.
Despite this agreement on the effects of stimulants on children with ADD, these three reviews had different conclusions and recommendations.Adelman and Compas (1977) rejected the clinical practice of using stimulant medication.Barkley (1977) supported established clinical practice while acknowledging the lack of demonstrated long-term effects.Whalen and Henker (1976) also supported the use of medication, based on behavioral effects from well-controlled studies that could not be dismissed as experimental artifacts, but with some qualifications due to attributional effects.
We can provide some explanation and reconciliation of the apparent discrepancies of these three reviews.First, the three reviews were based on different samples of the literature.The review that challenged the established clinical practice of using stimulant medication- Adelman and Compas, (1977)-reviewed only 34% (54/158) of the articles referenced by Barkley (1977) and only 37% (42/113) of the articles reviewed by Whalen and Henker (1976).However, the overlap of the literature reviewed by the two reviews that supported the established clinical practiceby Barkley and by Whalen and Henker-was not much different.Barkley's review included 34% (38/113) of the articles referenced by Whalen and Henker and 38% (54/143) of the articles referenced by Adelman and Compas (1977); and Whalen and Henker's review included 24% (38/158) of the articles referenced by Barkley and 29% (42/143) ofthe articles referenced by Adelman and Compas.
Second, because of an inconsistent use of multiple diagnostic labels in vogue at the time of the reviews, the three reviews emphasized different subgroups of subjects.Barkley (1977) used the labels hyperactivity and hyperkinesis without further description of the disorder; Whalen and Henker (1976) used the label hyperactivity (but pointed out that this was an "unfortunate misnomer" because inattention, impulsivity, and emotional lability accompany high activity levels); and Adelman and Compas (1977) used the term learning problems (but emphasized the multiple labels used in the literature they reviewed).
Third, the conclusions of the reviews emphasized different aspects of outcome.Adelman and Compas (1977) emphasized long-term outcomes on measures of learning and performance, Barkley (1977) emphasized the short-term outcomes on measures of manageability, and Whalen and Henker (1976) emphasized outcome measures of attribution.
Last, we concluded that despite a consistent view of the effects of stimulant medication on children, these three reviews differed in their conclusions and recommendations primarily because ofdifferences in their goal.The stated goal of Adelman and Compas (1977, p. 377) included criticism from a particular point of view (skepticism regarding drug treatment).Barkley (1977, p. 137) and Whalen andHenker (1976, p. 1113) stated more neutral goals, although the latter authors also included analysis from a sociocognitive perspective.
META-ANALYSES Cooper (1989) observed that the traditional review process typically lacks analytical precision because of biases associated with a reviewer's idiosyncratic perspective, failure to assess the size of the effects reported by studies reviewed, and imprecise combination of the volume ofevidence available across the studies reviewed.According to Cooper, meta-analysis offers a method that avoids some of the problems of the traditional "literary summary."In the early 1980s, three meta-analyses were published, based on the literature on short-term effects of stimulants on "hyperactive" children: Kavale (1982) published a meta-analysis in the educational literature; Ottenbacher and Cooper (1983), in the medical literature; and Thurber and Walker (1983), in the psychological literature.Thurber and Walker performed the simplest analysis by emphasizing an 156 overall drug-placebo effect size in a small number of existing studies free of methodological flaws.Kavale performed the most complicated analysis by evaluating the impact of a number of subject (Le., diagnosis, age, etc.) and design (i.e., control, blinding, etc.) variables to evaluate their impact on the drug-placebo effect size.Ottenbacher and Cooper asked the most specific question: They formulated effect sizes based on three comparisons (drug-control, drug-placebo, and placebo-control) to evaluate evidence of a "placebo effect" on outcome.
These three reviews appeared at about the same time, used a similar methodological approach, and addressed the same general topic.Despite these similarities, the conclusions of the three meta-analyses were quite different.Kavale's (1982) conclusions support the clinical practice of using stimulants.The overall effect of the use of stimulants was described as significant and not due to methodological weaknesses in the research.The positive effect on learning was considered to be verified, and the placebo effect was described as negligible.Thurber and Walker's (1983) conclusions offered weaker support for clinical practice and dismissed the small magnitude of this treatment effect on learning.Ottenbacher and Cooper (1983) agreed with Kavale in terms of the overall positive medication effect on the behavior and performance of children identified as hyperactive, but clearly disagreed about the drug effect in two areas.They considered the medication effect on academic performance to be small, and the placebo effect in the drug-versus-control conditions to be large.
We attempted to reconcile these apparent discrepancies in the three meta-analyses.We found that the actual effect sizes reported in the metaanalyses were similar.Because the three metaanalyses reported effect sizes for different classes ofdependent measures, selective comparisons were necessary to achieve an equivalent dependent variable for cross-review comparisons.For this purpose, we used two classes of effect size: the average effect size based on ratings of behavior and measures of attention (which individual comparisons in the meta-analyses indicated did not differ) and the average effect size based on measures of academic variables (IQ and achievement tests).In all three meta-analyses, the estimated effect on behavior is much larger than the estimated effect on achievement.Table 1 shows patterns extracted from these three meta-analyses.
OctoberlNovember 1993 The difference between the Kavale (1982) and Ottenbacher and Cooper (1983) estimates of effect size for the "placebo effect" remains unexplained.Ottenbacher and Cooper specifically included a contrast of placebo versus control, which yielded an effect size of .32,or about 30% of the reported drug-versus-placebo effect size.Even though this comparison was not a major part of the Kavale meta-analysis, an estimate of the placebo effect was small (an effect size of .07,or about 3% of the reported drug-versus-placebo effect size).Our measure of percentage overlap of references was low for these two meta-analyses, despite the stated purpose of each to review the literature as it existed in 1980.These two meta-analyses had only six references in common, which represents about 7% of the 86 references reviewed by Ottenbacher and Cooper and about 8% of the 74 articles reviewed by Kavale.Apparently, the placebo-control comparisons from these two meta-analyses were derived from different studies.Because Ottenbacher and Cooper specifically addressed the issue of a placebo effect, and the magnitude of the placebo effect they reported is consistent with the report of an important "literary summary" (e.g., Barkley, 1977, reported about a 30% placebo response rate), we accepted Ottenbacher and Cooper's conclusion that a significant and large placebo effect is manifested in the short-term response to stimulant medication.

REVIEWS FOR THE GENERAL PUBLIC
One of our coding variables from Cooper's (1988) taxonomy was Audience.We noted that most reviews that targeted "General Public" as the audience had "Criticism" as a goal.We also Exceptional Children noted that the profession of the first author of most reviews for the general public was usually different (i.e., the first author was a journalist) from that of the first author of reviews for "General or Specialized Scholars" (i.e., the first authors were educators, physicians, or psychologists).We present an analysis of three well-known reviews for the general public to isolate the reason(s) for criticism of the clinical practice of treating children with stimulants.
One of the first reviews for the general public was published as a book chapter by Schrag and Divoky (1975).Over 10 years later, another review for the public was published as a book chapter by McGuiness (1989).Recently, a review for the public was part of a magazine article by Kohn (1989).The goals and conclusions of these reviews were quite similar.
The goal of the Schrag and Divoky (1975) review was to expose the "myth of the hyperactive child" and to show the inappropriateness of "chemical intervention as a legitimate solution to the classic problem of controlling and making acceptable the behavior of children who annoy teachers, upset classroom routines, or otherwise fail to conform to adult expectations" (p.71).They acknowledged that, judged by parent or teacher reports, "The drugs clearly worked for a certain percentage of children: There was no shortage of positive reports, and there would be no shortage of people suggesting or demanding medication, and no shortage of doctors or clinics ready to prescribe" (p.107).Despite that admission, their main conclusion was that treatment of children with stimulants was viewed as behavioral control and thus was considered unacceptable, for political and social reasons.
The goal ofthe McGuiness (1989) review was to pose critical questions about the link between diagnosis of hyperactivity and treatment with stimulant drugs, which if answered negatively were assumed to offer "a powerful argument for abandoning our current practices of diagnosis and treatment " (p. 196).McGuiness noted that the literature has reported many clear effects of medication, but emphasized the consensus view that these effects were not specific to hyperactive children or to any specific physiological or neurological abnormality.McGuiness concluded that 25 years of research had failed to "pin down a disorder that does not exist" and that "drugs do not work.They help the teachers and parents, but they do nothing for children" (p.229).
The goal of the Kohn (1989) review was to question the impact of the emphasis among researchers on biological causes and explanations of hyperactivity.Kohn stated: "The most striking consequence of assuming that an unusually distractible or impulsive child is suffering from a disease is the tendency to tum to medication to solve the problem" (p.96).Kohn acknowledged that in most children stimulants are efficacious, but stated: Drugs do absolutely nothing for 25% to 40% (depending on whose estimate you trust) of hyperactive children....A large proportion of children who do respond to Ritalin also improve on a placebo.... Drugs do nothing to enhance actual academic achievement ... the effect is a temporary suppression of symptoms, not a cure (p.98).
Kohn, questioning why Ritalin is used in the first place, suggested that the drug "may have much greater relevance for stress reduction in caregivers than intrinsic value to the child" (p.98).
These reviews for the general public, with criticism as a goal, make the following points: (a) the literature does not support the notion of a paradoxical response of hyperactive children because normal children show increased attention and decreased undirected movement in response to stimulants; (b) the assumption that brain damage (or some other organic cause of the symptoms) predicts response to medication has not been supported in the literature; (c) the stimulants have effects on attention, concentration, or motivation, but no clear effect on academic performance or learning has been documented; (d) long-term effects of treatment with stimulants on adjustment of hyperactive children has not been documented; (e) when stimulants work in the short term, pharmacological intervention may be used as a crutch and may postpone or prevent the use of nonpharmacological intervention, which may be more effective in the long term.

CONCLUSIONS FROM RECENT REVIEWS
The observations made in the reviews for the general public are actually consistent with the conclusions of the reviews discussed earlier (e.g., Whalen & Henker, 1976; Barkley, 1977; Adelman & Com pas, 1977) and with recent reviews intended for different audiences (i.e., "Clinicians and Scholars").Consider the recent review by Jacobvitz, Stroufe, Stewart, and Leffert (1990), 158 which criticized current clinical practice, and the review by Stevenson and Wolraich (1989), which supported current clinical practice.Jacobvitz et al. acknowledged the short-term benefits outlined earlier but focused on the same limitations of stimulants noted by Schrag and Divoky (1975), McGuiness (1989), and Kohn (1989).As a conclusion, Jacobvitz et al. urged "greater caution and a more restricted use of stimulant treatment" (p.685).In contrast, Stevenson and Wolraich acknowledged the limitations outlined here but focused on the temporary suppression of symptoms.As a conclusion, they stated: "Stimulant medications are an effective treatment modality for most children with ADHD" (p.1193).
As part of our "review of reviews," we have extracted issues on which a consensus was expressed across the reviews discussed here (as well as across the larger number of reviews analyzed and discussed in the report of the UCI ADD Center to the U.S. Department of Education in Swanson, 1993).Figure 1 shows some generalizations derived from these reviews.In the top section of Figure 1, we outline the basic pattern of expected benefits that most reviews acknowledge.In the bottom section, we summarize the limitations acknowledged in most reviews.These generalizations are based on the reported effects (or lack of effects) of stimulant medication on children with ADD and do not reflect the interpretation of these effects expressed in the conclusions and recommendations of the reviews.

CURRENT QUESTIONS AND INVESTIGATIONS
As consensus has developed regarding the general effects of stimulant medication on children with ADD (Figure 1), more fine-grained issues have emerged.Recent reviews have pointed out new directions for research into stimulant limitations, effects on different behavioral domains, and interaction of stimulants with psychosocial treatments.The following critical questions have been addressed by recent reviews: 1. What are some boundary conditions that limit the effects of medication on school behavior and performance (Swanson et al., 1992)? 2. Does stimulant medication have an effect on the academic performance of children with ADD (Carlson & Bunner, 1993)? 3. Does stimulant medication have an effect on the aggression manifested by some children with ADD (Hinshaw, 1991)?
OctoberlNovember 1993 the clear beneficial effects of stimulants in children with ADD, but they do put those benefits in their proper perspective.
Academic Achievement Carlson and Bunner's (1993) article examined whether short-term gains can be translated into long-term improvements in academic achievement.They reviewed literature indicating that for children with ADD but without concurrent academic problems, stimulant medication clearly increases practice to a degree that should improve learning.Serious methodological problems in the literature were identified, including lack of random assignment of comparison groups, lack of control of dose or length of treatment with stimulants, and psychometric properties (lack of sensitivity, ceiling effects, etc.).In the absence of any definitive answer about the long-term effects of stimulants on children with ADD, Carlson and Bunner recommended individualized medication evaluations using standardized procedures for administering "real life" academic tasks, the results of which are communicated to the physicians to titrate dose.The Carlson and Bunner (1993) and Pelham et al. (in press) reviews offer descriptions of the state-of-the-art methods for performing a medication assessment using ecologically valid academic measures, as well as behavioral measures.

Aggression
The purpose of the Hinshaw (1991) review was to (a) assess the role of stimulant medication in the amelioration of aggressive behavior, and (b) discuss relevant methodologic, clinical, and theoretical issues that pertain to the role of medication in treating aggressive acts.He challenged the accepted belief that "whereas the core deficits of ADD-which are presumably biologically based-are best treated with pharmacological agents, aggressive behavior requires psychosocial intervention, preferably family-oriented, behavioral treatment" (p.303).He concluded that the literature suggests "small and usually nonsignificant effects of medication in the laboratory or playroom [but large effects in] naturalistic observations of aggression in classroom or outdoor play settings" (p.307).However, Hinshaw (1991) also concluded that any short-term amelioration of aggressive acts with stimulant medication is likely to be counteracted by (a) 160 compliance problems, (b) unmedicated periods in peer and neighborhood environments, and (c) the continuous, stressful interchanges that occur in the lives of these children.

CONCLUSION
We have described the methods we used to perform a "review of reviews," which suggested that a consensus exists about the expected benefits and the acknowledged limitations of stimulant medication in the treatment of children with ADD.We then considered specific reviews, and we discussed the issues that should be addressed in further reviews in this area.One of the most important issues to be addressed is the effect of combined psychosocial and pharmacological interventions.Many reviews ended with a recommendation that stimulants should always be used in combination with educational and behavioral interventions, but we found few references to empirical studies to support this common-sense recommendation.Even reviews of this area (Gadow, 1985;Pelham & Murphy, 1986) differ in their conclusions about the relative impact of behavioral and pharmacological components of multimodality treatment.Our "review of reviews" suggests that major methodological weaknesses or shortcomings in research preclude clear answers to questions about the impact of combined (or "multimodality") treatment on children with ADD.The next phase of research on the effects October/November 1993 of stimulants in children with ADD will address this issue.

TABLE 1 Effect
Sizes of Three Meta-Analyses National Institute of Mental Health (NIMH) Child and Adolescent Research Branch listed 10 topics that should be addressed in future studies of the effects of stimulants on children with ADD: (a) Why have no long-term effects been demonstrated?(b)In the short-term, how many children with ADD are nonresponders?(c)Do high doses impair learning?(d)Does state-dependent learning occur?(e)Do effects depend on age and IQ? (f) Do effects depend on comorbid conditions?(g)Are the effects of different stimulants the same?(h)Do attributions of success to the pill offset benefits?(i)Why are links to biological factors not well established?and(j) Why has length of treatment in most cases been so limited?These issues are being addressed in the NIMH Multisite Multimodality treatment study, now in the protocol development stage(Richters, Ar-