In recent years, practitioners and researchers in educational and psychological settings have compared raw scores from a test to evaluate the effectiveness of some academic instruction or other treatment. Naglieri (2012) illustrated how the comparison of raw scores over time can be misleading when students’ progress over time was monitored using words read per minute as a measure of reading skill. The issue is that some tests of skills, like reading or vocabulary, show a very strong age-to-age progression of raw scores. This progression reflects the typical changes in skills that occur as a child grows older as a combination of factors including maturation, learning from the environment as well as school. In order to calibrate change as a function of some specific instruction or intervention, the amount of change should be calibrated to the normal growth curve, not only the pretreatment level. This is a particularly important issue in treatment programs for individuals with ASD.

Expressive vocabulary is one variable that is often studied as a way to demonstrate improvement over time (e.g., Kasari et al. 2008). The choice of which expressive vocabulary test and score to use, has a profound impact on the result. Using the Kasari et al. study results as a guide, we obtained the standard scores using the normative tables from the Expressive Vocabulary Test (Williams 1997). The results shown in Fig. 3.1 suggest that there were changes in performance from Time 1 through Time 4 (12 month interval). Both treatment groups appear to have higher scores at Time 4. The interpretation of these data could lead to the conclusion that the treatments were effective but examination of the standard scores associated with these raw scores suggests a different conclusion as shown in Fig. 3.2.

When the raw scores are converted to standard scores (mean of 100 and SD of 15) shown in Fig. 3.2, the results suggest that although the raw scores increased over the 12-month interval the standard scores associated with these raw scores actually showed no improvement. That is, even though the two treatment (as well as the control) groups’ raw scores increased, the difference between those scores and the mean for the standardization group remained large. In fact, the average raw scores for the four age groups are 41, 48, 52, and 56. Therefore, we suggest that raw score improvement alone is insufficient to show treatment effectiveness. Standard score improvement provides an additional reference point that must be taken into consideration in order to determine if a treatment is sufficiently effective.

Consideration of the role reliability plays in evaluation of treatment change is essential because all measurements have some degree of error. In classical test theory, an obtained score is comprised of the true score plus error (Crocker and Algina 1986). For this reason, we should always report an obtained score with a range of values within which the person’s true score likely falls with a particular level of confidence. The size of the range is determined by the level of confidence and the reliability of the measurement; the higher the reliability the smaller the range. When reporting a T-score, for example, we state that a child earned a score of 50 (± 7); meaning that there is a 95 % likelihood that the child’s true IQ score falls within the range of 43–50. The range of scores (called the confidence interval) is computed by first obtaining the standard error of measurement (SEM) from the reliability coefficient and the SD of the score in the following formula (Crocker and Algina 1986):

The SEM is considered the average SD (68 % of the normal curve is in this range) of the theoretical distribution of a person’s scores around the true score. If one SEM is added to and subtracted from an obtained score, there is a 68 % chance (the percentage of scores contained within ± 1 SD) that the person’s true score is contained within that range. The SEM is multiplied by a z value of, for example 1.64 or 1.96, to obtain a confidence interval at the 90 or 95 % levels, respectively. The resulting value is added to and subtracted from the obtained score to yield the confidence interval. For example, the 95 % confidence range for a test score with a reliability of .95 and an obtained T-score of 60 (recall that a T-score is set to have a mean of 50 and SD of 10) is 57 (60  –  3) to 63 (60 + 3). It is important to note that the higher the reliability the smaller the interval of scores that can be expected to include the child’s true score. The smaller the range, the more precise the practitioners can be in their interpretation of the results, resulting in more accurate decisions regarding the child. The relationships between reliability and size of the confidence interval is provided in Fig. 3.3 for T-scores (M = 50; SD = 10). Confidence intervals should always be used for interpretation of all scores because they take measurement error into account at a specific level of probability.

The method of computing confidence intervals described above has been modified in recent years by some test authors (e.g., Wechsler and Naglieri 2006) to be theoretically more accurate than the approach of basing confidence intervals around obtained scores (Nunnally 1978). The modification involves centering the confidence interval on the estimated true score. This approach accounts for measurement error associated with the scores to provide a band of error that is centered on the estimated true score and thereby takes into account regression to the mean (Salvia and Ysseldyke 1981). Figure 3.4 shows the relationships between obtained scores from 30 to 70 and the confidence intervals associated with those scores using estimated true score-based method. This is the method used in most ability tests and some rating scales.

The SEM is also particularly important when scores from different raters are compared. When comparing scores earned on the same scale, it is critical to recognize that the lower the reliability the larger the SEM, and the more likely scores will differ as a function of measurement error i.e., the lower the reliability, the more likely there will be differences among scores. Inconsistent results that reflect measurement error can complicate the interpretation of pre- and postfindings, and make a clear understanding of an individual’s treatment progress more difficult to interpret. For example, when a researcher or practitioner is attempting to determine whether the several scores an individual has received are similar or significantly different, the answer to that question is directly related to each score’s reliability because the calculation of the SEM is based on reliability. In fact, the formula for the difference between two scores earned by an individual is calculated using the SEM of each score.

Applying this formula to T-scores, as shown in Fig. 3.5, we see that as the reliability goes down, the differences needed when comparing two scores increases dramatically. This means that scores from measures with reliability of .70 from two different teachers would have to differ by 15 T-score points to be significant at the 95 % level. This means that test scores with higher reliability reduce the influence of measurement error on the different scores. Clearly, in both research and clinical settings, variables with high reliability are needed for precision of measurement, but how are these coefficients evaluated?

Bracken (1987) provided suggested levels for acceptable for test reliability. He stated that scales that comprise a complete instrument should have at least an internal reliability estimate of .80 or greater and total test scales should have an internal consistency of .90 or greater. These guidelines should be further considered in light of the decisions being made. For example, if a score is used for screening purposes where over identification is preferred to under identification, a .80 reliability standard for a Total Score may be acceptable. If, however, scores from a scale contribute to important decisions then a higher (e.g., .95) standard should be deemed more appropriate (Nunnally and Bernstein 1994). We suggest that professionals evaluating the treatment of symptoms related to ASD use scores that have internal reliability estimates of .80 or higher. For scores comprised of several variables that have been combined, an internal reliability estimates of .90 or greater. Clinicians are advised not to use measures that do not meet these standards because there will be too much error in the measurement to allow for confidence in the result. This is especially important because the decisions clinicians make can have significant impact on the life of a child. Therefore, we urge the reader to carefully examine the reliability findings of any measure they choose to use.

We have stressed the need for norm-referenced measurement of symptoms related to ASD as well as the advantages of using measures that have high reliability so that greater accuracy can be achieved. The overarching goal is to use well-developed psychological tools when assessing individuals with ASD and particularly when evaluating the effects of any treatment. We will now illustrate how these measurement issues can be operationalized with a rating scale. Although all the issues related to reliability are well established, the issues surrounding psychometrics of treatment effectiveness are much more complex. There are many questions and an evolving set of possible solutions. Typically, researchers have studied both global changes in symptoms associated with a particular condition as well as specific behaviors, some of which may or may not be associated with a particular condition. In clinical practice, treatment goals are rarely set at the disorder level, instead, the focus is typically on general symptoms (e.g., improve peer socialization) and specific behaviors associated with a general symptom (e.g., increase ability to initiate conversation with peers).

In this section, we present a way to evaluate symptoms related to ASD on both global and specific levels, identify areas for treatment, and evaluate the effects of treatment. To do so, we will illustrate using information from the ASRS (Goldstein and Naglieri 2009). We choose to illustrate using this tool because it is nationally normed and provides several different types of global scores as well as measures of specific behaviors. In addition, the reliability of the scales is well documented and guidelines for assessing treatment change are also provided. We will begin with a brief explanation of the ASRS and then describe the steps needed to determine the current status of the individual who was rated, which scales and individual behaviors warrant attention, and how to assess treatment effectiveness.