Because special education requires strict individualization of student goals for the Individual Education Program (IEP), one critical challenge to educators and to researchers is how to assess progress. For educators, the challenge is the appropriateness of using standardized curriculum based assessments to assess progress of children with specialized instructional goals. For researchers, the challenge is to find a tool that can measure progress when each student has different goals, and therefore different milestones for what constitutes success when conducting group comparison studies. The standard procedure, to use nomothetic assessments, which assumes that all members of the group can be measured equivalently on a test and that expected outcomes for all members are similar, is not appropriate for special education or for COMPASS outcomes. That is, a standardized, norm-referenced outcome measure, such as language skills, is not consistent with the demands of special education that each student’s curriculum be individualized through the IEP. For example, even children who all have a communication goal, may not have the same communication goal to increase verbal language or sign language or picture symbol usage. Moreover, despite their common use in research for young children with ASD (Reichow 2012), broad constructs like language or IQ are relatively insensitive measures of change since improvement in these areas is typically associated with very young children rather than school-age children. Improvements in standard scores are not anticipated or appropriate IEP goals. IEP goals are personalized and focus on specific changes in behavior that are largely not captured by standard group-based or nomothetic tests. Accordingly, because success in special education, and thus with the COMPASS intervention, is measured by the achievement of student-specific IEP goals, we needed a measure of individual goal attainment as our outcome measure for competence. In creating such a measure, we asked the following questions (see below): (a) Could we develop a measurement approach that was useful for monitoring and assessing IEP goals that were different from child-to-child?; (b) Could we demonstrate that the measure was sensitive in detecting COMPASS outcomes?; and (c) Could we show that the measurement approach also was able to provide improvement and absolute scores that were comparable across individuals and groups, e.g., comparable between control and experimental group participants?

Specifically, as already mentioned, evaluation of the outcomes of special education programs at the student level requires an individualized measure of change, since we are measuring different outcomes for each student. Further, all students start at different baseline levels and require different intervention strategies. The question then is how do we measure the effectiveness of COMPASS when outcomes are different, baseline starting levels are different, and treatment plans are different? As noted above, our answer was to use a sensitive, idiographic approach for measuring individual student outcomes that also ensures psychometric equivalence for group comparisons. Idiographic refers to an individualized assessment approach. Idiographic assessment approaches are not suitable for outcomes that are based on standardized tests, like IQ or other norm-referenced tests. Typically, norm-referenced outcomes, or nomothetic approaches, have been favored in experimental studies because they represent quantitative data collected as mean scores that are readily applied in statistical designs. To answer our questions, our need for an idiographic methodology required that we adapt an individualized approach so that it would be suitable for quantitative data analysis similar to nomothetic approaches. Accordingly, we created a psychometric-equivalence tested approach to goal attainment scaling (PET-GAS) as an outcome measure. Goal attainment scaling is an ideal idiographic method that has been around for decades (Cytrynbaum et al. 1979). It also lends itself well to monitoring IEP goal accomplishment. Figure 3.2 shows a GAS template that is used for establishing goal benchmarks. Although GAS is readily accepted as a technique for measuring individual goal progression, before we could readily adopt and apply the use of GAS for our research, we had to address concerns expressed by researchers who questioned its validity for group design research (Schlosser 2004). Researchers worried that goals set for different individuals may not be equivalent. Some may be harder than others, some may be easier. Similarly, there was concern that progress rating steps may not be equivalently difficult for different individuals. Thus, differences in goal attainment between individuals may not reflect differences in the treatment they received, but differences in how hard or easy it was to achieve the goals.