Evidence of implementation quality includes what transpires between consultant and consultee. Fidelity is a critical aspect of implementation/intervention quality. The most common definition of fidelity is the degree to which a given intervention is implemented as intended (Moncher and Pribnz 1991; Yeaton and Sechrest 1981). Recall that fidelity is a critical aspect of intervention quality. In education, fidelity measurement has been typically assessed using two broad dimensions described as (a) structural fidelity and (b) process or procedural fidelity (Gersten et al. 2005; Odom 2009; O’Donnell 2008). Structural fidelity refers to the organizational/structural elements that characterize and describe the intervention and are somewhat dichotomous in nature in terms of measurement, in that they occur or don’t (e.g., IEP meeting occurred, goals were set, teacher meets daily with student, consultant met with teacher four times during the year) (Durlak and DuPre 2008; Gersten et al. 2005). On the other hand, process or procedural elements refer to the measurement of the actual behaviors of the teacher or intervener and tend to occur along a continuum (teacher employed joint attention in interaction). Both impact quality and are part of its measurement, and we applied both types of measures in our studies.

Initially we focused on what occurred during the first step of COMPASS, the opening consultation. Also recall in the EBPP framework the importance of clinical decision-making in integrating the overlapping areas of the EBP, the characteristics of the child (preferences, strengths), and the characteristics of the teacher (training, knowledge). Step one of COMPASS, the initial consultation, provides the platform for this clinical decision-making to occur. In the chapters that follow, we will review what we have learned about the intervention practice and practice outcomes. First, as shown in our Integrated Model, we examined how the implementation practice influences the intervention practice directly and also indirectly as a result of continuous performance monitoring and feedback from the practice outcomes.

As discussed in earlier chapters, the initial consultation is a 3-h goal-directed, parent-teacher, activity with three specific aims: (a) to develop a shared understanding of the child’s personal and environmental supports and challenges using the COMPASS competency framework (Ruble and Dalrymple 1996, 2002); (b) to use this information for identifying and generating high quality personalized teaching goals; and (c) to develop personalized teaching plans based on the information obtained in steps a and b for each goal. To guide our analysis of the quality of the initial consultation, we had the following questions: (a) Could we deliver high quality consultation reflected by positive exchanges between parents, teachers, and consultants? and (b) Do parents and teachers report that the initial consultation was high quality and that they were satisfied? The first question examines consultant quality using elements non-specific to COMPASS and thought to underlie good consulting generally, the second question examines quality in terms of adherence to specific desired elements of COMPASS.

Can we deliver high quality COMPASS consultation? Recall that the quality of the consultant-teacher interaction is a function of factors characteristic of general good consulting practice and of factors specific to COMPASS. Research on general elements of good consultation shows that effective processes and strategies used during consultation are based on collaboration, mutual respect, and parity among all team members (Brown et al. 2011). When consultees have input into identifying the problem and goals, they are more likely to be engaged throughout the entire process. This is critical because the real work happens once the consultant leaves. To understand how well the COMPASS consultants engaged teachers and parents as part of the process during the initial consultation, we evaluated a general element of good quality consultation or implementation quality—communication. Specifically, we evaluated the communicative exchanges that occurred during the critical stage of discussing the child’s COMPASS profile with the aim of developing a shared understanding of the child’s personal and environmental challenges and supports. Although the quality of communicative exchanges is important throughout COMPASS consultation and coaching, we focused on this first stage-setting activity to formally evaluate communication quality using an objective methodology. During the stage-setting activity, the COMPASS profile is reviewed by the parent and teacher and covers nine core areas of learning and development observed at home, in the community and at school: (a) likes, strengths, frustrations and fears, (b) adaptive skills, (b) problem behaviors, (c) social and play skills, (d) communication, (e) sensory preferences and dislikes, (f) learning skills, (g) environmental supports, and (h) environmental challenges. An example of the social and play skills assessment form used to guide discussion of the child’s skills at home and at school, with adults and with children is provided in the Fig. 5.2. Our manual on COMPASS provides transcribed examples of the discussion during this activity and also detailed case studies (Ruble et al. 2012).

Using data from our first randomized controlled single-blind trial of COMPASS (Ruble et al. 2010), we analyzed the verbal interactions between the consultant, teacher, and parents during the initial consultation expressed as speech acts and speech act exchanges (Ruble et al. 2011). Speech acts were defined as ‘‘a phrase or utterance, bounded by intonation, pauses, or grammar’’ (Sheridan et al. 2002, p. 311). Speech exchanges, on the other hand, represent the impact of a preceding speech act on the subsequent speech act among the different participants in the consultation. We applied the Psychosocial Processes Coding Scheme (PPCS) developed by Leaper (1991) which allowed us to examine the reciprocal influence and conversational intent and function of participants’ speech acts. Three types of speech acts identified were: (a) affiliative or positive speech acts, (b) distancing or negative speech acts, and (c) mixed, a combination of positive and negative speech acts. The speech acts were then transformed into one of three speech exchanges: affiliative, distancing, or mixed. An affiliative speech exchange, involves interactions in which one speaker’s collaborative or obliging speech act is followed by another speaker’s collaborative or obliging speech act (e.g., a consultant’s statement ‘‘This sounds like an issue related to lack of understanding of perspective taking’’ followed by a teacher’s statement ‘‘I agree’’). The second type of speech act exchange, distancing, involves interactions in which one speaker’s controlling or withdrawing speech act is followed by another speaker’s controlling or withdrawing speech act (e.g., a consultant’s statement ‘‘You really should do this instead …’’ followed by a teacher’s statement ‘‘I don’t agree with that…’’). The final speech act exchange, mixed, involves exchanges in which one speaker’s affiliative statement is followed by another speaker’s distancing statement, or vice versa (e.g., a consultant’s statement “Let me understand what you are saying” followed by a parent’s statement “No, I don’t think that you do understand”).

The three categories of speech acts are not equally desirable; affiliative acts are seen as ideal and distancing acts are not. Therefore, we hoped to find more affiliative acts and fewer distancing ones. We had a basic research question: Does COMPASS facilitate good consultant-teacher communication indicative of good quality as shown by high numbers of affiliative acts and low numbers of distancing ones? To answer our question, we analyzed a total of 13,826 speech acts and 9,310 speech exchanges from 18 COMPASS consultations. As we had hoped, and consistent with our research hypothesis that consultant-teacher communication would be high quality, the overwhelming majority of speech act exchanges were affiliative (93.6 %). Very few speech acts were coded as withdrawing and controlling (<2 % each). Intercorrelation analysis showed that speech exchanges were significantly associated with one another in expected ways. Affiliative speech exchanges were negatively associated with both distancing and mixed exchanges, r = −0.63, p < 0.001, and r = −0.99, p < 0.001, respectively, whereas distancing and mixed exchanges were positively associated, r = 0.57, p < 0.001. Thus, these analyses were able to show that, as hoped, consultants’ behavior was judged of high quality, as measured using general indicators of good communication.

We also wanted to know how parents and teachers perceived the initial consultation and if they report that we implemented the initial COMPASS consultation with good quality. Specifically, did we implement COMPASS with high fidelity and were parents and teachers satisfied? Analysis from study 1 showed that fidelity and satisfaction were high (Ruble et al. 2010). Teachers reported 96 % of the elements of COMPASS were implemented. For satisfaction, they reported a mean score of 3.7 from a total of 4, indicating high satisfaction. These results together indicate that the initial consultation was provided with high quality as measured using indicators specific to COMPASS. In study 2, we repeated the measures and replicated the findings. For fidelity, teachers reported that 92 % of the elements were implemented. Further, for satisfaction teachers reported a mean score of 3.6 out of a total of 4, indicating high satisfaction. Thus, we had consistent evidence of high quality implementation.

Next, we wondered if there was an association between our two measures of COMPASS consultation implementation quality, thus, we correlated scores from the COMPASS fidelity checklist with scores from the teacher satisfaction measure and found a significant correlation between the two (r = 0.41, p = 0.03). The correlation or overlap between the measures of quality provides evidence of convergent validity, which is one of the key indicators of construct validity. That is, our two measures were assessing related but also slightly different aspects of the same construct, consultant quality.

These findings led to a different question that emerged from our Integrated Model (Fig. 5.1). Specifically, we asked whether good quality implementation of the initial COMPASS consultation had any direct impact on intervention practice, i.e., teacher behavior? To answer this question, we correlated affiliative speech exchanges with one of the immediate expected outcomes from the initial consultation—IEP quality. Within the COMPASS model, IEP quality is a measure of intervention quality, in that improved IEP quality indicates that the teacher changed the IEP based on recommendations made following the consultation. Although we provide more description in the following chapter, briefly, IEP quality expected to change as a result of COMPASS was measured by evidence of best practice goals, as indicated by inclusion of goal targets recommended by the literature (i.e., a social goal, a communication goal, and independent work behavior goal) and by evidence that goals were well specified as indicated by their clarity, objectivity, and measurability. These were the targeted IEP elements we expected to change as a result of COMPASS. That is, we purposely identified a goal for each of the recommended domains and worked with the teacher and parent to make sure goals were clear, observable, and easily measured. Our results showed a direct and positive correlation between IEP quality and affiliative exchanges (r = 0.51, p < 0.008) and negative correlations between IEP quality and both distancing (r = −0.49, p = 0.03) and mixed speech exchanges (r = −0.49, p = 0.03). That is, the style and character of the communication between participants during the initial consultation was strongly associated with the quality of the goals generated. An affiliative, positive communication style was more likely to create a consultation environment leading to the production of high quality goals, than a communication style that was distancing or did not use predominantly affiliative exchanges. Good communication is one characteristic of a good working alliance and in this respect, these data are consistent with the general literature on the importance of therapeutic alliance on treatment process and outcomes.