Dependent variable change
Levels of integrity
High
Low or none
Desired direction
Confidence that the treatment package has an effect
No confidence that the treatment package has any effect
Increased risk of making a type I error (false positive) if treatment integrity data are not collected
No change
Confidence that the treatment package has no effect
No confidence that the treatment package has any effect
Increased risk of making a type II error (false negative) if treatment integrity data are not collected
Undesired direction
Confidence that the treatment package has no effect and may even be potentially harmful
No confidence that the treatment package has any effect
Increased risk of making a type II error (false negative) if treatment integrity data are not collected
In other conditions, the lack of treatment integrity coupled with no changes (e.g., a lack of an increase in social skills) or undesired changes in the dependent variables (e.g., an increase in stereotypical behaviors) may lead practitioners or researchers to conclude that the procedures were not effective. Procedures that are not effective should clearly be discontinued; however, it is possible that the treatment, had it been applied with integrity, might have been effective in that instance (in the field) or for all children evidencing a particular profile (in the research literature). Rejecting an intervention when it may actually be effective is considered to be a type II error. A lack of treatment integrity in these conditions would hinder the identification of potentially effective treatments.
Methods to Increase Treatment Integrity
The most commonly reported method to increase treatment integrity is performance feedback (PFB; Codding et al. 2005, 2008; DiGennaro et al. 2005, 2007; DiGennaro-Reed et al. 2010; Mortensen and Witt 1998; Mouzakitis 2010; Noell et al. 1997). Performance feedback typically consists of a meeting between a treatment agent and a supervisor, evaluator, or consultant. During this meeting, a variety of information can be shared. Feedback and praise can be delivered on the amount of correctly implemented components. The treatment agent and the observer can also discuss aspects of a plan that were not followed. Finally, some training method can be employed to ensure correct component implementation in the future. The failure to implement a plan with integrity may be due to potential skill deficits or a lack of fluency or automaticity. At times, the treatment agent may simply forget to implement all the steps of the intervention or they may have begun the process of drift. PFB is a method that can allow these issues to be addressed via the use of review, modeling, rehearsal, and role-play, if needed. A typical PFB session can last anywhere between 5 and 20 min (Reed and Codding 2011), with initial PFB sessions lasting much longer than later sessions.
Various components of PFB have been manipulated to examine how to make the process more efficient and effective. For example, Guercio et al. (2005) varied PFB private meetings with public postings of treatment integrity to train 30 staff members at a residential facility. Although the results of the study showed dramatic increases of integrity among all staff, it is unclear which PFB condition was superior. The amount of time between the observation period and the delivery of PFB has also been investigated. Noell et al. (1997) delivered PFB immediately after observation, while Codding et al. (2005) delivered PFB every other week—others have examined varying lengths of time in between. PFB is an effective way to increase treatment integrity, despite its distance from the initial observation; however, stronger, faster increases of treatment integrity were associated with shorter time lapses (Mortenson and Witt 1998).
While PFB has been demonstrated to be effective, investigations into the removal of this intervention evidence decreases in levels of treatment integrity (Noell et al. 1997; Witt et al. 1997). In order to deal with this issue, the process of fading (also referred to as systematic fading or dynamic fading) is recommended (Reed and Codding 2011; DiGennaro et al. 2005; Noell et al. 2000). Fading refers to the gradual decrease of PFB (i.e., thinning) over time that is contingent upon the demonstration of treatment integrity at specified criterion levels. For example, if a treatment agent is receiving a daily schedule of PFB and demonstrates integrity rates of 90 % or better for three consecutive observation sessions, then the schedule is thinned to once every other day.
Some investigation has been conducted into the essential components of PFB. While PFB is a procedure employed to ensure treatment integrity, PFB itself must be scrutinized for treatment integrity. Some have indicated that the essential components of PFB are praise and corrective feedback (Mouzakitis 2010). Corrective feedback refers to the process of delivering feedback on components that were incorrectly applied (or not applied at all) and the provision of training procedures to help correct skill deficits or improve automaticity. However, DiGennaro et al. (2005) conceptualized the PFB process as one that was aversive. In this conceptualization, treatment agents worked to obtain high rates of integrity in return for the removal of PFB. This is in contrast to Codding et al. (2008), in which the treatment agents rated the PFB process as rewarding and beneficial. However, these discrepant results can be due to the setting (e.g., an inner city private school vs. a suburban public school), the person delivering PFB (e.g., a university faculty member vs. agency supervisor), how PFB is used by the setting (e.g., as a teaching tool or as a way to evaluate staff dismissal), and perhaps even the personality characteristics of the individual delivering PFB himself or herself.
Self-monitoring as an intervention has also been investigated as a procedure to improve treatment integrity. Self-monitoring as a procedure would be attractive because it would decrease the reliance on other individuals observing and intervening with treatment agents, thus saving time for staff and resources for the agency as a whole. Self-monitoring as an intervention to improve treatment integrity shows some good results (Richman et al. 1988; Coyle and Cole 2004) and more rapid increases when paired with environmental prompts (Petscher and Bailey 2006; Burgio et al. 1990); however, overall, these results do not approach the speed and total amount of improvement that the PFB procedure offers.
Conclusions
Issues related to treatment integrity are of critical importance to treatment programs designed for children with ASD. Given the course and symptomology of the disorder, treatments targeting ASD tend to be intricate, and often have deep roots in theory that may not necessarily be apparent to many treatment agents. Given these issues, there needs to be considerable work to ensure treatment adherence, improve competence, and establish differentiation. Unfortunately, treatment integrity is an important construct that is not measured as often as it should be in both research and practice.
The recent interest in the application of EBI’s to treat ASD has the potential to increase awareness and interest in treatment integrity. Detrich (2008) suggests that environmental factors (such as the agency or stress levels of the family in the home) may play a considerable role in the selection and implementation of EBIs, to the point where various pieces of interventions might be combined to form unique treatment plans. While this may appear to be intuitively attractive to the clinician, the process does not necessarily equal a “mix-and-match” strategy—on the contrary, practitioners will need to work much harder in defining the treatment (i.e., independent variable), as well as the treatment outcomes (i.e., dependent variable) and a measurement strategy. This newly developed treatment protocol will need to be assessed for treatment integrity, so that agents can make an informed decision as to the effectiveness of the treatment.
Over the last 30 years, there has been an ever-increasing focus on the measurement of and interventions to improve treatment integrity. Direct observation and PFB appear to be the most commonly used (and most successful) measurement and assessment strategy. Attempts have been made to examine components of PFB to see how the process can be improved; however, it would be helpful to investigate what types of situations hinder PFB. For example, it is within the authors’ clinical experience that observations conducted by external individuals tend to be better received than those conducted by administrators or supervisors. This may partially explain some of the discrepancies in the field, but as of yet, there have been no investigations of the status of the observer upon the effectiveness of PFB.
In conclusion, the demonstration of treatment integrity within the context of the evidence-based movement in identifying interventions for ASD will serve to be a challenge that needs to be dealt with both in scientific literature as well as in practice. The level of treatment integrity adds another interpretative layer that deepens inferences made from outcome data. Ultimately, efforts to improve treatment integrity serve to develop better researchers and professionals that can make a difference in the lives of children diagnosed with ASD.
References
American Psychological Association. (2005). Policy statement on evidence-based practice in psychology. http://www.apapracticecentral.org/ce/courses/ebpstatement.pdf Accessed 15 Oct. 2012.
American Speech-Language-Hearing Association. (2010). Code of ethics [Ethics]. doi:10.1044/policy.ET2010-00309.
Barber, J. P., Gallop, R., Crits-Chirstoph, P., Frank, A., Thase, M. E., Weiss, R. D., et al. (2006). The role of therapist adherence, therapist competence, and alliance in predicting outcome of individual drug counseling: Results from the National Institute Drug Abuse Collaborative Cocaine Treatment Study. Psychotherapy Research, 16, 229–240. doi:10.1080/10503300500288951.CrossRef
Burgio, L. D., Engel, B. T., Hawkins, A. M., McCormick, K., Schieve, A, & Jones, L. T. (1990). A staff management system for maintaining improvements in continence with elderly nursing home residents. Journal of Applied Behavior Analysis, 23, 111–118. doi:10.1901/jaba.1990.23-111.PubMedCrossRef

Stay updated, free articles. Join our Telegram channel

Full access? Get Clinical Tree

