PCBH Program Evaluation

and Jeffrey T. Reiter2



(1)
Mountainview Consulting Group, Inc., Zillah, WA, USA

(2)
HealthPoint, Seattle, WA, USA

 



Electronic supplementary material: 

The online version of this chapter (doi:10.​1007/​978-3-319-13954-8_​8) contains supplementary material, which is available to authorized users.


Keywords
Program evaluationProductivityPatients per hourPatient volumePopulation penetrationModel fidelityRange of problemsPCP satisfactionRatio of new to follow-upsRatio of work-in to scheduledThird nextAverage number of visits per patientNo-show ratePatient satisfactionClinical effectivenessAssessing outcomes in adultsAssessing outcomes in youthsHealth-related quality of lifeIndicator-based screeningFuture research


“Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted.”

Albert Einstein


Although most clinicians prefer not to think about it, evaluating one’s service is an important part of making it successful. This is especially true for BHCs, who are often providing a new service and a new type of care. This chapter aims to take some of the pain out of program evaluation by providing specific guidance. We introduce four PCBH program evaluation domains—productivity, fidelity, patient satisfaction, and clinical effectiveness—and discuss how evaluating each can help grow a better service. We conclude with a brief discussion of studies that may help further refine our understanding of the PCBH model, just in case a potential researcher is reading.


Productivity


One of the most frequently asked questions when we talk to groups about the PCBH model is, “How many patients do you usually see in a day?” Productivity is indeed an important performance measure to track. It is arguably the most important, as a busy service is usually doing things right. However, there is more to “productivity” than the number of patients seen in a day. In fact, measuring the number of patients per hour is probably preferable to the number of patients per day; and measuring patient volume and population penetration is also a valuable productivity metric. Tracking trends in each of these over time provides very important feedback. We explore each of these types of productivity in this section. Note that data for productivity indices can usually be obtained rather easily from billing records.


Think Before You Count


In PCBH care, as in general medicine, BHCs may be compensated according to productivity, so a thoughtful approach to interpreting the data is important. Establishing productivity standards can be difficult, and the reasons for unexpectedly low productivity will be different from clinic to clinic. A BHC service in a larger clinic might expect to be busier than one in a smaller clinic, and this is often the case. However, much can also depend on the PCPs in the clinic. A small clinic with mostly biopsychosocially oriented PCPs might keep a BHC continuously busy, whereas a larger clinic with mostly biomedically oriented PCPs might be slower for a BHC. Similarly, one might expect that a busier clinic that sees a higher than average number of patients per day would also keep a BHC busier. Yet, because such clinics are so busy, PCPs may be reluctant to take the time for a BHC referral (especially in the early months of a BHC service, when its value might not yet be perceived). Clinics with a high turnover of PCPs or in a period of flux with their providers also typically produce fewer BHC referrals, because temporary providers have not developed practice habits that include utilization of a BHC. The schedule of the BHC provider can also influence referral rates. Full-time BHCs usually have much more of a presence in the clinic, with more opportunities to shape the practice habits of PCPs and to involve themselves in PCBH pathways and general clinical pathways, whereas part-time BHCs might have a harder time establishing themselves. The point here is that productivity data need to be interpreted in context. Many factors that influence productivity are under a BHC’s control, while others are not.


Patients per Hour

Perhaps the most commonly monitored metric for PCPs is the number of patients seen per hour. Measuring the number of patients per hour paints the most precise picture of the number of patient visits, because it is not influenced by the provider’s time out of clinic. For example, if a provider is on vacation for a week, her total number of patient visits for the month will be low; however, her patient-per-hour metric should be mostly unaffected because it is calculated using only the hours she is available for patient care.

We recommend setting a goal for this metric, as is done with PCPs in most clinics. For PCPs, the goal typically varies depending on the type of PCP, with MDs/DOs being expected to see more patients than PAs or ARNPs. In clinics that serve a complicated patient population, such as a homeless clinic or internal medicine, the goal may be lower, whereas in clinics that function more like urgent care the goal may be higher. A common goal for MDs/DOs is three patients per hour (using 15-minute visits; so the goal is to fill three of the four visit slots each hour).

When setting a goal for the BHCs, mimicking the model used for the PCPs often makes the most sense. It makes it easier for the clinic manager (or whoever does the tracking) to understand and follow and usually also provides a good fit with the clinic culture. As an example, in a clinic where most PCPs are required to see three patients per hour using 15-minute visits, a BHC using 30-minute visits would have a goal of 1.5 visits per hour. That is, the BHC’s visits are twice as long as the PCP’s, so the goal is half that of the PCP.


Patient Volume

Tracking the number (volume) of patient visits that a BHC sees on a monthly basis complements the patient-per-hour metric. A BHC service could conceivably perform well in terms of the number of patients per hour while simultaneously performing poorly in terms of volume. This most often happens when the BHC works limited hours or is blocked for a lot of meetings. When this type of discrepancy occurs consistently, an argument can perhaps be made for funding more BHC time in the clinic (though it also might mean the BHC attends too many meetings!).

Volume, like the patient-per-hour metric, is helpful in evaluating the impact of developments that occur in the service over time, such as implementation of clinical pathways that help the BHC reach more patients. A simple tally of patient visits is how volume is often calculated. However, Table 8.1 offers an alternative method for calculating volume. As an example, assume that a part-time BHC worked 11 days in August and saw 110 patients. His volume score for August would be 71% (110/11 × 14 = 0.71). If the clinic implemented a pathway program over the next 3 months, the expectation would be that his volume score would increase. Note that calculating this index is often preferable to merely tabulating the number of patients seen, because the index adjusts for days out of the clinic, as well as for the differing number of days in each month. Thus, it makes for more accurate month-to-month comparisons. However, many BHCs work varying hours depending on the day (e.g., 4 hours on Mondays and 10 hours on Thursdays), in which case tabulating the number of patients seen may be easier.


Table 8.1
A method for calculating patient volumea















1. Tally the number of days worked by the BHC

2. Tally the number of patient encounters the BHC completed for the month

3. Multiply the number of days the BHC worked (#1) by 14 to get the number of patient encounters the BHC would see if 100% productivea

4. Divide 2 above by 3 above

5. This is a productivity score, with a range of 0–100%


aAssumes a clinic day is always 7 hours, using 30-minute visits


Population Penetration

A third measure of productivity is the degree to which the BHC has penetrated the clinic population (i.e., the percentage of the population that has had a BHC visit). Typically this is measured over longer chunks of time, such as every 3 or 6 months, though it could be measured monthly. Combining this with the other two productivity metrics gives one the most accurate feel for how the service is performing. The goal, of course, is for all three of these metrics to be as high as possible. If the other two metrics are high, but penetration is low, the implication is that the BHC is keeping busy by seeing the same patients repeatedly.


Addressing Productivity Problems


Solutions to increasing productivity need to address a broad range of factors. These include clinic and system factors (e.g., the design of the BHC template, the transparency of the BHC schedule to other team members, the ease with which team members can access the BHC for same-day visits, etc.). Of course, core competency training for BHCs (see Chapter 5) and PCPs/RNs (see Chapter 6) is also of fundamental importance to optimal use of BHC services.

The Barriers to Using the BHC (BUB) Questionnaire and the Barriers to Same-Day Services (BUS) Questionnaire are useful surveys for understanding BHC productivity. They are available in Figures 10.​10 and 10.​11, respectively. One or the other may be administered every 3–6 months during the first 12–18 months of a BHC service, ideally as part of a provider or nursing meeting. In some cases, administrators and medical leadership need to be involved in solving productivity barriers that are identified. Many other factors also influence productivity, such as access, the BHC’s no-show rate, the number of warm handoffs, and others. Tracking these can also highlight the degree to which the BHC is working in the manner intended. We discuss these and other program metrics in the next section. The more metrics one tracks, the more possible it becomes to determine the factors responsible for low productivity (and, hence, potential solutions).


Model Fidelity


Model fidelity refers to the extent to which a BHC adheres to the PCBH model. Tracking this is important because an organization that implements the PCBH model must have some way of ensuring that indeed its BHCs are practicing the model. Day-to-day implementation needs to be consistent with implementation guidelines. Many organizations that intend to follow the PCBH model find that the BHCs they hire practice in a manner not at all consistent with the model. Thus, having metrics that illuminate how the BHC is practicing can be extremely helpful. In addition, fidelity metrics help BHCs understand how innovations or changes they make affect their overall service. Oftentimes, an innovation will improve one fidelity metric at the expense of another. Tracking this helps the BHC to evaluate whether to continue an innovation or perhaps to change it.


Range of Problems and Populations


A healthy BHC service should tend to a range of patient problems that go well beyond depression and anxiety. Included in a list of the most commonly referred problems should be some chronic medical conditions, lifestyle-influenced somatic complaints, subthreshold syndromes, and preventive visits, as well as the full gamut of psychological conditions in patients of all ages. The types of patients seen should also mirror the clinic’s population demographics. For example, if a clinic sees many pediatric patients, so too should the BHC. In such a case, problems common to that population (e.g., parenting problems, behavior problems, ADHD, etc.) should be among the most frequent referrals. If patient race, ethnicity, and/or primary language can be tracked, these should also be represented proportionally in the BHC’s visits. New EHRs often make collection of this data much easier.


BHC Impact on PCPs


Since the goal of PCBH care is to help PCPs be more efficient and effective with behavioral issues, evaluating success with this is important. A helpful practice is to survey PCPs annually as to their experience with the program and the impact of program services on their practice of medicine. Potential survey items are shown in Table 8.2. Responses are on a scale of 0–10, with 0 indicating “not true” and 10 indicating “completely true.” Be sure to instruct PCPs to not put their name on the survey (for anonymity), and administer it during a provider meeting to help ensure it gets done.


Table 8.2
Sample items for measuring PCP satisfaction with BHC service



























1. The BHC is easy for me to access (for warm handoffs or other helps).

2. Having the BHC in my clinic promotes better job satisfaction for me.

3. I am more likely to continue working here because we have the BHC service.

4. I am able to see more of my patients within the allotted appointment time because of the BHC service.

5. I make fewer referrals to outside (specialty) mental health because of the BHC service.

6. My ability to help patients with behavioral and mental health concerns in general is improved as a result of having the BHC service.

7. I talk more about behavioral and mental health issues with patients as a result of having the BHC service.

8. I am more likely to try a behavioral approach instead of medication since working with the BHC service.

9. I have less stress because of the help I get from the BHC service.

10. My patients find the BHC easy to access.

11. My patients tell me they like the way I coordinate care with a BHC.

Additional questions that could be asked include the percentage of continuing education funds spent on presentations related to behavioral health issues and how frequently PCPs/RNs utilize specific techniques the BHC taught that year (e.g., mindfulness, relaxation training, stress management training, etc.). The Barriers Questionnaires (see Chapter 10) can also be administered, to gain feedback about barriers to BHC access, which will hopefully diminish as the BHC service matures.

In addition to surveys, billing data can sometimes provide helpful feedback about PCP impact. Specifically, does access to PCP appointments improve after starting a BHC service? Is the PCP productivity increasing? Is PCP retention improving? Many fluid factors influence these areas, so change can be difficult to see and/or to attribute to a BHC service, but it can be worth an annual look regardless. Keep in mind that implementation is an incremental and ongoing process, involving frequent reeducation of PCPs and staff. Regular surveying can help one understand what education may be needed.


Ratio of New to Follow-Up Patients


A BHC might be seeing an impressive number of patients each day but, if they come from the same small pool of patients seen week after week, the overall penetration into the population will be poor. As such, measuring the ratio of new to follow-up patients is important. In our experience, a good goal to aim for is a 1:1 ratio; meaning on any given day the BHC is seeing roughly an equal number of new and established patients. A ratio favoring new over follow-up patients (e.g., 2:1) suggests that the practice is receiving a healthy influx of new patients but could be struggling to meet follow-up needs. When the ratio favors follow-up patients (e.g., 1:3), this may indicate that the BHC is sliding back into a therapy focus rather than a population health focus. The BHC that falls into a pattern of seeing patients for an excessive number of follow-ups quickly develops access problems, which often leads to frustrated PCPs slowing down or stopping new referrals.


Ratio of Work-In to Scheduled Patients


A primary goal of the PCBH model is to improve access to behavioral health services for both PCPs and patients. Both typically prefer same-day access, and the warm-handoff approach capitalizes on the teachable moment that increases the likelihood of a real behavior change. The goal is for the BHC to see as many same-day and warm-handoff patients (referred to here collectively as work-in patients) as scheduled appointments, but there are a number of factors that can make this difficult. One is the design of the schedule template. A template that reserves some appointments for work-ins is often helpful. The idea is not that work-ins could only be seen during those times but rather that having those built in provides a cushion that allows a busy BHC to take work-ins at various times.

The availability of BHC services during all clinic hours of operation is also helpful for accommodating work-ins. If the BHC service is part time, the PCP may hesitate in pursuing a warm handoff because it takes more time to find out if the BHC is around than it does to just have the patient scheduled. This problem may be worsened by variation in hours by part-time BHCs, as PCPs may eventually give up, thinking, “I never know when she is going to be here—so I just send a referral.” Understaffing BHC time also inhibits work-ins because if PCPs see the BHC struggling to meet demand, they may take pity on him. They may develop a habit of having their NA check the BHC’s schedule, and if it is full they may not pursue a warm handoff. Note that this may also affect overall productivity because while work-ins have by definition a 0% no-show rate, scheduled patients, of course, do not. Thus, missing a work-in opportunity because of a full schedule can be a double whammy if the scheduled patient fails to show. When BHCs are understaffed, a template that reserves half of the appointments for work-ins is recommended.


Days to Next-Available Appointment


Another way to assess access is to look at the number of days until a scheduled appointment is available with the BHC. This is referred to as the third next-available appointment (third next) metric and is commonly used for PCPs (Institute for Healthcare Improvement, 2014). The actual calculation is the average number of days, over a month, that a patient would need to wait for the third next available appointment with the BHC. Usually a manager will calculate this weekly and then obtain an average for each month. (The third next appointment is used because the “next appointment” could be open by a fluke such as a cancellation). The goal for this metric is zero days for PCPs (Institute for Healthcare Improvement, 2014), and we recommend the same for BHCs.

Only gold members can continue reading. Log In or Register to continue

Stay updated, free articles. Join our Telegram channel

Apr 9, 2017 | Posted by in PSYCHOLOGY | Comments Off on PCBH Program Evaluation

Full access? Get Clinical Tree

Get Clinical Tree app for offline access