In evaluations for programs aimed at improving client outcomes it is not uncommon for evaluators to interview those administering the "treatment" - that is, they ask they ask those individuals who deliver the program what they think of the program they run. It is also not surprising to discover those same individuals, who are clearly committed to offering that course of treatment, tend to appraise their own programs favourably and, when asked, typically demonstrate their success by proffering stories of perceived positive outcomes.
Today, I want to make a radical argument: until we can control for the possibility of a placebo effect, we should stop asking and/or relying on treatment providers/program creators for their thoughts on the success or failure of their own programs. Wait ... what?
Here’s the deal: what we are often seeing in such studies is the strong possibility of a version of what has been termed a “caregiver placebo effect." What is this? It's when individuals who are motivated to believe that a treatment is working (caregivers) appraise the treatment they provide favourably when there is no proof that it is working or, worse yet, demonstrable proof that it is not effective.
I see this a lot in relation to policing and other community safety programs. Positive evluations of a program's effectiveness too often boil down to little more than positive appraisals from those who are most heavily invested in its success. As a result, we get evaluations based on the beliefs of individuals who see exactly what they want to see. Despite its importance, the ‘caregiver placebo’ effect has rarely been studied. There are two possible reasons: 1. Few researchers study these types of evaluations outside of medicine and certainly few (if any) criminologists;
2. Clinical trials in medicine can control for placebo effects by double-blinding the study (ensuring that neither the treatment provider nor the patient know who is getting what intervention). Since standards for evidence are so low in relation to policing/community safety program evaluations, no one gives much thought to this program. If that's true - that we have no research on the “caregiver placebo effect” in policing - how do we even know that program evaluations are susceptible to this problem? Fortunately, we can turn to the field of psychology for an answer. The caregiver placebo effect is actually a well-known cognitive distortion that afflicts humans: confirmation bias. Confirmation bias (aka. 'seeing what you want to see') occurs when we selectively pick facts that appear to support those views we hold, while ignoring facts that challenge our positions. While some scholars in the fields of medicine and psychology have raised this issue in relation to evaluations of drug, alcohol and other counseling programs, the field of veterinary medicine actually provides more telling clues as to how this effect can work to distort perceptions in those administering treatments. Here’s some examples: - In a systematic review of five studies of the use of analgesics to treats cats with degenerative joint disease it was found that "the caregiver placebo effect was high" with cat caregivers rating 54-74% of placebo-treated cats as being successfully treated (Gruen et al. 2017). In other words, pet parents tended to see what they wanted to see. - In a study of dog lameness, researchers conducted a randomized, double-blinded, placebo-controlled study that found that both dog owners and veterinarians were susceptible to a caregiver placebo effect. They also found that "this effect was significantly enhanced with time" (Conzemius and Evans 2012), meaning the longer caregivers believed something, the more deeply rooted their beliefs became.
In support of the possibility of a caregiver placebo effect in policing, we could anecdotally point to every single time one of us has heard a police officer say they deliver the DARE (Drug Abuse Resistance Education) program because it “works”. I can similarly name, off the top of my head, several other untested or unsupported programs where pretty much the only proof of efficacy offered is suspect due to:
1. the possibility of a general placebo effect (ie every single course evaluation sheet offered after a training session in which instructors and course creators claim to have achieved significant behavioural changes in about 4-8 hours of classroom time)* or;
2. a potential caregiver placebo effect (such as when someone delivers a program and then rates their own results favourably. (Insert name of program du jour here)). So, where does that leave us? Hopefully, starting to rethink how we do police program evaluations and what we consider to be 'evidence.'
My position? Caregiver/service provider opinion of effectiveness - absent other independent evidence of outcome success rigorously obtained - should never be accepted as an indicator of program success.
*Aside from several possible cognitive biases that can lead one to make claims on course evaluations that are both favourable to the Instructor and untrue, we cannot rule out the possibility of “pain avoidance” as a factor behind some course evaluations, as in “if I say I learned lots and changed my behaviours, I might not ever have to take this course again!”
