One of the relative advantages of traveling so much is that I end up spending a lot of time looking at books at the airport. On my most recent trip, I bought several interesting books. Let’s start with This Idea is Brilliant and Richard Thaler’s discussion of the premortem. Developed by research psychologist Gary Klein, a premortem is a process whereby individuals launching a new venture sit down and seriously consider potential risks. Rather than applying hindsight after a problem has occurred (and program failure has happened or is inevitable), it challenges developers to use foresight to work out what they can do to prevent, minimize and/or respond to those things that are perfectly foreseeable (but nobody bothered to think about until after the fact). Here’s how Klein phrased it:
How does the premortem fit into evidence based policing? As you may remember, Larry Sherman developed a 3T approach to developing policing interventions. The 3Ts are: Targeting - identifying a high priority policing problem (ie. a place, crime pattern, type of offense) Testing - police strategies to fix the problem should be tested through scientific research to make sure they work (ie. reduce crime, increase arrests, improve community satisfaction). Tracking - once in place, solutions should be tracked over time to ensure they continue to work and, when needed, to make changes. For Sherman, one of the biggest reasons new policing programs go off the rails is because of ‘tracking’, or, more appropriately, the failure to track programs over time to ensure they continue to work after the initial assessment is done and the program judged a success. I agree with this. However, I am also of the view that a lot of new policing interventions (and studies) derail in the testing phase because researchers and program implementers failed to conduct a thorough premortem. Here’s a hypothetical example:
Police Service A decides to implement a hot spot strategy for its new bike patrol. Bike officers are tasked with showing up at select locations for 10-15 minutes at least once every shift.
What could possibly go wrong? Hmm .... And this is where Meltdown provides some further insights. You see, a lot of times, when people are asked to think about possible outcomes, they can come up with some fairly obvious errors, issues, bottlenecks and so on. However, what a premortem should do is help implementers also think through those issues that, while less likely to come to mind, are also foreseeable and therefore avoidable. To do this, Clearfield and Tilcsik suggest an exercise that challenges people to construct disaster scenarios based not on the current situation, but rather on what could happen two years from now. As they explain, instead of thinking of what risks might happen, ask “people to imagine ... a bad outcome has already occurred.”
The results are interesting: individuals tend to move away from a fairly narrow set of risks (ie. The bike cops didn’t fill out their reports correctly so the hot spot data was full of errors) to a more diverse array of potential problems (ie. There was local flooding and many streets are now under water, therefore the bike cops can't get to the hot spots and are busy with rescue activities). Using prospective hindsight in this way, challenges us to think beyond the 5 yard view - that is, where we typically live with our own intuitions and biases - to try to see a situation with the perspective and clarity one can achieve from taking a 30,000 foot view. Why aren’t premortems a standard part of program and research design? I’ll give the last word to Daniel Kahneman:
“most decision makers will trust their own intuitions because they think they see the situation clearly. It’s a special exercise to question your own intuitions.”
