One of my least favourite parts of being a researcher is you have to sit through conference presentations. I have never been good at sitting still. And, let’s be honest, most presentations are painful – from the individual who posts slides full of teeny-tiny bits of arithmetic no one can read, to the presenter who refuses to leave the stage because they have ‘just one more thing to say.’ Just painful.
Occasionally, though, an unpleasant experience of this type can yield an unintentional ‘aha’ or teachable moment. I had one such moment last year during a talk that could have been labeled ‘how not to do a literature review.’ The talk was on the presenter’s experience of conducting a literature review on a piece of technology he had already decided his police service needed. As he walked through his efforts at trying to locate only the research that backed up his pre-determined conclusions, he repeatedly expressed frustration that the literature didn’t yield the results he wanted, and he had to discard a lot of studies that didn’t back up his opinion. I literally face-palmed. On the upside, it did make me determined to spread the word: don't do this!
The technical term for this technique is ‘cherry-picking.’ It is the total opposite of anything looking like rigorous, quality research. In fact, it is the equivalent to another needs to be discredited practice, ‘calling a neighbouring police agency to ask what they do and then calling the response a ‘best practice’.’
So, if someone is serious about reviewing the research literature on a particular topic, what should they do? Allow me to present a brief introduction to one of my favourite research practices: the systematic review (SR). As I’ve explained it elsewhere,
A SR is a technique for “finding, sifting, sorting and synthesizing the findings of primary evaluations relevant to particular interventions” (Johnson et al. 2015: 460). In advance of beginning the finding phase of research, the researcher constructs a set of questions to be answered, as well as a set of well-defined inclusion and exclusion criteria (Akobeng 2005; Pawson 2006; Neyroud 2011). As Uman (2011: 57) explains, “Systematic reviews, as the name implies, typically involve a detailed and comprehensive plan and search strategy derived a priori [beforehand] ...” (Huey 2017).
How to begin:
Pick your research topic and be clear upfront about the parameters of what it is that you are interested in. For example, if you want to know what the research literature has to say about ‘police training’, what type of training are you interested in? Recruit training? In-service? Classroom? Online? Experiential?
Identify how you will search. A lot of people simply rely on our old friend, Professor Google. Professor Google is indiscriminate, meaning you will get a lot of junk that you will have to weed through. If possible, your best bet is to look through university search engines, scholarly databases, or, if that’s not possible, use Google Scholar.
Figure out what types of research you wish to include (what us research wonks call 'inclusion criteria'). Here’s a rough example from a recent discussion I had with a colleague on doing a systematic review on "qualitative research on women in policing in non-US countries".
inclusion criteria:
selected papers published in peer-reviewed journals* **
the paper contains the results of a qualitative research study (interviewing, ethnography, photovoice, etc).
the focus of the research is on one or more aspects of women's work in contemporary public policing (roles, experiences, challenges, etc).
the research was conducted in a country other than the U.S.
the paper contained a clear description of the methodology used.
4. Figure out what types of research you want to exclude (exclusion criteria).
exclusion criteria:
· the paper did not contain discussion of a study (ie. opinion pieces, commentaries, special issue introductions)
· the study topic was not on some aspect of women's work in contemporary public policing (ie. examined private
security, contained historical analyses, etc).
· the study was not published in English
· the paper contained no discussion of research methodology
· inability to access paper.
Once you’ve determined
1. What you’re looking for
2. How you will look for it
3. What you will include
4. What you will exclude
you are ready to get started ***. Good luck!
*My preference is, whenever possible, to work with published, peer-reviewed papers as opposed to what we call ‘grey literature’ – work that has not been subjected to any form of peer-review. This isn’t to say that all peer-reviewed academic literature is great – in fact, I teach classes on bad research – but that it has, presumably, been looked over by two or more independent experts.
** I also typically exclude Master’s theses. Despite universities claiming they require students to produce work of publishable quality, most thesis quality is fairly low.
*** At some undetermined future date, I will happily discuss how to analyze your SR data.
