Getting a Better Grip on Research: The Maze of the Most Busy Life

A common dilemma

At medical school and as a young doctor, I immersed myself in evidence-based medicine (EBM). I had the opportunity to work for some years in an academic unit and developed my critical appraisal skills to the point of performing some small-scale systematic reviews and meta-analyses.

Life changed when I joined a busy group practice—and as a parent of two young children life was, and continues to be, hectic. In reality, there was no time and, certainly, no reimbursement of the time and effort required to continue actively in personally critically appraising papers. And because I did not have time to do this often, it got harder to do and my skills were less sharp. Just knowing about advances in medicine is hard enough, let alone the evidence to support their use. Summarized resources from a trusted public sector source, although not perfect, supply succinct digests of evidence. I see the term EBM loosely used and widely abused by many, including many academically well-placed teachers. I want to be confident that I have found the ‘best’ answer, not just ‘an’ answer.

The longer I am in practice, the more I recognize the importance of personal care. Picking up cues, making astute observations and working with patients to ‘Cure sometimes, relieve often, comfort always’—the same art and skills that generations have valued—seems now at least as important to my patients as the results of those meta-analyses. But is there really a conflict between the two? I like this quote: ‘The job of the human being [in the digital age] is to become skilled at locating relevant valid data for their needs. In the sphere of medicine, the required skill is to be able to relate the knowledge generated by the study of groups of patients or populations to that lonely and anxious individual who has come to seek help’ (Muir Gray, 2001).

As an undergraduate, and especially in my GP training scheme, we spent a lot of time discussing consultation models and even longer preparing for our assessment of consultation skills as part of the Membership of the Royal College of General Practitioners examination. ‘Demonstrating familiarity with the common models of the consultation that have been proposed and how these models can be used to reflect on previous consultations in order to shape future consulting behaviour’ was etched on my consciousness (RCGP, 2007). But at the time, I have to admit my motivation was mostly driven by assessment.

I passed the examination and, after a time in practice, I recognized that this training meant a great deal. Many times I have been surprised at the disclosure that simple open questions produce and how, as a result, I am better able to see my patients in the context of their lives and worries—appreciating more often that their behaviour is more understandable when I try to take into account their emotions as well as their pathology. Sometimes a single, simple, open question saves me from making a big diagnostic mistake—a hint from the patient to consider a possibility that would not perhaps have occurred to me had I been working in a purely biomedical model.

So thank you MRCGP examiners and Pendleton et al. (2003), Stott and Davis (1979), Byrne and Long (1984), Neighbour (2004) and all of your co-authors. Mea culpa, I did not see it at the time, but I do now.

The science and the art

But even when the patient-centred consultation goes well and I am pretty confident about the evidence base for the main management options available, I am still surprised at how different patients choose different options; and on a related point, how different doctors advise different treatments for patients with the same condition.

A psychologist friend lent me Stephen Sutherland's book ‘Irrationality—the enemy within’ (Sutherland, 1992). I discovered that it is normal for human beings to ignore information that does not fit with their expectations, that late-coming information is in danger of being ignored while early information may be given undue emphasis, that easily remembered information may too be activated while more difficult information may be ignored, and that correlations may be conceived where none exist (Klein, 2005). I thought that I and my medical colleagues were all advising patients based on wide ranging and up-to-date knowledge and were presenting our findings in an understandable and unbiased way. But after reflecting on my own practice and discussing decision making with my partners, medical students attached to our practice and with our own GP registrars, I discovered this was far from the case.

Our normal approach to acquiring and using information and making decisions in consultations was the same as in other aspects of our lives. For some time, I did not recognize that the major influences on my practice were my undergraduate teachers, my own GP trainer and hospital specialists nor that information is unconsciously and automatically combined into my own mind map, which I activate, again automatically, most of the time for both diagnosis and management. I was thinking carefully about the patients—but I was using what I thought was the evidence rather than what I knew was the evidence. After a while, as contradictions emerged and I found a little time to look at Cochrane summaries and NICE guidance, I recognized that I and many of my colleagues were not always practising in line with the best quality evidence (Gabbay and le May, 2004), and moreover, we did not like it when new information came along to challenge our ways.

We were all using what the psychologists call System 1 processing most of the time. The human brain has a limit to the amount of information it is able to utilize in decision making. Various approaches are employed by Homo sapiens to enable a decision to be made in the face of large volumes of evidence, and these usually involve truncating the amount of information used in order to be able to make a ‘good enough decision’, an approach termed satisfycing (Gigerenzer, 2008). Appraising all of those data before coming to a conclusion about the optimal management is termed System 2 processing, but I and my colleagues do not have the time for that, and my EBM skills had fallen into disuse.

So I recognized that I often use pattern recognition as part of the diagnostic process. After all, I have seen common constellations of symptoms reflecting common conditions many times. Most times that process is fast and accurate, but I now recognize that that approach works only as long as the patient fits the expected pattern (Fletcher and Fox, 2006). If I am using a ‘recognition’ heuristic to arrive at a diagnosis, I now consciously activate an ‘evaluation’ heuristic to check out other alternatives and provide as much defence as possible against my own cognitive biases. And if there is no pattern emerging for me, then I fall back on the same hypothetico-deductive processes I used when less experienced, but I know they can fail me too (Elstein and Schwarz, 2002). More and more, I am aware that if I am using pattern recognition—and I still do that often—I need to protect against my inherent cognitive biases when diagnosing and in advising on management.

I do try hard to balance the need to make a reasonable attempt to exclude alternative diagnoses while not investigating to the nth degree. Many doctors are said to be natural Bayesians (Gill et al., 2005), but I struggle to employ computerized decision aids as a conscious, System 2 approach. I understand the principles of estimating the pretest probability, linking them with likelihood ratios, and I can see how they can help sequentially in estimating successive cycles of pre- and then post-test probabilities. There is an identified reluctance among doctors to use mathematically derived decision support even when it is shown to be superior to traditional clinical approaches (Sutherland, 1992). Am I really that biased? I am continuing to work at this, but it is comforting to know that the science of diagnosis is coming more under the spotlight and that I am not alone in still trying to make sense of this most difficult task (Straus, 2006).

Finding and using the best evidence

This is where my early interest in EBM has been both a help and a hindrance. I spent years learning how to find relevant published research and critically appraise it. But now, without ready access to a university library and trained information specialists, even obtaining the full paper is sometimes difficult. I know abstracts do not tell the full story and may even contain incorrect information (Pitkin et al., 1999). Yet, abstracts on Medline may be all I can obtain as so few journals are free access. Even when I can get the full paper, my critical appraisal skills have atrophied and I am less confident about my competence than when I used those skills regularly.

And the volume of data is amazing. In 2002, it was estimated that 7287 articles, potentially relevant to primary care, are published each month. Physicians trained in critical appraisal would take an estimated 628 hours per month to evaluate the articles (Alper et al., 2004). But worse, if I read a journal and find a paper relevant to my practice, how do I know if that is just an aberrant piece of research and that there are half a dozen other papers all showing I should do the opposite? Hardly ever does one paper tell me the whole story by setting its results in the context of the rest of the evidence (Clarke and Chalmers, 1998; Clarke et al. 2002). And that volume of material is before I try and keep up with guidelines, local clinical initiatives and important non-clinical information such as new contracts and policies from the Department of Health. So I have given up trying to keep up to date by reading.

That everyone should use the highest quality information to guide clinical decisions is now rarely seriously challenged (Reilly, 2004), but my standards were slipping. Pursing critical appraisal was unrealistic (Underhill and Pegler, 2005) and yet this grated against my basic training in medicine. Taking short cuts and then ‘hard wiring’ treatment decisions was happening—I was using available or more easily remembered data and ignoring new information that did not fit with my and my colleagues, perceptions of what the treatment should be. System 1 processing dominated the selection of my management decisions as it did my diagnostic approaches. I was activating the recognition heuristic without activating the evaluation heuristic often enough.

Finding the right information I can trust

A letter and two papers helped me. General practitioners see about 150 conditions repeatedly, but few of them very frequently (Lipman, 1998). Thirty-six problems account for two-thirds of problems presented. So if I could concentrate my limited time on the conditions I saw repeatedly instead of thinking I had to read journals, then I would be up to date for two-thirds of the pathology my patients brought to me.

It was reassuring to read that 72% of general practitioners say they use evidence-based summaries generated by others, and 84% say they use evidence-based practice guidelines or protocols. However, 95% believe that learning the skills of EBM is not the most appropriate method for moving to EBM (McColl et al., 1998).

Recognizing that I needed unbiased sources of evidence and would never have the time, resources or skills to personally search for and critically appraise all the published research, I began using predigested summaries of evidence. It went against my basic EBM training to rely on summaries of evidence produced by other people, but it was what other people were doing and there were even papers published advocating that radical approach (Slawson, 1994). Somehow knowing that others were doing this made it acceptable. The Cochrane Collaboration, Clinical Evidence, NICE, the Scottish Intercollegiate Guidelines Network, Clinical Knowledge Summaries (CKS) and the National Prescribing Centre (NPC) are just six organizations with robust and transparent methodologies for the production of high-quality evidence summaries. Critical appraisal is their stock trade and so they are better at it than me—my job means I need to see patients, not papers. And because they have clear transparent methods, I can trust them to do it to the highest standard possible.

I have found that coverage of the management of routine issues in most conditions repeatedly presenting to generalists is already good and continues to improve. I realized that most clinicians, like me, just need skills to understand the summaries; fully fledged critical appraisal can be largely the task of those producing the summaries. This has been liberating—I have thrown away the ‘guilt stack’—that tottering pile of unread journals in the corner of the living room that just seemed to induce guilt rather than wisdom. I now rely on three approaches to find the latest evidence. (see Box 1).

Whether I am foraging, hot-synching or hunting, information can be seen as a pyramid (see Fig. 1). The top of the pyramid usually leads quickly to the most useful (relevant and valid) information. A few things remain frustrating until you know where to look. For example, there is little on how to manage otitis externa in the top layers of the pyramid and it is only when you reach CKS that a summary of the evidence from a trusted, public sector source is found. The bottom section includes textbooks, individual journals and Medline. Although some of this information is useful, it is mixed in with an incredible amount of non-relevant and often out-of-date information. I start at the top of the pyramid and drill downwards, stopping as soon as the information needed is found. Medline still gets used quite a bit for the less common stuff.

OPEN IN VIEWER

Figure 1.

Slawson, D.C., Shaughnessy, A.F. The Information Mastery Pyramid.

The Information Pyramid builds upwards in terms of usefulness. Usefulness is a factor of relevance, validity, and the time taken to find the information. Relevance and validity need to both be high (or in mathematical language big), and time needs to be short (or small) So the usefulness equation is

Usefuless = relevance × validity time

Moving higher up on the pyramid leads to more useful (relevant and valid) information. The ultimate foraging, hot-synching or hunting tool would access all of these levels of evidence starting at the top, drilling down through the layers and in less than 30 to 40 seconds point the clinician towards the highest level of evidence which provides an answer to meet their needs. Whilst several pre-specified compilations of datasets are available, a resource which fully meets these exacting requirements is yet to be developed.

Professional development

Locally, I find that colleagues now sometimes look to me for advice about how they should manage patients. I have become involved in the teaching of medical students and GP registrars attached to the practice and that is a real boost for me. Together, we have devised an informal curriculum which builds from orientation (which includes the psychology of decision making), through basic skills to understanding and using summaries of evidence, and finally to foraging, hot-synching, hunting and explaining risks and benefits of options in terms patients can understand. (see Box 2).

I have to say that our local primary care trust (PCT) has been really helpful. We have an afternoon's protected learning time every 6 weeks when all the practices close and now, instead of having presentations from local consultants on clinical topics, we work in small multidisciplinary groups with colleagues from different practices. We decide what common condition we are going to discuss and we use trusted summaries of evidence as our sources. Of course, the most fascinating and powerful learning comes from hearing how different people manage the same condition—I have learnt that stories of patients impact on most people's memory better than the results of systematic reviews.

Of course, the NHS and PCT have not abandoned active implementation—‘doing things’ to the system or individuals in order to better implement clinical and cost-effectiveness decisions. We still get guidelines or local summaries of NICE guidelines. The prescribing adviser visits us and presents back to us our prescribing data and advises in therapeutic choices. We audit areas of interest to us and, of course, our practice is driven by the financial incentives in the Quality and Outcomes Framework.

But when we look at the evidence for these management approaches to changing clinical practice, we find they create an overall median effect size of around 10% (Grimshaw et al., 2004). As the Chief Medical Officer for England wrote ‘The way in which clinical decisions are made, the extent to which they depart from research evidence, and the factors that determine compliance with best practice have … been extensively studied … Despite this, the solution to the problem of clinical practice variation has not been found …’ (Donaldson, 2005). Of course, I do not expect the NHS to abandon implementation interventions, not least because sometimes there is a quick and widespread adoption of new practice disseminated via these traditional routes. But just to do that and not to consider how humans acquire and process information in the real world and then try to do something about that as well seems a little perverse.

Translating evidence for patients

So I feel a lot better equipped now in terms of knowing what the evidence actually is with things I see often, and I make the best attempt I can to find out if I do not know what to do about less common problems. But deciding with individual patients about what treatment is best for them is still tricky.

I am still signed up to the MRCGP principles of explaining the problem or diagnosis in appropriate language, incorporating some or all of the patient's health beliefs and to giving the patient the opportunity to be involved in significant management decisions. Of course, not all patients want to be involved in decision making—about a third, it seems, are quite prepared to leave decision making entirely to me, but about a third to a half want or at least appreciate some participation in the decisions taken and a minority actually want to take the decisions themselves.

My GP trainer told me the mnemonic ‘Oh, I see jays’ (OICJ) for giving information:

O Options ‘We could do X or Y or Z’

I Implications ‘If we do X (and this could be nothing), then its more likely that …. On the other hand if we do Y ….”

C Choice ‘On balance, I'd usually choose Y’. Often preceded by an interim step when the patient is asked what they think about those choices.

J Justification Sometimes not required, but if it is …. For example, ‘Here's a picture that explains the benefits of taking a statin to lower cholesterol in someone like you (see Fig. 2). Its shows a hundred people like you, and over the next 10 years, 80 would be fine but unfortunately 20 people will have a heart attack or a stroke. If we give all 100 people a statin and they all take it regularly that will mean five fewer heart attacks or strokes. So with treatment there are 85 people who don't have a heart attack or stroke and unfortunately despite treatment there are still 15 who do. What do you think is the right thing to do here?’

OPEN IN VIEWER

Figure 2.

Benefit from statin therapy over 10 years in 100 people at 20% cardiovascular risk. See www.npci.org.uk/therapeutics/cardio/cdlipids/patient_decision_aids/patient_decision_aid1.php [date last accessed 17.03.2009].

It sounds fine, and sometimes it is. I am used to using the key principles about communicating risk—and I try to communicate the trade-off between benefits and harms, I avoid purely descriptive terms of risk, for example ‘low risk’, I use natural frequencies (not percentages) and a consistent denominator, for example ‘1 in 100 are harmed but 5 in 100 will benefit’, I use absolute numbers (not relative) and use visual aids and probabilities (Paling, 2003). In our protected learning time, it has been really difficult for some colleagues to be able to understand the limited absolute benefits that the evidence shows. It is clear that we have some way to go to all of us being able to understand health statistics (Gigerenzer et al., 2008). We practise this together a lot.

Communicating risks and benefits in this way is an inexact science. But I am convinced that some, if not many, patients really appreciate the information being presented this way and I get a lot of satisfaction observing their increased understanding. They can still sometimes surprise me with their choices though.

A modern skill set

EBM has come a long way in little more than two decades. ‘Doing the right things for most people most of the time’ encompasses the flexibility required to incorporate the need for decision making to be based on both evidence and the patient's individual circumstances.

In many areas of medicine, the evidence is already synthesized. Some large randomized controlled trials (RCTs) and systematic reviews describe evidence so compelling that the right thing to do is clear. Caution is sometimes required given the inevitable uncertainty of some evidence, the selective reporting of clinical trials and the potential harms of being wrong about what is right—just what is the place of glitazones in type 2 diabetes?

So I realize that I need to review the tools and skills I use. In diagnosis and management, it is inevitable that heuristics and mindlines will remain part of clinical practice. But I will try and seek out clinical decision rules—fighting my aversion to anything that involves numbers in the diagnostic process.

It is unlikely and, indeed, unnecessary that each and every time common problems present I should hunt for the best information from current evidence. But a periodic check on the evidence base underpinning repeated clinical decision making means my mind map of how I ought to treat conditions that present repeatedly is up to date using valid summaries of evidence.

Communicating those results and then reaching a truly informed decision with individual patients remains a challenge. But I am certainly in a better position to do so, now I am more confident and I really know what the evidence is. Or, perhaps more importantly, I know more often when I do not know, and know more often where I might find out.