Challenges,solutions and future directions in evaluative research

There is a need for ‘step change in the historic performance of the health, health care and social care system’ that will depend on ‘skilful design and robust implementation of a range of initiatives, not just once but in a dynamic stream, rooted in and modified by information on impact: in short, rooted in intelligent evaluation that is sensitive to the complexity’. ¹ This call to arms appears in the foreword to a new collection of papers by leading thinkers that sets out the current state of the science for the ‘intelligent evaluation’ of complex health and public health interventions. ² It offers insights into methodological challenges and potential future directions across the broad disciplinary menu of evaluative research and proposes thoughtful approaches to the competing priorities of, on the one hand, rigour and generalizability, and on the other hand, appropriate scale, cost and the need for prompt results.

A particular trigger for the production of this volume was the launch in England of the NHS Five Year Forward View ³ which articulates not just the urgency of change for health care systems given new patterns of ill health but also the aspiration that such changes should no longer be centrally driven. The NHS is being encouraged to pursue local and regional experiments to bring about a range of new models of care; innovation is henceforth to be achieved from the ‘bottom up’. Writers involved in this volume felt that an authoritative ‘forward view’ of evaluative research methods was also required in response: bottom-up transformation has to be shown to work (using a range of appropriate research methods), underlying mechanisms and accommodating contexts need to be clarified and research evidence needs to be provided in a timely fashion to ensure effective dissemination and the adoption of optimal system and service change.

The Health Foundation (THF), the Medical Research Council (MRC) and the National Institute for Health Research (NIHR), via its Health Services and Delivery Research (HSDR) programme and Collaborations for Leadership in Applied Health Research and Care (CLAHRCs), together with Universities UK and AcademyHealth (the US health services research association), formed a partnership to develop the methodological underpinnings for evaluative research. A round table discussion in May 2015, involving these partners, set out an agenda for the democratization of evaluation via increased engagement between researchers and service leadership. ⁴ Key messages included the need to move beyond the unhelpful notion of service and research being two separate cultures – researchers can help service leaders to clarify goals, gather relevant evidence and identify proportionate approaches for evaluating planned changes – and the availability of a spectrum of study designs and methods to tackle challenges in evaluating complex and emergent services. This was followed by a meeting in London in June 2015 at which 90 leading researchers came together for two days of challenge and debate. The eight methodological domains that were the focus of plenaries and facilitated discussions were then drafted as essays. In the spirit of collective endeavour, drafts were shared with plenary speakers for critique and revision before they were sent to the two editors, who suggested final modifications. The essays were further revised in the light of independent reviewers.

Despite the plurality of methodological approaches, a number of themes emerge from across the essays. For example: the need for researchers from different disciplines to engage early and often with policy-makers, practitioners and funders through transparent and wide ranging discussions to ensure that common understandings are achieved about perceived objectives; the identification of relevant and feasible outcomes and the maintenance of appropriate boundaries between interested parties. Such discussions will often shift the focus from the traditional binary question of effectiveness towards a broader understanding of mechanisms, processes and outcomes of relevance to patients, practitioners and policy-makers with differing perspectives, priorities and timelines.

The essays also include a call to co-ordinate analyses of macro-, meso- and micro-level determinants of system change. Such contextual factors are too often dismissed as unique features, too intangible to define and yet they can act as barriers to the successful transplantation of service innovation to other settings. This is unhelpful. Instead, there is a need to identify generalizable elements of beneficial processes by exploring the mutual influence of the intervention with various, distinctly defined contextual components.

There is also agreement on the use of mixed and multiple methods. The importance of integrating observational research with experimental methods is now recognized. However, authors went further and provided examples of uniting established methods with innovative techniques to enhance the quality and utility of research, such as observational data alongside randomized trials to tackle external validity. ⁵

Such extensive analyses require access to health care data from multiple sources and sectors. This supports the need to improve the quality and comprehensiveness of routine data, particularly from non-acute settings and with regard to information on comorbidity and severity.

The value of critically applying and developing theories and models, for the evaluation both of intervention quality and of implementation, is endorsed, though any tendency to regard specific theories as inviolable was challenged.

Finally, we are urged to innovate by drawing more heavily from other disciplines. Health services research will benefit from more extensive collaborations with computer scientists, spatial analysts and mathematicians as well as with other fields such as systems biology and education research. Examples include the application of machine learning to code narrative data, computer adaptive testing to reduce response bias and interactive multimedia techniques to examine clinical decision making.

The volume may have missed important topics. It has not, for example, explored the application of artificial intelligence to modelling and analysis, and has barely mentioned relational methods such as network analysis or qualitative methods such as participatory research and action ethnography. More needs to be said about how best to develop the vital contributions from patients, the public and policy-makers. It was not, however, the intention to present an exhaustive account of the field. Instead, these essays reinvigorate the conversation both about plurality in methodological approaches and for sustained investment in new and diverse methods, whilst maintaining a keen eye on the need to achieve accurate, comprehensive, relevant, timely and generalizable results.