Improving Process Evaluations of Health Behavior Interventions

Abstract

This article reflects on the current state of process evaluations of health behavior interventions and argues that evaluation practice in this area could be improved by drawing on the social science literature to a greater degree. While process evaluations of health behavior interventions have increasingly engaged with the social world and sociological aspects of interventions, there has been a lag in applying relevant and potentially useful approaches from the social sciences. This has limited the scope for health behavior process evaluations to address pertinent contextual issues and methodological challenges. Three aspects of process evaluations are discussed: the incorporation of contexts of interventions; engagement with the concept of “process” in process evaluation; and working with theory to understand interventions. Following on from this, the article also comments on the need for new methodologies and on the implications for addressing health inequalities.

Keywords

public health process evaluation complex interventions behavior change qualitative methods

This article discusses current limitations of process evaluations of health behavior interventions and highlights several areas where practice could be improved by drawing on the social science literature, particularly with respect to complex interventions where process evaluation can be especially challenging. It argues, first, that the social science literature provides useful insights into the social world with which health behavior process evaluations are increasingly engaged. This is not a completely novel idea: complex interventions and corresponding evaluation approaches have, for some time, adopted more sociological perspectives (Bunton, Murphy, & Bennett, 1991; Burke, Joseph, Pasick, & Barker, 2009) as they evolve beyond the drug trial model (Craig et al., 2008b). There is scope, however, for further development in this direction. Second, the established tradition of process evaluation in the social sciences generates useful insights into the methodological and theoretical challenges that this type of evaluation presents and could be applied to the health behavior change field to a greater degree to improve evaluation practice. Much of this literature is located in the program evaluation field. This discussion is followed by a consideration of implications for the types of methodology required, and indications of ways in which applying social science approaches to interventions and process evaluations could improve how health inequalities are addressed.

This article draws on process evaluation literatures in two fields: health behavior interventions, which address aspects of healthy lifestyles such as physical activity and diet, and interventions in the social science field such as employment, education, and community development. Sociological literature of relevance, for example, theory relating to contextual factors, is also referenced. The limited interaction between process evaluation traditions in the health behavior and social science fields is reflected in their rather separate terminologies. The term process evaluation has a relatively specific meaning within health research, referring to an evaluation of a process of change that an intervention attempts to bring about in order, at least in principle, to explain how outcomes are reached. “Process” includes aspects of the intervention such as the recruitment of study participants, attrition rates, acceptability, and fidelity of the implementation. Process evaluations in health research are typically, but not always, nested within experimental studies including randomized controlled trials (Moore, 2010) and have been used increasingly in public health studies since the late 1990s (Steckler & Linnan, 2002). The practice of process evaluations and their role in evaluating interventions has also received attention, for example, in the updated Medical Research Council (MRC) guidelines for evaluating complex interventions (Craig et al., 2008b).

Examples of process evaluation in the social sciences can be found in a range of areas such as employment (Donaldson & Gooler, 2003), community development (Carvalho & White, 2004), education (Nesman, Batsche, & Hernandez, 2007), and international development (Gasper, 2000). However, the term process evaluation tends to be largely restricted to health, with a limited number of exceptions in the education literature. Furthermore, in the social sciences process evaluation is not a distinct concept as it is in health research: a variety of terms, concepts, and approaches are used. These include most commonly: realistic evaluation, theory-based evaluation, theory of change, logical frameworks, and logic models. Rogers summarizes further terms as “theory driven, theory oriented, theory anchored … intervention theory, outcomes hierarchies, program theory, and program logic (Rogers & Weiss, 2007, p.63), also noting that terms are not always used consistently. The commonality in these approaches is engagement with the evaluation of the process of an intervention. Some of these models such as logic models or realistic evaluation have already been adopted in health research (e.g. Bonell, Fletcher, Morton, Lorenc, & Moore, 2012; Byng, Norman, Redfern, & Jones, 2008; Judge & Bauld, 2001) but to a limited degree (Marchal, van Belle, van Olmen, Hoerée, & Kegels 2013).

Apart from linguistic and taxonomic differences, there are differences between the two fields in how processes evaluations are conceptualized and conducted. A comparison of the two areas, as will be demonstrated, reveals weaknesses in the health behavior process evaluation field in three key areas: context, process, and theory. Process evaluations of health behavior interventions are often detached from the intervention’s context, atomistic rather than joined up in considering elements of intervention processes, and divorced from theory. This article will illustrate where concepts and approaches to evaluation in the social sciences in these three areas are more developed and could be applied to improve health behavior process evaluations. The following section on context examines the underdevelopment of contextual factors of interventions in health behavior process evaluations, notes some relatively recent approaches to improving this area of process evaluation, and suggests further developments. The two sections on process and theory address a similar point about articulating theories of interventions more explicitly through explaining pathways of processes and how they are linked to intervention outcomes, noting ways in which social science process evaluations have engaged with these areas. The section on process focuses more on the modeling of intervention elements and the importance of linking process to outcomes. The section on theory addresses explanations of interventions at a slightly more abstract, generalized level, refers to links with theory beyond particular interventions, and also discusses different ways to apply theory in process evaluations.

Process Evaluation and Context

Health behavior interventions with multiple components are complex but are also set within complex settings (Shiell, Hawe, & Gold, 2008); understanding this context is critical for the generalizability of study findings. This is especially important when interventions are tailored to suit local conditions or in pragmatic trials, where the roll out of the intervention is locally or programmatically distinct since applications to other contexts may be uncertain. However, context is an underdeveloped aspect of process evaluations of health behavior interventions (Wells, Williams, Treweek, Coyle, & Taylor, 2012). A key issue is the tendency for evaluations to be acontextual, addressing the somewhat closed world of the intervention, particularly where process evaluations are nested within trials. Process is conceptualized as how the trial is implemented and received, and as such attention is paid to internal elements such as retention and fidelity. Context has been addressed to a limited degree in some health behavior process evaluations by including context as one of these elements for analysis (Moore, 2010). Additionally, context is a key consideration for the few health studies that have adopted Pawson and Tilley’s “Realist Evaluation” approach (Pawson & Tilley, 1997) which emphasizes three key elements: context, mechanism, and outcome, where the mechanism operates within a particular context to produce an outcome in a particular configuration (e.g., Byng et al., 2008).

Several perspectives within health behavior research have engaged with context more extensively, rather than simply identifying it as an element of process evaluation. For example, approaches to context may examine the organizational environment in which an intervention is implemented, which may have a significant effect on an intervention (Wells et al., 2012). Theoretical models relevant to organizational factors affecting implementation, such as Diffusion of Innovations and Normalisation Process Theory (Greenhalgh, Robert, Macfarlane, Bate, & Kyriakidou, 2004; Murray et al., 2010), have been applied to health behavior interventions to a limited extent and have been used as frameworks to understand how new ideas or initiatives are taken up by organizations. Socio-ecological perspectives take a broader approach and can be used to conceptualize context in terms of different levels, from the individual through to the family, the community, and the societal levels. They thus draw attention to context in its different scales and, further, may also attempt to explain how different levels are related to each other, for example how local organizations (the community level) and regulatory environments (the societal level) might complement health education for behavior change (the individual level) (Palareti & Berti, 2009; Stokols, Allen, & Bellingham, 1996). However, in practice interventions generally focus on one of these levels rather than linking a context, such as community organizations, with an individual health behavior (Richard, Gauvin, & Raine, 2011) and also tend to focus on the individual rather than family, community, or societal interventions in any case (Golden & Earp, 2012). A number of studies have addressed a wider context by adopting environmental approaches, but these have tended to be limited to physical environments such as access to green spaces (e.g., Charreire et al., 2012).

An additional consideration for context is that it changes and evolves over time (Pawson & Tilley, 2004; Wells et al., 2012) and thus requires a process evaluation that can accommodate dynamics of change. Hawe et al. provide an example of engaging with this issue in a more sophisticated way by theorizing an intervention as a “critical event in the history of a system” (Hawe, Shiell, & Riley, 2009, p.267), rather than a package of activities delivered. One then considers the settings, social networks of those involved and change over time, which form a system into which the intervention is introduced. The outcomes produced are conceptualized as emerging from the reactions produced in the “intervention field” rather than simply as a result of causal processes within an intervention (Virtanen & Uusikylä, 2004).

The section above has summarized various attempts in the health behavior literature to engage with context as a multi-level and dynamic factor in process evaluations. However, the wider social, cultural, and political environment is reflected in this literature to a lesser degree, although this is also a significant context with implications for the processes and outcomes of interventions and is one that social science perspectives can elucidate. Context can extend to economic, historical, and global factors, which have a distinct relevance to poverty and health inequalities through their influence on income, housing and other significant drivers of health (Edwards & Di Ruggiero, 2011; Marmot et al., 2010). In particular, a key challenge for health behavior interventions and evaluations is engagement with social, cultural and political contexts of groups that tend to have poor health, in contrast to the “rational behavior” of the health-seeking individual who often populates health behavior interventions at the conceptual level. Health behavior perspectives have tended to employ health psychology theories that have a bias toward cognitive processes of the individual and that incorporate the social world in a fairly limited way (Burke et al., 2009). This emphasis on “rational action” by autonomous individuals has remained despite the increasing incorporation of other perspectives noted above (Krumeich, Weijts, Reddy, & Meijer-Weitz, 2001). This perspective discounts the notion that “rational behavior” always occurs within a social context and that what is rational in one sociocultural setting may not be in another. Social contexts of low-income groups, elucidated by social science studies, can be starkly different from those of the professional policy maker or researcher and may indicate one reason why existing interventions are ineffective in addressing health inequalities. For example, shorter expected life spans in low-income groups may negate appreciation of long-term smoking risk and therefore interventions based on investing in a healthy older age will have little relevance (Lawlor, Frankel, Shaw, Ebrahim, & Smith, 2003). There are a number of texts that explore lifeworlds affected by deprivation and low incomes (e.g. Abrams, 2002; Ehrenreich, 2001) and others that specifically explore the relationship between sociocultural contexts and health behavior (Krumeich et al., 2001; Parry, Mathers, Laburn-Peart, Orford, & Dalton, 2012); these could be incorporated into process evaluation research to inform elements such as intervention recruitment or acceptability.

Bigger range sociopolitical perspectives from the social science literature may also be helpful, for example, Burke et al. (2009) comment on Bourdieu’s concept of “habitus” and Gidden’s concept of “structuration” and their relevance for understanding the dynamic relationship between social contexts and health behaviors. One theoretical point, more closely aligned to Bourdieu than Giddens, is that social environments such as class hierarchies shape people’s behavior, their sense of what is morally right and their ideas about what is logically coherent. This occurs without the individual necessarily being conscious that they have internalized the historical and social forces that shape their social world. Whitehead (2007) addresses the relationship between theory and interventions by providing a typology of interventions based on different theoretical ideas about health inequality, referring to approaches such as strengthening communities, and improving workplaces. This is not to overstate the scope for small or large scale social theories to explain social problems or how interventions might work: social theories of ‘big’ social problems always have potential for improvement (Chen & Rossi, 1980) and persistent social problems which resist intervention and change have been characterized as “wicked issues” within the social policy community (e.g., Page, 2000). Furthermore, the challenge is to incorporate these types of social science perspectives, which in illuminating context add to conceptual complexity, into approaches that can also accommodate the dynamic processes of interventions discussed by Hawe et al. (2009).

Process Evaluation and Process

Currently published process evaluations in health behavior research tend to address various elements of intervention implementation such as recruitment, fidelity, retention, and acceptability. For example, the Reach, Efficacy/Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework categorizes aspects of an intervention at different levels for: reach, efficacy, adoption, implementation, and maintenance (Glasgow, Vogt, & Boles, 1999). This approach to conducting a process evaluation is relatively atomistic, although useful in that it defines certain domains of interest for research activity. What these categories lack as a whole, however, is a sense of a process. It is somewhat implied in the chronological order of RE-AIMs elements, beginning with intervention “reach” and ending with “maintenance” of the intervention but is not explicit. The concept of a “process” indicates a series of linked events and change over time, and the sequential order of elements of the intervention is a core aspect of what the process is. For example, in smoking cessation group interventions, it is not just whether people attend that is important or how acceptable they find the intervention but what steps they progress through to become a non-smoker that are the substance of the intervention and account for its outcomes. This includes what events take place, such as peer support or quitting techniques supplied by the group leader as well as how participants respond to planned or unplanned elements of the intervention. The order of these may matter as well, for example peer support might be important for keeping the participant attending the group sessions long enough to learn sufficient quitting techniques from the leader. It is striking when one compares the list of elements that tend to be employed in health behavior process evaluations to approaches used in the social sciences literature, as the latter very often provide a model, represented by a graphic, which is structured around a conceptual timeline. This is represented by text boxes or similar, connected by arrows, where the arrows show the direction of time. Models start from the beginning of an intervention through to what is theorized to occur during the intervention to what the expected outcomes are (Margoluis, Stem, Salafsky, & Brown, 2009; Zantal-Wiener & Horwood, 2010). Many contain similar elements such as “input,” “output,” “outcome,” and “context” and are usually arranged in a fairly linear way to demonstrate a process or a causal chain (Nesman et al., 2007). Some distinguish between short or intermediate outcomes and longer term outcomes (McLaughlin & Jordan, 1999; Sidani & Sechrest, 1999). Some address different levels such as individual, interpersonal, collective (James & Meezan, 2002) or micro, meso, macro (Virtanen & Uusikylä, 2004). In other respects, these graphical models vary in the amount of detail provided, the exact elements they contain, and the overall style of graphic. Possibly the model most familiar to health behavior researchers is the realist evaluation one (Pawson & Tilley, 2004) that prescribes the three elements “mechanism, context, and outcome” and the configuration between them. Two other types of process models in use are “logic models” and “logical frameworks,” which are used for program design as well as evaluation. Logic models have been promoted by the Kellogg Foundation (Kellogg Foundation, 2004) and the term is used in public health literature to an extent (e.g., Petticrew et al., 2009). Logical frameworks have been used in international development for both monitoring and evaluation. They outline, in a relatively concise and prescriptive grid, the anticipated pathway that an intervention is predicted to take, taking assumptions including features of the external environment into account (e.g., Gasper, 2000). Logic models and logical frameworks, similarly to the other graphics described above, include within them a narrative about a process of change. These various ways of modeling process in the social science literature encapsulate a better grasp of the nature of process since they represent a chain of events that are linked to each other and that lead to intervention outcomes. The “atomistic elements” in health behavior process evaluations mentioned above could be arranged in a linked chain of events—they are not redundant—but do not explain a process since they only describe certain components, often in isolation. Social science-based models of process, by contrast, make explicit the idea of an intervention as introducing change over time and the possibility of explaining the various elements of a process of change through the causal links represented.

Social science process evaluations, as well as addressing process more explicitly, are also more likely to link processes with outcomes and therefore to address causal explanations to a greater degree. Where process evaluation has been adopted in health behavior studies it tends to be a rather separate enterprise, where process evaluation is nested within the main study that largely examines the outcomes of an intervention using quantitative data. Social science models such as those described above outline a chain of events that link input, process and outcomes and in doing so produce a causal account of an intervention. For example an account of social participation in a community project which included accounts of processes such as confidence building among participants would also be linked to an explanation of outcomes such as increased employment levels, with an explicit pathway explaining job-seeking behavior as being supported by greater confidence, which in turn was promoted by the community project: there is a narrative of events that take place, involving the participant and the intervention, in which the outcome is the last event. This representation of a causal chain provides a basis for explaining intervention outcomes in more depth and detail. Models can represent causality in different ways and do not have to be complex to be useful (Rogers, 2000; Virtanen & Uusikylä, 2004). Process in health behavior research reporting, however, tends to be divorced from analysis of outcomes. Demonstrating robust and convincing evidence for causal relationships in research can be difficult and therefore identifying causes through the modeling of processes will not necessarily be straightforward. However, process evaluation models of change processes at least represent a tentative explanation that can be explored further and can facilitate the structuring of future research activity around exploring causal pathways. Models that represent interventions are limited by the fact that they are, by their nature, simplistic and generally linear representations of complex, messy, and reiterative processes (Mackenzie & Blamey, 2005). Interactions between elements of an intervention and feedback loops occur (Davies, 2005) as well as interactions with external factors that make accurate description and modeling of causal chains more difficult. This indicates the need to develop more sophisticated process models which provide a basis for exploring and understanding relatively complex dynamics of change.

Process Evaluation and Theory

All academic disciplines engage with theory; the MRC guidance for complex interventions recommends commencing evaluation with theoretical considerations (Craig et al., 2008a). Theory has a practical use for process evaluation as it explains how an intervention is expected to have an effect (Weiss, 1997a). A single intervention may be explained by several different theories: for example, a weight loss group might be theorized to help people lose weight through weight loss education, social support, competition between members, regular monitoring of progress in weekly weigh-ins or through the role modeling of a group leader who has lost weight and encourages others to do the same. These could be regarded as competing theories or the process evaluator might want to consider two or more as complementary theories. “Programmes are theories incarnate” (Pawson & Tilley, 2004, p.3), although theories may be lurking rather than explicit, superficial rather than extensive, articulated fully in advance or addressed as an afterthought. Theory matters for improving interventions, for building and refining scientific knowledge, for removing inactive or harmful elements, and for providing a rationale for one type of an intervention over another. Theory also has implications for validity and reliability, as this will inform whether the intervention is likely to work elsewhere or for different populations (Michie, Fixsen, Grimshaw, & Eccles, 2009). Theory is needed at the level of social problems, models of programs and interventions, and the interaction of the program with its environment (Chen & Rossi, 1980). Theory in the health behavior literature, apart from a limited discussion of health psychology theory, tends to be more implicit than explicit and is often not discussed in depth when results are reported.

Several relevant social science models of process evaluation address theory more extensively. “Theory-based evaluation” conceptualizes and organizes evaluation based on the theory of how an intervention has an effect. Weiss (1997b) distinguishes between program theory which is defined as what is done to implement an intervention, and implementation theory, which attempts to explain the effect of the intervention on its recipients. A classic sociological text that analyzes the former is Lipsky’s “Street-Level Bureaucracy” which emphasizes the importance of the discretion of frontline public service employees (the street-level bureaucrats) for how public services are delivered (Lipsky, 1980). Tilley further distinguishes between what a program ought to be doing—“supposed to do theory” (STD)—and unexpected processes which he terms “otherwise/also does theory” (OAD) (Tilley, 2004). Theory may be adopted in process evaluations using different models of application of theory (Lipsey & Pollard, 1989) and theory-based approaches can be applied to suit the particular process evaluation (Blamey & Mackenzie, 2007). For example, Dickinson (2006) recommends combining realist evaluation and theory of change approaches within a critical realist framework, and Weitzman, Silver, & Dillman (2002) discuss using theory-based evaluation within a quasi-experimental study. Logic models and logical frameworks do not always explicitly mention theory in conceptualizations of process; however, logic models are often used as a conceptual tool to represent a program theory or theory of change. The Kellogg Foundation Logic Model Development Guide (Kellogg Foundation, 2004) includes a section on theory-based approaches, and several studies have explicitly integrated the two approaches by using logic models to articulate the intervention theory (Fear, 2007; Jordan, 2010; Judge & Bauld, 2001; Nesman et al., 2007; Nilsen, 2007).

Two major subsets of theory-based evaluation are “realist evaluation” and “theory of change.” Realist evaluation was developed by Pawson and Tillley (1997) who challenge the explanatory power of RCTs on the grounds that one needs to explore and understand generative mechanisms at work rather than simply noting which outcomes are correlated with which interventions. Their model, as described earlier, identifies three key elements—context, mechanism, and outcome—and their configuration. The other main subset is “theory of change,” promoted by the Aspen Institute (Anderson, 2005; Connell, Kubisch, Schorr, & Weiss, 1995). Theory of change is distinctive in that it adopts a participatory approach where the theory is “surfaced” and discussed or negotiated between the evaluator and project implementers at the beginning of an intervention; this theory then guides and focuses the evaluation (Hughes & Traynor, 2000; Mackenzie & Blamey, 2005). An intervention to provide additional sports sessions for inactive school children, for example, would be discussed to identify the theory of how the intervention is expected to bring about change; this might be to draw inactive children into sports though the appeal of the camaraderie of teams, in which case the process evaluator would examine social aspects of the sessions. Alternatively, the anticipated mechanism might be to provide extra practice and skills coaching to increase children’s confidence, in which case the process evaluation would concentrate on teaching and skills development. Although theory ought to be developed and articulated prior to an intervention in theory of change approaches, in practice if this has not happened an evaluator may have to uncover it in discussion with implementers (Moore, 2010). A theory may also be constructed by the evaluator rather than implementers (Chen & Rossi, 1980). Mason and Barnes (2007) advocate building theory as an intervention progresses; this could be useful where theory is unclear or where there are many unknown factors at the beginning. Depending on the evaluation approach, multiple theories may emerge from different stakeholders, including the evaluator who may be aware of competing theoretical possibilities (Chen & Rossi, 1980); this could be useful in comparing the utility of different theoretical explanations. There are, then, different ways to apply theory to evaluation activity; the benefit of constructing process evaluations around theory is that the evaluation activity is directed, coordinated, and mapped more closely onto the underlying rationale of the intervention. A process evaluation is thus constructed to answer pertinent research questions rather than comprising a standardized tick-box exercise covering elements such as recruitment and fidelity.

As well as using theory in different ways in process evaluations, the scale of theory is also a consideration. Lipsey distinguishes between “small” program theories which articulate how interventions operate and “big” sociological theory that is usually applied at more abstract or generalized levels (Leviton & Lipsey, 2007); this distinction might be helpful for analysis at different levels of context. Lipsey (1997) further argues that findings from multiple individual intervention evaluations ought to be reviewed together to build more generalized theory about interventions. Realist evaluation also aims to develop “middle range theory” that operates between big theory and particulars of individual interventions, and which can operate across commonalities in intervention approaches to draw lessons from more than one (Pawson & Tilley, 2004). One might, for example, investigate what types of interventions seem to work best for hard-to-reach groups and, after finding that interventions close to where people live tend to have higher recruitment and retention rates for this population, theorize that a key element in their participation is proximity to home, and perhaps develop further theory about the geographies of social participation for these groups. In health behavior interventions, however, this is difficult because there is a tendency for elements and mechanisms of interventions, and their connection to theory, to be under-specified in reporting of results (Michie & Abraham, 2004). The examples from the social science literature presented here demonstrate how models of theory can be adopted in process evaluations and how theory, which if specified, can be used explicitly and flexibly in different types of process evaluations to investigate different aspects and levels of interventions.

Implications for Methodology

Developing process evaluation practice in health behavior research and addressing broader social factors will require engagement with new areas and methodologies (Smith & Petticrew, 2010). Several examples of process evaluations of health behavior interventions cited in this article reflect a positive trajectory of engagement with the social world and with social science evaluation approaches to a degree. Nevertheless, a challenge for the improvements suggested in this paper is the associated methodology required to develop engagement with context, process, and theory further, particularly qualitative methodologies which have much to offer in this area. However, there have traditionally been reservations about methods across the paradigm divide, including in health behavior research. There is a tendency still for biomedical scientists to regard only experimental methods as robust (Albert, Hodges, Regehr, & Lingard, 2008) and therefore to reject the qualitative methods often necessary in process evaluation. Conversely, there have also been reservations about the value of randomized controlled trials (Mackenzie, O’Donnell, Halliday, Sridharan, & Platt, 2010). An alternative, more pragmatic, perspective is acknowledgment of how mixed methods add to explanation and rigor in health research. For example, a qualitative investigation of implementation processes may improve the internal validity of an intervention study. However, qualitative studies are not used in all trials or integrated well with quantitative data (Glenton, Lewin, & Scheel, 2011; Lewin, Glenton, & Oxman, 2009), and if they are carried out are not always reported in process evaluation findings (Moore, 2010).

Recommendations

Qualitative methods, traditionally used to a greater extent in the social sciences, bring several benefits to process evaluation research and to the examination of context, process, and theory. They are valuable for discovering unintended or unanticipated consequences that sometimes occur (Craig et al., 2008b; Tilley, 2004), which is useful for identifying, for example, contextual factors which may be difficult or impossible to predict. This type of discovery can be especially useful for building theory (Chen & Rossi, 1980) but currently represents untapped potential. The type of methodologies required are relatively open, exploratory approaches such as “grounded theory” based qualitative research that takes a “bottom-up” approach to theorizing, or ethnography. Ethnography is also a method ideally suited for complexity in health research (Huby, Hart, McKevitt, & Sobo, 2007) as it is holistic and ideal for collecting data in a natural setting (Hong et al., 2005); this could be useful in pragmatic trials for investigating complex processes and for understanding particular interactions between interventions and contexts. Additionally, ethnography is suited to exploring sociocultural factors as it provides an emic (insider) perspective rather than imposing an etic (external), “rational actor”, model of research participants. In other words, it can provide an insight into the research participant’s world and how they perceive a health issue rather than assuming participants’ ideas of appropriate health behavior are the same as those of a researcher or policy maker. The other major methodological challenge is the emerging area of “mixed methods,” particularly for linking process and outcome data where the data for the former may be largely qualitative and the latter largely quantitative, presenting difficulties such as how to combine paradigms (Munro & Bloor, 2010).

Discussion

Process evaluations of health behavior change interventions is a developing area; this article has suggested and discussed social science approaches that could benefit this field. It has argued that further development requires an expanded appreciation of context, a more direct engagement with the idea of process, better incorporation of theory and the adoption of new methodological approaches. Of course, these areas all relate to each other: theory-driven methodologies should attempt to explain how multiple processes interact over time to produce outcomes within a complex, possibly multilevel, intervention that itself interacts with a complex context. Theories of effective health behavior change interventions will develop and improve through better understanding of how processes operate within contexts.

Recommendations

More sophisticated health behavior process evaluations which draw on social science approaches should drive innovation and improvement in health behavior interventions. This has a particular relevance for decreasing health inequalities where little is known about effective approaches in different domains (Bambra et al., 2010). Health behavior process evaluations that incorporate wider socio-economic factors are particularly relevant for health inequality as these would lead to better engagement with relevant social contexts and drivers of health inequality as well as differential pathways through interventions by low-income groups. Current literature in health such as the Marmot Report (Marmot et al., 2010) is limited in that it tends to merely point to correlations with broad structural issues that produce health inequality rather than clearly identifying implementable health policy (Chandra & Vogl, 2010). This is not to claim that the social science literature has all the answers—areas of weakness have been noted—but to highlight the benefits of cross-disciplinary research between these two fields.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The work was undertaken with the support of The Centre for the Development and Evaluation of Complex Interventions for Public Health Improvement (DECIPHer), a UKCRC Public Health Research: Centre of Excellence. Funding from the British Heart Foundation, Cancer Research UK, Economic and Social Research Council (RES-590-28-0005), Medical Research Council, the Welsh Government and the Wellcome Trust (WT087640MA), under the auspices of the UK Clinical Research Collaboration, is gratefully acknowledged.

References

Abrams

(2002). Below the breadline: Living on the minimum wage. London, UK: Profile Books.

Albert

M. S. L.

Hodges

Regehr

Lingard

(2008). Biomedical scientists' perception of the social sciences in health research. Social Science and Medicine, 66, 2520–2531.

Anderson

(2005). The community builder's approach to theory of change: A practical guide to theory development. New York, NY: The Aspen Institute.

Bambra

Gibson

Sowden

Wright

Whitehead

Petticrew

(2010). Tackling the wider social determinants of health and health inequalities: evidence from systematic reviews. Journal of Epidemiology and Community Health, 64, 284–291. doi:10.1136/jech.2008.082743

Blamey

Mackenzie

(2007). Theories of change and realistic evaluation. Evaluation, 13, 439–455. doi:10.1177/1356389007082129

Bonell

Fletcher

Morton

Lorenc

Moore

(2012). Realist randomised controlled trials: A new approach to evaluating complex public health interventions. Social Science and Medicine, 75, 2299–2306.

Bunton

Murphy

Bennett

(1991). Theories of behavioural change and their use in health promotion: some neglected areas. Health Education Research, 6, 153–162. doi:10.1093/her/6.2.153

Burke

N. J.

Joseph

Pasick

R. J.

Barker

J. C.

(2009). Theorizing social context: Rethinking behavioral theory. Health Education & Behavior: The Official Publication of the Society for Public Health Education, 36, 55S–70S.

Byng

Norman

Redfern

Jones

(2008). Exposing the key functions of a complex intervention for shared care in mental health: Case study of a process evaluation. BMC Health Services Research, 8, 274.

10.

Carvalho

White

(2004). Theory-based evaluation: The case of social funds. The American Journal of Evaluation, 25, 141–160. doi:10.1016/j.ameval.2004.03.001

11.

Chandra

Vogl

T. S.

(2010). Rising up with shoe leather? A comment on fair society, healthy lives (The Marmot Review). Social Science & Medicine, 71, 1227–1230. doi:http://dx.doi.org/10.1016/j.socscimed.2010.07.006

12.

Charreire

Weber

Chaix

Salze

Casey

Banos

Oppert

J. M.

(2012). Identifying built environmental patterns using cluster analysis and GIS: Relationships with walking, cycling and body mass index in French adults. The International Journal of Behavioral Nutrition and Physical Activity, 9, 59.

13.

Chen

H.-T.

Rossi

P. H.

(1980). The multi-goal, theory-driven approach to evaluation: A model linking basic and applied social science. Social Forces, 59, 106–122.

14.

Connell

Kubisch

Schorr

Weiss

(1995). New Approaches to Evaluating Community Initiatives: Concepts, Methods and Contexts. Washington, DC: The Aspen Institute.

15.

Craig

Dieppe

Macintyre

Michie

Nazareth

Petticrew

(2008a). Developing and Evaluating Complex Interventions: New Guidance. London, UK: MRC.

16.

Craig

Dieppe

Macintyre

Michie

Nazareth

Petticrew

(2008b). Developing and evaluating complex interventions: The new medical research council guidance. British Medical Journal, 337, a1655.

17.

Davies

(2005). Scale, complexity and the representation of theories of change. Evaluation, 11, 133–149. doi:10.1177/1356389005055528

18.

Dickinson

(2006). The evaluation of health and social care partnerships: An analysis of approaches and synthesis for the future. Health & Social Care in the Community, 14, 375–383.

19.

Donaldson

S. I.

Gooler

L. E.

(2003). Theory-driven evaluation in action: Lessons from a $20 million statewide work and health initiative. Evaluation and Program Planning, 26, 355–366. doi:10.1016/s0149-7189(03)00052-1

20.

Edwards

Di Ruggiero

(2011). Exploring which context matters in the study of health inequities and their mitigation. Scandinavian Journal of Public Health, 39, 43–49. doi:10.1177/1403494810393558

21.

Ehrenreich

(2001). Nickel and dimed: On (Not) getting by in boom-time America. New York, NY: Metropolitan.

22.

Fear

(2007). Programme evaluation theory: The next step toward a synthesis of logic models and organisational theory. Journal of MultiDisciplinary Evaluation, 4, 13–25.

23.

Gasper

. (2000). Evaluating the ‘logical framework approach’ towards learning-oriented development evaluation. Public Administration and Development, 20, 17–28. doi:10.1002/1099-162x(200002)20:1<17::aid-pad89>3.0.co;2-5

24.

Glasgow

R. E.

Vogt

T. M.

Boles

S. M.

(1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health, 89, 1322–1327.

25.

Glenton

Lewin

Scheel

I. B.

(2011). Still too little qualitative research to shed light on results from reviews of effectiveness trials: A case study of a Cochrane review on the use of lay health workers. Implementation Science, 6, 53.

26.

Golden

S. D.

Earp

J. A.

(2012). Social ecological approaches to individuals and their contexts: Twenty years of health education & behavior health promotion interventions. Health Education & Behavior: The Official Publication of the Society for Public Health Education, 39, 364–372.

27.

Greenhalgh

Robert

Macfarlane

Bate

Kyriakidou

(2004). Diffusion of innovations in service organizations: Systematic review and recommendations. Milbank Quarterly, 82, 581–629. doi:10.1111/j.0887-378X.2004.00325.x

28.

Hawe

Shiell

Riley

(2009). Theorising interventions as events in systems. American journal of community psychology, 43, 267–276.

29.

Hong

Mitchell

S. G.

Peterson

J. A.

Latkin

C. A.

Tobin

Gann

(2005). Ethnographic process evaluation: Piloting an HIV prevention intervention program among injection drug users. International Journal of Qualitative Methods, 4, 1–12.

30.

Huby

Hart

McKevitt

Sobo

(2007). Addressing the complexity of health care: The practical potential of ethnography. Journal of Health Services Research and Policy, 12, 193–194.

31.

Hughes

Traynor

(2000). Reconciling process and outcome in evaluating community initiatives. Evaluation, 6, 37–49. doi:10.1177/13563890022209109

32.

James

Meezan

(2002). Refining the evaluation of treatment foster care. Families in Society, 83, 233–244.

33.

Jordan

G. B.

(2010). A theory-based logic model for innovation policy and evaluation. Research Evaluation, 19, 263–273. doi:10.3152/095820210x12827366906445

34.

Judge

Bauld

(2001). Strong theory, flexible methods: Evaluating complex community-based initiatives. Critical Public Health, 11, 19–38.

35.

Kellogg Foundation. (2004). Logic model development guide. Battle Creek, MI: Kellogg Foundation.

36.

Krumeich

Weijts

Reddy

Meijer-Weitz

(2001). The benefits of anthropological approaches for health promotion research and practice. Health Education Research, 16, 121–130.

37.

Lawlor

D. A.

Frankel

Shaw

Ebrahim

Smith

G. D.

(2003). Smoking and ill health: Does lay epidemiology explain the failure of smoking cessation programs among deprived populations? American Journal of Public Health, 93, 266–270.

38.

Leviton

L. C.

Lipsey

M. W.

(2007). A big chapter about small theories: Theory as method: Small theories of treatments. New Directions for Evaluation, 2007, 27–62. doi:10.1002/ev.224

39.

Lewin

Glenton

Oxman

A. D.

(2009). Use of qualitative methods alongside randomised controlled trials of complex healthcare interventions: methodological study. British Medical Journal, 339, b3496.

40.

Lipsey

M. W.

(1997). What can you build with thousands of bricks? Musings on the cumulation of knowledge in program evaluation. New Directions for Evaluation, 1997, 7–23. doi:10.1002/ev.1084

41.

Lipsey

M. W.

Pollard

J. A.

(1989). Driving toward theory in program evaluation: More models to choose from. Evaluation and Program Planning, 12, 317–328. doi:10.1016/0149-7189(89)90048-7

42.

Lipsky

(1980). Street-level bureaucracy: Dilemmas of the individual in public services. New York, NY: Russell Sage Foundation.

43.

Mackenzie

Blamey

(2005). The practice and the theory. Evaluation, 11, 151–168. doi:10.1177/1356389005055538

44.

Mackenzie

O’Donnell

Halliday

Sridharan

Platt

(2010). Do health improvement programmes fit with MRC guidance on evaluating complex interventions? British Medical Journal, 340, 401–403. doi:10.1136/bmj.c185

45.

Marchal

van Belle

van Olmen

Horerée

Kegels

(2013) Is realist evaluation keeping its promise? A review of published empirical studies in the field of health systems research. Evaluation, 18, 2, 192–212.

46.

Margoluis

Stem

Salafsky

Brown

(2009). Using conceptual models as a planning and evaluation tool in conservation. Evaluation and Program Planning, 32, 138–147. doi:10.1016/j.evalprogplan.2008.09.007

47.

Marmot

Allen

Goldblatt

Boyce

McNeish

Grady

Geddes

(2010). Fair society, health lives. Strategic review of health inequalities in England post-2010 (The Marmot review). London, UK: The Marmot Review.

48.

Mason

Barnes

(2007). Constructing theories of change. Evaluation, 13, 151–170. doi:10.1177/1356389007075221

49.

McLaughlin

J. A.

Jordan

G. B.

(1999). Logic models: A tool for telling your programs performance story. Evaluation and Program Planning, 22, 65–72. doi:10.1016/s0149-7189(98)00042-1

50.

Michie

Abraham

(2004). Interventions to change health behaviours: Evidence-based or evidence-inspired? Psychology & Health, 19, 29–49.

51.

Michie

Fixsen

Grimshaw

Eccles

(2009). Specifying and reporting complex behaviour change interventions: The need for a scientific method. Implementation Science, 4, 40.

52.

Moore

(2010). Developing a mixed methods framework for process evaluations of complex interventions: The case of the National Exercise Referral Scheme policy trial in Wales. PhD Thesis, Cardiff University.

53.

Munro

Bloor

(2010). Process evaluation: The new miracle ingredient in public health research? Qualitative Research, 10, 699–713.

54.

Murray

Treweek

Pope

MacFarlane

Ballini

Dowrick

May

(2010). Normalisation process theory: A framework for developing, evaluating and implementing complex interventions. BMC Medicine, 8, 63.

55.

Nesman

T. M.

Batsche

Hernandez

(2007). Theory-based evaluation of a comprehensive Latino education initiative: An interactive evaluation approach. Evaluation and Program Planning, 30, 267–281. doi:10.1016/j.evalprogplan.2007.04.004

56.

Nilsen

(2007). The how and why of community-based injury prevention: A conceptual and evaluation model. Safety Science, 45, 501–521. doi:10.1016/j.ssci.2006.07.009

57.

Page

(2000). Communities in the balance: The realities of social exclusion on housing estates. York, England: Joseph Rowntree Foundation.

58.

Palareti

Berti

(2009). Different ecological perspectives for evaluating residential care outcomes: Which window for the black box? Children and Youth Services Review, 31, 1080–1085. doi:10.1016/j.childyouth.2009.07.011

59.

Parry

Mathers

Laburn-Peart

Orford

Dalton

(2012). Improving health in deprived communities: What can residents teach us? Critical Public Health, 17, 123–136.

60.

Pawson

Tilley

(1997). Realistic evaluation. London, UK: Sage.

61.

Pawson

Tilley

(2004). Realist evaluation. London, UK: Sage.

62.

Petticrew

Tugwell

Welch

Ueffing

Kristjansson

Armstrong

Waters

(2009). Better evidence about wicked issues in tackling health inequities. Journal of Public Health, 31, 453–456. doi:10.1093/pubmed/fdp076

63.

Richard

Gauvin

Raine

(2011). Ecological models revisited: Their uses and evolution in health promotion over two decades. Annual Review of Public Health, 32, 307–326.

64.

Rogers

P. J.

(2000). Causal models in program theory evaluation. New Directions for Evaluation, 2000, 47–55. doi:10.1002/ev.1181

65.

Rogers

P. J.

Weiss

C. H.

(2007). Theory-based evaluation: Reflections ten years on: Theory-based evaluation: Past, present, and future. New Directions for Evaluation, 2007, 63–81. doi:10.1002/ev.225

66.

Shiell

Hawe

Gold

(2008). Complex interventions or complex systems? Implications for health economic evaluation. BMJ, 336, 1281–1283. doi:10.1136/bmj.39569.510521.AD

67.

Sidani

Sechrest

(1999). Putting program theory into operation. The American Journal of Evaluation, 20, 227–238. doi:10.1016/s1098-2140(99)00022-3

68.

Smith

R. D.

Petticrew

(2010). Public health evaluation in the twenty-first century: Time to see the wood as well as the trees. Journal of Public Health, 32, 2–7. doi:10.1093/pubmed/fdp122

69.

Steckler

Linnan

(Eds.). (2002). Process evaluation for public health interventions and research. San Francisco, CA: Jossey-Bass.

70.

Stokols

Allen

Bellingham

R. L.

(1996). The social ecology of health promotion: Implications for research and practice. American Journal of Health Promotion, 10, 247–251.

71.

Tilley

(2004). Applying theory-driven evaluation to the British crime reduction programme. Criminal Justice, 4, 255–276. doi:10.1177/1466802504048465

72.

Virtanen

Uusikylä

(2004). Exploring the missing links between cause and effect. Evaluation, 10, 77–91. doi:10.1177/1356389004043136

73.

Weiss

C. H.

(1997a). How can theory-based evaluation make greater headway? Evaluation Review, 21, 501–524. doi:10.1177/0193841x9702100405

74.

Weiss

C. H.

(1997b). Theory-based evaluation: Past, present, and future. New Directions for Evaluation, 1997, 41–55. doi:10.1002/ev.1086

75.

Weitzman

B. C.

Silver

Dillman

K.-N.

(2002). Integrating a comparison group design into a theory of change evaluation: The case of the urban health initiative. The American Journal of Evaluation, 23, 371–385. doi:10.1016/s1098-2140(02)00236-9

76.

Wells

Williams

Treweek

Coyle

Taylor

(2012). Intervention description is not enough: Evidence from an in-depth multiple case study on the untold role and impact of context in randomised controlled trials of seven complex interventions. Trials, 13, 95.

77.

Whitehead

(2007). A typology of actions to tackle social inequalities in health. Journal of Epidemiology and Community Health, 61, 473–478.

78.

Zantal-Wiener

Horwood

T. J.

(2010). Logic modeling as a tool to prepare to evaluate disaster and emergency preparedness, response, and recovery in schools. New Directions for Evaluation, 126, 51–64.