Assessing the Societal Impact of Research: The Relational Engagement Approach

Abstract

Marketing and policy researchers aiming to increase the societal impact of their scholarship should engage directly with relevant stakeholders. For maximum societal effect, this engagement needs to occur both within the research process and throughout the complex process of knowledge transfer. The authors propose that a relational engagement approach to research impact complements and builds on traditional approaches. Traditional approaches to impact employ bibliometric measures and focus on the creation and use of journal articles by scholarly audiences, an important but incomplete part of the academic process. The authors recommend expanding the strategies and measures of impact to include process assessments for specific stakeholders across the entire course of impact, from the creation, awareness, and use of knowledge to societal impact. This relational engagement approach involves the cocreation of research with audiences beyond academia. The authors hope to begin a dialogue on the strategies researchers can use to increase the potential societal benefits of their research.

Keywords

research impact relational engagement transformative consumer research societal benefit impact assessment

Journal of Public Policy & Marketing (JPPM) has a relatively high journal impact factor in part because it publishes research that grapples with timely and practical policy problems. For example, recent special issues of JPPM have focused on important social topics such as consumption constraints, social entrepreneurship, transformative consumer research, and marketplace diversity and inclusion. In a JPPM editorial, Editor in Chief David Stewart (2013) calls for increasing research that serves public and societal interests. This essay offers a critical reflection on the nature of research impact sought by journals such as JPPM. Much is at stake in defining the strategies and measures of research impact (Smith, Crookes, and Crookes 2013). The international business school accreditation body, Association to Advance Collegiate Schools of Business (AACSB), recently issued new standards to hold universities accountable; it states that high-quality intellectual contributions should “impact the theory, practice, and teaching of business and management” (AACSB International 2013, p. 16). Emerging progressive tendencies in marketing (e.g., transformative consumer research) promote exploration of new types of research that will result in new measures of impact (Davis and Pechmann 2013; Mick et al. 2012; Ozanne 2011; Özçağlar-Toulouse and Burroughs 2014). If research aimed at benefiting society takes center stage, how will this research be performed and evaluated? The purpose of this article is to reflect on traditional approaches to the meaning and assessment of impact, suggest a broader perspective across the complex process of societal impact, and encourage researchers to explore new forms of productive interaction with end users.

We propose a relational engagement approach, arguing that knowledge products created through persistent interactions between academics and other stakeholders are more likely to affect positive social change. Specifically, researchers employing relational engagement approaches work more directly with the external constituency they hope to serve. Thus, they can potentially forge good-quality relationships that involve reciprocal interactions and colearning. Collaborative research potentially delivers a wider range of direct research outputs (e.g., the ability to build stronger social networks through which insights can be shared) than traditional research approaches. We believe more researchers should work with invested stakeholders, sharing the power to define problems and create and use knowledge that can benefit society. In this way, the research process becomes more multivocal; that is, it includes the interests and insights of the end users to balance rigor against relevance. We hope to inspire more debate on how to better fulfill the implicit social contract that academic research should enrich society.

First, we highlight the problem of defining “research impact.” Second, we discuss traditional approaches to impact currently used in marketing. Third, we provide a framework for the strategies and measures of societal impact, which we call the relational engagement approach. Finally, we present exemplars in marketing that demonstrate the process of the relational engagement approach.

What is Research Impact?

Governments, academic institutions, and funding agencies increasingly want some form of accountability for the financial resources invested in academic research (Wiek et al. 2014). In the largest experiment on research impact assessment to date, the United Kingdom uses the Research Excellence Framework (REF) to assess the impact of published academic research and allocate funds to projects that meet their evaluative criteria (REF 2014). Yet assessing research impact is difficult and controversial. Consider the recent use of the commercial software Academic Analytics to measure academic productivity as one component of impact and the debate arising over its accuracy and use (Wexler 2015). Still, funding agencies that invest millions of dollars into research justifiably worry about the translation of efficacious findings into practice. For example, a funding agency of medical research (a field highly scrutinized for its societal impact) might question whether the research contributes to the amelioration of disease (Ioannidis 2004). For a research team to chip away at this laudable goal, they must complete studies with rigor, publish their findings, gain visibility, influence practitioners, integrate their work into common practice, and, finally, demonstrate some measurable improvement (Weiss 2007). It is a formidable task to engage in this full process of research impact.

Given the importance of this process, it is unsurprising that even the meaning of “impact” engenders considerable controversy. Confusion arises in part because “impact” is often used to capture four different points in the process: the creation of the research, the awareness of the findings, the use of the research, and the potential societal benefits of the research (De Jong et al. 2014). The first meaning of impact is the creation of a knowledge product: a journal article, book chapter, conference presentation, model, theory, decision aid, or innovation, to name but a few possible knowledge products. This stage is where academics most wisely invest their expertise. The second meaning of impact is the awareness of the findings, sometimes achieved by media interviews or press releases. The third meaning involves the use of the knowledge product, such as when a consumer, policy maker, or marketing manager adopts the research idea. For example, citation analysis captures the spread of a knowledge product, which is often the use of an article by another academic (Cote, Leong, and Cote 1991). The fourth meaning is when the knowledge product has societal benefits. For example, the definition of research impact by REF (2014) relates to this fourth meaning: “an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia.” The first three steps are necessary but insufficient for achieving societal benefits.

We first explore the traditional approaches used to assess impact in marketing. Of course, the meaning and assessment of impact varies widely on the basis of the interests of the party evaluating the research (Weiss 2007). University administrators, for example, need measures of impact for assessing educator promotion and tenure and often prefer more quantifiable measures that focus on the direct outcomes of research investments (Smith, Crookes, and Crookes 2013). To avoid using the term “impact,” and thus conflating its multiple meanings, throughout the article we instead specify the creation, awareness, use, and societal benefits of research.

Traditional Approaches to Research Impact

The traditional approaches to impact focus primarily on the creation of knowledge outputs—the number of publications in highly ranked journals—and the use of the knowledge products measured by citations (Malhotra 1996; Sprott and Miyazaki 2002). Many databases exist for counting citations, including Google Scholar, Scopus, and Thomson Reuters Web of Science. Tracing use through citation patterns is consistent with the view of science and knowledge production in which researchers produce decontextualized theoretical knowledge that is politically neutral and flows linearly from academics to the general populace (Murray and Ozanne 2009).

Scholars who publish highly cited articles have evidence that their work is being used. Citation metrics, though, quantify an individual researcher's influence primarily within the scholarly community. For example, an individualized “h-index” combines both the citations and the number of publications into a single score (i.e., a scholar with an index of h has published N papers, and h of those N papers have at least h citations each; Hirsch 2005). A high h-index denotes a scholar's high research creation and use within the academy. Institutional incentives reinforce this approach when these metrics are used in promotion and tenure processes. Similarly, the journal impact factor (JIF) serves as a metric for measuring the use of articles published in a journal (Shankar 2009). The JIF captures the average number of citations of articles published in the journal (Alexander, Scherer, and Lecoutre 2007). Journals with a high JIF are cited more than publications in lower or unranked journals. These bibliometric impact measures are easily available and quantifiable, making them the primary measure of research excellence (Aguinis et al. 2014; Smith, Crookes, and Crookes 2013; Wiek et al. 2014).

However, these assessment measures have shortcomings: review articles, generalist journals, and self-citations inflate the impact factor; a bias exists toward English-language journals; and widespread errors in reporting have been found (Smith, Crookes, and Crookes 2013). In addition, scientists may cite the most established and authoritative work to persuade reviewers of the value of their own research (Moed and Garfield 2004). Social motives may also inspire citation, such as rewarding a colleague or affirming an academic network. Consequently, citation counts are sometimes criticized for being biased toward a small group of journals, which are not necessarily read by consumers, practitioners, or policy makers (Erkut 2002).

Unintended consequences also arise when bibliometric measures are equated with societal benefits. Critics suggest that bibliometric measures undermine academic freedom because researchers are forced to preserve their reputation in a specific academic community and maximize in-field citations (Curtis 2007). Moreover, quality and citation counts are not necessarily correlated; a flawed article in a journal with a high JIF might be heavily cited by researchers attacking the article. This stance may also discourage multidisciplinary projects, which may take longer and are often published in journals in other fields. Researchers may even be incentivized to divide up research projects into smaller pieces to maximize their number of publications. Although bibliometric measures provide an easy way to compare researchers’ records, it may limit creativity and encourage scholars to play it safe, such as by selecting well-established research problems.

Furthermore, bibliometric indices do not measure the societal benefits of research, which involve more complex, dynamic, and indirect processes (Aguinis et al. 2014; Van Raan 2005). Societal benefits of research occur over time and are assessed from the perspective of the interests of different constituencies (Morton 2015). From a researcher's perspective, societal benefits are indirect outcomes of research over which they have far less control. For example, a researcher may devise and publish a theoretically rigorous and effective intervention program that is adopted by a governmental agency, but subsequently, through no fault of the researcher, the agency could cut the program's funding. Thus, the researcher created a knowledge product that was adopted, but no societal benefit occurred.

Traditionally, researchers are held accountable for the outputs over which they have direct control (i.e., knowledge products). The recent changes in the REF and AACSB call for researchers to have greater accountability for the broader effects of their research, leaving some to question the role of academic researchers in society. For example, do academics have a social contract with society, and are we accountable for ensuring that our research has positive societal outcomes? What is the value of unapplied research, whereby the practical outcomes may be years off and relevant end users are still unknown? We suggest that a middle-ground position exists that focuses more on process and complements traditional approaches: the relational engagement approach to research impact. In this approach, we argue that smart intermediary steps exist for researchers who aim for societal benefits in their research. These engagement practices represent a fair contract between the academy and society.

In the next section, we provide a framework for the societal benefit of research that begins to capture the complexity of this process. We explore the power of relational engagement research approaches to influence the knowledge creation, awareness, use, and societal benefit of research. We also discuss examples of marketing researchers who work more directly with other stakeholders.

The Relational Engagement Approach: Increasing Research's Societal Benefit

Societal benefit assessment is an emerging research priority with no clear consensus on best practices (Aguinis et al. 2014). The impact of research on society occurs over time and is a complex and indirect process with multidirectional influences (Morton 2015). The time lag between direct research outputs and societal benefits means that an attribution problem exists (Spaapen and Van Drooge 2011). Nevertheless, we integrate some of the most common features across existing approaches to provide an initial framework of this complicated process (De Jong et al. 2014; Morton 2015; Spaapen and Van Drooge 2011; Weiss 2007; Wiek et al. 2014).

As discussed previously, the traditional approach to research impact focuses more on the creation of direct research outputs, particularly knowledge products such as peer-reviewed articles, assessed using bibliometric measures. In Figure 1, the gray boxes trace the traditional approach to research impact most frequently employed. However, a relational engagement approach can generate additional direct outputs such as productive interactions, enhanced capacities, and social networks, which we discuss next. We offer the caveat that any distillation of such a complex and dynamic process will be an oversimplification. Many exceptions exist, but we offer this framework as an orienting tool for productive conversations on rethinking current practices.

Figure 1

A Framework for the Relational Engagement Approach to Impact

As we explain herein, documenting social benefits is fraught with difficulties. For example, in medical research, a 17-year time lag is estimated between medical trials demonstrating efficacy and consequent shifts in clinical practice (Weiss 2007). As an example from across all university fields, a recent textual mining and qualitative analysis of the 6,679 impact cases evaluated in the REF (2014) reported a three-to nine-year time lag, presumably among the best cases presented. Moreover, an astonishing 3,709 different pathways were found for research providing a societal benefit (Higher Education Funding Council for England [HEFCE] 2015). Despite the complexity of this challenge, there is increasing pressure for academic researchers to demonstrate societal benefits. We suggest that after creating direct research output, all researchers might focus on some of the direct and indirect process outcomes over which they have greater control. Figure 1 provides an overview of the general process of societal impact, beginning with the creation of direct research outputs and then moving to research awareness. As the dashed lines denote, researchers have less direct control over the later stages of research use and societal benefit. Moreover, even though Figure 1 presents this process as linear, a researcher might work on several incremental projects before building a critical body of research and gaining wider awareness and use. Alternatively, a research finding might be used and found inadequate, fueling additional research. Next, we explore how a relational engagement approach generates a broader portfolio of outputs.

Creation of Traditional and Nontraditional Direct Research Outputs

This process begins with the creation of direct research outputs emerging from the research process. As defined previously, the primary direct research outputs are externally validated knowledge products, such as published peer-reviewed work, presentations, theories, decision aids, and models, to name a few. Currently, these outputs are measured using bibliometric or scholarly output measures, such as number of publications and citations. We affirm that the peer-review process plays a crucial gatekeeper role and is arguably the best approach for ensuring that published research meets high standards of rigor. We also uphold the value of academic freedom and the importance of encouraging a portfolio of both basic and more practical research. Clearly, the academic stakeholder group is an important audience for research. Yet other important nonacademic stakeholders exist, including executives, marketing managers, policy makers, consumers, activists, and nonprofits.

Nonacademic Knowledge Products

Many marketing researchers are well aware of problems with the traditional strategies and measures of impact and are exploring alternatives to elevate the interests of external constituencies. At one end of the spectrum, this engagement is more basic, such as beginning the research process by focusing on problems deemed relevant for managerial stakeholders (Lehmann, McAlister, and Staelin 2011). Indeed, Stewart (2013) calls for more scholarly research that aids policy makers’ pressing concerns. Marketing researchers also engage with consumer problems with the goal of inspiring consumers and governments to take actions in their own self-interests (Epstein and Yuthas 2012; Viswanathan et al. 2009). Here, the basic goal is to create problem-focused knowledge of practical intent; for example, practical goals include improving packaging labels to help consumers make better choices and providing managers with better decision-making tools.

One challenge for achieving greater societal benefit is that academic journal articles are not necessarily written for or read by downstream stakeholders. Therefore, many researchers in marketing also create knowledge products for existing consumer, managerial, or governmental policies or practices. These researchers may target research outlets that their stakeholders read (e.g., JPPM, Harvard Business Review, news agencies, trade books). Table 1 provides concrete examples of the relational engagement approach within and outside the academy, including alternative knowledge products sensitive to external stakeholders.

Table 1

Relational Engagement Approach Examples Outside (vs. Inside) the Academy

	Inside the Academy	Outside the Academy
Knowledge products	Articles, chapters, books, theories, models, and measurement instruments	Government reports, policy briefings, guidelines, practitioner publications (e.g., trade publications, newsletters), case studies, textbooks, webinars, decision-making tools, and essays
Productive interactions (before, during, and after research)	Participation in academic conferences, panels, or roundtable discussions; invited presentations at universities; research workshops	Participation on advisory boards or policy development committees, collaborative research, informal meetings, media interviews, presentations, demonstrations, consultation with organizations, provision of expert testimony before a legislative body
Enhanced capacity	New research skills, enhanced understanding, new perspectives, new analytical techniques	New skills in research, analysis, reflection, and communication within a relational engagement approach; development of a shared language and common understanding; new models and theories; development of a community with shared concerns
Improved networks	Development of academic networks, membership in academic organizations, participating on committees, and holding offices in academic organizations	Development of social networks among policy makers, businesses, consumer groups, and nonprofits; participation on multidisciplinary research teams; requests for time from different stakeholder groups; faculty sabbatical at an organization or regulatory agency
Awareness and use	Academic conference presentations, blog posts on academic websites, awards	Traditional media coverage; websites promoting the research; telenovelas and videographies; altmetrics footprints (e.g., number of bookmarks, presence on social media); webometrics of the research (e.g., Mendeley, CiteULike, Tweets, Facebook, Google+, Reddit, Internet forums, LinkedIn); citations in blogs and web encyclopedias; noneducational entries on search engines; number of followers on social media; readers’ engagement with the research (e.g., number of likes and comments about the research); citations in the development of guidelines, policy, or regulations; citation in the development of standards, best practices, and textbooks

Notes: This incomplete list is meant to make the relational engagement approach framework concrete rather than to imply a linear process (i.e., many elements in one box might also apply to another box).

A relational engagement approach also proposes to nudge the production of knowledge products toward greater relevancy by including stakeholders in the creation process. As we explain next, this approach may even improve the accuracy of data on which the research is based. Research performed in collaboration with key stakeholders also yields additional direct research outputs, which do not necessarily arise when doing traditional research. Three possible sets of direct outputs of relational engagement research are productive interactions, enhanced capacities, and improved social networks (see the “Creation of Direct Research Outputs” column in Figure 1).

Productive Interactions

Productive interactions are “exchanges between researchers and stakeholders in which knowledge is produced and valued that is both scientifically robust and socially relevant“ (Spaapen and Van Drooge 2011, p. 210, emphasis added). Productive interactions do not assume a linear model of knowledge dissemination from researcher to user but rather assume that influences arise in complex networks that are dynamic and multidirectional. For example, a research-based decision model for better managerial decision making may be modified because of contextual demands, such as the reality that managers face time pressures or financial constraints. Thus, over time, a productive interaction between researchers and end users would develop a more rigorous and relevant model for use because key contextual features were included.

The productive interaction contrasts with the traditional approach, which casts the researcher as the epistemological teller who preserves objectivity by keeping stakeholders at arm's length and treating them as subjects and sources of data (Bourdieu 1984). The researcher questions, researches, and advises. However, this approach leaves the researcher susceptible to data errors, such as oversimplification and misinterpretation, when historical and relevant contextual factors are not incorporated in the data collection and analysis. A relational engagement approach recognizes the complex, dynamic, and value-laden terrain of social problems and aims to incorporate this into the research process.

Researchers within this approach take a more active role with constituencies who might benefit from the research, including consumers, activists, managers, and policy makers. At minimum, this relationship is a partnership, and the researcher engages with relevant stakeholders building on their everyday understandings, interests, and expertise. Research collaborations should “provide workable solutions to immediate concerns” (Ozanne and Saatcioglu 2008, p. 424). The researcher is one member of an assemblage seeking societal benefits. However, collaboration does not mean that academics relinquish their voice and become subservient to business, government, or consumer groups. Arguably, researchers might generate knowledge critical of managerial, policy, and consumer practices.

Yet when a network mobilizes around a specific cause, the potential for societal impact will be greater than if a lone individual advocates for a cause. Productive interactions between researchers and policy makers, whether within formal or informal partnerships, are crucial for maximizing societal benefit (Court and Young 2006; De Jong et al. 2014). So, relationship development requires active participation and time, but the researchers’ payoff is the opportunity to influence policy. By being “in the room” or “on the team,” greater mutual understanding may develop. For example, Wansink (2012) partnered with the U.S. Department of Agriculture on a study that demonstrated how specific payment system configurations could benefit the healthfulness of the foods purchased by high school students. This partnership between researchers and policy makers led to productive interactions with key stakeholders (i.e., school staff and students) that informed the design and relevance of the study as well as increasing the awareness and use of the research findings.

The researcher may be more deeply engaged, serving as the igniter or “provocateur” of change, and taking the role of researcher activist (Wansink 2012). This expansion to a more proactive and dynamic two-way relationship between researchers and the people affected by the research is assumed to develop knowledge that is more translatable (Ozanne and Saatcioglu 2008). Haas (1990) argues that although people working together on particular issues might come from different positions and contexts, they can be instrumental in creating new solutions. Researchers studying maternal health care in Jamaica provide an example of an active research process in which scholars worked in tandem with policy makers from inception to completion and proved effective in advancing policy (Court and Young 2006). In contrast, similar research policy initiatives on health finance reform failed to translate into policy action in nearby St. Lucia and St. Vincent, in part because policy makers were not involved in the process. The Jamaican case draws attention to the importance of productive interactions with people who can implement change.

Policy research is often more effective when it is disseminated before the policy actors’ positions are entrenched, which suggests that researchers should take advantage of the comment period that occurs before the government undertakes a study (Stewart 2013). The relational engagement approach to impact would also encourage an ideological expansion beyond examining existing social problems. Instead of waiting until the problem has become embedded, researchers can also be creative as the “identifiers of needs” while these needs are in the process of being socially constructed. For example, the large population of baby boomers who will be retiring without adequate savings means that scholars need to be anticipating and researching new forms of affordable housing to meet their needs.

Some scientific research requires productive interactions with stakeholders to produce good-quality observations. The Human Microbiome Project (also known as The American Gut Project) needs tens of thousands of participants worldwide to understand links between processed food and the well-being of our microbial ecosystems (humanfoodproject.com). Many examples exist in which lay people have better insights and access to data than scientists in understanding local ecosystems (Chambers 1997) or linkages between local contaminants and poor health outcomes (Minkler 2005).

Any researcher can engage in productive interactions before, during, or after research commences. These range from rigorous pretesting of experimental stimuli among a target audience and conducting in-depth interviews with managers and policy makers to presenting before stakeholder groups or participating on advisory boards (see the “productive interactions” row in Table 1). However, with the relational engagement approach, the depth, breadth, and quality of productive interactions would be greater because of its collaborative nature.

Enhanced Capacity

Enhanced capacity is another direct research output that can arise when external stakeholders collaborate, along with improved skills, knowledge, and understandings for all parties (see the “enhanced capacity” row in Table 1). For example, in the participatory photography research approach, collaborators learn photography skills, self-reflection, analysis, and communication, while researchers can discover local assets that can be leveraged in community interventions (Ozanne, Moscato, and Kunkel 2013). Hill et al. (2015, p. 160) studied a maximum security prison where prisoners were both informants and “coresearchers.” The prisoners learned research skills, and both prisoners and researchers forged an important understanding of the complexities of the illicit exchange market. These coresearchers presented their findings to the administration to forge new understandings on both sides of the prison bars (for an extensive review of the types of capacities that may be enhanced [e.g., forging a common language, creating educational benefits, building a shared purpose], see Wiek et al. 2014).

Improved Social Networks

A third potential direct research output of the relational engagement approach is improved social networks (see the “improved networks” row in Table 1). Network effects include building new contacts, increasing trust, strengthening bonds, increasing the flow of information, and building a sense of community (De Jong et al. 2014; Wiek et al. 2014). In relational engagement research, these network effects occur during the collaborative research process. However, networks can be built in other ways, such as when researchers take sabbaticals to work at organizations or regulatory agencies. For example, Marketing Science Institute has been building networks of academics and practitioners for more than 50 years.

One promising approach is to create networks of diverse partners organized around solving specific problems. Consider the EPODE European Network childhood obesity program, organized by alliances across university researchers, communities, practitioners, and policy makers, which aims to build local capacity for making behavior changes. This research is multidisciplinary, spanning different substantive areas of expertise, universities, and countries. Local programs are implemented with community and multistakeholder involvement but are supported by a social marketing approach based on the best scientific findings to date (http://www.epode-european-network.com).

Next, we examine the relational engagement approach in more depth. We then explore how both this approach and traditional approaches can engage more with stakeholders to increase awareness and use of research for potential societal impact.

Relational Engagement Measures of Direct Research Outputs

The relational engagement approach encourages collaboration with multiple and diverse stakeholders as part of the process of creating direct research outputs. We propose simple indices that measure the number and variety of research collaborators listed as authors and/or acknowledged in the notes of all knowledge products. An article with many nonacademic collaborators would have a high index, and an article with collaborators from a greater number of nonacademic categories (e.g., executives, marketing managers, policy makers, consumers, activists, nonprofits) would have an even higher index. It would be the responsibility of the knowledge product distributors (e.g., journals, news agencies) to provide a way to classify author and nonauthor collaborators this way, and it would be the authors’ responsibility to list this information appropriately (e.g., within manuscript notes, at minimum).

It is also possible to measure productive interactions, enhanced capacities, and improved social networks. De Jong et al. (2014) argue that there is an urgent need to study and map interactional processes. Many other researchers have also maintained that productive interactions are one of the most promising ways that research can have societal impact (Molas-Gallart and Tang 2011; Walter, Nutley, and Davies 2005). However, this is a nascent area of research. Currently, the study of productive interactions, enhanced capacities, and improved social networks consists of primarily case-based studies using interviews and focus groups and examining a range of direct and indirect interactions (see, e.g., De Jong et al. 2014; Molas-Gallart and Tang 2011; Walter, Nutley, and Davies 2005). For example, activities studied include formal and informal meetings, presentations, panels, and demonstrations. Of course, the relational engagement approach does not ensure that interactions are productive; bad interactions would not inspire the use of research for societal benefit.

Research Awareness

Following the creation of direct research outputs, potential users need to become aware of the findings. Scholars often raise awareness of traditional research at academic conferences, which are reasonable avenues to spread the news within the academic community. However, these approaches are less effective for other stakeholders given that they are infrequently attended by students, consumers, activists, managers, or policy makers. There are exceptions; for example, policy makers routinely attend the American Marketing Association Marketing & Public Policy conferences.

Within the relational engagement approach, however, awareness already exists for the collaborative stakeholder. Still, academics can increase awareness following the completion of their research (see the “Research Awareness” column in Figure 1). For example, any form of productive interaction could result in increasing awareness (see Table 1). This postresearch engagement could include testifying before legislative bodies, consulting with government agencies or organizations, conducting management training, engaging more directly with the media, and sharing their research with students. These efforts are aimed at making potential stakeholders aware of the research so it can be used. Academics can go even further by employing an impact strategy of pull versus push awareness.

Pull versus Push Awareness

Push strategies are traditional activities (e.g., media activities) that academics and their university public relations specialists can take to drive the findings to the stakeholders. For example, Journal of Research for Consumers publishes online academic articles but also offers shorter and simpler versions of these articles that are free and targeted to consumers. Pull awareness strategies are those activities that stakeholders increasingly take to access the findings. To the extent that relational engagement leads researchers and users to cocreate relevant knowledge products, these may have greater pull awareness.

Cocreation enhances pull awareness. For example, researchers working to enhance food well-being could work more directly with key informants (Block et al. 2011). They might create a website with materials that help families visualize food well-being, work with school lunch programs to create webinars, help restaurants redesign menus, and publicize research to the public through YouTube videos. Nimmon (2007) worked with immigrant women, creating photo novellas that captured their lived experiences of food practices as culturally embedded; this participant-generated photo documentation was then employed in health promotional materials for other immigrants. Ozanne, Moscato, and Kunkel (2013) argue that placing cameras in the hands of research participants is a way to gather and analyze group experiences, and subsequent photography exhibits can make the broader local community aware of the research. Creative ways to implement pull awareness may capture the heart of relevant stakeholders who are invited to attend and interact with the citizens, including business owners, government officials, and relevant activists (e.g., Findholt, Michael, and Davis 2011). All of these strategies can increase the awareness (and potential use) of research findings.

Social Media Interactions

While some pull awareness tactics may require major resource commitments, researchers are experimenting with new tools of awareness that are less resource intensive, such as posting their findings on Facebook, Twitter, and YouTube. Significant funds or organizational support are no longer necessary to make statements or spread provocative ideas. These tools require nothing more than access to the Internet and a creative message that resonates (Scaraboto and Fischer 2013). For instance, many researchers write blogs and actively use social media to generate awareness for ideas, philosophies, and movements. Robert Kozinets's blog, Brandthroposophy, has broad readership spanning academics, consumers, and managers. Research conducted online can also become activism through the way it is conducted. Netnography is an increasingly prevalent new research method incorporating all corners of the online realm (Kozinets 2010). When going beyond observation to take the form of active participation, the method can engage and begin two-way conversations with consumers on their own turf.

Although social media tools are more democratic, not everyone can make a difference; the author's name must still carry notoriety akin to a brand (e.g., McQuarrie, Miller, and Phillips 2013). By following a celebrity such as U2's Bono on Twitter or clicking “like” on a Facebook page, a relatively trivial act becomes a new form of protest, which runs the risk of creating a diluted and disengaged form of social awareness. Whether such activism in the digital realm evolves into “slacktivism”—a feel-good activism with minimal involvement or social affect—or mobilizes societal impact (e.g., the Arab spring) remains an empirical question.

Relational Engagement Measures of Research Awareness

These alternative awareness practices do not preclude awareness measurement. With a little creativity, new measures might be created to document awareness both quantitatively—such as through number of events and size of attendance—but also qualitatively, including by tracing new understandings that emerged or policy and community initiatives that arose from engagement and public discussions. For example, altmetrics are nontraditional metrics proposed as a way to assess nonacademic research awareness. The formal definition by its founder is “the creation and study of new metrics based on the web for analyzing and informing scholarship” (altmetric.org). Usage metrics are those most similar in form to traditional citation measures, and these include specific indicators, such as search results, clicks, views, downloads, and sales/holdings (including purchase by a library or article sales online). Usage metrics aim to measure nonacademic (in addition to academic) readership of journal and nonjournal media (Weiss 2007) and include tools such as PageRank, developed by commercial stakeholders such as Google (Yan and Ding 2011), which is based on counting the number and quality of links to knowledge products in addition to search engine results of their content (when adjusted for scholarly bibliometric measures such as citation counts, it is possible to measure impact on strictly nonacademic communities). These new metrics highlight the murky region between exposure, understanding, and use that are not easily untangled and indicate the need for additional research and refinement of our framework.

Some tools are designed to help disentangle the hierarchies of awareness quality. For example, the open access e-journal MRN Behavioral Marketing distributes working and accepted papers to nonacademics, and its download counts seem to be a reasonable indicator of research exposure. “Mentions,” which point to the discussion of scholarly work on the web, may include a variety of indicators including blog posts, comments, reviews, and attributions. Social media metrics, such as likes and shares, also provide some insight into the awareness and spread of information to the larger public (Roemer and Borchardt 2015).

Research Use

It is difficult to disconnect the awareness and use of research ideas. Whereas it is easy to measure exposure to an idea, it is far more difficult to determine whether the idea is adopted. Moreover, knowledge products can be used in a variety of ways (see the “Research Use” column in Figure 1). Again, we stress the complexity of tracing the process from the creation of direct research output to some eventual societal benefit, because research can be used in different ways. As we move to expand the assessment of societal impact beyond bibliometric measures, we need to consider carefully the different ways in which research can be used.

Conceptual Use of Research

The conceptual use of research is the traditional academic approach. Conceptual uses capture changes in the way that people think about their situation, which may or may not transfer to instrumental uses. However, the relational engagement approach may also enrich conceptual uses of research. Indeed, multiple collaborators with different societal roles necessitate an increased abstraction to address diverse perspectives. For example, research that reveals how low-literate consumers adeptly cope in the marketplace might attenuate stereotypes not only for retailers but also for the consumers themselves.

Instrumental Use of Research

We typically think of the instrumental use of research as use that informs consumer, managerial, or policy-related decision making, practice, or policy. Viswanathan et al. (2009) provide concrete recommendations to design effective educational programs among the poor to improve marketplace literacy, which is an example of research that could be used instrumentally. Although researchers traditionally discuss the implications of their results in their article's discussion section, the relational engagement model encourages researchers to employ strategies throughout the research process that more intentionally focus on instrumental use. In other words, rather than adding an implications section as a postresearch requirement or afterthought, the relational engagement approach means designing studies with stakeholders so that the instrumental use of the research is a meaningful core of the project, along with its conceptual use.

Political Use of Research

Research can be used to mobilize resources, such as by garnering support in political processes or justifying courses of action in an organization. For example, research on the dangers and prevalence of underage hookah smoking could be used to mobilize local policy support for age restrictions in hookah lounges (Griffiths, Harmon, and Gilly 2011). Like the instrumental use of research, its political use is practical; however, like the conceptual use of research, its political use is abstract, so that general policy can be implemented in as many situations and with as many instruments as possible.

Widespread Use of Research

Finally, on occasion, research can be widely adopted when findings lead to larger-scale shifts in thinking, moving beyond the original setting to new organizational and institutional uses (Weiss 1998). For example, the concept of a new sharing economy is one such idea that has spread like wildfire (Botsman and Rogers 2010) but is not necessarily political.

Relational Engagement Measures of Research Use

Whereas measures of research awareness include information consumption assessment, such as counting the number of times research knowledge products are clicked online, measures of research use include information application assessment, such as counting the number of times research knowledge products are used in nonacademic media (Weiss 2007). The altmetrics we propose as a measure of research awareness could also measure its use (e.g., mentions in blogs, comments, and reviews; Roemer and Borchardt 2015). Similarly, Weiss's (2007) logic model proposes the use of scientometrics to measure the number of times research is cited in newspapers, blogs, websites, and patents. Furthermore, a researcher's scientometrics could also be enhanced by measures of productive interactions, enhanced capacities, and social networks (see the “Relational Engagement Measures of Direct Research Outputs” subsection). If collaborators have scientometrics, these could be used to further enhance a researcher's own impact metrics. The logic model also proposes the use of surveys to gauge penetration of the information among nonjournal constituents. Finally, case studies could measure more complex instances of research use, such as using knowledge in the creation of new product, service, or nonprofit organization (REF 2014), which we discuss more in the next section. Such metrics, surveys, and cases could further measure use of knowledge products, productive interactions, enhanced capacities, and social networks.

Societal Benefit

The Trickle-Down and Trickle-Horizontal Approaches

Traditional approaches expect that research will benefit society by trickling down to the end users: scholars create academic knowledge products and push awareness of them to the correct stakeholders, who will hopefully translate the concepts correctly into practical use; finally, this information will benefit individuals and societies. Certainly, this is one path by which research findings may reach motivated users. However, we propose that this approach may not be effective, because it relies on unengaged connections with end users during creation, awareness, and use. We propose a trickle-horizontal approach of relational engagement of stakeholders throughout the research process as an additional pathway.

This approach has deep roots in Kurt Lewin's (1946) action research and has been resurrected more recently as participatory and community action research (Ozanne and Anderson 2010; Ozanne and Saatcioglu 2008). From this standpoint, consumer education is not a top-down process of providing guidance. Instead, education should be the practice of freedom (Freire 1970); criticism paves the way for societal transformation.

Relational Engagement Measures of Societal Benefit

It is also possible to assess research's downstream societal benefit, though this is challenging for a variety of reasons. The significant time lag between creating direct research outputs and societal benefits means that it is difficult to trace a clear path. Moreover, the research may not be used directly but may shift understandings or political processes, which can be more difficult to document. The research may be used in part or even misused or misapplied. Our theories of social change are woefully inadequate, and we really do not fully understand how a knowledge product can lead to a change in legislation, a shift in marketing strategy, or an adoption of a healthy behavior by a consumer. Finally, societal benefits may change over the short-term, intermediate, and long-term time horizons. Thus, we propose the use of case studies that permit the use of contextualized empirical evidence.

Given these complexities and the lack of quantitative data, the REF (2014) assessed the societal impact of research using case studies that were evaluated by panels of experts. Case studies were used to provide researchers maximum latitude to create an evidentiary trail that could span diversity across disciplines, projects, and contexts. This countrywide exercise led many people to conclude that “each piece of research results in a unique tapestry of impact,” and even a taxonomy using many different indicators could not capture this complexity (Penfield et al. 2014, p. 28). Although it is unlikely that a single quantifiable approach will ever emerge, different fields do provide some guidance here.

Again, drawing from the REF (2014), the actual evidence used to document societal impact provides potential directions. For example, research on health benefits has used evidence such as citations in clinical practice, use of technology in practice, and quality-adjusted life-years of a health intervention. Economic benefits have been demonstrated in new patents, licenses, products, revenue, commercialization of technology, business ventures, industrial investments, and the formation of industry partnerships, among others. The impact of research on policy has been demonstrated primarily by its informing governmental policy or engaging with governmental committees. Engagement arose in the form of giving oral or written evidence to legislative committees, research being used as evidence by a third party, research cited in a policy report, or the researcher acting as an advisor (HEFCE 2015).

Other case study approaches are similar to the REF (2014). For example, the Social Impact Assessment Methods approach stresses the role of productive interactions with stakeholders for increasing the societal benefits of research (Penfield et al. 2014; Spaapen and Van Drooge 2011). The Research Contribution Framework is also a case study approach to measuring societal benefit that emphasizes engagement/involvement and enhanced capacities (Morton 2015).

Marketing Exemplars Using Relational Engagement Approaches

Marketing researchers have already engaged in many research strategies for greater societal impact. Some researchers work toward the commercialization of their ideas and advocate that research should be transformative in the real world (Lodish 2005). Other researchers have initiated social enterprises to benefit consumers directly. Social enterprises merge the prosocial and business paradigms within specific sociocultural and political environments (Dart 2004). These enterprises take organizations to new levels as sustainable entities that benefit the constituents they serve. For example, DeBerry-Spence (2010) researches microentrepreneurs in Ghana and influences the socioeconomic well-being of those living in poverty. She collaborates with others through the MASAZI Visitor and Welcome Centre, an ongoing cultural redevelopment initiative she created that helps subsistence entrepreneurs earn a living wage. DeBerry-Spence's research addresses such complex issues as marketplace literacy, barriers to economic development, and conflicted identity projects of middle-class Africans living in subsistence marketplaces. She actively engages with her informants, whom she considers both “subjects” and beneficiaries of her research.

When embarking on such projects, an objective may be to encourage people to make positive changes in the lives of consumers, managers, and society alike. In DeBerry-Spence's (2010) work, for example, one outcome of this type of engaged research was the increased economic well-being of each microentrepreneur within the Visitor Centre. The ability to affect the individual lives of microentrepreneurs raises the overall quality of community life. This research also has some unintentional positive outcomes. Specifically, the researchers’ difficult process of establishing the Visitor Centre creates opportunities for other people looking to start businesses in Ghana by unveiling the cultural, social, and political complexities that may prevent entrepreneurs from being successful.

Critics might argue that the time researchers such as DeBerry-Spence spend on their ventures detracts from publishing more theoretically sophisticated research or from publishing a greater quantity of research. However, these authors would likely point out that the theoretical development of their research is richer from involvement in the kind of processes we identify as relational engagement approaches to societal impact. Likewise, these approaches often produce richer real-world data sets with the chance to produce more high-quality research publications, which improves “impact” even by traditional standards.

Consequently, for researchers to inspire societal impact, they may need to seek out communities and engage with consumers and groups on equal terms to cocreate and pursue transformative goals. For example, scholar and activist David Graeber (2009) explores notions of economic anthropology by organizing and attending various activist movements, including the Occupy Movement and the movement against World Economic Forum. Not satisfied with just the role of a public speaker, he was a facilitator of general assemblies and an organizer of lectures on legal issues, medical training, and nonviolent resistance. Thus, ethnographic immersion can become a two-way or multidirectional dynamic that moves starkly beyond observation to taking part in community activities. The community serves as a vital context for both the emergence and spread of new practices, creating potential for change and improvements in the marketplace and society. This happens through development of new practices as well as continuous self-reflection and monitoring.

Discussion

We present the relational engagement approach to research impact. This relational engagement occurs within a range of paradigms spanning experimental, survey, and qualitative field research aimed at a common interest toward societal effects. We are not suggesting that the relational engagement approach replace the traditional approach. Instead, we posit that this historical juncture calls for a more holistic articulation of research outputs and outcomes.

Consistent with the goals of public policy and transformative consumer research, the relational engagement approach to research does not claim neutrality but aims to enhance individual and collective well-being (Mick et al. 2012), striving toward goals such as social justice and diversity (Henderson and Williams 2013). Researchers who are interested in this approach often embed their goal of having a positive social effect throughout the research process. This might influence the use of multiple stakeholder and multidisciplinary teams that solicit diverse perspectives, theories, and tools and may be more likely to generate novel solutions for some of the most intractable social problems (Hill and Martin 2014; Tian et al. 2014). Notably, the best societal impact cases from the United Kingdom's REF (2014) assessment of societal impact were heavily multidisciplinary (HEFCE 2015). The goal of seeking practical effects might influence the research design by using data-gathering techniques that allow for more productive interactions with citizen consumers and practitioners, enhancing the capacities of the research through increased knowledge and improving social networks among researchers and other stakeholders of the research.

Implications of the Relational Engagement Approach for Nonacademic Stakeholders

The relational engagement approach has broad implications for nonacademic stakeholders. Consider the recent theorizing on the more encompassing construct of food well-being (Block et al. 2011). This research has suggested that the food = nutrients = health cultural discourse was mobilized in an era of scarcity. Yet now, in an era of relative abundance within developed countries, this discourse is dysfunctional, contributing to obesity and obsessions over fat, calories, and the body mass index. Indeed, these authors argue that this normalized discourse is partially responsible for creating a generation of paradoxical eaters. For example, people consume an entire box of fat-free cookies while trying to cut calories and lose weight. A cultural discourse better adapted to an era of fast food and overconsumption is “food as well-being” (Block et al. 2011). This involves shifting to a more holistic understanding of the role that food plays in our daily lives and overall health. To wit, this shift could help consumers question whether a box of fat-free cookies contributes to their well-being, rather than focusing on the fat-free information. Thus, the potency of the idea of “food as well-being” is not because this idea is retracing existing social reality. Instead, it acts as a map that opens up potential territory without charting which path is taken or exactly how social practices and institutions evolve.

The Block et al. (2011) article should be evaluated by traditional measures of scholarly impact, such as high citation counts (with many being outside the field of marketing and across different countries) and for winning the American Marketing Association's Thomas C. Kinnear award. Yet the food well-being framework was the foundation of an Institute of Medicine workshop on food literacy, in which it was used by members from policy, business, and academics (National Academy of Sciences 2015). Eventually, the societal benefit of this framework could be assessed by a case study or descriptive narrative of the shift in cultural discourse around well-being and how this shift led to a domino effect of social change. For example, policies focused on food well-being could lead to changes on front-of-package labeling and commercial messages, which in turn could change the way we discuss food at the family dinner table and in schools and restaurants. Market researchers could eventually pick up these changes in how we talk about food, leading to new ideas, products, services, and promotions.

Potential changes in the research process may be subtle. For example, academics in marketing often involve managers in research that is focused on the firm. We recommend that some academics in marketing more actively participate with managers when conducting research focused on societal and consumer well-being. For example, Wansink (2006) is known for his research on obesity and healthy consumption, which at first glance would appear to be a threat to the objectives of fast-food restaurant corporations such as McDonald's. However, Wansink and Hanks (2013) published research partially funded by McDonald's that investigates ways to help children reduce calorie intake by changing the content of McDonald's Happy Meals. Furthermore, Wansink is a member of McDonald's Corporation's Global Advisory Board. The traditional academic approach might have been for Wansink and Hanks to generate findings about children's meal bundles in the lab, without input from firms such as McDonald's. They might optimistically hope fast-food executives would read the publication or cede to pressure from outside groups to make recommended changes. In a prime example of a relational engagement approach, Wansink instead included McDonald's as an active research partner, which had several benefits. The researchers were invited to conduct research in the field, boosting the study's external validity. The partnership allotted a higher level of control within the field setting, increasing internal validity. This approach allowed for solutions consistent with the corporation's own needs (e.g., profitability, brand building, sustainability, resilience into the future). From a perspective of well-being, the study lessened contradictions between health-related values and the fast-food landscape. It had a societal impact because implementation was part of the study design. Foremost, because the managerial partner actively participated in the design of the research study, the company did not need to be cajoled into implementing the findings.

In addition to increasing the societal impact of research findings, academic/corporate partnerships also help reduce unproductive tensions and prejudices between constituencies (policy researchers vs. corporate managers). Productive interactions can be a direct research output, enabling researchers to influence managers (and vice versa). Such partnerships create open dialogue and diplomacy so that research is designed, implemented, and reported with input from the people with the power to enact corporate changes that may benefit their customers.

Implications of the Relational Engagement Approach for Academic Institutions

Assessing the societal impact of academic research is an increasingly important concern for universities, governing bodies, and researchers aiming to secure their roles in society (Naidoo, Shankar, and Veer 2011). With governing bodies working to assess the quality of academics’ work, a growing tension exists between traditional indices of scholarly impact and the far more demanding challenge to evidence societal impact (Polonsky 2008; Shugan 2003). We propose that a relational engagement approach can use existing traditional measures of scholarly impact. Yet a relational approach may also build the productive interactions, capacities, and social network to move the knowledge products beyond creation of direct research outputs to awareness and use and potential societal effects. As we discussed previously, traditional bibliometric measures are still important for the advancement of knowledge within academic circles. Academics must share their knowledge within their respective communities and hold each other accountable to high standards of rigor and quality. However, there are alternative ways to capture the process of influence that occurs beyond academic boundaries of research awareness, use, and societal impact. There are many ways that scholars can amplify the potential societal impact of their research, particularly in terms of increasing productive interactions. As such, we should also reward academics who aim to benefit society by focusing more on the process of change.

The REF (2014) in the United Kingdom advances such a notion. Undoubtedly, it is controversial to hold researchers responsible for the societal impact of their research, particularly if such accountability policies become dogmatic. For example, there are many instances in which basic research findings were believed to have practical applications, an assertion that, over time, was discovered to be wrong. Academic research in the field of finance proved to have an extreme impact on society in relation to the global financial crisis that began in 2007. So-called “breakthrough” research by David X. Li provided a model to value collateralized debt obligations, a market that had been stalled because no way existed to value these securities or evaluate their risks. Financial practitioners praised this research and used it to expand the market to trillions of dollars in the short run. In hindsight, it is clear that short-term interests in profits led to immense profits for a few and the near collapse of the financial market for the many (for more details, see Learmonth, Lockett, and Dowd 2012).

Theoretical Implications of the Relational Engagement Approach

From this perspective, theoretical explanation is, in part, the outcome of democratic dialogue taking place in the daily world of the informants (Habermas 1987). This assumes that theory that is useful for solving problems cannot be removed from the context of this day-to-day world. It is the researcher's job to mine cultural context for theoretical insight through discourse. This assumption is why building productive interactions and cocreating knowledge are so important for the relational engagement approach. Insights may indeed flow down from the researcher to end users. However, insights equivalently flow from the end users upward, as this research stance acknowledges and leverages important local expertise throughout the process. More often, however, insights emerge from dialogue and social interaction between researchers and stakeholders. Furthermore, various hybrid forms of theory can exist, such as a fusion of theory and practical knowledge (Kalb 2006) and theory-guided action (see also praxis in Murray and Ozanne 1991).

Researchers who are active in exploring new relational forms of research should be ever vigilant in identifying the overt and subtle ways that power relations are manifested within these community negotiations. All acts that bring about societal change are always provisional and evolving and, therefore, require constant negotiation and ongoing engagement. The key here is that the knowledge product is the result of dialogue among the researchers and representatives of an external constituency; it is relational, coconstitutive, and does not have a predetermined direction. For example, a marketing manager might focus on the ability to reach and transform customers and company employees through mutually beneficial, prosocial organizational processes (Smith, Drumwright, and Gentile 2010). For a consumer activist organization, meaningful research might help mobilize resources to alleviate injustice. Alternatively, policy makers could be more likely to use research that informs policies currently under consideration.

Limitations and Further Research

A relational engagement approach to evaluating and creating societal impact would understand that the researcher's position is not one that emanates only from the context of an academic office. Instead, a relational approach to societal impact views researchers as being an integral part of the wider community, and generating awareness and use for the research is part of the path to societal impact over time. Rather than being tangential to an academic's duties, room should exist in the academy for researchers to treat these activities as central to cocreating and learning how knowledge can be used for societal impact. Of course, these engagement activities are both resource and labor intensive, and if universities are seeking research that eventually has societal impact, additional administrative support may be needed. The demands for academics to create knowledge are great; demands to engage in the awareness, use, and societal benefit of research would be significant. The academy should be an island accessible to academics—however, some researchers should be rewarded more explicitly for building bridges to what are currently more distant shores.

Yet we know very little about the complex process of using research findings to affect societal change. Research findings do reach the broader public through other means, such as press releases, consulting, and teaching, but the process tends to be more top-down from academic researchers to end users. We do not know if and when readers of Men's Health or Cosmopolitan act on an article that reports research findings. Given that the REF (2014) found more than 3,700 unique paths to change, we are far from having a strong understanding of this very complex process (HEFCE 2015).

We end with a call for academics in the fields of public policy, marketing, and consumer research to seize the opportunity to expand both the meaning and nature of societal impact, particularly with regard to academic policy and in collaboration with university policy makers. The concepts “impactful,” “useful,” and “relevant” are inherently unstable, changing over time, and highly sensitive to political interests. Thus, it is not the purpose of this article to create a new code. Indeed, a proposal that allocates a one-size-fits-all metric is suspect; the complexity of social change will resist simple metrics. For example, imagine trying to untangle the relative impact of research that influences the policy process, the enforcement of policy, or the implementation of policy. Likewise, imagine weighing the relative importance of policy research with the potential for minor influence on millions of people versus policy research with the potential for critical, life-saving influence on a few dozen people. However, working toward academic policies that incentivize relational engagement approaches to scholarly impact will benefit our fields and the stakeholders of our research, despite the inherent complexity.

We propose an expansion of traditional conceptualizations of impact to include more relational engagement approaches, which will require higher levels of researcher involvement across the whole process. Rather than painting an image of the researcher as the objective expert, we advocate new roles for the researcher as “copreneur,” advocate, cocreator, and provocateur. A greater, deeper, and more sustainable form of impact may be facilitated through the imaginative use of various approaches and outlets, combined with a variety of promotional methods, personal interaction, multimedia activities, in- and out-of-field conferences, and partnerships. In doing so, marketing scholarship will more effectively achieve the shared goal of contributing to well-being for all citizens. Moreover, when researchers engage with a range of external constituencies, research outputs can avoid serving the interests of the few and most powerful and better serve the well-being of the many and most lacking.

References

AACSB International (2013), “Eligibility Procedures and Accreditation Standards for Business Accreditation,” http://www.aacsb.edu/∼/media/AACSB/Docs/Accreditation/Standards/2013-bus-standards-update.ashx.

Aguinis

Herman

Shapiro

Debra L.

Antonacopoulou

Elena P.

and Cummings

Thomas G.

(2014), “Scholarly Impact: A Pluralist Conceptualization,” Academy of Management Learning & Education, 13 (4), 623–39.

Alexander

Jennifer K.

Scherer

Robert F.

and Lecoutre

Marc

(2007), “A Global Comparison of Business Journal Ranking Systems,” Journal of Education for Business, 82 (6), 321–27.

Block

Lauren G.

Grier

Sonya A.

Childers

Terry L.

Davis

Brennan

, Ebert

Jane E.J.

, Kumanyika

Shiriki

, (2011), “From Nutrients to Nurturance: A Conceptual Introduction to Food Well-Being,” Journal of Public Policy & Marketing, 30 (Spring), 5–13.

Botsman

Rachel

and Rogers

Roo

(2010), What's Mine Is Yours: The Rise of Collaborative Consumption. New York: HarperCollins.

Bourdieu

Pierre

(1984), Distinction: A Social Critique of the Judgment of Taste. Cambridge, UK: Routledge.

Chambers

Robert

(1997), Whose Reality Counts? Putting the First Last. London: Intermediate Technology.

Cote

Joseph A.

Leong

Siew Meng

, and Cote

Jane

(1991), “Assessing the Influence of Journal of Consumer Research: A Citation Analysis,” Journal of Consumer Research, 1 (3), 402–10.

Court

Julius

and Young

John

(2006), “Bridging Research and Policy: Insights from 50 Case Studies,” Evidence & Policy, 4 (November), 439–62.

10.

Curtis

Bruce

(2007), “Academic Life: Commodification, Continuity, Collegiality, Confusion and the Performance Based Research Fund,” New Zealand Journal of Employment Relations, 32 (2), 1–16.

11.

Dart

Raymond

(2004), “The Legitimacy of Social Enterprise,” Nonprofit Management & Leadership, 14 (4), 411–24.

12.

Davis

Brennan

and Pechmann

Cornelia

(2013), “Introduction to the Special Issue on Transformative Consumer Research: Developing Theory to Mobilize Efforts that Improve Consumer and Societal Well-Being,” Journal of Business Research, 66 (8): 1168–70.

13.

DeBerry-Spence

Benet

(2010), “Making Theory and Practice in Subsistence Markets: An Analytic Autoethnography of MASAZI in Accra, Ghana,” Journal of Business Research, 63 (6), 608–16.

14.

De Jong

Stefan

Barker

Katherine

Cox

Deborah

Sveinsdottir

Thordis

and Van den Besselaar

Peter

(2014), “Understanding Societal Impact Through Productive Interactions: ICT Research as a Case,” Research Evaluation, 23 (2), 89–102.

15.

Epstein

Marc J.

and Yuthas

Kristi

(2012), “Scaling Effective Education for the Poor in Developing Countries: A Report from the Field,” Journal of Public Policy & Marketing, 31 (Spring), 102–14.

16.

Erkut

Erhan

(2002), “Measuring Canadian Business School Research Output and Impact,” Canadian Journal of Administrative Sciences, 19 (2), 97–123.

17.

Findholt

Nancy E.

Michael

Yvonne L.

and Davis

Melinda M.

(2011), “Photovoice Engages Rural Youth in Childhood Obesity Prevention,” Public Health Nursing, 28 (2), 186–92.

18.

Freire

Paulo

(1970), The Pedagogy of the Oppressed. New York: Herder and Herder.

19.

Graeber

David

(2009), Direct Action: An Ethnography. Edinburgh: AK Press.

20.

Griffiths

Merlyn A.

Harmon

Tracy R.

and Gilly

Mary C.

(2011), “Hubble Bubble Trouble: The Need for Education About and Regulation of Hookah Smoking,” Journal of Public Policy & Marketing, 30 (Spring), 119–32.

21.

Haas

E.B.

(1990), When Knowledge Is Power: Three Models of Change in International Organizations. Berkeley: University of California Press.

22.

Habermas

Jürgen

(1987), Theory of Communicative Action: Life-world and System: A Critique of Functionalist Reason, Vol. II. Boston: Beacon Press.

23.

HEFCE (2015), “The Nature, Scale and Beneficiaries of Research Impact: An Initial Analysis of Research Excellence Framework (REF) 2014 Impact Case Studies,” research report, http://www.hefce.ac.uk/media/HEFCE,2014/Content/Pubs/Independentresearch/2015/Analysis,of,REF,impactAnalysis_of_REF_impact.pdf.

24.

Henderson

Geraldine Rosa

and Williams

Jerome D.

(2013), “From Exclusion to Inclusion: An Introduction to the Special Issue on Marketplace Diversity and Inclusion,” Journal of Public Policy & Marketing, 32 (Special Issue), 1–5.

25.

Hill

Ronald Paul

and Martin

Kelly D.

(2014), “Broadening the Paradigm of Marketing as Exchange: A Public Policy and Marketing Perspective,” Journal of Public Policy & Marketing, 33 (Spring), 17–33.

26.

Hill

Ronald Paul

Rapp

Justine M.

Capella

Michael L.

and the Gramercy Gentlemen (2015), “Consumption Restrictions in a Total Control Institutions: Participation Action Research in a Maximum Security Prison,” Journal of Public Policy & Marketing, 34(Fall), 156–72.

27.

Hirsch

J.E.

(2005), “An Index to Quantify an Individual's Scientific Research Output,” PNAS, 102 (46), 16569–72.

28.

Ioannidis

John P.A.

(2004), “Materializing Research Promises: Opportunities, Priorities and Conflicts in Translational Medicine,” Journal of Translational Medicine, 2 (5), 1–8.

29.

Kalb

Don

(2006), “Uses of Local Knowledge,” in The Oxford Handbook of Contextual Political Analysis, Goodin

Robert E.

and Tilly

Charles

, eds. Oxford, UK: Oxford University Press, 579–96.

30.

Kozinets

Robert V.

(2010), Netnography: Doing Ethnographic Research Online. Thousand Oaks, CA: Sage Publications.

31.

Learmonth

Mark

Lockett

Andy

and Dowd

Kevin

(2012), “Promoting Scholarship That Matters: The Uselessness of Useful Research and the Usefulness of Useless Research,” British Journal of Management, 23 (1), 35–44.

32.

Lehmann

Donald R.

McAlister

Leigh

and Staelin

Richard

(2011), “Sophistication in Research in Marketing,” Journal of Marketing, 75 (July), 155–65.

33.

Lewin

Kurt

(1946), “Action Research and Minority Problems,” Journal of Social Issues, 2(4), 34–46.

34.

Lodish

Leonard M.

(2005), “Commentary on the 2004 ISMS Practice Prize Winner: A Business School Should Combine Academic Research and Its Application to the Real World,” Marketing Science, 24 (3), 332–33.

35.

Malhotra

Naresh K.

(1996), “The Impact of the Academy of Marketing Science on Marketing Scholarship: An Analysis of the Research Published in JAMS,” Journal of the Academy of Marketing Science, 24 (4), 291–98.

36.

McQuarrie

Edward F.

Miller

Jessica

and Phillips

Barbara J.

(2013), “The Megaphone Effect: Taste and Audience in Fashion Blogging,” Journal of Consumer Research, 40 (1), 136–58.

37.

Mick

David G.

Pettigrew

Simone

Pechmann

Cornelia

and Ozanne

Julie L.

(2012), Transformative Consumer Research for Personal and Collective Well-Being. New York: Routledge.

38.

Minkler

Meredith

(2005), “Community-Based Research Partnerships: Challenges and Opportunities,” Journal of Urban Health, 82 (2), ii3–ii12.

39.

Moed

Hank F.

and Garfield

Eugene

(2004), “In Basic Science the Percentage of ‘Authoritative’ References Decreases as Bibliographies Become Shorter,” Scientometrics, 60 (3), 295–303.

40.

Molas-Gallart

Jordi

and Tang

Puay

(2011), “Tracing ‘Productive Interactions’ to Identify Social Impacts: An Example from the Social Sciences,” Research Evaluation, 20 (3), 219–26.

41.

Morton

Sarah

(2015), “Progressing Research Impact Assessment: A ‘Contributions’ Approach,” Research Evaluation, 24 (4), 405–19.

42.

Murray

Jeff B.

and Ozanne

Julie L.

(1991), “The Critical Imagination: Emancipatory Interests in Consumer Research,” Journal of Consumer Research, 18 (2), 129–44.

43.

Murray

Jeff B.

and Ozanne

Julie L.

(2009), “The Critical Participant,” Journal of Marketing Management, 25 (7/8), 835–41.

44.

Naidoo

Rajani

Shankar

Avi

and Veer

Ekant

(2011), “The Consumerist Turn in Higher Education: Policy Aspirations and Outcomes,” Journal of Marketing Management, 27 (11/12), 1142–62.

45.

National Academy of Sciences (2015), “Food Literacy: How Do Communications and Marketing Impact Consumer Knowledge, Skills, and Behavior?—Workshop in Brief,” Institute of Medicine, (November).

46.

Nimmon

Laura E.

(2007), “Within the Eyes of the People: Using a Photonovel as a Consciousness-Raising Health Literacy Tool with ESL-Speaking Immigrant Women,” Canadian Journal of Public Health, 98 (4), 337–40.

47.

Ozanne

Julie L.

(2011), “Introduction to the Special Issue on Transformative Consumer Research: Creating Dialogical Spaces for Policy and Action Research,” Journal of Public Policy & Marketing, 30 (Spring), 1–4.

48.

Ozanne

Julie L.

and Anderson

Laurel

(2010), “Community Action Research,” Journal of Public Policy & Marketing, 29 (Spring), 123–37.

49.

Ozanne

Julie L.

Moscato

Emily M.

and Kunkel

Danylle R.

(2013), “Transformative Photography: Evaluation and Best Practices for Eliciting Social and Policy Changes,” Journal of Public Policy & Marketing, 32 (Spring), 45–65.

50.

Ozanne

Julie L.

and Saatcioglu

Bige

(2008), “Participatory Action Research,” Journal of Consumer Research, 35 (3), 423–39.

51.

Özçağlar-Toulouse

Nil

, and Burroughs

James E.

(2014), “Introduction to the Special Issue on Transformative Consumer Research: Taking the TCR Movement Global,” Journal of Marketing Management, 30 (17/18), 1723–27.

52.

Penfield

Teresa

Baker

Matthew J.

Scoble

Rosa

and Wykes

Michael C.

(2014), “Assessment, Evaluations, and Definitions of Research Impact: A Review,” Research Evaluation, 23 (1), 21–32.

53.

Polonsky

Michael

(2008), “What Is a Research Track Record?” Australasian Marketing Journal, 16 (July), 67–72.

54.

Research Excellence Framework (2014), “Results and Submissions,” http://results.ref.ac.uk/.

55.

Roemer

Robin Chin

and Borchardt

Rachel

(2015), Meaningful Metrics: A 21st Century Librarian's Guide to Bibliometrics, Altmetrics, and Research Impact. Chicago: Association of College & University Libraries.

56.

Scaraboto

Diane

and Fischer

Eileen

(2013), “Frustrated Fatshionistas: An Institutional Theory Perspective on Consumer Quests for Greater Choice in Mainstream Markets,” Journal of Consumer Research, 39 (6), 1234–57.

57.

Shankar

Avi

(2009), “Reframing Critical Marketing,” Journal of Marketing Management, 25 (7/8), 681–96.

58.

Shugan

Steven M.

(2003), “Journal Rankings: Save the Outlets for Your Research,” Marketing Science, 22 (4), 437–41.

59.

Smith

Kylie M.

Crookes

Ellie

and Crookes

Patrick A.

(2013), “Measuring Research ‘Impact’ for Academic Promotion: Issues from the Literature,” Journal of Higher Education Policy and Management, 35 (4), 410–20.

60.

Smith

N. Craig

, Drumwright

Minette E.

, and Gentile

Mary C.

(2010), “The New Marketing Myopia,” Journal of Public Policy & Marketing, 29 (Spring), 4–11.

61.

Spaapen

Jack

and van Drooge

Leonie

(2011), “Introducing ‘Productive Interactions’ in Social Impact Assessments,” Research Evaluation, 20 (3), 2011–18.

62.

Sprott

David E.

and Miyazaki

Anthony D.

(2002), “Two Decades of Contributions to Marketing and Public Policy: An Analysis of Research Published in Journal of Public Policy & Marketing,” Journal of Public Policy & Marketing, 21 (Spring), 105–25.

63.

Stewart

David W.

(2013), “Reinventing Marketing and Public Policy for the Twenty-First Century: An Editorial Statement,” Journal of Public Policy & Marketing, 32 (Spring), 1–5.

64.

Tian

Kelly

Sautter

Pookie

Fischer

Derek

Fischbach

Sarah

Luna-Nevarez

Cuauhtemoc

, Boberg

Kevin

, (2014), “Transforming Health Care: Empowering Therapeutic Communities Through Technology-Enhanced Narratives,” Journal of Consumer Research, 41 (2), 237–60.

65.

Van Raan

Anthony F.J.

(2005), “Fatal Attraction: Conceptual and Methodological Problems in the Ranking of Universities by Bibliometric Methods,” Scientometrics, 62 (1), 133–43.

66.

Viswanathan

Madhubalan

Sridharan

Srinivas

Gau

Roland

and Ritchie

Robin

(2009), “Designing Marketplace Literacy Education in Resource-Constrained Contexts: Implications for Public Policy and Marketing,” Journal of Public Policy & Marketing, 28 (Spring), 85–94.

67.

Walter

Isabel

Nutley

Sandra

and Davies

Huw

(2005), “What Works to Promoted Evidence-Based Practice? A Cross-Sector Review,” Evidence & Policy, 1 (September), 335–64.

68.

Wansink

Brian

(2006), Mindless Eating: Why We Eat More Than We Think. New York: Bantam-Dell.

69.

Wansink

Brian

(2012), “Activism Research: Designing Transformative Lab and Field Studies,” in Transformative Consumer Research for Personal and Collective Well-Being, Mick

D.G.

, Pettigrew

, Pechmann

, and Ozanne

J.L.

, eds. New York: Routledge, 67–88.

70.

Wansink

Brian

and Hanks

Andrew S.

(2013), “Calorie Reductions and Within-Meal Calorie Compensation in Children's Meal Combos,” Obesity, 22 (3), 630–32.

71.

Weiss

Anthony P.

(2007), “Measuring the Impact of Medical Research: Moving from Outputs to Outcomes,” American Journal of Psychiatry, 164 (2), 206–14.

72.

Weiss

Carol H.

(1998), “Have We Learned Anything New About the Use of Evaluation?” American Journal of Evaluation, 19 (1), 21–33.

73.

Wexler

Ellen

(2015), “Can Data Measure Faculty Productivity? Rutgers Professors Say No,” Chronicle of Higher Education (December 11), http://chronicle.com/article/Can-Data-Measure-Faculty/234595.

74.

Wiek

Arnim

Talwar

Sonia

O'Shea

Meg

and Robinson

John

(2014), “Toward a Methodological Scheme for Capturing Societal Effects of Participatory Sustainability Research,” Research Evaluation, 23 (2), 117–32.

75.

Yan

Erjia

and Ding

Ying

(2011), “Discovering Author Impact: A PageRank Perspective,” Information Processing & Management, 47 (1), 125–34.