Social Impact Indicators in the Context of the Roma Community: Contributions to the Debate on Methodological Implications

Abstract

Researching with methodologies focused on social impact in line with the SDGs is one of the priority orientations of the Horizon Europe program, as shown in the official European Commission document on impacts for this program. In this sense, researchers must forecast how their project will improve citizens' lives. Until now, many investigations showed the evaluation of the social impact through knowledge transfer activities that, although undoubtedly important, are not enough since the social impact is defined as the improvements derived from using the knowledge transferred to society. The search for the social impact of new research requires the introduction of impact indicators from the design, throughout the project development, and when the project ends. The introduction of indicators, in particular if they are decided in dialogue with the participants, allows not only to foresee a greater social impact but also to improve and adjust the methodology to be used. We explore this aspect in the context of research with social impact that starts from how the COVID-19 pandemic is increasing the inequalities suffered by the Roma population, causing the aggravation and creation of new problems and needs. Thus, we explain in detail how the selection of indicators that monitor the social impact, in dialogue with the Roma population, allows the design of research projects that are more appropriate to the current context.

Keywords

methods in qualitative inquiry qualitative evaluation social justice case study community based research

Introduction

The European Union has funded research in the Social Sciences and Humanities (SSH) in different ways, primarily through the EU Research and Innovation Framework Program (hereinafter, FP), initiated in 1983 and currently in the ninth edition. However, its effectiveness has been questioned and at the same time, many researchers have expressed concerns about the risk of decrease in SSH European funding. For example, the ‘European Alliance for Social Sciences and Humanities’ sent an open letter to the European Commission showing its concern in this regard (European Alliance for Social Sciences and Humanities, 2011). The argument is that, being SSH research funded with public funds, the aim is to guarantee that this investment is adequate, increasing the scientific competitiveness and excellence, the creation of wealth, productivity and social welfare (Reale et al., 2018), which requires measuring its quality and suitability.

Regarding suitability, different authors have demonstrated how SSH research can contribute to solving problems (Flecha et al., 2015). Some examples are the challenges of the European Commission on H2020 and Horizon Europe or the Sustainable Development Goals, adopted in 2015 by the United Nations. As König points out ‘Obviously, science – and new scientific knowledge – is key to understanding those problems, to alleviating them and also to preparing for potential fallouts. At the same time, this added a new layer to the ambitions of research funding policy. It has also renewed the quest to increase cooperation between different fields of science and scholarship, and has reinforced the growing demand for “impact”’ (2019, p. 4).

Until the mid-2000s, research was measured in terms of excellence, and the focus has broadened in recent years. There is talk of impact in several senses: scientific advances that are likely to stimulate knowledge production, political impact (possibility of providing solutions to perennial policy problems) and social impact (creating interventions to improve societal challenges) (Moore et al., 2017; Reale et al., 2018). The European Commission in the Interim Evaluation of Horizon 2020 also identifies three types of impact: scientific (create and disseminate high-quality new knowledge, skills, technologies and solutions to global challenges), economic (foster all forms of innovation, including breakthrough innovation and strengthening market deployment of innovative solutions) and societal (strengthen the impact of research and innovation in developing, supporting and implementing European Union policies and support the uptake of innovative solutions in industry and society to address global challenges) (Bruno & Kadunc, 2019).

Flecha advances knowledge in the field leading research in social impact and distinguishing between scientific impact, dissemination, political impact and social impact. The latter is the result of successfully achieving the three previous stages. Thus, ‘the social impact of research occurs when the published and disseminated results, which have been transferred to a policy or an initiative led by NGOs, produce improvements in relation to the stated objectives of society’ (Reale, et al., 2018, p. 300).

To address the orientation of research, we identify a similar distinction. For example, Kastrinos (2010) in the analysis of two FP calls identifies that there were more projects focused on dissemination than those projects fulfilling a mission or objective as indicated in the call priorities. The H2020 European Framework Programme (2014–2020) measured the impact of projects considering the publications, patents and intellectual property, mobility of researchers, etc. but the ninth edition of the FP, Horizon Europe, includes an approach that moves from the project results in publications to the impact trajectory in social impact. Therefore, the approach is to include key indicators classified according to key impact pathways, which allows tracking the impact through short, medium and long term indicators to obtain more accurate advancements over time (European Commission, 2018).

There is still discussion to define social impact. Although there is an agreement that this is the positive influence that research has on society, there are still doubts about how we understand the term ‘social’ and how we measure the related impact, that is, how we measure the improvements in society. Some authors refer that the impact can also be negative, which they call Grimpact (Derrick et al., 2018). However, social impact refers only to those positive results obtained from using the findings and contributions of a research project (Pulido et al., 2018).

This shift in research evaluation has also implied changes in the FP. Specifically, concerning H2020, it was raised that all projects include a social impact forecast ex-ante, during the project and after its implementation. This approach was needed for all projects that intended to be evaluated and funded by the FP, among other aspects. This approach has led to essential changes in how these projects are designed and developed. Focussing on the strategies that have already been successful in enabling projects to achieve social impact, the work of Aiello et al. (2020) highlights the following strategies: articulating from the beginning of a project the objective of realizing the social impact and a strategy to do so; meaningful stakeholder involvement throughout the project lifespan; use of previous contact networks in order to build up collaborations; coordination between the research activity and stakeholders' activity during the projects' duration; dissemination activities showing useful evidence and promoting public debate; the achievement of political impact as a way to realize the subsequent social impact. All these strategies involve in different ways the explicit pursuit of achieving impact and the relevance to include interactions with the different participants. In this framework, we contextualize our research about the Roma community.

This work discusses with five researchers how including social impact indicators have changed their research methodology. In SSH and other disciplines, it is common to use scientific indicators to measure scientific impact of research through the use of altmetrics, among others. With regards the social impact (concrete improvements for society resulting from the research results), recent advancements develop indicators to measure the social impact of research (European Commission, 2018). Thus, the definition of a social impact indicator implies the measurement of the social impact of the research results, which can be achieved in a short, medium or long term basis. Thus, we explain how incorporating social impact indicators, mostly when these are decided in dialogue with the Roma population, implies changing the methodology in three specific aspects. First, a better understanding of the study problem; second, a modification of the selection of social impact indicators and third, the contextualization of the indicators in the methodological design.

Indicators and Social Impact in SSH Research

The European Commission’s expert report ‘Monitoring the Impact of EU Framework Programmes’ includes three impact categories for research: scientific, societal and economic, in which Flecha develops the societal impact. According to the author, the social impact is different from the transfer and dissemination of results, specifically, it is a later stage and involves improvements derived from the use of the knowledge transferred to society (European Commission, 2018). Furthermore, in the report Flecha defines the social and political impact indicators for all sciences that informed the European Commission proposal for the monitoring and evaluation system of Horizon Europe for research and innovation 2021–2027.

While FP required to researchers to include in their methodology indicators that allow measuring dissemination and transfer, it is now necessary to include social impact indicators. This is a challenge for researchers. For instance, Godin and Doré (2005) pointed out that ‘systematic measurements and indicators on impact on the social, cultural, political, and organizational dimensions are almost totally absent from the literature’ (p.5). How to measure social impact and which indicators are more suitable is a recent discussion in SSH. However, it is already present in areas such as technology, agriculture or environment, where we can highlight projects that tried to create indicators to measure social impact such as the Assessments of the impacts of the Advanced Technology Program (ATP) (Ruegg & Feller, 2003), the Socio-Economic Analysis of the Impacts of Public Agricultural Research (ASIRPA) (Joly et al., 2015), or the Social Impact Assessment Method for research and funding instruments through the study of Productive Interactions (SIAMPI) (Molas-Gallart & Tang, 2011; Soler & Gómez, 2020). The Research Excellence Framework-REF (Higher Education Funding Council for England, 2011) of the United Kingdom was the first national system that includes the measurement of societal impact among the selection criteria for funding research. In the REF system, the assessment of social impact of research is developed through outputs or case studies. Greenhalgh et al. (2016) have identified up to 20 models of social impact on health research. Bornmann (2013) identified more than 60 impact indicators. This fact is justified because the definition of social impact, or ‘the broader impacts’, is still engaging academic debates and includes economic, cultural, political, environmental, or health impact.

The advancements in the definition of social impact does not imply less need to demonstrate its achievement. Wilsdon et al. (2015) mention that ‘research has a societal impact when auditable or recorded influence is achieved upon non-academic organization (s) or actor(s) in a sector outside the university sector itself – for instance, by being used by one or more business corporations, government bodies, civil society organizations, media or specialist/professional media organizations or in public debate’. As in the case of academic impact, societal impact needs to be demonstrated rather than assumed. In fact, “Evidence of external impact can take the form of references to, citations of or discussion of a person, their work or research results” (Wilsdon et al., 2015, p.6). The timing required to evaluate the social impact is another problem for researchers, as well as selecting appropriate indicators (De Jong et al., 2014).

Regarding measurements, for some authors, case studies are the most accepted methodologies when evaluating the social impact since they allow a better adaptation to the complexity of the different research and contexts. However, they are not exempt of criticism. They have been described as not very objective since they only report successful cases, which are very expensive and not comparable if they are not based on the same indicators (Bell et al., 2011; Godin & Doré, 2005).

A relatively recent approach uses computer systems to track the impact of research, for example, ‘Dimensions’, ‘ResearchFish’ or ‘VV-Impact Tracker’. These systems include alternative metrics tools (altmetrics) to measure societal impact (Priem et al., 2012; Robinson-Garcia et al., 2018). Altmetrics are usually based on activity on social media platforms, which relates to scholars or scholarly content. Typical examples of altmetrics include tweets, mentions in blog posts, readership counts on Mendeley, posts, likes and shares on social networks such as Facebook and Google Plus, and recommendations and ratings on F1000. However, altmetrics also comprise mentions in mainstream media or policy documents, as well as usage metrics such as full-text views and downloads, although these have been available long before the concept of altmetrics was introduced (Work et al., 2015, p.11). These indicators are characterized by generating data that might be automatically collected by computer programs (Wilsdon et al., 2015). However, altmetrics, or social networks in general, also has detractors when measuring social impact. For example, Haustein (2016) points out that there is no evidence that social networks indicate social impact. People talking in social networks about a research contribution does not imply that citizens talks about it, as it may be the scientific community in a more informal context. Bornmann and Haunschild (2017) refer to the fact that these new indicators are not related to research quality. When comparing the use of altmetrics with the use of peer-reviewed case-studies to assess social impact, recent research (Bornmann et al., 2019) concludes, that both measures ‘provide no support for the use of altmetrics to replace or even inform peer review in impact evaluation’ (Bornmann et al., 2019, p. 335). In the context of the social media, it is also important to highlight the social impact in social media methodology (Pulido et al., 2018), which is being replicated in different fields of knowledge.

Concerning the new types of data due to social networks and Internet of Things, new ways of analysing them to measure social impact emerge (Gupta et al., 2018). However, these analyses are based on existing data and some research suggests that to measure the social impact, it may be of interest to create specific indicators in consensus with stakeholders about the ones to be used. For example, in the case of biomedical sciences, the survey-based ‘best-worst scaling’ (BWS) method can be highlighted. In this method, participants are asked to evaluate the relevance for them of different types of impacts of the research. This research shows that ‘the general public and researchers value research impacts in different ways. However, it is also the case that when the two groups are in agreement, this is generally about matters that are seemingly more important and associated with wider social benefit (eg, life expectancy, cost of healthcare, job creation)’ (Pollitt et al., 2016, p.11). In this vein, it is interesting to use this ex-ante agreement to define the research impact that society wants. Although this factor is still a scarcely studied area, we will go deeper in this contribution.

Addressing the Inequalities Faced by the Roma Community with Social Impact Indicators

In many cases, research and interventions with the Roma community have focused on describing the population, using simplistic indicators that have not delved into the problem, nor found answers or solutions. This situation has contributed to perpetuate the exclusion of this community in various areas and countries, which is a significant concern at the European level. Thus, the report ‘A persisting concern: anti-gypsyism as a barrier to Roma inclusion’ (2018), elaborated by the European Union Fundamental Rights Agency (FRA) shows the situation of inequality and exclusion that the Roma people live in Europe with five axes: discrimination, harassment and hate crimes; education; poverty; occupation and health (European Union Agency for Fundamental Rights, 2018).

In all these areas, the exclusion of the Roma community is evident. For example, in education, the report states that in Europe, 48% of upper secondary school-age children do not attend school, and this percentage rises to 95% among young people of post-secondary or tertiary school age. There is also a higher dropout rate and school segregation in some countries. Despite having improved in recent years, this situation influences the Roma community to access the labour market on an equal opportunity basis. In particular, the rate of paid employment is much lower for Roma people aged 20–64 with 43% compared to 70% of the European average in 2015. Likewise, there are significant differences for young people with 63% of young people between 16 and 24 years who do not work or study compared to the European average of 12%.

The inequalities faced by the Roma are evident in the light of these data and have increased with COVID-19 (Arza Porras et al., 2020). All this points to the need not only to research on the Roma community, considering also the cultural diversity in research (Erden-Basaran, 2021; Islam, 2020) but also to achieve the social impact of research (Parthenis & Fragoulis, 2020; Pulido et al., 2020). Therefore, it is essential to ensure that all research undertaken is a conscious search for social impact, for the community’s improvement as, in social sciences, social improvement is amplified through an active approach and strategy of searching impact (Aiello et al., 2020; Khalfaoui et al., 2020; Matulič-Domadzič et al., 2020). To this end, it is necessary to include social impact indicators in the research design since, in this way, all the research will be oriented towards achieving community improvement. Thus, at the same time, research with social impact will mean improving and adjusting the methodology to the needs of the Roma population and overcoming the inequalities suffered by this group.

Methodology

This study explores how the inclusion of social impact indicators in the research design promotes the achievement of social impact. With this aim, we conducted five semi-structured interviews with researchers who worked with the Roma community and had achieved social impact. The guiding research question focused on the potential benefits of designing the indicators of social impact jointly with the Roma, with concrete questions about the differences between preparing research proposals considering or not the social impact, or to what extend can the target population contribute to the design of a research project. Furthermore, the questions raised during the semi-structured interviews included aspects such as how the Roma participants explained the improvements of their situation resulting from research.

The study used a purposeful sample as a strategy to select the researchers in the European context for having demonstrated to achieve social impact with the Roma community in their leadership and participation in projects, national or international, with results improving participants’ lives. This was the unifying criteria for all the selected researchers. Specifically, we reviewed research projects demonstrating social impact focused on Roma, to finally select researchers that have participated in total in 20 research projects funded by the FP of the European Commission and/or in 15 projects of national scope in Europe.

The participants age ranged from 27 to 50 years old (4 woman and 1 man) and they were contacted by email to share the aim of the study and inviting them to take part in an online semi-structured interview. All participants accepted and the semi-structured interviews were done using zoom (1) or google meets (4) during 2020 and the beginning of 2021, lasting between 15 and 40 minutes each. All the participants had the opportunity to provide additional information via email if they considered it appropriate. One of the participants did so by providing documentation.

The semi-structured interviews were recorded and analysed inductively, with the following key dimensions emerging from the data: (1) definition of the problem (2) co-creation of indicators and, (3) the need to contextualize these indicators. All the data was anonymized.

The study received the ethical approval of CREA Ethical Committee with reference number 20210111.

Results

From the analysis of the semi-structured interviews, three specific aspects emerge about how adding social impact indicators in the design of the methodology modifies the research methodology: (1) definition of the problem, (2) co-creation of indicators and (3) the need to contextualize these indicators. It is important to consider the role of egalitarian dialogue between researchers and participants in the identified aspects. In this vein, the inclusion of the voices of Roma people based on an egalitarian dialogue provides them a role in creating knowledge from their experiences. As a result, the researchers reshape their previous assumptions to expand their academic knowledge. The contributions that participants and researchers bring to the discussion are not valued depending on the position of power of the speaker, as validity resides in the arguments and the dialogic process beyond the academic status (Gómez et al., 2010).

Deeper Understanding of the Problem

In all the semi-structured interviews, the researchers explain this aspect because through dialogue with the people to whom the project is addressed, researchers achieve a better understanding of the problem. One of the researchers states that

‘Having the Roma in the project before even designing it allow to define and understand the problem better’ (R2).

Having a better knowledge of reality allows researchers to orient the investigation on those aspects that can improve the reality and contribute to solving the problems that the Roma community has.

‘In the research where all the research has been built in dialogue, besides having more possibilities to achieve that impact, it is not sure, but it increases the possibilities that there has been this impact, what you have clear is that what you are researching is socially relevant’ (R4).

In this example, the researcher shows that when dialoguing with the community, the research team ensures that they are researching socially important issues for the Roma community. Consequently, when the community is aware of the research and how to contribute, the possibility to achieve social impact increases. In this way, we can avoid doing research that is only relevant from a scientific point of view, avoiding bias derived from research only based on data and scientific knowledge and not on the knowledge provided by the Roma community.

The case of gender violence research in Roma women is an example of how researchers have a better understanding of the situation through dialogue with Roma women. In this vein, one interviewee referred to a research project focused on gender violence within the Roma community. If researchers consider only official data instead of expanding the knowledge with the community, it could lead them to assume that gender violence does not exist in this community. However, through the inclusion of the knowledge of Roma women, researchers identified that the Roma population is not accessing formal help and support mechanisms for victims. If we consider only the women who receive help and support as indicator, the information will be very limited. Instead, the involvement of the community from the beginning of the research has allowed researchers to know better the problem and thus, define more accurately the project objectives.

‘Many times, Roma women are, not many, but they are in those formal assistance mechanisms, what happens is that they do not say so. So, how can we detect the assistance strategies that they already have and, at the same time, generate protocols so that at the same time the public administration can learn from what the community itself is already doing? The methodological innovation of the project has been to work with the community’ (R2).

Finally, one of the interviewees refers to the importance of really want to know the problem and the community even before starting to collect the data. This implies that having a prior dialogue with the community allows the overcoming of barriers to access to data collection, influencing the proper development of the project.

‘For example, how to reach the Roma people to know the reality in which they are living has been easier for us, for example, to enter the schools, or to interview the families, to have dialogues with them when there had been a decision and mutual knowledge and dialogue before the collection of data of these social impact indicators. This has allowed us to overcome barriers or limitations that can occur in other research when there is no such previous dialogue’ (R3).

To sum up, incorporating the Roma population from the beginning of the research process allow researchers to better understand the problem, which in turn influences the aim of the research and, therefore, the indicators to be selected.

Co-creation of Impact Indicators in the Methodological Design

Once we know better the reality that we are studying and we are able to define the objectives and the impacts to achieve, we should select the indicators that will allow us to monitor whether we are on the right track to achieve the social impact throughout the project.

The adequate selection of indicators is crucial, as they influence all the research and guide the investigation process.

‘Introducing these indicators, can be in both quantitative and qualitative instruments when we design the instruments themselves, these indicators will mark us. In this context, we collected data through an online questionnaire and, in the interview scripts, the sections that we elaborated were thought out, and the search for information was designed and organized based on those indicators. If we had eight indicators, we tried to have all of them in each one of the instruments, that is why it is fundamental, it guides you in the research process’ (R3).

The inclusion of the Roma community from the beginning of the research is not only addressed to know more about the Roma, but also to include this community in the relevant decisions of the research project, facilitating to take join decisions. This aspect is especially relevant when addressing the indicators since it is the element that allows researchers to guarantee impact. When deciding jointly the impact indicators, researchers also get a deeper knowledge and can modify their methodologies accordingly.

‘To incorporate social impact indicators from the design [...] for example: reducing school dropouts, or increasing academic results, or reducing absenteeism, [...] so that they really lead us to achieve that impact it is fundamental to have put them in agreement with the participants from the beginning of the research, which is, in this case, the Roma population. Talking about these indicators with them allows us to have a better knowledge of the reality they live and to judge and revise our methodologies according to that reality’ (R3)

Furthermore, another interviewee refers to the importance of having an orientation towards the search for social impact from the beginning of the project design. First, as we have pointed out, the understanding of the problem is crucial, as well as the knowledge about what the community needs at the specific moment of implementing the project and what the research can contribute to. Second, the inclusion of the indicators in the project conceptualization itself is also a key element.

‘The indicators themselves, not the indicators per se but to include them in the development of the project, and somehow using, putting inside a tool that will allow that project to have an impact. The impact will not come just because, but because in the very conceptualization you foresee, you already want to make that project have an impact, there is a will there. And you do this with concrete things such as indicators’ (R6).

Another fundamental aspect is that these indicators should be focused on the research objective. In this line, social impact improves the research to solving the problem that we define with the Roma community.

‘It is not only about taking into account your expertise and looking for indicators of that place, that is if I want to have a social impact, but it is also the same if my line is economics or anthropology, it is about looking for indicators and analyzing what will really improve the situation’ (R4).

By selecting indicators for social impact, the information that would not appear in the dominant discourse emerges and can be captured by the researchers. For example, concerning education, one researcher refers to the context in which the pandemic started and how schools that encouraged family participation could organize themselves more quickly to provide tablets to their students, including Roma, allowing them to follow the lessons.

‘students will continue to receive the teaching-learning process we can identify in that way, because we have gone to look for what forms of organization, in this case, the school context, could facilitate reducing the potential negative impact it could have and on the contrary, obtain some positive impacts that would not appear in the dominant discourse’ (R3).

The selection of indicators according to the problem that we have been previously defined with the Roma community is not enough. The community itself can also participate in the selection, as they can bring ideas about the impact indicators that will enrich the process and results. In this vein, it is convenient to include their ideas on the basis of dialogue to support the overcoming of power relations. Furthermore, this will allow the creation of new indicators based on the Roma community participation.

Studies with the Roma community ‘are very focused on the problem, but not very focused on the solution. Many Roma families, and through the communicative methodology is what we see, is that when a person has a problem, they also have ideas of how it could be overcome. Collecting these ideas could be the germ of these social impact indicators because it would be the solution. To achieve social impact would be to overcome these problems. You can already include in these indicators the ideas the community has about how to overcome this problem. For example, in the case of all the hoaxes that have come out about the Roma community during the pandemic, an indicator of social impact could be what images or results have obtained that transform this image and show that it is false’ (R5).

Research with social impact implies the identification of evidence and contributions that overcome scientific fakes or hoaxes, in which, for example, the Roma community can be wrongly associated to a lack of interest in education. The introduction of indicators agreed upon with the participants in the methodological design implies focussing the object of study on this search. This will improve the Roma community academic results if the evidence is applied. Thus, one of the researchers explains:

‘If you involve the Roma community from the beginning, and with them, you define these indicators... It is a way that all the data and everything that you are collecting, and you are analyzing methodologically, also provide you with some data that will allow you to dismantle all the anti-gypsyism and racism that exists, that there was a lot during the pandemic. Often, the Roma community has even been considered guilty of spreading the virus, which occurred in La Rioja, and other episodes. Similarly, possible indicators would be how stereotypes and social images are improved or how they can be overcome. If these indicators were discussed in agreement with the Roma community, this would make the methodological design more limited and more aimed at collecting evidence to show that these stereotypes are false and, therefore, meet the objective of social impact’ (R5).

It is not important if the indicator is quantitative or qualitative, the important aspect is that selecting jointly with the involvement of citizens, researchers will be closer to achieving social impact, and reaching it through the methodology, implying an improvement of research

‘It is not that social impact can only be achieved with one type of methodology, as each social impact, each type of impact, will have its appropriate methodologies but, whether it is quantitative, qualitative or mixed, it will be much more socially relevant and more likely to be achieved if there is this citizen participation. In fact, in the “Monitoring” document on how Horizon Europe’s social impacts will be measured [...] one of the pillars is citizen participation [...] because it is considered that to the extent that you include this dialogue, you are more likely to achieve social impact. This process cannot be separated from the methodology because it is the path that will lead you to this social impact and having these indicators, a priori discussed, will undoubtedly improve the methodology’ (R5).

In this way, the selection of the indicators is also modified through the citizen’s participation as we first select indicators about a problem jointly decided and second, we incorporate the community into the selection and creation of indicators.

Contextualization of the Indicators

Finally, assuming that the indicators have been co-created with the community, it seems coherent to understand that their orientations will contribute to achieving improvements for society, which is the research objective with social impact. In this vein, an aspect appearing in the semi-structured interviews is that research oriented towards social impact cannot select merely descriptive indicators since we would obtain a limited vision of the reality. It is necessary to analyse the indicators in a context. For example, as one of the researchers explains, analysing the school absenteeism of Roma students during COVID-19 can be done with the attendance indicator, but it will only provide a biased vision without social impact. When analysing and looking for social impact, a broader context is taken into account and not only the fixed indicator.

‘With the theme of the COVID, school failure, absenteeism, from the official institutions has made a more abstract analysis of whether there is absenteeism or not, and that is it. [...] Talking to the (Roma) families there was a great fear. Suddenly, they would all go to school together because they wanted to see what impact that had. If you want to research educational success and reduction of absenteeism, not only on whether they go or not, (you have to take into account) at what moment this is happening and that the reasons can be very diverse, not only that “it’s a Roma community and that is it”. They were analyzed as saying “as they are Roma, it is normal that there is absenteeism”. We have to introduce other elements that help avoid absenteeism, for example, the school decided, that first month, to give a little more dialogue, (work) more from a distance. If the investigations this aspect is taken into account, because what you want is the educational improvement and the decrease of the absenteeism, you have to go with a wider data analysis’ (R4).

In another moment of an interview, a researcher mentions this aspect, pointing out that incorporating social impact indicators requires considering aspects that otherwise research would not have. Besides, those indicators are challenging to know without the community’s participation. This aspect is closely related to the more excellent knowledge of the problem achieved by discussing the indicators.

‘When we design the research, the fact of incorporating social impact indicators that improve the reality of the community we work with, if what we want is that they improve their lives or that their situation improves, we take into account aspects that otherwise we would not have, which would only be more descriptive. But to know those aspects that sometimes you don’t control, you have to talk to them’ (R4).

When investigating looking for the social impact, we do not pay attention only to the first indicator, which would be the lack of attendance of the Roma community at school. When talking with the Roma, when dialoguing, we see how this absence is justified by a fear of contagion and furthermore, this fear can be overcome. The social impact leads us to focus on how this fear can be overcome with reduced absenteeism and not on the first data of high absenteeism.

Similarly, we need to adjust our methodology to the needs of each moment. In order to obtain better chances to achieve social impact, in addition to including the Roma population, these voices must be diverse, from different fields, contexts and disciplines. This aspect will contribute to capture the multi-faceted nature of the problem and the moment when it occurs. All this allows us to adjust the methodology in our design to achieve social impact in diverse moments.

‘When the [COVID19] vaccine arrives, the challenge will be to vaccinate everyone and to communicate the relevance it has on the Roma population, which will generate an effort that has to be interdisciplinary. Moreover, if we do not see it from the perspective of social impact, we do not (see it). On the other hand, the question is to really adjust the methodology to the needs of each moment. To be able to capture what is needed at that moment’ (R2).

Another fundamental aspect is that researchers should assess them slightly differently because the goal in terms of the indicators is to show whether they are on the correct way to achieving social impact.

‘In the FP7 project, it was one of the things that said all the benefits, that from the beginning of the research design, there is that goal of social impact. It has to be defined from the beginning and this orientation of the research conditions all the methodological steps you take. How do you collect the data, how do you analyze, within the data collected, what information you prioritize, what is most useful for you to get those indicators to show you that there has been or that there can be a potential social impact’ (R3).

Finally, a relevant aspect is how the methodology itself leaves a mark on the participants and how it is a potential impact. Therefore, the use of methodologies that involve the participants is an opportunity to achieve social impact on the research process itself.

‘These indicators [of social impact decided with the community] are not only important but also necessary and offer a potential that has not yet been explored because it is very incipient. But it allows us not to miss the opportunity to optimize the potential impact of the methodology, which we know that leaves a mark and impacts the participants. That this impact is not random, based on any intervention, without thinking, but designed and elaborated from that look, from those indicators that offer an enormous potential of improvement from the beginning’ (R3).

All this implies that when using indicators, we must consider that they will serve to measure social impact, which must always be present since it has diverse implications.

Conclusions

Research demonstrates to have a crucial role in solving the problems that society faces. However, to achieve this goal, an adequate design of the entire research strategy is required. For instance, the European Commission through the research programme Horizon Europe includes novelties such as the need to guarantee that the improvement of society is achieved. This should be done through specific indicators (Bruno & Kadunc, 2019; European Commission, 2018). As the literature shows, there is still no agreement on which indicators should be used to measure research projects' social impact (Godin & Lane, 2013; Greenhalgh et al., 2016) but clear advances have been done in this regard (European Commission, 2018). The experience of previous programs confirms that community participation is key to improving social problems. For example, the report ‘Mission-oriented research and innovation in the European Union’ (2018) addresses in its Government Missions, how to engage citizens in co-designing, co-creating, co-implementing and co-assessing missions since it considers this participation a guarantee to achieve the mission, as developed by Mazzucato (2019).

Within a research project, a key aspect identified in all the interviews is that in the project design which is addressed to achieve social impact, the indicators should be agreed with the community. This implies that participation and egalitarian dialogue with the community are essential aspects even before the design of the project. It is not enough to think and design a project and then evaluating its social impact, it is about how the project is built from a bottom-up approach to achieve social impact. In this way, the social impact considered from the research design and specifically when deciding indicators, has methodological implications.

The community’s incorporation from the beginning of the research to selecting social impact indicators, allows a deeper knowledge of the problem. In turn, it allows defining the research objectives, determined by the social impact indicators, to be better designed and more appropriate to the reality being studied, more tightly and accurately formulated. Our conclusion is similar to other academic disciplines, for example, the case of biomedical sciences in which Pollitt et al. (2016) identify a broader social benefit in those research topics that concerned researchers and society. Although, in our case, it is not only that there is a greater social benefit because citizens and scientists agree, but also that this improvement is that scientists can give a better response to the social problem since they know better the reality before developing the study. For example, when looking for social impact indicators in egalitarian dialogue with the Roma community, we identify that the assistance to social resources for abused women is not an adequate indicator to measure the incidence of gender violence in the community because most Roma women do not use these resources. Of course, the objective of reducing gender-based violence among the Roma community is a shared objective between the community and the scientific team, but in the discussion of the indicators, researchers understand better the problem, going deeper into their reality. Without a previous dialogue addressing the indicators, there is a risk of focussing the project only on the formal aid mechanisms, leaving out the informal ones which are used by the community.

Another aspect is that the indicators can also be co-created with citizens. The document ‘Governing Missions in the European Union’ (Mazzucato, 2019) refers to co-creation as ‘to allow as many citizens as possible to engage in the mission-definition process at an early stage’ (p. 7). We suggest going a step further because it is not only to decide in an early stage with citizens what the indicators should be, as people can create and generate these indicators in a joint dialogue with researchers.

The Roma community suffers and has suffered discrimination in all areas of life. This discrimination is even identified in research since most of the research has focused on describing the problems, sometimes even blaming the Roma and far from contributing to improving their situation. They have perpetuated stigmas such as not caring about their children’s education. One of the social impact indicators may be precisely overcoming those myths that contribute to their social stigmatization. Giving visibility to the actions they are taking to overcome these stereotypes and using them as the indicators towards which we must go. In this view, the co-creation also implies that the indicator is created by them, by their actions, which in a way, is also empowering for citizens. This is important specially for vulnerable groups, but citizenship in general would also benefit.

The identified aspects can be applied to any methodology and field of investigation, not only with vulnerable groups. For example, another problem facing the world’s population is climate change. Deciding the indicators of climate change with the community could help to reduce the number of people questioning its existence, since they have decided the indicators that confirm its existence. At the same time, it would allow scientists to better understand what contributes to question the research results for some people, and include mechanism that minimize such risk. All this would lead to a greater involvement by increasing the number of people to address the changes necessary to face climate change.

These conclusions, although still exploratory, aim to contribute to the debate on the indicators to be used when evaluating social impact. More specifically, it is addressed to know how the active role that citizens can have in research has methodological implications that are challenging and exciting when designing research projects.

Footnotes

Acknowledgments

The authors would like to express their thanks to the professionals who participated in the interviews

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the European Social Fund and Spanish Agency of Research under the Ramon y Cajal grant number RYC2018-025,860-I.

ORCID iD

Gisela Redondo-Sama

References

Aiello

Donovan

Duque

Fabrizio

FlechaHolm

R.P.

Molina

Oliver

Reale

(2020). Effective strategies that enhance the social impact of social sciences and humanities research. Evidence & Policy, 17(1), 1–16. https://doi.org/10.1332/174426420X15834126054137

Arza Porras

Gil-González

Catalá Oltra

Francés García

González Angulo

M. E.

Rodríguez Camacho

M. F.

Sanchís Ramón

M. J.

Sanz-Barbero

Vives-Cases

La Parra Casado

(2020). COVID-19 Crisis: Impact on households of the Roma community. International Journal of Roma Studies, 2(2), 28–51. https://doi.org/10.17583/ijrs.2020.6242

Bell

Shaw

Boaz

(2011). Real-world approaches to assessing the impact of environmental research on policy. Research Evaluation, 20(3), 227–237. https://doi.org/10.3152/095820211X13118583635792

Bornmann

(2013). What is societal impact of research and how can it be assessed? A literature survey. Journal of the American Society for Information Science and Technology, 64(2), 217–233. https://doi.org/10.1002/asi.22803

Bornmann

Haunschild

(2017). Does evaluative scientometrics lose its main focus on scientific quality by the new orientation towards societal impact? Scientometrics, 110(2), 937–943. https://doi.org/10.1007/s11192-016-2200-2

Bornmann

Haunschild

Adams

(2019). Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF). Journal of Informetrics, 13(1), 325–340. https://doi.org/10.1016/j.joi.2019.01.008

Bruno

Kadunc

(2019). The next EU research & innovation investment programme (2021–2027). Based on the commission proposal for Horizon Europe, the common understanding between co-legislators and the partial general approach, both approved in April. https://rio.jrc.ec.europa.eu/sites/default/files/events/Bruno-Kadunc_The%20Next%20Eu%20RI%20Programme.pdf

De Jong

Barker

Cox

Sveinsdottir

Van den Besselaar

(2014). Understanding societal impact through productive interactions: ICT research as a case. Research Evaluation, 23(2), 89–102. https://doi.org/10.1093/reseval/rvu001

Derrick

G. E.

Faria

Benneworth

Budtz-Petersen

Sivertsen

(2018). Towards characterising negative impact: introducing grimpact. In Costas

Franssen

Yegros-Yegros

(Eds), Proceedings of the 23rd International Conference on Science and Technology Indicators. Centre for Science and Technology Studies (CWTS), Leiden University.

10.

Erden-Basaran

(2021). Protecting refugee students’ wellbeing after research. International Journal of Qualitative Methods, 20(1-14). https://doi.org/10.1177/16094069211025892.

11.

European Alliance for Social Sciences and Humanities - EASH (2011). Horizon 2020: Social sciences and Humanities research provides vital insights for the future of Europe. EASH. http://www.eash.eu/openletter2011/index.php?file=openletter.htm

12.

European Commission (2018). Monitoring the impact of the EU framework programmes. Publication Office of the European Union.

13.

European Union Agency for Fundamental Rights -FRA (2018). A persisting concern: anti-Gypsyism as a barrier to roma inclusion. FRA.

14.

Flecha

Soler-Gallart

Sordé

(2015). Social impact: Europe must fund social sciences. Nature, 528(7581), 193. https://doi.org/10.1038/528193d

15.

Godin

Doré

(2005). Measuring the impacts of science: Beyond the economic dimension. INRS.

16.

Godin

Lane

J. P.

(2013). Pushes and pulls (Hi)story if the demand pull model of innovation. Science, Technology and Human Values, 38(5), 621–654. https://doi.org/10.1177/0162243912473163

17.

Gómez

Racionero

Sordé

(2010). Ten years of critical communication methodology. International Journal of Qualitative Research, 3(1), 17–43. https://doi.org/10.1525/irqr.2010.3.1.17

18.

Greenhalgh

Raftery

Hanney

Glover

(2016). Research impact: A narrative review. BMC Medicine, 14(1), 78. . https://doi.org/10.1186/s12916-016-0620-8

19.

Gupta

Deokar

Iyer

Sharda

Schrader

(2018). Big data & analytics for societal impact: Recent research and trends. Information Systems Frontiers, 20(2), 185–194. https://doi.org/10.1007/s10796-018-9846-7.

20.

Haustein

(2016). Grand challenges in altmetrics: Heterogeneity, data quality and dependencies. Scientometrics, 108(1), 413–423. https://doi.org/10.1007/s11192-016-1910-9.

21.

Higher Education Funding Council for England (2011). REF2014: Assessment framework and guidance on submissions. http://dera.ioe.ac.uk/id/eprint/11755.

22.

Islam

(2020). Untold Stories: A study of sudanese and Syrian refugees in Estonia. International and Multidisciplinary Journal of Social Sciences, 9(1), 1–25. http://doi.org/10.17583/rimcis.2020.5131

23.

Joly

P.-B.

Gaunand

Colinet

Larédo

Lemarié

Matt

(2015). ASIRPA: A comprehensive theory-based approach to assessing the societal impacts of a research organization. Research Evaluation, 24(4), 440–453. https://doi.org/10.1093/reseval/rvv015

24.

Kastrinos

(2010). Policies for co-ordination in the European research area: A view from the social sciences and humanities. Science and Public Policy, 37(4), 297–310. https://doi.org/10.3152/030234210X496646

25.

Khalfaoui

García-Carrión

Villardón-Gallego

(2020). Bridging the gap: Engaging Roma and migrant families in early childhood education through trustbased relationships. European Early Childhood Education Research Journal, 28(5), 701–711. https://doi.org/10.1080/1350293X.2020.1817241

26.

König

(2019). Social sciences and humanities research matters - guidelines on how to successfully design, and implement, mission-oriented research programmes. In Centre for social innovation, impact of social sciences and humanities for a European research agenda valuation of SSH in mission-oriented research (pp. 26–29). Centre for Social Innovation.

27.

Matulič-Domadzič

Munté

De Vicente-Zueras

León-Jiménez

(2020). “Life starts for me again”. Social impact of psychology on programs for homeless people: Solidarity networks for the effectiveness of interventions. Frontiers in Psychology, 10, 3069. DOI:10.3389/fpsyg.2019.03069. https://10.3389/fpsyg.2019.03069.

28.

Mazzucato

(2019). Governing missions in the European union. Independent Expert Report https://ec.europa.eu/info/sites/default/files/research_and_innovation/contact/documents/ec_rtd_mazzucato-report-issue2_072019.pdf

29.

Molas-Gallart

Tang

(2011). Tracing ‘productive interactions’ to identify social impacts: An example from the social sciences. Research Evaluation, 20(3), 219–226. https://doi.org/10.3152/095820211X12941371876706

30.

Moore

Cameron

Martin

PE.

Paul O’Donnell

Pattinson

(2017). ‘Excellence R Us’: University research and the fetishisation of excellence. Palgrave Communications, 3(1), 17010. https://doi.org/10.1057/palcomms.2016.105

31.

Parthenis

Fragoulis

(2020). La principals’ views on policies and practices for the educational inclusion of Roma people. International Journal of Sociology of Education, 9(3), 234–261. http://doi.org/10.17583/rise.2020.5050

32.

Pollitt

Potoglou

Patil

Burge

Guthrie

King

Wooding

Grant

(2016). Understanding the relative valuation of research impact: a best–worst scaling experiment of the general public and biomedical and health researchers. BMJ Open, 6(8), e010916. http://doi.org/10.1136/bmjopen-2015-010916

33.

Priem

Groth

Taraborelli

(2012). The altmetrics collection. PloS One, 7(11), e48753. https://doi.org/10.1371/journal.pone.0048753

34.

Pulido

C. M.

Redondo-Sama

Sordé-Martí

Flecha

(2018). Social impact in social media: A new method to evaluate the social impact of research. PLoS ONE, 13(8), e0203117. https://doi.org/10.1371/journal.pone.0203117

35.

Pulido

C. M.

Villarejo-Carballido

Redondo-Sama

Gómez

(2020). COVID-19 infodemic: More retweets for science-based information on coronavirus than for false information. International Sociology, 35(4), 377–392. https://doi.org/10.1177/0268580920914755

36.

Reale

Avramov

Canhial

Donovan

Flecha

Holm

Larkin

Lepori

Mosoni-Fried

Oliver

Primeri

Puigvert

Scharnhorst

Schubert

Soler

Soòs

Sordé

Travis

Van Horik

(2018). A review of literature on evaluating the scientific, social and political impact of social sciences and humanities research. Research Evaluation, 27(4), 298–308. https://doi.org/10.1093/reseval/rvx025

37.

Robinson-Garcia

van Leeuwen

T. N.

Ràfols

(2018). Using altmetrics for contextualised mapping of societal impact: From hits to networks. Science and Public Policy, 45(6), 815–826. https://doi.org/10.1093/scipol/scy024

38.

Ruegg

Feller

(2003). A toolkit for evaluating public R & D investment: Models, methods, and findings from ATP's first decade. US Department of Commerce, Technology Administration, National Institute of Standards and Technology.

39.

Soler

Gómez

(2020). A citizen’s claim: Science with and for society. Qualitative inquiry, 26(8–9), 107780042093810. https://doi.org/10.1177/1077800420938104

40.

Wilsdon

Allen

Belfiore

Campbell

Curry

Hill

Jones

Kain

Kerridge

Thelwall

Tinkler

,(2015). The metric tide: Report of the independent review of the role of metrics in research assessment and management. Higher Education Funding Council for England (HEFCE).

41.

Work

Haustein

Bowman

T. D.

Larivière

(2015). Social media in scholarly communication. A review of the literature and empirical analysis of twitter use by sshrc doctoral award recipients. Canada research chair on the transformations of scholarly communication. University of Montreal.