Results of a pilot test of effects of an online resource on lay audience understanding of media reports on breast cancer research

Abstract

Objective:

This study aimed to assess the effects of eXamining Relevance of Articles to Young Survivors (XRAYS), an online resource that provides brief articles discussing recent research relevant to young women with or at risk for breast cancer, on users’ knowledge about information covered in media reports. An evaluator also assessed the degree to which XRAYS facilitated awareness of recent research findings and how users recommended improving the resource.

Design:

Participants were randomly assigned to an intervention condition, whose members read original news reports and the corresponding XRAYS review or to a control condition, whose members read only original news reports. Participants also discussed XRAYS’ utility and cultural appropriateness.

Setting:

Data were collected in a meeting room at a Facing Our Risk of Cancer Empowered (FORCE) conference. All participants were registered for a conference for women with or at risk for genetic breast or ovarian cancer.

Method:

Participants answered multiple choice questions about research before reading the assigned materials and again after reading the assigned materials. T-tests were used to assess pre-post-test change for both groups and to compare groups’ change in knowledge.

Results:

Both groups improved between pre- and post-test. The treatment group increased knowledge significantly more than the control group. Focus group results indicate that XRAYS is a valuable resource for identifying relevant recent research findings and for explaining limitations in research methods, relevance, and reporting quality. Results also indicate that it is critical for XRAYS and similar resources to be brief, use non-technical language and address the most recent trends in media coverage.

Conclusion:

Online resources that use plain language to help users to interpret science reporting critically help meet an important health education need.

Keywords

Evaluating media reporting lay audiences online resources research interpretation young breast cancer survivors

Breast cancer is the leading cause of cancer among women younger than 40 years and the second leading cause of cancer-related deaths among all women in the USA. Approximately 7% of women with breast cancer are diagnosed before age 40 (Assi et al., 2013). Breast cancer risk, aetiology, treatment, outcomes and related survivorship and quality of life concerns often vary between younger and older women (Freedman and Partridge, 2013; National Cancer Institute, 2014). Diagnosis at a young age is associated with a higher risk of recurrence; second malignancy and mortality, morbidity and quality of life issues (Freedman and Partridge, 2013). Young breast cancer survivors often have genetic mutations associated with breast cancer (Anders et al., 2009; Freedman and Partridge, 2013). Potential long-term effects of breast cancer therapies, such as early menopause, fertility impairment, lowered bone density and risk of second malignancies are of special relevance to the quality of life of younger breast cancer survivors (Finch et al., 2013; Freedman and Partridge, 2013). Cancer treatment and its after effects can lead to concerns about decreased productivity at home and work (Freedman and Partridge, 2013). Research suggests that emotional and social functioning after a breast cancer diagnosis is inversely proportional to age at the time of diagnosis (Anders et al., 2009).

Distinct issues related to genetics, cancer treatment side effects and resultant quality of life issues facing young women at high risk for breast cancer correspond to needs for information about the latest research and recommendations regarding genetic risk and testing as well as benefits and risks of prevention and treatment options. Young women frequently seek health information through the Internet and mainstream media (Pugh-Yi et al., 2018; Rowlands et al., 2015) but often find that it does not address their particular concerns, that it is difficult to evaluate or interpret, or even misleading (Fergie et al., 2013; Laugesen et al., 2015). Often the media miss or omit the nuance to a research report. Constrained by deadlines, space or relevant context, health media articles can appear to contradict one another, causing confusion in those they intend to inform (Dobransky and Hargittai, 2012; Freedman, 2013). Interpreting media reports on research findings, including determining the implications for younger women, is often challenging for readers. Atkin et al. (2008) report that information available through the Internet and mainstream media has important gaps in addressing the needs of young women diagnosed with breast cancer. These authors found that information about risk is often unclear, which may lead readers to be less likely to take preventive measures. Young women with breast cancer or at high risk of breast cancer need accurate, clearly presented information based on sound evidence to help them make informed decisions about their specific health needs.

Facing Our Risk of Cancer Empowered (FORCE) has developed the US Centers for Disease Control and Prevention (CDC)-funded eXamining Relevance of Articles to Young Survivors (XRAYS) programme. FORCE is a non-profit organisation with a mission of providing support, information and advocacy services to families facing hereditary breast or ovarian cancer. XRAYS is an online resource that provides brief articles summarising recent research relevant to young women with or at risk for breast cancer. XRAYS is on a website (www.facingourrisk.org/xrays) that presents a new article review approximately weekly. XRAYS articles rate the quality and relevance of research, the quality of media reporting, and offer questions that the user may want to address with health care providers. One critical aim of XRAYS is to summarise information about recent research to make it clearer for lay audiences.

This pilot test of XRAYS’ effectiveness aimed to address the following questions:

After reading a media article, do XRAYS users demonstrate more accurate knowledge of content than a control group that read only the media article?

Is the target audience more likely to read recent articles if they are reviewed in XRAYS than if they are not?

What do participants find to be useful about XRAYS?

What types of information would users like to learn from XRAYS?

Methods

To address these questions, the evaluation lead (R.H.P.Y.) collected pre- and post-test data and focus group data at the FORCE annual conference in June 2017. Conference participants include women with or at risk for hereditary breast or ovarian cancer and their carers seeking information about prevention and treatment. Thus, this setting was identified as ideal for assessing target audience response to XRAYS. Study procedures were reviewed and approved by the FORCE President (SF) and Vice President of Education, who were responsible for reviewing the ethics of proposed research with FORCE constituents. In addition, a member of the XRAYS steering committee (LR) reviewed the proposed procedures and made recommendations regarding protection of participants’ privacy. Participants were not asked to provide any personally identifying information or undertake any known risk. They were informed of the time and effort burden, advised that participation was voluntary and that they could stop participation at any time. Participants received a US$25 gift card as compensation.

Instruments

The evaluation lead developed a set of five multiple choice questions for each of three XRAYS summary reports about factual content in the article reviewed. The XRAYS scientific lead (P.W.) reviewed and approved each set of questions. The evaluation lead also developed a focus group protocol, which the scientific lead reviewed and approved.

Participants

Participants were 21 women between ages 18 and 45 years, who attended the FORCE conference and volunteered to participate in one of four groups. A total of 20 participants were white and one was African American. Two participants were health care providers.

Procedures

At the beginning of each session, the evaluation lead explained to all participants in both treatment and control groups that the purpose of the study was to assess whether reading an XRAYS review plus a media report on research increased understanding of the research more than reading the report alone. She emphasised that individual participants were not being evaluated, that it was essential to obtain accurate information about knowledge changes for each group and therefore critical for each participant to answer all questions independently, without discussing with any other participant. The evaluation lead then randomly assigned each participant to the treatment or control group and assigned each participant a random unique identifier. She monitored all participants while they answered pre- and post-test questions. All members, control and treatment, of each focus group read the same media article. All participants then answered the five questions corresponding to the article selected for their group session, using paper and pen. Participants were instructed to answer independently and not to discuss with each other. Both groups of participants then read a media article. Participants in the treatment group also read the corresponding XRAYS review. After reading the material, all participants answered the same questions, following the same instructions. All participants finished procedures within 10 min.

The scientific lead and evaluation lead developed questions by identifying relevant factual information in the reports, as well as research limitations, then drafting questions that would indicate whether readers knew these facts. All items were presented in multiple choice format. Sample questions are presented in Table 1. Items were scored as correct or incorrect.

Table 1.

Sample test questions.

Article Sample Item
Article 1 Tumor gene test may determine which breast cancer patients can forgo chemotherapy	Clinical risk is: a. Best assessed with genetic testing b. Estimated through family history and exposure to environmental toxins c. An outdated predictor of 10-year survival d. Indicated by age, menopausal status and estrogen receptor status of tumours and number of lymph nodes involved in cancer
Article 2 Is there a link between exercise and memory impairment for breast cancer survivors?	Which of the following is true? a. Exercise may improve breast cancer survivors’ memory by reducing stress b. Exercise reduces stress but is linked to memory impairment among breast cancer survivors c. Exercise has several benefits, but does not affect memory among breast cancer survivors d. The level of exercise necessary to improve memory among breast cancer survivors is associated with moderate health risks
Article 3 Can acupressure be used to treat cancer-related fatigue?	Which of the following is true? a. Cancer-related fatigue is not scientifically confirmed b. Cancer-related fatigue occurs during treatment, then is rarely an issue for breast cancer survivors c. About one-third of breast cancer survivors experience cancer-related fatigue d. Nearly all breast cancer survivors report cancer-related fatigue to their health care providers

After all the participants had completed post-test responses, they participated in a focus group facilitated by the evaluation lead and lasting approximately 45 min. Prior to the focus group, the evaluation lead described XRAY’s function and content, and provided printouts of XRAYS materials for all participants, including control groups. The following are the questions discussed during focus groups:

Had you heard of XRAYS before this focus group? If so, how did you learn about XRAYS?

Would you have learned about the information in the article you just read if it were not promoted by XRAYS?

How are you most likely to learn about recent research on cancer, survival and health?

Would XRAYS help you to learn about recent research related to health? If so, what features of XRAYS make it useful?

What types of articles would you like to see reviewed in XRAYS?

What kinds of health information are you most likely to look for in the media?

How do you use health information you learn about from the media?

Analysis

Knowledge improvement was assessed using within-groups t-tests to compare pre- and post-test mean percentages of correct responses for each group. A between-groups t-test then compared the changes in percentage of correct items for the treatment and control groups.

Focus group responses were analysed using content analysis to identify themes and highlights. The evaluation lead took notes during the focus group discussions. After discussion about each question, she confirmed with participants that she had accurately recorded key points. Following discussion, the evaluation lead thoroughly read all notes, extracted quotes and text directly responding to study questions and compared responses across individuals and focus group sessions, noting the number of individuals who had made key points. The evaluation lead identified points of agreement and disagreement, recommendations and recurring comments and questions, with a focus on comments that were likely to be useful in informing FORCE’s ongoing development of XRAYS as well as its communications planning.

Results

While there were 21 participants, one control group participant took her pre- and post-test instruments and one participant took away her pre-test instrument when they finished their sessions. This resulted in an analytic sample of 19 (11 treatment, 8 control) for the assessment of pre-post-test knowledge change. The responses of all 21 participants were included in the focus group analysis.

Knowledge change

Table 2 shows the results of the within-group t-tests. Table 3 shows the results of the between-group t-test.

Table 2.

Results of within-groups comparisons.

Group	N	Pre-test mean percentage correct responses (SD)	Post-test mean percentage correct responses (SD)	t-score (df)	p
Control	8	57.5 (12.82)	72.5 (23.75)	−2.05 (7)	0.08 (n.s.)
XRAYS	11	52.73 (20.54)	87.27 (16.18)	−7.29 (10)	0.000 (<0.001)

XRAYS: eXamining Relevance of Articles to Young Survivors; SD: standard deviation.

Table 3.

Results of between-group comparisons.

Group	N	Mean change score (SD)	t-score (df)	P
Control	8	15.00 (20.70)	−2.34 (18)	0.03
XRAYS	11	34.55 (15.72)	−2.34 (18)	0.03

XRAYS: eXamining Relevance of Articles to Young Survivors; SD: standard deviation.

Figure 1 illustrates changes in user knowledge before pre- and post-test.

Figure 1.

Pre-post-test changes in user knowledge.

At baseline, both groups answered an average of just more than half of the items correctly. While both groups increased knowledge between pre- and post-test, only participants in the XRAYS group increased their knowledge at a statistically significant level. A t-test comparing the average change between groups indicated that the increase in knowledge among XRAYS participants was significantly greater than the change for control group participants. While control group participants’ scores increased an average of 15%, XRAYS participants’ scores increased an average of 34.55%.

Focus groups

This section summarises key results of focus group discussions. Because all participants had seen XRAYS materials and heard a description of the resource before discussion, none referred to the knowledge change assessment during the discussion, and control and treatment group responses were analysed together.

Had you heard of XRAYS before this focus group? If so, how did you learn about XRAYS?

Of the 21 focus group participants, 18 reported that they had heard of XRAYS prior to participating. Responses indicated that participants had learned about XRAYS through FORCE communications materials, including the FORCE newsletter, website or Facebook, suggesting that these channels are effectively making the target audience aware of XRAYS.

Wouldyou have learned about the information in the article you just read if it were not promoted by XRAYS?

Only five participants indicated that they would probably have read the media article they reviewed if it had not been reviewed by XRAYS. One of these participants was a current acupressure patient who reviewed an XRAYS article about acupressure for breast cancer patients. Another was a nurse who regularly reviews recent research reporting. The remainder indicated that they view XRAYS as an effective tool for identifying which recent research findings are worth reviewing.

All participants indicated that they had sought information frequently after their initial diagnosis. Those who reported that significant time had passed since their initial diagnosis said that they no longer actively searched for new information. Several participants reported that they had become exhausted by excessive information and that they now screen it. Participants noted that they had limited time and reported that XRAYS helps them find the most important information quickly. One participant said,

When I was first diagnosed, I was looking stuff up for hours every day. Now, it’s too much coming at me all the time. I’m only going to read more if there’s a good reason-like news about a big breakthrough.

How are you most likely to learn about recent research on cancer, survival, and health?

Participants indicated that they mostly sought information about cancer- and health-related issues online and through personal conversations with family and friends. They indicated the importance of confirming the credibility of online resources. Sources respondents considered credible include: online discussions moderated by researchers, listservs and social media groups hosted by breast cancer advocate organisations, peer-reviewed journals, federal government health agencies, and websites hosted by well-known medical organisations such as the Mayo clinic.

WouldXRAYS help you to learn about recent research related to health? If so, what features of XRAYS make it useful?

All participants indicated that XRAYS was a valuable resource that they used or intended to use. Participants indicated that its use helped them to determine which new research is relevant and worth learning about. One woman said, ‘I think of XRAYS as like Snopes (an Internet fact-checking site) for breast cancer information’. Nearly all participants reported that XRAYS helped them to identify biases and limitations, such as sample size and characteristics, potential conflicts of interest and alternative explanations for results that they would not have known how to find on their own.

Other features that participants appreciated are presented in Table 4.

Table 4.

Focus group participant-identified positive features of XRAYS.

Clear explanation of research methods

Emphasis on information that is relevant for lay people making health-related decisions

Clear description of study methods and statistics for lay people who are considering study findings in their health decisions

Reporting that is not sensationalised

Realistic reporting that is neither alarmist nor complacent

Identifies the populations for whom results are relevant

Provides guidance rather than instruction

Explains the implications of readers’ decisions on health outcomes rather than telling readers what they should do without discussing why

Reminders to discuss issues with providers when making health-related decisions

Potential to facilitate discussion with providers who may tend to dismiss patients who want to discuss something they have seen in media reports

Assistance in weighing conflicting recommendations and reports

Links to original media and research articles

Little time lag between original article publications and XRAYS reviews

Frank discussion of risks and limitations, which online support groups are unlikely to provide

The participants who were health care providers recommended outreach education for health care providers, who could then inform their patients about XRAYS and use it to facilitate shared decision-making.

What types of articles would you like to see reviewed in XRAYS?

Participants’ recommendations for future XRAYS reviews are summarised in Table 5.

Table 5.

Focus group recommendations for future XRAYS reviews.

Continue to review articles on research with questionable methodology or conclusions. It is important to know about misinformation.

Continue to review poor media reporting and call attention to issues such as sensationalism or overstated conclusions

Review articles with findings relevant to groups at highest risk for breast cancer, recurrence and mortality

Review research relevant to African American women’s needs and risks

Review responses to celebrity statements about cancer and breast cancer, which often include misinformation, or a lack of information about resources and treatment options

Review any reporting with information about the latest treatment options

Review reports of information about personal behavioural choices and risks

Review older articles that may include the most recently available information relevant to health decisions (possibly include these as references in reviews of more recent findings that build on them)

Review articles relevant to men and include outreach messages that are more appealing to men

Several participants noted that XRAYS would be useful to friends and family members of survivors and previvors. Friends and family often share or discuss media articles, which can be tiring for patients who are overwhelmed or frustrated with responding to recommendations that contradict decisions they have made with health care providers. Participants thought that XRAYS could facilitate discussions that decrease this frustration by providing clear discussions of evidence and implications. One woman said, ‘I would use XRAYS to get my mom to stop pushing things she saw on Dr. Oz or some fake trying to make money on the Internet. I could say, “Mom, I looked it up. I don’t need that”’.

Some participants suggested adding a toolkit for critically evaluating articles about research findings to the XRAYS web page. FORCE may consider adding links to existing resources, such as Health News Review (www.healthnewsreview.org/toolkit) or developing a toolkit tailored for XRAYS audiences. Some participants also requested information about determining which sources are more and less credible. Others said that it would be useful to include a section on unproven or risky treatments.

Several participants said that XRAYS articles often contain challenging technical language and that they would like XRAYS to provide a glossary or text defining these words in the summary or hyperlink.

What kinds of health information are you most likely to look for in the media?

Participants reported that they searched for the most recent information about the following:

Prevention.

Screening.

Genetic testing.

Risk and protective factors.

Research relevant to health decisions at all stages of prevention, treatment, recovery and survivorship.

Information about children’s risk, treatment and decisions.

One participant said,

If there’s a new treatment that works, or research that shows there’s something I can do, I want to know. If they don’t know if it works, or there’s nothing I can do about it yet, I don’t need to see it.

Another said, ‘This is genetic. My kids could get it. I want to know everything I can about how to help them’.

How do you use health information you learn about from the media?

Most participants said they used the health information they acquired through the media to make personal decisions about preventive self-care (e.g. nutrition and physical activity). Several participants said they discussed information they had learned through media with their health care providers. They reported that media reports often facilitated discussions and shared decision-making with care providers. Some said that XRAYS would likely facilitate these discussions. One participant said, ‘I brought articles I read to my doctor a couple of times. He rolled his eyes and got impatient. I think if I brought this (XRAYS) and asked these questions he probably wouldn’t ignore me’.

All participants noted that the period immediately after a positive test for genetic risk or cancer diagnosis is a peak time for information seeking.

Discussion

All participants reported that XRAYS was a useful resource that addresses an important need for information among young women who have survived or are at genetic risk for breast cancer.

Recommendations for organisations that provide a resource for understanding recent research and evaluating media reporting included the importance of the following factors:

Offering clear, brief non-technical summaries of research findings, limitations and implications for decisions related to health and healthcare.

Offering suggested topics of discussion with healthcare providers.

Offering clear, brief critiques of any issues with media reporting, such as sensationalism or failure to report study limitations.

Identifying and reviewing media reports on topics of interest identified in focus groups.

Incorporating a glossary or other feature to facilitate the comprehension of technical vocabulary.

Including a tool that guides users in evaluating research reporting and conclusions.

Developing a dissemination strategy that includes outreach to and coordination with health care providers.

Considering disseminating information about the resource through targeted alerts.

Developing a communications plan that emphasises the value of the resource in facilitating shared decision-making with health care providers.

Conclusion

Results indicate that XRAYS is achieving one of its primary goals: namely, facilitating users’ learning from media reports on research related to breast cancer. In this small pilot sample, XRAYS users’ increase in correct answers about research covered by media articles was approximately twice that of study participants who read only the media article. While there is research indicating that media reports on science and health sometimes misrepresent information (Haneef et al., 2015) and there are some resources to help readers evaluate media news content (Gerrits et al., 2018; Rosentiel, 2013), a literature search did not yield information indicating that the efficacy of these tools has been tested. These initial results suggest that a tool developed to respond to audience needs, offering ratings of evidence quality, relevance and reporting quality can help readers to understand research reporting and its limitations.

Limitations

This study included a small sample of women. FORCE plans to replicate the findings with a larger sample with more power to assess the degree of improvement in the control group and to confirm the finding that knowledge increases significantly more when readers have access to XRAYS in addition to a media report.

In addition, participants were attendees at a FORCE conference and most had heard of XRAYS. It is likely that this sample of information seekers has a higher baseline level of knowledge and more experience reading about research than a broader sample would. These factors might be expected to lessen the effects of XRAYS for this sample. However, the group also may be biased in its assessment of FORCE’s credibility, willingness to critically evaluate media reports or otherwise differ from other young breast cancer survivors and previvors in ways that affect results. It would be valuable to assess the effects of XRAYS in a broader sample.

This study did not collect socioeconomic status data. Future pre-post-test comparisons should include the collection of these data, since it is important to reach diverse populations in health communications efforts targeting informed prevention and treatment decisions.

Footnotes

Acknowledgements

The authors thank all of the women who participated in the study for their time and insight.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This project was supported by the US Centers for Disease Control and Prevention (Cooperative agreement no. DP005404). Its contents are solely the responsibility of the authors and do not necessarily represent the views of the Centers for Disease Control and Prevention or the US Department of Health and Human Services.

ORCID iD

Robin H Pugh Yi

References

Anders

Johnson

Litton

et al . (2009) Breast cancer before age 40 years. Seminars in Oncology 36: 237–247.

Assi

Khoury

Dbouk

et al . (2013) Epidemiology and prognosis of breast cancer in young women. Journal of Thoracic Disease 5(Suppl. 1): S2–S8.

Atkin

Smith

McFeters

et al . (2008) A comprehensive analysis of breast cancer news coverage in leading media outlets focusing on environmental risks and prevention. Journal of Health Communication 13(1): 3–19.

Dobransky

Hargittai

(2012) Inquiring minds acquiring wellness: Uses of offline sources for health information. Health Communication 27: 331–343.

Fergie

Hunt

Hilton

(2013) What young people want from health-related online resources: A focus group study. Journal of Youth Studies 16(5): 579–596.

Finch

Valentini

Greenblatt

et al . (2013) Frequency of premature menopause in women who carry a BRCA1 or BRCA2 mutation. Fertility and Sterility 99: 1724–1728.

Freedman

(2013) Survival of the wrongest. Columbia Journalism Review, January 2. Available at: https://archives.cjr.org/cover_story/survival_of_the_wrongest.php

Freedman

Partridge

(2013) Management of breast cancer in very young women. Breast 22: S176–S179.

Gerrits

Kringos

Van Den Berg

et al . (2018) Improving interpretation of publically reported statistics on health and healthcare: The Figure Interpretation Assessment Tool (FIAT-Health). Health Research Policy and Systems 16(1): 20.

10.

Haneef

Lazarus

Ravaud

et al . (2015) Interpretation of results of studies evaluating an intervention highlighted in Google health news: A cross-sectional study of news. PLoS ONE 10(10): e0140889.

11.

Laugesen

Hassanein

Yuan

(2015) The impact of internet health information on patient compliance: A research model and an empirical study. Journal of Medical Internet Research 17(6): e143.

12.

National Cancer Institute (2014) High-penetrance breast and/or ovarian cancer susceptibility genes. Available at: www.cancer.gov/cancertopics/pdq/genetics/breast-and-ovarian/healthprofessional/page2#top

13.

Pugh-Yi

Rezende

Huynh

et al . (2018) XRAYS (eXamining Relevance of Articles to Young Survivors) Program Survey of information needs and media use by young breast cancer survivors and young women at high-risk for breast cancer. Health Communication 33(12): 1525–1530.

14.

Rosentiel

(2013) Six Questions that Will Tell you what Media to Trust. American Press Institute. Available at: www.americanpressinstitute.org/publications/six-critical-questions-can-use-evaluate-media-content/

15.

Rowlands

Loxton

Dobson

et al . (2015) Seeking health information online: Association with young Australian women’s physical, mental, and reproductive health. Journal of Medical Internet Research 17(5): e120.