Towards a shared understanding of the learning health system in a large academic-based health system: A qualitative analysis

Abstract

Healthcare delivery is currently undergoing major structural reform, and the Learning Health System (LHS) has been proposed as an aspirational model to guide healthcare transformation. As efforts to build LHS take considerable investment from health systems, it is critical to understand their leaders’ perspectives on the rationale for pursuing an LHS and the potential benefits for doing so. This paper describes the qualitative analysis of semi-structured interviews (n = 17) with health system leaders about their general perceptions of the LHS, description of key attributes and potential benefits, and perception of barriers to and facilitators for advancing the model. Participants universally endorsed the goal of the local health system aspiring to become an LHS. Participants identified many recognized attributes of LHS, though they emphasized unique attributes and potential benefits. There was also heterogeneity in participants’ views on what to prioritize, how to structure the local LHS within existing initiatives, and how new initiatives should be implemented. Improving conceptual clarity of attributes of the LHS would improve its potential in guiding future reform.

Keywords

learning health system health services research qualitative analysis

Background

As health systems in the United States (US) face pressure to improve patient care experiences, improve population health, and reduce the per capita cost of health care, learning health systems (LHS) have become an aspirational model.¹ The Institute of Medicine (IOM) defines LHS as, “a system in which, science, informatics, incentives, and culture are aligned for continuous improvement and innovation, with best practices seamlessly embedded in the delivery process and new knowledge captured as an integral by-product of the delivery experience.”² Similarly, the Agency for Healthcare Research and Quality (AHRQ) highlights numerous characteristics of an LHS, including “having leaders who are committed to a culture of continuous learning and improvement,” “capturing and analyzing data and care experiences to improve care,” and “continually assessing outcomes refining processes and training to create a feedback cycle for learning and improvement.”³ Thus, a true LHS is transformational, with research embedded into the practice of medicine and a culture that demands continuous improvements in quality and experience.

Over the past decade, use of and interest in the term LHS has grown, with an increasing number of health systems working to implement an LHS model. Multi-center and regional systems, and even patient-registries, have also been referred to as an LHS.^4–8 While standard definitions of LHS exist, studies suggest that interpretation and implementation vary.⁹ Similarly, metrics for successful implementation of LHS vary across settings, and thus, little is known about what constitutes successful LHS implementation and sustainment.¹⁰ Developing an LHS requires considerable investment from health systems, so understanding views on the value proposition from the leadership perspective is essential.

The purpose of this study was to assess how the somewhat abstract concepts of an LHS outlined by IOM and AHRQ were more specifically understood and operationalized by leaders at one academic medical center (AMC), a large hospital system affiliated with a medical school. This project was part of a larger initiative at the University of Alabama at Birmingham aimed at advancing the LHS model and served as a needs assessment for the initiative. Our research questions were: (1) What are the distinguishing features of an LHS generally? and (2) What are perceived benefits and challenges related to adopting the LHS model locally?

Methods

Interview guide development

The interview guide was developed based on a review of the literature, alignment with the research questions and consensus of the research team. Interview questions were revised at several points throughout the course of the study to improve flow and add probes to ensure the research questions were adequately addressed. See Figure 1 for an overview of the questions in the interview guide.

Figure 1.

Overview of interview guide questions.

Participants and Recruitment

The goal was to interview a diverse sample of academic medical center leaders, including leaders of clinical service lines, academic departments, and high-level health system administrators (ex: vice presidents and other senior leaders supervising units within the academic medical center). We used a combination of sampling strategies for recruitment. The team internally generated an initial list of potential interview participants who were then invited by email to participate in the study. The initial list of interviewees was selected purposively with the goal of interviewing leaders from health system administration, major clinical departments, and researchers who had been involved in work involving the health system.¹¹ We then further sought out individuals who had been involved in early stages of LHS planning through informal conversations external to our team with numerous local leaders, including some of the interviewees (thus representing a form of snowball sampling).¹² In some cases, interview participants made suggestions during or after the interview, unprompted. This second step was added to enhance the sample by identifying stakeholders that may not have been known to the study team and increasing the number of diverse perspectives included in the analyses.

Procedures

Researchers (KR, RD, TH) trained in qualitative methods conducted all interviews. Initially the interviews were conducted in-person, but the last three were conducted via Zoom due to COVID-19. Interviews were audio-recorded and then transcribed verbatim. Interviews were read and discussed by the research team as they were conducted.

Analysis

Data were analyzed using a hybrid deductive-inductive template approach.¹³ To start, three members of the research team (AV, KR, TH) read through the first several interviews and collectively developed an initial set of themes from each of the interview guide domains (barriers, benefits, strategies, distinguishing characteristics) and the AHRQ definition of LHS.³ The codebook was then applied to all interviews, while simultaneously employing an inductive approach to capture emergent themes. As potential new, emergent codes were identified, they were noted, reviewed by the team, and added if all three team members approved. All interviews were double-coded (AV, KR). The codebook and analytic plan were reviewed monthly. Codes, themes and disagreements were discussed and adjudicated with a third member of the study team (TH).

Results

We conducted semi-structured interviews with 17 health system leaders. Characteristics of the sample are displayed in Table 1. All but one of the participants were familiar with the term LHS, but some stated they knew very little about it. Among those who were familiar with the term, there was a great deal of heterogeneity in perceptions of the distinguishing features, priorities, and benefits of an LHS. The following are the major themes that emerged in the interviews. Exemplar quotations are displayed in Table 2.

Table 1.

Characteristics of interview participants.

	Total interviewees N = 17 (%)
Gender female	47.1
Under-represented minority	17.6
Physician	58.8
Academic leadership position	17.6
Health system leadership position	94.1

Table 2.

Exemplar quotations.

Theme/Code	Quotation
Distinguishing Features
Seamlessly capture process and outcomes data in real-time for the purpose of improving care delivery and patient experience	Ideally that means a health system and a whole enterprise is not fixed in its way of doing things, but is nimble enough to constantly be looking across the whole spectrum of care delivery, and assessing what is going well. What is not going so well?
Generation and dissemination of high-quality research evidence	It’s the dissemination of knowledge, so important. If you did something and you did it well, but you didn’t share it with me, I’m being potentially robbed of something that could help my patients
Availability of high-quality data and information technology systems	You’ve got to have a more robust informatics and analytics infrastructure and the ability to access broad swaths of data
	If you’re going to be a cardiologist, you know there’s inherent value to know somebody’s heart rhythm. If you’re a health system, it’s inherently valuable to know data
Strong support and engagement from leadership	I can tell you that there has to be that investment from the leadership standpoint in order for it to happen. I am currently living, trying to build those bridges without having that support and it is not easy. And, for someone who is not as passionate about it, it would be, it would create an obstacle that most people would not to try to get over
Stakeholder and patient driven	We are still very provider-centric. We create the care models that are based on what we think are right versus actually engaging patients and family to what you want for your care delivery. The path of learning health system, and a path of true value-based care is actually meeting the expectations of the patients and families
Dissemination and implementation research	I could do quality improvement and never sort of commit myself to studying what we did and what were the outcomes of the implementation. And then, studying, researching how other folks have implemented successfully and then, writing and publishing about the actual implementation process
Collaborative and not siloed	It’s not that one’s driving the other or one’s informing the other. It’s that it’s an integrated team that’s kind of in that constant learning, updating, improvement cycle… that constant cycle of learning, and improving, and finding new opportunities rather than it being somewhat siloed or informative to one, reactive to another
Benefits to becoming an LHS
Would strengthen the health system and research portfolio	I think, over the long haul, the research embedded in the system could generate really, impressive knowledge that then informs these operational people. All of our good care comes from effective research. There may be some lucky patients who scape by. But, the ones that are successfully treated are dependent on research that has been done sometime prior to them
Data to improve care	It enables you to make more objective decisions on how to better deliver that care. You end up with a higher quality of outcomes. At the end of the day, whether it’s left fewer errors making decisions where we’re choosing the right therapies for patients, not over-treating and not under-treating, and you are equitable in your gender and ethnicity in terms of how you’re delivering care it drives out the variability that I think ultimately is inherent in any kind of complex system
Shared resources and expertise can decrease costs	I do think that this is an opportunity to, again, bring the silos together and to look at, “Okay, what are you doing in this realm?”, and then, build that plan with intent based on that. I think we have so many opportunities; we spend so much money a lot of times on consultants, when we have the resources on this campus to do it ourselves
Synergistic with existing health system initiatives	Being an active medical center sets us up nicely to be able to formalize this process a lot faster and a lot easier. It’s culturally acceptable and known that people are doing research, education, we’re training, and things like that. I think this concept is not going to be anything like, earth-shattering new to anybody
Barriers to becoming an LHS
Siloes	There’s nothing that I think of or that I learn from one constituency that probably isn’t already in a startup phase in some other aspect of our organization
LHS as a buzzword for marketing	IT companies… seem to use it as a way to promote their software and their unique analytics and visionary sort of approaches… but I think it might not be as well-developed. It’s just, “hey, by the way, you could have a learning health system if you buy and use our stuff
Research must be easily integrated into clinical workflows	Part of it requires a different way of thinking for researchers. And this is a barrier I’ve encountered. It requires a special kind of researcher, as well, to realize this is going to look different. And yet, what I need the researcher to help me do is have it not be shoddy work but have it be rigorous
Cost and funding	I think funding will be a barrier. We can all think it’s really cool and exciting but then we’ll get down to who’s going to pay for it… That would be something but I don’t think that’s anything new. We have these challenges today. But with more excitement may come more opportunities, therefore more funding questions
Strategies to becoming an LHS
Investments from leadership	There’s such a drive to get the grant funding and to be productive with your research and it’s hard to integrate a lot of clinical activities into that. And same for clinical activities. Your productivity, you’ve got to see so many new patients, you’ve got to see so many returning patients, and then you might have a little bit of administrative time to catch up. And to try to drive clinicians to have exposure to research is very challenging. I think the health system would have to invest somehow and provide some sort of incentive
Investments from leadership	That goes back to the top leadership: you cannot have, when they go out of the room, one person being viewed as not committed to this in any form, because it’ll fail then, if the top leaders are not supportive of it
Leverage existing initiatives	It’s really about taking the assets that we have and using them to our own internal advantage

Distinguishing features of a learning health system

The most frequently discussed distinguishing feature was the use of data to either improve care or generate and disseminate research evidence. However, participants varied in their perceptions of the priority of these goals. Most agreed a key feature of an LHS should be to capture high quality process and outcomes data in real-time for the purpose of improving care delivery and patient experience. Some stated that improving the quality of care delivered was the primary purpose of an LHS and felt that to do this, a system should always be collecting and reviewing process and outcomes data for the purpose of identifying opportunities for improvement. For others, the generation and dissemination of high-quality research evidence was the most important priority in an LHS. Effective dissemination of evidence, both within the health system and externally, was frequently brought up as an important LHS activity. For example, one participated stated, “It’s the ability to adopt outside information and apply that within our organization.” Participants described the necessity of publishing or otherwise disseminating the results of projects to “collectively learn from each other”. However, some of the participants that felt care delivery to be the top priority stressed that LHS administrators must be mindful of how research is integrated into clinical care. They also stressed the importance of embedding research into clinical care without disrupting clinical workflows. For example, one interviewee said, “There would be a very limited tolerance [for disrupting clinical efficiency]. When we talk about embedding, there are finding ways to embed that don’t compromise flow.”

A prerequisite to using data for quality improvement and research is improving the access to this data through better information technology systems. Participants noted the current difficulty in accessing data from the health system and felt that moving forward these data need to be “user-friendly, timely, and capable of supporting operations.” Participants also noted that data from across the health system should be easily accessible, ideally from a single repository. Doing so may require a shift in emphasis on “protecting the medical record against violations” to the “accumulation of data to really make decisions and inform leaders of how care models and how business should be done”. However, some participant highlighted ethical considerations and barriers that must be addressed first (see barriers below).

Most participants described the LHS as a stakeholder-driven endeavor, with an often-overlooked stakeholder being patients. They asserted that the LHS should prioritize “what is meaningful and of value” to patients and their families and described the importance of making research questions related to care delivery focused on “what patients want.” Participants thought all LHS activities should be developed in collaboration, or coproduced, with patients and caregivers. For example, one participant reflected, “when we threw our family members out because we restricted visitors [during COVID], our outcomes got worse.” Others went a step further and discussed the importance of disseminating research evidence to patients. Several participants declared that as we learn about care delivery, learners should include, “not only the front-line staff but patients and families.” Some suggested the best way to achieve this is by ensuring patients and community members are present from beginning to end of LHS implementation planning.

Implementation science was an emergent feature in early interviews and was then added to the interview guide as a probe. Most, but not all, participants that were asked about implementation science were familiar with the term. Those who were familiar described it as a “key component” of an LHS. They expressed that everyone in an LHS should be studying the processes by which new innovations are implemented and lower value interventions are de-implemented. Many noted that they had been doing both implementation research and practice to some degree. Those who were not familiar with it, however, still felt that the ability to implement and spread evidence-based practices across health system units was a distinguishing feature of an LHS. Specifically, participants expressed the importance of having a system where best practices and research evidence could easily be disseminated across the health system. One participant stated that this type of system could significantly aid in “scalability and the ability to reduce barriers.”

When discussing how quality improvement fits in the LHS model, participants noted the importance of a culture that supports a continuous cycle of learning and improvement. This means you do not just conduct one project and stop. An organization must be “committed to learning about itself” continuously. The processes and activities in the continuous feedback loop of learning and improvement should be well defined. Many participants referenced the commonly employed, “Plan, Do, Study, Act” approach as a potential framework for quality improvement in the LHS.

To build the necessary culture of continuous learning and implementation, an endorsement and prioritization from “the top” is critically important. Committed leadership is particularly important when obstacles are faced, as one participant noted, “we’ve got champions that can help scale across those boundaries.” Other participants specifically described the importance financial commitments from health system leadership. They also stressed the importance of evidence-based leadership approaches, while others described how current leadership has already committed to a culture of collaboration and learning. Others pointed out that leadership must be easily accessible.

Participants described the LHS as an organized “system…not a project” and that learning happens at the level of the organization. For example, one leader stated that “the whole concept of a learning organization is that you’re learning as an organization, not as just a singular unit or an individual.” This requires partnerships between the various missions (education, research, patient care) of the health system.

Benefits to adopting an LHS model

Most participants agreed that pursuing an LHS would be a valuable endeavor. One frequently described benefit was access to data for the purposes of improving care delivery. Most participants were excited about the opportunities to improve patient care while generating new knowledge that would be made possible with improved ability to access real-time data. They also noted that this would make research and quality improvement projects easier to initiate and quicker to implement.

Participants stated that being an LHS would strengthen the academic medical center. They described how the culture necessary to be considered an LHS would result in increased employee satisfaction. They also noted that it would open opportunities for both clinicians and researchers to engage in research and quality improvement. In turn, this would improve the quality and breadth of the institution’s research portfolio. This also increases access to research and clinical trials for patients that get their care in the health system.

Participants noted the potential of an LHS to improve costs for both patients and the health system over time. Some participants noted that improved collaboration would promote awareness of and access to potential shared resources across the medical center. Other participants highlighted the potential for more research on cost-effectiveness. For example, one participant noted that focusing on identifying and testing the best interventions would lead to “identifying the most cost-effective interventions.” In an LHS, cost-effective interventions can be implemented and evaluated quicker. Indeed, another participant added that an important activity in an LHS is, “always tracking the money to see if we are saving the health system dollars and then use that to reinvest in other areas.” When the collection and analysis of cost data are seamlessly embedded into the health system’s culture and clinical workflows, resources can easily be assessed and reallocated where needed.

Last, participants highlighted that the LHS is synergistic with existing health systems initiatives. They expressed a belief that transitioning to an LHS does not require a major shift from the direction of prior reform efforts (which already included efforts to improve quality, efficiency, etc). In fact, many noted that the LHS would provide a platform to further advance initiatives in accountable care, quality improvement, population health, and value-based care.

Barriers to becoming a learning health system

When asked about challenges to becoming an LHS, the majority of participants spoke about issues associated with being in a large complex health care system: siloes, unaligned priorities, and cost. Most interviewees mentioned a need to remove “siloes”. These siloes were often described as unique groups from different hospital service lines or as individuals representing different parts of the health systems’ mission (like research, clinical care, and education).

Participants asserted that it is not just people and departments that are siloed. While the organization has “a lot of data surrounding us”, these data are also siloed. With multiple databases and different requirements for access, clinicians and researchers are not able to obtain data easily. They viewed this as a major hurdle given the importance of integrated, easily accessible data as noted above. Moreover, participants brought up that for-profit companies were using the LHS concept to market their software. Some expressed concern that this was more for marketing than substance.

Participants representing clinical departments raised concerns of how research could potentially interfere with clinical workflows. To truly embed research across the health system, researchers must understand and adapt their approach to the complexity of the clinical environment. The same participants also noted that lack of research training among many front-line clinicians could be a potential barrier to embedding research into clinical care. Those who highlighted the importance of prioritizing research in an LHS noted that this tension between research and clinical care could be because the idea of the LHS has not “permeated a lot of the clinical side of the organization.”

Cost was also noted as a barrier. Health systems must prioritize maximizing revenue and expanding the reach by attracting more patients. Indeed, current funding structures for LHS activities were frequently noted as inadequate, making it difficult to support the necessary embedded researchers. Participants raised the question “who is going to pay for it?”

Strategies to becoming a learning health system

Participants identified necessary steps to becoming an LHS. While leaders described funding of LHS activities as a barrier, they also saw health system investment as a necessary strategy to align researchers and clinicians. This included funding for embedded researchers, improved data and information technology infrastructures, and patient collaborators.

Leaders noted several existing programs within the health system that could be leveraged to advance the local LHS. For example, leaders noted an existing data aggregation tool, i2b2 (Informatics for Integrating Biology and the Bedside), as a program that sets the local LHS up to participate in some of the research and data analytics noted above.¹⁴ They also identified existing centers within the institution, such as one focused on outcomes research, that could support the necessary platform for collaboration and LHS infrastructure like data analysts and methodological consultants. In addition, they named several current initiatives, including the development of a clinical integrated network and the Medicare Shared Savings programs, as ideal incentives to spur investment in the LHS model.

Discussion

This study explored the perceptions of academic medical center leaders regarding the learning health system. We found that these leaders were familiar with the term, but perspectives varied on its distinguishing features, priorities, and strategies for implementation. Figure 2 summarizes the results of the interviews.

Figure 2.

Overarching themes from health system leadership on what LHS should be, what it is not, and potential strategies.

Participants identified two differing main priorities of the learning health system: enhanced care delivery and research. While the literature on LHS describes the importance of research embedded into clinical care, participants often described them as two separate and even sometimes competing priorities. Many noted barriers to integrating these two priorities. Thus, building capacity for embedded research is likely a next step in advancing the LHS in this health system. This will require up-front investment to align priorities, train embedded LHS researchers, and ensure early meaningful engagement with key stakeholders, including patients.^15–18

Leaders from this academic health system identified distinguishing features like those identified in other studies.^19–21 They identified core components of LHS, like a culture of continuous learning, collaboration, strong leadership, and a focus on using data to improve care delivery. However, participants’ overall understanding of the term and rationale for adopting the LHS model locally varied considerably. Some seemed to conceptualize a more limited model, such as individual departments using data to improve care delivery, while others conceptualized a more comprehensive transformation, such as a system-wide collaborative platform where best practices and research evidence are disseminated, implemented, and evaluated. Thus, a major barrier to advancing this LHS is a lack of shared definitions. To move forward, leadership will need to collaborate to develop a shared vision and key processes involved. Understanding how leaders conceptualize the term can help align perspectives, values and the overall mission; a necessary step that must be taken before other actors can be recruited to participate in in the system.^8,22 For example, the Kaiser Permanente Washington LHS Logic Model, maps out all the inputs, outputs and outcomes of an LHS.¹⁰ Collaborating to build such a plan, could help align definitions and create a shared vision amongst stakeholders that are key to adoption and implementation of the model.

Competing priorities could be a challenge to developing an LHS. To mature the LHS, leadership will need to develop processes to align multiple stakeholder preferences. For example, coproduction refers to a collaborative model of research that involves co-creation of research and care plans with patients, the public, payers, clinicians, service providers, and policy makers. Coproduction of the LHS vision, priorities and research projects with patients, clinicians, and health system leadership will be critical for a functioning LHS.^8,23–28 Evidence-based strategies to promote stakeholder engagement will need to be employed along with strategies to embed research into clinical workflows.²⁹

Implementation research and dissemination of evidence are necessary for an LHS.³⁰ Participants in the LHS want to know about ongoing projects so they can learn from the implementation successes of others and apply that knowledge more broadly. This will require several changes to the current system. The health system will need a platform to support collaboration and remove siloes between units, schools, and professions.³¹ Embedded researchers should have training in dissemination and implementation science and work closely with clinicians to design initiatives. Moreover, access to data from across the medical center is needed to assess outcomes for system-level implementation initiatives.

To support continuous learning, improvement, and implementation research, access to high quality, timely process and outcomes data are necessary. Participants frequently noted better access to quality data as a key strategy to advance the local LHS. While capabilities are improving, current reports suggest electronic health record data are challenging to access and requires substantial structuring to be useable in an LHS.^32,33 This too will require investment from health system leadership to ensure that data are appropriately prepared to serve the LHS objectives.

This study has some limitations. First, the study was conducted at a single AMC and recruitment was non-random. While there are common components of LHS, some aspects are likely specific to local context. Thus, perceptions may differ from other organizations aiming to implement an LHS and these results may not be applicable to all health systems. Second, we specifically asked participants about numerous topics including population health, value-based care, quality improvement, health services research, and implementation science, which might have prompted responses that otherwise would not have emerged from a more open-ended interview format.

Conclusion

Within a single academic medical center, there is strong endorsement for becoming an LHS, although the shared conception of what to prioritize and how to implement are still evolving. Participants highlighted important institutional features that could be leveraged to advance the LHS at the University of Alabama at Birmingham, like existing initiatives focused on health services research, quality improvement, and informatics. They also identified certain challenges that need to be overcome to become an LHS, like siloes across departments. Individuals responsible for advancing the local LHS will need to develop tailored strategies to remove these siloes and facilitate collaboration, specifically between researchers and clinicians. Investments in embedded researchers is also necessary. Last, improving conceptual clarity of distinguishing features and priorities of the LHS would improve its potential in guiding future reform.

Footnotes

Declaration of conflicting interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: AV has a financial relationship (employment) with Heart Rhythm Clinical Research Solutions, LLC and 3PH Alliance, Inc. All other authors have no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the UAB Heersink School of Medicine (Internal award from the Division of Preventive Med). Dr. Riggs was supported by a career development award from the National Institute of Arthritis and Musculoskeletal and Skin Diseases (K23 AR080224).

ORCID iD

Allyson L Varley

References

Etheredge

. A rapid-learning health system. Health Aff 2007; 26: w107–118. DOI: 10.1377/hlthaff.26.2.w107.

Institute of Medicine (IOM) . The learning healthcare system: workshop summary. Washington, DC: National Academies Press, 2007.

Agency for Healthcare Research and Quality . About learning health systems. Rockville, MA: Agency for Healthcare Research and Quality, 2019. https://www.ahrq.gov/learning-health-systems/about.html (accessed 13 January 2020).

Dixon-Woods

Campbell

Chang

, et al. A qualitative study of design stakeholders’ views of developing and implementing a registry-based learning health system. Implement Sci 2020; 15: 16. DOI: 10.1186/s13012-020-0976-1.

Seid

Hartley

Dellal

, et al. Organizing for collaboration: an actor-oriented architecture in ImproveCareNow. Learn Health Syst 2020; 4: e10205. DOI: 10.1002/lrh2.10205.

Aalsma

Aarons

Adams

, et al. Alliances to disseminate addiction prevention and treatment (ADAPT): a statewide learning health system to reduce substance use among justice-involved youth in rural communities. J Subst Abuse Treat 2021; 128: 108368. DOI: 10.1016/j.jsat.2021.108368.

Rubinstein

Warner

. CancerLinQ: origins, implementation, and future directions. JCO Clin Cancer Inform 2018; 2: 1–7. DOI: 10.1200/cci.17.00060.

Anderson

Mugavero

Ivankova

, et al. Adapting an interdisciplinary learning health system framework for academic health centers: a scoping review. Acad Med 2022: 97: 1564–1572. DOI: 10.1097/acm.0000000000004712.

Wouters

RHP

van der Graaf

Voest

, et al. Learning health care systems: highly needed but challenging. Learn Health Syst 2020; 4: e10211. DOI: 10.1002/lrh2.10211.

10.

Allen

Coleman

Mettert

, et al. A roadmap to operationalize and evaluate impact in a learning health system. Learn Health Syst 2021; 5: e10258. DOI: 10.1002/lrh2.10258.

11.

Palinkas

Horwitz

Green

, et al. Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Adm Policy Ment Health 2015; 42: 533–544. DOI: 10.1007/s10488-013-0528-y.

12.

Goodman

. Snowball sampling. Ann Math Stat 1961; 32: 148–170. DOI: 10.1214/aoms/1177705148.

13.

Saldana

. The coding manual for qualitative researchers. Thousand Oaks, CA: Sage Publications, 2021.

14.

Murphy

Weber

Mendis

, et al. Serving the enterprise and beyond with informatics for integrating biology and the bedside (i2b2). J Am Med Inf Assoc 2010; 17: 124–130. DOI: 10.1136/jamia.2009.000893.

15.

Isaacson

Simpson

. Governance and funding best practices for embedded research programs. Healthc 2021; 8 Suppl 1: 100433. DOI: 10.1016/j.hjdsi.2020.100433.

16.

Yano

Resnick

Gluck

, et al. Accelerating learning healthcare system development through embedded research: career trajectories, training needs, and strategies for managing and supporting embedded researchers. Healthc (Amst) 2021; 8 Suppl 1: 100479. DOI: 10.1016/j.hjdsi.2020.100479.

17.

Kraft

Caplan

Trowbridge

, et al. Building the learning health system: describing an organizational infrastructure to support continuous learning. Learn Health Syst 2017; 1: e10034. DOI: 10.1002/lrh2.10034.

18.

Gierisch

Hughes

Williams

Jr , et al. Qualitative exploration of engaging patients as advisors in a program of evidence synthesis: cobuilding the science to enhance impact. Med Care 2019; 57 Suppl 10 Suppl 3: S246–S252.

19.

Morain

Kass

Grossmann

. What allows a health care system to become a learning health care system: results from interviews with health system leaders. Learn Health Syst 2017; 1: e10015. DOI: 10.1002/lrh2.10015.

20.

Psek

Davis

Gerrity

, et al. Leadership perspectives on operationalizing the learning health care system in an integrated delivery system. EGEMS (Wash DC) 2016; 4: 1233. DOI: 10.13063/2327-9214.1233.

21.

Enticott

Braaf

Johnson

, et al. Leaders’ perspectives on learning health systems: a qualitative study. BMC Health Serv Res 2020; 20: 1087. DOI: 10.1186/s12913-020-05924-w.

22.

Fjeldstad

Johnson

Margolis

, et al. Networked health care: rethinking value creation in learning health care systems. Learn Health Syst 2020; 4. e10212. DOI: 10.1002/lrh2.10212.

23.

Elwyn

Nelson

Hager

, et al. Coproduction: when users define quality. BMJ Qual Saf 2020; 29: 711–716. DOI: 10.1136/bmjqs-2019-009830.

24.

Nelson

Dixon-Woods

Batalden

, et al. Patient focused registries can improve health, care, and science. BMJ 2016; 354: i3319. DOI: 10.1136/bmj.i3319.

25.

Damschroder

Knighton

Griese

, et al. Recommendations for strengthening the role of embedded researchers to accelerate implementation in health systems: findings from a state-of-the-art (SOTA) conference workgroup. Healthc (Amst) 2021; 8 Suppl 1: 100455. DOI: 10.1016/j.hjdsi.2020.100455.

26.

Kamal

Kirkland

Meier

, et al. A person-centered, registry-based learning health system for palliative care: a path to coproducing better outcomes, experience, value, and science. J Palliat Med 2018; 21: S61–S67. DOI: 10.1089/jpm.2017.0354.

27.

Batalden

Margolis

, et al. Coproduction of healthcare service. BMJ Qual Saf 2016; 25: 509–517. DOI: 10.1136/bmjqs-2015-004315.

28.

Redman

Greenhalgh

Adedokun

Co-production of Knowledge Collection Steering Committee , et al.. Co-production of knowledge: the future. BMJ 2021: 372: n434. DOI: 10.1136/bmj.n434.

29.

Keck

Hartley

Havens

, et al. Getting what is needed, when it’s needed: sharing information, knowledge, and know-how in a Collaborative Learning Health System. Learn Health Syst 2021; 5: e10268. DOI: 10.1002/lrh2.10268.

30.

Trinkley

Glasgow

, et al. How dissemination and implementation science can contribute to the advancement of learning health systems. Acad Med 2022; 97: 1447–1458. DOI: 10.1097/ACM.0000000000004801.

31.

Osuji

Frantsve-Hawley

Jolles

Embedded Research Conference Priorities and Methods Workgroup , et al.. Methods to identify and prioritize research projects and perform embedded research in learning healthcare systems. Healthc (Amst) 2020; 8: 100476. DOI: 10.1016/j.hjdsi.2020.100476.

32.

Bertagnolli

Anderson

Norsworthy

, et al. Status update on data required to build a learning health system. J Clin Oncol 2020; 38: 1602–1607. DOI: 10.1200/JCO.19.03094.

33.

Gould

Sharp

Nguyen

, et al. Embedded research in the learning healthcare system: ongoing challenges and recommendations for researchers, clinicians, and health system leaders. J Gen Intern Med 2020; 35: 3675–3680. DOI: 10.1007/s11606-020-05865-4.