Abstract
Background
The maturity of practices and infrastructure in the health care domain directly impacts the quality and efficiency of health care services. Therefore, various health care administrations (e.g. from hospital management to the nationwide health authority) need to assess and improve their operational maturity.
Objective
This study aimed to review and classify studies that propose/use maturity assessment or maturity models (MMs) as a vehicle to achieve operational excellence in the health care domain.
Method
To achieve this objective, we performed a multivocal literature review (MLR) – a form of systematic review that includes data from the grey literature (e.g. white papers and online documents) in addition to formal, peer-reviewed literature.
Results
Based on 101 sources, 80 from peer-reviewed literature and 21 from the grey literature, we identified 68 different MMs on, for example, telemedicine, care pathways and digital imaging. We reviewed them with respect to various aspects, including types of research and contribution, list of MMs proposed/used with their subject areas, elements of maturity/capability and application scope or scale. In the synthesis of empirical benefits of using MMs, two were found to be significant: (a) identifying issues and providing guidance for improvement in health care contexts, and (b) improving efficiency, effectiveness, performance and productivity.
Conclusion
This MLR provides an overview of the landscape and serves as an index to the vast body of knowledge in this area. Our review creates an opportunity to cope with the challenges in obtaining an overview of the state-of-the-art and practice, choosing the most suitable models or developing new models with further specialties.
Introduction
The maturity of practices, operations and infrastructure in the health care domain is of high importance. 1 However, these qualities often vary among health care provider units (e.g. hospitals) and in some cases are far from perfect. Establishing process thinking and achieving quality management is vital to assure service maturity and to improve service maturity continually in such a complex, dynamic and multidisciplinary domain.2,3
The notion of ‘maturity’ was first proposed by Phillip Crosby 4 and is defined as ‘the state of being complete, perfect, or ready’. 5 In a more general view, a maturity model (MM) is a conceptual framework that consists of a sequence of discrete maturity levels for a class of processes in one or more business domains, and represents an anticipated, desired or typical evolutionary path for these processes.6,7 MMs (also referred to as ‘capability frameworks’) have been widely used in many domains (such as industrial engineering, software engineering and information technology (IT)) for the purpose of process assessment and improvement.8–10
MMs have also been proposed in the health care domain with the purpose of assessing and improving the maturity of health care practices, operations and infrastructure.11,12 Some of these MMs focus on the specific sub-areas, such as telemedicine, care pathways, digital imaging, picture archiving and communication systems and facilities management. Given the large body of knowledge in this area, there is a need to review the scope and characteristics of available models together with their use and usefulness. health care practitioners, decision makers and researchers can then eliminate ‘reinventing the wheel’ to obtain the state-of-the-art and practice, choose the ‘right’ models that are most suitable for their needs or develop new models with further specialties. Therefore, this study is aimed at providing insight into and creating awareness on the MMs proposed as a vehicle to achieve operational excellence in the health care domain.
To achieve the above-mentioned objective, we performed a comprehensive literature review on MMs proposed and/or used in the health care domain. We used a multivocal literature review (MLR) 13 as the research method, which is a form of a systematic literature review (SLR) and includes grey literature (e.g. blog posts, white papers, presentations) in addition to the published (formal) academic literature. While a SLR is a common method used to conduct a literature review on published (peer-reviewed) literature on a given topic, the significance of including the grey literature in addition to the academic literature has been emphasised in the health care community.14–16
We used systematic guidelines13,17 to search for the sources and to categorise and synthesise them with respect to a number of review questions (RQs). We searched the academic literature using Google Scholar, PubMed and ScienceDirect, and we searched the grey literature using the regular Google search engine. In the remainder of this paper, we use the term ‘source’ to refer to a formally published paper/article or a resource from the grey literature. Through the RQs, we addressed the following properties of the sources: (a) types of research and contribution, (b) newly proposed versus existing MM used in the studies, (c) the aspects of maturity/capability addressed by the MMs (technology, business process, people, etc.), and (d) the application scope or scale of the MMs (e.g. single departments of a single hospital, multiple hospitals in a city, region or province). More specifically, the main contributions of this article can be summarised as providing:
An overview of the state-of-the-art and practice of 101 sources that propose or use MMs in the health care domain; A list of 68 MMs proposed or used in this domain, with details of selected examples; A classification scheme of the sources with respect to research type, contribution type, subject area, contained aspect, applied scope, scale of empirical evidence and reported benefits of the MMs; and A classification and synthesis of the sources with respect to the attributes mentioned above.
The remainder of this paper is structured as follows. Related work is presented in the following section. Next, we describe the research method and the review planning. The results of the literature review are then presented. Subsequently, we discuss our observations and suggestions, and potential threats to the validity of this MLR. Finally, we draw conclusions and make suggestions for future work.
Related work
Since this paper is an MLR on MMs in health care, the related works are the existing secondary studies on MMs in health care. A secondary study is a study of regular (primary) studies.
Using a literature search, we were able to identify 12 secondary studies on MMs in health care, as listed in Table 1. There are usually four types of secondary studies: systematic mapping (SM) studies, SLRs, MLRs and regular reviews (surveys). For each paper in Table 1, we also provide the title of the paper to show its scope, its type (SLR, SM, MLR or regular review) and the number of papers reviewed by the study.
Secondary studies on MMs in health care (ordered by year of publication).
MM: maturity model; PACS: picture archiving and communication systems; SLR: systematic literature review; SM: systematic mapping; MLR: multivocal literature review.
Our current MLR differs from the existing works in that it is the first comprehensive review on maturity assessment and MMs in health care, which covers both the formal and the grey literature and clusters subject areas and aspects studied so far. In terms of number of papers reviewed by a secondary study, it is seen from Table 1 that our MLR reviews also the highest number of papers (101 sources) compared to the existing secondary studies and derives a broad classification scheme.
Research method
We performed our study based on the guidelines for conducting SLR/MLR studies provided in the health care domain28,29 and also in other domains.13,17
The area of health care MMs is cross-disciplinary in nature, as it includes various stakeholders from health services and hospital management, to information systems (IS) and IT, each on the side of academia or practice. Furthermore, we found that some models in this area have been presented in the grey literature and have not been formally published. Conducting a typical SLR without including the grey literature would not have included many important and useful MMs that have emerged from practice and which are available in the grey literature. Therefore, we followed a natural approach by including the grey literature in the review and conducted an MLR. This decision is also in alignment with the literature considering the use of MLRs as a means to close ‘the gap between academic research and professional practice’. 13
Below, we discuss various aspects of our research method.
Goal and review questions
The goal of this study was to review and classify studies that propose/use maturity assessment or MMs in the health care domain. Based on this goal, we raised the following review questions (RQs):
• RQ 1 – Contribution and research types: ○ RQ 1.1 – Contribution types: How many sources have presented new MMs, methods/techniques, tools, metrics or processes in this area? ○ RQ 1.2 – Research types: What types of research methods were used by authors of the sources? Some sources only propose solutions without extensive validations, while some other sources present in-depth evaluation of their approach (e.g. rigorous empirical studies). • RQ 2 – Properties of the proposed MMs: ○ RQ 2.1 – MMs in this area: What models have been proposed in this area? What are their subject areas? ○ RQ 2.2 – Aspects of maturity/capability framework: What aspects (e.g. business process, organisational, technology and people) does each of the proposed MM address? ○ RQ 2.3 – Applicability scope/scale: What is the applicability scope/scale of the MMs? The models could be applied within the scope of, for example, single/multiple department(s) or single/multiple hospital(s). In this RQ, we consider the expected/possible applicability of the models, and not necessarily the scale of the actual empirical study presented in a given source. • RQ 3 – Properties of the empirical studies: ○ RQ 3.1 – Scale of empirical study: On what scales have the empirical studies been conducted, that is, what is the number of hospitals that the proposed MM has been applied in? Different from RQ 2.3, here we consider the ‘actual’ reported empirical results presented in a source. ○ RQ 3.2 – Reported benefits of MM applications: What benefits (quantitative or qualitative) have been reported as a result of applying the models? The application of such a model is sometimes claimed to be costly. Thus, enough benefits should be provided to ensure justifying the associated costs.
Multivocal review process
Figure 1 outlines the process that lies at the basis of this study, which consists of three phases: (a) search and selection of sources, (b) development of the classification scheme and data extraction and (c) systematic classification, synthesis and review. The process starts with the selection of sources in the academic and grey literature. Next, a classification scheme (map) is developed following a structured approach. The map is then used to conduct systematic classification, which is followed by the synthesis and reporting of the results.
An overview of the multivocal literature review process.
Search and selection of sources
To search for formally published (peer-reviewed) papers, we used Google Scholar, PubMed and ScienceDirect search engines. There have been numerous studies30–34 comparing the effectiveness, strengths and weaknesses of these academic search engines. We carefully reviewed these studies to be as objective and systematic as possible in our choice of search engines.
We also conducted a number of experimental search activities for papers in this area. We observed that the topic of MMs in health care is a cross-disciplinary subject and that many papers in this area (e.g. Sources 10, 17 and 24 in the pool of studies 35 ) are published in venues that address the domains outside of ‘core’ health care (e.g. management, IT and computer science), and thus are not covered (i.e. could not be found) in PubMed. Since Google Scholar has a much larger index (search space) than PubMed, first we used Google Scholar and then PubMed to populate the candidate pool of sources.
Through an iterative process, we developed the search string for this MLR study as: (maturity model OR maturity assessment) AND (health care OR healthcare OR health). We should note that the search process and source selection of this study was conducted in March 2018. Thus, sources published by that time had a chance to be included in our pool.
We performed the searches in our list of search engines using the above keywords. Using Google Scholar, we found 76 candidate formally published papers. Using PubMed, we found four additional candidate formally published papers which were not found through Google Scholar. They were Sources 13, 25, 80 and 90. Full details can be found in the study’s transparent online spreadsheet (https://goo.gl/Z4jxBd). Our search of ScienceDirect did not yield any additional papers.
To search for grey-literature sources, we used the Google search engine, which provides comprehensive access to the grey literature. 31 This popular web search engine (also the most widely used one according to eBizMBA, Inc. 36 ) has been used in many MLR studies in the past (e.g. in health 37 and management sciences 38 ).
As a result of the first search phase, we obtained an initial pool of 148 sources on which we applied the criteria for inclusion/exclusion and quality assessment.
Criteria for inclusion/exclusion and quality assessment of sources
We carefully defined the inclusion and exclusion criteria to ensure all relevant sources were included but not those which were out of scope. Our inclusion criteria were sources that proposed or used MMs in the health care domain; sources which were directly related to maturity assessment in the health care domain; and sources that were in English and with their full text accessible via our institutions’ subscriptions or freely available on the Internet. (Please note that one author of this article has a medical affiliation, giving us good enough to health care publication venues.)
In addition to the criteria above, we also defined and applied criteria for checking the quality of the sources. Quality assessment is specifically focused on determining the sources that report sufficient information to compare other studies in terms of answering defined review questions.
39
Therefore, we developed and used the following quality assessment criteria to ensure that sufficient information was reported in the sources:
Is there a clear statement that the study proposes/uses an MM for health care? Can it be inferred from the study that the MM is newly proposed, or it is only applied as already available? Can the organisational aspects that the MM involves (e.g. process, technology and people) be extracted from the study? Can the scope/scale in which the MM could be applied be extracted from the study?
All sources were examined in detail by the first three researchers in the voting stage with respect to the inclusion/exclusion and quality assessment criteria given above, and the sources which did not meet these criteria were excluded. Each of the first three researchers voted independently on to include or exclude each source. In the case of disagreements, discussions were conducted, and the fifth author who is a medical practitioner (pathologist) was consulted to reach a consensus.
As previously stated, the search process and source selection for this study was conducted in March 2018. Only a single paper from 2018 was found, 40 which was excluded in order not to provide a partial view for 2018. As a result, the time window for the pool sources was set to the yearly range of 1996–2017. After applying the inclusion/exclusion and quality criteria, 47 sources were excluded (see Figure 1), and our review pool was finalised with 101 sources.
Development of the classification scheme and data extraction
To conduct a structured and systematic classification of the sources, a classification scheme (systematic map) is needed.28,29 We developed our classification scheme through an iterative process of analysing the sources in the pool, identifying relevant list of attributes and refining them to derive the final map. Our goal was to categorise the sources to build a complete picture of the research area. Driven by our review questions, we identified the key attributes of the sources and refined them into a classification scheme using an iterative approach. Table 2 shows the classification scheme that we developed after applying the process described above.
Classification scheme (systematic map) developed and used in this study.
Taking our RQs and the classification scheme as the basis, we thoroughly reviewed the sources in our pool. In tagging each study based on the classification scheme, we incorporated as many explicit ‘traceability’ links between our mapping and the primary sources as possible. That is, we added comments to the cells of our mapping matrix by verbatim copy/paste of text from the source acting as the ‘traceability’ link. Such comments facilitated peer reviewing (that we conducted among the authors) and are also useful for other researchers reviewing the resulting paper pool and the Supplemental Material.
As in the initial voting stage, during the detailed analysis phase, three researchers extracted and analysed data from the portion of the sources and then peer reviewed each other’s extracted data. In the case of disagreements among the researchers, issues were resolved by holding discussions with all authors to ensure quality and validity of the data and their relevance in the health care domain.
Results
Our final pool included a total of 101 sources: 80 from the formal literature and 21 from the grey literature. For a complete list of these sources, please refer to Tarhan et al. 35 Below, we report a summary of the trends in the final pool of sources, followed by the results with respect to our RQs.
Figure 2 shows the cumulative number of sources per year by literature type (formally published vs. grey literature). We can see that the sources in both literature categories have been increasing steadily. While many scientists are publishing their work in this area, many practitioners/professionals are using online platforms such as blogs to post their work in this area as grey literature. We can see from Figure 2 that the topic has received active attention in recent years.
Number of sources per year.
Out of the 80 sources from the formal literature, 60 were journal papers, 13 were conference papers, four were theses, two were book chapters and one was a workshop paper. Out of the 21 sources from the grey literature, 13 were white papers, four were presentations, three were web pages and one was a blog post. Journal papers formed the majority (69%) of the pool, denoting a considerable level of maturity in the research field.
RQ 1: Contribution and research types
RQ 1.1: Contribution types
Figure 3 shows the annual cumulative breakdown of sources by contribution types. Under the figure, we also provide the references of the sources for each type. Please note that a source may contribute more than one contribution type. The top three contribution facets are new MMs, empirical (case) studies and methods/techniques, which have appeared in 68, 26 and 4 sources, respectively.
Sources by contribution type.
We can see that across different years, different contribution facets have been studied. The general observable trend is that new MMs are continuously being offered in this area. In addition, the trend with sources that have major contributions as empirical (case) studies is a promising sign of practice by the community.
In Figure 4, we present a bird’s-eye view of MMs by showing a word cloud of the MM names (these are discussed further below in a review of these MMs). As we can see, there have been many MMs focusing on the topics of IT, electronic systems and telemedicine. These topics highlight the importance of increasing digitalisation in this domain.
A word cloud of the maturity model (MM) names presented in the literature.
Four sources contributed methods and frameworks in this area. Source 2 presented a framework for a domain-specific business intelligence (BI) MM and applied it in health care. Source 22 developed a maturity assessment tool and technique in the subject of clinical governance. Source 59 proposed a MM for identity management and a procedure for applying the MM, and reported from practice on its successful use in two large Swiss hospitals. Source 74 proposed a method to identify a selected number of items from a previously validated quality improvement (QI) maturity tool as the basis for calculating organisational and system-level QI maturity scores in time.
As mentioned above, the contributions of 26 sources were empirical (case) results only. For example, the research in Source 12 sought further empirical evidence of IS and IT maturity in the context of the UK’s National Health Service (NHS) acute trust hospitals. A survey was used to collect data from more than 70 top and middle managers, representing four trust hospitals in the UK. Statistical analysis of these data provided evidence that 6/23 maturity characteristics identified in existing models can currently be used to differentiate the maturity of NHS acute trust hospitals. The aims of the study in Source 17 were to investigate the challenges of assessing process maturity in health-care institutions using a generic business process MM (i.e. OMG’s Business Process Maturity Model), and to explore the opportunities for future work that would facilitate a process maturity assessment and improvement in the health care domain. Source 20 reported an empirical audit of a health agency for preparedness with respect to the implementation of e-health management using the COBIT (Control Objectives for Information and Related Technology) framework. 41 Further insights into the pool of empirical studies in response to RQ 3 are discussed later in this article.
Two sources offered tools as one of their main contributions. In Source 54, the proposed health care quality maturity assessment model was incorporated in an interactive Microsoft Excel worksheet that visually displayed the quality maturity-level risk meters, which we considered under the ‘tool’ category. Another tool-presenting work was Source 64, which presented a software tool assisting the users in applying the General Practice Information Maturity Model.
Two sources contributed processes. Source 29 developed a capability maturity model (CMM)-based security risk assessment ‘process’ for patient-centred health care systems, and applied it in the context of personal health records. The CMM 10 is a framework developed based on the data collected from organisations that contracted with the US Department of Defense. The term ‘maturity’ in the CMM relates to the degree of formality and optimisation of processes, from ad hoc practices to formally defined steps to managed result metrics to active optimisation of the processes. The CMM’s aim is to improve software-development processes, but it has also been applied in other domains. Source 39 proposes a process for maturity assessment based on the MM introduced in that paper (i.e. the health care IT maturity model).
Five sources focused on ‘other’ types of contributions. Source 10 proposed a software process improvement model for the UK’s health care industry by using organisational maturity (based on ISO/IEC 15504 9 ) as the guideline. Source 16 discussed authors’ experiences in applying the CMMI for Services model to health care services, specifically for in-hospital pharmaceutical and respiratory services. Similarly, Source 28 shared the encountered experience and challenges that the authors faced during the design and implementation of three MMs for distinct improvement purposes in hospitals. Source 29 mapped the components of the CMM into the regulations of the American Health Insurance Portability and Accountability Act. Source 34 discussed opinions on using digital maturity assessment in order to understand the level of readiness for using technology in the UK’s NHS.
RQ 1.2: Research types
Based on the classification scheme described earlier, we classified the sources into seven types of research methods. Figure 5 shows the classification of the sources according to the type of research method they followed. We adapted these classifications from the SLR/MLR guidelines.13,17,28,29 Papers ranked under ‘solution proposal’ and ‘opinion papers’ used the least rigorous research methods, while papers conducting weak and strong empirical research were the most rigorous in terms research methods in general. Papers falling under the ‘experience’ category reported a form of experience in the topic.
Sources by type of research method.
Recall that for the research facet type, each study could be classified into only one category. In showing the classification in Figure 5, we also divided the sources by the two types of literature: academic literature and grey literature.
A large ratio of sources fell under ‘solution proposals’ (29 sources) and weak and strong empirical studies (31 and 20 sources, respectively), which is a positive sign of empiricism in this community. Twelve sources reported ‘opinions’. There were nine experience reports based on the ‘experience’ of their authors. No source was classified as a ‘philosophical paper’ or ‘other’.
Division of the sources by the two types of literature shows that most grey-literature sources were of ‘solution proposal’ or ‘opinion paper’ types and less of ‘weak empirical’ studies. No grey-literature sources used ‘strong empirical’ research methods. These observations are as one would expect, since grey-literature sources tend to utilise less rigorous research methods.
RQ 2: Various properties of MMs
RQ 2.1: Proposed MMs with their subject areas
We compiled a list of all the proposed MMs and collated them in Table 3. In total, 68 sources proposed 68 new models in this area. As can be seen from Table 3, there is vast diversity in the topic of MMs in this area. Thus, we also extracted the subject area of these models and provide an index of these subjects, as addressed by the sources, in alphabetical order in an online resource. 35 The most frequently occurring subjects by the models were: health services and IT (n = 6 each); public health, QI and safety culture (n = 4 each); BI, data analytics, e-health, electronic medical records (EMR), EMR adoption, hospital information system, integrated care, patient safety, security and telemedicine (n = 3 each).
List of proposed MMs by sources.
QI: quality improvement; EMR: electronic medical record.
RQ 2.2: Aspects covered by the MMs
We classified the aspects covered by the proposed MMs. The possible aspects (categories of classification) were process, technology, people and other aspects. Figure 6 shows the distribution of the sources with respect to these aspects. Note that the MM presented in a given source may cover more than one single aspect (e.g. Sources 1 and 2). Thus, the sum of the four categories in the figure is greater than number of sources (N = 101).
Mapping of sources with respect to the aspects of the proposed MMs.
The ‘business process’ aspect is covered in 65 sources. For example, Source 15 proposed an open government MM for participation/collaboration processes between public and government agencies. Source 30 developed a framework to measure maturity of regulations in health care, and Source 40 presented an MM for enterprise architecture and applied it in the context of the Veterans Health Administration. The ‘technology’ aspect of maturity is also covered in 65 sources. For example, Source 5 defined an MM to assess hospital IS. Source 23 proposed a cloud MM for the health care industry, and Source 33 described a digital imaging MM. The ‘people’ aspect of maturity is addressed in 46 sources. For example, Source 19 defined an assessment tool for patient safety culture in Manchester. Source 42 proposed a people capacity MM in medical record wards of Iranian hospitals. Source 100 defined a cooperation MM for hospitals. Under the ‘other’ category, 30 sources covered various aspects, including work culture, strategy, governance, leadership, interoperability, and data. Readers are encouraged to use Figure 6 to trace the references to the sources in each category and to access details of them via the study’s online spreadsheet (https://goo.gl/Z4jxBd).
Three example MMs focusing on different aspects of health care.
Table 4 shows the levels in the staged representation (or dimensions in continuous representation42,43) of three example models in different aspect categories.
RQ 2.3: Applicability scope/scale of the MMs
For RQ 2.3, we investigated the applicability scope of the MMs. As discussed earlier, the scope could be one of the following: a single department of a hospital/health care institution; multiple departments of a hospital; a single hospital/health care institution; multiple health care institutions; a city, region or province; government (nationwide); international; or ‘other’. Based on the classification of the sources into the above categories, Figure 7 shows a histogram. Remember that in eliciting the applicability scope of the MMs, we considered the expected/possible applicability of the models and not necessarily the scale of the actual empirical study presented in a given source. The sum of all bars in Figure 7 is 85. Please note that some sources did not report any applicability assessment of the MMs. Thus, their data (scope/scales) are not presented in Figure 7. Furthermore, for some other sources, the scope/scale classification could be multiple (e.g. for Source 15, the applicability scope/scale was both multiple hospitals and nationwide). We found that the most common application scope/scale of MMs was ‘multiple hospitals’, as addressed by 35 sources. For example, Source 4 applied a MM for care pathways to 11 hospitals in The Netherlands. To enable ‘precision medicine’, 44 Source 24 measured the maturity of BI in six hospitals in Italy. As another example, Source 75 assessed the relationship between organisational maturity and project success in health care based on data from seven hospitals in Portugal.
Sources with respect to the applicability scope/scale of the MMs.
RQ 3: Properties of empirical studies
RQ 3.1: Scales of empirical studies
To characterise the scale of the reported empirical studies, we focused on the number of health care institutions in which a proposed MM was applied. We also looked into other attributes of the empirical context, for example the number of physicians or other types of experts involved in the surveys, or regular participants who were interviewed, or the number of workshops, focus groups or expert panels that were organised to gather data for the maturity assessment. Note that the applicability scope/scale in RQ 2.3 differs from the empirical scale in RQ 3.1 in that in the case of the former, we assessed the ‘potential’ applicability, while in the latter we looked at the ‘actual’ reported empirical results presented in a source.
Out of the 101 sources, 56 reported quantitative information regarding the scale of their empirical studies. There were 33 sources that performed their applications in health care institutions (hospitals, medical centres or clinics), while the remaining 23 sources performed empirical work based on other forms, such as opinion surveys, workshops or focus groups with health care specialists, or interviews with experts. In the last group (‘other’ forms), seven sources used surveys with diverse stakeholders in the health care domain (e.g. Sources 70, 71, 84 and 88). Five other studies applied focus groups/expert panels for the evaluation of the MMs (e.g. Sources 76 and 77).
These sources typically structured their evaluations as case studies and involved experts from multiple departments within an institution (e.g. Sources 2 and 19). The sources that performed their empirical evaluations in multiple institutions (22 sources), on the other hand, varied greatly in terms of the number of institutions. Four sources applied their MMs on a large scale (i.e. more than 50 institutions). Sources 35 and 78 involved the most institutions in our list of sources (538 and 129 hospitals, respectively). Sources involving high numbers of institutions in their research typically conducted opinion surveys, which incorporated questions from MMs to gain an understanding of the state-of-the-maturity of health care in particular geographical regions. These included, for instance, the maturity of QI systems across Europe (Source 32), e-health care maturity in hospitals in Taiwan (Source 35) or the process management capability of hospitals in Switzerland (Source 78).
It is also important to note that a large majority of sources that reported quantitative data regarding the scale of their empirical studies originated from academia (52/55 sources). Only 3/21 sources (Sources 84, 88 and 97) in the grey literature provided quantitative information about the empirical studies that were conducted.
RQ 3.2: Empirical benefits of applying MMs
In addressing RQ 3.2, we extracted and synthesised the benefits reported by sources as the results of their empirical studies. In doing so, we focused particularly on the benefits reported through the application of the MMs, and whether these benefits were stated in quantitative or qualitative forms (or both). We observed that more than half of the sources (62/101) reported the benefits of applying MMs in qualitative terms, and only two sources reported benefits in quantitative terms (Sources 70 and 79). The remaining 37 sources did not provide any explicit discussions on the benefits of applying MMs.
As an example of quantitative benefits, Source 70 measured maturity of use for EMRs. A comparison of the quantitative data from before and after applying the maturity assessment showed an increase from 21% to 83% of physicians who would use EMRs as the principal method of record keeping. As another example, Source 79 compared procurement maturity with procurement performance in three Dutch hospitals. The gathered numerical measurements supported the hypothesis that an increase in maturity of organisation, processes and IT in hospitals would lead to better procurement performance, thus again showing a quantitative benefit of the proposed MM.
We then looked at the 62 sources which reported the benefits of applying MMs in qualitative terms. To understand the vast amount of qualitative data that we extracted from the papers better, we classified them using the ‘open coding’ (grounded theory) method 45 which is widely used in qualitative analysis. The process resulted in three classifications: (a) identifying issues and providing guidance for improvement (n = 17); (b) improving efficiency, effectiveness, performance and productivity (n = 12); and (c) ‘other’ benefits (n = 33). The classifications of the sources that report these types of benefits, together with selected examples, are shown in Table 5.
Qualitative benefits of applying MMs.
Based on these numbers, we can see that although the number of sources that reported on the benefits of applying MMs in concrete quantitative terms was limited, a vast number of sources reported on the benefits of using MMs in health care in qualitative terms.
Discussion
Observations and suggestions
In response to RQ 1, we reviewed sources by contribution and research type. The general observable trend showed that new MMs are continuously being offered in this area, and there were 68 new MMs proposed. Also, the trend of sources with an emphasis on and having major contributions as empirical (case) studies was an indicator of the progressive nature of MM adoption. In terms of research-method types, a large ratio of sources fell under the categories of ‘weak empirical study’ and ‘solution proposal’. Although this is noted as a good sign of emergence and empiricism in this area, there is a need for studies with stronger empirical evidence.
For RQ 2, we reviewed sources by various properties of MMs, including the identity of the proposed models, aspects covered and applicability scope/scale. We identified and listed MMs proposed by the sources, and also extracted and indexed the subject areas on which they focused. We believe the catalogue describing the available MMs and the index given in Tarhan et al. 35 could be useful to practitioners, decision makers and researchers who are interested in assessing the maturity of a given health care entity (e.g. a hospital) from a particular point of view.
The list of subject areas demonstrated a wide range of research and application, which is promising for further research and practice. Since the index of subject areas is very broad, detailed analysis of critical areas for potential future work is needed for various specialties in health care and technical domains. For example, we highlight areas such as practical aspects of patient safety (e.g. incorrect labelling on patients’ drug charts) and the clinical safety in medical and surgical departments. We were able to locate and review three MMs related to patient safety: (a) patient safety culture improvement tool (Source 76), (b) Manchester patient safety framework (Source 77) and (c) safety culture assessment in community (Sources 19 and 85). While these models are useful, we see a need for improvement to cover a broader range of clinical safety.
The MMs also varied in the aspects that they covered (e.g. technology, business processes and people). The majority of the models addressed both the ‘process’ and ‘technology’ aspects, and about half of the models covered the ‘people’ aspect. Almost one third of the models widened their coverage to include ‘other’ aspects such as culture, strategy, governance, leadership, interoperability and data. As the included aspects demonstrated, this area is multidisciplinary in nature and requires organisation of research and practice in such challenging settings. We believe that the increasing amount of digitalisation in health care will make the advantages of using guiding models as MMs even more significant in the upcoming years.
According to the applicability scope/scale of MMs, the majority of the models were targeted for use in ‘multiple hospitals’, followed by ‘a single hospital’, ‘city, region or province wide’, ‘nationwide’, ‘single departments of a hospital, and ‘multiple departments of a hospital’. As a result, there was a balanced variety in the applicability scale of the models. We suggest that the use of MMs among multiple hospitals or within governmental bodies (i.e. nationwide) will be useful, especially for the benchmarking of health care services.
RQ 3 focused on the application scale of the empirical studies performed and the benefits observed. Findings on the empirical applications of MMs showed that there was variety in the number of institutions involved and in the empirical research methods employed (e.g. case studies, surveys, interviews, workshops/focus groups). More than half of the sources reported on the scale of their works, and more than half of those reporting information about the scale of application performed their work directly in one or more health care institutions. More specifically, the MMs were empirically applied in various contexts, ranging from a single hospital to multiple hospitals and to health bodies of countries. Therefore, we argue that an important number of sources rely on a solid empirical basis. On the other hand, a considerable percentage of works, particularly in the grey literature, lacked application or empirical results.
Regarding the benefits of empirical applications, many of the sources reported on the usefulness and utility of the models through qualitative methods, but few presented concrete evidence of the benefits in using them. These benefits included, for instance, increased effectiveness and efficiency in health care service provisioning, improved patient satisfaction and improved service quality. Moreover, there is a need to investigate the relation between qualitative and quantitative benefits in future studies. Designing longitudinal studies to evaluate the effects of increasing maturity in combination with understanding these effects in quantitative terms (e.g. by using process analytics techniques such as process mining46–48) might be useful for this purpose.
Threats to validity
Here, we discuss the potential validity threats in the context of the four types adopted from Wohlin et al. 49
Internal validity
In the systematic approach utilised for source selection, search engines, search terms and inclusion/exclusion criteria were carefully defined to ensure that this review is repeatable. Still, the selection of search terms and search engines and the bias in applying exclusion/inclusion criteria can be considered as the limitations during this process. In order to mitigate the risk of missing some relevant sources, formal searching using defined keywords was conducted, and an adequate and inclusive basis has been collected while building the study pool. In order to minimise the bias related to researchers’ judgement and experience in applying inclusion/exclusion criteria, joint voting was applied after the initial source inclusion, and only sources passing the joint voting were selected for review.
Construct validity
This type of validity is concerned with the suitability of the RQs and the categorisation scheme used for the data extraction. Review questions were designed to cover our goal and were answered according to a categorisation scheme. To design a good categorisation scheme, we adapted standard classifications and also finalised the schema through several iterations. Another threat comes from the lack of empirical evidence in the primary sources. The majority of the grey literature was opinion or experience based, and as the source of knowledge was not typically revealed, we are faced with an epistemological problem, that is, we do not know how we know what we know. However, handling this limitation in study design by excluding the grey literature would leave an important source and related voice of practice out of the scope. Therefore, we decided to include these sources as they are in order to have a more complete profile in our review.
Conclusion validity
This type of validity requires reaching appropriate conclusions through rigorous and repeatable treatment. In order to ensure reliability of our treatments, the entire pool of primary sources was analysed, and the data were reviewed, extracted and synthesised by the authors. Following the systematic approach and describing the review questions and procedure ensured replicability of this study and created confidence that the results of a similar study would not have major deviations from our classification.
External validity
As described earlier, search terms in the source-selection approach resulted in having primary sources all written in the English language. However, a good proportion of industrial and collaborative works, in addition to academic studies, exists in our sources due to including the grey literature. This means that our inclusive process of article selection has led us to have an adequate basis for concluding results and that our pool contained sufficient information to represent the knowledge reported by other researchers and professionals. Nevertheless, the findings of this study are mainly within the specific area under study, and we have no intention of generalising our results beyond this field.
Conclusion
The complex and multidisciplinary nature of the health care domain brings significant challenges in terms of process and people management, and IT and IS. Moreover, the increasing digitalisation of health care services demands well-defined guidelines and success reports to rationalise and manage the transition process effectively. In response to those challenges, a wide variety of maturity assessment and MMs has been proposed and applied in the domain.
In this study, we have provided an overview of these models and classified them based on their various properties and applications using MLR as the research method. The results of our review indicate that MMs are continuously being offered under various subjects of health care and also empirically applied, and that the body of sources presents a good balance between research and practice, which is promising for conducting further studies in this area. Future research, however, is required to include results from a stronger empirical basis, especially for the reports in the grey literature.
The index of the subject areas concerned by the MMs is so broad that there is a need for deeper analysis of the sources with respect to detailed specialties of health care and technology. It is suggested that researchers should check the list of subject areas by the MMs reported in this study prior to proposing new models. This is especially important for eliminating identical studies (i.e. reinventing the wheel) as well as for creating opportunity on using or adapting already available MMs.
Finally, we believe that conducting longitudinal studies in different contexts by using quantitative or mixed methods (e.g. process mining) could better demonstrate the advantages of using MMs and therefore serve the community with stronger empirical evidence to advocate the utility of these models – particularly on the side of the practitioners.
Footnotes
Conflict of interest
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Contributorship
A.K.T., V.G. and O.T. performed the search, selection and pilot data extraction regarding the literature review, and S.G. was consulted for her medical knowledge in discussions as to whether to include/exclude the sources. A.K.T., V.G., O.T. and M.S. carried out the data extraction and analysis of all the included studies in the review, during which S.G. was actively consulted and involved in discussions in case of conflicts. All of the authors were involved in the manuscript design, writing, review and revision.
Ethical approval
Not applicable.
Funding
The author(s) received no financial support for the research, authorship and/or publication of this article.
Guarantor
A.K.T.
Peer review
This manuscript was reviewed by reviewers, the authors have elected for these individuals to remain anonymous.