Narrating in grey: An application to educational management information systems and accountability

Abstract

This research presents an application of a mixed narrative review, including grey literature, to broaden knowledge about the value of educational management information systems (EMIS) for accountability in higher education institutions (HEIs). The review was focused in understanding the relationships among quality management (QM), EMIS use and accountability. Analyzing 39 documents produced between 1990 and 2018, we confirm the tight QM–EMIS use–accountability relationship. A weak link between QM and EMIS use was found, resulting in a low accountability level, nevertheless e-maturity successfully describes the link between QM and EMIS use and legitimizes the HEIs in the society. Finally, the value of mixed narrative reviews, including gray literature, is demonstrated in the fields of management information systems and higher education.

Keywords

educational management information systems accountability higher education grey literature Chile

Theoretical framework

Problem statement

In Chile, while the government has shown a growing interest to increase the Higher Education quality, the educational system as a whole still has a low level of accountability, thereby affecting the public confidence toward Higher Education Institutions (HEIs) (Dussaillant and Guzmán, 2014; Riquelme Silva et al., 2018; Rojas Ríos and López Stefoni, 2016; Fleet et al., 2014; López et al., 2015).

On the other hand, an extensive ICT use nationwide, mainly in urban areas (Stager Koller et al., 2017; Stäger Koller and Núñez Tissinetti, 2015), an increasing ICT use in Chilean higher education (Brun and Hinostroza, 2014; González Bravo and Valdivia Peralta, 2015) and a growing market of Higher Education ERPs in Latin-America (Cassidy, 2006; Abdellatif, 2014), are observed. The latter is probably due to these Educational Management Information Systems (EMIS) providing information in challenging environments for quality purposes, both within HEIs, and in governments and quality agencies (Haris et al., 2017).

The starting problem addressed in this research is the discrepancy between a growing EMIS use and a still low accountability level in Chilean HEIs. On the other hand, a growing number of scholars are recognizing the value of narrative reviews and ‘grey literature’, for generating new knowledge beyond peer-reviewed journals, incorporating practice and case reports, among many other sources (Bae, 2014; Paez, 2017; Paré et al., 2015; Rumrill and Fitzgerald, 2001)

Considering the cases using the already mentioned approach (Benzies et al., 2006), a mixed narrative review (NR) is carried out in order to demonstrate the value of narrative reviews, which include grey literature, in emerging higher education issues.

Narrative reviews

Narrative literature reviews or narrative reviews (NR) are focused to identify, analyze, assess and interpret the existing knowledge on a specific topic, depending on the final product to which it is aspired (Guirao Goris, 2015). They tend to focus on interpretation, following a path which could include multiple open-ended meanings (Ayala, 2018). In this sense, it is valuable when the researcher is interested in connecting many sources either for purposes of deepening understanding of a broad or complex issue (Thorne, 2018) or reinterpretation/interconnection pursuing theory building (Baumeister and Leary, 1997). It is possible to identify quantitative narrative reviews, qualitative narrative reviews, and mixed methods reviews, being the latter used when both quantitative and qualitative research are involved (Bae, 2014).

NR critics have argued that this is the simplest type of literature review, a subjective first approach (Rumrill and Fitzgerald, 2001), which does not seek generalization or cumulative knowledge, without systematic and comprehensive literature searching, detailed explanations and methods regarding the review process (Paré et al., 2015; Rumrill and Fitzgerald, 2001). It has even been pointed out that NR is less comprehensive, conducted by unidentified experts, without an explicit statement of methods and being in the last place of the hierarchy of scientific methods (Aguilera Eguía, 2014).

Answering these critics, several authors have exposed methods to improve the quality of a NR: detail databases and keywords used, dates and search strategy, how the terms used were combined, the number of sources found (Guirao Goris, 2015) or final report sections clearly identified (Ferrari, 2015; Rumrill and Fitzgerald, 2001). Even have been suggested guidelines for evaluating the quality of certain reviews in terms of rigor and relevance (Paré et al., 2015) or recommendations to improve the searching and extracting process, or the final report (Vom Brocke et al., 2009; Bandara et al., 2011)

In a deeper view, some critics towards NR come from traditional quantitative perspectives, which underestimates the value of practice, reflection or social inquiry (Ng et al., 2015). The problem is that in fact, science moves on when it is capable of generating original and creative hypotheses from multiple directions, some of them germinal, and not necessarily coming from studies published in peer-reviewed journals: for example, a health-related clinical case, or a specific school-practice (Thorne, 2018). In fact, reflection could contribute to the development of educational practice at a high level, providing a critical awareness about the theories, applications of epistemological positions (Ng et al., 2015) or adding dimensions of insight/application perspectives, and providing critical analyses (Rumrill and Fitzgerald, 2001). In conclusion, some NR offer much deeper or “richer” information than a classical empirical analysis or meta-analysis (Rumrill and Fitzgerald, 2001)

Narrative reviews and grey literature

The NRs include in their scope a wide variety of sources, it thus becoming necessary to analyze the concept of grey literature and its value for scientific research.

According to the ‘Luxembourg Definition’, grey literature refers to print or electronic literature produced by government, academia, business and industry, and not controlled by commercial publishers. This can include materials such as unpublished studies, conference abstracts, conference proceedings, book chapters, government and agency reports, unpublished doctoral dissertations (Gokhale, 1998; Godin et al., 2015) or PowerPoint presentations, evaluation reports, standards/best practice documents, guidelines, working papers (Benzies et al., 2006).

The inclusion of grey literature has a heuristic value for NR, expanding the scope, adding contextual information (Benzies et al., 2006; Mahood et al., 2014; Adams et al., 2016; Paez, 2017) and in fact is recommended when the phenomenon is complex with multiple components (Benzies et al., 2006). Combining grey literature along with peer-reviewed sources may therefore provide a more balanced view of the evidence (Mahood et al., 2014), reducing the impact of publication bias (Adams et al., 2016; Gracia Leiva et al., 2019; Bellefontaine and Lee, 2014). It is because of this reason, that scholars are increasingly recognizing the value of incorporating ‘grey literature’ (Adams et al., 2017)

Quality management in higher education

Chile was the first Latin American country that initiated quality assurance practices in higher education as a state policy in 1990, through the Higher Education Council (Inga Ipanaque and Velásquez Silva, 2005). In their early years, the procedures were focused over a compulsory licensing of new private HEIs. In 2003, accreditation of institutions started, and in 2006 the National System of Quality Assurance was created by law (Lemaitre et al., 2011). Institutional accreditation remained in the hands of the newly founded CNA (National Accreditation Commission), being a requirement for students who wished to access the Credit with State Guarantee (CAE) to pursue undergraduate studies in a HEI, and making accreditation an imperative for institutions to operate – survive – in the system (Fleet et al., 2014).Through the years, the concept of quality eventually changed from an abstract standard to an HEIs’ response to a set of externally established benchmarks in terms of legitimacy and information about the purpose and outcomes. As a result, citizens can make informed decisions considering these factors in programs or institutions (Kinser, 2014).

Applied to higher education, a quality assurance (QA) view has been defined as “set of mechanisms and processes aimed at controlling, guaranteeing and promoting the quality of higher education institutions” (CNA-Chile, 2015: 4). Gradually, the QA concept has been replaced by quality management (QM), which emphasizes continuous development and improvement, rather than just responding to external certifications (Cabrera Lanzo, 2018; Lemaitre et al., 2012). In HEIs, without quality information, it is impossible to manage quality (Beard and Humphrey, 2014; Mora et al., 2009) and to advance from QA to QM involves not only control over a deregulated system, but also to regain academic promotion of quality (Zapata and Torre, 2012a).

Managers and political authorities are rethinking higher education models, and adjust them to meet the environmental requirements. It must be guaranteed the existence of a sustainable and well-funded framework to support the HEIs’ efforts. It is for this reason that in many countries, quality and continuous improvement, accountability and qualification frameworks in the higher education sector are at the forefront of national political agendas (Pucciarelli and Kaplan, 2016).

Accountability

Accountability is a very widespread concept and a term that has many meanings: from the efficient functioning of institutions and efficient learning in students, to the system’s ability to stimulate the economy (Speziale, 2012). In the last three decades, this concept has been developed in the US and subsequently in the world, from the prevailing concept in the 1970s – design of government structures that adequately address the problem of autonomy versus the supervision of institutions – towards a vision that originated in the 1990s, focused on systemic results of HEIs (McLendon et al., 2006). Accountability and governance are necessary to preserve the pact between higher education and society; they are central issues for developing countries, and governments must be sure that HEIs are addressing the needs of society in terms of qualified graduates, and research to improve the quality of life of the citizens (Hendel and Lewis, 2005). There are multiple forces that promote accountability, from the governments that promote a scarce allocation of resources based on the results of the institutions (Kai, 2009), to the professional authority that promotes monitoring and continual improvement of processes and outcomes, for example through the use of management information systems (Shavelson, 2010).

Accountability, expressed through relevant public information and transparency, are key factors for citizenship when choosing a HEI and it also enhances the ability of citizen control over the state or the Higher Education market (United Nations Development Programme, 2006; Altbach, 2009). While Chile must build its achievements on the basis of an institutional framework in which the formal role of accountability is sufficiently valued, the country is not fully prepared for higher education quality systems based on trust and social capital (Dussaillant and Guzmán, 2014). This is due to an extremely deregulated educational market since 1981 (during the dictatorship of Augusto Pinochet), and it has taken twenty-five years to change to a more comprehensive regulation in a cultural context of an open market, an advanced use of technology, but where distrust in institutions persists.

Equity, quality and effectiveness and therefore accountability have been promoted as indicators of system success in Latin America (Castro et al., 2017), and the first way to provide quality information to achieve transparency, accountability and success of an HEI, is to have adequate management information systems (Rivera et al., 2009; Altbach, 2009).

EMIS in higher education

HEIs are purchasing or developing information systems to manage academic, non-academic and administrative processes. This is largely because their use makes services cheaper, easier, faster and more accurate (Gunawardhana and Perera, 2015), and the implementation of cutting-edge technology helps streamline HEIs and transform their processes to achieve outstanding results (Dlamini, 2015). These EMISs must provide information to directors that allows them to make key decisions for the organization, in the right direction considering environment and internal processes (Bubel et al., 2015; Martins et al., 2019). This information facilitates management at different levels (Toro, 2012): inputs for governance, academic records, human resources, physical resources inventory, financial systems. Most of the efficient performance of a HEI depends on having updated retention and dropout data of students, academic progression and enrolment, graduates employability rates and salaries, courses satisfaction reports, curriculum plans validity and teaching loads, among many other indicators. This implies an effective management of a large amount of information, and providing it in user friendly way, which often involves the use of ERP systems for several processes (Bubel et al., 2015) and even the daily work of all members of the organization (Stensaker, 2018).

The strategical use of data mining plays an important role in the sustainable growth, success, and quality assurance of universities (Goyal and Vohra, 2012), and has become relevant to assess and improve HEI quality. Through implementing strategic planning and effective/efficient allocation of resources, HEIs can be more effective and have more opportunities to make right investments/improvements, which accelerate their sustainable growth, success, and reputation (Al-Sabaawi, 2015; Sart et al., 2015, November; Zahid and Khan, 2016).

The use of information is an intangible resource, which ends up being related to organizational culture or management styles, that allows a university to develop in a competitive environment (Bubel et al., 2015). As Weiss says, in the context of the results of her research about leadership and the use of new technologies in HEIs “organizational quality factors…[are]…defined and a correlation determined between an organization’s organizational quality score and overall satisfaction with the CIO and IT department” (Weiss, 2010: 169).

However, as early as 1998, Abby Rubin (1998) indicated that in the case of developing countries, MISs have been implemented in the educational field, but that “…implementation and use of accountability measures, however, has been varied, not always living up to the ideal of what could be expected in terms of making the connection with quality improvement” (Abby Rubin, 1998: 287). Today, this situation has not changed substantively (Rojas Ríos and López Stefoni, 2016), despite the fact that educational organizations seek certain objectives in a transcendent way, objectives that mobilize them in different areas, for example, in the ICT incorporation for educational management, expecting a quality certification (Harrison et al., 2014)

Higher education institutions face pressures from the government and society as a whole (Bernasconi, 2012, Fall; Pedraja-Rejas and Rodríguez-Ponce, 2015), to demonstrate quality and effectiveness, which translates into processes of self-assessment and quality assurance (Garnett and Ecclesfield, 2008). These external demands can revolve around the concept of public value: the ethos and values of any public organization, service provider or profession must be assessed according to their creation of value, understood as better outcomes, services and trust. This allows the building of trust from the environment towards these organizations, making them more adaptive and stimulating collaborative relationships with society to the benefit of their students and the entire educational system (Garnett and Ecclesfield, 2008).

To recognize the importance of educational management information systems is relevant, considering first, the changing and challenging historical breakpoint in HE in terms of quality and accountability, and secondly, the significance that this issue has for the countries to design Higher Education public policies which considers the broader environment where institutions are inserted (Opazo et al., 2019). In Chile, the value of provision of information, transparency and accountability is particularly notable, given the advanced ICT use but distrust toward institutions.

Aim

The aim of this study is to analyze the value of a narrative review which includes grey literature, to understand the relationship between the use of EMIS in higher education and accountability.

Materials and methods

Design and data collection

We conducted a narrative literature review (Bae, 2014; Baumeister and Leary, 1997; Cronin et al., 2008; Ferrari, 2015; Green et al., 2006; Rumrill and Fitzgerald, 2001) mainly using:

Databases EBSCO, Proquest and Redalyc.org from 1990 to 2018. 1990 was the year that Chile returned to democracy and the Constitutional Act on Education was issued (Ley Orgánica Constitucional de Enseñanza – LOCE).

Normative documents issued by CNA-Chile and other Chilean government departments and educational organizations.

The selection of relevant publications until December, 2018, was made based in four themes, operationalized in four keywords or key themes (Bearman and Dawson, 2013): Quality management/quality assurance (QM/QA), Chilean Higher Education, ICT Use and Information Systems. Sources that explicitly established a connection between the mentioned terms were selected. An exclusion criterion for the selection of relevant literature was not including articles exclusively focused on new technologies serving the teaching/learning process.

This process allowed the selection of 39 publications. Additionally, they were classified into two main categories: empirical and non-empirical. This general classification was utilized to allow the inclusion of public documents and books, not published in peer-reviewed journals.

Among the publications reviewed, eight were empirical, which used samples ranging between 10 accredited (Lemaitre et al., 2012) and 57 HEIs (López et al., 2015).

Data analysis

A narrative review was conducted, where different sources were analyzed according to key themes. The steps were the following:

Each source was read following the six questions suggested by Porritt et al. (2014), to determine relevancy and then again to collect information relevant to the research problem (Porritt et al., 2014). These six questions were related with a) time period covered b) language (English and Spanish) c) population/sample, d) phenomena stated in the main aim, e) study design and f) outcome measured (p. 48).

After that preliminary stage, each component was analyzed thematically, organizing the information collected in three nodes that articulate the analysis, for the results of this study. The findings were summarized, articulating emerging conclusions from the sources in analytical themes (Thomas and Harden, 2008). This last stage was based on the judgements and insights of the reviewers, and is an inductive analysis to answer the research problem. The purpose is going “beyond the content of original studies” (p. 7), using descriptive themes to infer more abstract meanings (Thomas and Harden, 2008).

Results

The result of analysis shows that 13 of the 39 documents, explicitly established the relationship between QA/QM and use of new technologies for management. These results are exposed in Table 1.

Table 1.

Sources that explicitly establish the relationship between the use of new technologies for management, and quality assurance/management in Chilean Higher Education.

Source	Type	Sample
1. Aguayo et al. (2009)	Empirical	75 HEIs
2. Bernasconi (2012, Fall)	Non Empirical	Non-applicable
3. Cárdenas and García (2014)	Non Empirical	Non-applicable
4. Fukushi (2010)	Non Empirical	Non-applicable
5. González Bravo and Valdivia Peralta (2015)	Non Empirical	Non-applicable
6. Lemaitre et al. (2012)	Empirical	10 universities
7. López et al. (2015)	Empirical	58 universities
8. Mora et al. (2009)	Non Empirical	Non-applicable
9. Rivera et al. (2009)	Empirical	13 universities
10. Rojas Ríos and López Stefoni (2016)	Empirical	4 universities
11. Zapata and Fleet (2012)	Empirical	56 universities
12. Zapata and Torre (2012a)	Empirical	23 universities (Ibero-America)
13. Zapata and Torre (2012b)	Empirical	30 universities (Ibero-America)

As shown in Table 1, all of the mentioned documentary sources consider the context of national accountability as a framework where information or ICT are essential. These findings are synthesized below, and their importance is analyzed in detail in the light of literature, identifying two major analytical themes: a) Quality management and ICT Use, b) Quality Management, ICT Use and accountability.

Quality management and ICT Use

In 2003, González reported that there were no systematic data regarding the computer resources implementation in Chilean HEIs, although “it could be said that all universities have computer resources for teachers and students…[and that]…a significant number has Intranet and technology classrooms that include access to teleconferencing” (González, 2003: 23). While there is now broad recognition of the value of EMIS for education, this development started towards the end of the 1960s and was closely linked to technological developments of the 1990s in the computing area, but had not yet reached its current expression at the beginning of the present century. In fact, as indicated by Marterer (2008), in 2002 it was possible to identify only two empirical studies in large universities worldwide about effects of ERP systems in higher education (United States and England).

Today, on the other hand, there is awareness in Chilean universities, particularly those that belong to the “Council of Rectors” (i.e., universities created before 1981), that ICTs and “Business Intelligence Mechanisms” are a fundamental part of their work, and the ongoing growth and complexity of HEIs, and the society’s needs and demands (Mora et al., 2009), have created new needs to be met from the management perspective (Fuentes Tapia and Valdivia Pinto, 2010; Mora et al., 2009; Busco et al., 2018), and that, complementary, information management is key in institutional accreditation (Fleet et al., 2014; Lemaitre et al., 2012).

As Lemaitre et al. (2012) pointed out, while improving the information management processes emerged as a response to external demands from accreditation processes, information had taken a leading role in quality management. This growing need has been answered from HEIs through incorporation of several technological tools, including ERPs (Enterprise Resource Planning) for academic management (Zapata and Torre, 2012a), being already a trend reported in other countries around the world.

Today, many Chilean universities utilize integrated ERPs for internal management (Zapata and Fleet, 2012): in most cases, these systems incorporate ‘academic management modules’ where managers can follow up on the students’ academic progress or to what extent professors and researchers are carrying out their professional activities. The institution’s complexity and maturity, will influence what use it will make of that information following the classification of intelligences listed by Terenzini (1993): Technical/Analytical, Issues or contextual.

Regardless of these specific differences, there has been growing recognition of EMIS value in the last decade, so that authors such as Bernasconi (2012, Fall) affirm that “not only in Chile, but also in Latin America, tools such as planning, business intelligence, market surveys and performance indicators have been strongly incorporated” (p. 13). It is very important that HEIs incorporate good information management practices that allow them to use quantitative information in a context of strategies that provide them with both meaning and purpose. This notion is directly related to the concept of e-maturity applied to the educational field – degree of provision, administration, and use of technology to support learning in the curriculum (Underwood, 2009) –, as a broader perspective (Bagozzi, 2007; Harrison et al., 2014; Nistor, 2014) that gives long term meaning to ICT use, as a way to achieve its objectives transcendently (for example, social or systemic determinants, quality or external certification; Harrison et al., 2014).

Quality management, ICT use and accountability

In the area of public visibility, Zapata and Fleet (2012) carried out an analysis of the web pages of 56 Chilean universities, and found that the institutions disseminate their information in a partial way, emphasizing aspects of academic offer rather than issues related to accountability (for example, qualification, financial aspects, etc.). In addition, they found that public and traditional private HEIs disseminate more information than new private universities. Something similar occurs when comparing accredited universities versus those that are not, being the former those that provide more information. The authors show how the quality of information is largely explanatory of institutional quality.

As an additional component, in Chile as in the whole world, since the 1990s the Institutional Analysis Offices (IAOs) have given a strong boost to the incorporation of EMIS. In Chile, they received an important boost with the quality assurance policies promoted by the state in the 2000s, following with some delay, the trends originated in United States, and followed by Europe and Asia (Opazo et al., 2019). It is important to mention that they are an important point of contact between universities and their environment, in two senses: IAOs provide official information to external bodies that require it, and provide the organization with information about the environment in which it operates (Rivera et al., 2009). Rivera et al. (2009) carried out a study in which they found that among 61 Chilean universities, 48 have a division in charge of information processing, support for decision-making and informing the environment (for example, state agencies). To the extent that authors recognize that the work of these units is based on collection, analysis and systematization of information, it is a problem not always to have online and updated managing data. Something similar happened until 2011, with national information systems: in that year the higher education public information started to be disseminated through websites such as Mifuturo.cl (Ferreyra et al., 2017), an initiative that provides information about degrees, enrollment values, length of careers, graduation rate, employability, among others (Espinoza and González, 2013). This is a clear example in Chile of how governments can use information and communication technologies to improve their systems, contributing to social development (Qureshi, 2009).

In any case, the problem regarding the significant and strategic use of information remains, and this is worrying as far this information finally feeds back institution quality in a process of continuous improvement. In the same line, it describes how the investment on technology should be guided by a project to lead the process along. In simpler words, HEIs should know what they are using technology for. Fukushi (2010) raises questions that organize socio-cognitive frames and place the organization regarding its technological reality. According to the author, the few Chilean HEIs using EMISs systematically do so motivated by the ‘know-what’ or ‘know-how’, but very few reach the higher stages, where they must ask about systemic understanding of growth or the “motivated creativity or ethics” (Fukushi, 2010: 14) with a strong social component in terms of social welfare. This is fully connected with theories like technological frames (Orlikowski and Gash, 1994) and with concepts expressed by Bagozzi (2007), in the sense that often ICT adoption models do not respond clearly to the question about the final purpose of the technology incorporation (Bagozzi, 2007: 245) which could even be social welfare in general (Bagozzi, 2007: 251)

This is also consistent with the findings of Aguayo et al. (2009), who reported that quality of strategic plans, management of material resources and information systems were among the most noticeable weaknesses of Chilean HEIs. Likewise, they indicate that a key factor that hinders optimal management control in the HEIs is the absence of an integrated management system (67%). That is consistent with managers’ interest in planning, developing information systems for such purpose (Cáceres, 2007), and with Solar et al. (2013), about the value of e-maturity in the field of strategic planning, delivery of information required by stakeholders and by law, among many other areas.

In short, although there is a greater recognition about the value of EMISs in Chilean HEIs, three types of problems are observed:

1) An insufficient number of universities have implemented adequate systems of institutional information management, with the subsequent low ICT use for management (Espinoza and González, 2013; Rojas Ríos and López Stefoni, 2016). For example, López et al. (2015) conducted a study with 48 accredited institutions until 2011, and found that when reviewing the results for accreditation, one of the difficulties experienced by HEIs is the limitation of information systems for decision-making, mainly in the planning area (López et al., 2015; Cáceres, 2007). It is interesting that the mentioned article by López et al. (2015), mainly aimed at evaluating the effects of accreditation in Chilean universities, ends up analyzing how these effects are connected with the society as a whole: as a way to balance the deregulated privatization of higher education.

2) Inside the Chilean academic environment, although there is recognition about the value of such systems for accountability and their connection with quality management (Zapata and Fleet, 2012), there is observed a wide and variable spectrum of incorporation of ERPs to their processes (Espinoza and González, 2013; Rojas Ríos and López Stefoni, 2016). The assessment about the real impact of its benefits (Abugabah and Sanzogni, 2010; Chaushi et al., 2018) would be variable.

This situation is due in part to the current paucity of national literature that explores this field and the diverse degrees of e-maturity among institutions. In those institutions that are just beginning a systematic incorporation of EMIS, it is sought to develop integrated systems of information to take decisions based on evidence. In more mature HEIs, a better management of indicators is carried out, which translates into concrete actions, for example, improvement of academic indicators (Lemaitre et al., 2012), being this consistent with the three tiers of institutional research mentioned by Terenzini (1993).

3) Complementing the aforementioned, long-term planning problems persist for many Chilean HEIs, and the ability of these long-term plans to guide an ICT use policy for management (Cáceres, 2007). In this line, Zapata and Torre (2012a) mention a study of 23 universities in seven countries showing greater progress in developing QA systems: Mexico, Costa Rica, Colombia, Chile, Argentina, Portugal and Spain. The authors reported that one positive result of QA processes, was the incorporation of more information for internal control processes and decision making (Zapata and Torre, 2012b), which implies the use of better information systems. The impact of this gradual transformation and valuation seems to be greater in Costa Rica, Colombia and Chile, and slightly less in Mexico, Portugal and Spain.(Zapata and Torre, 2012a). Even though these systems serve primarily the institutional and academic management, it is the governing authorities who perceive more clearly their benefits. Academic and professional associations, conversely, question their validity, reliability and usefulness (Zapata and Torre, 2012b).

Discussion

The results of this narrative review including grey literature are consistent with previous research about the value of EMISs in higher education and allow obtaining meaningful findings related with the Chilean Higher Education System.

In the first place, EMIS facilitate the implementation of quality managing mechanisms, and to achieve accountability. In the second place, the relationship between Quality Management and accountability in Chilean HEIs, is mediated by EMIS use. In Chile, as in the rest of the world (Speziale, 2012; Shavelson, 2010), there is recognition about the usefulness of ICT and ERP systems, for educational quality management and particularly for accountability outcomes. This has been stimulated by government regulations that demand transparency in institutional outcomes, and by the institutional and programs accreditation processes. These processes not only demand to demonstrate results and organizational health, but the self-evaluation process itself is developed thanks to EMIS that provide adequate information, which must be used at different levels of the organization, being clear the final purpose for which they are being used (e-maturity).

Notwithstanding the foregoing, the aforementioned processes have happened partially and gradually in Latin American countries, HEIs have been incorporating ERPs in the development of their processes, and managers have gradually valued such implementation. This positive perception is becoming increasingly consolidated due to the usefulness of this information –derived from accountability, and has also allowed an improving of their internal processes and academic outcomes, making them public, increasing their years of accreditation and thus gaining legitimacy in a highly distrustful society.

In the case of Chile, the differences among institutions regarding the above (Espinoza and González, 2013; López et al., 2015; Dooner et al., 2016), confirm that the country has not completely changed from a culture of quality assurance to quality management (CINDA, 2019), to the extent that the mechanisms described are probably still implemented in many institutions as a way of responding to external standards, rather than a deep understanding of the benefits of incorporating a quality culture, relating to the findings of Fukushi (2010), Aguayo et al. (2009) and López et al. (2015). This is most likely amplified by the fact that the increasing legitimation among managers is not equally developed among many academics, a phenomenon that has been found in other countries and described in the literature (Cardoso et al., 2013; Larrauri et al., 2012; Watty, 2006).

The technology evolution is a historical event inextricably influenced by current political and historical facts, where the participation of citizens and institutions plays a key role and the value of transparency for democracy is based on the right of citizens to judge public bodies (United Nations Development Programme, 2006; Qureshi, 2009). In this way, the ability of HEIs to achieve higher levels of e-maturity will allow them to build legitimacy in societies where distrust occupies an important place. In the words of Terziyan et al. (2015) the ability to generate trust towards higher education institutions is based on data and well-designed and significant performance indicators, which allows decisions involving all stakeholders in a system of accountability.

As Zapata and Torre (2012a: 276) say regarding the Chilean higher education system, “the HEIs’ quality assurance systems should strengthen their links with information systems and indicators or dashboards promoted by government agencies, which help to make macro decisions regarding university functioning, while allowing a process to be carried out of accountability that makes sense at a social level, outside the strictly university and political sphere”.

Regarding the methodology used in the present study, expanding literature reviews to include grey literature, though difficult, is increasingly important in the fields of information science and organizational management, . The quality of a science domain can be increased by considering these sources and can bridge the research-practice gap (Adams et al., 2017), providing a balanced view of the evidence and a sense of context to answer a research aim, mainly when there is paucity of available evidence in commercial publications (Paez, 2017), as happened in the application of EMIS to accountability. In these areas, grey literature provides current, up-to-date information, allowing a big picture about what is happening with a body of literature (Bellefontaine and Lee, 2014).

Often, those who work in university institutional management do not publish their findings or practical experiences in peer-reviewed journals, but rather in congresses, seminars, or congresses or systematize their experience into guidelines from government agencies. Mixed narrative reviews that consider grey literature provide a mechanism to integrate these experiences in an efficient way, contributing to the higher education system as a whole. This is particularly evident in the case of world higher education, in the segment of people grouped in the so-called third space: “a collegial space where university professional staff and academic staff work collaboratively on complex and multifaceted projects” (Veles and Carter, 2016: 520). These professionals, with different degrees of experience in publications and academic research, are in a historical period of recognition and evolution, and several projects are being generated in United States, Australia, England, aimed at recognizing and enhancing their contribution to higher education (Whitchurch, 2009; Whitchurch, 2015; Veles and Carter, 2016; Berman and Pitman, 2010). Thus, mixed narrative reviews constitute an efficient methodology that recovers the contributions of this third space, for example, in the use of new technologies for management.

Finally, in a complementary sense, the findings of this study open the possibility of future pure quantitative studies exploring the relationship between the variables mentioned, in particular, regarding the perceptions that managers will have about the use of new technologies and quality management processes.

Footnotes

ORCID iD

Luis González Bravo

References

Abby Rubin

(1998) Reforms of educational efficiency and quality in developing countries: An overview. Compare 28: 277–291.

Abdellatif

(2014) ERP in higher education: a deeper look on developing countries. 2014 International Conference on Education Technologies and Computers (ICETC). 73–78.

Abugabah

Sanzogni

(2010) Enterprise resource planning (ERP) system in higher education: a literature review and implications. International Journal of Human and Social Sciences 5.

Adams

Hillier-Brown

Moore

, et al. (2016) Searching and synthesising ‘grey literature’ and ‘grey information’ in public health: critical reflections on three case studies. Systematic Reviews 5: 164.

Adams

Smart

Huff

(2017) Shades of grey: guidelines for working with the grey literature in systematic reviews for management and organizational studies. International Journal of Management Reviews 19: 432–454.

Aguayo

Bustamante

Pérez

(2009) Una mirada a la gestión en las instituciones de educación superior chilenas [A look at management in Chilean higher education institutions]. Pontificia Universidad Católica de Chile, V Jornadas de Gestión Universitaria. Santiago, Chile.

Aguilera Eguía

(2014) ¿Revisión sistemática, revisión narrativa o metaanálisis? Revista de la Sociedad Española del Dolor 21: 359–360.

Al-Sabaawi

MYM

(2015) Critical success factors for enterprise resource planning implementation success. International Journal of Advances in Engineering & Technology 8: 496.

Altbach

(2009) Peripheries and centers: research universities in developing countries. Asia Pacific Education Review 10: 15–27.

10.

Ayala

(2018) Thinking of conceptual reviews and systematic reviews. Nursing Inquiry 25: e12264.

11.

Bae

(2014) Narrative reviews. Epidemiology and health 36.

12.

Bagozzi

(2007) The legacy of the technology acceptance model and a proposal for a paradigm shift. Journal of the Association for Information Systems 8: 243–254.

13.

Bandara

Miskon

Fielt

(2011) A systematic, tool-supported method for conducting literature reviews in information systems. In: Tuunainen

Nandhakumar

Rossi

, et al. (eds) Proceedings of the19th European Conference on Information Systems (ECIS 2011). Helsinki, Finland.

14.

Baumeister

Leary

(1997) Writing narrative literature reviews. Review of General Psychology 1: 311–320.

15.

Beard

Humphrey

(2014) Alignment of university information technology resources with the Malcolm Baldrige results criteria for performance excellence in education: A balanced scorecard approach. Journal of Education for Business 89: 382–388.

16.

Bearman

Dawson

(2013) Qualitative synthesis and systematic review in health professions education. Medical Education 47: 252–260.

17.

Bellefontaine

Lee

(2014) Between black and white: examining grey literature in meta-analyses of psychological research. Journal of Child and Family Studies 23: 1378–1388.

18.

Benzies

Premji

Hayden

, et al. (2006) State-of-the-evidence reviews: advantages and challenges of including grey literature. Worldviews on Evidence-Based Nursing 3: 55–61.

19.

Berman

Pitman

(2010) Occupying a ‘third space’: research trained professional staff in Australian universities. Higher Education 60: 157–169.

20.

Bernasconi

(2012, Fall) Management in Latin American universities: no longer an oxymoron. ReVista: Harvard Review of Latin America.

21.

Brun

Hinostroza

(2014) Learning to become a teacher in the 21st century: ICT integration in initial teacher education in Chile. Educational Technology & Society 17: 222–238.

22.

Bubel

Turek

Cichon

(2015) Multidimensional management of a higher education institution. International Journal of Arts & Sciences 8: 199–213.

23.

Busco

Dooner

d’Alencon

(2018) Universidad de Chile: self-assessment and its effects on university’s management. Higher Education 75: 431–447.

24.

Cabrera Lanzo

(2018) Más allá de la acreditación: gestión interna de la calidad. Una perspectiva institucional [Beyond accreditation: internal quality management. An institutional perspective]. In: CINDA and INQAAHE (eds) Seminario Internacional Educación Superior para el Siglo 21. Santiago, Chile.

25.

Cáceres

(2007) Planeación estratégica en universidades del Consejo de Rectores: evidencias del periodo 2000-2005. Calidad en la Educación: 108–150.

26.

Cárdenas

García

(2014) Gestión de procesos en educación superior: mitos, barreras y aportes [Process management in higher education: myths, barriers and contributions]. Universidad Católica de Chile, X Jornadas de Gestión De Educación Superior. Santiago, Chile.

27.

Cardoso

Rosa

Santos

(2013) Different academics’ characteristics, different perceptions on quality assessment? Quality Assurance in Education 21: 96–117.

28.

Cassidy

(2006) Education management information systems (EMIS) in Latin America and the Caribbean: Lessons and challenges. Inter-American Development Bank, 53.

29.

Castro

Rodríguez-Gómez

Gairín

(2017) Exclusion factors in Latin American higher education: A preliminary analyze from university governing board perspective. Education and Urban Society 49: 229–247.

30.

Chaushi

Ismaili

(2018) ERP systems in higher education institutions: Review of the information systems and ERP modules. 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO). 1487–1494.

31.

CINDA (2019) Fortalecimiento de la gestión de la calidad en las universidades chilenas. Resultados de tres décadas de aprendizaje nacional [Strengthening of quality management in Chilean universities. Results of three decades of national learning]. Santiago, Chile, 7.

32.

CNA-Chile (2015) Glosario de términos complementarios criterios de acreditación de pregrado [Glossary of complementary terms. Undergraduate accreditation criteria.] Santiago: CNA-Chile.

33.

Cronin

Ryan

Coughlan

(2008) Undertaking a literature review: a step-by-step approach. British Journal of Nursing 17: 38–43.

34.

Dlamini

(2015) The role of the strategic and adaptive Chief Information Officer in higher education. Education and Information Technologies 20: 113–140.

35.

Dooner

Armanet

Busco

, et al. (2016) Impacto de los procesos de autoevaluación en la gestión de pregrado y postgrado en la Universidad de Chile (2011-2014). Santiago de Chile: Comisión Nacional de Acreditación. Serie Cuadernos de Investigación en Aseguramiento de la Calidad: 216.

36.

Dussaillant

Guzmán

(2014) ¿ Por qué Finlandia puede prescindir de accountability formal en educación y Polonia no? Calidad en la Educación: 137–158.

37.

Espinoza

González

(2013) Accreditation in higher education in Chile: results and consequences. Quality Assurance in Education 21: 20–38.

38.

Ferrari

(2015) Writing narrative style literature reviews. Medical Writing 24: 230–235.

39.

Ferreyra

Avitabile

Paz

(2017) At a crossroads: Higher education in Latin America and the Caribbean: World Bank Publications.

40.

Fleet

Pedraja Rejas

Rodríguez Ponce

(2014) Acreditación institucional y factores de la calidad universitaria en Chile [Institutional accreditation and university quality factors in Chile]. Interciencia 39: 450–457.

41.

Fuentes Tapia

Valdivia Pinto

(2010) Incorporation of business intelligence elements in the admission and registration process of a Chilean university. Ingeniare. Revista chilena de ingeniería 18: 383–394.

42.

Fukushi

(2010) Marcos sociocognitivos útiles para comprender la relación entre estrategia y tecnología [Sociocognitive frames useful to understand relationship between strategy and technology]. Sungard Summit. San Francisco, USA.

43.

Garnett

Ecclesfield

(2008) Colloquium. British Journal of Educational Technology 39: 468–474.

44.

Godin

Stapleton

Kirkpatrick

, et al. (2015) Applying systematic review search methods to the grey literature: a case study examining guidelines for school-based breakfast programs in Canada. Systematic Reviews 4: 138.

45.

Gokhale

(1998) Grey literature varieties: definitional problems. Proceedings of the Third International Conference on Grey Literature. Luxembourg, 1997-11-13/14. 259–273.

46.

González Bravo

Valdivia Peralta

(2015) Posibilidades para el uso del modelo de aceptación de la tecnología (TAM) y de la teoría de los marcos tecnológicos para evaluar la aceptación de nuevas tecnologías para el aseguramiento de la calidad en la educación superior chilena [Possibilities for using TAM and technology frames models to assess the acceptance of new technologies in the chilean higher education quality assurance]. Revista Electrónica Educare 19: 181–196.

47.

González

(2003) Los nuevos proveedores externos de educación superior en Chile [The new external providers of higher education in Chile], Santiago, Chile: Instituto de Educación Superior Para América Latina y el Caribe.

48.

Goyal

Vohra

(2012) Applications of data mining in higher education. International Journal of Computer Science Issues (IJCSI) 9: 113.

49.

Gracia Leiva

Puente Martínez

Ubillos Landa

, et al. (2019) La violencia en el noviazgo (VN): una revisión de meta-análisis. Anales de psicología 35: 300–313.

50.

Green

Johnson

Adams

(2006) Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. Journal of Chiropractic Medicine 5: 101–117.

51.

Guirao Goris

SJA

(2015) Utilidad y tipos de revisión de literatura. Ene 9: 0–0.

52.

Gunawardhana

DNT

Perera

(2015) A study of the usage of information systems in higher education: An exploratory review. International Journal of Advanced Studies in Computers, Science and Engineering 4: 15–20.

53.

Haris

Washizaki

Fukazawa

(2017) Utilization of ICTs in quality assurance and accreditation of higher education: Systematic literature review. 2017 IEEE 6th International Conference on Teaching, Assessment, and Learning for Engineering (TALE). IEEE, 354–359.

54.

Harrison

Tomás

Crook

(2014) An e-maturity analysis explains intention–behavior disjunctions in technology adoption in UK schools. Computers in Human Behavior 34: 345–351.

55.

Hendel

Lewis

(2005) Quality Assurance of Higher Education in Transition Countries: Accreditation -- Accountability and Assessment. Tertiary Education and Management 11: 239–258.

56.

Inga Ipanaque

Velásquez Silva

(2005) La evaluación y acreditación de la calidad en las nuevas leyes de educación superior de América Latina [The quality assesment and accreditation in the new Higher Education Laws in Latin America]. Universidad Nacional Mayor de San Marcos, Comisión de Coordinación de Reforma Universitaria.

57.

Kai

(2009) A Critical analysis of accountability in higher education. Chinese Education & Society 42: 39–51.

58.

Kinser

(2014) Questioning quality assurance. New Directions for Higher Education 2014: 55–67.

59.

Larrauri

Espinosa

Rosario Muñoz

, et al. (2012) Los académicos frente a la gestión de la calidad en las instituciones de educación superior. Percepción y valoración [The academics face the management of quality in higher education institutions. Perception and valuation]. Avaliação: Revista da Avaliação da Educação Superior (Campinas) 17: 689–712.

60.

Lemaitre

Maturana

Zenteno

, et al. (2012) Cambios en la gestión institucional en universidades, a partir de la implementación del sistema nacional de aseguramiento de la calidad: la experiencia chilena [Changes in institutional management in universities, from the implementation of the national system of quality assurance: the Chilean experience]. Calidad en la Educación: 21–52.

61.

Lemaitre

Maturana

Zenteno

, et al. (2011) Cambios en la gestión institucional del aseguramiento de la calidad en las universidades, a partir de la implementación del Sistema nacional de aseguramiento de la calidad: la experiencia chilena [Changes in institutional management of quality assurance at universities, from the implementation of the national system of quality assurance: the Chilean experience].

62.

López

Rojas

López

, et al. (2015) Chilean universities and institutional quality assurance processes. Quality Assurance in Education 23: 166–183.

63.

Mahood

Van Eerd

Irvin

(2014) Searching for grey literature for systematic reviews: challenges and benefits. Research Synthesis Methods 5: 221–234.

64.

Marterer

(2008) Enterprise resource planning in higher education: A comparative case study. College of Education and Human Services. University of North Florida, 163.

65.

Martins

Branco

Gonçalves

, et al. (2019) Assessing the success behind the use of education management information systems in higher education. Telematics and Informatics 38: 182–193.

66.

McLendon

Hearn

Deaton

(2006) Called to account: Analyzing the origins and spread of state performance-accountability policies for higher education. Educational Evaluation and Policy Analysis 28: 1–24.

67.

Mora

Grünewald

Barros

(2009) Gestión de la información y benchmark en la educación superior. In: Arata

Rodríguez Ponce

(eds) Desafíos y perspectivas de la dirección estratégica de las instituciones universitarias [Challenges and Prospects for the strategic management of universities]. Santiago de Chile: Ediciones CNA-Chile, 403–430.

68.

Kinsella

Friesen

, et al. (2015) Reclaiming a theoretical orientation to reflection in medical education research: a critical narrative review. Medical Education 49: 461–475.

69.

Nistor

(2014) When technology acceptance models won’t work: Non-significant intention-behavior effects. Computers in Human Behavior 34: 299–300.

70.

Opazo

Díaz

Lemaitre

(2019) Análisis Institucional. Un Instrumento para la gestión eficaz [Institutional Analysis: An instrument for effective management] , Santiago, Chile: CINDA.

71.

Orlikowski

Gash

(1994) Technological frames: making sense of information technology in organizations. ACM Transactions on Information Systems (TOIS) 12: 174–207.

72.

Paez

(2017) Gray literature: An important resource in systematic reviews. Journal of Evidence-Based Medicine 10: 233–240.

73.

Paré

Trudel

M-C

Jaana

, et al. (2015) Synthesizing information systems knowledge: A typology of literature reviews. Information & Management 52: 183–199.

74.

Pedraja-Rejas

Rodríguez-Ponce

(2015) The quality assurance: A strategic imperative in higher education. Ingeniare. Revista chilena de ingeniería 23: 4–5.

75.

Porritt

Gomersall

Lockwood

(2014) Study selection and critical appraisal: The steps following the literature search in a systematic review. American Journal of Nursing 114: 47–52.

76.

Pucciarelli

Kaplan

(2016) Competition and strategy in higher education: Managing complexity and uncertainty. Business Horizons 59: 311–320.

77.

Qureshi

(2009) Social and economic perspectives on the role of information and communication technology for development. Information Technology for Development 15: 1–3.

78.

Riquelme Silva

López Toro

Ciruela Lorenzo

(2018) Propuesta de un modelo integral de accountability para la Educación Superior en Chile. Ciencia & trabajo 20: 107–115.

79.

Rivera

Astudillo

Fernández

(2009) Información y toma de decisiones: oficinas de análisis institucional en universidades chilenas. Calidad en la Educación 2009: 162–178.

80.

Rojas Ríos

López Stefoni

(2016) La acreditación de la gestión institucional en universidades chilenas. Revista electrónica de investigación educativa, 18.

81.

Rumrill

Jr Fitzgerald

(2001) Using narrative literature reviews to build a scientific knowledge base. Work 16: 165–170.

82.

Sart

Kırbaşlar

Şİ

Dülger

(2015, November) Why Do The Higher Education Institutions Need Big Data Management? ICERI 2015, 8th International Conference of Education, Research and Innovation. Sevilla, Spain.

83.

Shavelson

(2010) Measuring college learning responsibly: Accountability in a new era: Stanford University Press.

84.

Solar

Sabattin

Parada

(2013) A maturity model for assessing the use of ICT in school education. Journal of Educational Technology & Society, 16.

85.

Speziale

M-T

(2012) Differentiating higher education accountability in the global setting: a comparison between Boston University and University of Bologna. Procedia - Social and Behavioral Sciences 47: 1153–1163.

86.

Stager Koller

León Aceitón

Nuñez Tissinetti

(2017) Uso de internet en Chile. La otra brecha que nos divide [Internet use in Chile: the other gap that divides us]. Santiago, Chile: Centro de Estudios Digitales, Fundación País Digital.

87.

Stäger Koller

Núñez Tissinetti

(2015) Uso de internet en Chile: la otra brecha que nos divide [Internet use in Chile: the other gap that divides us]. Santiago de Chile: Centro de Estudios Digitales, Fundación País Digital.

88.

Stensaker

(2018) Academic development as cultural work: responding to the organizational complexity of modern higher education institutions. International Journal for Academic Development 23: 274–285.

89.

Terenzini

(1993) On the nature of institutional research and the knowledge and skills it requires. Research in Higher Education 34: 1–10.

90.

Terenzini

(2013) “On the Nature of Institutional Research” Revisited: Plus ça Change…? Research in Higher Education 54: 137–148.

91.

Terziyan

Golovianko

Shevchenko

(2015) Semantic portal as a tool for structural reform of the Ukrainian educational system. Information Technology for Development 21: 381–402.

92.

Thomas

Harden

(2008) Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Medical Research Methodology 8: 45.

93.

Thorne

(2018) Rediscovering the “Narrative” review. Nursing Inquiry 25: e12257.

94.

Toro

(2012) Gestión interna de la calidad en las instituciones de educación superior [Internal quality management in higher education institutions], Santiago, Chile: CINDA/ RIL Editores.

95.

Underwood

(2009) The impact of digital technology: a review of the evidence of the impact of digital technologies on formal education. BECTA, 27.

96.

United Nations Development Programme (2006) Desarrollo Humano en Chile: las nuevas tecnologías: ¿un salto al futuro?, Santiago de Chile: United Nations Development Programme.

97.

Veles

Carter

M-A

(2016) Imagining a future: changing the landscape for third space professionals in Australian higher education institutions. Journal of Higher Education Policy and Management 38: 519–533.

98.

Vom Brocke

Simons

Niehaves

, et al. (2009) Reconstructing the giant: on the importance of rigour in documenting the literature search process. Ecis. 2206–2217.

99.

Watty

(2006) Want to know about quality in higher education? ask an academic. Quality in Higher Education 12: 291–301.

100.

Weiss

(2010) Information technology management in higher education: An evidence-based approach to improving chief information officer performance . Ann Arbor: The University of North Carolina at Chapel Hill, 310.

101.

Whitchurch

(2009) The rise of the blended professional in higher education: a comparison between the United Kingdom, Australia and the United States. Higher Education 58: 407–418.

102.

Whitchurch

(2015) The Rise of Third Space Professionals: Paradoxes and Dilemmas. In: Teichler

Cummings

(eds) Forming, Recruiting and Managing the Academic Profession. Cham: Springer International Publishing, 79–99.

103.

Zahid

Khan

(2016) Impact of ICT Innovations on the quality of business research process in higher education institutions. International Journal of Engineering and Management Research (IJEMR) 6: 473–477.

104.

Zapata

Fleet

(2012) Mercado, rendición de cuentas e información pública en educación superior. Estudios pedagógicos (Valdivia) 38: 259–276.

105.

Zapata

Torre

(2012a) Impacto de procesos de aseguramiento de la calidad sobre las instituciones de educación superior: un estudio en siete países. In: Lemaitre

(ed) Aseguramiento de la calidad en Iberoamérica: educación superior informe 2012 [Quality assurance in Latin America: higher education 2012 report]. Santiago, Chile: Centro Interuniversitario de Desarrollo, CINDA, 117–153.

106.

Zapata

Torre

(2012b) Percepciones sobre impactos del aseguramiento de la calidad (AC) de la educación superior en Iberoamérica [Perceptions about impacts of quality assurance (QA) of higher education in Latin America]. Tercer Congreso de Investigación en Educación Superior. Santiago de Chile: Centro de Investigación Avanzada en Educación, Universidad de Chile.