Developing a brief data literacy measure using CFA and IRT

Abstract

Data literacy is essential in today’s workforce, yet existing data literacy assessments lack brevity, limiting their practical use. This study develops a brief data literacy instrument (BDLI) by streamlining the 25-item Global Data Literacy Benchmark (GDLB). Two studies were conducted. The first study (N = 408) explored item reduction and factor consistency using student participants, while the second study (N = 388) replicated findings using a diverse professional sample. Results indicated that a 12-item BDLI achieved similar factor structure and reliability compared to the original GDLB. Confirmatory factor analysis and item response theory confirmed the BDLI’s consistency and discriminatory power. These results suggest that the BDLI offers a practical tool for assessing an individual’s data literacy skills, which can be used to develop targeted learning opportunities at the individual or organizational level. The value and limitations of the BDLI are discussed.

Keywords

Data literacy item response theory CFA

Introduction

Data literacy (DL) is necessary for people to use and consume data intelligently and also enhances people’s relevance-marketability in the workforce (Brynjolfsson and Mitchell, 2017; Schüller, 2022). DL also improves people’s ability to recognize deceptive information and statements (Bond et al., 2006; George et al., 2013), especially in cases where the information is data-rich and data-derived. This, in turn, enables individuals to hold organizations and governments accountable (DATA-REVOLUTION-GROUP, 2014) for false and deceptive information. Consequently, DL allows individuals to become more informed and, thus more fully participate in society (Letouze, 2016; Unesco, 2019). In organizational settings, data-literate employees are more likely to use data-informed decisions that improve organizational processes, enhance decision quality, and lead to identifying new opportunities (George et al., 2013; McKinsey, 2009). This is what is referred to as a “data-driven culture” (Gupta and George, 2016; Mcafee et al., 2012; Ross et al., 2013).

An initial step in striving for such a culture entails first assessing the extent to which organizational members (upper-level executives, middle-level managers, lower-level workers) make decisions based on the insights gleaned from data (Gupta and George, 2016). Yet, many organizations suffer from a shortage of data literate employees (Manyika et al., 2011) and lack the expertise or assessment tools to evaluate the data literacy of their organization’s members (Cui et al., 2023). Whereas consultants and executive coaches are well-equipped to guide these leaders through the data literacy assessment process, there is a dearth of brief data literacy assessment tools.

The 25-item global data literacy benchmark survey instrument (GDLB) is often used to measure DL (DATATOTHEPEOPLE.ORG, 2018). However, its length may make it unsuitable for inclusion in assessments comprising multiple measures or questionnaires. In these instances, instrument brevity is a commodity. Previous studies indicate that research participants are more willing to complete instruments with fewer items (e.g. Carver, 1997; Robins et al., 2002). Additionally, brief instruments are useful when time is limited or there is risk of losing participants’ interest (e.g. Carver, 1997; Robins et al., 2001; Robinson et al., 2003). Crucially, there is evidence that many brief versions of instruments have nearly the same degree of reliability and validity compared to their long-form versions (e.g. Burisch, 1984, 1997).

This project had two objectives: (1) reduce the number of items required to reproduce the GDLB’s results and (2) explore whether samples of participants across different populations provide similar responses to this briefer version of the GDLB. These objectives were accomplished across two studies. Study 1 explored whether a reduced set of items derived from the GDLB would load on similar factors using a student sample. Study 2 replicated the first study’s results with a large, diverse sample of professionals whose jobs require some degree of data literacy. Lastly, we examined 15-item and 12-item versions of the instrument with a composite of the student and professional samples from Study 1 and Study 2.

Literature review

This literature review provides a comprehensive overview of data literacy, highlighting its importance, challenges, and potential strategies for improvement. By addressing these aspects, stakeholders can better understand how to cultivate data literacy in various contexts.

Importance of data

Describing data as the “new oil” speaks to the newfound importance of data literacy in the 21st century for individuals and society (Holeni, 2020). Stepping back and looking at basic literacy, one finds that many across the world do not possess basic literacy skills, limiting their ability to fully engage and participate in society (Letouze, 2016; Unesco, 2019). The lack of data literacy skills challenges individuals and organizations, impacting their ability to successfully compete in today’s data-centric world. Yet, when did data become important?

The collection and analysis of data represent core steps in a research methodology, supporting discovery for hundreds of years; data predates computers. Researchers recognized the importance of information literacy and computer literacy in the 1970s and 1980s (Noble, 1984; Spitzer et al., 1998), yet the focus on data literacy did not appear until after the turn of the 21st century (Hunt, 2005; Schield, 2005; Stephenson and Schifter Caravello, 2007). Why did this take so long?

The focus on data literacy coincides with the convergence of several key technical capabilities. Continued increases in computer processing power, the ability to store and process vast amounts of data, and the ability to leverage advanced algorithms did not become widely available until around 2007 (Friedman, 2017). And while the conceptual underpinnings of advanced analytics, such as artificial intelligence, trace back to the 1940s (Buchanan, 2005), after this convergence, organizations moved quickly to leverage data and analytics for competitive advantage (Davenport, 2006; Lavalle et al., 2011; Manyika et al., 2011). Organizations scrambled to leverage the new capabilities associated with these advancements.

To manage the rapid growth of data and analytics and develop their data resources as a strategic asset, organizations created the Chief Data Officer (CDO) position (Xu et al., 2016). From the first recognized CDO in 2003 (Zhang et al., 2017), 90% of large organizations now employ a CDO (Gartner, 2016). The US government also recognizes the potential value available from internal data sources, as evidenced by a 2019 law mandating that federal agencies adopt the CDO role (Shibu, 2019).

The widespread adoption of the CDO role presents a unique view of the importance of data versus other major technology initiatives, such as Enterprise Resource Planning (ERP), e-commerce, and the Internet. Each of these technological advances greatly enhanced business capabilities, yet the technology was integrated into the organization and did not result in the creation of persistent C-Suite roles like the CDO (Kirby, 2020).

Further, the importance of data continues to dominate organizational priorities. According to recent Society for Information Management surveys, data-related investments, such as AI, analytics, and data integration, remain top priorities for organizations (Johnson et al., 2023, 2024). Enhancing data literacy skills remains a critical priority for organizations.

Challenges associated data literacy

To improve data literacy, one must contend with several challenges: organizational issues, lack of a clear definition, and a lack of existing data-literate individuals.

Many organizations struggle with data-related initiatives. The vast majority of large-scale technology investments fail to achieve desired results, with a 50%–70% failure rate (Deloitte, 2015; Tabrizi et al., 2019; Waid, 2019). For data initiatives, CDOs report similar failure rates (Bennett, 2016). The rapid increase in CDO appointments suggests companies have taken a “knee jerk” approach to harnessing value from data, rather than taking the strategic approach, recognizing that success is less about the technology and more about strategy, alignment, and creating capabilities within the workforce (Davenport, 2006; Feeny and Willcocks, 1998; Pothier and Condon, 2020; Ross et al., 1996). The exemplars that find success with these initiatives do so with a focus on developing digital strategies aligned with their core business strategies, developing a data-centric culture, and creating a data-literate workforce, leaving their less capable peers struggling to find success (Bean, 2018; Dykes, 2019; McKinsey Analytics, 2019).

The lack of a common definition for data literacy presents another critical challenge. Corrall (2019) provided a model highlighting the interdisciplinary nature of data literacy, presenting a complex view of how various forms a literacy intercept with data literacy at the center. The literature suggests that one who is data literate needs to leverage advanced algorithms (Alpaydin, 2016), be curious (Dykes, 2019; Markham, 2020), possess statistical knowledge (Schield, 2005), understand data privacy/security and ethical issues (Bhargava et al., 2015; Markham, 2020), transform raw data into meaning analytical content (Gummer and Mandinach, 2015), understand data quality (Lawson and Desroches, 2019), and understand the digital data infrastructure (Gray et al., 2018), and possess many more skills.

Also, the skills needed to be data-literate vary depending on a person’s role. The skills needed for a data-literate researcher (Koltay, 2017), will differ from those needed for an accountant (AICPA, 2021; Appelbaum et al., 2021), for a journalist (Gray et al., 2012), for business students (Pothier and Condon, 2020), for statisticians (Schield, 2006), and for public servants (Bonikowska et al., 2019), as a few examples. The definitions and skill requirements describe a person with vast specialized skills, or a unicorn. A simpler approach is needed.

Recognizing the confusion surrounding data and analytics, Kirby (2021) attempted to demystify the subject by presenting a framework placing data literacy in context of numeracy/statistics, problems-solving, and domain knowledge, aimed at addressing a specific business issue. The framework further encourages the selective use of the term “data analytics,” by proposing a scaffolding of data skills with the following levels: Basic Data Skills, Data Analysis, Data Analytics, and Data Science.

A skills gap exists. While no shortage exists of data analytics courses (Almgerbi et al., 2022), the skills gap suggests a greater need for basic data-savvy employees rather than high-end data scientists (Manyika et al., 2011). Davenport and Harris (2017) believe that while digital initiatives must start at the top of the organization, a lack of savvy executives persists (Weill et al., 2021), and the gap between firms capable of leveraging data and digital initiatives and those that can not continues to grow (D'ignazio, 2017; McKinsey Analytics, 2019). Most organizations recognize the disruptive nature of data and analytics, but struggle to respond (Downes and Nunes, 2013).

Improving data literacy

Ridsdale et al. (2015) provide a starting point with their comprehensive synthesis of data literacy strategies and best practices, which recognizes the interdisciplinary nature, providing a conceptual framework for the collection, management, evaluation, and application of data-related tasks. This research defined data literacy as “the ability to collect, manage, evaluate, and apply data in a critical manner,” (Ridsdale et al., 2015) which provides a generic definition to cover the disparate approaches found in existing data literacy educational efforts.

The lack of clarity regarding what it means to be data literate remains. College graduates believe they possess data literacy skills (Pothier and Condon, 2020), while employers continue to report that recent college graduates lack basic data literacy skills (Strauss, 2016). A means of assessing data literacy skills will help.

Consistent with the Ridsdale et al. (2015) framework, the “Global Data Literacy Benchmark” (GDLB) (Crofts, 2018) helps organizations of all types assess their level of data literacy, allowing for a focused approach to a starting. Nath and Kirby (2022) prepared factor analysis on this instrument, identifying three factors: (1) Data Analysis, Storytelling, & Decision-Making, (2) Data Acquisition, Navigation, & Quality Assessment, and (3) Data Wrangling, which involves the modeling and enrichment of data.

Data literacy represents a critical skill to actively participate in today’s data-driven world. It enables individuals and organizations to make informed decisions, drive innovation, and improve efficiency. While there are challenges in improving data literacy, various strategies can be employed to overcome these obstacles. By prioritizing data literacy, educational institutions, organizations, and individuals can better harness the power of data to achieve their goals.

Project method

The aim of this project was to create a briefer and more consistent alternative to the GDLB instrument. To accomplish this aim, we sought to develop a brief data literacy instrument (BDLI) by reducing the number of instrument items, improving item parsimony, and improving item clarity.

Reducing instrument items

Nath and Kirby’s (2022) exploratory analysis of the 25-item GDLB identified three factors: Factor 1 (F1) involves data analysis, storytelling, and decision-making skills; Factor 2 (F2) involves data acquisition, navigation, and quality assessment; Factor 3 (F3) involves data manipulation, modeling and enrichment (“data wrangling”). We sought to reduce the GDLB to 15 items or fewer, using an equal number of items for each factor. Our reasoning for reducing the scale to 15 items was based on previous psychometric evidence in scale reduction and pragmatism. This number of items was chosen because brief instruments comprising two, three, and five-item factors can produce similar reliability and validity as their more numerous item counterparts (e.g. Graham et al., 2011; Muck et al., 2007). In some cases, instruments can be reduced to a few items or a single item (e.g. Benet-Martínez et al., 2002; Campbell et al., 1976) without significant losses in reliability. Pragmatically, five items for each factor allows for some assurance in the similarity between factors for weighting purposes, similarity in completion time requirements, and has intuitive appeal as each factor can be used individually across similar study procedures. Items were selected based on the largest within-factor zero-order correlations from Nath and Kirby’s (2022) exploratory factor analyses.

Improving item parsimony and improving item clarity

The principle of parsimony indicates that it is good psychometric practice to connect a single behavior with a single psychological construct. Some of the items in the original GDLB required participants to rate two behaviors simultaneously. This approach may create ambiguity regarding what the participants are rating. For instance, if a participant were asked to affirm the statement, “I am good at interpreting and cleaning data.” In this question, is the participant evaluating their ability to understand data, arrange data in an interpretable way, or both? If both, are these skills equally weighted in the participants’ rating? Thus, we examined the reduced list of GDLB items that described two or more behaviors and then created a new item or multiple new items that attempted to maintain the conceptual core(s) of the original item(s). Lastly, many psychological constructs are difficult to quantify (e.g. uncanniness; Windsor, 2019). Once operational definitions are established, the language used in the psychometric instrument should clearly connect to the operational definition. Thus, psychometric measures must provide users with clear terms and simple language (Simms, 2008). For this reason, we examined each GDLB’s item syntax and revised where necessary.

The instrument uses a capability approach, whereby a participant rates their ability to perform a given task with questions starting with “I can. . .,” followed by a specific data-related task. A seven-point Likert-type scale measures each response from a 1 (Strongly disagree) to 7 (Strongly agree).

Study 1—Student sample

Participants were recruited across three U.S. universities in 2022. The final student sample comprised 408 students. Of these participants, 57% were women, with 44% in the 19–24 age group, 46% in the 25–44 age group, and 10% aged 45 or older. Approximately 20% of the students reported being in year 5+ of school, some of which came from graduate degree programs. The student sample reported a diverse set of majors, including accounting, finance, marketing, business administration, psychology, biology, clinical counseling, cybersecurity, computer science, economics, and engineering. When asked about the quantitative nature of their field of study, 53% indicated either generally non-quantitative or minimally quantitative, while 47% indicated moderately, highly, or extremely quantitative. When asked about their level of comfort with formal analytics, 33% indicated they were either very uncomfortable or somewhat uncomfortable, 58% indicated they were somewhat comfortable, and 9% indicated they were very comfortable.

Study 2—Workforce sample

Working professional participants were invited to complete a Qualtrics survey shared on Linked-In, Facebook, a post to an academic list-serve, and an invitation to members attending a professional association conference; all surveys were completed in the year 2022. An attention check was added to the survey to ensure diligent effort by respondents. The final sample comprised 388 working professionals. This sample reported a diverse set of occupations, ranging from professors, business executives, accountants, software engineers, commercial realtors, school administrators, project managers, counselors, and some 50+ additional occupations. When asked about the quantitative nature of their work, 28% indicated either generally non-quantitative or minimally quantitative, while 78% indicated moderately, highly, or extremely quantitative.

The working professional sample comprised 49% males and 51% females, but generally older, with only 4% in the 19–24 age group, 66% in the 25–44 age group, and 30% age 45 or older. The workforce sample reported higher levels of comfort with formal analytics, with 26% responding that they were either very uncomfortable or somewhat uncomfortable, 52% indicating they were somewhat comfortable, and 22% indicating they were very comfortable.

Study 1 & 2 results

To test the brief data literacy instrument (BDLI) using the factors as identified by Nath and Kirby (2022), a three-factor a priori exploratory factor analysis was conducted using the 15-question instrument. The factor analysis was prepared with the R statistical computing environment (R-CORE-TEAM, 2023) using varimax rotation and retaining factor loadings exceeding 0.40.

The results indicate that the 15-question instrument reported three factors with a total variance of 67.5%, which compares to the original 25-question instrument’s total variance of 63.7%. The Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy value of 0.95 indicates the sample size for this study is adequate (Shrestha, 2021). The results conceptually replicated the 25-question GDLB’s factor loadings.

Table 1 shows the factor loadings for Study 1 and Study 2, with factor loadings for all participants and partitioned subsets for student and workforce respondents. The loadings indicate expected results, with the exception of questions 5, 8, and 15. Question 5 loads to F1 as expected in Study 2 and total results but loads to F2 in Study 1, both with weak correlations. Question 8 unexpectedly loads to F3 across Study 1 and Study 2. Question 15 loads to F3 as expected, yet with weak correlations. These results suggest that Questions 5, 8, and 15 may require removal, creating a 12-question instrument with four questions per factor.

Table 1.

15 Item-version brief data literacy survey factor loadings for Study 1, Study 2..

		All surveys (n = 796)			Students (n = 408)			Workforce (n = 388)
Ref	Question	F1	F2	F3	F1	F2	F3	F1	F2	F3
Q01	Identify connections across data sources	0.67			0.65			0.71
Q02	Analyze data for decision-making	0.79			0.77			0.80
Q03	Use visualizations to explore data	0.61		0.41	0.59		0.45	0.65
Q04	Assess practical problems with data	0.65	0.42		0.66	0.42		0.67
Q05	Create tables and charts to present data	0.43				0.46		0.55		0.47
Q06	Find appropriate data across multiple sources		0.69			0.74		0.44	0.57
Q07	Determine trustworthiness of data		0.50			0.50			0.44
Q08	Identify errors within data sources			0.60			0.62			0.55
Q09	Access multiple types of data		0.75			0.80			0.62
Q10	Search through multiple types of data sources		0.74			0.75			0.69
Q11	Conduct data cleansing using multiple methods			0.69			0.68			0.68
Q12	Enhance data sources with metadata			0.72			0.71			0.67
Q13	Correct errors within data sources			0.81			0.83			0.75
Q14	Convert data to support analysis needs			0.73			0.74			0.70
Q15	Prepare data for others.			0.50			0.53			0.48

F1—data analysis, storytelling, and decision-making, F2—data acquisition, navigation, and quality assessment, F3—data wrangling. Factors r < 0.40 not shown.

Composite 12-question results

Next, a composite study was prepared to examine how well the 12-question BDLI loads against the three factors as compared to the 15-question instrument, using the composite sample of 796 responses (408 students and 388 workforce participants), using a confirmatory factor analysis method to expand upon the results of Study 1 and 2.

Factor analysis for the briefer 12-question instrument resulted in three factors with a total variance of 72.9%, which compared to the original 25-question instrument’s total variance of 63.7%. The Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy value of 0.93 indicates the sample size for this study is adequate (Shrestha, 2021).

Table 2 reports the factor loadings for the composite study, with factor loadings for all participants and partitioned subsets for student and workforce respondents. Across the full sample and for the student and workforce subsets, the 12-question instrument produced loadings as expected with reasonable correlations to each factor (r > for all loadings). The superior factor loadings and parsimony of the 12-question instrument support its use. Table 3 reports the mean scores and standard deviation for each item and composite scores for all participants and student and workforce subsets. In the ANCOVA, after controlling for student versus workforce, results showed no significant difference between F1 and F2 (p > 0.05) while showing significant differences between the scores for F1 and F3 (p < 0.001) and between scores for F2 and F3 (p < 0.001). No difference was found between student and workforce scores (p > 0.05). Consistent with Nath and Kirby (2022), across all participants, the highest scores were reported for F2, followed by F1 and F3, respectively.

Table 2.

Brief data literacy survey 12 item-version factor loadings.

		All surveys (n = 796)			Students (n = 408)			Workforce (n = 388)
Ref	Question	F1	F2	F3	F1	F2	F3	F1	F2	F3
Q01	Identify connections across data sources	0.68			0.65			0.73
Q02	Analyze data for decision-making	0.78			0.78			0.79
Q03	Use visualizations to explore data	0.61	0.40		0.59		0.44	0.66
Q04	Assess practical problems with data	0.65			0.67	0.42		0.68
Q06	Find appropriate data across multiple sources		0.69			0.73		0.45	0.56
Q07	Determine trustworthiness of data		0.51			0.51			0.42
Q09	Access multiple types of data		0.76			0.80			0.63
Q10	Search through multiple types of data sources		0.75			0.77			0.69
Q11	Conduct data cleansing using multiple methods			0.69			0.68			0.68
Q12	Enhance data sources with metadata			0.73			0.72			0.68
Q13	Correct errors within data sources			0.78			0.80			0.74
Q14	Convert data to support analysis needs			0.73			0.74			0.69

F1—data analysis, storytelling, and decision-making, F2—data acquisition, navigation, and quality assessment, F3—data wrangling. Factors r < 0.40 not shown.

Table 3.

Brief data literacy survey results.

		All surveys (n = 796)		Students (n = 408)		Workforce (n = 388)
Ref	Question	M	SD	M	SD	M	SD
Q01	Identify connections across data sources	5.38	1.25	5.39	1.24	5.37	1.26
Q02	Analyze data for decision-making	5.56	1.17	5.60	1.09	5.51	1.25
Q03	Use visualizations to explore data	5.20	1.32	5.18	1.35	5.22	1.28
Q04	Assess practical problems with data	5.45	1.19	5.47	1.17	5.43	1.20
Q06	Find appropriate data across multiple sources	5.54	1.13	5.63	1.13	5.43	1.12
Q07	Determine trustworthiness of data	5.42	1.28	5.54	1.28	5.29	1.27
Q09	Access multiple types of data	5.49	1.17	5.66	1.15	5.31	1.17
Q10	Search through multiple types of data sources	5.62	1.15	5.75	1.12	5.48	1.17
Q11	Conduct data cleansing using multiple methods	4.95	1.41	4.79	1.51	5.12	1.27
Q12	Enhance data sources with metadata	4.24	1.59	3.92	1.66	4.58	1.44
Q13	Correct errors within data sources	4.68	1.48	4.50	1.58	4.87	1.34
Q14	Convert data to support analysis needs	5.03	1.41	4.83	1.48	5.23	1.29
Composite data literacy score		5.21	0.95	5.19	0.97	5.24	0.92
F1—data analysis, storytelling, and decision-making		5.40	1.04	5.41	1.02	5.38	1.07
F2—data acquisition, navigation, and qual. assessment		5.52	0.99	5.65	0.99	5.38	0.98
F3—data wrangling		4.72	1.26	4.51	1.35	4.95	1.11

CFA and IRT data analysis plan

In Study 1 and Study 2, exploratory factor analysis was used to replicate the three factors found in Nath and Kirby (2022) using our revised items with two unique samples. To examine the instrument using a more rigorous set of tests, the brief data literacy instrument (BDLI) was evaluated using an integrated confirmatory factor analysis (CFA) and item response theory (IRT) approach. Decision-making about whether to further reduce the total number of items in each factor was determined based on CFA model fit indices. If item reduction improved the overall model fit of each factor, the reduced model was kept. The decision to remove each item was based on both Study 1 and Study 2’s EFA results and the CFA standardized error for each item within each factor. For instance, item Q05 was removed from factor 1 because it showed the weakest correlation with the 5-item model in the EFA analysis in Study 1 and Study 2 and the largest standardized error in the CFA (R code is available upon request). Once the factors achieved satisfactory model fit indices, a CFA analysis was conducted on all three factors to determine the total instrument model fit, allowing for correlated factors. Finally, an IRT was performed to replicate CFA results and to determine item spread.

CFA analysis

Lin (2022) recommendations for CFA in R (R-CORE-TEAM, 2023) using the lavaan R package (Rosseel, 2012) were used to examine the model fit indices of the three BDLI factors. Following the Bean and Bowen (2021) approach with insights from Lin (2022), we examined four commonly used fit measures: model chi-square, comparative fit index (CFI), Tucker-Lewis index (TLI), and the root-mean-square error of approximation (RMSEA). The model chi-square test is a significance test used to assess exact sample model fit to the population model. However, it is often considered to be over-sensitive to sample size (Bean and Bowen, 2021; Lin, 2022), so approximate fit indices are often used in addition to estimate model fit. Standards in model fit cutoffs outlined in West et al. (2012) were used. The comparative fit index (CFI) and Tucker-Lewis Index (TLI) are used to evaluate approximate model fit using an incremental approach, with a standard cutoff of 0.95 or higher. The root-mean-square error of approximation (RMSEA) is an approximate model fit index that uses an absolute approach. The RMSEA point estimate for close fit is ⩽.05, and reasonable fit is ⩽.08, with a 90% confidence interval range of 0.01–0.08 (Lin, 2022).

Based on the EFA results of Study 1 and Study 2, each five-item and four-item version of the three BDLI factors were analyzed. According to the model chi-square results, Factors 1 and 2 did not show exact model fit for either the 5-item or 4-item model (p < 0.05). According to the model chi-square results, Factor 3 showed exact model fit (p > 0.05) for both the 5-item and 4-item models. All models showed adequate CFI and TLI scores (>0.95). The 4-item models for Factors 2 and 3 showed reasonable approximate fit (Lin, 2022), but the upper confidence intervals exceeded recommended cutoffs. Factor 3, for both the 5-item and 4-item models, met all recommended cutoffs. However, the 4-item model did show improvement for each fit index, which makes it preferred to the 5-item model.

The three-factor 4-item model of the BDLI did not show exact model fit but did show adequate CFI and TLI values (>0.95), and the RMSEA value showed reasonable approximate model fit (0.056) as well as acceptable 90% confidence intervals. These results led us to use IRT analyses on each factor’s 4-item version. See Table 4 for details.

Table 4.

CFA model fit indices for the three factors.

Model	χ2	CFI	TLI	RMSEA [90% CI]
F1 (Q01, Q02, Q03, Q04, Q05)	17.733	0.993	0.987	0.057 [0.057,0.030]
F1 (Q01, Q02, Q03, Q04, Q05)	p = 0.003	0.993	0.987	0.057 [0.057,0.030]
F1 (Q01, Q02, Q03, Q04)	6.844	0.997	0.991	0.055 [0.014,0.103]
F1 (Q01, Q02, Q03, Q04)	p = 0.033	0.997	0.991	0.055 [0.014,0.103]
F2 (Q06 + Q07 + Q08 + Q09 + Q10)	47.857	0.978	0.955	0.104 [0.078, 0.132]
F2 (Q06 + Q07 + Q08 + Q09 + Q10)	p < 0.001	0.978	0.955	0.104 [0.078, 0.132]
F2 (Q06 + Q08 + Q09 + Q10)	11.740	0.994	0.981	0.078 [0.039, 124]
F2 (Q06 + Q08 + Q09 + Q10)	p = 0.003	0.994	0.981	0.078 [0.039, 124]
F3 (Q11 + Q12 + Q13 + Q14 + Q15)	9.777	0.998	0.995	0.035 [0.000, 0.067]
F3 (Q11 + Q12 + Q13 + Q14 + Q15)	p = 0.082	0.998	0.995	0.035 [0.000, 0.067]
F3 (Q11 + Q12 + Q13 + Q14)	0.137	1.000	1.000	0.000 [0.000, 0.019]
F3 (Q11 + Q12 + Q13 + Q14)	p = 0.934	1.000	1.000	0.000 [0.000, 0.019]
F1 + F2 + F3 4-item model	178.391	0.978	0.971	0.056 [0.047,0.065]
F1 + F2 + F3 4-item model	p < 0.001	0.978	0.971	0.056 [0.047,0.065]

IRT data analysis

Following guidelines from Bean and Bowen (2021), we conducted IRT analyses for each factor using the R statistical computing environment (R-CORE-TEAM, 2023). We assessed item fit for each of the factors using the mirt (Chalmers, 2012) and GAIPE (Bean and Bowen, 2021) R packages. Using the index S-χ2, we examined IRT slope and location parameters, and visually inspected the IRT plots, as recommended by Bean (2021) and Kılıç et al. (2023).

Table 5 provides the item graded response model parameter estimates for items within the three factors. The slope parameter (a) provides a measure of how well each question discriminates the measured level of each factor trait. Using the threshold of a > 1.75 as suggested by Baker (2001), the results indicate the questions within each factor provide a very high level of discrimination. Q02 (Analyze data for decision-making) produces the greatest discrimination for Factor 1, while Q09 (Access multiple types of data) and Q13 (Correct errors within data sources) produce the greatest discrimination for Factors 2 and 3, respectively. Thus, these overall results provide good evidence that the BDLI is a potential tool for assessing an individual’s DL strengths and weaknesses. Appendix A contains the 12-item BDLI.

Table 5.

Item graded response model parameter estimates.

Question	a (se)	b₁ (se)	b₂ (se)	b₃ (se)	b₄ (se)	b₅ (se)	b₆ (se)
F1—data analysis, storytelling, and decision-making
Identify connections across data sources	2.69 (0.17)	−2.85 (0.19)	−2.29 (0.13)	−1.64 (0.09)	-1.00 (0.06)	−0.11 (0.05)	1.08 (0.07)
Analyze data for decision-making	3.80 (0.30)	−2.90 (0.20)	−2.34 (0.13)	−1.67 (0.09)	−1.06 (0.06)	−0.23 (0.05)	0.85 (0.06)
Assess practical problems with data	2.23 (0.14)	−3.04 (0.21)	−2.27 (0.13)	−1.50 (0.09)	−0.88 (0.07)	0.16 (0.06)	1.22 (0.08)
Create tables and charts to present data	3.09 (0.21)	−3.34 (0.27)	−2.44 (0.14)	−1.64 (0.09)	−1.01 (0.06)	−0.11 (0.05)	1.00 (0.07)
F2—data acquisition, navigation, and quality assessment
Find appropriate data across multiple sources	3.00 (0.20)	−3.88 (0.48)	−2.68 (0.17)	−1.79 (0.09)	−1.12 (0.07)	−0.23 (0.05)	0.99 (0.07)
Determine trustworthiness of data	1.77 (0.12)	−4.00 (0.40)	−2.80 (0.19)	−1.82 (0.11)	−1.11 (0.08)	−0.20 (0.06)	1.14 (0.09)
Access multiple types of data	3.54 (0.26)	−3.72 (0.46)	−2.42 (0.14)	−1.63 (0.08)	−0.99 (0.06)	−0.21 (0.05)	0.94 (0.06)
Search through multiple types of data sources	3.30 (0.23)	−3.40 (0.32)	−2.52 (0.15)	−1.75 (0.09)	−1.16 (0.07)	−0.36 (0.05)	0.82 (0.06)
F3—data wrangling
Conduct data cleansing using multiple methods	2.57 (0.16)	−2.84 (0.18)	−1.90 (0.10)	−1.13 (0.07)	−0.47 (0.06)	0.33 (0.05)	1.36 (0.08)
Enhance data sources with metadata	2.30 (0.14)	−2.07 (0.12)	−1.22 (0.08)	−0.59 (0.06)	0.09 (0.05)	0.86 (0.07)	1.90 (0.10)
Correct errors within data sources	3.35 (0.23)	−2.27 (0.12)	−1.42 (0.08)	−0.88 (0.06)	−0.27 (0.05)	0.50 (0.05)	1.48 (0.08)
Convert data to support analysis needs	3.00 (0.20)	−2.46 (0.14)	−1.71 (0.09)	−1.14 (0.07)	−0.67 (0.06)	0.25 (0.05)	1.36 (0.08)

The (b) parameters in Table 5 report the theta level at which an individual has a 50% likelihood of selecting the given point on the Likert scale. Figures 1 –3 provide the item probability function plots for each factor. These plots demonstrate the steady progression, by level ability, for each item.

Discussion

The present study aimed to test the consistency of the factors identified by Nath and Kirby (2022), using a shortened version of the GDBL. The factor analysis results across the diverse student and workforce samples demonstrate the briefer instrument produces the same underlying factors found in the GDBL. These findings were consistent across student and workforce samples, using the 15-question and 12-question instruments, with better results from the 12-question instrument.

The three factors of F1—Data Analysis, Storytelling, and Decision-Making, F2—Data Acquisition, Navigation, and Quality Assessment, and F3—Data Wrangling identify the work needed to leverage data for greater insight and decision-making. F1 involves applying the appropriate analytical techniques, algorithms, and models, and creating compelling messages to support the analysis and decision-making process. F2 involves the sourcing, collection, and assessment of the veracity of the data, which is often stored in multiple locations, systems, and formats. F3 involves cleaning and modeling processes necessary to support the analysis and decision-making processes. These factors reflect the core capabilities needed for someone that is data literate. Furthermore, the BDLI provides a briefer and more rigorously validated instrument. This instrument could be used to explore both basic research questions regarding data literacy and assessment questions regarding individuals’ data literacy. Our results support use of the 12-item BDLI by academics and corporate trainers to measure data literacy.

As organizations pursue increased data literacy skills, they are faced with confusion about what constitutes data literacy (Bhargava et al., 2015). Firms recognized as leaders in building data-driven cultures recognize the need and invest heavily in comprehensive, ongoing training (Brown et al., 2019; Landi, 2019), while many firms recognize the need, but struggle with knowing how to start (Columbus, 2014).

The findings of significantly lower scores for F3 across student and workforce participants, suggest that basic data wrangling skills are generally lacking. F1 scores suggest participants feel confident making data-driven decisions from provided data. F2 suggests that participants feel confident collecting data from various sources, but generally lack the skills to clean, model, and enrich data, limiting their ability to conduct meaningful analysis. This finding suggests academics and practitioners should create learning experiences to enhance basic data wrangling skills to enhance overall data literacy skills.

Data analytics involves a continuum of skills ranging from basic knowledge of data to advanced analytics involving complex algorithms, which numeracy and statistical skills (Kirby, 2021). The terms “data analysis” and “data analytics” are often used interchangeably, yet they refer to opposite ends of the skills continuum, which leads to confusion and contributes to the lack of traction for organizations seeking to improve skills. Individuals in possession of the full range of skills are highly desirable yet in short supply. Organizations must define the relevant data literacy skills for individuals to be successful in their roles, then create learning pathways to develop these individuals. The BDLI provides a starting point for educators and company trainers.

In developing the briefer data literacy instrument, we recognize the limitation that this research does not seek to measure the results of data literacy on a particular outcome. Future research should examine how data literacy impacts results across different industries and disciplines.

A composite study uses the same instrument data as used in Study 1 and Study 2 but removes the three weakest questions. We recognize that presenting the 12-question instrument to an audience may be preferred, it seems reasonable that an individual’s response to each question is independent and not influenced by the remaining questions.

Footnotes

Appendix A

Acknowledgements

The authors would like to acknowledge Dr. Ravi Nath, who was instrumental in the early work on this effort. Dr. Nath passed in January 2024.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Compliance with ethical standards

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.

Informed consent

Informed consent was obtained from all individual adult participants included in the study.

ORCID iD

Joseph Kirby

Author biographies

Joseph Kirby, Assistant Professor at Bellevue University, specializes in the impact of information leadership on firm performance. With a DBA from Creighton University and 30 years of industry experience in finance, systems, and operations, his work bridges academic research and practical applications in organizational leadership and data-driven decision-making.

Jerome Lewis, Associate Professor of Psychology at Bellevue University, specializes in Victimology, Moral Psychology, and Political Psychology. His research explores the victim role and reactions to victims, and he leads the Victim Role Research Group (VRRG), recruiting undergraduate and graduate research assistants for collaborative studies.

References

AICPA (2021) CPA Evolution Model Curriculum. AICPA.

Almgerbi

De Mauro

Kahlawi

, et al (2022) A systematic review of data analytics job requirements and online-courses. Journal of Computer Information Systems 62(2): 422–434.

Alpaydin

(2016) Machine Learning. The New AI, Cambridge, MA, MIT Press.

Appelbaum

Showalter

Sun

, et al (2021) A framework for auditor data literacy: A normative position. Accounting Horizons 35(2): 5–25.

Baker

(2001) The Basics of Item Response Theory. ERIC.

Bean

Bowen

(2021) Item response theory and confirmatory factor analysis: Complementary approaches for scale development. Journal of Evidence-Based Social Work 18(4): 597–618.

Bean

(2021) Using R for Social Work Research. Columbus, OH: Ohio State University Press.

Bean

(2018) Big data executive survey 2018.

Benet-Martínez

Leu

Lee

, et al (2002) Negotiating biculturalism: Cultural frame switching in biculturals with oppositional versus compatible cultural identities. Journal of Cross-Cultural Psychology 33(5): 492–516.

10.

Bennett

(2016) How Chief Data Officers Can Tackle Formidable Roadblocks Including People, Culture and Internal Resistance. Gartner: Gartner Website.

11.

Bhargava

Deahl

Letouzé

, et al (2015) Beyond data literacy: Reinventing community engagement and empowerment in the age of data.

12.

Bond JR

F C

Depaulo

(2006) Accuracy of deception judgments. Personality and Social Psychology Review 10(3): 214–234.

13.

Bonikowska

Sanmartin

Frenette

(2019) Data Literacy: What It Is and How to Measure It in the Public Service. Statistics Canada.

14.

Brown

Herring

, et al. (2019) The Analytics Academy: Bridging the Gap Between Human and Artificial Intelligence. McKinsey.

15.

Brynjolfsson

Mitchell

(2017) What can machine learning do? Workforce implications. Science 358(6370): 1530–1534.

16.

Buchanan

(2005) A (very) brief history of artificial intelligence. AI Magazine, 26, 53.

17.

Burisch

(1984) Approaches to personality inventory construction: A comparison of merits. American Psychologist 39(3): 214–227.

18.

Burisch

(1997) Test length and validity revisited. European Journal of Personality 11(4): 303–315.

19.

Campbell

Converse

Rodgers

(1976) The Quality of American Life: Perceptions, Evaluations, and Satisfactions. New York, NY: Russell Sage Foundation.

20.

Carver

(1997) You want to measure coping but your protocol's too long: Consider the brief COPE. International Journal of Behavioral Medicine 4(1): 92–100.

21.

Chalmers

(2012) Mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software 48(6): 1–29.

22.

Columbus

(2014) 84% Of enterprises see big data analytics changing their industries’ competitive landscapes in the next year. Available at: https://www.forbes.com/sites/louiscolumbus/2014/10/19/84-of-enterprises-see-big-data-analytics-changing-their-industries-competitive-landscapes-in-the-next-year/#6a5aa2317de1

23.

Corrall

(2019) Repositioning data literacy as a mission-critical competence.

24.

Crofts

(2018) Global data literacy benchmark.

25.

Cui

Chen

Lutsyk

, et al (2023) Data literacy assessments: A systematic literature review. Assessment in Education Principles Policy and Practice 30(1): 76–96.

26.

D'ignazio

(2017) Creative data literacy: Bridging the gap between the data-haves and data-have nots. Information Design Journal 23: 6–18.

27.

DATA-REVOLUTION-GROUP (2014) Accessible data: open data, accountability and data literacy [Online]. Available at: https://www.undatarevolution.org/data-revolution/ (accessed 22 February 2020).

28.

DATATOTHEPEOPLE.ORG (2018) We’re data to the people [Online]. Available at: https://www.datatothepeople.org/ (accessed 22 February 2020).

29.

Davenport

Harris

(2017) Competing on Analytics: Updated, With a New Introduction: The New Science of Winning. Boston, MA: Harvard Business Press.

30.

Davenport

(2006) Competing on analytics. Harvard Business Review 84: 98.

31.

Deloitte (2015) Your guide to a successful ERP journey.

32.

Downes

Nunes

(2013) Big bang disruption. Harvard Business Review 91(1): 44–56.

33.

Dykes

(2019) Data Curiosity: How To Cultivate An Inquisitive Workforce [Online]. Forbes. Available at: https://www.forbes.com/sites/brentdykes/2019/10/10/data-curiosity-how-to-cultivate-an-inquisitive-workforce/#7d72b1735471 (accessed 11 October 2019).

34.

Feeny

Willcocks

(1998) Core IS capabilities for exploiting information technology. Sloan Management Review 39: 9–21.

35.

Friedman

(2017) Thank You for Being Late: An Optimist's Guide to Thriving in the Age of Accelerations (Version 2.0, With a New Afterword). New York, NY: Picador/Farrar Straus and Giroux.

36.

Gartner (2016) Gartner Estimates That 90 Percent of Large Organizations Will Have a Chief Data Officer by 2019 [Online]. Stamford, CN: Gartner. Available at: https://www.gartner.com/en/newsroom/press-releases/2016-01-26-gartner-estimates-that-90-percent-of-large-organizations-will-have-a-chief-data-officer-by-2019 (accessed 15 August 2019).

37.

George

Carlson

Valacich

(2013) Media selection as a strategic component of communication. MIS Quarterly 37(4): 1233–1251.

38.

Graham

Nosek

Haidt

, et al (2011) Mapping the moral domain. Journal of Personality and Social Psychology 101(2): 366–385.

39.

Gray

Chambers

Bounegru

(2012) The Data Journalism Handbook: How Journalists Can Use Data to Improve the News. Sebastopol, CA: O’Reilly Media, Inc.

40.

Gray

Gerlitz

Bounegru

(2018) Data infrastructure literacy. Big Data & Society 5(2): 2053951718786316.

41.

Gummer

Mandinach

(2015) Building a conceptual framework for data literacy. Teachers College Record 117(4): n4–22.

42.

Gupta

George

(2016) Toward the development of a big data analytics capability. Information Management 53(8): 1049–1064.

43.

Holeni

(2020) Data is the new oil of the 21st century. CIGFARO Journal (Chartered Institute of Government Finance Audit and Risk Officers) 21(4): 12–14.

44.

Hunt

(2005) The challenges of integrating data literacy into the curriculum in an undergraduate institution. IASSIST Quarterly/International Association for Social Science Information Service and Technology 28(2): 12–12.

45.

Johnson

Maurer

Torres

, et al (2024) A preview of the 2023 SIM IT trends study. MIS Quarterly Executive 22: 83–124.

46.

Johnson

Torres

Maurer

, et al (2023b) The 2022 SIM IT issues and trends study. MIS Quarterly Executive 22: 55–98.

47.

Kılıç

Koyuncu

Uysal

(2023) Scale development based on item response theory: A systematic review. International Journal of Psychology and Educational Studies 10(1): 209–223.

48.

Kirby

(2021) Demystifying and developing a framework for “data analytics”. Economics and Business Journal: Inquiries and Perspectives 12(1): 54–68.

49.

Kirby

(2020) The impact of functional TMT technology leadership roles on firm financial outcomes. In: UNIVERSITY, C. (ed.).

50.

Koltay

(2017) Data literacy for researchers and data librarians. Journal of Librarianship and Information Science 49(1): 3–14.

51.

Landi

(2019) Amazon plans to spend $700M to retrain a third of its workforce for data, analytics roles [Online]. Available at: https://www.cnbc.com/2019/07/11/amazon-plans-to-spend-700-million-to-retrain-a-third-of-its-workforce-in-new-skills-wsj.html (accessed December 2020)

52.

Lavalle

Lesser

Shockley

, et al (2011) Big data, analytics and the path from insights to value. MIT Sloan Management Review 52: 21–32.

53.

Lawson

RHT

Desroches

(2019) The Impact of Big Data on Finance Now and in the Future. Montvale, NJ: Institute of Management Accountants.

54.

Letouze

(2016) Should ‘data literacy’ be promoted? [Online]. Available at: https://unstats.un.org/unsd/undataforum/blog/should-data-literacy-be-promoted/ (accessed 20 February 2020).

55.

Lin

(2022) Confirmatory Factor Analysis (CFA) in R with lavaan. UCLA: Statistical Consulting Group.

56.

Manyika

Chui

Brown

, et al (2011) Big Data: The Next Frontier for Innovation, Competition, and Productivity. McKinsey Global Institute.

57.

Markham

(2020) Taking data literacy to the streets: Critical pedagogy in the public sphere. Qualitative Inquiry 26(2): 227–237.

58.

Mcafee

Brynjolfsson

Davenport

, et al (2012) Big data: The management revolution. Harvard Business Review 90: 60–68.

59.

McKinsey (2009) Hal Varian on how the Web Challenges Managers. McKinsey Technology, Media & Telecommunications.

60.

McKinsey Analytics (2019) Catch Them if you can: How Leaders in Data and Analytics Have Pulled Ahead. McKinsey Analytics.

61.

Muck

Hell

Gosling

(2007) Construct validation of a short five-factor model instrument. European Journal of Psychological Assessment 23(3): 166–175.

62.

Nath

Kirby

(2022) An empirical examination of the factors of data literacy. Journal of Digital Science 4(1): 3–20.

63.

Noble

(1984) Computer literacy and ideology. Teachers College Record 85(4): 602–614.

64.

Pothier

Condon

(2020) Towards data literacy competencies: Business students, workforce needs, and the role of the librarian. Journal of Business & Finance Librarianship 25(3–4): 123–146.

65.

R-CORE-TEAM (2023) A language and environment for statistical computing [Computer software].

66.

Ridsdale

Rothwell

Smit

, et al (2015) Strategies and best practices for data literacy education: Knowledge synthesis report.

67.

Robinson

Gibson

Khan

, et al (2003) Validity of two oral health-related quality of life measures. Community Dentistry and Oral Epidemiology 31(2): 90–99.

68.

Robins

Hendin

Trzesniewski

(2001) Measuring global self-esteem: Construct validation of a single-item measure and the Rosenberg self-esteem scale. Personality and Social Psychology Bulletin 27(2): 151–161.

69.

Robins

Trzesniewski

Tracy

, et al (2002) Global self-esteem across the life span. Psychology and Aging 17(3): 423–434.

70.

Rosseel

(2012) Lavaan: An R package for structural equation modeling. Journal of Statistical Software 48(2): 1–36.

71.

Ross

Beath

Goodhue

(1996) Develop long-term competitiveness through IT assets. Sloan Management Review 38: 31–42.

72.

Ross

Beath

Quaadgras

(2013) You may not need big data after all. Harvard Business Review 91: 90+.

73.

Schield

(2005) Information literacy, statistical literacy, data literacy. IASSIST Quarterly 28: 6.

74.

Schield

(2006) Statistical literacy survey analysis: Reading graphs and tables of rates and percentages. In: Proceedings of the Sixth International Conference on Teaching Statistics, 2006. International Statistical Institute and International Association for for Statistical Education [Online]. Available at: http://www.stat.auckland.ac.nz/~iase

75.

Schüller

(2022) Data and AI literacy for everyone. Statistical Journal of the IAOS 38(2): 477–490.

76.

Shibu

(2019) New law mandates chief data officers at federal agencies. govloop.

77.

Shrestha

(2021) Factor analysis as a tool for survey analysis. American Journal of Applied Mathematics and Statistics 9(1): 4–11.

78.

Simms

(2008) Classical and modern methods of psychological scale construction. Social and Personality Psychology Compass 2(1): 414–433.

79.

Spitzer

Eisenberg

Lowe

(1998) Information Literacy: Essential Skills for the Information Age. Syracuse University, New York: ERIC Clearinghouse on information and technology.

80.

Stephenson

Schifter Caravello

(2007) Incorporating data literacy into undergraduate information literacy programs in the social sciences. Reference Services Review 35(4): 525–540.

81.

Strauss

(2016) These are the Skills Bosses Say New College Grads do not have. Forbes.

82.

Tabrizi

Lam

Girard

, et al (2019) Digital transformation is not about technology. Harvard Business Review 13: 1–6.

83.

UNESCO (2019) UNESCO - Literacy Website [Online]. Available at: https://www.unesco.org/en/literacy (accessed 22 February 2020).

84.

Waid

(2019) Solving The Last Mile Problem For Data Science Project Success. Forbes.

85.

Weill

Stephanie

Shah

(2021) Does your C-Suite have enough digital smarts. MIT Sloan Management Review, 62(3).

86.

West

Taylor

(2012) Model fit and model selection in structural equation modeling. In: Hoyle

(ed.) Handbook of Structural Equation Modeling. New York, NY: Guilford Press, pp. 209–231.

87.

Windsor

(2019) What is the uncanny? British Journal of Aesthetics 59(1): 51–65.

88.

Zhan

Huang

, et al (2016) The Value of Chief Data Officer presence on Firm Performance. PACIS, p.213.

89.

Zhang

Lee

Wang

, et al (2017) Chief data officer appointment and origin: A theoretical perspective.