AI Literacy Scale for Undergraduate Design Students: Development and Validation

Conceptualization of AI Literacy

AI literacy was initially articulated as the ability to understand, evaluate, and effectively use AI systems to solve problems (Long & Magerko, 2020). Early models focused on conceptual knowledge of AI, basic tool operation and awareness of AI applications. More recent work has broadened the construct to include ethical awareness, critical evaluation of AI outputs and appreciation of AI’s societal impacts, moving beyond a purely technical view (Biagini, 2024; Li et al., 2024; Ng et al., 2021a).

General AI literacy frameworks developed for K-12 and STEM education typically foreground computational thinking, algorithmic reasoning and subject-specific problem solving (Chan, 2024; Kim et al., 2022). These models have proved useful for structuring curricula in mathematics or computer science, where problems are relatively well-defined and success criteria are stable. However, they implicitly assume that tasks can be decomposed into discrete steps and evaluated against fixed rubrics. This assumption sits uneasily with the conditions of design education, which is characterized by ill-structured problems, open-ended briefs, and the need to balance functional, aesthetic and ethical considerations in situated contexts (Park, 2025; Romero et al., 2018).

AI Literacy in Design Education

Within design education, AI literacy must extend well beyond tool proficiency. It demands co-creative engagement with intelligent systems, the capacity to critically evaluate AI-generated outputs, and a reflexive understanding of AI’s socio-cultural implications (Copper et al., 2024; Evangelidis et al., 2024; Ng et al., 2024; Schauer et al., 2025). Empirical studies in architecture, art, and media programs have begun to explore how generative AI supports ideation, style exploration, and rapid prototyping (Michalak & Ellixson, 2025; Sako, 2024). These contributions show that AI can expand the search space of design solutions and accelerate iteration, but they typically rely on course-specific rubrics or qualitative reflections rather than validated measurement models.

A key limitation of existing AI literacy frameworks, when imported into design schools, is that they under-specify competencies that are constitutive of design practice. UNESCO-inspired models, for instance, highlight human-centered mindset, ethics, and system-level awareness, yet say little about authorship, aesthetic ownership or visual bias in AI-generated imagery—issues that are pervasive in communication and product design (Miao & Shiohira, 2024; Romero et al., 2018). Likewise, ABCE-type scales capture attitudes, behaviors, cognition, and ethics at a general level but do not account for iterative prototyping, multimodal expression or human–AI co-creation in studio workflows (Ng et al., 2024; Zhang et al., 2023). As a result, technical and ethical knowledge may be measured, but capacities such as orchestrating AI across the design process, integrating user feedback into AI configurations or framing AI within complex socio-technical systems remain largely invisible.

AI Literacy Assessment Methods

In parallel with conceptual work, a growing number of instruments have been developed to assess AI-related competencies. Self-report scales have been proposed for general student populations (Ng et al., 2024; Yim & Su, 2024), for pre-service teachers (Ning et al., 2025) and medical students (Karaca et al., 2021). These tools typically examine combinations of knowledge, self-efficacy, ethical awareness and behavioral intentions, and their factor structures provide useful baselines for operationalizing AI literacy in formal education (Ng et al., 2021b, 2022). Beyond self-report, performance-based and contextualized assessment methods have emerged, including project rubrics and scenario-based tasks that evaluate how learners actually deploy AI tools in authentic activities (Chiu et al., 2022; Copper et al., 2024). While these approaches are rich in contextual detail, they are often difficult to standardize across institutions and may lack rigorous psychometric validation.

From the perspective of design education, both strands of work share two important limitations. First, they are predominantly rooted in non-creative disciplines and rarely capture design-specific competencies such as aesthetic reasoning, speculative exploration, narrative framing or system-oriented problem definition (Romero et al., 2018; Schauer et al., 2025). Second, they typically treat AI as an add-on technology used at isolated stages—such as idea generation or data analysis—rather than as a co-creative and systemic actor embedded throughout the design process.

The present study responds to these gaps by constructing the AI-CIEI Scale, which explicitly aligns its dimensions with the epistemic logic of design education while retaining the psychometric rigor of established AI literacy instruments. The next section introduces the theoretical framework that links UNESCO’s AI competency domains, design-specific theories and the four dimensions of the AI-CIEI model.

Cognition and Collaboration (CC)

CC combines UNESCO’s Understanding and Being domains with insights from human–AI co-creativity. It captures students’ conceptual grasp of AI principles (e.g., data–model–output relationships, generative mechanisms and limitations) and their ability to position AI as a creative collaborator rather than a black-box tool. Building on cognitive models of AI literacy (Long & Magerko, 2020) and empirical studies of AI-augmented design practice (Gmeiner et al., 2023; Song et al., 2024), this dimension emphasizes whether students can form accurate mental models of how AI systems work, interpret and interrogate AI-generated artifacts, and negotiate agency and responsibility between human designers and AI systems in collaborative workflows. In this way, CC goes beyond generic “AI knowledge” constructs by foregrounding relational and collaborative cognition specific to design studios.

Implementation and Integration (II)

II primarily operationalizes UNESCO’s Doing domain and aspects of System Design within the procedural structure of Design Thinking and information literacy. It focuses on the practical capacities required to embed AI tools into end-to-end design workflows, including selecting appropriate tools for different stages (research, ideation, prototyping, testing), formulating and refining prompts, orchestrating data and integrating AI outputs with traditional design software and techniques (Brown, 2008; Lu & Petiot, 2014; Weisgrau et al., 2024; Zhou & Schofield, 2024). II is thus distinguished from generic “application” dimensions by emphasizing workflow-level integration and orchestration across iterative cycles, rather than isolated tool skills.

Ethics and Critical Judgment (EJ)

EJ builds directly on UNESCO’s Being domain and critical AI literacy research. It conceptualizes ethics as an ongoing practice of judgment embedded in concrete design decisions, rather than as abstract awareness. In design contexts, this includes identifying algorithmic bias, representational harms and stereotyping in AI-generated visuals; reasoning about copyright, data provenance and consent in training data; and considering privacy, accessibility and environmental sustainability of AI-enabled solutions (Jobin et al., 2019; Mittelstadt et al., 2016; Ng et al., 2024; Wu et al., 2023). By linking these debates to specific design projects, EJ emphasizes that design students must not only “know about” AI ethics but also diagnose and address ethical tensions in their own AI-mediated practice.

Innovation and Systemic Thinking (IS)

IS fuses UNESCO’s System Design domain with Systems Thinking and strategic design perspectives. It focuses on how students mobilize AI to approach complex, open-ended and multi-stakeholder problems: framing design briefs at the system level rather than as isolated artifacts; identifying feedback loops, trade-offs and unintended consequences of AI-infused solutions; coordinating AI use across project stages, touchpoints, and team members; and exploring future-oriented scenarios and speculative applications of AI in services, products and environments (Buchanan, 1992; Deroncele-Acosta et al., 2025; Senge & Sterman, 1992). Conceptually, IS is therefore distinguished from Design Thinking’s iterative cycles: whereas Design Thinking mainly structures process-level iteration, IS emphasizes system-level framing, causal reasoning and long-term consequences in AI-mediated design ecosystems.

Taken together, the four AI-CIEI dimensions constitute a discipline-specific reinterpretation of UNESCO’s AI competency architecture for design education. CC recasts Understanding and Being through human–AI partnership; II embeds Doing and parts of System Design within iterative design workflows; EJ grounds Being in situated ethical judgment; and IS extends System Design into strategic, systemic and future-oriented design practice. This theoretically anchored framework guided the construction of the AI-CIEI items and the subsequent validation of its four-factor structure. For transparency, we note that the initial conceptualization stage briefly considered a five-dimensional structure, with a provisional fifth dimension labeled AI-driven Design Creativity and Innovative Thinking (AI-supported ideation expansion in studio work); expert feedback suggested that these competencies were better treated as part of Innovation and Systemic Thinking (IS) rather than as a standalone construct, and the final framework therefore retains four dimensions for conceptual parsimony. A conceptual diagram (Figure 1) visually summarizes these linkages between UNESCO domains, design-specific theories and the AI-CIEI dimensions.

OPEN IN VIEWER

Figure 1.

Theoretical framework of the AI-CIEI model.

Operationally, these four dimensions were translated into item clusters in the AI-CIEI Scale. Items under Cognition and Collaboration focus on students’ understanding of AI principles and their perceived partnership with AI in design tasks, whereas Implementation and Integration items tap into the use of AI tools across different design stages and their orchestration with conventional software. Ethics and Critical Judgment items probe how students diagnose and respond to ethical tensions in AI-mediated projects, and Innovation and Systemic Thinking items assess their ability to frame AI-enabled solutions at the system level and anticipate long-term consequences. Table 1 presents the final item pool organized according to these four dimensions.

OPEN IN VIEWER

Table 1.

AI-CIEI Scale for Undergraduate Design Students.

Dimension/items	Reference
Cognition and Collaboration (CC)	Gmeiner et al. (2023), Long and Magerko (2020), Miao and Shiohira (2024), Song et al. (2024)
1. I understand the basic principles and processes of AI in generating visual, 3D, and interaction-based design.( CC1)
2. I can distinguish the core differences between generative AI and traditional design tools. (CC2)
3. I can identify differences in creative logic and detail between AI-generated and human-designed work. (CC3)
4. I believe AI should serve as a collaborative tool, with designers responsible for creative direction and outcomes. (CC4)
5. I understand the roles and limitations of AI across design stages such as research, ideation, prototyping, and testing. (CC5)
6. I can integrate my professional understanding to define AI’s role and contribution in the design process. (CC6)
Implementation and Integration (II)
7. I understand how the type of training data influences the design style generated by AI.(II1)	Brown (2008), Lu and Petiot (2014), Weisgrau et al. (2024), Zhou and Schofield (2024)
8. I am proficient in using AI tools for visual, 3D, and interaction-based design expression. (II2)
9. I can generate design content aligned with specific themes by optimizing prompts. (II3)
10. I can create initial drafts using AI and refine details with traditional software. (II4)
11. I can generate personalized design solutions using user data. (II5)
12. I can integrate professional software and AI tools to improve efficiency. (II6)
13. I explore and adopt emerging AI tools in design projects. (II7)
Ethics and Critical Judgment (EJ)	D. Ng et al. (2024), Jobin et al. (2019), Miao and Shiohira (2024), Mittelstadt et al. (2016), Wu et al. (2023)
14. I understand that AI tools may involve gender, cultural, or esthetic bias.(EJ1)
15. I am concerned about the sources and copyright of AI training data. (EJ2)
16. I value privacy and data security in AI-generated content. (EJ3)
17. I can identify and improve design issues that may cause discrimination or user difficulty in AI-generated outputs. (EJ4)
18. I believe designers should be responsible for the social impact of AI-generated designs. (EJ5)
19. I can assess the environmental sustainability of AI-generated design solutions in terms of materials and techniques. (EJ6)
20. I examine whether AI outcomes are fair to diverse populations and seek better tools or algorithms when necessary. (EJ7)
21. I assess whether AI-generated content fits the needs and habits of different user groups. (EJ8)
Innovation and Systemic Thinking (IS)	Buchanan (1992), Deroncele-Acosta et al. (2025), Kyselova (2024), Senge and Sterman (1992)
22. I can systematically plan how AI integrates and collaborates throughout the design process. (IS1)
23. I can assess the value and application scenarios of AI in various projects. (IS2)
24. I can identify risks such as data bias or design homogenization in AI-based design. (IS3)
25. I can facilitate AI collaboration and information sharing in interdisciplinary design teams. (IS4)
26. I can extract innovative directions from AI-generated solutions to enhance user experience. (IS5)
27. I can optimize AI parameters or data based on user feedback. (IS6)
28. I can evaluate the cost-effectiveness of AI in terms of time, human effort, and resources. (IS7)

Stage 1: Scale Development and Item Generation

Based on the conceptual definition of “AI literacy among undergraduate design students,” this study developed a theoretical framework comprising multiple sub-dimensions, integrating international research on AI literacy, and the specific characteristics of design education. The initial qualitative phase primarily involved systematic literature analysis and expert interviews. Drawing especially from the competency structure outlined in UNESCO’s AI Competency Framework for students (Miao & Shiohira, 2024), and combining insights from pedagogical practices in design disciplines, the research team generated an initial pool of measurement items aligned with the theoretical framework. This resulted in an initial item bank consisting of five provisional dimensions and 40 items. Importantly, this five-dimensional structure served as a coverage-oriented conceptual scaffold at the item-generation stage. Following expert content validation and cognitive interviews, experts recommended consolidating the framework into a more parsimonious four-dimensional structure for conceptual clarity and cross-subfield applicability, which was implemented prior to the quantitative validation.

Initial Item Development

To improve the content validity and disciplinary relevance of the scale, an expert validation procedure was implemented. Seven experts (two AI scholars, one expert in education, and four experts in design) completed a structured feedback form on the dimensional structure, item clarity, and applicability across design domains. For each item, they judged dimensional relevance (“belongs/does not belong”), clarity (“clear/not clear”), and retention (“retain/revise/delete”). Item-level content validity indices (I-CVI) were calculated as the proportion of experts indicating that an item belonged to its intended dimension, and a scale-level index (S-CVI/Ave) was obtained by averaging the I-CVI values across all items. All items showed satisfactory I-CVI values (minimum I-CVI = 0.83, range = 0.83–1.00), and the S-CVI/Ave was 0.94, indicating adequate content representativeness of the preliminary item pool.

Based on the experts’ feedback, the research team systematically revised the scale in four ways (Boateng et al., 2018): (1) merging overlapping dimensions to improve conceptual clarity, (2) removing items with limited applicability or redundancy, (3) rewording items to ensure relevance across different design subfields, and (4) refining ambiguous or overly technical wording to enhance measurability.

This process reuslted in pre-EFA qualitative purification (40 → 28) and dimensional refinement (5 → 4). In line with the experts’ structured recommendations, the 40-item pool underwent an item-level decision process (retain/revise/delete/merge) to improve cross-domain relevance and reduce redundancy. At the dimension level, experts noted that the initial five-dimensional structure was somewhat verbose and recommended refining it into four clearer dimensions—Cognition and Collaboration (CC), Implementation and Integration (II), Ethics and Critical Judgment (EJ), and Innovation and Systemic Thinking (IS)—thereby motivating a consolidation of overlapping constructs prior to EFA.

Correspondingly, the original fifth provisional dimension (AI-driven Design Creativity & Innovative Thinking), which was initially designed to capture AI-supported creative ideation and innovation in design, was integrated into Innovation and Systemic Thinking (IS), because experts and students consistently perceived substantial conceptual overlap, and several items under the provisional fifth dimension were judged as overly specific or redundant.

To ensure full traceability, we provide (a) an item-level record of the 12 items removed/merged during the 40 → 28 purification, including decision rationales (Appendix Table AX), and (b) a dimension-mapping table documenting how the five provisional dimensions were consolidated into the final four dimensions (Appendix Table AY).

Following expert validation, five undergraduate students from different design majors (two from visual communication design, two from product design, and one from environmental design) participated in cognitive interviews to assess item comprehensibility and practical relevance. Individual 30 to 40-min sessions employed a think-aloud protocol with follow-up probes (Lu & Petiot, 2014; Willis, 2004). The interviews highlighted issues of linguistic clarity, domain applicability, semantic redundancy, and user-centered phrasing, which informed further targeted revisions. After these two rounds of refinement—expert evaluation with CVI analysis and student cognitive interviews—the pool was reduced from 40 to 28 items, and the framework was consolidated from five provisional dimensions to four retained dimensions (see Table 1; traceable records are provided in Appendix Tables AX and AY).

Prior to the formal data collection, a pilot study was conducted to evaluate the clarity, content validity, and internal consistency of the initial AI literacy scale. In mid-to-late Feb 2025, the research team distributed the questionnaire to undergraduate design students across different regions in China via online platforms. A total of 105 responses were collected, among which 88 were considered valid. During the pilot testing, participants generally reported clear understanding of each item, indicating good content and face validity of the instrument for the target population (Hardesty & Bearden, 2004; Haynes et al., 1995). As shown in Table 2, all four subscales demonstrated excellent internal consistency, with Cronbach’s α coefficients ranging from .885 to .915. These values exceed the commonly accepted threshold of 0.70 for adequate internal consistency (Hair et al., 1998; Nunnally, 1994). These results suggest that the scale demonstrates strong internal reliability and is suitable for subsequent formal administration and structural validation. These preliminary results established a strong foundation for structural validation through EFA and CFA.

OPEN IN VIEWER

Table 2.

Reliability Coefficients (Cronbach’s α) of Each Dimension in the Pilot Study (n = 88).

Dimension	Number of items	Cronbach’s α
Cognition and Collaboration (CC)	6	.915
Implementation and Integration (II)	7	.910
Ethics and Critical Judgment (EJ)	8	.908
Innovation and Systemic Thinking (IS)	7	.885

Participants and Data Collection

This study adopted a cross-sectional survey design targeting undergraduate design students in China. Participants were recruited from eight public universities located in four provinces (Zhejiang, Hunan, Jiangsu, and Yunnan). These institutions all host established design programs (e.g., product/industrial design, visual communication, interaction design, environmental design). A network-based convenience cluster sampling strategy was used: tutors at these universities were contacted in advance and invited to assist with data collection by distributing an online survey link to their students in design-related courses. The link was shared via institutional learning management systems and class WeChat groups. Participation was entirely voluntary. Before accessing the questionnaire, all students were informed about the purpose of the study, the anonymity and confidentiality of their responses, and their right to withdraw at any time. No course credit or monetary incentives were provided.

The survey was administered and collected using Wenjuanxing (www.wjx.cn), a widely used online questionnaire platform in China. Data collection took place in March 2025. A total of 600 questionnaires were distributed. After excluding cases with substantial missing values, straight-line or patterned responses, and completion times below one-third of the median (indicating potential inattentive responding), 485 valid responses were retained for analysis, yielding an effective response rate of 80.8%. The AI-CIEI scale employed a 5-point Likert format ranging from “Strongly Disagree (1)” to “Strongly Agree (5),” allowing for fine-grained assessment of AI literacy among design students (DeVellis & Thorpe, 2021).

To enhance the representativeness of the findings, demographic data were also collected (Boateng et al., 2018). Among the valid participants, 51.1% were male and 48.9% female. Participants were distributed across academic years: 34.6% freshmen, 25.6% sophomores, 20.4% juniors, and 19.4% seniors. The sample encompassed a diverse range of design disciplines: 30.9% in Visual Communication Design, 15.5% in Product Design (including Industrial Design), 11.5% in Interaction Design, 17.1% in Environmental Design, 8.7% in Digital Media, 6.8% in Art and Technology, and 9.5% in other fields. This disciplinary diversity supports the scale’s applicability across various domains of design education. Additionally, data on AI-related training and experience were collected. 4.3% of respondents had never received AI training; 33.6% had training 1 to 2 times, 41.6% had training 3 to 5 times, and 20.4% had training more than 5 times. Regarding usage experience, 35.5% had used AI for less than 6 months, 23.9% for 6 months to 1 year, 27.0% for 1 to 2 years, and 13.6% for more than 2 years.

Taken together, the sample covers multiple academic years, diverse design majors, and eight public universities across four provinces. It therefore provides a reasonably broad empirical basis for the subsequent Exploratory Factor Analysis (EFA), Confirmatory Factor Analysis (CFA), and reliability and validity testing, while remaining a non-probability sample that is most representative of undergraduate design students in similar institutional contexts. A comprehensive summary of the participants’ demographic characteristics is presented in Table 3.

OPEN IN VIEWER

Table 3.

Demographic Characteristics of the Sample (n = 485).

Category	Subgroup	Frequency	Percentage (%)
Gender	Male	248	51.1
	Female	237	48.9
Academic year	Freshman (year 1)	168	34.6
	Sophomore (year 2)	124	25.6
	Junior (year 3)	99	20.4
	Senior (year 4)	94	19.4
Major	Visual communication design	150	30.9
	Product design (Industrial design)	75	15.5
	Interaction design	56	11.5
	Environmental design	83	17.1
	Digital media	42	8.7
	Art and technology	33	6.8
	Others	46	9.5
Training times	Never	21	4.3
	1–2 times	163	33.6
	3–5 times	202	41.6
	More than 5 times	99	20.4
Usage duration	Less than 6 months	172	35.5
	6 months–1 year	116	23.9
	1–2 years	131	27
	More than 2 years	66	13.6

Stage 2: Exploratory and Confirmatory Factor Analyses

Exploratory Factor Analysis (EFA)

To evaluate the structural validity of the AI-CIEI Scale for Undergraduate Design Students, an Exploratory Factor Analysis (EFA) was first conducted on the initial pool of 28 items using data from 485 valid responses. Preliminary tests confirmed the data’s suitability for factor analysis, with a high Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy (KMO = 0.934). Bartlett’s Test of Sphericity was also significant (χ² = 6535.767, df = 378, p < .001), indicating sufficient correlations among items for factor extraction (Osborne & Costello, 2005; see Table 4). To address the potential risk of common method variance associated with the exclusive use of self-report measures, Harman’s single-factor test was also conducted by loading all 25 items of the final AI-CIEI Scale into an unrotated principal component analysis.

OPEN IN VIEWER

Table 4.

KMO and Bartlett’s Test of Sphericity for the 28-Item AI-CIEI.

Test item	Value
KMO measure of sampling adequacy	0.934
Bartlett’s test of sphericity
Approx. chi-square	6,535.767
Degrees of freedom (df)	378.000
Significance (Sig.)	<.001

In accordance with best practices for identifying theoretically correlated latent constructs, a common factor model rather than principal components analysis was adopted. Specifically, principal axis factoring (PAF) with direct oblimin rotation was employed to uncover the underlying factor structure. To determine the appropriate number of factors, multiple criteria were used. In addition to the Kaiser criterion and theoretical expectations, Horn’s parallel analysis was conducted based on 1,000 randomly generated datasets, using the 95th percentile of random eigenvalues as the comparison benchmark. Four factors with eigenvalues greater than 1 (8.538, 2.991, 2.918, and 1.961) were extracted, jointly accounting for 58.600% of the total variance (52.056% after extraction). Horn’s parallel analysis further supported the retention of four factors: the first four observed eigenvalues (8.538, 2.991, 2.918, and 1.961) were all larger than the corresponding 95th-percentile eigenvalues from the randomly generated data, whereas the fifth observed eigenvalue fell below its random counterpart (see Figure 3). This pattern provides additional empirical justification for the four-factor solution adopted in this study. These factors corresponded closely to the hypothesized dimensions and were labeled as: (1) Cognition and Collaboration, (2) Implementation and Integration, (3) Ethics and Critical Judgment, and (4) Innovation and Systemic Thinking. Inspection of the pattern and structure matrices indicated a clean factor solution, with no salient cross-loadings (all secondary loadings < 0.30).

OPEN IN VIEWER

Figure 3.

Parallel analysis for AI-CIEI.

Internal consistency for each provisional subscale was satisfactory, with Cronbach’s α values ranging from .850 to .888. For each subscale, corrected item–total correlations (CITCs) were inspected together with factor loadings. Within the Ethics and Critical Judgment dimension, CITCs ranged from approximately 0.32 to 0.70; EJ4 and EJ8 showed the lowest CITCs (around 0.32–0.38), clearly below the other EJ items (all ≥0.66). Within the Innovation and Systemic Thinking dimension, IS3 displayed the weakest item–total statistics, with the lowest squared multiple correlation and a marked increase in Cronbach’s α if the item it was deleted. Consistent with these item-level diagnostics, inspection of the PAF pattern matrix indicated that EJ4, EJ8, and IS3 exhibited the weakest primary loadings on their respective factors (0.389, 0.423, and 0.376, respectively) and accounted for limited unique variance beyond conceptually similar items. In line with recommended criteria for item retention (Nunnally, 1994), these three items were removed to enhance the clarity of the factor structure and to improve internal consistency. Following their removal, Cronbach’s α for the Ethics and Critical Judgment subscale increased from .850 to .880, and Cronbach's α for Innovation and Systemic Thinking increased from .872 to .904. Overall, these results support a robust four-dimensional factor structure. The final version of the scale comprising four dimensions and 25 items, served as the basis for the subsequent confirmatory factor analysis (CFA; see Table 5).

OPEN IN VIEWER

Table 5.

Exploratory Factor Analysis of the Initial 28-item AI-CIEI (PAF with Oblimin Rotation).

Factor/items	Factor loading	CITC	Eigenvalue	Explained variance (%)	Cumulative variance (%)	Cronbach’s α
CC			8.538	30.492	30.492	.888
CC1	0.687	.693
CC2	0.739	.685
CC3	0.723	.713
CC4	0.741	.715
CC5	0.739	.693
CC6	0.769	.719
II			2.991	10.683	41.175	.881
II1	0.732	.675
II2	0.731	.664
II3	0.702	.657
II4	0.719	.660
II5	0.727	.671
II6	0.719	.675
II7	0.697	.667
EJ			2.918	10.423	51.598	.850
EJ1	0.725	.697
EJ2	0.710	.677
EJ3	0.770	.690
EJ4	0.389	. 382
EJ5	0.703	.676
EJ6	0.718	.663
EJ7	0.699	.666
EJ8	0.423	. 319
IS			1.961	7.002	58.600	.872
IS1	0.709	.750
IS2	0.688	.732
IS3	0.376	. 228
IS4	0.638	.709
IS5	0.641	.709
IS6	0.670	.712
IS7	0.663	.728

Note. Factor loadings are pattern coefficients from principal axis factoring with direct oblimin rotation. CITC = corrected item–total correlation within each provisional subscale. Only factor loadings ≥ .30 are shown. Bold italicized rows indicate the three items (EJ4, EJ8, IS3) that were retained in the 28-item EFA table for diagnostic transparency but were later removed before CFA due to low primary loadings and relatively weak item–total statistics. Cronbach’s α values refer to the 28-item provisional subscales.

As an initial diagnostic of common method variance, Harman’s single-factor test was performed by loading all 25 items of the final scale into an unrotated principal component analysis. The first factor accounted for 33.70% of the total variance, which is well below the 40% to 50% threshold commonly taken to indicate serious common method bias (Podsakoff et al., 2003). Moreover, four factors with eigenvalues greater than 1 jointly explained 63.32% of the variance, consistent with the hypothesized four-factor structure rather than a dominant general factor. These findings suggest that common method variance is unlikely to be a dominant concern in the present data.

Confirmatory Factor Analysis (CFA)

To further validate the structural validity of the AI-CIEI Scale for Undergraduate Design Students, this study conducted a Confirmatory Factor Analysis (CFA) using AMOS 24.0. Before estimating the model, univariate normality was examined for the 25 retained items. Descriptive statistics indicated moderate negative skewness, with skewness values ranging from −1.20 to −0.61, and kurtosis values ranging from −0.75 to 1.02. These distributions do not represent extreme departures from normality and are typical for positively oriented attitudinal items. In line with recommendations that maximum likelihood (ML) estimation performs adequately with five-point Likert-type items when distributions are approximately normal and sample size is sufficient, the items were treated as approximately continuous and the CFA model was estimated using ML (Finney & DiStefano, 2006; Hair et al., 2010; Rhemtulla et al., 2012).

Based on 485 valid responses, the hypothesized four-factor model was specified with four latent variables—Cognition and Collaboration (CC), Implementation and Integration (II), Ethics and Critical Judgment (EJ), and Innovation and Systemic Thinking (IS)—and the model was estimated using ML. Model fit was evaluated using standard fit indices. The model and fit results are shown in Figure 4.

OPEN IN VIEWER

Figure 4.

Confirmatory factor analysis fit results.

Following the model fit evaluation criteria proposed by Hair et al. (2010), multiple fit indices were examined, including χ²/df, GFI, AGFI, CFI, TLI, RMSEA, and SRMR (see Table 6). The results indicate that the model demonstrated an excellent fit: χ²/df = 1.223 (below the threshold of 3), GFI = 0.950, AGFI = 0.939, CFI = 0.990, and TLI = 0.989, all exceeding the recommended cutoff of 0.90. Furthermore, RMSEA =0.021 and SRMR = 0.0308 were well below 0.05, indicating a high degree of consistency between the hypothesized model and the observed data.

OPEN IN VIEWER

Table 6.

Model Fit Indices of the Four-Factor CFA Model (n = 485).

Fit index	Criteria	Value
χ2	Lower is better	328.926
χ2/df	<3.000	1.223
GFI	>0.900	0.950
RMSEA	<0.080	0.021
SRMR	<0.080	0.0308
AGFI/CFI/TLI	>0.9000	0.939/0.990/0.989

Regarding internal consistency, the Cronbach’s α coefficients for the four dimensions were .888 (CC), .881 (II), .880 (EJ), and .904 (IS), all substantially exceeding the commonly accepted threshold of 0.70. As shown in Table 7, the Composite Reliability (CR) values for all dimensions were above 0.880, further supporting the scale’s reliability (Bagozzi & Kimmel, 1995).

OPEN IN VIEWER

Table 7.

Reliability and Validity Test of the Scale (n = 485).

Factor	Item	Unstandardized estimate	S.E.	Z-value	p-Value	Standardized estimate	Cronbach’s α	CR	AVE
CC	CC1	1				0.746	.888	0.888	0.569
	CC2	.974	0.062	15.710	***	0.727
	CC3	1.058	0.063	16.794	***	0.772
	CC4	1.081	0.065	16.631	***	0.769
	CC5	1.001	0.062	16.145	***	0.742
	CC6	1.063	0.064	16.609	***	0.768
II	II1	1				0.725	.881	0.881	0.515
	II2	.948	0.064	14.813	***	0.711
	II3	.992	0.068	14.588	***	0.707
	II4	.965	0.066	14.621	***	0.708
	II5	.999	0.067	14.910	***	0.720
	II6	1.009	0.067	15.060	***	0.728
	II7	.994	0.066	15.061	***	0.722
EJ	EJ1	1				0.745	.880	0.880	0.551
	EJ2	.993	0.061	16.279	***	0.758
	EJ3	.996	0.062	16.065	***	0.753
	EJ5	.95	0.061	15.574	***	0.727
	EJ6	.951	0.060	15.850	***	0.748
	EJ7	.911	0.059	15.441	***	0.720
IS	IS1	1				0.805	.903	0.904	0.61
	IS2	1.032	0.053	19.472	***	0.803
	IS4	.941	0.051	18.451	***	0.764
	IS5	.95	0.051	18.627	***	0.769
	IS6	.92	0.05	18.400	***	0.763
	IS7	1.016	0.054	18.815	***	0.781

Note. CR = composite reliability; AVE = average variance extracted.

***

Indicates p < .001.

In terms of convergent validity, three widely recognized criteria were satisfied: (1) all standardized factor loadings exceeded 0.50 (Bailey & Ball, 2006); (2) the Average Variance Extracted (AVE) for each dimension exceeded 0.50 (Bagozzi & Kimmel, 1995); and (3) all CR values surpassed 0.70 (Bagozzi, 1981; see Table 7). These results provide strong evidence of the scale’s convergent validity.

For discriminant validity, the Fornell–Larcker criterion was used, which compares the square root of each factor’s AVE with its inter-factor correlations (see Table 8). The findings showed that the square root of AVE for each construct was greater than its correlations with other constructs, confirming satisfactory discriminant validity.

OPEN IN VIEWER

Table 8.

Discriminant Validity Test of Latent Factors (n = 485).

Factor	IS	EJ	II	CC
IS	0.781
EJ	0.401	0.742
II	0.405	0.342	0.718
CC	0.608	0.412	0.366	0.754

Note. Bolded values on the diagonal are the square roots of the AVE values for each construct. Off-diagonal values represent the inter-construct correlation coefficients.

To further verify the robustness and superiority of the four-factor model, this study also tested and compared four alternative models: a one-factor model (all items loaded onto a single latent variable), a two-factor model (combining CC and II into one factor and EJ and IS into another), a three-factor model (with CC as a standalone factor and the remaining three factors merged), and the hypothesized four-factor model. As shown in Table 9, the four-factor model demonstrated the best fit across all indices (e.g., χ²/df = 1.223, CFI = 0.990, TLI = 0.989, RMSEA = 0.021, SRMR = 0.0308), meeting or exceeding all recommended thresholds. In contrast, the other models performed substantially worse, with the one-factor model in particular showing poor fit (χ²/df = 10.955, RMSEA = 0.143). These findings suggest that the four-factor structure aligns more closely with the data and possesses greater explanatory power and discriminability than simplified alternatives.

OPEN IN VIEWER

Table 9.

Competing Model Comparison: Results of Confirmatory Factor Analysis.

Model	χ2	df	χ2/df	GFI	AGFI	CFI	TLI	RMSEA	SRMR
Four-factor model	328.926	269	1.223	0.95	0.939	0.990	0.989	0.021	0.0308
Three-factor model	1730.793	275	6.294	0.668	0.608	0.761	0.739	0.105	0.2135
Two-factor model	2520.632	275	9.166	0.57	0.491	0.631	0.598	0.130	0.2016
One-factor model	3012.535	275	10.955	0.525	0.439	0.551	0.510	0.143	0.1353

Note. GFI = Goodness-of-Fit Index; AGFI = Adjusted Goodness-of-Fit Index; CFI = Comparative Fit Index; TLI = Tucker-Lewis Index; RMSEA = root mean square error of approximation; SRMR = standardized root mean square residual.

In summary, the CFA results provide strong empirical support for the proposed four-factor structure of the AI Literacy Scale for Undergraduate Design Students. The scale exhibits sound psychometric properties, including robust structural validity, internal consistency, convergent validity, and discriminant validity, and can serve as a reliable instrument for future empirical research and theoretical testing.

Measurement Invariance Across Gender

Before interpreting gender-related differences, measurement invariance of the four-factor AI-CIEI model was examined using multi-group CFA in AMOS. A sequence of increasingly constrained models was estimated for male and female students (M1–M5: configural, metric, scalar, factor variance–covariance, and strict invariance), and model comparisons were evaluated using changes in CFI and TLI, with absolute decreases in CFI ≤ 0.01 and decreases in TLI ≤ 0.05 indicating invariance (Vandenberg & Lance, 2000). As shown in Table 10, all models exhibited good absolute fit, and the changes in CFI and TLI between adjacent models were very small (all |ΔCFI| ≤ 0.003, |ΔTLI| ≤ 0.002). These results support configural, metric, and scalar invariance of the AI-CIEI across gender, indicating that the measurement and factor-level structural properties of the scale are highly comparable for male and female design students.

OPEN IN VIEWER

Table 10.

Measurement Invariance Across Gender for the AI-CIEI Scale.

Model	χ2	df	χ2/df	RMSEA	TLI	ΔTLI	CFI	ΔCFI
M1	645.334	538.000	1.200	0.020	0.980	—	0.983	—
M2	679.315	559.000	1.215	0.021	0.979	−0.001	0.980	−0.003
M3	695.930	584.000	1.192	0.020	0.981	0.002	0.982	0.002
M4	712.300	594.000	1.199	0.020	0.981	0.000	0.981	−0.001
M5	722.270	619.000	1.167	0.019	0.984	0.003	0.983	0.002

Note. M1 = configural invariance (baseline, unconstrained); M2 = metric invariance (factor loadings equal); M3 = scalar invariance (factor loadings and item intercepts equal); M4 = factor variance–covariance invariance (factor variances and covariances equal); M5 = strict invariance (factor loadings, item intercepts, and residual variances equal).

Stage 3: Criterion-Related Validity

To examine criterion-related (concurrent) validity within the nomological network, a MIMIC-type structural equation model was specified instead of separate regressions on factor scores. Gender, academic year, major, AI training frequency, and AI usage duration were included as observed exogenous variables. The four AI literacy dimensions: Cognition and Collaboration (CC), Implementation and Integration (II), Ethics and Critical Judgment (EJ), and Innovation and Systemic Thinking (IS)—were modeled as latent factors with their respective items as indicators, consistent with the confirmatory factor analysis (CFA) results. Direct paths were specified from each exogenous variable to each latent factor, while correlations among exogenous variables and residual covariances among the four latent factors were freely estimated. The model was estimated using AMOS with maximum likelihood estimation, and criterion-related validity was evaluated based on global fit indices and standardized path coefficients. The structural model showed excellent fit to the data, χ²(374) = 412.86, χ²/df = 1.10, p = .081, GFI = 0.946, AGFI = 0.933, CFI = 0.997, TLI = 0.996, IFI = 0.997, RMSEA = 0.015. These indices indicate that introducing the background and experience variables did not compromise the good measurement properties of the four-factor AI-CIEI model.

Table 11 reports the standardized path coefficients from the exogenous variables to each AI literacy dimension. Overall, academic year emerged as the only consistent predictor. Higher academic year was associated with slightly higher scores on Cognition and Collaboration (β = .102, p < .05), Implementation and Integration (β = .123, p < .01), Ethics and Critical Judgment (β = .135, p < .05), and Innovation and Systemic Thinking (β = .105, p < .05). In contrast, gender, major, AI training frequency, and AI usage duration showed only weak and non-significant associations with the four latent dimensions (all |β| ≤ .04, ps > .20). This pattern suggests that, once measurement error is taken into account, demographic and exposure variables explain only a modest proportion of variance in AI literacy, and that other individual or contextual factors may play a more substantial role.

OPEN IN VIEWER

Table 11.

Regression Analysis (n = 485).

Predictor	CC (β, p)	II (β, p)	EJ (β, p)	IS (β, p)
Gender	.018, n.s.	.014, n.s.	.032, n.s.	.029, n.s.
Academic year	.102*, .024	.123**, .009	.135**, .004	.105*, .027
Major	.013, n.s.	.019, n.s.	−.038, n.s.	.013, n.s.
Training times	.009, n.s.	.017, n.s.	−.007, n.s.	.018, n.s.
Usage duration	.032, n.s.	.041, n.s.	.038, n.s.	.016, n.s.

Note. Entries are standardized path coefficients (β) from the MIMIC SEM model.

n.s. = non-significant.

*

p < .05. **p < .01.

Summary of Key Findings

The present study developed and validated the AI-CIEI Scale as a discipline-specific instrument for assessing AI literacy among undergraduate design students. Drawing on UNESCO’s AI Competency Framework, Design Thinking, Systems Thinking, and human–AI co-creativity, a four-factor structure was theorized and empirically supported: Cognition and Collaboration (CC), Implementation and Integration (II), Ethics and Critical Judgment (EJ), and Innovation and Systemic Thinking (IS; Brown, 2008; Long & Magerko, 2020; Miao & Shiohira, 2024; Senge & Sterman, 1992). Exploratory and confirmatory factor analyses indicated that this structure showed good fit and satisfactory psychometric properties, with all dimensions demonstrating high internal consistency, convergent validity, and discriminant validity.

These results suggest that AI literacy in design education is not reducible to general “AI knowledge” or isolated tool skills. Instead, it comprises intertwined cognitive, procedural, ethical, and systemic competencies that align with the studio-based, project-driven nature of design curricula (Romero et al., 2018; Schauer et al., 2025). By making these dimensions explicit and measurable, the AI-CIEI Scale provides an empirically grounded framework for describing how design students currently engage with AI in their studies and complements existing general-purpose AI literacy instruments developed in non-creative disciplines (Ng et al., 2024; Yim & Su, 2024).

Background and Experience Variables

The MIMIC structural model further examined how demographic and experience-related variables are associated with the four AI-CIEI dimensions. Academic year emerged as a small but consistent predictor: students in higher years reported slightly higher levels of Cognition and Collaboration, Implementation and Integration, Ethics and Critical Judgment, and Innovation and Systemic Thinking. This pattern is consistent with research suggesting that prolonged engagement with studio projects and exposure to more complex briefs can gradually deepen students’ conceptual understanding and strategic use of digital tools, including AI, even when explicit AI instruction is limited (Park, 2025; Romero et al., 2018).

By contrast, gender, major, AI training frequency, and AI usage duration showed weak and non-significant associations with all four dimensions once measurement error was controlled. This finding complicates assumptions that short-term workshops or self-reported usage time automatically translate into higher AI literacy, echoing concerns that decontextualized, tool-centric training has limited impact on deeper competencies (Copper et al., 2024; Ng, Leung, Chu, & Qiao, 2021b). It may reflect that many training opportunities remain tool-centered and weakly integrated into design practice, or that students’ self-directed use of AI tends to focus on a narrow set of functions (e.g., quick image generation or text polishing) rather than the broader competencies captured by the AI-CIEI Scale. From a design-education perspective, the modest explanatory power of these variables indicates that program-level factors—such as how AI is embedded in studio teaching, assessment, and critique—may be more influential than individual background characteristics.

Educational Implications and a Tentative Developmental Pathway

Within these empirical boundaries, the AI-CIEI results nonetheless have several implications for curriculum design. First, the clear differentiation between the four dimensions supports the idea that AI literacy for designers should be cultivated as a profile of competencies rather than a single score (Ng et al., 2024). Programs can use the scale diagnostically to identify whether particular cohorts are relatively stronger in, for example, Implementation and Integration but weaker in Ethics and Systemic Thinking, and then target pedagogical interventions accordingly. Second, the dimension structure and item content suggest a plausible sequence of learning goals, even though the present data do not track longitudinal development. A tentative developmental pathway can therefore be proposed as a conceptual application of the model rather than as a direct empirical finding. At an introductory level, teaching activities can focus on strengthening Cognition and Collaboration by helping students build accurate mental models of AI mechanisms and experiment with human–AI co-creation in low-stakes tasks, in line with calls for AI literacy to move beyond tool operation to collaborative sense-making (Gmeiner et al., 2023; Long & Magerko, 2020). At an intermediate level, project-based modules can emphasize implementation and integration, requiring students to orchestrate AI tools across research, ideation, prototyping and testing and to reflect on workflow design (Brown, 2008; Weisgrau et al., 2024). More advanced courses can foreground ethics and critical judgment, asking students to analyze bias, authorship, sustainability, and accessibility issues in AI-mediated design solutions and to justify their decisions in critiques and documentation, building on UNESCO’s ethical AI recommendations and critical AI literacy work (Miao & Shiohira, 2024; Ng et al., 2024). Finally, capstone or strategic-design projects can be used to foster innovation and systemic thinking by engaging students with multi-stakeholder problems, speculative scenarios and service or system-level briefs in which AI is treated as one actor among many (Buchanan, 1992; Deroncele-Acosta et al., 2025). In this way, the AI-CIEI Scale offers a conceptually grounded roadmap for scaffolding AI literacy in design curricula, while recognizing that the proposed pathway requires future longitudinal and experimental research for empirical verification.

Limitations and Future Research

Several limitations should be acknowledged. First, the data were collected using a network-based convenience sample of design students from eight public universities in four provinces (Zhejiang, Hunan, Jiangsu, and Yunnan). Although this multi-site sampling strategy enhances sample diversity across institutions and specializations, the findings cannot be assumed to generalize to all design programs or institutional types in China. Future research could employ stratified or probability sampling to further examine the robustness of the AI-CIEI across different regions, tiers of universities and program structures, and, where possible, extend the validation to international contexts through cross-national comparison and measurement invariance testing.

Second, the study relied on self-report measures and cross-sectional data. As such, the findings speak to perceived competencies and correlational associations rather than to actual performance or developmental change. Future work could combine the AI-CIEI Scale with performance-based tasks, portfolio analysis or studio critique assessment to triangulate students’ AI literacy.

Third, the background variables included in the MIMIC model were relatively coarse. More fine-grained indicators—such as the quality and content of AI training, the types of projects in which AI is used, or students’ prior experience with coding and data—may reveal stronger and more nuanced relationships with the AI-CIEI dimensions.

In addition, academic year was modeled as an observed covariate in the MIMIC analysis rather than as a grouping variable. Although measurement invariance was established across gender groups, it remains unclear whether the AI-CIEI operates equivalently across different stages of design education. Future studies with larger and more balanced samples at each year level could explicitly test configural, metric, and scalar invariance across academic years, thereby clarifying whether the four-factor structure is stable throughout students’ progression.

Finally, the developmental pathway outlined above is intentionally framed as a conceptual and pedagogical application of the model, informed by theory and item content rather than by direct longitudinal evidence. Subsequent research should design interventions and follow-up studies to test whether structured curricula aligned with this pathway lead to measurable gains in specific AI-CIEI dimensions.