Self-assessment model for achieving the intended learning outcomes

Abstract

The academic performance of learners can be measured by determining the learning results with the intended learning outcomes (ILOs). ILOs principles have encouraged learners to improve their competencies. The existing approaches for evaluating learner capacity lack a correlation between the actual learning outcomes and the ILOs. This study has proposed a new self-assessment model that allows learners to realize their ILO achievement during the learning processes towards enhancing their competencies. The Self-Regulated Learning Management (SRLM) system, implemented the ILO-based Self-Assessment Model (ISAM), has been developed for this research. The personalized learning paths in SRLM are constructed based on Bloom’s taxonomy, the Structure of Observed Learning Outcome (SOLO), and the Constructive alignment. The experimental study was conducted on ninety-two (N = 92) university learners learning in the Database Management course. They were asked to complete a voluntary survey rating their achievement on learning outcomes of the course. The results showed that learners who were provided the ILO achievement information during the learning processes performed better academically than their peers in the same class. This conclusion was derived from t test analyses on the dataset of learners’ test scores in the experimental course and is statistically significant. Simultaneously, most learners who participated in the research expressed satisfaction with the proposed model.

Keywords

Academic performance intended learning outcomes (ILO)ILO achievement learning management system self-assessment

Introduction

Regardless of their nature, whether formal education, business-oriented knowledge enhancement, or individual self-training, training programs require regular evaluation of training results (Andrini, 2016; Elnaga and Imran, 2013). Typically, this evaluation involves two distinct approaches: formative assessment and summative assessment. Formative assessment occurs continuously throughout the training period, while summative assessment occurs upon completing the training content (Bennett, 2011; Havnes et al., 2012; Knight, 2002). This report proposes a new approach to evaluating training results, complementing the formative and summative assessment methods mentioned above. This approach allows learners to assess their acquired skills and knowledge through test scores and compare their achievement of ILOs with the requirements of the subject. It is particularly suitable for individual learning environments, enabling learners to gauge their current knowledge based on ILO achievement. As a result, learners can independently adjust their learning strategies to meet their individual needs and learning objectives better.

To effectively assess learners’ skills and knowledge in meeting subject requirements, creating a flexible learning path that caters to their diverse learning needs (Muhammad et al., 2016; Xu et al., 2012). This report presents the development of adaptable learning paths within the same subject. Each learning path, designed according to the proposed methodology of this study, includes a range of learning and assessment activities that are categorized based on Bloom’s taxonomy and the Structure of Observed Learning Outcomes (SOLO), enabling the comparison of knowledge difficulty (Adams, 2015; Agustinsa et al., 2021; Forehand, 2010; Wu and Lin, 2022). Consequently, each node within the learning path represents a distinct level of knowledge, spanning from basic memorization to the application and creation of new knowledge. Additionally, nodes at the same knowledge level within the learning path exhibit varying difficulty levels, ranging from pre-structural to extended abstract. Each node within the learning path is closely linked to at least one subject’s ILO, ensuring that all learning activities align with measuring the subject’s learning outcomes. As each ILO consists of a verb and a statement of subject matter content, it determines the learner’s ability. Therefore, when learners fulfill the requirements of the ILOs, they also demonstrate their current competencies and skills. Nodes within the learning path can be presented in different formats, such as text, images, media, website links, or external files, to provide learners with a diverse and rich learning experience.

Constructive alignment (CA) was introduced by Biggs and colleagues in 1996 (Biggs, 1996). CA consists of “construct” and “align.” In the “construct” stage, the instructor constructs the learning content and assesses the knowledge for the subject, and from there, the learner follows the predetermined learning path to complete the subject. After completing the learning content and assessments, the learner’s feedback serves as the basis for the instructor to “align” the content of the subject to meet the learners’ expectations and make the subject content more relevant.

Although the CA approach has demonstrated its advantages in creating appropriate learning content for learners, the “construct” and “align” contents are aimed at meeting the ILOs of the topic, course, and program. However, the CA approach does not provide a necessary tool for instructors to compare and contrast the actual learning outcomes of learners with the intended learning outcomes predetermined for the course. ILOs are what learners are expected to be able to do after completing the course, including a verb that defines the competency, typically determined according to Bloom’s taxonomy of knowledge, and a subject matter content descriptor. For example, an ILO is defined as follows: “Describe an algorithm for finding paths in the graph.” This study introduces an ISAM model for learners to self-assess their current competencies. ISAM allows learners to measure their competencies in the topic, course, and program using the “ILO Achievement” value. ILO achievement is the level or proportion of actual learning outcomes compared to ILOs. The higher the percentage (up to 100%), the more the learner meets the topic, course, or program requirements. ISAM model is an essential supplement to the self-assessment factor in self-regulated learning and contributes to realizing the characteristics of outcome-based education. The study poses two research questions: (1) Do learners who receive information about ILO Achievement during the learning process achieve better learning outcomes than other learners in the same class? and (2) Are learners satisfied with the ISAM model in terms of ease of understanding, ease of use, and meeting expectations?

The remaining sections of this report are organized as follows: the theoretical foundations for the report are presented in the Background section, followed by the Methods section, which outlines the data collection methods and research variables. The Results section presents the findings obtained from the experimental implementation of the ISAM model in a real-life university course. The Discussion section discusses the opinions and comparisons with related studies. Finally, the Conclusion and Future Works section presents the study’s current conclusions and the following steps for applying the ISAM model in some domains.

Background

Taxonomies of learning

In 1956, Benjamin Bloom and his colleagues developed a classification taxonomy outlining essential levels of intellectual behavior in learning (Bloom, 1956a, 1956b). Bloom’s taxonomy encompasses three domains: cognitive, affective, and psychomotor. The cognitive domain, most prevalent in education today (Monrad et al., 2021; Stringer et al., 2021), is divided into six levels by Bloom, progressing from basic recall to the highest level, evaluation. The stages of cognitive thinking identified by Bloom are Knowledge, Comprehension, Application, Analysis, Synthesis, and Evaluation.

In 2002, Krathwohl presented a revised version of Bloom’s classification to reintegrate its ideas and incorporate new knowledge since 1956 (Krathwohl, 2002). This revised cognitive classification aims to expand learning outcomes, focusing on “maintenance” and “transfer” (Forehand, 2005; Nkhoma et al., 2017).

Another taxonomy is the Structure of the Observed Learning Outcome (SOLO), developed by John Biggs and Kevin Collis in the 1980s (Biggs and Collis, 1982). This framework, rooted in the cognitive theories of Piaget and Gagné, suggests that new knowledge is constructed upon prior experiences and cognitive structures (Gagné, 1974; Piaget and Cook, 1952). SOLO, widely utilized for assessing learning outcomes (Svensäter and Rohlin, 2023; Wu and Lin, 2022), differs from Bloom’s by organizing difficulty horizontally.

The SOLO taxonomy classifies knowledge into five levels: Prestructural, Unistructural, Multistructural, Relational, and Extended Abstract.

1. Prestructural: the learner has a lack of understanding of the topic. They may not have the necessary prior knowledge or comprehension of the subject matter.

2. Unistructural: the learner acquires basic knowledge. They can identify and describe basic information but do not yet connect it to a broader context

3. Multistructural: At this stage, the learner can understand and articulate multiple isolated aspects of the topic. They can recognize and describe various elements but may struggle to integrate or see connections between them.

4. Relational: Learners at this level can make connections between the various elements of the topic. They understand how different components relate to each other and can present their knowledge in a more integrated and comprehensive manner.

5. Extended Abstract: This is the highest level of SOLO framework; learners possess an advanced understanding. At this stage, learners not only understand the topic in depth, but also can apply their knowledge in innovative and abstract ways. They demonstrate critical thinking skills, make connections across different domains, and show creativity and originality in their understanding and application of knowledge.

The utilization of the SOLO framework not only generates insights but is also employed in developing content assessments (Stålne et al., 2016; Svensäter and Rohlin, 2023). It provides a systematic approach for instructors to evaluate learner comprehension levels. Moreover, beyond content assessment, the SOLO framework is utilized in formulating learning objectives and outcomes across disciplines. By incorporating SOLO into assessment content creation, instructors design tests that progress from simple to complex, thereby facilitating the assessment of learners’ abilities across a spectrum from simple to advanced levels.

Outcome-based education

Learning outcomes are defined as the expected achievements of learners upon completion of a learning process (Cedefop, 2017). These outcomes are typically categorized into three levels: learning outcomes for a specific topic, learning outcomes for a course within a program, and learning outcomes for the entire program (Rao, 2020). Identifying learning outcomes at the outset of designing a course or training program is crucial to provide clear direction for developing teaching and learning materials, activities, and assessments of learners’ competencies (Asim et al., 2021; Gurukkal, 2020).

Learning outcomes are classified into two types: intended learning outcomes and actual learning outcomes (Alfauzan and Tarchouna, 2017).

• Intended learning outcomes refer to what instructors or training institutions expect learners to achieve and are determined before the training process. These outcomes typically remain unchanged within a training cycle, such as a semester or a class, but may be reviewed and adjusted for the next training cycle.

• Actual learning outcomes, also known as achieved learning outcomes, reflect learners’ competencies at a specific point in time. These outcomes evolve throughout the learning process and tend to increase over the training cycle (Akpinar, 2009). Training methods are designed to align actual learning outcomes with intended learning outcomes. When this alignment is achieved, learners are considered competent.

Various methods are available for aligning learning outcomes with the training process, with one of the most prominent being the Outcome-based Education (OBE) approach (Gurukkal, 2020; Rao, 2020). William Spady first proposed OBE in the 1990s to shift the focus of education towards what learners learn rather than what they are taught (Spady, 1994). In OBE, all training activities are designed and operated to determine what the learner can do after completing a topic, course, or program, rather than focusing solely on what the instructor does to impart knowledge. OBE embodies a constructivist approach to education, wherein learners progressively enhance their learning abilities through studying materials, engaging in learning activities, and undergoing competency assessments (Al-Huneidi and Schreurs, 2013; Asim et al., 2021).

In OBE, learning assessment activities play a crucial role in ensuring they align with the predetermined Intended Learning Outcomes (ILOs) (Crespo et al., 2010). Previous studies have introduced various learner assessment methods within the OBE model, including Criterion-referenced assessment (CRA) (Gurukkal, 2020), rubric-based assessment (Biggs and Tang, 2014; Sasipraba et al., 2020), and peer-assessment (Almuhaideb and Saeed, 2020). However, these assessment methods may not directly measure actual learning outcomes, thus necessitating the development of new assessment methods capable of measuring learners’ achievement of ILOs during the training process.

Constructive alignment

Constructive Alignment (CA) is an educational approach rooted in Outcome-Based Education (OBE) (Crespo et al., 2010). CA serves as a framework for designing and developing teaching programs, aligning teaching, learning, and assessment activities to achieve desired learning outcomes. Biggs and colleagues introduced CA in 1996 in their article titled “Enhancing teaching through constructive alignment” (Biggs, 1996) and revised it in 2011 in the book “Teaching for Quality Learning at University: What the learner does” (Biggs and Tang, 2011). The principle of constructive alignment is fundamentally derived from the theory of constructivism, based on observations and scientific research on how people learn (Jaiswal, 2019). CA is considered the main principle behind current requirements for curriculum specifications, statements of future learning outcomes, and evaluation criteria (Ali, 2018; Jaiswal, 2019). The core principle of the educational system developed under CA is that all components of the teaching program must be designed so that learning activities and assessment tasks are appropriate for the expected learning outcomes.

In the context of Constructive Alignment (CA), instructors must ascertain what learners can do upon completing each learning unit (Rao, 2020). The purpose of assessment in CA is to furnish a comprehensive tool for educating learners and monitoring progress. Assessing learner competencies often involves providing feedback information (Biggs, 2014; Crespo et al., 2010), which learners can utilize to enhance their learning and academic performance. Additionally, this feedback information can aid instructors in adjusting their teaching methods to meet learners’ needs (Lwin et al., 2020; Rao, 2020). According to Biggs, the core principle of CA is that a sound teaching system must adapt its teaching and assessment methods to align with the learning activities outlined in the ILOs, thereby supporting all aspects of the educational system for effective learner learning (Biggs and Tang, 2010, 2011).

Instructors and learners collaborate to enhance the quality of the subject, with instructors determining the ILOs, designing Teaching and Learning Activities (TLAs) and Assessment Tasks (ATs), and constructing a learning pathway for learners. Learners construct knowledge by engaging in TLAs and ATs to meet the requirements outlined in the ILOs, while also providing feedback to align the TLAs and ATs of the subject.

Despite the advantages of guiding the design of TLAs and ATs to meet predetermined ILOs, CA also presents limitations, requiring a significant amount of time and effort in designing teaching methods and assessments (Ali, 2018; Loughlin et al., 2021). This limitation underscores the need for a more proactive assessment method to be used in courses and training programs based on the principles of CA.

Materials and methods

The study poses two research questions: (1) Do learners who receive information about ILO Achievement during the learning process achieve better learning outcomes than other learners in the same class? and (2) Are learners satisfied with the ISAM model in terms of ease of understanding, ease of use, and meeting expectations?

To address these two research questions, we have developed an Learning Management System (LMS) named SRLM based on the ISAM model. The backend of SRLM is built using the C# programming language on the . NET 7.0 platform and the Entity Framework Core library. It utilizes the Ardalis Clean Architecture along with techniques such as Dependency Injection and Restful API to create a structured API for interaction with the frontend. The frontend portion is developed using HTML5, SCSS, and Typescript with the Angular 10 framework, incorporating libraries such as Bootstrap 4 and Syncfusion. The SRLM database is implemented on Microsoft SQL Server with shared cloud services.

ISAM

The ISAM model provides different learning paths for each learner in the same class. The learning activities within each path are divided into two types: Teaching and Learning Activities (TLA) and Assessment Tasks (AT). TLAs can be presented in various forms such as lectures, readings, or videos to provide knowledge on a specific topic. When constructing TLAs, the instructor determines the ILOs that the TLA aims to achieve, as well as the weighting of each ILO within that TLA. ATs are built based on the question bank of the subject, with each question having a different difficulty level determined by the SOLO. When constructing an AT, the instructor selects the number of questions for each topic of the subject that the AT intends to assess, including the difficulty level of the questions. The system randomly selects questions from the question bank that meet the selection criteria and generate the exam according to the instructor’s requirements. Each AT, once generated, is assigned an AT Complexity Value (ACV).

A C V = \sum_{1}^{5} ({S O L O}_{ư e i g h t} * {S O L O}_{v a l u e}) / 100

(1)

ACV was developed from the concept of “SOLO average” (Brabrand and Dahl, 2009). Formula (1) outlines the calculation method employed to determine the ACV. The SOLO_value takes values from 1 to 5, corresponding to the five levels of difficulty of the questions according to SOLO. The SOLO_weight represents the weight of each level of difficulty in the AT. The ACV value is the sum of the SOLO_weight multiplied by the SOLO_value for each level of difficulty.

Similarly, since each AT of the subject is used to assess at least one ILO of the subject, and each ILO is measured by at least one AT, the ACV value mentioned above can be used to evaluate the complexity of an ILO, known as the ILO Complexity Value (ICV) as described in formula (2). The ILO_percentages represent the proportion of each ILO in each AT, and n is the number of ATs of the subject. Table 1 presents the ACV values of the experimental subject, and Table 2 presents the ICV values of each ILO. In Table 1, “Quiz on ERD” has the highest ACV value, indicating that it has the highest level of difficulty among the ATs of the subject.

Table 1.

The difficulty levels of each question group are weighted based on the SOLO and ACV values of each AT in the experimental course.

AT	SOLO value					ACV
AT	1	2	3	4	5	ACV
Quiz on database system	20	20	50	10		2.5
Quiz on database concept, architecture, and components	10	20	20	50		3.1
Quiz on data model	25	30	10	10	25	2.8
Quiz on ERD	10	10	40	20	20	3.3
Quiz on EERD	20	30	30	20		2.5
Quiz on relational data model and mapping techniques	20	20	20	20	20	3.0

Table 2.

ILO complexity values.

AT	ILO
AT	Explain the concepts and components of database and the database management system.	Analyze the given real world problems’ requirements in order to model it using conceptual modeling tools.	Demonstrate an understanding of the relational data model.
Quiz on database system	5	5
Quiz on database concept, architecture, and components	10	5	5
Quiz on data model	5		5
e	5	10	5
Quiz on EERD	5	5	10
Quiz on relational data model and mapping techniques	5	5	10
ICV	2.9	3.0	2.9

Figure 1 shows the creation of an Assessment Task (AT) using the ISAM model in the experimental system. In this process, for each learning objective (ILO) that the AT aims to assess, the instructor will select the level of knowledge according to Bloom’s taxonomy, the difficulty level of each question according to SOLO, and the number of questions at each level. The system will randomly select questions from the question bank of the subject to create the desired AT.

I C V = \sum_{1}^{n} (A C V * {I L O}_{p e r c e n t a g e s}) / \sum_{1}^{n} {I L O}_{p e r c e n t a g e s}

(2)

Data collection

This study was conducted from May to October 2023 at the Faculty of Information Technology and Communication (ICT), University X, Country Y. The Institute Research Board of University X approved the research for ethical considerations through the Certificate of Approval number MU-CIRB 2022/125.211.

Ninety-two learners (N = 92), aged between 19 and 23 years old, voluntarily participated in the research by signing an Informed Consent Form. The participating learners were randomly assigned to either the control group or the experimental group, with 45 learners (48.91%) in the control group and 47 learners (51.09%) in the experimental group. At the end of the study period, all learners completed all the tasks designed for the research, including participating in the experimental course and responding to relevant questionnaires. These learners had prior experience with online learning in various courses on the online learning management system during the COVID-19 pandemic. All participating learners were second-year learners majoring in Information Technology, with a GPA of 2.0 or above at the time of joining the research. The experimental course used in this study was Database Management System (DBMS). Table 3 lists the five topics covered in this course, teaching and learning activities (TLAs), and assessment tasks (Ats) designed for each topic, according to the DBMS course outline in the Faculty of ICT, University X.

Table 3.

List of topics, TLAs, and ATs in the DBMS course.

Topic	TLA/AT	Type	Required
Introduction to database systems, database concept, architecture, and components	Introduction to database systems	Lecture	X
	Key terms for database system	Reading
	Quiz on database system	Assessment	X
	Database concept, architecture, and components	Lecture	X
	Quiz on database concept, architecture, and components	Assessment	X
Data models and business rules and constraints	Data models and business rules and constraints	Lecture	X
	Key terms for data model	Reading
	Quiz on data model	Assessment	X
Entity-relationship diagram (ERD)	Entity-relationship diagram	Lecture	X
	Key terms for ERD	Reading
	Quiz on ERD	Assessment	X
Extended entity relationship diagram (EERD)	Extended entity relationship diagram	Lecture	X
	Key terms for EERD	Reading
	Quiz on EERD	Assessment	X
Relational data model and mapping ERD to relational database schema	Relational data model and mapping ERD to relational database schema	Lecture	X
	Quiz on relational data model and mapping techniques	Assessment	X

We want to assess the level of understanding of learners participating in research on the role and effects of learning outcomes. A questionnaire consisting of 10 positively framed questions regarding the definition and role of learning outcomes, as listed in Table 4, was distributed to the learners. The participating learners responded to their knowledge of these questions on a 5-point Likert scale. Additionally, the participating learners completed a pre-test consisting of ten multiple-choice questions, with one correct option, covering the knowledge related to the content presented in the DBMS course. The content of the pre-test questions was discussed and agreed upon by the DBMS lecturers at the Faculty of ICT before being conveyed to the learners to ensure accuracy, rationality in form, and relevance to the content of the course.

Table 4.

List of survey questions on the role and impact of learning outcomes.

Code	Question content
Q1	Learning outcomes are measurable.
Q2	Learning outcomes help lecturers and learners to have the same understanding of the objectives and essential points to know or achieve in a subject or training program.
Q3	Learning outcomes emphasize applying and integrating learned knowledge rather than simply remembering it.
Q4	Learning outcomes help to clearly define a path to learner success when participating in a subject or training program.
Q5	Learning outcomes are the foundation for building teaching and learning materials, tests, and assessing learners’ knowledge.
Q6	Learning outcomes are explicitly presented and are directly related to teaching and learning materials and tests used in the subject.
Q7	Learning outcomes connect learners’ learning with the materials provided during the course.
Q8	Learning outcomes help learners understand the assessment conditions and objectives of the tests in the subject.
Q9	Learning outcomes must be communicated clearly and accurately before starting a course or training program.
Q10	Learning outcomes should be referenced throughout the training process.

At the commencement of the research, the research team sent an email to each participant, which includes an account (i.e., username and password) for logging into SRLM, which was developed by the research team. The faculty provided the list of learner email addresses. This email also included an instructional video on how to use the functionalities within the experimental system Figure 2.

Within this LMS, the learning paths are developed following the ISAM approach, with the content encompassing the learning topics, TLAs, and ATs. The LMS records the start and end times of each TLA and AT activity completed by the learners. At any given time, learners in the experimental group can view their ILO achievement, specifically for each ILO of the course, as shown in Figure 3. Additionally, as illustrated in Figure 4, to improve their ILO achievement, learners in the experimental group can access detailed information on learning topics, TLAs, and ATs related to each ILO, allowing them to supplement their learning on these topics and improve their ILO achievement. Therefore, the only difference between the control and experimental groups is that learners in the experimental group have access to detailed information about their ILO achievement and the learning activities and assessments related to each ILO.

At the end of the course, all learners completed a post-test consisting of questions similar to those in the pre-test, but with the order of questions and answer choices shuffled. Additionally, a questionnaire was conducted among all learners to assess their satisfaction with the functionalities of the ISAM system. Satisfaction in the context of this study is defined as the ease of understanding, ease of use, and alignment with the learners’ expectations of the LMS system designed and developed according to ISAM.

Variables

The independent variable in this study is the ILO achievement. The control group, which uses the ISAM system without information on their level of compliance with the course’s learning outcomes defined in the ILOs, is contrasted with the experimental group, which receives continuous information on ILO achievement and ILO details while using the ISAM system. ILO achievement allows the experimental group to self-monitor their compliance with the course’s learning outcomes. Therefore, the dependent variable in this study is the learning results and the learner’s satisfaction in each group.

Results

The effectiveness of information regarding ILO achievements

Table 5 presents descriptive statistics on the average pre-test and post-test scores for each group of learners. Table 5 shows that the number of learners in the control and experimental groups is equal (N = 45 and N = 47). The average pre-test score in the control group is higher by 0.26, corresponding to 6.53% compared to the experimental group. However, in the post-test, the average score of the experimental group is higher than the control group by 0.22, corresponding to 4.09%. Comparing within each group, the average score of the post-test in the control group increases by 1.14 compared to the average score of the pre-test, corresponding to a rate of 26.89%. This rate is lower than the score increase rate in the experimental group, which is 1.62, corresponding to 40.70%. The standard deviation values of the two groups in each test range from 1.98 to 2.32, which are not excessively large, indicating relative consistency in the average score values of each group. The two groups’ other descriptive statistics values, such as minimum, maximum, median, and mode, are similar. Therefore, the initial observation is that the experimental group has better post-test results than the control group.

Table 5.

The descriptive statistics to analyze the pre-test and post-test scores of the learners.

Group	Test	Count	Mean	Std	25%	50%	75%	Max	Median	Mode
Control group	Pre-test	45.0	4.24	1.98	3.0	4.0	6.00	9.0	4.0	6.0
Control group	Post-test	45.0	5.38	2.32	4.0	5.0	7.00	10.0	5.0	4.0
Experiment group	Pre-test	47.0	3.98	2.01	3.0	4.0	5.00	9.0	4.0	5.0
Experiment group	Post-test	47.0	5.60	2.13	4.0	5.0	7.00	10.0	5.0	4.0

The pre-test and post-test consisted of ten multiple-choice questions, each with one correct answer. Therefore, the score for each test ranged from 0 to 10. Table 6 presents the frequency of each score range in the tests for each group of learners. The frequency data shows that for the control group, the score of six appeared most frequently in the pre-test, with 11 occurrences and four learners scoring above six in the pre-test. In the post-test, the number of learners scoring above six increased to 19. For the experimental group, in the pre-test, the scores of four and five had the highest frequency, appearing 10 times, and eight learners scored above five in the pre-test. In the post-test, the number of learners scoring above five increased to twenty-one.

Table 6.

The frequency distribution of scores among each group of learners in each test.

Group	Test	Score(s)
Group	Test	0	1	2	3	4	5	6	7	8	9	10
Control group	Pre-test	1	5	1	9	9	5	11	3	1	0	0
Control group	Post-test	1	1	1	1	7	8	7	5	4	7	3
Experiment group	Pre-test	2	3	5	9	10	10	3	2	2	1	0
Experiment group	Post-test	1	1	1	4	10	9	6	7	3	5	0

Number: frequency of each score.

To verify the assertion that learners in the experimental group have better academic performance than those in the control group, or in other words, learners who are provided with information about ILO achievement will perform better academically than their peers in the same class. We conducted t-tests on the learners’ learning results participating in the study.

The t test is valid only when applied to data that meets the criteria of normal distribution and equal variance. Therefore, we first conducted tests to assess the normality of the scores for each group of learners. The results of the Shapiro-Wilk test for normality are presented in Table 7, indicating that the test values for each group of learners have p-values >.05. Thus, it fails to reject the Shapiro-Wilk test’s null hypothesis, which suggests that the data was drawn from a normal distribution. Since the learners’ scores follow a normal distribution, Bartlett’s test assessed the equal variance between each pair of learner score datasets. The results of Bartlett’s test, shown in Table 8, reveal that all p-values for each pair of datasets are more significant than .05. Therefore, we fail to reject the null hypothesis of Bartlett’s test, which suggests that all input samples are from populations with equal variances. Based on the results of the Shapiro-Wilk test and Bartlett’s test, it can be concluded that the learner score datasets in this study satisfy the assumptions required for conducting the t test.

Table 7.

The Shapiro-Wilk test results about the normality of the distribution of each dataset.

Group	Test	Shapiro-Wilk test
Group	Test	Statistic	p-value
Control group	Pre-test	0.95	.07
Control group	Post-test	0.97	.19
Experiment group	Pre-test	0.97	.21
Experiment group	Post-test	0.96	.14

Table 8.

The Bartlett’s test results about the equal variances among each pair of datasets.

Data group 1	Data group 2	Bartlett’s test
Data group 1	Data group 2	Statistic	p-value
Control group, pre-test	Experiment group, pre-test	0.01	.93
Control group, post-test	Experiment group, post-test	0.31	.58
Control group, pre-test	Control group, post-test	1.09	.30
Experiment group, pre-test	Experiment group, post-test	0.17	.68

Table 9 provides the results of an independent two-sample t test on the learner score datasets. The p-values for tests are .52 and .64, respectively, more significant than .05. Therefore, we fail to reject the null hypothesis of the independent two-sample t test, which states that two independent samples have identical average values. In other words, the test results indicate no statistically significant difference in the pre-test scores between the two groups of learners and the post-test scores between the two groups of learners.

Table 9.

The results of an independent two-sample t test conducted on datasets.

Data group 1	Data group 2	Independent two-sample t test result
Data group 1	Data group 2	T	dof	Alternative	p-value	CI 95%	Cohen-d	Power
Control group, pre-test	Experiment group, pre-test	0.64	89.92	Two-sided	.52	[−0.56, 1.09]	0.13	0.10
Control group, post-test	Experiment group, post-test	−0.47	88.57	Two-sided	.64	[−1.14, 0.71]	0.10	0.076

We want to evaluate the effectiveness of the proposed self-assessment method in this study. A paired t test on two related samples of scores was conducted for each group of learners. Table 10 displays the paired t test results on each corresponding pair of datasets. Unlike the results shown in Table 9 above, both p-values in the paired t test on two related samples of scores were found to be less than .05. Therefore, this test result rejects the null hypothesis of the paired t test that the two related samples have identical average values. In other words, the higher post-test scores compared to the pre-test scores in each group of learners are statistically significant.

Table 10.

The results of the paired t test conducted on datasets.

Data group 1	Data group 2	paired t test results
Data group 1	Data group 2	T	dof	Alternative	p-value	CI 95%	Cohen-d	Power
Control group, pre-test	Control group, post-test	−2.59	44	Two-sided	.01	[−2.02, −0.25]	0.53	0.93
Experiment group, pre-test	Experiment group, post-test	−3.84	46	Two-sided	.00	[−2.46, −0.77]	0.78	1.00

Along with the conclusions drawn from the descriptive statistics in the previous section, it can be concluded that when provided with information about ILO achievement, learners perform better academically compared to their peers in the same class, and this finding is statistically significant.

The satisfaction of learners with studying according to the ISAM model

The questionnaire at the end of the course consists of questions to assess learners’ satisfaction regarding the usefulness of the ISAM model. Table 11 lists 15 questionnaire questions used to gauge learner satisfaction in this approach. The questions are divided into six groups: LP, RE, IA, OS, EE, and OR. The first four groups (i.e., LP, RE, IA, and OS) are designed in a 5-Likert scale format with response options as (1) Strongly disagree, (2) Disagree, (3) Neither agree nor disagree, (4) Agree, (5) Strongly agree. Next, the EE group is designed as Yes/No questions. Finally, at the end of the survey, there are three open-ended questions belonging to the OR group to capture additional user feedback. Descriptive statistical results for the four groups of questions designed in a 5-Likert scale format are presented in Table 12. Accordingly, learners provided average feedback for survey questions ranging from 3.60 to 3.96, indicating satisfaction with the ISAM model. The median and mode values are both 4.0, indicating that at least 50% of learners responded to the survey questions at level 5 (i.e., “Strongly Agree”), and the majority were satisfied with a 4.0 and above. Figure 5 presents the frequency values of learner responses. Note that only responses from the experimental group of learners were considered in the statistical analysis for questions belonging to the IA group.

Table 11.

The questionnaire inventory assessing learners’ satisfaction with the ISAM.

Questions group	Code	Question content
Learning path (LP)	LP1	The experimental system displays the CNs in each learning path, which helps me understand what activities must be done to complete the course.
	LP2	The experiment system shows that a learning path is easy to understand for me.
	LP3	The font size, font name, colors, and images of the learning path displayed in the experiment system make it easy for me to study.
Revision (RE)	RE1	The revision function provided in the experiment system helps me feel respected and heard by the university.
Revision (RE)	RE2	After my revision, the flexible learning activity provided by the experiment system met my expectations.
ILO achievement (IA)	IA1	The ILO achievement function of the experiment system helps me understand the course’s learning outcomes.
	IA2	The ILO achievement function of the experiment system helps me understand my achievement in the course’s learning outcomes.
	IA3	Experiment system functions that allow me to review the learning content, learn additional knowledge, and retake the assessments help me better understand the subject matter contents.
	IA4	Information on how well I achieved the course’s learning outcomes helps me control the completion of the course requirements without direct instruction from the instructor.
Overall satisfaction (OS)	OS1	Overall, are you satisfied with this proposed learning model?
Ease and meet expected (EE)	EE1	Overall, do you think this model is easy to use for you?
Ease and meet expected (EE)	EE2	Overall, do you think the information in this model is as accurate as you expected?
Open (OR)	OR1	Do you have any comments or suggestions for improving and modifying a function related to the learning paths?
	OR2	Do you have any comments or suggestions for improving and modifying the function related to the revision function?
	OR3	Do you have any comments or suggestions for improving and modifying the visualization of ILO achievement?

Table 12.

The statistical analysis of the responses provided by learners regarding the usefulness of the ISAM model.

Code	Count	Mean	Std	Min	25%	50%	75%	Max	Median	Mode
LP1	92.0	3.92	0.70	1.0	4.0	4.0	4.0	5.0	4.0	4.0
LP2	92.0	3.96	0.80	1.0	4.0	4.0	4.0	5.0	4.0	4.0
LP3	92.0	3.78	0.89	1.0	3.0	4.0	4.0	5.0	4.0	4.0
RE1	92.0	3.77	0.77	1.0	3.0	4.0	4.0	5.0	4.0	4.0
RE2	92.0	3.70	0.84	1.0	3.0	4.0	4.0	5.0	4.0	4.0
IA1	47.0	3.85	0.83	1.0	3.0	4.0	4.0	5.0	4.0	4.0
IA2	47.0	3.77	0.79	1.0	3.0	4.0	4.0	5.0	4.0	4.0
IA3	47.0	3.94	0.79	1.0	4.0	4.0	4.0	5.0	4.0	4.0
IA4	47.0	3.89	0.81	1.0	3.0	4.0	4.0	5.0	4.0	4.0
OS1	92.0	3.60	0.80	1.0	3.0	4.0	4.0	5.0	4.0	4.0

The statistical analysis involved conducting the Kruskal-Wallis test on the learners’ responses, as illustrated in Figure 5. With a calculated p-value of .96, the null hypothesis cannot be rejected, indicating that there are no significant variances in the median responses across the groups. Essentially, the mean values of the learners’ responses lack statistical importance. This outcome addresses research question RQ2: Are learners’ content with the ISAM model concerning comprehensibility, usability, and meeting expectations? The statistical data from the average scores of the survey questions reveals that learners are content with the user-friendly and comprehensible features of the SRLM system. The functionalities within the SRLM system effectively meet the learners’ expectations.

The question EE1 and question EE2 questions were used to evaluate the overall opinions of learners regarding the ISAM model implemented in the DBMS course. Out of 92 learner responses, 69 learners agreed with the question EE1, and 75 learners agreed with the EE2 question. Therefore, the corresponding percentages of learners who agreed with these questions were 75.00% and 81.52%. The results of the binomial test analysis, with a probability of learner agreement of 78.26% (the average of 75% and 81.52%), yielded p-values of .45 and .53, both of which were greater than 0.05. Thus, there was a failure to reject the null hypothesis of the binomial test, which stated that “The probability of success in a Bernoulli experiment is p.” This led to the conclusion that there was a probability of more than 78.26% that learners agreed that the ISAM model was easy to use and met their expectations for an LMS system.

Discussion

This study introduces a novel approach for learners to self-assess their actual learning outcomes in comparison to the learning objectives of a given course. The ISAM self-assessment model complements the self-assessment aspect of the self-regulated learning theory, which previous researchers have extensively examined (Min and Nasir, 2020; Viberg et al., 2020). The ISAM model presented in this study shares similarities with Zi Yan’s research, which focuses on providing learners with self-assessment tools at every stage of self-regulated learning (Yan, 2020). However, while Yan’s study concluded that self-assessment is necessary at all stages of self-regulated learning, the results of this study contribute a more straightforward model for implementing self-assessment in conjunction with constructive alignment in constructivist learning systems. The ISAM model offers learners a mechanism to align their learning with the objectives of a given course, augmenting previous self-regulated learning models introduced by scholars (Boekaerts et al., 2012; Pintrich and Garcia, 2012; Zimmerman and Schunk, 2011).

Conclusion and future work

ILO, or Intended Learning Outcomes, represents the expectations instructors or training institutions have for learners upon completing a training program. The empirical findings of this study demonstrate that regular information about actual learning outcomes, referred to as ILO achievement, may leads to better learning outcomes compared to peers in the same class. Additionally, most learners expressed satisfaction with the ISAM model for self-assessment based on ILO, as presented in this report. The ISAM model holds potential for application in university training environments and various other forms of training, including short-term training in businesses, skill enhancement training for company employees, and integration training for new staff. Further research is necessary to assess the suitability of the ISAM model across different fields and to refine the model for broader applications.

Footnotes

Acknowledgements

We are grateful to the instructors and learners who participated in this research. Their willingness to share their time and expertise was essential to this project.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research is partially supported by the Faculty of Graduate Studies and Graduate Studies of Mahidol University Alumni Association.

ORCID iDs

Phat Nguyen Huu

Preecha Tangworakitthaworn

Lester Gilbert

Author Biographies

Phat Nguyen Huu was born in Hochiminh City, Vietnam, in 1984. He received a B.S. degree in Information and Technology from Hoa Sen University, Vietnam, in 2008 and an M.Sc degree from Bordeaux 1 University, France, in 2013. In 2020, he became a PhD student at the Faculty of Information and Communication Technology, Mahidol University, Thailand. His expertise includes software development, database design and management, and testing. His research interests are applied Information and Technology in education, educational technology and online learning.

Preecha Tangworakitthaworn is an Assistant Professor at the Faculty of Information and Communication Technology, Mahidol University, Thailand. He received his PhD in computer science from the University of Southampton, UK, in 2014. His current research interests include technology enhanced learning, e-learning system engineering, outcome-based education, game-based learning and serious game.

Lester Gilbert is a senior lecturer in Computer Science at Department of Electronics and Computer Science, University of Southampton, UK. His research focuses on e-learning, e-assessment, service-oriented architectures, and technology enhanced learning.

References

Adams

(2015) Bloom’s taxonomy of cognitive learning objectives. Journal of the Medical Library Association: JMLA 103(3): 152–153.

Agustinsa

Muchlis

Maizora

(2021) Level of student understanding in solving geometry problems based on taxonomy of SOLO (structure of observed learning outcomes). Journal of Physics: Conference Series 1731: 012055.

Akpinar

(2009) Validation of a learning object review instrument: relationship between ratings of learning objects and actual learning outcomes. International Journal of Doctoral Studies 4(4): 291–302.

Al-Huneidi

Schreurs

(2013) Constructivism based blended learning in higher education. In: Information systems, e-learning, and knowledge management research: 4th world summit on the knowledge society, WSKS 2011, revised selected papers 4, Mykonos, Greece, 21–23 September 2011, 581–591. Springer.

Alfauzan

Tarchouna

(2017) The role of an aligned curriculum design in the achievement of learning outcomes. Journal of Education and e-learning Research 4(3): 81–91.

Ali

(2018) The design of curriculum, assessment and evaluation in higher education with constructive alignment. Journal of Education and E-Learning Research 5(1): 72–78.

Almuhaideb

Saeed

(2020) Fostering sustainable quality assurance practices in outcome-based education: lessons learned from ABET accreditation process of computing programs. Sustainability 12(20): 8380.

Andrini

(2016) The effectiveness of inquiry learning method to enhance students’ learning outcome: a theoritical and empirical review. Journal of Education and Practice 7(3): 38–42.

Asim

Vaz

Ahmed

, et al. (2021) A review on outcome based education and factors that impact student learning outcomes in tertiary education system. International Education Studies 14(2): 1–11.

10.

Bennett

(2011) Formative assessment: a critical review. Assessment in Education: Principles, Policy & Practice 18(1): 5–25.

11.

Biggs

(1996) Enhancing teaching through constructive alignment. Higher Education 32(3): 347–364.

12.

Biggs

(2014) Constructive alignment in university teaching. HERDSA Review of Higher Education 1(3): 5.

13.

Biggs

Collis

(1982) Evaluating the Quality of Learning: The SOLO Taxonomy (Structure of the Observed Learning Outcome). Cambridge, MA: Academic Press.

14.

Biggs

Tang

(2010) Applying constructive alignment to outcomes-based teaching and learning. In: Training Material for “Quality Teaching for Learning in Higher Education” Workshop for Master Trainers. Kuala Lumpur, Malaysia: Ministry of Higher Education, 23–25.

15.

Biggs

Tang

(2011) Teaching for Quality Learning at University: What the Student Does. 4th edition. New York, NY: McGraw-Hill Education.

16.

Biggs

Tang

(2014) Constructive alignment: an outcomes-based approach to teaching anatomy. In: Teaching Anatomy: A Practical Guide. Berlin, Germany: Springer, 31–38.

17.

Bloom

(1956a) Taxonomy of Educational Objectives. Vol. 1: Cognitive Domain, the Classification of Educational Goals. New York, NY: McKay, 20–24.

18.

Bloom

(1956b) Taxonomy of educational objectives. In: Affective Domain. Harlow, UK: Longmans, Vol. 2.

19.

Boekaerts

Musso

Cascallar

(2012) Self-regulated learning and the understanding of complex outcomes. Hindawi 2012: 1–2.

20.

Brabrand

Dahl

(2009) Using the SOLO taxonomy to analyze competence progression of university science curricula. Higher Education 58(4): 531–549.

21.

Cedefop (2017) Defining, Writing and Applying Learning Outcomes: A European Handbook. Thessaloniki, Greece: Cedefop.

22.

Crespo

Najjar

Derntl

, et al. (2010) Aligning assessment with learning outcomes in outcome-based education. In: IEEE EDUCON 2010 conference, Madrid, Spain, 14–16 April 2010, 1239–1246. IEEE.

23.

Elnaga

Imran

(2013) The effect of training on employee performance. European Journal of Business and Management 5(4): 137–147.

24.

Forehand

(2005) Bloom’s taxonomy: Original and revised. Emerging Perspectives on Learning, Teaching, and Technology 8: 41–44.

25.

Forehand

(2010) Bloom’s taxonomy. Emerging Perspectives on Learning, Teaching, and Technology 41(4): 47–56.

26.

Gagné

(1974) Instruction and the conditions of learning. Psychology of School Learning: Views of the Learner 1: 153–175.

27.

Gurukkal

(2020) Outcome-Based Education: An Open Framework. New Delhi, India: Sage Publications Sage, 1–4.

28.

Havnes

Smith

Dysthe

, et al. (2012) Formative assessment and feedback: making learning visible. Studies In Educational Evaluation 38(1): 21–27.

29.

Jaiswal

(2019) Using constructive alignment to foster teaching learning processes. English Language Teaching 12(6): 10–23.

30.

Knight

(2002) Summative assessment in higher education: practices in disarray. Studies in Higher Education 27(3): 275–286.

31.

Krathwohl

(2002) A revision of bloom’s taxonomy: an overview. Theory Into Practice 41(4): 212–218.

32.

Loughlin

Lygo-Baker

Lindberg-Sand

(2021) Reclaiming constructive alignment. European Journal of Higher Education 11(2): 119–136.

33.

Lwin

, et al. (2020) Feedback analysis in outcome base education using machine learning. In: 2020 17th International conference on electrical engineering/electronics, computer, telecommunications and information technology (ECTI-CON), Phuket, Thailand, 24–27 June 2020, 767–770. IEEE.

34.

Min

Nasir

(2020) Self-regulated learning in a massive open online course: a review of literature. European Journal of Interactive Multimedia and Education 1(2): e02007.

35.

Monrad

Bibler Zaidi

Grob

, et al. (2021) What faculty write versus what students see? Perspectives on multiple-choice questions using bloom’s taxonomy. Medical Teacher 43(5): 575–582.

36.

Muhammad

Zhou

Beydoun

, et al. (2016) Learning path adaptation in online learning systems. In: 2016 IEEE 20th international conference on computer supported cooperative work in design (CSCWD), Nanchang, China, 4–6 May 2016, 421–426. IEEE.

37.

Nkhoma

Lam

Sriratanaviriyakul

, et al. (2017) Unpacking the revised bloom’s taxonomy: developing case-based learning activities. Education + Training 59(3): 250–264.

38.

Piaget

Cook

(1952) The Origins of Intelligence in Children. New York, NY: International Universities Press.

39.

Pintrich

Garcia

(2012) Self-regulated learning in college students: knowledge, strategies, and motivation. In: Student Motivation, Cognition, and Learning. London, UK: Routledge, 129–150.

40.

Rao

(2020) Outcome-based education: an outline. Higher Education for the Future 7(1): 5–21.

41.

Sasipraba

Bantha Navas

Nandhitha

, et al. (2020) Assessment tools and rubrics for evaluating the capstone projects in outcome based education. Procedia Computer Science 172: 296–301.

42.

Spady

(1994) Outcome-Based Education: Critical Issues and Answers. Arlington, TX: American Association of School Administrators.

43.

Stålne

Kjellström

Utriainen

(2016) Assessing complexity in learning outcomes–a comparison between the SOLO taxonomy and the model of hierarchical complexity. Assessment & Evaluation in Higher Education 41(7): 1033–1048.

44.

Stringer

Santen

Lee

, et al. (2021) Examining bloom’s taxonomy in multiple choice questions: students’ approach to questions. Medical Science Educator 31(4): 1311–1317.

45.

Svensäter

Rohlin

(2023) Assessment model blending formative and summative assessments using the SOLO taxonomy. European Journal of Dental Education 27(1): 149–157.

46.

Viberg

Khalil

Baars

(2020) Self-regulated learning and learning analytics in online learning environments: a review of empirical research. In: Proceedings of the tenth international conference on learning analytics & knowledge, Frankfurt, Germany, 23–27 March 2020, 524–533.

47.

Lin

(2022) SOLO taxonomy-based knowledge structure with subjective items. Journal of Intelligent and Fuzzy Systems 43(1): 265–278.

48.

Wang

Chen

, et al. (2012) Personalized learning path recommender based on user profile using social tags. In: 2012 fifth international symposium on computational intelligence and design, Paris, France, 24–26 October 2011, 511–514. IEEE.

49.

Yan

(2020) Self-assessment in the process of self-regulated learning and its relationship with academic achievement. Assessment & Evaluation in Higher Education 45(2): 224–238.

50.

Zimmerman

Schunk

(2011) Self-regulated learning and performance: an introduction and an overview. In: Handbook of Self-Regulation of Learning and Performance. Oxfordshire, UK: Taylor & Francis, 15–26.