A Deep Learning Framework With Visualisation for Uncovering Students’ Learning Progression and Learning Bottlenecks

Abstract

Educational process mining aims (EPM) to help teachers understand the overall learning process of their students. Although deep learning models have shown promising results in many domains, the event log dataset in many online courses may not be large enough for deep learning models to approximate the probability distribution of students’ learning sequence due to a lack of participants. This study proposes a deep learning framework to help uncover the learning progression of learners. It aims to produce a graphical representation of the overall educational process from event logs. Our framework adopts the Smith–Waterman algorithm from the bioinformatics field to evaluate general learning sequences generated from deep learning models. Using our framework, we compare the performance of a deep learning model with the modified cross-attention layer and a model without modification and find that the modified model outperforms the other. The contribution of this framework is that it enables the use of neural architecture search techniques to uncover students’ general learning sequence irrespective of the dataset’s size. The framework also helps educators identify education materials that present as learning bottlenecks, enabling them to improve the materials and their respective layout order, thus facilitating student learning.

Keywords

deep learning students’ learning progression sequence alignment algorithm transformer learning bottlenecks educational process mining

Introduction

Many academic institutions utilise Learning Management System (LMS) to improve learning and teaching experience (Aldiab et al., 2019). Those institutions have been delivering their courses through LMS platforms in recent years where students are allowed to access different course materials freely throughout the duration of an enrolled course. In general, LMS platforms are considered to be more cost-effective than traditional classrooms (Bartley & Golek, 2004). In particular, the Modular Object-Oriented Dynamic Learning Environment (Moodle), considered to be one of the most prominent open-source LMS globally, has been considered as an efficient LMS that provides sets of learning-centric resource in the literature (Mustafa & Ali, 2023).

While students interact with the Moodle platform to access course content such as videos, assignments, lecture notes, and quizzes, Moodle helps teachers monitor students’ activity with its logging system (Shrestha & Pokharel, 2021). However, the log data generated by Moodle or other LMS platforms is difficult for teachers to comprehend, making it challenging to use the data to improve the student learning experience. Therefore, Educational Data Mining (EDM) and Educational Process Mining (EPM) have emerged as methods to obtain useful educational insights from raw data.

EDM-related algorithms originate from classical regression and classification data mining techniques and aim to extract simple patterns and data correlations. EDM often used to predict student performance, detect undesirable behaviours and automatic evaluations (Hernández-Blanco et al., 2019). However, information extracted from EDM alone is not sufficient to produce an overview visualization of the learning process and does not consider the process as a whole (Bogarín et al., 2018). In contrast, EPM is a specialised area that focuses on improving comprehension of the overall learning process.

Specifically, EPM is process-centric and assumes event sequences as the data type with each event associate with a single process instance or online activities. In summary, EPM focuses on the overall process as a whole, in contrast EDM focuses on significant patterns that emerge from data (Bogarín et al., 2018). For example, EDM would focus on predicting students’ final exam score using other tests score in the context of a Moodle course. On the other hand, EPM algorithms would analyse the sequences of events from the beginning of the course up until the final exam.

There are three main types of EPM (Bogarín et al., 2018): learning process discovery mining, conformance analysis and process model analysis. According to Bogarín et al. (2018), the main goal of educational process discovery mining is to give the instructor a clear visualisation of the student learning behaviour model. Conformance analysis, in contrast, helps the instructor to analyse whether a presumed behaviour model corresponds to the behaviour shown in event logs. Process model analysis aims to elaborate and extend the use cases of a given process model, which can help instructors to find learning bottlenecks and relationships between students in a course by comparing different process models.

Learning process discovery remains one of the most challenging tasks in EPM. Process discovery is challenging because a myriad of possible behavioural models can be drawn from any one set of event logs. This makes finding the behavioural model that best describes the general learning process out of all possible candidate models akin to finding a needle in a haystack. In the context of EPM, event logs can be produced from multiple learning progression models convoluted together, as different students may have different learning processes. As such, it is extremely difficult to generate a general learning process model from educational event logs.

Furthermore, students are allowed to access different course materials freely throughout the duration of an enrolled course in an LMS setup such as Moodle. Therefore, the log events created by students accessing different learning materials are not causally dependent on one another according to the definition originating from the alpha algorithm (Van der Aalst et al., 2004). Although a process model connecting all course materials with a parallel edge can be the correct model, it may not be a particularly useful model as the model is too generic (de Medeiros et al., 2007).

Consequently, matrices of values such as preciseness (de Medeiros et al., 2007) have been developed to confine the vagueness of the process model being mined as a representation of the corresponding event logs. However, as the number of highly diverse traces increases in the event log of an LMS, less precise models are allowed to be mined from the event log. Furthermore, as students often differ in their online learning behaviour (Hsiao et al., 2019), how to generalise the different types of learning behaviour and produce a model that is useful for educators to visualise remains a challenge in EPM.

Recent advancements in deep learning research have encouraged a widespread adoption of neural network architectures based on transformers (Vaswani et al., 2017) for analysing sequential data. However, in many online courses, the event log dataset may not be large enough for deep learning model (Du et al., 2020) to accurately approximate the probability distribution of general students’ learning sequences due to a limited number of students. Moreover, a general deep learning framework for uncovering the learning sequence of students is lacking in the literature due to the absence of a settled evaluation method. This study proposes a novel deep learning framework to help uncover students’ learning sequence. Specifically, this study aims to develop a framework to ensure that each segment in the general students’ learning sequence proposed by deep learning models is backed by subsequence that can be found within the events data set.

In our framework, we adopt the Smith-Waterman sequence alignment algorithm from bioinformatics to compare a sequence generated from a deep learning model to the original dataset and assess its similarity, enabling more precise evaluation.

We demonstrate the deep learning framework using a Moodle course that introduces statistical concepts to 94 students across three cohorts, with 33 course materials in total. Ultimately, this framework enables the use of neural architecture search techniques to uncover the general learning sequence of students, irrespective of the size of the dataset. Using our framework, we demonstrated how two deep learning models are evaluated and selected. On top of that, we design a convenient visualisation for users to see the learning process from the model output.

Background

There are four major areas of study related to EPM according to Bogarín et al. (2018): intention mining, sequence pattern mining, sub-graph mining and processing mining. Intention mining aims to model the processes according to the intention of the actors rather than the actual process (Khodabandelou et al., 2013). Sequence pattern mining and sub-graph mining focuses on searching interesting patterns or sub-process that may describe the underlying process whilst process mining focuses on extracting the overall underlying process from event logs. All in all, process mining remains one of the most challenging areas in EPM.

In EPM, there are two major approaches to construction of learning process models: Bottom-up approaches (sometimes refer as local search) and top-down approaches. Bottom-up approaches build learning process model through merging small local patterns whilst top-down approaches build learning process model through modifying a prior learning process model (Trcka & Pechenizkiy, 2009). The alpha algorithm (Van der Aalst et al., 2004) and heuristics miner algorithm (Weijters et al., 2006) are considered to be bottom-up approaches. Genetic process mining (de Medeiros et al., 2007) and fuzzy process mining (Günther & van der Aalst, 2007) adopted top-down strategies.

Over the years, different approaches developed their own conformance checking (Rozinat & Van Der Aalst, 2006) methodologies and metrics to find discrepancies and commonalities between the model behaviour and the observed behaviour. (Bogarín et al., 2018). Although deep learning had been widely adopted in EDM to predict learning outcomes and detect undesirable student behaviours and automatic evaluation (Hernández-Blanco et al., 2019), it is rarely used in EPM to construct a learning process model.

The alpha algorithm (Van der Aalst et al., 2004), one of the earliest process mining algorithms, extracts a workflow process model based on a myriad of event sequences, which are called traces. In essence, the algorithm classifies the relationships between events into three major types: causal relation, potential parallelism and neither. From these relationships, the algorithm constructs two sets A and B in which all of the events in A are causally related to B. The events within each set should be neither causally related nor parallel with one another. For every pair (A, B), a place (a type of node in a Petri net) is added to connect all of the events from A to B. A start event (an event without incoming links) and an end event (an event with only incoming links) are then added to the process model to order the intermediate processes of the process graph.

The heuristics miner algorithm (Weijters et al., 2006) uses a value ranging from −1 to 1 to express how strongly two events relate to one another in three ways. Like the alpha algorithm, the heuristic miner algorithm takes consecutive causal relations between two events into account. By normalising the frequency of directed consecutive causal relations, the heuristic values between two single events can be calculated. A threshold can then be applied to the heuristic matrix to determine whether there should be an edge connecting two events in the process model.

Another major improvement brought by the heuristic miner algorithm is the ability to detect length-one (a repeating event) and length-two loops (two consecutive repeating events). The heuristic miner algorithm specifically counts the number of length-one and length-two loops in the event log and normalises the value. A similar approach is applied to detect long-term dependence. The heuristic miner algorithm specifically counts the number of times an event A is followed by another event B at a later point in a trace and normalises the value with the total occurrence of A. Heuristic miner algorithms allow a user to pick a threshold to filter out low-frequency occurrences, which makes the algorithm more robust to noise (Bogarín et al., 2018).

Genetic process mining is designed to properly address non-free choices (de Medeiros et al., 2007), invisible tasks and duplicate tasks. Genetic process mining involves five major steps: reading an event log, building a population, computing fitness, selecting from the population through genetic operations and returning possible candidates.

Genetic process mining starts off with a population of models. Each model within the population has the same set of activities as those found in the event log. A new generation of models is selected from the old population according to their fitness. Genetic operators, such as crossover (merging two models) and mutation (the modification of a model), are applied to the newly selected models to increase the diversity of the new population. The genetic algorithm’s fitness function, which comprises formulas for both completeness (the model can reproduce the given set of event traces) and preciseness (the model cannot parse more than the given set of event traces), has added much value to the global search framework.

Fuzzy process mining (Günther & van der Aalst, 2007) algorithms are a branch of process mining algorithms that use a fuzzifier (van Dongen & Adriansyah, 2010) to cluster events into groups. Performing clustering before the mining process gives fuzzy process mining an advantage in handling highly unstructured behaviour and substantial numbers of events (Bogarín et al., 2018). The output of fuzzy process mining is a fuzzy net that can be interpreted as a more abstract representation of the underlying behaviour model (Bogarín et al., 2018). Fuzzy algorithms adopt both local and global strategy techniques to construct the process model (Gupta, 2014), and they remain one of the most popular algorithms in EPM research (Bogarín et al., 2018).

Methodology

In response to the challenges stated above, we propose a novel learning process discovery EPM framework using deep learning models and a dynamic visualisation methodology. The proposed framework differs from other frameworks in that it adopted the Smith–Waterman sequential alignment algorithm from bioinformatics to compare a sequence generated from a deep learning model to the original dataset, thereby enabling a more detailed evaluation of the alignment between the sequence and the dataset. This approach allows for a more nuanced analysis of the data over each time step of the sequence. We formalise our notion starting with Definitions 1–5 below.

Definition 1 (course). Let $A$ be a set of educational materials stored in LMS. A course $C$ contains a set of educational materials $C \subseteq A$ . Each student $l$ can interact with $C$ if and only if the student is enrolled in course $C$ .

Definition 2 (event traces and event log). Let $T_{c}$ be a set of actions that students are allowed to perform by the LMS within a course $C$ . An event trace $σ \in T_{c}^{*}$ is a sequence of actions, and an event log is a bag of event traces ${W_{o}}_{c} \subseteq T_{c}^{*}$ . $W_{o} \subseteq T^{*}$ implies that ${W_{o}}_{c} \subseteq T_{c}^{*}$ even if c is not specified, and the $o$ in $W_{o}$ refers to the original event log.

Definition 3 (original sequence of events). A sequence of events is an arbitrary sequence of actions denoted by $S_{o} \in T^{*}$ . For instance, given a set of educational materials in course $C = {a, b}$ , a sequence with length 4 is denoted as ${s_{o}}^{4}$ (e.g., ${s_{o}}^{4} = aaba$ ), where ${s_{o}}^{4} \in {S_{o}}^{4}$ and the fourth element is defined as ${s_{o}}_{4}$ . Hence, ${s_{o}}_{4}$ must be an element of $T_{c}$ (i.e., ${s_{o}}_{4} \in T_{c}$ ).

Definition 4 (start event and end event). Similar to the start and end places in a Petri net, a start event $s_{}$ and an end event $s_{< e o s >}$ are added at the beginning (i.e., ${s_{o}}_{0}$ ) and at the end (i.e., ${s_{o}}_{m + 1}$ ) of each element in the set ${W_{o}}_{c}$ to create the set $W_{c} ∋ {W_{o}}_{c}$ . Hence, the sequence of events is a superset of the original sequence of events. $S \in {s_{}, s_{< e o s >}, T^{*}}^{*}$ (e.g., if ${s_{o}}^{4} = aaba$ , a sequence of events is $s^{6} = s_{} a a b a s_{< e o s >}$ ), and the maximum length of a trace $s$ is defined as $m$ .

m = \max_{s \in W_{c}} | s |

(1)

Using the above definitions, we can now enrich the definition of event traces from $σ \in T_{c}^{*}$ to $σ \in S$ . More specifically, $σ^{w_{i}}$ refers to a particular event trace $i$ being present in the event log $W_{c}$ starting with $s_{}$ event and ending with $s_{< e o s >}$ , where c is implied. $σ^{w_{i}}$ is padded with $s_{}$ (s) after $s_{< e o s >}$ to ensure $| σ^{w_{i}} | = m$ . $σ_{\leftarrow}^{w_{i}}$ is defined as the left-shifted event trace where the first event is removed from the sequence and $s_{}$ is added at the end of the sequence to maintain $| σ_{\leftarrow}^{w_{i}} | = m$ . After $s_{}, s_{< e o s >} and s_{}$ are introduced, the number of events becomes $E = | T | + 3$ .

Definition 5 (general learning sequence, general learning decision instance). Given $W_{c}$ (a bag of event traces $s$ , where $s \in W_{c}$ ) produced from a set of students $L$ within a course c in the sample space denoted $A$ , the most likely learning decision instance $d_{t + 1}$ is defined in (2):

d_{t + 1} = {\begin{cases} \underset{s_{t + 1} \in S}{argmax} P (s_{t + 1} | W_{c}, s^{t}) & \exists s_{t + 1} : | s | \geq t + 1 \\ s_{< e o s >} & o t h e r w i s e \end{cases}

(2)

where

P

is the conditional probability of the next learning decision that a set of students

L

would be likely to make given an arbitrary sequence

s^{t}

with length

t

given

W_{c}

d_{t + 1}

is very similar to the frequency approach often used in heuristic process mining. By introducing the notion of the relative time step

t

, we are essentially mining the causal dependence between

s^{t}

and

s_{t + 1}

. We then parameterise

d_{t + 1}

d_{t + 1}^{θ}

, which gives the following definition of a general learning decision instance

d_{t + 1}^{θ}

d_{t + 1}^{θ} = \underset{s_{t + 1} \in S}{argmax} P_{W_{c}} (s_{t + 1} | θ, s^{t})

(3)

Ultimately, we want to find a model that can help us to identify a general learning sequence, given as (4)

\max \prod_{t = 0}^{m} P_{W_{c}} (s_{t + 1} | θ, s^{t})

(4)

From the above definitions, any deep learning models can be used for $d_{t + 1}$ parameterisation. However, the model being used requires two major constructs to model both time step and conditional relationships. Hence, we propose using a transformer model (Vaswani et al., 2017) with modifications for parameterisation. The model comprises a positional embedding layer and a masked self-attention layer, which can efficiently model the time step and conditional relationship.

To demonstrate the framework, a minor modification is made to the transformer model’s position embedding layer (Vaswani et al., 2017), as shown in (5)

p_{t, j} = {\begin{cases} \sin \frac{t}{m^{j / D}} & i f j i s e v e n \\ \cos \frac{t}{m^{(j - 1) / D}} & i f j i s o d d \end{cases}

(5)

where d is the size of the embedding. As such, a user can choose the model complexity based on the size of the embeddings

D

. Each trace in the set

W_{c}

is padded with

s_{< p a d >}

such that each trace has length m. With

X

defined as the padded

W_{c}

, a masked single-head attention mechanism of the model is defined in equations (6) and (7) with

Q = W_{q} X

K = W_{k} X

and

V = W_{v} X

U_{o, p} = {\begin{cases} - \infty & o < p \\ 0 & o \geq p \end{cases}

(6)

A_{m a s k e d} = s o f t m a x (\frac{Q K^{T} + U}{\sqrt{d}})

(7)

The output of the masked single-head attention layer is $A_{m a s k e d} V$ . From equation (7), we can observe that the upper triangle of the matrix $A_{m a s k e d}$ contains only zeros caused by the matrix $U_{o, p}$ after passing the $softmax$ function. This mechanism fits well into the original $P_{W_{c}} (s_{t + 1} | θ, s^{t})$ framework design, as matrix $U$ forces the model to compute the probability distribution of $s_{t + 1}$ given $s^{t}$ . As the cross-attention mechanism in the original design is not masked, we modify the cross-attention layer into $A_{m o d i f i e d}$ and the output of the layer into $O_{m o d i f i e d}$

A_{m o d i f i e d} = s o f t m a x (\frac{Q_{r i g h t} K_{l e f t}^{T} + U}{\sqrt{d}})

(8)

V_{a b s t r a c t i o n} = W_{a b s t r a c t i o n} Q_{r i g h t}

(9)

O_{m o d i f i e d} = A_{m o d i f i e d} V_{a b s t r a c t i o n}

(10)

We add a linear layer neural network between the left side (similar to the encoder part (Vaswani et al., 2017)) and right side (similar to the decoder part (Vaswani et al., 2017)) of the network acting as an abstraction operation. Furthermore, as shown in the model structure diagram in Figure 1, there is at least one masked operation between the input and output to reinforce the model to learn the conditional probability distribution of $s_{t + 1}$ given $s^{t}$ .

We train the model to approximate $P_{W_{c}} (s_{t + 1} | s^{t})$ by feeding event traces $σ^{w_{l}}$ , where $l \in [0,)$ , to the input, and we compute the categorical cross-entropy loss (as shown in equation (11)) from the output of the model against the ground truth $σ_{\leftarrow}^{w_{l}}$ as shown to update $θ$ using the gradient descent

l o s s = \sum_{t = 0}^{m - 1} \sum_{e = 0}^{E - 1} y_{\leftarrow e, t}^{w_{l}} \log {P^{θ} (σ^{w_{l}})}_{e, t}

(11)

where

y_{\leftarrow}^{w_{l}}

is the one-hot vector notation of

σ_{\leftarrow}^{w_{l}}

. In this way, we can input any arbitrary

s^{t}

to get an approximation of

P_{W_{c}} (s_{t + 1} | s^{t})

, where

t \leq m

After parameterisation of the original construct, $P_{W_{c}} (s_{t + 1} | θ, s^{t})$ is approximated using a modified transform layer. We can sample over $P_{W_{c}} (s_{t + 1} | θ, s^{t})$ to obtain the general learning sequence stated in Definition 5 with an initial sequence $s^{m} = [s_{}, s_{}, \dots, s_{}]$ as the input value. Although more sophisticated approaches, such as a beam search (Borgeaud & Emerson, 2019), can be adopted into this framework to sample over $P_{W_{c}} (s_{t + 1} | θ, s^{t})$ , the naive greedy approach (i.e. pick the event with the highest probability) is introduced as the baseline for evaluation. Finally, as there is a time step component $t$ introduced in the output format of this framework, the general learning sequence can be visualised using an animated state transition graph $A S T G = [s_{} \to s_{< e v e n t X >} \to \dots]$ , in which the state transition graph shows $s_{< a >} \to s_{}$ if and only if a and b are two distinct events. The ASTG formulation captures the essential information for educators and course instructors, as we are interested in the general learning progression.

Based on this framework, we created a web application for users to train their models and visualise the output of the probabilistic model using a Sankey diagram and the animated state transition graph. Figure 2 presents an instance of an animated state transition graph. In contrast to other frameworks, the animated state transition graph lays out the information in the time domain rather than only in the space domain. This helps to avoid overwhelming users with the spaghettified diagrams (van Dongen & Adriansyah, 2010) introduced by heuristic mining and the complex notations (Bogarín et al., 2018) introduced by fuzzy abstraction.

In our framework, a sequence alignment algorithm is used to evaluate the general learning sequence $G L S$ against $W_{c}$ . The Smith–Waterman algorithm is widely adopted in bioinformatics research to find the best subsequence alignment between a base sequence and a query sequence (Xia et al., 2021).

The Smith–Waterman algorithm comprises two main steps: matrix filling and backtracking (Xia et al., 2021). The matrix filling step computes the alignment matrix, of which its maximum value can be regarded as the local alignment score between $σ^{w_{i}}$ and the $GLS$ . Consequently, it can be used as a parametric evaluation of the $G L S$ against $W_{c}$ .

The parametric alignment evaluation score is defined as $h_{G L S} = \frac{1}{L} \sum_{l = 0}^{L - 1} \max H^{l}$ , where L is the total number of traces in $W_{c}$ and $H$ is the alignment matrix. Concretely, a linear gap penalty is adopted as follows, where $r$ denotes the match and mismatch scores (i.e., if $σ_{t}^{w_{i}} = {G L S}_{t}$ , then $r = 2$ , and $r = - 1$ otherwise) and $r_{g a p} = 1$ refers to the gap penalty.

H_{i, j}^{l} = \max {\begin{cases} \begin{array}{l} H_{i - 1, j - 1}^{l} + r (σ_{i - 1}^{w_{i}}, {G L S}_{j - 1}) \\ H_{i, j - 1}^{l} - r_{g a p} \end{array} \\ \begin{array}{l} H_{i - 1, j}^{l} - r_{g a p} \\ 0 \end{array} \end{cases}

(12)

From $H$ , the optimal local alignment between $σ^{w_{i}}$ and $G L S$ can be produced by the backtracking phase of the Smith–Waterman algorithm using the backtracking matrix $B_{t r a c e}^{l}$ .

{B_{t r a c e}}_{i, j}^{l} = \underset{i, j}{argmax} {\begin{cases} \begin{array}{l} H_{i - 1, j - 1}^{l} + r (σ_{i - 1}^{w_{i}}, {G L S}_{j - 1}) \\ H_{i, j - 1}^{l} - r_{g a p} \end{array} \\ H_{i - 1, j}^{l} - r_{g a p} \end{cases}

(13)

In essence,

{B_{t r a c e}}_{i, j}^{l}

is an element in the backtracking matrix

B_{t r a c e}^{l}

that stores which pair of

{i, j}

H_{i, j}^{l}

is computed from. For example, if

H_{i - 1, j}^{l} - r_{g a p}

is the largest value of the set

{H_{i - 1, j - 1}^{l} + r (σ_{i - 1}^{w_{i}}, {G L S}_{j - 1}), H_{i, j - 1}^{l} - r_{g a p}, H_{i - 1, j}^{l} - r_{g a p}, 0}

{B_{t r a c e}}_{i, j}^{l}

would be assigned as

{i - 1, j}

. If

H_{i - 1, j - 1}^{l} + r (σ_{i - 1}^{w_{i}}, {G L S}_{j - 1})

H_{i, j - 1}^{l} - r_{g a p}

and

H_{i - 1, j}^{l} - r_{g a p}

are less than or equal to zero, and

{B_{t r a c e}}_{i, j}^{l}

is marked as {0}. From

B_{t r a c e}^{l}

, the trace starts from the position

i_{e n d}, j_{e n d}

, where

{i_{e n d}, j_{e n d}} = \underset{i, j}{argmax} H^{l}

. The trace then proceeds to the next

{i_{e n d - 1}, j_{e n d - 1}}

given by

{B_{i, j}}_{,}^{l}

. If

{i - 1, j}

is picked, then a gap is inserted after

σ_{i - 1}^{w_{i}}

to align with

G L S

. Similarly, if {

i, j - 1}

is picked, then a gap is inserted after

{G L S}_{j - 1}

to align with

σ^{w_{i}}

By aligning $G L S$ and $σ^{w_{i}}$ , the support $n_{s u p p o r t}$ of $G L S$ in this framework can be measured precisely to each time step $t$ and defined as $n_{t}^{s w}$ , which is the number of traces with a subsequence that optimally align with $G L S$ at time step $t$ .

{s u p p o r t}_{t} = \sum_{l = 0}^{L - 1} n_{t}^{s w} t \in [0, m)

(14)

The support measurement not only reveals the section in which $W_{c}$ supports $G L S$ precisely but can also help to trim $G L S$ to be more concise. Consequently, this can provide an abstraction of the general learning sequence similar to fuzzy algorithms but without explicitly clustering events. However, using ${s u p p o r t}_{t}$ to perform a trim is different from the frequency filtering used in heuristic process mining because $n_{t}$ originates from the optimal local alignment from the Smith–Waterman algorithm, which only searches for optimal subsequence alignment. This, in turn, makes ${s u p p o r t}_{t}$ and $n_{t}^{s w}$ sequence- and time step-dependent.

One major advantage of constructing a probabilistic model is that the model can represent a huge sample space. In our case, the size of the sample space is ${| T |}^{m}$ , which is astronomically huge compared with the size of the event log $| W_{c} |$ . Although it is fortunate that we have such a powerful model as the transformer (or other kinds of deep learning models) that can be used out of the box, sampling over the probabilistic model might be prone to high variance. In light of this, we enforce that each time step in $G L S$ requires at least ${s u p p o r t}_{t} \geq 1$ , defined as ${G L S}^{1}$ . Precisely, we define ${G L S}^{x}$ , where x is the support lower bound.

Furthermore, the support evaluation can be used as an indicator to determine learning bottlenecks incrementally. Formally, we can use equation (15) to formalise learning bottlenecks.

| {G L S}^{x} | - | {G L S}^{x + δ} | = k

(15)

If $δ$ is large and $k$ is larger than zero, we can consider the events only bounded by ${G L S}^{x + δ}$ as a significant learning bottleneck over $W_{c}$ . To produce a better visualisation of the learning bottlenecks, a series of ${A S T G}^{x}$ where $x \in [1, F]$ are dynamically presented in our web application, where $F = \max_{t} {s u p p o r t}_{t}$ . In this framework, a learning bottleneck is defined as a set of learning materials rather than a single material. To visualise a learning bottleneck in terms of educational materials, series of animated state transition graphs are dynamically presented by incrementing $δ$ by 1 according to equation (16)

| {A S T G}^{x} | - | {A S T G}^{x + δ} | = k

(16)

From equation (16), we can see that the cardinality of $A S T G$ changes with a decrease of the support lower bound if $k$ is larger than zero. More precisely, a decrease of the support lower bound encourages more educational materials to be included in $G L S$ . This, in turn, increases the cardinality of $A S T G$ .

With the formula stated above, we can derive learning bottlenecks in terms of a set of educational materials. Let $H_{s l b}$ be the set of educational materials in an ${A S T G}^{s l b}$ , where $s l b$ is the support lower bound. Given $x = s l b$ , the minimum $s l b$ that an ${A S T G}^{x}$ can have without changing the set $H_{x}$ is defined as $β$ . Similarly, the maximum $s l b$ that an ${A S T G}^{x}$ can have without changing the set $H_{x}$ is defined as $α$ , such that $H_{β_{2}} \subset H_{α_{1}}$ where $α_{1} + 1 = β_{2}$ . $Γ_{G L S}$ is defined as the maximum number of the $A S T G$ topology with a different set of educational materials given a $G L S$ .

Ω_{g} = {\begin{cases} \begin{array}{l} β_{g} - β_{g + 1} & , 0 < g {< Γ}_{G L S} \end{array} \\ \begin{array}{l} 0 & , g {= Γ}_{G L S} \end{array} \end{cases}

(17)

{H_{d}} = {H_{g}} - {H_{g + 1}}

(18)

Educational materials can be grouped based on $H_{d}$ and assigned $Ω_{g}$ as the degree to which they present an obstacle to the learning process (i.e. many students find it difficult to further explore new educational materials). In other words, we evaluate each learning material in $H$ by their power to drop the support lower bound given a decrease in the cardinality of the $A S T G$ . This, in turn, quantifies learning bottlenecks as sets of material that obstruct the progression of students’ learning process.

In summary, our proposed deep learning framework consists of the following phases: data pre-processing, model training, general learning sequence extraction, general learning sequence alignment, general learning sequence trimming and general learning sequence presentation.

The data pre-processing phase is a process of filtering out unnecessary or redundant log events. Some events are logged multiple times for each dynamic interaction, which creates an imbalance in the event log, whereas others are logged fewer times per interaction. To extract meaningful interactions in a probabilistic base framework, $W_{c}$ is normalised into $W_{c}^{r},$ where each educational material is logged r times per access. The model training phase is a process of training deep learning models. In this study, we adopt the state-of-the-art deep learning model (transformer) and propose an improved version of it. However, any kind of deep learning model can be adopted. One advantage of augmenting the data from $W_{c}$ to $W_{c}^{r}$ is that it empowers the positional embedding in transformer shown in equation (5) to have a better representation on the position t. The representation power of a distinct event time step increases r-fold with r distinct positional embedding vectors.

General learning sequence extraction is a process of sampling a general learning sequence from any probabilistic model. As there may be an imbalance between the number of traces in $W_{c}^{r}$ and the probabilistic model complexity, the general learning sequence produced from the model will suffer from high bias. Therefore, it is necessary to adjust $G L S$ using $W_{c}^{r}$ . General learning sequence trimming is a process of trimming the general learning sequence and is carried out for two reasons. The first reason is to ensure that ${G L S}^{1}$ represents $W_{c}^{r}$ accurately, which echoes the preciseness defined in genetic process mining (de Medeiros et al., 2007). The second reason to trim the general learning sequence further (i.e., to generate ${G L S}^{> 1}$ ) is to provide a series of general learning sequences for learning bottleneck identification and for sequence simplification without explicit clustering.

After trimming, ${G L S}^{1}$ is parsed into an (animated) state transition graph ${A S T G}^{1}$ . Each educational material is assigned a value $Ω_{g}$ , which represents its power to drop the support lower bound of the animated state transition graph when introduced to the graph. The last step is to convert the general learning sequence and the animated state transition graph presentation into an interface without overwhelming the user. Figure 3 shows the phases in the deep learning framework. Concretely, we demonstrate different phases of the framework using a Moodle course that is detailed in the result section.

Figure 1.

Modified transformer block. Note. The original design (Vaswani et al., 2017) separates the left part of the model and regards it as an encoder and the right part of the model as a decoder. ‘Modified single-head cross attention layer’ refers to the cross-attention layer with the masked operation as well as the abstraction layer.

Figure 2.

Animated state transition graph of a general learning sequence. Note. The red dot in the middle of an animated state transition graph represents the current instance of a general learning sequence. One important distinction between a state transition graph and an animated state transition graph is that there is a time step component represented by the animation in an animated state transition graph. At time step 1, the current state is ‘[Video]_Introduction_to_Sampling_Distribution’. In a state transition graph, the current state can either transition left or down. However, in an animated state transition graph, it can only go down, as revealed in the later time step 2. The degree of freedom upper bound of an animated state transition graph is ${| T |}^{t}$ or, more precisely, ${| T - 1 |}^{t}$ . With t > 2, the degrees of freedom upper bound of an animated state transition graph is greater than the degrees of freedom upper bound of a state transition graph, which is only ${| T |}^{2}$ .

Figure 3.

Phases of the deep learning framework for uncovering the learning progression of learners.

Results

We applied our deep learning framework to a Moodle course that introduces statistical concepts to 94 students across three cohorts and we have obtained the consent of 28 students for using their log event data in Moodle. During the course, each student is free to access the course materials apart from the post-test (final exam). The post-test (final exam) is only available at the end of the three-week course. Since Moodle logs are available at site and course level, we collected the log event data from the Moodle log database by selecting the course ID. Each event or action is related to a user’s accessing a page or course material in Moodle. For instance, if a student watched a video in Moodle, it would be recorded the event log as video log.

There are four teaching sessions spanning over the three-week course. In the first teaching session, we introduce the Moodle course and ask the students to complete a pre-test. We also introduce the topic ‘Sampling Distribution’ in this session. A week after the first session, we host another teaching session to go over the topic ‘Central Limit Theorem’, summarizing the teaching materials that should be covered. In the third session, we ask students to cover the topic ‘Confidence Interval and Hypothesis Testing’ in preparation for the post-test in the final session. In the fourth session, we ask the students to complete the post-test.

There are 33 course materials listed in order according to the layout of the Moodle page. The course contains four major sections, covering sampling distribution, the central limit theorem, the confidence interval and hypothesis testing. Each section contains different types of educational materials, including quizzes, videos and simulations. At the start and the end of the course, students are strongly encouraged by teachers to complete a pre-test and post-test, respectively. Tables 1 –4 show the educational materials in the order of their layout in Moodle by topic.

Table 1.

Course Materials in the Statistical Concepts in Higher Education Course Topic ‘Sampling Distribution’.

Type	Title
Video	Introduction to basic statistics
Video	Introduction to sampling
Video	Introduction to sampling distribution
Video	Four distributions page
Video	Sampling distribution of sample mean
Simulation	Sampling simulation
Quiz	Definition of sampling distribution
Quiz	Calculation of sampling distribution of sample mean

Table 2.

Course Materials in the Statistical Concepts in Higher Education Course Topic ‘Central Limit Theorem’.

Type	Title
Video	Central limit theorem page
Quiz	Definition of central limit theorem
Simulation	Central limit theorem simulation
Video	Calculation of probability page
Quiz	Computation of probabilities

Table 3.

Course Materials in the Statistical Concepts in Higher Education Course Topic ‘Confidence Interval’.

Type	Title
Video	Introduction to confidence interval page
Quiz	Interpret confidence interval
Quiz	Calculate confidence interval
Simulation	Confidence interval simulation
Video	Effect of confidence level and sample size on confidence interval
Quiz	How changing confidence level and sample size influence confidence interval
Notes	Notes for confidence interval

Table 4.

Course Materials in the Statistical Concepts in Higher Education Course Topic ‘Hypothesis Testing’.

Type	Title
Video	Introduction to hypothesis testing
Quiz	Determine the null and alternative hypotheses
Video	Hypothesis testing for mean
Video	Determine critical region
Quiz	Determine the critical region
Quiz	Hypothesis testing for a mean with known variance
Notes	Notes for p-value confidence interval and hypothesis testing
Simulation	Hypothesis testing simulation
Notes	Extension knowledge for hypothesis testing
Video	p-value
Video	Type I and II error

Although the animated state transition graph reveals the structure of the Moodle course and the section order of the Moodle course layout, only quizzes are included in the graph, with all other materials treated as noise. This phenomenon is caused by the imbalanced event counts between quizzes and other educational materials: each attempt at a question in a quiz is recorded repeatedly as a quiz event, whereas only one event is recorded when the other educational materials are accessed.

In light of this, the event log $W_{statistics course}$ was pre-processed into $W_{statistics course}^{r}$ , which refers to balancing each log event count to be repeated $r$ times and removing the Moodle course homepage access event ‘\\core\\event\\course_viewed’. By removing this event and rebalancing the log count with r, a high-resolution view of the general learning sequence can be mined. $W_{statistics course}^{5}$ was used as the event log with $m = 398$ and $l = 27$ as we also filter out student whom only access one course material. Through event base data augmentation, the positional embedding layer provides a better representation on each event time step. With r = 5, there are 5 distinct positional vectors to represent each event time step.

To demonstrate the evaluation metrics

h_{G L S}

of our deep learning framework, we compare two deep learning models that differ only by the modification illustrated in equation (10). We tested whether the modified cross-attention layer outperforms the model without the modification, we ran 10 tests against

W_{statistics course}^{5}

on both models. We used 150 as the word embedding size for each model, and each model was trained with 35 epochs. After training each model, a naive greedy approach (select the event via highest probability) was used to extract the most likely general learning sequence from the distribution approximator with an initial sequence

s^{m} = [s_{< b o s >}, s_{< p a d >}, \dots, s_{< p a d >}]

as the input value. During each test,

h_{G L S}

was computed for comparison, and the results are shown in Table 5.

Table 5.

$h_{G L S}$ Mean Comparison of a Model With the Modification Shown in equation (10) and a Model Without the Modification.

Model	n	$h_{G L S}$ Mean	$h_{G L S}$ Std Deviation	$h_{G L S}$ Std Error Mean
Without the modification	10	26.81	6.60	2.09
With the modification	10	35.69	10.34	3.27

Table 5 shows that the model with the masked cross-attention layer significantly outperformed the model without the masked layer. Finally, we performed an independent sample t-test, and the results show that our mask modification on the cross-attention layer achieved a significantly higher $h_{G L S}$ value with $p < 0.05$ .

The blue line in the graph in Figure 4 represents the support of the general learning sequences produced by the model with the modified cross-attention layer measured against the event log $W_{statistics course}^{5}$ , and the area shaded in light blue around the line represents the 95% confidence interval. Similarly, the orange line shown in the graph represents the support of the general learning sequences produced by the model with the original cross-attention layer measured against the event log $W_{statistics course}^{5}$ , and the area shaded in light orange around the line represents the 95% confidence interval.

Figure 4.

The support value computed per time step using equation (14) on the general learning sequences produced from Table 5. Note. The accumulated trace count refers to the number of traces with a length greater than the GLS time step t.

The black line shown in the graph represents the accumulated trace count up to time step $t$ . For instance, at $t = 1$ , there are 28 event traces with lengths greater than 1. As the time step $t$ progresses towards $m$ , the accumulative trace count decreases towards 1. The accumulative trace count can be regarded as the resource for the Smith–Waterman algorithm to search for the optimal subsequence. Overall, the general learning sequences produced from the model with the modified cross-attention layer perform significantly better than those produced from the model with the original cross-attention layer given an accumulated trace count larger than 18.

Although a long trace may have a better likelihood of aligning with a general learning sequence with a longer subsequence, the section of subsequence alignment may not necessarily be supported towards the end of the sequence. As a result, there is a dip in the support for around one third of the general learning sequences produced from both the masked and non-masked models when the accumulated trace count drops below 18.

These results revealed that the deep learning model with a masked cross-attention layer outperformed other models in the region in which the accumulated trace count is larger than 30. Using the deep learning model with better performance, we mined ${A S T G}^{1}$ (with $h_{G L S} = 44.56$ ), as shown in Figure 5, which follows the course structure outlined in Figure 6.

Figure 5.

${A G L S}^{1}$ for uncovering the learning progression of learners with the highest $h_{G L S}$ of the 10 trials (44.56). Note. This is an instance capture of ${A S T G}^{1}$ . ‘BOS435’ is the tag name of the start event $s_{}$ . The dynamics can be visualised in our web application, shown in Figure 2.

Figure 6.

The structure of the online learning course Statistical Concepts in Higher Education. During the course, we guided the students to follow the designated structure. Moodle includes the event ‘\\core\\event\\course_viewed’, which is triggered when a student accesses the front page of the Moodle course. Figure 7 captures an instance of the ${A S T G}^{1}$ generated from our web application when $W_{statistics course}$ was loaded without data pre-processing.

Overall, ${A S T G}^{1}$ shows that the students followed the course design (Figure 7). Figure 8 shows the $Ω_{g}$ of each educational material in ${G L S}^{\geq 1}$ in orange and the minimum support lower bound in blue. The ‘[Quiz] Calculate Confidence Interval’ quiz had the highest power ( $Ω$ = 7) to drop Beta (disrupt the learning progression).

Figure 7.

${A S T G}^{1}$ produced without data cleaning of $W_{statistics course}$ . Note. The animated state transition graph before data cleaning shows only quizzes in the online learning course Statistical Concepts in Higher Education as each attempt at a question in a quiz is recorded repeatedly as a quiz event, whereas only one event is recorded when the other educational materials are accessed. ‘BOS435’ represents the start sequence event.

Figure 8.

The power to terminate the learning progression ${A S T G}^{\geq 1}$ s computed from different sets of course materials measured using $Ω_{g}$ as defined in equation (17). Note. Beta is defined in equation (17) as the minimum support lower bound of ${A S T G}^{β \leq x \leq α}$ .

Less significant learning bottlenecks were the set {‘Notes for Confidence Interval’} and the set {‘Pre Test’, ‘[Video] Introduction to Basic Statistics’, [Quiz] Definition of Sampling Distribution}. Using such an analysis, course instructors can identify which topics require most attention. In addition to the parametric analysis of the learning bottleneck, our web application allows course instructor to visualise the topology change.

Figure 9 shows the topology of the animated state transition graph ( ${A S T G}^{6}$ ). ${A S T G}^{6}$ reveals the learning progression of students who have overcome the major learning bottleneck ‘[Quiz] Calculate Confidence Interval’.

Figure 9.

Animated state transition graph topology ${A S T G}^{6}$ . Note. Once students overcome the learning bottleneck ‘[Quiz] Calculate Confidence Interval’ (labelled in the red box), they would follow the progression shown in the green box.

In summary, the results show that the deep learning models are able produce a graphical representation of the overall educational process from event logs. The results also demonstrate how Smith–Waterman algorithm from the bioinformatics field evaluates general learning sequences generated from deep learning models. The results show that modifying the cross-attention layer in deep learning models can better approximate the distribution of general learning sequences, yielding significantly better results with p < .05 under our deep learning framework using 28 students’ event logs from a Moodle course that introduces statistical concepts with 33 course materials and found that the modified model outperforms the other. Last but not the least, the results also show that a significant learning bottleneck ‘[Quiz] Calculate Confidence Interval’ can be found by examine against the general learning process model under the framework.

Discussion and Conclusion

There are two types of search strategies for tackling the EPM process model discovery challenge: local search strategy and global search strategy. Frequency-based methods, such as the heuristics process mining algorithm and alpha algorithm, are regarded as local strategies, whereas sampling-based methods, such as the genetic algorithm, are regarded as global strategies. Although the framework proposed in this study is grounded on frequency, the parameterisation step allows global search methods to be incorporated into the search algorithm. There are seven phases in this framework: data pre-processing, model training, sequence extraction, sequence evaluation and sequence alignment, sequence trimming, state transition graph extraction and bottleneck evaluation.

In general, it is difficult to approximate the probability distribution with a small dataset, especially in a model with many parameters such as deep learning models. A large dataset not only provides sufficient data for deep learning models to construct a general learning sequence with higher support but also provides a variety of subsequences to find learning bottlenecks with higher fidelity. The Smith–Waterman algorithm is chosen in this study rather than the Needleman–Wunsch algorithm for sequence alignment because the Smith–Waterman algorithm finds the optimal subsequence alignment instead of the global alignment. The Needleman–Wunsch sequence alignment algorithm is designed to find the optimal alignment between the base sequence and the query sequence. This algorithm assumes that the lengths of the base sequence and the query sequence are the same. In contrast, the Smith–Waterman local sequence alignment algorithm does not make this assumption. As for learning progress discovery, we do not assume that the length of the general learning sequence and the length of a randomly sampled learning trajectory in the event log are the same. Therefore, we adopted the Smith-Waterman algorithm in our framework. Table 6 summarises the major differences between the deep learning framework proposed in this study and previous frameworks for uncovering the learning progression of learners.

Table 6.

Comparison of the Deep Learning Framework and Other Frameworks for Uncovering Learning Progression.

	Deep learning framework	Other frameworks
Output degrees of freedom	${\| T \|}^{m}$ or $F * {\| T \|}^{m}$	${\| T + P \|}^{2}$ (P is the number of non-event related operations)
Output	General state transition sequence and ${A S T G}^{> 1}$	Specific (Petri net, fuzzy model, heuristic net)
Evaluation type	Parametric and time-point specific evaluation (optimal subsequence alignment)	Parametric evaluation
Visualisation	Series of animated state transition graphs ordered by support value lower bound	Graph with values on the edges and nodes
Learning bottleneck metric	Optimal subsequence alignment voting	Frequency base (Bogarín et al., 2018)
Learning bottleneck evaluation	$Ω_{g}$ over a set of educational materials in an ${A S T G}^{1}$ and ${A S T G}^{x}$ topology changes over changes of Beta	Qualitative analysis
Learning progress problem specificity	Specific	Non-specific

If the number of students in a course (and thus the data size) is small, course instructors may find it difficult to train a model with a large number of parameters. Therefore, deep learning models with fewer parameters should be adopted if there are only a small number of traces in the log. In this study, the model complexity (i.e., the embedding size) and training epochs are fixed for comparison;

Figure 10 illustrates how neural architecture search techniques can be merged into our deep learning process mining framework. The adoption of a neural architecture search framework would allow for the auto-tuning of model hyperparameters such as embedding size. In essence, a neural architecture search aims to find the best neural architecture given a dataset, a definition of the dedicated search space and an evaluation metric. In our deep learning framework, we properly define the event log $W_{c}$ and the parametric evaluation $h_{G L S}$ as the evaluation metric for general learning sequence evaluation. The only step required to move from our deep learning framework to a neural architecture framework is a proper definition of the neural architecture search space. With sufficient computational resources, a larger search space can be incorporated into the framework, and new models can be discovered through extensive neural architecture search in the future.

Figure 10.

Deep learning framework for uncovering the learning progression of learners with a neural architecture search.

The deep learning framework presented in this study can be applied in pedagogical design of Massive open online courses (MOOCs) host in LMS (Oh et al., 2019). Oh et al. (2019) points out that many e-learning design principles, such as practice domain-specific problem-solving skills, were applied to a limited degree regardless of levels of course difficulty. In contrast, Oh et al. (2019) found that those principles related to the organization and presentation of MOOCs content are important. One of the major contributions of the deep learning framework presented in this study is that a set of graphs is presented to the teachers for them to re-organise the course layout for students to overcome the learning bottlenecks whilst other frameworks only provide a metric base evaluation on the learning bottlenecks.

In conclusion, we propose a novel framework that allows deep learning models to perform learning sequence discovery. This framework enables the use of neural architecture search techniques to uncover the general learning sequence of students regardless of the size of the dataset. Sequence alignment algorithm from bioinformatics is adopted in the evaluation, allowing for a detailed evaluation of the support of each time-step of the general learning sequence. Using our framework, we suggest that the modification of the cross-attention layer in deep learning models helps to approximate the distribution of general learning sequences and produce a significantly better result with p < .05. Finally, we quantify learning bottlenecks as sets of material that obstruct the progression of students’ learning process and formally as the set of materials that significantly lower the support lower bound in an animated state transition graph.

Ethical Statement

Ethical Approval

The submitted work is original and has not been published elsewhere in any form or language, partially or in full. The authors have included the ethical approval and all of the relevant declaration statements in the submission of this manuscript.

Data Availability Statement

The data given this article are available from the corresponding author on reasonable request.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Informed Consent

The authors have obtained the approval of the ethics committee of the University for research involving humans and the informed consent of the human participants in this study.

ORCID iD

Siu-Cheung Kong

Author Biographies

Siu-Cheung Kong is Research Chair Professor of E-Learning and Digital Competency in the Department of Mathematics and Information Technology and Director of Artificial Intelligence and Digital Competency Education Centre at The Education University of Hong Kong. His research interests include pedagogy in digital classroom, teacher development, mathematics education, computational thinking education, STEM education, artificial intelligence literacy education, metaverse literacy education, and policy on digital technology in education.

Chun Yan Enoch SIT received his Bachelor’s degree in Medical Engineering, MSc in Computer Science and M Phil in Biomedical Engineering from the University of Hong Kong. His research interests include algorithm design and learning analytics.

References

Aldiab

Chowdhury

Kootsookos

Alam

Allhibi

(2019). Utilization of learning management systems (LMSs) in higher education system: A case review for Saudi Arabia. Energy Procedia, 160, 731–737. https://doi.org/10.1016/j.egypro.2019.02.186

Bartley

S. J.

Golek

J. H.

(2004). Evaluating the cost effectiveness of online and face-to-face instruction. Journal of Educational Technology & Society, 7(4), 167–175. http://www.jstor.org/stable/jeductechsoci.7.4.167

Bogarín

Cerezo

Romero

(2018). A survey on educational process mining. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(1), e1230. https://doi.org/10.1002/widm.1230

Borgeaud

Emerson

(2019). Leveraging sentence similarity in natural language generation: Improving beam search using range voting. arXiv, Preprint 1908.06288.https://doi.org/10.48550/arXiv.1908.06288

de Medeiros

A. K. A.

Weijters

A. J.

van der Aalst

W. M.

(2007). Genetic process mining: An experimental evaluation. Data Mining and Knowledge Discovery, 14(2), 245–304. https://doi.org/10.1007/s10618-006-0061-7

Yang

Hung

Shelton

(2020). Educational data mining: A systematic review of research and emerging trends. Information Discovery and Delivery, 48(4), 225–236. https://doi.org/10.1108/IDD-09-2019-0070

Günther

C. W.

van der Aalst

W. M. P.

(2007). Fuzzy mining – adaptive process simplification based on multi-perspective metrics. Lecture Notes in Computer Science, 4714, 328–343. https://doi.org/10.1007/978-3-540-75183-0_24

Gupta

E. P.

(2014). Process mining a comparative study. International Journal of Advanced Research in Computer and Communications Engineering, 3(11), 5. https://doi.org/10.17148/ijarcce

Hernández-Blanco

Herrera-Flores

Tomás

Navarro-Colorado

(2019). A systematic review of deep learning approaches to educational data mining. Complexity, 2019. https://doi.org/10.1155/2019/1306039

10.

Hsiao

Huang

J. C.

Huang

A. Y.

O. H.

Yin

C. J.

Yang

S. J.

(2019). Exploring the effects of online learning behaviors on short-term and long-term learning outcomes in flipped classrooms. Interactive Learning Environments, 27(8), 1160–1177. https://doi.org/10.1080/10494820.2018.1522651

11.

Khodabandelou

Hug

Deneckere

Salinesi

(2013). Process mining versus intention mining. In Lecture notes in business information processing (pp. 466–480). https://doi.org/10.1007/978-3-642-38484-4_33

12.

Mustafa

A. S.

Ali

(2023). The adoption and use of moodle in online learning: A systematic review. Information Sciences Letters (Online), 12(1), 341–351. https://doi.org/10.18576/isl/120129

13.

Chang

Park

S. W.

(2019). Design review of MOOCs: Application of e-learning design principles. Journal of Computing in Higher Education, 32(3), 455–475. https://doi.org/10.1007/s12528-019-09243-w

14.

Rozinat

A. A.

Van Der Aalst

W. M. P.

(2006). A Conformance testing: Measuring the fit and appropriateness of event logs and process models. In Lecture notes in computer science (pp. 163–176). https://doi.org/10.1007/11678564_15

15.

Shrestha

Pokharel

(2021). Educational data mining in moodle data. International Journal of Informatics and Communication Technology, 10(1), 9. https://doi.org/10.11591/ijict.v10i1.pp9-18

16.

Trcka

Pechenizkiy

(2009). From local patterns to global models: Towards domain driven educational process mining. International conference on intelligent systems design and applications (ISDA), Pisa, Italy, 30 November 2009–02 December 2009. https://doi.org/10.1109/isda.2009.159

17.

Van der Aalst

Weijters

Maruster

(2004). Workflow mining: Discovering process models from event logs. IEEE Transactions on Knowledge and Data Engineering, 16(9), 1128–1142. https://doi.org/10.1109/TKDE.2004.47

18.

van Dongen

B. F.

Adriansyah

(2010). Process mining: Fuzzy clustering and performance visualization. Lecture Notes in Business Information Processing, 43, 158–169. https://doi.org/10.1007/978-3-642-12186-9_15

19.

Vaswani

Shazeer

Parmar

Uszkoreit

Jones

Gomez

A. N.

Polosukhin

(2017). Attention is all you need. Advances in Neural Information Processing Systems, 30, 6000–6010. https://doi.org/10.48550/arXiv.1706.03762

20.

Weijters

van Der Aalst

W. M.

De Medeiros

A. A.

(2006). Process mining with the heuristics miner-algorithm. Technische Universiteit Eindhoven Tech Rep WP , 166(July 2017), 1–34.

21.

Xia

Cui

Zhang

Tang

Peng

Huang

Yang

Liao

(2021). A review of parallel implementations for the Smith–Waterman algorithm. Interdisciplinary Sciences: Computational Life Sciences, 14, 1–14. https://doi.org/10.1007/s12539-021-00473-0