Learner Experiences During the Design-Based Research Process for a Problem-Based Instructional Design Course

Abstract

The discrepancy between students’ understanding of domain knowledge and their actual competencies to use tacit knowledge for a given task has been an issue to consider when instruction is designed, implemented, and revised. In this study, the researcher examines teaching and learning processes in which a Problem-Based Learning (PBL) design model was crafted and used for an advanced instructional design course in the educational technology field. The focus is on the researcher‘s ongoing and recursive reflective experiences in reviewing design components of the model, analyzing the motivators and difficulties students faced, and offering alternative methods to improve the initial design. The findings show that the PBL approach played an instrumental role in creating real challenges that the target professionals may experience. On the other hand, observed difficulties in PBL include intrinsic and extrinsic cognitive loads associated with the complexity of the instructional design process, self-regulated learning skills, learning beliefs, and supporting strategies used for team dynamics. The patterns identified in the behaviors and perceptions of effective and struggling students are interpreted. Based on the findings, new constructs of the model were suggested for further improvement.

Keywords

problem-based learning design-based research situated learning PBL challenges scaffolding negative feedback collaboration with SMEs formative research

Introduction

Instructional Design (ID) is a somewhat fuzzy concept for new students in the field to easily make sense of, define, and use appropriately. At the same time, advanced students often find it challenging when faced with a gap between knowing what ID is and applying tacit knowledge to a real-world problem as ID practitioners typically do in the workplace. The difficulties students experience may relate to the nature of ID as an ill-structured problem (Jonassen, 2000) and insufficient real-life experiences encountered by students during their academic programs. In reality, ID professionals are frequently asked to design instruction for subject content not familiar to them, make sense of data available in relation to the instructional conditions to be designed, create solutions, and conduct projects in collaboration with other professionals with different expertize, both inside and outside of organizations. Achieving these goals and meeting expectations entails more than understanding domain knowledge, i.e., the theories, principles, and models provided in textbooks. Rather, it requires the use of those complex cognitive and metacognitive competencies a professional possesses, such as, planned inquiry, evidence-based decision making, thoughtful interpretation, and creative solutions, as well as self-regulatory competencies in controlling their own learning process. When it comes to the development of higher order thinking skills, researchers draw attention to new teaching and learning approaches that reflect paradigm shifts in our society (APA Work Group of the Board of Educational Affairs, 1997; Cognition and Technology Group at Vanderbilt, 1992; Reigeluth, 1997). In this respect, experiential learning (Kolb, 1984), constructivism (Duffy & Jonassen, 1992), problem-based learning (Barrows, 1996), and situated learning (Lave & Wenger, 1991) have been advocated. In this common active learning environment, students are encouraged to deal with real problems, take risks, and enjoy the freedom to generate creative solutions.

Insights into the nature of ID as a systemic process (Molenda, 2009), a cyclical process reflecting real life ID practices (Dorsey et al., 1997; Gustafson, 2000; Tessmer & Wedman, 1990), and a complex problem-solving procedure (Foshay et al., 2003) point to the importance of understanding the interrelationship between essential ID constructs and the creative application of knowledge grounded in research and practice in order to design instruction. In addition, they lead to a question regarding how ID programs ensure that students practice, internalize, and use a set of competencies (i.e., such as the application of systemic processes or heuristics) and enhance proficiencies that can steer their careers after graduation.

One main obstacle is identified when new approaches, such as experiential PBL in educational technology programs, are implemented. Specifically, when individual courses in ID programs focus on one or two discrete phase(s) of ID, such as “Analysis and Design” or “Development and Evaluation,” they may function well in introductory courses that are designed to provide a foundation in the field. However, the isolated focus of each course is not sufficient to build a complex learning environment that meaningfully engages students in the whole ID process as demanded by most ID related jobs, whether it is executed linearly, cyclically, or both, in their daily practices.

Given that the part-task for each stage of the process and the whole-task (Merrill, 2002; van Merrienboer, 1997) are equally important for acquiring domain knowledge and skills, the lack of fidelity and complexity embedded into isolated learning events is problematic because it is less likely to foster well-rounded skills that can be transferred to a future profession. It is not unusual to see that employers express their concerns about skill deficiencies between what they expect from new employes and what they actually possess (Horn, 2014; Kuhlmann, 2013a). Hansen argues, “There is a direct disconnect between education and employability, where employers view universities and colleges as the gatekeepers of workforce talent, yet those same institutions aren‘t prioritizing job skills and career readiness” (2021, p. 3). Moreover, it seems unreasonable to expect students to synthesize knowledge from fragmented experiences without meaningful opportunities to do so.

The purpose of this study is to examine PBL as a design principle used to support the complex ID process in an advanced ID course. This study looks at how the PBL framework framed by the researcher was implemented and functioned to promote problem-based learning. The analysis and discussion are developed around the constructs of the PBL model, scaffolding strategies, and pedagogical challenges encountered during the PBL process. The patterns identified in the behaviors and perceptions of effective and less effective students are described and interpreted.

Problem-Based Learning

PBL as an instructional method has been considered a potential approach to foster higher cognitive and metacognitive skills including analytical, reasoning, and decision making, as well as self-regulated, reflective learning (Barron et al., 1998; Brush & Saye, 2014; Dunlap, 2005). The existing PBL models (Barrows, 1986; Duffy & Cunningham, 1996; Ge & Land, 2004; Hmelo-Silver, 2004; Hung, 2006; Jonassen, 1997; Savery, 2009) provide insights into its theoretical assumptions and core components that interplay within the learning environment represented by each model. The theoretical conception of PBL draws on experiential learning (Kolb, 1984), constructivism (Duffy & Jonassen, 1992), and situated learning (Lave & Wenger, 1991). These perspectives have overlapped in so far as the instructional process is driven by real-life, ill-structured problems; during the application of problem solving, learners are placed at the center of the instruction and their active role in learning is emphasized. Contextualized knowledge construction is underscored through nurturing meaningful experiences, reflection, self-regulation, and collaboration. The implication for designing PBL instruction can be delineated as (1) it is crucial to engage students in an open and active learning process, in which they deal with real-world problems through involvement in constant negotiations and sense-making; (2) since PBL is likely to place more cognitive demand upon students, and more on less prepared students, selecting appropriate scaffolding and making on-going modifications to course strategies and evaluation methods should help avoid using one set of criteria for different student teams; and (3) learners’ own strategies to manage the learning process should be encouraged and successful strategies that work well in the PBL environment should be promoted.

Along with the PBL theoretical assumption, it is noted that different kinds of PBL problems may lead to different learning outcomes (Jonassen, 2000). For instance, ill-structured problem solving pursues creative solutions, drawing on multiple perspectives and evidence. It is inevitable that this process inherently requires using varied types of knowledge that build on one another (Smith & Regan, 2005). Accordingly, different types of knowledge base (i.e., declarative, principles, procedures, problem-solving, etc.) necessitate diverse approaches and alternative ways of nurturing (Gagne, 1985). This entails that any single learning perspective may not effectively address combined learning objectives (Strobel & van Barneveld, 2009). Attention is given to an eclectic approach (Reigeluth, 1999) that supports drawing on both traditional (e.g., behaviorist or supplantive) and non-traditional (e.g., constructivist or generative) principles in order to teach the varied types of effective skill sets or to address relevant learning needs (Cronje, 2006).

The successful adoption of PBL requires a critical transition of teaching and learning beliefs and practices. Research has documented that for students who are accustomed to passive learning or have rarely been exposed to self-regulated learning, the behavioral and psychological changes necessary for PBL might not occur or may occur very slowly, due to underlying beliefs in what learning and teaching should be (Albanese, 2000; Georgiou et al., 2002; Maclellan & Soden, 2003). To support smooth transitions, researchers have suggested critical discussion on the epistemological beliefs of both students and instructor (Song et al., 2007). Discussing the value assigned to course activities is necessary for reflecting on the new roles of students and instructor in the PBL setting. In addition, to foster learners’ active engagement in the learning process, integrated support systems for learning activities are examined and discussed.

Hannafin et al. (1999) have categorized scaffolding strategies into four groups to support the cognitive and metacognitive process for complex problem solving. Conceptual scaffolding supports the reasoning process by identifying key conceptual knowledge relevant to a given problem and builds structures that can illustrate conceptual organization. Metacognitive scaffolding supports the thinking process, reflection, and learning management. Procedural scaffolding provides available resources and tools and makes them accessible to learners. Strategic scaffolding supports alternative approaches that help reduce cognitive load by aiding selection of information, evaluating resources, and linking new to old knowledge.

Hmelo-Silver and Barrows (2006) highlight strategies an expert used for students’ reasoning and the self-directed learning process. Their research underscored the importance of interactive knowledge construction through modeling, scaffolding, and fading during small group discussions. Delclos and Harrington (1991) found that problem-solving training improved students’ monitoring ability. The treatment group that was instructed to self-monitor their own problem-solving process excelled at difficult problems and use of metacognitive strategies.

In Dunlap (2005), reflective journal writing was used as an instructional tool to foster reflection as well as to examine changes in students’ perceived self-efficacy while they were participating in PBL. Students were required to write a journal entry once every three weeks throughout the semester, and the activity was guided by a series of questions. Journal writing provided a useful tool in the study to help engage students in reflection on their performance and in observing changes in self-efficacy perceptions. King (1991) found that the guided questions and answers used for the problem-solving process increased performance during cognitive and metacognitive activities in a pair group.

Ariely and Wertenbroch (2002) assessed how self-imposed deadlines affect the performance of participants. For participants who had self-control problems and failed to set deadlines optimally, flexibility did not affect their performance positively. For participants without self-control problems, self-imposed strategies helped them to manage procrastination. However, self-imposed deadlines were less effective than the deadlines established by the instructor.

However, despite its popularity in education, there are ongoing debates around the effectiveness of PBL (Hmelo-Silver et al., 2007; Kirschner et al., 2006; Norman & Schmidt, 1992; Strobel & van Barneveld, 2009). Hung (2011) explained the conflicting findings in PBL research by examining research methods and implementation. Regarding the implementation issue, he points out, “a variety of student and instructor behaviors that occurred during actual implementation were contrary to PBL‘s theoretical assumption and could have confounded the PBL research” (p.548). Most research studies have discussed the effectiveness of PBL by focusing on student learning outcomes. The controlled settings and instructional intervention that was implemented led researchers to overlook the impact of numerous uncontrollable variables on student learning outcomes. Hung calls for further research that examines the actual implementations, discusses what has actually been done and what factors interact and influence successful or less successful learning outcomes, and what differences are found between the conception of PBL and its application to classrooms.

Considering the insights and concerns regarding PBL and the unique needs of this research study, the researcher found that the existing models do not quite fit the learning conditions of this study. Therefore, it became necessary to develop a new design framework that could provide practical guidance for selection of strategies and for support of students. The researcher identified essential components within the research context and drafted the initial framework. Following this, the course structure, strategies, and activities were deliberately aligned with the framework. The initial model was also used when the researcher measured learning effectiveness.

Formative Research

This design-based research examines PBL as a design principle which was used to support the complex ID process in an advanced ID course. Wang and Hannafin (2005) identify the characteristics of design-based research: “(a) pragmatic; (b) grounded; (c) interactive, iterative, and flexible; (d) integrative; and (e) contextual. In essence, the design-based research methodology support systematic, systemic, research-based, data-driven ID, implementation, and evaluation to make the designed interventions practical, context-specific, and ultimately generalizable to similar contexts. The same researchers categorize several similar methodologies into the design-based research group. Some of them consist of design-based research (Design-Based Research Collective, 2003), design experiments (Brown, 1992; Collins, 1992), developmental research (Richey, Klein, & Nelson, 2004), and formative research (Reigeluth & Frick, 1999).

In this current study, the researcher adopted the formative research methodology (Reigeluth & Frick, 1999) which was proposed to guide and support educators when they engage in the reflective process of reviewing and improving their own instructional practices. Drawing on formative research, the researcher in this study formulated the tentative PBL model for an advanced ID course and used it for the target context for examination and revision. The design model building process falls into the “Designed Case To Improve an Existing Theory” (p.639) as one of the formative research methods. The process consists of “(1) Select a design theory, (2) Design an instance of the theory, (3) Collect and analyze formative data on the instance, and (4) Revise the instance, and Repeat the data collection and revision cycle.” (p. 639)

Initial Design Model

The initial PBL framework was developed from existing relevant research studies and the researcher‘s experience as university faculty and a learning design practitioner. The focus of the PBL model (Table 1) is on increasing relevance to a real-life ID practice whereby the students stumble through unexpected challenges, apply knowledge to the problems, make evidence-based decisions, and experiment with alternative solutions when their prior knowledge do not work for a given problem. The process requires the students to take the lead in tackling issues and collaborate with a project partner, the instructor, and the clients/SMEs. The preliminary PBL principles include: (1) Pivot towards real problems, real people, and real challenges; (2) Help shift from passive to active doers; (3) Adapt and respond to needs; (4) Rethink how to establish relationships with students; and (5) Balance structure, autonomy, and dialogue.

Table 1.

PBL Design Principles and Strategies for Advanced Learning Design Skill Development.

Design Principle	Strategies
Pivot towards real problems, real people, and real challenges	– Use problems giving mutual values to all members involved. – Adapt class requirements to PBL and create the project conditions reflecting real-world practices involving a Subject Matter Expert (SME), authoring tools on demand, and a design document, in doing so help students harness from the activities. – Mindful that a real, complex problem does not always enhance experiences. Limit scope to a given time. Avoid heavy resource intensive projects.
Help shift from passive to active doers	– Assure students are ready for what is required next. Keep ambiguity from scaring/confusing students. – Discuss, not explain how PBL differs from a conventional learning, what makes it works, rationales behind the approach, and expected roles and responsibilities. – Prepare clear guidelines for requirements, and acceptable and unacceptable behaviors.
Adapt and respond to needs	– Give heads-up for foreseen and potential challenges that could interfere with the PBL process. – Require project logs or reflection papers to keep all members in the loop and recognize issues earlier. – Instant intervention does not always work or desirable. Offer scaffolding when it is necessary for students to quickly stay out of difficulties and manage challenges. – Provide exemplary or non-exemplary work samples. – Share best practices that can model working with SMEs professionally.
Rethink how to establish relationships with students	– Do not micromanage but facilitate. Empower students to plan, monitor, and evaluate process and outcomes. – Utilize peer support and peer learning. If one team gets off the track, the other team gives advice and assistance. Give a chance to inspire and heal each other.
Balance structure, autonomy, dialogue	– Become a supportive member of the learning community. – Hold regular team meetings to avoid pitfalls of lack of teacher presence. – Offer options to choose communication tools and modes students prefer. – Require a regular team project meeting and scaffold it if needed.

More discussion about each component is presented in conjunction with the findings of the study in the analysis and discussion section.

Instance of the Model

The prototype of the PBL model was used for a graduate course taught by the researcher in the field of educational technology in a state university. The course objectives were to build a deeper understanding of the foundation of ID and to enhance skills for the real-world demands of workplaces. Five residential students and three online students enrolled in the course. All eight students were aspiring to become ID practitioners after graduation, as instructional designers, instructional technologists, or digital content developers. This blended course was offered face-to-face via the D2L Brightspace (D2L) learning management system and the Adobe Connect synchronous conferencing tool.

The content base of the course topics was even more important in this student-led learning setting. The large group synchronous (online and face-to-face) meetings were held for the first and last meetings of the semester. The remaining interactions took place through individual team meetings with the instructor. To help activate students’ prior knowledge and connect it to their problems, the course content, readings, and resource links were organized according to ID skill sets (e.g., ADDIE) and shared through D2L. In addition, the guidelines and work samples (e.g., design documents, prototypes, and formative evaluation) were provided. Online discussions were required through the course system during the first half of the semester in which the students reviewed the readings and reflected on the topics and their implications for their projects. This activity provided the instructor with useful information to ascertain how well the students understood and reflected on the topics. As the project progressed, the discussion assignment was gradually decreased to allow students more time to concentrate on the project itself.

Once the learning goals and objectives were discussed in class, the students were empowered to decide how to proceed with their ID projects. For example, it was required for the students to (1) work on real-life, not hypothetical, learning problems. At the beginning of the semester, they were provided with options to select problems from a list given by the instructor or to propose their own; (2) work with Subject Matter Experts (SMEs)/clients who were committed to regularly meet with the students, review and provide feedback on the project in progress; (3) to use multiple sources to collect and analyze relevant information/constraints around the problem and the context; (4) use e-learning design software such as Articulate Storyline, Captivate, or other tools suggested by the client. In doing so, the students were assisted in acquiring technology skill sets that are in high demand in contemporary workplaces; (5) to rationalize their process, decision, and final products drawing on evidences, relevant research, and needs.

Six out of eight of the students were grouped into two-member teams and two students chose individual projects (Table 2).

Table 2.

Students’ Design Projects, SMEs, and Authoring Software.

Team	Project	SME/Client	Design Software
A (2*)	Workshop for mediation	Faculty**	Articulate Storyline
B (2)	Library search engine tutorial	Faculty librarian**	Articulate Storyline
C (1)	Library reference tool tutorial	Faculty librarian**	Captivate
D (2)	Customer support system	A design company	Web design tools
E (1)	Research methods	Faculty & Staff***	Captivate

*Numbers in parenthesis mean the number of team members.

**Faculty members in the university the student teams attended.

***Faculty and staff of the university the student worked for.

Data Collection

To examine how the supporting strategies suggested in the PBL framework worked for this research context, the researcher used the students’ online discussion postings, bi-weekly project progress reports, design documents, working artifacts, final artifacts, the semester-end course survey data, and the instructor‘s notes. In addition, a post-project interview was conducted to gain further understanding of the perceived learning and satisfaction of the students. Three students participated in the interview. The interview questions were sent to them to obtain their written responses in advance and follow-up phone interviews were held to gain clarification and more information. The interviews lasted 20 to 40 min. The questions asked concerned their experiences in activities, assignments, learning support, flexibility, barriers, and suggestions.

Analysis and Findings

The analysis and interpretation were conducted based on the information gathered from the course activities, the anonymous semester-end survey, and the post-project interview data from the pre- and follow-up interviews. The students’ comments cited in this section are excerpted from the course survey and the written responses collected prior to the post-project interview. To protect the identity of the participants, the pseudonyms assigned to the participants are used.

Realness of Problems, People, and Challenges

In this study, the genuineness of PBL requirements played an instrumental role to create a complex and real-world environment in terms of time constraints, pressure to adapt to new roles and responsibilities, resource limitations, and the people involved. The quotes below are taken from the anonymous end-of-semester course survey (Survey).

It was the most challenging class I have ever taken. However, it was also very beneficial because it allowed me to push myself to accomplish the work that was required of me.

I really liked the fact that creating a project was the central objective to the course. I feel like we gained some wonderful experiences through actually developing a real self-paced training and working with SMEs.

One of the students who taught in a state college explained how this comprehensive design project helped to link tacit knowledge to the student‘s own course design:

The PBL approach provided the opportunity for me to create a project centered around a problem I wanted to address related to design and delivery of instruction in a course that I teach to college freshmen for [the name of the university the student worked for]; to better understand principles of instructional design in order to implement them in the redesign of lessons for that course; and to learn and use software (Adobe Captivate) as an authoring tool. (Carey)

The freedom to choose options posed benefits as well as challenges. The options including a problem to work on, design software, team members, experts to work with, and communication tools for meetings with the instructor motivated the students to learn and posed real constraints.

The benefit of choosing topics is that the topics are of interest and I was highly motivated to learn them. I suggest allowing students to choose topics in future classes. (Survey)

I liked the fact that there was a choice with the topics (that some were pre-selected and others were chosen). I was a part of a group, in which the topic was pre-selected by the professor, and I found that the experience seemed more authentic than the alternative. For example, as someone who is given the task of developing training, the developer may not have a lot of prior knowledge about the subject matter. This, I believe, is an actual challenge that developers are faced with on a regular basis. (Mary)

In case the students worked on an unfamiliar topic, the condition created the authentic barriers to handle.

Our topic was very unique that made it very interesting to learn and develop a training module for it. However, because it is an abstract subject and the success of the lesson depends on the learner‘s experience, it was hard to design a training that would reflect that self-reflection concept that the SME was conveying through his workshops. (Jane)

Each project team partnered with SMEs/clients who had expertize in the area of the subject content or/and authoring software. The SMEs who supported the projects underway did so through regular meetings, review, and feedback, and thus practically assisted the student designers’ learning experiences.

Working with [the SME] in this project has been a smooth process because [the SME] has knowledge of the audience and the content. At the meeting, besides discussing the needs assessment, [the SME] also suggested a list of crucial and optional features she wanted incorporated in the tutorial. I have read stories about uncooperative SMEs, but this is not the case with [the SME.] Having an awesome SME alleviates certain anxieties that come with being a novice instructional designer. (Mary)

I was fortunate to work with a solid team (SME and IT staff). All were essential for the successful completion of my project. (Carey)

In contrast, those who believe they did not receive supportive assistance from their SMEs struggled to complete tasks on time and were less confident with their design decision on working products.

SME was supportive but not really helpful in terms of my bringing the project to fruition. It might be helpful in the future, to show the SME an example of past projects that students have already completed-- so that they have a more accurate understanding of what is expected for this course. (Jane)

Some of the students who did not clearly grasp the role of SMEs/clients or their own role as designers with responsibility had difficulty in explicating why a certain part of the instruction they designed or that SMEs wanted may or may not increase the probability of learning success. This was a similar dispute to those often experienced by ID professionals (Kuhlmann, 2013b), thus it also provided the students with an important issue to reflect on. However, fostering clear awareness and expectations about the roles of participants would have ensured that all students participated in cooperative work in an appropriate way.

The SME was genuinely encouraging and supportive of everything that we did. It was challenging to get our SME to point, where the SME was adherent to believing that self-paced training was something worth pursuing- but was open to trying it. (Mary)

Tailored Supports

Although complex problem-solving skill development was the primary objective of the course, different learning perspectives (Reigeluth, 1992; Smith & Regan, 2005) were adopted because it remained necessary to support several types of knowledge involved in the ID process (i.e., declarative, conceptual, and problem-solving). It entails that the role of the instructor was not limited to that of a tutor as described in Barrows (1996). It was necessary for the instructor to provide direct answers when it was assessed that doing so could accelerate progress or reduce the cognitive load on students (e.g., when developing a prototype or working with difficult SMEs) (van Merrienboer et al., 2003). At the same time, the instructor‘s role as the facilitator was pivotal to guide the process, advise, and collaborate with students (Hmelo-Silver & Barrows, 2006). The supports encompassed strategies coaching, articulating, offering clarifications, and encouraging reflection as in the Collins et al. (1989). All quotes below are from the end of semester course survey.

Time with [the instructor] was used for asking questions, clarifying my understanding (especially related to my understanding of principles), and monitoring progress.

[The instructor] typically referred me to specific learning materials/evidence-based research and provided instruction and written feedback before a team meeting, and then she elaborated on the materials and her comments during meetings.

[The instructor] gave good suggestions about the project, throughout the semester. It was especially helpful conferencing with her about concerns, clarification, and suggestions about the project throughout the semester.

[The instructor] defined the outcomes of the course and explained how my project might meet them; additionally, she provided written and verbal feedback on my project, its progress, and my interactions with team members.

The importance of task analysis cannot be overstated in the ID process (Morrison et al., 2011; Smith & Regan, 2005). Although the students chose the project topics that interested them, they struggled to use their analysis to determine a lesson scope referring to the learning objectives each lesson is supposed to achieve within a given timeframe. The appropriateness of the project scope in breadth and depth needed to be determined in the early phase of the ID process as indicated in the following student quotes.

The feedback was very helpful and helped my group correct major mistakes and narrow the scope of our project. (Mary)

[The instructor] was instrumental in helping me define the scope and focus of my project and the steps needed to bring my project to fruition. (Survey)

Readiness for PBL

The degree of acceptance and level of involvement in this PBL setting were linked to the students’ long-established learning beliefs/values and learning habits derived from those beliefs. It was challenging to resolve misunderstandings from different perspectives and help the students accommodate to new roles. Researchers have discussed the perception and attitude towards the teaching beliefs grounded in course design as one of the predictors of student performance (Wu & Tsai, 2006). In this current study, resistance was observed in those students who struggled to carry out the project in a team. The students reported that they preferred direct instruction (lecture) and step by step guidance over student-led learning in which they were supposed to make sense of available data, search for resources, and learn from peers. The underperforming students seemed not to consider the topic discussion in individual meetings or course readings to be a legitimate teaching and learning approach. As a result, they believed that they did not receive support from the instructor.

The two students worked together in a struggling team. At the course survey, one of the members identified the perceived causes that brought about team conflicts, including that the other student did not contribute to the project, the workload was uncontrollable due to the inability of the other member, and communication breakdown. It is noteworthy that both students rarely verbalized difficulties during the semester and infrequently initiated contact with the instructor, while their counterparts, who seemed motivated to explore the PBL features, scheduled meetings more often, at least once a week on average. As a result, reluctance to engage in candid communication kept the students from resolving conflicts during the process. At the post-project interview, one student said that s/he was afraid of any consequences that might arise from discussing concerns with the team member and the instructor. It is difficult to see that different personal belief alone brought difficulties in their team. Other factors, fear of giving negative feedback coupled with low self-confidence (Bandura, 1997) in the challenging tasks contributed to overall unsatisfactory experiences and outcomes.

Project Management

Early in the semester, all eight students reported that they were excited to have a chance to practice skills with e-learning design software (Articulate Storyline and Captivate) that are in high demand in ID careers. However, seven out of the eight students said that, due to limited experience, they had little confidence in using those tools and their application to their projects in this course. Despite similar levels of technical readiness across the students, they used diverse strategies to cope with the same challenge. The more motivated students actively related learning goals to their needs (e.g., qualifications described in job advertisements) and used available resources to remedy skill deficiencies. For example, Alice, Mary, and Sophia learned software through online video tutorials provided by an e-learning design software company (e.g., Articulate Storyline), Mary and Alice took workshops offered by a commercial training company (e.g., Linda.com), and Carey worked with a supporting staff member who was available in her organization.

Some different approaches were taken by the struggling students. They had a difficult time identifying their skill gaps and how to acquire the skills, if identified. Recapitulating Perkins et al. (1986), Jonassen (2000) relates their findings to task persistence and effort to learning success:

If students do not believe in their ability to solve problems, they will most likely not exert sufficient cognitive effort and therefore not succeed. Their self-confidence of ability will predict the level of mindful effort and perseverance that they will apply to solving the problem.” (p.72)

In this present study, low self-confidence apparently discouraged some students from voluntarily exploring new methods and stopped them from exerting effort to complete the given tasks. In consequence, they were less enthusiastic about attempting alternatives or the available resources provided by the instructor. Although they had a chance to observe their peer students who successfully overcame similar difficulties, the observation did not lead to learning from peers.

Communication

The project team meetings were held on a regular basis. The students were asked to discuss their progress and sought assistance while the instructor offered clarification, examples, resources, and feedback. All quotes below are taken from the anonymous course survey.

One of the things I liked best about this class was individual meetings with instructors. I think the meetings were FAR more meaningful than class meetings via Adobe Connect [web-based synchronous meeting tool].

This mode of instruction was very new to me. At first, I was overwhelmed because I did not have the guidance that I usually get from group lectures. However, as time passed, I realized that I still had the teacher‘s support through the group‘s weekly meetings and that really helped me throughout the semester.

Instructional support was spot-on, and [the instructor] provided helpful suggestions for additional support/resources/materials based on my needs and the progress of my project.

To maintain ongoing interactions, the students could select communication modes—face-to-face, via phone, via the Desire2Learn BrightSpace course management system, and/or via the Adobe Connect synchronous communication system.

My interactions with [the instructor] were by phone, email, and mostly Adobe Connect since I live four hours away from [acronym of the university]; many were scheduled although some were not, especially if I had a time issue. Interactions with team members were by email but mostly in person since my team members were readily accessible. (Survey)

Discussion

The PBL model designed for an advanced ID course helped create a situated learning environment in which the student designers coped with real challenges that the target professionals may experience and created evidence-based solutions. The student-led approach was well accepted overall by most students. The SMEs acknowledged the students’ efforts and the quality of the products. All five SMEs responded that they planned to use the products for their target audiences. As PBL requires new roles and responsibilities of the students, the role of the instructor also transitioned (Maudsley, 1999). In that sense, this research context underscores that student-centeredness in PBL necessitated far more ideas, forethought, and scrutiny to design instruction and follow-up interventions (Table 3). Individual team discussion on a regular basis helped address the different needs of each team, provided personalized supports, kept track of team progress, and adjusted the predetermined requirements (e.g., deadlines, frequency of team meetings). In addition, the tailored scaffoldings and flexible options in this study were well accepted. However, not all students were capable of harnessing freedom and flexibility as in Ariely and Wertenbroch (2002). Even in the PBL setting, strict enforcement of deadlines and clearly stated written information better served those who had lower self-regulatory skills. Structured, clear guidance (Turner & Merrill, 2021) in the form of written guidelines, criteria, and a checklist benefited the students. These measures directed students to the course expectations, as well as enormously saving the instructor‘s time. Conceptual scaffolding (Hannafin et al., 1999) worked well when it was adopted for scaffolding tasks analysis and determining the project scope.

Table 3.

Revised Design Principles and Strategies and Revision.

Design Principle	Strategies	Updates
Pivot towards real problems, real people, and real challenges	– Use problems giving mutual values to all members involved. – Adapt class requirements to PBL and create the project conditions reflecting real-world practices involving SMEs, authoring tools on demand, and a design document, in doing so help students harness from the activities. – Mindful that a real, complex problem does not always enhance experiences. Limit scope to a given time. Avoid heavy resource intensive projects.	– Prescribe written protocols for roles of and supports from SMEs.
Help shift from passive to active doers	– Assure students are ready for what is required next. Keep ambiguity from scaring/confusing students. – Discuss, not explain how PBL differs from a conventional learning, what makes it works, rationales behind the approach, and expected roles and responsibilities. – Prepare clear guidelines for requirements, and acceptable and unacceptable behaviors.	– Tailor materials/tutorials that better support less effective students with low prior knowledge.
Adapt and respond to needs	– Give heads-up for foreseen and potential challenges that could interfere with the PBL process. – Require project logs or reflection papers to keep all members in the loop and recognize issues earlier. – Instant intervention does not always work or desirable. Offer scaffolding when it is necessary for students to quickly stay out of difficulties and manage challenges. – Provide exemplary or non-exemplary work samples. – Share best practices that can model working with SMEs professionally.	– Create cases in which students can identify issues faced during working with SMEs and come up with strategies to cope with the issues. – Attend to struggling students more closely for personalized feedback and resources.
Rethink how to establish relationships with students	– Do not micromanage but facilitate. Empower students to plan, monitor, and evaluate process and outcomes. – Utilize peer support and peer learning. If one team gets off the track, the other team gives advice and assistance. Give a chance to inspire and heal each other.	– Make peer support substantial by requiring both positive and negative/corrective peer feedback, so that make both practices routine parts of peer review/support (Carmichael, 2014). – Encourage students to share ideas how they handle negative feedback and make criticism less threatening, such as careful listening, any ideal ratio of praise to criticism.
Balance structure, autonomy, dialogue	– Become a supportive member of the learning community. – Hold regular team meetings to avoid pitfalls of lack of teacher presence. – Offer options to choose communication tools and modes students prefer. – Require a regular team project meeting and scaffold it if needed.	– Use an anonymous mid-term survey soliciting open feedback on teamwork and class activities (Kamp et al., 2013).

To make this initial PBL usable (Table 3), it is critical to understand what students, particularly less effective students, prefer and address this (Ertmer et al., 2009). It was apparent that reluctance to give and receive feedback hindered team collaboration in an underperforming team. Regarding the effectiveness of peer feedback, there are conflicting insights offered (e.g., Buckingham & Goodall, 2019; Chappelow & McCauley, 2019). For example, peer process feedback is effective for underperforming students that lead to more contributions to group work (Kamp et al., 2013). On the other hand, criticism/negative feedback is viewed as causes of anxiety or fear that are less likely to improve performance. Paul Green (cited in Berinato, 2018) states, “People who received negative feedback, we found, were far more likely to seek a new partner for their next task than those who received confirming feedback.” (p.32). Other researchers pay more attention to different conditions that can affect changes to behaviors and performance after the recipients receive feedback (Smither et al., 2005). For the purpose of fostering team dynamics in PBL, it would be more beneficial to focus on the factors leading to voluntary, open discussion of their team performance. Given that team conflicts are not unusual (Brownell & Jameson, 2004) and are often predictable, preparing the embedded and contingent scaffoldings (Saye & Brush, 2002) seems necessary. For example, tailored scaffoldings such as the preparatory instruction discussing the importance of peer feedback, structured guidance that requires both positive and negative feedback for peer evaluation (Carmichael, 2014), and ways to communicate concerns with team members would be incorporated into the different stages of team development.

Use of real-world problems did not guarantee that all students equally lead active learning. Self-regulatory strategies, i.e., plan, evaluate, and make decisions on issues, rather than being led by the instructor, were pivotal in this PBL context. Davidson and Sternberg (1998) noted,

Metacognitive skills help the student strategically (a) encode the nature of the problem and form a mental model or representation of its elements, (b) select appropriate plans and strategies for reaching the goal, and (c) identify and conquer obstacles that impede progress” (p.48).

In this current study, the differences were apparent when the students were asked to personalize their learning. The requirement posed a greater challenge to those students who lacked the adequate level of metacognitive skills. More effective students had a wider repertoire of strategies. They initiated conversations, asked questions more often, took advantage of the available resources, and brought ideas (e.g., alternative ways to conduct usability testing when the target audience was not accessible) and suggestions to the individual team meetings with the instructor. They were willing to take advantage of challenges presented in the tasks and considered it an opportunity to improve their skills for a future career. They rationalized their requests (e.g., adjusting the scope of the lesson, extension for assignments) and remained more reflective on their experiences throughout the process. Their active and reflective performances were presented in the quality of participation and work and the final products they submitted. Time resource management (Ruiz-Gallardo et al., 2016) truly matters to all participants in the PBL process. The struggling students benefit from the instructor imposing rules over free options (e.g., the deadline assigned to the assignments, choice of regularity of the meetings) and guided instruction for unfamiliar topics and activities.

For the struggling students, the complex problem solving and student-directed process added more load and emotional distress (e.g., active initiation and more accountability). This response was found in common among those who experienced team conflicts and or who struggled to voluntarily carry out the tasks. These students failed to meet deadlines or ended up not participating in part of the course activities (e.g., D2L discussion). Notably, the struggling students attributed difficulties to uncontrollable causes (Schunk, 1991), such as incompetent team members, instructional supports that mismatched their beliefs (e.g., guided instruction), and less effective SMEs. The types of attributional response seemed to also link to project management (e.g., plan, control, and reflection). These students struggled to monitor their progress and rarely sought help from the instructor, their peers, the SMEs, or other available resources. Although the research suggests that students learn more effectively from peers who successfully overcome similar obstacles than from expert models (Bandura, 1977), this did not apply to the students in this study. In sum, the students’ prior experience in PBL might influence their acceptance, efforts, and attitudes towards the activities. However, the aforementioned dispositions and learning habits of those struggling students seemed to be more critical and deterred them from adjusting their learning approaches to the attainment of the goals.

The level of support from outside SMEs/clients had a serious impact on students’ motivation and the overall quality of the work in PBL (Kapranos, 2015). Provision of protocols and guidelines should help explicitly explain expectations and types of support to be given to students with whom SMEs work.

The project scope in breadth and depth should be determined in the early phase of the ID process. Timely scaffolding (e.g., helping to link content knowledge to their analysis, use mapping charts to depict the topic analysis, and create a visual representation) for task analysis ensured that the students identified the hidden tasks/topics from the initial analysis and determined the focused tasks/topics for their lesson to be developed. It was tremendously important to involve the SMEs in this process (Kapranos, 2015). The SMEs checked the accuracy of the content and helped determine the hours and resources they expected to be used for each project.

Despite the caution of some researchers (e.g., Johnson et al., 2000), success in the learning process and outcomes was not associated with the mode of interaction chosen by the students. Even though the PBL activities in this course involved high demand for interaction and complex tasks, the online students outperformed their counterparts. They were more active, put more effort into tasks, and fully benefited from online interactions. Researchers have explained performance variations online as possibly caused by personalities or learning style differences (Su et al., 2005).

Conclusion

This study examined the researcher‘s reflective process of designing, implementing, and revising the PBL model for an advanced ID skill development course. The research process drew on the core parts of design-based research in terms of, “(1) pragmatic; (b) grounded; (c) interactive, iterative, and flexible; (d) integrative; and (e) contextual” (Wang & Hannafin, 2005, p. 7). This is significant because individual teaching and learning experiences take place under unique learning conditions in terms of learners, contexts, and topics. The situational constraints require context-specific methods to achieve the stated learning goals and ongoing improvement. In addition, situationality (Reigeluth & Frick, 1999) was addressed to increase generalizability, that is, the extent to which the model can be used for a wider range of settings.

When situationalities are incorporated into the theory, the theory becomes useful for a broader range of situations. At a very minimum, your research report should describe as completely as possible the situations under which the theory was applied in your study, so that others may draw conclusions about situationalities (p. 22)

Despite the obvious benefits of PBL in this research, there are several questions that need to be answered. For example, how can struggling students be better assisted in participating in the collaborative nature of the ID process and feeling fulfilled in a learner centered context? Reflective journal writing (Lahteenmaki & Uhlin, 2010), mid-term feedback (Kamp et al., 2013), and structured guidance that requires positive and negative feedback as constructive criticism for peer review (Carmichael, 2014) would help the students share their needs and ideas in a less obtrusive way. However, if students are reluctant to ask for help, it is not easy to accurately assess the needs of individual students. Often, it makes a situation more difficult when the students do not consider their peers a useful resource to learn from, rather to rely on the instructor. It is necessary to further explore the heart of successful team culture: under what conditions students build trust within a team, so as to glean ideas from each other and offer constructive feedback. Along with it, identifying factors that cause negative team dynamics and corresponding strategies to improve team performance should offer useful insights. It is expected that the gaps (barriers/difficulties) identified in this study and research-based solutions for the gaps in the future would allow the researcher to experiment with the solutions as new constructs of the model and increase practical applicability over time (Brown, 1992).

Footnotes

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author received no financial support for the research, authorship and/or publication of this article.

ORCID iD

Yun Claire Park

References

Albanese

(2000). Problem-based learning: Why curricula are likely to show little effect on knowledge and clinical skills. Medical Education, 34(9), 729–738. https://doi.org/10.1046/j.1365-2923.2000.00753.x

APA Work Group of the Board of Educational Affairs (1997, November). Learner-centered psychological principles: A framework for school reform and redesign. Washington, DC: American Psychological Association.

Ariely

Wertenbroch

(2002). Procrastination, deadlines, and performance: Self-control by precommitment. Psychological Science, 13(3), 219–224. https://doi.org/10.1111/1467-9280.00441

Bandura

(1977). Self-efficacy: Toward a unifying theory of behavioral change. Psychological Review, 84(2), 191–215. https://doi.org/10.1037/0033-295X.84.2.191

Bandura

(1997). Self-efficacy: The exercise of control. W H Freeman/Times Books/ Henry Holt & Co.

Barrows

H. S.

(1986). A taxonomy of problem-based learning methods. Medical Education, 20, 481–486.

Barrows

H. S.

(1996). Problem-based learning in medicine and beyond: A brief overview. In Wilkerson

Gijselaers

W. H.

(Eds.), Bring problem-based learning to higher education: Theory and practice (pp. 3–12). San Francisco: Jossey-Bass.

Barron

Schwartz

Vye

Moore

Petrosino

Zech

Bransford

, & The Cognition and Technology Group at Vanderbilt. (1998). Doing with understanding: lessons from research on problem- and project-based learning. The Journal of the Learning Sciences, 7(3&4), 271–311.

Berinato

(2018, January-February). Negative feedback rarely leads to improvement: An interview with Paul Green. Harvard Business Review.

10.

Brown

A. L.

(1992). Design experiments: Theoretical and methodological challenges in creating complex interventions in classroom settings. Journal of the Learning Sciences, 2, 141–178. http://dx.doi.org/10.1207/s15327809jls0202_2

11.

Brownell

Jameson

D.A.

(2004). Problem-based learning in graduate management education: An integrative model and interdisciplinary application. Journal of Management Education, 28, 558–577.

12.

Brush

Saye

(2014). An instructional model to support problem-based historical inquiry: The persistent issues in history network. Interdisciplinary Journal of Problem-Based Learning, 8(1), 79–103. https://doi.org/10.7771/1541-5015.1409

13.

Buckingham

Goodall

(2019). The feedback fallacy. Harvard Business Review, 97(2), 92–101.

14.

Carmichael

S. G.

(2014, June). Everything you need to know about giving negative feedback. Harvard Business Review.

15.

Chappelow

McCauley

(2019). What good feedback really looks like. Harvard Business Review.

16.

Cognition and Technology Group at Vanderbilt. (1992). The Jasper experiment: An exploration of issues in learning and instructional design. Educational Technology Research and Development, 40(1), 65–80.

17.

Collins

(1992). Toward a design science of education. In Scanlon

O’Shea

(Eds.), New directions in educational technology (pp. 15–22). New York: Springer-Verlag. http://dx.doi.org/10.1007/978-3-642-77750-9_2

18.

Collins

Brown

J. S.

Newman

S. E.

(1989). Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. In Resnick

L. B.

(Ed.), Knowing, learning, and instruction: Essays in honor of Robert Glaser (pp. 453–494). Lawrence Erlbaum Associates, Inc.

19.

Cronje

(2006). Paradigms regained: toward integrating objectivism and constructivism in instructional design and the learning sciences. Educational Technology Research and Development, 54(4), 387–416. https://doi.org/10.1007/s11423-006-9605-1

20.

Davidson

J. E.

Sternberg

R. J.

(1998). Smart problem solving: How metacognition helps. In Hacker

D. J.

Duniosky

Graesser

A. C.

(Eds.), Metacognition in educational theory and practice (pp. 47–68). Lawrence Erlbaum Associates.

21.

Delclos

V. R.

Harrington

(1991). Effects of strategy monitoring and proactive instruction on children’s problem-solving performance. Journal of Educational Psychology, 83(1), 35–42. https://doi.org/10.1037/0022-0663.83.1.35

22.

Design-Based Research Collective. (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5–8.

23.

Dorsey

L. T.

Goodrum

D. A.

Schwen

T. M.

(1997). Rapid collaborative prototyping in an instructional development paradigm. In Dills

C. R.

Romiszowski

A. J.

(Eds.), Instructional development paradigm. (pp. 445–466). Educational Technology Publications.

24.

Duffy

T. M.

Cunningham

D. J.

(1996). Constructivism: implications for the design and delivery of instruction. In Jonassen

(Ed.), Handbook of research for educational communications and technology (pp. 170–198). Macmillan.

25.

Duffy

Jonassen

(1992). Constructivism and the technology of instruction: A conversation. Philadelphia, PA: Lawrence Erlbaum.

26.

Dunlap

J. C.

(2005). Problem-based learning and self-efficacy: How a capstone course prepares students for a professor. Educational Technology Research and Development, 53(1), 65–85. https://doi.org/10.1007/BF02504858

27.

Ertmer

P. A.

Glazewski

Ottenbreit-Leftwich

Goktas

Collins

Kocaman

(2009). Facilitating technology-enhanced problem-based learning (PBL) in the middle school classroom: An examination of how and why teachers adapt. Journal of Interactive Learning Research, 20(1), 35–54. Retrived from http://www.edci.purdue.edu/ertmer/docs/AECT07_PBL-Challenges.pdf

28.

Foshay

W.R.

Silber

K. H.

Stelnicki

(2003). Writing training materials that work. San Francisco: Jossey-Bass/Pfeiffer.

29.

Gagne

R. M.

(1985). The conditions of learning and theory of instruction. New York: Holt, Rinehart & Winston.

30.

Georgiou

S. N.

Christou

Stavrinides

Panaoura

(2002). Teacher attributions of student failure and teacher behavior toward the failing student. Psychology in the Schools, 39(5), 583–595.

31.

Land

S. M.

(2004). A conceptual framework for scaffolding ill-structured problem-solving processes using question prompts and peer interactions. Educational Technology Research and Development, 52(2), 5–22. https://doi.org/10.1007/BF02504836

32.

Gustafson

K. L.

(2000). Designing technology-based performance support. Educational Technology, 40(1), 38–44.

33.

Hansen

(2021, May 18)). The U.S. education system isn’t giving students what employers need. Harvard Business Review.

34.

Hannafin

Land

Oliver

(1999). Open learning environments. In Reigeluth

C. M.

(Ed.), Instructional design theories and models: A new paradigm of instructional theory (Vol. 2, pp. 115–140). Lawrence Erlbaum Associates.

35.

Hmelo-Silver

C. E.

(2004). Problem-based learning: what and how do students learn? Educational Psychology Review, 16(3), 235–266. https://doi.org/10.1023/B:EDPR.0000034022.16470.f3

36.

Hmelo-Silver

C. E.

Barrows

(2006). Goals and strategies of a problem-based learning facilitator. Interdisciplinary Journal of Problem-Based Learning, 1(1). https://doi.org/10.7771/1541-5015.1004

37.

Hmelo-Silver

C. E.

Duncan

R. G.

Chinn

C. A.

(2007). Scaffolding and achievement in problem-based and inquiry learning: A response to Kirschner, Sweller, and Clark (2006). Educational psychologist, 42(2), 99–107.

38.

Horn

(2014, August 14). Disruption in higher ED is growing, but not instantaneous. Forbes .

39.

Hung

(2011). Theory to reality: A few issues in implementing problem-based learning. Educational Technology Research and Development, 59(4), 529–552.

40.

Hung

(2006). The 3C3R model: A conceptual framework for designing problems in PBL. Interdisciplinary Journal of Problem-based Learning, 1(1), 55–77. PBL theory to reality 549. https://doi.org/10.7771/1541-5015.1006

41.

Jonassen

D. H.

(1997). Instructional design models for well-structured and ill-structured problem-solving learning outcomes. Educational Technology Research and Development, 45(1), 65–94. https://doi.org/10.1007/BF02299613

42.

Jonassen

D. H.

(2000). Toward a design theory of problem solving. Educational Technology Research and Development, 48(4), 63–85. https://doi.org/10.1007/BF02300500

43.

Johnson Aragon Shaik Palma-Rivas . (2000). Comparative analysis of learner satisfaction and learning outcomes in online and face-to-face learning environments. Journal of Interactive Learning Research, 11(1), 29–49.

44.

Kamp

R. J.

Dolmans

D. H.

Van Berkel

H. J.

Schmidt

H. G.

(2013). The effect of midterm peer feedback on student functioning in problem-based tutorials. Advances in Health Sciences Education : Theory and Practice, 18(2), 199–213. https://doi.org/10.1007/s10459-012-9364-1

45.

Kapranos

(2015). PBL for doctoral students in collaboration with SMEs: “thinking like a professional engineer.” Open Journal of Social Sciences, 3, 57–63. https://doi.org/10.4236/jss.2015.36012

46.

King

(1991). Effects of training in strategic questioning on children’s problem-solving performance. Journal of Educational Psychology, 83(3), 307–317. https://doi.org/10.1037/0022-0663.83.3.307

47.

Kirschner

P. A.

Sweller

Clark

R. E.

(2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41(2), 75–86. https://doi.org/10.1207/s15326985ep4102_1

48.

Kolb

(1984). Experiential learning: Experience as the source of learning and development. Upper Saddle River, NJ: Prentice-Hall.

49.

Kuhlmann

(2013a). Do you really need an instructional design degree? The Rapid E-Learning Blog. Retrieved from http://www.articulate.com/rapid-elearning/do-you-really-need-an-instructional- design-degree/.

50.

Kuhlmann

(2013b). How to overcome the challenge of working with subject matter experts. The Rapid E-Learning Blog. https://blogs.articulate.com/rapid-elearning/how-to-overcome- the-challenge-of-working-with-subject-matter-experts/?utm_source=REL&utm_medium= Email&utm_campaign=RELFEED.

51.

Lave

Wenger

(1991). Situated learning. Legitimate peripheral participarion. Cambridge, England: Cambridge University Press.

52.

Lahteenmaki

Uhlin

(2010). Developing reective practitioners through PBL in academic and practice environments. In Barrett

Moore

(Eds.), New approaches to problem-based learning (pp. 158–171). Routledge.

53.

Maclellan

Soden

(2003). Expertise, expert teaching and experienced teacher's knowledge of learning theory. Scottish Educational Review, 35(2), 110–120.

54.

Maudsley

(1999). Roles and responsibilities of the problem based learning tutor in the undergraduate medical curriculum. BMJ (Clinical research ed.), 318(7184), 657–661. https://doi.org/10.1136/bmj.318.7184.657

55.

Merrill

M. D.

(2002). First principles of instruction. Educational Technology Research and Development, 50(3), 43–59. https://doi.org/10.1007/BF02505024

56.

Molenda

(2009). Origins and evolution of instructional systems design. In Handbook of improving performance in the workplace (Vol. 1, pp. 53–92). San Francisco, CA: Pfeiffer.

57.

Morrison

G. R.

Ross

S. M.

Kalman

H. K.

Kemp

J. E.

(2011). Designing effective instruction (6th). Wiley.

58.

Norman

Schmidt

H. G.

(1992). The psychological basis of problem-based learning: A review of the evidence. Academic Medicine, 67(9), 557–565. https://doi.org/10.1097/00001888-199209000-00002

59.

Perkins

D. N.

Hancock

Hobbs

Martin

Simmons

(1986). Conditions of learning in novice programmers. Journal of Educational Computing Research, 2(1), 37–55.

60.

Reigeluth

C. M.

(1992). The imperative for systemic change. Educational Technology, 32(11), 9–13.

61.

Reigeluth

C. M.

(1997). Instructional theory, practitioner needs, and new directions: Some reflections. Educational Technology, 37(1), 42–47.

62.

Reigeluth

C. M.

(1999). What is instructional-design theory and how is it changing. Instructional-design theories and models: A new paradigm of instructional theory, 2, 5–29.

63.

Reigeluth

C. M.

Frick

T. W.

(1999). Formative research: A methodology for creating and improving design theories. In Reigeluth

C. M.

(Ed.), Instructional design theories and models (Vol. 2). A new paradigm of instructional theory (pp. 633–651). Erlbaum.

64.

Richey

R.C.

Klein

Nelson

(2004). Developmental research: Studies of instructional design and development. In Jonassen

(Ed.) Handbook of research for educational communications and technology (2nd ed., pp. 1099–1130). Lawrence Erlbaum Associates, Inc.

65.

Ruiz-Gallardo

González-Geraldo

J. L.

Castaño

(2016). What are our students doing? Workload, time allocation and time management in PBL instruction. A case study in science education. Teaching and Teacher Education, 53, 51–62. https://doi.org/10.1016/j.tate.2015.10.005

66.

Saye

J. W.

Brush

(2002). Scaffolding critical reasoning about history and social issues in multimedia-supported learning environments. Educational Technology Research and Development, 50(3), 77–96.

67.

Savery

J. R.

(2009). Problem-based approach to instruction. In Reigeluth

C. M.

Carr-Chellman

A. A.

(Eds.), Instructional design theories and models: building a common knowledge base (Vol. 3, pp. 143–165). Routledge. Taylor and Francis, Publishers, NY & London.

68.

Schunk

D. H.

(1991). Self-efficacy and academic motivation. Educational psychologist, 26(3-4), 207–231.

69.

Smither

J. W.

London

Reilly

R. R.

(2005). Does performance improve following multisource feedback? A theoretical model, meta-analysis, and review of empirical findings. Personnel Psychology, 58(1), 33–66. https://doi.org/10.1111/j.1744-6570.2005.514_1.x

70.

Smith

P. L.

Regan

T. J.

(2005). Instructional design. Wiley Jossey-Bass Education. USA.

71.

Song

Hannafin

Hill

J. R.

(2007). Reconciling beliefs and practices in teaching and learning. Education Technology Research Development, 55, 27–50. https://doi.org/10.1007/s11423-006-9013-6

72.

Strobel

van Barneveld

(2009). When is PBL more effective? A meta-synthesis of meta-analyses comparing PBL to conventional classrooms. Interdisciplinary Journal of Problem-Based Learning, 3(1). https://doi.org/10.7771/1541-5015.1046

73.

Bonk

C. J.

Magjuka

R. J.

Liu

Lee

S. -h.

(2005). The Importance of interaction in Web-Based education: A program-level case study of online MBA courses. Journal of interactive online learning, 4(1).

74.

Tessmer

Wedman

J. F.

(1990). A layers of necessity instructional development model. Educational Technology Research and Development, 38(2), 77–85. https://doi.org/10.1007/BF02298271

75.

Turner

Merrill

(2021, May 25). Using structure to promote equity and engagement in live remote sessions. Educause Review.

76.

van Merrienboer (1997). Training complex cognitive skills: A four-component instructional design model for technical training. Educational Technology Publications.

77.

van Merrienboer

J. J. G.

Kirschner

P. A.

Kester

(2003). Taking the load off a learner’s mind: instructional design for complex learning. Educational Psychologist, 38(1), 5–13. https://doi.org/10.1207/S15326985EP3801_2

78.

Wang

Hannafin

M. J.

(2005). Design-based research and technology-enhanced learning environments. Educational Technology Research and Development, 53(4), 5–23.

79.

Y.-T.

Tsai

C. C.

(2006). University students’ internet attitudes and internet self-efficacy: A study at three universities in Taiwan. Cyber Psychology and Behavior, 9(4), 441–450. https://doi.org/10.1089/cpb.2006.9.441