Abstract
The purpose of this study is to review and analyse the importance of responding to feedback from students attending my courses on their learning outcomes and their satisfaction with various elements of my courses: one traditional course titled Solid Mechanics and Computational Analysis and a learner-centred course titled Fluid and Electrical Drive Systems. Central Queensland University utilises online evaluations for each course in each term of the year through a student experience survey. With careful consideration and evaluation of these data using a new student feedback evaluation framework developed by the author, some appropriate teaching interventions are developed and implemented during each term. The research question is to analyse the effectiveness of this framework by collecting the same data through subsequent student experience surveys. The student experience survey data for my courses showed that student satisfaction was increasing gradually for best managed courses, indicating that proactive implementation of appropriate responses to student feedback on their learning journey is effective in improving both student satisfaction and learning.
Introduction
It is the utmost task of the tertiary educational sector to develop programmes to promote meaningful student learning. Modern computers and learning management systems (LMS) incorporating information technology (IT) help us to manage the daily learning and teaching activities and student feedback. Both traditional and online student feedback systems are popular. With advancement of internet-based technology, evaluating effectiveness of teaching and learning processes based on student ratings has become most important for administrative review and assessment for quality assurance. Students undertake a formative review of the courses at appropriate intervals. In most cases, this happens at the end of each term/semester. 1 Student feedback has a great influence not only on quality assurance of course delivery but also on development of individual lecturer’s good practices towards improving students’ quality of learning by knowing each student’s needs. Otani et al. 2 correctly articulated that student feedback had a significant impact on transformation and modification, improvement in lecture materials, course delivery, classroom management, assessment methods and engagement and interaction with students. There are some important factors that potentially affect course and lecturer evaluation by the students. These are: student interest in the course content, size of the class, the level of class being taught, course difficulty, characteristics of students and lecturers.3,4
There are some benefits of online course evaluations focusing on time and cost and quick reporting of results. 5 In this system, students can provide their feedback at any time during the feedback period. Redish and Smith 6 stressed the need to prepare students for professional practice through implementing a theoretical framework of the dynamics of thinking and learning. They believed that this could best be developed based on ‘cognitive science’. The authors illustrated that most of the current models addressing curriculum, assessment and instruction had not yielded any ‘modern knowledge of how students learned’ as they were derived from ‘limited conceptions of learning’. The authors pointed out that the conceptions were fragmented and poorly demarcated with regard to subject matter knowledge. Therefore, coordination of those aspects of student learning relating to curriculum, assessment and instruction could be better achieved through subject matter focused cognition and learning with courses based on ‘scientifically credible and shared knowledge’. On the other hand, Atman et al. 7 reiterated that students might not always put their design knowledge into practice, probably because their problem-solving skills are not related to the problems in context.
With a view to address reasons for this gap in student learning, this paper considers two courses, one traditional course titled Solid Mechanics and Computational Analysis and a learner-centred course titled Fluid and Electrical Drive Systems, for better focussing on curriculum, assessment and instruction. Project-based learning (PBL) courses such as Fluid and Electrical Drive Systems incorporate realistic industry problems to develop students’ design skills that can be applied to solve real life problems. This paper elucidates student cognition and learning by student feedback systems of various forms. The effectiveness of a new student feedback evaluation framework developed by the author (presented later) is evaluated considering meaningful student learning and closing the available gaps. An earlier version of this paper 8 was presented at the AAEE Conference in Coffs Harbour, NSW, Australia, 2016.
Traditional evaluation
In the early 1920s, University of Wisconsin 1 first implemented student evaluation by collecting feedback from students so that lecturers would understand students’ needs. In a traditional system of evaluation, a set of questionnaire along with a rating scheme in printed form is distributed to students at the end of a semester to seek out their comments. Course lecturers are not generally involved in the feedback system. Other lecturers can manage the process. Analysis software or Microsoft Excel can be used to process student feedback data. This has been found to be very effective in the past as a guide9,10 for instructors to identify their strengths and weaknesses in teaching courses and find ways to improve the quality of meaningful learning for students by addressing identified shortfalls of teaching skills. Uttl et al. 11 recently presented some points on good characteristics of teachers. These included not only putting an enormous contribution towards student learning but also showing sound knowledge of course, availability, friendliness, approachability, enthusiasm, care and fun to students. Recently, Pereira et al. 12 focused on students’ perception on effectiveness and relevance of feedback on assessment methods and self-regulation of learning. Through a set of open-ended and closed questions, the feedback results showed that more relevant, effective and positive feedback was given by students whose course outcomes were assessed under learner-centred methods compared to those assessed using traditional methods.
Online evaluation
This process is usually controlled by internal student evaluation through an online student feedback system.
13
The online system is generally used to evaluate course materials and learning and teaching strategies in terms of the lecturers’ quality assurance, performance, course content infrastructure and IT tools.
1
At Central Queensland University (CQU), an LMS called Moodle is used for this purpose. The concept of self-evaluation of course performance is for quality assurance purposes and ongoing development of the students’ learning journey. Students’ perspectives in their assessment of courses and teaching are still often viewed with scepticism even though the reliability and validity of students’ perceptions are regularly confirmed. Solicited student feedback data managed by the university LMS can be the basis of self-regulatory changes in teaching practices to further enhance course performance outcomes. The course performance matrix (CPM), discussed later, is populated through the responses/feedback given by students. Various colour codes are used to highlight different levels of course performance. By closing the loop of student feedback, continuous improvement to produce better student learning outcomes is achievable. In this setting, it is most important to understand how a new approach evolved because of the feedback, how it works in practice and how we know that it has contributed in some measurable way to better engagement that is demonstrated by improved learning outcomes. Gaertner
13
pointed out that the quality development of courses requires continuous cycles of: setting objectives, planning activities, evaluation and derivation of new measures. This self-evaluation process is very much in line with what is in place at CQU. Schildkamp and Visscher
14
defined self-evaluation as a systematic information gathering process initiated by a university for facilitating decision making, university wide learning and fostering improvement of the students’ learning journey. Among many proposed in the literature, Gaertner
13
considers that the ideal steps for processing feedback information are:
Perception – Lecturers must observe and understand student feedback. Interpretation – Lecturers can identify the explanations for the results. Action – Specific approaches and measures are implemented to optimise teaching outcomes. Re-evaluation – Post-implementation evaluation of the measures is undertaken and a further evaluation cycle evolves.
The above model includes individual, university/school and context characteristics and is presented schematically in Figure 1.
Model for reflective practices, 13 where SEP is self-evaluation portal and SFB is student feedback.
Traditional versus online evaluation methods
Both the traditional and online evaluation methods work effectively for the purpose of course and teaching assessment to facilitate decision making to promote professional development and career advancement, promotion and tenure. Table 1 illustrates some simple comparisons of the effectiveness considering different aspects of student rating calculations.
Evaluation of traditional versus online methods.
The literature search discussed above has illustrated that there have been many studies conducted to work on students’ feedback to improve course content and delivery, develop new good teaching practices and ultimately enhance students’ effective learning experiences. However, there are still many studies that can be conducted to employ a general model to treat feedback and close the loop. In this study, a new self-regulatory assessment of student feedback model is developed by the author of this paper to change teaching practices so as to enhance a good student learning environment. As a result, when proactively closing the loop of feedback, it can be seen that students’ evaluations on feedback rate and satisfaction on courses can be improved over the CQU corporate target and student retention can thereby be improved. The next section presents how CQU manages student feedback data through the CPM.
CPM at School of Engineering and Technology, CQU
Students of School of Engineering and Technology (SET) at CQU provide student experience survey (SES) feedback on their courses through Moodle. The students are giving constructive, effective and relevant feedback on various things regarding the courses they are enrolled in within a set period towards the end of each term. Students provide this feedback in their own time and anonymously. The feedback areas are Moodle Navigation, Learning Resources, Assessment Task, Assessment Requirements, Assessment Return and Assessment Feedback. The students provide a rating on a five-point Likert scale
11
in each area and a CPM processes these ratings to provide overall student satisfaction rankings of courses through colour coding. CQU’s rating system is defined as 1 for strongly disagree, 2 for disagree, 3 for neither agree nor disagree, 4 for agree and 5 for strongly agree. This rating is for course evaluation only. However, both lecturer and course evaluation is also popular.
11
CQU sends the following request out to the students: We invite you to use this anonymous survey to tell us what you think of this course. We will use the information you provide to help us enhance the overall course design, delivery and outcomes as part of the course enhancement process (aggregated data may also be used for academic research purposes). We expect that our courses will provide you with a positive learning experience and enable you to reach your potential, meeting the following outcomes: 1. providing you with opportunities to develop the appropriate knowledge, skills and attitudes. 2. communicating clear expectations, in terms of what you need to do in order to succeed in the course. 3. using an active and collaborative approach, to engage and challenge you to succeed. 4. using assessments that allow you to demonstrate the extent of your learning.
CQU corporate targets on student satisfaction and response rates.
Methods
Student feedback data obtained through SES is considered to be statistically viable at SET, CQU, if there are more than 10 students in a course and 50% or more students provide feedback. The author has examined two courses, one utilising a learner-centred method and another based on a traditional method, whose student numbers are more than 10. The traditional course titled Solid Mechanics and Computational Analysis and the learner-centred course titled Fluid and Electrical Drive Systems are considered in this study to focus on their students’ feedback and corresponding teaching interventions performed through a new student feedback assessment framework (SFAF) developed by the author of this paper modifying the model of Gaertner. 13 The SFAF helps to form CPMs to assess student’s feedback and its intervention in different courses. The steps of this new SFAF method are set out in Table 3.
Steps of processing student feedback.
SET, CQU, administers this process through discipline leaders in each discipline: Civil, Electrical and Mechanical Engineering. The course coordinator of each course prepares a Annual Course Enhancement Report (ACER) using some strategies (Table 3, for example) to propose various measures based on the student feedback received. This process is assisted by the CQU NEXUS system that is designed to support a range of academic processes related to course and programme accreditation, operation and reporting. Some of the tasks undertaken by NEXUS are production of course profiles (CPs), examination papers, ACERs, etc. The ACERs administered by each discipline leader are placed before the programme committee for further broader scale discussion. The final recommendations for good teaching practices due to student feedback approved by the programme committee are populated to the CP of this course offering in the next term. Students, the course coordinator and lecturers can all see the feedback and corresponding recommendations in the CP. The CP is then considered as a quality assurance document to carry out appropriate measures to achieve a better student learning journey.
Results
Students’ actual feedback on both courses is now presented through classifications, ranking and summarising following the Moodle/CPM method detailed previously. The student feedback considered here covers from 2012 to 2016. Associated teaching interventions also proposed for both courses are also indicated.
Because of the multi-campus teaching mode at CQU, a team teaching method is adopted for the CQU Rockhampton, Mackay, Gladstone and Bundaberg campuses where both courses are offered. Different lecturers conduct tutorial sessions, laboratory classes and workshops at different campuses. Students like to see similar learning experiences provided at all campuses and a high degree of uniformity is therefore necessary. As such, delivery should be mostly the same in all significant respects: lectures, tutorial sessions, test rigs, laboratory sheets, etc. As a course coordinator, the author tries to maintain standard approaches to assessment tasks, similar processes for responding to student queries and uniformity in marking processes. As this situation is complex, any difference in approach in innovative teaching practices can result in dissatisfaction among student cohorts that will almost certainly be reflected in their feedback. As these issues are not general in nature over different years, they are not populated in the feedback process in Table 3.
Student feedback and teaching intervention for solid mechanics and computational analysis
This course is examination based and there are three assignments with weightings of 20% each, a workbook with a pass/fail criterion and a final examination with 40% marks. Typical feedback from a student in 2014 focusing on various questions set by CQU through its Moodle site can be seen in Table 4. The last two rows of Table 4 introduce open-ended questions for students to highlight both positive and negative aspects of the course with the responses by a single student. The course coordinator can see good delivery areas and areas with issues. Table 5 presents specific feedback and related teaching practices introduced because of the constructive feedback obtained through the SESs.
A sample of feedback on this course (one student’s feedback, 2014).
Positive and negative feedback and teaching practice intervention.
ACER: Annual Course Enhancement Report.
Figures 2 and 3 illustrate the performances of this course over the years from 2010 to 2016. Good student learning journeys are reflected in student satisfaction. From the initial design stage of the course, it was being improved gradually. This course was regarded as a good course as students’ satisfaction was above the corporate target. There are some other factors that play important roles to yield student dissatisfaction. In 2015, the satisfaction score was 3.6, below the target. The feedback was related to problems of connection of students’ home computers to the university Finite Element Software (FEA), Strand 7. The IT department also faced problems in resolving the issue. As a result, teaching performance measured through student satisfaction fell. In 2016, I introduced ‘AnyDesk’ in place of ‘Citrix’ to connect to the university FEA software and it worked well. I also recorded an FEA tutorial on a simple truss problem during week 9 showing how to model the problem, impose loading and boundary conditions and finally how to populate displacement, strain and stress information through a static analysis to prepare a good report. The students were happy and indicated ‘this course helped a lot to learn’ and the satisfaction score jumped up to 4.4 (Figure 2), well above the corporate target. Rather than lecturer’s technical knowledge on course/unit, better management of the course Moodle site through regular updates, effective communication, etc. can also enhance students’ learning. Students can provide good feedback about critical issues. Felton et al. 18 and Gezgin 19 pointed out one interesting aspect of student feedback – attractive lecturers were rated better compared to the elderly lecturers. In 2016, I realised one trend of the student mind set – if students are not happy, they generally give detailed feedback quickly when the feedback system opens. If they are happy they give brief feedback somewhat later. The response rate was also high at 70% (Figure 3), hence statistically the data were deemed acceptable.
Student satisfaction.
Student feedback rate.
Student feedback and teaching intervention for fluid and electrical drive systems
This is a PBL course and utilises a compulsory teaching portfolio submitted by the students through Moodle at the end of the course to allocate students’ grades based on their claims and solid supporting evidence of those claims. The evidence and examples are provided by the students from various works: individual – reflective journals, demonstrated problem solutions; and workbook and team work – two project reports and two laboratory reports (one in electrical drive systems and another in fluid drive systems). The students’ feedback was obtained through the course Moodle site from 2012 to 2016. Table 6 presents specific feedback and related teaching practices introduced because of the comprehensive feedback obtained through SES.
Positive and negative feedback and teaching practice intervention.
ACER: Annual Course Enhancement Report.
Discussion
The student feedback through an SES is obtained focusing on various points stated in Table 4 including ease of access to the course Moodle site, resources provided, assessment tasks, detailed feedback on returning assessments on time, traditional versus PBL modes, etc. They can comment on other aspects such as face-to-face versus distance support, timetabling of conducting classes and laboratory support for distance students. The student feedback is processed as per the information given in Table 3. After classicisation of the feedback, important classes of feedback that are statistically viable over the years are put into Tables 5 and 6. For example, one type of feedback from a single student, not an accurate representation from the students, is not considered in those Tables 5 and 6.
Redish and Smith 6 pointed out concerns regarding requirements for better student retention and integration of theory and practice into engineering programmes. They also reported some studies urging more focus in engineering education on integration of traditional engineering fundamentals and professional skills in real world situations. One stream of engineering fundamentals is related to design of new components and checking existing structures and failure analysis. Other streams promote general professional skills such as problem solving, creativity, time management, team work, cost analysis, etc., touching on core concepts of the course. This paper focusses on both these requirements and they are in line with Purdue University’s future engineering model. 20 Redish and Smith 6 also pointed out that the engineering community was not particularly effective in achieving necessary changes in engineering education and current practices to keep pace with technological changes. They stressed that current approaches in L&T through assessment, instruction, curriculum, etc. incorporated theory and models that were not in line with ‘modern knowledge’. However, this paper and a new four-point strategy proposed in Mandal 21 are effective towards innovative learning and teaching practices for creating a better student learning journey (Figure 2) based on student feedback. The connections of the above points in relation to my study on student feedback and appropriate teaching interventions should be described clearly. Referring to Table 6 relating to ‘course content’ and ‘assessment task’ of student feedback, I intervened my teaching methods for curriculum development in a way that I incorporated more design and control related contents on fluid machinery in weekly delivery and two industry related project scopes, one in automotive hydraulic brakes and another in hydraulic air-craft lading gears. The influence of my teaching interventions on the trend of student feedback is important. The student feedback in 2012, and 2013 (Table 6 on assessment task) urged that the assessment task should be related to learning outcomes of the course, and in 2014 identified the assessment task was interesting and in 2017 (not in Table 6), the student feedback was ‘very detailed understanding on hydraulics and fluid machinery, the course content was interesting and relevant to the projects’.
Design and delivery of quality engineering education is not only to educate students for work readiness but also make them better engineers when working within other discipline areas. Therefore, inter-disciplinary projects, working closely with industry, student feedback and related interventions are important.
The last column of both Tables 5 and 6 contains information on teaching interventions adopted because of student feedback on the author’s courses. They also include comments on the effect/impact of adopted teaching practices through student feedback in subsequent terms. If students can see that their feedback is well respected through changes in CPs and course Moodle sites and appropriate good teaching practices are in place, student satisfaction data can be improved. For example, the lack of recording of lectures (Table 5 feedback in 2012) did not appear again and were replaced by positive comments on how recordings helped through students being able to listen at a suitable time. This is a clear example of the impact and evidence of good teaching practices being able to enhance students’ learning.
In the context of multi-campus delivery at CQU, the concept of teaching teams is in place and applies to both courses. The course coordinator and the facilitators of a teaching team’s course delivery should ensure realistic expectations from the students and consistency in course advice from the teaching team members. Otherwise problems are created and student satisfaction results fall. It is better to assess and address students’ expectations from the course at the start of the course. It is very interesting to note from Table 6 that addressing students’ constructive feedback on projects and course textbook in 2013 resulted in positive feedback from other students in 2014 on these points.
It is clear from analysing their feedback that students like to see a good course with industry linked projects, productive and timely feedback from facilitators on assessment items, good learning resources and regular communication with the students. These are the core aspects of good teaching practices such as the four-point strategy that the author of this paper presented in a teaching and learning conference in 2015. 21 Additional aspects are also essential such as proper/standard documentation on format and grading of course assessment tasks through an assessment criteria sheet, reflective journal template, workbook template, etc. Assessment criteria for assignment tasks should be clear and easy to follow. They should be free from confusion and misunderstanding regarding assignment tasks. One of the most important things to ensure as a coordinator of a course is that the lecturers are knowledgeable, approachable and helpful and responsive to students’ enquiries in a short period of time. 21 Another aspect of good multi-campus teaching delivery is the coordination among the teaching team to ensure that common expectations are being presented to students. This can be reviewed and managed through regular team meetings, setting weekly tasks and goals, etc. As a course coordinator, it is also necessary to guarantee that sufficient feedback to enhance students’ learning is provided, that delays in giving feedback are avoided and that assessment items are error free.
An easy to navigate course Moodle site is a key to producing better student satisfaction. A complex Moodle site can create unnecessary frustration for students as they need more time to locate appropriate items and so on. The structure of the course Moodle site should be well organised and not be at all disjointed. It is necessary to remove any unnecessary/unrealistic course expectations from CP documentation and the Moodle site. A good welcoming message and clearly stated lecturer expectations of the students should be recorded and put at the top of the site. A brief discussion of the lecturer’s views on teaching, research and the like can help. It is also helpful to similarly provide prior communication to the students regarding assignment items, their mark distributions, formative and summative feedback strategy of students’ assignments, modes of communication by telephone, student forums, weekly common e-mails and after-hours blackboard collaborative (BBC) sessions. Weekly postings of good lecture slides and lecture recordings, tutorial questions, handwritten step-by-step problem solutions and recordings to the Moodle site are very helpful. Additional support lecturers/guest lecturers along with their recordings being made available on the course Moodle site are also valuable. The course coordinators should make sure that all lecture files and recordings are easy to download. The CP is another important avenue to achieving good student satisfaction. Detailed information on assessment items, due date, feedback date, etc. should be accurate. There should be no variations in submission date, feedback date, etc. between the CP and the Moodle site.
On student feedback relating to student satisfaction, the SES quantified that distance students were less satisfied compared to on-campus students in my courses. There may be a few reasons: they are working and studying part-time, have family responsibilities, have less communication with lecturers and other students and have no scope to ask questions during lecturing sessions.
On the relationship between happiness and learning outcomes of courses, the important aspects are: creating excellent student learning journeys by providing reflective teaching practices, classroom management, engagement and interaction – the ‘four-point’ strategy. 22 By providing a proper student learning platform, the lecturers’ main goal is to achieve course learning outcomes. As a result, students can be happier and this can be quantified by student satisfaction data through their feedback. The corporate target of student satisfaction is 4.0 in 5.0 Likert scale provided the student number giving feedback is 10 or more and the feedback rate is 50% or more. 22
For distance students, the course coordinator must ensure availability of proper learning and teaching support similar to on-campus students. After-hours communication can resolve most problems relating to learning and teaching issues. BBC in Moodle is one of the best communication channels with which to have after-hours collaboration with distance students. Through the BBC session, discussion on weekly tutorial problems/lectures, etc. can be performed. For their residential schools, a detailed work sheet should be provided to the students before they come to the campus. It should include the various set tasks in order such as welcoming, group laboratory schedule time, booking a computer lab for group discussion and lab report writing, library tour and formal discussion with the course coordinator. Lecturers and laboratory technicians should interact with students more closely during residential schools. As this is the only face-to-face time with distance students, it can be very productive if the lecturers provide the theoretical background to the lab concept/task, emphasise the usefulness of the laboratory to achieve particular learning outcomes, etc.
By following the learning and teaching good practice methods (Table 3) and by studying student feedback and introducing appropriate teaching interventions, student satisfaction and attrition rates can be improved.
Conclusions
A new student feedback evaluation framework for achieving good teaching practices is described and evaluated after several years of implementation by properly responding to students’ constructive feedback on two courses. The new framework facilitates the identification of appropriate teaching interventions for improving course delivery in subsequent terms, focusing on scholarship in engineering education. Based on an analysis of student feedback data related to interventions in various years from 2012 and further feedback received in later years, the following conclusions can be made:
The proposed framework is a good tool for identifying teaching interventions for improving the quality of teaching. Regular communication with students is important. Facilitators/lecturers fruitful feedback on assignments/projects/lab reports, etc. should be provided with minimal delay. In PBL courses, project scopes should be clear and industry linked. Caring attitudes towards students are also very important. Proper coordination of team teaching delivery of courses is essential.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.