Assessing learning outcomes in electrical engineering education: A case study from Saudi Arabia

Abstract

This paper presents the main features of a study carried out by the higher education authority in the Kingdom of Saudi Arabia for developing a practical framework for formulating and assessing learning outcomes in electrical engineering education. The proposed learning outcomes were based on a compilation of different international accreditation frameworks. The formulated learning outcomes were grouped into four learning areas: (1) engineering sciences, (2) engineering analysis and investigation, (3) engineering design and (4) engineering practice. The paper also presents the main elements of a proposed standardized exit exam to test the developed learning outcomes. A table of specification was constructed that maps the developed learning outcomes with the test questions distributed over various learning levels. It allows the transformation of the developed learning outcomes into balanced questions to be used in an exit exam. The paper also discusses the implications of the proposed exit exam on electrical engineering education, and presents the results of a trial exam carried out recently.

Keywords

assessment electrical engineering exit exam learning outcomes

The engineering education environment is changing as information and communication technologies are having greater impact, and innovation is becoming increasingly essential. Companies, operating in a highly competitive environment, are more inclined to value engineering graduates who possess additional non-technical qualities as well as technical know-how for the job. Employers as well as academic accreditation entities are putting pressure on to incorporate robust assessment techniques into engineering programmes. Therefore, the assessment of learning outcomes has become a primary focus for engineering education today.^1,2

An outcome-driven assessment process, if carefully designed and implemented, can be useful at different levels. It can provide critical information on whether graduates have acquired the knowledge and skills defined by predetermined educational objectives. The assessment process can also convey useful information to faculty and administrators on the effectiveness of the design and delivery of the educational programme. It can also develop, in the long term, instruments to obtain comparable information on what students actually learn across different engineering colleges.^3,4

Quality assurance requirements, the primary drivers of assessment, are likely to be more imperative in developing countries. Engineering education in the Kingdom of Saudi Arabia, for example, is less than half a century old. However, with the large expansion of the industrial sector as result of oil revenues, the country has seen a large increase in its engineering colleges, both public and private. Currently, the Kingdom has 21 public and 3 private universities offering engineering education. More than half of these colleges are less than five years old. This rapid expansion was, however, not initially met with adequate assessment of student learning outcomes. Although a number of engineering colleges are accredited, there are sustained complaints from the industry about the unsuitable quality of the educational product compared to the demands and expectations of the labour market. Undoubtedly, this issue presents a big challenge to educators to design effective learning strategies and to provide adequate tools for their assessment. The Ministry of Higher Education in Saudi Arabia strongly encourages the universities to seek accreditation. However, private and newly founded public engineering institutions present real challenges in this respect. In attempt to tackle these, it has launched a study to implement an exit exam for engineering graduates across the country.

The study was carried out over a period of two years and involved as a first step the elaboration of a ‘qualifications framework’. This was followed by the setting up of the exam structure. The authorities also decided to implement the exit exam, on a trial basis, in the first semester of the academic year 2013–2014. The objective of this paper is to provide and discuss details of this study as applied to electrical engineering education.

It should be noted that few countries in the world (see, e.g., Ref. 5) impose mandatory testing of graduating engineering students. But large-scale voluntary direct assessments via standardised tests are carried out by various non-governmental assessment agencies in many countries.^6–9 Other experiences of assessment use popular alternatives like ‘curriculum-embedded’ assessments and student portfolios.^9–11 In addition to such direct measures of student learning outcomes, surveys are generally used within the realm of assessment.^9,10,12

Research methodology

The study carried out at the request of the Ministry of Higher Education in Saudi Arabia lasted two years and involved essentially the following tasks. The first stage involved an extensive review of available literature on the development of frameworks for learning outcomes in engineering education in order to compile previous related works, experiences, and lessons learned. The literature review covered experiences from various countries worldwide. The review also covered independent and important projects on learning outcomes such as the Accreditation Board for Engineering and Technology (ABET),¹³ Conceiving- Designing-Implementing-Operating (CDIO),¹⁴ European Accredited Engineer framework (EUR-ACE),¹⁵ Assessment of Higher Education Learning Outcomes project (Tuning-AHELO),¹⁶ UK standard for professional engineering competence (UK-SPEC),¹⁷ standards for Engineers Australia (EA),¹⁸ National Academy of Engineering (NAE)¹⁹ and International Engineering Alliance (IEA).²⁰ The second task involved the development of a local framework for engineering learning outcomes for various disciplines (including electrical engineering) while the final task consisted in the development of the structure of the proposed exit exam. In the course of the study, the products of each stage were reviewed locally and by international experts through several workshops organized by the Ministry of Higher Education.

The formulated learning outcomes were grouped into four learning areas: (1) basic sciences and engineering fundamentals, (2) engineering analysis and investigation, (3) engineering design and (4) engineering practice. A comparison between NCAHE formulated learning outcomes and other international frameworks was carried out as part of the present work. Table 1 shows a sample of such a comparison related to the ‘Basic Sciences and Engineering Fundamentals' skill.

Proposed framework for electrical engineering learning outcomes

Electrical Engineering (EE) skills include the knowledge of the fundamentals of various electrical engineering principal areas, namely electric power and machines, communications and electronics as well as control systems. Electrical engineering graduates should be able to build on their acquired skills pertaining to the basics of engineering sciences as well as engineering design, analysis, investigation and practice skills, and consequently acquire electrical engineering discipline-level skills and demonstrate their knowledge of the basic concepts governing various electrical engineering phenomena and the associated design and operating principles.^21,22

The proposed framework for learning outcomes was organised into the four aforementioned learning areas. Within each learning area, the content is further defined by a set of abilities. Each ability is composed of two major parts:

The ability statement which broadly defines what an engineering graduate in his field/discipline should know and be able to do.

The descriptive statements of the learning outcomes (LO) associated with the ability. These LOs describe in greater detail the knowledge and skills eligible for testing.²³

The following is a description of one of these abilities as an illustrative example.

A. Learning area: Engineering sciences

A.1 Ability (EE1): The ability to identify electrical components and interpret their characteristics and material contents.

Associated Learning Outcomes:

Graduates who possess this ability (EE1) should be able to:

Identify characteristics of electrical circuit components and materials, such as resistance, inductance, capacitance, conductors, semiconductors and dielectrics.

Explain the meaning, symbols and notations associated with various electrical quantities and master the usage of – and conversion between – their units of measurements.

Demonstrate familiarity with the characteristics and composition of electrical components as well as their behavior under both direct and alternating current (d.c., a.c.) excitations.

State the principles of electrostatic, electromagnetic and magneto-static fields, electric potential and current density.

TABLE 1

Example of comparison of NCAHE formulated skills with other frameworks

Skill	ABET	Conceiving- Designing-Implementing - Operating (CDIO)	EUR-ACE Framework Standards for the Accreditation of Engineering Programs	Tuning-AHELO framework of Learning Outcomes	NCAHE Framework of Learning Outcomes
Basic Sciences and Engineering Fundamentals	An ability to apply knowledge of mathematics, sciences, and engineering.	DISCIPLINARY KNOWLEDGE AND REASONING	Knowledge and understanding of the scientific and mathematical principles underlying their branch of engineering.	The ability to demonstrate knowledge and understanding of the scientific and mathematical principles underlying their branch of engineering.	BS1. The ability to apply university level mathematical skills to analyze and solve engineering problems.
		Knowledge of underlying mathematics and sciences
		Core engineering fundamental knowledge
			A systematic understanding of the key aspects and concepts of their branch of engineering.	The ability to demonstrate a systematic understanding of the key aspects and concepts of their branch of engineering.	BS2. The ability to utilize knowledge of chemistry and physics to solve engineering problems.
		Advanced engineering fundamental knowledge, methods and tools
			Coherent knowledge of their branch of engineering including some at the forefront of the branch.	The ability to demonstrate comprehensive knowledge of their branch of engineering including emerging issues.	EF1. The ability to apply knowledge of engineering fundamentals and information technology. This ability encompasses, also, the integration of knowledge from other engineering disciplines with graduate's engineering discipline.

Proposed test structure

The proposed test is paper based, planned over a 3-hour session and consists of a total of 50 multiple choice questions. The test is open to holders of a Bachelor degree in Electrical Engineering as well as students in the final year of such programmes. Books, lecture notes, or similar materials are not allowed in the test. Necessary reference sheets, equations and relevant data are provided if needed. The detailed structure of the test is clarified with the construction of the Table of Specifications (Table 2). The table of specifications is a map that facilitates the transformation of the developed learning outcomes into balanced questions.

The first column of Table 2 shows the learning area, namely engineering sciences, engineering analysis & investigation, engineering design, engineering practice. The second column indicates the alpha-numeric code assigned to the ability, as outlined in the earlier section. The third column contains the codes of learning outcomes as specified in the previous section. Different learning outcomes are grouped – if necessary – according to question allocation requirements. The need for grouping some learning outcomes stems from the fact that the number of learning outcomes could exceed the suggested maximum number of questions in a particular learning area. The grouping of complementary learning outcomes would ensure that at least one learning outcome in a particular group will be tested. The fourth column indicates the number of questions allocated to each learning outcome group.

The fifth column, distributed over three learning levels (knowledge & comprehension, application & analysis, synthesis & evaluation), specifies the question distribution among the three learning levels. For example, for the third ability (EE3), there is a total of (9 + 5 = 14) questions assigned to the LO group (EE3-01 to EE3-07) of the first ability (EE3). Nine questions are allocated to the learning outcomes (EE3-01, EE3-02, EE3-06, EE3-07) and five to the learning outcomes (EE3-03, EE3-04, EE3-05). A total of six of fourteen questions are assigned to the first learning level (knowledge & comprehension) while the other eight questions are assigned to the second learning level (application & analysis). It is important to note that the distribution of questions among the learning outcomes follows a careful process which ensures an adequate coverage for different learning areas (vertical allocation) as well as for various learning levels (horizontal allocation). In this regard, the distribution of questions among the three learning levels is 15 questions (30%) for knowledge & comprehension, 25 questions (50%) for application & analysis, and 10 questions (20%) for synthesis & evaluation. The distribution, on the other hand, of questions among the four learning areas is 10 questions (20%) for engineering sciences, 20 questions (40%) for engineering analysis & investigation, 10 questions (20%) for engineering design, and 10 questions (20%) for engineering practice. The sixth column shows the distribution of questions among different subject areas as well as the total number of questions. The seventh column shows the percentage of questions allocated to each learning area while the eighth column specifies the total allocated time (in minutes) to each learning area. The last column indicates the average question time (in minutes) as calculated from the preceding two columns. This average represents only general guidance as it is understood that some questions may require longer or shorter time.

TABLE 2

Table of specifications

Learning Area	Skills Code (Ability)	LO-Code Group	Suggested Number of Questions	Assigned Allocations Among Learning Levels			Suggested Distribution by Major Subject Areas (Within EE Discipline) and Associated # of Question		% of Test	Total Allocated Time (Minutes)	Average Question Time (Minutes)
				Knowledge and Comprehension	Application and Analysis	Reasoning (Synthesis and Evaluation)	Major Subject Area	#
Engineering Sciences	EE1	EE1-01	2	2	0	0	Power, Machines & Control	1	20% (10 Questions)	30	3
		EE1-04					Communications & Electronics	1
		EE1-02	3	3	0	0	Power & Machines	1
		EE1-03					Control	1
							Communications & Electronics	1
	EE2	EE2-01	3	3	0	0	Power & Machines	1
		EE2-02					Communications & Electronics	1
							Control	1
		EE2-03	2	0	2	0	Communications & Electronics	1
		EE2-04					Power, Machines & Control	1
		EE2-05
Engineering	EE3	EE3-01	9	4	5	0	Power	2	40%	70	3.5
Analysis and Investigation		EE3-02					Machines	2	(20 Questions)
		EE3-06					Communications	2
		EE3-07					Electronics	2
							Control	1
		EE3-03	5	2	3	0	Power & Machines	2
		EE3-04					3	Control
		EE3-05
	EE4	EE4-01	3	1	2	0	Communications & Electronics	2
		EE4-02					Control	1
		EE4-03
		EE4-04	3	0	2	1	Power, Machines & Control	1
		EE4-05					Communications & Electronics	2
Engineering	EE5	EE5-01	2	0	1	1	Power, Machines,	2
Design		EE5-02					Control,		20%	40	4
		EE5-03					Communications & Electronics		(10 Questions)
		EE5-04
		EE5-05	2	0	2	0	Power, Machines & Control	1
		EE5-06					Communications & Electronics	1
	EE6	EE6-01	2	0	2	0	Power, Machines & Control	1
		EE6-06					Communications & Electronics	1
		EE6-02	2	0	2	0	Power, Machines & Control	1
		EE6-07					Communications & Electronics	1
		EE6-08						1
		EE6-03	2	0	0	2	Power, Machines & Control	1
		EE6-04					Communications & Electronics	1
		EE6-05						1
Engineering	EE7	EE7-01	1	0	1	0	Power, Machines, Communications, Electronics & Control	1	20%	40	4
Practice		EE7-02						1	(10 Questions)
		EE7-04						1
		EE7-03	2	0	1	1	Power, Machines & Control	1
		EE7-05					Communications & Electronics	1
		EE7-06						1
	EE8	EE8-01	1	0	0	1	Power, Machines, Communications, Electronics & Control	1
		EE8-02
		EE8-03
		EE8-05
EE8-04	2	0	0	2	Power, Machines & Control	1
							Communications & Electronics	1
	EE9	EE9-01	2	0	1	1	Power, Machines & Control	1
		EE9-04					Communications & Electronics	1
		EE9-02	2	0	1	1	Power, Machines & Control	1
		EE9-03					Communications & Electronics	1
			50	15 (30%)	25 (50%)	10 (20%)		50	100%	180

Pilot study

A pilot study of the developed framework was conducted in the first semester 2013–2104 at the National Center for Assessment in Higher Education (NCAHE) of Saudi Arabia. The pilot study comprised two test sessions on examinee populations from 13 universities as outlined in Table 3.

TABLE 3

Participation data of pilot study

#	University	(Test population) Electrical engineering
		Session # 301	Session # 302
1	King Abdulaziz University	3	3
2	Um Al Qora University	3	2
3	Taeif University	8	7
4	King Khalid University	10	10
5	KF-UPM University	34	33
6	Al Baha University	25	25
7	King Saud	9	8
8	Najran University	12	10
9	Tabouk University	11	11
10	Hail University	8	8
11	Arar University	6	6
12	King Faisal University	5	4
13	Jizan University	27	24
	TOTAL	161	151

All the examinees were students in the last semester prior to their graduation. In order to reduce test application time, the test was divided into two equivalent forms (denoted form#1 and form#2). Each form has 25 questions (i.e. items) and is administered to one group divided into two subgroups in each university. Two methods of analysis were used (classical and Item Response Theory (IRT)) in order to investigate the test characteristics and effectiveness of item options.

Table 4 shows the descriptive statistics for both forms. The Alpha index for test reliability was 0.667 and 0.672 for form#1 and form#2 respectively.

TABLE 4

Descriptive statistics

Form	Items	Mean	SD	Mean P	Mean Rpbis	Alpha
#1	25	8.805	3.455	0.354	0.172	0.667
#2	25	9.919	3.911	0.399	0.220	0.672

The item parameters are provided in Table 5 for the two forms.

TABLE 5

Summary statistics for all calibrated items

Form 1						Form 2
Parameter	Items	Mean	SD	Min	Max	Mean	SD	Min	Max
a	24	0.815	0.176	0.461	1.153	0.839	0.148	0.495	1.087
b	24	1.253	0.837	−0.426	2.635	1.441	1.110	−0.246	3.458
c	24	0.191	0.012	0.160	0.207	0.209	0.013	0.168	0.227

a = item discrimination; b = item difficulty; c = pseudo-guessing

For form #1, the mean of discrimination is 0.82, the mean of difficulty is 1.25, and the mean of guessing is 0.19 within the existing range. Similar values for item parameters were found for form # 2. However, an examination of the item-person map for form #1 and form #2 (see Figs 1 and 2, respectively) shows that there is an interval of abilities, from −4 to 0.8, which is not covered by the range of item difficulties (b). Therefore, it is necessary to add some relatively easy items to cover this gap in the item-person map.

Fig. 1

Parameter (b = item difficulty) by Theta (Form#1).(Reproduced with the permission of Assessment Systems Corporation (ASC).)

Fig. 2

Parameter (b = item difficulty) by Theta (Form #2). (Reproduced with the permission of Assessment Systems Corporation (ASC).)

The pilot study also investigated the quality of items in terms of item fit and distractor effectiveness. The IRT standardised (z) residual for items (z Resid) were not statistically significant, thus indicating a reasonable fit, namely for form #1 from (0.23, p = 0.81) to (0.91, p = 0.36), and for form #2 from (0.12, p = 0.90) to (1.54, p = 0.12).

Each test item has four response options, with one correct answer and three distractors. The results for item fit indicated that there is a need to revise five items in each form in terms of quality of distractors.

Conclusions

This paper has presented the essential features of an ambitious plan by the higher education authorities in Saudi Arabia to set up an exit exam for the Kingdom's engineering graduates including electrical engineers. The plan was set up after two years of extensive study by local staff and international reviewers and if carefully implemented could have a positive impact on electrical engineering education there. Results of the exit exam will be conveyed to the different engineering colleges and will be used to correct any shortcomings in the curriculum.

However, a number of issues remain controversial in this plan, and are still under debate among the different stakeholders. The exam results may serve to rank the different universities which could affect government spending on them. Newly formed universities are concerned that their structures are not yet solid enough to compete with the other well-established colleges. Students and parents are concerned that the exam results, if communicated to employers, could affect students' chances of recruitment. Faculty members raise the concern that the students will ultimately focus their efforts on studying for the exit exam rather than the actual curriculum. Multiple choice questions may be inappropriate for testing high-level cognitive skills.

A trial test was carried out for around 310 examinees from various national universities. The tests results yielded useful feedback on the quality of the questions and point out a need to refine some of them. The other important issue to be investigated is how to map the scores of the students to their achievement in the tested learning outcomes and to compare the performance of the various universities.

Footnotes

Acknowledgments

We are grateful to the National Center for Assessment in Higher Education, Saudi Arabia for its generous grant.

References

1 Yildirim

T. P.

Shuman

L. J.

Besterfield-Sacre

, ‘Model eliciting activities: assessing engineering student problem solving and skill integration processes’, Int. J. Eng. Educ., 26 (2010), 831.

2 Moore

Williamson

, Assessment of learning outcomes, The Higher Education Academy – Engineering Subject Centre, June 2008, available online at http://www.engsc.ac.uk/, last accessed 7 January 2013.

3 Bogue

Shanahan

Marra

Cady

, ‘Outcomes-based assessment: driving outreach program effectiveness’, Leadership Manag. Eng, 13 (2013), 27.

4 Leydens

Moskal

Palevich

, ‘Qualitative methods used in the assessment of engineering education’, J. Eng. Educ., 93 (2004), 65.

5 Ferrer

, Educational Assessment Systems in Latin America: Current Practice and Future Challenges (Preal, Washington, DC, 2006).

6 ACER (Australian Council for Educational Research), available online at http://www.acer.edu.au/tests/gsa, last accessed 7 January 2013.

7 Harris

K.-L.

James

, ‘The course experience questionnaire, graduate destination survey, and learning and teaching performance fund in Australian higher education’, in Dill

and Beerkens

, (Eds), Public Policy for Academic Quality: Analyses of Innovative Policy Instruments (Springer, New York, 2010), pp. 103–124.

8 National Institute for Learning Outcomes Assessment (NILOA), available online at www.learningoutcomeassessment.org/documents/NILOA_ForWeb_5-22.pdf, last accessed 7 January 2013.

9 Nusche

, Assessment of learning outcomes in higher education: a comparative review of selected practices, OECD Education Working Paper No. 15, 2008 (OECD, Paris, 2008).

10.

10 Augeri

, et al., Assessment of student learning outcomes, The University of Iowa, available online at http://www.uiowa.edu/∼outcomes/documents/FocusGroups-SUMMARY.pdf, accessed 7 January 2013.

11.

11 Kuh

G. D.

Ewell

P. T.

, ‘The state of learning outcomes assessment in the United States’, Higher Educ. Manag. Policy, 22 (2010), 1.

12.

12 Bers

T. H.

, ‘The role of institutional assessment in assessing student learning outcomes’, New Dir. Higher Educ., 141 (2008), 31.

13.

13 ABET, Criteria for Accrediting Engineering Programs (ABET, Baltimore, MD, 2010).

14.

14 Crawley

E. F.

Malmqvist

Ostlund

Brodeur

, Rethinking Engineering Education: the CDIO approach (Springer, New York, 2007).

15.

15 European Network for Accreditation of Engineering Education (EUR-ACE), http://www.ihep.org/assets/files/gcfp-files/EUR-ACESTANDARDS.pdf, last accessed 7 January 2013; and https://www.oecd.org/education/highereducationandadultlearning/40256023.pdf.

16.

16 Organization for Economic Co-Operation and Development (OECD), available online at www.oecd.org/dataoecd/46/34/43160507.pdf, last accessed 7 January 2013.

17.

17 UK-SPEC, Engineering Council UK, The accreditation of higher education programmes - UK standard for professional engineering competence (Engineering Council UK, London, 2008).

18.

18 Engineers Australia (EA), Engineers Australia National Generic Competency Standards - Stage 1. Competency Standards for Professional Engineers (Engineers Australia, Barton, 2005).

19.

19 National Academy of Engineering (NAE), The Engineer of 2020: Visions of engineering in the new century (National Academies Press, Washington, DC, 2004).

20.

20 International Engineering Alliance (IEA), available online at http://www.washingtonaccord.org/IEA-Grad-Attr-Prof-Competencies-v2.pdf, last accessed 7 January 2013.

21.

21 Butun

Erkin

H. C.

Altintas

, ‘A new teamwork-based PBL problem design for electrical and electronic engineering education: a systems approach’, Int. J. Elect. Enging Educ., 45 (2008), 110.

22.

22 Erdil

Bilsel

, ‘Curriculum design to revitalise electrical engineering education at Eastern Mediterranean University’, Int. J. Elect. Enging Educ., 42 (2005), 234.

23.

23 Kennedy

, Writing and Using Learning Outcomes: A practical guide (University College Cork, Ireland, 2007).