Immediate and sustained effect of a service-learning course grounded in public problem analysis

Abstract

This study aims to propound an instructional design for a service-learning course grounded in public problem analysis and verify its immediate and sustained outcomes through the actual instructional process. A quasi-experimental research design was adopted wherein students who were enrolling in a service-learning course were considered as a treatment group, while those not enrolling in the course were categorized into a comparison group. Three-wave surveys were distributed at the beginning, end, and 6 months after the end of the course. Our difference-in-differences (DID) analysis showed that the project-based service-learning experience seems to be negatively related to perceived problem-solving ability but positively associated with students’ public service motivation (PSM) in the short term. However, the above influence did not sustain as time progressed. This study helps to clarify the relation between service-learning courses and students’ problem-solving ability as well as PSM in the field of public affairs education. It can also serve as a foundation for efforts to improve subsequent instructional plans of service-learning courses.

Keywords

Project-based service-learning public affairs education quasi-experimental research design difference-in-differences problem-solving ability public service motivation

Introduction

Traditionally, knowledge exploration and talent cultivation have been two crucial functions that are central to the role played by a university. However, over the past few decades, the idea of engaged scholarship has been discussed extensively (Boyer, 1996). Some scholars have advocated a third mission of the university (Etzkowitz et al., 2000), namely, faculty members should come out of the ivory tower of their research and move from pure knowledge exploration to commercial applications to enhance national competitiveness. In addition to economic development, other aspects such as environmental sustainability, cultivation of humanist concern, and civic value and duty have been included as tasks that universities can undertake. In other words, universities are expected to assist in the attainment of the sustainable development goals through organizational management, knowledge exploration, and social interaction—also termed as university social responsibility (USR) (Ramos-Monge et al., 2017).

In this context, the Taiwan Ministry of Education launched a USR Program in 2017 to encourage universities to contribute to regional development by collaborating with community or other institutions. Many Taiwanese universities practice USR through curriculum designs that combine both service and learning. According to the data released by the Taiwan Open Government Data Platform, 155 out of 158 higher education institutions in Taiwan offered service-based courses as of 2015.¹ Such courses are either offered by an academic department to pursue academic specialty with problems in the public sphere to develop scholarship of engagement (Barker, 2004) or provided by a center for general education at a university to give students an opportunity of concrete practice for civic education (Shumer et al., 2012).

Many studies on service learning have investigated students’ learning experiences in specific subject areas or have analyzed the effects of combining professional courses of a particular field with service learning (Case et al., 2021; Gujarathi and McQuade, 2002; Long et al., 2011). Of these, most studies have explored the fields of health and medicine and nursing (Salam et al., 2019). Furthermore, extant research has indicated that in addition to enhancing students’ professional competencies (Lambert et al., 2022) and social and communication skills (Sass and Coll, 2015), service-learning experience can strengthen their civic consciousness and enhance their participation in public affairs (Coelho and Menezes, 2021; Winston, 2015). Although service learning is a helping and altruistic action grounded in a concern for social problems, research designing service-learning courses from the perspective of public affairs education is scant. Moreover, little research has analyzed the effects of service-learning courses from a more long-term perspective according to our knowledge.

Based on the discussion above, this study propounds an instructional design for a service-learning course grounded in public problem analysis and verifies its immediate and sustained outcomes through the actual instructional process. The instructional design proposed in this study was adopted in a general education course in a public university in Taiwan. Through a quasi-experimental design, a three-wave longitudinal study was conducted on course takers and non-course takers, and the difference-in-differences method (DID) was used to analyze the immediate and sustained effects of the course.

Literature review

Sigmon (1994) considered that the concept and content of service learning should comprise two elements, namely, service and learning, which could be further classified into four types, namely, service–LEARNING, SERVICE–learning, service–learning, and SERVICE–LEARNING, depending on the extent of importance attached to each element in a service program. Based on Sigmon’s classification, Furco (1996) further differentiated between different types of service programs according to the main beneficiaries and the goals of such programs. In Furco’s view, service programs such as volunteering and community service have service-oriented goals, with the programs’ service recipients as the main beneficiaries. Contrarily, service programs such as field education and internship have learning-oriented goals, with service providers as the main beneficiaries. Service learning differs from the aforementioned service programs as it emphasizes the mutual benefits between service providers and recipients, with both educational learning and serving others being the programs’ desired goals.

Though there are many types of service programs, as far as service-learning courses offered by universities are concerned, the goals are to provide service for the community or disadvantaged groups, promote the sustainable development of society, and fulfill USR. Furthermore, through the experiential education approach, service and the specialisms learned are combined, whereby students can bring their service experiences to their classrooms and reflect on the knowledge that they gained (Dewey, 1938). In other words, service-learning courses offered by universities deviate from the essence of education if there is only service (or labor) but no learning. A well-designed service-learning course should equally emphasize both service and learning, and with service as the medium, enable both service providers (students) and recipients to benefit from it.

Gronlund (1995) proposed three types of learning domains: cognitive, affective, and psychomotor. The present study contends that a service-learning course should be able to help students grow in the cognitive and affective dimensions. As for the psychomotor dimension, only the service contents combined with some special professional fields (e.g., nursing) are more likely to be displayed. As a result, this study focuses its discussion on cognition and affective dimensions.

In the cognitive dimension, service learning provides an opportunity for students to put what they have learned in the classroom into practice, thereby deepening their understanding of the knowledge and concepts learned (Balakas and Sparks, 2010; Novak et al., 2007; Resch and Schrittesser, 2021). Moreover, before providing service, students must understand the problems faced by service recipients and formulate service plans in response to their needs. Hence, students’ problem-solving ability can be strengthened even if the contents of service are not entirely linked to their specialized fields.

Past empirical studies have also supported the relation between service-learning experiences and problem-solving ability. Yorio and Ye (2012) examined 40 papers published between 1993 and 2010 on service-learning outcomes and concluded that service-learning activities helped improve students’ understanding of social issues, self-understanding, and cognitive development. In particular, the random effects that service-learning activities brought to cognitive development were the strongest, spanning across cognitive areas such as academic performance, management skills, writing skills, and the ability to think critically and solve problems.

With college freshmen as their research subjects, Zheng et al. (2013) examined the effects of project-based service learning on students’ learning outcomes. Their findings revealed that if instructors could give students instant feedback and problem prompts during the process of community service, it would positively influence students’ performance in creative thinking, finding innovative solutions, and problem description.

By adopting an experimental research design, Guo et al. (2016) revealed that students with service-learning experiences displayed better problem-solving ability, especially with respect to the handling of ill-structured problems. In addition, their study discovered that students who had the chance to actually apply the knowledge that they learned to the service fields also had greater commitment, concentration, and persistence in classrooms, which further enhanced their problem-solving ability through classroom learning.

Liu (2006) investigated students’ learning experiences and outcomes through their participation in a service learning. The results of quantitative analysis indicated that the service action itself, reflection on service experiences, and sharing and celebration of service outcomes all positively influenced the students’ problem-solving ability. The interview results further revealed that the service-learning course improved students’ problem-solving ability by increasing their understanding of the structural factors behind the phenomenon of problems and enabling them to free themselves from old perspective frameworks (e.g., they no longer merely looked upon the service recipients from the perspective of disadvantaged groups).

Besides helping in enhancing students’ problem-solving ability on the cognitive level, service-learning courses are expected to change students’ attitudes, beliefs, and values on the affective level. As noted above, USR emphasizes the exploration of public issues and cultivation of humanist concern as well as academic specialisms to solve local problems. Service-learning courses, which consider the application of USR as one of its objectives, should also be able to contribute to the formation of civic consciousness and realization of humanist values. These purposes are akin to the notion of public service motivation (PSM) discoursed in the public administration field.

Perry and Wise (1990) define PSM as “an individual’s predisposition to respond to motives grounded primarily or uniquely in public institutions and organization” (p. 368). They also indicated that PSM comprises three types of motives: rational, norm-based, and affective. Rational motives refer to individuals’ drive to engage in public service because in the process, they can come to perceive their influence over the solution of public problems. Norm-based motives refer to individuals’ drive to participate in public service because they consider showing concern for public affairs as one of their duties as citizens. Affective motives refer to individuals’ drive to participate in public service owing to their great love for society and compassion for others. Perry (1996) further constructed a measurement scale, considering PSM as primarily containing the following four dimensions: compassion (COM), self-sacrifice (SS), commitment to public interest (CPI), and attraction to public policy making (APM). These four dimensions can correspond to the aforementioned three motivational bases, respectively.

The link between service-learning courses and PSM lies in the former that presents students with an opportunity to contribute their efforts toward helping their service recipients solve their problems while empathizing with their situations. Regarding the rational and norm-based dimensions of PSM, previous studies have indicated that students can gain the awareness that as citizens, they should undertake a certain obligation toward society through the process of public service (Conway et al., 2009). Furthermore, they can become cognizant of having the ability to influence the formulation of public policy or participate in discussion (Rocha, 2000) and even have higher commitment to civic engagement (Einfeld and Collins, 2008; Nokes et al., 2005). With respect to the affective dimension of PSM, service processes give students the opportunity to understand other people’s needs, empathize with others’ situations (Liu, 2006), and arouse their enthusiasm to serve other people (Behar-Horenstein et al., 2015).

While comparing lower division (freshman and sophomore) with upper division (junior and senior) undergraduates, Ferrari and Bristow (2005) observed that the former displayed a greater sense of CPI, a stronger tendency to respond to social justice, and more willingness to serve other people at their own expense. Ferrari and Bristow believe that the possible reason was that the site where the study was carried out was a private Catholic university that emphasized human dignity, diversity, and the spirit of helping the poor and disadvantaged. The lower division also had more opportunities to witness the values and spirit emphasized by the university through courses or community service programs, thus displaying stronger PSM.

Using a quasi-experimental research design, Seider et al. (2011) analyzed the relation between a service program and students’ PSM. Their results indicated that the students participating in the service program had significantly higher PSM than those who were not involved in the program. Service experiences enabled the participants to acquire deeper understanding of social inequality, which consequently strengthened their hope for exercising their influence on society by using their own specialized knowledge. This can be considered the manifestation of the rational dimension of PSM.

Compared with the two previous studies that focused on the service-learning courses within universities, Ward (2014) used the data of a three-wave longitudinal study to understand whether the service programs of AmeriCorps helped improve the program participants’ PSM. AmeriCorps, an organization founded in 1994 by the Corporation for National and Community Service in the U.S.A., provides services for communities in need, non-profit organizations, or schools across America. Ward's study indicated that compared with those not participating in AmeriCorps’ service projects, the participating members’ sense of commitment and civic consciousness significantly improved after the services ended and this positive influence lasted for at least 7 years. In other words, AmeriCorps’ service programs had sustained influence on the participants’ PSM.

Dinour and Kuscin (2021) conducted pre–post surveys for 142 U.S. undergraduates who completed a fixed amount of service time as a requirement of a course to discover whether service-learning experience leads to any change in students’ PSM level. Their findings revealed that the students who experienced service learning tend to report a higher level of PSM in the post-survey, irrespective of the types of service they provided. As regards PSM sub-dimensions, service-learning experience also presented positive impact on students’ CPI, COM, and SS.

Based on the above discussion, experiential learning activities with service as their core in theory ought to enhance students’ problem-solving ability and PSM. However, it is still unclear whether the service-learning course designed on the basis of public affairs education approach can better reinforce the two desirable outcomes— problem-solving ability and PSM. Thus, this study proposes a service-learning course design grounded in public problem analysis. We collected data through the actual implementation of the instructional plan to examine the immediate and sustained outcomes of the course.

A service-learning course design grounded in public problem analysis

Course design

This study proposes a project-based service-learning course design based on the approach of public affairs education. This instructional design was implemented in a semester’s 2-credit undergraduate course offered by a center of general education in a public university in Taiwan.² Students from any academic department in the university were eligible to enroll in this course; however, the awarded credits would not count toward the degree for those who were majoring in public administration. The purpose of this course was to cultivate students’ interest in public service and their problem-solving skills through experiential learning. Students were asked to conduct a project related to public affairs and propose their solutions to the societal problems of interest. Subsequently, they provided service for the target groups according to their proposal. This process intended to strengthen their ability to analyze the problems, their sense of responsibility toward public affairs, and their compassion toward disadvantaged groups. Figure 1 illustrates the conceptual framework of this study.

To enhance problem-solving ability, the course required students to first discover the public problem that they found interesting and seek a thorough understanding of the problem background in literature or news reports. Public problems can cover a wide range of unresolved social issues such as poverty or gender inequality. However, the students were encouraged to focus on local community problems such that the scope of chosen problems does not exceed the limits of their ability. Then, they would identify the causes and propose feasible solutions based on the results of in-depth interviews or participant observations. Some examples of the selected public problems were social isolation of older adult community living alone, poor academic performance of indigenous children from a remote community, etc.

With respect to PSM, the following mechanisms are expected to strengthen students’ rational, norm-based, and affective motives:

1. Rational motives: Through the multi-stage process of implementing projects (identifying problems, reviewing literature, collecting data, proposing solutions, etc.), students realize their role in solving societal problems wherein they find solutions for the target groups using their own expertise or abilities, thereby exercising an influence in solving social problems.

2. Norm-based motives: In the process of analyzing the problems, students realize that every citizen is responsible to show concern for public issues. If everybody cares merely for the issues of immediate relevance to their own lives, society can hardly advance in a holistic direction.

3. Affective motives: Students had the opportunity to perceive the seriousness of a problem through face-to-face interaction with the target groups or through participant observations. In the process of providing service for the target groups, they could also more profoundly understand others’ predicaments, which in turn aroused their compassion toward disadvantaged groups and their willingness of self-sacrifice for public interests.

Instructional methods

Prior research has indicated that well-designed service-learning courses should at least include four stages of learning–action, reflection, demonstration, and celebration (Fertman et al., 1996; Kaye, 2010). The service-learning course designed by this study contained both elements of project implementation and service practice; accordingly, the learning process covered the following six stages:

1. Preparation: The instructional method in this stage was primarily classroom discussion. The first week started with the introduction of the course to enable students to understand the requirements and implementation process of the course. Students were also informed about the purpose of the underlying research that they were voluntary to participate in; their decision about whether or not to participate will not affect their current or future relations with the instructor. In the second and third week, students selected the public problems of interest, and then, they held discussion in the class to limit the scope of the problems as well as plan out the methods to be used later for data collection.³

2. Observation: In this stage, the instructional method was primarily field observation for understanding the causes of the problems. From the fourth to sixth week, students collected data at the site of their practice according to the discussion in the previous 3 weeks. In the process, the instructor helped revise the questionnaires or interview outlines proposed by each student group to ensure that they were helpful for the subsequent formulation of solutions.

3. Proposal: Class discussion was the main instructional method in this stage. After obtaining an understanding of the causes of the problems based on the data they collected, students formulated their solutions and presented them in class. Other students and the instructor then offered suggestions for revision.

4. Service: The experiential teaching model was adopted in this stage, where students provided actual services to target groups based on the solutions they had proposed. During their service provision, the instructor contacted service receiving organizations (or their representatives) to learn about students’ service practices, or conducted on-site visits to understand the status of students’ service implementation.

5. Reflection: At this stage, the self-directed learning model was adopted. Based on the service experience gained from the course, students completed their project reports. The report included a description of the research problems, data collection methods, content of the solutions proposed, process of service implementation, and reflection on service learning.

6. Demonstration and feedback: The presentation instruction method was used at this stage. Students turned the content of their project reports into posters to be publicly displayed and presented on campus. In the process, they explained to the audience their observations and thoughts on their services and responded to the questions from the audience. Finally, the instructor gave her feedback on the content.

Research method

Data source

This study implemented the proposed instructional plan in a general education course offered in a public university in Taiwan. A total of 27 freshman and sophomore students from different departments of the university undertook the course.⁴ This study employed a quasi-experimental research design to analyze the pre-and post-event results of the treatment and comparison groups to ascertain the influence of the course on students’ problem-solving ability and PSM. In the first lesson, this study informed the students taking the course about the purpose of the research project and invited them to participate in it. The students were also requested to recommend a student of the same sex, from the same department, and at the same year-level as them to join the research as members of the comparison group.

In the second lesson, a questionnaire pretest (Wave 1) was administered to the students through an online survey platform during the class to obtain the necessary data. Simultaneously, students not taking the course were also invited through email or text message to respond to the same questionnaire. In the 18th lesson (last week of the semester), the research team again invited the treatment and comparison groups to complete the first posttest questionnaire (Wave 2) to understand the immediate effects of the course. Finally, half a year after the end of this course, the members of both groups were once again invited to take the second posttest (Wave 3) to understand the persistence of the course effects. Figure 2 presents the survey administration process.

The respondents of the questionnaires in Wave 1 survey were 27 course takers, 27 non-course takers, and 13 students who either withdrew from the course or were original paired-up partners, totaling 67 persons.⁵ Those who returned the questionnaires in Wave 2 survey were 27 course takers, 25 non-course takers, and 6 who either withdrew or were original paired-up partners, adding up to 58 persons. In Wave 3 survey, those who responded to the questionnaires were 17 course takers, 14 non-course takers, and 5 students who either withdrew or were original paired-up partners, thus accounting for 36 persons.

Variables and measures

The primary outcomes variables in this study were Problem-solving Ability and five PSM sub-dimensions (Attraction to Public Service [APS], Commitment to Public Value [CPV], COM, SS, and APM). Problem-solving Ability was measured using 10 out of the questionnaire developed by Pan and Wang (2003). The original questions were revised to better fit with the design content of this course (6-point Likert scales: 1 = strongly disagree and 6 = strongly agree). A higher self-rating score means higher perceived problem-solving ability. Public service motivation was measured using the questionnaire developed by Kim et al. (2013). The original scale comprises 16 items used to operationalize four PSM sub-dimensions, namely, APS, CPV, COM, and SS. As APM has also been considered as an important subdimension of PSM (Perry, 1996), the present study further incorporated Kim’s (2009) scale on APM to fully reflect the concept of PSM. Six-point Likert scales were adopted for all questions (1 = strongly disagree and 6 = strongly agree). A higher average score indicates a higher level of PSM (APS, CPV, COM, SS, and APM). The above questions were asked in all three waves of survey.

Wave 2 survey additionally asked the course takers to give their opinions on how helpful the course was to them to obtain their subjective evaluation of the influence of the course. Six-point Likert scales were adopted for the relevant questions (1 = not helpful at all and 6 = very helpful). A higher score indicates a more positive perception of the course by the respondents.

Given that individual factors other than the course may also affect students’ problem-solving ability and PSM, this study included several control variables in the analysis: Male (1 = male; 0 = female); Having a Minor/Second Major (1 = yes; 0 = no); Grade (1 = freshman; 0 = sophomore); three dummy variables representing the respondents’ college (College of Social Sciences, College of Public Affairs, and Others). All variables and their measures are presented in Appendix.

Method

This study used the DID method in analyzing whether significant differences existed between the treatment and comparison groups in problem-solving ability and PSM. The operational method was to separately calculate the differences between the treatment group’s posttests and pretest and those between the comparison group’s posttests and pretest; the disparity between the two calculations represents the effects produced by the treatment (Gertler et al., 2016). To find out the persistence of the effects of the course, this study also calculated the disparity between the pretest and the second posttest and verified whether the difference between the treatment and comparison groups reached the level of statistical significance. The DID equation can be described as follows:

Immediate effects of course treatment = (treatment group’s first posttest–treatment group’s pretest) – (comparison group’s first posttest–comparison group’s pretest)

Sustained effects of course treatment = (treatment group’s second posttest–treatment group’s pretest) – (comparison group’s second posttest–comparison group’s pretest)

To ensure that the DID analysis produces unbiased estimation, the parallel trend assumption had to be met, that is, the outcome variables of both the treatment and comparison groups must present identical variation trends without treatment intervention (Gertler et al., 2016). As this study could not collect the pretest data of several time points, it was unable to analyze, through plotting or verifying the pretest data, whether the variations in the problem-solving ability and PSM of both groups indicated an identical trend before treatment intervention (Wing et al., 2018). However, following Fredriksson and Oliveira’s (2019) recommendation, this study adopted the sampling strategy that matched the course takers (treatment group) with non-course takers (comparison group) who are identical regarding sex, grade, and major such that both groups could be as homogeneous as possible. We expected that the outcome variables could thus manifest identical variation trends across two groups under the situation where there was no treatment intervention.

As the number of respondents responding to the three-wave surveys differed, however, there were situations where values were missing among certain paired groups (e.g., respondents of the treatment group responded to the pretest and posttest questionnaires, but members of the comparison group paired up with them only responded to the pretest questionnaire). This study thus handled and analyzed the data in two approaches. The first approach conducted the analysis using the partially paired data, which included all those who had responded to the pre-and posttests. In other words, those who had responded to the pretest and posttest questionnaires but whose paired-up partners had no complete data were still included in the analysis. This approach also treated those who withdrew midway and their original activity partners as members of the comparison group in performing the analysis. A total of 58 respondents (27 from the treatment group and 31 from the comparison group) replied to the pretest and Wave 1 questionnaires, while only 36 respondents (17 from the treatment group and 19 from the comparison group) replied to the pretest and Wave 2 posttest questionnaires.

The second approach conducted the analysis using fully paired data, which only included the complete pairs that had responded to the pretest and posttest questionnaires. In other words, if a member of a pair did not reply to any wave of the survey, then the members of that paired group would be entirely excluded. There were 25 pairs providing complete data in the pretest and Wave 1 posttest, totaling 50 respondents, while only 10 pairs provided data in the pretest and Wave 2 posttest, with a total of 20 respondents. The description of the sample is presented in Table 1.

Table 1.

Sample description.

	Pretest and 1^st posttest partially paired data		Pretest and 2^nd posttest partially paired data		Pretest and 1^st posttest fully paired data		Pretest and 2^nd posttest fully paired data
Attributes	Frequency	Percentage, %	Frequency	Percentage, %	Frequency	Percentage, %	Frequency	Percentage, %
Course enrollment
Course takers	27	46.6	17	47.2	25	50.0	10	50.0
Non-course takers (include the withdrawn)	31	53.5	19	52.8	25	50.0	10	50.0
Sex
Male	26	44.8	11	30.6	22	44.0	2	10.0
Female	32	55.2	25	69.4	28	56.0	18	90.0
Department
Computer science and information engineering	1	1.7	1	2.8	1	2.0	0	0.0
Law	2	3.5	1	2.8	1	2.0	0	0.0
Public finance	3	5.2	3	8.3	0	0.0	0	0.0
Real estate & built environment	8	13.8	4	11.1	8	16.0	2	10.0
Business administration	1	1.7	0	0.0	1	2.0	0	0.0
Finance and cooperative management	1	1.7	0	0.0	1	2.0	0	0.0
Leisure and sport management	5	8.6	2	5.6	2	4.0	0	0.0
Accountancy	8	13.8	4	11.1	8	16.0	2	10.0
Economics	1	1.7	0	0.0	0	0.0	0	0.0
Social work	6	10.3	5	13.9	6	12.0	4	20.0
Sociology	22	37.9	16	44.4	22	44.0	12	60.0
Grade
Freshman	45	77.6	27	75.0	44	88.0	18	90.0
Sophomore	13	22.4	9	25.0	6	12.0	2	10.0

Results

This section first presents the subjective evaluation of the respondents regarding the influence of the course, and then discusses the analysis results of the DID method. Table 2 indicates that the rating about the course given by the 27 course takers was the lowest in “cultivation of problem-solving ability,” with the average being 4.44. The standard deviation of this item, however, was the highest among all the items (SD = 0.97), implying a certain degree of difference in the rating among the course takers. Furthermore, the course takers gave the highest rating of the course in “cultivation of compassion,” with an average score of 5.04. The average scores of remaining items were all over 4.6, implying that overall, the course takers considered that this service-learning course did at least “somewhat helped” in enhancing their PSM.

Table 2.

Subjective evaluation of the course impact.

Subjective course evaluation	Obs	Mean	SD	Min	Max
CE Q1 this course enabled me to improve my problem-solving ability.	27	4.44	0.97	2	6
CE Q2 this course enabled me to become willing to provide assistance to the service recipients at my own expense.	27	4.67	0.83	3	6
CE Q3 this course enabled me to enhance my public service motivation.	27	4.78	0.64	4	6
CE Q4 this course enabled me to have greater enthusiasm for participating in public service.	27	4.81	0.62	4	6
CE Q5 this course enabled me to perceive the importance of public values (relative to private interests).	27	4.96	0.65	4	6
CE Q6 this course enabled me to become more compassionate toward the service recipients.	27	5.04	0.71	4	6

However, the course takers’ subjective evaluation regarding the influence of the course might not truly reflect the learning outcomes. As the questionnaires directly asked for the respondents’ rating of the course, the course takers, for the reason of feeling embarrassed to give negative views, might give higher scores to all items. Hence, this study invited the respondents to fill out the problem-solving ability and PSM scales in the three waves of the survey, to facilitate comparing the difference between the course takers and non-course takers in their pretest and posttest scores.

We first performed an analysis using the partially paired data. Table 3 presents the results of the DID analysis of problem-solving ability with control variables using the pretest and first posttest data. As the first posttest was conducted in the last week of the course, the analysis result presents the immediate effects of the service-learning course. According to Table 3, the variable Time in the PS Q5 and PS Q10 models displayed positive and significant influence. This means that the posttest average responses of Questions 5 and 10 related to problem-solving ability were significantly higher than the pretest average responses. This represented that along with the increase in the time of attending the school, the students became more confident that they could systematically think about the problems and apply what they had learned to solve them. However, the results presented here are the overall trend of all the respondents, which might not necessarily mean that such change stemmed from the course.

Table 3.

Immediate effect of the course on problem-solving ability (partially paired data).

Problem-solving ability	PS Q1		PS Q2		PS Q3		PS Q4		PS Q5
Problem-solving ability	Coef.	SE	Coef.	SE	Coef.	SE	Coef.	SE	Coef.	SE
Time (1 = 1^st posttest, 0 = Pretest)	0.10	0.22	−0.03	0.20	0.06	0.15	0.06	0.21	0.36	0.21†
Group (1 = Treatment, 0 = Comparison)	0.06	0.21	0.25	0.19	−0.06	0.15	−0.25	0.18	0.24	0.20
Time*Group	−0.06	0.27	−0.08	0.27	−0.32	0.21	−0.14	0.26	−0.29	0.26
Male	−0.05	0.21	−0.29	0.16†	−0.22	0.18	0.17	0.18	−0.05	0.20
Having a minor/Second major	0.16	0.36	−0.39	0.33	0.27	0.27	0.34	0.33	0.38	0.22†
Grade	−0.04	0.24	−0.13	0.22	0.09	0.13	−0.06	0.18	0.08	0.16
College of social sciences	−0.04	0.24	−0.08	0.20	0.02	0.20	0.19	0.21	−0.01	0.21
College of public affairs	0.24	0.22	0.28	0.18	−0.11	0.15	0.19	0.17	0.11	0.20
Intercept	4.68	0.31	5.15	0.24	5.48	0.22	4.95	0.29	4.79	0.22
Observations	116		115		115		116		115
R-squared	0.025		0.069		0.124		0.071		0.057
	PS Q6		PS Q7		PS Q8		PS Q9		PS Q10
Problem-solving ability	Coef.	SE	Coef.	SE	Coef.	SE	Coef.	SE	Coef.	SE
Time (1 = 1^st posttest, 0 = Pretest)	−0.16	0.26	−0.06	0.21	0.23	0.28	0.23	0.19	0.42	0.18*
Group (1 = Treatment, 0 = Comparison)	0.19	0.23	0.29	0.21	0.40	0.28	−0.03	0.17	0.32	0.17†
Time*Group	0.16	0.32	−0.05	0.29	−0.15	0.35	−0.30	0.24	−0.60	0.24*
Male	0.18	0.37	−0.60	0.29*	0.24	0.25	0.09	0.17	0.19	0.15
Having a minor/Second major	−0.94	0.58	−0.49	0.45	0.50	0.38	−0.24	0.29	−0.12	0.30
Grade	−0.28	0.24	−0.33	0.20†	0.13	0.21	−0.04	0.18	−0.10	0.20
College of social sciences	0.07	0.39	−0.34	0.31	−0.31	0.31	0.31	0.19	0.18	0.18
College of public affairs	0.21	0.21	0.07	0.20	0.22	0.27	0.13	0.19	0.04	0.18
Intercept	5.07	0.37	5.49	0.33	3.74	0.35	4.74	0.22	4.68	0.25
Observations	116		116		116		116		116
R-squared	0.100		0.110		0.130		0.078		0.081

†<0.1, *p < 0.05, **p < 0.01, ***p < 0.001.

Moreover, the variable Group manifested positive and significant influence in the PS Q10 model (p < 0.1), which is to say that compared with the comparison group, the treatment group on average had a higher self-rating with respect to their ability to use what they have learned to solve the problems in life. This may be due to the influence of the service-learning course but may also be a result of the possibility that the students with higher confidence in their own problem-solving ability simultaneously had a higher tendency to choose to enroll in this course.

Finally, the variable Time*Group showed significant influence in the PS Q10 model, while its regression coefficient was negative (Coef. = −0.6, p < 0.05). In other words, as regards whether one could use what one has learned to solve the problems in life, treatment group members’ sense of confidence did not increase as much as their non-course takers counterparts. Instead of enhancing the students’ confidence in using what they have learned to solve the problem, this service-learning course seemed to have negatively influenced them.

Table 4 presents analysis results of PSM with control variables using the pretest and first posttest data. The variable Group positively and significantly influenced COM (p < 0.1), implying that relative to the comparison group, the treatment group was generally more compassionate. However, the variable Time*Group did not show significant influence in the COM model, representing that the service-learning course was limited in increasing the students’ compassion. Meanwhile, the variable Time*Group displayed positive influence on the APM model (p < 0.1). This implies that the service-learning course not only improved students’ understanding of the issues that they cared for, but also triggered their interest in policy participation and desires to exercise an influence in the formulation of public policies.

Table 4.

Immediate effect of the course on PSM (partially paired data).

Public service motivation	APS		CPV		COM		SS		APM
Public service motivation	Coef.	SE	Coef.	SE	Coef.	SE	Coef.	SE	Coef.	SE
Time (1 = 1^st posttest, 0 = Pretest)	−0.23	0.16	0.01	0.16	0.01	0.18	−0.10	0.23	−0.28	0.18
Group (1 = Treatment, 0 = Comparison)	0.16	0.15	0.07	0.13	0.25	0.14†	0.36	0.24	−0.14	0.20
Time*Group	0.03	0.20	−0.14	0.19	−0.09	0.20	0.23	0.29	0.42	0.25†
Male	−0.09	0.15	−0.08	0.17	0.16	0.14	0.03	0.26	0.18	0.16
Having a minor/Second major	0.004	0.23	−0.07	0.27	−0.27	0.23	−0.18	0.28	−0.52	0.25*
Grade	0.02	0.16	0.10	0.14	−0.19	0.16	−0.11	0.26	−0.10	0.17
College of social sciences	0.07	0.16	0.06	0.18	0.22	0.16	0.01	0.29	0.11	0.19
College of public affairs	0.03	0.16	−0.09	0.16	−0.03	0.16	0.08	0.24	0.15	0.19
Intercept	4.87	0.20	4.97	0.25	4.76	0.20	3.96	0.29	4.51	0.24
Observations	116		116		116		116		116
R-squared	0.074		0.043		0.072		0.091		0.087

†<0.1, *p < 0.05, **p < 0.01, ***p < 0.001.

Tables 5 and 6 are the analysis results of problem-solving ability and PSM with control variables using the pretest and second posttest data. As the second posttest was administered half a year after the course ended, the analysis results present the sustained effects of the service-learning course. Tables 5 and 6 revealed that the variable Time*Group showed no significant influence in any model, implying that this service-learning course neither changed students’ confidence on problem solving nor increased their PSM (and its sub-dimensions) as time progressed.

Table 5.

Sustained effect of the course on problem-solving ability (partially paired data).

Problem-solving ability	PS Q1		PS Q2		PS Q3		PS Q4		PS Q5
Problem-solving ability	Coef.	SE	Coef.	SE	Coef.	SE	Coef.	SE	Coef.	SE
Time (1 = 2^nd posttest, 0 = Pretest)	0.11	0.27	<0.001	0.25	0.05	0.20	0.05	0.24	−0.05	0.26
Group (1 = Treatment, 0 = Comparison)	0.44	0.29	0.35	0.30	−0.004	0.22	−0.22	0.27	0.30	0.31
Time*Group	−0.22	0.35	0.18	0.32	0.12	0.26	0.07	0.31	0.11	0.34
Male	−0.19	0.34	−0.24	0.22	−0.19	0.21	0.44	0.27	−0.25	0.30
Having a minor/Second major	−0.12	0.50	0.12	0.55	0.34	0.31	0.02	0.46	0.10	0.33
Grade	−0.23	0.29	−0.23	0.25	0.17	0.21	−0.07	0.25	0.06	0.26
College of social sciences	−0.31	0.39	−0.09	0.27	0.07	0.23	0.32	0.29	−0.18	0.30
College of public affairs	−0.37	0.30	0.16	0.17	−0.003	0.22	−0.11	0.24	−0.11	0.29
Intercept	5.06	0.42	5.20	0.26	5.32	0.26	4.88	0.33	5.09	0.29
Observations	72		72		72		72		72
R-squared	0.063		0.118		0.096		0.075		0.079
	PS Q6		PS Q7		PS Q8		PS Q9		PS Q10
Problem-solving ability	Coef.	SE	Coef.	SE	Coef.	SE	Coef.	SE	Coef.	SE
Time (1 = 2^nd posttest, 0 = Pretest)	−0.05	0.29	0.32	0.24	−0.05	0.32	0.05	0.23	0.32	0.23
Group (1 = Treatment, 0 = Comparison)	0.06	0.26	0.49	0.24†	0.50	0.37	0.02	0.22	0.42	0.24†
Time*Group	0.17	0.37	−0.14	0.32	0.35	0.46	0.12	0.27	−0.43	0.27
Male	0.36	0.29	−0.39	0.31	−0.37	0.44	0.22	0.24	0.38	0.20†
Having a minor/Second major	−0.45	0.67	−0.17	0.35	0.46	0.54	−0.02	0.25	0.08	0.31
Grade	0.09	0.38	−0.35	0.26	0.06	0.28	−0.01	0.27	−0.30	0.24
College of social sciences	0.04	0.34	−0.57	0.32†	−0.58	0.51	0.09	0.23	0.37	0.20†
College of public affairs	0.10	0.29	−0.65	0.26*	−0.64	0.37†	−0.31	0.21	−0.01	0.20
Intercept	4.89	0.46	5.60	0.33	4.26	0.46	4.92	0.28	4.72	0.26
Observations	72		72		72		72		72
R-squared	0.102		0.179		0.174		0.081		0.142

†<0.1, *p < 0.05, **p < 0.01, ***p < 0.001.

Table 6.

Sustained effect of the course on PSM (partially paired data).

Public service motivation	APS		CPV		COM		SS		APM
Public service motivation	Coef.	SE	Coef.	SE	Coef.	SE	Coef.	SE	Coef.	SE
Time (1 = 2^nd posttest, 0 = Pretest)	−0.05	0.21	0.16	0.18	0.11	0.21	0.02	0.26	−0.28	0.26
Group (1 = Treatment, 0 = Comparison)	0.10	0.21	0.05	0.18	0.19	0.20	0.18	0.27	0.14	0.27
Time*Group	−0.04	0.25	−0.06	0.22	−0.14	0.24	0.09	0.36	0.36	0.34
Male	0.14	0.19	0.18	0.19	0.30	0.18	0.24	0.35	0.16	0.29
Having a minor/Second major	−0.35	0.30	−0.18	0.24	−0.47	0.20*	−0.93	0.40*	−0.80	0.36*
Grade	−0.11	0.21	−0.05	0.21	−0.24	0.21	−0.22	0.25	−0.45	0.27
College of social sciences	0.29	0.20	0.18	0.22	0.41	0.20*	0.48	0.39	0.29	0.31
College of public affairs	−0.09	0.19	−0.33	0.18†	−0.18	0.17	−0.39	0.34	−0.27	0.27
Intercept	4.94	0.23	5.00	0.30	4.78	0.22	4.06	0.39	4.72	0.34
Observations	71		71		71		71		71
R-squared	0.087		0.154		0.170		0.201		0.142

†<0.1, *p < 0.05, **p < 0.01, ***p < 0.001.

Given that the partially paired data could not have ensured the homogeneity between the treatment and comparison group, we alternatively conducted the analysis using fully paired data. This approach enabled us to verify the robustness of the analysis results displayed previously. It should be noted that the analysis results using fully paired data will not be biased only if the complete pairs are not significantly different from the incomplete pairs in terms of the key outcome variables (Problem-solving and PSM). When considering the immediate effect (the pretest and Wave 1 posttest), there were only 2 incomplete pairs (4 samples) and performing a statistical test of difference was not suitable. Thus, we only performed the test for the sustained effect (the pretest and Wave 2 posttest), which had a total of 17 incomplete pairs and 10 complete pairs. The results indicated that no significant difference was observed between the complete and incomplete pairs with regard to the scores of the 10 items on problem-solving ability and the five PSM dimensions.

Tables 7 –10 present the analysis results using fully paired data. Tables 7 and 8 display the course’s immediate effects on problem-solving ability and PSM, and Tables 9 and 10 demonstrate the results of its sustained effects. Consistent with the analysis results using the partially paired data, Table 7 indicates that the course negatively influenced perceived problem-solving ability, that is, the training of the service-learning course may lessen the students’ confidence in using what they have learned to solve problems. Conversely, Table 8 displayed that the regression coefficient of Time*Group in the APM model was positive but not significant, implying that the course did not affect the students’ APM as previously suggested. Finally, Tables 9 and 10 are consistent with the analysis results using the partially paired data, implying that the service-learning course seemed only to have immediate effects, with no continued role to play as time progressed.

Table 7.

Immediate effect of the course on problem-solving ability (fully paired data).

Problem-solving ability	PS Q1		PS Q2		PS Q3		PS Q4		PS Q5
Problem-solving ability	Coef.	SE	Coef.	SE	Coef.	SE	Coef.	SE	Coef.	SE
Time (1 = 1^st posttest, 0 = Pretest)	0.16	0.22	<0.001	0.24	0.04	0.18	<0.001	0.24	0.29	0.24
Group (1 = Treatment, 0 = Comparison)	0.15	0.22	0.31	0.21	−0.11	0.17	−0.26	0.18	0.22	0.22
Time*Group	−0.12	0.28	−0.17	0.30	−0.23	0.23	−0.12	0.28	−0.21	0.30
Male	0.47	0.20*	−0.19	0.25	−0.33	0.28	0.28	0.22	0.21	0.33
Having a minor/Second major	0.35	0.27	−0.83	0.56	0.66	0.19**	−0.54	0.46	0.42	0.38
Grade	0.004	0.28	−0.20	0.32	0.06	0.17	−0.26	0.22	0.05	0.16
College of social sciences	0.48	0.25†	0.04	0.29	−0.15	0.31	0.37	0.24	0.24	0.35
College of public affairs	0.45	0.23†	0.35	0.25	−0.23	0.18	0.40	0.18*	0.20	0.24
Intercept	4.01	0.37	5.06	0.39	5.69	0.36	4.99	0.32	4.58	0.39
Observations	100		99		99		100		99
R-squared	0.073		0.074		0.128		0.102		0.046
	PS Q6		PS Q7		PS Q8		PS Q9		PS Q10
Problem-solving ability	Coef.	SE	Coef.	SE	Coef.	SE	Coef.	SE	Coef.	SE
Time (1 = 1^st posttest, 0 = Pretest)	<0.001	0.27	0.04	0.22	0.40	0.31	0.20	0.20	0.40	0.20*
Group (1 = Treatment, 0 = Comparison)	0.18	0.23	0.39	0.21†	0.52	0.30†	−0.06	0.17	0.30	0.18
Time*Group	−0.04	0.33	−0.16	0.30	−0.32	0.39	−0.28	0.25	−0.60	0.26*
Male	−0.08	0.61	−0.45	0.45	0.53	0.34	0.40	0.21†	0.30	0.17†
Having a minor/Second major	0.58	0.51	−0.77	0.24**	−0.10	0.22	0.57	0.44	0.51	0.52
Grade	0.10	0.32	−0.77	0.23**	−0.16	0.21	−0.05	0.18	−0.02	0.26
College of social sciences	−0.25	0.63	−0.24	0.49	−0.01	0.40	0.53	0.24*	0.28	0.21
College of public affairs	−0.08	0.26	0.31	0.23	0.52	0.32	0.19	0.20	0.08	0.20
Intercept	5.02	0.67	5.68	0.49	3.55	0.43	4.50	0.30	4.51	0.33
Observations	100		100		100		100		100
R-squared	0.031		0.165		0.180		0.097		0.088

†<0.1, *p < 0.05, **p < 0.01, ***p < 0.001.

Table 8.

Immediate effect of the course on PSM (fully paired data).

Public service motivation	APS		CPV		COM		SS		APM
Public service motivation	Coef.	SE	Coef.	SE	Coef.	SE	Coef.	SE	Coef.	SE
Time (1 = 1^st posttest, 0 = Pretest)	−0.25	0.19	0.06	0.19	0.01	0.20	−0.14	0.26	−0.28	0.21
Group (1 = Treatment, 0 = Comparison)	0.14	0.16	0.14	0.15	0.26	0.15†	0.33	0.25	−0.11	0.22
Time*Group	0.10	0.23	−0.14	0.21	−0.08	0.23	0.24	0.32	0.42	0.28
Male	0.20	0.24	0.22	0.23	0.30	0.21	0.32	0.46	0.40	0.25
Having a minor/Second major	−0.31	0.33	−0.33	0.22	−0.20	0.21	−0.77	0.32*	−0.86	0.24***
Grade	−0.06	0.22	0.07	0.17	−0.22	0.21	−0.43	0.30	−0.05	0.22
College of social sciences	0.38	0.26	0.37	0.26	0.35	0.23	0.32	0.49	0.40	0.28
College of public affairs	0.21	0.19	0.03	0.19	0.03	0.19	0.38	0.25	0.30	0.24
Intercept	4.61	0.35	4.61	0.33	4.62	0.31	3.95	0.52	4.17	0.36
Observations	100		100		100		100		100
R-squared	0.089		0.058		0.074		0.125		0.086

†<0.1, *p < 0.05, **p < 0.01, ***p < 0.001.

Table 9.

Sustained effect of the course on problem-solving ability (fully paired data).

Problem-solving ability	PS Q1		PS Q2		PS Q3		PS Q4		PS Q5
Problem-solving ability	Coef.	SE	Coef.	SE	Coef.	SE	Coef.	SE	Coef.	SE
Time (1 = 2^nd posttest, 0 = Pretest)	<0.001	0.38	−0.10	0.41	−0.10	0.20	−0.40	0.32	−0.50	0.31
Group (1 = Treatment, 0 = Comparison)	0.01	0.38	0.30	0.39	−0.21	0.23	−0.69	0.29*	−0.19	0.36
Time*Group	0.10	0.44	0.40	0.47	0.20	0.30	0.60	0.40	0.60	0.42
Male	<0.001	0.33	−0.75	0.27*	0.25	0.21	<0.001	0.46	<0.001	0.70
Having a minor/Second major	0.94	0.82	<0.001	0.80	1.11	0.18***	−0.11	0.42	−0.11	0.31
Grade	0.65	0.82	0.25	0.44	0.66	0.14***	0.23	0.25	0.02	0.25
College of social sciences^a	−0.07	0.27	−0.50	0.22*	−0.14	0.23	0.04	0.45	−0.18	0.50
Intercept	4.07	0.84	5.05	0.44	5.19	0.26	5.16	0.48	5.43	0.62
Observations	40		40		40		40		40
R-squared	0.074		0.205		0.207		0.181		0.106
	PS Q6		PS Q7		PS Q8		PS Q9		PS Q10
Problem-solving Ability	Coef.	SE	Coef.	SE	Coef.	SE	Coef.	SE	Coef.	SE
Problem-solving Ability	Time (1 = 2^nd posttest, 0 = Pretest)	−0.10	0.30	0.30	0.32	0.20	0.52	<0.001	0.30	0.30	0.31
Group (1 = Treatment, 0 = Comparison)	−0.16	0.29	0.22	0.28	0.40	0.49	<0.001	0.29	0.21	0.32
Time*Group	0.10	0.41	−0.10	0.41	0.20	0.59	<0.001	0.33	−0.30	0.37
Male	0.50	0.49	<0.001	0.30	0.25	0.62	0.75	0.37†	0.25	0.20
Having a minor/Second major	2.61	0.45***	−0.17	0.27	<0.001	0.60	1.00	0.18***	0.94	0.52†
Grade	2.13	0.18***	−0.12	0.23	0.18	0.58	1.04	0.18***	0.54	0.34
College of social sciences^a	0.07	0.44	−0.29	0.27	−0.07	0.40	0.54	0.30†	0.32	0.22
Intercept	2.98	0.52	5.14	0.32	3.47	0.66	3.46	0.32	4.03	0.39
Observations	40		40		40		40		40
R-squared	0.426		0.094		0.127		0.268		0.142

†<0.1, *p < 0.05, **p < 0.01, ***p < 0.001.

^aThere were only two samples from the College of Public Affairs. They were thus incorporated into the category of Others (reference group).

Table 10.

Sustained effect of the course on PSM (fully paired data).

Public service motivation	APS		CPV		COM		SS		APM
Public service motivation	Coef.	SE	Coef.	SE	Coef.	SE	Coef.	SE	Coef.	SE
Time (1 = 2^nd posttest, 0 = Pretest)	−0.03	0.30	0.18	0.25	0.08	0.31	−0.10	0.34	−0.03	0.39
Group (1 = Treatment, 0 = Comparison)	0.03	0.26	−0.05	0.23	−0.004	0.25	−0.004	0.30	0.11	0.37
Time*Group	0.03	0.35	−0.13	0.30	−0.08	0.34	0.43	0.41	0.27	0.44
Male	−0.13	0.30	0.38	0.35	0.44	0.31	0.75	0.43	0.17	0.42
Having a minor/Second major	−0.04	0.27	0.74	0.24**	0.54	0.19**	−0.71	0.27*	−0.41	0.33
Grade	0.67	0.26*	0.93	0.22***	0.61	0.19**	0.08	0.21	0.30	0.28
College of social sciences^a	0.38	0.21†	0.56	0.31†	0.53	0.31†	0.75	0.40†	0.33	0.39
Intercept	4.14	0.33	3.73	0.40	3.94	0.37	3.55	0.42	3.85	0.40
Observations	40		40		40		40		40
R-squared	0.208		0.268		0.143		0.213		0.112

†<0.1, *p < 0.05, **p < 0.01, ***p < 0.001.

^aThere were only two samples from the College of Public Affairs. They were thus incorporated into the category of Others (reference group).

Discussion

Based on the research findings, we further discuss the relation between the service-learning course and problem-solving ability/PSM, including both immediate and sustained effects.

Service learning and problem-solving ability

The DID analysis results indicated that the service-learning course grounded in public problem analysis did not remarkably help the students’ problem-solving ability (cogitation about problems, identification of causes, collection of required data, etc.); rather, it weakened their sense of confidence in using what they have learned to solve the problems in life. In other words, unlike the findings of previous research (Guo et al., 2016; Liu, 2006; Yorio and Ye, 2012; Zheng et al., 2013), the service-learning course designed based on the approach of public affairs education seems to be not helpful in enhancing students’ problem-solving ability. This result implies that it might be difficult to improve problem-solving skills by simply taking a couple of courses, especially when the underlying problems are more complex or involved in a wider range.

Nevertheless, before coming to the above conclusion, considering other factors is also necessary. First, this course hoped to enable the students to acquire a sense of having the capacity to exercise some influence over the handling of specific problems through conducting a project related to public affairs. Although such a design was well-intentioned, the problems that the students proposed right from the start may have been too big. After all, the dimensions involved in public problems are broad, with many also involving structural factors (e.g., regulation and resource constraints) that students have little control over. Thus, the scope that the students could handle was relatively limited. Moreover, given that most course takers did not possess the specialized knowledge in public problem analysis, they had to spend more time narrowing down the problems and interacting with the stakeholders to propose feasible solutions. This may explain how the students’ confidence in their problem-solving ability was negatively affected.

Second, some service organizations could not allow the students to provide service according to their proposals, and instead, they hoped that students would provide service according to the service items and goals set by the organizations. Consequently, some students became mere service providers and could not apply the results of their project to actual service delivery. In addition, some organizations expected long-term service commitment and were not inclined to accept short-term service provided in combination with courses. If students can only provide short-term services, or their service time is restricted by their semester term, this will cause some degree of disturbances to the service organizations given that students needed to be given training before engaging in service (Tryon et al., 2008). This reality prevented some student groups from finding suitable organizations to deliver service in connection with the problems that interested them. They were forced to change their topics or revise their solutions. This may in turn affect their sense of confidence in their problem-solving abilities.

Service learning and PSM

Previous research has indicated that participating in service activities could awaken individuals’ civic consciousness and compassion; individuals may thus aspire to improve public problems through civic engagement or delivery of public services (Conway et al., 2009; Liu, 2006; Rocha, 2000). However, the analysis of this study has mixed findings on the relation between the service-learning course and PSM. When performing the DID analysis using the partially paired data, the results provided evidence for the positive effect of the service-learning course on the APM dimension of PSM. However, when the fully paired data were used for analysis, the positive relation between the service-learning course and APM did not reach the significance level.

We believe that the cause of the inconsistent results should not lie in the use of two extra incomplete pairs in the partially paired data, but rather in the inclusion of data of the respondents who enrolled but later withdrew from the course. According to our observation, the respondents who later withdrew had high interest in the course at the beginning of the semester. They felt displeased and determined to withdraw from the course as some group members exhibited free-rider behavior. These respondents, who had withdrawn but still responded to the posttest questionnaires, may be very much akin to the treatment group members in terms of relevant attributes (e.g., they also had the interest to participate in service activities). Including those in the comparison group would help this study bring the homogeneity between the treatment and comparison groups closer. Hence, even though the analysis of the fully paired data did not support the positive relation between the service-learning course and APM, the analysis results of the partially paired data regarding APM still provided valuable information.

Immediate and sustained effects of the service-learning course

Our DID analysis results indicated that the service-learning course design proposed by this study only produced immediate effects but no sustained ones, irrespective of using the partially or fully paired data. This result contrasts with Ward (2014), who found that AmeriCorps’ service programs enhanced participants’ PSM and such positive influence could even persist for at least 7 years. One possible explanation is that the returned samples size of our first posttest was 86.5% of that of the pretest, while the returned sample size of the second one only reached 53.7%. This loss of samples between the two posttests might be a reason why the sustained effects were not obvious. The other more possible explanation is that most members participating in AmeriCorps service programs would provide a year-long full-time service, while some were providing part-time service, yet with their minimum service hours also reaching 300.⁶ The requirement of this service-learning course for the students, however, was only a minimum of 20 h of service. Excluding the preparation, observation, and proposal stages, the period during which the students actually provided service for the target groups concentrated within 9 weeks. The services, whether in terms of length or intensity, were limited. Once the service activities ended, the students’ experiential awareness remained fresh. In the process of writing their project reports, they reflected on their service-learning experiences, and thus, with respect to problem-solving ability and some dimensions of PSM, some changes were observed among the students. Once the course’s immediate effects cease to drive students’ continued participating in service activities after the course ended, expecting that the changes brought about by the course would persist as time progressed is hard.

Conclusion

This study proposed a service-learning course grounded in public problem analysis and took a general education course of a public university in Taiwan as its instructional practice field. Through a quasi-experimental research design, the study conducted three waves of questionnaire survey before and after the course, seeking to understand whether the instructional design of this course helped enhance students’ problem-solving ability and PSM. The DID analysis results indicated that the project-based service-learning course in the field of public affairs tended to decrease, rather than increase, the students’ confidence in using what they had learned to solve problems. However, the course’s negative influence on the students’ problem-solving ability was found to show only immediate effects without persisting as time progressed. Furthermore, for the relation between the service-learning course and PSM, this study failed to obtain consistent conclusions. The analysis performed on the partially paired data showed that the course could raise the students’ aspiration for policy participation (one subdimension of PSM); yet when the fully paired data were analyzed, the course was shown to have no significant influence on the various dimensions of PSM.

In view of the results of this study, several directions can be considered for the improvement of service-learning courses grounded in public affairs education. First, appropriate adjustment should be made to the proposed instructional plan according to the situations of target students. If such courses are to be offered as general education courses, with students mostly from underclassman levels, instructors can contact and negotiate with one or two specific service organizations beforehand to obtain their consent, allowing students some autonomy over service contents. For instance, there is an online platform named “RuRo,” which allows remote community or schools in Taiwan to report their problems (needs), and then the plateform would help match proposers to the problem with someone who can provide services to solve them.⁷ Instructors may first examine the requests for service proposed by various organizations and choose one problem area that is most suitable for students to work further. Then, instructors can contact the proposer to discuss possible flexibility on the service contents as per their needs. In this manner, students can ideate the service contents in one to two specific problem areas. Given that the boundary for an operable space has been demarcated in advance, the solutions that students propose accordingly will be more feasible, thereby reducing their sense of frustration. If the course is offered as an elective for departments (institutes) related to public affairs, offering it at upperclassman levels can be considered. As upperclassmen have more experience in report making and project inquiry, this can reduce student’s duration of data collection in the course. They also have an opportunity to put the results of their project investigation into practice through actual service delivery.

Second, most service-learning courses offered in Taiwanese universities last only one semester, and some also provide service in the form of groups or teams. This requires time for building group consensus and cultivating understanding in terms of problem definition and service program formulation. If students have to complete the projects and actually provide service within one semester (18 weeks), they may not be able to effectively digest what they have learned. Therefore, it seems more appropriate if the course design of this study can be implemented in the form of a two-semester course. Whether for problem-solving ability or PSM, constant learning stimuli and contacts are required for them to manifest more obvious progress, and only through continuous accumulation of learning outcomes can the course be expected to have persisting influence on students.

Despite its promising potential, certain limitations still exist in this study, making it unsuitable to overgeneralize its analysis results. First, as this study could not know who was going to enroll in the course in advance, it could not conduct pretest observations at several time points. Such limitation rendered it unable to provide data-based evidence regarding whether the parallel trend assumption was satisfied through plotting or testing. In relation to this limitation, this study attempted to make the treatment and comparison groups as homogenous as possible through the matching strategy, which is expected to raise the internal validity of the analysis results. Second, this study only investigated the effects of a service-learning course offered in a Taiwanese public university. Subsequent studies are recommended to adopt similar instructional plans and research designs and implement them in the service-learning courses of different universities to verify the external validity of the research results of this study.

In conclusion, this study clarifies the relation between service-learning courses, students’ problem-solving ability and PSM in the field of public affairs education. Our findings can also serve as a foundation for efforts to improve subsequent instructional plans of service-learning courses. Considering that the current literature has informed us little about sustained effect of service-learning courses and that this study has a relatively narrow scope, we call for future research to examine the extent to which service-based courses can promote students’ sense of citizenship or inclination to engage in public affairs over a long period of time. A qualitative research approach is also recommended to discover why service-based courses affect (or not affect) students’ attitude, motivation, or behavior.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported in part by the Taiwan Ministry of Education (PSL1090128).

ORCID iD

Wan-Ling Huang

Notes

Variables and their descriptions

Construct	Variables	Questions
Problem-solving ability	PS Q1	When facing a problem, I believe I can solve it.
	PS Q2	Before solving a problem, I will first think about what kind of problem I am facing.
	PS Q3	I think before solving a problem, I should first know its causes.
	PS Q4	In the process of solving a problem, I will often gather related materials.
	PS Q5	When confronted with a problem requiring solution, I will first ponder over the methods and steps to solve it.
	PS Q6	I will work together with other people to jointly solve a problem.
	PS Q7	I think it is necessary to compare the possible outcomes produced by each method in solving a problem.
	PS Q8	I will use scientific methods, such as “doing experiments”, to solve a problem.
	PS Q9	After I come up with a method to solve a problem, I will carefully carry it out.
	PS Q10	I will apply the methods I have learned to solve the problems I encounter in life.
Public service motivation	APS	I admire people who initiate or are involved in activities to aid my community.
		It is important to contribute to activities that tackle social problems.
		Meaningful public service is very important to me.
		It is important for me to contribute to the common good.
	CPV	I think equal opportunities for citizens are very important.
		It is important that citizens can rely on the continuous provision of public services.
		It is fundamental that the interests of future generations are taken into account when developing public policies.
		To act ethically is essential for public servants.
	COM	I feel sympathetic to the plight of the underprivileged.
		I empathize with other people who face difficulties.
		I get very upset when I see other people being treated unfairly.
		Considering the welfare of others is very important.
	SS	I am willing to risk personal loss to help society.
		I believe in putting civic duty before self.
		I am willing to risk personal loss to help society.
		I would agree to a good plan to make a better life for the poor, even if it costs me money.
	APM	I am interested in making public programs that are beneficial for my country or the community I belong to.
		Sharing my views on public policies with others is attractive to me.
		Seeing people get benefits from the public program I have been deeply involved in brings me a great deal of satisfaction.
Subjective Course evaluation	CE Q1	This course enabled me to improve my problem-solving ability.
	CE Q2	This course enabled me to become willing to provide assistance to the service recipients at my own expense.
	CE Q3	This course enabled me to enhance my public service motivation.
	CE Q4	This course enabled me to have greater enthusiasm for participating in public service.
	CE Q5	This course enabled me to perceive the importance of public values (relative to private interests).
	CE Q6	This course enabled me to become more compassionate toward the service recipients.
Control variables	Male	What is your sex?
	Having a minor/Second major	Are you doing a second major or a minor?
	Grade	What grade are you in now?
	College	Social sciences, public affairs, and others (including business, law, and electrical engineering and computer science)

References

Bardach

Patashnik

. A Practical Guide for Policy Analysis. Fifth Edition. Thousand Oaks, CA: CQ Press/SAGE Publications, Inc, 2016.

Balakas

Sparks

. Teaching research and evidence-based practice using a service-learning approach. The Journal of Nursing Education 2010; 49(12): 691–695.

Barker

From service learning to the scholarship of engagement: a taxonomy of civic renewal in American higher education. In: Paper presented at the Annual meeting of the American Political Science Association, Chicago, IL, 2004.

Behar-Horenstein

Feng

Roberts

, et al. Developing dental students’ awareness of health care disparities and desire to serve vulnerable populations through service-learning. Journal of Dental Education 2015; 79(10): 1189–1200.

Boyer

. The scholarship of engagement. Bulletin of the American Academy of Arts and Sciences 1996; 49(7): 18–33.

Case

Schram

Jung

, et al. A meta-analysis of the effect of adapted physical activity service-learning programs on college student attitudes toward people with disabilities. Disability and Rehabilitation 2021; 43(21): 2990–3002.

Coelho

Menezes

. University social responsibility, service learning, and students' personal, professional, and civic education. Frontiers in Psychology 2021; 12: 617300.

Conway

Amel

Gerwien

. Teaching and learning in the social context: a meta-analysis of service learning's effects on academic, personal, social, and citizenship outcomes. Teaching of Psychology 2009; 36(4): 233–245.

Dewey

. Experience and Education. New York, NY: Macmillan Company, 1938.

10.

Dinour

Kuscin

. Charity-and project-based service learning models increase public service motivation outcomes among dietetic students in a community nutrition course. Public Health Nutrition 2021; 24(13): 4268–4276.

11.

Etzkowitz

Webster

Gebhardt

, et al. The future of the university and the university of the future: evolution of ivory tower to entrepreneurial paradigm. Research Policy 2000; 29(2): 313–330.

12.

Einfeld

Collins

. The relationships between service-learning, social justice, multicultural competence, and civic engagement. Journal of College Student Development 2008; 49(2): 95–109.

13.

Ferrari

Bristow

. Are we helping them serve others? Student perceptions of campus altruism in support of community service motives. Education 2005; 125: 404–413.

14.

Fertman

White

. Service Learning in the Middle School: Building a Culture of Service. Columbus, OH: National Middle School Association, 1996.

15.

Fredriksson

Oliveira

. Impact evaluation using difference-in-differences. RAUSP Management Journal 2019; 54(4): 519–532.

16.

Furco

. Service-learning: a balanced approach to experiential education. In: Expanding Boundaries: Serving and Learning. Washington DC: Corporation for National Service, 1996, 2–6.

17.

Gertler

Martinez

Premand

, et al. Impact Evaluation in Practice. Washington, DC: The World Bank, 2016.

18.

Gronlund

. How to Write and Use Instructional Objectives. Fifth Edition. Englewood Cliffs, NJ: Pearson, 1995.

19.

Gujarathi

McQuade

. Service-learning in business schools: a case study in an intermediate accounting course. Journal of Education for Business 2002; 77(3): 144–150.

20.

Guo

Yao

Wang

, et al. The effects of service learning on student problem solving: The mediating role of classroom engagement. Teaching of Psychology 2016; 43(1): 16–21.

21.

Kaye

. The Complete Guide to Service Learning: Proven, Practical Ways to Engage Students in Civic Responsibility, Academic Curriculum, & Social Action (Revised & Updated Second Edition ed.). Minneapolis, MN: Free Spirit, 2010.

22.

Kim

(2009) Revising Perry's measurement scale of public service motivation. The American Review of Public Administration 39(2), 149–163.

23.

Kim

Vandenabeele

Wright

, et al. Investigating the structure and meaning of public service motivation across populations: developing an international instrument and addressing issues of measurement invariance. Journal of Public Administration Research and Theory 2013; 23(1): 79–102.

24.

Lambert

Mehta

Nelson

, et al. Development of professional competencies through service learning: an exploratory study. Critical Reviews™ in Physical and Rehabilitation Medicine 2022; 34(3): 31–40.

25.

Liu

R-L

. A study of developmental process and learning outcomes in service-learning curriculum focusing on social issues. Curriculum & Instruction Quarterly 2006; 19(3): 113–138. (in Chinese) .

26.

Long

Lee

Federico

, et al. Developing leadership and advocacy skills in medical students through service learning. Journal of Public Health Management and Practice 2011; 17(4): 369–372.

27.

Nokes

Nickitas

Keida

, et al.

Does service-learning increase cultural competency, critical thinking, and civic engagement?

The Journal of Nursing Education 2005; 44(2): 65–70.

28.

Novak

Markey

Allen

. Evaluating cognitive outcomes of service learning in higher education: a meta-analysis. Communication Research Reports 2007; 24(2): 149–157.

29.

Pan

I-Y

Wang

M-F

. A study on the effects of the play-based elementary science teaching. Journal of Taipei Municipal Teachers College 2003; 34: 157–172. (in Chinese) .

30.

Perry

. Measuring public service motivation: an assessment of construct reliability and validity. Journal of Public Administration Research and Theory 1996; 6(1): 5–22.

31.

Perry

Wise

. The motivational bases of public service. Public Administration Review 1990; 50(3): 367–373.

32.

Ramos-Monge

Audet

Barrena-Martínez

. Universities as corporate entities: the role of social responsibility in their strategic management. In: Emeagwali

(ed). Corporate Governance and Strategic Decision Making. Norderstedt, Germany: BoD–Books on Demand, 2017.

33.

Resch

Schrittesser

. Using the service-learning approach to bridge the gap between theory and practice in teacher education. International Journal of Inclusive Education 2021. Advance online publication.

34.

Rocha

. Evaluating experiential teaching methods in a policy practice course: the case for service learning to increase political participation. Journal of Social Work Education 2000; 36(1): 53–63.

35.

Salam

Awang Iskandar

Ibrahim

DHA

, et al. Service learning in higher education: a systematic literature review. Asia Pacific Education Review 2019; 20(4): 573–593.

36.

Sass

Coll

. The effect of service learning on community college students. Community College Journal of Research and Practice 2015; 39(3): 280–288.

37.

Seider

Rabinowicz

Gillmor

. The impact of philosophy and theology service-learning experiences upon the public service motivation of participating college students. The Journal of Higher Education 2011; 82(5): 597–628.

38.

Shumer

Lam

Laabs

. Ensuring good character and civic education: connecting through service learning. Asia Pacific Journal of Education 2012; 32(4): 430–440.

39.

Sigmon

. Linking Service with Learning. Washington, DC: Council of Independent Colleges, 1994.

40.

Tryon

Stoecker

Martin

, et al. The challenge of short-term service-learning. Michigan Journal of Community Service Learning 2008; 14(2): 16–26.

41.

Ward

. Cultivating public service motivation through AmeriCorps service: a longitudinal study. Public Administration Review 2014; 74(1): 114–125.

42.

Wing

Simon

Bello-Gomez

. Designing difference in difference studies: best practices for public health policy research. Annual Review of Public Health 2018; 39: 453–469.

43.

Winston

. Reflections upon community engagement: service-learning and its effect on political participation after college. Journal of Higher Education Outreach Engagement 2015; 19(1): 79–103.

44.

Yorio

. A meta-analysis on the effects of service-learning on the social, personal, and cognitive outcomes of learning. Academy of Management Learning & Education 2012; 11(1): 9–27.

45.

Zheng

Wang

Yin

. Correlation analysis of scaffolding creative problem solving through question prompts with process and outcomes of project-based service learning. In: Paper presented at the 2013 ASEE Annual Conference & Exposition, Atlanta, GA, 2013.