Integrating industry-crowdsourced projects in university capstone courses: A comparative study using parametric statistics and sentiment analysis

Introduction

In the evolving landscape of higher education, integrating real-world applications and experiential learning methods has become increasingly crucial for producing industry-ready graduates (Che et al., 2021). One avenue through which universities are striving to achieve this goal is by exploring innovative pedagogical approaches that break away from the traditional classroom setting. Within this context, the present study investigated the efficacy of industry crowdsourcing as a novel pedagogical method within capstone courses, specifically in the Bachelor of Science in Management program. Often considered the culmination of a student’s academic journey, capstone courses traditionally utilize textbook cases and business strategy presentations as primary teaching tools (Alstete and Beutell, 2016). However, this approach may only sometimes align with the ever-changing demands of the modern industry, thereby necessitating a pedagogical shift. Considering this, our research piloted an industry crowdsourcing method, wherein students were matched with 16-weeks real-world projects that covered a broad spectrum of the program’s core subjects, such as marketing, management, and finance, to name a few. These projects, sourced from local business associations, allowed students to apply theoretical knowledge in a practical context while addressing genuine industry challenges. To assess this new teaching method’s impact and effectiveness, we compared two sections of the capstone course – one following the conventional textbook-based approach and the other embracing the industry-crowdsourced model. The comparative analysis encompassed data from the university’s grading and course opinion survey systems, and to ensure a holistic understanding, we further incorporated data from third-party course rating platforms. Thus, this study examines the potential benefits of industry crowdsourcing in enhancing student engagement and learning outcomes and contributes to the broader discourse on innovative teaching methodologies in higher education.

The research question (RQ) was focused on whether a new industry crowdsourcing andragogy method for teaching the Bachelor of Science in Management capstone course would be effective. The goal of the current study was to collect primary data from fourth-year Bachelor of Science in Management students at an accredited university in the U.S. Two sections were to be compared. First, the traditional method of teaching the capstone course is using textbook cases and business strategy presentations. The second section featured industry crowdsourcing to match students with a 16-weeks project focused on one or more of the program’s core subjects (marketing, management, economics, finance, human resources management, organizational behaviour, consumer behaviour, statistics, operations optimization, and supply chain management). Data was to be collected from the university grading and course opinion survey systems. Data was also to be collected from third-party sites to triangulate the findings.

Literature review

Capstone courses culminate experiences integrating knowledge and skills acquired throughout an undergraduate’s academic journey. As higher education evolves, so do the methods of delivering these crucial courses. Problem-based learning (PBL) is an instructional method where students actively explore real-world problems and challenges to acquire deeper knowledge. Problem-based learning often involves students working over an extended period to research, design, refine, and present a project. This method has its roots in the constructivist theories of education, mainly drawing from the works of Dewey (1986), Piaget (1973), and Vygotsky and Cole (1978). Problem-based learning places students in the driver’s seat of their education as they work on complex problems and projects over an extended period. Problem-based learning’s theoretical foundation is grounded in constructivism, emphasizing the importance of learners constructing their understanding and knowledge through experiences. Problem-based learning is characterized by a central driving question that drives the entire project, promoting curiosity and inquiry (Kokotsaki et al., 2016). Students have significant autonomy over the direction, process, and outcome of their projects, and these projects are designed around real-world problems or challenges. Students reflect on their work and the learning process, often involving teamwork, allowing them to develop communication and collaboration skills. The projects culminate in a presentation or demonstration to a public audience.

Problem-based learning offers multiple benefits, including a deeper understanding of the problem, development of skills, increased engagement, and preparing the students for real-world challenges (Larmer et al., 2015; Salvador et al., 2023). Students get the opportunity to engage with content deeply and practically, allowing them to understand and retain knowledge more effectively. Besides knowledge, PBL helps students develop various skills, from research and problem-solving to communication and team collaboration. Students benefit from the hands-on, practical projects that are more engaging than traditional instruction methods. Problem-based learning prepares students for real-world challenges by making them problem-solvers and critical thinkers. However, traditional assessment methods may not be suitable for PBL, requiring educators to think innovatively about evaluating student work (Ertmer and Simons, 2006). Problem-based learning can be more time-consuming than traditional teaching methods, requiring resources (materials and expert input) that are only sometimes readily available. Problem-based learning is flexible and can be applied across various domains as it is not restricted to any subject or grade level. Problem-based learning has been successfully implemented across disciplines, from sciences to the arts and elementary to post-secondary education (Bell, 2010). Some tools that educators can use in the assessment in PBL include rubrics, self-assessments, peer evaluations, and reflection journals. Problem-based learning offers a transformative approach to education, placing students at the centre of the learning process. However, successful implementation of PBL requires thoughtful planning, resources, and assessment strategies that align with its unique characteristics.

Industry crowd-sourced capstone courses are a progressive and innovative approach to education, offering students an unparalleled opportunity to interact directly with industry problems, technologies, and professionals (Cullers et al., 2017; Hall et al., 2020; McKiernan, 2018). They involve industries submitting real-world challenges, often unsolved or ongoing within their operations, for students to address, ideate upon, and solve (Palacin-Silva et al., 2017; Zaugg et al., 2021). The origin of the industry crowdsourced approach can be traced to the broader concept of crowdsourcing, referring to the act of sourcing solutions or input from a large group of people, typically from an online community rather than from traditional employees or suppliers (Sindlinger, 2010; Sloane, 2011).

Industry crowdsourced capstone courses offer students real-world experience as they tackle genuine industry challenges, preparing them for their professional lives and bridging the gap between academia and industry (Palacin-Silva et al., 2017; Porras et al., 2018). Students also gain diverse problem-solving skills as they deal with real-world problems that often do not have a single solution, promoting various perspectives and innovative thinking (Wurdinger and Allison, 2017). These projects also offer the students networking opportunities through direct interaction with industry professionals (Martonosi and Williams, 2016). Students also benefit from enhanced engagement because engaging with real industry challenges can be more motivating and relevant for students than hypothetical scenarios (Burns and Chopra, 2017; Goldberg et al., 2014).

Industry crowdsourced capstone courses have some challenges as they vary in complexity, and not all are suitable for a capstone course’s duration or students’ expertise level (Palacin-Silva et al., 2017). There are also concerns related to intellectual property because as students work on real industry problems, concerns about intellectual property rights and confidentiality can arise. Also, academics will have to address the challenges in assessing solutions to real-world problems, as these can be more challenging than traditional academic problems. Industry crowdsourced capstone courses need clear guidelines regarding expectations, deliverables, and feedback mechanisms from the university and the industry. Also, promoting interdisciplinary teams can be beneficial because the best solutions often come from diverse perspectives. The success of these courses also depends on providing students with mentors from the industry so that they receive appropriate guidance and feedback.

The hypotheses derived from the review of the literature are:

H1

Students grades would be the same or higher in the industry crowd-sourced capstone course as compared to the control course.

H2

Student course opinion responses would be higher for the industry crowd-sourced capstone course.

H3

External student comments would triangulate and corroborate the above, with more positive sentiments for the industry crowd-sourced capstone course.

Methods

In this study, the investigators adopted a post-positivist epistemological stance, thereby emphasizing the necessity for empirical, quantitative data to assess their proposed hypotheses about variations in student learning across the two distinct capstone course methodologies. This post-positivist approach was slightly moderated during the application and interpretation of ML results. The literature was systematically assessed employing the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) methodology. Within the PRISMA framework, the breadth of the search progressively narrows as publications are meticulously screened for their pertinence to the research questions based on criteria such as keywords, methodological constructs, and research design (for instance, empirical studies). This screening commences with an evaluation of abstracts, culminating in a comprehensive review once a refined list of relevant articles is compiled. The search keywords included: “Industry crowdsourcing”, “Pedagogy”, “Capstone course”, “16-weeks industry projects”, “Traditional capstone course”, and “Third-party course rating systems”.

Where possible, the analysis to answer the hypotheses was performed using parametric statistical techniques. First, grades and student survey data were collected across two independent sections of the same BS management capstone course during two back-to-back terms at the same university. The first section was the traditional method, and the second was the industry crowd-sourcing andragogy. Normalcy tests were used to evaluate the grade field. T-tests were used to compare the grades and the course opinion ordinal scale values between the traditional versus the industry crowd-sourced capstone course. Next, three third-party course rating systems were accessed to download student comments concerning the industry crowd-sourced capstone course. These comments were stored in two text files: INDUSTRY and TRADITIONAL.

Machine learning was applied to analyse the student comments and then interpreted by the researcher comparing with the quantitative results of the grade and course opinion survey scores. Descriptive statistics were used to present the characteristics of the two different course section datasets, namely the mean, standard deviation (SD), median, and proportion, where applicable. Based on the RQ and hypotheses, the unit of analysis was how student perceptions of capstone course andragogy, and the actual student learning were determined by measuring the grade as well as the course opinion survey score. This was a between-group comparison, with one dependent variable (grade) and secondly with the course opinion survey response. Therefore, with only two-course sections being evaluated in the current study, this ruled out ANOVA or multivariate parametric tests.

Since the course opinion survey response scores were anonymous, there was no reliable method to link this to the student or their capstone course grade. This meant that a pair-wise parametric test could not be used (even though it would be more powerful if the context had allowed for that). An additional complexity in the design was that the number of students was different across both sections of the capstone courses (the traditional vs the crowd-sourced format: N1 = 28, N2 = 29). Thus, separate bivariate techniques would be needed for grades and the course survey score. Since normal distribution tests were being performed on grade and course opinion scores, it was hoped that the variances would be roughly equal across the grades of the course sections. However, the course opinion survey responses could often vary widely. Therefore, as a post-positivist safeguard, independent sample t-tests with assumed unequal variances were applied to grade and course opinion survey scores to test the first two hypotheses. Two-tailed t-tests were applied despite the hypothesis containing an implied direction because it was reasonably possible that the grade or course opinion survey score could decrease instead of increase or remain the same across the two course sections.

Classification types of ML would be appropriate to analyse patterns in the text comments from the third-party student rating system data since these could be linked only to the course section (traditional or crowd-sourced) but not the student. The ML conceptual workflow is illustrated in Figure 1. This shows a left-to-right progression using ML techniques incrementally, in an ad hoc fashion. According to the results obtained in one step, the next technique was selected. The results section will discuss only the critical ML techniques and their outputs. First, the data was imported from the three sources and only the third provided sufficient anonymous student comments. The text files were pre-processed to record only relevant nouns and verbs, thus eliminating punctuation and sentence fillers like “a, in, as, or, the, was.” Next, the various ML techniques were applied incrementally, according to whether the results of a previous step were significant or interesting. A classification type of ML would be appropriate to analyse patterns in the text comments from the third-party student rating system data since these could be linked only to the course section (traditional or crowd-sourced) but not the student. The ML conceptual workflow is illustrated in Figure 1. This shows a left-to-right progression using ML techniques incrementally, in an ad hoc fashion. According to the results obtained in one step, the next technique was selected. The results section will discuss only the critical ML techniques and their outputs. First, the data was imported from the three sources and only the third provided sufficient anonymous student comments. The text files were pre-processed to record only relevant nouns and verbs, thus eliminating punctuation and sentence fillers like “a, in, as, or, the, was.” Next, the various ML techniques were applied incrementally, according to whether the results of a previous step were significant or interesting.

OPEN IN VIEWER

Figure 1.

ML conceptual analysis workflow (left to right).

Sentiment Analysis (SA) was selected as an ML classification algorithm to conduct the course opinion survey response analysis. The goal of the SA would be to understand if the students thought the industry crowdsourced course design was better, the same, or worse compared to the traditional pedagogy. SA is also referred to as opinion mining, pertains to the use of computational techniques to determine and categorize opinions and sentiments expressed in text, especially to decide whether the writer’s stance towards a particular topic, product, or theme is positive, negative, or neutral. Sentiment analysis commonly applies variations of Natural Language Processing (NLP) to extract subjective information from textual sources to determine whether the sentiment is positive, negative, or neutral toward certain entities or topics of interest that were deductively identified from a literature review. Early versions of SA were accomplished using strict rule-based methods, but modern approaches to SA leverage lexicons of positive and negative words and interpretation of the context using adjacent words. This is the approach the authors applied for the current study, to utilize a modern form of SA in ML where contextual keyword processing was applied to detect emotional nuances. ML is the only technological research technique capable of this complexity for data analytics at the time of writing.

To enhance our understanding of the effectiveness of the industry crowdsourcing andragogy method for teaching the Bachelor of Science in Management capstone course, we conducted a comprehensive thematic analysis of the qualitative data obtained from student opinion surveys and third-party review sites. This analysis involved a coding process where comments were categorized into emergent themes that reflect students’ perceptions and experiences. Several key themes were identified, including real-world applicability, student engagement, collaboration experiences, and the perceived relevance of the industry projects to their future careers. Each comment was examined for underlying sentiments and perspectives, allowing us to capture the student experiences that quantitative data alone could not reveal. For instance, many students expressed that the industry-sourced projects provided a tangible connection to the business world, enhancing their engagement and motivation in the course.

In addition to thematic analysis, we employed sentiment analysis to quantify the positive, negative, and neutral sentiments expressed in the student comments. Using natural language processing tools, we analysed the frequency and context of words, which provided a sentiment score for each comment. This allowed us to assess the overall sentiment trend for each course format. The sentiment analysis results were then cross-referenced with the thematic findings to ensure an interpretation of the qualitative data. By integrating these qualitative insights with the quantitative analysis of course grades and survey scores, we presented a more nuanced and comprehensive evaluation of crowdsourcing andragogy’s effectiveness. This in-depth qualitative analysis, therefore, not only confirmed the quantitative findings but also enriched our understanding of the student learning experience, highlighting the value of industry collaboration in enhancing educational outcomes in capstone courses.

The first author was the principal investigator (PI). The PI designed the study, wrote the initial paper, conducted the analysis, interpreted the results, and served as the presenting author. The second author completed the literature review, assisted in designing the study, wrote the remaining paper, performed quality assurance, and served as the corresponding author. The PI was certified as research professional and certified for conducting research involving human subjects. The PI obtained ethical clearance to conduct the study from the internal research review board. The PI was employer-funded, and there was no specific external funding for the project.

Results and discussion

The descriptive statistics from the course sections (N1 = 28, N2 = 29) indicated no significant differences between these or the general population of BS management students at the university. A z-test of each sample’s mean age and SD compared to the program’s historical mean indicated no difference. Since the grade was being evaluated using manipulation (the andragogy of the crowd-sourced course), the grades could not be compared with the historical population. The mean student age at the end of the course was 22.3 (SD = 0.9) for the traditional and 22.5 (SD = 0.8) for the crowd-sourced version. Both course sections had approximately equal gender balances (54% female vs 52% female for the second section). These gender proportions did not differ from the historical statistics reported for the last 2 years.

As hypothesized, there was no statistically significant difference in final grade between the two sections but a significant difference in the course opinion survey scores. The mean for the traditional capstone course grade was 3.51 (SD = 0.401). In comparison, the average of the industry crowd-sourced course was 3.55 (SD = 0.319). Grades were on a 4.0 scale, the standard for colleges in the U.S. The results of the independent samples two-tailed t test comparing the tradition with the industry crowd-sourced capstone course grade indicated no significant difference (t [59] = 0.709, p < .05). Therefore, the first hypothesis (H1) was accepted: Students grades would be the same or higher in the industry crowd-sourced capstone course as compared to the control course.

Next, the course opinion survey scores were evaluated, again using independent samples and two-tailed t-tests assuming unequal variances. The result indicated there was a significant difference. The mean survey response for the traditional course was 3.49 (SD = 1.016), while the average score for the crowd-sourced course was 4.49 (SD = 0.3). The independent samples’ two-wailed t test with unequal variance results were t [59] = 0.043, p < .05, and considering the second mean was higher, we can interpret this as the students indicated they preferred the industry crowd-sourced capstone course better than the traditional andragogy method. Therefore, we can accept the second hypothesis (H2): Student course opinion responses would be higher for the industry crowd-sourced capstone course.

Classification ML techniques were applied to the student comments downloaded from the three third-party course rating systems. They were contrasted across the two capstone course sections (traditional vs industry crowd-sourced). First, the student comments were processed and tokenized to remove unimportant phrases and punctuation. Then, the frequencies were tabulated. An example of a comment from the traditional capstone was: “Cases not related to today’s jobs … cases too old” (source: authors data). In contrast, a representative comment from the industry crowd-sourced capstone was: “Industry projects allowed us to work independently from teammates but with company teams” (source: authors data).

Next, the tokenized student comments were passed through SA scoring based on the VADER algorithm. The results of that classification are shown in the heat map of Figure 2, which shows the sentiment positive, negative, neutral, and composite scores across the two courses (see bottom legend). The heat map in Figure 2 could be visually interpreted to mean more positive comments by the students taking the industry crowd-sourced course and hardly any positive comments by students taking the other. Both courses had some negative comments and a lot of neutral comments. The composite score is higher for the crowd-sourced course.

OPEN IN VIEWER

Figure 2.

SA of student comments across two courses.

Next, the three ML models (hierarchical tree analysis, linear regression, and neural networks) were applied to the data after SA. Several interim procedures were done, as shown in the conceptual workflow (Figure 1), such as scoring the tokens and the sentiments, selecting the coefficient columns to report, and building tables and diagrams of the estimates.

Overall, the sentiment in the “INDUSTRY” text file is overwhelmingly positive. The comments indicated satisfaction and appreciation for various aspects of the industry projects, the course material application, the teaching, and the overall experience. We analysed the individual student comments from the third-party site for the INDUSTRY course.

• “Interesting motivated” - Positive sentiment.

The analysis of the student comments from the third-party site for the traditional version of the course (TRADITIONAL) were as follows:

• “Hard to understand” - Negative sentiment.

Overall, the sentiment in the “TRADITIONAL” text file is predominantly negative, with several complaints and frustrations expressed about various aspects such as understanding, relevance of content, time constraints, team dynamics, and grading.

The critical statistical estimates from the ML models applied to analyse the student comments across the two courses are summarized in Table 1. The abbreviations in Table 1 refer to accuracy and effect sizes, namely Mean Sum Error (MSE), Root Mean Sum Error (RMSE), Mean Absolute Error (MAE), and coefficient of determination, which is also known as the r² effect size. MSE calculates the mean of the squares of the deviations (or errors) between the predictor and the actual value. RMSE is the square root of the mean of squares between the model’s fit compared to a baseline null model. In other words, a higher RMSE, typically over 20%, would imply that the model is less accurate than ML techniques with a lower score.

OPEN IN VIEWER

Table 1.

Effect size estimates of student comments by each ML model.

Model	MSE	RMSE	MAE	r2
ML decision tree	0.3465	0.5858	0.5888	0.0009
ML neural network	0.0035	0.0605	0.0429	0.9887
ML linear regression	0.0001	0.0011	0.0011	0.9988

The MAE is a simple yet effective index of the difference between predicted and actual values. A higher MAE of more than 20% may indicate a poor ML model. However, the managerial cutoff for RMSE and MAE depends on the context, that is, the institution’s intended application of the ML model. For example, if the ML model is developed in an exploratory nature for use in identifying strong patterns from big data where follow-up research studies will be conducted, then a higher RMSE and MAE could be accepted, ranging from 20% and higher. Also, the ML model error tolerance level depends on the life cycle of the research project, such as allowing lower accuracy for early phase research and development and increasing the accuracy expectations through further iterations of the ML model in additional studies.

Finally, the r² effect size is the benchmark used by researchers with positivist ideologies where statistical techniques are being used. Effect size r² represents the proportion of variance in the target variable that the independent features did not explain or predict. The receiver operating curve (ROC) is also sometimes used to measure the accuracy of specific ML techniques, such as support vector machines (SVM), which was not applied in the current study. The ROC represents the area captured in common between the test and actual data. The confusion metric can be calculated to count the number of correctly classified target variables using all features. Precision is the proportion of true positives within the records predicted as positive recall is the proportion of true positives in the data (correct classifications). The effect size estimates are usually more robust than the confusion matrix and ROC accuracy metrics when there is a large sample, as in the current study. Consequently, we assert that the r2 and MSE are the most reliable metrics across different studies.

The best models were linear regression, with a perfect effect size of 100% and a very low error rate of 0.1% (not surprising due to the SA being applied to only two documents). The next step was to analyse the estimates from the sentiment scores, which are shown in Table 2.

OPEN IN VIEWER

Table 2.

ML SA of student comments by course pedagogy versus linear regression.

Course	Positive	Negative	Neutral	Compound/LR
Industry crowd-sourced capstone	0.3338	0.0009	0.6658	0.9877
Traditional capstone	0.0168	0.0447	0.9379	−0.1902

It is clear from Table 2 that the compound score from linear regression shows that the industry crowd-sourced capstone course session (0.9877) had by far the most positive student comments (0.3338), while the coefficient for the traditional capstone course session was −0.1902 (lower overall sentiment effect with only 0.0168 positive comments). Interestingly, the crowd-sourced version had no negative student comments and fewer neutral comments (0.6658) as compared to the traditional instance, with 0.0447 negative words and many neutral words (0.9379). Thus, we can accept the last hypothesis (H3) that external student comments would triangulate and corroborate the above with more positive sentiments for the industry crowd-sourced capstone course.

Future research directions

While the current study focused on a Bachelor of Science degree in Management program, future research could explore the impact of industry crowdsourcing in other academic disciplines or at different educational levels. Future research studies could also target a longitudinal study to track students who participated in the industry crowdsourced courses to determine the long-term benefits, such as job placements, career advancement, and the applicability of their capstone projects in real-world settings.

Researchers can also consider conducting in-depth interviews or focus groups with students to gather qualitative data on their experiences, providing a richer understanding of the reasons behind their preferences and any challenges they faced. There is also a need for exploring other ML models or techniques, such as Support Vector Machines (SVM) or ensemble methods, to analyse student comments and provide different perspectives and insights into student sentiments.

Additionally, future researchers will need to gather feedback from industry partners or other external stakeholders who benefited from the student projects, as their perspective on the quality, relevance, and applicability of the projects would provide a holistic view of the program’s impact. Future studies can also compare the industry crowdsourced approach with other innovative teaching methods to determine which provides the best outcomes regarding student learning, engagement, and real-world applicability.

As mentioned in this study, there are several challenges in industry crowdsourcing for researchers. Hence, there is a need to investigate the challenges faced by educators and administrators in implementing the industry-crowdsourced approach, from sourcing industry projects to integrating them into the curriculum. This would provide insights for other institutions looking to adopt a similar approach. Researchers could also consider assessing the economic impact of the crowdsourced projects on the companies or industries involved, as this would help quantify the real-world value generated by these student projects.

Conclusion

The introduction of industry crowdsourcing as a novel pedagogical approach within an American Bachelor of Science degree program showcased the potential of aligning academic pursuits with real-world industry challenges. By establishing an unincorporated business consulting organization, the capstone course allowed students to immerse themselves in diverse short-term industry projects sourced from five local chambers of commerce associations. These projects spanned a wide array of sectors, from Internet marketing design and eco-tourism planning to biomedical device optimization, thereby providing a holistic exposure to real-life business scenarios.

By comparing this innovative approach with the traditional capstone course in the same semester, the study aimed to draw evidence-based comparisons. While academic performance, gauged through grades, demonstrated parity between the two courses, the student experience vastly differed. The course opinion surveys underscored a clear preference for the industry-crowdsourced approach, revealing a heightened level of student satisfaction and engagement. This sentiment was further corroborated by analysing independent third-party course rating platforms, where the crowdsourced version consistently garnered more positive feedback than the traditional textbook case-based approach.

The incorporation of industry crowdsourcing not only ensured academic rigor but also resonated with students’ aspirations for practical, real-world relevance. The study thus underscores the importance of evolving pedagogical strategies to bridge the academia-industry gap, ensuring curricula remain dynamic, relevant, and in tune with contemporary business challenges. Embracing such industry-academia collaborations could be the touchstone for training graduate students to be ready for a future career that is better matched to industry needs.