Changes in Perceptions and Experiences Following a Generative AI Learning Program

Abstract

Artificial Intelligence is changing the way people work and live. However, there is a good deal of fear and uncertainty with respect to this technology. This study investigated a learning and familiarization program regarding generative AI, specifically ChatGPT. Employees at a consultancy completed a program designed to familiarize the employees with ChatGPT and walk them through a variety of use cases. A pre- and post-training survey was used to collect data on employees’ perceptions and experiences. This study revealed that the training was effective in improving the employees’ confidence in using generative AI, their feelings of familiarity with the tool, their usage frequency, the complexity of usage, and their feelings of having been trained on the tool. In addition, the training shifted attitudes from general concerns and fears to greater understanding and more practical expectations regarding ChatGPT.

Keywords

AI implementation ChatGPT familiarization generative AI perceptions training use cases

Introduction

Artificial Intelligence (AI), defined as “systems that seek to provide the intellectual processes characteristic of people” (National Academies of Sciences, Engineering, and Medicine, 2021, p. 127), is a disruptive technology, radically changing our lives. Even people who do not deliberately choose to use AI technology are affected, as internet search engines use AI to provide responses, recommender engines on Amazon or Netflix, and advertisements on social media are all based on AI. Individuals and organizations have a wide variety of interest and willingness to engage and explore the opportunities afforded by AI. However, given the ever-increasing capabilities, many are motivated to learn by a fear of falling behind their colleagues and competitors.

This paper discusses the implementation and evaluation of a training and familiarization program at a consultancy. The employees experienced a month-long, self-directed program that included guided experiences interacting with Generative AI (GenAI), specifically ChatGPT. The effectiveness of this training in terms of employees’ attitudes and usage of GenAI was evaluated by a short survey, delivered before and after the training program. This paper discusses the training program and the impact of the training on the consultants’ attitudes and usage of GenAI following the training.

Research indicates that people and companies have both strongly positive and negative attitudes toward AI. As with new technology in general, a lack of understanding is common and can lead to both negative attitudes as well as excessive optimism (Castagno & Khalifa, 2020; Vogel et al., 2023). Fear of job displacement or expected changes in jobs cause concern and rejection of the technology (Morikawa, 2017). Concerns about ethical issues are another potential barrier (Gîngută et al., 2023; Na et al., 2022). At a company level, concerns about intellectual property are front and center. Some companies have mandated that AI will not be used for any work purposes, given the fear of losing IP and competitive advantage. Other companies are experimenting with AI and investigating ways to effectively implement the technology.

For those companies that want to encourage the use of AI, a variety of tools exist. These include providing a safe instantiation of AI, where employees can use the tool without fear of their data being sent out to the internet. Training is another option: companies indicate clear support for using AI when they train employees to work with the tools. In addition, clear communications, explicit regulations regarding AI usage (Castagno & Khalifa, 2020; Greenwald, 2017), organizational and leadership support (Castagno & Khalifa, 2020; Suseno et al., 2022), and explicit use cases for AI can all drive adoption. The explicit use case is an important factor, as it shows employees how AI can fit into their workflow and how it can be used to solve particular problems.

One particularly relevant study (Adams et al., 2024) investigated college students and their usage and perceptions regarding ChatGPT. This study found that students were interested in using ChatGPT for academic work, but they expressed concerns regarding the lack of clear guidance on how to use the tools. Students saw an opportunity to use the AI to “nurture self-regulated learning” (Adams et al., 2024, p. 13630). Students expressed concerns about relying too heavily on AI and diminishing their critical and creative thinking abilities.

The research question addressed in this work is as follows: Does a GenAI training and familiarization program affect trainee perceptions (e.g., confidence, familiarity) and use of the technology?

Approach

The approach in this work was to administer a survey prior to the GenAI training experience, provide the training experience, and administer the survey a second time upon completion of the experience. The training was offered over a four-week period. Personnel consented (opt-in) to have their data included in this analysis. Approval for this study was granted through the UNCo Institutional Review Board.

Training and Familiarization Program

The training program was implemented in a tool called Adoptify that provides a gamified learning experience with a set of exercises and reflections. For the GenAI training experience, the tasks required participants to engage with the content for approximately 30 minutes per week. Details of the program are provided in Table 1. General information about AI, potential uses, intellectual property concerns, and other material was provided via an eLearning module. Upon completion of that background information, trainees began the series of interactions with ChatGPT. The tasks were crafted to demonstrate specific use cases that employees might use in daily life (e.g., meal planning, trip planning) and in work (e.g., brainstorming and refining ideas, summarizing meeting transcripts, analyzing data in a spreadsheet). Gamification elements of Adoptify include a leaderboard where players are awarded points for completing exercises in a timely manner and receiving peer nominations.

Table 1.

Overview of the GenAI Training Program.

Lessons	Topics	Detail
AI basics	Assess perceptions and experiences GAI and intellectual property ChatGPT basics Create a GPT account Using ChatGPT	• Complete the survey. • Take an eLearning module. • Read an information page with a link to a community of practice. • Task—Follow a tutorial to create an account. • Task—Create and tailor a meal plan.
The AI paradigm shift	New ways of working The role of human expertise AI Chatbots and Google Search Apply AI to your role Use AI to explore a new role	• Consider how AI can improve your workflow, taking over certain tasks to free time for more important tasks. • Task—Create a travel itinerary to your hometown (first do it manually, then get ChatGPT to do it). • Task—Do a google search and a ChatGPT search. Engage in a conversation. Compare results and reflect. • Consider how you might use AI in your work and write out next steps you plan to take. • Task—Use a detailed prompt (provided) to explore different roles within the company and how a person might prepare for that role.
Common AI tasks	Use AI to. . . Create a client personal Write in a specific style Brainstorm a new solution Analyze a meeting transcript Analyze a spreadsheet	• Task—Use a prompt to start a conversation where ChatGPT assumes a persona. • Task—Upload text and have ChatGPT rewrite it in different voices. • Task—Provide context and prompt AI to generate ideas. Reflect on their utility. • Task—Put a meeting transcript (provided) into ChatGPT and request a summary with guided, detailed prompts. • Task—Provide data (CSV / Word) and request analyses.
Share your knowledge	Share your AI experience Assess perceptions and experiences	• Reflect and share your thoughts. • Complete the survey.

The lessons were developed to adhere to a key principle of adult learning: namely, that adults need to see how the course material is relevant to their daily lives and work (Merriam & Bierema, 2013). Further, the training included reflection points after each task, where participants were specifically asked to consider how the learnings could be incorporated into their lives and work (Chuang, 2021; Merriam & Bierema, 2013; Powell & Bodur, 2019). In addition, a Community of Practice was established around the use of generative AI, and training participants were encouraged to join (Lave & Wenger, 1991).

Interactions with AI were completed in a company instantiation of ChatGPT, so participants did not need to worry about security concerns. The program was designed so trainees began interactions with more simple tasks that they could easily complete while seeing tangible benefits. Scaffolding was provided through detailed instructions and initial simple tasks that gradually increased in complexity. The meal planning example was simple and effective in showing how ChatGPT could provide useful recommendations and how it could refine these recommendations based on subsequent user inputs. As trainees progressed through the learning experience, tasks became more complex and involved. One of the final tasks was uploading a spreadsheet and performing statistical analyses.

One key lesson included in the experience was that interactions with ChatGPT were not “one and done,” but they involved iteration and conversation to get more refined results. The exercises gave people experience with the tool, they learned nuances of interacting, and they learned a set of practical use cases. Trainees saw specific examples of how ChatGPT can be valuable in daily life and in work contexts.

Survey

Prior to beginning the Adoptify experience, participants completed a short survey, asking about their experiences and perceptions using AI. Participants were given the same survey at the end of the experience. Results were analyzed to identify changes in experiences and perceptions following the training. The survey was delivered as part of the training experience. It was offered in Microsoft Forms and included an option to participate in a study for the University of Northern Colorado (UNCo). Nearly 95% of the learning experience participants agreed to allow their data to be included in the UNCo study.

The survey included eight questions asking about experiences and familiarity with AI and six demographic questions. Among the AI questions, five were multiple choice Likert-scale type questions and three were open-ended text questions. Multiple choice questions are listed below:

How confident do you feel that you could use AI for useful outcomes? (rated 1–5, with 1 being “not at all” and 5 being “highly confident”)

How familiar are you with AI? (rated 1–5, with 1 being “not at all” and 5 being “I’m an expert”)

How frequently do you intentionally use AI? (eight options were provided, ranging from “never” to “multiple times a day”)

How have you used AI? (seven options were provided, ranging from “I have heard of AI but never used it” to “I create AI tools or chatbots”)

To what extent have you been trained on AI? (rated 1–5, with 1 being “not at all” to 5 being “highly trained”)

One open-ended question asked how participants had used AI. Two open-ended questions asked about perceived concerns and benefits of AI.

Outcomes

Quantitative Findings

The program participation was high, with approximately 80% of the employees at the consultancy at least starting the learning experience. Approximately 45% of employees completed the entire experience, including the final survey. Of the employees who agreed to have their data included in the UNCo study, 239 had completed the pre-training survey and 119 completed the post-training survey. A multivariate analysis of variance (MANOVA) was conducted to assess overall differences between pre-training survey and post-training survey groups across four outcome variables: confidence, familiarity, frequency of AI use, and AI skill level.

Responses were quantified for the rating scales. Results revealed a statistically significant multivariate effect of Test Time (Pre vs. Post) on the combined dependent variables, Wilks’ Λ = .922, F(4, 310) = 6.52, p < .001, indicating that participants’ outcomes collectively improved following the training. Further, training was used as a grouping/independent variable called Training Type. The omnibus MANOVA shows that Training Type has a strong multivariate effect on the four outcome variables (Wilks’ Λ = .722, F = 5.92, p < .001).

A one-way ANOVA, with pre- and post-training session as the dependent variable, was performed on the data using SPSS. Table 2 identifies the key quantitative findings from the study. It shows the overall averages on these performance metrics. Participants reported greater confidence and familiarity with AI following the training. They reported using AI more frequently and they reported using AI for more sophisticated types of tasks (e.g., developing tools and chatbots or teaching, versus personal use). These results indicate that the personnel feel more confident and familiar with AI and they engage in more frequent and complex usage of AI. They feel they have received training. Table 3 gives specific examples of the types of quantitative changes that were reported in the pre- and post-training surveys, listing specific types of questions that revealed differences.

Table 2.

Pre-Post Survey Results from the Gen AI Pre-Post Survey (df = 310).

Metric	Pre-test mean rating	Post-test mean rating	Difference (Post – Pre)	Significance
Confidence	3.59	3.93	+0.34	p < .0002
Familiarity	2.99	3.39	+0.40	p < .005
Deliberate usage frequency	4.69	5.79	+0.90	p < .0001
Usage types	4.10	4.39	+0.29	p < .0003
Training	1.85	2.87	+1.02	p < .001

Table 3.

Specific Quantitative Results from the GenAI Pre-Post Survey.

Metric	Relevant Item: The percentage of personnel who said . . .	Pre-test percent	Post-test percent	Difference (Post – Pre) (%)
Metric	Relevant Item: The percentage of personnel who said . . .	N = 239	N = 119	Difference (Post – Pre) (%)
Confidence	They feel “Confident” or “Very Confident”	53.5	73.9	20.4
Familiarity	They feel “Very Familiar” with AI tools	23.8	44.5	20.7
Deliberate usage frequency	They use AI “More than once a week”	52.8	75.6	22.8
Usage types	They use AI for more complex tasks: • Professional usage • Teaching others to use AI • Developing tools or chatbots	84.1 20.1 6.3	94.1 31.9 11.8	10 11.8 5.5
Training	They received at least “a fair amount” of AI training	17.1	63.9	46.8

Qualitative Findings

The survey asked participants to identify (1) the ways in which they deliberately use AI, (2) their top concerns with respect to AI and, (3) the key benefits they envision. Thematic analysis of the results were conducted and chi-squared analyses performed to determine differences. Results are shown in Figures 1 to 3.

Figure 1.

Percentage of respondents answers to the question “How do you use AI?” in the pre- and post-training survey.

Figure 2.

Percentage of respondents answers to the question “What concerns do you have about AI?”

Figure 3.

Percentage of respondents answers to the question “What benefits do you see with respect to AI?”

In the post-training survey, participants were more likely to indicate using AI for data analysis or to say many uses. A chi-squared analysis revealed differences between the pre- and post-training responses regarding uses: χ²(10) = 46.9, p < .005.

Results indicate that the participant concerns focused on accuracy of results, data privacy and security, and ethical issues. A chi-squared analysis revealed no differences between the pre- and post-training responses regarding concerns: χ²(10) = 15.2, p > .10.

With respect to benefits, the participants in the pre- and post-test indicated that productivity (e.g., time savings and increased efficiency) was the most commonly reported benefit, followed by supporting creativity through idea generation. A chi-squared analysis revealed differences between the pre- and post-training responses regarding benefits: χ²(6) = 21.7, p < .005. In the pre-test, participants indicated that better outcomes (e.g., advances in healthcare, science, and innovation) were a potential benefit. In the post-test, participants identified more personal uses for AI. These results suggest a deeper understanding of GenAI capabilities and limitations based on experiences in the training and familiarization program.

Interestingly, despite productivity being the most commonly cited benefit by the vast majority of participants (approximately 80%), few of the participants claimed to use ChatGPT specifically for productivity (see Figure 1, where approximately 5% of participants in the post-test claimed to use AI for productivity). While this might initially appear to be a disconnect, the concept of “productivity” was captured or implied in the other uses provided by participants (e.g., edit/summarize, text, brainstorm, research, data analysis).

Potential Limitations

While the findings are encouraging, a few limitations should be noted. Not all participants completed both the pre- and post-tests, introducing potential bias. Future studies should focus on improving participant retention and exploring whether completers and non-completers differ in important ways. The survey and training were offered at a single consultancy, so results might not apply across different companies and industries. Further, the survey was not framed as an evaluation of the training (e.g., no questions asked, “How effective did you find this training?”). It was developed to assess perceptions, beliefs, and interactions. However, the shift in these factors reveal findings of interest to those who want to develop or deliver training on Generative AI technology.

Conclusion

These findings consistently indicate the positive impact of the training experience. They emphasize the importance of providing employees with an opportunity to use ChatGPT and walking them through relevant use cases. The value of use cases is that they show people, specifically, how they can use an AI tool to support them in everyday tasks. The step-by-step guidance on these tasks also enabled the participants to complete the exercises, experiencing success. One interesting point to note is that participants in this study did not echo the concerns that the students identified in Adams et al. (2024). Participants in this current study did not express a fear that AI would limit their critical thinking or creativity. This might have been due, in part, to the specific use cases in the training program, none of which directly “threatened” these higher mental processes but left the human in control.

Another critical aspect is providing a culture of innovation, where people are encouraged to experiment, are given the tools needed to do that work, and are given the grace to fail. If that last component is not in place, if failure is punished, a culture of innovation cannot be said to exist. The success of AI implementation relies on seeing clearly how AI can be integrated into workflows. These use cases supported that recognition. Further, a work culture that promotes innovation and delivers consistent messaging from leadership is another key element in successful implementation of AI in the workplace.

Footnotes

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: This work was performed as part of a course at the University of Northern Colorado and no funding was provided. The lead author is a consultant at the company that participated in the research.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Adams

Chuah

Devadason

Azzis

M. S. A.

(2024). From novice to navigator: Students’ academic help-seeking behaviour, readiness, and perceived usefulness of ChatGPT in learning. Education and Information Technologies, 29(11), 13617–13634. https://doi.org/10.1007/s10639-023-12427-8

Castagno

Khalifa

(2020). Perceptions of artificial intelligence among healthcare staff: A qualitative survey study. Frontiers in Artificial Intelligence, 3, Article 578983.

Chuang

(2021). The applications of constructivist learning theory and social learning theory on adult continuous development. Performance Improvement, 60(3), 6–14. https://doi.org/10.1002/pfi.21963

Gîngută

Munteanu

V. P.

Ștefea

Noja

G. G.

(2023). Artificial intelligence and consultancy services: Perspectives of organizational and ethical concerns. In de la Iglesia

D. H.

de Paz Santana

J. F.

López Rivero

A. J.

(Eds.), New trends in disruptive technologies, tech ethics and artificial intelligence (pp. 219–232). Springer.

Greenwald

(2017). How AI is transforming the workplace; Artificial intelligence is changing the way managers do their job—from who gets hired to how they’re evaluated to who gets promoted. The Wall Street Journal. Eastern Edition. https://www.proquest.com/wire-feeds/how-ai-is-transforming-workplace-wsj-2/docview/1876656072/se-2

Lave

Wenger

(1991). Situated learning: Legitimate peripheral participation. Cambridge University Press.

Merriam

S. B.

Bierema

L. L.

(2013). Adult learning: Linking theory and practice. John Wiley & Sons, Incorporated.

Morikawa

(2017). Who are afraid of losing their jobs to artificial intelligence and robots? Evidence from a survey (Global Labor Organization Discussion Paper, No. 71). Global Labor Organization (GLO).

Heo

Han

Shin

Roh

(2022). Acceptance model of artificial intelligence (AI)-based technologies in construction firms. Buildings, 12(2), 90.

10.

National Academies of Sciences, Engineering, and Medicine. (2021). Human-AI teaming: state of the art and research needs. The National Academies Press. https://doi.org/10.17226/26355

11.

Powell

C. G.

Bodur

(2019). Teachers’ perceptions of an online professional development experience: Implications for a design and implementation framework. Teaching and Teacher Education, 77, 19–30. https://doi.org/10.1016/j.tate.2018.09.004

12.

Suseno

Chang

Hudik

Fang

E. S.

(2022). Beliefs, anxiety, and change readiness for artificial intelligence adoption among human resource managers: The moderating role of high-performance work systems. The International Journal of Human Resource Management, 33(6), 1209–2936.

13.

Vogel

Strina

Said

Schmallenbach

(2023). The evolution of artificial intelligence adoption in industry. Artificial Intelligence and Social Computing, 72, 139–150.