The effect of instruction on pragmatic routines in academic discussion

Abstract

This study investigates the effect of instruction on the acquisition of pragmatic routines used in academic discussion, specifically expressions of agreement, disagreement, and clarification. Thirty-seven learners, including an experimental group of 26 students and a control group of 11 students, participated in the study. Five intact classes received instruction across four 50-minute lessons. Input consisted of authentic language samples extracted from the MICASE corpus; instruction included noticing and production activities. Evaluation in the form of a pretest/posttest was administered through a computer-delivered oral-production test, which allowed for free production in a researcher-controlled context. The task included 30 items, including 10 agreement, 10 disagreement, and 10 clarification scenarios. Two-tailed paired-sample t-tests revealed that production of both speech acts and targeted expressions increased significantly in the experimental condition, whereas the control group did not show significant change. The results show that instruction has a significant positive effect on the use of expressions as measured by oral production in conversation simulations. We attribute this positive outcome to two components, instruction and means of evaluation.

Keywords

Corpus-based materials formulaic sequences instructional effects oral production pragmatic routines pragmatics speech acts

I Introduction

This study investigates the influence of instruction on the learning of formulaic sequences for use in academic discussion. The formulaic sequences that we targeted are pragmatic routines that convey the intended function (i.e. illocutionary force) of conversational contributions.¹ They include expressions such as That’s right, You’re right, and That’s true to indicate agreement; Yeah but, I agree… but, and I don’t think so for disagreement; What do you mean and You’re saying for other-clarification; and What I mean for self-clarification. Without such pragmatic routines, speaker intent, especially in multi-speaker discussion groups, is often difficult to interpret.

Participating in academic discussion and group work is part of the English for Academic Purposes (EAP) curriculum. Teaching academic discussion presents challenges both in authenticity of the language samples and the mode of measuring the effects of instruction. Pragmatically appropriate authentic language samples are only rarely presented in English as a second language (ESL)/English as a foreign language (EFL) textbooks (Ishihara & Cohen, 2010). To meet the challenge of authenticity we identified expressions and discourse excerpts by using a corpus of academic spoken English, and to meet the challenge of evaluation we created a conversation simulation task. The following two sections review relevant studies that explore the teaching of formulaic language and the means by which such knowledge has been measured.

II Teaching formulaic language

Approaches to teaching formulaic sequences can be found in the teaching of both vocabulary and pragmatics. Within the former, instruction addressing formulaic sequences has varied from relatively incidental to very focused. Schmitt, Dörnyei, Adolphs, and Durow (2004) targeted 20 expressions of high academic value that appeared in the normal EAP curriculum at their university. Each expression appeared at least once in the teaching materials during the two to three months between pretest and posttest, and teachers drew attention to formulas at some point in the course. In contrast, Jones and Haywood (2004) took an intensive approach, scheduling two hours per week for instruction over 10 weeks in an academic program, integrating the use of reading passages modified by adding formulaic sequences with and without input enhancement in the form of highlighting, training in holistic production, and use of sequences in writing. In the most explicit presentation, Alali and Schmitt (2012) introduced formulaic sequences via translation on Microsoft PowerPoint slides (one minute per sequence) and used a fill-in-the blank exercise sheet for written review and verbal repetition for oral review. Allowing 10 minutes per day, they presented 10 idioms a day for three days (10-minute introduction) in class sessions that included grammar instruction, and one session of written or oral review, or double grammar (in lieu of review), depending on the experimental condition.

In contrast to the teaching of formulaic sequences more generally, the teaching of pragmatic routines always includes input in context and often includes metapragmatic information. Olshtain and Cohen (1990) presented instruction on apologies in three 20-minute lessons; one lesson contrasted the expressions I’m sorry and Excuse me, another focused on intensification, and the last on semantic formulas. The instruction employed model dialogues, discussion of differences between the expressions, explicit descriptions of differences (metapragmatic information), and an intensification scale for English apologies. In addition, practice in both listening and production was included.

Takimoto (2006) presented input-based instruction contrasting structured input and structured input plus reactive explicit feedback. Instruction focused on lexical/phrasal downgraders and syntactic downgraders in English requestive forms for 40 minutes, twice weekly for two weeks. Using a longer period of instruction with a broader focus, Németh and Kormos (2001) presented five classes on lexical instruction and five classes on argumentation skills that included conversation strategies, words, and expressions that can be used in discussion, including pragmalinguistic markers expressing opinion, agreement, and disagreement.

Bardovi-Harlig and Vellenga (2012) used contextualized input and guided metapragmatic noticing during pair work. Three 50-minute periods of instruction presented conventional expressions for a range of functions once a week for three weeks; input consisted of excerpts from Friends, an American sitcom. Sociopragmatic information was delivered through directed noticing exercises designed to help learners to recognize placement, co-occurrence, and meaning. Instruction did not include opportunities for production.

Whereas most pragmatic instruction is conducted in special units within a curriculum, pragmatics instruction was an integral part of a semester-long communication course for very advanced students studied by House (1996). Two sections of the course received input that included lists of the pragmatic routines, and students were given tapes of their own role-play production with transcripts. Both groups had extensive production practice and received feedback with rules, but one group received explicit metapragmatic information and the other received none.

As this brief review shows, there is a range of approaches to the teaching of formulaic language. With the exception of Alali and Schmitt (2012), all studies presented target-language contextualized use of the expressions. Studies that favor input include Schmitt et al. (2004), Takimoto (2006), and Bardovi-Harlig and Vellenga (2012). Instruction with input and production was implemented by Olshtain and Cohen (1990), House (1996), and Jones and Haywood (2004). Metapragmatic input was supplied directly by Olshtain and Cohen (1990), as well as House (1996), and through metapragmatically guided noticing by Bardovi-Harlig and Vellenga (2012).

III Measuring knowledge of formulaic sequences

Formulaic sequences have been measured in a variety of ways, ranging from controlled written tasks to free oral and written production. Jones and Haywood (2004) used a C-test in which blanks and letters provided cues to the formula, and Schmitt et al. (2004) used a C-test that also gave a meaning gloss: “I’ve been watching the news report and they say that there’s a go___cha____that the international debts of poorer countries might be cancelled (this will probably happen)” (Schmitt et al., 2004, p. 73). Reporting that learners lost sight of the context when trying to fit expressions into a C-test, Revier (2009) developed a 3 × 3 matrix to test verb-article-noun sequences from which participants selected one word from each column (tell a joke, keep a secret, or non-idiomatically, tell a secret, keep the joke). Roever (2012) investigated learner identification of pragmatic routines using a multiple choice task in English. Alali and Schmitt (2012) used four tests to assess the learning of English formulaic sequences, two testing recall—the translation of Arabic into English (form recall) and translation of English into Arabic (meaning recall)—and two testing recognition—multiple-choice selection of English expressions for Arabic (form recognition) and multiple-choice selection of Arabic given English expressions (meaning recognition).

At the opposite end of the evaluation spectrum are production tasks that allow learners to determine their own contributions. These include essays (Jones & Haywood, 2004), oral role-plays (House, 1996), and opinion-gap activities (Németh & Kormos, 2001). Takimoto (2006) bridges the divide between free production and controlled tasks by balancing oral role-plays and written discourse completion tasks (DCTs) with listening and acceptability judgment tasks. Olshtain and Cohen (1990) used written DCTs.

Mixing controlled contexts with free production, Bardovi-Harlig and Vellenga (2012) employed a computer-delivered oral DCT in which 32 scenarios were presented aurally while students read them (Bardovi-Harlig, 2009). The task elicited conversational turns in which students either initiated the turn or responded to another speaker’s turn.

A comparison of these studies shows that learners generally score higher on the controlled written tasks than on the open-ended tasks; and although the desired outcome for pragmatics instruction is improved oral production, in free production tasks learners may not produce the targeted formulaic sequences (see Jones & Haywood (2004) for essays and Németh & Kormos (2001) for opinion gap activities). Similarly, studies of instructional effects on the production of speech acts reveal that given open-ended tasks (e.g. phone calls (Huth, 2006) and academic discussions (LoCastro, 1997)), learners may not produce the speech act that was the object of instruction, and as a result may not produce the context for the pragmatic routines. Moreover, when learners work in groups or pairs—i.e. when the task is a good representation of conversation—what one speaker says influences what another speaker says. One student’s early disagreement may eliminate the context for a later disagreement by another student. Speakers’ previous contributions influence their own subsequent contributions because speakers must balance the content and tenor of their turns. In sum, as assessment instruments, conversations provide learners with neither obligatory contexts nor identical opportunities for everyone, whereas focused written tasks ranging from C-tests to multiple-choice likely elicit explicit knowledge.

IV Research question

Previous studies show that change in production of specific speech acts in conversational contexts is particularly difficult to evaluate. This leaves open the question of whether lack of change is due to a lack of effect of instruction or difficulty in measurement. Our goals for this study were (a) to develop and test the efficacy of pragmatics instruction of one type of formulaic sequence, namely pragmatic routines, in the context of academic discussion, and (b) to measure potential change by using a controlled conversation simulation that avoids the assessment pitfalls of free conversation while retaining spontaneity, turn-taking, and the unpredictable nature of conversation. A secondary goal was to avoid the possibility of drawing on learners’ explicit knowledge when presented with written tasks. We thus addressed the following research question keeping in mind both our pedagogical and assessment goals:

Does instruction have a positive effect on the use of pragmatic routines as measured by oral production in conversation simulations?

V Materials development

As early as Williams (1988) and as recently as Ishihara and Cohen (2010), researchers have described commercially available language textbooks as impoverished sources of input for second-language pragmatics. Following the call in pragmatics teaching and research for empirically-informed instructional materials, and a request from the teachers in our ESL program for assistance with academic discussion, we undertook the development of corpus-based teaching materials and activities for this study. The materials development proceeded in three stages: the identification of authentic expressions related to agreements, disagreements, and clarifications in academic discourse; the identification of examples from a corpus; and the development of teaching materials and activities. We selected the Michigan Corpus of Academic Spoken English (MICASE) (Simpson, Briggs, Ovens, & Swales, 2002) as the source of authentic language use for our materials development because it documents the use of academic spoken English on a major Midwest campus similar to the one where our study took place. MICASE includes 1.8 million words of transcribed speech from almost 200 hours of recordings from the University of Michigan, including lectures, classroom discussions, lab sessions, seminars, and advising sessions.

1 Identifying authentic expressions and contexts in academic discourse

a The expressions

We examined the chapters on academic discussion and conversation in supplemental textbooks used in the program at the time of the study (Porter & Grant, 1998; Skillman, & McMahill, 1996). We extracted all the expressions suggested by the textbooks as relevant to agreement, disagreement, and clarification and compared them with naturally occurring expressions found in MICASE.²

We searched MICASE for each textbook expression and recorded the number of times an expression appeared in the corpus and the number of distinct speech events in which it occurred. Our target frequency was 40 occurrences per million words—a frequency that Biber, Conrad, and Cortes (2004) identify as frequent for lexical bundles—but we also considered a second threshold of 10 occurrences per million words as used in earlier work by Biber, Johansson, Leech, Conrad, and Finegan (1999). Many of the expressions presented in textbooks are much longer than the expressions that recur in the academic spoken corpus, so we reduced longer expressions from the textbooks to shorter ones that recurred in the corpus. For example, Porter and Grant (1998) listed You’re right about that as an agreement expression, but only one instance of the full expression was found in 1.8 million words and You’re right about only occurred twice.³ However, You’re right occurred 139 times in 50 different transcripts at a frequency of 75 occurrences/million words. You’re right thus met the frequency requirement (Biber et al., 2004). In addition to the 11 expressions gleaned from the textbooks, six additional expressions were selected from the corpus based on previous research on academic discussion (Bardovi-Harlig & Salsbury, 2004) (Table 1).⁴ The right-hand column in the table gives the source of each expression.

Table 1.

Formulaic sequences identified for instruction with frequency counts from MICASE.

Formulaic sequence	Function	Raw number	Number of texts	Frequency (per million)	Source
Yeah, but	Disagreement	233	78	120+	Bardovi-Harlig & Salsbury (2004)
What I mean	Clarification	195	65	100+	Skillman & McMahill (1996)
Okay, but	Disagreement	170	64	90+	Corpus
You’re saying	Clarification	165	59	90+	Skillman & McMahill (1996)
That’s right	Agreement	167	63	90+	Skillman & McMahill (1996), Porter & Grant (1998)
You’re right	Agreement	139	50	75+	Porter & Grant (1998)
That’s true	Agreement	118	57	65+	Skillman & McMahill (1996), Porter & Grant (1998)
I agree ^*	Agree/Disagree	69	29	35+	Skillman & McMahill (1996), Porter & Grant (1998)
Do you mean	Clarification	67	33	35+	Corpus
I have a question	Clarification/ questioning	65	46	35+	Porter & Grant (1998)
What you’re saying	Clarification	63	36	35	Skillman & McMahill (1996)
I don’t think so	Disagreement	56	33	30+	Porter & Grant (1998)
Good point	Agreement	34	24	18+	Corpus
Your point	Agree, Disagree, Clarification	33	14	10+	Corpus
I agree with ^*	Agreement	27	16	10+	Skillman & McMahill (1996), Porter & Grant (1998)
In other words	Clarification (self)	23	13	10+	Skillman & McMahill (1996), Porter & Grant (1998)
I agree but ^*	Disagreement	18	14	10	Corpus

The counts for I agree include I agree with, and I agree but, which are also reported separately. Because MICASE is not set up to search for discontinuous sequences (as in I agree that there’s, definitely a long way to go, but I definitely think there’s been a lot of improvement; MICASE identifier STP095SU139), the 69 occurrences of I agree were checked manually for the presence of but in the same turn.

Seventeen expressions met or exceeded the frequency thresholds and were identified for instruction: six agreement expressions, four disagreement expressions, two self-clarifications, and five requests for clarifications. The expressions with their frequency counts are listed in descending order of frequency in Table 1. Seven expressions occurred between 65–120 times per million words, another five occurred above 30 per million words, and the final five occurred above 10 times per million words.

We also examined the discourse function (illocutionary force) of each expression. Whereas some expressions, such as I don’t think so uniquely signal a disagreement, ostensible agreements such as I agree often occur as disagreements in the sequence, I agree… but. As the corpus search showed, all agreement expressions could be used as the first part of a disagreement marker in accordance with the American English preference for the strategy agree-before-disagree (Pomerantz, 1984). The two most common disagreement expressions were Yeah but and Okay but, neither of which occurred in the textbooks but are frequent in American English, appearing in MICASE 120+ times/million words and 90+ words/million words, respectively. I agree but exemplifies the agree-before-disagree strategy; the corpus reveals that in 17 of the 18 occurrences I agree was separated from but by 1–19 words (mean length, 9.29 words) as illustrated by I agree that there’s, definitely a long way to go, but I definitely think there’s been a lot of improvement (MICASE identifier STP095SU139).

b Identifying conversational excerpts for input

After we identified the expressions, we selected examples that occurred across speaker turns and included an unambiguous use of the expression and a clear referent. To enhance the salience of the corpus examples, we deleted non-sequential turns from multi-party conversations, substituted common words for technical words to increase comprehensibility (e.g. ‘cover’ for ‘capsid’ in a biology transcript), and reduced repetition or repair (cf. Ishihara & Cohen, 2010). However, turns containing the target expression were not modified, but rather appeared in our teaching materials as they do in the corpus. Examples (1) to (4) provide a sample of the input for agreement, disagreement, other-, and self-clarifications. Target expressions are underlined here, but were not necessarily emphasized in the student materials.

(1) Agreement

A: Good, thanks. Okay, the last thing that I was considering was a translation task, and I can see that as being something very messy.

B: Yeah, I can too.

A: So…

B: Um, cuz, well, let’s see… with German and Spanish, you’re not gonna have character problems. Chinese and Japanese you are gonna have, character problems.

A: Oh that’s true. [Transcript ID: OFC355SU094]

(2) Disagreement

A: I saw this being a poem very much between, the narrator and the eyes of the poor, and not the narrator and this woman. I just saw that Baudelaire needed a situation for the narrator to be in so he could have this experience. But I didn’t really see it as a central part of the text.

B: So, on the one hand I agree that, it’s the relationship between him and these people he sees that brings out the conflict in him, but you know the woman there is used to be kind of this other half of him. [Transcript ID: SEM545MG083]

(3) Request for clarification

A: What do you mean you write backwards?

B: Yeah, do you mean that you write vertically instead of horizontally?

A: If I’m going to add up some numbers. Where do I start when I add?

B: You compute backwards. Okay. [Transcript ID: LES165JG121]

(4) Self-clarification

A: I think that the Beasties, what’s, what surprises me about them is uh that everything they’ve done is so good. I mean not across the board, obviously, but what I mean is that uh, that I think that there’s a, um, that a lot of their, success and attention, has been pretty well deserved. [Transcript ID: OFC105SU068]

VI Method

In order to investigate the effect of instruction on the use of pragmatic routines we implemented a pretest-intervention-posttest design with experimental and control groups.⁵ The study was carried out in intact EAP classes.

1 Participants

A total of 37 students participated in either the experimental (instructed) or the control (repeated-test) condition. Four teachers participated in the instructed condition.

a The experimental group

Five intact communications classes participated in the experimental group. Four classes were sections of Level 5 Communication in the Intensive English Program and one was a section of Academic Discussion, an elective English-language support course for matriculated students at the university. Two sections were taught by one teacher, and the remaining three sections were taught by three other teachers.

Students can be described as low-advanced learners. The students in the Intensive English Program (IEP) are in the fifth level of a seven-level program. Level placement was determined by scores on a 3.5-hour, four-part institutional placement exam used by the IEP with reading, writing, grammar, and listening components. The range for the institutional Test of English as a Foreign Language (TOEFL) at Level 5 is 420–450.

Although intact classes received instruction, only students who completed both the pretest and posttest and attended at least three of the four hours of instruction were included in the study. Thirty-eight students attended the pretest. Thirty-one students completed both the pretest and posttest, but only 26 attended three of the four hours of instruction. Twenty students (77%) attended all four hours of instruction and the remaining six students (23%) attended three out of four hours. The 26 students represented six language backgrounds (Arabic, 8; Chinese, 10; Japanese, 1; Korean, 3; Spanish, 3; Portuguese, 1); 12 students were female and 14 male.

b The control group

Two Level 5 Communication classes were recruited in the subsequent session for the control group. Sixteen students attended the pretest, but only 13 attended both the pretest and posttest. Two of those students did not successfully record their responses to the pretest and were thus excluded, leaving 11 students in the control group. The students represented seven language backgrounds (Arabic, 4; Chinese, 2, and 1 each of Japanese, Kazakh, Korean, Portuguese, and Spanish); 10 students were male and one was female.

The control group was included to gauge the influence of taking the test twice. The students completed the pretest and the posttest at approximately the same spacing as the experimental groups (14–18 days). One class was delayed by a severe thunderstorm, which prevented it from coming to the language lab, and it was rescheduled for the next class meeting following the weekend. There was no significant difference in the performance of the experimental and control groups at the time of the pretest (see Results).

2 Instructors

In order to avoid researcher bias, the four teachers already assigned to the communication classes at the time of the study delivered the instruction to the experimental group. The teachers were given the lesson plans and all materials for the four hours of instruction. A research associate met with the teachers to discuss the units. Teachers were not informed of the specific research goals of the study. Two of the IEP teachers had 10 years of ESL/EFL teaching experience. The third teacher reported two years of ESL experience. The Academic Discussion instructor had 30 years of ESL/EFL experience.

3 Instrument (pretest/posttest)

The pre- and posttest was designed to simulate academic conversation in group work while providing comparability across speakers. Prior to the production of the recorded test, 46 items were piloted in written format to ensure that students recognized the intention conveyed by the scenario as evidenced by their production of agreements, disagreements, and clarifications. Items that were unclear or that elicited speech acts other than those targeted were eliminated. Items that showed a high production of targeted speech acts were included in the oral dialogue completion task; this is a type of oral DCT that has been used with increasing frequency as a measure of oral production in pragmatics (Bardovi-Harlig, 2009; Nickels, 2006; Schauer, 2007).

The resulting computer-delivered oral-production test includes two examples, two practice items, and 30 test items randomly arranged in two sections of 15 items each (see Appendix). The two sections were counterbalanced in Versions A and B. The 30 items included 10 agreement, 10 disagreement, and 10 clarification scenarios. Each item started with a brief description of the topic (e.g. your group is discussing transportation, newspapers, or learning English) and then for agreements and disagreements gave learners a specific opinion. Students saw the descriptions and their position on the screen as shown in Examples (5) and (6). After students heard and read the setting and their position, they heard a classmate’s turn to which they responded. The narrator and the classmate alternated between a male and female voice so that learners could easily distinguish the classmate’s turn from the narrator’s.

(5) Agreement item

Narrator (visual and audio): Your group is discussing the way that people communicate. You have the same opinion as your classmate.

Classmate’s turn (audio only): People spend too much time talking on the phone these days.

[Screen only] You say:

(6) Disagreement item

Narrator (visual and audio): Your group is talking about the news and media. You think that newspapers like The New York Times and The London Times are still very important.

Classmate’s turn (audio only): Nobody reads newspapers these days.

[Screen only] You say:

Agreement and disagreement items were further divided into items that explicitly stated the respondent’s position relative to their classmate’s (5) and items that stated the content of the respondent’s position (6). There were five of each type for both agreements and disagreements. The words ‘agree’ and ‘disagree’ were not used in the prompts to avoid priming expressions using those words.

Clarifications were divided between other-clarifications and self-clarifications. Other-clarification items prompted students to check their understanding of their classmate’s turn (7). Self-clarification items gave students a sentence that they needed to restate, showing them a screen that said “people look confused,” a signal that the students should clarify their contributions (8).

(7) Other-clarification item

Narrator (visual and audio): Your group is talking about motivating other people. Your classmate uses a proverb.

Check your understanding of the saying.

Classmate’s turn (audio only): You can lead a horse to water, but you cannot make it drink.

[Screen only] You say:

(8) Self-clarification item

Narrator (visual and audio):You and your group are discussing the best way to learn math. You say, “Daily practice with problems is important for learning math.”

[Screen only] People look confused

[New screen] You say:

4 Instruction

Four 50-minute lessons were developed. Instructional targets were distributed across the lessons (Table 2). The first lesson introduced four agreement expressions, That’s right, You’re right, Good point, and That’s true. The second lesson covered two more agreement expressions I agree and I agree with, and introduced the disagreement expressions Yeah but, Okay but, and I agree but.⁶ The third lesson introduced self-clarification expressions What I mean and In other words and requests for other-clarification Do you mean, I have a question, and Your point. The fourth lesson continued with requests for clarification You’re saying and What you’re saying. Each lesson recycled previous expressions and provided scaffolded opportunities for controlled practice of the expressions introduced in the current and previous lessons.

Table 2.

Lesson outline.

Lesson	Focus	Expressions
One	Agreement	That’s right, You’re right, Good point, That’s true
Two	Agreement	I agree, I agree with
	Disagreement	Yeah but, Okay but, I agree but
Three	Self- and other-clarification	What I mean, In other words, Do you mean, I have a question, Your point
Four	Other-clarification	You’re saying, What you’re saying

Each lesson included three primary elements: noticing of expressions in context, explicit metapragmatic information concerning use, and opportunities for production. Each lesson began with a warm-up activity, followed by multiple focused-noticing activities, and ended with production activities that gave the students the opportunity to use the expressions to which they had been exposed up to that point. The allotted time for an activity was usually given as a range on the lesson plans (5–7 minutes, for example). Keeping in mind that some activities involved more than one aspect, the instructional time suggested on the lesson plans was 107–137 minutes for noticing, 42–48 minutes for metapragmatics, and 51–61minutes for production activities for a total of 200 minutes of instruction.

Input was delivered through both reading and listening tasks, and included at least three noticing events for each expression, which may have been augmented by interactions in pair- and group-work and class discussion. Focused-noticing activities included circling and underlining expressions to enhance noticing of discontinuous elements (notably I agree… but), filling in tables to emphasize asymmetries, and writing expressions down. Listening activities were accompanied by Microsoft PowerPoint slides that provided visual context. For example, an image of a molecule was displayed while learners listened to an academic discussion about chemistry. When each activity was completed, the instructor reviewed the answers with the entire group.

After the noticing activities, the instructor provided explicit metapragmatic information about the expressions. For instance, following a noticing activity in which learners identified disagreement expressions in a transcript, the instructor explained that the expressions used for disagreements were the same as those used for agreements, with the addition of but. Then, the learners calculated the distance between the agreement expression and the word but in the transcript. In another activity, after listening to dialogues and circling self-clarification expressions, the learners were told the expression What I mean could be followed by either is or by and that by was used with a re-statement of the word or phrase being clarified.

Finally, the learners engaged in production activities, featuring a variety of games and interactive activities. For example, learners played a game to practice agreement expressions. In groups of three, one learner read a statement, another agreed with the statement, and the third judged whether the learner had used a target expression to agree. The students received statement cards and play money for the judge to ‘pay’ students who accurately supplied a target expression. Roles alternated to give students an opportunity to agree and judge. In another activity, learners practiced both agreements and disagreements in pairs by looking at ambiguous pictures where the photography (close-ups and odd angles) made it difficult to identify the image. One learner offered a suggestion as to what the picture depicted, and the other drew a card indicating whether he should agree or disagree with his partner, and then did so using one of the target expressions. In the final lesson, learners used all four types of expressions that they had been exposed to over the course of the treatment by playing a board game in which they rolled dice and moved pieces around a game board. Squares on the board contained statements and, in order to advance, learners produced speech acts indicated on cards they drew in relation to the statements on the squares on which they landed. The production activities gave each student in a group an equal number of turns, but the specific speech acts they produced depended on the random order of the cards in the deck and the roll of the dice.

Following the procedures used by Bardovi-Harlig and Vellenga (2012), instructors were provided with a detailed lesson plan for each lesson with explicit instructions as well as an accompanying student handout. To ensure the fidelity of the lessons across instructors and sections, instructors also completed a checklist to record each activity as it was accomplished.

At the end of each lesson, the instructors collected the handouts from the students to reduce the likelihood that some students would study the expressions, creating unequal exposure across the group. Instructors took attendance and noted anything they thought relevant to the study.

5 Procedure

The four classes that met daily (Level 5 Communication) took the pretest in the first week. Lessons 1 and 2 were taught in the second week, allowing a day in between for activities that had not been covered the preceding day, and Lessons 3 and 4 were taught in the third week.⁷ The posttest was given the day after the instruction ended. With pretest and posttest, the treatment spanned 14 days in total.

One class had double sessions (110-minute classes) and met twice a week (Academic Discussion). The students took the pretest the first meeting of the first week, then received Lessons 1 and 2 in the second meeting. Lessons 3 and 4 were given on the first class of the second week. The teacher reported completing one activity from Lesson 2 during the following class meeting. The posttest was given the second class of the second week. With pretest and posttest, the treatment spanned nine days in total. The more intensive delivery in Academic Discussion was planned to better fit the established schedule for that class.

The classes met in a language computer lab for the pretest and posttest. The task was computer-delivered, and students listened through individual headsets. They simultaneously heard and read each scenario on the screen, and then heard a turn (without written support) immediately after the scenario. Next they saw a screen that showed only “You say,” and then they provided an oral response. Learners were given 10 seconds to respond. Responses were recorded through headset microphones onto digital files. Versions A and B were loaded onto alternating computers. No learner sat next to someone completing the same version of the task.

6 Analysis

The task yielded 2220 responses, which were transcribed and coded. The authors and an assistant each transcribed one quarter of the data, and checked another quarter. The responses were then coded by the first and second author in two cycles: the first cycle coded speech acts, the second coded for targeted expressions. Speech acts form the context for the expressions; if a disagreement is not performed, a disagreement expression will not be used. If a student produced the targeted speech act (i.e. an agreement after an agreement prompt), the response received a score of 1 point, if it did not, it earned no points. There were 10 points possible for agreements and disagreements and 5 points for each of self- and other-clarifications. The percentage of items receiving 1 point for a targeted speech act was calculated for each student for agreements, disagreements, and self- and other-clarifications, separately for pretest and posttest (Table 3). Inter-rater reliability for speech act identification was 92%.

Table 3.

Production of speech acts before and after instruction by experimental and control groups.

Speech acts	k	Experimental group (n = 26)							Control group (n = 11)
		Pretest		Posttest		Paired-samples t-test			Pretest		Posttest		Paired-samples t-test
		M	SD	M	SD	t	p	d ^*	M	SD	M	SD	t	p	d ^*
Agreements	10	.677	.180	.862	.142	5.91	.001	1.19	.627	.347	.691	.311	.921	.379	0.28
Disagreements	10	.681	.240	.858	.133	3.54	.002	0.73	.582	.322	.636	.242	.711	.493	0.22
Other-clarification	5	.415	.271	.831	.202	8.19	.001	1.65	.455	.391	.545	.336	1.17	.271	0.36
Self-clarification	5	.362	.330	.585	.362	3.38	.002	0.67	.364	.398	.382	.275	.209	.839	0.07

d = Cohen’s d (effect size).

A second coding examined the targeted speech acts for the targeted expressions for agreements, disagreements, and self- and other-clarifications. Uses of I don’t think so were also coded because it is a high-frequency expression in MICASE (Table 1) and the only disagreement expression from the corpus that the students used on the pretest. Only the first expression in a response was coded for the purposes of this assessment of instructional effectiveness; a response could only score a maximum of 1. Thus, the response “Yes, that’s true. I agree with that” (S15, I-3) was scored only as an example of That’s true. Only well-formed expressions, such as That’s true, earned one point; interlanguage forms such as that true earned no points, but were coded as attempts and will be discussed qualitatively. There were 10 points possible for expressions that occurred in the context of agreements and disagreements and 5 points for each of self- and other-clarifications. The percentage of items receiving 1 point for a targeted expression were calculated for each student for agreements, disagreements, and self- and other-clarifications, separately for pretest and posttest (Table 4).

Table 4.

Use of pragmatic routines before and after instruction by experimental and control groups.

Pragmatic routines	k	Experimental group (n = 26)							Control group (n = 11)
		Pretest		Posttest		Paired-samples t-test			Pretest		Posttest		Paired-samples t-test
		M	SD	M	SD	t	p	d ^*	M	SD	M	SD	t	p	d ^*
Agreements	10	.131	.159	.500	.291	6.70	.001	1.42	.227	.190	.273	.195	0.58	.578	−0.18
Disagreements	10	.212	.188	.412	.261	3.36	.002	0.67	.282	.199	.264	.180	0.00	.831	−0.07
Other-clarification	5	.046	.130	.308	.345	4.00	.001	0.92	.073	.135	.073	.185	−0.22	1.000	0.00
Self-clarification	5	.092	.228	.294	.367	2.96	.007	0.62	.000	.000	.036	.081	1.49	.167	none

d = Cohen’s d (effect size).

VII Results

Both the number of appropriate speech acts and the number of targeted expressions significantly increased from pretest to posttest in the experimental group. The control group showed no significant gains. The overall pretest performance of the experimental and control groups was within 4 points of each other. Paired t-tests showed that there was no significant difference at the pretest between the experimental and control groups in any category.

1 Speech acts

Paired t-tests were used to compare the percentage of speech acts produced between pretest and posttest for each of agreements, disagreements, other-, and self-clarifications. The changes in percentage of production for speech acts from pretest to posttest were normally distributed as appropriate for the paired t-test. The results of the two-tailed paired-samples t-tests showed that the production of speech acts increased significantly in the experimental condition, p < .002 (Table 3). Applying a Bonferroni correction to the alpha level .05 to control for the four tests sets the alpha level at p < .0125, which is met by all speech acts in the experimental group. The effect size (measured by Cohen’s d) is large for agreements and other-clarifications and medium for disagreements and self-clarification. The number of agreements in agreement contexts increased from 67.7% at the pretest to 86.2% at the posttest, disagreements from 68.1% to 85.8%, other-clarifications from 41.5% to 83.1% and self-clarifications from 36.2% to 58.5%.⁸ We attribute these increases to instruction since the control group did not show significant improvement when only taking the test twice.

2 Targeted expressions

Paired t-tests were used to compare the percentage of targeted expressions used between pretest and posttest for each of agreements, disagreements, other-, and self-clarifications. The change scores from pretest to posttest were normally distributed as appropriate for the paired t-test. The results of the two-tailed paired-samples t-tests showed that the production of pragmatic routines used to perform each speech act type increased significantly in the experimental condition, p < .007 (Table 4). Applying a Bonferroni correction to the alpha level .05 to control for the four tests sets the alpha level at p < .0125, which is met by all speech acts in the experimental group. The effect size (measured by Cohen’s d) is large for expressions appearing in agreements and other-clarifications and medium for disagreements and self-clarification. The use of expressions for agreements increased from 13.1% at the pretest to 50% at the posttest (That’s right), 21.2% to 41.2% for disagreements (I agree with you… but…), 4.6% to 30.1% for other-clarification (Do you mean…?), and 9.2% to 29.2% for self-clarifications (What I mean is…). The control group did not show a significant change (Table 4).

Learners used some expressions more than others, despite equally balanced input for the expressions (Table 5).⁹ For agreements, I agree with was the preferred expression before instruction, with I agree a distant second. In the posttest, learners increased their use of I agree with about 2.5 times. They also added That’s right and That’s true and increased their use of You’re right and I agree (Table 5).

Table 5.

Expressions used before and after instruction by experimental group.

Speech acts
Agreements			Disagreements			Clarification-other			Clarification-self
Expression	Pretest	Posttest	Expression	Pretest	Posttest	Expression	Pretest	Posttest	Expression	Pretest	Posttest
You’re right	4	14	Yeah, but	0	28	Do you mean	2	5	What I mean	1	16
That’s right	0	23	Okay, but	0	9	What do you mean	2	24	I mean	11	16
That’s true	1	20	I don’t think so	55	52	You’re saying	1	7	What I’m saying	0	6
I agree	7	11	I agree but	0	10	Your point	1	3	In other words	0	0
I agree with	22	59	Creative agreement + but	0	8	I have a question	0	0
Good point	0	3				What you’re saying	0	1
Total	34	130		55	107		6	40		12	38

Prior to instruction, 100% (55) disagreement expressions were I don’t think so. After instruction, I don’t think so represented less than half of the 107 disagreement expressions produced, 55 of which comprised the four targeted expressions that had not been used prior to instruction. A direct No was traded for Okay but by Learner T5 in (9).

9. Pretest: “No, the newspaper in these days is: isn’t like that”

Posttest: “Okay, but I think (nn) that reading newspapers is very important and that we should [2 syll].”

Another learner (T1), disagreeing with the interlocutor’s claim that fast food is healthy, replaced “in my opinion” with “yes but” hedged by “sometimes” in the posttest (10).

10. Pretest: “In my opinion fast food is not healthy enough hmm ’cause they, maybe all the food is fried food”

Posttest: “Sometimes yes, but most of the time fast food is not healthy enough”

Agree-before-disagree expressions (e.g. Yeah but; Pomerantz (1984), Bardovi-Harlig & Salsbury (2004)) were absent in pretest disagreements, but were used at the posttest. Learners also produced discontinuous agree-before-disagree expressions found in “I agree with you but” (T1, I-7), “Yes, I agree with that but” (S19, II-11) and “Yeah, studying grammar is very important but I think is more important to speak to other people in the language” (S4, I-7). They also used a variety of agreement expressions followed by but to create a disagreement including “Yeah, you are right, but I think New York is more fashional than Bloomington” (S8, I-9) and “You made a good point but I think communication with others is best way to improve English” (S3, I-7).

Other-clarification showed the lowest percentage of expressions per turn of any of the categories in the pretest. Nevertheless, the number of expressions used went from 6 to 40, more than six times more. The greatest increase was found in What do you mean? with two uses in the pretest and 24 in the posttest. For example, Learner S13 modified his comprehension check of a classmate’s use of “banned” (11).

11. Pretest: “Banned it means uh not allowed”

Posttest: “What do you mean by banned? Is it like forbididded?”

A parallel increase is found in self-clarification with the use of What I mean, which was used only once in the pretest, but 16 times after instruction as illustrated by Learner S11’s explanation of her original statement “I’m not saying that I won’t do it, just that other people should contribute” in (12).

12. Pretest: “I’m not sure what they are doing, but other people needs to help others”

Posttest “Well, what I mean by that is you need to contribute because we are a group so you need to do the same work as me”.

I mean was the favorite self-clarification expression at the pretest and increased by five additional uses after instruction.

3 Attempts at expressions

Similar to findings in earlier studies (Bardovi-Harlig, 2009; Bardovi-Harlig & Vellenga, 2012; Wildner-Bassett, 1994; Yorio, 1989), interlanguage forms of expressions were attested. Students produced 19 interlanguage forms of expressions used for agreement and other-clarification at the posttest (disagreements and self-clarifications were all well-formed). Agreements exhibited 130 well-formed expressions (Table 5) and 14 attempts and other-clarifications exhibited 40 well-formed expressions and five attempts. Agreement expressions That’s true and I agree (with) were realized as That true and I’m agree (with) seven times each. The clarification expression What do you mean was realized as What you mean five times.

VIII Discussion

The results show that instruction had a significant positive effect on the use of expressions as measured by oral production in conversation simulations. We attribute this positive outcome to two components, teaching and evaluation, and we discuss each in turn.

1 Teaching

The difference in the pretest and posttest scores shows the instruction of expressions to be highly successful with medium to large effect sizes. In addition, the number of speech acts also increased significantly, also with medium to large effect sizes. The increase in relevant speech acts is important not only because speech acts constitute the context for the instructed expressions, but also because improvement in the production of the speech acts reveals an increase in pragmatic performance with or without the targeted expressions. Although speech acts were not the direct target of instruction, the combination of input and practice lead to clearer contributions by learners in simulations of academic discussions.

Learners used new expressions that they had not used before instruction and they expanded their use of expressions already in play. Use of I agree with increased by 37 tokens (59 up from 22) and Yeah but, What do you mean, and That’s right increased by 20 uses from zero (for Yeah, but and That’s right) to two uses (for What do you mean). Thus, instruction not only introduced learners to new expressions but helped them consolidate their knowledge of expressions in their repertoire. Due to the unfortunate omission of I don’t think so from the planned input, we are unable to observe the effect of instruction on that pragmatic routine. However, it is interesting to note that in the absence of examples in the input, I don’t think so lost ground, dropping from 100% of all pragmatics routines produced with disagreements to only 49% as new expressions were added.

The instructed and control group students started at the same level, but those who received no instruction changed by fewer than 5 percentage points in their use of pragmatic routines. In contrast, the instructed learners showed group improvement scores of 20–35 percentage points depending on the speech act. This result, coupled with the general low scores before instruction, suggest that Roever’s (2012) claim that learners in ESL settings can be expected to get some pragmatic routines for free may be overly optimistic.

Both the pragmatic expressions and the materials developed for the input component of this study were excerpted from MICASE, and although there is no independent measure of the efficacy of authentic compared to invented materials, we responded to the call for use of authentic materials in pragmatics more generally. Given that textbooks overestimate the length and complexity of expressions, and that our expressions verified against MICASE are shorter, our use of the shorter, frequently used expressions may have enhanced learnability and contributed to the success of instruction.

2 Evaluation

The conversation simulation balanced two concerns for evaluation of pragmatics, oral production and equal opportunity for speakers. The task asked students to provide time-pressured responses. This reduced the likelihood of their drawing on explicit knowledge compared to performance of written pragmatic production tasks, and it greatly reduced the amount of explicit knowledge likely to be encouraged by the written means used for evaluation reviewed earlier, which include multiple choice questions, C-tests, glossed c-tests, and matrices. The conversation simulation also provided all of the learners with identical opportunities to agree and disagree with classmates as well as to ask for clarification and offer clarification of their own turns. Learners had sufficient freedom to determine their own responses, thus simulating conversation, consistent with the goals of pragmatics while encouraging learners to focus on communication, a situation that we believe promotes answering by feel rather than explicit knowledge.

The combination of oral production and control in our conversation simulation allows the benefits of instruction to show through. It neither over-estimates learners’ knowledge by providing multiple-choice items or crossword puzzle-style hints, nor under-estimates learners’ knowledge by using production activities in which learners could fail to take a turn (or not be afforded a turn by other speakers) that would allow them to demonstrate their knowledge. Moreover, the conversation simulation gives every learner multiple opportunities to produce the target speech act. Innovations in teaching pragmatics, whether speech acts or pragmatic routines and expressions, require means of evaluation in which students can demonstrate their progress and researchers can track it.

The pretest scores suggest that self- and other-clarifications may be harder for learners to perform without instruction other than agreements and disagreements. The posttest improvement suggests that clarifications are not harder to learn, however. Nevertheless, review and revision of the clarification items may be warranted. It was a challenge to create contexts in which the task could realistically tell a learner that he did not understand what a classmate had said. In the case of asking for an interpretation of You can lead a horse to water, but you can’t make it drink, so many learners did not understand the proverb that it might have added an additional layer of difficulty to the item. Similarly, telling learners that other people did not understand them using the prompt “people look confused” could be integrated with other prompts with the same intention. Whatever refinement awaits these items, the task nevertheless showed that students improved in the realization of both speech acts and expressions.

3 Future directions

The increase in speech acts was accompanied by an increase in the number of targeted pragmatic routines. The increase in disagreements also saw an increase in the use of the more direct, but dispreferred pragmatic routine I(’m) disagree which increased from 14 uses in the pretest to 33 in the posttest. Although we did not anticipate this increase prior to the study, we would now recommend that future instruction provide pre-emptive negative evidence. Consistent with the instruction used here, we suggest a lesson that includes an online search of MICASE to emphasize the low frequency of I disagree compared to the other disagreement expressions in the lessons.

One of our goals in designing the instructional activities for this study was to include oral production activities. Bardovi-Harlig and Vellenga (2012) speculated that the lack of production opportunities may have led to lower gain scores for orally-assessed conventional expressions. Oral production during instruction may be particularly important for studies (and classrooms) that use oral assessment. Direct evaluation of the contribution of the oral production activities is possible by adding production activities to a replication of Bardovi-Harlig and Vellenga’s instruction on conventional expressions or removing it from the present study. Given the field’s interest in improving instruction, it seems that enriching the former study would be the preferable route.

This study addressed a variety of desiderata in the teaching of pragmatics, from using authentic language as input to using spoken production for evaluation of instructional effects. We hope that studies like this will help move pragmatics instruction from the domain of research studies to second/foreign language classrooms.

Footnotes

Appendix: Conversation simulation

The full task with audio may be retrieved from IRIS http://www.iris-database.org/iris/app/home/index

Acknowledgements

We would like to thank the instructors who administered the lessons, Roosevelt Faulkner for his help with transcription, teacher support, and data collection, Stephanie Dickinson of the Indiana Statistical Consulting Center, Indiana University, and the Center for Language Technology and Instructional Enrichment (CeLTIE), Indiana University, and their staff for providing technical expertise in recording and formatting the computer tasks and assistance during the elicitation sessions. Audio-editing was done by Natasha Branch (CeLTIE).

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Notes

References

Alali

F.A.

Schmitt

(2012). Teaching formulaic sequences: The same as or different from teaching single words? TESOL Journal, 3, 153–180.

Bardovi-Harlig

(2009). Conventional expressions as a pragmalinguistic resource: Recognition and production of conventional expressions in L2 pragmatics. Language Learning, 59, 755–795.

Bardovi-Harlig

Salsbury

(2004). The organization of turns in the disagreements of L2 learners: A longitudinal perspective. In Boxer

Cohen

A.D.

(Eds.) Studying speaking to inform second language learning (pp. 199–227). Clevedon, UK: Multilingual Matters.

Bardovi-Harlig

Vellenga

H.E.

(2012). The effect of instruction on conventional expressions in L2 pragmatics. System, 40, 1–13.

Biber

Conrad

Cortes

(2004). If you look at …: Lexical bundles in university teaching and textbooks. Applied Linguistics, 25, 371–405.

Biber

Johansson

Leech

Conrad

Finegan

(1999). Longman grammar of spoken and written English. London, UK: Longman.

House

(1996). Developing pragmatic fluency in English as a foreign language: Routines and metapragmatic awareness. Studies in Second Language Acquisition, 18, 225–252.

Huth

(2006). Negotiating structure and culture: L2 learners’ realization of L2 compliment-response sequences in talk-in-interaction. Journal of Pragmatics, 38, 2025–2050.

Ishihara

Cohen

A.D.

(2010). Teaching and learning pragmatics: Where language and culture meet. London, UK: Longman.

10.

Jones

Haywood

(2004). Facilitating the acquisition of formulaic sequences: An exploratory study in an EAP context. In Schmitt

(Ed.), Formulaic sequences: Acquisition, processing and use (pp. 269–300). Amsterdam, The Netherlands: John Benjamins.

11.

LoCastro

(1997). Pedagogical intervention and pragmatic competence development. Applied Language Learning, 8, 75–109.

12.

Németh

Kormos

(2001). Pragmatic aspects of task performance: The case of argumentation. Language Teaching Research, 5, 213–240.

13.

Nickels

E.L.

(2006). Interlanguage pragmatics and the effects of setting. In Bardovi-Harlig

Félix-Brasdefer

Omar

(Eds.), Pragmatics and language learning (Vol. 11, pp. 253–280). Honolulu: National Foreign Language Resource Center, University of Hawaii.

14.

Olshtain

Cohen

A.D.

(1990). The learning of complex speech act behaviour. TESL Canada Journal/Revue TESL du Canada, 7, 46–65.

15.

Pomerantz

(1984). Agreeing and disagreeing with assessments: Some features of preferred/dispreferred turn shapes. In Atkinson

J.M.

Heritage

(Eds.), Structures of social action: Studies in Conversation Analysis (pp. 57–101). Cambridge, UK: Cambridge University Press.

16.

Porter

P.A.

Grant

(1998). Communicating effectively in English: Oral communication for non-native speakers. (2nd ed.). Boston, MA: Heinle & Heinle.

17.

Revier

R.L.

(2009). Evaluating a new test of whole English collocations. In Barfield

Gyllstad

(Eds.), Researching collocations in another language: Multiple interpretations (pp. 125–138). London, UK: Palgrave, Macmillan.

18.

Roever

(2012). What learners get for free: Learning of routine formulae in ESL and EFL environments. ELT Journal, 66, 10–21.

19.

Schauer

G.A.

(2007). Finding the right words in the study abroad context: The development of German learners’ use of external modifiers in English. Intercultural Pragmatics, 4, 193–220.

20.

Schmitt

Dörnyei

Adolphs

Durow

(2004). Knowledge and acquisition of formulaic sequences: A longitudinal study. In Schmitt

(Ed.), Formulaic sequences: Acquisition, processing and use (pp. 55–86). Amsterdam, The Netherlands: John Benjamins.

21.

Simpson

R.C.

Briggs

S.L.

Ovens

Swales

J.M.

(2002). The Michigan corpus of academic spoken English. Ann Arbor, MI: The Regents of the University of Michigan. Retrieved from quod.lib.umich.edu/m/micase/

22.

Skillman

McMahill

(1996). Springboard to success: Communication strategies for the classroom and beyond. Upper Saddle River, NJ: Prentice Hall Regents.

23.

Takimoto

(2006). The effects of explicit feedback on the development of pragmatic proficiency. Language Teaching Research, 10, 393–417.

24.

Wildner-Bassett

M.E.

(1994). Intercultural pragmatics and proficiency: ‘Polite’ noises for cultural appropriateness. IRAL, 32, 3–17.

25.

Williams

(1988). Language taught for meetings and language used for meetings: Is there anything in common? Applied Linguistics, 9, 45–58.

26.

Yorio

(1989). Idiomaticity as an indicator of second language proficiency. In Hyltenstam

Obler

L. K.

(Eds.), Bilingualism across the lifespan (pp. 55–72). Cambridge, UK: Cambridge University Press.