Comparing Paired-Stimulus Preference Assessments With In-the-Moment Reinforcer Analysis on Skill Acquisition: A Preliminary Investigation

Abstract

Today, the use of formal preference assessments, including paired-stimulus preference assessments, is widely utilized to help determine which items to use as reinforcers during intervention. A second way to determine potential reinforcers is to analyze multiple dimensions of a stimulus in the moment, a procedure known as in-the-moment reinforcer analysis. Although paired-stimulus preference assessments are widely used, there is no experimental evidence that extensive advance preference assessments actually produce higher rates of learning than in-the-moment reinforcer analysis. The present study compared rates of learning on a simple expressive labeling task when correct responses were reinforced with items selected based on extensive formal paired-preference assessments versus items selected by a teacher using in-the-moment analysis of reinforcer effects. The results indicated no clear difference in skill acquisition, but there were clear differences in terms of efficiency and maintenance.

Keywords

autism paired-preference assessment preference reinforcement

One hallmark of applied behavior analysis (ABA) is the provision of positive reinforcement to increase desired behaviors. Positive reinforcement has been defined as the presentation of a stimulus causing an increase in frequency of targeted behavior in the future (Cooper, Heron, & Heward, 2007). Researchers have shown that a wide variety of stimuli can be utilized to increase desired behavior or decrease undesired behavior, including food (e.g., Schreibman, 1975), toys (e.g., Leaf, Sheldon, & Sherman, 2010), praise (e.g., Schreibman, 1975), tokens (e.g., Ayllon & Azrin, 1965), and engagement in stereotypic behaviors (e.g., Rincover & Newsom, 1985). Researchers have utilized reinforcement to increase a variety of behaviors, including social skills (e.g., Leaf, Dotson, Oppenheim, Sheldon, & Sherman, 2010), language (e.g., Petursdottir, Carr, Lechago, & Almason, 2008), and academic and pre-academic tasks (e.g., Leaf, Sheldon, & Sherman, 2010). Although reinforcement is widely used with individuals diagnosed with autism spectrum disorder (ASD), it is often difficult for clinicians, parents, and teachers to identify potential reinforcers that can be utilized to increase adaptive behaviors for them.

This difficulty has led to the use of formal preference assessments to assist in identifying potential reinforcers. Formal preference assessments are procedures utilized by clinicians to identify which stimuli are preferred or nonpreferred by the learner, with the presumption that the preferred stimuli are more likely to function as reinforcers during teaching. There are several types of formal preference assessments that are utilized in clinical practice, including interviews (e.g., Piazza, Fisher, Hagopian, Bowman, & Toole, 1996), single stimulus approaches (e.g., Green et al., 1988), paired-stimulus preference assessments (e.g., Fisher et al., 1992), multiple stimulus without replacement (e.g., Restar & Noell, 2008), and multiple stimulus with replacement (e.g., DeLeon & Iwata, 1996). Researchers have demonstrated a strong correlation between an item being identified as highly preferred and the effectiveness of that stimulus as a reinforcer (e.g., Carr, Nicolson, & Higbee, 2000).

Paired-stimulus preference assessments (Fisher et al., 1992) are commonly implemented in clinical settings providing intervention for students diagnosed with ASD (Graff & Karsten, 2012). Based on the learner’s selections, the teacher creates a hierarchy from the learner’s most preferred item to least preferred item. The most preferred items are then used as reinforcers during teaching opportunities. The first study to evaluate the paired-stimulus preference assessment was conducted by Fisher and colleagues (1992). They compared the use of single stimulus preference assessment to a paired-stimulus preference assessment for students diagnosed with physical, intellectual, and developmental disabilities, using a concurrent operant paradigm. Results indicated that, although both procedures were accurate in identifying highly preferred stimuli, the paired-stimulus preference assessment was able to detect greater differentiation among items. In 1994, Windsor, Piche, and Locke compared a group presentation method, staff ranking method, and a paired-preference method to identify preferences among food and drink stimuli for eight adults with developmental disabilities. Results of the study indicated that the group presentation method and paired-preference assessment yielded similar results. However, the results indicated that staff rankings did not match the paired-preference assessment in differentiating between the different stimuli.

In 2012, Graff and Karsten surveyed 406 professionals, including Board Certified Behavior Analysts (BCBAs), on their use of a wide variety of formal preference assessment procedures. Fifty-two percent of all respondents reported using at least one direct formal preference assessment. Furthermore, 36% of all respondents reported previously using a paired-preference assessment. Finally, 45% of BCBAs and/or Board Certified assistant Behavior Analysts (BCaBAs) reported using a preference assessment on at least a monthly basis; 3% reported using formal preference assessments several times a day, 18% reported using formal preference assessments at least once a month, and 24% reported using formal preference assessments somewhere in between. Although the survey had its limitations in terms of representative assessment and self-reporting, it is safe to assume that many certified behavior analysts utilize formal preference assessments as common clinical practice.

The use of formal preference assessments is the “gold” standard in identifying potential reinforcers. Another procedure that can be utilized to determine reinforcement value of various stimuli is in-the-moment reinforcer analysis. In-the-moment reinforcer analysis requires a teacher to analyze current circumstances and several other factors to identify which stimulus is likely to have the highest reinforcement value at any given moment, rather than making decisions based on prior formal assessments. In-the-moment reinforcer analysis takes into account (a) learner affect (e.g., body language), (b) learner interaction with the item, (c) how often the item was utilized during previous sessions or trials, (d) ongoing improvement in student responding following access to a specific reinforcer, (e) likelihood of untested items having reinforcing properties based on having similar properties to known reinforcers, and (f) teacher’s ability to condition alternative items as reinforcers, as described by Leaf et al. (2012).

Leaf and colleagues (2015) were the first to evaluate the use of in-the-moment reinforcer analysis; they compared a formal paired-stimulus preference assessment teaching condition to an in-the-moment reinforcer analysis teaching condition for three individuals diagnosed with ASD. In this study, participants had 4 min to sort poker chips by color. Two consequence conditions were compared for their effect on the rate of responding: (a) contingent delivery of reinforcing stimuli identified through paired-stimulus preference assessment and (b) contingent delivery of reinforcing stimuli chosen via in-the-moment reinforcer analysis. Results showed no significant difference in rate of sorting across the three conditions. However, in-the-moment reinforcer analysis was far more efficient across all three participants. The researchers concluded that, due to the long duration required to complete paired-preference assessments, the use of formal paired-preference assessments might not be justified in the selection of potential reinforcers for motivating individuals diagnosed with ASD to perform simple tasks. The researchers did state that future research was still needed.

The purpose of the present preliminary study was to evaluate the utility of a formal paired-stimulus preference assessment as compared with in-the-moment reinforcer analysis for teaching two individuals diagnosed with ASD expressive language tasks. The target skill of acquiring new vocabulary used in the present study is potentially more demanding than the chip sorting task used in the previous study by Leaf et al. (2015) and, therefore, differential reinforcement effects might be more evident. Outcome measures used in the present study included rate of skill acquisition as measured by probes, maintenance of targets acquired during teaching, efficiency of teaching, and performance during acquisition. In addition, we evaluated which reinforcers were utilized by the teachers in the in-the-moment analysis condition.

Method

Participants

Wayne, a 4-year-old boy with autistic disorder, had a Wechsler Preschool and Primary Scale of Intelligence–IV (WPPSI-IV) full scale IQ score of 86, a Vineland Adaptive Behavior Scales (VABS) composite score of 83, a Gilliam Autism Rating Scale–2 (GARS-2) Autism Quotient of 100 (probability of autism: very likely), a Peabody Picture Vocabulary Test (PPVT-IV) standard score of 92 (32nd percentile), and a Social Skills Improvement System–Parent Version (SSIS-P) standard score of 92. At the time of the study, Wayne was attending a general education preschool classroom. Wayne was conversational and displayed beginning play skills (e.g., imaginative play, parallel play). Wayne also displayed several stereotypic behaviors including twirling items, lining up items, stacking items, and making repetitive statements. Prior to the study, Wayne had received 7 months of intensive early behavioral intervention with a focus on decreasing stereotypic behaviors and increasing compliance, learning-how-to-learn skills, social behaviors, range of interests, and general knowledge.

Terry, a 5-year-old boy diagnosed with Autistic Disorder, had a WPPSI-IV full scale IQ score of 101, a VABS composite score of 84, GARS-2 Autism Quotient of 76 (probability of autism: possible), a PPVT-IV standard score 73 (2nd percentile), a SSIS-P standard score of 66, and an Autism Diagnostic Observation Schedule (ADOS) total score of 17 (consistent with autistic disorder). At the time of the study, Terry was attending a general education preschool classroom. Terry could speak in three- to four-word sentences and displayed beginning play skills (e.g., parallel play, cooperative play, and game play). Prior to the study, Terry had received 16 months of intensive early behavioral intervention with a focus on increasing language, social behaviors, cognitive processing, and general knowledge.

Teachers

Three teachers were utilized in this study, all who were employed for at least 1 year by a private agency that provides intensive behavioral treatment for children with ASD. Each teacher had a similar history with each of the participants. Each participant had a total of two teachers; one teacher was randomly assigned to the preference assessment condition and one teacher was randomly assigned to the in-the-moment reinforcer analysis condition. All teachers were trained in the principles of ABA (e.g., how to identify reinforcement) and ASD. Initial training for all teachers lasted approximately 2 months (40 hr per week) prior to them working independently with children, after which they received ongoing supervision and training. Each teacher received ongoing monthly supervision from a licensed psychologist with more than 40 years of experience in the field of ABA and ASD. All teachers utilized in this study had over 5 years of experience working in the field of ABA and ASD. Two teachers had a master’s degree in education and social work, and one had a bachelor’s degree in psychology.

Teachers were kept blind from each other’s findings; the teachers were not allowed to discuss the results of any sessions or the reinforcers utilized within each condition. Therefore, the teachers assigned to the in-the-moment reinforcer analysis did not know the results of the paired-stimulus preference assessment. Teachers were responsible for running the assessment and assessment condition sessions they were randomly assigned to. Due to scheduling logistics, only three teachers could be utilized. Therefore, one teacher served as the teacher in the paired-stimulus preference assessment condition for one student and the in-the-moment reinforcer analysis condition for the other student. As each participant had different results in the formal paired-stimulus preference assessments, carryover for that teacher serving in two different conditions was not anticipated. In addition, having more than one teacher in the clinical condition adds strength to the generality of the procedure.

Setting

Research sessions took place in a small research room located in the clinic where the children received their usual intervention services. The research room contained three desks, chairs, a video camera, couch, and various instructional materials.

Pre-Baseline

Reinforcer identification

To determine potential reinforcers to be utilized in the paired-stimulus preference assessment, the researcher (first author) emailed each member of the participant’s clinical team (i.e., teachers and program supervisors) requesting a list of all items (e.g., light toys, bouncy balls, figurines) or social activities (e.g., red light-green light game, tickles, high fives) that the participant enjoyed. The researcher asked that each member respond via email and not discuss the list with other members of the team. Once the researcher received each member’s list, the researcher created a list including all items identified by members of the clinical team.

The researcher emailed each member of the participant’s clinical team the list of potential reinforcers and asked each person to numerically rank the list from the participant’s presumed most preferred item to least preferred item. The researcher asked that members of the clinical team not discuss their rankings with each other. Once the researcher received all of the rankings, he averaged the rankings across all members of the clinical team. The top seven presumed preferred items from this ranking were used in the paired-stimulus preference assessment and in the in-the-moment reinforcer analysis. The researchers added an additional three items that were not identified by the clinical team, ensuring that not all of the items were the most preferred toys for each participant.

Paired-stimulus preference assessments

Prior to baseline, two paired-stimulus preference assessments, across 2 days, were conducted. This was done to ensure that preferences remained consistent over time. The three items identified as most preferred across the two preference assessments were used during teaching during the paired-stimulus preference assessment condition.

Prior to starting the first paired-stimulus preference assessment, the participants were provided with the opportunity to play with each item for approximately 30 s. Next, the teacher conducted the paired-stimulus preference assessment. The paired-stimulus preference assessment consisted of the teacher holding out two items (one in each hand) and asking the participant to select an item to play. After physically selecting the item, the participant was provided 30-s access to it. The teacher socially engaged with the participant and the item. This continued until all items were compared against each other item one time.

The second day of the paired-stimulus preference assessment was identical to the first day with the exception that the participant did not have 30-s access to each item before the assessment. The three items selected most frequently (i.e., toy cars, spinner, and stickers) across the 2 days were used as the reinforcers during the paired-stimulus preference assessment and all 10 items were used during the in-the-moment reinforcer analysis condition. Participants only had access to these items during research sessions, to minimize satiation effects.

Skills Taught

Wayne was taught to expressively label the names of 12 pictures displaying comic book characters (e.g., Wolverine, Captain America, Aqua Man, etc.). The selection of these skills was made based on Wayne’s supervisor’s recommendation that Wayne needed to increase his pop culture knowledge so that he could better interact with his peers. Terry was taught to state associations of things that go together (e.g., Q: What goes with a castle? A: A King); in total, Terry was taught 12 different targets. The selection of these skills was based on Terry’s supervisor’s recommendation that he needed to increase his ability to make associations to prepare for school. Targets for both participants were taught in sets (three sets for paired-stimulus preference assessment condition and three sets for in-the-moment reinforcer analysis condition). Each set consisted of two targets that were randomly assigned to either of the two conditions and were then randomly assigned to one of the three sets.

General Procedure

Research sessions were conducted 3 to 5 days per week and lasted approximately 20 min. Sessions were conducted once per day. The research study consisted of three conditions: baseline, intervention, and maintenance. Due to the experimental design, some skills could be in intervention while other skills could be in either maintenance and/or baseline. Each research session started with probe trials to determine either baseline levels, if the participant was acquiring the targeted skill (during intervention condition), or maintenance levels. Probe trials were conducted in three sets (i.e., Set 1, Set 2, and Set 3) with each set consisting of 12 trials: six trials for the targets assigned to the paired-preference stimulus assessment condition and six trials for the targets assigned to the in-the-moment reinforcer analysis condition. During the first session of intervention, only intervention trials were conducted. On all other sessions, there was a 1- to 2-min break between probe trials and the two teaching conditions. The order of the teaching sessions was randomly determined prior to the research session. Once participants reached mastery on all their targets, they received full probes for three research sessions (maintenance).

Probe Trials

Each targeted skill received three probe trials each for a total of 12 trials per set. A probe trial began with the teacher providing an instruction to the participant (e.g., “What is his name?” or “What goes with a restaurant?”). The participant was allowed 5 s to respond. If the participant responded correctly or incorrectly, the teacher provided neutral feedback (e.g., “Thanks”); if the participant did not respond within 5 s of the instruction, the teacher informed the participant that he needed to respond (e.g., “You need to try.”). No reinforcement, prompting, or priming occurred during full probe sessions.

Teaching Conditions

Paired-stimulus preference assessment condition

Each paired-stimulus preference assessment condition session consisted of 10 total teaching trials. The teachers who were randomly assigned to this condition ran every session. The three top items selected in the paired-stimulus preference assessment were evenly dispersed across the 10 teaching trials. In this condition, the teacher provided only items that were identified as highly preferred (utilizing an item that was selected 67% to 100% of the time, dependent upon the item) as a reinforcer in the paired-stimulus preference assessment. The order of delivery of the reinforcers was predetermined prior to each session. Only one item was provided to the participant per trial.

A teaching trial began with the teacher holding up a picture (e.g., a picture of Captain America or a picture of a castle) and providing a discriminative stimulus (e.g., “What is his name?” or “What goes with a castle?”). The participant was given 5 s to respond to the discriminative stimulus. If the participant responded correctly, the teacher praised him (e.g., “Good Job”) and provided 10-s access to the predetermined reinforcing stimulus. The teacher also interacted with the participant while he was accessing the tangible reinforcer. If the participant responded incorrectly, or did not respond within 5 s of the instruction, the researcher provided negative feedback and information about the correct response, for example, “That’s not it; his name is Captain America” or “Nope; a king goes with a castle.” This continued until the teacher had implemented all 10 teaching trials.

In-the-moment reinforcer analysis condition

Each in-the-moment reinforcer analysis session consisted of 10 teaching trials. The teachers who were randomly assigned to this condition ran every session. The same 10 toys that were evaluated in the paired-stimulus preference assessment were available for the teacher to select as a reinforcer, but the teacher had complete discretion over which item to use on each occasion of reinforcement. The teacher could select the same reinforcer during every teaching trial or the teacher could select a different reinforcer during every teaching trial. As the teacher was not privy to the results of the paired-stimulus preference assessment, in this condition, the teacher could select the item which was identified as highly preferred by the student in the paired-stimulus preference assessment (e.g., an item selected during every opportunity) or the teacher could select an item that was identified as nonpreferred by the student during the paired-stimulus preference assessment (e.g., an item selected in less than 20% of opportunities).

The teacher made his or her analysis based on several factors. It is important to note that these were guidelines, as the approach was flexible and the assessment could change moment to moment. First, the teacher would evaluate participant affect. If the participant was smiling, laughing, or appeared “happy” then the teacher would select the stimulus more frequently across the 10 trials. If the participant sighed, frowned, or appeared “bored” then the teacher would select the stimulus less frequently across the 10 trials. Second, if the participant made statements that they were enjoying the item or having fun then the teacher would select the stimulus more frequently across the 10 trials. Conversely, if the participant said that they did not enjoy the item or were not having fun then the teacher would not select the stimulus across the 10 trials. Third, the teacher would evaluate how often the stimulus was selected within the session and across recent sessions (e.g., looking at previous sessions data sheets). If the teacher evaluated that the item was selected in more than 70% of trials across the previous three sessions or more than 70% of trials within a session, they would select a different stimulus to utilize as a reinforcer. This was done to avoid satiation and to address deprivation of the stimulus. Fourth, the teachers attempted to condition the stimulus similar to the procedure conducted by Leaf et al. (2012). If the participant started playing with the stimulus in a manner similar to how they played with other preferred items, the teacher would continue to select that stimulus more frequently across the 10 trials. If the participant did not play with the stimulus in novel ways or in a manner similar to how he played with other toys, that item was not selected across the 10 trials. Teaching trials in the in-the-moment reinforcer analysis were identical to teaching trials in the preference assessment condition, with the exception of the range of reinforcers provided.

Dependent Variables and Measures

There were five dependent variables evaluated in this study. The primary dependent variable was the number of targets that reached mastery criterion. Mastery criterion was set as the participant responding 100% correctly during probe trials (i.e., responding correctly on all six probe trials) for three consecutive sessions. The second dependent variable was the participants’ ability to maintain the skills taught following intervention. During probe trials, two types of responses could be recorded. Correct responding was defined as the participant verbally stating the name of the picture presented to him (e.g., stating “Captain America” when shown a picture of Captain America) or verbally stating an appropriate association that went with the picture shown (e.g., stating “a king” when shown a picture of a castle). Incorrect responding was defined as the participant verbally stating an incorrect name when presented with a picture (e.g., stating “Wolverine” when shown a picture of Captain America), verbally stating an incorrect association (e.g., stating “a barn” when shown a picture of a castle), or not responding to the instruction.

The third dependent variable was an efficiency measure. We measured efficiency in three ways. First, we evaluated total amount of teaching time it took participants to reach mastery criterion on skills taught during each of the two teaching conditions. For the paired-stimulus preference assessment condition, total time included both the time participants spent in teaching sessions and the time participants spent in the two paired-preference assessments. For the in-the-moment reinforcer analysis condition, total time included only time the participant spent in the teaching sessions, since no extra time was necessary for reinforcement assessments. Second, we measured total number of teaching trials required for participants to reach mastery criterion for skills taught with each of the two conditions. Third, we measured total number of teaching sessions required for the participants to reach mastery criterion for skills taught with each of the two conditions.

The fourth dependent variable was participant responding during both of the teaching conditions. For this measure, we assessed the percentage of correct trials and the percentage of incorrect trials across all teaching sessions. Correct and incorrect responding during the teaching conditions had an identical definition to correct and incorrect responding during probe trials. The final measure evaluated was the stimulus selected by the teacher during the in-the-moment reinforcer analysis condition. The researchers evaluated the percentage of times the teacher selected each of the 10 stimuli across all in-the-moment reinforcer analysis sessions.

Design

An adapted alternating treatments design was used to evaluate the effects of the two teaching conditions. We hypothesized that participants would acquire skills rapidly and that there might not be differences across the two teaching conditions. This would minimize the functional control; therefore, we combined the adapted alternating treatment design with a multiple probe design across behavior sets. The addition of the multiple probe design shows that skills were only acquired when and only when intervention was implemented, demonstrating stronger experimental control.

Interobserver Agreement

The teacher running the session scored the participant’s responses during every session. A second observer (first author) simultaneously and independently recorded participant responses during 38.2% of probe sessions, 50% of paired-stimulus preference assessment sessions, and 52.1% of in-the-moment reinforcer analysis sessions. Interobserver agreement was calculated by totaling the number of agreements divided by the number of agreements plus disagreements and multiplying by 100%. Percentage agreement was 100% across all probe sessions and teaching conditions.

Procedural Fidelity

To assess procedural fidelity, an independent observer (e.g., research assistant) recorded the researcher’s behaviors during 50% of preference assessment sessions and during 52.1% of in-the-moment reinforcer analysis sessions.

Correct instructor behaviors for the preference assessment session consisted of the teacher (a) holding up the correct card, (b) providing the correct discriminative stimulus, (c) allowing 5 s for the participant to respond, (d) providing a reinforcer for correct responding only, (e) only selecting one of the three reinforcing items as determined by the paired-stimulus preference assessment conducted prior to the intervention condition, (f) selecting the reinforcing items three or four times throughout the entire session, and (g) providing negative feedback and informative feedback following errors. Procedural fidelity was 100% for the preference assessment conditions.

Correct instructor behaviors for the in-the-moment reinforcer analysis session consisted of the teacher (a) holding up the correct card; (b) providing the correct discriminative stimulus; (c) allowing 5 s for the participant to respond; (d) providing a reinforcer for correct responding only; (e) selecting the reinforcing stimulus based on either the participant’s affect (e.g., smiles or frown), verbal statements (e.g., enjoyment or disapproval), and/or not selecting the same item more than 70% of trials; (f) conditioning the stimuli as preferred items similar to procedures identified by Leaf and colleagues (2012); and (g) providing negative feedback and informative feedback following errors. Procedural fidelity was 100% for the in-the-moment reinforcer analysis condition.

Results

Skill Acquisition and Maintenance

Figures 1 and 2 display the skill acquisition results and maintenance results for Wayne and Terry, respectively. Wayne reached mastery criterion for all six sets of skills taught. Across all three sets, Wayne reached mastery criterion for skills taught during the in-the-moment reinforcer analysis condition in fewer sessions than for skills taught during the paired-stimulus preference assessment condition. During the assessment of maintenance, Wayne’s mean correct responding across all three sets of skills was 97.2% (range = 83.3%-100%) and 98.6% (range = 83.3%-100%) for skills taught during the preference assessment condition and the in-the-moment reinforcer analysis condition, respectively.

Figure 1.

Wayne’s results.

Figure 2.

Terry’s results.

Terry reached mastery criterion for all six sets of skills taught. Across all three sets, Terry reached mastery criterion for skills taught during the in-the-moment analysis in fewer sessions than for skills taught during the paired-stimulus preference assessment condition. During the assessment of maintenance, Terry’s mean correct responding across all three sets of skills was 86.1% (range = 0%-100% across sessions) and 88.2% (range = 0%-100% across sessions) for skills taught during the preference assessment condition and the in-the-moment reinforcer analysis condition, respectively.

Efficiency

Across both participants, the in-the-moment reinforcer analysis condition was more efficient than the preference assessment condition in terms of total time, total number of sessions, and total number of trials (see Table 1).

Table 1.

Efficiency Data.

Participant	Condition	Total assessment time	Total teaching time	Total time	Total number of trials	Total number of sessions
Wayne	In-the-moment reinforcer analysis	0 min	64 min 41 s	64 min 41 s	100	10
Wayne	Stimulus Preference Assessment Condition	89 min 49 s	57 min 43 s	147 m 32 s	150	15
Terry	In-the-moment reinforcer analysis	0:00	57 m 54 s	57 m 54 s	130	13
Terry	Stimulus Preference Assessment Condition	81 min 11 s	74 m 7 s	155 m 18 s	170	17

Responding During Teaching

Results for responding during teaching were mixed across the two participants. Although there was an advantage for the in-the-moment reinforcer analysis condition for Wayne’s targets, Terry responded with a nearly equivalent level of accuracy across both conditions. Wayne’s percent correct responding during teaching trials was 91% for the in-the-moment reinforcer analysis condition and 79.3% during the paired-stimulus preference assessment condition. Terry responded correctly on 89.3% of teaching trials for the in-the-moment reinforcer analysis condition and 92.3% of trials during the paired-stimulus preference assessment condition.

Distribution of Stimulus Selection

Figure 3 represents participants’ selection of stimuli during the paired-stimulus preference assessment (represented by bar graph) and the therapist selection of stimuli during the-in-the moment reinforcer analysis (represented by line graph). For Wayne, the two items that were delivered most frequently during the in-the-moment reinforcer analysis condition were two big toy cars (20.8%) and a toy space ship (15.5%). For Wayne, overall, the teacher in the in-the-moment reinforcer analysis condition selected items that were not used in the preference assessment condition more frequently than items that were used in the preference assessment condition. For Terry, the two items that were delivered most frequently during the in-the-moment reinforcer analysis condition were the Buzz Lightyear spinning toy (26.4%) and a bubble wand (22.9%); neither of these items were identified as the top three toys via the paired-preference assessment. For Terry, overall, the teacher in the in-the-moment reinforcer analysis condition selected items that were not used in the preference assessment condition more frequently than items that were used in the preference assessment condition.

Figure 3.

Percentage of participant selection of stimuli during paired-stimulus preference assessment (bar graph) and therapist selection of stimuli during in-the-moment reinforcement analysis (line graph).

Discussion

The purpose of this study was to determine whether there would be a differential rate of skill acquisition for two young children diagnosed with ASD when a teacher used the top three toys identified by a formal paired-stimulus preference assessment versus a teacher using in-the-moment reinforcer analysis to select among the same 10 items. Overall, the results were similar across the two teaching conditions. Although participants were able to acquire skills under both conditions, the in-the-moment reinforcer analysis was considerably more efficient across the two participants. One participant (Wayne) responded correctly on more teaching trials during the in-the-moment reinforcer analysis condition, while the other participant (Terry) responded nearly equivalently across both conditions. Therefore, the results showed that there were no real differences in terms of skill acquisition, maintenance, or responding during teaching when a teacher utilized the results from a formal paired-stimulus preference assessment or when a teacher selected stimuli at his or her discretion. Even though the results are similar in terms of skill acquisition, maintenance, and responding during teaching, the results for this study indicated for the two participants that conducting paired-stimulus preference assessments provided no real advantage. The paired-stimulus preference assessments, however, carried the disadvantage of increasing the time and effort required for participants to reach mastery on skills.

The results of this study have several practical clinical implications for those working with individuals diagnosed with ASD. First, formal preference assessments have long been considered the “gold” standard in behavioral intervention. They have been promoted as the most accurate technique to identify stimuli likely to have the highest reinforcing value and, thus, likely to be most effective when used during teaching. This study, however, demonstrated that children with ASD may be able to learn skills just as effectively without spending time on formal preference assessments, as long as instructors are guided by relevant in-the-moment information.

One of the two participants did show a modestly faster rate of learning with the paired-stimulus preference assessment condition, requiring under 58 min to master all targets, versus 64 min to master all targets in the in-the-moment reinforcer analysis condition. However, the savings of 6 min in teaching time was vastly overshadowed by the additional 89 min to complete the paired-stimulus preference assessment. When teaching individuals with learning difficulties, any incremental gain in speed of acquisition through formal preference assessments must be weighed against the reduced teaching time that remains after completing the assessments, which is a point that seems to be too often overlooked. The average amount of time to complete a single paired-preference assessment in this study was approximately 41 min. This amount of time becomes a more substantial number as the frequency of formal preference assessments increases (e.g., daily or even several times a day).

According to Graff and Karsten (2012), 3.8% of all BCBAs or BCaBAs reported that they implement formal preference assessments on a daily basis. Based on our average amount of time spent conducting a paired-preference assessment (41 min), if these assessments were conducted on a daily basis (260 days), this would result in approximately 10,660 min per year of time spent just in formal preference assessments. Although some teachers may not implement formal preference assessments of this length, clearly there are professionals who may. Given the extent of deficits individuals diagnosed with autism face, and the importance of maximizing teaching time to overcome these deficits, it would seem that the time it takes to implement formal preference assessments might be unjustified and better spent on direct teaching. Even if the clinicians utilize a more efficient formal preference assessment (e.g., multiple stimulus without replacement), provide shorter durations of access to the stimulus during formal preference assessments, or conduct formal preference assessments less frequently, it still is additional time that may be utilized elsewhere.

Third, this study might raise the question of how much training professionals should receive prior to working with individuals diagnosed with ASD. Each teacher in this study had received intensive training (i.e., 40 hr per week) for an extensive period of time (e.g., approximately 2 months) followed by 5 years of experience and ongoing training and supervision. Although it might take a professional a lot of training hours to be proficient in utilizing in-the-moment analysis and far less training hours to learn how to conduct a formal preference assessment, the training may have additional benefits (e.g., learning how to condition reinforcement or learning how to conduct in-the-moment assessments in other areas). Given the deficits that individuals diagnosed with ASD face, and the vast improvements professionals can make when implementing quality behavioral intervention, it is critical that staff be trained so that they can make in-the-moment analysis of reinforcers without having to utilize time-consuming, even if formulaic, protocols.

This study is not without its limitations, which precludes us from making global confirmative statements about the use of formal preference assessments. For one, only high functioning individuals diagnosed with ASD participated in this study; thus, it is not known whether the results would be similar for individuals who would be considered lower functioning. Second, the study only consisted of two children and, therefore, the results of this study would be considered a preliminary analysis. In this study, we utilized a single-subject design which controls for many threats to internal validity (Campbell & Stanley, 1963) but does not control for many threats to external validity. When evaluating newer procedures, it is important to first ensure that a procedure is internally robust prior to evaluating the procedure in a larger group design. Nevertheless, future researchers should evaluate in-the-moment reinforcer analysis methods with a wider range of individuals diagnosed with ASD and within a larger group design.

Third, we only utilized a paired-stimulus preference assessment, so it is not known what the results would have been if we had utilized other formal preference assessments (e.g., multiple stimulus without replacement). Although the results may be different if we evaluated the in-the-moment reinforcement analysis with other formal preference assessments, the additional time of the formal preference assessments would more than likely add unneeded time. Another limitation is that the majority of items used throughout the study were identified as preferred to the participant by the clinical team. This may have stacked the deck in favor for the in-the-moment reinforcement analysis condition. The researchers did include three items that were identified as not being preferred for each of the participants, and the teachers assigned to in-the-moment reinforcement analysis condition were unaware of which items were identified as preferred or not preferred by either formal preference assessment or by interviews. Therefore, it is highly unlikely that the deck was stacked in favor of either condition. Nevertheless, future researchers may wish to evaluate both procedures utilizing items that are not identified as preferred.

Along these lines, in both conditions, the teachers provided social praise paired with the provision of the stimulus. Therefore, it is not known how much the social praise may have effected responding. Although pairing social praise with a tangible item is a more natural way to provide reinforcement, it does provide a possible confound and, therefore, should be parsed out in future research. Furthermore, it is possible that participants satiated to the preferred items in the preference condition, which may have an effect on responding. Although this is a possibility, it is also highly unlikely given the brief access that participants had with each item.

Finally, the research included teachers who were well trained in ABA and ASD, who had more than 5 years of experience, and who knew the participants. Therefore, it is not known what the effects would be using a novice teacher. It is also not know, empirically, how to effectively train staff to implement in-the-moment reinforcer analysis. This is an area that would warrant future research. Along these lines, it would be important for both future researchers and clinicians to evaluate treatment fidelity on each level of the procedure (e.g., conditioning or using learner affect to determine reinforcers) to ensure that the procedures are being implemented correctly.

This study compared the use of reinforcing stimuli selected based on the results of formal paired-stimulus preference assessments to a teacher using in-the-moment reinforcer analysis in providing reinforcement. Results of the study showed that skill acquisition occurred across both teaching conditions; however, the amount of time a participant spent during the paired-stimulus preference assessment condition was substantially longer than time spent in the in-the-moment reinforcer analysis condition. Therefore, the results of this study indicate that the use of paired-preference assessments was not warranted for our two participants.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Ayllon

Azrin

(1965). The token economy: A motivational system for therapy and rehabilitation. New York, NY: Appleton-Century-Crofts.

Campbell

D. T.

Stanley

J. C.

(1963). Experimental and quasi-experimental designs for research. Chicago, IL: Rand McNally College.

Carr

J. E.

Nicolson

A. C.

Higbee

T. S.

(2000). Evaluation of a brief multiple-stimulus preference assessment in a naturalistic context. Journal of Applied Behavior Analysis, 33, 353–357.

Cooper

J. O.

Heron

T. E.

Heward

W. L.

(2007). Applied behavior analysis. Upper Saddle River, NJ: Pearson Education.

DeLeon

I. G.

Iwata

B. A.

(1996). Evaluation of a multiple-stimulus presentation format for assessing reinforcer preferences. Journal of Applied Behavior Analysis, 29, 519–533.

Fisher

W. W.

Piazza

C. C.

Bowman

L. G.

Hagopian

L. P.

Owens

J. C.

Slevin

(1992). A comparison of two approaches for identifying reinforcers for persons with severe and profound disabilities. Journal of Applied Behavior Analysis, 25, 491–498.

Graff

R. B.

Karsten

A. M.

(2012). Assessing preference of individuals with developmental disabilities: A survey of current practices. Behavior Analysis in Practice, 5, 37–48.

Green

C. W.

Reid

D. H.

White

L. K.

Halford

R. C.

Brittain

D. P.

Gardner

S. M.

(1988). Identifying reinforcers for persons with profound handicaps: Staff opinion versus systematic assessment of preferences. Journal of Applied Behavior Analysis, 21, 31–43.

Leaf

J. B.

Dotson

W. H.

Oppenheim

M. L.

Sheldon

J. B.

Sherman

J. A.

(2010). The effectiveness of a group teaching interaction procedure for teaching social skills to young children with a pervasive developmental disorder. Research in Autism Spectrum Disorders, 4, 186–198.

10.

Leaf

J. B.

Leaf

R. B.

Alcalay

Leaf

J. A.

Ravid

Dale

. . . Oppenheim-Leaf

(2015). The utility of formal preference assessments for individuals diagnosed with autism spectrum disorder. Education and Training in Autism and Developmental Disabilities, 50, 199–212.

11.

Leaf

J. B.

Oppenheim-Leaf

M. L.

Leaf

Courtermanche

A. B.

Taubman

McEachin

Sherman

J. A.

(2012). Observational effects on the preferences of children with autism. Journal of Applied Behavior Analysis, 45, 473–483.

12.

Leaf

J. B.

Sheldon

J. B.

Sherman

J. A.

(2010). Comparison of simultaneous prompting and no-no prompting in two-choice discrimination learning with children with autism. Journal of Applied Behavior Analysis, 43, 215–228.

13.

Petursdottir

A. I.

Carr

J. E.

Lechago

S. A.

Almason

S. M.

(2008). An evaluation of intraverbal training and listener training for teaching categorization skills. Journal of Applied Behavior Analysis, 41, 53–68.

14.

Piazza

C. C.

Fisher

W. W.

Hagopian

L. P.

Bowman

L. G.

Toole

(1996). Using a choice assessment to predict reinforcer effectiveness. Journal of Applied Behavior Analysis, 29, 1–9.

15.

Restar

J. L.

Noell

G. H.

(2008). Evaluating preference assessment for use in the general education population. Journal of Applied Behavior Analysis, 41, 447–451.

16.

Rincover

Newsom

C. D.

(1985). The relative motivational properties of sensory and edible reinforcers in teaching autistic children. Journal of Applied Behavior Analysis, 18, 237–248.

17.

Schreibman

(1975). Effects of within-stimulus and extra-stimulus prompting on discrimination learning in autistic children. Journal of Applied Behavior Analysis, 8, 91–112.

18.

Windsor

Piche

L. M.

Locke

P. A.

(1994). Preference testing: A comparison of two presentation methods. Research in Developmental Disabilities, 15, 439–455.