Abstract
Despite potential benefits of tobacco-free campus policies, compliance remains a challenge. Observational measures hold the most promise in determining compliance with these policies. There is need for further study to determine validity of observational measures of compliance with tobacco-free campus policies. The purpose of this study was to determine the validity of two observational measures of compliance with a tobacco-free campus policy: direct observation of violators and cigarette butts. Data collection took place over a 1-year time period. Direct observation was operationally defined as the number of observed violators in hot spots. A cigarette butt protocol previously found to be reliable was used to count the number of butts in campus hot spots. Results indicated a positive relationship between number of violators observed per visit and number of cigarette butts collected. Although most of the hot spots exhibited two or fewer violators per visit and 100 butts or fewer per collection, the data points outside this range supported a positive association between observed violators per visit and cigarette butts. The findings support that direct observation of violators is a valid measure of compliance compared to cigarette butts. Given available resources, using one or the other as evaluation measures is warranted.
Tobacco use and exposure to secondhand smoke (SHS) are the single most preventable causes of death in the United States (U.S. Department of Health & Human Services, 2006). The American College Health Association (ACHA) recommends that college and university campuses promote tobacco-free environments in an effort to improve overall well-being among students, staff, and visitors (ACHA, 2011). Colleges and universities have made progress in reducing tobacco use and exposure to SHS by adopting smoke- or tobacco-free campus policies. As of July 3, 2014, there were at least 1,372 campuses with 100% smoke- or tobacco-free policies with no exemptions (includes entire campus, both indoors and out; American Nonsmokers’ Rights Foundation [ANRF], 2014). Of these, 938 have a tobacco-free policy in which no form of tobacco is allowed (ANRF, 2014).
Despite the potential benefits of these smoke- and tobacco-free campus policies, one common challenge is compliance. There tends to be a perceived lack of enforcement on campuses with current smoke- or tobacco-free policies (Etter, Ronchi, & Perneger, 1999; Gerson, Allard, & Towvim, 2005; Halperin & Rigotti, 2003; Plaspohl, Parrillo, Vogel, Tedders, & Epstein, 2012). According to Harris, Stearns, Kovach, and Harrar, (2009), anecdotal reports suggest there is little compliance with smoke- and tobacco-free campus policies. Mamudu, Veeranki, He, Dadkar, and Boone (2012) asked university personnel about noncompliance with the tobacco-free policy. They found that although only 5.6% and 1.7% of the total study participants reported that they had smoked cigarettes and used other tobacco products, respectively, on campus since the policy was enacted, 80.8% reported that they observed others not complying with the policy. There may be a disconnect with regard to perceived and actual compliance of these campus policies. Yet little research has examined the extent of policy compliance and effectiveness of enforcement (Halperin & Rigotti, 2003; Rodgers, 2012).
In this study, policy compliance is conceptualized as an action arena in Ostrom’s (2005) Institutional Analysis and Development (IAD) Framework. The IAD posits that policy development, from adoption to implementation and evaluation, occurs in different action arenas at different levels. These three nested levels include the (1) constitutional level that set the rules that govern institutions (e.g., state laws), (2) collective choice level at which rules are adopted (e.g., institution of higher education), and (3) operational level where the policy directly affects individuals (e.g., students, faculty, and staff; Ostrom, 2005). In the case of tobacco-free campus policies, institutions of higher education (collective level) take action (i.e., action arena) to debate and adopt tobacco-free campus policies. After the policy is adopted, it is implemented requiring specific actions by the affected community (i.e., action arena). Compliance is one critical element of policy implementation. Policy compliance is an operational-level construct that is based on an individual’s decision whether or not to use tobacco in areas covered by the campus policy (Fallin, 2011). The IAD has been used to guide understanding of the implementation of tobacco control policies and related policy compliance (Fallin, 2011; Martinez, 2009).
As part of the policy implementation process, there is a critical need for valid and reliable data collection measures to assess compliance on smoke- and tobacco-free college campuses to guide policy evaluation (Bower & Enzler, 2008; Halperin & Rigotti, 2003; Harris et al., 2009). The current measurement and evaluation research investigating compliance with smoke- and tobacco-free policies on college campuses, albeit limited, includes several modes of data collection: self-report (Lechner, Meier, Miller, Wiener, & Fils-Aime, 2012; Mamudu et al., 2012; Plaspohl et al., 2012), assessment of written policies (Boyce, Mueller, Hogan-Watts, & Luke, 2009; Halperin & Rigotti, 2003; Lee, Goldstein, Klein, Ranney, & Carver, 2012), and observational measures such as cigarette butts (Fallin, 2011; Fallin et al., 2012; Ickes, Hahn, McCann, & Kercsmar, 2013; Lee, Ranney, & Goldstein, 2013; Seitz et al., 2012) and direct violator observation (Etter et al., 1999; Harris et al., 2009; Ickes et al., 2013). One disadvantage of self-report measures of compliance is social desirability bias (Fallin et al., 2012). Observational measures hold the most promise in determining actual compliance with tobacco-free campus policies (Fallin et al., 2012).
A few studies have used observational measures such as direct observation of violators or individuals complying with the policy and cigarette butts (Fallin, 2011; Fallin et al., 2012; Harris et al., 2009; Ickes et al., 2013; Lee et al., 2013; Seitz et al., 2012) to evaluate compliance with smoke- and tobacco-free policies on college campuses. Harris et al. (2009) directly observed smokers’ compliance with a smoke-free campus policy that prohibited smoking within 25 feet of buildings during 120 observation intervals over 3 weeks. Similarly, observations occurred pre- and post-policy at a university in Switzerland. The proportion of smokers in the “no-smoking” area of the cafeteria decreased from 16% to 3% (p = .02; Etter et al., 1999). In another evaluation of an intervention to improve compliance with a tobacco-free campus policy, trained Ambassadors observed and approached violators of the policy (Ickes et al., 2013). During the 4-week intervention, ambassadors monitored campus hot spot areas throughout the week; in all, 529 policy violators were observed and 332 approached. Although direct observations seem to provide meaningful data, research is warranted to determine the validity and feasibility of this compliance assessment.
Cigarette butts provide another unique source of observational data to evaluate compliance with smoke- and tobacco-free policies (Lee et al., 2013). Fewer cigarette butts over time may indicate fewer cigarettes smoked (i.e., compliance with the policy) and reduced exposure to SHS (Lee et al., 2013). Cigarette butts also provide a direct measure of the potential impact on the campus environment (Lee et al., 2013; Sawdey, Lindsay, & Novotny, 2011). The Tobacco-Free Compliance Assessment Tool (TF-CAT) was designed to measure compliance with tobacco-free policies by counting cigarette butts in specific locations (Fallin et al., 2012). Fallin et al. (2013) report strong interrater reliability of the TF-CAT. This measure has been used to measure change in compliance pre- and post-policy intervention (Fallin et al., 2013; Ickes et al., 2013). Lee et al. (2013) used cigarette butts to determine if there was more smoking on campuses with no restrictions or partial restrictions compared to campuses with 100% smoke-free policies. Findings indicated fewer cigarette butts at campuses with100% smoke-free policies (Lee et al., 2013). Seitz et al. (2012) also used cigarette butts to determine the number of violators within areas covered by a partial smoke-free policy. Based on number of cigarette butts, there were at least 7,861 violations of the policy during a 30-day period (Seitz et al., 2012).
There are, however, limitations to using cigarette butts as the sole means of measuring compliance with tobacco-free campus policies. Cigarette butt data collection is time- and labor-intensive. It is not feasible to count cigarette butts on an entire college campus, particularly on large campuses (Fallin et al., 2012). During 1 week of data collection, Fallin et al. (2012) reported over 31 hours, or approximately 0.80 full-time equivalent, spent collecting and counting cigarette butts on hot spots throughout the campus. In addition, use of cigarette butt data as an evaluation measure for tobacco-free policies does not include smokeless or other forms of tobacco. Unlike traditional cigarettes, which are routinely discarded on the ground, there may be no “evidence” left behind of smokeless tobacco use (Fallin et al., 2012).
Considering the limited research on smoke- and tobacco-free campus policy evaluation, there is not a single accepted method to assess actual compliance with smoke- or tobacco-free policies (Fallin et al., 2012). Cigarette butt pickup is a reliable observational measure, but the protocol is time-consuming and may be impractical especially on large campuses. The purpose of this study was to determine the validity of direct observation of violators compared to cigarette butts as measures of assessing compliance with a tobacco-free campus policy. It was hypothesized that number of violators observed would be associated with number of cigarette butts on a tobacco-free campus.
Method
This measurement study combined and analyzed direct observation and cigarette butt data collected on one large, southeastern public university campus from three different time points during one calendar year—May 2012 to April 2013. Data collection took place during a 1-year Ambassador intervention to promote compliance with a university policy prohibiting tobacco use outside and inside any property owned by the university. The aim of the Tobacco-Free Take Action! Ambassador program is to train individuals to approach violators of the tobacco-free policy using an approved scripting technique (Ickes et al., 2013). Ambassadors were part-time employees of the university who observed violators, and reminded and reported violators of the policy. Since the focus of this study was to compare two observational measures of compliance, only data related to direct observations and cigarette butts were included. This study was exempt from institutional review board approval because no identifying human subject data were collected. For the purposes of this study, direct observation was operationally defined as the number of observed violators of the tobacco-free policy in campus hot spots in a given time period, adjusted by the corresponding number of ambassador visits to the hot spot during the same period. We used the TF-CAT (Fallin et al., 2012) protocol to collect and count cigarette butts on the ground in the same campus hot spots.
Campus hot spot locations were selected based on the ambassadors’ observational rounds through campus, as well as areas where policy violations had previously been reported (Fallin et al., 2012; Ickes et al., 2013). The four hot spots are centrally located on campus and are considered “heavy traffic” areas (Table 1). Over the 1-year time period, trained ambassadors visited Hot Spot A 17 times, Hot Spot B 127 times, Hot Spot C 37 times, and Hot Spot D only 5 times. Ambassadors spent the majority of their time at Hot Spot B due to the large number of violators consistently observed in that location. Ambassadors spent little time at Hot Spot D due to the times of the day that ambassadors were available to observe violators as well as the available resources. Complaints about Hot Spot D were typically during the evenings when ambassadors were unavailable to work.
Description of Campus Hot Spots Selected for Cigarette Butt Collection and Direct Observation
Measures
Direct Observation of Violators
Trained ambassadors observed and approached violators of the tobacco-free policy in the four campus hot spots. Data were collected during the ambassadors’ working hours, typically between 10 a.m. and 2 p.m. Ambassadors completed a site-specific checklist to document time spent at each hot spot. The checklist included location of hot spot, date, time of arrival and departure, and number of violators (Hahn et al., 2012; Ickes et al., 2013). After each day, ambassadors input the data using an online data collection tool created with Qualtrics software (Qualtrics, LLC, 2013).
Cigarette Butts
Cigarette butt data were collected by the trained ambassadors during 3-day periods in each of August 2012, December 2012, and April 2013. Data were collected using a validated protocol, the TF-CAT, a direct observation method (Fallin et al., 2012). Cigarette butts were collected from campus grounds on three consecutive days at the same time each day. Due to weather conditions and scheduling logistics, it was not always possible to collect data on the same days of the week. Boundaries for each hot spot were noted so collection locations remained consistent. Since the first collection day in each 3-day time period represented a much larger count of cigarette butts (due to a buildup since the previous collection), the sum of the counts on Days 2 and 3 were used in the analysis to reflect a representative picture of tobacco use at each location.
Data Analysis
The direct observation and cigarette butt data were summarized using descriptive and graphical methods. For each of the hot spots during each time period, the total number of violators observed during all ambassador visits was divided by the number of ambassador visits in the period. The cigarette butts counted from Days 2 and 3 were summed for each time period. A scatter plot was used to graph the number of violators observed per visit versus cigarette butts collected.
Results
As shown in Table 2, Hot spot B had the highest number of observed violators per ambassador visit, ranging from 1.88 to 13.24 over the three time periods. The other three hot spots had ratios of violators to visits that ranged from 1.25 to 2.00. These results are partially consistent with the butt data: Two of the areas (Hot Spots B and D) routinely had more than 100 cigarette butts during the study, whereas the other two consistently had less than 50 for all three time periods. The scatter plot shown in Figure 1 provides a graphical display of the number of cigarette butts collected (during Days 2 and 3) versus the observed violators per visit for Hot Spots A to C. Hot Spot D was omitted from this graph since ambassadors made very few visits to this hot spot based on few reported violations during ambassador working hours, resulting an incomplete assessment of violators at all times of day. The scatter plot indicates a positive relationship between number of violators observed per visit and number of cigarette butts collected. Although most of the hot spots exhibited two or fewer violators per visit and 100 cigarette butts or fewer per collection, the data points outside this range support a positive association between cigarette butts and violators per visit.
Descriptive Summary, by Hot spot and Time Period, of Observed Violators, Ambassador Visits, Ratio of Number of Violators to Number of Ambassador Visits, and 2-Day Cigarette Butt Totals
NOTE: T1 for cigarette butt data: August 2012; T1 for violator data: May to August 2012; T2 for cigarette butt data: December 2012; T2 for violator data: September to November 2012; T3 for cigarette butt data: April 2013; T3 for violator data: January to March 2013.

Scatter Plot of 2-Day Cigarette Butt Total Versus Number of Observed Violators per Visit
Discussion
The purpose of this study was to compare two observational measures of compliance with a tobacco-free campus policy: direct observation of violators and number of cigarette butts. There was a positive association between observed violations of the tobacco-free policy per visit and cigarette butts in campus hot spots. This finding supports the validity of direct observation as a measure of compliance with smoke- and tobacco-free campus policies compared to cigarette butts (Fallin et al., 2012). Cigarette butt data collection and direct observations have been used previously as observational measures (Fallin, 2011; Fallin et al., 2012; Fallin et al., 2013; Harris et al., 2009; Ickes et al., 2013; Lee et al., 2013; Seitz et al., 2012). The cigarette butt protocol has been shown to have interrater reliability (Fallin et al., 2012). However, previous research has not investigated the validity of direct observation compared to cigarette butts as measures of compliance.
There are pros and cons to using cigarette butts and direct observation as policy compliance evaluation measures. One advantage of using cigarette butts is that they indicate the number of cigarettes smoked on campus and exposure to SHS (Lee et al., 2013). Measuring cigarette butts also showcases the direct impact on the campus environment (Lee et al., 2013; Sawdey et al., 2011). In addition, collection of cigarette butts can be used to investigate the differential impact of policy strength across campuses (Lee et al., 2013). However, there are disadvantages to using cigarette butts as a measure of policy compliance. Data collection is time- and labor-intensive (Fallin et al., 2012; Fallin et al., 2013). It is often not feasible to count cigarette butts on an entire campus, particularly on large college campuses with property spread out over hundreds of acres (Fallin et al., 2012). Another concern is that cigarette butt data exclude other forms of tobacco, such as smokeless tobacco (including e-cigarettes), which are covered by 100% tobacco-free campus policies (ANRF, 2014). Unlike traditional cigarettes, which are routinely discarded on the ground, there may be no “evidence” left behind by smokeless tobacco, e-cigarettes, or other forms of tobacco use (Fallin et al., 2012). The ACHA (2011) recommends tobacco-free campus policies, including all forms of tobacco, as the gold standard. As more and more campuses adopt these policies (ANRF, 2014), it is necessary to develop and use valid measures to assess policy compliance related to all forms of tobacco use.
Direct observation of individuals violating smoke- or tobacco-free policies has also been used as a measure of policy compliance (Etter et al., 1999; Harris et al., 2009; Ickes et al., 2013). Direct observation provides a general indicator of the number of individuals violating campus smoke- and tobacco-free policies. These data enable trend comparisons in policy compliance over time, particularly when targeting campus hot spots (e.g., high-traffic areas). Direct observation provides the potential to discern violators who use multiple forms of tobacco, as covered by 100% tobacco-free campus policies (ACHA, 2011). Since college campuses have seen a trend in increased use of emerging tobacco products, particularly e-cigarettes (ACHA, 2012), evaluation of actual compliance (based on all tobacco products) may be achieved only through direct observation. Although data collection of observations may also be time-intensive, one advantage of direct observation as an evaluation strategy is that the mere presence of observers (e.g., individuals collecting data) may change subsequent tobacco use behavior, thereby improving compliance (Fallin, 2011). According to Harris et al. (2009), both passive and active enforcement strategies increase compliance.
Direct observation as the sole measure has the potential to serve two purposes: compliance evaluation and as an intervention to improve compliance with smoke- and tobacco-free campus policies. Direct observation could be an efficient use of time and resources, often cited as barriers to compliance efforts (Halperin & Rigotti, 2003; Harbison & Whitman, 2008). As more campuses implement smoke- and tobacco-free policies (ANRF, 2014), there is a need to maximize the potential impact of these policies. The mere presence of individuals observing violators of campus policies could make the difference in overall policy compliance and related policy outcomes. Data reported here provide additional support for an Ambassador program to promote compliance on smoke- or tobacco-free college campuses (Hahn et al., 2012; Ickes et al., 2013). Ambassadors who are trained to observe policy violators as well as approach and report violators may be cost-effective and have long-term impact on campus compliance and enforcement efforts.
Ambassadors are identified with buttons and over time become recognized throughout campus. This may have affected the positive association between direct observation and cigarette butts in campus hot spots in this study. The goal behind an Ambassador intervention is to create a positive environment of compliance (Ickes et al., 2013) to promote effective smoke- and tobacco-free campus policies. Future research is warranted to investigate the validity of direct observation on campuses with and without Ambassador programs to determine if the findings reported here relate to the ambassadors themselves or to surveillance by general observers. Regardless, as health promotion professionals advocate for adoption of smoke- and tobacco-free campus policies, there is a need to evaluate policy compliance to determine overall impact on the campus community.
Health promotion professionals assist communities and institutions to adopt health-promoting policies, but, oftentimes, the details of how the policies will be put into place or implemented are lacking. There is a dearth of research on the implementation phase of health policy development (Fallin, 2011). More research is needed to better understand the action arena (Ostrom, 2005) of policy implementation (e.g., compliance) once policies are adopted. The IAD framework guides health promotion practice by providing guidance on the different levels of policy development that must be considered to achieve optimal health outcomes (Ostrom, 2005). In the case of tobacco-free campus policies, evaluation of the operational level (e.g., how well individuals comply with the policy) is a critical component to consider when campuses implement smoke- or tobacco-free campus policies.
Limitations
There are several limitations to this study. First, although cigarette butt data collection occurred throughout the year, more favorable weather conditions during certain times of year may lead to tobacco users congregating outside. This limitation is minimized since data collection occurred during summer, winter, and spring seasons. Second, although data collection occurred at multiple campus hot spots, campus hot spots may shift over time. Direct observation and cigarette butt data may not have been generalizable to all campus areas. Finally, although the ambassadors made more visits to the hot spots with the greatest number of policy violators (as evidenced by the 2-day butt totals), the choice of which hot spot to monitor on any particular visit was subjective and was necessarily associated with observed violator counts. Related to this issue, we noticed anecdotally that hot spot locations may be time-of-day sensitive, with tobacco users congregating around classroom buildings during the daytime hours when ambassadors were working and around the library in evening hours when ambassadors were not available. Future research is warranted to investigate validity of observational measures when ambassadors work split shifts and/or when multiple paired teams observe throughout the day.
Conclusion
The ACHA (2011) emphasizes the need to plan, maintain, and support consistent compliance efforts and enforcement of smoke- and tobacco-free policies. With almost 1,400 smoke- or tobacco-free campuses across the United States (ANRF, 2014), valid and feasible measures of compliance are necessary to evaluate the impact of smoke- and tobacco-free campus policies (Glassman, Reindl, & Whewell, 2011). Solely relying on self-report measures may limit the ability to accurately assess policy compliance and health outcomes, slowing the progress of health promotion practice (Fallin et al., 2012). We found that direct observation is a valid measure of policy compliance compared to counting cigarette butts. Furthermore, direct observation may serve two purposes: evaluation and an intervention to promote compliance. Future research is needed to investigate the reliability and validity of measures to assess policy compliance.
Footnotes
Acknowledgements
We would like to thank the efforts of our Ambassadors in their tireless work. In addition, we would like to acknowledge University of Kentucky Campus Services for supporting the Tobacco-Free Take Action! Ambassador program.
This work was supported by the National Center for Research Resources and the National Center for Advancing Translational Sciences, National Institutes of Health (Grant Nos. UL1TR000117, TL1 TR000115, or KL2 TR000116). The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.
