Abstract
BACKGROUND:
Workplace violence is a global problem that includes actions collectively defined as bullying as perpetrated by a work colleague.
PURPOSE:
Two distinct studies were conducted to assess the feasibility of using an abridged 6-item scale within the 21- item Negative Acts Questionnaire-Revised (NAQ-R) designed to assess workplace bullying.
METHODS:
The pilot study was a psychometric review of the 21-item NAQ-R, and the main study was conducted to determine the reliability and the validity of using a 6-item version. Cronbach’s alpha assessed the internal consistency of the two versions of the NAQ-R. Validity was assessed using logistic regression with theoretically related constructs with a pilot study (n = 420) using the 21-item scale, followed by the main study (n = 11,874) using the 6-item version.
RESULTS:
Both the pilot study (21-item)and the main study (6-item) versions of the NAQ-R demonstrated high internal consistency (Cronbach’s alpha 0.93–0.86, respectively). Similar to the 21-item, the 6-item version had a significant impact on the intent to remain on the job, being personally affected, and supported the protective role of the organizational climate against exposure to negative acts.
CONCLUSION:
This study supports using the 6-item NAQ-R, which can reduce respondent burden and streamline data gathering and analysis.
Introduction and background
Early investigation of workplace bullying began in the 1970s when Carroll Brodsky published his seminal work, The harassed worker [1]. This 1976 book was a ground breaking analysis of over a thousand cases filed in California and Nevada for Worker’s Compensation by workers who stated that they were ill or unable to work because of harassment by employers or coworkers. This was the first examination of the phenomenon now known as workplace bullying. A decade later, European scholars conceptualized the phenomenon and explored its antecedents, consequences, attributes, and most importantly its measurement and quantification [2–9].
Workplace bullying has been defined as “repeated actions and practices directed against one or more workers, that are unwanted by the victim, carried out deliberately or unconsciously, that cause humiliation, offense, and distress and can interfere with work performance and/or cause unpleasant working environment” [3, 10]. However, research on this topic is challenged by the lack of a standard definition of workplace bullying and a common underlying theoretical framework [11–15]. Previous literature defines and measures the phenomenon of bullying by a range of behaviors including: specific negative acts, work-related bullying, person-related bullying, psychological or physical bullying, and organizational bullying. This literature is additionally complicated by the subjective versus objective nature of the measurement of bullying. The bullying experience is dependent upon the victims’ subjective perceptions and the observability of the bullying behavior. Accordingly, some researchers have used subjective measures (a statement that includes “I feel”), while others have utilized objective tools (a list of behaviors experienced that do not mention the word bullying”), and some have used both [16].
Many instruments have been tested to assess workplace bullying but researchers have struggled to develop a single valid and reliable tool that adequately measures workplace bullying [10, 17–23]. The most often cited instrument is the Negative Acts Questionnaire (NAQ) developed and tested by Einarsen and Hoel [18]. It was written in Norwegian and contained 29 items with a Cronbach alpha of 0.87–0.93. The English translation is known as the Negative Acts Questionnaire-Revised (NAQ-R) [10, 26].
The NAQ-R focuses on negative acts, and all items are written in behavioral terms with no reference to the term bullying. This objective approach has the benefit of allowing participants to respond to each item without having to label themselves as being bullied or not [27]. The responses measure frequency of occurrence from “never” to “daily”. A one-time incident is not sufficient to be considered as bullying [7, 18]. After responding to these behavioral items, participants are introduced to a definition of bullying at work and they must indicate whether or not they considered themselves as targets of bullying at work invoking their subjective perspective [27]. The tool further assesses exposures to bullying and related behaviors in the previous six months that include the experience of bullying, frequency of encounters, duration of experience, self-labeled victimization from bullying, and the main perpetrators [7, 18]. There are three NAQ-R sub-scales reflecting several important dimensions: person-related bullying, work-related bullying, and sexual harassment [7, 27]. The NAQ-R correlates with measures of psychological health and well-being (r = –0.42), intention to quit the job (r = –0.36), and self-assessed job performance (r = –0.24) [7, 27].
Recent evidence suggests that a shorter NAQ-R may suffice as a measure of workplace bullying [25, 28–30]. Despite demonstrated reliability and validity of the different versions (Table 1), there is no consensus on the fewest number and which specific items need to be included in a shorter version.
Use, reliability, and validity evidence for selected various forms of the Negative Act Questionnaire
Use, reliability, and validity evidence for selected various forms of the Negative Act Questionnaire
The purpose of this paper is to examine the psychometric properties of a proposed six item NAQ-R scale. The pilot study is a psychometric review of the 21-item Negative Acts Questionnaire-Revised (NAQ-R), and the main study tests the reliability and validity of a reduced six-item version (NAQ-R). Individual impact, organizational impact, and the supportiveness of the organizational climate are used as constructs to validate the NAQ-R in its different-item versions. Selection of these constructs was guided by the Organization of Work framework (OOW) [31] and the Workplace Bullying theoretical framework proposed by Einarsen et al. [3]. The OOW framework has been used to conceptualize how bullying affects exposure to psychological stress and physical hazards that may impact the victim’s health [31]. The Workplace Bullying model based on the transactional theories of stress, the Cognitive Activation Theory of stress, and the Affective Events Theory encompasses most of the antecedents and consequences of bullying [3, 32].
Two distinct studies, referred to as the pilot study and main study within the upcoming sections, were conducted with employees of state government public sector agencies in an eastern US state. A pilot study was conducted using the 21-item scale to determine the psychometrics of the 21-item NAQ-R and guided the decision to test a reduced six-item version of the NAQ-R in the main study. The survey was initially pilot tested in one agency (n = 420) and subsequently shortened before being distributed to the employees of other agencies in the main study. A main study was then conducted to test the psychometric properties of the shortened six-item version (n = 11,874). The study utilized a Participatory Action Research (PAR) framework in design and implementation. A Project Advisory Group (PAG) that included state government managers and union representatives identified agencies that would represent a cross-section of government functions. The PAG also helped ensure that the survey would be understood and well-received by potential respondents, and that it would be administered in a way that would protect respondents’ confidentiality while still resulting in a high survey response rate. The study was approved by the University of Maryland, Baltimore (UMB) Institutional Review Board.
Participants received an introductory email from the research team, management, and union leaders. Next, an email was sent by the Principal Investigator (PI) that included the survey hyperlink, project details and confidentiality guarantees. This was followed by two reminder emails sent at one-week intervals. Survey responses were uploaded directly into the secure UMB server.
The 15–20 minutes surveys included 81 questions that assessed the extent, severity, and impact of coworker conflict, plus the work organizational climate and demographics. A paper version was available for participants without access to email or upon request. The paper version was delivered by mail with pre-paid postage for return delivery, or via email as a PDF file.
Sample
Pilot sample
The pilot study was conducted in a service agency (n = 420) with a 72% response rate.
Main study sample
The main study involved employees of four agencies, collectively representing the major functions performed by state government, namely administrative, regulatory, field, and institutional services with a 92% response rate. Two of the larger agencies (n = 3996) were primarily administrative, one with some service and regulatory functions (n = 2581). A third large agency was responsible for both administrative and field activities (n = 4325), and a fourth agency included three large regionally-based semi-autonomous mental health centers that provide largely institutional care to the chronically mentally ill (n = 972). Both the pilot and the main study samples are further described in the results section and in (Table 2).
Description of the Pilot Study and Main Study Samples
Description of the Pilot Study and Main Study Samples
aNumbers (N) may not sum to total due to missing data.
NAQ-R pilot study (21-items)
The survey was pilot tested utilizing the 21-item NAQ-R (Cronbach’s alpha = 0.93). The NAQ-R consists of 21- objective items describing different kinds of behaviors that may be perceived as bullying if they occur on a regular basis. All items are written in behavioral terms without reference to the word bullying. This allows participants to respond to each item without the bullying label. The responses for these questions were as follows: daily or almost daily, more than once a week, more than once a month, at least once during the past six month, and not in the past six months or never.
Responses to each of the 21-item NAQ-R were analyzed individually at first. As the model necessitated dichotomous data, responses were collapsed and summed into exposed (Yes = 1) and not exposed (No = 0) to negative acts. Participants who experienced a negative act daily or almost daily, more than once a week, more than once a month, at least once during the past six months were categorized as exposed (Yes = 1); and participants who reported no exposure to any negative acts in the past six months or never were categorized as not exposed (No = 0).
NAQ–R main study (6-items)
The 21-item NAQ-R was subsequently shortened to a 6-item version for the main study (Cronbach’s alpha = 0.90). The decision to use fewer items was guided by the participatory action research approach of the study plus previous evidence that a shortened scale would have good reliability and validity [29, 33]. The six NAQ-R items included the following negative acts: “within the last six months, how often at work have you: (1) been humiliated or ridiculed; (2) had insulting or offensive remarks made about you; (3) been intimidated with threatening behavior; (4) been ignored or shunned; (5) been subjected to excessive teasing and sarcasm; (6) been shouted at or targeted with spontaneous anger or rage.” The responses for these questions were as follows: daily or almost daily, more than once a week, more than once a month, at least once during the past six month, and not in the past six months or never.
Similar to the 21-item NAQ-R in the pilot sample, the responses to the 6-item version were analyzed individually at first, then the responses to each NAQ were summed and collapsed into two categories, exposed to NAQ “Yes” (daily to at least once during the past six months) and “No” defined as not exposed to any NAQ in the past six months or never.
Bullying
After responding to the NAQ-R objectives items, the standard definition of bullying was then given and the participants subjectively indicated whether or not they consider themselves as bullied. The participants were then asked how often they had experienced workplace bullying in the previous six months. Responses for these questions were as follows: daily or almost daily, more than once a week, more than once a month, at least once during the past six months, and not in the past six months or never.
The measured constructs other than bullying (individual impact, organizational impact, supportiveness of the organizational climate, and demographics) were used to validate the NAQ-R and are described below.
Individual impact: was measured by three questions which asked how the experience of negative acts during the last six months negatively affected ‘your’ work; influenced ‘your’ intention to remain in current job; and negatively affected ‘you’ personally using a five-point Likert-type Scale: 0 = not at all; 1 = not much; 2 = somewhat; 3 = a lot; and 4 = very much. Each item was collapsed into two categories (No = not at all, and Yes = all other categories combined).
Organizational impact: was measured by seven yes or no questions assessing actions taken by participants who experienced or were exposed to negative acts. Item selection was guided by the fact that reporting, filing a grievance, submitting a workers compensation, and litigation affect the organization in terms of costs associated with the procedures, staff time, providing help and support, as well as compensation [34–37]. These seven questions were reduced to three dichotomous (yes/no) variables: reporting (reporting the act or behavior to senior staff member, union or organization and/or filing an incident report); taking action (seeking counseling or help from employee assistance program; transferring to another position/worksite or shift); and taking time off/charging leave credits.
Supportiveness of the organizational climate index: was composed of eight questions with response options (0 = False, 1 = True). These questions included if the respondents 1) were treated with respect and fairness by supervisors and 2) were treated with respect and fairness by each other; 3) were listened to by supervisor; 4) believed that the organization values and cares about them; 5) thought employees treated each other aggressively; 6) had knowledge of grievance process if they felt they were treated unfairly; 7) needed to put up with a lot of tough treatment from those in authority; and 8) could speak up if they thought there was a problem. The index demonstrated acceptable internal consistency with Cronbach’s alpha of 0.74. A total index score was derived by summing the number of questions, thus a higher score represented better work atmosphere, where the scores ranged from 0–8 with a mean of 5.10±2.19. The index was analyzed in tertiles: 0–5 = low supportive organizational climate, 6–7 = moderately supportive organizational climate, and 8 = high supportive organizational climate.
Demographics and work factors
Workers were asked to provide gender, age, race, if their job included bargaining unit membership, the agency where they worked, and employment tenure.
Data analysis
Descriptive statistics were conducted for the NAQ-R questions; and inter-item correlations were calculated between each of the sub-items. To test for internal consistency reliability, Cronbach’s Alpha was calculated for the 21 and the six-item NAQ-R.
To evaluate the construct validity, we conducted a Pearson’s product-moment correlation between the NAQ-R scores (6 and 21 items) and the subjective response to bullying. The construct validity was evaluated using separate logistic regression models for the 21 and six-item NAQ-R in relation to other related constructs e.g., supportiveness of the organizational climate and individual impact. Analyses were conducted using SPSS 19.0 [38].
Results
Descriptive analysis
The pilot study sample (n = 420) was 61.6% female, 85.3% white, and more than two thirds (69.5%) were greater than 45 years of age. More than half of the pilot sample (53.7%) had a job tenure of more than five years and (57.9%) were a member of the professional bargaining unit (Table 2).
The main study sample (n = 11,874) was 51.9% male; 85.3% white, and less than two thirds (59.8%) were greater than 45 years of age. More than half of the main study sample (56 %) had a job tenure of more than five years and were members of professional bargaining unit (Table 2).
The frequency distribution of the specific NAQ-R items is presented in (Table 3) by pilot study (21 NAQ-R items) and main study (six NAQ-R items) samples. Among the six NAQ-R items in both the pilot and main study samples, the most common negative act reported was “being ignored or shunned” (23% and 30% respectively). The least common negative act was “being subjected to excessive teasing and sarcasm” (10%) in the pilot sample, and “being intimidated with threatening behavior” (14%) in the main study sample. Almost 10% of the participants in both samples reported having experienced bullying in the prior six months (Table 3).
Frequency Distribution [n (%)]for the specific NAQ-R items in the Pilot and the Main Study Sample
Frequency Distribution [n (%)]for the specific NAQ-R items in the Pilot and the Main Study Sample
Most of the participants in the pilot sample who reported some type of negative acts stated that it negatively affected their work (87.7%). More than one third (34.5%) reported that it influenced their intent to remain in their job; and 40% said it negatively affected their personal life (Table 4).
More than half of the participants in the main study sample experienced some individual impact as depicted in (Table 4). Nearly two thirds reported that it negatively affected their work, about half of the participants reported that it influenced their intent to remain in their job, and 63% reported that it had negatively affected their personal life (Table 4).
Almost two thirds (59.2%) of the pilot sample versus 40% in the main study sample reported one or more actions with organizational impact. One third (32%) of those in the pilot sought some kind of help and 15% charged leave credits compared to 11.4% and 14% in the main study sample, respectively (Table 4).
Frequency Distribution of Individual and Organizational Impact and Supportiveness of Organizational Climate in the Pilot and Main Study Sample
aNumbers (N) may not sum to total due to missing data.
The NAQ-R indicated a high internal consistency with a Cronbach’s Alpha of 0.93 for the 21-items in the pilot sample, and 0.86 for the 6-item version in the pilot agency and 0.90 in the main study sample.
To evaluate the construct validity testing of association using Pearson correlations between the 21 and 6 items NAQ-R (used as an index) and the participants subjective measure of bullying item demonstrated significant positive correlations in the pilot 21 and 6 items NAQ-R, and the main study sample 6 items NAQ-R (r = 0.7; p < 0.01).
The Chi square association (Table 5) between the pilot 21 and 6 items NAQ-R, collapsed into two categories, exposed to NAQ “Yes” and “No”, and individual impact questions showed that the intent to remain on the job (94.7% and 72% respectively) and being negatively affected personally (93.5% and 69.3% respectively) were significant within both the 21 and 6 items NAQ-R versions in the pilot sample (p < 0.05). For the main study sample, the three individual impact questions (negatively affected work (98.1%), affected personally (97.9%), and intent to remain on the job (98.3%)) showed significant association with the 6 items-NAQ-R collapsed into two categories, exposed to NAQ “Yes” and “No”, (p < 0.05).
Descriptive Statistics and Chi-Square Associations for Individual Impact and Supportiveness of Organizational Climate Within the Pilot Study and Main Study by by Exposure to Negative Acts
Descriptive Statistics and Chi-Square Associations for Individual Impact and Supportiveness of Organizational Climate Within the Pilot Study and Main Study by by Exposure to Negative Acts
aNumbers (N) may not sum to total due to missing data.
Organizational impact measured by reporting, seeking help, and charging leave credits were not significantly associated with any version of the NAQ-R in either the pilot or the main study samples. This can be attributed to the nature of the data where numbers as low as zero were noted in some cells, to the type of questions asked to participants, or to the quality of the measure itself or the absence of an association. Accordingly, the research team decided to exclude this measure from the regression analysis to assess the validity of the 6 vs. 21 item NAQ-R.
The pilot 21 and 6 items NAQ-R, and the main study 6 items-NAQ-R version were significantly associated with organizational supportiveness (p < 0.05) with higher frequency of low supportive organizational climate reported among those who were exposed to negative acts (51.8% and 46.1% for the pilot 21 and 6 items NAQ-R respectively, and 60.4% for main study sample 6 items-NAQ-R version).
To further assess validity, binary logistic regression was conducted to determine the association between NAQ-R versions and related theoretical constructs, namely: individual impact illustrated by the intent to remain on the job, being negatively affected personally, negatively affecting their work, and the supportiveness of the organizational climate (Table 6).
Estimated association (odds ratio(95% confidence interval) for the NAQ-R (21 and 6) and Individual Impact; and Supportiveness of the Organizational Climate Within the Pilot Study and Main Study
Both NAQ-R versions were significantly associated with both intent to remain on the job and negatively affecting the participants personally (Table 6). The odds ratio of the intent to remain on the job was 18.0 for the 21- item NAQ-R, compared to 11.0 for the 6-item NAQ-R (pilot sample) and 2.59 for the main study sample where only the 6-item NAQ-R was used. The odds ratio of negatively affecting the participants personal was 16.4 for the 21-item version and 11.7 for the 6- items (pilot sample) compared to 2.16 in the main study sample.
A supportive organizational climate was significantly associated with all the NAQ-R item combinations (Table 6), with a protective odds ratio for moderate (0.40) and high (0.27) support in the 21-item, indicating less exposure to negative acts with the increase in the supportiveness of the organizational climate. The 6-item odds ratio were similar for both pilot and main study samples:0.25 (moderate) and 0.12 (high) support the protective impact of the supportive organizational climate and reducing the risk for exposure to negative acts.
This study supports the validity and reliability of the 6 and 21 item NAQ-R. The level of evidence demonstrated in this paper is similar to other shorter versions of the NAQ-R where Cronbach’s alpha ranged between 0.81–0.91 [8, 39–42].
We found that experiencing negative acts had significant individual impact in terms of the intent to remain on the job and being negatively affected personally, in both versions. Our findings are also supported by a meta analysis that showed that workplace bullying was associated with increased levels of intention to leave, decreased levels of organizational commitment and of job satisfaction [32] and by validation of measures of bullying in relation to stress, intent to leave, job satisfaction, and other related theoretical constructs [8, 40–45]. The individual impact of negatively affecting work was significant within the main study sample, supporting the validity of the construct. The lack of significance within the pilot sample may be attributed to the cell size and the smaller number of responses, but it showed consistent odds ratio and confidence intervals.
Working within an unsupportive organizational climate with poor psychosocial climate, where the participants are not listened to or valued, nor treated with respect and fairness, is associated with bullying [5, 46–48]. Both versions of the NAQ-R (6 and 21 items) in the pilot sample and the main study sample showed the validity of the items in relation to supportiveness of the organizational climate, where it was a significant protective factor.
The major strength of these analyses is testing and validating the 6-item NAQ-R in a pilot study and in a very large sample (main study) within the US public sector. Additional strengths include a high response rate and the Participatory Action Research approach to the design and implementation of the study [49]. In addition, further validation for the protective effect of strong organizational support in preventing bullying in this large public section, U.S. workforce sample is an additional important strength and finding.
Limitations can be attributed to using a cross-sectional design which limits the possibility of making causal inferences. Cell size of organizational impact as a covariate within the pilot sample was too small to be included in the regression model to test the validity of the 21 and 6-items. Validity of the measures relied on the participants’ self-report, hence the potential for information bias. Instruments currently available to measure workplace bullying use self reported data which inherently results in reporting bias as it is impossible to determine if the bullying exposure is perceived or actual. Possible alternative to control information bias include the use of more objective administrative data such as incidents reported, turnover and transfer data. Other limitations include use of a convenience sample of the participants within the public sector which limits the generalizability of the results to the private sector or other work settings.
While the results support using a smaller NAQ-R (6-item version) further analysis using principal component analysis and structural equation models would add to the strength of supporting such a recommendation, taking into account the dialogue on the dimensionality of the construct of workplace bullying and the its latent nature [29, 30]. Longitudinal studies are also needed to demonstrate the reliability and validity of the variation in items (number and questions) over time. Furthermore, item response theory for item selection for reduced versions of NAQ-R also may provide a psychometrically guided decision for item selection.
Conclusion
This paper provided a comparative psychometric review of selected items for a 6-item vs 21-item version of the NAQ-R. This large study of unionized public sector workers documents the reliability and validity of using the 6-item NAQ-R. The study also provides important corroborating evidence that negative acts and bullying reported here have a significant individual impact on intent to remain on the job and being affected personally, and supportiveness of the organizational climate playing a protective role in relation to exposure to negative acts.
The findings of this study support the use of a six-item instrument to measure perceived bullying as it is reliable, valid, and easier to administer. This will facilitate a better understanding of workplace bullying and its consequences while reducing respondent (and organizational) burden.
Conflict of interest
None to report.
Footnotes
Acknowledgments
This paper was supported by Grant Number R01OH009072 from the National Institute for Occupational Safety and Health. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of NIOSH. We thank the agencies and unions that were valuable partners in all aspects of this work. We would also like to express gratitude to the dedicated workers who agreed to participate in this project and who are determined to improve the work environment for all.