Video Data Analysis and Police Body-Worn Camera Footage

BWC Technology and Footage

Law enforcement agencies in the United States have adopted BWCs rapidly over the last 7 years (Suss et al. 2018). Following the deaths of Michael Brown in Ferguson, Missouri and Laquan McDonald in Chicago, Illinois, the White House released its report of the President's Task Force on 21st Century Policing (2015), recommending the widespread adoption of police BWCs in the United States (cited in Crow et al. 2017). Shortly thereafter, the United States Department of Justice began to provide funding for law enforcement agencies to purchase BWCs. In 2020, as a result of the George Floyd murder by police officers in Minneapolis, Minnesota, state legislatures began to mandate the use of BWCs by police (e.g., New Jersey, Ohio, and Illinois) as well as the release of video footage to the public (e.g., California). These measures by Federal, state, and local legislatures were part of the drive to make police more accountable and transparent regarding their actions. In addition, BWCs are suggested to generate a civilizing effect on police when interacting with citizens. That is, their self-awareness is increased because they are on video and so they behave in a more polite manner (Boivin et al. 2017).

While the primary intent of BWCs is to improve police behavior, their existence as a data collection tool cannot be ignored. As with other types of video (closed circuit, phone-based, and private surveillance footage), researchers must acknowledge and confront significant limitations that accompany these data. Furthermore, as a data collection tool, BWCs should be viewed in the same light as other police-related datasets like calls for service, crime incidents, arrests, traffic stops, and field interviews.

Limitations of BWC Footage

Before commencing VDA applications, researchers should recognize the limitations of video at the outset. In accessing the video, researchers are likely to most easily obtain data from police department YouTube channels and perhaps from existing departmental BWC archives or databases (Uchida et al. 2022). The limits of YouTube releases include the unknown editing and redaction that is likely to take place. Rarely are raw, unedited videos released to the public, thus researchers must attend to blurring of faces, cuts, and omitted audio as possibilities in that avenue of access. Those sampling and using data from such publicly available platforms must recognize the need to address issues of neutrality and balance. Specifically, agencies uploading data may not be representative of police in general, and the uploaded elements may, similarly, have bias in curation in terms of presentation and included footage. Critical incidents posted on YouTube by LAPD in 2019, for example, represent 36 extreme incidents selected from among the 4 million videos the agency stored in 2019 and which were not shared in whole or in part with the public (Uchida and Anderson 2021). Additionally, the more general limitations outlined below regarding BWC footage would apply to such uploaded video.

Police departments that are willing to share video with researchers may be inclined to pick cases that are believed to be interesting for the question at hand, but this presents problems of generalizability depending on the research question posed and might approximate a convenience sample or sampling on the dependent variable. Even if researchers can access police data directly and randomly sample videos based on traffic stops or other encounters enumerated in police record management systems (RMS) there are still at least four major problems. First, storage space is an issue requiring some departments to routinely delete video to maintain storage and reduce costs (Solomon, Connor and Schmitz 2021). Police departments have different retention policies for video footage, and in some cases, the footage is automatically purged from the system. This may make some sampled cases unavailable, especially those which do not include an arrest, crime, or use of force, which will trigger video retention. Second, video files may be excluded from view because of ongoing investigations. Sytsma, Chillar and Piza (2021) could not access video for specific cases because internal policies prohibited their viewing of “live” and active investigations. Third, video files that are associated with a sampled event may be missing or “untagged” and hard or impossible to locate. This could be due to officers’ not categorizing their footage correctly, a mismatch within the department's data system between elements and identifiers associated with the video, and other factors (Uchida and Anderson 2021). The fourth problem involves the failure to record a video due to non-activation of a sampled event. In some cases, officers do not turn on their cameras (inadvertently or otherwise), though they may write up an incident or arrest report. Recent research by Katz and Huff (2022), using a sample of Phoenix Police Department data, suggests that non-activation of BWCs is substantial and systematically associated with departmental policies as well as a variety of individual, neighborhood, and situational factors. These gaps speak to important validity issues and suggest that the mundane and currently investigated cases may be systematically missing from sample frames.

Once data are obtained, video and audio quality are often cited as barriers to analysis. In addition, researchers have documented the placement of the BWC as problematic when viewing the footage because they are typically mounted on an officer's chest area and clothing and can be dislodged during scuffles or become obstructed by an officer's arms in some tactical positions (Suss et al. 2018; Terrill and Zimmerman 2022). Moreover, the angle of the body camera cannot capture everything that is occurring in front of an officer and does not really capture video of the officer at all. Mateescu, Rosenblat, and Boyd (2016) concluded that even slight movements by the officer can impact what the BWC captures, and the fisheye panoramic view is not reflective of what the wearer is seeing and thus assessing whether stimuli provoke certain responses could be problematic (Boivin et al. 2017). Since the BWC, if activated, “goes where the action is” it represents a set of tradeoffs between perspective and accessibility (Grimshaw 1982).

The limitations and cautions noted above are recognized, to a varying degree, across the growing body of research using directly accessed BWC footage (see Appendix 1). The initial VBSSO-based research includes Mell's (2016) unpublished dissertation coding procedural justice measures from BWC and Voigt et al.'s (2017) research analyzing police-citizen courtesy using the audio track of BWC during traffic stops. Since those two projects at least seven more have drawn on agency BWC footage, coded it, and published peer-reviewed research, with three projects appearing in 2021. Given the likely growth in use of BWC going forward, a systematic comparison of data derived from this source and IPSSO fills an important gap in the methodological literature.

Finally, the ethical framework from which BWC footage is accessed must be carefully weighed by researchers. The principles of beneficence, respect for persons, justice, and respect for law and public interest apply to the access and use of footage for research purposes. For example, to answer some research questions, the use of footage within a private residence may be prohibited whereas review of BWC from public encounters may be available, due to the concern with privacy and lack of consent of citizens balanced against the benefit of answering the particular research question. Further, interstate and international variations in the protection of individual privacy may supersede research interests. Nassauer and Legewie (2022, Chapter 4) offer a general outline of the balance that should be considered in terms of the benefits of the video usage and the uniqueness of the data collected, analysis performed, and the cost-effectiveness of the research overall against the disadvantages and risks to officers and citizens who are involved.

The VDA Context and SSOs

The comparison of VBSSO to IPSSO, and threats to reliability and validity, can be considered along the three dimensions that Nassauer and Legewie (2019 and 2021) have outlined. For example, the criterion of natural behavior is parallel to that of reactivity in IPSSO. Spano's (2003, 2007) research on IPSSO methodology has, for example, surfaced significant variation in what is observed based on observer gender and time in the field. The impacts he assessed were small but statistically dependable. In comparison, the VDA of BWC lacks a reactivity component that is performative or muted (by the officer impressing an in-person observer or avoiding certain activities) and it is not limited to the observer's single interpretive reference point. That is, multiple persons can view and code video and researchers can determine if codes are reliably applied (Makin, Willits and Brooks 2020) or need to be discarded (Schafer, Hibdon and Kyle 2022). Although the BWC lacks the intrusiveness of in-person observation and has the strength of multiple reviews at lower cost, it does not eliminate or solve the problems that have been raised involving IPSSO research (Spano 2007). For example, mounting of the camera on a participant in the police-citizen encounter might be ascribed as a systematic error or threat to validity of measuring aspects of the police-citizen encounter. This could be akin to that of “over identification” with a research subject's perspective which can be present to a degree within the individual observer. Grimshaw (1982) argues that, for the purposes of addressing some research questions, the type of mount described here can yield important data as it will track officer movements and perhaps be a proxy for attention and focus. However, since mounting is built in to BWC data collection, it is unseen by the video observer, instead perhaps being accepted as “the way things are.” In the context of VDA of BWC, we would frame the concerns regarding data validity and reliability by exploring the extent to which they are not neutral and have suboptimal capture (Nassauer and Legewie 2021).

To explicitly assess the validity and reliability of BWC, we need to consider separate audio and visual channels from which data can be coded. For example, if we are coding physical actions or emotional responses between police and citizen, the process is limited by camera movement and obstruction (Makin et al. 2019). Conversely, the data available to assess officer behavior is largely restricted to verbalization of commands, tone, word choices, and the like, since visual data cannot be ascertained apart from hand movement, gun display, and perhaps the rapidity of body movements. The second officer on the scene, however, would afford opportunities to capture some lead officer behavior and bridge this concern with validity of BWC data. Suspect, victim, or witness behavior may have higher validity in the visual channel since the camera is focused on them, although imperfectly. A complication for VBSSO is establishing a consistent timeline for multiple officer videos, however, timestamps and synchronization may afford reviewers opportunities, with much time cost, for developing accurate timelines and establishing temporal order. Chest-mounted microphones, conversely, likely offer a better record of officer compared to citizen utterances. In sum, we surmise that we would observe differential reliability and validity across these two channels of data that can be drawn from BWC, which may impact the reliability of some common measures of behaviors measured in police-citizen encounters.

The Site

The LAPD serves the second largest city in the United States, Los Angeles, California. With 9,500 sworn personnel and over 2,800 civilian employees, the Department covers the city's 472 square miles. In 2015 the LAPD began its initial deployment of BWCs. Two divisions, Mission and Newton began implementation in September 2015 and are the focus of this study as they were the first two patrol divisions in the Department to receive BWCs. They were specifically selected based on their locations and the differences in their call types and activities. Details regarding both divisions and the overall methodology and the purpose of the larger SSO study are available elsewhere (McCluskey et al. 2019).

Methods

In June 2015, eight observers were trained in a classroom setting on the IPSSO instruments, including discussions of coding protocols, group viewing of vignettes, and a series of field training rides consistent with protocols used in prior studies (Mastrofski et al. 1998). After finalizing all procedures and instruments, observers conducted the initial SSOs in August (Mission Division) and September (Newton Division) of 2015 prior to BWC deployment. Officers were randomly selected for observation in both divisions. The same process was followed in June 2016 for the second wave of observations, after BWC had been deployed for approximately 9 months.

To select the random sample of officers (and up to five alternates) for participation in the SSOs, staff secured a list of all eligible patrol officers with the cooperation of Captains at each Division. The randomly selected officer was identified prior to roll call by the research team (and without input from police personnel to avoid bias). Observers attended division roll calls for all observation shifts and notified the selected officers about the ride-along. For each 6-hour observation period, staff observed the interactions between the assigned officer (O1) and the citizens during each encounter. As LAPD employs two-person patrol cars, each SSO included the randomly selected primary officer (O1) and his/her partner (O2) for that shift. The primary officer observed in the encounter refers to the officer who took the lead in the decision making and had the most interaction with citizens. LAPD tactics, such as assigned roles in “contact and cover” allowed for observers to accurately determine which officer would take the decision making lead prior to the commencement of citizen contact.

Throughout both waves of the SSOs, observers spent 725 h riding with and collecting observational data on the encounters between officers and citizens. A total of 124 ride-alongs (71 from Wave I and 53 from Wave II) were completed for both Newton and Mission Divisions. Completed Wave II rides were then randomly sampled to obtain BWC video from the observed officers to assess and compare data from 102 videos involving 53 events at which IPSSO personnel also collected data. Seven non-encounter events with a total of 13 videos recorded by Officer 1 (O1) and Officer 2 (O2), and 46 citizen encounter events comprising 89 videos were sampled. Duration and distribution of videos across events and encounters are presented in Table 1. VBSSO involved the review of unenhanced video and assessment of the audio and video clarity for each of the 102 videos and coded responses using parallel measures from IPSSO as well as unique questions describing the quality of audio and video rendered by BWC.

OPEN IN VIEWER

Table 1.

Duration and Content of Body-Worn Camera (BWC) Videos Reviewed.

Duration	Frequency	Percentage (%)
Less than 2 min	1	1
2–5:59 min	20	19.6
6–10:59 min	24	23.5
11–15:59 min	21	20.6
16–25:59 min	15	14.7
26–31:59 min	8	7.8
32–45:59 min	9	8.8
46–59:59 min	1	1
60–90:59 min	3	2.9
Total	102	100
Number of videos in event/encounter	Events (non-encounter)	Encounters
	n = 7	n = 46
1	2	7
2	4	37
3	1	0
4		2
Total videos reviewed	13	89

For video reviews, all were conducted on LAPD workstations using a streaming feed to standard desktop monitors. Headphones were used to evaluate the audio stream to isolate reviewers from office background noise. Time spent answering questions on the form ranged from under five minutes (n = 14 videos) to more than an hour (n = 5 videos), with the average time spent being almost 20 min (x̄ = 19.7; SD = 24.7) across the 102 videos (examples of video quality questions assessed by reviewers can be found in Appendix 2).

Analysis and Findings

The series of analyses that follows first examines the BWC audio channel and then assesses the video stream separately to appraise the quality of each and includes the VBSSO assessment of the causes of degradation, noise, or interference in each of the signal channels.

Reliability of the BWC Record

As noted, the reliability of audio and video is unlikely to be equal across officer and non-officer participants (suspects, victims, other first responders, and witnesses) present due to camera and microphone placement. We anticipate, from the extant literature noted by BWC users and coders, that citizen audio will be less reliable when compared to officer audio.

Overall Audio Clarity

Observers conducting video SSO assessed the number of participants’ speech that was audible in the unenhanced BWC reviewed. Participants are defined as officers other than the recording officer, citizens, and other service providers (e.g., emergency medical personnel and other first responders) who are captured by the BWC. As shown in Table 2, the full content of participants’ speech was audible in 19 videos, and most of participants’ content was audible in 45 videos, indicating 63 percent of videos featured full or most of participants’ speech. A separate evaluation of the audio quality of the recording officer, upon which the microphone/camera was mounted was made. Using the same categorical assessment (also in Table 2, right-hand panel) observers coded the BWC footage for the audio clarity of the recording officer's speech in the 98 videos where the recording officer spoke. In 45 cases full audio clarity was noted, 39 more were coded as most of the recording officer's audio was captured, or slightly more than 85 percent of the videos reviewed had fully or mostly discernible audio accounts of the recording officer's spoken words. If we dichotomize officer and participant audibility (1 = full/most, 0 = else) a chi-square test indicates that they are dependent χ² (1, n = 98) = 29.4, p < .001, suggesting that as surmised, the audio channel of the BWC has superior reliability for officers.

OPEN IN VIEWER

Table 2.

Overall Assessment of Audio Quality for Participants and Recording Officer.

Proportion audible	Participants (citizens, officers, and other service providers) (n = 102)		Recording officer (n = 98)
	N	Percentage (%)	N	Percentage (%)
None	1	1.0%	0	0.0
Little	9	8.8%	2	2.0
Some	28	27.5%	12	12.2
Most	45	44.1%	39	39.8
Full	19	18.6%	45	45.9

A related line of inquiry involves enumeration of the sources that led to degradation in the audio clarity. Regarding participants’ speech, any interference was coded, and interference that lasted for more than 25 percent of the entire video was also noted in Table 3. For example, distance between the participant and officer was noted as problematic in 46 of 102 videos, and it presented an issue for 25 percent or more of the video's duration in 22 instances. Background noise presented an issue in 65 videos and was an issue for 25 percent or more of the video's duration in 38 instances. When the presence of background noise is used to predict dichotomized officer audibility χ²(1, n = 98) = 10.4, p < .01, and participant audibility χ² (1, n = 102) = 6.1, p < .01, there is significant dependence. With regard to dichotomized participant audibility, we also find significant dependence with distance χ² (1, n = 102) = 10.5, p < .01, as well as statistical dependence with participant's speech χ² (1, n = 102) = 4.2, p < .05. These are unlikely to be randomly distributed in police encounters. Specifically, background noise is likely a substantial problem in traffic encounters on busy streets, whereas indoor encounters in private spaces are likely to feature minimal distances or extraneous noise. To test this, we dichotomized video into those occurring “on the street” (n = 40) against those not occurring on the street (n = 62, includes indoors and outside, in yards or on porches) and ran a series of tests comparing sources of audio degradation presence to determine if the location is systematically related to problems in the audio channel. Background noise was noted as a problem in 83 percent of on-the-street videos, and the comparison with non-street videos yielded a significant contrast χ² (1, n = 102) = 10.0, p = .002. Similarly, distance was noted as a source of audio problems in street encounters 58 percent of the time, yielding χ² (1, n = 102) = 4.1, p = .043, indicating statistical dependence. Thus, reliability of the audio channel, that is completeness due to absence of interference, is systematically linked to location and encounter types. If we return to the differential reliability for the recording officer as compared to participants, we find the street encounters account for a significant proportion of officer audio loss χ² (1, n = 98) = 4.1, p = .043, but the difference, although in the same direction, is not statistically dependable for participants χ² (1, n = 102) = 3.0, p = .086.

OPEN IN VIEWER

Table 3.

Sources of Audio Quality Degradation for Participants and Recording Officer.

Interference source	Participants (citizens, officers, and other service providers) (n = 102)				Recording officer (n = 98)
	Any		>25%		Any
	N	Percentage (%)	N	%	N	Percentage (%)
Distance	46	45.1	22	21.6%	N/A	N/A
Background noise	65	63.7	38	37.3%	45	45.9
Obstructed mic	5	4.9	3	2.9%	3	3.1
Speech volume	38	37.3	9	8.8%	N/A	N/A
Radio volume	N/A	N/A	N/A	N/A	19	19.4
Other	5	4.9	4	3.9%	19	19.4
Two or more problems	58	56.9	19	18.6%	29	29.6

Overall Video Clarity

The quality of the video obtained from the cameras in 102 videos was assessed in two primary ways. First, lighting, color, light sources, and obstructions in all 102 videos were evaluated. This allows for a sense of how often BWC video suffers from quality issues in the sample and yields a suboptimal data source in the form of the unenhanced video. Results of the initial quality assessment are presented in Table 4. Second, the adequacy of video for describing citizen participants’ characteristics (e.g., apparent age, sex, and race/ethnicity) as judged by the coders and factors reducing this capacity were explored.

OPEN IN VIEWER

Table 4.

Overall Assessment of Video Quality (n = 102).

	N	Percentage (%)
Quality of lighting
Ideal	32	31.4
Inconsistent	57	55.9
Unhelpfully bright	3	2.9
None/Outline only visible	10	9.8
Color rendered
Vibrant	22	21.6
Easily discernible	24	23.5
Dull	37	36.3
Black and White	18	17.6
Footage indiscernible	1	1.0
Light sourcesa
Sunlight	56	54.9
Streetlight	31	30.4
Flashlight	18	17.6
Headlights	7	6.9
Indoor lighting	5	4.9
Obstruction occurred
Any obstruction	29	28.4
Obstruction >25% of video	10	9.8
Primary types noted
Officer's body part	8	7.8
Officer's clothing	7	6.9
Officer's equipment	6	5.9
Seatbelt	4	3.9

Note: ^aMultiple light sources are possible in a single video, hence the total sums to more than 100%.

Coders noted obstructions that concealed BWC footage in 28 percent of videos (n = 29) and obstructions that persisted for more than 25 percent of the duration of the video that occurred in 10 instances (10%). The primary sources of obstruction were noted to be officer's body parts, clothing, and equipment.

Additional examination of the video quality assessed the adequacy of video with respect to determining citizen characteristics. Citizen participants (not including officers or other service providers, but victims, suspects, and people in need of assistance that call 911) were visible in 89 videos in the sample. These videos were isolated in our analysis and VBSSO coders assessed whether the video was of sufficient quality to make a description of citizen participants (Table 5).

OPEN IN VIEWER

Table 5.

Video Quality for Determining Citizen Participants’ Characteristics and Sources of Interference (n = 89).

	N	Percentage (%)
Citizen characteristics in video
Not discernible	8	9.0
Guess characteristics	55	61.8
Confident description possible	26	29.2
Sources of interferencea
Lighting	36	40.4
Distance	28	31.5
Camera angle	44	49.4
Obstruction	6	6.7

Note: ^aMultiple interference sources are possible in a single video, hence the total sums to more than 100%.

Coders determined that 9 percent of videos (n = 8) yielded video footage that allowed no identification of citizens from the recording. In 62 percent of cases (n = 55), the observer determined that citizen characteristics such as apparent gender, age, and ethnicity/race could be guessed (not confidently) by viewing the video. Finally, the sources of interference that contributed to the inability to confidently discern citizen characteristics were noted. Lighting (40%, n = 36) and distance (32%, n = 28) were common causes of interference with identification of citizen characteristics, however, surprisingly, camera angle, in nearly half of the videos (n = 44) was noted as an obstacle to making a confident description. If we dichotomize videos into those where confident description can be made (n = 26) compared to those of lower quality (n = 63) we find that confident descriptions are possible in 35 percent of non-street videos, but 20 percent of on-the-street videos χ² (1, n = 89) = 2.4, p = .124, consistent with findings above with respect to direction, but not statistically significant.

Time of day (lighting) and outdoor encounters (distance) may systematically be correlated with what VBSSO can derive from the BWC data source, suggesting that triangulation of citizen characteristics, from official police records for example, would be an important consideration for researchers.

Comparison of IPSSO and VBSSO

As noted, VBSSO and IPSSO have addressed parallel questions, but have not assessed concordance and discordance across modalities. To address this gap, the 46 encounters from IPSSO and VBSSO were aligned for a comparison across commonly measured SSO elements. The IPSSO had nested elements of citizens coded within encounters allowing for coding age, race, gender, and apparent social class, and assessing each observed citizen's interaction (e.g., whether arrested, handcuffed, searched, and disrespected) with officer/police actions tied to each individual in the IPSSO. The encounter-based VBSSO did not allow for citizen-level assessments to the same degree because of difficulties ascertaining who was speaking, citizen characteristics, and the like, as noted in the previous section. Thus, the unit of analysis for VBSSO was the encounter itself (an aggregate of all citizens interacting with police at a particular event). VBSSO posed questions of whether police arrested anyone, if officers were disrespectful toward any citizen, if any citizens were in conflict, and so on. Aggregation of IPSSO citizens’ experiences up to the common level of encounter was used to better match and merge across observation conditions. Based on the results from the reliability study above we surmise that SSO elements reliant on spoken words will have higher concordance for officer and citizen comparisons and less concordance will be observed for elements reliant on the visual record.

The second issue regarding comparability stems from the IPSSO being tasked to follow the “lead decision-maker” in the two-officer team during an encounter. Due to LAPD's “contact and cover” tactical approach, this was often clearly stated when officers began a shift and if it was to be altered it was decided prior to exiting the patrol car. The video, in contrast, offers no such clear signifier to a viewer, and the researcher conducting VBSSO was not privy to the coding of the IPSSO and was never the same observer. This accommodation of aggregation is one that may yield noncomparable results when there are many citizens on-scene and some interact extensively with the “cover” officer. This will be viewable on video, but likely not a focal point for the IPSSO coder, tasked with trailing the decision maker upon exiting the vehicle. Regardless, the two sources should allow for preliminary assessment of IPSSO and VBSSO illuminating areas of strength and weakness for the latter in terms of its value as a platform for future VDA.

To assess the correspondence between VBSSO and IPSSO, a series of parallel dichotomous indicators were coded from the observation data for each of the 46 encounters. The items are consistent with the current areas being addressed by VBSSO research including indicators of police use of coercion (e.g., police verbal threats), procedural justice (e.g., police displays of respectful treatment toward citizens), and citizen states and behaviors (e.g., citizens being apparently intoxicated). These data are presented in the forms of comparison of coding (1 = yes, 0 = no, not present) whether the presence of an indicator was observed in each form of SSO. Next, a concordance measure, which computes the percentage of agreement between IPSSO and VBSSO is computed. Finally, the last two columns present Gwet's Agreement Coefficient (AC) which approximates a measure of observer agreement correcting for chance concordance and the lower bound of Gwet's AC point estimate within the 95% confidence interval (Gwet 2014, 2021; Klein 2018). The comparison measures include three domains where both modalities of SSO have played a role including police legal and coercive actions, procedural justice, and citizen states and behaviors.

The IPSSO and VBSSO comparison commences with legal and coercive actions presented in the top panel of Table 6. Results indicate that, in reference to Landis and Koch's (1977) criteria for benchmarking agreement, the paired observations have substantial to almost perfect agreement regarding whether an encounter involved an arrest, handcuffing, reaching for or brandishing a weapon, whether police issued a threat to a citizen, or had a firm grip hold on a citizen. Finally, whether the police conducted a search had only fair agreement across the SSO modalities, however, IPSSO observation yielded nearly twice as many encounters (n = 17) as VBSSO (n = 9) where a search was determined to have been conducted. The VBSSO, with multiple records in two-officer units, does offer a comprehensive view of police interaction on a scene, rather than the IPSSO which is limited to a primary focus on the single officer. Nevertheless, a general pattern emerges, that IPSSO generally yields more instances of the presence of each indicator in the encounters in the sample. We cannot discern whether this is definitively overcounting IPSSO or undercounting VBSSO, but it is suggestive of a divergence in the methods.

OPEN IN VIEWER

Table 6.

Legal/Coercive Actions, Procedural Justice, and Citizen States in Police-Citizen Encounters (n = 46).

	In-person	Video-based	Concordance	Gwet's AC	Lower bound 95% CI
Police legal/coercive actions
Arrest	7	6	93.5%	0.91	0.81
Handcuff	13	11	87.0%	0.79	0.61
Search	17	9	78.3%	0.63	0.40
Police reach for weapon	6	3	89.1%	0.87	0.74
Police brandish weapon	4	2	95.7%	0.95	0.88
Police threat to citizen	2	0	95.7%	0.95	0.89
Police firm grip C	2	2	91.3%	0.91	0.80
Procedural justice elements
C Participation in encounter	36	35	76.1%	0.63	0.40
P display neutrality	21	12	71.7%	0.48	0.20
Police respect	35	33	69.6%	0.50	0.23
Police disrespect	2	3	89.1%	0.88	0.76
Citizen states/behaviors
Injured/ill citizen	4	4	91.3%	0.90	0.83
C mental illness	6	2	82.6%	0.79	0.63
C Alc/drug use	3	0	93.5%	0.93	0.85
Any C conflict with police	14	11	80.4%	0.68	0.46
Citizen disrespect	7	5	82.6%	0.78	0.60
Citizen respect	33	37	73.9%	0.59	0.34

Procedural justice (coding exemplars can be found in Appendix 2) is captured in four measures in the middle panel of Table 6. The concordance of police disrespect is high, and Gwet's AC indicates substantial agreement on that measure. Participation, neutrality, and indicators of police respect, however, only generate slight to fair levels of agreement as measured by the lower bounds of Gwet's AC. This is at odds with our supposition that measures rooted largely in the spoken word would show higher concordance and suggests that subjectivity may also be higher in these indicators.

The last domain of measures, citizen states, and behaviors, includes six indicators of citizen state, presented in the bottom panel of Table 6. These were coded with regard to whether any citizen present showed evidence of injury or illness, apparent mental illness, influence of alcohol or drugs, conflict with police, disrespect toward police, or respectful behavior toward police. In comparisons of citizen states across SSO conditions, almost perfect agreement is found in identifying the presence of injured or physically ill citizens in encounters and regarding evidence of drug or alcohol use. In the latter, however, VBSSO yielded no cases where citizens had apparent alcohol or drug use compared to three in IPSSO, offering some evidence that the methods diverge in this area. Moderate levels of agreement were obtained for whether a citizen was observed to be mentally ill, disrespectful, or in conflict with the police. Finally, citizen respect showed a fair level of agreement, according to the statistical benchmark, and was one measure, along with police disrespect, where VBSSO produced a greater number of observed indicators when compared with IPSSO.

The single-item comparisons of concordance suggest that the complexity of encounters and the quality of BWC audio/video may contribute to discrepancies between the SSO modalities. Though the sample is small (n = 46), indices of concordance were calculated for police coercive actions (summing seven binary concordance scores for handcuffing, displaying weapon, and other action.; x̅ = 6.3), four indicators of procedural justice concordance (x̅ = 3.07), and six indicators of citizen states and behavior concordance (x̅ = 5.04) to generate three dependent measures in Table 7. These measures were then regressed on indicators of BWC quality and the complexity of the coded encounter (results are not shown, Supplemental materials outline measures and results are shown in Appendix 3). Two measures of BWC quality focus on contrasting those cases with 25 percent audio interference (17% of cases) and if any video was obstructed for 10 percent or more of the encounter (28% of cases). Four measures of complexity reflect the count of citizens (x̅ = 2.98) and police on scene (x̅ = 3.26), binary measures of whether the encounter occurred on the street (39% of cases), and whether the encounter lasted less than 15 min (35% of cases) round out this domain representing the complexity and possibly attention fatigue for observers.

OPEN IN VIEWER

Table 7.

Descriptive Statistics: Concordance Measures, Audio/Visual Problems, and Complexity (n = 46).

	Source data	Min	Max	Mean	St. Dev.
Dependent measures
Police coercive Actions	VB and IP	2	7	6.30	1.17
Procedural justice (PJ)	VB and IP	0	4	3.07	1.04
Citizen sates and behavior	VB and IP	3	6	5.04	1.11
High subjectivity index	VB and IP	1	6	4.78	1.38
Low subjectivity index	VB and IP	1	6	5.35	1.06
Independent measures
Audio problems (GT 25%)	VBSSO	0	1	0.17	0.38
All video 10% or more obstruction	VBSSO	0	1	0.28	0.46
Number of police on scene	IPSSO	2	14	3.26	2.38
Number of citizens on scene	IPSSO	1	15	2.98	2.76
IPSSO <15 min	IPSSO	0	1	0.35	0.48
Street encounter	VBSSO	0	1	0.39	0.49

Note: IPSSO = pioneered in-person systematic social observation; VBSSO = video-based systematic social observation.

Ordinary least squares (OLS) regression models were estimated on all three concordance indices, with resulting dot-and-whisker plots presented for models using point estimates and unadjusted standard deviations as shown in Figure 1. Bonferroni adjustment of p-values yielded only one statistically reliable estimate, with the number of police on scene having a substantially negative impact on concordance for police coercion. Given the small sample size, patterns of coefficient signs offer a second evaluative approach. Specifically, all six audio/video problem estimates indicate reduced concordance, which is a pattern consistent with expectations. Shorter encounters, those less than 15 min suggest positive concordance, consistent with lower complexity, and the pattern of street encounter coefficients suggest the anticipated attenuation of concordance.

OPEN IN VIEWER

Figure 1.

Estimates and unadjusted standard deviation dot-and-whisker plots for ordinary least squares (OLS) coefficients.

An alternative approach to comparing the VBSSO and IPSSO concordance is to consider that individual items vary along the dimensions of high subjectivity and low subjectivity. Put differently, some items require applying a more complex set of decision rules to observed behavior and might be considered latent items, compared to explicit or manifest measures with lower subjectivity. To address this difference (which is embedded in items in various indices), we examined the content of definitions for indicators. To account for subjectivity, we opted for six-item scales reflecting higher or lower subjectivity as shown in Table 7. Lower subjective items include: arrest, handcuffing, brandish weapon, reach for weapon, any citizen injury, and whether a search occurred. Higher subjective items include citizen displays of respect and disrespect, police displays of respect and disrespect, whether citizens exhibited signs of mental illness, and whether any citizens displayed conflict toward the police. The pattern of findings, reflected in the lower two plots as shown in Figure 1, suggests a similar pattern to the regressions noted above. However, there is no statistically significant predictor of either index once Bonferroni correction adjustments were applied to significance levels.