Evaluation Review

Involvement in bullying at school is detrimental to students’ mental and physical health; however, school antibullying programs have not been found to be uniformly successful. Self-reported frequency of involvement in bullying victimization and perpetration, often used as outcome measures in intervention impact evaluation studies, may be subject to response shift, particularly in intervention conditions. Such differential shifts could lead to biased estimates of intervention effects.

This study investigated the presence of reconceptualization, reprioritization, and recalibration response shift, resulting from intervention implementation.

The study subjects were Grade 8 students (n = 3,382, 53% female) in the 35 schools participating in the Cyber Friendly Schools Project, a longitudinal group-randomized intervention trial. Response shift was assessed by comparing traditional and retrospective pretest measures of bullying involvement, as well as testing for measurement invariance over time in the Forms of Bullying Scale (FBS) using confirmatory factor analyses.

No evidence of response shift was found, indicating students’ understandings of bullying behavior remained stable over time. These findings also demonstrate the applicability of the FBS in longitudinal studies involving adolescents.

While response shift was not present in our study, researchers conducting program evaluations in other contexts are advised to consider testing for this potential source of bias in their studies.

The impact of surveying on individuals’ behavior and decision making has been widely studied in academic literature on market research but not so much the impact of monitoring on economic development interventions.

To estimate whether different monitoring strategies lead to improvement in participation levels and adoption of best practices for coffee production for farmer who participated in TechnoServe Agronomy Training Program in Rwanda.

Farmers were identified randomly for monitoring purposes to belong to two different groups and then selected depending on the additional criterion of having productive coffee trees. We estimate treatment-on-the-treated and intention-to-treat effects on training attendance rates and farmers best-practice adoptions using difference-in-differences estimation techniques.

Farmers were randomly identified to a high or low monitoring with different type and frequency of data collection and selected if they had productive coffee trees as part of the monitoring strategy.

Attendance to training sessions by all farmers in the program and best-practice adoption data for improving coffee yield.

We find that monitoring led to surprisingly large increases in farmer participation levels in the project and also improved best-practice adoption rates. We also find that higher frequency of data collection has long-lasting effects and are more pronounced for low-attendance farmers.

Monitoring not only provides more data and a better understanding of project dynamics, which in turn can help improve design, but can also improve processes and outcomes, in particular for the least engaged.

Recent technological advances have much potential for improving police performance, but there has been little research testing whether they have made police more effective in reducing crime.

To study the uses and crime control impacts of mobile computing technology in the context of geographically focused “hot spots” patrols.

An experiment was conducted using 18 crime hot spots in a suburban jurisdiction. Nine of these locations were randomly selected to receive additional patrols over 11 weeks. Researchers studied officers’ use of mobile information technology (IT) during the patrols using activity logs and interviews. Nonrandomized subgroup and multivariate analyses were employed to determine if and how the effects of the patrols varied based on these patterns.

Officers used mobile computing technology primarily for surveillance and enforcement (e.g., checking automobile license plates and running checks on people during traffic stops and field interviews), and they noted both advantages and disadvantages to its use. Officers did not often use technology for strategic problem-solving and crime prevention. Given sufficient (but modest) dosages, the extra patrols reduced crime at the hot spots, but this effect was smaller in places where officers made greater use of technology.

Basic applications of mobile computing may have little if any direct, measurable impact on officers’ ability to reduce crime in the field. Greater training and emphasis on strategic uses of IT for problem-solving and crime prevention, and greater attention to its behavioral effects on officers, might enhance its application for crime reduction.