Farmer Field Schools for Improving Farming Practices and Farmer Outcomes: A Systematic Review

Intermediate and final outcomes

We included all eligible studies to answer review question (1), irrespective of whether they reported impacts on intermediate or final outcomes. The review focused primarily on agricultural outcomes, including agricultural yields (production per unit of land), and net revenues (profits per unit of land) as indicated in the study protocol (Waddington et al., 2012a). We examined secondary intermediate outcomes including farmer knowledge and capacity and adoption of new approaches (e.g. reduction in pesticide use). We also examined secondary final outcomes including health, environment and empowerment outcomes such as feelings of self-esteem. Table 1 shows the different ways of measuring outcomes reported in included studies. Data on outcomes along the theory of change are reported outcome by outcome, for all studies reporting that particular outcome. Unless otherwise specified, outcome data are not reported along the theory of change by individual study.

Other data

To answer review question (2) we included evidence on barriers to and enablers of FFS effectiveness, diffusion and sustainability. This included process and implementation information together with measures of beneficiaries' attitudes and experiences with FFS.

Review question (1): What are the effects of farmer field schools on intermediate and final outcomes, for FFS particiants and neighbour farmers?

Studies eligible for inclusion in the quantitative synthesis used experimental or quasi-experimental study designs. Study designs which collected longitudinal data at baseline and endline and those using cross-sectional (endline) data only were included. In addition, data needed to be collected at the farm or household level contemporaneously in both groups. Studies that used the following methods of allocating FFS to participants were eligible:

allocation rules based on prospective randomised or quasi-randomised (e.g. alternate) assignment (randomised controlled trials or RCTs, and quasi-RCTs);

We included studies which used statistical matching (e.g. propensity score matching or PSM, or covariate matching), regression adjustment (e.g. difference-in-differences or DID, and single difference regression analysis, instrumental variables or IV, estimation and Heckman selection models), as well as other cross-sectional or longitudinal designs which used less rigorous approaches. Given the breadth of designs included, we conducted rigorous assessment of internal validity based on risk of bias categories (see below).

Excluded studies are those which did not use a comparison group design, or employed less than a six-period ITS design. For example, Tin (2009) used a pre-test post-test design with no comparison group, and Armen et al. (2009) collected post-test data among field school participants only (see Appendix B).

Review question (2): What are the enablers of and barriers to FFS effectiveness, diffusion and sustainability?

We included qualitative studies and studies using descriptive statistics which met the following criteria:

reported on interventions as identified as “farmer field schools”, although not necessarily the same interventions as those included in the review of effects (review question 1);

We adopted a two-stage approach to inclusion of the qualitative studies, which, in addition to removing studies based on the usual relevance criteria (intervention, population, relevance to research question, study type and location), removed studies of particularly low quality in the first round (Thomas et al., 2003; Spencer et al., 2003), using the criteria set out in point 4 above. We then assessed the quality of the included studies using a detailed quality appraisal checklist in the second round, as described below.

Given the limited reporting of programme and contextual characteristics in the impact and qualitative evaluation literature, in the final stages of the review we systematically searched for implementation documentation (see Appendix A for details), collecting data on project, programme and implementation characteristics which we linked to the impact evaluations in order to conduct more in-depth analysis of moderators. This analysis was conducted a posteriori.

Review question (1): Critical appraisal of studies of effects

Studies were critically appraised according to risk of bias in internal validity and external validity (generalisability), and publication bias. The assessment of risk of bias was based on: 1) quality of attribution methods (addressing confounding and sample selection bias); 2) the possibility of spillovers to farmers in comparison groups; ¹² 3) outcome and analysis reporting biases; and 4) other sources of bias (Appendix C). Risk of bias was assessed on both study design and implementation of the evaluation methodology. “Low-risk-of-bias” studies were identified as those in which clear measurement of and control for confounding was made, including selection bias; there were no sources of unobserved confounding which were likely to affect our degree of confidence in the findings; intervention and comparison groups were described adequately (in respect of the nature of the interventions being received) so that risks of spillovers or contamination were small; and where reporting biases and other sources of bias were unlikely. We rated the likelihood of spillover effects as low when comparison farmers were living in a different village from the farmer field school participants.

Studies were identified as “medium risk of bias” when there were moderate threats to validity of the attribution methodology, or likely risks of spillovers or contamination (arising from inadequate description of intervention or comparison groups or possibilities for interaction between groups such as when they are from the same community), or possible reporting biases.

“High risk of bias studies” were all other studies, including those where the study design was of questionable internal validity (such as those where comparison groups were not matched on observables, differences in covariates were not accounted for in multivariate analysis or where there were serious threats to the validity of the statistical procedure used to deal with attribution), or where there was evidence for spillovers or contamination, and reporting biases were evident. Two reviewers (JH and HJW) undertook the critical appraisal, the results of which are presented in Appendix F. ¹³

The assessment of external validity of studies (generalisability) was based on two sources of information. Firstly, data were collected on the sampling methodology to assess whether the whole population or a random or purposive sample of FFS farmers and neighbours were covered. Second, the socioeconomic and demographic characteristics of farmers in the included studies were compared to average characteristics of farmers to assess whether FFS farmers were more or less representative of farmers in developing countries. We attempted to reduce publication bias by searching for and including unpublished studies in the review. However, we also assessed the likelihood of file-drawer effects resulting from selective outcome reporting and conducted statistical tests for under-reporting of small sample studies with negative or insignificant findings using funnel plots and Egger's (Egger et al., 1997) tests for those outcomes with more than ten observations.

Review question (2): Critical appraisal of studies examining barriers and enablers

We assessed the quality of included studies using an adapted version of the Critical Appraisal Skills Programme checklist (CASP, 2006), making judgments on the adequacy of reporting, data collection, presentation, analysis and conclusions drawn. The checklist is presented in Appendix C. In accordance with our inclusion criteria we filtered out studies of particularly low quality (Hannes, 2011) and studies where questions 1–5 on the checklist were assessed as “No” were excluded at this stage. The critical appraisal was conducted by two reviewers independently (results reported in Appendix F). Any discrepancies were resolved through discussion, with a third person acting as an arbitrator.

Calculation of effect size

We calculated effect size estimates, standard errors and 95 per cent confidence intervals using data provided by included studies, where possible. We report two types of effect sizes in this review. All outcomes were measured using continuous variables. We calculated response ratios to measure effects, which are expressed as the difference in outcome in the intervention group as a proportion of the outcome in the comparison group. The response ratio (RR) is centred around 1, which is the point of “no effect”; the distance above and below the no-effect point are translatable as percentage changes in the outcome in the treatment group over the comparison, giving the same interpretation as a risk ratio. Thus, an RR of 1.10 translates as a 10 per cent average increase in the treatment condition, while an RR of 0.90 would translate as a 10 per cent average reduction.

The response ratio has the twin advantages of ease of interpretation and ease of calculation, since it requires less information than the standardised mean difference to compute. However, the use of the response ratio is subject to limitations for some variables, since it is only meaningful when the outcome is measured on a true ratio scale that has a natural zero point, although is unlikely to be equal to zero in practice (Borenstein et al., 2009). Thus, we calculated RR for some of the outcomes, including pesticide use, environmental measures estimated from pesticide use, production yield and revenues, and variables measuring probabilities such as disease incidence, and empowerment. In the cases of knowledge test scores and indices of adoption of practices, we estimate the Hedges' g (sample size corrected) standardised mean difference (SMD) which measures the effect size in units of standard deviation of the outcome variable.

We have used appropriate formulae to calculate zero-order and partial effect sizes, respectively, from (unadjusted) bivariate and (adjusted) multivariate specifications. Appendix D provides details of all effect size calculations.

However, caution is needed regarding the synthesis since most of the studies included in the review of effects, including all of the studies assessed as being of medium risk of bias, have used multivariate specifications in their analyses. As such, the meta-analysis synthesises both partial and bivariate effect sizes. However, effect sizes are only strictly comparable in studies employing a common model, meaning that “suitable proxies for the same constructs [i.e. the outcome variable, treatment variable and covariates] are included in all the studies being synthesized” (Keef & Roberts, 2004, p. 103). This is due to the degree of correlation between the variables included in the model, over and above any problems due to model misspecification. In other words, the partial effect size based on the regression coefficient measures the treatment effect “holding all other variables constant”. It is therefore measuring a different quantity to the bivariate relationship where the treatment effect is correlated with the other explanatory variables. The partial and bivariate effect sizes are equal only where coefficients of explanatory variables are the same in treatment and comparison groups (the “constant slopes” assumption), as would be indicated, for example, by insignificant interaction terms between treatment variable and covariates (ibid.).

As indicated in Appendix D, several solutions have been proposed to address this problem. Becker and Wu (2007) notably propose drawing on the variance–covariance matrix from the included studies to estimate generalised least squares meta-regression analysis. However, there are several problems in applying Becker and Wu's solution. First, not all multivariate models control for the same covariates, nor should models estimated for different study designs using data collected in different contexts necessarily do so. And where they do use common covariates, the variance–covariance matrices are seldom reported and difficult to obtain. Indeed, we were not able to identify any included studies in this systematic review which reported the full variance–covariance matrix.

We have opted not to eliminate studies which did not report bivariate effect sizes from the synthesis, due to the loss of information this would entail and the likely high risk of bias in included studies which did not use multivariate specifications (since none of the studies used randomised assignment or similar design-based methods to reduce biases). While the risk of bias assessment evaluated likely specification errors, included studies did not report diagnostic test statistics for multicollinearity or results of models estimated with full interaction terms. Our RR and g effect size calculations therefore implicitly assume zero correlation between treatment effect variable and other covariates in the model. However, many of the commonly used covariates (socioeconomic status, demographic characteristics and location) in the farmer field schools literature are likely to be correlated with the treatment, violating the constant slopes assumption. With the aim of partially ameliorating this problem, where different studies did report bivariate and partial effect sizes for a particular outcome, we have conducted sensitivity analysis to examine evidence for systematic differences (with the corollary that these are indirect comparisons across studies, not direct comparisons of bivariate and partial effect sizes within the same study).

Dependent effect sizes

We only included one effect estimate per study in a single meta-analysis. Where multiple outcomes were reported from alternate specifications, we selected the specification according to likely lowest risk of bias in attributing impact, for example the most appropriately specified regression equation, which may in some instances be the least parsimonious. In some cases, where studies reported multiple dependent effect sizes, for example according to different outcome sub-groups (such as the paper by Ricker-Gilbert et al., 2008, which reported simple, intermediate and complex knowledge), we calculated a “synthetic effect size” based on the sample-weighted average, using appropriate formulae to recalculate variances according to Borenstein et al. (2009, chapter 24), making covariance assumptions as necessary (more information is available in Appendix D). We used the same approach where studies reported multiple effect sizes according to different follow-up periods, although we also discuss in the report differences in reported follow-ups for those studies which did so. Where studies reported multiple effect sizes according to sub-groups of participants, we report data on relevant sub-groups separately (as in the case of FFS participants and neighbours).

We report data in the meta-analysis according to the paper in which the effect size originated. However, we attempted to avoid synthesising dependent effect sizes from multiple studies in any single meta-analysis by linking papers prior to analysis, and conducting sensitivity analysis as necessary. This was deemed necessary in the case of Feder et al. (2004) and Yamazaki and Resosudarmo (2008), which replicate analysis of the same data. In a number of cases, information has been collected on the same programme at the same or different periods of time, but in most cases this did not cause problems for the analysis since the papers either reported data on different outcome constructs which are eligible for inclusion in separate meta-analyses, or they did not report sufficient information on standard deviations or outcome means to extract effect sizes.

For example, David and Asamoah (2011) reported knowledge outcomes based on a survey of farmers in 2007-08 in Ghana, while Gockowski et al. (2010) reported on adoption and yields for farmers in the same programme in 2005; David (2007) and Wandji et al. (2007) also reported outcomes in Cameroon for the same multi-country programme in West Africa as Gockowski et al. (2005), although the latter only reported impacts on adoption and yields in the project areas of Nigeria. Chi et al. (1999) and Murphy et al. (2002) both collected data on the FAO National IPM Programme in Vietnam, but only Murphy reported sufficient information to estimate the effect size. Mutandwa and Mpangwa (2004) and Maumbe and Swinton (2003) both reported on cotton IPM-FFS in Sanyati District, Zimbabwe, but only Mutandwa and Mpangwa presented statistical information. Dankyi et al. (2005) and Carlberg et al. (2012) reported different outcomes for the same programme, as did Reddy and Suryamani (2005) and Pananurak (2010) for the same Indian programme; DANIDA (2011) and Islam et al. (2006) reported data on the same Bangladesh programme but for different outcomes and different years. Similarly, Friis-Hansen and Duveskog (2012) and Friis-Hansen et al. (2004) collected data on empowerment and adoption outcomes, respectively, for the FAO's East African IPPM-FFS Pilot Project in Uganda, while Davis et al. (2012) collected data on agricultural outcomes for the expansion phase of the same project. Torrez et al. (1999) reported on the pilot of the IPM programme for potatoes in Bolivia evaluated in Bentley et al. (2007), although Bentley and colleagues do not report sufficient information on sample distribution to calculate effect sizes.

There are cases, however, where effect sizes were likely to be dependent. ¹⁴ Hiller et al. (2009) and Waarts et al. (2012) both reported on the same outcomes for a pilot and up-scaled tea IPPM project in Kenya, and Pananurak (2010) and Wu (2010) reported, respectively, short-term (one-year follow-up) and medium-term (three-year follow-up) outcomes from the same survey of households in China for yields. In these cases we included results for the scaled-up project (Waarts et al., 2012) and longer-term follow-up (Wu, 2010) in meta-analysis, discussing differences by follow-up time period in the report narratively. ¹⁵ In the case of the Indonesian National IPM Programme, two papers reported analyses of the same data (Feder et al., 2004a; Yamazaki & Resosudarmo, 2008). Given the prominence of these papers in the literature, in this case we opted to assess sensitivity of meta-analysis findings to inclusion of these studies rather than calculate a synthetic effect size (Appendix G reports results from sensitivity analysis excluding these studies; results are not affected).

Rejesus et al. (2010) and Huan et al. (1999) estimated impacts on the National IPM Programme in Vietnam, collecting data at different points in time. Only a subset of observations appeared to be from the same province of Long An. However, we used the results from Rejesus on the grounds of preference according to risk of bias, dropping the higher-risk-of-bias observations in Huan et al. (1999). Finally, there were two instances of studies which reported two measures of the same adoption construct – Yang et al. (2005) and Khan et al. (2007), which both reported pesticide use and cost outcomes – in which we included only pesticide costs in the pooled meta-analysis of pesticide adoption.

We have identified effect sizes in the meta-analysis forest plots by author citation and country location in the main text. However, we also show the same forest plots identified by programme name in Appendix G, so that readers can see which FFS programmes are included within each meta-analysis.

Unit of analysis

We provided an assessment of the unit of analysis error for the included studies based on whether the included studies account for differences in demographic and socioeconomic household and village characteristics across individuals in different clusters (Appendix F). For those studies that reported moderate or high probability of relevant unit of analysis error, corrections were applied to the standard errors and the confidence intervals of the effect size using the following formula (Higgins & Green, 2011):

S E c o r r e c t e d = S E u n c o r r e c t e d * √ [ 1 + ( m - 1 ) * I C C ]

where m is the number of observations per cluster and ICC is the intra-cluster correlation coefficient, which we assume to be o.05. ¹⁶ We used this formula to correct for likely unit of analysis errors in four studies (Bunyatta et al., 2006; Palis, 1998; Rejesus et al., 2010; Ricker-Gilbert et al., 2008). The nature of reporting on cluster assignment in the original studies in the majority of cases meant that we were not able to reach firm conclusions about unit of analysis errors. We therefore conducted sensitivity analysis in which additional studies assessed as being of “unclear risk” were corrected for unit of analysis errors (Appendix G).

Review question (1): Effectiveness synthesis and sensitivity analysis

We synthesised quantitative information on impacts using inverse-variance weighted statistical meta-analysis. We implemented random effects meta-analysis because we can reasonably expect effect sizes to differ across studies due to a range of factors including contextual variation (e.g. relating to the location, type of crop, beneficiary groups, intervention design and implementation process and follow-up period) and study design, over and above the effects of chance alone on findings. Random effects meta-analysis produces a pooled effect size with greater uncertainty attached to it, in terms of wider confidence intervals than a fixed effect model.

We reported effect sizes on predefined sub-groups of interest, including FFS participants and neighbouring farmers potentially exposed to knowledge, to assess the extent of spillovers. We also investigated sources of effect heterogeneity according to contextual moderators and factors relating to study design. As noted above, there are important issues relating to comparability of bivariate and partial effect sizes (Keef & Roberts, 2004; Becker & Wu, 2007; Aloe & Thompson, 2013), due to likely collinearity between treatment variable and commonly used covariates such as socioeconomic status and demographic characteristics. We therefore conducted sensitivity analyses to examine whether there were any systematic associations with effect size magnitude.

Effect sizes shown using forest plots were synthesised in inverse-variance weighted random effects meta-analysis, estimated using Stata software (Stata Corporation, College Station, TX, USA). All response ratio analyses were conducted using the natural logarithm of the effect size, with results exponentiated back to the response ratio metric for forest plots and discussion.

Review question (2): Barriers and enablers synthesis

For the synthesis of evidence relating to question (2), we conducted the synthesis in two stages, using the hypothesised programme theory as our overall framework throughout. After having completed the detailed coding of all of the included studies we re-reviewed the coding and identified descriptive findings which remained close to the findings in the primary studies (following Thomas & Harden, 2008). We summarised the descriptive findings across studies, using headings corresponding to the key stages of the FFS programme theory (and the categories used for data extraction) to structure our synthesis. The full text of this synthesis is provided in Appendix H.

The descriptive synthesis provided the basis for a more analytical synthesis. We analysed the descriptive findings across studies in detail, and identified themes relating to potential barriers and enablers of FFS effectiveness. We then reviewed and compared the original FFS programme theory with the emerging evidence on barriers and enablers, and revised the theory and assumptions to reflect the findings of our synthesis.

Integrated synthesis (review questions 1 and 2)

We used the programme theory (Figure 4) as a framework for integrating the findings from the two syntheses (Noyes et al., 2011) with the aim of providing an integrated narrative synthesis along the causal chain addressing the objectives of the review. We used an iterative approach, in which the intermediate outcomes and assumptions underlying the causal chain were further developed as part of the analytical approach. We used programme names to link studies included in quantitative, qualitative and project portfolio analyses to facilitate moderator analysis. We discuss the implications of the findings for policy, implementation and research.

Pesticide use for IPM/IPPM farmer field schools

Measures of pesticide use usually took the form of the volume of pesticides used per unit of land area or the number of sprays per cropping season, or total pesticide expenditure. For these outcomes, we have estimated response ratio (RR) effect sizes. Reduced pesticide demand is considered a beneficial outcome, since all interventions included in this meta-analysis promoted IPM or IPPM farmer field schools (with or without complementary input and/or marketing interventions). Reductions in pesticide use – and therefore positive impacts of FFS training – are measured as values of RR between 0 and 1. Increases in pesticide use over the comparison group are measured as values of RR greater than 1. RR are interpretable as percentage changes over the comparison.

The meta-analysis suggests that demand for pesticides, measured in terms of amount sprayed or total expenditure, is estimated on average to be significantly lower among IPM/IPPM FFS graduates than comparison farmers (RR=0.68, 95% CI=0.57, 0.81; Q=172, Tau-sq=0.09, I-sq=90%; 18 observations), but not IPM/IPPM FFS interventions which included additional intervention components of input or marketing support (RR=0.77, 95% CI=0.30, 1.96; Q=158, Tau-sq=0.90, I-sq=98%; 4 observations) (Figure 15). Furthermore, and commensurate with the lack of knowledge gains reported above, we did not find evidence for diffusion to neighbouring farmers, estimating non-significant reductions in pesticide demand for this group on average (RR=0.91, 95% CI=0.66, 1.26; Q=47, Tau-sq=0.12, I-sq=85.2%; 8 observations). ²⁹

There is, however, substantial variation in effect sizes which appears to be in large part due to heterogeneity across studies (indicated statistically by high values of I-squared and Tau-squared). We therefore also conducted additional sensitivity and moderator analyses to explore the heterogeneity. Similarly to knowledge outcomes, we found that less rigorous study designs (Table 9) and high-risk-of-bias studies were likely to overestimate impacts on pesticide adoption (Figure 16 reports the forest plot excluding high-risk-of-bias studies). Indeed the best estimate of average effects on pesticide use for IPM/IPPM curricula FFS suggests a reduction in pesticide use of 23 per cent on average (RR=0.77, 95% CI=0.61, 0.97; Q=40, Tau-sq=0.07, I-sq=83%; observations=8). Impacts measured over shorter-term follow-up periods (up to two years) tended to be bigger than longer-term periods, as did studies which measured pesticide costs. The heterogeneity analysis for pesticide use by neighbouring farmers suggests results are not sensitive to dropping high-risk-of-bias studies (RR=0.95, 95% CI=0.64, 1.39; Q=45, Tau-sq=0.14, I-sq=91%, 5 observations). Results are also not sensitive to exclusion of the studies by Feder et al. (2004) and Yamazaki and Resosudarmo (2008) (see also forest plots in Appendix G).

OPEN IN VIEWER

15

Pesticide use for IPM/IPPM FFS participants and IPM neighbours versus non-FFS comparison farmers

OPEN IN VIEWER

16

Pesticide use for FFS participants and neighbours excluding high-risk-of-bias studies

We conducted additional analyses of moderators to explore the variation in findings across studies according to intervention and participating farmer characteristics (Table 10). The results suggest that the intervention characteristics which are associated with systematically smaller and insignificant impacts on pesticide use programmes implemented at national scale, programmes which involved complementary input or marketing components (as opposed to “pure” IPM/IPPM), and programmes in Latin America; in contrast, the results suggest impacts on pesticide use were biggest for cotton crops. We did not find differences for other variables describing intervention activities or length of implementation, with the exception of lack of impact for the single FFS for which it was clear that “farmer practice” control plots were not implemented (Labarta, 2005). However, the analysis by characteristics of participating farmers suggests socioeconomic status may affect pesticide adoption, as measured by education levels and landholdings. Field schools targeting women also seemed to have lower levels of adoption, although due to limited data available on many of the interventions and farmer characteristics variables, these results should be interpreted cautiously.

Sensitivity analysis for IPM-FFS neighbour farmers suggests that there may be diffusion in terms of reduced pesticide use for communities growing cotton over the short term (follow-up periods less than two years after FFS training), where FFS farmers are relatively highly educated. This finding is robust to exclusion of high-risk-of-bias studies (Table 11). The overall results also suggest diffusion effects among FFS communities in which participants had larger landholdings, although these findings are neither statistically significant nor robust to exclusion of high-risk-of-bias studies. It does not seem unreasonable to expect there to be diffusion of simple practices like reduced pesticide use due to interaction between FFS graduates and their communities, at least in the short term; indeed, as noted above, one study did find evidence for diffusion of “simple” knowledge (Ricker-Gilbert et al., 2008). However, those studies which examined sustainability found that any initial adoption among neighbouring farmers in terms of pesticide use fell within a few years (e.g. Wu, 2010, in China; Pananurak, 2010, in India). In addition, evidence from the one programme implemented at scale does not suggest any effects on diffusion (the Indonesia National IPM Programme evaluated by Feder et al., 2004 and Yamazaki & Resosudarmo, 2008). The findings in terms of diffusion should be interpreted with caution due to the limited sample size and problems of confounding in bivariate analysis. We also examined heterogeneity for FFS neighbour groups using intervention design variables which might reasonably be expected to foster diffusion (i.e. farmer exchanges and field days, efforts at local institutionalisation, FAO involvement, years of implementation of the programme), but did not find systematic differences across studies according to these characteristics, since many were poorly reported in the included studies and project documents we not available.

Finally, we explored robustness of effect size moderators in multivariate meta-regression analysis (Table 12). The analysis across the full sample (specification 1) suggests studies which were assessed as being of high risk of bias, which targeted cotton crops, were significantly associated with larger effects on pesticide use, while neighbour farmers had significantly smaller effects. We did not find significant correlations between effect size and whether studies measured outcomes by pesticide costs (rather than use), length of intervention and follow-up, or programme scale (results not reported). In a second meta-regression analysis for the subsample of studies in which education data were available (specification 2), we also tested for the association between years of education (continuous variable) and variation in effects on pesticide use. Due to the limited sample size we included only those variables found to be statistically significant in specification 1. The results suggest that FFS which targeted farmers with a relatively greater number of years of education, and those targeting cotton crops, were significantly associated with bigger decreases in pesticide use, both for FFS participants and IPM-FFS neighbours.

Other reported measures of practices adopted

Many studies, including those which were assessing field schools providing training in technologies other than IPM/IPPM, reported measures of numbers of improved practices adopted. Studies used a range of outcomes including indices of adoption and numbers of practices adopted, and probability of farmers adopting positive practices (Appendix E). Similar to the variables used to measure knowledge outcomes, given the lack of a natural scale in many such outcomes, we considered it appropriate to calculate standardised mean difference effect sizes.

The forest plot shows similar positive average impacts on adoption among FFS participants (SMD=0.63, 95% CI=0.32, 0.94; Q=3,192, Tau-sq=0.314, I-sq=99.6%; 14 observations) as compared with non-FFS farmers across the whole sample (Figure 17). ³⁰ The findings do not suggest differences on average for IPM and other (non-IPM/IPPM) FFS curricula. No studies estimated effects on IPM-FFS neighbour farmers. The very high level of variability in effect sizes is likely to substantial between-study heterogeneity. Similar to other outcomes, we found medium risk of bias studies tended to produce effects of smaller magnitude (SMD=0.22, 95% CI=0.06, 0.38; Q=10, Tau-sq=0.02, I-sq=80%, 3 observations) than high risk of bias studies (Figure 18). Other factors relating to study design, effect size calculation and implementation appeared to be associated with variation in effects (Table 13), but we were unable to explain the variation in effects further using meta-regression analysis (results not reported).

OPEN IN VIEWER

17

Beneficial practices adopted by IPM and other curricula FFS-participants

OPEN IN VIEWER

18

Beneficial practices adopted by risk of bias status

Impacts on farmers' time use

A small number of IPM-FFS studies also measured the burden of labour time (Birthal et al., 2000; Tripp et al. 2005; Khan et al., 2007; Mancini & Jiggins, 2008; Wu, 2010), estimating increases with adoption in India (Birthal et al., 2000; Mancini & Jiggins, 2008). Birthal et al. (2000) noted that the increased time burden may be due to both time monitoring crops as well as harvesting bigger yields. The study found an increase of 25 per cent in labour time on IPM cotton farms as compared with farms using pesticides, only about one-tenth of which was borne by women handpicking insect larvae, while one-third was due to time harvesting larger yields on FFS farms. However, the study was classified as high risk of bias because the FFS had a larger average farm size and more irrigation than the comparison group, and these differences were not controlled in the analysis. Mancini and Jiggins (2008) noted the burden of time for plant protection shifted towards women household members in India, and suggested therefore that availability of women might be a factor determining adoption of

IPM. Indeed, the relative costs of pesticide and labour may be important factors determining adoption. One study with medium risk of bias which found modest effects on outcomes noted pesticide costs were not a high share of variable production costs and dwarfed by costs of labour (Praneetvatakul & Waibel, 2006). In this study, farmers with more farm area per household member were also more likely to drop out of field school training due to labour shortages and high opportunity costs of labour.

Impacts on yields

Meta-analysis findings suggest that FFS training does on average lead to higher yields among FFS graduates as compared with non-FFS farmers, both for IPM/IPPM and other field school training (including other production techniques and/or additional input or marketing support) (Figure 19).

OPEN IN VIEWER

19

Yields for IPM/IPPM and other FFS farmers and IPM neighbours versus non-FFS comparison

There appears to be substantial between-study heterogeneity (as measured by I-squared). Sensitivity analyses (Table 14) to explore possible sources of heterogeneity proceeded as per other outcomes. For IPM/IPPM FFS participants, we found average effects to be small but statistically significant for medium risk of bias studies, representing an estimated increase of 13 per cent in yields on average (RR=1.13, 95% CI=1.04, 1.22; Q=53, Tau-sq=0.008, I-sq=81%; 11 observations) (Figure 20). Similarly, we estimated average effects to be smaller for more rigorous quasi-experimental evaluation designs, and for outcomes measured in terms of yields as opposed to monetary measures of output. For outcomes measured at follow-up periods longer than two years (excluding high-risk-of-bias studies), there is no evidence for statistically significant effects on yields. Moderator analysis suggests FFS have been relatively more successful in boosting yields of staples and vegetables in Africa using IPPM curriculum. Outcomes for FFS neighbours were fairly homogeneous across studies and analysis does not suggest outcomes are sensitive either to risk of bias or length of follow-up.

OPEN IN VIEWER

20

Yields for FFS participants and neighbours: excluding high-risk-of-bias studies

We did not find evidence for spillovers to neighbour farmers living within FFS communities across all studies (RR=1.00, 95% CI=0.98, 1.03; Q=13, Tau-sq=0.000, I-sq=53%; observations=7) (Figure 19), when we exclude high-risk-of-bias studies (RR=1.02, 95% CI=0.97, 1.08; Q=9, Tau-sq=0.002, I-sq=66%; 4 observations) (Figure 20) or indeed for the majority of individual studies. ³¹

Notably, as indicated in Table 14, we did not find significant effects for FFS participants (nor neighbours – results not reported) for the two programmes implemented at national scale (Indonesia and Vietnam National IPM Programmes). Two studies replicated analysis of the Indonesian data (Feder et al., 2004 and Yamazaki & Resosudarmo, 2008), coming to different conclusions regarding effectiveness in terms of agricultural outcomes. Feder et al. (2004) found no effect of FFS on trained or exposed farmers' adoption of reduced pesticides or rice yields, mentioning poor implementation delivery as a possible explanation for poor performance (although not providing any evidence of this). They suggested scaling up of the programme may have had a negative effect on the average quality of trainers and their commitment to bottom-up approaches, given many of them may have been experienced extensionists trained to deliver using top-down methods in the past. Moreover, they noted that delays in transfers of funds to the field training organisers meant that FFS were not fully synchronised with the rice-growing season calendar and supplies of materials were irregular, and suggested that this may have had a negative impact on the quality of knowledge achieved by FFS graduates.

Yamazaki and Resosudarmo (2008) re-analysed the data from Feder et al. (2004), but additionally controlled for time exposed to FFS prior to FFS training, since FFS were carried out in the same villages for different farmers over a number of seasons, and time after graduation. This was in principle a sensible approach, given the authors' hypothesis about bigger short-term rather than longer-term effects, and they did find that allowing for time variables produced positive impacts of FFS training on rice yields in the short term. However, like Feder et al. (2004), they did not find any significant impact on adoption as measured by changes in pesticide costs (as reported in Figure 15). ³² We examined sensitivity of our findings by excluding these studies from analysis, in part because of potential spillovers suggested by Van den Berg and Jiggins (2007). Our results were insensitive to their exclusion (Table 14), as well as exclusion of studies with possibilities for other studies with possible spillovers (Rejesus et al., 2010) and active control groups (Davis et al., 2012 in Uganda) (see forest plots in Appendix G).

We conducted additional moderator analyses to assess whether findings vary by FFS curriculum, length of implementation, crop, region and farmer characteristics (Table 15). Due to the important policy relevance of the outcome, we examined whether findings differed when we excluded high-risk-of-bias studies, and we limit the following discussion to these results. We were able to identify significantly positive impacts among FFS graduates on yields for IPM-FFS training (9% increase; 6 observations) and FFS with complementary input or marketing support (20% increase; 3 observations), growing cotton (9% increase; 3 observations) and other staples/vegetables (37% increase; 4 observations). In addition, FFS which had been implemented for longer than two years experienced more significant effects (17% increase; 6 observations) than those which had been operating for less (no significant difference; 5 observations). However, there was no evidence for effects in programmes operating at national scale (no significant difference; 3 observations). We also found that only field schools which involved farmers of relatively high education levels exhibited significantly positive impacts on yields (9% increase; 3 studies), although we did not find any significant differences for landholdings. We were not able to identify significantly positive effects on average for any other sub-groups, due to the limited numbers of studies available.

Consistent with the moderator analysis for adoption, we did not find significantly positive effects for rice FFS on yields. One study suggested that additional yield gain in technologically advanced rice production systems might be small and difficult to measure by recall surveys (Praneetvatakul & Waibel, 2006). In contrast, we estimated significant effects for cotton farmers in terms of both adoption and yields. Two of these studies based on evaluations of (genetically modified) Bt cotton programmes (Pananurak, 2010; Wu, 2010) found benefits of high yields and reduced pesticide use in incorporating improved seeds in the FFS curriculum.

The positive estimated effects of FFS for shorter-term studies (and conversely insignificant effects for longer-term studies) were found for both adoption and yields outcomes. However, most data are from indirect comparisons over programmes in different contexts with different follow-up periods. Several studies directly compared impacts over short and longer time periods among the same farmers. Praneetvatakul and Waibel (2006) reported impacts at one and three years after the intervention for FFS farmers in the Philippines, while Pananurak (2010) and Wu (2010) reported impacts at one and four-year follow-ups in China. Neither found significant differences in impacts for yields (or adoption) over these time periods, and both concluded FFS graduates retained their knowledge and continued applying IPM practices over time. However, more long-term impact evaluations are necessary to assess whether benefits are indeed sustained over time as these authors suggest.

We also examined heterogeneity of diffusion from IPM field schools to non-participating neighbour farmers (Table 16). The only studies which measured diffusion did so for programmes which had been implemented for more than two years. An initial positive diffusion effect found in China was not sustained (Wu, 2010), with both yield and knowledge gains in the neighbouring farmer groups diminishing considerably over time. ³³

In the final analysis of yields, we attempted to explain the heterogeneity using meta-regression (Table 17). Specification 1, which includes the full sample of 35 studies reporting yields effect sizes, confirms much of the bivariate analysis, namely staples and vegetable crops tended to exhibit the biggest impacts, rigorous quasi-experimental studies tend to show significantly smaller impacts than other studies, and publication bias due to small study effects may be present (as indicated by Egger's regression test).

Specifications 2 adds follow-up period and specification 3 additionally includes average education of FFS-participants (both variables are measured in years); these specifications reduce the sample size due to missing observations. To conserve degrees of freedom, the fourth specification excludes variables which are not statistically significant in specification 3. The analysis suggests impacts measured over longer periods were generally smaller while impacts were significantly bigger for more educated farmer participants. As per other outcome variables, years of education significantly increased the explanatory power of the model, suggesting field schools which targeted better educated farmers were also more effective in improving yields. In addition, an interaction between FFS-participant education and the neighbour farmer dummy variable is not statistically significant (results not reported), suggesting that field schools targeting better educated farmers may also have been more effective in promoting diffusion to neighbours. However, given the small sample size and relative instability of coefficients over specifications, these results should be interpreted cautiously. Until further evidence becomes available on the relationship between programme effects and characteristics of participating farmers such as education, income and social standing, we will not be able to conclude whether targeting farmers of higher socioeconomic status does lead to bigger increases in agricultural outcomes for FFS participants or neighbours.

Impacts on net revenue

It is possible that the production technologies promoted in FFS may lead to improved net revenues or profits (monetary value of production less costs), even where they do not improve yields, due to reduced reliance on pesticides and other purchased inputs. ³⁴ We differentiated FFS programmes providing only training in IPM and those providing training in other (non-IPM/IPPM) curricula, from “FFS-plus” programmes which provided additional components of support for inputs and/or marketing (Figure 21). ³⁵ At least two points are worth noting here, the first being that the impacts on net revenues are larger in magnitude than yields, as we might also expect based on the bigger average impacts on pesticide expenditure than pesticide use. For the “FFS-plus inputs/marketing” farmer group in particular, we estimated revenues on average to be 150 per cent greater, although over a small sample size with wide dispersion in the pooled confidence interval (RR=2.57, 95% CI=1.18, 5.58; 4 observations) and substantial estimated heterogeneity due to contextual factors (I-squared=96%, Tau-squared=0.56). For IPM-FFS participants, ³⁶ revenues increased by a lesser amount. The results were sensitive to exclusion of high-risk-of-bias studies (Figure 22 and Table 18); for medium-risk-of-bias studies we estimated revenues to increase by 19 per cent on average (RR=1.19, 95% CI=1.11, 1.27; Q=1, Tau-sq=0, I-sq=0%; 2 observations).

OPEN IN VIEWER

21

Net revenues for FFS farmers and neighbours versus non-FFS comparison

A second point of note is that the pooled effect size suggests that there may have been spillovers to non-participant neighbour farmers, although only partially in terms of magnitude of effect as compared with FFS graduates (RR=1.08, 95% CI=1.03, 1.15; Q=1, Tau-sq=0, I-sq=0%; 2 observations). However, the analysis excluding high-risk-of-bias studies (Figure 22) comprises only two interventions in China and Pakistan (Pananurak, 2010), both of which found positive effects in terms of revenues which were not reflected either in reductions in pesticide use or in improved yields, casting doubt on the credibility of the findings. In contrast, the increase in revenues for FFS participants in the same study was mirrored by improvements in both yields and pesticide reduction.

OPEN IN VIEWER

22

Net revenues excluding high-risk-of-bias studies

The sensitivity and moderators analysis (Table 18) suggests that there may be differences in impacts on net revenues by crop, region and curriculum, but there were simply too few rigorous studies to draw conclusions.

Finally, we examined heterogeneity of revenues using meta-regression (Table 19). The meta-regression indicated that FFS which were provided alongside inputs and/or marketing support lead to significantly better outcomes, when controlling for crop type, study design, and farmer. Although heterogeneity remained high (I-sq=78%), the small sample size limited our ability to explore other sources of differences across studies. However, no evidence was found for publication bias or for significant effects for other variables such as FFS-participant education (results not reported).

Effects on environment

Five studies of IPM/IPPM farmer field schools reported effects on environmental outcomes; in three cases we were able to calculate effect sizes and standard errors (Appendix E). One study measured positive effects on soil fertility of IPPM-FFS among FFS participants in Uganda (RR=1.79, 95% CI=1.15, 2.78; Friis-Hansen et al., 2004). Four further studies measured impacts of five IPM-FFS interventions on environmental outcomes using the environmental impact quotient (EIQ) score (Kovach et al., 1992), which estimates changes in outcomes indirectly based on reported reductions in chemical pesticide use. ³⁷ The findings of the meta-analysis for EIQ scores are suggestive of benefits to FFS graduates (Figure 23). Walter-Echols and Soomro (2005) also estimated reductions in EIQ in India (RR=0.68) and Pakistan (RR=0.58) but did not report information to calculate standard errors so are not included in the meta-analysis.

OPEN IN VIEWER

23

Environment outcomes: environmental impact quotient (EIQ)

EIQ is the only outcome for which estimated effects on non-participant neighbour farmers were significantly positive across all studies regardless of risk of bias. However, given the EIQ is estimated from reported pesticide use, for which we did not find any impacts on neighbour farmers, it seems unlikely that these significant effects on average are common. Indeed, when we excluded high-risk-of-bias studies from the analysis (including results for neighbour farmers from Cavatassi et al., 2011, for which we have approximated standard errors) the results for FFS participants remained significant (RR=0.61, 95% CI=0.48, 0.78; Q=3, Tau-sq=0.01, I-sq=33%; 3 observations), while those for neighbours were no longer statistically significant (RR=0.70, 95% CI=0.43, 1.14; Q=0.5, Tau-sq=0.00, I-sq=0%; 2 observations) (Figure 24).

OPEN IN VIEWER

24

Environmental impact quotient (EIQ) excluding high-risk-of-bias studies

Effects on health

Health outcomes were measured in four studies, all of which were assessed as being of high risk of bias. We were able to calculate effect estimates and confidence intervals in two studies (Amera, 2009; Labarta, 2005) (Appendix E). ³⁸ One study in Nicaragua compared self-reported health outcomes among FFS participants with those of non-participants (Labarta, 2005). The results, shown in Figure 25, suggest – counter-intuitively – that FFS beneficiaries experienced greater respiratory difficulties, though cases of eye irritation, stomach ache and blurred vision were not significantly different. In addition, neighbour farmers experienced greater eye irritation than non-FFS comparison farmers. However, the specifications used also found that the length of time after graduation from FFS significantly reduced the incidence of respiratory difficulties.

OPEN IN VIEWER

25

Health outcomes in Nicaragua (Labarta, 2005)

It appears that adoption of practices was a problem in Labarta (2005), as the authors indicated that 5 of the 13 FFS did not include a conventional “farmer practice” (control) plot; of the 8 that did, half observed lower yields in the IPM plot compared with the control plot, while 6 observed lower net revenue in the IPM plot compared with the conventional plot. In these cases, when comparative trials of IPM found higher revenues or yields in the IPM plot relative to the conventional plot, farmers were more likely to adopt the IPM practices included in the curriculum. A second study (Amera, 2009) reported perverse results in terms of pesticide poisoning in Kenya (RR=1.10, 95% CI=0.93, 1.30), despite significant estimated reductions in pesticide use (RR=0.61, 95% CI=0.52, 0.71).

It is difficult therefore not to conclude from these studies that reverse causality is driving the counter-intuitive outcomes, where farmers who most recently participated did so because of an existing respiratory problem for which analysis was unable to account. ³⁹

Effects on empowerment

A total of four studies collected quantitative data on some measure of empowerment (Appendix E). Some outcomes were reported more clearly than others, and in only one case were we able to calculate confidence intervals (Van Rijn, 2010). Thus, Rusike et al. (2004) reported “empowerment attitudes” and Hiller et al. (2009) measured change over time in farmers' perceptions of empowerment; further details on the definition of empowerment were not reported. In contrast, Friis-Hansen and Duveskog (2012) reported empowerment indices for innovation uptake, access to services, engaging with markets and collective action/social relations, from which we calculated a pooled point estimate. In the case of Van Rijn (2010), we calculated a pooled variable measuring the probability that farmers felt improvements in self-esteem, including feeling capable of solving problems in the field, feeling comfortable in giving an opinion, and participating in the community (RR=2.13, 95% CI=1.46, 3.12). Figure 26 shows the forest plot for empowerment outcomes, including point estimates only for those studies where confidence intervals could not be calculated. The limited evidence suggests beneficial impacts, which is supported by qualitative evidence reported in Chapter 5, although further quantitative studies are needed to support the validity findings.

OPEN IN VIEWER

26

Empowerment outcomes (includes studies without standard error estimates)

A study of the IFAD-FAO East African FFS programme examined impacts on production by gender (Davis et al., 2012). The study showed high rates of FFS participation by women, who comprised two-thirds of field school participants in Kenya and half of those in Uganda. While women did not seem to have problems attending the FFS, female FFS participants did not benefit significantly more than female non-participants in both of these countries in terms of improved income. While women form a large part of the agricultural labour force, they may not be the household or community decision-makers and therefore the agents of change in adoption of practices. However, women FFS participants did benefit more than women non-participants in Tanzania, where women's participation made up one-third of the FFS programme intake. Davis et al. (2012) also found that increases in productivity for participants with no formal education were greater than for any other group of farmers, suggesting the experiential learning and demonstration focus of the FFS appears to allow low-literacy farmers to actively participate and learn.

General inputs

As with any programme FFS require a range of different inputs at the outset in order to establish the programme and start delivering services to participants. There was a lack of systematic reporting of the resources devoted to individual FFS interventions and assessment of the quality of the services delivered to farmers in the included studies. However, some studies reported a lack of (timely) funding and inadequate provision of resources, including inputs and finances, as potential reasons for implementation failures (Pedersen et al., 2008; Najjar, 2009; Winarto, 2004). These included insufficient and incidental provision of inputs, discrepancies between budgeted and received payments, payment delays and difficulties with logistics and dissemination. In addition, selection of inappropriate FFS sites and a lack of adequate follow-up and support during pest outbreaks were noted as constraints.

Facilitators

The facilitator is an important input in FFS programmes, and according to the programme theory adequate training of facilitators is a key assumption for the FFS training to lead to improved knowledge and skills among participating farmers. If facilitators are not adequately trained the quality of the training received by farmers may be lower, affecting knowledge formation, adoption and final outcomes.

Six of the seven studies that reported on training of facilitators suggested that there were issues related to a lack of appropriate training, resources and ongoing support of facilitators (DANIDA, 2011; Gottret & Córdoba, 2004; Hofisi, 2003; Pedersen et al., 2008; Rola & Baril, 1997; Simpson, 1997). Examples of deficiencies included gaps in the curriculum covered in the training of facilitators and lack of focus on participatory techniques and facilitation skills. The studies also highlighted issues related to a lack of ongoing support or backstopping, and lack of inputs to support facilitators in their roles.

One theme which was not included in the original programme theory relates to the selection and characteristics of FFS facilitators. Four studies suggested appropriate criteria for selecting facilitators were important for identifying candidates suitable to be facilitators (DANIDA, 2011; Hofisi, 2003; Machacha, 2008; Winarto, 2004). These studies found that rather than high levels of education, characteristics such as personal attitude, maturity, literacy, leadership skills and experience with farming might be more important for facilitators to perform their role successfully. Facilitators not having the right characteristics from the outset appeared to have influenced their ability to successfully perform the role of FFS facilitator.

Moreover, a large difference between farmers and facilitators in terms of socioeconomic characteristics may prevent farmers from fully participating and making suggestions or raising concerns (Simpson, 1997). Finally, the gender of the facilitator might be important, in particular in more conservative contexts. Some studies found that women preferred female facilitators (Hofisi, 2003; Van Der Wiele, 2004), suggesting that FFS programmes aiming to target women farmers should take account of this when selecting facilitators.

The characteristics of the facilitators and the training they receive in turn effect another theme which was highlighted by several of the included studies: the relationship between facilitators and participants. Three of the studies suggested an imbalance in the farmer–facilitator relationship as a potential barrier to farmers' learning and adoption (Isubikalu, 2007; Najjar, 2009; Simpson, 1997), with farmers not feeling comfortable enough to admit when they did not understand something, or to voice concerns or make suggestions.

Targeting criteria and procedures

While not included in the original programme theory our synthesis suggests a key assumption in the FFS programme theory may be that the right farmers are targeted and reached by the FFS. The targeted farmers must be willing and able to participate in FFS training throughout the full season, and be able to implement FFS practices in their fields. Those studies that provided information on targeting, selection procedures and group composition, have been linked to FFS effectiveness analysis above, and are covered in a separate piece on FFS targeting (Phillips et al., 2014).

Available evidence suggests that structural factors such as socioeconomic status, gender and cultural norms influence both who is targeted and who is able to participate in FFS. In some cases targeting procedures privileged the elite and more affluent (DANIDA, 2011; Simpson, 1997; Van de Fliert, 1993), or excluded women and the poor (DANIDA, 2011; Najjar, 2009; Simpson, 1997). In other cases poverty and ill health prevented farmers from participating (Gottret & Córdoba, 2004; Najjar, 2009). In yet another case, an absence of targeting criteria was suggested as a barrier to effectiveness (Pedersen et al., 2008).

While several studies found that women were able to participate in FFS (Hofisi, 2003; Machacha, 2008; Najjar, 2009; Rola & Baril, 1997), other studies found that women were often not allowed to participate by their husbands or could not participate due to time constraints and lack of access to resources and land (DANIDA, 2011; Hofisi, 2003; Najjar, 2009; Van de Fliert, 1993, Van Der Wiele, 2004).

Farmers' education levels and motivation to participate was another key determining factor highlighted by existing evidence (Gottret & Córdoba, 2004, Hofisi, 2003). The most commonly cited reason to join a field school was to improve knowledge, skills and livelihoods (DANIDA, 2011; Friis-Hansen, 2008; Hofisi, 2003, Najjar, 2009). Several studies found high levels of drop-out (Friis-Hansen, 2008; Gottret & Córdoba, 2004; Machacha, 2008; Najjar, 2009; Rola & Baril, 1997; Winarto, 2004). Unmet expectations of hand-outs or loans were the most commonly cited explanation in the literature (Friis-Hansen, 2008; Hofisi, 2003; Machacha, 2008; Najjar, 2009). Additional reasons for drop-outs included lack of interest, access and time (Gottet & Córdoba, 2004; Machacha, 2008; Najjar, 2009).

Overall, the review identified several key barriers related to targeting, such as inappropriate selection criteria and targeting procedures, and structural barriers to participation such as gender, poverty and cultural norms. If there is not a considered approach to targeting, this may result in farmers participating for the wrong reasons and ultimately dropping out. Alternatively, beneficiaries may not have the right characteristics, such as education levels, or access to land and resources, to be able to fully benefit from the FFS training. FFS should target farmers with appropriate education levels, motivation and access to land, and those who do not live too far away to attend the weekly FFS sessions.

Approach to learning

A key characteristic of FFS design is that programmes should follow a participatory approach, based on theories of adult education and discovery-based learning (Pontius et al., 2002). A participatory, bottom-up approach to the training is considered important in communicating complex concepts, strengthening problem-solving capabilities and ultimately convincing farmers to adopt the practices promoted in FFS.

Over half of the included studies included themes pertaining to the way in which FFS were delivered in the field (DANIDA, 2011; Dolly, 2009; Friis-Hansen, 2008; Gottret & Córdoba, 2004; Hofisi, 2003; Isubikalu, 2007; Machacha, 2008; Najjar, 2009; Pedersen et al., 2008; Rola & Baril, 1997; Van de Fliert, 1993, Winarto, 2004). Some of the studies represented cases where FFS were delivered in a top-down manner, using a transfer of technologies approach rather than according to the original approach characterising FFS (Isubikalu, 2007; Pedersen et al., 2008; Najjar, 2009; Simpson, 1997). The top-down approach seemed to have occurred at different levels – a bias towards talk, rather than practice, experiments being managed by the facilitators rather than farmers, or the delivery of the FFS in terms of the overall approach, budgeting, monitoring and inputs provided.

Nevertheless, a greater proportion of the included studies represented cases where FFS were delivered in a participatory manner (DANIDA, 2011; Dolly, 2009; Friis-Hansen, 2008; Gottret & Córdoba, 2004; Hofisi, 2003; Machacha, 2008; Rola & Baril, 1997; Van de Fliert, 1993; Winarto, 2004), ⁴⁰ with farmers taking part in field trials and development of the curriculum. According to the participating farmers of one FFS, their active experimentation and information-sharing enhanced their learning and increased knowledge and ownership of the resulting farming systems.

Related to this, there is consistent evidence that due to the complexity of the curriculum, observability is important for farmers to trust the messages promoted in FFS and to develop analytical skills (Dolly, 2009; Hofisi, 2oo3; Machacha, 2008; Palis, 2002; Simpson, 1997; Van Der Wiele, 2004; Winarto, 2004). The observability of IPM practices and their relative advantage compared with conventional methods was found to be important in enhancing farmers' confidence and trust in IPM messages. Similarly, in a few cases facilitators were unable to demonstrate observable benefits from FFS practices and this acted as a barrier to adoption.

The extent to which the FFS in the included studies were delivered according to the original principles upon which the intervention was designed varied, ranging from approaches to learning similar to traditional extension, to “true FFS”. Where implementation in practice was characterised by a more top-down, transfer of technology approach farmers were not able to see and practice IPM, and this appeared to have been a barrier to farmers developing their knowledge and analytical skills.

The evidence highlighted the importance of empirical study, successful demonstration plots, and visibility of benefits in other farmers' plots in convincing farmers to adopt new practices. The apparent importance of observability and relative advantage of FFS practices in farmers' adoption is consistent with Rogers' theory of diffusion (2003), which suggests peoples' perceptions of these characteristics are among the important attributes determining adoption of innovation.

FFS content and coverage

The FFS curriculum is another major component of the intervention. For farmers to learn and adopt the practices promoted in FFS, an assumption is that the content of the curriculum is relevant to the needs of farmers, appropriate for the local context in terms of the crops it covers and feasible for farmers to implement in their fields. Most of the included studies reported themes relating to the FFS curriculum (DANIDA, 2011; David, 2007; Dolly, 2009; Gottret & Córdoba, 2004; Hiller et al., 2009; Hofisi, 2003; Isubikalu, 2007; Mancini et al., 2007; Najjar, 2009; Palis, 2002; Simpson, 1997; Van de Fliert, 1993; Van Der Wiele, 2004).

From the studies it appears that in most cases the FFS curriculum was relevant and appropriate to the local context (DANIDA, 2011; Dolly, 2009; Gottret & Córdoba, 2004; Hiller et al., 2009; Hofisi, 2003; Mancini et al., 2007; Van de Fliert, 1993), for instance by responding to local concerns over economic and environmental costs of pesticides (Dolly, 2009; Mancini et al., 2007).

However, in some cases FFS curricula appeared to lack relevance to the local context (Isubikalu, 2007; Najjar, 2009; Simpson, 1997), and the major barrier in these cases seemed to be that the curriculum was not sufficiently tailored to the needs and resources of the farmers it was targeting. For instance, in some cases the practices were seen to be too labour- or time-intensive (Isubikalu, 2007) or failed to address the issues that were of highest priority among farmers (Simpson, 1997).

Farmers may have a range of concerns, such as water management, fertilisation, diversification and marketing. A failure to incorporate a broader range of concerns in the curriculum was perceived by farmers to be a weakness in several studies (DANIDA, 2011; Mancini et al., 2007; Najjar, 2009; Van de Fliert, 1993, Winarto, 2004), who expressed that the curriculum was not sufficiently comprehensive in its coverage. Therefore a broader focus might encourage participation in FFS, and could potentially also facilitate greater improvement of agricultural outcomes. Nevertheless, as one case from Kenya demonstrates (Pedersen et al., 2008), trying to cover too many topics or crops in one cycle may put the technical quality of the intervention at risk.

Limited evidence suggests that in some cases the complexity of IPM made it difficult for farmers to implement all the IPM practices on their crops (Gottret & Córdoba, 2004; Van de Fliert, 1993; Winarto, 2004). Some of the analytical tools used were too complex and perceived as impractical in relation to the time, energy and resources they required (Winarto, 2004). For instance, the use of forms to record field sampling with formulae to calculate percentages for damages and prevalence of insects was found to be of little practical use for the farmers, who abandoned this in favour of simply recording what they observed in their fields (Van de Fliert, 1993). On the other hand, some studies found that including indigenous and practical knowledge, rather than theoretical concepts, may enhance farmers' understanding of FFS practices (David, 2007; Hofisi, 2003).

Related to the complexity of the curriculum is the discourse and language used for communication in FFS. An assumption not included in our original programme theory, but which emerged from the synthesis is that the curriculum needs to be communicated in a language farmers understand. Evidence from Indonesia suggested that when facilitators used unfamiliar foreign and scientific terms which were part of their vocabulary, such as “economic threshold level” (ETL) and “ecosystem”, these terms were not easily understood by farmers (Van de Fliert, 1993; Winarto, 2004). On the other hand, the use of common concepts and metaphors, such as “natural enemy” (musuh alami) or “farmers' friends” (teman petani), were more easily understood and facilitated knowledge formation and adoption (Dolly, 2009; Winarto, 2004).

Moreover, farmers might not be sufficiently fluent in the national language to be able to comprehend the FFS content (DANIDA, 2011; Najjar, 2009), and in one case farmers' understanding was improved when facilitators used the local language (Van de Fliert, 1993).

Policy context and institutional set-up

FFS are implemented within the context of existing and emerging policies, institutional structures and industry activities. A range of actors are involved in agricultural extension, including international donors, national, regional and local governments, research institutions, NGOs and various industry actors. Several studies noted that diverging institutional incentives and objectives of these various actors influence how FFS are implemented in practice, with a tendency for some of the participatory elements to be eroded (Isubikalu, 2007; Pedersen et al., 2008; Simpson, 1997).

There is some evidence suggesting that the existence of conflicting agricultural policies can act as a barrier to adoption of FFS practices. In some cases, the involvement of a multitude of institutions each promoting their own mandate resulted in each impacting on the design, content and implementation of FFS, crowding out the beneficiary needs and interests (Isubikalu 2007, Simpson, 1997). In other cases, the institutional legacy of existing extension systems influenced the implementation of FFS (Isubikalu 2007; Mancini et al., 2007; Van de Fliert, 1993; Winarto, 2004). Institutional structures associated with earlier top-down agricultural extension systems, such as subsidised input schemes, trickle-down messages and off-the-shelf technology promotion, remained and contradicted the bottom-up, participatory approach of FFS.

Other evidence suggested that the power of the pesticide industry and their continued links with the extension system can act as a barrier to adoption and diffusion of FFS practices. In at least two cases, studies found that the pesticide industry maintained close links with the extension system at the local level (Mancini et al., 2007; Van de Fliert, 1993), paying commission to extension workers and local cooperatives to promote pesticides, hampering farmers' efforts to practice IPM.

Community context and social capital

Several of the included studies highlighted the role of the community in influencing farmers' practices (Dolly, 2009; Machacha, 2008; Najjar, 2009; Palis, 2002; Simpson, 1997; Van Der Wiele, 2004; Winarto, 2004). One of the hypothesised assumptions in the programme theory was that for sustained adoption of FFS practices it was important “farmers are convinced others will do the same”. There was limited evidence to support this assumption, which was based on the suggestion by Feder et al. (2004a) that lack of adoption of IPM practices by neighbouring farmers might curtail the effectiveness of the intervention, as pests from their fields may re-infest the fields of adopters, eventually leading to disadoption of IPM by FFS participants. In one case farmers in Indonesia found it difficult to practise new strategies when the rest of the community continued applying old practices (Winarto, 2004). However, the reason for this seems to be that they faced discouragement and disbelief from their untrained peers, rather than a fear of re-infestation.

In our original programme theory we suggested that a “high degree of social cohesion” might be an assumption that needs to hold for knowledge to diffuse to neighbouring farmers not participating in FFS, but we did not include it as a factor that might affect adoption among FFS participants. Several of the included studies suggested that existing social capital and reach of social networks may also influence adoption among FFS participants (Dolly, 2009; Gottret & Córdoba, 2004; Palis, 2002; Simpson, 1997). In some cases high levels of social cohesion and a tradition of collective action from existing farmer groups may have encouraged a willingness to learn and succeed with the training, facilitating adoption (Dolly, 2009; Gottret & Córdoba, 2004; Palis, 2002). On the other hand, in one case low levels of social capital and little sense of community in the FFS villages may have been a barrier to FFS effectiveness.

Access to inputs

One of the assumptions included in the programme theory was that farmers have access to any complementary inputs necessary to adopt FFS practices. Five studies highlighted issues related to availability of inputs, labour and markets (DANIDA, 2011; Machacha, 2008; Mancini et al., 2007; Najjar, 2009; Van Der Wiele, 2004). In all cases the lack of access to inputs, capital and/or markets were suggested as factors influencing uptake of FFS practices or final outcomes of the FFS training. These issues are common challenges for farmers, and are not particular to FFS programmes, but in some cases they appear to have remained a barrier for FFS farmers to adopt FFS practices and fully benefit from adoption.

Sustainability

FFS aim to provide farmers with skills which enable them to solve problems for themselves. An implicit assumption behind this is that the FFS training will enable farmers to continue with FFS practices and deal with any new challenges after they graduate. The sustainability of these skills and practices, along with their successful diffusion to broader farmer communities are an important outcome for cost-effectiveness of FFS interventions. Eleven out of 20 studies discussed factors affecting sustainability of FFS (DANIDA, 2011; David, 2007; Dolly, 2009; Friis-Hansen, 2008; Gottret & Córdoba, 2004; Karanja-Lumumba et al., 2007; Machacha, 2008; Simpson, 1997; Van de Fliert, 1993; Van Der Wiele, 2004; Winarto, 2004).

Based on the descriptive themes, all studies highlighted that ongoing support and follow-up are important for sustainability of FFS practices and the establishment and sustainability of FFS-related activities (DANIDA, 2011; David, 2007; Dolly, 2009; Gottret & Córdoba, 2004; Karanja-Lumumba et al., 2007; Machacha, 2008; Simpson, 1997; Van Der Wiele, 2004; Winarto, 2004). In particular, included studies identified a lack of technical assistance and backstopping from researchers and extensionists to support farmers in continuing development of local practices (Dolly, 2009; Gottret & Córdoba, 2004; Simpson, 1997; Winarto, 2004). The studies that reported on follow-up group activities taking place suggested that active follow-up and continued support by the implementing agency, encouragement to establish farmer clubs and additional sessions on club formations may facilitate the establishment of sustainable and effective groups and practices.

In addition, four studies also suggested that particular group characteristics may affect sustainability of farmer groups and FFS-related practices (DANIDA, 2011; David, 2007; Machacha, 2008; Van Der Wiele, 2004). Studies reporting successful follow-up farmer activities suggested consistent membership, good leadership, collective goals and a supportive group environment might be important in maintaining FFS groups and providing impetus for further farmer-led initiatives.

Overall, the studies suggested that lack of ongoing support and follow-up is an important barrier to the sustainability of the FFS approach. In the absence of formal follow-up, the implicit assumption is that the FFS training is sufficient to enable farmers to continue with FFS practices and deal with any new challenges. However, the evidence suggested that this might not hold in most cases, as farmers expressed the need for additional follow-up support and technical backstopping to be able to continue the development of local practice. A lack of formal follow-up activities therefore appears to be a barrier to sustainability of FFS practices.

An assumption missing from the original theory of change is that the group characteristics may play a crucial role in enabling sustainability of FFS practices and related activities. Based on the available evidence, it appears that choosing groups with common goals, good leadership and high attendance rates might be important enablers of sustainability of FFS-related practices.

Perceived outcomes: gender and empowerment

Proponents of FFS suggest empowerment is a key outcome of FFS. While few impact evaluation studies looked at these outcomes, empowerment and improved gender relations were frequently highlighted in the qualitative literature (DANIDA, 2011; Dolly, 2009; Friis-Hansen, 2008; Friis-Hansen et al., 2012; Hofisi, 2003; Machacha, 2008; Mancini et al., 2007; Najjar, 2009; Simpson, 1997, Van Der Wiele, 2004; Winarto, 2004). Evidence based on participants' perceptions suggests FFS may influence empowerment positively. Some studies also suggest participation in FFS may lead to women's empowerment and improve gender relations.

Farmer targeting and participation

A key assumption in the FFS programme is that the “right” farmers are targeted and reached by the FFS. The targeted farmers must be willing and able to participate in FFS training throughout the full season, and be able to implement FFS practices in their fields.

Overall, the review identified several key barriers related to targeting, such as inappropriate selection criteria and targeting procedures, and structural barriers to participation such as gender, poverty and cultural norms. If there is not a considered approach to targeting, this may result in farmers participating for the wrong reasons and ultimately dropping out. Alternatively, beneficiaries may not have the right characteristics, such as education levels, or access to land and resources, to be able to benefit fully from the FFS training. We were able to assess whether there were differences in knowledge gains among field schools which targeted farmers based on education levels and land size, but due to limited data provided in most included studies the results, though suggestive of benefits for higher socioeconomic status, were not conclusive.

General inputs

FFS require a range of different inputs at the outset in order to establish the programme and start delivering services to participants. Some studies reported a lack of (timely) funding and inadequate provision of resources, including inputs and finances, as potential reasons for implementation failures (Pedersen et al., 2008; Najjar, 2009; Winarto, 2004). These included insufficient and incidental provision of inputs, discrepancies between budgeted and received payments, payment delays and difficulties with logistics and dissemination. In addition, selection of inappropriate FFS sites and a lack of adequate follow-up and support during pest outbreaks were noted as constraints for IPM field schools.

Facilitator training, performance and support

The facilitator is an important input in FFS programmes, and adequate training of facilitators is a key assumption in the theory of change for the FFS training to lead to improved knowledge and skills among participating farmers. Six of the seven studies that reported on training of facilitators suggested that there were issues related to a lack of appropriate training, resources and ongoing support of facilitators (DANIDA, 2011; Gottret & Córdoba, 2004; Hofisi, 2003; Pedersen et al., 2008; Rola & Baril, 1997; Simpson, 1997). Examples of deficiencies included gaps in the curriculum covered in the training of facilitators and lack of focus on participatory techniques and facilitation skills. The studies also highlighted issues related to a lack of ongoing support or backstopping, and a general lack of inputs to support facilitators in their roles.

One issue relates to the selection and characteristics of FFS facilitators. Studies in Bangladesh, Indonesia, Kenya and Zimbabwe suggested appropriate criteria for selecting facilitators were important for identifying candidates suitable to be facilitators (DANIDA, 2011; Hofisi, 2003; Machacha, 2008; Winarto, 2004). These studies found that rather than high levels of education, characteristics such as personal attitude, maturity, literacy, leadership skills and experience with farming may be more important for facilitators to perform their role successfully. Facilitators not having the right characteristics from the outset appear to have been less successful in performing their role as a FFS facilitator.

A big difference between farmers and facilitators in terms of socioeconomic characteristics might prevent farmers from fully participating and making suggestions or raising concerns (Simpson, 1997). Some studies also suggested that the gender of the facilitator is important, in particular in more conservative contexts (Hofisi, 2003; Van Der Wiele, 2004).

The characteristics of the facilitators and the training they receive in turn effect another theme which was highlighted by several of the included studies: the relationship between facilitators and participants. Three of the studies suggested an imbalance in the farmer–facilitator relationship as a potential barrier to farmers learning and adoption (Isubikalu, 2007; Najjar, 2009; Simpson, 1997), with farmers not feeling comfortable enough to admit when they did not understand something, or to voice concerns or make suggestions.

Approach to learning

A key characteristic of the design of FFS is that they follow a participatory approach, based on theories of adult education and discovery-based learning (Pontius et al., 2002).

Over half of the qualitative studies included themes pertaining to the way in which FFS were delivered in the field (DANIDA, 2011; Dolly, 2009; Friis-Hansen, 2008; Gottret & Córdoba, 2004; Hofisi, 2003; Isubikalu, 2007; Machacha, 2008; Najjar, 2009; Pedersen et al., 2008; Rola & Baril, 1997; Van de Fliert, 1993; Winarto, 2004). The extent to which the FFS were delivered according to the original principles upon which the intervention was designed varied, ranging from approaches to learning similar to traditional extension, to bottom-up FFS. Where implementation in practice was characterised by a more top-down, transfer of technology approach, the qualitative evidence suggests that farmers were not able to see and practice IPM, which appears to have been a barrier to farmers developing their knowledge and analytical skills. However, we were not able to test whether this made a difference to FFS effectiveness in quantitative analysis due to inadequate reporting of intervention delivery in those (and linked) studies.

Related to this, the qualitative evidence suggests that due to the complexity of the curriculum, observability is important for farmers to trust the messages promoted in FFS and to develop analytical skills (Dolly, 2009; Hofisi, 2oo3; Machacha, 2008; Palis, 2002; Simpson, 1997; Van Der Wiele, 2004; Winarto, 2004). The observability of IPM practices and their relative advantage compared with conventional methods was found to be important in enhancing farmers' confidence and trust in IPM messages. Similarly, in a few cases facilitators were unable to demonstrate observable benefits from FFS practices which may have acted as a barrier to adoption.

FFS content and coverage

For farmers to learn and adopt the practices promoted in FFS, the content of the curriculum should in theory be relevant to the needs of farmers, appropriate for the local context in terms of the crops it covers and feasible for farmers to implement in their fields. Most of the qualitative studies reported themes relating to the FFS curriculum (DANIDA, 2011; David, 2007; Dolly, 2009; Gottret & Córdoba, 2004; Hiller et al., 2009; Hofisi, 2003; Isubikalu, 2007; Mancini et al., 2007; Najjar, 2009; Palis, 2002; Simpson, 1997; Van de Fliert, 1993; Van Der Wiele, 2004). It appears that in most cases the FFS curriculum was indeed relevant and appropriate to the local context (DANIDA, 2011; Dolly, 2009; Gottret & Córdoba, 2004; Hiller et al., 2009; Hofisi, 2003; Mancini et al., 2007; Van de Fliert, 1993), for instance by responding to local concerns over economic and environmental costs of pesticides (Dolly, 2009; Mancini et al., 2007).

However, in some cases the FFS curriculum appears to have lacked local relevance (Isubikalu, 2007; Najjar, 2009; Simpson, 1997), and the major barrier in these cases seemed to be that the curriculum was not sufficiently tailored to the needs and resources of the farmers it was targeting. A failure to incorporate a broader range of concerns in the curriculum was perceived by farmers to be a weakness in several studies (DANIDA, 2011; Mancini et al., 2007; Najjar, 2009; Van de Fliert, 1993; Winarto, 2004).

In some cases the complexity of IPM made it difficult for farmers to implement all the IPM practices on their crops (Gottret & Córdoba, 2004; Van de Fliert, 1993; Winarto, 2004). Some of the analytical tools used were too complex and perceived as impractical in relation to the time, energy and resources they required (Winarto, 2004).

Related to the complexity of the curriculum is the discourse and language used for communication in FFS, which needs to be in a language farmers understand (DANIDA, 2011; Dolly, 2009; Najjar, 2009; Van de Fliert, 1993; Winarto, 2004).

Policy context

Field schools are inevitably embedded within a policy and community context which can act to promote or hinder FFS effectiveness. Some evidence suggested that the existence of conflicting agricultural policies can act as a barrier to adoption of FFS practices. In some cases, the involvement of a multitude of institutions each promoting their own mandate resulted in each impacting on the design, content and implementation of FFS, crowding out the beneficiary needs and interests (Isubikalu 2007; Simpson, 1997).

In other cases, the institutional legacy of existing extension systems influenced the implementation of FFS (Isubikalu 2007; Mancini et al., 2007; Van de Fliert, 1993; Winarto, 2004). Institutional structures associated with earlier top-down agricultural extension systems, such as subsidised input schemes, trickle-down messages and off-the-shelf technology promotion, remained and contradicted the bottom-up, participatory approach of FFS.

There was also some evidence suggesting the power of the pesticide industry and continued links with the extension system acted as a barrier to adoption and diffusion of FFS practices. At least two studies found that the pesticide industry maintained close links with the extension system at the local level (Mancini et al., 2007; Van de Fliert, 1993), paying commission to extension workers and local cooperatives to promote pesticides, hampering farmers' efforts to practise IPM.

Several studies suggested that policy context was key for FFS effectiveness and sustainability, although no studies provided any systematic analysis (Praneetvatakul & Waibel, 2006; Pananurak, 2010; Wu, 2010). In particular, high levels of subsidies for inputs, including pesticides, might have provided strong incentives against IPM (Pananurak, 2010). Incentives to use pesticides, such as subsidies, and different market-based as well as institutional disincentives to the adoption of IPM, may have reduced adoption rates (Praneetvatakul & Waibel, 2006; Pananurak, 2010). This was the case in Thailand where, after a period of high-level support for FFS in the Department for Agricultural Extension, a change of leadership in the department reversed priorities towards pesticide-based crop protection and the FFS programme declined (Praneetvatakul & Waibel, 2006). In China, plant protection stations at township and county levels were found to be involved in pesticide sales due to shortages of operation funds, resulting in a diversion of human resources or even outright subversion of extension agents' attitudes towards pesticide use (Wu, 2010).

Community and social capital

Another emerging theme was the potential role of existing social capital in influencing FFS effectiveness. Several of the qualitative studies highlighted the role of the community in influencing farmers' practices (Dolly, 2009; Machacha, 2008; Najjar, 2009; Palis, 2002; Simpson, 1997; Van Der Wiele, 2004; Winarto, 2004).

There was limited evidence to support the assumption that sustained adoption of FFS practices requires community-wide uptake (Feder et al., 2004a). In one study, farmers in Indonesia found it difficult to practise new strategies when the rest of the community continued applying the existing practices (Winarto, 2004). However, the reason for this seems to be that they faced discouragement and disbelief from their untrained peers, rather than a fear of re-infestation.

Several of the qualitative studies suggested existing social capital and reach of social networks may also influence adoption among FFS participants (Dolly, 2009; Gottret & Córdoba, 2004; Palis, 2002; Simpson, 1997). In some cases high levels of social capital and a tradition of collective action from existing farmer groups may have encouraged a willingness to learn and succeed with the training, and facilitated adoption (Dolly, 2009; Gottret & Córdoba, 2004; Palis, 2002). On the other hand, in one case low levels of social capital and little sense of community in the FFS villages may have been a barrier.

Access to inputs

Five studies highlighted issues related to availability of inputs, labour and markets (DANIDA, 2011; Machacha, 2008; Mancini et al., 2007; Najjar, 2009, Van Der Wiele, 2004). In all cases the lack of access to inputs, capital and/or markets were suggested as factors influencing uptake of FFS practices or final outcomes of the FFS training. These issues are common challenges for farmers, and are not particular to FFS programmes, but they appear to have remained a barrier for FFS farmers to adopt FFS practices and benefit fully from adoption. Indeed, the analysis suggested field schools involving farmers who were of higher socioeconomic status (measured by education and landholdings) tended to have greater impacts in terms of adoption as well as agricultural outcomes.