PROTOCOL: A mixed methods systematic review on the effects of arts interventions for at-risk and offending children and young people on behavioural,psychosocial,cognitive and offending outcomes

Types of studies

We will include randomised and non-randomised controlled trials and quasi-experimental study designs. We will not include studies that did not employ a control or comparator group. We will include qualitative studies that were conducted alongside intervention trials that investigated the experiences and perceptions of participants, and that offer insight into the barriers and facilitators associated with delivering and receiving arts interventions. We will include qualitative and mixed methods studies that are focused on the delivery of an arts intervention and explored aspects of the process of intervention delivery from the perspectives of those delivering and those who are participants in the intervention and/or their carers/family members or significant agents (e.g., probation officers, victims). We will include studies from any global setting.

Types of participants

We will include studies that include children and young people (8–25 years) who are either identified as at-risk of offending behaviour (secondary populations) or already in the criminal justice system (tertiary populations).

Types of interventions

We will include studies of interventions involving arts participation. Arts participation will include involvement in artistic and creative activities. Studies which include arts participation as an intervention on its own or alongside other interventions, such as mentoring, will be included. We will include studies that use art as therapy (a form of psychotherapy) and as a medium to address emotional difficulties.

Examples of arts interventions in included studies are:

Music making

We will include studies that compare arts interventions to either no intervention, usual care, other types of arts intervention or non-arts control. The intervention will involve organised arts interventions targeted to the population. We will not include associational studies between arts participation and offending behaviour.

Types of outcome measures

Duration of follow-up

We will consider outcomes at the following time-points: short-term immediately post-intervention to <3 months; medium-term 3 to <12 months post intervention; long-term >1 year post intervention. Where studies report multiple follow-ups within a single time-point range we will preferentially extract as follows: short term, the closest follow-up point to the end of the intervention. Medium- and long-term: the latest timepoint reported.

Types of settings

We will include studies employing arts interventions in any setting including (i) juvenile correctional facilities, prisons, other residential settings, (ii) community and workplace settings, and (iii) schools.

Electronic searches

We consulted Campbell guidance on searching for studies (Kugley et al., 2016). Our search strategy will include expert advice from information services experts at Brunel University London Library. We will agree an appropriate time frame for our review in discussion with the YEF and Advisory Boards. We will search relevant key databases including AMED, Academic Search Complete; APA PsycInfo; CINAHL Plus; ERIC; SocIndex; SportDiscus (via EbscoHost), Medline (via Ovid), CENTRAL, Web of Science, Scopus, PTSDPubs, Performing Arts Periodicals Database, Sage, the US National Criminal Justice Reference Service databases, the Global Policing Database, and the National Police Library.

We will search Web of Science: Conference Proceedings Citations Index and the British Library EThOS database (dissertations). We will use a combination of controlled vocabulary, i.e. medical subject headings (MeSH), and free text terms to identify published articles. In addition, we will check reference lists of reviews and retrieved articles for additional studies. An example search strategy can be found in Appendix 1 and will include separate search strings for identifying quantitative and qualitative studies. To identify the population of interest we will use the search filter proposed by the Canadian Health Libraries Association (CHLA, 2022). We will use and adapt the Cochrane highly sensitive search filter to identify randomised controlled trials (RCTs) (Higgins et al., 2021), a validated filter for identifying non-randomised controlled studies (Waffenschmidt et al., 2020) and the University of Texas School of Public Health (University of Texas, 2022) filter for identifying qualitative studies which has been demonstrated to show good performance in sensitivity and specificity (Wagner et al., 2020).

Our searches will be worldwide and include studies from any country. We will agree the provision and support for translation of potentially relevant papers into English, and provide explicit justification for excluding non-English papers as per MECCIR (R36).

Titles and abstracts will be independently screened by two reviewers to identify potential sources of disagreement. These will be discussed and reviewed by a third senior author in the team. Two reviewers will then screen the full texts of potentially relevant studies and apply the inclusion and exclusion criteria, with recourse to a third reviewer for any records where there is uncertainty.

We will check the references of relevant systematic reviews found in our searches.

Searching other resources

In searching other sources we will seek expert advise. We are working with an expert Advisory Board convened by Campbell and the Youth Endowment Foundation for this purpose. We will search the WHO International Clinical Trials Registry Portal (ICTRP). We will conduct a grey literature search of databases such as Arts and Humanities Citation Index and ProQuest using our search terms. In discussion and agreement with our Advisory Board we will conduct a selected website search including the National Criminal Justice Arts Alliance and other websites with a specific focus on young people and the criminal justice system. We will conduct an Advanced Google Scholar search and sift the first 100 returns using search terms from our search strategy as appropriate. We will include reports that are authored and meet our inclusion criteria. We will use our inclusion and exclusion criteria to select grey literature. We will use our stated approach to extraction and quality assessment. We will revise our grey literature searching alongside advice from experts on the project Advisory Board.

Description of methods used in primary research

The interventions will involve organised arts interventions targeted to the population. We will not include associational studies between arts participation and offending behaviour. We will include controlled study designs. Studies that compare arts interventions to either no intervention, usual care, other types of arts intervention or non-arts control will be included. We will also include studies comparing one type of arts intervention to another. Studies using any qualitative research method to examine context, intervention assumptions, implementation process (including barriers and facilitators), and mechanisms of impact and outcomes will be included. Qualitative studies evaluating how an arts intervention works may be conducted as independent studies or alongside controlled study designs.

Selection of studies

Two review authors will independently assess the titles and abstracts of potential trials identified by the search strategy for their eligibility. We will obtain the full text of studies we think are eligible, or if the eligibility of a study is unclear from the title and abstract. We will exclude studies that do not match the inclusion criteria (see ‘criteria for considering studies for this review’). We will resolve disagreements between review authors regarding inclusion by discussion. If we cannot reach agreement, a third review author will assess relevant studies, and a majority decision will be made. We will not anonymise studies prior to assessment. We will include a PRISMA study flow diagram in the full review to document the screening process.

Data extraction and management

We will frame our work with the PRISMA guidance for reporting systematic reviews (Page et al., 2021). Two review authors will independently extract data from all included studies using a standardised and piloted data extraction form. They will resolve discrepancies and disagreements by consensus. In cases where consensus cannot be achieved, a third review author will assess the article, and a majority decision will be made.

We will extract the following data from quantitative studies included in the review

Study characteristics (aims/objectives, study design, sample size, description of the sample, country, recruitment year(s) and procedure, conflict of interest, funding source)

We will extract the following data from qualitative studies in the review

Study characteristics and context (aims/objectives study design, sampling approach, description of the sample, country, recruitment year(s) and procedure, conflict of interest, funding source)

As arts interventions are complex we will extract detailed information regarding the intervention guided by the MRC guidance on process evaluations of complex interventions (Moore et al., 2015) and items on the TiDiER (Hoffmann et al., 2014) and Cert (Slade et al., 2016) checklists framed by a focus on why, what, who, where, when, how much, how well, tailoring and modifications.

Assessment of risk of bias in included studies

One reviewer will assess the risk of bias or study quality of included studies with recourse to a 2nd reviewer where there is uncertainty. We will use the Cochrane Risk of Bias tool (Higgins et al., 2011) to evaluate included controlled trials. We will assess the following domains for each study: Random sequence generation; Allocation concealment; Blinding of participants and providers; Blinding of assessors; Incomplete outcome data; Selective reporting; Other sources of bias.

We will take a risk to rigour approach to evaluating qualitative studies (Noyes et al., 2018) using the CASP tool for qualitative research (CASP UK) to appraise the rigour and significance of the sampling, data, collection, analysis and reporting of results.

Measures of intervention effect

For continuous outcome measures we will express the size of the intervention effect using the mean difference (MD) when all studies utilised the same measurement scale, or the standardised mean difference (SMD) when studies used different scales, with 95% confidence intervals. When we pool data from different scales for which the direction of interpretation varies, we will normalise the direction of the scales to a common direction. In order to aid interpretation of the pooled effect size, we will back-transform the SMD to the most commonly used outcome scale on the basis of the median standard deviation from trials using that scale when possible.

For dichotomous outcomes we will report the Relative Risk, Odds Ratio or Risk Difference where available from individual included studies. In the event that we pool data in a de novo analysis, we will preferentially report the relative risk as the effect size of interest but will also report the risk difference.

Unit of analysis issues

Unit-of-analysis issues refer to issues regarding clustering (individuals randomised/allocated in clusters), crossover designs, and studies with multiple outcome measurement time-points.

For studies with more than two eligible active treatment groups that are included in a meta-analysis as separate interventions, we will divide the number of participants in the control group between active treatment groups, to avoid double counting (Higgins et al., 2021). For cluster-RCTs, we will seek direct estimates of the effect from an analysis that accounted for the cluster design. When the analysis in a cluster trial does not account for the cluster design, we will use the approximately correct analysis approach, presented in the Cochrane Handbook (Higgins et al., 2021). For cross-over studies, we will only include data from the first phase of the study, when they are available. However, we do not anticipate finding cross-over studies as the design is not appropriate for this research question.

Criteria for determination of independent findings

Where we identify multiple reports for a single study we will only include data from that study once in any given analysis. Where a study reports multiple outcome domains with some conceptual overlap that fit one of our stated outcome domains, the research team will agree which of the measures conceptually best matches our outcome of interest and will only include that measure. This decision will not be made on the basis of the results of these outcomes.

Dealing with missing data

Where there are insufficient data presented in the study report to enter into an analysis, we will request the missing data from the study authors. We will preferentially calculate and extract effect sizes derived from intention to treat analyses. We will evaluate the potential risk of bias introduced by missing data in our assessment of risk of bias, within the domain ‘Incomplete Outcome Data’ and explore the impact of risk of bias through sensitivity analyses.

Assessment of heterogeneity

We will deal with heterogeneity by only combining studies that examine similar interventions. To estimate statistical heterogeneity, we will calculate the Chi² statistic, the between-study variance (Tau²) and the proportion of this variance not due to sampling error (I²). We will use these measures, together with visual inspection of the forest plots to form judgements about heterogeneity. If we identify substantial heterogeneity, we will report it and explore possible causes by prespecified subgroup analysis.

Assessment of reporting biases

We will consider the potential influence of small study biases on review findings. We will use funnel plots to visually explore small study biases where there are at least 10 included studies in a meta-analysis.

Data synthesis

We will conduct separate analyses of the quantitative evidence for the following comparisons: Arts intervention versus no intervention or usual practice; Arts interventions versus non-arts control; Arts interventions versus other arts intervention.

We will pool studies of arts interventions in the primary analysis, including different types of arts, delivery mode and setting. We will use a random-effects model to account for the anticipated heterogeneity between studies. For each comparison of interest, we will conduct separate analyses at short, medium and long-term follow-up. For the primary analysis, we will pool data from studies regardless of the specific population. Where there are inadequate data to enable statistical pooling we will conduct a narrative synthesis of the evidence. For head-to-head comparisons of different types of arts intervention, we will only pool studies if the intervention and comparators are conceptually similar.

In the event that we conduct a narrative synthesis we will separately synthesise studies within the comparisons outlined above, guided by the SWiM guideline (Campbell et al., 2020). We will first summarise all arts interventions in the primary synthesis, including different types of arts, delivery mode and setting and explore potential heterogeneity of treatment effects between studies by considering intervention setting (Custodial, community or school-based interventions) and population age (children and adolescents 8–18 years; young adults 18–25 years.) We will report effect sizes for each reported outcome of interest with estimates of precision. We will include all relevant studies for each comparison and outcome and document the size and risk of bias of those studies in our reporting. We do not plan to present this synthesis in a tabular or graphical format.

See below for details of synthesis of qualitative research.

Subgroup analysis and investigation of heterogeneity

Where there are adequate data and significant heterogeneity is observed in a meta-analysis (I ² ≥ 50%, p < 0.10), we will explore subgroup analyses of quantitative results by type of intervention. To explore whether there is a difference in effects between subgroups, we will use the test for subgroup differences (Deeks et al., 2020).

We will explore the following subgroups:

Intervention setting: Custodial, community or school-based interventions.

Population Age: Children and adolescents 8–18 years; young adults 18–25 years.

We will also take an inductive approach to narratively exploring other potentially important sources of heterogeneity, for example, group vs individual therapy, the use of specific intervention characteristics such as incentives to participation, types of offending (violent, sexual, non-violent).

Sensitivity analysis

When sufficient data are available, we will explore the impact of risk of bias for the primary analyses, by repeating the analyses and excluding studies rated at high risk of bias.

Treatment of qualitative research

We will take a thematic approach to analysing and synthesising data from qualitative studies. This will include line-by-line reading for extraction and preliminary coding, development of descriptive themes and refinement of analytical themes (see Thomas and Harden, 2008). We will conduct our thematic analysis with attention to the complexity of arts interventions for children and young people at-risk or in the criminal justice system. We will map themes from the findings of the qualitative studies to theoretical domains of complexity relating to the intervention itself, the population, implementation of the intervention and the specific context that may impact on the process of delivering and engaging with the interventions. Table 2 outlines the complexity framework for qualitative analysis. We will take a reflexive approach and consult with Advisory Board to seek advice about relevance of themes for policy and practice.

Summary of findings and assessment of the certainty of the evidence

We will use the GRADE system to rank the level of certainty of the evidence (Schünemann, 2020). We will do this for both pooled effects and where we have used narrative synthesis. The GRADE approach uses five considerations (risk of bias, consistency of effect, imprecision, indirectness, and publication bias) to assess the certainty of the body of evidence for each outcome, and uses the following criteria to describe the confidence in the evidence:

high: we are very confident that the true effect lies close to that of the estimate of the effect;

We will decrease the grade rating by one (−1), two (−2), or three (−3) levels, up to a maximum of −3, (or very low) for any criteria, based on the level of concern it raises.

We will use GRADE CERQUAL to asses our confidence in the qualitative evidence. CERQual has four components (methodological limitations, relevance, adequacy and coherence) and uses the following criteria for judging confidence in the body of literature (Lewin et al., 2015)

methodological limitations—the extent to which there are problems in design or conduct of primary studies that contributed evidence to the review

relevance—the extent to which evidence in primary study is applicable (perspective, population, phenomenon of interest, setting)

coherence—the degree to which primary studies provide convincing explanations for patterns

adequacy—the degree of richness and quantity/scope of data

Confidence will be decreased if there are serious or very serious limitations in the design or conduct of the study, the evidence is not relevant to the study objectives, the findings/conclusions are not supported by the evidence or the data are of inferior quality and inadequate in supporting the findings. Confidence will be increased if the study is well designed with few limitations, the evidence is applicable to the context specified in the objectives, the findings/conclusions are supported by evidence and provide a convincing explanation for any patterns found or the data supporting findings are rich and of high quality.