Evaluating the use of text-message reminders and personalised text-message reminders on the return of participant questionnaires in trials,a systematic review and meta-analysis

Abstract

Background:

Randomised controlled trials are widely accepted as the gold standard research methodology for the evaluation of interventions. However, they often display poor participant retention. To prevent this, various participant interventions have been identified and evaluated through the use of studies within a trial. Two such interventions are participant short message service reminders (also known as text-messages) and personalised participant short message service reminders, designed to encourage a participant to return a study questionnaire. While previous studies within a trial have evaluated the effectiveness of these two retention strategies, trialists continue to spend both time and money on these strategies while the evidence remains inconclusive.

Methods:

This systematic review and meta-analysis compared the use of short message service reminders with no short message service reminder and personalised short message service reminders with non-personalised short message service reminders, on participant retention. Eligible studies were identified through advanced searches of electronic databases (MEDLINE, EMBASE and Cochrane Library) and hand-searching of alternative information sources. The review primary outcome was the proportion of study questionnaires returned for the individual study within a trial primary analysis time points.

Results:

Nine eligible studies within a trial were identified, of which four compared short message service versus no short message service and five compared personalised short message service versus non-personalised short message service. For those that compared personalised short message service versus non-personalised short message service, only three were deemed appropriate for meta-analysis. The primary outcome results for short message service versus no short message service concluded that short message service led to a statistically non-significant increase in the odds of study questionnaire return by 9% (odds ratio = 1.09, 95% confidence interval = 0.92 to 1.30). Similarly, comparison of personalised short message service versus non-personalised short message service concluded that personalised short message service caused a statistically non-significant increase in odds by 22% (odds ratio = 1.22, 95% confidence interval = 0.95 to 1.59).

Conclusion:

The effectiveness of both short message service and personalised short message service as retention tools remains inconclusive and further study within a trial evaluations are required. However, as short message services are low in cost, easy to use and generally well accepted by participants, it is suggested that trialists adopt a pragmatic approach and utilise these reminders until further research is conducted. Given both the minimal addition in cost for studies already utilising short message service reminders and some evidence of effect, personalisation should also be considered.

Keywords

SMS personalised text-message retention SWAT study within a trial randomised controlled trials

Background

Randomised controlled trials (RCTs) are considered the gold standard research methodology to evaluate the effectiveness of both healthcare and social care interventions;¹ however, they often display poor participant retention.^2,3 When designed for use within research and clinical practice, the response rate for patient-reported outcome measures (PROMs) often falls below 50%.⁴ This is true for remote questionnaires, by which PROMs are commonly administered.⁵ The non-return of questionnaires may introduce bias, reduce statistical power or jeopardise a study’s internal and external validity.^6–9 Consequently, trials may be prematurely terminated or require additional funding and resource to counteract the impact of poor retention, thus contributing to research waste.^10–12

One strategy often employed within RCTs to increase the return of participant questionnaires is the use of short message service (SMS) reminders.^13,14 As a retention tool, SMS reminders (also known as text-message reminders) are relatively low in cost, easy to implement, rapid to send and can be both automated and personalised.^15,16 They are also widely accessible and inclusive, with a report published in the year 2020 evidencing that 95% of all UK adults use a mobile telephone.¹⁷ However, the evidence for their effectiveness as a retention tool remains both inconclusive and inconsistent.

Previous systematic reviews of retention methods concluded the use of electronic prompts and reminders to have either a small, positive impact or no impact on participant retention, with no statistical significance.^12,18,19 Furthermore, authors of the most recently published Cochrane review,¹² considered the evidence to be either low or very low GRADE. While four studies within a trial (SWATs) were found to compare the impact of SMS reminders versus no reminder (independent of other retention strategies), a meta-analysis of theses SWATs does not appear to have been performed.^20–22 For the use of personalised SMS reminders specifically, the evidence is far less abundant. Two previous studies reported conflicting findings, with personalised SMS reminders increasing and decreasing participant retention, neither of which showed statistical significance.^23,24

In addition to this, working as a member of the research team for the PROMoting THE Use of SWATs (PROMETHEUS) programme²⁵ highlighted additional SWATs, which had both evaluated the use of SMS reminders and were not included in previous reviews. This programme was led by York Trials Unit and was designed to pump prime host trial teams with the funding and support to conduct a recruitment or retention SWAT.²⁵ Therefore, to update the limited evidence with that of additional SWAT evaluations, we undertook a systematic review and meta-analysis to evaluate the effectiveness of SMS reminders versus no reminders and personalised SMS reminders versus non-personalised SMS reminders on participant retention.

Methods

The protocol was prospectively pre-registered on PROSPERO, registration ID: CRD42020227342.

Inclusion and exclusion criteria

Eligible studies were randomised SWAT design, which were embedded within a host RCT, with an adult population (aged ≥ 18 years) and which collected PROMs data using study questionnaires, for which the proportion returned was evaluated as the study’s primary outcome. Due to the nature of the review, two different interventions and their corresponding comparators were eligible for inclusion (see Table 1).

Table 1.

Summary of the review eligibility and ineligibility study criteria.

PICOS framework	Eligibility criteria	Ineligibility criteria
Population	Participants aged ≥ 18 years.	Participants aged < 18 years.
Intervention	An SMS reminder (either personalised or non-personalised) sent to the participant to encourage their return of a study questionnaire	Any electronic reminder alternative to an SMS, for example, e-mail reminder, application notification or phone call
Comparator	No SMS reminder or a non-personalised SMS reminder	Any electronic reminder alternative to an SMS, for example, e-mail reminder, application notification or phone call
Outcome	Primary outcome. The proportion of participants who returned either a postal or electronic study questionnaire at the primary analysis time point defined by the specific retention trial
Study design	SWATs designed as an RCT embedded within a host RCT. SWATs of a factorial RCT design	Host trials designed as cluster design RCTs and all non-randomised study types. SWATs designed as non-randomised study types

SMS: short message service; SWAT: study within a trial; RCT: randomised controlled trial.

Information sources

An advanced online search of the Cochrane Library, MEDLINE and EMBASE databases was performed from inception to June 2023 (see Supplemental material – Appendix 1). The following information sources were hand-searched by one reviewer: Trial Forge website, F1000 Research – Open Research Platform, The Northern Ireland Network for Trials Methodology Research SWAT Repository and the reference lists of both review-included studies and relevant systematic reviews identified through searching (see Supplemental Appendix 2). Members of the PROMETHEUS programme were also contacted to identify any relevant SWATs. This programme was led by York Trials Unit and was designed to pump prime host trial teams with the funding and support to conduct a recruitment or retention SWAT, therefore, being deemed as a relevant information source.²⁵

Screening and data extraction was performed by two independent reviewers, for which a third independent reviewer resolved any discrepancies. Data were collected using a pre-piloted data extraction tool (see Supplemental Appendix 3) to capture the key information of individual studies (including the population type, intervention, comparator, study design) and their measured outcomes.

Review outcomes

The primary outcome was the proportion of participant study questionnaires returned at the SWAT primary analysis time point, reported as the number of questionnaires returned, of the total to be returned (provided as an odds ratio (OR) and 95% confidence interval (CI)). Where a corresponding OR and 95% CI were not reported, they were calculated by the review primary author.

Secondary outcomes were the proportion of participants who returned a study questionnaire at other study time points and the cost-effectiveness of the intervention, defined as the cost of retaining one additional participant.

Data synthesis

Narrative synthesis and meta-analysis compared the use of SMS reminders versus no reminder and personalised versus non-personalised reminders. As aspects of text messaging dependency have been shown to decline with increasing age,²⁶ subgroup analysis for age was also performed (both < and ≥ 65 years). The age of 65 years was chosen as adults of this age and older are often considered ‘older adults’.²⁷ With previous research also evidencing participant response to both postal and electronic questionnaires to be significantly greater for shorter, rather than longer, questionnaires, a subgroup analysis of questionnaire page length was deemed important to explore.¹⁴ Aligned with the review by Partha Sarathy et al.,¹⁸ short and long questionnaires were categorised as both < and ≥ 10 pages, respectively. In addition, as the impact of participant attrition is greatest for trials with long-term follow-up,²⁸ subgroup analysis of the duration of time between enrolment of SWAT participants to the host trial and the SWAT primary analysis time point (both ≤ and > 6 months) was undertaken. These subgroups were chosen as trial participation of > 6 months has been observed as a strong barrier to participation.²⁹

A post hoc subgroup analysis compared the intent for a reminder to be received either before, or after, a retention time point, defined as both a prompt and reminder, respectively, by Gillies et al.¹³

All analyses used a fixed effects model. Funnel plots were generated to assess for publication bias and the Cochrane Risk of Bias 2 tool was completed by two independent reviewers to evaluate the quality of included studies.

Factorial SWATs with at least one eligible intervention and comparator arm were included. The data from ineligible SWAT arms were not extracted or included within the analysis. For factorial SWATs with more than one eligible intervention or comparator arm, the data from the arms were combined assuming that no evidence of interaction between the review eligible intervention and additional intervention had been reported by the authors.

The cost-effectiveness analysis, as detailed in Supplemental Appendix 4, derived the cost of retaining one additional participant through use of the intervention.

Results

Eight studies, which reported a total of nine SWATs, were deemed eligible for inclusion (see Figure 1).^{20–24,30–32} Keding et al. sequentially embedded two independent SWATs within the same host trial.²¹ Four SWATs compared the use of SMS versus no SMS^20–22 and five compared personalised versus non-personalised SMS^{23,24,30–32} (see Table 2).

Figure 1.

The PRISMA 2020 flow diagram of the studies identified, screened and included within the review.

Table 2.

The PICOS summary table for all studies included within the review.

Lead author (year of publication)	Host trial information				SWAT information
Lead author (year of publication)	Trial acronym	Study design	Research area	Location	Populationn = total number of participants randomised (n = total number of participants analysed)	Intervention(s)	Comparator(s)	Primary outcome(s)	Study design
SMS versus no SMS
Bradshaw (2020)	BEEP	Two-armed parallel RCT	Dermatology	UK hospitals	Parents, the majority of which were mothers.n = 1394(n = 1394)	SWAT arm 1:Non-personalised SMS sent to the participant 1 day before their study questionnaire was sent (either electronically or via post). SMS were sent prior to the 3-, 6-, 12- and 18-month questionnaires. Participants also sent a letter at 22 months informing them that they will receive a £10 shopping voucher at their 24-month visit.SWAT arm 2:Non-personalised SMS sent to the participant 1 day before their study questionnaire was sent (either electronically or via post). SMS were sent prior to the 3-, 6-, 12- and 18-month questionnaires. Participants also received a letter at 22 months, which enclosed a £10 shopping voucher	SWAT arm 3:No SMS sent to the participant. Participants sent a letter at 22 months informing them that they will receive a £10 shopping voucher at their 24-month visit.SWAT arm 4:No SMS sent to the participant.Participants also received a letter at 22 months, which enclosed a £10 shopping voucher	1. Collection of data via the chosen method of questionnaire (postal or electronic) at 3, 6, 12 and 18 months.2. Collection of the BEEP host-trial primary outcome at 24 months, during a home or clinic visit	2 × 2 factorial design
Keding (2016)Trial 1	ACUDep	Three-armed pragmatic RCT	Acupuncture and depression	Primary care practices in Yorkshire and the North of England	Mixed gender population.n = 523(n = 523)	Participants were sent a non-personalised SMS on the same day as their 3-month postal study questionnaire	No SMS was sent to the participant	The proportion of participants who returned a valid 3-month study questionnaire to the trial team	Two-armed parallel RCT
Keding (2016)Trial 3	ACUDep	Three-armed pragmatic RCT	Acupuncture and depression	Primary care practices in Yorkshire and the North of England	Mixed gender population.n = 523(n = 523)	Participants were sent a non-personalised SMS 4 days after the 9-month study questionnaire was posted	No SMS was sent to the participant	The proportion of participants who returned a valid 9-month study questionnaire to the trial team	Two-armed parallel RCT
Starr (2015)	SUSPEND	Three-armed parallel RCT	Ureteric colic	Urology departments of UK hospitals	Mixed gender population.n = 418^a(n = 418^a)	Participants were sent a non-personalised, pre-notification SMS prior to both the 4- and 12-week questionnaires.Two SWAT arms ineligible for review^b:Participants who did not return their 4- or 12-week study questionnaire were also randomised to receive either an e-mail or postal reminder. The data from these two SWAT arms were not eligible for or included within this review	No SMS was sent to the participant	The questionnaire response rate at 4- and 12-week study time points	2 × 2 partial factorial design
Personalised SMS versus non-personalised SMS
Cochrane (2020)	OTIS	Cohort RCT	Fall prevention in the elderly	Participant homes in the United Kingdom	Mixed gender population.n = 403(n = 283)	Participants were sent a personalised SMS 4 days after their 4-month questionnaire was posted. The SMS was personalised with the participants title and surname	Participants were sent a non-personalised SMS 4 days after their 4-month questionnaire was posted	The proportion of participants who returned their 4-month postal questionnaire	Two-armed parallel RCT
Coleman (2021)	MiQuit-3	Two-armed parallel RCT	Smoking cessation in pregnancy	UK hospital antenatal clinics	Female population.n = 194(n = 189)	SWAT arm 1:Participants were sent a personalised SMS 1 week prior to their 36-week gestational follow-up study phone call. The SMS was personalised with the participant’s name.SWAT arm 2:Participants were sent a personalised SMS 1 day before their 36-week gestational follow-up study phone call	SWAT arm 3:Participants were sent a non-personalised SMS 1 week prior to their 36-week gestational follow-up study phone call.SWAT arm 4:Participants were sent a non-personalised SMS 1 day before their 36-week gestational follow-up study phone call	The completion rate, defined as the proportion of the questionnaires completed via phone call within the 14-day follow-up window.The secondary outcome of the SWAT was the proportion of questionnaires completed via any method of follow-up (including phone call, postal, e-mail/web or SMS)	2 × 2 factorial design
Cureton (2021)	GRASP	2 × 2 factorial design	Shoulder pain	UK muscoskeletal clinics	Mixed gender population.n = 618(n = 618)	Participants were sent a personalised SMS at the same time as their 6-month follow-up study questionnaire was posted. The SMS was personalised with the participant’s title and preferred name	Participants were sent a non-personalised SMS at the same time as their 6-month follow-up study questionnaire was posted	The questionnaire response rate, defined as the proportion of 6-month GRASP follow-up questionnaires returned by participants	Two-armed parallel RCT
Herbert(2024)	MAGIC	Two-armed parallel RCT	Paediatric surgery	UK hospitals	Mixed gender population.n = 100(n = 100)	Participants were sent a personalised SMS 24–48 h prior to their telephone call follow-up, which was scheduled 14 days post-surgery. The SMS was personalised with the participant’s first name	Participants were sent a non-personalised SMS 24–48 h prior to their telephone call follow-up, which was scheduled 14 days post-surgery	The proportion of participants who were successfully contacted and completed any of the questionnaires, over the telephone within the follow-up window (day 14 + 7 days)	Two-armed parallel RCT
Mitchell (2020)	KReBS	Two-armed parallel RCT	Knee replacement bandage	UK hospitals	Mixed gender population.n = 1470(n = 1465)	Participants were sent a personalised SMS 4 days after the 12-month study questionnaire was posted. The SMS was personalised with the participants’ title and surname	Participants were sent a non-personalised SMS 4 days after the 12-month study questionnaire was posted	The primary outcome was the proportion of participants who returned a 12-month study questionnaire	Two-armed parallel RCT

SWAT: study within a trial; SMS: short message service; RCT: randomised controlled trial.

Total number of participants for the two review eligible SWAT arms only.²²

SWAT arm data not extracted by reviewers or included within the review.

Narrative synthesis

SMS versus no SMS

Of the four SWATs, either a two-armed parallel RCT (n = 2), 2 × 2 factorial (n = 1), or partial factorial (n = 1) design was used (see Table 2). In addition to SMS reminders, Bradshaw et al. evaluated the impact of a participant monetary incentive but reported no interaction between interventions meaning the SWAT data could be appropriately combined to compare SMS versus no SMS exclusively.²⁰ Starr et al.²² also evaluated the use of a postal/email reminder, which was ineligible for review; therefore, only the data from the two SWAT arms comparing SMS and no SMS was extracted. In light of this, the SWAT authors reported no evidence of interaction between the use of SMS reminders and postal/email reminders.

Collectively the SWATs studied a mixed gender population of similar age (mean ages 31.3 to 41.0 years old) and recruited from UK-based hospitals (see Supplemental Appendix 5). However, the SWAT population size varied considerably (range of 418 to 1394 participants; see Table 2). The SMS intervention for all four SWATs were non-personalised and sent only once per follow-up time point used. Each included the host trial acronym, reference to the arrival of the study questionnaire and it’s return or completion. All SMS were sent prior to the study questionnaire, except for Keding et al. Trial 3 (see Table 2).²¹ Each SWAT sent questionnaires primarily via post (see Supplemental Appendix 5), excluding Bradshaw et al. which utilised both electronic and postal questionnaires dependent on participant preference.²⁰ Where reported, questionnaire page length varied considerably (ranging from 4 to 12 pages; see Supplemental Appendix 5).

Two of the four SWATs provided a measure of the participants socioeconomic status. Bradshaw et al.²⁰ reported similar median values for the English Index of Multiple Deprivation (IMD) across the four SWAT arms (reported as 6 for those who received an SMS and monetary incentive, 5 for an SMS and no monetary incentive and 6 for no SMS, either with or without a monetary incentive). Therefore, across the arms, the overall median decile for the English IMD was 5, suggesting that for this SWAT, the participants were generally from areas of average depravity. For the SWAT by Keding et al.,²¹ the mean age of participant’s leaving education was 18.1 years (SD = 3.89). The participant employment status was also reported as 214 (41%) working full time, 100 (19%) working part time, 65 (12%) looking after home, 61 (12%) unable to work, 21 (4%) in full-time education, 18 (3%) retired, 30 (6%) ‘other’ and 14 (3%) provided no information. None of the four SWATs reported on race or ethnicity.

The two SWATs by Bradshaw et al. and Starr et al. were the only two to evaluate the return of questionnaires at subsequent time points to the SWAT primary end point.^20,22 Across the SWATs, the primary end point varied considerably (ranging from 4 weeks post-randomisation to 9 months; see Supplemental Appendix 5). Data from the return of questionnaires at 24 months studied by Bradshaw et al.²⁰ was deemed irrelevant as it was collected via a face-to-face visit and not a remote data collection method – and as such was not extracted.

For the two SWATs by Keding et al., the primary OR and corresponding 95% CI was calculated by the review primary author.²¹ This showed the return to be greater for the control group, than the intervention group (ORs = 0.87 and 0.92, respectively). Whereas Bradshaw et al. and Starr et al. found the proportion of returned questionnaires at the SWAT primary analysis time point was greater for the intervention group (ORs = 1.17 and 1.24).^20,22 All results were not considered statistically significant. At subsequent time points, these findings showed the opposite for Starr et al. (OR = 0.97) and fluctuated for Bradshaw et al. (ORs = 1.02, 0.87 and 1.03).

Personalised SMS versus non-personalised SMS

Across the SWATs, either a two-armed parallel RCT (n = 4) or 2 × 2 factorial design (n = 1) was used (see Table 2). Coleman et al. evaluated the timing and personalisation of SMS factorially;³⁰ however, as there was no evidence of interaction between interventions, the SWAT data were combined to evaluate personalisation exclusively. For all five SWATs, the number of recruited participants (range of 100 to 1470 participants), the mean participant age (range of 27.1–77.8 years) and research area varied considerably (see Table 2 and Supplemental Appendix 5). However, most recruited a mixed gender population from either a hospital or clinic within the United Kingdom.

SMS interventions across all five SWATs were sent only once per follow-up time point used; however, the timing in which they were sent varied (see Table 2). The intervention and comparator SMS included the host trial acronym, reference to the arrival or phone call receival of the questionnaire and the return or completion of it. They were issued at the same time and frequency; however, personalised SMS included the participant’s name. The method of questionnaire delivery and page length (range of 4.5 to 15 pages) varied considerably (see Supplemental Appendix 5).

Of the five SWATs, only Coleman et al. provided a measure of the participant’s ethnicity.³⁰ In this SWAT, 168 participants (88.9%) self-reported their ethnicity as Caucasian. This finding was similar for each of the SWAT arms (92.4% for those who received the personalised SMS and 85.6% for those who received a non-personalised SMS). None of the five SWATs reported on the participant’s socioeconomic status.

Each SWAT evaluated the primary outcome at one time point only (see Supplemental Appendix 5). For Coleman et al.³⁰ and Herbert et al.,³² the duration of time between host trial enrolment and the SWAT primary analysis time point differed for each participant due to the nature of the studies (see Supplemental Appendix 5). Coleman et al. was also the only SWAT to place a time restriction on questionnaire completion (14 days). For Cureton et al.,³¹ Herbert et al.³² and Mitchell et al.,²⁴ the proportion of returned questionnaires was greater for the intervention group (ORs: 1.63, 1.49 and 1.09, respectively, see Table 2). The opposite was true for Cochrane et al.²³ and Coleman et al.³⁰ (ORs: 0.64 and 0.61, respectively). The OR and corresponding 95% CI for Cureton et al. was calculated by the review primary author and had a CI, which could be considered statistically significant (95% CI = 1.01–2.63).

Meta-analysis

Studies by Coleman et al. and Herbert et al. were not included in the meta-analysis as their questionnaires were completed via phone call.^30,32 It is possible that the introduction of human-to-human interaction may stimulate a different response to that of either postal or electronic remote data collection.

SMS versus no SMS

The primary analysis, which included all four SWATs that compared SMS versus no SMS, favoured the use of SMS for the return of participant study questionnaires (OR = 1.09, 95% CI = 0.92 to 1.30, I² = 0%, see Figure 2). Subgroup analysis of the two SWATs with a questionnaire length of 10 pages or greater (Keding et al.²¹), favoured the use of no SMS (OR = 0.90, 95% CI = 0.66 to 1.23, I² = 0%). In contrast, the three studies with a duration of 6 months or less between participant enrolment to the host trial and the SWAT primary analysis time point (Bradshaw et al.,²⁰ Keding et al.²¹ Trial 1 and Starr et al.²²), favoured the use of SMS (OR = 1.13, 95% CI = 0.93 to 1.37, I² = 0%). The results of this analysis were identical to that for the use of an SMS ‘prompt’ rather than ‘reminder’. The influence of mean participant age could not be evaluated as this was below 65 years for all four SWATs.

Figure 2.

Meta-analysis of studies evaluating SMS versus no SMS on the return of study questionnaires at the individual SWAT primary analysis time points.

Personalised SMS versus non-personalised SMS

Primary analysis of the three SWATs that compared personalised versus non-personalised SMS favoured the use of personalised SMS on participant questionnaire return (OR = 1.22, 95% CI = 0.94 to 1.59, see Figure 3); however, there was heterogeneity (I² = 16%). Subgroup analysis of the two SWATs with a mean participant age of 65 years or older (Cochrane et al.²³ and Mitchell et al.²⁴) favoured the use of personalised SMS (OR = 1.07, 95% CI = 0.78 to 1.47, I² = 0%). The results of this analysis were identical to that for the use of an SMS ‘prompt’ rather than ‘reminder’. Similarly, subgroup analysis of Mitchell et al.²⁴ and Cureton et al.,³¹ which used questionnaires of 10 pages or more, favoured personalised SMS (OR = 1.24, 95% CI = 0.95 to 1.62); however, with moderate heterogeneity (I² = 46%). Again, for SWATs with 6 months or less between participant enrolment to the host trial and the SWAT primary analysis time point (Cochrane et al.²³ and Cureton et al.³¹), personalised SMS were favoured (OR = 1.53, 95% CI = 0.96 to 2.63, I² = 0%).

Figure 3.

Meta-analysis of studies evaluating personalised SMS versus non-personalised SMS on the return of study questionnaires at the individual SWAT primary analysis time points.

Publication bias and risk of bias

Funnel plots for SWATs comparing SMS versus no SMS and personalised versus non-personalised SMS suggested the review primary analyses suffered very little and some publication bias, respectively. The Cochrane Risk of Bias 2 tool (see Supplemental Appendix 6) concluded that all eight of the nine SWATs showed ‘some concern’; however, no SWAT had a rating of ‘high risk’ for any domain.

Cost-effectiveness analysis

The only SWAT to report the cost of both a personalised and non-personalised SMS was Cochrane et al.²³ (£0.096 and £0.048, respectively). Therefore, these values informed the cost-effectiveness analysis. Analysis showed the cost to retain an additional participant when using an SMS compared with no SMS is £4.80. The cost to retain an additional participant when using a personalised compared with a non-personalised SMS is £1.60, in addition to the cost of sending a non-personalised SMS, resulting in a total of £1.65.

Discussion

When comparing the use of SMS versus no SMS, SMS favoured a non-significant increase in the return of participant questionnaires (see Table 3), as aligned with the findings of previous reviews. Similarly, when compared with non-personalised SMS, personalised SMS favoured a non-significant increase in participant questionnaire return (see Table 3). As the increase in odds was much greater for the latter comparison, it should be considered that personalisation, rather than the medium of personalisation, has a greater impact on retention. Therefore, it could be useful to incorporate personalisation for other retention tools.

Table 3.

The pooled OR (95% CI) for the primary, subgroup and post hoc analyses for both the comparison of SMS versus no SMS and personalised versus non-personalised SMS.

Analysis	Pooled OR (95% CI)
Analysis	SMS versus no SMS	Personalised versus non-personalised SMS
Primary analysis
The return of study questionnaires for the SWAT primary analysis time point	1.09 (0.92 to 1.30)	1.22 (0.95 to 1.59)
Subgroup analysis
Mean participant age (< 65 years)	No meta-analysis performed	No meta-analysis performed
Mean participant age (≥ 65 years)	No meta-analysis performed	1.07 (0.87 to 1.47)
Study questionnaire page length (< 10 pages)	No meta-analysis performed	No meta-analysis performed
Study questionnaire page length (≥ 10 pages)	0.90 (0.66 to 1.23)	1.24 (0.95 to 1.65)
Duration of time between participant host trial enrolment and the SWAT primary analysis time point (≤ 6 months)	1.13 (0.93 to 1.37)	1.53 (0.96 to 2.43)
Duration of time between participant host trial enrolment and the SWAT primary analysis time point (> 6 months)	No meta-analysis performed	No meta-analysis performed
Post hoc analysis
The use of an SMS sent prior to the study questionnaires arrival/date to be completed (SMS prompt)	1.13 (0.93 to 1.37)	No meta-analysis performed
The use of an SMS sent following the study questionnaires arrival/date to be completed (SMS prompt)	No meta-analysis performed	1.07 (0.78 to 1.47)

OR: odds ratio; CI: confidence interval; SMS: short message service; SWAT: study within a trial.

While the review findings provide an important contribution to the existing research, they are not conclusive. Aligned with Trial Forge Guidance 2, the inconsistency in the direction of the intervention’s effect for the individual studies, suggests that further SWAT evaluations are required.³³ Therefore, as informed by this same guidance,³³ we suggest that trialists adopt a pragmatic approach of utilising personalised SMS as a low cost, easy to use intervention, until further research suggests otherwise. This is superior to using interventions (of potentially higher cost) that have no evidence of a positive effect on retention.

As the impact of retaining one additional study participant can vary between studies, there is no uniform measure for the clinical significance and impact of a retention tool. Therefore, trialists should consider the use of an intervention such as SMS reminders on a study-specific basis.

Strengths and limitations

In contrast to previous research, this review evaluated the use of SMS (both personalised and non-personalised), independent of other electronic retention tools, therefore, answering a more specific research question. In addition, the review included three newly published SWATs, which were not included in the review by Gillies et al. No study presented a ‘high’ risk of bias and the review’s primary analyses displayed either none or minimal heterogeneity. While the review is susceptible to language bias, due to including only English reported studies, the advanced electronic database search, which was not restricted by language, returned no further relevant studies.

For this review, a wide range of both published and unpublished information sources were searched, mostly by two independent reviewers. The host trials within which the SWATs were embedded mostly studied mixed gender populations and spanned a diverse collection of research areas, increasing the results generalisability. The minimally reported socioeconomic information appears to align with that for the UK population, with UK residents leaving education aged 19.3 years on average (as reported in 2018)³⁴ and 52.7% of residents in England and Wales recorded as either employed or self-employed in 2021.³⁵ Similarly, a Median Decile of English IMD value of 5 suggests an average deprivation status for England.³⁶ However, as seven SWATs did not report socioeconomic information, this cannot be confidently determined for the review findings overall.

The review findings are only applicable to studies within the United Kingdom, which utilise questionnaires completed via remote data collection methods and for comparison of SMS versus no SMS specifically, a narrow participant age range. Of the nine SWATs, only one reported on participant ethnicity, jeopardising the generalisability of the review findings to the wider UK population. For the one SWAT that did report on ethnicity, the 2021 Census data, which report 81.7% of residents in England and Wales to be White,³⁷ suggests that a greater percentage of Caucasians were included in the SWAT compared to the UK population. However, this should be improved for most research as ethnic minorities are commonly under-represented in trials.³⁸

It was also not considered that the individual host trials measured the return of study questionnaires by their own definition, which may have differed in completeness and time of receival. It is also possible that the host trials implemented additional SWATs, or retention strategies, which were not accounted for. The review may have suffered some publication bias and some subgroup analyses presented heterogeneity. In addition, the ORs calculated by the review primary author may be less accurate than those reported by SWAT authors, as the latter have access to raw data and may adjust for covariates.

Conclusion

Although not statistically significant, the use of an SMS increases participant retention and retention can be further increased when SMSs are personalised. Therefore, as SMSs are low in cost, easy to implement and generally well accepted by participants, it is suggested that future trials adopt a pragmatic approach and use personalised SMS to potentially increase participant retention. Future evaluations of both personalised and non-personalised SMS in populations outside of the United Kingdom or evaluations of SMS compared with no SMS within populations aged either below 30 or above 40 years of age, remain useful.

Supplemental Material

sj-docx-1-ctj-10.1177_17407745251320888 – Supplemental material for Evaluating the use of text-message reminders and personalised text-message reminders on the return of participant questionnaires in trials, a systematic review and meta-analysis

Supplemental material, sj-docx-1-ctj-10.1177_17407745251320888 for Evaluating the use of text-message reminders and personalised text-message reminders on the return of participant questionnaires in trials, a systematic review and meta-analysis by Laura Doherty, Catherine Arundel, Elizabeth Coleman, Ailish Byrne and Katherine Jones in Clinical Trials

Supplemental Material

sj-docx-2-ctj-10.1177_17407745251320888 – Supplemental material for Evaluating the use of text-message reminders and personalised text-message reminders on the return of participant questionnaires in trials, a systematic review and meta-analysis

Supplemental material, sj-docx-2-ctj-10.1177_17407745251320888 for Evaluating the use of text-message reminders and personalised text-message reminders on the return of participant questionnaires in trials, a systematic review and meta-analysis by Laura Doherty, Catherine Arundel, Elizabeth Coleman, Ailish Byrne and Katherine Jones in Clinical Trials

Supplemental Material

sj-docx-3-ctj-10.1177_17407745251320888 – Supplemental material for Evaluating the use of text-message reminders and personalised text-message reminders on the return of participant questionnaires in trials, a systematic review and meta-analysis

Supplemental material, sj-docx-3-ctj-10.1177_17407745251320888 for Evaluating the use of text-message reminders and personalised text-message reminders on the return of participant questionnaires in trials, a systematic review and meta-analysis by Laura Doherty, Catherine Arundel, Elizabeth Coleman, Ailish Byrne and Katherine Jones in Clinical Trials

Supplemental Material

sj-docx-4-ctj-10.1177_17407745251320888 – Supplemental material for Evaluating the use of text-message reminders and personalised text-message reminders on the return of participant questionnaires in trials, a systematic review and meta-analysis

Supplemental material, sj-docx-4-ctj-10.1177_17407745251320888 for Evaluating the use of text-message reminders and personalised text-message reminders on the return of participant questionnaires in trials, a systematic review and meta-analysis by Laura Doherty, Catherine Arundel, Elizabeth Coleman, Ailish Byrne and Katherine Jones in Clinical Trials

Supplemental Material

sj-docx-5-ctj-10.1177_17407745251320888 – Supplemental material for Evaluating the use of text-message reminders and personalised text-message reminders on the return of participant questionnaires in trials, a systematic review and meta-analysis

Supplemental material, sj-docx-5-ctj-10.1177_17407745251320888 for Evaluating the use of text-message reminders and personalised text-message reminders on the return of participant questionnaires in trials, a systematic review and meta-analysis by Laura Doherty, Catherine Arundel, Elizabeth Coleman, Ailish Byrne and Katherine Jones in Clinical Trials

Supplemental Material

sj-docx-6-ctj-10.1177_17407745251320888 – Supplemental material for Evaluating the use of text-message reminders and personalised text-message reminders on the return of participant questionnaires in trials, a systematic review and meta-analysis

Supplemental material, sj-docx-6-ctj-10.1177_17407745251320888 for Evaluating the use of text-message reminders and personalised text-message reminders on the return of participant questionnaires in trials, a systematic review and meta-analysis by Laura Doherty, Catherine Arundel, Elizabeth Coleman, Ailish Byrne and Katherine Jones in Clinical Trials

Footnotes

Declaration of conflicting interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: L.D., C.A. and E.C. declare that they have previously received funding from the PROMETHEUS programme (MR/R013748/1) and an NIHR CTU infrastructure grant (NIHR132547). All other authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Laura Doherty

Catherine Arundel

Supplemental material

Supplemental material for this article is available online.

References

Torgerson

. Designing randomised trials in health, education and the social sciences: an introduction. 1st ed. Basingstoke: Palgrave Macmillan, 2008.

Treweek

Gillies

. Trial retention: it’s time for Cinderella to go to the ball. On Medicine, 1 September 2017, https://blogs.biomedcentral.com/on-medicine/2017/09/01/trial-retention-its-time-for-cinderella-to-go-to-the-ball/ (accessed 11 April 2024).

Frampton

Shepherd

Pickett

, et al. Digital tools for the recruitment and retention of participants in randomised controlled trials: a systematic map. Trials 2020; 21: 478.

Warwick

Hutrya

Politzer

, et al. Small social incentives did not improve the survey response rate of patients who underwent orthopaedic surgery: a randomized trial. Clin Orthop Relat Res 2019; 477: 1648–1656.

Neve

Van Benthem

Stigglebout

, et al. Response rate of patient reported outcomes: the delivery method matters. BMC Med Res Methodol 2021; 21: 220.

Nunan

Aronson

Bankhead

. Catalogue of bias: attrition bias. BMJ Evid Based Med 2018; 23(1): 21–22.

Page

Persch

. Recruitment, retention, and blinding in clinical trials. Am J Occup Ther 2013; 67(2): 154–161.

Fogel

. Factors associated with clinical trials that fail and opportunities for improving the likelihood of success: a review. Contemp Clin Trials Commun 2018; 11: 156–164.

Patino

Ferreira

. Internal and external validity: can you apply research study results to your patients? J Bras Pneumol 2018; 44(3): 183.

10.

Williams

Tse

DiPiazza

, et al. Terminated trials in the clinical trials.gov results database: evaluation of availability of primary outcome data and reasons for termination. PLoS One 2015; 10: e0127242.

11.

Salman

RA-S

Beller

Kagan

, et al. Increasing value and reducing waste in biomedical research regulation and management. Lancet 2014; 383: 176–185.

12.

Serdar

Cihan

Yucel

, et al. Sample size, power and effect site revisited: simplified and practical approaches in pre-clinical, clinical and laboratory studies. Biochem Med 2021; 31: 010502.

13.

Gillies

Kearney

Keenan

, et al. Strategies to improve retention in randomised trials. Cochrane Database Syst Rev 2021; 3: MR000032.

14.

Edwards

Roberts

Clarke

, et al. Methods to increase response to postal and electronic questionnaires. Cochrane Database Syst Rev 2009; 2009: MR000008.

15.

Schwebel

Larimer

. Using text message reminders in health care services: a narrative literature review. Internet Interv 2018; 13: 82–104.

16.

Hughes-Morley

Torgerson

. SWAT 35: personalised text messages versus standard text message prompts for increasing response to postal questionnaires. SWAT Store – The Northern Ireland Network for Trials Methodology, 2015, https://www.qub.ac.uk/sites/TheNorthernIrelandNetworkforTrialsMethodologyResearch/SWATSWARInformation/Repositories/SWATStore/ (accessed 11 April 2024).

17.

Ofcom. Adults’ media use & attitudes report2020, 2020, https://www.ofcom.org.uk/__data/assets/pdf_file/0031/196375/adults-media-use-and-attitudes-2020-report.pdf (accessed 11 April 2024).

18.

Partha Sarathy

Kottam

Parker

, et al. Timing of electronic reminders did not improve trial participant questionnaire response: a randomized trial and meta-analyses. J Clin Epidemiol 2019; 122: 70–77.

19.

Clark

Gillies

Torgerson

, et al. Evidence pack – retention: electronic prompts (ID Ret2). Trial Forge, 2020, https://www.trialforge.org/resource/evidence-pack-retention-electronic-prompts-id-ret2/ (accessed 12 April 2024).

20.

Bradshaw

Montgomery

Williams

, et al. Two-by-two factorial randomised study within a trial (SWAT) to evaluate strategies for follow-up in a randomised prevention trial. Trials 2020; 21: 529.

21.

Keding

Brabyn

MacPherson

, et al. Text message reminders to improve questionnaire response rates. J Clin Epidemiol 2011; 79: 90–95.

22.

Starr

McPherson

Forrest

, et al. SMS text pre-notification and delivery of reminder e-mails to increase response rates to postal questionnaires in the SUSPEND trial: a factorial design, randomised controlled trial. Trials 2015; 16: 295.

23.

Cochrane

Welch

Fairhurst

, et al. An evaluation of a personalised text message reminder compared to a standard text message on postal questionnaire response rates: an embedded randomised controlled trial. F1000Res 2020; 99: 154.

24.

Mitchell

Cook

Dean

, et al. An embedded randomised controlled retention trial of personalised text messages in an orthopaedic setting. F1000Res 2020; 9: 591.

25.

Doherty

Parker

Arundel

, et al. PROMoting the use of studies within a trial (PROMETHEUS): results and experiences from a large programme to evaluate the routine embedding of recruitment and retention strategies within randomised controlled trials routinely. Res Methods Med Health Sci 2022; 4: 113–122.

26.

Ferraro

. Does age impact text-message dependence? J Gen Psychol 2018; 145(2): 199–207.

27.

Age

. Later life in the United Kingdom 2019, 2019, https://www.ageuk.org.uk/globalassets/age-uk/documents/reports-and-publications/later_life_uk_factsheet.pdf (accessed 25 April 2024).

28.

Herbert

Kasza

. Analysis of randomised controlled trials with long-term follow-up. BMC Med Res Methodol 2018; 18: 48.

29.

Desai

. Recruitment and retention of participants in clinical studies: critical issues and challenges. Perspect Clin Res 2020; 11(2): 51–53.

30.

Coleman

Whitemore

Clark

, et al. Pre-notification and personalisation of text messages to increase questionnaire completion in a smoking cessation pregnancy RCT: an embedded randomised factorial trial. F1000Res 2021; 10: 637.

31.

Cureton

Marian

Barber

, et al. Randomised study within a trial (SWAT) to evaluate personalised versus standard text message prompts for increasing trial participant response to postal questionnaires (PROMPTS). Trials 2021; 22: 502.

32.

Herbert

Papaioannou

Loban

, et al. Personalised versus standard text message prompts for increasing trial participant response to telephone follow-up: an embedded randomised controlled retention trial. Trials 2024; 25: 108.

33.

Treweek

Bevan

Bower

, et al. Trial Forge Guidance 2: how to decide if a further Study Within A Trial (SWAT) is needed. Trials 2020; 21(1): 33.

34.

Office for National Statistics. Milestones: journeying into adulthood, 2019, https://www.ons.gov.uk/peoplepopulationandcommunity/populationandmigration/populationestimates/articles/milestonesjourneyingintoadulthood/2019-02-18#:∼:text=Between%201998%20and%202018%2C%20the,17.8%20years%20to%2019.3%20years (accessed 12 November 2024).

35.

Office for National Statistics. Economic activity status, England and Wales: Census 2021, 2022, https://www.ons.gov.uk/employmentandlabourmarket/peopleinwork/employmentandemployeetypes/bulletins/economicactivitystatusenglandandwales/census2021 (accessed 12 November 2024).

36.

Ministry of Housing, Communities and Local Government. The English indices of deprivation 2015. Gov.UK, 2019, https://assets.publishing.service.gov.uk/media/5a7f42a940f0b6230268e6b3/English_Indices_of_Deprivation_2015_-_Statistical_Release.pdf (accessed 22 November 2024).

37.

Gov.UK. Population of England and Wales, 2022, https://www.ethnicity-facts-figures.service.gov.uk/uk-population-by-ethnicity/national-and-regional-populations/population-of-england-and-wales/latest/#:∼:text=The%202021%20Census%20data%20shows,other'%20ethnic%20group%20(6.2%25) (accessed 12 November 2024).

38.

Dawson

Banister

Biggs

, et al. Trial Forge Guidance 3: randomised trials and how to recruit and retain individuals from ethnic minority groups – practical guidance to support better practice. Trials 2022; 23: 672.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB

0.01 MB

0.02 MB

0.01 MB

0.02 MB

0.04 MB