Measuring factors associated with telehealth use by people who use mental health services: A psychometric analysis of a theoretical domains framework questionnaire

Abstract

Introduction

Telehealth has the potential to improve access to mental health care, especially for people living in rural and remote regions. Yet, telehealth accessibility remains a challenge in Australia, and there is a scarcity of appropriate, psychometrically sound tools for evaluating telehealth use by mental health service users. The aim of this study was to adapt and validate a scale for measuring factors associated with mental healthcare telehealth use.

Methods

A 39-item scale was adapted from the Theoretical Domains Framework questionnaire (TDFQ); a 14-domain framework for measuring implementation of evidence-based practice in health research and service delivery. Since use of the TDFQ in the service user and telehealth space is novel, we adapted and piloted the TDFQ using a rigorous and iterative consultation process and analysis. The study sample included 208 people who use mental health services (52% male). Preliminary analysis identified 32-items for inclusion in the exploratory factor analysis. Internal reliability and construct validity were also analysed.

Results

The resulting 21-item telehealth adaption of the TDFQ includes four factors, each with good internal reliability: satisfaction and habit (6 items), knowledge and training (5 items), benefit and value (6 items), and emotionality (4 items). Both the total scale and individual factors were positively associated with telehealth use.

Discussion

The telehealth adaption of the TDFQ is a psychometrically sound tool for assessing factors associated with the use of telehealth by people who use mental health services.

Keywords

Telehealth‌Service user patient experience exploratory factor analysis theoretical domains framework reliability and validity Australia

Introduction

Australia is facing a shortage of mental health providers to meet the treatment needs of the one in five Australians who experience a mental illness in any one year.¹ This situation worsened during the COVID-19 pandemic, with a 47% increase in workforce shortages in the healthcare sector, including mental health services.² Further to this, 30% of Australians live in rural and remote regions.³ While the prevalence of mental illness in these areas is proportionate to metropolitan cities, access to mental healthcare, including psychology and psychiatry, is significantly more limited.⁴

Telehealth, defined as the delivery of healthcare using information and communication technology (ICT) to facilitate audio, visual and/or data exchange between patients and healthcare providers,⁵ stands to be a powerful tool for expanding access to care. For instance, telehealth has been shown to reduce financial burden, employment disruption and travel expenses, particularly for rural and remote residents and those with poor access to transport,⁶ and allows for scheduling flexibility for the provider and service user. In mental health settings, telehealth can be the preferred modality for service provision in certain psychiatric conditions (e.g., OCD) and patient cohorts (e.g., forensic settings). It is also increasingly being used for early intervention in co-responder models in partnership with police and ambulance services.⁷ Telehealth also decreases the risk of airborne infectious disease transmission, such as COVID-19,⁶ an important consideration for both mental health patients and clinicians, particularly those with chronic health conditions.

The effectiveness of telehealth has been extensively researched. Eze et al.'s (2020) umbrella review found that telehealth-delivered mental healthcare was as effective as in-person delivery in 90% of reviews focusing on mental health.⁸ Snoswell et al.'s (2021) systematic review of 38 high quality meta-analyses also revealed that telehealth delivered care was as clinically effective, or more clinically effective, than traditional care across telepsychology and telepsychiatry care.⁹ In our own research, 89% of service users reported that tele-mental health allowed them to feel safe, maintain control, communicate their needs, and have their privacy protected.¹⁰

Recognising the potential of telehealth to meet the needs of people who access mental health services and anticipating a rise in mental health challenges post-pandemic, Australia's ‘Group of Eight’ universities,¹¹ along with peak bodies in psychiatry¹² and psychology,¹³ have called for the permanent integration of telehealth into routine mental health practice. Despite this advocacy, and substantial investment by the Australian Government to improve access to psychologists via the Better Access Initiative,¹⁴ telehealth accessibility remains a prominent issue in Australia.¹⁵ To address key identified barriers to telehealth use, understanding the perspectives and experiences of people who use mental health services is critical.^16,17

Notably, there is a dearth of end-users’ voices in prior tele-mental health research, with studies focusing primarily on service provider perspectives.¹⁸ In particular, there are no validated and reliable tools for measuring factors associated with the use of telehealth by people who use mental health services in the Australian context. Moreover, the use of international measures is constrained by methodological shortcomings, including a focus on outdated or specific hardware and systems, small, nonrepresentative samples,^19–23 lack of piloting and psychometric evaluation,^20,21,23 and adoption of items from outdated questionnaires.²³ These limitations render current international measures unsuitable for adaption in the Australian mental health context, highlighting the requirement for a tool to address this need.

The present study

To address the scarcity of psychometrically sound tools for measuring factors associated with telehealth use by mental health service users, we assessed the adaptability of the Theoretical Domains Framework Questionnaire (TDFQ) for this context. The TDFQ is a highly cited, flexible tool designed for application in healthcare settings.²⁴ The tool was developed to measure the 14 domains of the Theoretical Domains Framework²⁵: (1) knowledge; (2) skills; (3) social/professional role and identity; (4) beliefs about capabilities; (5) optimism; (6) beliefs about consequences; (7) reinforcement; (8) intentions; (9) goals; (10) memory, attention and decision processes; (11) environmental context and resources; (12) social influences; (13) emotions; and (14) behavioural regulation.²⁶

The TDFQ is intended for use with the Behaviour Change Wheel (BCW); the most comprehensive and practically useful methodology available for developing behaviour change interventions.²⁷ After identifying the domain of intervention using the TDFQ, the BCW suggests a target intervention based on a synthesised list of 93 behaviour change tactics. The TDFQ was therefore considered well suited for informing practical guidance to address identified telehealth implementation challenges in the current study.²⁷

Methods

The present study formed part of a larger mixed methods research project, which sought to improve the uptake of telehealth in mental health services within the Hunter New England (HNE) region of New South Wales, Australia. Ethical approval for this research was obtained from the HNE Health Research Ethics Committee (ETH01176).

Questionnaire adaption

The TDFQ has been applied in a range of research and healthcare contexts, including for intervention design and identifying factors influencing service adoption by providers.²⁸ However, to our knowledge, this study represents the first attempt to adapt the TDFQ to assess telehealth service uptake by people who access mental health services. To do this, we refined existing items and added telehealth specific items. We then sought expert review of the adapted TDFQ from healthcare providers (psychologists, social workers, psychiatrists, and mental health service managers) to determine suitability for the mental health service context. After consultation, 34 items from the original TDFQ, and five purposely designed questions were included in the initial questionnaire. The five purpose designed questions can be found in Table 2 and cover privacy and confidentiality (item 35), digital accessibility (item 36), physical limitations (item 37), control and convenience (item 38), and quality of communication between patient and clinician (item 39). The 5-point response scale of the TDFQ was maintained (1 = strongly disagree to 5 = strongly agree), however, six items (7, 24, 31, 32, 35, and 37) were reverse scored (1 = strongly agree to 5 = strongly disagree) to ensure that higher values on all items reflected positive endorsement of the item. The resulting 39-item questionnaire was piloted among mental health peer workers to obtain feedback on item readability, clarity, and face validity. As a result, minor amendments were made to the wording of some questions. The final 39-item questionnaire, corresponding original TDFQ domains, and item inclusion or exclusion from the final scale can be found in Table 2.

Sample and recruitment

Service users were eligible to participate in the study if they were i) aged 18 years or older, ii) an Australian resident, and iii) a person who uses a HNE mental health service, which provides specialist drug and alcohol, psychiatry, psychotherapy, and acute mental health care across metropolitan, regional, and rural regions of the HNE district. An anonymous online self-report questionnaire was disseminated (i) via unpaid social media (i.e., Facebook, Twitter, Reddit) campaign, (ii) on iPads in a community mental health service with the support of a peer worker, and (iii) through text messages sent by HNE mental health and drug and alcohol services. Upon completion of the survey, participants were given the choice to be entered into a draw for an iPad Tablet as reimbursement for their time.

Data collection

All data were collected using REDCap data collection software²⁹ between June and December 2022. Participants were provided with the following definition of telehealth prior to completing the 39-item adapted questionnaire: ‘Telehealth is the secure transmission of a clinical consultation by videoconference over the internet. Telehealth allows patients to have flexible delivery consultations with health professionals without the need and inconvenience of travel. Telehealth allows patients to talk to, hear and see health professionals using high-speed internet, cameras, speakers, and monitors at both locations’, which reflects how telehealth operates within the HNE mental health services. They were then asked to provide demographic, mental health service use, and telehealth use information (see Table 1). This included open ended questions asking about barriers and facilitators to telehealth use, which are reported elsewhere.¹⁰ The survey took approximately 20 min to complete.

Table 1.

Patient sample characteristics

Demographic Questions	Categories	N (%)
Gender	Male Female Non-binary	108 (51.9) 96 (46.2) 4 (1.9)
Age	18–25 years 26–35 years 36–45 years 46–55 years 56–65 years 65 years or older	8 (3.8) 35 (16.8) 78 (37.5) 58 (27.9) 23 (11.1) 6 (2.9)
Indigenous status	Aboriginal Not Aboriginal Prefer not to say	45 (21.6) 150 (72.1) 13 (6.3)
Birth country	Australia Other	197 (94.7) 11 (6.3)
Education	Postgraduate degree Graduate diploma/certificate Bachelor's degree Diploma/advanced diploma Certificate Secondary education Primary/pre-primary education	4 (1.9) 5 (2.4) 15 (7.2) 21 (10.1) 62 (29.8) 88 (42.3) 13 (6.2)
Health Service	Mental health Alcohol and other drug (AOD) Both mental health and AOD	34 (16.3) 126 (60.6) 48 (23.1)
Use of telehealth	Never Rarely (<10% of appointments) Occasionally (10–30% of appointments) Often (up to 50% of appointments) Regularly (>50% of appointments)	25 (12.0) 35 (16.8) 67 (32.2) 34 (16.3) 47 (22.6)

Note. Use of telehealth was specifically for service users’ mental health and/or AOD appointments.

Table 2.

Adaption of the TDFQ for telehealth use among people who use mental health services

TDFQ Domain	Adapted Item	Item result	Reason/factor
Knowledge	1. I understand why telehealth is used	Excluded before EFA	Low variability in responding
	2. I know how to use telehealth	Included in final scale	Knowledge and training
	3. I know when I will be offered telehealth	Included in final scale	Knowledge and training
Skills	4. I have been shown how to use telehealth for my appointments	Included in final scale	Knowledge and training
Skills	5. I have been given the opportunity to use telehealth for my appointments	Included in final scale	Knowledge and training
Social/professional role and identity	6. Using telehealth is my preferred way to access appointments	Included in final scale	Satisfaction and Habit
	7. It is up to me to ask for the telehealth option for my appointments (R)	Excluded before EFA	Low correlation with outcome
	8. When I use telehealth, I find it works seamlessly with other aspects of my healthcare	Included in final scale	Satisfaction and Habit
Beliefs about capabilities	9. I feel confident to use telehealth for my healthcare appointments	Excluded after EFA	Loading on >1 factor
	10. I am capable of using telehealth for my healthcare appointments	Excluded before EFA	High correlation with item 11
	11. I feel confident to ask for telehealth even when it has not been offered to me	Excluded before EFA	Low correlation with outcome
	12. I have the option to choose telehealth for my healthcare appointments	Excluded before EFA	High correlation with item 6
	13. For me, using telehealth for my healthcare appointments is easy	Included in final scale	Benefit and Value
Optimism	14. Even when I feel uncertain about my ability to use telehealth, I usually expect that things will work out okay	Excluded after EFA	Loading on >1 factor
	15. When I use telehealth, I feel positive about my healthcare	Included in final scale	Emotionality
	16. I do not expect anything will prevent me from using telehealth for my healthcare appointments	Included in final scale	Satisfaction and Habit
Beliefs about consequences	17. I believe that telehealth will lead to benefits for clients of the service	Included in final scale	Benefit and Value
	18. In my view, using telehealth for my appointments is practical	Excluded before EFA	High correlation with item 20
	19. In my view, using telehealth for appointments is worthwhile	Included in final scale	Benefit and Value
Reinforcement	20. My healthcare provider encourages me to use telehealth	Excluded after EFA	No factor loading > .40
Reinforcement	21. My friends, family, and/or community think that using telehealth is a good idea	Excluded after reliability testing	Fifth factor unreliable
Memory, attention, and decision making	22. Booking telehealth appointments is part of my regular healthcare routine	Included in final scale	Satisfaction and Habit
	23. I can maintain focus during telehealth appointments just as well as when I attend the service in person	Included in final scale	Satisfaction and Habit
	24. I am more likely to need a reminder (sms/email/phone call) for a telehealth appointment than for face-to-face appointments (R)	Excluded after EFA	No factor loading > .40
Environmental context and resources	25. I have enough support to attend my appointments using telehealth	Excluded after EFA	Loading on >1 factor
	26. I can talk to my healthcare provider about any problems I have using telehealth	Excluded after EFA	Loading on >1 factor
	27. Using telehealth for my appointments saves me time	Included in final scale	Benefit and Value
	28. Using telehealth for my appointments saves me money	Included in final scale	Benefit and Value
Social influences	29. People who are important to me think that I should use telehealth for my appointments with my HNE healthcare provider	Excluded after reliability testing	Fifth factor unreliable
Social influences	30. I can count on support from family, friends, and/or community to help me use telehealth	Excluded after reliability testing	Fifth factor unreliable
Emotions	31. Using telehealth makes me feel nervous or anxious (R)	Included in final scale	Emotionality
	32. Using telehealth makes me feel distressed or upset (R)	Included in final scale	Emotionality
	33. I am able to use telehealth even when I feel stressed	Included in final scale	Emotionality
Behavioural regulation	34. I have discussed how telehealth will be used in my appointments	Included in final scale	Knowledge and training
Purpose designed items	35. I am concerned about the confidentiality or privacy of using telehealth (R)	Excluded after reliability testing	Increased reliability of Emotionality if dropped
	36. I own, have access to, or can afford to purchase a device capable of facilitating telehealth appointments.	Excluded after EFA	No factor loading > .40
	37. I have hearing and/or sight limitations that may limit my ability to use telehealth (R)	Excluded before EFA	Low correlation with outcome
	38. When I use telehealth services, I have more control over the time and place of my appointments, which is convenient for me	Included in final scale	Benefit and Value
	39. When I use telehealth, I feel the quality of the communication will be just as good as if it were face-to-face	Included in final scale	Satisfaction and Habit

Note. (R) refers to reverse scored items; EFA = exploratory factor analysis.

Data analysis

Data were analysed using IBM Statistical Package for the Social Sciences (SPSS) version 29.³⁰ Descriptive statistics were used to explore the demographic characteristics of the sample. The mean and standard deviation for each item total score was calculated to ensure variability in responding.

Exploratory factor analysis

Since the TDFQ has not previously been adapted for service users, we conducted an exploratory factor analysis (EFA); a statistical method for identifying order and structure in multivariate data.³¹ Specifically, EFA is used to test a pool of items or questions to identify those meaningfully measuring a construct (item reduction), and groups these items to show the possible underlying dimensions of a questionnaire.³² For this analysis, we followed Watkin's EFA best practice guide.³² To ensure that the data were appropriate for factor analysis, a Kaiser-Meyer-Olkin (KMO) test of sampling adequacy and Bartlett's test of sphericity were conducted. Mardia's estimates were checked for multivariate skew and kurtosis using WebPower online software (https://webpower.psychstat.org/). Prior to conducting the EFA, correlations among items and the outcome variable were explored for inclusion. The EFA was conducted using Principal Axis (PA) factoring extraction method with Promax rotation, which allows for more realistic representation and interpretation of the data.³² A value of between .40 and .90 was set as the acceptable factor loading coefficient. Three items per factor are considered acceptable, however four items are considered robust³² and was chosen as the minimum for this study.

Internal reliability

Cronbach's alpha coefficient (a) and corrected item-total correlations were used to measure internal reliability. A Cronbach's alpha range of .80 to .95 was used to identify reliability of the total scale and each factor, and a corrected item-total correlations of greater than .40 were used to identify internal reliability of each item. Cronbach's alpha was used to check whether removal of any items would increase factor reliability.

Construct validity

To ensure that the final scale measured telehealth use, Spearman's correlations (r_s) were used to correlate each factor and the total scale against a question asking how often respondents used telehealth for their mental healthcare appointments (see Table 1). Significant and positive correlations were expected to show that higher scores were associated with increased telehealth use. Spearman's correlations between the final scale and each factor, telehealth use, and demographic variables explored whether demographic factors influenced responding. Figures with a p-value of less than .05 were considered statistically significant.

Results

Respondent characteristics

In total, 337 service users responded to the survey invitation, with 208 completing the survey and constituting the final sample. Participants in the final sample were majority male (n = 108, 51.9%), between 36 and 55 years of age (n = 136, 65.4%), and people who use alcohol or other drug (AOD) services (n = 126, 60.6%) (Table 1). Also included in the sample were 45 Indigenous Australians, representing 21% of the sample.

Exploratory factor analysis

KMO values for the full scale and each item (.938) and Bartlett's tests of sphericity (Chi-square = 4735.55, p < .001) show that the sample and data were appropriate for EFA. Bivariate correlations among items and the outcome variable resulted in the removal of five items prior to EFA. Three items (7, 11, 37) were removed for not correlating significantly with the outcome variable. Items 10 and 11, items 6 and 13, and items 19 and 20 were highly correlated (r = .806, .813, and .820, respectively) and assessing similar concepts, showing redundancy. Items 10, 6, and 20 were retained for higher correlation with the outcome variable. A further one item (item 1) was removed for low variability in responding (mean = 4.36, SD = .72). Therefore, 32 of the 39 items were included in the EFA. A significant value for Mardia's estimates (p < .001) shows that multivariate skew and kurtosis were present in the data. Therefore, PA factoring with Promax rotation was used as this method makes no distributional assumptions.³²

Prior to EFA, internal reliability testing indicated high redundancy among the 32 items (a = .952). The eigenvalues and scree plot resulting from PA factoring indicated extraction of a five-factor solution. At this stage, 7 items were omitted due to no factor loading above .40 (items 20, 24 and 36), or loading onto more than one factor (items 9, 14, 25 and 26). See Table 2 for item inclusion and exclusion.

Internal reliability

The internal reliabilities of factor one, ‘satisfaction and habit’ (six items; a = .893), factor two, ‘knowledge and training’ (five items; a = .825), factor three, ‘benefit and value’ (six items; a = .897), and factor four ‘emotionality’ (four items; a = .808) were good. Factor five, which appeared to be measuring support and encouragement, included only three items (21, 29, and 30) and had an internal reliability below the acceptable threshold of .800 (a = .715). Additionally, two items on factor five (21 and 30) showed an increase in internal reliability if removed. Therefore, factor 5 was removed from the questionnaire. The remaining 21-item scale showed excellent internal reliability (a = .944), and removing any other items led to a decrease in Cronbach's alpha, showing that they were essential for the questionnaire's reliability. See Table 3 for internal reliabilities, factor loadings, and item statistics for the final 21-item questionnaire. All resulting subscale scores indicate that higher values reflect agreement with that subscale.

Table 3.

Final questionnaire items, factors, and factor descriptives

	Cronbach's alpha	Mean	Standard deviation	Factor loading	Item-total correlation
Full scale: TDFQ-21 item telehealth adaption	.944	77.09	15.03
Factor One: satisfaction and habit	.893	21.25	5.19
1. I can maintain focus during telehealth appointments just as well as when I attend the service in person		3.74	1.07	.714	.767
2. When I use telehealth, I find it works seamlessly with other aspects of my healthcare		3.57	.99	.686	.773
3. When I use telehealth, I feel the quality of the communication will be just as good as if it were face-to-face		3.57	1.10	.660	.735
4. I do not expect anything will prevent me from using telehealth for my appointments		3.76	.98	.573	.696
5. Booking telehealth appointments is part of my regular healthcare routine		3.22	1.10	.532	.646
6. Using telehealth is my preferred way to access appointments		3.38	1.18	.515	.685
Factor Two: knowledge and training	.825	17.82	4.40
7. I know how to use telehealth		3.95	.10	.809	.635
8. I have been shown how to use telehealth		3.28	1.18	.728	.707
9. I know when I will be offered telehealth		3.56	1.12	.717	.690
10. I have been given the opportunity to use telehealth		3.78	1.08	.713	.705
11. I have discussed how telehealth will be used in my appointments		3.26	1.07	.568	.561
Factor Three: benefit and value	.897	23.55	4.74
12. When I use telehealth services, I have more control over the time and place of my appointments		3.83	1.06	.660	.698
13. For me, using telehealth for my appointments is easy		3.82	1.06	.403	.697
14. I believe that telehealth will lead to benefits for clients of the service		4.01	.89	.522	.695
15. In my view, using telehealth for my appointments is worthwhile		4.02	.88	.578	.773
16. Using telehealth for my appointments saves me time 17. Using telehealth for my appointments saves me money		4.013.86	.96.97	.687.578	.815.671
Factor Four: emotionality	.808	14.51	3.26
18. Using telehealth makes me feel distressed or upset (R) 19. Using telehealth makes me feel nervous or anxious (R) 20. I am able to use telehealth, even when I feel stressed		3.723.423.64	1.961.111.00	.690.664.424	.769.724.475
21. When I use telehealth, I feel positive about my healthcare		3.72	. 93	.698	.511

Note. Variance explained = eigenvalues; percentage of telehealth variance explained by the full scale and each factor; (R) indicates reverse scored items.

Construct validity

Spearman's correlation coefficients were calculated for the total questionnaire, each factor, telehealth use, and demographic factors. As expected, telehealth use was significantly and positively correlated with the total scale (r_s = .490, p < .001), factor one (r_s = .482, p < .001), factor two (r_s = .348, p < .001) factor three (r_s = .405, p < .001), and factor four (r_s = .404, p < .001). Age was negatively and significantly correlated with factor three (r_s = -.143, p = .041). There were no other significant correlations between demographic factors and the final scale or each factor.

Discussion

The aim of this study was to assess the adaptability of the TDFQ for measuring factors associated with telehealth use among people who use mental health services in Australia, and to provide a preliminary validation and reliability of the resulting scale. The final scale consisted of 21 items (Cronbach's a = .944) loading onto four factors including (i) satisfaction and habit, (ii) knowledge and training, (iii) benefit and value, and (iv) emotionality; each with good internal reliability (Cronbach's a = .808 - .897). The EFA supported the extraction of a five-factor scale, however psychometric evaluation identified problematic internal reliability of factor five which was thus removed. Further, the total scale and each of the four factors are significantly associated with self-reported telehealth use, showing that higher factor values were associated with increased telehealth use.

The resulting scale and its factors show a unique composition in relation to the original domains of the TDFQ.^33,34 Of the 12 TDFQ domains included in the EFA, items from 10 domains were included in the final scale. The TDFQ domains ‘reinforcement’ and ‘social influences’ were not represented in the final scale, however items from these domains were represented in the fifth unreliable factor, which appeared to be capturing support and encouragement. Five items were purpose designed in consultation with expert stakeholders, covering privacy and confidentiality (item 35), digital accessibility (item 36), physical limitations (item 37), control and convenience (item 38), and quality of communication between patient and clinician (item 39). Of these items, two were included in the final scale (items 38–39), while three (items 35–37) did not meaningfully contribute to the factor structure. While this is a somewhat surprising outcome, it is possible that individuals who are concerned about privacy and security may not have responded to the questionnaire for these reasons. It is also possible that people had less concerns around privacy as some of the telehealth services were embedded in public hospital settings which also provides in-person care. While our sample only included a small proportion of older adults (3% were aged 65 year or older) which may be a limitation due to digital accessibility barriers among this population,³⁵ other research shows that older people are interested in engaging with telehealth for mental health,³⁶ showing that further research is required to examine the sentiment of this group more fully.

Prior research reinforces the importance of patient-centred approaches to telehealth implementation.^16,18 Our adaption of the TDFQ allows people who use mental health services to be activity engaged in their tele-mental health care delivery by directly assessing factors that are associated with their telehealth use. An important consideration for factor three ‘benefit and value’ is the significant negative correlation with age, indicating that younger people may value the benefit and value associated with telehealth use more than those of older ages. Importantly, this telehealth adaption of the TDFQ was developed in the post-COVID-19 pandemic landscape, making it a timely and appropriate tool for inclusion in tele-mental health research and clinical implementation.

It is important to acknowledge that this study did not include measures of social inclusion or socioeconomic status. This represents a significant limitation, since people who experience social exclusion across multiple domains report more mental health problems and additional barriers to accessing telehealth services.³⁷ In fact, responses to the open-ended questions in our survey suggested that some participants, such individuals from CALD backgrounds and First Nations people, encountered barriers to the use of telehealth that led to a sense of disempowerment.¹⁰ While our sample included 21% First Nations representation, a proportion greater than the percentage of First Nations people living in the HNE region where the study was conducted (7.1%), the adaption was not designed for use among this population, and is thus unlikely to align with First Nations ways of knowing.³⁸ Additionally, while the HNE region of NSW is represented by a regional and rural population, the proportion of the sample living rurally is unknown. It is also important to note that this study used a specific definition of telehealth that did not include digital apps, websites, or other programs that offer mental health treatments and support asynchronously. Given the proliferation of digital mental health programs in recent years, exploring the utility of the TDFQ in this context is an important future direction.

Conclusion

To our knowledge, this is the first tool adapted from the TDFQ to measure factors associated with telehealth use among people who use mental health services in Australia. The resulting brief, easy to administer, and reliable self-report scale has the potential to address identified gaps in telehealth research including non-rigorous methodologies and a lack of patient-centredness. The scale has a coherent factor structure specific to the Australian tele-mental healthcare context, and with consideration of the scale's limitations, its practical use in research and clinical practice has the potential to address mental healthcare inaccessibility for many Australians. Future research should consider combining the TDFQ-telehealth adaption with theories of social justice and including non-digital and/or culturally aligned research methods to establish the scale's reliability and validity among older adults and populations who experience social exclusion, including those from lower SES, those who are CALD, and First Nations peoples. To further establish the scales psychometric validity, future research should also explore convergent and divergent validity, and test-retest reliability.

Footnotes

Acknowledgements

The authors would like to thank the research participants and the peer workers who supported them for their time and valuable contribution to this research. We would also like to thank Kate Simpson, Pam Davis, Fionna Murphy, Brendan Flynn, Luke Wolfenden, and Laura Wall for their contributions as collaborators on this project. Artificial intelligence (AI) was not used in the writing of this manuscript.

Author contributions

JW was involved in the project design and data collection, conducted the analysis and interpretation of the data, and drafted all sections of the manuscript. MH (project lead), and PB provided feedback throughout the interpretation and writing of the manuscript CT was involved in project design, data collection, and adaption of the questionnaire. RG supported data collection within mental health services. RC, CB, RW, and JR were involved in the expert review of the adapted questionnaire. FKL was involved in project design. All authors provided feedback and reviewed and approved the final manuscript.

Declaration of conflicting interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was funded by the Hunter Medical Research Institute (HMRI) Advanced Out of Hospital Care (AOHC) Telehealth Research Initiative (Grant ID number: G2011010).

Hunter Medical Research Institute, (grant number G2011010).

ORCID iDs

Jessica Wilson

Rahul Gupta

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

Australian Bureau of Statistics. Two in five Australians have experienced a mental disorder. 2023. Accessed 10 January 2024. https://www.abs.gov.au/media-centre/media-releases/two-five-australians-have-experienced-mental-disorder#:∼:text=More%20than%20two%20in%20five,Bureau%20of%20Statistics%20(ABS).

Australian Government: Jobs and Skills Australia. 2023 Skills Priority List. 2023. Accessed 10 January 2024. https://www.jobsandskills.gov.au/sites/default/files/2023-10/2023%20SPL%20Key%20Findings%20Report.pdf.

Australian Bureau of Statistics. Population. Accessed August 7, 2023. https://www.abs.gov.au/statistics/people/population.

National Rural Health Alliance Inc. Mental Health in Rural and Remote Australia. 2017. Accessed September 26, 2023. https://ruralhealth.org.au/sites/default/files/publications/nrha-mental-health-factsheet-dec-2017.pdf.

Royal Australian College of General Practitioners. On demand telehealth services. Accessed August 8, 2023.

Fisher

Davey

Magin

. Telehealth for Australian general practice: The present and the future. Australian Journal of General Practice 2022; 51: 626–629.

Nataliansyah

Merchant

KAS

Vakkalanka

, et al. Virtual partnership addressing mental health crises: Mixed methods study of a coresponder program in rural law enforcement. JMIR Ment Health 2023; 10: e42610–e42610.

Eze

Mateus

Cravo Oliveira Hashiguchi

. Telemedicine in the OECD: An umbrella review of clinical and cost-effectiveness, patient experience and implementation. PLoS One 2020; 15: e0237585.

Snoswell

Chelberg

De Guzman

, et al. The clinical effectiveness of telehealth: A systematic review of meta-analyses from 2010 to 2019. J Telemed Telecare 2023; 29: 669–684.

10.

Heinsch

Wilson

. Client perceptions of safety and fulfilment of needs when accessing telemental health services: A mixed-methods study. 2024Unpublished Manuscript.

11.

Group of Eight Australia. COVID-19 road map to recovery: a report for the nation. 2021. Accessed 10 October 2023. Available at: https://go8.edu.au/wp-content/uploads/2020/05/Go8-Road-to-Recovery.pdf.

12.

The Royal Australian and New Zealand College of Psychiatrists. College continues telehealth advocacy. 2022. Accessed 10 October 2023. Available at: https://www.ranzcp.org/news-analysis/college-continues-telehealth-advocacy.

13.

Australian Psychological Society. Psychologists welcome telehealth extension. 2021. Accessed 10 October 2023. Available at: https://psychology.org.au/about-us/news-and-media/media-releases/2021/psychologists-welcome-telehealth-extension.

14.

Department of Health. The Hon Greg Hunt MP. Permanent telehealth to strengthen universal medicare. 2021. Accessed 9 October 2023. available at: www.health.gov.au/ministers/the-hon-greg-hunt-mp/media/permanent-telehealth-to-strengthen-universal-medicare#:∼:text=The%20Hon%20Greg%20.

15.

Wade

Soar

Gray

. Uptake of telehealth services funded by medicare in Australia. Aust Health Rev 2014; 38: 528–532.

16.

Bailey

Gurgol

Pan

, et al. Early patient-centered outcomes research experience with the use of telehealth to address disparities: Scoping review. J Med Internet Res 2021; 23: e28503.

17.

Thomas

Haydon

Mehrotra

, et al. Building on the momentum: Sustaining telehealth beyond COVID-19. J Telemed Telecare 2022; 28: 301–308.

18.

Standing

McDermott

M-L

, et al. The paradoxes of telehealth: A review of the literature 2000-2015. Syst Res Behav Sci 2018; 35: 90–101.

19.

Bakken

Grullon-Figueroa

Izquierdo

, et al. Development, validation, and use of English and spanish versions of the telemedicine satisfaction and usefulness questionnaire. J Am Med Inform Assoc 2006; 13: 660–667.

20.

Demiris

Speedie

Finkelstein

. A questionnaire for the assessment of patients’ impressions of the risks and benefits of home telecare. J Telemed Telecare 2000; 6: 278–284.

21.

Yip

Chang

Chan

, et al. Development of the telemedicine satisfaction questionnaire to evaluate patient satisfaction with telemedicine: A preliminary study. J Telemed Telecare 2003; 9: 46–50.

22.

Hirani

Rixon

Beynon

, et al. Quantifying beliefs regarding telehealth: Development of the whole systems demonstrator service user technology acceptability questionnaire. J Telemed Telecare 2017; 23: 460–469.

23.

Parmanto

Lewis Jr.

Graham

, et al. Development of the telehealth usability questionnaire (TUQ). Int J Telerehabil 2016; 8: 3–10.

24.

Demir

Wickens

, et al. Exploring the behaviour change wheel and the theoretical domains framework in interventions for mobile phone driver distraction: A scoping review. Accident Analysis and Prevention 2024; 195: 107369–107369.

25.

Michie

Johnston

Abraham

, et al. Making psychological theory useful for implementing evidence based practice: A consensus approach. Quality and Safety in Health Care 2005; 14: 26–33.

26.

Cane

O’Connor

Michie

. Validation of the theoretical domains framework for use in behaviour change and implementation research. Implement Sci 2012; 7: 37.

27.

Kolodko

Schmidtke

Read

, et al. LetsUnlitterUK: A demonstration and evaluation of the behavior change wheel methodology. PloS one 2021; 16: e0259747–e0259747.

28.

Atkins

Francis

Islam

, et al. A guide to using the theoretical domains framework of behaviour change to investigate implementation problems. Implement Sci 2017; 12: 77.

29.

Harris

Taylor

Thielke

, et al. Research electronic data capture (REDCap)—A metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009; 42: 377–381.

30.

IBM Statistics for Windows, Version 28. IBM Corp., 2021.

31.

Tucker

MacCallum

. Exploratory factor analysis. 1997:1-459. Unpublished Manuscript.

32.

Watkins

. Exploratory factor analysis: A guide to best practice. Journal of Black Psychology 2018; 44: 219–246.

33.

Huijg

Gebhardt

Crone

, et al. Discriminant content validity of a theoretical domains framework questionnaire for use in implementation research. Implement Sci 2014; 9: 11.

34.

Huijg

Gebhardt

Dusseldorp

, et al. Measuring determinants of implementation behavior: Psychometric properties of a questionnaire based on the theoretical domains framework. Implement Sci 2014; 9: 33.

35.

Wilson

Heinsch

Betts

, et al. Barriers and facilitators to the use of e-health by older adults: A scoping review. BMC public Health 2021; 21: 1–12.

36.

Christensen

Wilson

Hansen

, et al. A qualitative study of patients’ and providers’ experiences with the use of videoconferences by older adults with depression. Int J Ment Health Nurs 2021; 30: 427–439.

37.

Heinsch

Tickner

Kay-Lambkin

. Placing equity at the heart of eHealth implementation: A qualitative pilot study. Int J Equity Health 2022; 21: 38.

38.

Gregor

Gunawardena

Imran

, et al. Indigenous knowledge-sharing interventions in Australia and the use of information and communication technology: A scoping review. Australas J Inf Syst 2023; 27: 1–27.