Design and development of a Telemedicine Assessment Toolkit (TAT) for the assessment of audiovisual telemedicine encounters

Abstract

Although the use of audiovisual telemedicine has grown in recent years especially during recent COVID-19-related lockdowns, evidence shows there is still a lack of tools that can be used for the assessment of telemedicine encounters. The few validated questionnaires that are available for assessing telemedicine encounters are not often used. Non-validated questionnaires dominate research, leading to results that cannot be compared or extrapolated to other research or medical sites. Development of standard measures for the assessment of telemedicine encounters has been advocated by stakeholders. The objective of this study is to provide a comprehensive set of measures by developing a conceptual approach and a preliminary Telemedicine Assessment Toolkit (TAT) for the assessment of audiovisual telemedicine encounters. A two-step conceptual approach was used to identify potential domains and sub-domains by qualitative analysis of a pool of questions from studies published from 2016 to 2021. Questions were adopted from validated questionnaires or generated to represent the underlying concept of each sub-domain, resulting in a core block of comprehensive questions. A toolkit is proposed with question-measures that cover the sub-domains relevant to the assessment of telemedicine encounters. This study recommended 11 domains to be used for the assessment of telemedicine encounters: “usability,” “patient satisfaction,” “patient–provider interaction,” “patient perspectives,” “telemedicine readiness,” “qualitative feedback,” “comparison to standard (in-person) care,” “privacy,” “technology,” “patient feeling,” and “patient costs.” Of the 11 domains, 26 underlying sub-domains were created. From the subdomains, a 30-question core block was proposed. The core-block together with a precursor block aimed to retrieve demographic/patient characteristics and, together with a customizable clinical outcomes block, complete the comprehensive toolkit. The toolkit, upon testing and validation, would enable researchers and system owners to assess patient-oriented aspects of audiovisual telemedicine encounters more accurately and accelerate the adoption of common audiovisual telemedicine assessment measures.

Keywords

Telemedicine technology assessment survey questionnaire

Introduction

Telemedicine provides continuity of care by enabling the exchange of medical information from one site to another via electronic communications with the main goal of improving the health of patients. Telemedicine as a routine mode of delivering healthcare has continued to further accelerate with improvements in communication infrastructure and coordinated global response to crises such as disease outbreaks. Telemedicine services are conducted using different systems like telephonic, store and forward (asynchronous), digital remote monitoring, and more recently wearable devices. Synchronous audiovisual encounters or video consultations have grown in use due to the enhanced capacity of real-time visual cues important for patient engagement, clinical observation, and visual assessment.¹

Despite the growing adoption of audiovisual telemedicine, insufficiency in the scientific evidence derived from telemedicine assessment persists, particularly in effectiveness, impact on patient management, and costs.² Systematic reviews continue to highlight the need to standardize populations, interventions, and outcome measures to reduce study heterogeneity, facilitate meta-analysis and develop controlled studies with larger samples.^2,3

Non-validated measures are used in assessing telemedicine encounters,^4,5 leading to inability to combine and compare study results. Validated questionnaires specific for telemedicine such as the Telemedicine Satisfaction Questionnaire (TSQ), the Telehealth Usability Questionnaire (TUQ), Telemedicine Satisfaction and Usability Questionnaire (TSUQ), the Patient Assessment of Communication in Telemedicine (PACT), and the Telemedicine Satisfaction Scale Questionnaire (TESS) are not often used despite being available in the literature.^6–10 Limitations contributing to the low use of validated questionnaires identified in the literature include neglect of key assessment criteria such as patient views and access to care, while focusing on satisfaction and clinical effect.^11,12 The lack of built-in mechanisms for continuous improvement despite the rapid change in technology over time is also an issue with current validated assessment tools.⁷ Because of the lack of standard measures, the consistency of assessment strategies and the development of standardized measures are research priorities across health care.¹³

Standard measures covering the essential elements of a discipline are needed because they enhance the ability of researchers to combine data from several studies, improve the strength of evidence and broaden its applicability to several groups of people and situations.¹⁴ Several initiatives have been launched to develop tools to collect uniform datasets.^14–16 The Global Tobacco Control Surveillance program by the Centers for Disease Control and Prevention-CDC,¹⁷ and the Patient Reported Outcomes Management Information System-PROMIS¹⁸ used approaches that integrate the collective wisdom of researchers within their field to design comprehensive assessment tools. Similarly, in developing the quality of life questionnaires,^19–21 the methodologies utilized consensus domains to address the problem of heterogeneity.

Prior literature describe how research can conceptualize attributes into domains.^22,23 When combined, theories and concepts help researchers to understand and explain what needs to be comprehensively measured.²⁴ To achieve general acceptance and adoption, uniform measures for the assessment of telemedicine encounters sourced from existing literature should be developed around a conceptual approach, similar to the work done in other health fields.

A previous study retrieved questions from validated and non-validated questionnaires used by researchers in the literature from January 2021 through November 2021,²⁵ creating a question pool that can be analyzed to identify conceptual domains.

The goals of this study were (1) to identify the most common domains for the assessment of audiovisual telemedicine encounters, (2) to propose a conceptual approach with key domains and sub-domains to be used for the assessment, (3) and to develop a preliminary toolkit with comprehensive measures to be used for the assessment of audiovisual telemedicine encounters.

Methods

We used a two-step conceptual approach that led to the development of a preliminary comprehensive toolkit for the assessment of audiovisual telemedicine encounters. We integrated qualitative and quantitative techniques in a mixed method design, as identified in prior literature.²⁶

Domain and sub-domain identification and questions categorization

Previous research identified 702 unique question stems used to assess telemedicine encounters.²⁵ Two authors (RA and GD) reviewed 702 questions to identify domains using open coding. Open coding for unique domains continued until saturation occurred with no new domain being identified. Existing literature was reviewed to ensure no domains were overlooked. Then, two authors (RA and GD) proceeded to associate each question to a domain working independently. To categorize the question stems that the two authors assigned to different domains, author 1 (RA) reviewed the non-matched items coded by author 2 (GD), while GD reviewed the non-matched items coded by RA. Disagreements were resolved by consensus based on discussion of the keywords associated with each criterion to identify a set of consensus codes and domains (Table 1).

Table 1.

Frequency distribution of analyzed question items by domain/sub-domain.

Domain n(%)	Sub-domain	N	% All analyzed stems	%sub-domain
Telemedicine readiness 98(16.0)	Future use intent	38	6.2	38.8
	Previous experience	15	2.4	15.3
	Likelihood to recommend others	23	3.8	23.5
	Preference	22	3.6	22.4
Patient perspective 87(14.2)	Benefits	36	5.9	41.4
	Provider's skills and knowledge	19	3.1	21.8
	Convenience	19	3.1	21.8
	Clinical examination	13	2.1	14.9
Patient–provider interaction 83(13.5)	Patient–provider relationship	30	4.9	36.1
	Shared decision making	29	4.7	34.9
	Clarity of communication	24	3.9	28.9
Technology 67(11.0)	Equipment readiness	24	3.9	35.8
	Video quality	23	3.8	34.3
	Audio quality	20	3.3	29.9
Patient satisfaction 64(10.4)	Overall satisfaction	31	5.1	48.4
	Satisfaction with system	18	2.9	28.1
	Satisfaction with process	15	2.4	23.4
Patient costs 52(8.5)	Time	31	5.1	59.6
Patient costs 52(8.5)	Money	21	3.4	40.4
Usability 48(7.8)	Ease of use	20	3.3	41.7
	Ease of set-up	15	2.4	31.3
	Device usability	13	2.1	27.1
Comparison with in-person care 43(7.0)	Comparison with standard care	43	7.0	100.0
Patient feeling 27(4.4)	Comfort with encounter	27	4.4	100.0
Privacy 23(3.8)	Respect for privacy	23	3.8	100.0
Qualitative feedback 21(3.4)	Improvement suggestions	21	3.4	100.0

For each domain, a similar approach was used to identify sub-domains. The question stems from each sub-domain were reviewed for their sentence structure and archetype questions that best represent the sub-domain (Table 2).

Table 2.

Question measures for Telemedicine Assessment Toolkit Core Questionnaire generated from underlying sub-domains for Telemedicine encounters, uniformly adopted response scale and item sources

Representative question measure	Sub-domain	Source	Response scale
I find online consultation a positive means to achieve my health needs.	Benefits		Strongly Disagree-1, Disagree -2, Neutral-3, Agree-4, Strongly agree-5.
I have a good understanding of the next steps in my care.	Benefits
I could see that the clinicians showed enough knowledge during the online consultation.	Provider's skills and knowledge
I received care that suited me.	Convenience
I believe not receiving a physical examination affected my care.	Clinical examination
I will use online consultation again.	Future use intent	TSQ^a
I have used online consultation in the past.	Previous experience
I would recommend online consultation to others.	Likelihood to recommend others
I obtained better access to healthcare services by using online consultation.	Preference for Telemedicine	TSQ^a
I had necessary equipment for the online consultation.	Equipment readiness
I did not have any technical problems with the online consultation.	Equipment readiness
I could see all people involved in the online consultations clearly for the majority of the videoconference time.	Video quality	ITU^b
I did not have trouble seeing the clinicians during the online consultation.	Video quality	ITU^b
I could hear all people involved in the online consultations clearly for the majority of the videoconference time.	Audio quality	ITU^b
I experienced a personal connection with the clinician during the online consultation.	Patient–provider relationship
I was asked by the clinician for my concerns pertaining to my health problems.	Shared decision making	PACT^c
I was able to communicate clearly with the clinician.	Clarity of communication
I was able to understand the plan explained by the clinician.	Clarity of communication
Overall, I was satisfied with the quality of service provided during the online consultation.	Overall satisfaction	TSQ^a
I was satisfied with the structural features of the online consultation.	Satisfaction with system	TUQ^d
I was satisfied with how the online consultation was operated by the clinic.	Satisfaction with process
I did not need assistance while using the online consultation system.	Ease of use	TSQ^a
I found it easy to set up the online consultation.	Ease of set-up
I was able to operate the devices needed for the online consultation.	Device usability
I saved time by using the online consultation.	Time
I saved money by using the online consultation.	Money
I think the visit provided over the online consultation is the same as in-person visit.	Comparison with in-person care	TUQ^d
I was able to share sensitive information without concern about unintentional disclosure..	Respect for privacy
I felt comfortable during the online consultation with the clinician.	Comfort with encounter
I would like to suggest one aspect for the improvement of the online consultations……… (Free format)	Improvement suggestions		Qualitative

International Telemedicine Union (ITU) real time video communication scale.

Telemedicine Satisfaction Questionnaire (TSQ).

Patient Assessment of Communication in Telemedicine (PACT) Questionnaire.

Telemedicine Usability Questionnaire (TUQ).

Tool kit item generation

For each of the selected sub-domains, we listed the unique question stems coming from the previously published article.²⁵

If an existing validated assessment tool had been used to assess the sub-domain, one or more specific questions from that validated questionnaire were selected for inclusion in the tool kit. For the novel sub-domains, one or more questions were generated from the list of question-stem of the sub-domain, and if needed, the item was rewritten slightly to match the uniform answers scale we adopted. The uniform answer scale used was a 5-point Likert agreement scale from strongly disagree to strongly agree.

The final comprehensive question list for the toolkit core block is a matrix of all questions (item-measures) in domains that were considered important for widespread acceptance and implementation in line with prior recommendations for robust telemedicine assessment^27,28 (Table 2).

Results

At the end of the domain identification step, 19 core domains emerged. The domains were: technology (n = 67, 9.5%), comparison to in-person care (n = 65, 9.3%), patient satisfaction (n = 64, 9.1%), future/previous use (n = 57, 8.1%), communication (n = 53, 7.6%), patient costs (n = 52, 7.4%), provider evaluation (n = 49, 7.0%), patient view of impact (n = 49, 7.0%), usability (n = 48, 6.8%), access to care (n = 42, 6.0%), qualitative feedback (n = 27, 3.9%), patient feeling (n = 27, 3.9%), privacy (n = 23, 3.3%), likelihood to recommend telemedicine (n = 23, 3.3%), non-relevant questions (n = 16, 2.3%), patient demographic and characteristics (n = 15, 2.1%), overall rating (n = 11, 1.6%), reason to use telemedicine (n = 8, 1.1%), and clinical outcome (n = 6, 0.9%).

The domain “access to care” represents the patient's opinions on how well the telemedicine encounter was able to deliver quality care to the patient at the time and place the patient needed it most, while the domain “patient view of impact” represents how well the telemedicine encounter was able to fulfill the psycho-social needs of the patient as assessed by the patient. The “clinical outcome” domain assesses how the telemedicine encounter affected the individual's daily functioning and well-being.²⁹

The authors removed five domains. The non-relevant questions domain (n = 16, 2.3%) included questions of a general nature, for example, “Do you use social media” or “did you travel last month to Hubei province,” etc. The domain “reason to use telemedicine (n = 8 1.1%)” was considered of little relevance given few questions in the category, the domain “overall rating” (n = 11, 1.6%) was excluded because, after testing, the toolkit aims to provide a single composite score for summative analysis. Clinical outcomes domain (n = 6, 0.9%) was found to be already standardized for a wide range of conditions in medicine within the Patient Reported Outcome Management Information System (PROMIS).³⁰ The domain “patient demographic and characteristics” (n = 15, 2.1%) includes questions that the electronic system could auto generate, for example, patient health condition, demographic and location or type of device connection.

Among the remaining 14 domains, some domains overlapped, and the authors decided to combine them. The domain “provider evaluation” and “communication” were merged into a domain called “patient–provider interaction.” Similarly, the “likelihood to recommend” and “future/previous use” were merged into “telemedicine readiness.” Finally, “access to care” and “patient view of impact” were merged under “patient perspectives.”

After an additional analysis of the questions, sub-domains were identified for each of the domains. Five sub-domains that had less than 13 questions or 2% of the total number of questions among the domains were removed from further analysis: scheduling, technical support availability, other costs, location of encounter, telemedicine advantages and disadvantages. The final 26 sub-domains under the 11 domains representing 613 questions of the initial 702 questions are reported in Table 1.

Question measures generated for toolkit

The demographic and clinical domains identified as already having established sources from which data could easily be obtained to complete a comprehensive assessment were designated as customizable blocks for the toolkit. Along with the core block of encounter assessment questions generated in this study, the three blocks created a three-block functional structure for the proposed tool kit.

The comprehensive core block of question measures listed in the toolkit is organized under 26 sub-domains, assembled under the 11 domains, which are relevant to assessing telemedicine encounters.

For each sub-domain, a search was done among validated questionnaires to find one or more question measures that summarize all questions identified for that specific core concept. For instance, for the sub-domains, “overall satisfaction,” “future use,” “ease of use,” and “ preference for telemedicine,” we used one question each from the TSQ.⁶ For the sub-domains, “satisfaction with system” and “comparison to in-person care,” we used one question each from the TUQ.⁷ For the sub-domain “shared decision making,” we used one question from the PACT questionnaire.¹⁰ For the sub-domain “audio quality,” we used one question from the international telecommunications union (ITU) scale for real time video communication, and two questions from the same ITU scale was used for the sub-domain “video quality.”³¹ Questions adopted from each validated questionnaire and their sources are indicated in Table 2 with an illustration for the sub-domain “video quality” in Table 3b.

For the 17 sub-domains where a question from a validated questionnaire could not be found, we generated a new question as illustrated using the sub-domain “clarity of communication” in Table 3a. The comprehensive matrix of 30 questions for the core block, with a 5-point Likert agreement scale from strongly disagree to strongly agree and associated sub-domains is reported in Table 2.

Table 3a.

Illustration of a question measure for “clarity of information” sub-domain generated from existing non-validated items.

Clarity of information	Generated toolkit question
Were you able to communicate clearly with your provider?	I was able to communicate clearly with the clinician.
The clarity of the follow up appointment report?
Did your physician clearly communicate your care plan and follow up instructions?
The recommendations and plans suggested by the doctor regarding my bone health were clear.
Did you receive any written information regarding your virtual clinic follow-up appointment, either at your earlier face to face appointments or following contact by the virtual clinic coordinators?
The clinician exposed things clearly or in an easy way for me to understand?
Explanation and treatment plan were clear
My clinician explained things in a way that was easy to understand.
Were you supplied clear information about any questions pertaining to your health?
Satisfaction with clarity of information received from staff

Table 3b.

Illustration of question measure for “audio quality” sub-domain adopted from an existing validated question.

Sub-domain: Audio quality	Adopted toolkit question
I could easily talk to the health care provider during the [cIinic_video] visit.	I could hear all people involved in the online consultations clearly for the majority of the videoconference time. From: International Telecommunication Union(ITU) real time video conference scale.
I could easily hear communication using the video conferencing system
I was satisfied with the quality of the sound (audio) during the video consultation.
Satisfaction with: Ease of hearing and seeing the remote practitioner
I was satisfied with the sound quality during the teleconsultation
I was able to hear the clinician clearly.
I could HEAR the specialist clearly
Can hear provider and be heard
I could easily hear what the clinician was saying
The clinician could hear me what I spoke
During your most recent visit, were you able to clearly hear your provider?
I could hear the other people clearly for the majority of the videoconference time.
Were you satisfied with the audio quality of the telepsychiatry equipment?
Regarding the audio quality
It was easy to hear my health care provider during my online visit.
How satisfied were you with the quality of the audio?
I was satisfied with the sound quality.
The quality of the sound was….
How did you hear the specialist?
The audio connection worked well.

Discussion

We have presented a novel conceptual approach and proposed a toolkit with comprehensive measures for assessing audiovisual telemedicine encounters. The goal is to reduce heterogeneity with non-validated measures and improve the use of validated measures in assessing telemedicine encounters. This approach aims to create meaningful measures from the outset as advocated by psychometricians.^32,33

Conceptual approach

This study recommended 11 domains: patient satisfaction, patient–provider interaction, patient perspectives, patient feeling, patient costs, technology, privacy, telemedicine readiness, usability, qualitative feedback, and comparison to standard (in-person) care. These domains have received substantial contributions from researchers in prior telemedicine encounter research and we applied consensus judgments and objective quantitative rules to conceptualize and organize them under coherent domains and sub-domains. This approach is consistent with prior methodology that use representative review of the work that has been done to provide the basis for formulating a more comprehensive model of information system success,³⁴ and evaluating usability of mobile applications.³⁵ Contents of each domain were then interpreted based on what is known in the literature.

The results of our analysis, while being consistent with other studies, provided more insight towards a comprehensive assessment strategy. For instance, we found that, patient satisfaction was a focus of many researchers with 10.4% of analyzed questions aiming to assess satisfaction, similar to other studies which identified satisfaction as a focus of researchers, to the neglect of other aspects.¹¹ In addition, we also found that four other domains, patient perspective (14.2%), telemedicine readiness (16.0%), patient–provider interaction (13.5%), and technology (11.0%) were of higher relevance to researchers based on the number of questions asked by researchers to assess these domains, suggesting that researchers in the studies viewed the other domains as more relevant for comprehensively assessing telemedicine encounters. The latent need for objective domains is believable because satisfaction has known limitations as a complex measure, varying based on different factors,^28,36 such as user expectation which is affected by culture and values.

The conceptual approach adopted in this study currently includes the domain “comparison to standard (in-person) care,” since it was considered relevant by researchers in the literature, but we consider it a tentative domain, diminishing in relevance in the future. It is foreseeable that the comparisons between telemedicine and in person care will gradually become unnecessary as funding, a skilled workforce and new clinical models of care, propel patients and the healthcare systems to embed telemedicine as routine delivery modality, a scenario predicted by two recent studies.^37,38

Toolkit construction

During the process of developing measures, we generated 30 questions representing the underlying conceptual sub-domains (Table 3). The questions cover 94.9% of the potential sub-domains identified from the initial item pool. Each question was designed to measure the essential characteristic of the individual 26 sub-domains and written to be specific and precise, without ambiguous or unfamiliar terms. We made use of word choices using lay language appropriate for different patient reading levels, while avoiding double-barreled phrases and social desirability bias which occurs when respondents conceal their true opinion to make themselves look good to the observer. This approach is consistent with prior instrument development research during the phase of generating measures within the information systems field.³⁹

Most questions were linked to the 5-point Likert agreement scale while one question was designed to enable qualitative feedback. Despite the rapid change in technology over time, the lack of built-in mechanics for continuous improvement remains an issue with the current validated assessment tools.⁷ The qualitative response available in the toolkit will enable system owners and researchers to gather feedback for improvement.

The proposed Telemedicine Assessment Toolkit (TAT) comprises three functional blocks providing assessment of the telemedicine encounters: the demographic block, the core block, and the clinical outcomes block. This adopts a similar method as used in the CDC global tobacco surveys where a core questionnaire set in provided with optional customizable question sets.¹⁷ The demographic block should retrieve from the electronic medical records of the provider demographic and patient characteristics information such as age, sex, socioeconomic status, educational status, patient location, and technology literacy. The core block has 30 questions covering 11 domains as described in this study, while the clinical outcomes block asks questions from the patient reported outcome health measures from the PROMIS system. PROMIS computer adaptive measures are standardized for documenting the impact of illness and health care interventions on an individual's daily functioning and well-being.²⁹ We propose that the toolkit will allow the researchers or providers to add the specific disease states from the PROMIS to the final questionnaire to be administered.

Since short-term patient assessments and long-term impacts of telemedicine encounters are needed for a successful assessment strategy, we recommend that the toolkit should be administered at regular intervals post-visit. This is essential to establishing disease impact patterns in different contexts over the timespan of multiple telemedicine encounters.

The TAT would enable different users stakeholders within telemedicine to achieve specific goals and tasks. Providers would find it a helpful tool when improving patient–provider interaction, process, and structure of care delivery as well as clinical performance. Third party payers and managed care organizations would find the TAT useful in justifying reimbursement algorithms with comparing in-person to audiovisual telemedicine visits. Researchers will find TAT a simplified tool to adopt in whole, or at the least select measures with uniform word choice which would enhance comparability between studies and simplify pooled analysis of data. From a societal perspective, policymakers and regulators would be able to use the TAT to conduct pilot tests and rely on pooled data from research in the field in designing health, privacy, or licensing policies. Additionally, industry and professional groups would find TAT a comprehensive material for developing generally acceptable standards for telemedicine encounters.

We acknowledge that this research may not have captured every questionnaire available in the literature since the article used to retrieve the questions had a time frame from 2016 to 2021.²⁵ Also, only patient-focused questionnaires with audiovisual encounters were included. Questions focused on provider assessments, questions not written in English, and questions to be used for pediatric patients were excluded from the study which provided the question stems that were analyzed.²⁵ The conceptual approach and toolkit may not extrapolate to other telehealth systems, for example, remote monitoring, wearables, or asynchronous modes of telemedicine and studies in other languages. There may also be potential limitation due to non-inclusion of the home health setting in the referenced article.²⁵ Also, we generated two questions for four of the sub-domains: clarity of communication, equipment readiness, benefits and video quality, a limitation which reflects a need for further refinement of these four multidimensional sub-domains in future studies.

Future studies should focus on “goodness of fit” testing, refinement, calibration by collecting feedback from experts, and validation by field testing while conducting empirical investigations of the relationships between the questions. Also, a systematic review of the articles published since 2021 similar to the one used in this paper should be performed to verify that the TAT questions would still be able to cover all the questions asked.

The toolkit will allow researchers and telemedicine systems to standardize encounter assessment and increase data pooling, statistical power, and interpretation from small settings and diverse telemedicine platforms. While telemedicine could be used to promote and facilitate continuity of care, barriers still exist. For example, according to a survey of hospital and physician practice managers, one-third of telemedicine platforms are not integrated with electronic health records at all and only 17% were fully integrated.⁴⁰ The TAT presented in this paper could also serve as first step towards a fully integrated care.

Conclusion

This study uses a systematic and conceptual approach to create comprehensive measures using mixed qualitative and quantitative methods. The toolkit proposed in this study, upon testing and validation, should enable researchers and system owners to assess patient-oriented aspects of audiovisual telemedicine encounters more accurately. The toolkit will also accelerate the attainment of standardized measures and adoption of common audiovisual telemedicine assessment measures.

Footnotes

Acknowledgments

We thank the College of Allied Health Sciences, Interdisciplinary Health Sciences Department and the staff of the Greenblatt Library at Augusta University for the use of library materials and assistance in obtaining full text articles. We acknowledge the support of the staff of the Center for Writing Excellence at Augusta University for editorial review of the toolkit items.

Author contributions

All authors contributed meaningfully to conceptualization, article selection, writing, reviewing, and editing this article.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Raphael Agbali

Andrew Balas

Vahe Heboyan

Jeane Silva

Steven Coughlin

Francesco Beltrame

Gianluca De Leo

Data availability statement

Data sharing not applicable to this article as no datasets were generated or analyzed during the current study.

References

Orlando

Beard

Kumar

. Systematic review of patient and caregivers’ satisfaction with telehealth videoconferencing as a mode of service delivery in managing patients’ health. PLoS One 2019; 14: e0221848. 2019/08/31.

Ekeland

Bowes

Flottorp

. Methodologies for assessing telemedicine: A systematic review of reviews. Int J Med Inform 2012; 81: 1–11. 2011/11/23.

Agbali

Balas

Beltrame

, et al. A review of audiovisual telemedicine utilization and satisfaction assessment during the COVID-19 pandemic. Int J Technol Assess Health Care 2021; 38: e2. 2021/12/21.

Hajesmaeel-Gohari

Bahaadinbeigy

. The most used questionnaires for evaluating telemedicine services. BMC Med Inform Decis Mak 2021; 21: 36. 2021/02/04.

Weaver

Lukowski

Wichman

, et al. Human connection and technology connectivity: A systematic review of available telehealth survey instruments. J Pain Symptom Manage 2021; 61: 1042–1051.e1042. 2020/10/18.

Yip

Chang

Chan

, et al. Development of the Telemedicine Satisfaction Questionnaire to evaluate patient satisfaction with telemedicine: A preliminary study. J Telemed Telecare 2003; 9: 46–50.

Parmanto

Lewis

Jr Graham

, et al. Development of the telehealth usability questionnaire (TUQ). Int J Telerehabil 2016; 8: 3.

Bakken

Grullon-Figueroa

Izquierdo

, et al. Development, validation, and use of English and Spanish versions of the telemedicine satisfaction and usefulness questionnaire. J Am Med Inform Assoc 2006; 13: 660–667.

Morgan

Kosteniuk

Stewart

, et al. The telehealth satisfaction scale: Reliability, validity, and satisfaction with telehealth in a rural memory clinic population. Telemedicine and e-Health 2014; 20: 997–1003.

10.

Agha

Schapira

Laud

, et al. Patient satisfaction with physician–patient communication during telemedicine. Telemedicine and e-Health 2009; 15: 830–839.

11.

Jackson

McClean

. Trends in telemedicine assessment indicate neglect of key criteria for predicting success. J Health Organ Manag 2012; 26: 508–523.

12.

Izower

Liao

Kim

, et al. A Proposed Patient-Inclusive Methodology for Developing and Validating Telehealth Surveys that Include Social Determinants of Health. In: AMIA Annual Symposium Proceedings 2021, p.565. American Medical Informatics Association.

13.

Hollander

Ward

Alverson

, et al. Creating a framework to support measure development for telehealth. Washington, DC: National Quality Forum, 2017.

14.

Cohen

Thompson

Yates

, et al. Implementing common data elements across studies to advance research. Nurs Outlook 2015; 63: 181–188.

15.

National Institutes of Health. NIH Common Data Elements Program, https://heal.nih.gov/data/common-data-elements (2023, 2023).

16.

National Library of Medicine. NIH Common Data Element Resource Portal_Home, https://wayback.archive-it.org/org-350/20200602040150/https:/www.nlm.nih.gov/cde/index.html (2020, accessed March 21 2023 2023).

17.

Palipudi

Morton

Hsia

, et al. Methodology of the global adult tobacco survey—2008–2010. Glob Health Promot 2016; 23: 3–23.

18.

DeWalt

Rothrock

Yount

, et al. Evaluation of item candidates: The PROMIS qualitative item review. Med Care 2007; 45: S12.

19.

EJB

Xiao

LYS

Xin

, et al. Developing item banks to measure three important domains of health-related quality of life (HRQOL) in Singapore. Health Qual Life Outcomes 2020; 18: 1–14.

20.

Ware

Jr Sherbourne

. The MOS 36-item short-form health survey (SF-36): I. Conceptual framework and item selection. Med Care 1992; 30: 473–483.

21.

The Development of the World Health Organization Quality of Life Assessment Instrument (the WHOQOL). Berlin, Heidelberg: Springer Berlin Heidelberg, 1994, pp. 41–57.

22.

Tao

LeRouge

Smith

, et al. Defining information quality into health websites: A conceptual framework of health website information quality for educated young adults. JMIR human Factors 2017; 4: e6455.

23.

Gärdenfors

. Semantic knowledge, domains of meaning and conceptual spaces. In: Meusburger P, Werlen B and Suarsana L (eds) Knowledge and space. Vol 9. Gemeinnützige (GmbH): Open Springer, 2017, pp.203–219.

24.

Wade

Gray

Carati

. Theoretical frameworks in telemedicine research. J Telemed Telecare 2017; 23: 181–187.

25.

Agbali

Balas

Beltrame

, et al. A review of questionnaires used for the assessment of telemedicine. J Telemed Telecare 2023: 1357633–231166161. 20230409. DOI: 10.1177/1357633(231166161

26.

Caffery

Martin-Khan

Wade

. Mixed methods for telehealth research. J Telemed Telecare 2016; 23: 764–769.

27.

Fatehi

Smith

Maeder

, et al. How to formulate research questions and design studies for telehealth assessment and evaluation. J Telemed Telecare 2017; 23: 759–763.

28.

Langbecker

Caffery

Gillespie

, et al. Using survey methods in telehealth research: A practical guide. J Telemed Telecare 2017; 23: 770–779. 2017/07/22.

29.

Cella

Hays

. A patient reported outcome ontology: Conceptual issues and challenges addressed by the patient-reported outcomes measurement information system®(PROMIS®). Patient Relat Outcome Meas 2022; 13: 189–197.

30.

Knapp

Harst

Hager

, et al. Use of patient-reported outcome measures and patient-reported experience measures within evaluation studies of telemedicine applications: Systematic review. J Med Internet Res 2021; 23: e30042.

31.

Caffery

Smith

. Investigating the quality of video consultations performed using fourth generation (4G) mobile telecommunications. J Telemed Telecare 2015; 21: 348–354.

32.

Anderson

Gerbing

. Predicting the performance of measures in a confirmatory factor analysis with a pretest assessment of their substantive validities. Journal of Applied Psychology 1991; 76: 732.

33.

Nunnally

. Psychometric Theory. 2nd ed. New York: Mcgraw Hill Book Company, 1978.

34.

DeLone

McLean

. Information systems success: The quest for the dependent variable. Inf Syst Res 1992; 3: 60–95.

35.

Hoehle

Venkatesh

. Mobile application usability. MIS Q 2015; 39: 435–472.

36.

Garcia

Adelakun

. A conceptual framework and pilot study for examining telemedicine satisfaction research. J Med Syst 2019; 43: 1–14.

37.

Smith

Thomas

Snoswell

, et al. Telehealth for global emergencies: Implications for coronavirus disease 2019 (COVID-19). J Telemed Telecare 2020; 26: 309–313.

38.

Thomas

Haydon

Mehrotra

, et al. Building on the momentum: Sustaining telehealth beyond COVID-19. J Telemed Telecare 2022; 28: 301–308.

39.

MacKenzie

Podsakoff

. Construct measurement and validation procedures in MIS and behavioral research: Integrating new and existing techniques. MIS Q 2011; 35: 293–334.

40.

Ingenious Med. New Survey Shows Telehealth Growth, https://ingeniousmed.com/new-survey-shows-telehealth-growth (2020, accessed 2 May).