Efficacy of a Digital Diabetes Logbook for People With Type 1,Type 2,and Gestational Diabetes: Results From a Multicenter,Open-Label,Parallel-Group,Randomized Controlled Trial

Abstract

Background:

In a randomized controlled trial, the efficacy of a digital diabetes diary regarding a reduction of diabetes distress was evaluated.

Methods:

A randomized controlled trial with a 12-week follow-up was conducted in 41 study sites across Germany. Key eligibility criteria were a diagnosis of type 1, type 2, or gestational diabetes and regular self-monitoring of blood glucose. Participants were randomly assigned (2:1 ratio) to either use the digital diabetes logbook (mySugr PRO), or to the control group without app use. The primary outcome was the reduction in diabetes distress at the 12-week follow-up. All analyses were based on the intention-to-treat population with all randomized participants. The trial was registered at the German Register for Clinical Studies (DRKS00022923).

Results:

Between February 11, 2021, and June 24, 2022, 424 participants (50% female, 50% male) were included, with 282 being randomized to the intervention group (66.5%) and 142 to the control group (33.5%). A total of 397 participants completed the trial (drop-out rate: 6.4%). The median reduction in diabetes distress was 2.41 (interquartile range [IQR]: −2.50 to 8.11) in the intervention group and 1.25 (IQR: −5.00 to 7.50) in the control group. The model-based adjusted between-group difference was significant (−2.20, IQR: −4.02 to −0.38, P = .0182) favoring the intervention group. There were 27 adverse events, 17 (6.0%) in the intervention group, and 10 (7.0%) in the control group.

Conclusions:

The efficacy of the digital diabetes logbook was demonstrated regarding improvements in mental health in people with type 1, type 2, and gestational diabetes.

Keywords

randomized controlled trial diabetes distress digital health application type 1 diabetes type 2 diabetes gestational diabetes

Introduction

Diabetes mellitus is a long-term condition characterized by hyperglycemia due to insulin deficiency or insulin resistance.¹ People with diabetes need to perform various self-management tasks several times a day on a day-to-day basis to manage glucose levels. These tasks include self-monitoring glucose levels, keeping track of meals and physical activity, and considering factors such as stress or illness that may influence glucose levels.^2,3 The success of diabetes self-management is therefore a decisive prognostic factor for the course of diabetes.^2-4 However, the daily commitment to diabetes self-management is often perceived as a burden by many people with diabetes.^5,6 On one hand, people with diabetes need to acquire the skills and knowledge necessary to successfully master the various self-management tasks. At the same time, the constant need for diabetes self-management and the constant effort required to motivate oneself to perform these tasks can be emotionally distressing.^5-7

To help people manage their diabetes and its instrumental and emotional challenges, a digital diabetes diary can be a useful tool.⁸ Digital diabetes diaries are designed to support daily diabetes self-management by reducing the complexity of handling various diabetes data such as glucose values, carbohydrate content of meals, level of physical activity, and other glucose-affecting factors. They can also motivate people with diabetes to engage in their continued self-management by providing automated, personalized feedback.⁸ By replacing paper and pencil diaries, digital diabetes diaries aim to simplify self-management and reduce the burden of diabetes self-management for people with diabetes in daily life.⁹

A reduction in the perceived burden of self-management is an important person-reported outcome (PRO). Thus, PRO measures are essential in understanding the efficacy of digital diabetes diaries. Person-reported outcomes have become increasingly important in diabetes research in recent years.^10,11 These outcomes can help define the clinical utility and efficacy of new interventions and diabetes technologies.^10,12,13 The importance of PRO has also been recognized by regulatory bodies in assessing a new intervention’s added value.¹⁰ In diabetes research, one of the most important PRO is diabetes distress.⁷ Diabetes distress is a specific emotional reaction to diabetes and related self-management tasks.⁶ Elevated diabetes distress is quite common in people with diabetes with prevalence rates up to 40%.⁷ In general, higher diabetes distress scores are associated with lower quality of life, lower adherence, and unfavorable diabetes self-management.⁷ In addition, there is evidence that higher levels of distress may promote depressive symptom onset,¹⁴ and current evidence also suggests that diabetes distress is associated with cardiovascular health.¹⁵

The efficacy of digital diabetes diaries regarding PROs has not been systematically evaluated in randomized controlled trials (RCTs).¹⁶ An early study by Drion et al⁹ evaluated the effect of a digital diabetes diary on quality of life but found no significant differences compared to a standard paper diary. However, the study may have been underpowered because only 63 people with type 1 diabetes were included. Consequently, there is a lack of evidence from rigorous and large-scale trials that include people with both type 1 and type 2 diabetes.

Considering efforts in multiple countries to make digital health interventions prescribable and thus part of reimbursement plans of health insurance companies (e.g., Digital Healthcare Act in Germany, UK Medicines and Medical Devices Act 2021 in the United Kingdom),¹⁷ research on the efficacy of digital health interventions such as diabetes diaries is needed as part of evidence-based medicine. For this reason, the National Institute for Health and Care Excellence (NICE) and the German Federal Institute for Drugs and Medical Devices (BfArM) have set up evidence standards frameworks for the evaluation of digital interventions.¹⁷

Therefore, an RCT to evaluate the efficacy of a digital diabetes diary was conducted. The aim of the study was to analyze whether the use of a digital diary that supports people with diabetes with their self-management would lead to a reduction in diabetes distress as a primary endpoint when compared to a control group not using this digital solution. The superiority of the digital diabetes diary regarding secondary glycemic outcomes and secondary PRO was also evaluated.

Methods

Study Design

The study was a multicenter, open-label, parallel group, RCT with a 12-week follow-up. It was conducted in 41 secondary care practices specialized in the treatment of people with diabetes throughout Germany. Each study site consisted of at least one diabetologist and a study nurse. Ethical approval was obtained from the State Chamber of Physicians of Baden-Wuerttemberg (F-2020-121) as the primary vote as well as from the 13 local State Chambers of the participating study sites. Study conduct was in alignment with the declaration of Helsinki (Fortaleza, Brasil, October 2013), good clinical practice as well as European Union and national regulations. The study was registered at the German Clinical Trials Register (DRKS00022923). Details of the study protocol are published elsewhere.¹⁸

Participants

People with type 1 diabetes, type 2 diabetes, and gestational diabetes who performed self-monitoring of blood glucose (SMBG) on a daily basis were eligible to participate in the study. All types of diabetes were included to demonstrate the overall efficacy across types of diabetes as every person with diabetes can potentially benefit from a digital diabetes diary. Further inclusion criteria were age ≥16 years, last obtained HbA1c value <12% (107.6 mmol/mol), possession of a smartphone compatible with the intervention app (Android ≥5.0; iOS ≥11.4), willingness to accept the terms of conditions of the app, German language skills, and provision of informed consent. Exclusion criteria comprised the use of a continuous glucose monitoring (CGM) system, use of a digital diabetes diary or digital application for therapy documentation in the last 3 months before study entry, any medical conditions contradicting with diabetes therapy and study participation, and simultaneous participation in another clinical trial. The gender of participants was assessed via self-report (female, male, diverse). All participants were informed about the study both verbally and in writing and provided written informed consent.

Randomization and Masking

Participants were randomized to either the intervention group receiving the app for digitally logging diabetes data or the control group performing treatment-as-usual without the use of an app. Randomization was done in a 2:1 ratio favoring the intervention group. A computer-generated randomization sequence (Research Randomizer: www.randomizer.org) was used with block randomization and an initial block size of six and three for subsequent participants. The study centers received sealed opaque envelopes containing the group allocation for a single participant. Randomization was done by the study coordinating center that was not involved in recruitment. Owing to the nature of the intervention, blinding of participants and study sites was not possible.

Procedures

After obtaining informed consent, data were collected at baseline (visit 1) and at the follow-up visit (visit 2), which was 12 weeks after visit 1. At baseline, PROs were collected using standardized questionnaires. Demographic and medical data were documented in case report forms (CRFs) by the study centers based on medical records. Self-monitoring of blood glucose data of the previous 2 weeks before the baseline visit were downloaded using a secure data download system (DIABASS, Balingen, Germany) or with the practices’ own software and transmitted to the study coordinating site as a pseudonymized file. As part of the routine quarterly assessments, a blood sample was drawn and sent to a central laboratory to assess HbA1c (Automated Glycohemoglobin, Analyzer HLC-723G11; Tosoh; normal range 21-43 mmol/mol, [4.1%-6.1%]). When no collection of a blood sample was scheduled, the most recent HbA1c values from medical records were entered into the CRF due to regulatory requirements. After completion of the baseline visit, participants were allocated to one of the two study groups by the study center based on the randomization envelope. For participants in the control group, a date was scheduled for the follow-up visit, and participants continued with their usual mode of therapy and documentation. For participants in the intervention group, the intervention app (mySugr, mySugr GmbH, Austria) was downloaded on participants’ personal smartphone, and they were shown the functions of the app and how best to use them. The app is designed to support people with diabetes with their day-to-day diabetes management through the integration of several diabetes data such as glucose levels, estimated HbA1c values, data on medication/insulin intake, and entries on diet, weight, blood pressure, and activity as well as stress level. These data can be shared with the diabetes team prior to an appointment. The app presents users with an overview of diabetes management using easy-to-interpret reports with traffic light colors and suggests areas for improvement of self-management using pattern detection. However, the app does not make suggestions for adjustments of therapy. Psychological elements are also included, such as motivational challenges and positive feedback. A description of the intervention app is provided in Supplementary Table 1.

The follow-up visits in both groups took place 12 weeks after the baseline visit. Participants completed the same questionnaires as at baseline, and the study centers entered relevant changes in medical treatment into the CRF. Furthermore, all SMBG data collected since the baseline visit were downloaded from the glucose meters.

Outcomes

The primary outcome was the between-group difference in reduction of diabetes distress from baseline to follow-up. Diabetes distress was assessed using the German version of the Problem Areas in Diabetes (PAID) scale.¹⁹ The PAID questionnaire consists of 20 items with a scale ranging from 0 to 100 with higher scores indicating higher diabetes distress. A PAID score ≥17 is considered moderate diabetes distress with an indication to reduce this distress.²⁰ A prespecified hierarchical order of secondary outcomes was defined¹⁸: HbA1c, either assessed in a central laboratory or drawn from medical files; diabetes self-management using the Diabetes Self-Management Questionnaire (DSMQ)²¹; diabetes-specific empowerment using the German Diabetes Empowerment Scale (GDES)²²; percentage of glucose values <70 mg/dL (3.9 mmol/L) based on SMBG data; number of SMBG per day; percentage of glucose values between 70 and 180 mg/dL (3.9–10 mmol/L) based on SMBG data; percentage of glucose values >180 mg/dL (10 mmol/L) based on SMBG data; mean glucose based on SMBG data; general self-efficacy using the General Self-Efficacy Scale (GSE)²³; quality of life using the World Health Organization (WHO)-5 well-being index²⁴; and therapy satisfaction using the diabetes-related problems and satisfaction with insulin treatment (DSat) questionnaire.²⁵ In addition, participants in the intervention group completed an adapted version of the Mobile App Rating Scale to assess usability and satisfaction of the app.²⁶ Objective data on the use of the intervention app during the entire study period was derived from a pseudonymized study server.

For SMBG-related glycemic outcomes, the 2 weeks prior to the follow-up visit were used and compared to the 2 weeks prior to the baseline visit. All PRO measures were transformed to a scale from 0 to 100 with higher values indicating higher expression in the direction of the assessed characteristic.

Adverse events (AEs) and severe adverse events (SAEs) were collected by study centers using standardized AEs forms. In addition to AEs, rates of severe hypoglycemic (<54 mg/dl [3.0 mmol/L]) and hyperglycemic (>250 mg/dL [13.9 mmol/L]) events were analyzed using SMBG data of the whole study period.

Statistical Analysis

The sample size was calculated based on an anticipated effect size of 0.3 standard deviations, a two-sided alpha of 0.05, a power of 0.80, and a 2:1 randomization ratio.¹⁸ A total of 396 evaluable participants were needed. Anticipating a 15% drop-out rate, the recruitment goal was 466 participants. To account for multiplicity of testing, a prespecified hierarchical testing procedure was chosen, testing every outcome at a P level of .05 until an outcome became nonsignificant. Formal statistical testing was stopped after the first nonsignificant findings, but 95% confidence intervals (95% CI) are still reported.

Data for all outcomes showed a nonnormal distribution (Supplementary Table 2). Thus, for all analyses, robust regression analysis was chosen with the outcome measure at follow-up as the dependent variable, group allocation as predictor, and the baseline value of the respective outcome as covariate.

The intention-to-treat population was used for all analyses, consisting of every participant that was randomized. Missing values were replaced using multiple imputations based on the jump-to-reference approach with the control group as reference.²⁷ A sensitivity analysis was prespecified using the per-protocol population for all outcomes. Definition of the per-protocol population was based on the following criteria: meeting all eligibility criteria, completing the follow-up visit within a time frame of 42 to 137 days, using the intervention app on at least 10% of study days (intervention group), not using any digital diabetes diary (control group), and not using CGM during the study period. Prespecified moderator analyses were conducted analyzing the potential moderating effects of type of diabetes, gender, age, type of therapy, and number of SMBG measurements. For this, an interaction term of the moderator variable with group allocation was entered in the robust regression model. As a safety analysis, the incidence rate ratios of AE and SAE between the two groups were analyzed using negative-binomial regression analysis. In addition, the rate of severe hypoglycemic and hyperglycemic measurements (<54 mg/dL [3.0 mmol/L], >250 mg/dL [13.9 mmol/L]) per 1000 blood glucose measurements was analyzed between groups using negative-binominal regression analysis. Safety analyses were independent of the hierarchical testing. SPSS 29.0 (IBM Corp., Armonk, NY) and R version 4.2.2 were used for all statistical analyses.

Role of the Funding Source

The study design was initially proposed by the study coordinating center and was reviewed, discussed, and approved by the funder. The funder had no involvement in the collection, analysis, interpretation, and writing of the clinical trial report.

Results

From February 11, 2021, to June 24, 2022, a total of 424 participants were included in the study with 282 being randomized to the intervention group (66.5%) and 142 to the control group (33.5%). The last patient out was on October 4, 2022. Recruitment was stopped earlier due to the lower-than-anticipated drop-out rate of 6.4%. As planned, 397 participants completed the 12-week follow-up visit, with 18 and 9 participants lost to follow-up in the intervention and control group, respectively (Figure 1). Excluded from the per-protocol population were 32 participants of the intervention and 8 participants of the control group, leaving 357 participants for the sensitivity per-protocol analyses. Overall, data completeness was high with just 3 missing data for the primary endpoint (Supplementary Table 3).

Figure 1.

Trial profile.

Participants had a mean age of 52 years with a mean diabetes duration of 10 years (Table 1). Gender was equally distributed within the groups. Most of the participants had type 2 diabetes, followed by women with gestational diabetes, and type 1 diabetes. Insulin therapy was the most frequent type of diabetes therapy, followed by oral antidiabetic medication (OAD). Glycemic management of participants was rather sufficient with a mean HbA1c of 7.1%. The mean duration between the two study visits was 93.6 ± 14.2 days in the intervention group and 93.4 ± 13.6 days in the control group (P = .871).

Table 1.

Baseline Characteristics of the Intention-to-Treat Population.

	Intervention group (n = 282)	Control group (n = 142)
Age (in years)	51.2 (14.6)	52.8 (16.3)
Gender
Female	142 (50.4%)	71 (50.0%)
Male	140 (49.6%)	71 (50.0%)
Divers	0 (0%)	0 (0%)
Body mass index (kg/m²)	33.0 (7.3)	31.5 (6.7)
Type of diabetes
Type 1 diabetes	37 (13.1%)	16 (11.3%)
Type 2 diabetes	188 (66.7%)	101 (71.1)
Type 3 diabetes^a	0 (0%)	2 (1.4%)
Gestational diabetes	57 (20.2%)	23 (16.2%)
Diabetes duration (in years)	9.1 (10.1)	10.5 (12.1)
Diabetes therapy
Lifestyle	120 (42.6%)	61 (43.0%)
OAD	147 (52.1%)	67 (47.2%)
Incretins	53 (18.8%)	33 (23.2%)
Insulin	174 (61.7%)	93 (65.5%)
Insulin pump	3 (1.1%)	2 (1.4%)
Number of long-term complications^b	0.63 (1.00)	0.77 (1.21)
HbA1c
%	7.1 (1.5)	7.1 (1.5)
mmol/mol	54 (16.4)	54 (16.3)

Data are n (%), mean (SD). OAD = oral antidiabetic medication.

Type 3 diabetes includes maturity-onset diabetes of the youth (MODY) and pancreatogenic diabetes.

List of long-term complications: retinopathy, neuropathy, nephropathy, peripheral arterial occlusive disease, coronary heart disease, myocardial infarction, and stroke.

User data from the pseudonymized study server were available for 249 participants of the intervention group (88.3%). Overall, app use was quite high with registered app use on a median of 92.6% (interquartile [IQR] range: 57.7%-100%) of study days (Table 2). Figure 2 shows the distribution of the app use and indicates that, for example, more than 80% of participants used the app on more than 50% of study days. Over the whole study period, 3.1 ± 2.5 blood glucose entries per day were logged per person, 18.4% of participants made use of the option to add meal entries and 7.8% of participants uploaded meal pictures (Table 2).

Table 2.

Use of the Intervention App.

Intervention group n = 282	M (SD)/Md (IQR)
Days on which intervention app was used
Number of people with user data	249 (88.3%)
Mean (SD)	78.1% (28.5%)
Median (IQR)	92.6% (57.7%-100%)
Blood glucose entries per day
Number of persons with blood glucose entries	249 (88.3%)
Per day of use	3.7 (2.6)
Per study day	3.1 (2.5)
Meal entries per day
Number of people with meal entries	52 (18.4%)
Per day of use	2.6 (1.8)
Per study day	1.2 (1.5)
Meal pictures per day
Number of people with meal pictures	22 (7.8%)
Per day of use	1.8 (1.7)
Per study day	0.3 (0.3)

Figure 2.

Cumulative distribution of app use in the intervention group.

The intention-to-treat analysis showed that diabetes distress improved from a median score of 18.59 to 15.00 in the intervention group and slightly increased from 16.25 to 16.56 in the control group (Table 3). The baseline-adjusted diabetes distress at follow-up was significantly lower in the intervention group compared to the control group (model-based treatment effect: −2.20, 95% CI: −4.02 to −0.38, P = .0182) which indicates that the reduction in diabetes distress was associated with app use in the intervention group. Figure 3 shows that the reduction in diabetes distress in the intervention group started at PAID scores of 17 and higher and was evident across the whole range of elevated diabetes distress (Figure 3A), while in the control group, only a slight regression to the mean of highly elevated PAID scores was seen (Figure 3B). Model-based change in diabetes distress was higher in the intervention group (Figure 3C).

Table 3.

Primary and Secondary Outcomes in the Intention-to-Treat Analysis.

	Intervention group (n = 282)		Control group (n = 142)		Model-based adjusted treatment effect (95% CI)^a	P
	Baseline	Follow-up	Baseline	Follow-up	Model-based adjusted treatment effect (95% CI)^a	P
Primary outcome
Diabetes distress (PAID)	18.59 (7.50 to 31.25)	15.00 (6.25 to 26.27)	16.25 (7.50 to 33.11)	16.56 (8.75 to 30.00)	−2.20 (−4.02 to −0.38)	.0182
Secondary outcomes
HbA1c (%)	7.00 (5.90 to 7.93)	6.63 (5.90 to 7.40)	7.05 (6.00 to 8.10)	6.90 (5.80 to 7.50)	−0.02 (−0.12 to 0.09)	.7504
Diabetes self-management (DSMQ)	77.08 (66.67 to 85.42)	78.17 (68.75 to 86.08)	76.19 (63.96 to 85.42)	75.00 (64.58 to 85.42)	1.13 (−0.84 to 3.10)	^b
Diabetes empowerment (GDES)	68.94 (48.48 to 86.88)	72.73 (54.24 to 90.91)	66.67 (47.78 to 85.30)	66.67 (47.17 to 86.67)	2.70 (−1.22 to 6.63)
% <70 mg/dL	0.00 (0.00 to 1.92)	0.00 (0.00 to 2.50)	0.00 (0.00 to 1.84)	0.00 (0.00 to 2.88)	−0.10 (−0.43 to 0.23)
Number glucose measurements per day	2.86 (1.39 to 5.04)	2.58 (1.33 to 4.00)	3.23 (1.42 to 5.79)	2.55 (1.15 to 4.07)	0.17 (−0.19 to 0.53)
% 70 to 180 mg/dL	83.81 (60.44 to 100)	88.48 (71.25 to 98.90)	82.97 (66.09 to 100)	86.72 (70.53 to 98.46)	0.36 (−2.23 to 2.95)
% >180 mg/dL	14.02 (0.00 to 32.46)	10.89 (0.00 to 25.91)	13.90 (0.17 to 32.97)	12.17 (0.00 to 30.96)	−0.08 (−2.86 to 2.70)
Mean glucose (mg/dl)	144.07 (121.69 to 170.67)	136.33 (117.67 to 159.19)	147.44 (119.00 to 171.34)	140.39 (122.71 to 157.24)	0.48 (−3.44 to 4.39)
Self-efficacy (GSE)	80.00 (66.67 to 90.00)	80.00 (66.67 to 90.00)	76.67 (64.58 to 93.33)	73.33 (66.67 to 86.67)	1.07 (−0.96 to 3.11)
Quality of life (WHO-5)	64.00 (44.00 to 76.00)	65.87 (48.00 to 76.00)	64.00 (44.00 to 76.00)	64.39 (48.00 to 76.00)	0.63 (−2.44 to 3.70)
Treatment satisfaction (Dsat)	72.00 (62.00 to 82.00)	76.00 (66.00 to 85.24)	70.00 (61.50 to 80.50)	76.00 (63.63 to 82.50)	0.81 (−1.61 to 3.23)

Data are median (interquartile range). CI = confidence interval. Analyses are based on a robust regression controlled for respective baseline variable with group allocation as predictor.

Negative values in favor of the intervention group.

Formal statistical testing was stopped due to nonsignificant finding.

PAID = Problem Areas in Diabetes questionnaire. DSMQ = Diabetes Self-Management Questionnaire. GDES = German Diabetes Empowerment Scale. GSE = General Self-Efficacy questionnaire. WHO-5 = World Health Organization Well-being questionnaire. DSat = diabetes-related problems and satisfaction with insulin treatment questionnaire.

Figure 3.

Cumulative distribution of baseline and follow-up scores of diabetes distress for the intervention group (A), for the control group (B) and distribution of model-based change in diabetes distress for each group (C).

The baseline-adjusted HbA1c at follow-up was not significantly different between the groups (P = .7504, Table 3). Because of the hierarchical testing, formal inference statistical testing was stopped. Descriptively, diabetes self-management and diabetes empowerment improved in the intervention group, while no change or even deterioration was observed in the control group. Glycemic outcomes also indicate that there was a nominal improvement in the intervention group, however, with similar improvements in the control group.

Regarding safety, the rate of low glucose values <54 mg/dL (3.0 mmol/L) per 1000 blood glucose measurements was significantly lower in the intervention group (3.42 vs 4.54 events, P = .0484, Table 4). In addition, the rate of high glucose values >250 mg/dL (13.9 mmol/L) per 1000 blood glucose measurements over the whole study period was significantly lower in the intervention group compared to the control group (35.02 vs 52.29 events, P = .0021, Table 4). There were 9 SAE of 9 participants in the intervention group (3.2%) and 4 SAE of 4 participants in the control group (2.8%). No SAE was related to the use of the intervention app, and no severe hypoglycemia or ketoacidosis occurred. Incidence rate ratios of AE and SAE were not significantly different between groups (all P > .5; Table 4). A detailed description of all AEs and SAEs are given in Supplementary Tables 4 and 5.

Table 4.

Safety Parameter.

	Intervention group (n = 282)	Control group (n = 142)	Between-group difference^a	P
Incidence rate ratio (IRR) of adverse and severe adverse events
Adverse events	17 (6.0%)	10 (7.0%)	0.95 (0.41-2.19)	.9062
Severe adverse events	9 (3.1%)	4 (2.8%)	1.51 (0.40-5.67)	.5408
Incidence rate ratio (IRR) of events per 1000 glucose measurements
Number <54 mg/dL^b	3.42	4.54	0.75 (0.57-0.99)	.0484
Number >250 mg/dL^b	35.02	52.29	0.67 (0.52-0.86)	.0021

Data for the whole study period from baseline to follow-up were extracted.

IRR based on negative-binominal regression analysis with control group as reference.

Analysis based on all available glucose data (intervention group: n = 184; control group: n = 90).

Sensitivity analyses with the per-protocol population corroborated the results of the intention-to-treat analyses (Supplementary Table 6). There was a significant reduction in diabetes distress in the intervention group compared to the control group (−2.43, 95% CI: −4.49 to −0.37, P = .0211). Moderator analyses indicated no significant moderation of type of diabetes indicating that the effect on diabetes distress was not dependent on the type of diabetes (P_interaction = .5762, Supplementary Table 7). Additional moderator analyses also indicated no significant moderation of age, gender, type of therapy, and number of SMBG (Supplementary Table 7). Participants of the intervention group were highly satisfied with the app (Supplementary Figure 1) with an overall rating of 4.3 ± 0.7 (scale: 1-5).

Discussion

This multicenter RCT showed the efficacy of a digital diabetes logbook to reduce diabetes distress when compared to treatment-as-usual without the use of an app. The per-protocol analysis corroborated this finding and also indicated that the intervention effect was larger when analyzing participants who met all requirements of the study protocol and used the app on at least 10% of study days. Interestingly, the effect of the intervention can be seen at moderate diabetes distress levels (PAID sums core ≥17), indicating that there was an improvement across the whole range of clinically meaningfully elevated diabetes distress levels. The model-based treatment effect revealed an approx. 11% reduction in diabetes distress in the intervention group. Thus, the use of the digital diabetes diary was associated with a clinically significant reduction in diabetes distress in a diverse population of people with type 1, type 2, and gestational diabetes. Furthermore, the effect was not moderated by type of diabetes, age, gender, and type of therapy.

Importantly, frequency of app use was high with 50% of intervention participants using the app on more than 92.6% of study days. This demonstrates a high level of usability and satisfaction with the app. In addition, the drop-out rate was very low (6.4%). The high frequency of app use and the low drop-out rate indicate that the effects on diabetes distress are not biased due to a potential performance or attrition bias. The low drop-out rate is rather unusual for studies on digital health interventions and in contrast to many other studies in this field.^28-30

The study was conducted mirroring the current health care situation in Germany, with study visits aligned with the routine quarterly visits and no additional study visits in between to reduce a study effect as good as possible. Furthermore, 41 study centers recruited a broad range of people with diabetes with all types of diabetes therapy (age 16-75, type 1 diabetes, type 2 diabetes, gestational diabetes). Therefore, the external validity and generalization of the study can be considered to be high.

For the secondary outcomes, an improvement of the intervention group was seen for almost all outcome variables, but none of the tested effects reached a significance level of P < .05. However, regarding glycemic outcome parameters, it must be noted that they were already in a favorable range at baseline for most of the participants.^31,32 Thus, there was no urgent indication for a further reduction of the HbA1c value in most study participants, especially for women with gestational diabetes. This may explain the comparatively small reduction in HbA1c. The systematic review and meta-analysis by Moschonis et al²⁸ also demonstrated that higher efficacy of digital health interventions can be achieved with higher baseline HbA1c values. Regarding the glucose safety parameters, there were significant reductions in severe hyperglycemic and severe hypoglycemic glucose measurements in the intervention group. However, level 1 hypoglycemia and hyperglycemia did not significantly improve. This indicates that although overall glycemic management was not improved, the intervention app may contribute to avoiding highly elevated glucose levels that increase the risk for ketosis as well as avoiding seriously low glucose values which increase the risk for severe hypoglycemic events.

Regarding secondary PROs, baseline values indicated a good level of psychosocial well-being with the median WHO-5 score being comparable with the German standard sample.²⁴ This may also explain the lack of effects on secondary outcomes.

Existing evidence of other digital health interventions demonstrate that an improvement in PRO cannot be seen as low-hanging fruit. Previous studies have often failed to demonstrate the effects of digital health interventions on psychosocial outcomes,^9,33-35 with significant effects resulting from more intensive cognitive behavioral therapy [CBT]-based digital interventions in people affected by more severe mental health problems.^29,30 The digital health intervention tested in the current RCT can be considered a holistic diabetes app that integrates glucose, contextual and self-management-related data into a single app, to support people with diabetes with their daily diabetes self-management. However, as a digital diabetes diary it did not contain specific CBT elements. Thus, the app is designed for all people with diabetes performing regular SMBG, and not specifically for people with mental health issues. The beneficial effects of this general digital diabetes diary on improved diabetes distress may be due to the motivational features [e.g., motivational challenges, gamification] of the tested app that is designed to simplify and structure daily diabetes management. The effect is of clinical importance as it can be easily implemented in routine care and can be used by most people with diabetes.

When interpreting the results, the following limitations must be considered. First, the follow-up period of three months was short due to the requirements of the BfArM for the evaluation of digital applications that require studies to be fully conducted within 12 months. However, real-world data suggests prolonged effects of mySugr over a 6-month period with high engagement.³⁶ Second, due to the nature of the app, use of a CGM system was an exclusion criterion that may limit generalizability, especially for people with type 1 diabetes. Third, for regulatory reasons, blood samples for the determination of HbA1c in the central laboratory could only be obtained if a routine blood draw had already been scheduled. Therefore, not all HbA1c values were determined in the central laboratory but in the local laboratories at the study centers. However, consistency of method was guaranteed as the method of determining HbA1c for an individual was kept constant within an individual. Furthermore, baseline HbA1c and PAID levels were rather low. This may be due to the inclusion criteria, which did not include fixed cut-off levels for neither HbA1c nor PAID. In addition, the inclusion of women with gestational diabetes most likely led to lower mean HbA1c values. Finally, no information on social determinants of health was available.

Conclusions

In this 12-week, multicenter, RCT, the efficacy of a digital diabetes diary in reducing diabetes distress was demonstrated. Thus, an important aspect of psychosocial well-being of people with type 1, type 2, and gestational diabetes was significantly improved. The study was conducted in accordance with the BfArM’s requirements for digital health applications and thus fulfills the demands of the German Digital Healthcare Act. Because of the beneficial effects of the digital diabetes diary on diabetes distress, an outcome that falls within the requested positive care effects of the BfArM, it is eligible to be listed as a diabetes app that can be prescribed. The results of the study have therefore important implications for routine clinical care, as they demonstrate the potential of the digital diabetes diary to be a feasible and highly accessible tool for people with diabetes. The study showed a low drop-out rate and high frequency of app use, suggesting that the intervention could have a broad reach and significant impact to reduce diabetes burden for people with diabetes. These results highlight the potential of the app to provide an effective, scalable, and practical solution for people with diabetes and health care providers to facilitate diabetes care and reduce the burden of diabetes caused by the demands of daily diabetes self-management.

Supplemental Material

sj-docx-1-dst-10.1177_19322968241239870 – Supplemental material for Efficacy of a Digital Diabetes Logbook for People With Type 1, Type 2, and Gestational Diabetes: Results From a Multicenter, Open-Label, Parallel-Group, Randomized Controlled Trial

Supplemental material, sj-docx-1-dst-10.1177_19322968241239870 for Efficacy of a Digital Diabetes Logbook for People With Type 1, Type 2, and Gestational Diabetes: Results From a Multicenter, Open-Label, Parallel-Group, Randomized Controlled Trial by Dominic Ehrmann, Norbert Hermanns, Katharina Finke-Gröne, Timm Roos, Johanna Kober, Vanessa Schäfer, Michael Krichbaum, Thomas Haak, Ralph Ziegler, Lutz Heinemann, Cosima Rieger, Ezgi Bingol, Bernhard Kulzer and Stephan Silbermann in Journal of Diabetes Science and Technology

Supplemental Material

sj-docx-2-dst-10.1177_19322968241239870 – Supplemental material for Efficacy of a Digital Diabetes Logbook for People With Type 1, Type 2, and Gestational Diabetes: Results From a Multicenter, Open-Label, Parallel-Group, Randomized Controlled Trial

Supplemental material, sj-docx-2-dst-10.1177_19322968241239870 for Efficacy of a Digital Diabetes Logbook for People With Type 1, Type 2, and Gestational Diabetes: Results From a Multicenter, Open-Label, Parallel-Group, Randomized Controlled Trial by Dominic Ehrmann, Norbert Hermanns, Katharina Finke-Gröne, Timm Roos, Johanna Kober, Vanessa Schäfer, Michael Krichbaum, Thomas Haak, Ralph Ziegler, Lutz Heinemann, Cosima Rieger, Ezgi Bingol, Bernhard Kulzer and Stephan Silbermann in Journal of Diabetes Science and Technology

Footnotes

Acknowledgements

The authors thank Marius Kraus and Iris Vesper for study management and Julia Ehrmann and Susanne Bokeloh for their help with clinical project management and coordination. They thank the following study centers for the recruitment of participants: Diabetes Zentrum Mergentheim, Prof Dr Thomas Haak, Bad Mergentheim; BAG Diabeteszentrum, Dr Dietrich Tews, Gelnhausen; MVZ DiaMedicum Würzburg, Dr Dominik Bergis, Würzburg; Diabetologie Frechen, Dr Florian Wirges, Frechen; Endokrinologikum Ulm, Prof. Dr Werner Kern, Ulm; Diabetologische Schwerpunktpraxis Hückelhoven, Dr Gernot Sachs, Hückelhoven; MVZ DiaMedicum Bad Mergentheim, Dr Simon Vidal, Bad Mergentheim; Gemeinschaftspraxis Dres. Denger & Pfitzner, Dr Thomas Pfitzner, Friedrichsthal; Praxis Dr Barion, Dr Ralf Barion, Niederkassel; Gemeinschaftspraxis Dres. Zweigle/Zweigle, Dr Bernhard Zweigle, Aalen; Gesundheitszentrum Markdorf, Dr Volker Kroll, Markdorf; Praxis Dr Woitek & Kollegen, Dr Cornelia Woitek, Wurzen; Diabetespraxis Rheine, Dr Clio Roussos, Rheine; Diabetespraxis Buxtehude, Dr Oliver Schubert-Olesen, Buxtehude; Diabetologische Schwerpunktpraxis, Dr Winfried Keuthage, Münster; Diabetologische Schwerpunktpraxis Rosenheim, Prof. Dr Michael Hummel, Rosenheim; Diabeteszentrum am Marienplatz München, PD Dr Martin Füchtenbusch, München; St. Josefskrankenhaus—Innere Medizin, Diabetologie, Prof. Dr Erhard Siegel, Heidelberg; Klinik für Innere Medizin Schopfheim, Dr Michael Maraun, Schopfheim; Diabeteszentrum Neckar-Odenwald, Dr Carsten Iannello; Mosbach; Diabetes-Zentrum Potsdam, Dr Uta Rieger, Potsdam; Gemeinschaftspraxis Dres. Neureither/Schumacher, Dr Sabine Schumacher-Herold, Heidelberg; diabendo Praxisgemeinschaft, Dr Stephan Arndt, Rostock; Hausärztlich Internistische Gemeinschaftspraxis—Dres. Cseke und Friese, Dr Almos Cseke, Gießen; Gemeinschaftspraxis Dres. Kaltheuner/Schultens-Kaltheuner/v. Boxberg, Dr Matthias Kaltheuner, Leverkusen; Zentrum für Diabetologie Bergedorf, Dr Jens Kröger, Hamburg; Diabetes-Zentrum Hannover-Nord, Ralph Geldmacher, Hannover; MVZ im Altstadt-Carree Fulda, Dr Jörg Simon, Fulda; Diabetologische Schwerpunktpraxis Neuss, Dr Kirsten Holtappels, Neuss; Hausarzt- und Diabetologische Schwerpunktpraxis Hohenmölsen—Weiβenfels, Dr Karsten Milek, Hohenmölsen; MVZ am Bahnhof Spandau, Dr Uta Dorothea Stephan, Berlin; Diabetes-Zentrum Neustadt, Dr Olaf Ney, Neustadt am Rübenberg; Praxis Dr Lange, Dr Martina Lange, Rheinbach; Diabeteszentrum Ludwigsburg, Dr Jörg Gloyer, Heike Flohr, Ludwigsburg; Gemeinschaftspraxis Dr med. Schreiber u. Petra Werkmeister, Petra Werkmeister, Volkertshausen; Diabetologische Schwerpunktpraxis, Dr Iris Donati-Hirsch, Dortmund; Diabetologische Schwerpunktpraxis Dres. Sommer/Milnik, Dr Alexander Milnik, Aschaffenburg; Diabeteszentrum Osnabrück, Dr Markus Graf, Osnabrück; Diabetespraxis, Dr Hermann Braun, Berlin; Gemeinschaftspraxis Dres. Münch, Dr Christian Münch, Immenhausen; Diabetologie am Nordbahnhof, Dr Burkhard Schnückel, Paderborn.

Abbreviations

AE, adverse events; BfArM, German Federal Institute for Drugs and Medical Devices; CBT, cognitive behavioral therapy; CGM, continuous glucose monitoring; CI, confidence interval; CRF, case report form; DSat, Diabetes-related Problems and Satisfaction with Insulin Treatment Scale; DSMQ, Diabetes Self-Management Scale; GDES, German Diabetes Empowerment Scale; GSE, General Self-Efficacy Scale; IQR, inter-quartile range; NICE, National Institute for Health and Care Excellence; OAD, oral antidiabetic medication; PAID, Problem Areas in Diabetes Scale; PRO, person-reported outcomes; RCT, randomized controlled trial; SAE, severe adverse events; SMBG, self-monitoring of blood glucose; WHO-5, World Health Organization Well-being scale.

Author Contributions

DE and NH analyzed the data. DE, NH, and BK wrote the manuscript. DE, NH, and BK designed the study and coordinated the study conduct. KF-G, TR, and MK were responsible for study conduct and monitoring, and revised the manuscript. CR contributed to the study management. JK, VS, RZ, LH, EB, TH, and SS contributed to the discussion and revised the manuscript. DE, NH, BK, CR, and SS had full access to all the analyzed data. All authors had final responsibility for the decision to submit for publication.

Data Sharing

Anonymized data, without any demographic identifiers, underlying the results and analysis can be made available to researchers upon reasonable request to the corresponding author after publication. A data access agreement needs to be signed in advance.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: DE reports Advisory Board member fees from mySugr, Dexcom Germany, and Roche Diabetes Care as well as honoraria for lectures from Berlin-Chemie AG, Sanofi-Aventis, Dexcom Germany, and Roche Diabetes Care. NH reports Advisory Board member fees from Abbott Diabetes Care and Insulet as well as honoraria for lectures from Berlin-Chemie AG, Becton Dickenson, Sanofi Germany, Roche Diabetes Care, and Dexcom Germany. TR reports honoraria for lectures from Berlin-Chemie AG. JK is an employee of mySugr. VS is an employee of Roche Diabetes Care Deutschland. TH reports consulting fees from Eli Lilly, NovoNordisk, Sanofi, Boehringer Ingelheim, and Abbott Diabetes Care as well as honoraria for lectures from Abbott Diabetes Care, Sanofi, and Eli Lilly. RZ reports consulting fees from Roche Diabetes Care and mySugr as well as honoraria for lectures from Roche Diabetes Care, Dexcom, VitalAire, NovoNordisk, and Abbott Diabetes Care. He participated in data safety monitoring boards or advisory boards of Roche Diabetes Care, mySugr, Dexcom, NovoNordisk, and Eli Lilly. LH reports consulting fees from Roche Diabetes Care, Lifecare, Medtronic, Spiden, Embecta, Dexcom, Onetwenty, Perfood, Boydsense, Pharmasense, Unomedical and Sinocare. CR is an employee of Roche Diabetes Care. EB is an employee of mySugr. BK reports Advisory Board member fees from Abbott Diabetes Care, Embecta, Roche Diabetes Care, Novo Nordisk, Berlin-Chemie AG and Dexcom Germany as well as honoraria for lectures from Sanofi Germany, Novo Nordisk, Abbott Diabetes Care, Roche Diabetes Care, Berlin-Chemie AG, Embecta, Dexcom, and Feen. In addition, he reports support for travel and fees for scientific meetings from Sanofi, Roche Diabetes Care and Berlin-Chemie AG as well as unpaid obligations as workshop leader and member of working groups of the German Diabetes Association. SS is an employee of Roche Diabetes Care. The remaining authors have nothing to disclose.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The study was funded by Roche Diabetes Care GmbH.

ORCID iDs

Dominic Ehrmann

Norbert Hermanns

Michael Krichbaum

Ralph Ziegler

Lutz Heinemann

Supplemental Material

Supplemental material for this article is available online.

References

ElSayed

Aleppo

Aroda

, et al. 2. Classification and diagnosis of diabetes: standards of care in diabetes—2023. Diabetes Care. 2022;46(suppl 1):S19-S40.

ElSayed

Aleppo

Aroda

, et al. 5. Facilitating positive health behaviors and well-being to improve health outcomes: standards of care in diabetes—2023. Diabetes Care. 2022;46(suppl 1):S68-S96.

Davis

Fischl

Beck

, et al. 2022 national standards for diabetes self-management education and support. Diabetes Spectr. 2022;35(2):137-149.

Powers

Bardsley

Cypress

, et al. Diabetes self-management education and support in type 2 diabetes: a joint position statement of the American Diabetes Association, the American Association of Diabetes Educators, and the Academy of Nutrition and Dietetics. Diabetes Educ. 2017;43(1):40-53.

Polonsky

Anderson

Lohrer

, et al. Assessment of diabetes-related distress. Diabetes Care. 1995;18(6):754-760.

Fisher

Polonsky

Hessler

. Addressing diabetes distress in clinical care: a practical guide. Diabet Med. 2019;36(7):803-812.

Skinner

Joensen

Parkin

. Twenty-five years of diabetes distress research. Diabet Med. 2020;37(3):393-400.

Shan

Sarkar

Martin

. Digital health technology and mobile devices for the management of diabetes mellitus: state of the art. Diabetologia. 2019;62(6):877-887.

Drion

Pameijer

van Dijk

Groenier

Kleefstra

Bilo

. The effects of a mobile phone application on quality of life in patients with type 1 diabetes mellitus: a randomized controlled trial. J Diabetes Sci Technol. 2015;9(5):1086-1091.

10.

Agiostratidou

Anhalt

Ball

, et al. Standardizing clinically meaningful outcome measures beyond HbA1c for type 1 diabetes: a consensus report of the American Association of Clinical Endocrinologists, the American Association of Diabetes Educators, the American Diabetes Association, the Endocrine Society, JDRF International, The Leona M. and Harry B. Helmsley Charitable Trust, the Pediatric Endocrine Society, and the T1D Exchange. Diabetes Care. 2017;40(12):1622-1630.

11.

Wee

PJL

Kwan

Loh

DHF

, et al. Measurement properties of patient-reported outcome measures for diabetes: systematic review. J Med Internet Res. 2021;23(8):e25002.

12.

US Department of Health and Human Services FDA Center for Drug Evaluation and Research; US Department of Health and Human Services FDA Center for Biologics Evaluation and Research; US Department of Health and Human Services FDA Center for Devices and Radiological Health. Guidance for industry: patient-reported outcome measures: use in medical product development to support labeling claims: draft guidance. Health Qual Life Outcomes. 2006;4:79.

13.

Varma

Richman

Ferris

Bressler

. Use of patient-reported outcomes in medical product development: a report from the 2009 NEI/FDA Clinical Trial Endpoints Symposium. Invest Ophthalmol Vis Sci. 2010;51(12):6095-6103.

14.

Ehrmann

Kulzer

Haak

Hermanns

. Longitudinal relationship of diabetes-related distress and depressive symptoms: analysing incidence and persistence. Diabet Med. 2015;32(10):1264-1271.

15.

Ehrmann

Chatwin

Schmitt

, et al. Reduced heart rate variability in people with type 1 diabetes and elevated diabetes distress: results from the longitudinal observational DIA-LINK1 study. Diabet Med. 2023;40(4):e15040.

16.

Bonoto

de Araujo

Godoi

, et al. Efficacy of mobile apps to support the care of patients with diabetes mellitus: a systematic review and meta-analysis of randomized controlled trials. JMIR Mhealth Uhealth. 2017;5(3):e4.

17.

Essen

Stern

Haase

, et al. Health app policy: international comparison of nine countries’ approaches. NPJ Digit Med. 2022;5(1):31.

18.

Ehrmann

Eichinger

Vesper

, et al. Health care effects and medical benefits of a smartphone-based diabetes self-management application: study protocol for a randomized controlled trial. Trials. 2022;23(1):282.

19.

Hermanns

Kulzer

Krichbaum

Kubiak

Haak

. How to screen for depression and emotional problems in patients with diabetes: comparison of screening characteristics of depression questionnaires, measurement of diabetes-specific emotional problems and standard clinical assessment. Diabetologia. 2006;49(3):469-477.

20.

de Wit

Pouwer

Snoek

. How to identify clinically significant diabetes distress using the Problem Areas in Diabetes (PAID) scale in adults with diabetes treated in primary or secondary care? evidence for new cut points based on latent class analyses. BMJ Open. 2022;12(3):e056304.

21.

Schmitt

Gahr

Hermanns

Kulzer

Huber

Haak

. The Diabetes Self-Management Questionnaire (DSMQ): development and evaluation of an instrument to assess diabetes self-care activities associated with glycaemic control. Health Qual Life Outcomes. 2013;11(1):138.

22.

Bergis

Ehrmann

Hermanns

Kulzer

Haak

. Lässt sich Empowerment bei Menschen mit Diabetes messen? [Is empowerment measurable in persons with diabetes?]. Diabetol Stoffwechs. 2012;7(S 01):9.

23.

Schwarzer

Jerusalem

. Generalized self-efficacy scale. J Weinman, S Wright, & M Johnston, Measures in health psychology: a user’s portfolio. Causal Control Beliefs. 1995;35:37.

24.

Brähler

Mühlan

Albani

Schmidt

. Teststatistische prüfung und normierung der deutschen versionen des EUROHIS-QOL lebensqualität-Index und des WHO-5 wohlbefindens-index [Test statistics and standardisation of the German versions of the EUROHIS-QOL Quality of Life Index and the WHO-5 Well-being Index]. Diagnostica. 2007;53(2):83-96.

25.

Kulzer

Bauer

Hermanns

Bergis

. Entwicklung eines Problemfragebogens fuer Diabetiker zur Identifikation von Schwierigkeiten im Umgang mit der Krankheit [Development of a questionnaire for the assessment of diabetes related problems and satisfaction with insulin treatment]. Verhaltenstherapie. 1995;5(suppl 1):A72.

26.

Messner

Terhorst

Barke

, et al. The German version of the Mobile App Rating Scale (MARS-G): development and validation study. JMIR Mhealth Uhealth. 2020;8(3):e14479.

27.

Carpenter

Roger

Kenward

. Analysis of longitudinal trials with protocol deviation: a framework for relevant, accessible assumptions, and inference via multiple imputation. J Biopharm Stat. 2013;23(6):1352-1371.

28.

Moschonis

Siopis

Jung

, et al. Effectiveness, reach, uptake, and feasibility of digital health interventions for adults with type 2 diabetes: a systematic review and meta-analysis of randomised controlled trials. Lancet Digit Health. 2023;5(3):e125-e143.

29.

Nobis

Lehr

Ebert

, et al. Efficacy of a web-based intervention with mobile phone support in treating depressive symptoms in adults with type 1 and type 2 diabetes: a randomized controlled trial. Diabetes Care. 2015;38:776-783.

30.

Newby

Robins

Wilhelm

, et al. Web-based cognitive behavior therapy for depression in people with diabetes mellitus: a randomized controlled trial. J Med Internet Res. 2017;19(5):e157.

31.

ElSayed

Aleppo

Aroda

, et al. 6. Glycemic targets: standards of care in diabetes—2023. Diabetes Care. 2022;46(suppl 1):S97-S110.

32.

Battelino

Danne

Bergenstal

, et al. Clinical targets for continuous glucose monitoring data interpretation: recommendations from the international consensus on time in range. Diabetes Care. 2019;42(8):1593-1603.

33.

Murray

Sweeting

Dack

, et al. Web-based self-management support for people with type 2 diabetes (HeLP-Diabetes): randomised controlled trial in English primary care. BMJ Open. 2017;7(9):e016009.

34.

Choi

Bruno

, et al. Impact of MyDiabetesPlan, a Web-based patient decision aid on decisional conflict, diabetes distress, quality of life, and chronic illness care in patients with diabetes: cluster randomized controlled trial. J Med Internet Res. 2020;22(9):e16984.

35.

Hilmarsdóttir

Sigurðardóttir

ÁK

Arnardóttir

. A digital lifestyle program in outpatient treatment of type 2 diabetes: a randomized controlled study. J Diabetes Sci Technol. 2021;15(5):1134-1141.

36.

Debong

Mayer

Kober

. Real-world assessments of mySugr mobile health app. Diabetes Technol Ther. 2019;21(S2):S235-S240.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.05 MB

0.24 MB