Towards responsible digital health implementation: A mixed-methods exploratory study developing a tool to assess workforce experience

Step 1: Re-analysis of a scoping review

Step 2: Survey and priority-setting

The themes mapped on the modified Work-System Model were classified following the five categories of the Quintuple Aim.^12,29 All themes assigned to Care-team well-being were selected and converted to survey items. The survey, designed to support prioritisation rather than statistical inference, used a six-point scale (0 = no priority; 5 = high priority) to elicit priority scores from participants. We used purposive and snowball sampling via institutional mailing lists of the centres where the authors work, and professional networks to recruit survey participants. This was chosen in order to recruit an information-rich group with direct experience in DHT implementation. Participants were eligible if they were HCPs (e.g., physicians, nurses) or hospital employees (e.g., IT specialists, project managers) with current or recent involvement in DHT-implementation within hospital settings. The survey was open throughout November 2024 and administered via the Research Electronic Data Capture (REDCap) web application.^30,31 The survey was offered in Dutch, and later translated to English, for this publication (supplementary materials, Text S3). The survey aimed to obtain scores for the priority-setting of items, which would inform further qualitative exploration and tool development. We included submissions with 100% item completion. After obtaining the scores, we used a reweighted range voting approach using the Reweighted Priority-Setting (REPS) tool to obtain a ranked list of themes based on priority scores. ³² Participants score each theme on a fixed scale and these scores are the input for the REPS tool. The tool uses reweighted range voting to reweigh each participant’s influence after assigning a rank to a theme. The sum of the individual participant’s scores on previous ranked themes result in a decline of their weight for assigning a rank to the next theme. The higher their previous scores on ranked themes, the lower their weight. This makes REPS suitable for situations where the goal is to reflect group priorities, ³³ even in small, information-rich samples. Note that it is not a consensus method. We used the unmodified range voting method in the tool. Any ties during ranking with the tool were resolved by selecting the theme with the highest unweighted sum score first. The REPS-tool’s top-10 ranked themes were selected for qualitative exploration. It functioned as a pragmatic funnel, providing focus from the broad evidence base to specific topics of interest for in-depth qualitative exploration and checklist development. As a sensitivity check, we also performed an unweighted sum score ranking of all themes.

Step 3: Focus-groups and interviews

Two focus-groups of approximately 120 minutes each were organised through online video calls in December 2024 to facilitate participation across sites. Participant recruitment was done similarly to that for the survey. We used purposive sampling to capture variation in profession, and hospital type among stakeholders. Consistent with our constructivist perspective and qualitative study aim, we did not seek a statistically representative sample. Instead, we aimed to capture a broad range of perspectives through purposive sampling across professional roles and organisations. Topic guides were developed to ensure a semi-structured approach for the focus-groups (Supplementary materials, Text S4). In the focus-groups we aimed to explore why each prioritised theme matters to HCPs in terms of well-being, and to explore observable indicators that a priority theme is adequately addressed during implementation. Focus-groups were moderated by TB, with MO observing. Recordings were collected online via Microsoft Teams, where initial computer-generated verbatim transcripts were de-identified and further hand corrected by one researcher (TB) using the recordings. The same topic guide was used for three one-on-one online semi-structured interviews with purposefully sampled participants. With these interviews, we aimed to reach additional depth to the topic. One researcher (TB) conducted the interviews. Both the focus-groups and the interviews were conducted in Dutch. We used Braun and Clarke’s reflexive thematic analysis to analyse the data. ²⁷ Coding and theme development were iterative. TB conducted initial open coding in Dutch, followed by clustering codes into candidate themes. MO reviewed the developing themes. Discrepancies and interpretive decisions were discussed between TB and MO until consensus on the thematic structure was reached. To enhance reflexivity, themes were also discussed with co-authors, who were medical specialists, implementation experts, and epidemiologists. Throughout the analysis, we used analytic memos to document coding decisions and theme development Themes were then translated to English. Based on the richness and relevance of the collected data, we considered the dataset to provide sufficient information to support the development of an implementation oriented checklist. Ultimately, for this publication, themes were translated to English.

Step 4: Tool development

Results from the prioritised themes in Step 2 and the RTA in Step 3 were used to generate the tool’s items. These items were themes re-phrased as guiding questions or prompts to support structured reflection on whether and how each aspect was addressed during DHT-implementation. Qualitative analyses in Step 3 provided additional context for each item of the tool. Items were merged when survey themes and qualitative insights consistently reflected the same underlying mechanism. Only themes supported by both prioritisation data and qualitative findings were retained as separate tool items, ensuring that the final tool balanced comprehensiveness with feasibility. Draft items were iteratively refined for clarity and feasibility using feedback from co-authors.

Acceptance

Acceptance of DHTs was described as dependent on the perceived balance between short-term effort and long-term benefit. Participants acknowledged that initial implementation requires additional time for training and patient instruction, with efficiency gains becoming apparent later. Where these expected future benefits were considered credible, staff were more willing to tolerate the initial investment. As one participant noted:

‘’By actually showing them (NB. HCPs) what it will bring them (…) So that way (…) it costs at the beginning, it requires an investment of time, but the return is greater.’’ (P9)

Participants noted that acceptance differed across professional groups, often related to variation in experience and familiarity with DHT rather than age. While broad inclusion of HCPs during implementation was viewed as important, several argued that once a DHT demonstrates value, organisations should establish norms to ensure adoption, even among reluctant staff.

Champions

Champions, trusted and motivated colleagues within the same professional group, were seen as critical to align other team members. Unlike general enthusiasts, champions actively mobilise others, translate innovations into practice, and sustain momentum. Their influence was strongest when they operated as peers within their team. Misalignment between champion and affected team often undermined uptake. Participants emphasised that multiple champions are often needed to bridge teams and ensure broad support.

Process & development

Early involvement of HCPs in the development and implementation of DHTs was repeatedly identified as essential to ensure usability and prevent frustration. Subsequently, participants described multidisciplinary working groups, including representatives from each affected role as effective in surfacing problems early. Not only was this related to HCPs, but the inclusion of IT specialists and legal advisors was also considered important, to avoid pushing decisions onto clinicians that lie outside their expertise. Furthermore, transparent communication about realistic timelines was another recurring theme: professionals cautioned that the process often spans years rather than months, and that unclear expectations could lead to demotivation or resistance. And lastly, hospitals with a centralised DHT-desk or coordination team were regarded as particularly effective in supporting implementation, connecting relevant stakeholders, and avoiding duplication of efforts. In contrast, its absence was perceived as a shortfall to unburden the HCP.

Role of the professional

Clarity about roles and responsibilities was viewed as a prerequisite for sustained engagement with DHTs. Ambiguity around who is responsible for particular tasks or decisions was said to cause inefficiency, frustration, and reduced motivation for the HCP. Conversely, clear implementation plans in which responsibilities are explicitly defined were perceived as facilitating smooth adoption, positively reflecting on the HCP. Participants stressed that DHT should be designed to make work easier rather than creating additional (administrative) burden.

Communication

Participants highlighted the role of departmental leads, such as head nurses, as crucial intermediaries who translate organisational information into practically relevant messages for the frontline staff. Timely updates were described as motivating, whereas lack of communication was associated with uncertainty and resistance. As one professional noted:

“That’s already a first step in creating motivation: giving people a little stimulus by showing: we’re working on this. (…) Then you already have some awareness of what’s going on and what’s coming. (…) If you’re informed at an earlier stage, (…) that really contributes to motivation.” (P8).

Training

Participants stressed that training should be structurally embedded in the organisation, for example, offered during working hours and incorporated into onboarding programmes, to ensure that all professionals are adequately supported. Crucially, participants distinguished between two types of competences: technical skills, such as navigating an application, which can be taught relatively easily, and clinical competences, such as deciding how to act on remotely monitored patient data, which require more context-specific discussion and role clarity. Without support across these domains, HCPs will likely fall back on old routines or avoid the technology altogether.

Cross-cutting observation: Difficulty articulating HCP-focused impacts

Across themes, participants found it challenging to articulate how DHTs affected their well-being, even when directly prompted to reflect on it. Discussions often shifted toward patient benefits or workflow efficiency rather than personal or collective impacts on staff. This difficulty was unexpected, as one of the main aims of the focus-groups and interviews was to explore well-being more explicitly. The finding itself is therefore meaningful: it illustrates how the well-being of HCPs remains a more implicit, under-discussed topic in digital transformation processes. Participants reflected on this imbalance, noting that data and evaluation mechanisms predominantly focus on patients rather than professionals. The cardiologist in our focus-group observed:

“See, we measure a lot for patients. It’s a bit funny right? Because (…) we ask healthcare professionals far less.” (P1).

This underscores both the need and the challenge of systematically integrating HCP well-being into DHT-implementation. It also helps explain why the resulting tool emphasises work-related factors influencing staff experience, rather than well-being outcomes.