Multicentric Design and Validation of a Set of Democratized Hospital Patient Reported Experience Measures in Spain

Abstract

In recent years, an increasing number of Patient Reported Experience Measures (PREMs) have been enabled to explore and guide improvements. The aim is to establish a set of questionnaires for assessing hospitals care, share results, and perform a transparent formal cognitive and psychometric validation process for some of these. A set of 23 questionnaires was developed through literature review and discussion with professionals from one Spanish tertiary hospital, and supplemented with Net Promoter Scores (NPS). The hospital piloted the questionnaires, receiving 400,719 responses (November-2022-November-2025). These were described for reference benchmarking. Questionnaire dashboards were enabled to all hospital professionals Intranet. Five of the PREMs underwent content validation:focus groups/interviews in the aforementioned hospital(July-2024-September-2025) and psychometric validation(June–2025-January-2026) involving 11 hospitals from 4 Spanish regions, including an item-bank proposal and scoring. A cultural adaptation process was performed into Catalan and English, involving additional hospitals. The hospital received 400,719 responses from 23 questionnaires (24% overall response-rate). The main average NPS values (range across services) were: adult patients hospitalization/adults referring to children’s hospitalization (77ad/70adch;55–84), emergencies without hospitalization (71/52), external face-to-face consultations (83ad/75ch;38–83). Through five questionnaires tested with focus groups, items were nunaced, deleted or added. The questionnaires proved psychometrically robust. The process per questionnaire involved between 106-1,353 patients per validation wave. The cultural adaptation process was formally completed with minor changes. Fully-validated questionnaires are generated and detailed with data from several patient profiles and without specific units biases enabling wide patient experience analyses. The robust validated questionnaires can inspire other hospitals future value-based healthcare.

Keywords

patient reported experience value based healthcare transitional care

Introduction

Electronic Patient Reported Experience Measures are of increasing interest in the participation strategies for hospitals quality improvement. They complement sources like claims, suggestions or complaints documents. Many questionnaires have been developed to generically cover the corresponding assessment in different settings within the hospital care and recently many are appearing to approach the specific aspects from different “care problems” (also named, in value based healthcare, integrated practice units¹). Furthermore, measurable involvement projects play an increasing role, with the need for full person-centered tools like ePREMs to work with.

The most widely used questionnaires in hospital settings such as Picker,² HCAHPS³ or coordinated care P3CEQ,⁴ as well as many relevant specific and local ones,^5,6 play a wide role in achieving to understand patient needs and cover the aforementioned challenges. They’re increasingly used even if, for example, only a small number of them are proposed among ICHOM standard sets.⁷ Accompanying shorter versions of these with final Net Promoter Scores (NPS)⁸ also enables to work with complementary easy to understand overall scores.

Nevertheless, many of these existing questionnaires have problems to overcome. Some are only available for hospitalization assessment leaving an absence of a robust set covering the wide range of hospital fields with a common philosophy. Many do not have a formal published transparent detailed cognitive or psychometric validation process available or where designed as satisfaction and not as experience questionnaires. Some could be considered too large (>15 items) and some miss some dimensions that might be relevant to the patient like pain control or spirituality need.

PREMs should be used in the context of hospital citizen involvement strategies such as ICE-VH,⁹ and dashboards should be freely available to facilitate the daily practiceand integrated decision-making . Achieving this scale-up requires a formally multicentric, randomized, co-created set of questionnaires sharing a common philosophy and available in multiple language – a gap that remains unaddressed in countries like Spain.

The main aim of this study was to share the learnings from developing a set of questionnaires and the formal validation of a subset for their use in value-based-health-care driven tertiary hospitals and show the formal design in clinical and citizen participation environments.

Methods

Development Phase. The First Set of Questionnaires

To establish the questionnaires a first literature search was performed searching for questionnaires preferably designed or implemented in Spanish for different types of care (emergencies;day-hospital;hospitalization…). Wordings and new items were adopted after groups of professionals providing discussion and feedback. These hospital professionals were both experienced with years of practice and part of the directive healthcare committee, as well as from the information systems and citizen attention hospital departments. Items adequacy to be covered through 5-Likert or dichotomic formats discussions took place empowering the decision of this response format to the patients. Also several wordings were adjusted to express an objective experience questionnaire and not a satisfaction questionnaire (we avoided specific and uncertain words: “assess…”;“how did it look like”;“good hands”;“what’s your opinion about”;“How was …?” or using examples as a formula). The questionnaires’ item focus was also to cover aspects that could be always assessable by patients and changeable by the hospital management team. For example a patient can’t assess the “equipment quality”. Finally, the items speech was adapted to “easy-to-read” and “neutral-gender” language criteria (which would not apply in English). As a result, a set of questionnaires were automatically sent to patients via SMS links following each clinical encounter from October 2022 to November 2025. The questionnaires were always finished with a Net Promoter Score (NPS) question. The questionnaires were consecutively developed and the start dates are shared in Table 1.

Table 1.

Set of Questionnaires Answers Distribution in One Tertiary Hospital; Net Promoter Score Values (November 2022- November 2025)

Questionnaire answers by Children/Adults – Setting or type of care	Formal validation	Start date for collection	Net Promoter Score (NPS)	NPS Clinical units range*	NPS Women	Promoter (% 9.10)	Detractor (% 1-6)	Answers (Participation rate %)
1.CH- Home hospital		Apr-24	99		99	99	0	60 (73%)
2.AD- Home hospital		Dec-22	85		83	88	3	344 (15%)
3.CH- Day hospital		Apr.24	83		81	86	5	503 (11%)
4.CH- Neonatology		Feb-24	82		80	88	6	66 (12%)
5.CH-Face-to-face visits*	X(F2FCEXCH)	Jan-24	80	67-91	79	84	4	9,646 (21%)
6-AD-NonRX Tests		Feb-25	78		78	81	3	3,399 (27%)
7-10. CH- Hospitalization [includes 3 add. questionnaires: ICU, REA, EM]		Jul-23	76 (EM 63)		74	82	5	1,110 (14%)
11. AD-Day hospital		Dec-22	77		75	81	4	7,662 (23%)
12.AD- Major ambulatory surgery	X(MASAD)	Dec-22	75	63-79	75	82	5	2,854 (31%)
13.AD-Radio-Diagnoses (same questions used for non RDX tests)		Dec-22	73		73	78,8	3,8	80,326 (25%)
14.CH-Emergencies without in-stay		Apr-24	68		65	77	9	3,694 (18%)
15.AD-Obstetrics		Dec-22	75		-	83	8	773 (14%)
16-19.AD-H Hospitalization Refer [includes 3 add. ICU, REA, EM]	X(HOSPAD)	Nov-22	70(Emergencieswith stay 51	54-84	67	76	6	12,632 (32%)
20.AD- Face-to-face visits*	X (F2FCEXAD)	Nov-22	70	35-91	65	76	6	178,949 (27%)
21.Extractions (blood, urine and other fluids)		Apr-24	55		54	67	12	24.544 (15%)
22.AD- Emergencies without In-stay	X (EMwoAD)	Sep-23	51		49	66	14	31.878 (25%)
23.AD- Telematics visits		Nov-22	22 (29; 2025)	-5-51	26	48	26	33,997 (16%)
TOTAL			64.4			72.9	8.5	400,719 (24%)

AD: Adult, CH: Children, ICU: Intensive Care Unit; REA: Reanimation Unit, EM: Emergencies, H: Hospitalization.

*with N>50 patients.

**includes hospital pharmacy attended patients.

Patient fatigue burden was dealt compacting a reanimation unit, emergencies before hospitalization, and intensive care unit multi-response conditional questionnaires proposing to be sent together to assess the hospitalization process.

A limitation of the current work is that these extensions will still need to be validated. Also each PREM was programmed limited to be sent a maximum of once every 3 months to each patient impacting response rates. Accessibility and trust to care are a relevant limitation in some of the questionnaires considering the Catalan Framework for the patient experience dimensions.¹⁰ That said, in regards to accessibility to care (general waiting list, not the waiting in the hospital hallway), this was claimed in one of the paediatric interviews, but authors think that it is an outstanding topic that should have its own questionnaire. In addition, patients did not request trust questions and considered the number of items fair, in some way delegating the trust in covering adequately the rest of the experience dimensions. Thus, an option could be working on optional additional items on accessibility in the PREM bank, not fatiguing the patient with questions if they can’t be solved or if they are out of scope.

Questions were deemed unsuitable for neonatology and obstetrics purposes and so specific questionnaires were built. Extractions (blood…) and radio-diagnostics settings questionnaires used were established based on already existing historical paper versions. Transition to primary care questionnaires¹¹ and end-of-life questionnaires are in development and were not included in the current manuscript. The questionnaire result dashboards were made available to all the professionals on the hospital’s Intranet. We called this process the PREMs democratization to raise more awareness and data-based decisions and actions.

Five of the aforementioned questionnaires entered into a process of formal validation from July 2024 to December 2025 (highlighted at Table 1).

Cognitive Testing Phase. Focus Groups and Interviews

Five patient focus groups were performed in cooperation with the Patient Experience Institute. The invitation process included patients being diversity inclusive (non-native, medical/surgery diseases, sex, age). This qualitative design was proposed as a must to the quantitative psychometric approach. Interviews were not the first choice because of the burden of work and because we trust that the topic was not sensitive enough allowing the value-added of hearing and getting inspired by others. Patients agreed with the items ordered by care pathway even if this meant to combine Likert and dichotomic outcomes. The whole process re-inforced the objective of developing an experience and not a satisfaction Spanish validated questionnaire wording, though, “assess”, “sufficient”, “correct”, “punctuate”, “satisfactory” or “adequate” were words that had to be avoided. The sentence to introduce the questionnaires was related with reporting regarding the experience, trying to even avoid the word “correct”, given that reporting the experience was the only introduction perspective minimizing methodological critiques.

The focus groups on-line were developed from July-2024 to September-2025. The patients included had been treated in each of the settings in the hospital in the last 2 years. Patients were selected by asking for support from the citizen attention unit to the heads of service and direct key known motivated professionals from the hospital. Patients were presented with each of the questions and were requested to assess qualitatively the clarity, pertinence, relevance and wording suggestions per item as per part of an experience questionnaire. After the items were presented, patients were asked if any important topic was not included and if any overall suggestions were to be included. A Parents focus group (for the pediatric scope) was too short in participation, and so was an e-Delphi to try to gather the cognitive debriefing, so additional 8 interviews in August-2025 were performed to achieve enough minimal robust patient feedback.

The Validation Pyschometrics Phase

The following sections detail the work performed:

- Setting/sites involved in validation waves: the five questionnaires were administered in two consecutive waves separated by 2 weeks in 11 Spanish hospitals summoned via SMS or mails (centre-dependant) between June-July 2025 (adults) and October-2025-January-2026 (paediatrics);

- Sampling/recruitment and sample sizes: each participating centre was instructed to administer questionnaires on a randomly selected day, stratified by care type (hospitalization, emergencies…) encouraging to reach enough patients for a distribution analysis. If additional instructions were required, maximum efforts for a thirty patients minimum per type, and adding days to accomplish it was given as a theoretical value. Centres exceeding 2,000 selected patients had their sample capped at 350, through random selection, to preserve comparability across 11 hospitals of different sizes and regions.

- Analysis performed: (1) Exploratory/Confirmatory Factor analyses were performed (EFA-CFA) validating the model questionnaire: This was carried out to re-inforce the coherence of an underlying questionnaire structure and group the items that measured the experience construct, (2) reliability (internal consistency and test-retest) and validity (internal structure, discriminant validity and convergent validity) considering McDonald values because the data include categorical data, (3) Scoring establishment.

- Missing-data handling: For the EFA, only patients with complete data were included. Given the high rate of 'not applicable’ responses to spirituality and telemedicine items, these were handled separately and proposed as part of the additional item bank rather than the main psychometric model.

- Additional analytic decisions: Non-included “out-of-structure” items are proposed as an additional item bank. The main rationale is that factor analysis should not cause a loss of replicable items on topics of patient interest. Moreover, additional items could be needed for very specific purpose intervention assessments. For example, diversity or trust items can be added. An additional supplementary appendix 9 was added to clarify the minimals required in the framework by April 2026.

The Validation Cultural Adaptation Phase

The five questionnaires in Spanish were culturally adapted as final experience measures into Catalan and English following ISPOR guidelines steps¹² (October-2025 -March-2026). Bilingual healthcare sector professionals (who lived and worked several years in countries where both languages per comparison are official) translated and reached a consensus on discrepancies. The process was repeated with a back-translation and re-consensus. Finally cognitive debriefing interviews (template in the Supplementary Index) with native patients with the cooperation of two totally additional Spanish hospitals as well as from a centre in the United States and from Ireland.

Results

The Development Phase

The initial search identified several Spanish-language adult PREMs covering various hospital settings. Some examples were: emergencies,¹³ day-hospital,¹⁴ hospitalization,¹⁵ major ambulatory surgery,¹⁶ face-to-face visits,¹⁷ telematics visits,¹⁸ home hospital¹⁹ or questionnaire to assess children experience.²⁰ Almost all validations were made in some specific units, unicentric, based on only some profiles of patients, or missed types of validation analyses.

An initial set of 23 questionnaires were available in the hospital before the 5 questionnaires validation process. The average NPS was 64.4 (8.5% of detractors) (Range by setting: 22-99) enabling a full Table 1 The average NPS shown slightly lower consistent values in women as well as higher across adults referring children care. The response rate was 24%

Cognitive Testing Phase

Nine men and nineteen women provided validation to the questions through the focus groups. For example, new items to be added, as well as some to be deleted. When items focus was shared across questionnaire, the proposals from one focus group were extended to others. Wording changes occurred in most items. Almost all the suggestions were covered; a small number weren’t. A reason was the “unavailability to change care”. For example: they proposed to assess the increasing presence/absence of psychological support, which was seen as non-economically manageable by 2025. Finally, some unfolding questions suggestions were not implemented for the sake of the questionnaire length: (1) differentiating the pain management in the emergency admission process, during attention, at discharge, or (2) differentiating video-conference and phone calls when asking whether you would you have liked: face-to-face or virtual visit (Table 2).

Table 2.

Main Questionnaire Improvement Changes From the Focus Groups and Interviews on the Patient Reported Experience Questionnaires (July 2024 -September 2025)

Setting	Patients invited attended the last 2 years per setting*	Number of final participants by gender*	Items deleted or nuanced	Item added	Main additional changes
AD-Face-to-face visits	10 Men;	Focus group:	-The”¿Were you asked if you wanted your medical information shared with anyone other than yourself?” question was not considered relevant in the this context	-An item was added on attending on-time.	-Wording/writing changes without outstanding were proposed in all items
	10 Women	4 Men;		- Three more items suggested: accessibility+ signalling and visit-time adaptation (these were suggested on other questionnaires)
	10 Women	4 Women		-Patients requested to duplicate by type of professionals but this was against patient burden fatigue design policy
AD- Hospitalization	7 Men;	Focus group:	-Availability to listen was seen as already included by an overarching “way to treat” item	-Physical centre accessibility	The aforementioned items were suggested in the focus groups and transerred to the rest of questionnaires
	5 Women	4 Men;		-Within the centre, orientation signals
	5 Women	3 Women		-Contact information availability
AD-Major ambulatory surgery	5 Men;	Focus group:	-An item on “ Whether or to whom information should be given”	-Mobility to arrive and risks information items were suggested but the following item analysis showed that this was already covered	-The “whether to who information should be given” question was converted and kept in other questionnaires as “were you asked whether you wanted the healthcare team to share medical information with anyone other than yourself?”
	8 Women	0 Men;		-Contact information at discharge was requested
	8 Women	5 Women		-Accessibility items added, as well as intimacy item coming from another questionnaire
AD-Emergencies without in-stay	1 Men;	Focus group:	-“How was the discharge information” was asked in 2 item ways	Suggestion to unfold or separate questions in different items: the “way to treat”, by 5 types of professionals (nursers/physicians differentiation was finally applied although this was considered to be hardly ever done as a model).	Wording/writing changes without outstanding were proposed in all items
	6 Women	1 Men;
	6 Women	4 Women
Families of children-Face-to-face visits	4 Men;	Focus group:	Intimacy item was suggested to be wiped out if “(confidentiality)” was not explicitly added as an example concept. Finally, as per confirmed in other questionnaires, “curtains” or “body management” were added as examples with the same purpose of covering details and interpretation	Consider that adolescence might not be the same than younger children	It is better to talk about “appointment” than “visit”.
	11 Women	0 Men;		New item on accessibility to answers to doubts in case of incidents	Differentiate by type of professional
	+ 18 invited	3 Women		Regarding to the clear given information “Upcoming visits”
	+ 18 invited	+7 complementary interviews		Regarding to the clear given information “Upcoming visits”

AD: Adults.

*Both invitations and participation were subject to the hospital’s professionals and patients availability and cooperation . Patients decided for each item if they wanted a dichotomic or a 5-level answer to explain their answer.

The Psychometric Validation Phase

The final sample included in each of the questionnaires and waves a range from 177 to 1,353 patients (Table 3). All five questionnaires demonstrated adequate psychometric coherence (Table 4, Table 5). The face-to-face consultation (F2FCEXAD) and emergency without hospitalization (EmwoH) questionnaires yielded a three-factor structure, enabling dimension-level scoring. The remaining three questionnaires were best represented by a single factor, supporting a global score but limiting sub-dimension analyses. Internal consistency was excellent across the five questionnaires and both validation waves. The number of items referred as additional item bank (relevant-for-the-patient items but not coherent as a construct with others psychometrically) were 1-6 per questionnaire. The internal consistency was always excellent with and without combining waves(re-test). The total-item correlations were generally high for all the items. The questionnaires in general did show a good ceiling/floor effect profile as per scoring analysis, and the construct validity showed clear good results in the discrimination among known groups and the convergence with the “NPS question”. The representativeness of the sample including the scoring depending on sex, age and type of care is presented. Some items, specially those with high “non-applicable” burden, resulted into an additional item bank set (Supplementary Index Tables).

Table 3.

Final Patients Completing PREM in the Eleven Centres Psychometric Validation

	C1	C2	C3	C4	C5	C6	C7	C8	C9	C10	C11	Totals
>700 beds hospitals	X			X	X						X	Totals
Wave 1
Adults-Face-to-face visits (F2FCEXAD)	457	14	5	np	350*	*	35	59	277	51	104	1,353
Adults-Hospitalization (HospAD)	73	21	1	np	93	148	30	46	158	100	35	705
Adults-MAS (MASAD)	np	20	2	225	45	48	38	58	53	104	71	664
Adults-Emergency without admit (EMwoAD)	np	17	np	np	194	89	27	41	32	45	65	510
Paediatrics-CEX (F2FCEXPED)	np	24	22	np	np	np	np	np	np	25	106	177
Wave 2 (15 days later)
Adults-Face-to-face visits (F2FCEXAD)	79	20	4	np	350*	*	64	32	221	17	100	887
Adults-Hospitalization (HospAD)	9	20	0	np	70	171	51	37	143	36	28	676
Adults-MAS (MASAD)	np	14	0	256	32	44	1	50	2	np	59	454
Adults-Emergency without admit (EMwoAD)	1	11	np	np	141	53	43	21	27	13	61	371
Paediatrics-CEX (F2FCEXPED)	np	np	21	np	np	np	np	np	np	np	85	106

Acronyms. CEX: Face-to-Face External consultations; MAS: Major Ambulatory Surgery; C1 to C11 are from 4 regions in Spain: Andalucia, Asturias, Catalonia and Madrid.

np: Did not Participate.

*There was confusion in the coding and call for participation between 2 centers with similar names.and more than 2,000 patients were involved so a randomization was involved to avoid an overweight This table presents the participation once missings were managed (patients included in the factorial analyses).

Table 4.

Psychometric Results of the 5 Patient Reported Experience Mesures Validation With Patients in the Eleven Participating Centres (1/2)

Questionnaire	Exploratory and Confirmatory Factor Analysis [initial items] → Final number of factors and for scoring without additional items bank	Internal structure (CFA)	Reliability					Validity
Adults-CEX	[16 items] → 3 Factor and 14 items	>Fit indices: RMSEA=0.050 (IC90%: 0.043–0.057), CFI=0.995, TLI=0.993 and SRMR=0.024	>Stability across waves					>Known-groups validity (see Supplementary Index Tables)
	Structure validated by standarised charges:	>Standardized loadings: majority λ>0.70 (2 items < 0.70).			Average (SD)	ω McDonald		>Convergent validity: NPS recommendation vs Questionnaire Score Spearman’s ρ = 0.65 (p<0.001) (see Table B1/B2 Supplementary Index)
	F1:Environment and Information to arrive F2:Discharge/disease/treatment information		Wave	N	Global punctuation	Global scale
	F3:Contact info + Organize + Participate + pain controL + Treat		1	1,353	3.87 (0.69)	0.967
	2 Items apart due to the factor analysis scores:		2	887	3.88 (0.70)	0.969
	- “wanting the consultation as face-to-face or telematics” and		>Internal Consistency
	- “Did the method used to inform of the appointment work well? For example, letter, text message, or app.”			Overall	Factor 1	Factor 2	Factor 3
			N items	14	3	7	4
			ω McDonald	0.967	0.858	0.956	0.826
Adults-Hospital	[17 items]-→ 3 Factor and 14 items	>Fit indices: RMSEA=0.049 (IC90%: 0.039–0.058), CFI=0.996, TLI=0.995 and SRMR=0.041	>Stability across waves					>Known-groups validity (see Supplementary Index Tables)
	F1: Accessibility (P6-P8)	>Standardized loadings: majority λ>0.70 (4 items < 0.70).			Average (SD)	ω McDonald		>Convergent validity: NPS recommendation vs Questionnaire Score Spearman’s ρ = 0.69 (p<0.001) (see Table B1/B2 Supplementary Index)
	F2: Clinical information and discharge (P10, P11, P20)		Wave	N	Global punctuation	Global scale		> Criterion validity (concurrent): Questionnaire Score vs Picker positive score, Spearman’s ρ = 0.38 (p=0.025)*
	F3: Person-centred care (P12-P18, P21)		1	705	3.63 (0.61)	0.956
	3 items apart due to the factor analysis scores:		2	565	3.66 (0.58)	0.961
	- Spirituality question and sub-question (2 items)		>Internal Consistency
	-“Were tyou asked if someone should be informed appart from you?”			Overall	Factor 1	Factor 2	Factor 3
			N items	14	3	3	8
			ω McDonald	0.957	0.791	0.936	0.908
Adults-MAS	[15 items ]-→ 1 Factor and 12 items	>Fit indices: RMSEA=0.048 (IC90%: 0.034–0.063), CFI=0.996, TLI=0.994 and SRMR=0.032	>Stability across waves					>Known-groups validity (see Supplementary Index Tables)
	Al factors are coherent together as one factor.	>Standardized loadings: majority λ>0.70 (2 items < 0.70).			Average (SD)	ω McDonald		>Convergent validity: NPS recommendation vs Questionnaire Score Spearman’s ρ = 0.61 (p<0.001) (see Table B1/B2 Supplementary Index)
	3 item apart due to the factor analysis scores:		Wave	N	Global punctuation	Global scale
	-“Were tyou asked if someone should be informed appart from you?”		1	664	3.86 (0.58)	0.936
	- Pain item		2	454	3.87 (0.64)	0.953
	- Discharge item		>Internal Consistency
				Overall/Factor 1
			N items	12
			ω McDonald	0.943

CEX: Face-to-Face External consultations; F1: Factor 1, F2: Factor 2; F3: Factor 3; FG: Focus Group; MAS: Major Ambulatory Surgery; NPS: Net Promoter Score.

*Number of items as per significance given Geomin Rotated Loadings **only performed at one of the hospitals.***logically, no dimension was priviliged in the analysis.

**The correlation was significant and the absolute value was moderate, in a context where both questionnaires are conceptually different (Picker was designed as a continuous psychometric scale).

Table 5.

Psychometric results of the 5 Patient Reported Experience Mesures validation with patients in the eleven participating centres (2/2)

Questionnaire	Exploratory and Confirmatory Factor analysis [initial items] → Final number of factors and for scoring without additional items bank	Internal structure (CFA)	Reliability				Validity
Adults-Emergency	[14 items ]-→ 1 Factor and 9 items	>Fit indices: RMSEA=0.056 (IC90%: 0.035–0.077), CFI=0.997 TLI=0.996 and SRMR=0.017	>Stability across waves				>Known-groups validity (see Supplementary Index Tables)
	Al factors are coherent together as one factor.	>Standardized loadings: majority λ>0.70 (1 item < 0.70).			Average (SD)	ω McDonald	>Convergent validity: NPS recommendation vs Questionnaire Score Spearman’s ρ = 0.77 (p<0.001) (see Table B1/B2 Supplementary Index)
	5 items accounted apart from the overall construct:		Wave	N	Global punctuation	Global scale
	- Spirituality questions (2 items)		1	510	3.60 (1.00)	0.960
	- Were you asked if someone should be informed appart from you?”		2	371	3.62 (0.98)	0.960
	- Accompanied item		>Internal Consistency
	-Pain item			Overall/Factor 1
			N items	9
			ω McDonald	0,933
Paediatrics-CEX	[16 items ]-→ 1 Factor and 10 items	>Fit indices: RMSEA=0.089 (IC90%: 0.066–0.113), CFI=987, TLI=982 and SRMR=0.038	>Stability across waves				>Known-groups validity (see Supplementary Index Tables)
	Al factors are coherent together as one factor even if the exploratory factor analysis proposed 2 factors.	>Standardized loadings: majority λ>0.70 (2 items < 0.70			Average (SD)	ω McDonald	>Convergent validity: NPS recommendation vs Questionnaire Score Spearman’s ρ = 0.61 (p<0.001) (see Table B1/B2 Supplementary Index)
	6 items were set apart mainly due to that adolescence and symptoms items had lots of “does not applies”, the type of connectivity channel was answered by everyone as a “yes” (information channel worked), and the item about providing contact information at the end of the visit assessed compliance with a center process.		Wave	N	Global punctuation	Global scale
			1	177	4.68 (0.44)	0.952
			2	106	4.73 (0.46)	0.969
			>Internal Consistency
				Overall/Factor 1
			N items	11
			ω McDonald	0.958

The Validation Cultural Adaptations Phase

The main changes both to Catalan and English were word changes (not sentences) due to linguistic nuances. We could group them in conceptual and semantic equivalence to the clinical context (“body handling” vs “patient handling and transfers”), syntactic and stylistic refinement (“dispuso” vs “pudo disponer”), lexical standardization (“schedule” vs timetable”) or grammatical precision (“Alta Alta” vs “En el momento del alta”). This was similar in English where formality instead of jargon, and ease of comprehension provoked debates of one word against another in the consensus phases between bilingual care professionals as detailed in the Supplementary Index table 6. The feedback from the patients was more scarce even finding final minor ortographic errors, typographical inconsistencies, literal errors or similar but always little inconsistencies. Even if small, all the experience -not satisfaction items had minimal changes respecting the learnings from all the steps of the overall validation process.

In summary, five questionnaires were formally validated and are ready for implementation: F2FCEXAD, HOSPAD, CMAD, EmwoAD, F2FCEXPED. Eighteen additional questionnaires sharing the same wording philoshophy are available upon request. Any language use, and adaptations into new languages can but should be done requesting the process and with free centralised copyright given by the authors.

Discussion

The overall NPS of the first 400,079 hospital answers from the total set of questionnaires initially developed was 64.4 which can serve as a basis when aggregating all type of hospital services and even to establish yearly goal management objectives. Nevertheless, the range of values among settings (types of care) is high, proving that, assessing a whole center with a unique value, is quite simplistic. Values ranged from above 80s in children care, or 70s in hospitalization care, to 60s in emergency settings and 20s in virtual visits, which remains an expectable challenge. Given the response rate above 20% - higher than others reported with SMS systems²¹ - and number of available answers, the NPS values could serve as a reference for many tertiary public hospitals. The high response rate can be related with the inclusion of “dealing with too many questions” strategies. The high participation on the whole set of questionnaires endorses the number of items, principles and wording policy of these questionnaires. The wording of items are similar in the full set of questionnaires, thus, given the transparent explained cognitive validation process, it re-inforces the plausible validity of the rest of questionnaires that were not validated by the time of this manuscript and that are available upon request in Spanish and Catalan. Finally, as items are presented following the patient pathway, this should facilitate to process to remember and expose experience logics.

The transparent process of the focus groups is one of the strengths of the cognitive testing part of the validation process. Qualitative approaches to listen directly to patients needs aren’t so frequent in PREMs questionnaire validation processes. The whole set of items after this exercise cover coherently most of the main PREMs dimensions established by different frameworks²² (treat, information, accessibility, environment…). The items include variability of cross-cultural items from deeper information need analysis to shared-decision-making. Formally, PREMs used in hospitals should have a transparent validation of such a process.

The transparent and psychometric multi-sized-region-centric experience-oriented validation is quite another study strength. Even if factor analysis is questioned in the literature as a tool for validating PREMs,^23,24 authors trust that it has a value not to be deleted as an additional analysis for covering the building including content consistency. The addition of a proposal for a validated additional item bank also both compensates the dangers of a lost of items and ensures, keeping scores calculation available, that any assessment need can be added in terms of benchmarking making easy to empower assessable improvement actions.

As mentioned, some settings had already separated options of questionnaires available in Spain. To tour knowledge most existing tools lack validation for online use or high participation rates. Furthermore, few have undergone transparent validation with randomized patients across diverse units, including items on facts that can’t be changed by the hospital or understood by patients, or not proven adaptive to patient fatigue, amongst others. Furthemore, as the main limitation the specific test-retest in our validation was not totally paired (same invited but could not be exactly same respondents).

As explained, discussions occurred during the psychometric and cognitive validation phase to avoid proposing satisfaction instead of experience questions. We decided to leave numerical 1 to 5 with “-” and “+” on non-dichotomic items, instead of providing category labels reducing ceiling effects. This could be seen as following satisfaction-style questionnaires; after working wordings and focus groups, it was seen as the easier comprehension. Neutral intermediate answer was not over-reported and it makes dashboarding-colors easier. Exceptionally, a satisfaction-oriented question was left, for example, because of its’ power for experience decision making: the item asking if you would have liked a virtual visit instead of a face-to-face. Finally, the use of a Likert format and an overall introducing statement inviting to report regarding the “patient experience” was proposed after big debates considering that is is not fully reprehensible as explictely asking for an “agreement” but also because patients participation phases re-approved they felt comfortable with not using specific words (of course each item results can’t be analyzed by quantitative averages but as frequencies analyses).

The current set of generic PREMs can be used in usual practice with frameworks such like MATRICS²⁵ and as mant specific disease PREMs items overlap with that of generics, centralized strategies such as having dashboards with a common philosophy set of questionnaires enabling selection by ICD-10 diagnoses, seem initially more profitable. Finally, having complete response PREM Likert matrixes will reduce missing answers against questionnaires that split answers depending on the patient consecutive answers or Computer-Assisted Questionnaires.

The set of valid PREMs strategy covers and is aligned with that of value-based healthcare and participation principles given.²⁶ The main future lines are validating other questionnaires from the basal set, expanding the cultural adaptations, or setting additional questionnaires like an <<end-of-life PREM>> or <<PREMs answered directly by adolescents/children>>.

Conclusion

Hospital care quality must be improved every day as well as the tools to assess this improvement including the patient experience perspective. The results demonstrate the feasibility to develop and share a set of several languages robust and transparent based framework generic validated PREMs across a big spectrum of hospital settings including paediatrics and full-patient involvement, and not limited to a number of services. This shouldinspire integrated Value-Based-Healthcare strategies and taking managerial decision-making based on patients opinions.

Supplemental Material

Supplemental Material - Multicentric Design and Validation of a Set of Democratized Hospital Patient Reported Experience Measures in Spain

Supplemental Material for Multicentric Design and Validation of a Set of Democratized Hospital Patient Reported Experience Measures in Spain by Emmanuel Gimenez, Marta Aguayo, Laura Muñoz, Maria José Rodriguez, Carlos Bezos, Marina Martínez, Montserrat Martínez1, Albert Esplugas, Ana Marti in Journal of Patient Experience

Footnotes

Acknowledgements

To all the bilingual people having worked/lived/been attended in the health sector during years/native in both Spain (Catalonia) and English speaking countries including Canada, United States, Ireland and United Kingdom. The authors thank the Institut Català de la Salut as an umbrella institution and all the participating hospitals for the unpaid collaboration. Hospital Universitario Virgen del Rocío de Sevilla (Antonio Cervera), Hospital Universitario Central de Asturias (Bernabe Fernandez), Hospital Universitario Príncipe de Asturias (Madrid) (Marta Macías), Institut Català de la Salut (Margarita Garcia), Hospital Universitari Germans Trias (Irene Jimenez), Hospital Universitari dr.Josep Trueta de Girona (Elisabet Jordà, Pere Rimbau), Hospital Verge de la Cinta de Tortosa (Soledad Lucas), Hospital de Viladecans (Jose Luis Moreno), Hospital Universitari de Bellvitge (Irene Feliu, Silvia Millat), Hospital Universitari Arnau de Vilanova de Lleida (Maria Bonjorn), Hospital Joan XXIII de Tarragona (Neus Camañes), Hospital Universitari Vall d’Hebron Hospital (Melissa Bradbury, Emma Pastó, Anna Oliver-Seguí and Laia Humbert), Benard Health & Science Consulting (Angèle Benard), Marcos Seneca Garcia (Parc Taulí de Sabadell), Institut Gutman (Elena Hernandez i Montserrat Bernabeu); Expertise Valeur en Santé; Coalition priorité cancer au Québec, Canada (Eva Villalba), Trinity St James’s Cancer Institute (Grainne Smith), Juan Diego Gonzalez (West Alton Gloor Medical Clinic, Texas) and Biocat (Nuria Castany). The authors also thank David Romero and Francisco Cidoncha (Information Systems Vall d'Hebron Hospital). Additionally to RateNow for providing the technological platform used to administer the survey waves to patients in the participating hospitals. This system ensured appropriate information to patients and enabled detailed segmentation of responses by hospital, questionnaire and survey wave.

ORCID iD

Emmanuel Gimenez

Ethical Considerations

Study presented and approved by the Drug Research Ethics Committee and Medication Projects Committee of Vall d’Hebron University Hospital on November 26, 2021. (PR(AG)584/2023). All methods were carried out in accordance with relevant guidelines and regulations or Declaration of Helsinki.

Author Contributions

The manuscript has been submitted with the consent of the authors who made significant contribution to the concept, design, acquisition, analysis or interpretation of data, as well as provided important intellectual content, approved the final version and agreed to be accountable for the work.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: LM (statistician) works for Databioexp and received a non-conditional funding from RateNow to do the psychometric analysis of the five Patient Reported Experience Measures. RateNow is a company specialised in measuring patient experience through digital surveys delivered to patients’ mobile phones, enabling real-time analysis in a business-intelligence platform powered by artificial intelligence. RateNow technology was used in this study to perform the psychometric validation, sending and building SMS invitations with the questionnaires to patients from all participating hospitals.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Additional Comments

Each care centre questionnaires uses but must be requested as per scientific property, per traceability of use, and possible improvement versions to prems@vallhebron.cat, and in copy to info@iexp.es iexp@iexp.es and info@ratenow.es. Any publication using the questionnaires should include this manuscript as a reference and refer to a webpage from one of the institutions where these questionnaires are displayed. The scientific responsible duty to report to the authors includes using one of the additional items for specific replicable and intent to evaluation use and for the authors versions improvement. No economic charge is related with the intellectual property. The non-validated (the other 18) questionnaires by june 2026 can also be delivered upon reasonable request to prems@vallhebron.cat.

Supplemental Material

Supplemental material for this article is available online.

References

Muñoz

Barba

Abadias

, et al. Hospital transformation: From departmental specialties to integrated knowledge areas. An analysis of health outcomes and the experience of patients and professionals. Med Clin (Barc). 2025;165(5):107174.

Jenkinson

Coulter

Bruster

. The Picker Patient Experience Questionnaire: development and validation using data from in-patient surveys in five countries Int J Qual Health Care. Int J Qual Health Care. 2002;14(5):353-358.

Weinick

Becker

Parast

, et al. Emergency Department Patient Experience of Care Survey: Development and Field Test. Rand Health Q. 2014;4(3):5.

Rijken

Menting

, et al. Assessing the experience of person-centred coordinated care of people with chronic conditions in the Netherlands: Validation of the Dutch P3CEQ. Health Expect. 2022;25(3):1069-1080.

Ferrè

De Rosis

Murante

, et al. Systematic and continuous collection of patient-reported outcomes and experience in women with cancer undergoing mastectomy and immediate breast reconstruction: a study protocol for the Tuscany Region (Italy). BMJ Open. 2021;11(1):e042235.

Fernandez

Fond

Yves

, et al. Measuring the Patient Experience of Mental Health Care: A Systematic and Critical Review of Patient-Reported Experience Measures. Patient Prefer Adherence. 2020;14:2147-2161.

Internal Consortium for Health Outcomes Measurement (ICHOM) Standard Sets . Pregnancy and ChildBirth. Available at: https://www.ichom.org/patient-centered-outcome-measure/pregnancy-and-childbirth/

Adams

Walpola

Schembri

Harrison

. The ultimate question? Evaluating the use of Net Promoter Score in healthcare: A systematic review. Health Expect. 2022;25(5):2328-2339.

Marti

Grau

Gimenez

, et al. J Healthcare Qual ResA practical model to implement the patient participation in tertiary hospitals (ICE model). J Healthc Qual Res. 2024:S2603-S6479.

10.

Unitat d’Avaluació d’Experiència de Pacient, Gerència de Gestió Ciutadana . Àrea de C. iutadania, Innovació i Usuari, Servei Català de la Salut. In: Marc de l’experiència de pacient al sistema de salut de Catalunya. Barcelona: Departament de Salut; 2025. Available at: https://scientiasalut.gencat.cat/bitstream/handle/11351/12539/marc-experiencia-pacient-sistema-salut-catalunya-2025.pdf?sequence=4&isAllowed=y

11.

Aller

Vargas

Coderch

, et al. Development and testing of indicators to measure coordination of clinical information and management across levels of care. BMC Health Serv Res. 2015;15:323.

12.

Wild

Grove

Martin

, et al. Principles of Good Practice for the Translation and Cultural Adaptation Process for Patient-Reported Outcomes (PRO) Measures: Report of the ISPOR Task Force for Translation and Cultural Adaptation. Value in Health. 2005;8(2):94-104.

13.

Bull

Crilly

Latimer

Gillespie

. Establishing the content validity of a new emergency department patient-reported experience measure (ED PREM): a Delphi study. BMC Emerg Med. 2022;22(1):65.

14.

Mula-Domínguez

Rivera-Sequeiros

. Valoración de la satisfacción del paciente en el hospital de día onco-hematológico del Hospital Universitario Virgen Macarena. SANUM. Revista científico-sanitaria. 2024;8(4):28. Available at: https://revistacientificasanum.com//wp-content/uploads/vol8n4/vol8n4-articulos-pdf/sanum_v8_n4_a2.pdf

15.

Bertran

Viñaras

Salamero

, et al. Spanish and Catalan translation, cultural adaptation and validation of the Picker Patient Experience Questionnaire-15. J Healthc Qual Res. 2018;33(1):10-17.

16.

García

Pancorbo

Rpdríguez

, et al. The development and validation of a questionnaire to evaluate the user satisfaction of major outpatient surgery. Enf Clin. 2021;11(4):146-154.

17.

Subdirección General de Calidad Asistencial . Seguridad y Evaluación. Cuestionario EMCA Calidad Percibida. Consultas Externas; 2024. Available at: https://www.murciasalud.es/documents/197507/5357937/Cuestionario+EMCA+Consultas+Externas.pdf/20c07bff-9e15-e7b0-6b19-ae9693128968?version=1.1&t=1708435176432

18.

Agency for Health Research and Quality (AHRQ) . TeleHealth Satisfaction Questionnaire (TSQ). Available at: https://digital.ahrq.gov/sites/default/files/docs/survey/telehealthsatisfactionquestionnaire_comp.pdf

19.

Osakidetza . Informe de la encuesta de satisfacción de hospitalización a domicilio. 2014. Available at: https://www.euskadi.eus/contenidos/informacion/obid_gestion/eu_obid/adjuntos/hospitalizacion_dom.pdf

20.

Martínez-Roda

Ruiz-Romero

Torres-Ruiz

García-Garmendia

. Evaluación de la experiencia de niños y padres en un servicio de Pediatría. Journal of Healthcare Quality Research. 2021;36(4):217-224.

21.

Vainieri

De Rosis

PREM Observatory . Systematic digital use of PREM for quality enhancement in Italian Hospitals. Available at: https://www.networkjci.it/wp-content/uploads/2022/11/10.11.22_VainieriDerosisPREM_Ieo.pdf

22.

Katz

. EIT Health, Implementing Value-Based Health Care in Europe: Handbook for Pioneers, 2020.

23.

Picker . Validation and realiability of Picker surveys, 2024. Available at: https://picker.org/wp-content/uploads/2024/03/Validation-Reliability-of-Picker-Surveys-WEB.pdf

24.

Sizmur

Graham

Bos

. Psychometric evaluation of patient-reported experience measures: is it valid? Int J Qual Health Care. 2020;32(3):219-220.

25.

Gimenez

Watson

Cossio-Gil

. Decoding patient-reported measures (PRMs) use in clinical practice: How and for what? The MATRICS framework. J Healthc Qual Res. 2023:S2603-S6479.

26.

European Commission . Defininf Value in ‘Value Based Healthcare. Available at: https://health.ec.europa.eu/system/files/2019-11/2019_defining-value-vbhc_factsheet_en_0.pdf

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.04 MB