Testing for Differential Item Functioning within the EQ-5D

Method

Data for this study were derived from the ongoing Efficacy of Nitric Oxide in Stroke (ENOS) trial, investigating the use of glyceryl trinitrate therapy following acute stroke. ⁸ ENOS has recruited patients in more than 150 hospitals in 16 different countries. At around 90 days following the stroke event, trial personnel record a range of clinical data for each patient. At the same time, the EQ-5D questionnaire is completed either by the patients themselves or by their carers (proxies), depending on the former’s capabilities.

Each patient record included scores for the modified Rankin Scale (mRS) and for the Barthel Index (BI). These instruments have been validated extensively as clinically relevant measures in cerebrovascular disease.^9,10 The mRS is a 6-point ordinal classification of increasing disability and dependence, ranging from “no symptoms” (mRS0) through “severe” (mRS5), the latter implying the patient is bedridden, is incontinent, and requires constant attention. In the trial context, a further class (mRS6) is assigned to “dead,” although no mRS6 records were included in the present analysis. The BI assesses patient performance in 10 activities of daily living related to self-care and mobility. Scores vary continuously between 0 and 100, a higher score implying greater independence in physical functioning. All patients in the ENOS trial also completed the Short Zung Depression Scale, which possesses a score range of 25 to 100. A higher score indicates a more severe problem with mood, although Zung scores are known to be positively associated with extent of physical illness also. ¹¹

In a recent study of stroke patients, ¹² EQ-5D index scores were mapped from observed mRS scores using regression analysis. We replicated this mapping using the ENOS data, although data availability offered the possibility of a superior model specification. Compared with the Rankin classification, the Barthel Index assesses particular, as opposed to generalized, aspects of physical limitation. BI scores have been used previously as a predictor of EQ-5D index scores in stroke studies, either alone^13,14 or alongside Rankin scores. ¹⁵ Arguing that both the BI and the mRS neglected psychological aspects of quality of life in stroke patients, a German study ¹⁶ added a specific depression measure as a predictor of index scores. In line with these approaches, we added the BI and Zung Depression variables to our mapping model.

In the earlier study, ¹² all patients were located in and around Oxford, United Kingdom, and no proxy responses were included. We recognized that the inclusion of the relevant dummy variables in the mapping procedure for the ENOS data could establish the existence, or otherwise, of DIF with respect to region-specific and proxy responses. As the assignment of index scores to health states follows standard EQ-5D algorithms, any differences in mean scores observed as a result of including the dummy variables would be attributable to differences in the distributions of reported dimension levels. In principle, the regression models could be estimated using any of the national value sets of index scores, although we chose to employ the UK set. The ENOS trial is being coordinated from the United Kingdom, and the majority of patients have been recruited by UK sites. The use of the UK set, moreover, facilitates a direct comparison with the previous analysis. ¹² The index scores in the UK value set range from 1 (vector 11111, no health problems in any domain) to −0.59 (vector 33333, severe problems in all health domains).

The EQ-5D index score is determined by severity levels on 5 separate dimensions and is, in effect, a summary measure. A particular score is not unique to one configuration of health problems only and might be associated with several different health state vectors. For example, in the UK value set, vectors 13232, 23231, and 32322 all translate to an index score of −0.06 at 2 decimal places, whereas both 12321 and 13212 translate to 0.33. The mapping analysis, therefore, might succeed in signaling the presence of DIF in the instrument, but it would be unable to identify which of the dimensions had constituted the source. To illuminate this question, we constructed logistic regression models, as employed routinely in assessing DIF. ¹⁷ These predicted the likelihood of a particular problem severity level being chosen on each of the 5 dimensions, given the latent characteristics as measured by the mRS, BI, and Zung scores, plus the proxy and region dummy variables. All analyses were conducted using SPSS 16.0 (SPSS, Inc., an IBM Company, Chicago, IL).

Ideally, each country contributing records would have been assigned a dummy variable, but this was precluded by the skewed distribution of responses. Most records in our sample (58.3%) originated from the United Kingdom, and the numbers of patients recruited in each of the remaining countries were relatively small. We therefore divided the non-UK data into 2 further regions. The continent of Asia provided 25.1% of the responses, including those from sites in Singapore and Malaysia (10.5% of all responses), India and Sri Lanka (7.0%), and China, including Hong Kong (6.9%). The final region was a residual, the “rest of world.” This provided the remaining 16.5% of responses, with most coming from Poland (5.8%), Romania (4.5%), and Australia/New Zealand (3.0%).

Results

The sample comprised records for 1462 patients. Of these, 24.4% were in the least severe mRS categories (mRS0 and mRS1), 40.7% were slightly or moderately disabled (mRS2 and mRS3), and 34.8% were severely disabled, requiring assistance in attending to bodily needs (mRS4 and mRS5). Patients’ Rankin scores were closely associated with sex and age. Proportionately fewer females than males were assessed as occupying the 2 least severe categories (18.2% v. 28.8%, respectively), and more females than males were classified as being severely disabled (42.7% v. 29.2%, χ² = 49.6, P < 0.01). Patients classified as mRS5 were, on average, 6.2, 8.4, 11.0, and 13.2 years older than those classified as mRS4, mRS3, mRS2, and mRS1, respectively (one-way analysis of variance, all differences significant at 5%). The mean (SD) BI and Zung Depression scores for the sample were 80.3 (24.9) and 51.0 (16.6).

The mean (SD) EQ-5D index score for the full sample was 0.53 (0.37). A skew in the data was evident, the median being 0.64 (interquartile range [IQR], 0.26−0.81). The mean (SD) VAS score was 65.63 (22.25), the median being 70 (IQR, 50–80). Table 1 displays the distribution of EQ-5D descriptions. These data suggest that, at 3 months, limitations on usual activities were being experienced by around three-quarters of the sample, with around two-fifths reporting no mobility or self-care problems. Severe problems with pain and discomfort and with anxiety and depression were relatively rare. In total, 103 different health state vectors were represented in the data set, including both of the extreme states. Only 11.0% of the sample recorded no health problems (level 1) on all 5 dimensions, whereas 35.8% of records each included at least one level 3 response.

OPEN IN VIEWER

Table 1

Distribution of EQ-5D Responses, by Dimension (%)

	EQ-5D Dimension
EQ-5D Level	Mobility	Self-Care	Usual Activities	Pain and Discomfort	Anxiety and Depression
1	37.8	43.9	23.2	54.4	53.5
2	49.5	39.2	44.9	41.7	41.0
3	12.8	16.9	31.9	4.0	5.5

EQ-5D questionnaires were completed by carers or proxies, rather than by the patients themselves, in 436 cases (29.8%). Understandably, the proportion of EQ-5D descriptions recorded by proxy was strongly associated with mRS score (χ² = 310.9, P < 0.01). Although 13.5% of records for all patients below mRS3 (mRS0 to mRS2) were completed by proxies, 40.1% and 87.1% of mRS4 and mRS5 patient records, respectively, were proxy reported. The mean BI score of patients reporting by proxy was significantly lower than that of patients recording themselves (50.0 v. 83.6, t = 21.9, P < 0.01), and the mean Zung Depression score was significantly higher (56.6 vs 49.7, t = 5.9, P < 0.01).

Model 1 in Table 2 uses ENOS data to replicate the structure of the model constructed by the previous researchers using Oxford data. ¹² The modified Rankin Scale is an ordinal classification, and scores are represented as binary dummy variables. As with the Oxford model, mRS1 is the reference category in model 1, and the constant term therefore corresponds to the mean index score for mRS1. The beta coefficient for each of the mRS scores in model 1 is significantly different from zero. Given the coefficients in model 1, the mean EQ-5D index scores for mRS states 0 through 4 amount to 0.93, 0.85, 0.71, 0.55, and 0.28, respectively. In all cases, these lie within 0.03 of the index scores that had been estimated in the earlier Oxford study. The ENOS and Oxford results differ in 2 respects, however. First, the adjusted coefficient of determination for the Oxford model was, at 0.45, considerably lower than the 0.66 reported for model 1. Second, the mean index score for mRS5 using model 1 amounts to −0.15, considerably lower than the −0.06 reported for mRS5 for Oxford. It should be noted that the authors of the Oxford study reported underprediction of the poorest health states, which they considered explicable in part by the small number of records from severely disabled patients in their own sample (7.9% assessed at mRS4 or mRS5 v. 34.8% of ENOS). By contrast, the ENOS predictions were more accurate, in the sense that the 6 mean index scores by mRS category for the ENOS sample corresponded to the mean scores estimated using model 1 to within 2 decimal places.

OPEN IN VIEWER

Table 2

Linear Regressions Predicting EQ-5D Index and Visual Analog Scale (VAS) Scores

	Model 1			Model 2			Model 3			VAS
	β	SE	P Value	β	SE	P Value	β	SE	P Value	β	SE	P Value
Constant	0.85	0.01	0.00	0.39	0.05	0.00	0.42	0.05	0.00	89.98	4.80	0.00
Modified Rankin Scale score (dummy)
0	0.08	0.03	0.00	0.05	0.02	0.03	0.05	0.02	0.03	3.57	2.05	0.08
2	−0.14	0.02	0.00	−0.08	0.02	0.00	−0.08	0.02	0.00	−3.58	1.47	0.02
3	−0.30	0.02	0.00	−0.13	0.02	0.00	−0.14	0.02	0.00	−7.75	1.65	0.00
4	−0.57	0.02	0.00	−0.16	0.03	0.00	−0.18	0.03	0.00	−5.73	2.39	0.02
5	−1.00	0.02	0.00	−0.35	0.05	0.00	−0.37	0.05	0.00	−12.46	4.66	0.01
Barthel Index				0.01	0.00	0.00	0.01	0.00	0.00	0.14	0.04	0.00
Zung Depression				−0.01	0.00	0.00	−0.01	0.00	0.00	−0.58	0.04	0.00
Proxy dummy							0.00	0.02	0.92	2.88	1.51	0.06
Rest-of-world dummy							−0.04	0.02	0.01	−6.48	1.36	0.00
Asia dummy							−0.04	0.01	0.01	−2.75	1.22	0.02
Adjusted R2			0.66			0.70			0.70			0.42

SE, standard error.

Model 2 in Table 2 adds the BI and Zung Depression scores to the model 1 structure. The coefficients for both variables are significant and of the expected sign. The higher coefficient of determination and the continued significance of the mRS coefficients support the view that the new variables augment rather replace the mRS classifications. Model 3 adds the dummy variables for region and proxy recording to model 2. This model is suggestive of DIF with respect to region but not to proxy reporting. Non-UK patients evidently offered EQ-5D descriptions that translated to lower index scores at any given mRS score level. Table 2 also displays a replication of the model 3 formulation using the VAS score, rather than index score, as the dependent variable. The VAS model paralleled the index score model, in terms of signs on coefficients and significance, although the coefficient for the proxy dummy came closer to statistical significance.

The extent to which differences in distributions of descriptors correspond to differences in index scores depends on the algorithms used to construct the value set. In an international trial setting, the choice of value set is essentially arbitrary, although only a limited number of national value sets have been verified to date. By way of experiment, we replicated the Table 2 regression models using value sets from the United States ¹⁸ and Japan. ¹⁹ These yielded essentially similar results (not shown)—namely, insignificant coefficients for the proxy variable and negative coefficients for the regional dummies (i.e., lower mean scores for the non-UK regions). The adjusted coefficients of determination were marginally higher, however. In fact, these results are unsurprising, as the full sets of the 243 EQ-5D index scores valued for each country are highly correlated (r > 0.95 for each pair of sets). This having been said, the range of national value sets is steadily increasing, and differential responsiveness on the part of some European sets has already been identified. ²⁰

Table 3 presents binary logistic regression models estimated using backward stepwise elimination. These predict whether either moderate or severe health problems (level 2 or 3), as opposed to no problems (level 1), were reported for each of the EQ-5D’s dimensions. The independent variables were those for the Table 2 models, although the mRS4 and mRS5 variables were excluded because no patients with the highest mRS scores gave level 1 responses. The odds ratios (ORs) are easily derived in a logistic regression because the beta coefficient equals the log of the OR. Two indications of model quality are provided. Nagelkerke’s generalized coefficient of determination ²¹ is a measure of the effect size of the model, ²² and the area under the receiver operating characteristic curve assesses capacity to discriminate. The area under the curve (AUC) is equivalent to the probability that the model will rank a randomly chosen positive instance higher than a randomly chosen negative instance, assuming positive indeed ranks higher than negative. The statistic takes the range 1 (certainty) to 0.5 (i.e., equivalent to random ranking). ²³

OPEN IN VIEWER

Table 3

Stepwise Logistic Regressions Predicting Problems More Than Level 1 on 5 EQ-5D Dimensions

	Modified Rankin Scale Score
	0	1	2	3	Barthel Index	Zung Depression	Proxy dummy	Rest-of-World Dummy	Asia Dummy	Constant	Nagelkerke R2	AUC
Mobility											0.63	0.91
β	−5.89	−3.70	−2.44	−2.23	−0.07	0.02			0.59	8.20
SE	0.97	0.67	0.64	0.62	0.01	0.01			0.19	1.03
P value	0.00	0.00	0.00	0.00	0.00	0.00			0.00	0.00
OR	0.00	0.03	0.09	0.11	0.93	1.02			1.80
Self-care											0.78	0.95
β	−5.41	−2.64	−0.84		−0.16				−0.49	14.63
SE	1.72	0.39	0.23		0.01				0.24	1.20
P value	0.00	0.00	0.00		0.00				0.05	0.00
OR	0.00	0.07	0.43		0.86				0.62
Usual activities											0.67	0.93
β	−4.38	−2.81	−0.93		−0.09	0.04	0.73		−0.57	8.71
SE	0.54	0.37	0.36		0.02	0.01	0.33		0.22	1.64
P value	0.00	0.00	0.01		0.00	0.00	0.03		0.01	0.00
OR	0.01	0.06	0.40		0.92	1.04	2.07		0.57
Pain and discomfort											0.26	0.70
β	−1.21	−0.57		0.34	−0.01	0.03	0.38	0.70	1.07	−1.53
SE	0.34	0.20		0.16	0.00	0.01	0.17	0.18	0.16	0.39
P value	0.00	0.00		0.04	0.00	0.00	0.03	0.00	0.00	0.00
OR	0.30	0.56		1.40	0.99	1.03	1.47	2.02	2.91
Anxiety and depression											0.39	0.69
β			0.29			0.09	0.37	0.67	0.71	−5.06
SE			0.16			0.01	0.18	0.19	0.17	0.31
P value			0.08			0.00	0.04	0.00	0.00	0.00
OR			1.33			1.09	1.45	1.96	2.04

OR, odds ratio; SE, standard error.

Table 3 indicates that higher BI scores decreased the likelihood of problems being reported for the first 4 dimensions, although not for anxiety and depression. Reporting problems was less likely at Rankin scores signaling low disability and dependence, especially mRS0 and mRS1. As with the BI, Rankin scores failed to predict responses on the anxiety and depression dimension. A higher Zung Depression score made reporting problems more probable, for all dimensions except self-care. Proxy responses were more likely to indicate problems than patient responses on the usual activities, pain and discomfort, and anxiety and depression dimensions. Non-UK respondents were more likely than UK patients to indicate problems on the pain and discomfort and anxiety and depression dimensions. Respondents in Asia were also more likely to report mobility problems but were less likely to report problems with self-care and with usual activities. Compared with the first 3 dimensions, the models were less successful in predicting problems for the pain and discomfort and anxiety and depression dimensions. This having been said, problems in these dimensions were less prevalent in the sample than those in the other dimensions (Table 1), and none of the clinical instruments contained an explicit pain measure.

Having discriminated between those reporting problems and those reporting no problems (Table 3), we constructed corresponding binary logistic models to discriminate between those with extreme or severe problems (level 3), as opposed to moderate problems (level 2). Estimation for each dimension entailed excluding those in the sample reporting no problems (level 1), as per Table 1. Again, these models resulted from backward stepwise elimination, and Table 4 presents the results. Only higher-level mRS scores were included as an independent variable, because patients with the lower mRS scores rarely reported problems. Higher BI scores reduced the likelihood that severe problems would be reported, for all dimensions except pain and discomfort, although a high Rankin score was influential in predicting severity on the mobility and usual activities dimensions only. Higher Zung Depression scores predicted severe problems for pain and discomfort and anxiety and depression. Proxy respondents were more likely to indicate moderate than severe problems on the usual activities dimension. When problems in self-care were reported, non-UK subjects were more inclined to rate them as severe. Asian subjects were less inclined to rate pain-discomfort problems as severe. As with those of Table 3, the models were less successful in predicting problems for the pain and discomfort and anxiety and depression dimensions.

OPEN IN VIEWER

Table 4

Stepwise Logistic Regressions Predicting Severe (Level 3) Problems on 5 EQ-5D Dimensions

	Modified Rankin Scale Score 4 or 5	Barthel Index	Zung Depression	Proxy Dummy	Rest-of-World Dummy	Asia Dummy	Constant	Nagelkerke R2	AUC
Mobility								0.63	0.95
β	1.56	−0.09					0.82
SE	1.09	0.01					1.19
P value	0.15	0.00					0.49
OR	4.77	0.91
Self-care								0.58	0.91
β		−0.09			0.70	0.99	2.73
SE		0.01			0.42	0.34	0.36
P value		0.00			0.10	0.00	0.00
OR		0.92			2.00	2.69
Usual activities								0.50	0.86
β	0.79	−0.06		−0.47			3.15
SE	0.25	0.01		0.24			0.55
P value	0.00	0.00		0.05			0.00
OR	2.20	0.94		0.63
Pain and discomfort								0.16	0.65
β			0.05			−1.11	−5.31
SE			0.01			0.46	0.72
P value			0.00			0.01	0.00
OR			1.05			0.33
Anxiety and depression								0.28	0.70
β		−0.01	0.08				−7.12
SE		0.01	0.01				1.06
P value		0.09	0.00				0.00
OR		0.99	1.09

OR, odds ratio; SE, standard error.

Summarizing the results in Tables 3 and 4, proxy respondents were more likely than patients to report problems on the usual activities, pain and discomfort, and anxiety and depression dimensions. They were no more likely to report severe problems on the latter 2 dimensions but, with respect to usual activities, were less likely to report the problems as severe. Respondents completing questionnaires in Asian sites were more likely than UK respondents to report problems for mobility and anxiety and depression but less likely for usual activities. In addition, they were less likely to record problems in self-care but, when such problems were reported, were more likely to rate them as severe. The situation with respect to pain and discomfort was the reverse: Asian respondents were more likely than UK respondents to record problems but, when such problems were reported, were less likely rate them as severe. Finally, respondents from the rest of the world were more likely to report problems for pain and discomfort and anxiety and depression only, although, when reported, problems in self-care were more likely to be assessed as severe. These variations in the impact of the dummy variables by level suggest that the DIF identified was nonuniform.

Discussion

Our mapping of EQ-5D index scores on stroke patients’ modified Rankin scores produced accurate predictions and a better fit than had a previous analysis. ¹² This fit was improved further, albeit by a modest amount, by including additional clinical measures in the model. Building on this mapping, we were able to test whether the existence of 2 forms of DIF were sufficient to affect either the index scores or the distribution of dimension levels.

With respect to patient/proxy reporting, disparities in judgment in level of health problem were observed for 3 dimensions—namely, limitations on usual activities, pain and discomfort, and anxiety and depression. Proxies were more likely to report health problems than were patients, although less severe ones in the case of usual activities. The divergences were not sufficiently large to result in significant differences in index scores between patient and proxy reports, presumably because, first, moderate or severe problems with pain and discomfort and anxiety and depression were reported the least frequently (Table 1) and, second, the different opinions on severity with respect to limitations on usual activities were, in some degree, compensating. The significance of proxy overstatement on the VAS was marginal (Table 2). Our findings are broadly consistent with those of most other quality-of-life studies of stroke patients, which conclude that proxy respondents overestimate patients’ functional impairment and dependence compared with the estimates of the patients themselves. ²⁴ The timing of the ENOS data collection might well have been important in this context: a proxy-patient pairing study using the EQ-5D ⁷ noted that differences between responses decreased with time after the stroke event, especially beyond 1 month.

DIF across regions seems more influential than DIF by proxy. Distributions of reported levels of problems for similar clinical states differed by region (Tables 3 and 4), and these translated into significantly different mean index and VAS scores (Table 2). Both were higher for UK responses than for those of responses from Asia and the rest of the world. To the best of our knowledge, regional variation in EQ-5D response has been observed previously in 2 multinational trials only. First, a Rasch analysis was conducted on records from a large (n = 10,205) trial of treatment for schizophrenia, which recruited across 10 countries in Western Europe. ²⁵ Rasch analysis tests for differences in item response patterns, based on the assumption of an underlying uniformity in latent characteristics. The study concluded that national response patterns were congruent across the dimensions with a notable exception: patients were more likely to indicate no problems on the mobility dimension in 9 countries but less likely to do so in the tenth. The authors speculated that the behavior of the outlier might have been accounted for by translation imprecision, “cultural habits,” or small sample size.

The second and more recent study ²⁶ examined data from a large randomized controlled trial (n = 11,118) of treatment for diabetes. Records were available from 20 countries grouped into 3 regions: Asia, Eastern Europe, and established market economies, including the United Kingdom. Logistic regression models were fitted to predict the reporting of health problems on EQ-5D dimensions from clinical histories, risk factors, and sociodemographic characteristics, plus the regional dummy variables. This second study has obvious parallels with the present examination of ENOS data: it employed the same statistical method to detect DIF and constructed broadly similar regional classifications (around two-thirds of ENOS’ “rest-of-world” records originated from Eastern Europe). However, the subjects in the 2 trials differed in their EQ-5D profiles. Only 4% of the diabetes patients recorded a severe health problem in at least 1 domain, and 38% reported no problems on any, compared with 35.8% and 11.8%, respectively, for the ENOS sample. The diabetes study was therefore able to produce models analogous to Table 3 but not to Table 4, while the low prevalence of severe health problems made proxy reporting unnecessary. It was discovered that, allowing for the control variables, diabetes patients in Eastern Europe were significantly more likely to report health problems on all 5 dimensions, relative to patients from established market economies. In contrast, and also relative to market economy respondents, diabetes patients from Asia were less likely to report a problem on the mobility, self-care, and usual activities dimensions but more likely to report one on the anxiety and depression dimension.

In view of the differences in the diseases and countries being considered in these 2 studies and our own, it is unsurprising that, although region-related DIF is evident, no consistent structure emerges. In none of the studies can the explanation for DIF be found in the existing data. However, we note that, whatever that explanation, DIF in quality-of-life instruments used in multinational settings is not confined to the EQ-5D. Interregional variations in quality of life after stroke as measured by the SF-36 instrument have been observed, even following adjustment for prognostic case mix and care quality. ²⁷ In the cancer field, the QLQ-C30 instrument ²⁸ has been translated extensively and used frequently in international studies. An analysis of a large sample of previously collected QLQ-C30 results revealed degrees of DIF by region relative to the response patterns of UK subjects. ²⁹ The variation in response styles was ascribed to sociocultural differences, for example, whether or not the society in which respondents lived embodied an ethic of conformity or of individuality, was tolerant of assertiveness and criticism, or fostered aversion to ambiguity or uncertainty. ³⁰ DIF was most pronounced between responses from the United Kingdom, on one hand, and Eastern Europe and Asia, on the other, a result that bears a strong resemblance to our own findings. Cultural differences might well lie at the root of the explanation for DIF within the EQ-5D also; indeed, they have been offered as an explanation for differences in population health state utilities across the national value sets. ³¹

We conclude that, according to this study of stroke patients, differential item functioning with respect to patient-v.-proxy responses to the EQ-5D is detectable, although the impact on the mean index scores is insignificant. DIF with respect to multinational responses is detectable, but in this case, the impact on index scores is significant. Whether such a bias constitutes a problem, from the point of view of using trial results to inform decisions, remains debatable. ¹⁷ Suppose, for example, that we elected to use our DIF analysis to disinfect the trial results. Table 2 (model 3) indicates that, for the same latent health characteristics, non-UK patients provided EQ-5D descriptions that translate to index scores 0.04 lower than those provided by UK patients. By implication, had identical patients been responding in the United Kingdom rather than elsewhere, their scores would, on average, have been 0.04 higher. By weighting the index scores by the proportions of patients by region, we might calculate a “true UK” mean index score for the entire sample, using both UK utility values and UK EQ-5D distribution of levels. This mean would be 0.55, rather than the present 0.53. Alternatively, we could simply ignore DIF and take the results at face value. We might accept that all measurement entails error margins and that the error introduced by DIF is acceptable. We could question the identification itself, by arguing that health state utility is more than a few latent clinical characteristics, although this argument does weaken the case for the popular practice of mapping utilities onto clinical outcomes. ³² Either way, it would be foolish to make recommendations until further multinational EQ-5D studies of stroke, as well as other medical conditions and circumstances, have been undertaken.