Assessment of the internal construct validity of the revised High-Level Mobility Assessment Tool for traumatic orthopaedic injuries

Abstract

Objective:

To determine whether the revised High-Level Mobility Assessment Tool (HiMAT) was valid for measuring mobility for people with multi-trauma orthopaedic lower limb injuries.

Design:

Cross-sectional study.

Subjects:

Participants with lower limb multi-trauma orthopaedic injuries.

Methods:

One complete revised HiMAT was obtained for 106 people within 12 weeks of being allowed to fully weight bear. Rasch analysis was used to assess the overall fit of the model for individuals and items, differential item functioning, local dependency, targeting of items and dimensionality.

Results:

The mean revised HiMAT score was 10.5 (SD = 6.8) with a range of 5–30. Rasch analysis of revised HiMAT showed adequate overall fit to the model (P = 0.29) with no misfitting items (fit residual SD = 0.69) or persons (fit residual SD = 0.62). The scale showed good internal consistency (Person Separation Index = 0.91). One item (hopping) demonstrated disordered thresholds, however this item had good fit to the model in all other aspects. The revised HiMAT was unidimensional, and no differential item functioning was detected for gender or age. The revised HiMAT was well targeted for this group with a range of items across all ability levels.

Conclusion:

The results of this study support the internal construct validity of the revised HiMAT as a well-targeted, unidimensional measure of high-level mobility with no ceiling or floor effect for males and females recovering from multi-trauma orthopaedic lower limb injuries.

Keywords

Mobility limitation outcome assessment (healthcare)trauma Rasch analysis

Introduction

Traumatic lower limb injuries are a major global health concern, and the leading cause of all trauma hospitalizations in adults below the age of 65 years.¹ The US Centers for Disease Control and Prevention reported 13.6 million non-fatal traumatic injuries to adults aged 20–49 per year.² Survival rates are high (~90%) due to advances in medical technology and the establishment of specialized trauma centres.³ Although people with multi-trauma orthopaedic injuries may be expected to make a good functional recovery because they are young, mounting evidence suggests continued morbidity well beyond standard fracture healing times.^4–8 Persistent mobility limitations restrict participation rates for a range of employment, social and leisure roles.^9,10 Many measures commonly used to quantify outcome following orthopaedic trauma do not quantify advanced mobility activities,^11–13 so may be susceptible to a ceiling effect.

Although successful return to work and social integration relies on a combination of factors, the capacity to perform high-level mobility skills may be important. Indeed, the ability to mobilize and return to high-level tasks, including running, have been described as key factors in a patient’s outcome.^14,15 In order to identify the contribution of mobility limitations to participation restrictions and reduced health-related quality of life, an objective unidimensional measure of high-level mobility is needed. The High-Level Mobility Assessment Tool (HiMAT) was originally developed to evaluate the advanced mobility skills required for participation in a range of day-to-day activities following traumatic brain injury.¹⁶ Initial psychometric evaluation showed the HiMAT to have high inter-rater (intraclass correlation coefficient, ICC = 0.99) and test–retest (ICC = 0.99) reliability. It has recently been revised using advances in Rasch analysis,¹⁷ reducing the original set of 13 items to 8. The four stair and one bounding item were removed. The revised HiMAT comprises items ranging from a 10-m walk test to running, skipping, hopping and bounding. Performances are measured by a stopwatch or tape measure and each is converted to a score from 0 to 4. Higher scores represent better performance. Although the revised HiMAT has been validated for people with a range of neurological disorders,¹⁸ the validity of the HiMAT for multi-trauma orthopaedic injury has not been reported.

A recent study by Williams et al. (in press PM&R; accepted for publication 12 September 13) indicated that the revised HiMAT was more responsive to change and less susceptible to a ceiling effect than existing scales in a small sample of 43 people with multi-trauma orthopaedic lower limb injuries. Further, revised HiMAT scores appear to be less susceptible to changes in upper limb function, pain, and emotional status. Outcome measures should be validated for the populations in which they are intended for use. To date the revised HiMAT has only been validated in populations with neurologically based diagnoses.^16–18 Since the revised HiMAT may have some advantageous psychometric properties for this population, it is important to investigate its internal construct validity. Therefore, the aim of this study was to investigate the internal construct validity of the revised HiMAT for people with multi-trauma orthopaedic lower limb injuries.

Methods

This study was approved by Epworth HealthCare Human Research and Ethics Committee (HREC No. LR10213). One complete revised HiMAT score¹⁷ was obtained from people attending an outpatient rehabilitation unit with multi-trauma orthopaedic lower limb injuries sustained following either motor vehicle or work accidents. Participants who were over 18 at the time of injury and had sustained lower limb or pelvic trauma including fractures or severe soft tissue injury were sequentially recruited once they could walk a minimum of 20 m without a gait assistive device. Participants were excluded if they had sustained (1) concomitant lower limb peripheral nerve injuries including lumbo-sacral plexus injury; (2) a traumatic brain injury (Glasgow Coma Score ≤13); or (3) a premorbid medical condition that limited mobility.

The testing procedure for the revised HiMAT was conducted according to the published instructions.^6,17 Briefly, this includes a timed performance for several walking items, as well as running, skipping, hopping and bounding. Performances are timed over the middle 10 m of a 20-m track. The distance of the bounding item is measured in centimetres from the toe of the take-off leg, to the heel of the landing leg. All data were de-identified and pooled for Rasch analysis. Analyses were undertaken using the RUMM2030 program.¹⁹

Given the heterogeneous nature of the multi-trauma orthopaedic population we used the 4-point Bethesda Scale to classify the degree of lower limb injury sustained by this cohort.⁴ All fractures of each participant were rated as follows: (1) uncomplicated fracture with no major joint disruption, no motor loss and no significant soft tissue injury; (2) multiple uncomplicated fractures, excluding joint surface disruption or nerve lesion; (3) fracture involving major joint surface, including minor nerve lesions; and (4) severe crush fractures or major soft tissue lesion, including severe nerve lesions. However, we excluded all those with peripheral nerve and lumbo-sacral plexus injuries from this study, so Bethesda Scale scores of 4 were based on severe crush fractures and soft tissue injury severity. Each individual is assigned a score related to the highest rating obtained for each injury. For example, if a person sustains four lower limb fractures that are rated as 1, 2, 2 and 3, the Bethesda Score assigned to that person is 3.

Data analysis

Rasch analysis was undertaken using the RUMM2030 program¹⁹ to assess the internal construct validity of the revised HiMAT. This included an assessment of the overall fit of the items, response format, internal consistency reliability, differential item functioning, local dependency, targeting of items, and dimensionality. The procedures undertaken were consistent with published guidelines.^18,20–23

The overall aim of Rasch analysis is to assess the degree to which the scale, and its items, behaves in a manner consistent with the expectations of the underlying Rasch mathematical measurement model. It is expected that patients with low levels of the underlying trait (in this case mobility) will experience difficulty performing the various tasks specified in each of the HiMAT items, while those with high levels of mobility will successfully complete more tasks and thereby achieve higher overall scores. To assess the fit of the scale to the Rasch model a number of fit statistics are used. A non-significant chi-square probability value indicates the overall hierarchical ordering of the items was consistent across all levels of the underlying trait (i.e. mobility). The overall fit to the model is also assessed using fit residual values, with a fit residual standard deviation value that exceeds 1.5 suggesting misfit. To assess the fit of individual items and persons it is expected that the fit residual values should fall within the range of ±2.5.

Potential problems with the response format of the scale is indicated by the presence of disordered thresholds. A threshold is the point between two response categories where either response is probable (i.e. the probability of scoring a 0 or a 1 is 50/50).²⁰ One of the most common sources of item misfit occurs as a result of inconsistent use of category options, resulting in disordered thresholds. This can be resolved by collapsing adjoining categories.^22,23

Differential item functioning (DIF) is a form of item bias that occurs when different groups within the same sample (e.g. males and females) respond in a different manner to an item despite equal levels of the underlying characteristic (i.e. mobility limitation) being measured. In this sample we examined DIF for gender and age. The targeting of the revised HiMAT was assessed to ensure sufficient items assessed the full range of patient’s ability, with no evidence of floor or ceiling effects. Internal consistency reliability was evaluated using the Person Separation Index. The Person Separation Index may be interpreted in the same way as the Cronbach alpha used in Classic Test Theory, with a value of 0.70 considered the minimally accepted value.²⁴

One of the key requirements of a good scale is unidimensionality. This is assessed in RUMM2030 using a two-step process. Principal components analysis is conducted on the residual correlation matrix to identify two sets of intercorrelated items. Rasch-derived scores on these two sets are then compared using a series of t-tests. If fewer than 5% of these tests are significantly different (or the lower bound of the confidence interval contains 5%) then the scale can be considered unidimensional.

Fit to the Rasch model requires that the response to one item is not influenced by the response to another (i.e. all items must be locally independent). For example if a person can walk 100 m they must be able to walk 10 m, therefore these two items would not be locally independent. The presence of local dependency was evaluated by inspecting the residual correlation matrix for items showing correlations above 0.2. The impact of local dependency on the Person Separation Index value was assessed by creating ‘testlets’ or summary scores of any pairs of locally dependent items.²² The Person Separation Index values obtained using testlets were then compared with the original Person Separation Index values, and the difference evaluated. A substantial difference between the two values indicates the presence of local dependency, and suggests that the internal consistency of the scale may be artificially inflated.

Results

One complete revised HiMAT score was obtained for 106 patients with multi-trauma orthopaedic lower limb injuries. All patients were attending outpatient therapy and were in the early rehabilitation phase of their recovery (Table 1). All patients had sustained high-velocity severe multi-trauma orthopaedic injuries; 75 (71%) had unilateral lower limb involvement and 31 (29%) had bilateral lower limb involvement. There was variability in patient physical ability with a mean revised HiMAT score of 10.5 (SD = 6.8), ranging from 2 to 30. The mean score for the Bethesda Scale was 2.6 (SD = 0.75, range 1–4), with 14 (13%) having uncomplicated fractures (BS = 1), 18 (17%) having sustained multiple uncomplicated fractures (BS = 2), 71 (67%) sustaining a fracture or fractures involving a major joint surface (BS = 3) and 3 (3%) having severe crush fractures or major soft tissue injuries (BS = 4).

Table 1.

Patient demographics.

Descriptor	Mean (SD)	Range
Age (years)	37.3 (13.3)	17–79
Time post injury (days)	126.4 (92.7)	6–652
Time post weight bearing (days)	77 (74.5)	0–552
Bethesda Score	2.6 (0.8)	1–4

Rasch analysis

Rasch analysis of the eight items from the revised HiMAT showed adequate overall fit to the model (P = 0.29), indicating the hierarchical ordering of all the items is consistent across the range of mobility performance. Residual mean fit values for revised HiMAT items were 0.15 (SD = 0.69) and 0.12 (SD = 0.62) for individuals, suggesting no problems with misfitting items or persons. No individual person fit residuals exceeded 2.5, indicating no outliers that did not fit the model, despite the wide range of orthopaedic trauma sustained by this cohort. All revised HiMAT items demonstrated model fit with no fit residual residuals above 2.5 (Table 2). Item 7 ‘hop’ exhibited disordered thresholds, suggesting the possibility the scoring for this item may not be working as intended (Table 3). However, the majority of participants found this item hard to perform, with 90 of the 106 unable to hop 10 m on their more affected leg. In all other respects, the hopping item demonstrated adequate fit to the Rasch model, therefore no changes were made to the scoring.

Table 2.

Individual item fit statistics of the revised HiMAT.

No.	Item	Location	SE	Fit residual	df	Chi square	df	P-value
1	Walk	−0.109	0.188	−0.595	89	1.792	1	0.181
2	Walk backward	−2.919	0.186	−0.826	89	0.895	1	0.344
3	Walk on toes	−1.352	0.156	−0.249	89	0.720	1	0.397
4	Walk over obstacle	−1.654	0.189	−0.808	89	0.167	1	0.683
5	Run	2.432	0.209	0.988	89	0.533	1	0.465
6	Skip	2.112	0.208	−0.583	89	1.111	1	0.291
7	Hop	2.478	0.198	0.226	89	1.724	1	0.189
8	Bound	−0.990	0.153	0.669	89	2.709	1	0.099

SE, Standard error; df, degrees of freedom.

Table 3.

Frequency distribution of responses to the revised HiMAT.

Item	Total	Response category frequency
		0	1	2	3	4
Walk	106	0	49	28	21	8
Walk backward	106	1	27	44	25	9
Walk on toes	106	22	34	15	25	10
Walk over obstacle	106	2	53	28	18	5
Run	106	54	37	11	4	0
Skip	106	71	22	8	4	1
Hop	106	90	7	3	5	1
Bound	106	32	29	18	15	12

The Person Separation Index was 0.91 (Cronbach alpha α = 0.93), indicating the revised HiMAT had good internal consistency reliability in this population. An examination of the residual correlation matrix showed evidence of local dependency with two clusters of items, grouping the four walking items together, and the four non-walking items (run, skip, hop, bound). Two testlets were created based on these clusters and the full analysis repeated as per the method proposed by Nilsson and Tennant.²² A series of t-tests showed only one case (0.9%) with a significant difference between the testlets. The Person Separation Index for testlets (a = 0.79) was slightly lower than that recorded for the original analysis (a = 0.85), however it remained within acceptable levels.²⁰

No differential item functioning was detected for gender or age, indicating that males and females, and younger and older participants, performed similarly on the revised HiMAT items given the same level of mobility. The revised HiMAT was well targeted for this group, with a range of items across all ability levels (Figure 1). No participant achieved the highest or lowest possible scores, indicating no ceiling or floor effect.

Figure 1.

Targeting map for person and item thresholds. The upper section of the figure shows the distribution for the participants. The lower section of the figure shows the distribution of the items.

Discussion

Detailed examination of the revised HiMAT using Rasch analysis found support for its internal construct validity for measuring high-level mobility following lower limb multi-trauma orthopaedic injuries. There is a spread of items across all mobility levels with no evidence of a floor or ceiling effect. These results are consistent with previous Rasch findings conducted on the revised HiMAT for other neurological populations.^17,18 One strength of the revised HiMAT is that it appears to be unidimensional, with no apparent ceiling effect. Consequently, it may assist clinicians to demonstrate the need for ongoing therapy to achieve the high-level mobility skills required for participation in many of the day-to-day activities commonly performed in this predominantly young male population.

One item (‘hop’) displayed disordered thresholds, but showed no other indication of misfit to the Rasch model. This may have been due to the relatively small number of participants (16/106) who were able to perform the hopping task. This group were evaluated early in their rehabilitation phase (i.e. within the first 12 weeks following the orthopaedic specialist’s decision to allow full weight bearing) (Table 1). The participants had a high burden of injury, with 31 (29%) sustaining bilateral lower limb injuries, and 71 (67%) recording a Bethesda Score of 3, indicating trauma to the articular surface of the hip, knee or ankle. It is therefore not unexpected that many people would find the ‘hop’ task very difficult. Although only a few participants were able to perform the hopping task, a decision was made to retain the hopping item in the scale. Hopping has been demonstrated to be a very high-level task that is an important clinical test for people with lower limb injuries.^16,25,26 It is likely that more people would be able to perform the hopping task as they progress through the latter stages of rehabilitation. A key aspect of any mobility assessment scale is that it contains sufficient items across a spectrum of ability to quantify the performance of people with more and less severe injuries, and to detect change as mobility improves.

The revised HiMAT displayed a degree of local dependency, suggesting that some items were dependent on the response to others. Four of the eight items on the revised HiMAT are specific to walking, therefore some degree of overlap, or local dependency may be expected. There was, however, no evidence of multi-dimensionality among these items. The combination of the walking items into a subtest resulted in a decrease in the Person Separation Index value, but not to the degree that would suggest local dependency was responsible for artificially inflating the apparent internal consistency of the revised HiMAT. This finding supports the retention of all items in the scale, providing a range of items of varying difficulty levels. This is particularly important in a mobility scale used to guide treatment planning and goal setting during rehabilitation.

The revised HiMAT appears to be well targeted for use in this population with a wide spread of items across all levels of subject ability (Figure 1). However, the revised HiMAT is only appropriate once a person is able to walk without a gait assistive device. Prior to independent ambulation, an outcome measure designed for the inpatient phase of rehabilitation, such as the Functional Independence Measure (FIM)¹¹ or the Functional Ambulation Classification (FAC),²⁷ may be more appropriate.

One limitation of this project is the relatively small sample size. The data were collected from 106 participants, which is at the lower bounds of the acceptable minimum required for Rasch analysis.²⁸ However, participants were predominantly young and male and therefore representative of the multi-trauma orthopaedic population.^2,3,5 In addition, testing was undertaken relatively soon after full weight bearing was allowed. However, as noted above, this is a key time point when clinicians require outcome measures that can justify ongoing therapeutic intervention. Anecdotally, once individuals no longer require gait assistive devices to mobilize, funding bodies increasingly question the need for ongoing therapy and require evidence of improvement over time.

Further testing may be required on individuals in the later stages of rehabilitation following extensive lower limb orthopaedic trauma, or those with a single isolated injury, to determine if the revised HiMAT items are discriminative for high-performing patients with advanced mobility goals. Additional investigation is needed to determine whether the revised HiMAT is appropriate for use with other types of lower limb trauma, such as peripheral nerve injury, isolated ligamentous and soft tissue injuries, or joint replacement.

In conclusion, the results of this study support the internal construct validity of the revised HiMAT for people recovering from multi-trauma orthopaedic lower limb injuries. Further investigation of the revised HiMAT is required to determine the test–retest, inter-rater reliability and minimal detectable change score for people with multiple traumatic orthopaedic lower limb injuries.

Clinical messages

Rasch analysis supported the internal construct validity of the revised HiMAT for measuring high-level mobility following multi-trauma lower limb orthopaedic injuries.

The revised HiMAT is unidimensional, with no item bias for age or gender, and is well targeted with no evidence of a ceiling effect for this population.

Footnotes

Acknowledgements

The authors wish to acknowledge the assistance of Tim Strahan, Physiotherapy Department, Epworth HealthCare during the data collection phase of this project.

Author contributions

BH and GW developed the study design. MK undertook recruitment, data collection and data analysis. BH and JP undertook the Rasch analysis. All authors contributed to the writing of this manuscript.

Conflict of interest

The authors declare that there is no conflict of interest.

Funding

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

References

Rhodes

Aronson

Moerkirk

Petrash

. Quality of life after the trauma center. J Trauma 1988; 28: 931–938.

Centers for Disease Control and Prevention. Injury prevention & control: data & statistics (WISQARS^TM). http://www.cdc.gov/injury/wisqars/index.html (accessed 18 December 2012).

Gabbe

Simpson

Sutherland

. Functional measures at discharge: are they useful predictors of longer term outcomes for trauma registries? Ann Surg 2008; 247: 854–859.

Ponsford

Hill

Karamitsios

Bahar-Fuchs

. Factors influencing outcome after orthopedic trauma. J Trauma 2008; 64: 1001–1009.

Butcher

MacKenzie

Cushing

. Long-term outcomes after lower extremity trauma. J Trauma 1996; 41: 4–9.

Michaels

Madey

Krieg

Long

. Traditional injury scoring underestimates the relative consequences of orthopedic injury. J Trauma 2001; 50: 389–396.

Soberg

Finset

Bautz-Holter

Sandvik

Roise

. Return to work after severe multiple injuries: a multidimensional approach on status 1 and 2 years postinjury. J Trauma 2007; 62: 471–481.

Horwitz

Schuerer

DJE

. Trauma rehabilitation outcome scoring. Curr Opin Crit Care 2008; 14: 445–450.

Doukas

Hayda

Frisch

. The Military Extremity Trauma Amputation/Limb Salvage (METALS) study outcomes of amputation versus limb salvage following major lower-extremity trauma. J Bone Joint Surg Am 2013; 95A: 138–145.

10.

MacKenzie

Bosse

Pollak

. Long-term persistence of disability following severe lower-limb trauma – Results of a seven-year follow-up. J Bone Joint Surg Am 2005; 87A: 1801–1809.

11.

Granger

Hamilton

Sherwin

. Guide for the use of the uniform data set for medical rehabilitation. New York: Uniform Data System for Medical rehabilitation Project Office, Buffalo General Hospital, 1983.

12.

Jennett

Bond

. Assessment of outcome after severe brain damage: A practical scale. Lancet 1975; 1: 480–484.

13.

Williamson

Gabbe

Sutherland

Wolfe

Forbes

Cameron

. Comparing the responsiveness of functional outcome assessment measures for trauma registries. J Trauma 2011; 71: 63–68.

14.

Lindstrom

Roding

Sundelin

. Positive attitudes and preserved high level of motor performance are important factors for return to work in younger persons after stroke: a national survey. J Rehabil Med 2009; 41: 714–718.

15.

Trickett

Mudge

Price

Pallister

. A qualitative approach to recovery after open tibial fracture: the road to a novel, patient-derived recovery scale. Injury 2012; 43: 1071–1078.

16.

Williams

Robertson

Greenwood

Goldie

Morris

. The high-level mobility assessment tool (HiMAT) for traumatic brain injury. Part 2: Content validity and discriminability. Brain Inj 2005; 19: 833–843.

17.

Williams

Pallant

Greenwood

. Further development of the High-level Mobility Assessment Tool (HiMAT). Brain Inj 2010; 24: 1027–1031.

18.

Williams

Hill

Pallant

Greenwood

. Internal validity of the revised HiMAT for people with neurological conditions. Clin Rehabil 2012; 26: 741–747.

19.

Andrich

Lyne

Sheridan

Luo

. RUMM2030. Perth: RUMM Laboratory Pty Ltd, 2010.

20.

Pallant

Tennant

. An introduction to the Rasch measurement model: An example using the Hospital Anxiety and Depression Scale (HADS). Br J Clin Psychol 2007; 46: 1–18.

21.

Williams

Pallant

Greenwood

. Further development of the High-level Mobility Assessment Tool (HiMAT). Brain Inj 2010; 24: 1027–1031.

22.

Nilsson

Tennant

. Past and present issues in Rasch analysis: The Functional Independence Measure (FIM™) revisited. J Rehabil Med 2011; 43: 884–891.

23.

Da Rocha

Chachamovich

de Almeida Fleck

Tennant

. An introduction to Rasch analysis for psychiatric practice and research. J Psychiatr Res 2013; 47: 141–148.

24.

Lohr

. Assessing health status and quality-of-life instruments: Attributes and review criteria. Qual Life Res 2002; 11: 193–205.

25.

Fitzgerald

Lephart

Hwang

Wainner

. Hop tests as predictors of dynamic knee stability. J Orthop Sports Phys Ther 2001; 31: 588–597.

26.

Reid

Birmingham

Stratford

Alcock

Giffin

. Hop testing provides a reliable and valid outcome measure during rehabilitation after anterior cruciate ligament reconstruction. Phys Ther 2007; 87: 337–349.

27.

Holden

Gill

Magliozzi

Nathan

Piehl-Baker

. Clinical gait assessment in the neurologically impaired. Reliability and meaningfulness. Phys Ther 1984; 64: 35–40.

28.

Linacre

. Sample size and item calibration stability. Rasch Measure Trans 1994; 7: 97–116.