Abstract
This article describes the development and validation of the Modified Occupational Questionnaire (MOQ), a measure of meaningful time use. The measurement of meaningful time use is a key element of occupationally focused inquiry (Farnworth, 2003). Although experienced and suitably occupation-focused clinicians are able to evaluate the meaningfulness of time use through detailed interview, a paucity of useful tools exists to measure meaningful time use in the context of large-scale quantitative investigations.
A wide range of different methods has been designed to collect information about time use, including stylized questionnaires or time estimation techniques (Robinson, 1999); time diaries and time budgets (Harvey, 1993; Robinson, 1999); direct observation; and more complex methodologies, such as Csikszentmihalyi’s (1991) Experience Sampling Method. With the exception of the complex and expensive Experience Sampling Method, all the techniques share the same limitation for the occupational therapist: They fail to evaluate the personal meaning ascribed to each activity (Robinson, 1999).
When we set out to investigate the relationship between meaningful use of time and self-reported health in a group of 18- to 25-yr-old unemployed Australians, we needed an efficient and accurate method of collecting information about both the objective (what the person was doing) and the subjective (how the person felt about it) dimensions of time use. Although the Australian government is recognized as a leader in time-use surveys (Harvey, 1993; Ironmonger, 1999), the data collected do not include information on the subjective qualities of activities. This situation is true of almost all wide-scale time-use investigations led by governments.
We explored a range of time-diary formats reported in the literature (Farnworth, 2003; Harvey, 1993; Ironmonger, 1999). Many of these formats also failed to explore the subjective dimensions of time use. The time-diary format that most closely matched our needs was the Occupational Questionnaire (Smith, Kielhofner, & Watts, 1986).
The Occupational Questionnaire is a time diary that asks respondents to report on what they do “on a typical day” in half-hour blocks from 5:00 a.m. to midnight. To investigate the subjective qualities of activities, respondents are also requested to classify each activity according to (1) its type, (2) their competence at it, (3) how much they value it, and (4) how much they enjoy it (Smith et al., 1986). The addition of such questions about the subjective dimensions of activity is a useful strategy to allow deeper exploration of time use (Michelson, 1999). The questionnaire has demonstrated acceptable test–retest reliability and good concurrent validity (68% and 82% agreement in activities undertaken, respectively; Smith et al., 1986) and has been used in several small-scale studies (e.g., Ebb, Coster, & Duncombe, 1989; Kielhofner & Brinson, 1989; Stewart & Craik, 2007) to measure time use and elements of occupational engagement.
After initial examination of the Occupational Questionnaire within the context of our study, we determined that modifications were required before its use with unemployed young people. The remainder of this article describes the modification process and the piloting, refinement, and evaluation of the psychometric properties of the new measure, the MOQ.
Method
This study was approved by the University of Sydney Human Research Ethics Committee.
Modifications and Piloting
The Occupational Questionnaire was presented to a panel of doctoral students and occupational therapy academics to evaluate its face and content validity. A range of modifications was suggested, including expansion of the categories of activity and rewording of the subjective scales to more effectively capture information about meaningful time use in the context of unemployment.
The process of developing questions that allowed the examination of the meaningfulness of time use was complex. Dictionary definitions of meaningfulness include elements of “value,” “importance,” and “significance” (Soanes & Stevenson, 2005; Yallop et al., 2005), and occupationally focused researchers describe elements of connectivity, congruence, coherence, sociocultural values, and a sense of “making a contribution” (Hvalsøe & Josephsson, 2003; Jonsson & Josephsson, 2005). We selected four areas to investigate the meaningfulness of each activity: (1) category of activity, (2) the reason the person was doing the activity, (3) value of the activity to the individual, and (4) perceived value of the activity to society.
The activity category question was used to classify the productiveness of the activity on the basis of a hierarchy of “worklikeness.” The expanded categories included (1) work; (2) unpaid work; (3) caring for self; (4) sport; (5) recreation/leisure; (6) socializing; (7) “chilling”/doing nothing; (8) rest; and (9) other. Because worklikeness of activities is likely to be linked to subjective definitions of activity category, no attempts were made to define each category, thereby allowing participants to self-define each category. Worklike activities were hypothesized to be related to a greater sense of contribution to society and possibly related to a greater sense of meeting the sociocultural expectations that all adults should work (Winefield et al., 2002). In addition, participation in worklike activities may enhance opportunities to access leisure; many authors have suggested that unemployed people feel that their free time is not leisure because it is only through work that one “earns” leisure (Lobo, 1999; Øian, 2004; Pettifer, 1993).
The question related to the reason the person was doing the activity (“I had to do it,” “I wanted to do it,” or “I had nothing else to do” from Experience Sampling Method) sought to examine the purposefulness of the activity. Nonproductive use of time has been related to boredom (Csikszentmihalyi, 1991; Farnworth, 1998; Feather & Bond, 1994; Roberts, Lamb, Dench, & Brodie, 1989; Roche, 1990) and occupational deprivation (Whiteford, 2000; Wilcock, 1998). Having a reason for doing activities is likely to promote a better sense of coherence and promote best congruence between what people do and what they and others expect them to be doing. Engaging in activities because one has to is also likely to be related to obligations to other people and may suggest better connectivity between people and their community.
Questions related to the value of the activity (to people and their perception of the value of the activity to society, rated on a 5-point scale: 1 = not at all valuable, 2 = not very valuable, 3 = somewhat valuable, 4 = quite valuable, and 5 = very valuable) tap most directly into the meaningfulness of the activity. Individually valued activities are likely to promote a better sense of coherence and congruence, and participation in activities that are valued by society are likely to be related to a stronger sense of contribution to society, connectivity, and meeting sociocultural expectations. Because employment is recognized as a primary means of establishing personal status and identity (Jahoda, 1981), these value questions were considered especially important for the examination of meaningful time use for unemployed people. When considered together, these four parameters of time use were hypothesized to represent the construct of meaningful time use, especially in the context of unemployment.
Additional modifications were made. The time period measured was expanded to a full 24 hr but, to minimize respondent burden, time blocks were increased from 30 min to 1 hr. A risk associated with increasing time blocks to 1 hr is that activities of short duration may be underreported. Respondents were also asked to report on what they did yesterday, rather than for a hypothetical typical day.
Version 1 of the MOQ was piloted with a group of occupational therapy students to seek feedback about its content and ease of use. The students were a sample of convenience; however, most fell into the same age group as our target population, and they completed the questionnaire during university vacation. In response to feedback, activity categories were expanded to include additional categories of (1) caring for others, (2) housework, (3) travel, and (4) playing with children. A prompt of “please write what you did in this space” was added to each 1-hr block because many respondents left this space blank. Respondents did not report difficulties associated with the 1-hr reporting format.
Version 2 of the MOQ was further piloted with another group of occupational therapy students. Completeness of data was enhanced, and respondent feedback was positive. Thus, the format was retained.
Participants and Data Collection
During April 2007, Version 2 of the MOQ was completed by 272 unemployed 18- to 25-yr-olds residing in New South Wales, Australia’s most populous state. Participant recruitment and data collection were conducted online by means of a third-party research company. Forty-four participants were excluded: 21 because of substantial amounts of missing data and 23 who reported on weekend time use. Weekend time use was excluded because weekend and weekday time use may be different, even for unemployed people. Demographics of the final sample are summarized in Table 1. No significant differences were found in the demographics of those participants reporting on weekday or weekend time use.
Demographics of the Final Sample (N = 228)
Data Analysis
We evaluated both the external and the internal validity of the MOQ.
External Validity.
External validation incorporates a range of methods for evaluating whether an instrument effectively measures what it purports to measure. One of the most common measures of external validity is concurrent validity, which is determined by having participants complete, in addition to the instrument under review, a second, “gold standard” measure that evaluates the same, or a closely related, construct. A gold standard measure of time use in this circumstance would be the Australian Time Use Survey (Australian Bureau of Statistics [ABS], 2008b).
In the context of this study it was not feasible, nor would it have been appropriate, to have participants complete the Australian Time Use Survey in addition to the MOQ. Because both instruments require participants to describe their daily pattern of activity, responses from one would directly influence responses on the other, therefore risking an artificial inflation of the correlation between instruments.
The external validity of the MOQ was evaluated by comparing the mean length of time spent in activities by participants using the MOQ and the mean length of time spent in activities by unemployed 18- to 25-yr-olds in the 2006 Australian Time Use Survey ([TUS06] a separate group of participants; ABS, 2008a). Only data for weekday time use were included in this analysis. High correlations between MOQ data and the TUS06 data would support the external validity of the MOQ.
MOQ activities were coded according to ABS criteria (ABS, 2008b) by the first author (Scanlan) and an independent coder. Coding discrepancies were corrected by consensus. The ABS (2008a) granted access to confidentialized microdata from the TUS06 to allow a secondary data analysis to be conducted. For each 18- to 25-yr-old unemployed person reporting on weekday time use in the TUS06, total time spent in different categories of activity was determined by analyzing episode duration and the primary activity reported (using the “purpose of activity” code). Although secondary activities are important, only primary activities were used because the MOQ format did not allow for reporting of secondary activities. The ABS coding methodology allows for activities to be examined on three levels of detail: fine (e.g., handiwork, crafts; 223 categories); moderate (e.g., games, hobbies, arts, crafts; 79 categories); and coarse (e.g., recreation and leisure; 10 categories). Because time-use data did not conform to the assumptions of normality and homogeneity of variance required for parametric tests, comparisons were made using the Spearman’s ρ statistic (Huber, 1981).
Internal Validity.
Rasch modeling was used to evaluate the construct validity, scale validity, and statistical validity of the MOQ. The data matrix included two facets (people and items) and 24 hourly observations for each person. Each hour of the day yielded one line of data (activity category, reason for doing the activity, value to self, and perceived value to society). Thus, to derive a single-measure score for each participant, we used the many-faceted Rasch analysis program FACETS (Version 3.64.0; Linacre, 1987–2008). This analysis approach allowed for the examination of the contribution of each parameter of time use for each activity and across the course of a day for each person. Rasch analysis is a common method of psychometric evaluation (Bezruczko, 2005; Bond & Fox, 2007) and is based on two assertions. As they apply in this study, those assertions are that (1) the easier the item is, the more likely it is that people will affirm it, and that (2) people who demonstrate more ability are more likely to affirm more difficult items (Bond & Fox, 2007).
In the context of this study, the construct being measured was “meaningful time use,” and the items being tested were “category of activity,” “reason for doing the activity,” “value to self,” and “perceived value to society.” Person ability was the extent to which participants filled their time with meaningful activity.
Several different quality control criteria have been established in the literature to assist in the evaluation of validity and reliability of assessment tools. In this study, we investigated the rating scale validity, item function and person performance validity, and statistical validity of the MOQ, as well as the construct validity.
In “rating scale” models, such as those used in the MOQ, the examination of category ordering and Rasch–Andrich thresholds is important for ensuring scale validity. Where there are several response categories, the “average measures” for each category should demonstrate monotonic progression (e.g., it should be more difficult to affirm that an activity is very valuable to me than somewhat valuable to me: Bond & Fox, 2007; Linacre, 1999a). Rasch–Andrich thresholds indicate the ability level at which each category becomes modal. Disordered thresholds suggest category redundancy because some categories are never modal (Andrich, 1996; Linacre, 1999a). If categories are not disordered, it is generally recommended that redundant categories be combined with adjacent, theoretically similar categories (Bond & Fox, 2007). Although the monotonic progression of thresholds is the primary concern, it has been suggested that thresholds should progress by between 1.4 and 5 logits for maximal rating scale functionality (Linacre, 1999b).
Item function and person performance validity allows for the examination of whether the instrument was used in the way in which it was intended. If each individual interprets each item in the same way, then data from each of the items and each of the people will show good fit to the model. Rasch models present two forms of fit statistics: infit and outfit statistics (presented as mean squares and standardized scores). Infit or outfit mean squares of >2.0 may distort or degrade the model; mean squares of 1.5–2.0 do not degrade the model but add little to it; mean squares of 0.5–1.5 are helpful in the measurement process; and mean squares <0.5 are not degrading to the model but may lead to inflated reliability or separation statistics (Wright & Linacre, 1994). The “optimal” range for standardized infit and outfit statistics is −2 to 2; however, this statistic is known to be vulnerable to inflation in the context of large sample sizes (Linacre, 2003; Smith, Rush, Fallowfield, Velikova, & Sharpe, 2008).
Good item and person fit to the Rasch model expectations suggests that the construct being measured is unidimensional and that the instrument demonstrates acceptable construct validity (Baghaei, 2008). In this study, the acceptable range of infit and outfit mean squares was set at 0.5–1.5, with particular attention paid to people or items with infit or outfit mean squares >2.0 because they represent particularly erratic scoring and may therefore be at risk of degrading to the model.
Rasch models also allow for the examination of two important features of statistical validity: (1) person and item reliability indexes and (2) the person separation index. Person and item separation indexes are conceptually equivalent to Cronbach’s α and report on the internal consistency of the instrument (Smith, Wakely, de Kruif, & Swartz, 2003; Wright & Stone, 1999). Reliability indexes of >0.81 are considered “good” and those >0.91 are considered “very good” (Fisher, 2007). The person separation index is the capacity of the instrument to stratify people into meaningfully separate groups. Separations should be no less than 2 and separations >3 are considered “good” (Fisher, 2007).
In addition to all of these factors, construct validity is further evaluated through examination of the item–person distribution map. The examination of the item–person distribution map allows for identification of item gaps and ceiling and floor effects. Where the construct is well represented by the measure, the items will demonstrate good distribution across all levels of person ability.
To prepare the MOQ data for analysis using FACETS software, scoring hierarchies were required for each scale. The scales for value were most straightforward: 0 = not at all valuable through 4 = very valuable. Initially, the reason for doing was scaled as 0 = nothing else to do, 1 = wanted to do it, and 2 = had to do it; however, this scaling was soon dichotomized to 0 = nothing else to do and 1 = had a reason for doing it (because the original three categories did not form a logical hierarchy). At the first stage of data organizing and before analysis, an additional activity category of “study” was added because participants frequently reported such activities under the “other” category. Activity categories were scaled according to their work likeness: 0, rest; 1, “chilling”/doing nothing; 2, socializing; 3, playing with children; 4, travel; 5, recreation/leisure; 6, sport; 7, caring for self; 8, housework; 9, caring for others; 10, unpaid work; 11, study; and 12, work.
Results
External Validity
Spearman’s rank-order correlation coefficients between the mean length of time spent in different weekday activities from the MOQ and the TOS06 were as follows:
Fine level of detail: .657 (significant at α = .001)
Moderate level of detail: .863 (significant at α = .001)
Coarse level of detail: .903 (significant at α = .001).
Internal Validity
Rating Scale Validity.
Each rating scale was analyzed individually. Apart from the dichotomized “reason for doing the activity” scale, all rating scales required adjustment.
For the value scales, average measures for categories were ordered correctly but Rasch–Andrich thresholds were disordered, suggesting that some categories may be indistinguishable from one another. In such circumstances, combining or collapsing theoretically similar, contiguous categories is recommended to improve rating scale validity (Bond & Fox, 2007). For further analyses, the scales were collapsed to three points: 0, not at all valuable; 1, of some value (combining a little valuable and somewhat valuable); and 2, of considerable value (combining quite valuable and very valuable). With these collapsed scales, thresholds progressed appropriately. Average measures and thresholds for each category are presented in Table 2.
Rating Scale Statistics for Items
Note. MnSq = mean square.
Reported in log-odds units (logits).
Initial analysis of the activity category scale indicated that numerous categories were ranked close to one another (i.e., the average overall score of people scoring contiguous categories did not change very much) and that several were disordered (i.e., the average score of people scoring contiguous categories did not progress). Before collapsing categories, we removed an 8-hr block of sleep, because the presence of such a large proportion of time spent in one activity appeared to corrupt the integrity of the scale (i.e., caused Rasch–Andrich thresholds to be disordered because the category of rest was modal across a wide range of person abilities). For 126 respondents (55%), this time period was 11:00 p.m. to 7:00 a.m. For 3 respondents (1%), two blocks of sleep were removed (totaling 8 hr), and for 63 respondents (28%), passive activities temporally related to sleep (such as watching TV, browsing the Internet, or getting ready for bed) were removed because they slept for <8 hr. For each respondent, 16 one-hr blocks of activity were included in the final analyses.
The final activity category scale included 4 points: 0, “chilling”/doing nothing; 1, inactive (combining rest and recreation/leisure); 2, active (combining socializing, caring for self, housework, sport, and travel); and 3, worklike (combining care for others, play with children, unpaid work, study, and work). The categorization of recreation/leisure and playing with children may require further explanation. Time spent in recreation/leisure is considered inactive in this context because almost all of this time was spent in screen-dependent pursuits such as watching television or using the Internet. Time spent playing with children was quite infrequent but was most often associated with caring activities (e.g., bathing or babysitting). Using this four-category scale (arrived at through an iterative process of category combination based on the worklikeness conceptualization and reanalysis of the data), the hierarchy was correctly ordered and Rasch–Andrich thresholds progressed monotonically (although the threshold progressions were lower than the recommended 1.4 logits; Bond & Fox, 2007). Results are summarized in Table 2.
Item Function and Person Performance Validity.
All items demonstrated acceptable fit with the Rasch model, as is summarized in Table 3. All item infit and outfit mean square statistics fell within the range of 0.77–1.31, which is considered excellent (Fisher, 2007). Standardized infit and outfit statistics for all items were outside of the recommended range of –2 to 2. These statistics are especially vulnerable to large sample sizes (Linacre, 2003; Smith et al., 2008), and in the context of these analyses, the sample size approached 4,000 (16 observations for each of the 228 participants). Therefore, these figures are not considered concerning because they add little to the understanding of the validity of the instrument.
Statistics for Item Functioning
Note. MnSq = mean square; ZStd = standardized mean square.
Reported in log-odds units (logits).
Person performance validity was also examined. Data from 177 (78%) people fit the model. The data for 12 people (5%) were overfitting (infit or outfit mean square <0.5), and the data for 39 people (17%) were considered erratic (infit or outfit mean square >1.5). Only a small minority (n = 8, 4%) of people had data with infit or outfit mean squares of >2.0.
Statistical Validity.
The item reliability index was 1.00, suggesting excellent internal consistency. The person reliability index was also very good (0.93). The person separation index was 3.52, indicating that the instrument stratified people into approximately five statistically distinguishable groups (according to the equation (4PSI+1)/3; Wright & Masters, 2002).
Construct Validity.
The person–item distribution map (Figure 1) reveals that most of the items fall toward the bottom half of the scale, suggesting that the MOQ captures only information about the basic elements of meaningful time use. This situation creates a risk that people who have higher levels of meaningfulness in their time use may not be effectively evaluated with this instrument.
Person–item distribution map.
Discussion
The MOQ demonstrated good psychometric properties as an instrument for the measurement of the basic elements of meaningful time use, especially in the context of quantitative research projects. In clinical practice, an interview may be required in addition to completion of the MOQ to explore more complex elements of meaningful time use.
High correlations between the MOQ and the TUS06 (at the moderate and coarse levels of detail) suggested strong external validity and that the MOQ is a useful instrument for collecting information about patterns of time use, especially when only moderate detail is required. At the fine level of detail, the MOQ demonstrated insufficient correlation with the TUS06 to be considered valid. When fine detail is required, more time-consuming and expensive methods, such as the Australian Time Use Survey (ABS, 2008b), may be preferable. The loss of validity at this fine level of detail may be related to the 1-hr reporting format of the MOQ or may be caused by insufficient participants in this project to account for all the potential variability in time use across all 223 categories of activity.
In considering the method used to examine the external validity of the MOQ, we should exercise some caution in interpretation. Given the equivalence in age of the samples and the temporal proximity of the measurement points (within 1 yr of one another), it appears likely that our results are accurate reflections of the external validity of the MOQ. However, because the people who completed the MOQ and those who completed the TUS06 were entirely separate samples, there is a chance that the high correlations reported are caused by some spurious factor. Conversely, we cannot definitively say that the actual time-use patterns of participants who completed the MOQ and the 18- to 25-yr-old unemployed people who completed the TUS06 would be similar. Between 2006 and 2007, patterns of time use for young people may have changed enough that the correlations are actually understated.
The MOQ also demonstrated sound internal validity; however, some elements do require further examination: the reasonably large number of people whose data failed to conform to the expectations of the Rasch model; the scale construction for the activity category item; and the lack of “difficult” items to effectively measure people who demonstrate higher levels of meaningful time use. The primary concern in the evaluation of the MOQ’s internal validity is the reasonably large proportion of people who failed to conform to the expectations of the Rasch model. Examination of the patterns of time use for the 39 people whose data were considered erratic revealed that this was most often the case when the person spent large chunks of time engaged in worklike activities (the category deemed most difficult to affirm) and a large chunk of time “chilling”/doing nothing (the category deemed easiest). Such patterns of time use are uncommon for this group of participants but would not necessarily be considered abnormal: People who consider themselves to work hard or work long hours may believe it necessary to “do nothing” to rejuvenate themselves. Further investigation is clearly warranted. In samples that include employed participants, such failure to fit may be less frequent because time-use patterns may demonstrate more variability, creating more flexibility within the model.
In addition, it appears unusual that a 13-point scale such as the activity category scale should need to be collapsed to a 4-point scale to achieve satisfactory item functioning. This situation is likely to be, at least in part, a result of the constricted range of activities undertaken by this group of young unemployed people. Consistent with theories of occupational deprivation (Whiteford, 2000; Wilcock, 1998), most people did not do very much with their day; many stayed inside their homes engaged only in rest, doing nothing, or taking part in screen-dependent activities. If participants had enjoyed a broader range of activities, it might not have been necessary to collapse the scale as far as we did. If the MOQ is used with employed groups, it is likely that the scale will not require such significant simplification.
Finally, consideration should be given to the addition of items that represent more difficult elements of meaningful time use to improve construct validity. In a study of the characteristics of meaningful activity for people with mental illness, Hvalsøe and Josephsson (2003) identified elements such as autonomy, a sense of happiness and satisfaction, future orientation and goal-directed activity, and doing things that are acknowledged by and that promoted a sense of being needed by others. Potentially, the addition of items exploring such elements may improve the construct validity of the MOQ; however, careful consideration must be given to the subsequent increase in respondent burden and to ensure that all items are interpreted in the same way by all participants. It may also be the case that detailed examination of the meaningfulness of a person’s time use is not possible by way of brief quantitative survey.
Conclusion
From these results, the MOQ can be considered a simple and easy-to-use source of information about the basic elements of meaningful time use when studying larger samples. Its ease of use also allows for its application in clinical settings, although it should be used in conjunction with an interview to further explore the more complex elements of meaningfulness.
Further research should explore the use of the MOQ in populations other than unemployed people. Although the sound construct validity demonstrated by the MOQ in this study suggests that the MOQ will be useful across populations, further research, especially with people who are involved in a broad range of activities, may assist in the refinement of item scales and calibration.
Further studies in this project will explore the implications of the ways in which 18- to 25-yr-old unemployed people spend their time, as well as the associations between meaningful use of time and health in this group.
Footnotes
Acknowledgments
We thank Mike Linacre for his expert assistance with data analysis methods, Joshua Teoh for his assistance and diligence in providing independent coding of all MOQ activities according to 2006 Time Use Survey criteria, and all the participants, who so willingly gave of their time.
This research was funded, in part, by a Postgraduate Research Support Scheme grant from the Faculty of Health Sciences, University of Sydney.