Wanted—Measurement in Home Health Needed

Introduction

Measurement tools are only useful to the extent that they provide accurate information such as a ruler to measure length or a scale to measure weight. Such physical measures like 25 pounds or 105 degrees are easy to imagine given our direct experiences. More difficult to imagine are instructional outcomes such as knowledge about health. Psychological traits such as cognitive impairment or depression become even harder to imagine because these labels are mere abstractions for complex human behaviors and thoughts. Perhaps most challenging to imagine and measure are cultural creations believed to be abstractions of a complex interplay between social, physical, and mental forces. These abstractions are referred to as constructs. Consider, for example, medication risk assessment, a problem both in terms of prediction and consequence. ¹ Why do patients who transition across health care settings—including the home—emerge from their medication experiences with such varied outcomes?

Being able to measure a complex construct like medication risk enables us to discover its relationships with other variables, a process known as validation and considered essential for scientific research in nursing and health care. ² Validation will ultimately explain the construct within a larger context—a theory—in the same manner in which we measured weight and learned its connection to calories. The reliable linkage between scores derived from measuring instruments and predicted outcomes—validation—advances science in all fields. The conclusion that instruments to measure medication management risk lack high quality in clinical practice validation according to reviews^3,4 highlights the need to design validation studies concurrently with instrument development. Accurate measurement of medication risk begins with its presumed components in the same manner as other complex constructs, for example, depression presumed to have its component connections to loss of interest in activities, feelings of hopelessness, and so on.

Content Validity

Validity as a general term in measurement has evolving definitions ⁵ but generally refers to the accuracy of inferences made on the basis of scores. For example, can we infer with accuracy that a specific score on a test of depression tells us that a specific patient will avoid social contacts? Or attempt to end their life? If so, then that test for depression is valid for that specific purpose of prediction. Can a tool for medication risk predict that a specific patient will be rehospitalized due to medication adverse events within 60 days? If so, that tool is valid for this specific purpose. Tools may be valid for one purpose and not another.

Explanation and prediction are ultimate goals of science, hence the need for valid measuring tools that enable prediction. The search for components of medication risk is a process commonly referred to as content validity, aided by statistical indexes that gauge item relevance. ⁶ Content validity may enable accurate prediction if items on the measuring tool accurately reflect the components of the construct being measured. One must ask, “Do the items represent an integral sample of the entire universe of admissible components? Are the items accurate representations of the construct being assessed?” If they are not, then the tool may be measuring something, but not the construct (trait, ability, etc.) that is desired for this purpose. Validation of a tool often begins with a focus on its content (item representativeness) before establishing its ability to make accurate predictions. If predictions are confirmed, a set of relationships among variables can be established and a theory posited (and tested) to explain those relationships. The importance of theory is described by a “clarion call" to the discipline of nursing. ⁷ The first step in this process, however, is the development of an accurate measuring tool so that relationships can be uncovered.

Consider a concrete example. We developed a Medication Management Risk Tool by culling the literature on medication-related problems, specifically those factors believed to be related to medication mismanagement. From a larger pool of potential items, a combination of clinical expertise and literature searches reduced the collection of reasonably admissible items to 20 (e.g., number, novelty, and type of medications prescribed at discharge; patient habits such as use of multiple pharmacies, hoarding medications, and storage procedures in the home; patient capabilities such as vision, swallowing, cognition, literacy, and delivery skills; and patient attitudes such as fear of side effects and health beliefs).

This pool of items that potentially contribute to the measurement of medication risk was administered to 12 health care providers with recognized expertise in the management of hospital to home transitions who rated each item as being essential, useful (but not essential), and not needed. From these assessments, a preliminary tool was developed and edited for clarity by a small sample of literacy experts. Finally, the trial version of the instrument shown in Table 1 will be field tested over the course of 1 year at a large home care agency with an average daily census of about 4,000 home health patients. The tool will be administered at the start of home health services as part of the overall patient assessment process.

OPEN IN VIEWER

Table 1.

SCIC Indicators of High Risk for Medication Management Risk Tool.

Medication regimen risk (conducted in the hospital, at intake or at start of care) □ Twelve (12) prescriptions or more at hospital discharge (1 point) □ Three (3) new medications ordered at hospital discharge (2 points) □ On a high alert medication? (1 point) □ Any one of the high alert medications are the following: Warfarin, Insulin, Plavix and Oral Hypoglycemic agents (2 points) □ An ordered medication route is other than oral form: liquid, injected, inhaled, etc. (1 point) □ An ordered medication is taken every 1-2 hours (1 point) □ An ordered medication has an alternating daily dose/ tapering or increasing (2 points) □ Patient prescriptions are from two (2) or more physicians (1 point) □ None of the above (0 points) ____Sub-total Risky patient behaviors (conducted in the home or MD office or clinic as a brown bag medication review) □ Receives medications from multiple pharmacies (1 point) □ Tendency to hoard or retain medication refills for discontinued medications (1 point) □ Sometimes shares medications with others in the same household (either shares their medications or takes others medications) (1 point) □ No structured way to track medication taking (1 point) □ History of non-adherence or potential for non-adherence (2 points) □ History of drug abuse (1 point) □ None of the above (0 points) ____Sub-total Physical/cognitive limitations that incur risk (conducted in the home or MD office or clinic as a brown bag medication review) □ Swallowing difficulty (1 point) □ Difficulty opening medication bottles or drawing up or injecting medications (1 point) □ Low vision/visual disturbances (1 point) □ Had a fall this year that was possibly medication related (1 point). Refer to STOPP criteria □ English is second language and medication labels are in English (1 point) □ Cognitive deficit (identified in history, or potential via screening) (2 points)  Or, of the medicine chosen to discuss:   • does not knows what to do if misses a dose   • does not know how often to take or how many to take in 1 day □ History of depression or positive screen for depression (1 point) □ Limited literacy based on assessment findings (1 point) □ None of the above (0 points) ____Sub-total ___ Grand total (add up sub-totals from Sections 1, 2, and 3) Instructions Have patient gather all medications in the house to bring to central location (e.g., brown bag to MD appointment). If patient was just discharged from the hospital, ask patient to include all medications taken prior to admission. Reconcile medications found in the house or on the medication list with current medication orders. Note if patient is on any high-alert medications. Review medication labels to determine if any medication is out of date and if they are filled from different pharmacies. Select one medication container—preferably a medication that is taken more than once per day. Ask patient to describe purpose of medication and how often it is taken per day. Ask patient to open cap and place number of pills he or she takes per day in the hand. Have patient self-administer one of the medications, or if not appropriate due to timing, to pour and drink some water from the glass. Ask patient, “How do you keep track of the medicines you take every day?” Ask patient, “What would you do if you forgot to take this medication?” Determine possible non-adherence using ABCD Are you afraid of side effects related to any of your medications? Do you Believe that your medications are helping you? Do you have any concerns about the Cost of your medications? Any other concerns about any of your medications? Screened as possible Depression Note: It is important to ask questions about medication adherence in a “matter of fact” non-judgmental manner so that the patient feels comfortable with this topic. Scoring: 0-27 points possible ____0: low risk ____1-8: moderate risk ____9-16: high risk ____>16: extreme risk Risk level of 9 or greater: Consider medication focused visit for further assessment (occupational therapy, speech, and social work) and medication management education during the first week of care. Select appropriate interventions on care plan using your organization or agency’s care planning tools.

Notes. SCIC = Sutter Center of Integrated Care; STOPP = Screening Tool of Older Persons’ Potentially Inappropriate Prescriptions

These data will be analyzed to understand how the items converged on meaningful dimensions for the purpose of reducing the tool into a short form. For example, perhaps patient characteristics such as cognitive deficits, literacy, language, vision, or history of falls all measure the same basic trait, but 1 item in particular might measure it best according to the statistical analysis. If so, that 1 item (e.g., literacy) alone can be used as a single item to measure that scale. Another item, perhaps number of new medications, alone could be used to tap another scale such as medication complexity within the larger tool. Many measurement tools achieve their predictive power via different but conceptually related scales that total into a composite score. The ultimate goal is to identify items that comprise relatively independent scales, each scale providing information about the multidimensional construct in an additive manner. If possible, a single item representing each scale most reliably could be used to create a practical short form of perhaps 3 to 4 items to measure the whole construct. Some fields (e.g., psychology, education) develop scales that contain perhaps 50 or more items; yet very short forms (even 2-3 items) are desirable in health care, especially when used as screening tools.

Prediction and Theory

Once content validity is assured—the items in fact do represent a larger universe of potentially admissible items that tap the essence of the construct—then attention is directed to predictive validity. Do scores on the instrument truly predict who is at greatest risk of medication mismanagement and the resultant problem of rehospitalization or some other outcome measure? The entire focus during this phase of instrument development is assuring that predictions based on scores are generally accurate. The prediction of a negative outcome, of course, raises “red flags” so that appropriate steps can be taken to avoid the negative consequence, in this case seeking alternatives to risky medications or changing the environmental circumstances linked to negative outcomes. We are exploring options to assess this tool’s predictive validity at our agency. One option is correlating scores with the number of individual patient medication adverse events. Another option is correlating scores with the number of hospital readmissions due to a medication adverse event. These options presume that patients scoring in the high-risk range will experience more medication management difficulty, whereas low-risk patients will experience fewer adverse events.

When predictive validity is established, there is a need to explain the phenomenon of medication risk in a larger context, hence the need to develop a theory to explain findings. Relationships hypothesized by the theory created to explain the construct are tested and supported or disproven, often called construct validity. The theory of medication risk, like any other theory of risk (e.g., school drop-out risk), may be depicted by a conceptual model (theoretical framework) for the purpose of easing communication. Geometric shapes showing linkages are common and especially useful for suggesting which factors require testing to support the underlying theory. The theory may include personal factors (e.g., depression and health literacy), social (e.g., family and support), environmental (e.g., integration of health care providers), cultural, economic, or whatever influences are deemed to be most useful and potent for the purpose of explaining outcomes. A good model suggests hypothesized relationships (e.g., often shown as arrows) that need testing for support. A poor model would be ambiguous and not amenable to empirical testing. A good scientific theory of medication risk should be “fertile” in the sense that it gives rise to testable hypotheses. But it must also be capable of being disconfirmed.

One mark of potential scientific value of any measure is its dependability as well as its validity. Dependable measures (e.g., bathroom scale) are often referred to as trustworthy in qualitative research or reliable in quantitative research. Reliability is commonly assessed over time, called test–retest and across items within a scale, called internal consistency. They can also be checked across difference versions of the same instrument, Form A and Form B, commonly referred to as parallel forms. The essence of reliability is consistency, and many constructs are presumed to be stable (hence, consistent or reliable). This is especially true in psychology (e.g., personality measures) and education (e.g., vocational interests). But reliability may not be the mark of a good measure if the underlying construct is truly dynamic and marked by constant changes (e.g., mood).

Little is known about medication risk in the context of a larger body of knowledge other than findings that reveal costly, common, and preventable medication-related problems, especially in the geriatric population. ⁸ Yet one reasonable conclusion is that people are “at risk” of something whenever medication is ordered, and the older the patient, the greater the risk and the more difficult the challenge. It is clear that we do not have sufficient knowledge to make accurate predictions at the individual level with regard to potentially inappropriate medicines. We need to know who benefits the most and who suffers the most from the same array of medications and its protocol. This level of knowledge may come from theory, itself the result of measurement, validation, and the search for relationships. As noted by esteemed psychologist Kurt Lewin and paraphrased, “Nothing is as practical as a good theory.”

Conclusion

For any theory to influence a complex practice such as home health, it must have credibility that derives from solid empirical evidence, not anecdote. Theories are supported, hence made credible, by empirical testing that begins with measurement and follows guidelines that assure conclusions are supported by the scientific research process and sound, critical reasoning. ⁹ When evaluated against the standards of validity and reliability, the measuring instruments can be used to test hypothesized relationships that help solve the medication risk puzzle—and many other puzzles—by evaluating its explanatory theory. Once we explain why home health patients are distributed so widely on a medication risk scale (continuum), we can use that knowledge to improve the health outcomes of all patients.

One confounding variable affecting the ability to establish predictive validity is the inability to blind nurses to initial risk assessments, likely leading to more interventions in the high-risk group hence mitigating risk, the desired action in the long term. Establishing validity poses practical challenges when conducting scientific research.Yet proper planning of data collection and critical thinking about the value of measurement in an informal research project may ultimately lead to higher quality of care in ways that circumvent the strict requirements of the scientific method (e.g., blinding, control groups, random assignment, etc.). Following sound principles of measurement facilitates evidence-based care delivery. As a sector, home health must make data-driven decisions and apply sound principles of reasoning if we continue to be valued by our stakeholder partners and patients for the provision of high-quality health care.