Evaluation & the Health Professions

Preventive case management was implemented by Sharp Healthcare of San Diego with the intention of complementing primary care for geriatric patients enrolled in PacifiCare’s Secure Horizons Medicare plan. This article presents patient outcomes after 12 months of participation. The program featured an annual screening questionnaire, appointment monitoring, disease education, and self-management support. It used a prospective design, tracking randomly assigned treatment (n= 1,537) and control patient samples (n = 1,542) for 12 months. Outcomes included physical and mental health status; hospital, ER, and nursing home use; hospital days and expenditures among persons having an inpatient stay; and primary care physician visits. Utilization data were obtained from Sharp Healthcare systems and from screening questionnaires. No statistically significant main effects were found, but persons with three or more independent activity of daily living limitations were about half as likely to have a nursing home admission if they were in case management rather than in the control group.

Subscores resulting fromthe administration of high-stakes tests to candidates for credentials in the health professions are desirable for two reasons. First, failing candidates want a profile of performance to plan future remedial studies. Second, training institutions want a profile of performance for their graduates to better evaluate their training. The validity of the interpretation or use of subscores depends on a summative judgment based on a combination of reasoning and empirical analyses, known as validation. We describe this reasoning process and show that with a large credentialing test the validity of any subscore interpretation or use can and should be studied systematically. Validity evidence should be established to support the interpretation and use of subscores that we intend to report. Some principles arise in this study related to the validity of subscores, and some procedures are proposed to help testing program personnel better validate the use of subscores.

A primary question that must be resolved in the development of tasks to assess the quality of physicians’ clinical judgment is, “What is the outcome variable?” One natural choice would seem to be the correctness of the clinical decision. In this article, we use data on the diagnosis of urinary tract infections among young girls to illustrate why, in many clinical situations, this is not a useful variable. We propose instead a judgment weighted by the relative costs of an error. This variable has the disadvantage of requiring expert judgment for scoring, but the advantage of measuring the construct of interest.

Volunteers (n = 52) and temporary employees (n = 49) at a nonprofit, health care program for the elderly in Australia completed measures on community self-efficacy, sense of community, and caregiver satisfaction and stress. Results indicated that both samples of respondents experienced a relatively strong sense of common mission to help the elderly, but volunteers, compared to employees, reported stronger self-efficacy about making a difference in their community. Volunteers, in comparison to temporary (or casual, the preferred term in Australia) employees, also reported a stronger sense of reciprocal responsibility to help their peers but stronger disharmony among members. Employees compared to volunteers reported greater satisfaction as a caregiver. Implications suggest marked differences in experiences of eldercare among volunteer and temporary employees working with the elderly.

Reporting of incidents or near incidents because of medical devices in French hospitals relies on procedures following European and national guidelines. The authors intend to evaluate hospital staff knowledge on these surveillance procedures as a marker of appropriate application. A telephone survey is conducted on a sample of Paris University hospital staff (n = 327) using a structured questionnaire. Two-hundred sixteen persons completed the questionnaire. The response rate was lower among physicians, especially surgeons paid on an hourly basis. Rates of correct answers were different according to age, seniority, job, and department categories. Physicians and nurses correctly answered questions on theoretical knowledge more often than the other job categories. However, on questions dealing with actual practice conditions, correct answers depended more on age and seniority with a U-shaped distribution (minimum rates in intermediate categories of age and seniority).

To assess the quality of residency education programs at an academic medical center for purposes of enhancing individual graduate medical education programs, we asked residents and fellows (N = 419) to evaluate their training programs using a Web-based questionnaire (response rate = 70%). Kruskal-Wallis tests, factor analysis, correlations, generalizability/decision studies, and mean plots were used to examine trainee responses and to assess the questionnaire’s measurement properties. Exploratory factor analysis indicated that the instrument had a threefactor structure that correlated highly with overall program rating. Cronbach’s alpha exceeded .80 for all factors, and decision studies revealed that 13 to 23 raters were needed to obtain G-coefficients greater than .70. Mean plots showed that the instrument could discriminate within and among training programs at the item level and the factor level, which should help target improvements across graduate training programs within large institutions.