Validation of a Cystic Fibrosis Medication Knowledge Questionnaire

Abstract

Low adherence to cystic fibrosis (CF) treatment is associated with poor health outcomes, while knowledge of the disease and medication regimen can positively influence adherence. This study’s purpose was to develop and validate a questionnaire to help determine CF medication knowledge of pediatric patients and caregivers. Our questionnaire had 37 items: 22 selected-response and 15 open-response questions. We used validation processes from the Standards for Educational and Psychological Testing. CF experts analyzed validity evidence based on content. Then, the questionnaire was field tested with 17 pediatric patients and 18 caregivers. The correlation between age and medication knowledge was medium (r = .33), but was not significant (P = .189). Cronbach’s α for the overall test was .84. Participants thought the questionnaire was important and suitable, with a few minor suggestions to improve wording. Strong validity evidence indicates the questionnaire could be used to assess medication knowledge and allow more personalized education to improve adherence.

Keywords

validity validation pediatrics cystic fibrosis

Introduction

Cystic fibrosis (CF) is a “chronic, terminal illness that primarily affects the lungs and pancreas.”¹ Treatment is daily, burdensome, and time-consuming,² which can create a challenge for patients with CF and their caregivers to adhere to treatment regimens. Adherence is described as the “extent to which a person’s behavior (in terms of taking medications, following diets, or executing lifestyle changes) coincides with medical or health advice.”³

Adherence to medications in young, chronically ill patients is a long-standing concern in the health care community.^4,5 This is particularly true for patients with CF, whose treatment is intense and complex.^4-9 Medication adherence estimates differ across various reports and studies. Some estimates range from 48% to 63%,^10,11 while others range from 35% to 68%.¹² The low rate of adherence includes both pulmonary and gastrointestinal/dietary therapies. Poor adherence to these therapies can lead to increased hospital admissions, decreased pulmonary functioning, and shorter life spans.^6,13,14 In this era of new treatments and improved lifespans for patients with CF,^9,15 the importance of adherence cannot be underestimated.

Two factors that influence adherence include knowledge of the disease and knowledge of the characteristics of the regimen.⁴ These understandings have been identified as important for both patients and their caregivers, and some evidence suggests that greater knowledge is associated with higher adherence.^4,13,16-19 Furthermore, Mameniskiene et al²⁰ reported that lack of knowledge is associated with poor health outcomes and increased mortality. There are other factors that can contribute to low adherence, such as “complex regimens, negative side effects, and inconsistent efficacy,”⁴ but the focus of this study is on patient and caregiver knowledge of CF medications.

It is not just adult patients who have been shown to have adherence issues but also pediatric patients under the age of 18 years, many of whom rely on their caregivers for help with their medications. Thus, adherence issues begin at a young age, and it is while patients with CF are still children that they need to become knowledgeable about their medications so they can gradually become more independent as they mature into adolescents and adults. Much of the education regarding medications is conducted with caregivers, who then become responsible for educating their children. However, it is not uncommon for young adults with CF to lack the necessary medication knowledge they need to maintain their health,^21-23 indicating caregivers may need more support with educating their children about their medications and regimens.

It is important to assess how well young patients with CF and their caregivers understand their medications and the reasons why they are supposed to take them as a preliminary step to increasing therapy adherence. Questionnaires are available that determine measures of adherence to medications and barriers to medication adherence.^7,19 However, there is limited evidence of questionnaires to determine the medication knowledge of patients with CF or of their caregivers. The purpose of this study was to develop and validate a CF medication questionnaire that focuses on young patient and caregiver knowledge of the medication regimen for CF. The questionnaire will provide important information for health care professionals to enable them to improve the education provided to patients with CF and their caregivers. This, in turn, will help the CF population understand the contribution their medications can make to the enhancement of their lifestyles and to their prolonged health, and will hopefully strengthen adherence.

Method

This research was reviewed and approved by the provincial Health Research Ethics Board, an affiliated board with Memorial University. All participants gave informed consent.

The validation process examines and measures whether the instrument does what it purports to do. Validity does not reside in the instrument. Instead, validation rests with the inferences that are made from the results of the test, or in this case, the questionnaire.^24-26 For this study, we intended to validate that the inferences made about the medication knowledge of young patients with CF and their respective caregivers, from their responses to the questionnaire, were warranted.

Questionnaire Development

We developed a questionnaire intended to address 2 concerns noted in the adherence literature: medication knowledge and characteristics of the medication regimen. First, a literature search on CF questionnaires confirmed there was a need for a questionnaire on medication knowledge for children with CF and their caregivers. Then, 2 members of the research team, a pharmacist and a pediatrician, both of whom work in CF clinics, identified the fundamental concepts about CF medications that should be included in the questionnaire. These concepts were airway clearance therapy, nutritional supplements, pancreatic enzymes, airway openers, mucous thinners, antibiotics, and other medications. Next, an education specialist worked with the 2 health professionals and an additional health professional to design the questionnaire. Details of the validation process are reported in this article as part of the results.

The team determined that a mixture of selected response, fill in the blank, and short answer questions would be best suited to determine the medication knowledge of patients under 18 and of their caregivers. Several preliminary drafts were created and discussed by the team, and then distributed to other professionals as part of validation.

Two versions of the questionnaire were piloted. The first was a patient version that CF patients 12 years of age and older should be able to complete independently. This same version can be administered to CF patients who are younger than 12 years, with an adult reading the questionnaire to them and explaining terminology, as necessary. The second version was a caregiver version, with the same questions as the patient version, but slightly modified for the caregiver.

There were 37 items: 22 selected response and 15 open response questions. The first 3 questions asked demographic information. The next set of questions asked participants to list all their medications, their medication regimens, and why they were taking these medications. Following these questions, there were specific sets of questions that were related to the previously indicated concepts (see Table 1 for sample; graphics and medication picture cards are not included). It was not considered necessary for all participants to complete all the questions in the questionnaire. For example, question 10 in the nutritional supplements section asked if the patient took supplements on a regular basis. If the response was no, then the participant would skip to question 13 in the pancreatic enzymes section.

Table 1.

Summary of 5 Sources of Validity Evidence From the 2014 Standards.

Source of Evidence	Description
Evidence based on test content	The content of the test/questionnaire should align with the construct being measured and should be appropriate to the interpretations made from the results. Experts judge the themes, wording, format, and administration guidelines. Construct representation, construct relevance, and scoring are examined.
Evidence based on response processes	Assumptions are made about the cognitive processes required. For example, if items are designed to assess problem solving, then it should be determined if test/questionnaire takers have to problem solve to respond to the item or if they simply have to remember the solution.
Evidence based on internal structure	The relationships among test/questionnaire components and test/questionnaire items should conform to the construct being measured. A test/questionnaire may have a single dimension to be measured or may have several components that are distinct from each other.
Evidence based on relations to other variables	If the intended interpretation indicates a relationship of the construct with other variables, then there should be a relationship with test/questionnaire results and the other variables. For example, there should be convergent evidence for the relationship between licensure exam results and job performance. High exam results should be expected to predict good job performance.
Evidence for validity and consequences for testing	Consequences of test/questionnaire use may or may not be relevant to validity. For example, if the test/questionnaire users make an inappropriate decision based on the results and there was construct irrelevance, then this can be a source of invalidity. For example, if a patient could not read the questionnaire, the physician thinks the patient did not know how to do his/her therapy, and the physician recommends patient education on airway clearance therapy. This would be a negative consequence that is a validity issue.

Once the team was satisfied with the content and wording of questions, a graphic artist created a child-friendly version of the questionnaire. This version had illustrations, color, and large print. A caregiver version was also created, as well as picture cards of the medications.

Participants and Procedure

At the time of the study, there were only 18 patients with CF in Newfoundland and Labrador who were younger than 18 years of age and old enough to participate in our study, so it was not possible to have more patient participants. Based on the knowledge and experiences of the educational specialist and pediatrician, patients younger than 5 years old were not considered mature enough to participate. We approached the 18 patients and their caregivers when they came to the CF clinic for consent to pilot the questionnaire. Seventeen patients and 18 caregivers agreed to participate (n = 35). Seven patients ranging from 5 to 10 years of age and 10 patients ranging from 13 to 17 years of age participated. The education specialist read the questionnaire to each patient and audio-recorded the procedure. The questionnaire was read to all the patients because some patients could not read: some were young and others had reading difficulties. Thus, reading to all the patients provided consistency of administration. The medication picture cards were used with all the patients.

Two of the health care professionals administered the questionnaire to the caregivers, who completed the questionnaires independently. Caregivers were also provided with the medication picture cards.

After each questionnaire was administered, each patient and caregiver was asked questions about the suitability of the questionnaire, including questions about content, appropriate language, clarity, and formatting. See Table 2 for interview questions. The interviews were transcribed, and how the participants responded about their medications was compared to the pertinent patient medication profiles for accuracy. This helped determine if the participants understood the questions. For example, if patients or caregivers consistently answered a particular medication question inaccurately, it may have been due to the wording of the question rather than lack of medication knowledge.

Table 2.

Interview Questions That Followed the Administration of the Questionnaire.

Now I have a few questions for you about the questionnaire. There are no right or wrong answers, and the questions are about what you think. The questions are meant to help make the questionnaire better for other people.
1. What did you like about this questionnaire?
2. What did you not like about this questionnaire?
3. What is one thing, or more, we can do to make the questionnaire better?
4. Do you think the questionnaire is too long, too short, or just right?
5. What do you think of the pictures of the medications?
6. What about the words we used—were they okay or were some of them too hard to understand?
7. Are there any other questions you think we should ask on the questionnaire?
8. Do you think it is important that patients know a lot about their medications? Why do you think this?
9. Do you think it is important that parents/guardians/caregivers know a lot about their children’s medications? Why do you think this?
10. Is there anything else you would like to add?

We used the most recent Standards for Educational and Psychological Testing²⁷ for our validation processes. These validation procedures are commonly used in both the social and health sciences, as were earlier versions of the Standards.^24-26 These processes provided evidence to support the inferences and conclusions that were made from the results of the questionnaire. We used evidence based on the content of the questionnaire to determine if the concepts, wording, and formatting were appropriate to the construct being measured. This included reviewing the questionnaire for construct representation, determining face validity (wording and formatting that is appropriate to young learners), and examining construct irrelevance by checking to see if the responses of the participants were influenced by irrelevant factors such as the reading level of the questions or the clarity of the administration procedures. We also analyzed the internal structure of the questionnaire to determine the relationship among the items in each section, and for the overall questionnaire (reliability).

Evidence Based on Content

Construct Representation

This was established through consultation with experts on CF. One of the investigators for this study is the pharmacist affiliated with both the Pediatric and Adult Cystic Fibrosis Clinics. He is also a pharmacy professor with the university and a researcher in CF. One of the other investigators is a pediatrician and Clinic Head of a Pediatric Cystic Fibrosis Clinic. The third content expert is a pharmacy professor. The questionnaires were also sent to a pediatrician in another province who specializes in CF.

These experts studied the questions for construct representation, which is how well the breadth and depth of CF medication knowledge is represented in the questionnaire (see Table 3 for sample questions from the questionnaire). They also studied the word choice for accuracy. In addition, the CF clinic team at the hospital where we conducted the study was given the questionnaires to validate for content. The team included a doctor, nurse practitioner, dietician, physiotherapist, and social worker, all of whom specialized in CF. After each team member had the opportunity to study the questionnaires individually, we met with the team as a group to discuss their thoughts and address any concerns.

Table 3.

Sample Items From Patient Questionnaire (Graphics Not Included).

SECTION C
The following questions tell us what you know about your CF medications.
Nutritional Supplements:
10. Do you take any supplements on a regular basis? ◻Yes ◻No If no, skip ahead to question 13.
11. Check all the supplements you take. Check how often you take them.
	Once a Day	Twice a Day	Three Times a Day
AquADEKs ◻
Centrum Forte ◻
Flintstones ◻
Vitamin D ◻
Vitamin E ◻
Iron ◻
Calcium ◻
12. What is the most important reason why patients with CF take supplements? Please answer in the box.

Face Validity

The education specialist and pediatrician edited several drafts, reviewing the educational literature on reading, and examining numerous questionnaires for children as part of the development and editorial process. The patients and caregivers were also asked questions in the follow-up interviews about the clarity of the language and the formatting.

Construct Irrelevance

This was determined in 3 ways. First, the ages of the patients ranged from 5 to 17 years of age so not all would be able to read the questionnaire independently, and some of the older patients had diagnosed reading difficulties. To ensure that reading did not interfere with the patients’ comprehension of the questions, the education specialist read the questionnaire to all patients. This was not necessary for the caregivers as all of them were able to read the questionnaire independently. Second, during the interviews the participants were shown the medication picture cards. This was to strengthen the inferences we made about patient and caregiver knowledge of their medications as we did not want the names of the medications to be the only indication of medication knowledge. Third, we used SPSS to correlate the ages of the patients with their ability to answer the questions about their medications to determine if age was a factor in comprehending the questionnaire.

Face Validity and Construct Irrelevance

We developed administration guidelines for the questionnaire. These guidelines were examined to ensure the reading level and clarity of instructions was appropriate for caregivers so they could independently self-administer the questionnaire. The administration guidelines also ensured consistent administration for all participants, so that how participants completed the questionnaire did not interfere with participant comprehension of the questionnaire.

Evidence Based on Internal Structure (Reliability)

We used SPSS to determine the internal structure of each of the medication knowledge scales and the overall test, using Cronbach’s α. In addition to questions in each section about medication knowledge, each participant was asked why it was important for patients with CF to take the type of medication that was the focus of the scale. We determined the internal structure of these questions across all the scales.

Results

Evidence Based on Content

Construct Representation

The content experts confirmed that the questions were appropriate for content and wording. All agreed that the questions were medically accurate and measured knowledge of CF medications and medication regimens. This strengthened the argument that construct representation was strong. Underrepresentation was not a problem—all the important concepts of the construct of CF medications were addressed in the questionnaire.

Face Validity

Prior to the piloting, the education specialist and pediatrician agreed that the wording and formatting were appropriate to the age range. During the pilot, most caregivers said the words were “fine,” although 3 of them mentioned that the term “nutritional supplements” was not comprehensive and suggested alternate terms, such as “vitamins.” The only other term that was mentioned by 2 of the caregivers was “airway clearance therapy”; they recommended the term “physio.” Most of the patients did not recognize the term airway clearance therapy. They suggested using physio, therapy, or exercises.

Patients and caregivers thought the graphics on the patient questionnaire would appeal to a range of ages. Some of the patients mentioned that the graphics made the questionnaire “fun” and “not boring.” Patients and caregivers alike thought the length of the questionnaire was “just right.” All participants made positive comments about the formatting of the questionnaire. Thus, face validity was deemed to be strong.

Construct Irrelevance

Due to reading the questionnaire to the patients, reading did not interfere with comprehension of the questions, thus reducing construct irrelevance. Three of the caregivers did not use the medication picture cards or comment, but the remaining 14 thought they were helpful for identifying the medications. Some of the patients thought the names of the medications were “hard,” but all the patients believed the picture cards helped them, and one patient’s comment represented many of their thoughts, “I think the pictures really helped, because sometimes I wouldn’t recognize the name of the medication.” One of the adolescent patients who had a reading difficulty expressed, “It helped me a lot because the way I learn it, I got to see the pictures and see what it looks like so it might help other people like me.” Thus, the medication picture cards helped so that patients who were young or had reading difficulties were not disadvantaged, which decreased construct irrelevance.

There was a medium correlation r = .33 between the ages of the patients with their knowledge of their medications, but it was not significant P = .189. All the caregivers were adults so we did not consider age a factor for the caregivers. Thus, neither reading nor age interfered with understanding of the questionnaire, reducing construct irrelevance.

Face Validity and Construct Irrelevance

The administration guidelines were considered to be comprehensible by the authors and the clinic team, thus strengthening face validity and decreasing construct irrelevance. The directions would be appropriate for the caregivers to use independently, but not appropriate for independent use by children. However, future use of the instrument would most likely be in clinical settings where young patients would have the questionnaire administered to them.

Evidence Based on Internal Structure

The internal structure was acceptable for the questionnaire. Cronbach’s α ranged from .70 for the section on pancreatic enzymes to .86 for the section on antibiotics (see Table 4). Cronbach’s α for the overall test was .84, indicating that generally the questions were suitable for the purpose of the questionnaire.

Table 4.

Internal Structure for Each of the Scales.

Scale	Cronbach’s α
Nutritional supplements	.75
Pancreatic enzymes	.70
Airway openers	.76
Mucous thinners	.82
Antibiotics	.86

Cronbach’s α for patient and participant knowledge of the importance of the different medications was .693, which was lower than the scales and overall test, but considered acceptable for newly created scales.²⁸

Discussion

Validity is more than a measurement principle, it is a social value that has meaning whenever evaluative judgements and decisions are made.²⁹ This comprehensive view of validity combines scientific inquiry with logical argument to justify interpretations that are made from tests, questionnaires, or any forms of assessment scores and results.^29,30 Our study indicates strong validity evidence, using both quantitative measurement and empirical argument to support the inferences that can be made from the results of the questionnaire we developed.

The responses from the participants about their medications and regimes were verified by the CF team, strengthening our conclusion that the patients and caregivers understood the questions. Hence, the questionnaire could be used to assess patient and caregiver CF medication knowledge. This utilizes evidence of validity based on consequences, allowing for more robust and personalized educational interventions to improve adherence.

Patients and caregivers consistently made positive comments regarding content, organization, and length. The participants also found the medication picture cards helpful. These qualitative results strengthen validity evidence based on content.

The participants were asked to suggest additional questions for the questionnaire, but none of the participants had suggestions. Participants were asked for general ideas to improve the questionnaire, such as content changes or additions, formatting, and vocabulary. The only suggestions have already been mentioned, such as changing the term “airway clearance therapy” to “physio.” Some caregivers mentioned that their children had reading difficulties, and they were concerned about them being able to read the questions, but for young children and those with reading difficulties, the questionnaire should be administered by an adult or another appropriate person.

Limitations of this study include the small sample size, due to the small numbers of patients with CF in our province. This limited the quantitative analysis that could be conducted, such as factor analysis; however, the qualitative validation is strong. Our study could be replicated in other jurisdictions to increase the quantitative component and confirm or supplement our findings. A second limitation is that although we interviewed each of the participants, we could not conduct focus groups to see if a group discussion would have triggered some of the participants to have additional thoughts about the questionnaire. Due to their CF, the patients and caregivers could not be in the same room. Also, they lived in different areas of the province, and travel and time away from school and work was required for many of them to attend clinic, thus reducing the amount of time they could spend with us.

Poor adherence to medical regimens for patients with chronic conditions has been cited as “the single, greatest cause of treatment failure.”⁶ It can lead to increased morbidity, increased mortality, and reduced quality of life.^6,13,14,20 Studies of patients with CF, particularly children, have consistently shown adherence rates of less than 50%. It is an acknowledged, serious, and persistent problem. Recommendations have been made that medication knowledge of patients and their caregivers should be assessed as one of the ways to address adherence.⁷ Thus, this questionnaire will help fill this gap in the literature on CF.

Consequences of such a questionnaire could help improve adherence and result in improved health and quality of life for patients with CF. Future recommendations for research would be to assess the use of the questionnaire with patients and caregivers as a tool to develop an education plan for young patients.

Author Contributions

BF: Contributed to conception and design; contributed to acquisition, analysis, and interpretation; drafted manuscript; critically revised manuscript; gave final approval; agrees to be accountable for all aspects of work ensuring integrity and accuracy.

JH: Contributed to conception and design; contributed to acquisition; critically revised manuscript; gave final approval; agrees to be accountable for all aspects of work ensuring integrity and accuracy.

MJS: Contributed to conception and design; contributed to acquisition; critically revised manuscript; gave final approval; agrees to be accountable for all aspects of work ensuring integrity and accuracy.

TL: Contributed to design; contributed to acquisition; critically revised manuscript; gave final approval; agrees to be accountable for all aspects of work ensuring integrity and accuracy.

Footnotes

Acknowledgements

Caroline Cashin and Jennifer Voisey.

Authors’ Note

Poster presented at the North American Cystic Fibrosis Conference, Phoenix, Arizona. Abstract published as a supplement to Pediatric Pulmonology.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Research Grant provided by the School of Pharmacy, Memorial University of Newfoundland.

References

Quittner

Tolbert

Regoli

Orenstein

Hollingsworth

Eigen

Development of the role-play inventory of situations and coping strategies for parents of children with cystic fibrosis. J Pediatr Psychol. 1996;21:209-235.

Arias Llorente

Bousoño García

Díaz Martín

JJ.

Treatment compliance in children and adults with cystic fibrosis. J Cyst Fibros. 2008;7:359-367.

Haynes

. Introduction. In: Haynes

Taylor

Sackett

eds. Compliance in Health Care. Baltimore, MD: John Hopkins University Press; 1979:1-7.

Modi

Quittner

AL.

Barriers to treatment adherence for children with cystic fibrosis and asthma: what gets in the way?

J Pediatr Psychol. 2006;31:846-858.

Rapoff

MA.

Adherence to Pediatric Medical Regimens. New York, NY: Kluwer Academic/Plenum; 1999.

Bauman

Wright

Leickly

. Relationship of adherence to pediatric asthma morbidity among inner-city children. Pediatrics. 2002;110(1 pt 1):e6.

Quittner

Modi

Lemanek

Ievers-Landis

Rapoff

MA.

Evidence-based assessment of adherence to medical treatments in pediatric psychology. J Pediatr Psychol. 2008;33:916-936.

Gudas

Koocher

Wypij

Perceptions of medical compliance in children and adolescents with cystic fibrosis. J Dev Behav Pediatr. 1991;12:236-242.

Sawicki

Goss

CH.

Tackling the increasing complexity of CF care. Pediatr Pulmonol. 2015;50(suppl 40):S74-S79.

10.

Eakin

Bilderback

Boyle

Mogayzel

Jr Riekert

KA.

Longitudinal association between medication adherence and lung health in people with cystic fibrosis. Am J Respir Crit Care Med. 2011;10:258-264.

11.

Quittner

Zhang

Marynchenko

. Pulmonary medication adherence and healthcare utilization in cystic fibrosis. Chest. 2014;146:142-151.

12.

Shakkottai

Kidwell

Townsend

Nasr

SZ.

A five-year retrospective analysis of adherence in cystic fibrosis. Pediatr Pulmonol. 2015;50:1224-1229.

13.

Balfour

Armstrong

Holly

. Development and psychiatric validation of a cystic fibrosis knowledge scale. Respirology. 2014;19:1209-1214.

14.

O’Donohue

Fullen

BM.

Adherence of subjects with cystic fibrosis to their home program: a systematic review. Respir Care. 2014;59:1731-1746.

15.

Riekert

Eakin

Bilderback

Ridge

Marshall

BC.

Opportunities for cystic fibrosis care teams to support treatment adherence. J Cyst Fibros. 2015;14:142-148.

16.

Basketter

Sharples

Bilton

Knowledge of pancreatic enzyme supplementation in adult cystic fibrosis (CF) patients. J Hum Nutr Diet. 2000;13:353-361.

17.

Ievers

Brown

Drotar

Caplan

Pishevar

Lambert

RG.

Knowledge of physician prescriptions and adherence to treatment among children with cystic fibrosis and their mothers. J Dev Behav Pediatr. 1999;20:335-343.

18.

Bandura

Health promotion by social cognitive means. Health Educ Behav. 2004;31:143-164.

19.

Simons

Blount

RL.

Identifying barriers to medication adherence in adolescent transplant recipients. J Pediatr Psychol. 2007;32:831-844.

20.

Mameniskiene

Sakalauskaite-Juodeikiene

Budrys

People with epilepsy lack knowledge about their disease. Epilepsy Behav. 2015;46:192-197. doi:10.1016/j.yebeh.2015.03.002.

21.

Siklosi

Gallagher

McKone

EF.

Development, validation, and implementation of a questionnaire assessing disease knowledge and understanding in adult cystic fibrosis patients. J Cyst Fibros. 2010;9:400-405.

22.

Nolan

Desmond

Herlich

Hardy

Knowledge of cystic fibrosis in patients and their parents. Pediatrics. 1986;77:229-235.

23.

Conway

Pond

Watson

Hamnett

Knowledge of adult patients with cystic fibrosis about their illness. Thorax. 1996;51:34-38.

24.

Kane

. Validation. In: Brennan

, ed. Educational Measurement. 4th ed. Westport, CT: American Council on Education & Praeger; 2006:17-64.

25.

Messick

Validity. In: Linn

, ed. Educational Measurement. 3rd ed. New York, NY: Macmillan; 1989:13-103.

26.

Sireci

. Packing and unpacking sources of validity evidence: history repeats itself again. In: Lissitz

, ed. The Concept of Validity: Revisions, New Directions, and Applications. Charlotte, NC: Information Age; 2009:19-37.

27.

American Education Research Association, American Psychological Association, & National Council on Measurement in Education. Standards for Educational and Psychological Testing. Washington, DC: American Educational Research Association; 2014.

28.

Nunnally

JC.

Psychometric Theory. New York, NY: McGraw-Hill; 1978.

29.

Messick

Standards of validity and the validity of standards in performance assessment. Educ Meas. 1995;14(4):5-8.

30.

Downing

Haladyna

. Validity and its threats. In: Downing

Yudkowsky

, eds. Assessment in Health Professions Education. New York, NY: Routledge; 2009:21-55.