National Role Delineation Study of the Board Certification for Advanced Diabetes Management

Abstract

Purpose

The purpose of this comprehensive role delineation study of the Board Certification for Advanced Diabetes Management (BC-ADM) examination was to ensure its content validity.

Method

The comprehensive role delineation study was conducted in 3 phases. In the first phase, a panel of 12 currently credentialed BC-ADM subject matter experts produced a list of practice domains, a unique set of content subdomains for each, and a series of knowledge statements for every subdomain. In the second phase, a validation study survey consisting of 3 sections was created. The first section was to rate each subdomain on three factors: criticality, frequency, and point in career at which knowledge is first used. The second section asked respondents to estimate the percentage of the examination to dedicate to the four domains. The third section captured demographic information of the respondents. A total of 667 BC-ADMs and 18 physicians were invited to take the survey. In the third phase, the subject matter expert panel analyzed the survey results and determined the weight that the domains and subdomains should have on the 150-item BC-ADM examination.

Results

Final domain weights and the corresponding number of items for the BC-ADM examination are provided in this report; these constitute the BC-ADM examination blueprint.

Conclusions

The national role delineation study reported here for the BC-ADM examination ensures that it is reflective of the current practice and required knowledge of the advanced diabetes manager.

History of BC-ADM

Board Certification for Advanced Diabetes Management (BC-ADM) has been an advanced practice certification recognized by stakeholders in the diabetes community since its inception in 2001.^1-3 The American Association of Diabetes Educators (AADE) and the American Nurses Credentialing Center (ANCC) jointly sponsored the BC-ADM examination from 2001 through 2010. The BC-ADM examination was developed and administered as 4 separate examinations to clinical nurse specialists and nurse practitioners, registered dietitians, and registered pharmacists.⁴ Any applicant who obtained the initial BC-ADM credential sat for the examination under the authority and standards of the ANCC/AADE.

In 2009, a new combined examination was designed, converting the BC-ADM into a single examination for all 4 roles. When the examination was combined in 2009, a comprehensive role delineation study was performed on the roles of clinical nurse specialists, nurse practitioners, registered dietitians, and registered pharmacists by Parsa International, Inc (Annandale, Virginia) on behalf of the ANCC and AADE. The BC-ADM examination was found to be appropriate and applicable to each role. In 2010-2011, the AADE purchased the certification from the ANCC, becoming the sole managing body. Following this purchase, the first candidates sat for the examination under the management of the AADE through its contract testing vendor, Castle Worldwide, Inc, in June 2011. In 2012, the BC-ADM Oversight Committee made the decision to add physician assistants, medical doctors, and doctors of osteopathic medicine to the group of qualified candidates. This decision—in combination with the passage of several years since performance of the last role delineation study—necessitated performance of a comprehensive role delineation study.

Comprehensive Role Delineation Study for BC-ADM

In February 2014, the AADE assembled a panel of subject matter experts (SMEs) to begin the comprehensive national role delineation study for the BC-ADM examination. The purpose of a role delineation study is to identify the critical components of work in a particular role, profession, or occupation, as well as the knowledge and skills required to perform that work.⁵ Best practice and professional testing standards dictate that the content of certification examinations be linked to the current practice of qualified entry-level testing candidates.⁵ Conducting a role delineation study is essential for ensuring that a certification examination demonstrates content validity—that is, that the content of the examination is reflective of contemporary practice.^5-9 With that goal in mind, a panel that was representative of the BC-ADM certificant population was tasked with developing a set of content domains, subdomains, and knowledge statements associated with the BC-ADM role. These content domains formed the national survey, which ultimately provided evidence to establish the latest weighted test content outline, or “blueprint,” for the examination. This article outlines the process of conducting the role delineation and reports the findings from the study. Furthermore, it presents the new BC-ADM examination blueprint.

Method

Phase 1: Defining the BC-ADM Practice

In the first of the 3 phases of the role delineation study, a panel of 12 SMEs met to delineate the role of the BC-ADM and identify the essential knowledge required for competent entry-level practice. The panel was multidisciplinary, included newly certified and experienced BC-ADMs, and was geographically and demographically representative of the BC-ADM population (Table 1). This meeting was facilitated by a psychometrician from Castle, the testing agency responsible for managing and delivering the BC-ADM examination. The panel began by reviewing and refining the target audience statement and the eligibility requirements—an important first step that ensures that all panelists have an understanding of the scope of practice of the BC-ADM as well as the minimum level of education and experience required to obtain the certification. The panel determined that the BC-ADM target audience statement should read as follows:

The BC-ADM skillfully manages complex patient needs and assists patients with therapeutic problem-solving. Within their discipline’s scope of practice, healthcare professionals who hold the BC-ADM certification adjust medications, treat and monitor acute and chronic complications and other comorbidities, counsel patients on lifestyle modifications, address psychosocial issues, and participate in research and mentoring.

Table 1

Panel of Subject Matter Experts

Sex	Discipline	Location
Female	PA	Arizona
Female	RD	South Carolina
Female	RN, APN	Louisiana
Female	RN, APN	New York
Male	RPh	Utah
Female	RPh	Virginia
Female	RPh	Illinois
Female	RD	North Carolina
Male	RPh	West Virginia
Female	RD	Colorado
Female	RN, APN	Texas
Female	RN, CNS	Texas

Abbreviations: APN, advanced practice nurse; CNS, clinical nurse specialist; PA, physician assistant; RD, registered dietitian; RPh, registered pharmacist.

Next, the panel reviewed the 2009 test content outline, which listed 8 domains of practice. Practice domains define the major responsibilities or activities associated with a job, credential, or profession. The previous outline included a mixture of content domains delineating the knowledge required for competent practice as a BC-ADM and performance domains delineating the tasks performed by BC-ADMs. It was determined prior to the role delineation meeting that only content domains capturing the knowledge required to practice as a BC-ADM would be developed as part of the current study. Content domains were the focus of this study rather than performance domains because the multidisciplinary nature of the credential prevents the identification of a common set of core job tasks that are performed by candidates from the all the eligible disciplines, despite the fact that they share a common body of knowledge about diabetes education. Therefore, the panel consolidated the 8 existing practice domains into 4 broadly focused content domains (Table 2) that captured the knowledge required by BC-ADMs from all the eligible disciplines.

Table 2

Comparison of the 2009 and 2015 Test Content Outline Domains

BC-ADM Domains
2009	2015
Foundational Knowledge	Assessment and Diagnosis
Assessment and Data Collection	Planning and InterventionEvaluation and Follow-up
Diagnosis/Problem Identification	Leadership and Advanced Professional Practice
Planning and Intervention
Evaluation
Public and Community Health
Quality Improvement and Research
Leadership and Professional Practice

Abbreviation: Board Certification for Advanced Diabetes Management.

Working in small groups and as a large group, the panel next identified content subdomains to further specify the critical knowledge required for competent practice in each content domain. In the Assessment and Diagnosis domain, the panel developed 10 subdomains; in the Planning and Intervention domain, there were 7 subdomains; and the Evaluation and Follow-up domain and the Leadership and Advanced Professional Practice domain each had 5 subdomains. Finally, the panelists developed knowledge statements for each of the 27 subdomains. Knowledge statements describe the specific body of factual or procedural information needed for successful function within the subdomain of practice.¹⁰ For instance, in the Collaboration and Coordination of Care subdomain, the panel reasoned that competent entry-level practice for the BC-ADM would require working knowledge of interdisciplinary specialties and community resources.

Thus, at the end of this first phase, the SMEs had produced a list of practice domains, a unique set of content subdomains for each, and a list of knowledge statements for every subdomain. A postmeeting evaluation survey showed that the panelists believed that they had accomplished the meeting objectives, included balanced input from each discipline, and produced a defensible description of the knowledge required for entry-level BC-ADM practice.

Phase 2: Creating the Validation Study

In phase 2, a validation study was undertaken to validate the content domains, subdomains, and knowledge statements produced in phase 1. Specifically, the validation study was designed to determine whether current BC-ADM certificants perceived that the essential knowledge defined by the panel in phase 1 is necessary for successful job performance for entry-level BC-ADMs. The methodology for the validation study, including the development of a 3-part online validation survey, was recommended by Castle. The first section of the survey was designed to elicit ratings for each subdomain on three factors: criticality of the knowledge, frequency with which the knowledge is used, and point in career at which the knowledge is first used. For criticality, survey respondents were asked to rate the extent to which they believed harm (physical, emotional, financial, etc) would result if a BC-ADM did not have the knowledge described in the subdomain. The rating for criticality ranged from 1 (no harm) to 5 (extreme harm). In addition to criticality, the survey respondents evaluated each subdomain based on how frequently a newly certified BC-ADM would be expected to rely on the knowledge described in the subdomain. The rating for frequency ranged from 1 (never) to 5 (daily). Finally, the survey respondents assessed the point at which a newly certified BC-ADM would first be expected to have the knowledge described in the subdomain. The rating choices were 1 (not at all), 2 (within the first 6 months after certification), and 3 (after the first 6 months of certification). The purpose of the point-in-career question was to ensure that the knowledge that is included on the BC-ADM examination is at an appropriate level for the candidates, rather than being too advanced for entry-level BC-ADM candidates (ie, not required until >6 months after certification). The results of the survey revealed that all of the content subdomains described knowledge that was necessary within the first 6 months of being certified; therefore, no subdomains were deleted on the basis of the point-in-career ratings. Thus, the first section of the survey yielded 3 sets of scores: criticality, frequency, and point in career.

The mean criticality ratings were multiplied by the mean frequency ratings to arrive at a relative importance weighting for each subdomain. Because the relative importance weightings provided an indication of which pieces of knowledge were deemed most critical and most frequently used, they were used to determine the weightings of each content domain in the BC-ADM test blueprint. Specifically, content subdomains that were rated as critical to the BC-ADM role and performed more frequently were determined to be more important for competent performance on the job and were represented more heavily on the examination. The AADE had the option of excluding subdomains with lower relative importance ratings, but all subdomains were retained after a careful review. It was decided that even the few subdomains with lower relative importance scores were still high on either frequency or criticality; therefore, it was desirable to retain them as testable content.

The second section of the validation survey asked respondents to estimate the percentage of the examination that should be allotted to each of the 4 domains. The third and final section captured demographic information necessary to evaluate the representativeness of the sample of respondents.

All current BC-ADMs who had not opted out of e-mail correspondence were invited to participate in the survey. In addition, because allopathic and osteopathic physicians had recently become eligible for the credential, it was important to survey that practice group. A sample of 18 physicians was included in the invitation to the study. In the end, 667 BC-ADMs and 18 physicians were invited to participate in the study through a series of invitation and reminders delivered on a prescribed schedule. All invitees were promised confidentiality and were offered 3 continuing education units to be used for BC-ADM renewal. Physicians were also eligible to be entered into a drawing to win 1 of 2 $100 Amazon gift cards. The survey was open from May 5 to May 30, 2014.

Phase 3: Analyzing the Validation Study Survey and Weighting Domains

In Phase 3, the Castle psychometrician analyzed the survey responses to determine the weight that each domain should have on the 150-item BC-ADM examination. In addition to determining domain weights, it was necessary to calculate subdomain weights. This weighting specifies how many examination items are needed from each subdomain for each examination form, and it was calculated by summing all the subdomain-level relative importance weightings within a domain. This value became the denominator in the calculation. The numerator was the individual subdomain score multiplied by the domain weight. The resulting calculation allowed the AADE to assign a number of items to each subdomain.

Results and Discussion

Of the 685 BC-ADMs and physicians invited to take the validation survey, 183 (26.7%) provided ratings for all 27 subdomains in the survey. An additional 78 respondents (11.4%) provided partial but usable data for at least 1 subdomain, with the majority of those completing ratings for more than half the subdomains. Approximately half the respondents (n = 261) were nurses (53%), with 18% of respondents working as pharmacists, 15% as registered dietitians, 3% as physicians, and 1.5% as physician assistants and with the remainder electing not to respond to this question. More than 80% of respondents were between 36 and 65 years old, with 50% falling between 51 and 65 years old. More than 89% were female, and >80% were white. Of 50 US states, 45 were represented with at least 1 respondent. More participants were from California (7%) and Texas (7%). Most respondents had a master’s degree (64%) and ≥11 years of experience (62%); the majority worked full-time (87%); and 35% of respondents were new certificants (≤2 years).

The mean criticality, frequency, and importance ratings are provided in Table 3. Mean criticality ratings ranged from 1.99 to 5.57. Recall that a rating of 2 indicated that “minimal harm” would result if a BC-ADM did not have the knowledge described in the subdomain, while a rating of 4 indicated that “substantial harm” would result. Average frequency ratings ranged from 2.59 to 4.87, with a score of 3.00 indicating that a newly certified BC-ADM would rely on the subdomain knowledge every few days to weekly.

Table 3

Mean Criticality, Frequency, and Relative Importance Ratings

Domain: Subdomain	Mean Criticality	Mean Frequency	Relative Importance
Domain 1: Assessment and Diagnosis
1.1. Therapeutic interview (eg, motivational interviewing, open-ended questions)	2.973	4.678	14.008
1.2. Comprehensive history and physical exam techniques across the life span (eg, functional status, sensory, fundoscopic, medication, and complementary alternative medication review)	3.984	4.795	19.232
1.3. Physiology and pathophysiology relating to prediabetes, diabetes, complications, and comorbidities	3.690	4.691	17.498
1.4. Self-care behavior and mental health assessment	3.140	4.560	14.482
1.5. Health care resource assessment (eg, insurance, finances, social support, health care team)	2.719	4.234	11.714
1.6. Standards of diabetes care and clinical practice guidelines pertaining to assessment (eg, ADA, AACE, AADE)	3.679	4.800	17.779
1.7. Analysis of complex data sets (eg, continuous glucose monitoring, blood glucose results, insulin pump software)	3.866	4.431	17.451
1.8. Screening and diagnostic criteria for types of diabetes and patients at risk for diabetes across the lifespan (eg, children, pregnancy, MODY, LADA)	3.332	4.165	14.178
1.9. Synthesis of information from assessment and diagnostic test results	4.099	4.763	19.588
1.10. Formulation of differential diagnoses/prioritization of problem list	3.358	4.602	15.552

Domain 2: Planning and Intervention
2.1. Standards of diabetes care and clinical practice guidelines pertaining to intervention (eg, ADA, AACE, AADE)	3.644	4.732	17.402
2.2. Nutrition and exercise prescription (eg, lifestyle interventions, MNT, DSME/S)	3.332	4.646	15.546
2.3. Pharmacologic therapy options and interventions to manage diabetes and related conditions	4.573	4.874	22.316
2.4. Surgical options for diabetes management, including their risks and benefits (eg, weight loss, remission of diabetes, nutritional deficiencies)	2.738	3.204	9.912
2.5. Technology options (eg, insulin pump, CGM, apps/software)	3.423	4.055	14.256
2.6. Individualization and prioritization of care	3.342	4.574	15.477
2.7. Collaboration and coordination of care (eg, referrals, case management)	2.664	4.017	10.918

Domain 3: Evaluation and Follow-up
3.1. Goal setting using AADE7 Self-care Behaviors™	2.684	4.479	12.172
3.2. Standards of diabetes care and clinical practice guidelines pertaining to follow-up care (eg, ADA, AACE, AADE)	3.380	4.643	15.813
3.3. Patient outcomes and expected treatment results reflecting evidence-based practice (eg, clinical, quality of life)	3.674	4.695	17.409
3.4. Goal achievement (eg, SMART goals, identifying obstacles, adherence)	2.797	4.464	12.664
3.5. Plan modification	3.204	4.594	14.909

Domain 4: Leadership and Advanced Professional Practice
4.1. Regulatory, accreditation/recognition, disease management, and reimbursement programs and standards for institutions and providers.	2.614	3.839	10.566
4.2. Program development and continuous quality improvement	2.114	2.919	6.637
4.3. National health initiatives, prevention strategies, and public health and policy trends	2.157	2.940	6.851
4.4. Scholarship (eg, research, dissemination, mentoring)	2.144	2.944	6.854
4.5. Community programs and outreach	1.987	2.587	5.519

Abbreviations: AACE, American Association of Clinical Endocrinologists; AADE, American Association of Diabetes Educators; ADA, American Diabetes Association; CGM, continuous glucose monitoring; DSME/S, diabetes self-management education and support; LADA, latent autoimmune diabetes in adults; MNT, medical nutrition therapy; MODY, maturity onset diabetes of the young; SMART, specific, measurable, attainable, relevant, and time-bound.

Relative importance scores (Table 3) revealed the 3 subdomains with the highest relative importance: pharmacologic therapy options (domain 2, subdomain 3), synthesis of information from assessment and diagnostic test results (domain 1, subdomain 9), and comprehensive history and physical examination techniques (domain 1, subdomain 2). The 3 subdomains with the lowest relative importance were community programs and outreach (domain 4, subdomain 5), program development (domain 4, subdomain 2), and national health (domain 4, subdomain 3).

Mean point-in-career ratings ranged from 2.06 to 2.53 (on a 3-point scale, with 2.00 indicating knowledge used within 6 months). Upon review of all 4 measures (criticality, frequency, importance, point in career), the AADE elected to retain all subdomains but decided that subdomains with low ratings would be weighted less on the BC-ADM test blueprint.

Following the collection of the validation survey results, Castle produced the BC-ADM weighted test content outline (blueprint) by analyzing the survey results and determining the weight that each subdomain should have on the 150-item BC-ADM examination. During phase 1, prior to the validation study, the SME panel had provided recommendations for how much each domain should be weighted on the examination. As part of the validation study, survey respondents were also asked to identify the percentage of the BC-ADM examination that should be devoted to each of the 4 domains. The recommendations of the SME panel and the survey respondents were remarkably similar, as shown in Table 4. Final domain weights and corresponding number of examination items are also indicated in Table 4; these constitute the BC-ADM weighted test content outline or examination blueprint. The final 2014 BC-ADM blueprint shown in Table 4 is also available online at http://castleworldwide.com/aade/AppSystem/6/Public/Resource/AADE_Candidate_Handbook.pdf.

Table 4

Examination Blueprint

Domain	SME Panel Results, %	Survey Results, %	Final Domain Weights, %	Exam Questions, n
1: Assessment and Diagnosis	30.83	30.5	31	47
2: Planning and Intervention	34.42	31.9	32	48
3: Evaluation and Follow-up	24.67	25.0	25	37
4: Leadership and Advanced Professional Practice	9.25	12.5	12	18
Total			100	150

Abbreviation: SME, subject matter expert.

Conclusions

Role delineation studies are conducted to ensure content validity for national certification examinations.^7-9 This report describes the methodology of a role delineation study for the BC-ADM examination under the guidance a psychometrician and with input from a panel of currently credentialed BC-ADM SMEs. The SME panel produced 4 content domains, a unique set of content subdomains for each, and a series of knowledge statements for every subdomain. This content outline was validated through an online survey with BC-ADM certificants representing a variety of professions, practice settings, and professional experience, as well as a group of 8 physicians. The survey respondents validated the criticality, frequency, and importance of subdomains, and their ratings resulted in setting the overall distribution of content across 4 practice domains. This validation study informed the next actions—updating the test bank and refreshing old items—which will keep the BC-ADM examination reflective of the current practice and required knowledge of the advanced diabetes manager.

Relevance

The first BC-ADM examinations aligned to the new test specification were launched in June 2015. For further information, see https://www.diabeteseducator.org/education-career/certification/bc_adm.

References

Daly

Kulkarni

Boucher

. The new credential: advanced diabetes management. J Am Diet Assoc. 2001;101:940-943.

Valentine

Kulkarni

Hinnen

. Evolving roles: from diabetes educators to advanced diabetes managers. Diabetes Educ. 2003;29:598-602.

Barlow

Crean

Heizler

Mulcahy

Springer

. Diabetes educators: assessment of evolving practice. Diabetes Educ. 2005;31:359-372.

Daly

. Advanced practice care in diabetes. Diabetes Educ. 2003;29:592-595.

Henderson

Smith

. Job/practice analysis. In: Knapp

Anderson

Wild

, eds. Certification: The ICE Handbook. Washington, DC: Institute for Credentialing Excellence; 2009:123-148.

Gray

Jaff

Slovut

. The first 10 years of the American Board of Vascular Medicine. Vasc Med. 2015;20:69-73.

Roberts

Hughes

. What do orthopaedic nurses do? Implications of the role delineation study for certification. Orthop Nurs. 2013;32:198-206.

Villanueva

Thompson

Macpherson

Meunier

Hilton

. The neuroscience nursing 2005 role delineation study: implications for certification. J Neurosci Nurs. 2006;38:403-408.

McMillan

Heusinkveld

Chai

Miller Murphy

Huang

. Revising the blueprint for the Oncology Certified Nurse (OCN) examination: a role delineation study. Oncol Nurs Forum. 2002;29:E110-E117.

10.

Brannick

Levine

Morgeson

. Job Analysis: Methods, Research, and Applications for Human Resource Management. 2nd ed. Thousand Oaks, CA: Sage; 2007.