Assessing Preservice Teachers’ Cultural Competence With the Cultural Proficiency Continuum Q-Sort

Abstract

This article describes an educational design research program situated within a professional development school that led to the development of the Cultural Proficiency Continuum Q-Sort (CPCQ). The CPCQ is a tool that enables teacher educators to systematically examine preservice teachers’ cultural competence concerning students who are minoritized, marginalized, and otherized within PreK–12 schools. The author provides background and rationales for the need of the CPCQ together with a discussion of the educational design research program that facilitated the design of the CPCQ. Findings for this study are discussed discursively addressing the following question: (a) What are the rationales and chain of decisions that warranted advancing the development of the CPCQ through each phase of an educational design research program? and (b) What are participants’ perceptions of the emerging tool’s effectiveness for facilitating critical self-reflection, inquiry, and dialogue concerning students who are minoritized, marginalized, and othered within PreK–12 schools? This study demonstrated that educational design research together with professional development schools are an ideal context to develop tools in a real-world setting aimed to address issues around racial and social justice and cultural competence within teacher education programs and PreK–12 schools and classrooms.

Keywords

adult education/development mixed-methods research methodology teacher assessment teacher education/development teacher research urban education

The U.S. Census Bureau (2015) projected that the U.S. population would experience a majority-minority demographic transformation (i.e., browning of America; Frey, 2018) by 2044; for children under 18, this transformation is projected to occur as early as 2020. This phenomenon, specific to the United States, is when the number of one or more racial or ethnic minorities is greater than the White population. However, as of 2014, this transformation had already begun in U.S. public schools (National Center for Educational Statistics, 2016), prompting education researchers to discuss implications. One major implication is the discrepancy between the PreK–12 teacher workforce and the majority-minority student population (Easton-Brooks, 2019; Milner, 2020a; U.S. Department of Education, 2016). Furthermore, this discrepancy is complicated by the racialized context, which is understood as the daily reality of racism, that both teachers and students are socialized into by living in the United States (Ladson-Billings & Tate, 1995; Milner, 2017; Stovall, 2006). The overwhelmingly White teacher workforce together with the racialized context highlights the lack of cultural competence among teachers (Castro, 2010; Gay, 2010; Howard, 2010; Ladson-Billings, 2017; Milner, 2020b; Villegas & Lucas, 2002).

Cultural competence in education, broadly, is understood as a teacher’s “ability to successfully teach students who come from cultures other than [their] own” (Moule, 2011, p. 5). It is important to note that culture, although traversing race¹ and ethnicity², is a different concept. Throughout this article, culture is understood as the medium that is social and shapes how individuals learn and evolve. Howard (2010) added that enhancing teachers’ cultural competence will increase their capacity to “gain a comprehensive understanding of how cultural knowledge is acquired, expressed, maintained, and transformed across space and time” (p. 113). Subsequently, tools are needed that empower preservice teachers (PSTs) to both interrogate their cultural competence³ together while also enabling teacher educators to assess and codify PSTs’ current understanding of their cultural competence.

The goal of this article is to provide education researchers with a model for how to employ educational design research (EDR) principles together with the aims of asset-based pedagogies (e.g., culturally relevant pedagogy, cultural proficiency) in developing tools to assess and codify PSTs’ cultural competence. Accordingly, in this article, I provide the context and rationale for the creation of a tool aimed to assess and codify PSTs’ cultural competence. Next, I describe the EDR program used to guide the development of the Cultural Proficiency Continuum Q-Sort (CPCQ), a tool designed to assess and codify PSTs’ cultural competence. Findings are embedded within the methodology section and written discursively, allowing me to achieve the goal of this article by providing the reader with the rationales and chain of decisions that support the advancement and development of the CPCQ within an EDR program.

Sociocultural Gap and Cultural Competence

White teachers and others (i.e., non-White teachers) hold dispositions (e.g., racial color blindness) that are not beneficial in addressing the holistic needs of students from diverse racial and ethnic backgrounds (Bonillia-Silva, 2017; Castro, 2010; Milner, 2020). Gay (1993) and Castro (2010) asserted that teachers lack the capacity to address students’ holistic needs because of the sociocultural gap, which is the social and cultural distance between educators and students who are minoritized,⁴ marginalized, and otherized. In light of the recent demographic transformation within the U.S. public school student population, the sociocultural gap is a significant concern (Gay, 1993; Howard, 2010; Ladson-Billings, 2017). Reason for concern is warranted because consequences of the sociocultural gap, consciously or unconsciously, lead to problematic teachers’ beliefs, dispositions, and interactions (e.g., color blindness, cultural conflicts, myth of meritocracy, low expectations, and deficit and context-neutral mind-sets) that negatively impact the learning experience for students who are minoritized, marginalized, and otherized within PreK–12 schools and classrooms (Milner, 2020b).

Within the context of teacher education, attention to the sociocultural gap can lead PSTs to espouse and implement equity-minded teaching practices (e.g., centering race, eliminating inequality, and authentic caring) within their PreK–12 classrooms (Bensimon et al., 2007). The sociocultural gap negatively affects the instructional process and student achievement (Khalifa at el., 2016), but as Ladson-Billings (1998) indicated, teachers struggle to close these gaps because they “possess only a surface understanding of culture—their own or anyone else’s” (p. 261). Subsequently, offering instruction that empowers PSTs to integrate their own racial and cultural identities together with formative assessments aimed to assess and codify cultural competence can serve as a remedy for mitigating the sociocultural gap (Castro, 2010; Mansour, 2009).

Researchers have developed tools (e.g., self-reporting surveys or inventories) that enable teacher educators and researchers to assess PSTs’ cultural competence (Bennett, 1993; Marshall, 1996; Neville et al., 2000; Siwatu, 2007). However, self-reporting surveys and inventories have limitations. A significant limitation is that self-report surveys do not allow for follow-up queries by participants who have questions about the survey’s constructs and items or to explain their selections made during the surveying process (Ladson-Billings, 1998). According to Ladson-Billings (1998), a tool aimed to assess teachers’ cultural competence should be structured and include the following approaches: (a) situated pedagogies, (b) teaching cases, and (c) reflective practices. These approaches put forth by Ladson-Billings would allow for and empower PSTs to have follow-up queries about the constructs and concepts embedded within the tool. In addition, these approaches would enable teacher educators to assess PSTs’ cultural competence together with using data generated by PSTs to codify their cultural competence.

Q-methodology (Q) can be used to develop a tool to assess and codify PSTs’ cultural competence while adhering to Ladson-Billings’s (1998) recommendations (Stephenson, 1935). Q, also referred to as Q-sorting or Q-technique, has been used across many disciplines and in many formats (e.g., cards, online, or static worksheet; Watts & Stenner, 2005). Few Q-sorts have been designed to assess educators’ cultural competence implicated in the sociocultural gap. Yang and Montgomery (2013) developed a traditional Q-sort (i.e., cards) using three theoretical dimensions of multicultural education to capture PSTs’ and teacher educators’ attitudes toward student diversity. Yang and Montgomery’s (2013) study provided useful information; however, Q-sorts that use card formats or online formats, although mobile, require an individual (i.e., facilitator) that is skilled in the Q-sorting process (e.g., implementation and data analysis; Watts & Stenner, 2005).

Thus, for this study, I aimed to design and validate a Q-sort that is a static worksheet and features the Cultural Proficiency Continuum (the Continuum; defined in the following) to enable PSTs together with teacher educators or facilitators to assess and codify their cultural competence. Furthermore, I used the worksheet format because I wanted to design a tool that is transportable and could be used in a host of teaching and learning applications. For example, a worksheet Q-sort can be used by an individual to self-assess their cultural competence or used as a professional development activity to facilitate individual or group dialogue.

Guiding Design Perspective

The teleological aims (e.g., transformational approaches and social action; Banks, 2013) of multicultural education guided the espousal of the perspectives, methods, and pedagogical frameworks used to design the CPCQ. Multicultural education comprises a host of approaches related to educational equity for students and families who are minoritized, marginalized, and otherized. Over the years, multicultural education has evolved into a concept, a reform movement, and a process that can be situated epistemologically within many approaches for comprehensive school and individual transformation.

An epistemological approach known as bricolage (Lévi Strauss, 1962) was used to design the CPCQ. Bricolage allows individuals to reimagine, improvise, or appropriate preexisting materials (Chandler, 2002). A bricoleur serves as the “experienced tinker/handy person who uses, as much as possible, those materials that happen to be available” (Akker et al., 2006, p. 21). Bricolage was appropriate for this research and situated milieu because learning occurs within three contexts: (a) structured, (b) intentional, and (c) unanticipated (Golombek & Johnson, 2004; Gravemeijer, 1994). Accordingly, within this study, I embodied the disposition of a bricoleur; I relied heavily on existing methods and approaches, thoughtful reflections, and critical friends to facilitate the design of the CPCQ—a notion referred to as theory-guided bricolage (Gravemeijer, 1994).

Addressing a Problem Within a Professional Development School

The study occurred within a secondary English and social studies professional development school (PDS) at an R1 university located in a small metropolitan area in Pennsylvania. A PDS is characterized as a school-university partnership, which is a partnership between a teacher education program and PreK–12 organization (e.g., a school district/division or an individual school community; National Association for Professional Development Schools, 2018). The partnering organizations, the town in which the PDS was located, and the teacher preparation program were comprised of predominantly White individuals. Furthermore, most PSTs within the PDS were White (mostly women) and sought and secured employment in urban schools. Urban schools are educational settings in which “outside of school factors such as housing, poverty, and transportation are directly connected to what happens inside the school” (Milner, 2012, p. 559). Moreover, urban schools have a majority-minority student population, are underresourced, and have the least qualified or credentialed teachers (Milner, 2006, 2012; Welsh & Swain, 2020). With this framing of the problem, it was important for me to develop a Q-sort featuring the Continuum that could demonstrate usefulness for assessing and codifying PSTs’ cultural competence and facilitate dialogue regarding urban schools (Lindsey et al., 2009; Singleton, 2015). Most importantly, borrowing from Ladson-Billings and Tate (1995), the CPCQ honors and elevates the lived experiences of minoritized, marginalized, and otherized students. This thinking aligns with Howard (2010), in which he proffered that teachers who can engage with semiotics (i.e., signs and symbols) germane to students from diverse racial and ethnic backgrounds are better prepared to espouse pedagogical practices that are more accessible for students who attend urban schools and come from culturally rich backgrounds.

At this particular PDS, many of the PSTs had limited experience engaging in conversations about diverse racial and ethnic students and issues specific to diversity, equity, and inclusion within U.S. public schools. The importance of preparing new, predominantly White teachers to teach students attending urban schools served as a key motivation for this study (Castro, 2010). Subsequently, the aforementioned problem warranted the development of the CPCQ, which is aimed to assess and codify PSTs’ cultural competence as well as understand the sociocultural factors that maintain or mitigate the sociocultural gap between PSTs and their prospective minoritized, marginalized, and otherized students.

Methodology (With Embedded Findings)

Educational design research was used in this study to bridge theory to research, ultimately having implications for enhancing teachers’ cultural competence, teaching, and student learning. I used pragmatist approaches to facilitate the design of the CPCQ from conception within a real-world setting—a PDS (Cole et al., 2018). Barab and Squire (2004) offered that EDR “is not so much an approach as it is a series of approaches, with the intent of producing new theories, artifacts, and practices that account for and potentially impact learning and teaching in naturalistic settings” (p. 2).

The EDR model used in this study was the Compleat [sic] Cycle of Design Research (CCDR; Lamberg & Middleton, 2009; Middleton et al., 2008). CCDR is an iterative and cyclical process used to conduct “design research from inception of an idea, to creation of . . . artifacts, to testing and upscaling innovations for a broader audience” (Lamberg & Middleton, 2009, p. 233). CCDR is divided into two larger phases—artifact design (Phases 1–3; Cobb et al., 2003) and artifact implementation research (Phases 4–7; Fishman et al., 2013)—fundamental in bridging theory, research, and practice (see Figure 1). I discursively discuss the process and chain of decisions that took place within the CCDR: Phase 1: conceptualizing the problem and establishing theoretical intent and Phase 2: designing the cultural proficiency continuum Q-sort to assess and codify PSTs’ cultural competence. The findings for this study are embedded within the methodology section of this article. Furthermore, this study took place in a PDS, and all participants (e.g., critical friends, community members, and PSTs) played a vital role in designing the CPCQ.

Procedures for Designing the Cultural Proficiency Continuum Q-Sort

Phase 1: Conceptualizing the problem and establishing theoretical intent

The first phase of the CCDR involved conceptualizing the problem and developing theoretical intent for the CPCQ (Gravemeijer, 1994; Lamberg & Middleton, 2009). Thus, the problem addressed in this study was to backward design a Q-sort, featuring the Continuum, that could be used to assess and codify PSTs’ cultural competence supporting subsequent critical self-reflection, inquiry, and dialogue to examine and mitigate potential sociocultural gaps.

The Educational Philosophy Q-Sort* (see Badiali, 2005) was used as a structure to guide the backward design of the CPCQ. Q is a process by which respondents are forced to rank/sort ideas that illuminated their subjectivities about a given topic (e.g., teaching beliefs or practice; Prasad, 2001; Stephenson, 1935). Q-sorts have psychometric properties that point to subjective beliefs that may materialize in tangible forms (e.g., discourse) for the purpose of observation and study (Brown, 1986). Embedded within the Q is a sorting/ranking process that preoccupies respondents while engaging with a Q-sort. This preoccupation is a convenient distraction, which consequently mitigates socially desirable responding (Fluckinger, 2014). Along with Q, the Continuum (see Figure 2) was utilized to assess constructs (i.e., q-factors) and language to assess and codify how PSTs responded to issues regarding equity, diversity, and inclusion within urban schools (Lindsey et al., 2009). In addition, Figure 2 illustrates how the left side of the Continuum contains “culturally proficient behaviors” that are “reactive and focus on tolerance and the other as being problematic” (Lindsey et al., 2009, p. 112), whereas the right side of the Continuum illustrates culturally proficient behaviors that are “proactive and focus on transforming one’s practice” (Lindsey et al., 2009, p. 112). The resulting product was a mock-up of the CPCQ.

Thus, the theoretical intent for the CPCQ was to create a learning opportunity through a disorienting dilemma. A disorienting dilemma is an “acute internal and personal crisis” (Taylor, 2002a, p. 298) that is informed by sociocultural factors that are dependent on one’s personal histories and experience. Within the context of this study, the q-set items (i.e., vignettes) within the CPCQ, which center racial and cultural discourse, are positioned as potentially disorienting dilemmas or triggers (Milner, 2007). PSTs’ reactions to q-set items are dependent on their frames of reference, cultural competence, and comfortability with talking about race and educational inequities in the areas of diversity, equity, and inclusion (Castro, 2010; Gay, 1993; Singleton, 2015; Villegas & Lucas, 2002). Accordingly, when a disorienting dilemma presents itself within the scope of the CPCQ, this tension (e.g., ambivalence and cognitive dissonance) is seen as useful and envisioned to activate personal transformation (e.g., increasing one’s cultural competence) through a process of critical self-reflection, inquiry, and subsequent dialogue (Taylor, 2002b). The disorienting dilemma together with the dialogic properties of the CPCQ positions the tool as a vehicle to illuminate PSTs’ sociocultural gaps together with increasing their cultural competence.

Phase 2: Designing the cultural proficiency continuum Q-sort

Phase 2 involved “generating . . . a design—a kind of operational definition of the problem-as-theorized” (Lamberg & Middleton, 2009, p. 234). In this study, I theorized that PSTs did not have opportunities to reflect on and discuss how their cultural competence was affecting their teaching beliefs and practices regarding minoritized, marginalized, and otherized students. Many of the PSTs within the PDS were looking for a teaching position at an urban school. Furthermore, these PSTs had limited experience with engaging with semiotics germane to urban schools. Thus, the development of the CPCQ was warranted for introducing PSTs to a host of culturally proficient behaviors (i.e., semiotics) germane to urban schools. In addition, the CPCQ would provide evidence—qualitative data—unique to each PST, which could be used to assess and codify their cultural competence.

Mock-up of the Cultural Proficiency Continuum Q-Sort

The mock-up of the CPCQ was developed using Wiggins and McTighe’s (2005) backward design concept. Q-sorts have three components: (a) q-factors (constructs), (b) q-set items (vignettes), and (c) subcategories (situated contexts; Brown, 1986; Stephenson, 1935). The components of the CPCQ were: (a) q-factors (n = 6; the Continuum; see Figure 2), (b) q-set items (n = 30; vignettes of culturally proficient behaviors that take place in urban schools), and (c) categories (n = 5; subcategories: attitude, empathy, policy, professionalism, and teaching practice).

In addition, Q-sorts, particularly those that are static worksheets, have additional components (Badiali, 2005). These additions are designed to increase the usefulness and mobility of the Q-sort. The CPCQ’s add-ons included a rating guide and making-sense section—a list of definitions for each level on the Continuum. The rating guide enabled respondents to visualize their sorts; the making-sense section allowed respondents to reflect on the potential implications of their sorts.

The mock-up CPCQ (i.e., first iteration) was tested and its effects (i.e., disorienting dilemma) observed on critical friends and PSTs. Subsequently, the mock-up showed promise in meeting its intended aims, but before prototyping with PSTs, critical friends suggested modifying, validating, and establishing reliability for the q-set items. Furthermore, due to the CPCQ’s intended purpose and observed effects, it was essential to demonstrate validity and reliability (i.e., social trustworthiness) for the q-set items.

Addressing q-set items for the first iteration of the CPCQ

Addressing q-set items (n = 30) within the first iteration of the CPCQ was a rigorous and lengthy process. Developing q-set items (e.g., vignettes) can be done using a variety of qualitative and quantitative methods and criteria (Ozer, 1993). Appropriately using Ozer’s (1993) recommendations, I developed an iterative and cyclical process to construct and validate q-set items.

The CPCQ uses racial and cultural discourse, which is highly subjective; it was important to develop a process espousing democratic participation. Furthermore, the CPCQ was designed to emulate day-to-day interactions that take place within majority-minority student populations and urban public schools. The schema was used to guide the construction of the q-set items for the first iteration of the CPCQ. Moreover, Figure 3 illustrates a mixed-methodological design that was used to construct/modify, validate, and establish reliability for q-set items (Moss & Haertel, 2016). This process was repeated until q-set items proved to be valid and reliable and demonstrated usefulness for examining PSTs’ cultural competence.

Q-set development for the first iteration of the CPCQ

A systematic examination of news outlets, social media, academic journals, educational case law, and personal stories was undertaken to develop a pool of q-set items (e.g., vignettes) that mapped onto each q-factor (i.e., the Continuum; see Figure 2) and subcategory within the CPCQ germane to students who attend urban schools. The q-set’s composition is a heterogeneous set of sociocultural interactions that represent each subcategory (e.g., attitude–teaching practice) and each level on the Continuum. Found within each q-set item is the reenactment of a culturally proficient behavior demonstrating an individual or individuals approving or disproving a form of oppression (e.g., racism, sexism, ableism, homophobia, xenophobia, and classism).

Validating q-set items for the first iteration of the CPCQ

Critical friends and subject matter experts (SME) were used to validate (face, construct, and content) each q-set item (n = 30), representative of each subcategory and each level on the Continuum. Validity in this study was understood as the appropriateness, relevance, and usefulness of each q-set item together with the methods (e.g., think-aloud protocol with critical friends and SME) used to establish validity (Donaldson et al., 2009; Leung, 2015). The final composition of the initial q-set was as follows:

five q-set items, one for each subcategory, that represent the culturally proficient behavior/interaction, cultural destructiveness—see the difference; stomp it out;

five q-set items, one for each subcategory, that represent the culturally proficient behavior/interaction cultural incapacity—see the difference; make it wrong;

five q-set items, one for each subcategory, that represent the culturally proficient behavior/interaction, cultural blindness—see the difference; dismiss it;

five q-set items, one for each subcategory, that represent the culturally proficient behavior/interaction, cultural precompetence—see the difference; recognize what you don’t know;

five q-set items, one for each subcategory, that represent the culturally proficient behavior/interaction, cultural competence—see the difference; understand the difference that difference makes;

five q-set items, one for each subcategory, that represent the culturally proficient behavior/interaction, cultural proficiency—see the difference; respond positively and affirmingly.

Establishing reliability for q-set items for the first iteration of the CPCQ

Koo and Li (2016) suggested that researchers should try to obtain at least three raters across 30 heterogeneous subjects (i.e., q-set items or vignettes) whenever possible when conducting a reliability study. The first recommendation was met with the fact that I recruited 14 participants to participate in a focus group (FG1) to rate/judge each q-set item. Second, the CPCQ had 30 heterogeneous q-set items. The focus group included critical friends, community members, and PSTs. Ensuring reliability was important for two reasons: (a) When applicable, I wanted to utilize and maintain fundamental democratic principles (e.g., cogenerative dialogue and checks and balances) throughout this research process, and (b) I wanted to ensure that each q-set item mapped on to its intended construct (i.e., levels within the Continuum; see Figure 2).

I conducted focus group activities with FG1 participants. First, I presented and discussed each of the definitions within the Continuum (i.e., q-factors; constructs) with participants. After the focus group, participants, via a survey, were asked to assign/rate one of the Continuum definitions to each q-set item or vignette. While completing the survey, FG1 participants were encouraged to offer written feedback on q-set items. At the conclusion of the focus group, I estimated the interclass correlation coefficient (ICC) and 95% confidence interval (CI) to obtain interrater reliability for the q-set (n = 30). ICCs and 95% CIs were estimated using the Statistical Package for the Social Sciences (SPSS) Version 26, with a mean-rating (k = 14), absolute-agreement, and one-way random effect model (Koo & Li, 2016).

For FG1, the ICC single measure = 0.60, 95% CIs [0.48, 0.74]. Hallgren (2012) provided

commonly-cited cutoffs for qualitative ratings of agreement based on ICC values, with IRR being poor for ICC values less than .40, fair for values between .40 and .59, good for values between .60 and .74, and excellent for values between .75 and 1.0. (p. 9)

Based on these recommendations, the CPCQ’s q-set’s interrater reliability is good. The q-set’s good reliability warranted prototyping the CPCQ; however, some issues needed to be addressed before prototyping within the PDS. First, FG1 participants indicated that Continuum definitions (i.e., q-factors) were ambiguous and difficult to differentiate during the rating process. The large range between the lower (.48) and upper bound (.75) single-measure 95% CI signaled the aforementioned difficulty.

In developing the content for the emergent Q-sort and FG1 materials, I used an earlier edition of Lindsey et al.’s (1999) cultural proficiency. The later version of Lindsey et al. (2009) addressed the ambiguous nature of the Continuum by offering clear and decisive definitions when compared to the earlier edition. Furthermore, I paraphrased some of the definitions (i.e., cultural destructiveness, cultural precompetence, cultural competence, and cultural proficiency) to mitigate the ambiguity and fluidity of the Continuum. Subsequently, I incorporated newer definitions in the making-sense section for subsequent iterations of the CPCQ and materials for future focus groups.

FG1 participants (i.e., raters) indicated that some q-set items needed to be eliminated or modified (n = 11)—I remedied these q-set items with critical friends before prototyping. The remedy for the majority of the q-set items had to do with how the subjects (e.g., perpetrator or change agent) and objects (e.g., victimized or liberated) were situated within the vignettes (Freire, 1970). Because of the dichotomous nature of the vignettes, raters had difficulty deciding who should (i.e., subjects or objects) be the focus of observation while assigning ratings. An example of this dilemma is shown in Table 1.

Table 1

An Example of How a q-Set Item (i.e., Vignette) Was Modified

Before	A high school student from a high poverty school with a 4.0 GPA and a perfect score on the SAT has received a scholarship from an Ivy League School. However, this scholarship offer is accredited to their socioeconomic status.
After	Many suspect that a student from a high poverty school with a 4.0 GPA and a perfect score on the SAT has received a scholarship from an Ivy League School due to their socioeconomic status.

The vignette shown in Table 1 was rated cultural precompetence, which denotes the limitations of an organization or individual’s ability to fully make sense of social ills that affect students who are minoritized, marginalized, and otherized in PreK–12 schools (Lindsey et al., 2009). Embedded in the vignette is the notion that students who attend schools with the majority of students coming from economically challenged backgrounds cannot gain admittance to an Ivy League university without some form of institutional assistance rather than their personal merit. This notion is part of a larger social phenomenon (e.g., systemic racism and deficit mind-sets and low expectations), which in most cases goes unchecked without any problematization on behalf of the perpetrator (Ladson-Billings & Tate, 1995; Milner, 2020).

Before modifications, the vignette, first, did not have a subject—perpetrator. Second, this vignette is in the attitude subcategory within the CPCQ. However, it did not project like an individual’s attitude toward the issue embedded in the vignette. Attitude within the context of the CPCQ is understood as “learned predispositions to respond to an object or class of objects in a favorable or unfavorable way” (Fishbein, 1967, p. 257). Thus, the remedy for this vignette was first adding a perpetrator (i.e., many suspect). Second, the vignette was restructured to project a group’s attitude toward the social issue within the vignette. These modifications enabled raters as well as participants (i.e., CPCQ respondents) to better see the intended meaning of the vignette and rate/react accordingly.

Prototyping the first iteration of the CPCQ

After undertaking an iterative and rigorous process to develop a useful set of q-set items, it was time to implement the emergent CPCQ with PDS teacher interns (i.e., PSTs). Within the PDS, I was one of four consultants who were tasked with facilitating weekly meetings that gave teacher interns the space to reflect and make inquiries about challenges (e.g., critical incidents) that arose during their field experience. However, my role was modified, which was to introduce teacher interns to activities that allowed them to reflect on their own cultural competence together with the lived experience of minoritized, marginalized, and otherized students within the context of school.

Due to my access to teacher interns, convenience sampling was used to recruit participants (n = 17; see Table 2) for this study (Palinkas et al., 2015). Furthermore, my positionality within the PDS, together with my level of engagement with participants, situated me in this study as a participant observer, meaning that my role as a participant while prototyping the CPCQ with PDS teacher interns was more salient than that of the researcher (Creswell & Poth, 2018). However, this situatedness was warranted for a couple of reasons. First, I needed a first-person point of view while designing and prototyping the CPCQ, a point of view that enabled me to make first-hand observations and address issues that emerged while designing and implementing the CPCQ. Second, these first-hand observations put me in the best position to make data-informed decisions regarding impending modifications to the CPCQ. All participants (i.e., PDS teacher interns) agreed to participate in this study.

Table 2

Secondary Professional Development School (PDS) Participants

Role Within the PDS	Race (Self-Reported)	Gender (Self-Reported)	Total (N = 18)
Consultant (i.e., researcher/designer/consultant)	Black	Man	1
English teacher Interns	White	Women	9
	White	Men	3
	Mexican	Woman	1
Social studies teacher intern	White	Women	4

PDS teacher interns were divided into four groups (i.e., PDS consultant groups), two of which met during the fall semester and two during the spring semester. Meetings with consultants lasted for 6 weeks. The first consultant group I engaged with had three participants: three women, one Mexican and two White. Each week, I introduced teacher interns to different activities aimed to facilitate critical self-reflection, inquiry, and dialogue about diversity, equity, and inclusion in the classroom. The six6week agenda was as follows: Week 1, formal introductions and ice-breaking activities; Week 2, prototype the CPCQ; Weeks 3 and 4, one-on-one and whole-group semistructured stimulated recall interviews designed to collect data regarding teacher interns’ experience with the CPCQ; and Weeks 5 and 6, PD activities aimed to facilitate critical self-reflection and dialogue regarding any critical incidents, especially those in connection with students and families who are minoritized, marginalized, and otherized.

All of the teacher interns from the first consultant group indicated that the CPCQ raised their awareness of the problematic interactions that happen in urban schools. Moreover, they found the CPCQ effective and appropriate for increasing their awareness regarding the lived experiences of diverse student populations. One intern said that the CPCQ helped them recognize “what my students are going through, and how can I use that to either help us in the classroom or how can I find strategies and ways to allow that student to be successful as they possibly can” (PDS teacher intern, personal communications, September 29, 2016). In addition, teacher interns provided constructive feedback that led to the modification of q-set items and the instructions for the q-sorting process. Lastly, teacher interns suggested that I develop a procedure to facilitate follow-up discussions with participants after they engaged with the CPCQ.

Summary for the first iteration of the CPCQ

Evidence showed that the CPCQ was developing into a viable tool. The CPCQ helped the researcher facilitate critical self-reflection and dialogue concerning teacher interns’ subjectivities regarding majority-minority student populations and urban schools. However, evidence showed that it was necessary to repeat the iterative and cyclical process (see Figure 3) to further improve q-set items as well as instructions for the q-sorting process. The second round of development aimed to improve the CPCQ’s social trustworthiness and refine the CPCQ’s implementation process.

Addressing q-set items for the second iteration of the CPCQ

The initial round provided key insights from critical friends and teacher interns. Before addressing the q-set items for the second iteration of the CPCQ, I addressed issues raised by critical friends and teacher interns. First, I made modifications to the CPCQ’s rating guide. Next, I developed a semistructured interview protocol. Thereafter, I engaged in the second round of q-set development (see Figure 3) for the second iteration of the CPCQ.

Modifying the rating guide

In the first iteration of the CPCQ, teacher interns were asked to tally up the sum of the ratings in each column that represented each level of the Continuum. I replicated this process from Badiali’s (2005) q-sort, as previously described, but found via observations, critical friends, and teacher interns that there was no need to have participants tally their scores. Specifically, I found that the tallying process detracted from the initial aims of the CPCQ, which was to assess PSTs’ cultural competence via a dialogic process. Thus, I replaced the tallying process with a process that enabled facilitators to locate opportunities for inquiry via warranting q-set items that emerged during the q-sorting process. During the process of developing q-set items shown in Figure 3, each vignette was rated or assigned to a level on the Continuum within each subcategory of the rating guide. Consequently, a predetermined order (i.e., Cultural Destructiveness; 1 - Cultural Proficiency; 6) emerged in which sorts could unfold within each subcategory within the rating guide. However, those sorts that were not sorted the predetermined order, precisely, those sorted two or more ordinal values from their predetermined values, those sorts became known as warranting sorts. Moving forward, warranting sorts that emerged during participants’ engagement with the CPCQ became a subject of inquiry. This modification fundamentally changed how participants understood their sorts, effectively changing the utility of the CPCQ. I modified the instructions (see Table 3) for the CPCQ to reflect these changes.

Table 3

Significant Modification to the Instructions for the Rating Guide Section of the Cultural Proficiency Continuum Q-Sort (CPCQ)

First iteration

When you have completed the CPCQ, go back and revisit each category. Write the number that you have assigned to each meme within each of the five categories to the right of the corresponding letter—A, B, C, D, E, F—in the spaces provided below in the rating guide. Add up the values in each column to determine what level of cultural proficiency aligns with your present attitude and beliefs. Grouped together, organized in columns, these memes are narratives that represent each level of cultural proficiency in the context of culturally diverse memes in public schools.

Second iteration

You have completed sorting the vignettes within the CPCQ; go back and revisit each category. Write the number that you have assigned to each vignette within each of the five categories to the right of the corresponding letter—A, B, C, D, E, F—in the spaces provided below in the rating guide. Ideally, your final results in each row would read 1, 2, 3, 4, 5, 6, but because cultural proficiency is a fluid phenomenon, these numbers may not align in numerical order. The final step in your analysis is to locate the culturally proficient interactions that are 2 or more points higher or lower than the ideal number by each letter. For example, if a row reads 2, 1, 5 , 4, 6, 3, you would circle the numbers 5 and 3. Each number you circle in each row represents an opportunity for inquiry and dialogue for that particular culturally proficient behavior. Remember, this is not a judgment.

Constructing the semistructured interview protocol

At the advisement teacher interns, I developed a semistructured interview protocol (see Appendix A). The interview protocol, a new artifact, was used to engage participants after they completed the CPCQ. Broadly, the interview protocol included (a) questions about participants’ perceptions of the q-sorting process and (b) a set of questions designed to examine participants’ cultural competence regarding each of their warranting q-set items. The first question addressed research questions concerning the design process and theoretical intent for the CPCQ. The second question asked PSTs to “(a) identify and name the social problem, (b) analyze the causes of the social problem, and (c) find solutions to the social problem” (Smith-Maddox & Solórzano, 2002, p. 69).

Validating q-set items for the second iteration of the CPCQ

The problematic q-set items for the second iteration of the CPCQ was addressed after FG1 activities and before prototyping the first iteration of the CPCQ with teacher interns. The modification to problematic q-set items was data-informed (e.g., ICC and critical friends); after completing modifications, this demarcated the second iteration of the CPCQ.

Establishing reliability for the q-set items for the second iteration of the CPCQ

Fourteen individuals were recruited to participate in a focus group (FG2) aimed to improve the reliability of the q-set items for the second iteration of the CPCQ. The modifications made to focus group materials, aforementioned, led to a well-planned focus group. In addition, during the the time between FG1 and FG2, seven7 months, I gained a deeper understanding of Lindsey et al.’s (2009) Cultural Proficiency framework, which ensured an effective presentation of the Continuum during FG2 activities. After FG2 participants engaged with the presentation, participants, via a survey, were asked to assign/rate q-set items using the Continuum.

After surveys were completed, ICCs and 95% CIs were estimated using SPSS Version 25, with a mean-rating (k = 14), absolute-agreement, and one-way random effect model. FG2 ICC single measure = 0.73, 95% CI [0.63, 0.83]. The ICC measure for FG2 is good, a 0.13 increase compared to FG1, which indicated more agreement (Hallgren, 2012). In addition, the 95% CI [0.62, 0.83] for FG2 significantly improved compared to FG1’s 95% CI [0.48, 0.74]. This pointed to a lower-bound increase of 0.14, moving it from a fair reliability to a good reliability, and an upper-bound increase of 0.09, moving it from good to excellent reliability (Hallgren, 2012). Second, the range between the lower and upper bounds between FG1 (.26) and FG2 (.21) decreased, showing that modifications made to the focus group presentation enabled FG2 to make clearer distinctions while rating q-set items. The ICC estimations from FG2 showed that the second iteration of the CPCQ’s 30 q-set items (see Appendix B) were reliable; the greater consensus among raters warranted prototyping with the next group of PDS teachers.

Prototyping the second iteration of the CPCQ

The second iteration of the CPCQ was prototyped with the third PDS consultant group. The consultant group had three participants (e.g., teacher interns), two women and one man, all of whom were White. I repeated the same Week 1 and Week 2 PDS consultant meeting protocols used while prototyping the first iteration of the CPCQ. After teacher interns completed the q-sorting activity, their perceptions of and experience with the CPCQ was gathered one-on-one via the newly developed interview protocol. The interview protocol was effective in gathering teacher interns’ understandings of minoritized, marginalized, and otherized students via unpacking their warranting sorts. Furthermore, data yielded by the interview protocol as well as the modified rating guide instructions enabled me to assess and codify teacher interns’ cultural competence.

Summary for the second iteration of the CPCQ

The modifications made to the CPCQ during the second round of q-set development yielded a better product. During prototyping, teacher interns clearly understood the purpose of the CPCQ, a notion that increased the effectiveness of implementing the Q-sort. All teacher interns from the third consultant group found the subsequent dialogue via the interview protocol helpful in making them aware of what they did not know regarding majority-minority student populations and urban schools. One teacher intern said: “It made me reflect a lot, because it put me in situation to think about things I never really thought about before . . . um, so I thought it was beneficial, it gave me a lot of things to think about” (PDS teacher intern, personal communications, February 23, 2017). The same intern said that the CPCQ, compared to other methods of having intercultural dialogue, allowed them to focus on specific things (i.e., culturally proficient behaviors) regarding social issues found in majority-minority student populations and urban schools. Sentiments from teacher interns and my observations triangulated the effectiveness of the CPCQ, a notion that warranted further development of the Q-sort.

Discussion, Implications, and Future Research

Findings from this EDR program aimed to design a Q-sort formatted into a static worksheet featuring the Continuum were embedded in the narrative within the methodology section of this article. This approach (i.e., embedded findings) is common for studies such as this because it enables the researcher to offer transparency by demonstrating rigor together with the rationales and chain of decisions that warranted advancing the design of an artifact or product within an EDR program (Fishman et al., 2013). Specific to this study, a sequence of iterative and cyclical findings, qualitative and quantitative, yielded the CPCQ. The CPCQ is a stand-alone tool that has demonstrated effectiveness for systematically studying educators’ cultural competence and facilitating subsequent dialogue concerning the lived experience and racial and social injustices of students attending urban schools (Lindsey et al., 2009; Singleton, 2015). In addition, the CPCQ demonstrated usefulness for assessing and codifying PSTs’ cultural competence. Most important, participants observed and reported that the CPCQ was useful in facilitating self-reflection, inquiry, and dialogue about the sociocultural interactions (i.e., culturally proficient behaviors) that take place with urban schools as well as the factors (e.g., race, ethnic, and social class) that maintain the sociocultural gap.

This research demonstrated that EDR programs within teacher education programs (e.g., PDS) can yield pedagogical artifacts and products (i.e., the CPCQ) beyond the scope of traditional constructions of teaching and learning (e.g., math and science teaching methods; Fishman et al., 2013). An overarching goal of EDR is to engineer particular forms of “learning and systematically studying those forms of learning within the context defined by the means of supporting them” (Cobb et al., 2003, p. 9). Learning within this study involved formatively assessing and codifying PSTs’ cultural competence via a disorienting dilemma (i.e., the CPCQ) through self-reflection, inquiry, and dialogue within the context of their practicum/field experience (Taylor, 2002b). PSTs self-reported personal limitations (i.e., sociocultural gaps) regarding their understanding of students who come from racial and cultural backgrounds different than their own as well as their problematic perceptions of urban schools. Most importantly, PSTs found the CPCQ and subsequent dialogue safe and useful for illuminating their blind spots regarding their minoritized and culturally rich students.

This type of learning—cultural competence—within the scope of EDR is an emergent inquiry. Furthermore, this study demonstrated that learning outcomes aimed to help PSTs develop tools (i.e., self-reflection, inquiry, and dialogue) and motivation to increase their cultural competence can happen within the context of self and their teaching practice and at the benefit of students who are minoritized, marginalized, and otherized within PreK–12 classrooms.

Implications

This study demonstrated that EDR is a viable methodology for researchers who espouse variants of critical theory (e.g., critical race theory, critical feminist theory, DisCrit, LatCrit, queer theory), culturally relevant/responsive teaching, and multicultural education to design curricula and pedagogical artifacts and products aimed to increase PreK–12 educational stakeholders’ cultural competence. An imagined consequence of EDR programs aimed to address racial and social inequities and problematic teaching dispositions is to yield products that situate PSTs within discourses that elevate the voices and narratives of students and families who are minoritized, marginalized, and otherized within PreK–12 schools and classrooms (Golombek & Johnson, 2004; Ladson-Billings & Tate, 1995). Subsequently, curricula and pedagogical artifacts and products (e.g., the CPCQ) yielded from EDR programs with the aforementioned aim can be incorporated within teacher education programs to assess and develop PSTs’ cultural competence. Furthermore, data yielded from these imagined artifacts can be used to provide evidence of cultural competence or culturally relevant/responsive teaching dispositions to teacher education accreditation programs such as the Council for the Accreditation of Educator Preparation (CAEP) or Association for Advancing Quality in Educator Preparation (AAQEP) as well as state-level departments of education.

Future Research

Evidence demonstrated that the CPCQ is beginning to take the form that Ladson-Billings (1998) put forth, which is a tool that assesses and codifies PSTs’ cultural competence using situated pedagogies, teaching case methods, and reflective practices. This evidence warranted advancing the development of the CPCQ to Phase 3 of the CCDR (see Figure 1). Phase 3 of the CCDR involves testing the CPCQ in a study aimed to work out the acceptability and usefulness of the CPCQ for teacher educators and PSTs and to see the CPCQ in action under rigorous conditions of implementation (Lamberg & Middleton, 2009).

While prototyping the CPCQ during Phase 2 of the CCDR, additional artifacts begin to emerge that significantly improved the feasibility of the CPCQ. The first supporting artifact that emerged was the semistructured interview protocol, detailed previously. This interview protocol yielded data that enabled me to codify PSTs’ reactions to or dialogue about their warranting sorts (i.e., vignettes). Subsequently, while reflecting on preliminary findings, I began to think about how the CPCQ may have greater impact and usefulness for teacher educators. Thus, I decided to design additional artifacts (e.g., pre- and postsurvey, rubric, and an implementation procedure) to supplement the CPCQ and interview protocol. Comprehensively, these artifacts together could yield quantitative and qualitative data (i.e., a thick description) designed to assess and codify PSTs’ cultural competence (Geertz, 1994). Implications of such data would allow teacher educators to assess and codify PSTs’ cultural competence systematically. Furthermore, these data can be used to provide evidence of PSTs’ cultural competency to teacher education accreditation programs such as CAEP or AAQEP as well as state-level departments of education.

Accordingly, the next steps within this EDR program are to refine the CPCQ and design supplementing artifacts aimed to enhance the effectiveness as well as the mobility of the CPCQ. When understood as a multidimensional whole, the CPCQ is a primary data source that yields data for educational leaders to facilitate school transformation. After developing supplementary artifacts, I will conduct a pilot/feasibility and field study (see Figure 1) in relevant contexts (e.g., teacher education programs) to assess the CPCQ together with supplemental artifacts for robustness and effectiveness.

Footnotes

Appendix A

Appendix B

Second Iteration of the Cultural Proficiency Continuum Q-Sort (CPCQ) Rated Q-Set Items

CPCQ Subcategory	Q-Set Item (i.e., Vignette)	Focus Group 2 Continuum Rating
Attitude	Many suspect that a student from a high poverty school with a 4.0 GPA and a perfect score on the SAT has received a scholarship from an Ivy League School due to their socioeconomic status.	Cultural precompetence
Attitude	Black students have the same opportunities and access to attend college as their white peers.	Cultural blindness
Attitude	Students who wear dreadlocks as a hairstyle are a distraction in the classroom.	Cultural incapacity
Attitude	It should be required for students to remove hair accessories that are associated with their culture, tribe, sexuality, or religion.	Cultural destructiveness
Attitude	All students who live in the United States deserve a quality public education, even those students whose parents are undocumented immigrants.	Cultural proficiency
Attitude	All faculty and staff within a school setting have equal ability to transfer knowledge to students in regards to life, culture, character, and/or academic content.	Cultural competence
Empathy	A 10th-grade male died as a result of a car accident during an illegal street race. A group of teachers will not attend the funeral because of the activity that led to the student’s cause of death.	Cultural incapacity
Empathy	A public school teacher is grounded by the philosophy of their religion. Throughout their career as an educator, they have learned to be welcoming to others’ cultural and religious beliefs even if those beliefs are in conflict with their own.	Cultural competence
Empathy	In two weeks, the 8th grade teaching cluster will be getting a new math teacher who is Muslim. Next week’s professional development is dedicated to learning about the customs and etiquette of Islam to ensure that the new colleague feels welcomed in their new school.	Cultural proficiency
Empathy	An Asian student scored an 85 percent on today’s math quiz. The teacher expressed disappointment in the student by saying, “Come on, you are Asian! You should’ve scored better than this.”	Cultural precompetence
Empathy	A school counselor advised a biracial student that they would appear more professional if they perm their hair (removing the kinks) for their upcoming college visits.	Cultural destructiveness
Empathy	Your colleague was pulled over by the police on the way to work this morning. You hear other colleagues in the break room discussing the reasons why they were pulled over, and all of the suspected reasons were related to what they may have done (i.e., speeding, running a stop sign, etc.), rather than acknowledging that they may have been pulled over for being a person of color driving a luxury car.	Cultural blindness
Policy	A group of teachers and students are protesting the athletic department at a high school for changing the school’s logo to a silhouette of a Native American in a headdress; they have secured a place on the agenda for the next school board meeting to give a presentation on why this logo is offensive and trivializes Native Americans.	Cultural proficiency
Policy	A school that does not consider factors related to culture and ethnicity in their decision-making process is more democratic than a school who does consider factors related to culture and ethnicity in their decision-making process.	Cultural blindness
Policy	A school leader makes a “diversity hire” to introduce new ideas around school culture, curriculum, leadership, and/or supervision.	Cultural precompetence
Policy	A principal implemented a policy that all faculty and staff have to participate in quarterly potlucks; they are to bring a traditional dish specific to their culture, as a way to facilitate cross-cultural discourse.	Cultural competence
Policy	Students in a high-poverty school who are absent 10 consecutive days in a marking period are referred to special education for excessive absenteeism.	Cultural incapacity
Policy	A member of the school board is proposing a policy where English is the official language for all school related business in a community where the majority of the students’ and families’ primary language is Spanish.	Cultural destructiveness
Professionalism	A teacher asked the class who has traveled out of the country. A Black male raises his hand and states “I have.” The teacher asked the student three follow-up questions to corroborate his story.	Cultural incapacity
Professionalism	During lunch duty, teachers had a discussion in the presence of students about tomorrow’s district-wide racial sensitivity training. A teacher commented, “Why do we have to attend this training, the United States is post-racial because we have a Black president.”	Cultural blindness
Professionalism	The social committee at an elementary school is planning an awards ceremony and dinner for the staff and faculty. Six of their colleagues are vegan because of their religious beliefs. As a result, the committee has added a vegan selection to the menu to accommodate their colleagues’ religious beliefs.	Cultural competence
Professionalism	The reading specialist expressed some concerns to the Vice Principal about how her male students emasculates her during whole group instruction. The Vice Principal said, “Don’t worry about it, that’s just how boys behave.”	Cultural precompetence
Professionalism	A history teacher refuses to acknowledge black history month and does not want to teach content about black history in the United States.	Cultural destructiveness
Professionalism	An AP English teacher has assigned a research paper where the students were required to write about their religion. The teacher did not let their own religious beliefs cloud their judgement; the teacher researched the student’s religion while grading the paper to offer relevant constructive feedback back on the arguments made within the paper.	Cultural proficiency
Teaching practice	A novice teacher in a school where the majority of the students are African-American and Latino uses hip-hop as a way to make science relevant in the student’s social and cultural context.	Cultural competence
Teaching practice	An algebra teacher who teaches in a school where the majority of the students are on free and reduced lunch requires that each student buy a specific graphing calculator for Algebra, which retails for over 100 dollars.	Cultural blindness
Teaching practice	A teacher created a seating chart that placed ‘unteachable students’ in the back of the classroom, all of whom were male: 2 Latinos and 6 African Americans. In the same seating chart, her ‘good students,’ all white, were seated in preferential seating.	Cultural incapacity
Teaching practice	A teacher puts together a petition to request a textbook company to remove the narrative about the enslavement of Africans in the United States in their history textbook to accommodate teachers’ discomfort with discussing this sensitive topic.	Cultural destructiveness
Teaching practice	A teacher in a Title 1 school (high percentage of children from low-income families) checks homework for completeness rather than correctness as a strategy to improve student efficacy.	Cultural precompetence
Teaching practice	There is an increase of LGBTQIA+ students reporting that they are being harassed both verbally and physically in United States public schools. A group of students and teachers are researching the best practices to implement and create a supportive environment for these students to voice their concerns in their school.	Cultural proficiency

Note. The singular pronouns their or they are used as gender-neutral pronouns aimed to mitigate gender bias.

Notes

Author

DWAYNE RAY CORMIER, PhD, is an assistant professor in the School of Education at Virginia Commonwealth University, Oliver Hall, Room 4077, 1015 Main Street, Richmond, VA 23284-2020; cormierd2@vcu.edu . His research focuses on examining and bridging the sociocultural gap between teachers and their students who are minoritized, marginalized, and otherized within PreK–12 school systems.

References

Akker

J. V.

Gravemeiger

McKenney

Nieveen

(2006). Introducing educational design reserach. In Akker

J. V.

Gravemeiger

McKenney

Nieveen

(Eds.), Educational design research (pp. 3–7). Routledge.

American Psychological Association. (2020). Publication manual of the American Psychological Association.

Badiali

(2005). Standards for supervision of curriculum development. In Gordon

S. P.

(Ed.), Standards for instructional supervision: Enhancing teaching and learning (pp. 171–190). Eye on Education.

Banks

J. A.

(2013). Multicultural education: Characteristics and goals. In Banks

J. A.

McGee Banks

C. A.

(Eds.), Multicultural education: Issues and perspectives (8th ed., pp. 3–23). John Wiley & Sons, Inc.

Barab

Squire

(2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1), 1–14.

Bennett

M. J.

(1993). Towards ethnorelativism: A developmental model of intercultural sensitivity. In Paige

R. M.

(Ed.), Education for the intercultural experience (pp. 109–135). Intercultural Press.

Bensimon

E. M.

Rueda

Dowd

A. C.

Harris

F. I.

(2007). Accountability, equity, and practitioner learning and change: In honor of William M. Plater. Metropolitan Universities, 18(3), 25–45.

Bonilla-Silva

(2017). Racism without racists: Color-blind racism and the persistence of racial inequality in america. Rowman & Littlefield Publishers.

Brown

S. R.

(1986). Q technique and method: Principles and procedures. In Berry

W. D.

Lewis-Beck

M. S.

(Eds.), New tools for social scientists: Advances and applications in research methods (pp. 57–76). Sage.

10.

Castro

A. J.

(2010). Themes in the research on preservice teachers’ views of cultural diversity: Implications for researching millennial preservice teachers. Educational Researcher, 39(3), 198–210.

11.

Chandler

(2002). Semiotics: The basics. Routledge.

12.

Cobb

Confry

diSessa

Lehrer

Schauble

(2003). Design experiments in educational research. Educational Researcher, 32(1), 9–13.

13.

Cole

Penuel

O’Neill

(2018). Cultural-historical activity theory approaches to design-based research. Routledge.

14.

Creswell

J. W.

Poth

C. N.

(2018). Qualitative inquiry and research design: Choosing among five approaches (4th ed.). Sage.

15.

Delgado

Stefancic

(2017). Critical race theory: An introduction (3rd ed.). New York University Press.

16.

Donaldson

S. I.

Christie

C. A.

Mark

M. M.

(2009). What counts as credible evidence in applied research and evaluation practice? Sage.

17.

Easton-Brooks

(2019). Ethnic matching: Academic success of students of color. Rowman & Littlefield.

18.

Fishbein

(1967). Readings in attitude theory and measurement. Wiley.

19.

Fishman

B. J.

Penuel

W. R.

Allen

A.-R.

Cheng

B. H.

Sabelli

(2013). Design-based implementation research: An emerging model for transforming the relationship of research and practice. National Society for the Study of Education, 112(2), 136–156.

20.

Fluckinger

C. D.

(2014, July-September). Big five measurement via q-sort: An alternative method for constraining socially desirable responding. SAGE Open. https://doi.org/10.1177/2158244014547196

21.

Freire

(1970). Pedagogy of the oppressed. Bloomsbury Academic.

22.

Frey

W. H.

(2018). Diversity explosion: How new racial demographics are remaking America. The Brookings Institution.

23.

Gay

(1993). Building cultural bridges: A bold proposal for teacher education. Education and Urban Society, 25(3), 285–299.

24.

Gay

(2010). Culturally responsive teaching: Theory, research, and practice (2nd ed.). Teachers College Press.

25.

Geertz

(1994). Thick description: Towarad an interpretive theory of culture. In Martin

McIntyre

L. C.

(Eds.), Readings in the philosophy of social science (pp. 213–231). The MIT Press.

26.

Golombek

P. R.

Johnson

K. E.

(2004). Narrative inquiry as a mediational space: Examining emotional and cognitive dissonance in second-language teachers’ development . Teachers and Teaching: Theory and Practice, 10(3), 307–327.

27.

Gravemeijer

(1994). Developing realistic mathematics education. CD-Beta-Press.

28.

Hallgren

K. A.

(2012). Computing inter-rater reliability for observational data: An overview and tutorial. Tutorial in Quantitative Methods for Psychology, 8(1), 23–34.

29.

Howard

T. C.

(2010). Why race and culture matter in schools: Closing the achievement gap in America’s classrooms . Teachers College Press.

30.

Khalifa

M. A.

Gooden

M. A.

Davis

J. E.

(2016). Culturally responsive school leadership: A synthesis of the literature. Review of Educational Research, 86(4), 1272–1311.

31.

Koo

T. K.

M. Y.

(2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15, 155–163.

32.

Ladson-Billings

(1998). Teaching in dangerous times: Culturally relevant approaches to teacher assessment . Journal of Negro Education, 67(3), 255–267.

33.

Ladson-Billings

(2017). The (r)evolution will not be standardized. In Paris

Alim

H. S.

(Eds.), Culturally sustaining pedagogies: Teaching and learning for justice in a changing world (pp. 141–156). Teachers College Press.

34.

Ladson-Billings

Tate

W. F.

(1995). Toward a critical race theory of education. Teachers College Record, 97(1), 47–68.

35.

Lamberg

T. D.

Middleton

J. A.

(2009). Design research perspectives on transitioning from individual microgenetic interviews to a whole-class teaching experiment. Educational Researcher, 38(4), 233–245.

36.

Leung

(2015). Validity, reliability, and generalizability in qualitative research. Journal of Family Medicine and Primary Care, 4(3), 324–327.

37.

Lindsey

R. B.

Robins

K. N.

Terrell

R. D.

(1999). Cultural proficiency. Corwin.

38.

Lindsey

R. B.

Robins

K. N.

Terrell

R. D.

(2009). Cultural proficiency (3rd ed.). Corwin.

39.

Mansour

(2009). Science teachers’ beliefs and practices: Issues, implications and research agenda. International Journal of Environmental & Science Education, 4(1), 25–48.

40.

Marshall

P. L.

(1996). Multicultural teaching concerns: New dimensions in the area of teacher concerns research? The Journal of Educational Research, 89(6), 371–379.

41.

Middleton

Gorard

Taylor

Bannan-Ritland

(2008). The “compleat” design experiment: From soup to nuts. In Kelly

A. E.

Lesh

R. A.

Baek

J. Y.

(Eds.), Handbook of design experiment: Innovations in science, technology, engineering, and mathematics learning and teaching (pp. 21–46). Routledge.

42.

Milner

H. R.

IV . (2006). Preservice teachers’ learning about cultural and racial diversity: Implications for urban education. Urban Education, 14(4), 343–375.

43.

Milner

H. R.

IV . (2007). Race, culture, and researcher positionality: Working through dangers seen, unseen, and unforeseen. Educational Researcher, 36(7), 388–400.

44.

Milner

H. R.

IV . (2012). But what is urban education? Urban Education, 47(3), 556–561.

45.

Milner

H. R.

IV . (2017). Where’s the race in culturally relevant pedagogy? Teachers College Record, 119, 1–32.

46.

Milner

H. R.

IV . (2020a). Fifteenth annual AERA Brown Lecture in Education Research: Disrupting punitive practices and policies: Rac(e)ing back to teaching, teacher preparation, and brown. Educational Researcher, 49(3), 147–160.

47.

Milner

H. R.

IV . (2020b). Start where you are, but don’t stay there: Understanding diverisity, opportunity gaps, and teaching in today’s classrooms (2nd ed.). Harvard Education Press.

48.

Moss

P. A.

Haertel

E. H.

(2016). Engaging methodological pluralism. In Gitomer

D. H.

Bell

C. A.

(Eds.), Handbook of research on teaching (5th ed., pp. 127–248). American Educational Research Association.

49.

Moule

(2011). Cultural competence: A primer for educators (2nd ed.). Wadsworth Cengage Learning.

50.

National Association for Professional Development Schools. (2018). School-university partnerships. https://napds.org/journal/

51.

National Center for Educational Statistics. (2016). Digest of education statistics. https://nces.ed.gov/programs/digest/d16/tables/dt16_203.50.asp

52.

Neville

H. A.

Lilly

R. L.

Duran

Lee

R. M.

Browne

(2000). Construction and initial validation of the color-blind racial attitudes scale (CoBRAS). Journal of Counseling Psychology, 47(1), 59–70.

53.

Ozer

D. J.

(1993). The q-sort method and the study of personality development. In Funder

D. C.

Parke

R. D.

Tomlinson-Keasey

Widaman

(Eds.), Studying lives through time personality and development (pp. 1–11). American Psychological Association.

54.

Palinkas

L. A.

Horwitz

S. M.

Green

C. A.

Wisdom

J. P.

Duan

Hoagwood

(2015). Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Administration and Policy in Mental Health, 42(5), 533–544.

55.

Prasad

R. S.

(2001, November-December). Development of the HIV/AIDS q-sort instrument to measure physician attitudes. Family Medicine, 33(10), 722–778.

56.

Singleton

G. E.

(2015). Courageous conversations about race: A field guide for achieving equity in schools (2nd ed.). Corwin.

57.

Siwatu

K. O.

(2007). Preservice teachers’ culturally responsive teaching self-efficacy and outcome expectancy beliefs. Teaching and Teacher Education, 23(7), 1086–1101.

58.

Smith-Maddox

Solórzano

D. G.

(2002). Using critical race theory, Paulo Freire’s problem-posing method, and case study research to confront race and racism in education. Qualitative Inquiry, 8(1), 66–84.

59.

Stephenson

(1935). The study of behavior: Q-technique and its methodology. The University of Chicago Press.

60.

Stovall

(2006). Forging community in race and class: Critical race theory and the quest for social justice in education. Race Ethnicity and Education, 9(3), 243–259.

61.

Strauss

C. L.

(1962). The science of the concrete. In C. L. Strauss, The savage mind. The University of Chicago Press.

62.

Taylor

E. W.

(2002a). Analyzing research on transformative learning theory. In Mezirow

& Associates (Eds.), Learning as transformation: Critical perspectives on a theory in progress (pp. 285–328). Jossey-Bass.

63.

Taylor

E. W.

(2002b). Critical reflection and transformative learning: A critical review. ACE Journal of Lifelong Learning, 26, 77–95.

64.

U.S. Census Bureau. (2015, March 15). New census bureau report analyzes U.S. population projections. https://www.census.gov/newsroom/press-releases/2015/cb15-tps16.html

65.

U.S. Department of Education. (2016). The state of racial diversity in the educator workforce.

66.

Villegas

A. M.

Lucas

(2002). Preparing culturally responsive teachers: Rethinking the curriculum. Journal of Teacher Education, 53(1), 20–32.

67.

Watts

Stenner

(2005). Doing q methodology: Theory, method and interpretation. Qualitative Research in Psychology, 2, 67–91.

68.

Welsh

R. O.

Swain

W. A.

(2020). (Re)defining urban education: A conceptual review and empirical exploration of the definition of urban education. Educational Researcher, 49(2), 90–100.

69.

Wiggins

McTighe

(2005). Backwards design. In G. Wiggins & J. McTighe, Understanding by design (Second Edition). Association for Supervision and Curriculum Development.

70.

Wirth

(1945). The problem of minority groups. In R. Linton, The science of man in the world crisis (pp. 347–372). Columbia University Press.

71.

Yang

Montgomery

(2013). Gaps or bridges in multicultural teacher education: A Q study of attitudes toward student diversity. Teaching and Teacher Education, 30, 27–37.