Development and validation of the Cultural Responsiveness Assessment Measure (CRAM): A self-reflection tool for mental health practitioners when working with First Nations people

Abstract

Background:

The purpose of this study was to develop and to validate a measure of cultural responsiveness that would assist mental health practitioners across a range of disciplines, in Australia, to work with Indigenous clients.

Aim:

The Cultural Responsiveness Assessment Measure (CRAM) was developed to provide a tool for practitioners and students to evaluate their own culturally responsive practice and professional development.

Method:

Following expert review for face validity the psychometric properties of the measure were assessed quantitatively, from the responses of 400 mental health practitioners.

Results:

Confirmatory Factor Analysis yielded a nine factor, 36 item instrument that demonstrated strong convergent and discriminant validity as well as test-retest reliability.

Conclusions:

It is anticipated that the CRAM will have utility as both a learning tool and an assessment measure, for mental health practitioners to ensure that services are culturally responsive for Aboriginal and Torres Strait Islander people.

Keywords

Cultural responsiveness Indigenous self-reflection tool Australian Aboriginal and Torres Strait Islander

Introduction

Cultural responsiveness is recognised as a core competency for all psychology students and practitioners in Australia, as well as in other countries such as the United States and Canada (American Psychological Association, 2015; Canadian Psychological Association, 2021). The Australian Psychology Accreditation Council (2019) has included cultural responsiveness as a requirement for all levels of psychology qualification and practice. Beyond psychology, the Australian Association of Social Workers (AASW) Practice Standards state that ‘social workers practice respectfully and inclusively with regard to culture and diversity’ (Australian Association of Social Workers, 2023, p. 10). Such requirements place importance on methods for both students and practitioners to monitor and assess their cultural responsiveness for working with Indigenous people. This group has found mainstream health and mental health services culturally unsafe (Kendall & Barnett, 2015) and its members suffer mental illness at higher rates than the wider community with a suicide rate that is double that of the rest of the Australian population (Silburn et al., 2014). Self-assessment measures can improve the accuracy of self-reflection across multifaceted competency domains and enable practitioners to identify and monitor gaps in practice, and facilitate professional development (Rice et al., 2022). To aid self-assessment of the core competency of cultural responsiveness, this research aimed to develop and validate a measure for mental health practitioners. Structured from the elements of cultural responsiveness found within the literature (P. Smith et al., 2021) and client interviews (P. Smith et al., 2023a, 2023b) the Cultural Responsiveness Assessment Measure (CRAM) provides a tool for systematic self-evaluation across nine domains of culturally responsive practice. It is anticipated that the CRAM will provide a tool for mental health practitioners in a range of disciplines, to work in ways that will help to close the gap in mental health, which has been found to be culturally unsafe for Indigenous people (Kendall & Barnett, 2015; McGough et al., 2018).

Cultural responsiveness for practitioners is a recursive process that sets reflexivity at the centre of a dynamic and ongoing movement where the reflexive self, initiates exploration into the domains of learning, experience, and discovery for best outcomes for Indigenous people (P. Smith et al., 2022). A culturally responsive mental health workforce will help to decolonise psychology by setting aside the ethnocentric practices of the dominant culture, giving a place to Indigenous epistemologies and worldviews (Dudgeon & Walker, 2015). As a dynamic and ongoing process cultural responsiveness requires that individuals continue to monitor their progress and consider areas of strength and deficit, and so the need for a self-assessment instrument is a matter of great importance to inform sound professional practice.

A measure or scale or assessment instrument can capture the depth and breadth of multifaceted constructs beyond the limits of a single item (Boateng et al., 2018). Cultural responsiveness is a complex and multifaceted construct that is an overarching term for multiple discrete domains that are also interrelated (P. Smith et al., 2021). This multifaceted construct requires an assessment instrument that is holistic and all-encompassing enabling systematic self-evaluation across all domains. Previous measures which have sought to evaluate practitioner capabilities in cross-cultural or multi-cultural contexts have been mostly designed for populations outside of Australia, and have largely focused on knowledge, attitudes and awareness of culture and health rather than practitioner responsiveness or capabilities in working with Indigenous people (West et al., 2018). The CRAM is based on a framework that emerged from a concept analysis of cultural responsiveness found within the literature (P. Smith et al., 2021) and was subsequently developed into a model of reflexivity for teaching and learning (P. Smith et al., 2022). This model with nine domains centred on the reflexive self is recursive, non-binary and non-linear, prompting the need for ongoing practitioner self-evaluation. In a further study by the authors (P. Smith et al., 2023a), which sought to test this model with Indigenous former clients of mental health practitioners, knowledge and awareness were clear themes. Further, culturally safe practice that is free of racism and includes practitioner reflexivity were also identified in the responses from participants, indicating that current concepts of cultural responsiveness need to now move forward to a more expansive definition and understanding of this construct.

The aim of the present study was to develop a psychometrically sound instrument to assist mental health professionals and students at a practical level, to work more effectively with Indigenous people. The study sets out to develop the CRAM and to assess the psychometric properties of the newly developed measure.

Method

The term Indigenous is used respectfully within this article as a term to identify both Aboriginal and Torres Strait Islander people who are the First Nations people of Australia, and the first author, a Gamilaroi First Nations researcher, was assisted with cultural advice and support from an Aboriginal advisory group that included Gamilaroi Elders.

There are three important steps to creating a scale that is valid and reliable: item development, scale development and scale evaluation (Boateng et al., 2018). Item development sets forth an initial set of items that are deemed to be representative of the construct that is to be measured. Scale development is the process of organising the items into a structure that is user friendly, clear, and logical, and scale evaluation seeks to ensure that the instrument is psychometrically rigorous, and tests validity and reliability and in respect of construct validity, assuring that it measures constructs as hypothesised (Churchill, 1979).

Item development

Item development commenced from the conceptual model of cultural responsiveness (P. Smith et al. 2022), which is based on the literature review (P. Smith et al., 2021) and qualitative interviews (P. Smith et al., 2023a, 2023b), using both deductive and inductive methodologies (Morgado et al., 2017). This meant that items were initially deductively generated based on the following themes of cultural responsiveness (P. Smith et al., 2022): Awareness, Knowledge, Inclusive Relationships, Cultural Respect, Cultural Safety, Social Justice/Human Rights, Self-Reflection, Cultural Humility, and Cultural Competencies. To strengthen the validity of the scale an inductive analysis of data from a qualitative study (P. Smith et al., 2023a) were also considered with a view to discovering any additional emerging themes (Morgado et al., 2017). From these themes a pool of 50 items was generated to elicit responses for practitioners to evaluate their level of cultural responsiveness.

Scale development

To evaluate face validity of the item pool, a panel of academics and practitioners, both Indigenous and non-Indigenous, were consulted to provide feedback on the suitability and appropriateness of the initial 51 items, and whether they would adequately measure changes to dimensional aspects of cultural responsiveness. Responses from the thirteen panel members focused on specific items, and there were no recommendations to remove any of the domains, apart from a single comment that Knowledge and Awareness could be difficult for respondents to differentiate. Recommendations by each individual panel member for change of items and wording were incorporated after consensus by the research team and this process resulted in a reduced pool of 42 items, with six domains retaining 5 items and three domains with 4 items, shown in Table 1. All panel members were consulted a second time to review the latest version of the instrument which incorporated the panel’s suggested amendments, and to provide opportunity for any additional comments or suggestions. Responses from this second round were integrated following review and consensus by the research team. These responses indicated that the items, fifteen of which would be reverse scored, and the nine domains of the CRAM were deemed to encapsulate the construct of cultural responsiveness. The panel also approved the response format of a 7-point Likert Scale, with choices ranging from ‘Strongly disagree’ to ‘Strongly agree’. Seven-point Likert scales possess greater sensitivity and accuracy than scales with less points, so the 7-point scale was adopted for this study (Finstad, 2010).

Table 1.

Items and wording of the CRAM initially proposed by the expert panel.

Awareness

1. Past history does not impact the mental health of Indigenous people today. (Deleted)

2. There are many needs and issues affecting Indigenous communities, individuals, and families that differ from those in the wider community.

3. I can specifically name some of the causes that disadvantage the mental health of some Indigenous people in this country.

4. I believe that all Indigenous cultures are the same regardless of their location. (Deleted)

5. I understand the types of strength and resilience that have enabled Indigenous people to resist colonisation.

Knowledge

6. I have actively sought information and knowledge about Indigenous people.

7. I have attempted to learn the history of Indigenous Australia.

8. Indigenous people are better off since colonisation.

9. I have made attempts to learn about local Indigenous culture.

10. Indigenous people need to take more responsibility for their own mental health. (Deleted)

Inclusive relationships

11. I would seek to work with family and/or community members of Indigenous clients including having them present during sessions.

12. In order to assist me and if appropriate I would seek the help of an Indigenous mentor.

13. Indigenous healers are important.

14. Accessing community members or Indigenous mentors is not important for successful outcomes.

Cultural respect

15. I respect Indigenous culture and cultural values.

16. My own culture influences the way that I think and behave.

17. For some Indigenous people, spirituality has an important place within their culture.

18. Indigenous culture is not an important part of Indigenous people’s social and emotional wellbeing.

Cultural safety

19. I would make my practice more culturally safe, in ways such as seeking the guidance of Indigenous clients.

20. I take time with clients to explain therapeutic practice or to answer questions.

21. Changes to the workplace environment would not make it more culturally safe.

22. Asking an Indigenous client about their culture is not important for best outcomes in therapeutic practice.

Social justice and human rights

23. I understand the meaning of my own position of power and privilege in relation to therapeutic practice.

24. I understand how the dominant culture has disadvantaged Indigenous people.

25. I understand how power and privilege and their psychological impacts have disadvantaged Indigenous people.

26. Decolonising mental health is not an important part of therapeutic practice.

27. Social justice means treating everybody the same. (Deleted)

Self-reflection

28. I frequently take the time to reflect on my thoughts, emotions, and behaviours in relation to therapeutic practice.

29. I use self-reflection to evaluate my values and my professional practice.

30. I have used self-reflection to make changes to professional practice or to further learn.

31. Self-reflection following a client session does not better inform subsequent sessions.

32. Reflecting on my own cultural values and their impact on others will not help me to work more effectively with my clients. (Deleted)

Cultural humility

33. Listening is an important first step in understanding a client’s culture.

34. I am open to constructive criticism even if it causes me some discomfort.

35. I can learn from Indigenous clients who are knowledgeable of their own culture.

36. Understanding the cultural values of others does not contribute to better outcomes in therapeutic practice.

37. When working with Indigenous clients I strive to address deficits in my knowledge and ways of working.

Cultural competencies

38. Deep listening is an important intervention when working with Indigenous people.

39. Working holistically with Indigenous clients is important.

40. I have used an Indigenous-specific therapeutic practice or an instrument that is validated by and for Indigenous people. (Deleted)

41. Specific interventions for Indigenous people are not necessary for best outcomes in therapeutic practice.

42. The Western medical model of mental health is appropriate for all people.

Deleted denotes items omitted following confirmatory factor analysis. Final items of the scale are in Appendix A.

Scale evaluation

The second aim of this study was to evaluate the psychometric properties of the newly developed CRAM. Following ethics approval from the university’s Human Research Ethics Committee, this second stage of scale development consisted of a pilot study of the 42-item instrument, as an online survey. Participants were able to complete the survey anonymously or opt to provide an email address and complete the survey a second time, 1 month later for assessment of test-retest reliability.

Participants

Mental health practitioners and trainees across a variety of disciplines were invited to participate through electronic advertisements and printed flyers resulting in the participation of 400 mental health practitioners, with their demographics shown in Table 2.

Table 2.

Participant demographics (N = 400).

Gender
Female	322	(81%)
Male	68	(17%)
Non-binary	6	(1.5%)
Rather not say	2	(0.5%)
Current role
Psychologist	163	(37%)
Social worker	19	(4%)
Mental health nurse	7	(2%)
Counsellor	13	(3%)
Student in a mental health discipline	180	(41%)
Other	56	(12%)
Years of experience	8.06 (mean)	10.05 (SD)

Procedure

Participants were asked to complete the survey, hosted on Qualtrics^TM (2022; Provo, UT) and each participant provided online informed consent prior to undertaking the survey. Participants were also asked to provide their email address if they were willing to be contacted again to complete the survey once more a month later for the purposes of test-retest reliability

Materials

Cultural Responsiveness Assessment Measure (CRAM)

The psychometric qualities of the newly developed CRAM were the focus of the present study and as such are reported in the results section below.

Cultural Safety Training Questionnaire

The survey also asked participants as a measure of convergent validity, to respond to the items of the Cultural Safety Training Questionnaire (CSTQ; Ryder et al., 2017), which is a 15 item self-report scale. This instrument is a measure of cultural safety and evaluates attitude change across two factors: transformative unlearning and critical thinking for students and academic staff involved in Aboriginal Health. This instrument was reported as having overall adequate reliability for the two thematic areas with Cronbach’s α > .70, (Boateng et al., 2018) and test-retest reliability with overall intraclass correlation (ICC) = 0.7 (Ryder et al., 2017). Reliability analysis for this sample indicated Cronbach’s α of .82.

Riverside Life Satisfaction Scale

To assess divergent validity participants completed the Riverside Life Satisfaction Scale (RLSS; Margolis et al., 2019), a 6-item one-factor self-report scale that assesses contentment with life, desire for change and absence or presence of envy. It is considered to possess strong correlation (r = 0.95) with another similar measure the Satisfaction with Life Scale (Diener et al., 1985) as well as high internal consistency (α = 0.75) and test-retest reliability (Margolis et al., 2019). Reliability for this sample was estimated with Cronbach’s α = .86.

Analysis

Exploration of the dimensions and content of factors is a necessary first step before assessment of validity and reliability can be conducted (Boateng et al., 2018). Confirmatory Factor Analysis (CFA) using Jamovi Project (2022) statistical software explored both the content and the relationships between factors of the CRAM. CFA was selected as there was already an a priori hypothesis of the factor structure, and CFA is used where the purpose is to test theory that has already been established and where factors have been determined (Kääriäinen et al., 2011). CFA seeks to test whether the measures of a construct, in this case cultural responsiveness, are consistent with a priori understandings from literature and qualitative data of that construct (Awang, 2015). The measure in this study was derived from the theoretical model (P. Smith et al., 2022) developed from the published literature review (P. Smith et al., 2021) and qualitative interviews (P. Smith et al., 2023a). Thus, for this study which was hypothesis or theory driven rather than data driven, CFA was deemed to be the best method because it tests the structure of existing theory, unlike inductive methodologies (Kääriäinen et al., 2011; Maruyama, 1998). Exploratory Factor Analysis (EFA) was not deemed satisfactory for this study because it is data driven and it is adopted in research where there is no a priori assumption of the associations of variables (Kääriäinen et al., 2011).

The total sample of 400 survey responses were included in the CFA, and this number is deemed to be a suitable sample size for CFA (Cattell, 1978; Gorsuch, 1983). Based on a rating system convention for sample sizes in factor analysis by Comrey and Lee (1992) this study’s sample would rate good (>300).

Results

Confirmatory factor analysis

All items demonstrated positive factor loadings (Table 3) and CFA where items with factor loadings with a standard estimate coefficient of <0.4 were deleted. This accords with the convention of Tabachnick and Fidell (2000) where 0.4 as a minimum, accounts for more than 10% of the item’s variance and ‘as a rule of thumb, only variables with loadings of 0.32 and above are interpreted’ (p. 625). This process resulted in reduction of the instrument to 36 items, with all factors possessing at least 3 items. The 36 items, together with the scale instructions, are shown in Appendix A (Figure 1).

Table 3.

Factor loadings with deleted items in bold.

Factor	Item	Estimate	SE	Z	p	Standard estimate
Awareness	A1	0.410	0.0892	4.59	<.001	0.302	Deleted
	A2	0.738	0.0897	8.24	<.001	0.582
	A3	0.994	0.0971	10.24	<.001	0.737
	A4	0.401	0.0709	5.65	<.001	0.331	Deleted
	A5	0.722	0.0931	7.76	<.001	0.503
Knowledge	K1	1.350	0.0566	23.88	<.001	0.929
	K2	1.173	0.0508	23.10	<.001	0.911
	K3	0.574	0.0721	7.96	<.001	0.397
	K4	1.125	0.0681	16.53	<.001	0.731
	K5	0.429	0.0834	5.14	<.001	0.262	Deleted
Incl. relationships	IR1	0.734	0.0578	12.69	<.001	0.615
	IR2	0.761	0.0475	16.01	<.001	0.738
	IR3	0.879	0.0508	17.30	<.001	0.777
	IR4	0.708	0.0746	9.49	<.001	0.479
Respect	CR1	0.528	0.0348	15.18	<.001	0.700
	CR2	0.484	0.0485	9.97	<.001	0.498
	CR3	0.516	0.0334	15.42	<.001	0.724
	CR4	0.526	0.0480	10.96	<.001	0.536
Safety	CS1	0.628	0.0461	13.63	<.001	0.652
	CS2	0.619	0.0522	11.87	<.001	0.582
	CS3	0.560	0.0817	6.86	<.001	0.352
	CS4	0.613	0.0674	9.09	<.001	0.456
Justice	SJ1	0.704	0.0471	14.94	<.001	0.677
	SJ2	0.935	0.0395	23.68	<.001	0.923
	SJ3	0.985	0.0428	23.02	<.001	0.907
	SJ4	0.766	0.0709	10.80	<.001	0.521
	SJ5	0.316	0.1095	2.89	.004	0.149	Deleted
Self-reflection	SR1	0.984	0.0446	22.08	<.001	0.880
	SR2	1.053	0.0422	24.94	<.001	0.946
	SR3	0.982	0.0442	22.22	<.001	0.884
	SR4	0.620	0.0704	8.82	<.001	0.432
	SR5	0.550	0.0867	6.35	<.001	0.318	Deleted
Humility	CH1	0.454	0.0235	19.29	<.001	0.822
	CH2	0.451	0.0359	12.56	<.001	0.595
	CH3	0.589	0.0277	21.30	<.001	0.878
	CH4	0.604	0.0622	9.72	<.001	0.478
	CH5	0.705	0.0537	13.12	<.001	0.626
Competencies	CC1	0.716	0.0369	19.42	<.001	0.835
	CC2	0.750	0.0424	17.69	<.001	0.783
	CC3	0.487	0.0971	5.02	<.001	0.263	Deleted
	CC4	0.718	0.0697	10.30	<.001	0.510
	CC5	0.717	0.0609	11.77	<.001	0.574

Figure 1.

Final model structure of the CRAM. (R) denotes reverse scored items.

With 400 responses in the CFA for a 36-item instrument, a response-to-item ratio of 11 to 1 was achieved, indicating a suitable sample size for validity and stability of factor loadings (Boateng et al., 2018). CFA fitness indices which assist in understanding how the data support or fit the instrument, returned mixed results. The Root Mean Square Error of Approximation (RMSEA) showed satisfactory Absolute Fit (0.08) meeting the <0.09 threshold suggested by McNeish et al. (2018) but the Comparative Fit Index (CFI) and Tucker-Lewis Index (TLI) as measures of Incremental Fit were slightly below acceptance levels of >0.90 at 0.79 and 0.77 respectively (Awang, 2015). However, these cannot be discounted as irrelevant (McNeish et al., 2018) and rather than asking whether there is exact fit it can be more relevant to ask the degree of lack of fit and what meaning this then has for the instrument, which in this case may not affect its validity (Browne & Cudeck, 1992; McNeish et al., 2018). Marsh et al. (2004) have cautioned against the universal application of cutoff values for fit indices in that these can be affected by sample size (under approximately 500) and model complexity, such as the nine-factor measure that comprises the CRAM. Nonetheless, the CFA fit indices indicated that the original a priori nine factor model performed well overall and that a more parsimonious model, with less factors, was not indicated, supporting the structure of the cultural responsiveness model.

Internal reliability

The 36 items comprising the total CRAM demonstrated strong overall internal reliability (α = .94), and reliability coefficients for all factors are presented in Table 4. The individual factors in general showed acceptable reliability with α ⩾ .7 being acceptable (Boateng et al., 2018) apart from Awareness (α = .62) and Cultural Safety (α = .58) which were retained for reasons outlined in the discussion.

Table 4.

Internal reliability coefficients.

Factor	Cronbach’s α (N = 400)
Awareness	0.62
Knowledge	0.81
Inclusive relationships	0.72
Cultural respect	0.68
Cultural safety	0.58
Social justice	0.81
Self-reflection	0.83
Cultural humility	0.73
Cultural competencies	0.74
Total scale	0.94

Test-retest reliability

In respect of test-retest reliability, which measures the temporal stability of the instrument, 150 participants responded to the invitation to complete the measure a second time and were emailed a link to complete the survey again. There were 50 responses received at timepoint 2 which is deemed a suitable sample size within the minimal acceptable error rates proposed by McMillan and Hanson (2014). Additionally, a sample of 50 involved in 2 timepoint responses delivers a correlation with power = 90%, p < .05 with the null hypothesis at 0 (Bujang & Baharum, 2017). Generally, a response rate of 32% to electronic surveys is deemed average (Shannon & Bradshaw, 2002), and the response rate to this survey at timepoint 2 was 33%. This is considered a good response rate when there are often diminishing returns for multiple surveys attributed to what has been called survey fatigue (Porter et al., 2004). The mean total score on the CRAM at timepoint 1 was 6.46 and at timepoint 2 was 6.44. The Pearson product-moment correlation coefficient was used in this instance, as a measure of reliability (Boateng et al., 2018) and a correlation coefficient of 0.5, p < .001, indicated large effect size for test-retest reliability (Cohen, 1988) and suggesting that the scale assesses a stable construct that shows some changeability.

Construct validity

For purposes of construct validity, measures of convergent and discriminant or divergent validity were used employing the total sample (N = 400). In terms of convergent validity total scores on the CRAM were compared using the Pearson product-moment correlation with the total scores on the Cultural Safety Training Questionnaire (CSTQ; Ryder et al., 2017), an instrument that was deemed to measure similar constructs as the CRAM. A correlation of 0.805, p < .001 indicated a large effect size (Cohen, 1988).

Evaluation of divergent validity was conducted by comparison with the Riverside Life Satisfaction Scale (Margolis et al., 2019) which measures constructs deemed to be fairly conceptually distinct from those measured by the CRAM, despite both measures similarly investigating personal beliefs and values, the foci of the two instruments are diverse. A result of 0.195, p < .001 indicated that there was small correlation between the scores on these two instruments (Cohen, 1988), supporting the position that the CRAM is a novel instrument that is not simply reflecting other measures (Boateng et al., 2018), in this case a measure of a generally positive view of one’s life.

Discussion

The Cultural Responsiveness Assessment Measure (CRAM) has been developed as a tool for mental health practitioners to self-assess across all components of culturally responsive practice when working with Indigenous people. The CRAM is underpinned by a literature review (P. Smith et al., 2021), a conceptual model (P. Smith et al., 2022) and qualitative interviews with Indigenous clients of mental health practitioners (P. Smith et al., 2023a; 2023b). This paper detailed the development and initial psychometric evaluation of the CRAM, in accordance with recognised scale creation guidelines (e.g., Boateng et al., 2018). The initial results of psychometric testing, as presented above, suggest an instrument with strong face validity, strong internal reliability, and test-retest reliability. Item factor loadings were also considered suitable for the nine factors, supporting the a priori model of the instrument. Consideration was given to excluding the Awareness and Cultural Safety factors from the instrument, due to low reliability, but these factors were retained because they did not detract from the CRAM’s overall reliability. These two factors in this instrument, also serve as a reminder to mental health professionals, in their self-assessment, that they are still important parts of culturally responsive practice, as identified in the related qualitative interview study of Indigenous clients of mental health practitioners (P. Smith et al., 2023a; 2023b). The CFA also indicated that there was some overlap between factors, and future psychometric evaluations may eventually indicate a reduced factor structure for the CRAM is warranted. However, the current evaluation justified all domains of the conceptual model and retained the nine-factor structure of the CRAM.

The strong test-retest reliability calculated between the timepoint 1 and timepoint 2 data suggests stability of the instrument across two administrations, but unlike what would be found with a higher coefficient, it suggests that the measure also possesses elements of change and fluctuation. This would be expected with an instrument such as this, in that it assesses constructs for some people, particularly trainees, whose results from timepoint 1 to timepoint 2 may be influenced by a learning effect (Hart, 2014).

In relation to convergent validity the strong correlation between the CSTQ and the CRAM, suggests that the two measures partially assess a similar construct. In comparison, a small correlation with the RLSS which was considered to be discriminant or divergent from the CRAM, supports the position that the two measures deemed to be conceptually diverse are also sufficiently statistically distinct. These two results support the construct validity of the CRAM and provide evidence to support its ability to measure the dimensions of cultural responsiveness.

It was the intention of the authors from the outset that the CRAM would be an instrument that would assist both students and practitioners within mental health to monitor their cultural responsiveness, and it is noted that 41% of survey respondents identified as students and participants were from a range of disciplines and with varied experience levels. It is possible that results from a sample with a higher percentage of experienced practitioners could deliver different results. However, the results of this study indicate both the utility and relevance of the CRAM as a novel instrument across a range of mental health disciplines and for a variety of experience levels. Most similar instruments, including the CSTQ, measure single or a limited number of components of what is now proposed as being definitive of cultural responsiveness. The CRAM provides a holistic conceptualisation of cultural responsiveness, based on the broad and multi-factor model identified in research (e.g., P. Smith et al., 2021, 2023a). This detailed model and measure provides a practical framework for practitioners, by identifying multiple factors and skills that can be specifically targeted in the development of cultural responsiveness. Thus, the CRAM provides a broad base for practitioners to assess relevant skills and attitudes when working with Indigenous peoples. This instrument moves mental health practice beyond competence models and the acquisition of knowledge and toward a more self-reflexive and collaborative approach.

This study was limited to mental health students and practitioners within the Australian context, and studies assessing the applicability of the CRAM for other locations is needed. Future cross-cultural studies may find that some modifications for local culture may be necessary. For example, items specifically mentioning an Australian context could be modified to specify other cultural contexts for psychometric evaluation. Additionally, it is hoped that the CRAM could offer a foundation for an instrument to assist those who are involved in other disciplines beyond mental health. The scale and its factor structures may also offer a starting point that will have international applicability for practitioners whose work involves First Nations people. However, as to the suitability of the CRAM for use with non-Indigenous minority or marginalised groups such as refugees, immigrants, and ethnic minorities, this would require major structural changes. For example, specific items about Elders, spirituality, colonisation, and traditional healers, would need to be carefully reconsidered.

The results of the present study support the proposition that cultural responsiveness is a multi-faceted construct in terms of assessment and skill development. The CRAM total score demonstrated strong internal reliability and construct validity with acceptable test-retest reliability, and support for the nine-factor model of the instrument.

Conclusion

This study outlines the development of an instrument that may assist mental health students and practitioners to support Indigenous clients in ways that are culturally responsive. The CRAM, as a self-report instrument offers mental health practitioners a way to understand and assess their own areas of strength as well as those areas that need further attention when they are working with Indigenous clients.

Footnotes

Appendix A

Acknowledgements

It is acknowledged that the work of this study took place on the traditional lands of the Gamilaroi and Anaiwan nations of northwest New South Wales, Australia, and respects are paid to Elders past, present and emerging. Acknowledgement is also given to the Aboriginal Advisory Group, which assisted with cultural advice and support.

Author contributions

All authors contributed to the research and all versions of the manuscript. PS conducted all stages of the research and prepared the draft version of the manuscript. KR, NS and KU contributed to all stages of the research and proofread draft manuscripts. Statistical analyses were reviewed by NS.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The first author, (PS) a proud Gamilaroi man, is supported by a University of New England Aboriginal and Torres Strait Islander Higher Degree by Research scholarship under the Australian Government Research Training Program (RTP).

ORCID iD

Peter Smith

Data availability

Data may be made available on reasonable request to the corresponding author.

References

American Psychological Association. (2015). Competencies for psychology practice in primary care. http://www.apa.org/ed/resources/

Australian Association of Social Workers. (2023). AASW practice standards. https://aasw.asn.au/practice-standards-2023/

Australian Psychology Accreditation Council. (2019). Accreditation standards for psychology practice. Australian Government. http://www.psychologycouncil.org.au/

Awang

(2015.) Validating the measurement model: CFA. In Awang

(Ed.). SEM made simple: A gentle approach to learning structural equation modeling. (pp. 54–74). MPWS Rich Publication.

Boateng

Neilands

Frongillo

Melgar-Quiñonez

Young

(2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer. Frontiers in Public Health, 6, 149. https://doi.org/10.3389/fpubh.2018.00149

Browne

Cudeck

(1992). Alternative ways of assessing model fit. Sociological Methods and Research, 21(2), 230–258.

Bujang

Baharum

(2017). A simplified guide to determination of sample size requirements for estimating the value of intraclass correlation coefficient: A review. Archives of Orofacial Sciences, 12(1), 1–11.

Canadian Psychological Association. (2021). Proposed revisions for the 6th revision of the accreditation standards for doctoral and residency programs in professional psychology. https://cpa.ca/docs/accreditation/

Cattell

(1978). The scientific use of factor analysis. Plenum.

10.

Churchill

(1979). A paradigm for developing better measures of marketing constructs. Journal of Marketing Research, 16(1), 64–73. https://doi.org/10.2307/3150876

11.

Cohen

(1988). Statistical power analysis (2nd ed.). Erlbaum.

12.

Comrey

Lee

(1992). A first course in factor analysis. Erlbaum.

13.

Diener

Emmons

Larsen

Griffin

(1985). The satisfaction with life scale. Journal of Personality Assessment, 49, 71–75. https://doi.org/10.1207/s15327752jpa4901_13

14.

Dudgeon

Walker

(2015). Decolonising Australian psychology: Discourses, strategies and practice. Journal of Social and Political Psychology, 3(1), 276–297. https://doi.org/10.5964/jspp.v3i1.126

15.

Finstad

(2010). Response interpolation and scale sensitivity: Evidence against 5-point scales. Journal of Usability Studies, 5, 104–110.

16.

Gorsuch

R. L.

(1983). Factor analysis (2nd ed.). Erlbaum.

17.

Hart

(2014). Evaluation of ATAPS cultural competence training for the mental health workforce [Unpublished report]. Australian Indigenous Psychologists Association.

18.

Jamovi Project. (2022). Jamovi (Version 2.3). Computer software. https://www.jamovi.org

19.

Kääriäinen

Kanste

Elo

Pölkki

Miettunen

Kyngäs

(2011). Testing and verifying nursing theory by confirmatory factor analysis. Journal of Advanced Nursing, 67(5), 1163–1172. https://doi.org/10.1111/j.1365-2648.2010.05561.x

20.

Kendall

Barnett

(2015). Principles for the development of Aboriginal health interventions: Culturally appropriate methods through systemic empathy. Ethnicity and Health, 20(5), 437–452. https://doi.org/10.1080/13557858.2014.921897

21.

Margolis

Schwitzgebel

Ozer

Lyubomirsky

(2019). A new measure of life satisfaction: The Riverside Life Satisfaction Scale. Journal of Personality Assessment, 101(6), 621–630. https://doi.org/10.1080/00223891.2018.1464457

22.

Marsh

Hau

Wen

(2004). In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler’s (1999) findings. Structural Equation Modeling, 11, 320–341. https://doi.org/10.1207/s15328007sem1103_2

23.

Maruyama

(1998). Basics of structural equation modeling. Sage.

24.

McGough

Wynaden

Wright

(2018). Experience of providing cultural safety in mental health to Aboriginal patients: A grounded theory study. International Journal of Mental Health Nursing, 27(1), 204–213. https://doi.org/10.1111/inm.12310

25.

McMillan

Hanson

(2014). Sample size requirements for establishing clinical test-retest standards. Ear and Hearing, 35(2), 283–286. https://doi.org/10.1097/01.AUD.0000438377.15003.6B

26.

McNeish

Hancock

(2018). The thorny relation between measurement quality and fit index cutoffs in latent variable models. Journal of Personality Assessment, 100(1), 43–52. https://doi.org/10.1080/00223891.2017.1281286

27.

Morgado

Meireles

Neves

Amaral

Ferreira

(2017). Scale development: Ten main limitations and recommendations to improve future research practices. Psicologia: Reflexão e Crítica, 30, 1–20. https://doi.org/10.1186/s41155-016-0057-1

28.

Porter

Whitcomb

Weitzer

(2004). Multiple surveys of students and survey fatigue. New Directions for Institutional Research, 2004(121), 63–73. https://doi.org/10.1002/ir.101

29.

Qualtrics. (2022). Qualtrics XM [Computer software]. https://qualtrics.com/core-xm/survey-software/

30.

Rice

Schutte

Cosh

Rock

Bannister

Sheen

(2022). The utility and development of the competencies of professional psychology rating scales (COPPR). Frontiers in Education, 7, 818077. https://doi.org/10.3389/feduc.2022.818077

31.

Ryder

Mackean

Ullah

Burton

Halls

McDermott

Edmondson

(2017). Development and validation of a questionnaire to measure attitude change in health professionals after completion of an aboriginal health and cultural safety training programme. The Australian Journal of Indigenous Education, 48(1), 24–38. https://doi.org/10.1017/jie.2017.37

32.

Shannon

Bradshaw

(2002). A comparison of response rate, response time, and costs of mail and electronic surveys. The Journal of Experimental Education, 70(2), 179–192. https://doi.org/10.1080/00220970209599505

33.

Silburn

Robinson

Leckning

Henry

Cox

Kickett

(Eds.). (2014). Preventing suicide among Aboriginal Australians (2nd ed.). Commonwealth of Australia.

34.

Smith

Rice

Schutte

Usher

(2021). Cultural responsiveness for mental health professionals working with Aboriginal and Torres Strait Islander clients: A concept analysis. Australian Psychologist, 56(6), 446–457. https://doi.org/10.1080/00050067.2021.1974281

35.

Smith

Rice

Schutte

Usher

(2022). Reflexivity: A model for teaching and learning cultural responsiveness in mental health. Australian Psychologist, 57(4), 209–214. https://doi.org/10.1080/00050067.2022.2078648

36.

Smith

Rice

Schutte

Usher

(2023a). Cultural responsiveness; listening to the voices of Australian Indigenous people: A qualitative study. School of Psychology, University of New England. (Manuscript submitted for publication).

37.

Smith

Rice

Schutte

Usher

(2023b). Healing through meaning as an aspect of spirituality for Indigenous Australians: A qualitative study. AlterNative. Advance online publication. https://doi.org/10.1177/11771801231193169

38.

Tabachnick

Fidell

(2000). Using multivariate statistics (4th ed.) Allyn and Bacon.

39.

West

Mills

Rowland

Creedy

. (2018). Validation of the first people’s cultural capability measurement tool with undergraduate health students: A descriptive cohort study. Nurse Education Today, 64, 166–171. https://doi.org/10.1016/j.nedt.2018.02.022