Challenges in the Ethical Review of Clinical and Biomedical Research in Malaysia: A Mixed Methods Study

Abstract

Empirical evidence of the ethical review of clinical and biomedical research in Malaysia is limited. We have conducted a convergent mixed methods research, which comprises an online survey and semistructured interviews to examine the challenges in the ethical review of clinical and biomedical research. Data collected reveal that the ethics review process is inconsistent, duplicate, and inadequate. The results indicate a strong need for a centralized ethical review mechanism and a national system for mutual recognition of ethics reviews to improve the effectiveness and efficiency of the ethics review system in Malaysia. A joint research ethics committee review between Malaysia and sponsoring countries for multinational research should also be encouraged as it could address the concerns of the lack of expertise and training.

Keywords

ethical review clinical trials biomedical research challenges research ethics committee Malaysia

Introduction

The establishment of an effective research ethics committee (REC) is one of the critical ethical infrastructures of research organizations, which is necessary for the promotion of ethical conduct of research and protection of human subjects (Hyder, Dawson, Bachani, & Lavery, 2009; Silverman et al., 2015). However, literature reveals that REC members in low- and middle-income countries (LMICs) are lacking in expertise and have limited review competencies to conduct an ethical review of biomedical research or clinical trials (Barchi, Matlhagela, Jones, Kebaabetswe, & Merz, 2015; Barchi, Singleton, & Merz, 2014; Bhowmick, Banerjee, Sikdar, & Chatterjee, 2014; Dauda & Dierickx, 2014; Nyika et al., 2009).

The US National Institutes of Health, Fogarty International Center (NIH-FIC), a leading funder of international research ethics education for resource-limited settings, established a funding mechanism in the year 2000 to support ethics training program in response to the increasing need for research ethics expertise in LMICs (Ali, Kass, Sewankambo, White, & Hyder, 2014). The NIH-FIC has been awarding ethics training grants to support scholars from Africa and the Middle East to attend research ethics and bioethics training. These long-term ethics training programs range from Diploma to Postgraduate Diploma and Master's level of studies. The training includes providing opportunities to the Chair and members of RECs of LMICs in the following activities: observe how the medical and public health institutional review boards work; participate in mock ethical review sessions as well as in real REC sessions at universities in the United States. These preparatory courses will subsequently build the capacity of RECs to conduct scientific and ethical reviews of international collaborative research, and strengthen RECs in LMICs especially those in Africa and the Middle East (Ndebele et al., 2014; Silverman, Edwards, Shamoo, & Matar, 2013).

In Malaysia, all the biomedical research and clinical trials involving human subjects are reviewed by the 13 registered RECs in the country. These RECs have been established mainly by hospitals, universities, medical institutions, and the National Institute of Health. The National Pharmaceutical Control Bureau (NPCB) of the Ministry of Health (MOH) Malaysia is responsible for the registration and oversight of all the RECs. NPCB will carry out a service audit for each registered REC every three years, to ensure the entity complies with the Malaysian Guideline for Good Clinical Practice (Malaysian GCP), Malaysian regulatory requirements, and other established guidelines (Hooi Yan, Mohd Salim, Siti Nurani, & Wah Yun, 2019). To date, there is little reported evidence of the views and experiences of REC members, scientists, and research stakeholders about the ethical review process in Malaysia. A study conducted by Kaur (2011) demonstrates that the ethics review process in Malaysia has failed to adequately provide meaningful protection to human subjects of clinical trials; there is a lack of formal and systematic training for the ethics committee members, and the institutional structures of ethics committees are inadequate.

The objective of this study is to identify the challenges of the ethics review of biomedical research and clinical trials in Malaysia from the perspectives of scientists, REC members’ and research stakeholders. This study will also recommend ways to address these challenges effectively. The findings of this study will have two major contributions: firstly, enable the regulatory of RECs to understand the issues and problems arising from the ethical review of clinical trials or biomedical research; secondly, how to resolve the issues as well as addressing the problems adequately so that the RECs can provide meaningful protection to human subjects.

Research Method

Study Design

This study is an empirical, cross-sectional, convergent mixed methods research, which comprises an online survey and semistructured interviews. The procedures for implementing a convergent design are presented in the procedural flowchart in Figure 1 (Creswell & Plano Clark, 2018). As shown in this diagram, the quantitative and qualitative strands were implemented within the same phase of the research process, and each had an equal emphasis in this study. Specifically, the online survey is the quantitative arm dealing with scientists and research stakeholders to identify the challenges faced by RECs in conducting an ethics review of biomedical research and clinical trials in Malaysia. Concurrently, the REC members and scientists were invited for a semistructured interview. The one-on-one semistructured interviews method is used in this study because it can accomplish two tasks: provide an in-depth exploration of the critical challenges faced by REC members during the deliberation and ethical review process; and formulate solutions to meet the challenges appropriately and adequately. The strength of the interview method is that the researcher can examine the ethical review process in depth based on the experts’ specific points of view during the interview sessions, and to explore the real-life processes of an ethics review (Dove & Garattini, 2018).

Figure 1.

Flowchart for convergent mixed methods design procedures.

The large-scale survey of scientists, REC members, and other research stakeholders provides quantitative data about the challenges of RECs in Malaysia, which can be generalized to a population; while semistructured interviews give an in-depth understanding from the angles of a few people. The survey has a three-fold function: improves reliability; offsets the weaknesses inherent in a semistructured interview; and enhances the external validity of the study. Therefore, this convergent mixed methods design helps the researcher obtain an in-depth qualitative exploration as well as a rigorous, high-power quantitative examination of the challenges of the ethical review process of RECs in Malaysia (Creswell & Plano Clark, 2018).

Online Survey Instrument and Participants

The respondents of this study are confined to scientists, REC members, and research stakeholders who are involved in biomedical research and clinical trials in Malaysia. We identified the respondents of this study in cooperation with the secretariats of 13 RECs that are registered with the MOH and 11 hospital clinical research centers in the different states of Malaysia. An online survey program known as Qualtrics was used to conduct the survey. An invitation email with a unique link to the questionnaire was sent to 1667 scientists and research stakeholders in Malaysia. The email explained the nature of the study and guaranteed anonymity of the individual participants. A reminder email was sent to each recipient of the questionnaire subsequently.

The survey questionnaire was developed based on two sources of information: first, the survey instruments used in previous studies that sought to identify challenges and capacity needs of RECs in Africa (Ateudjieu et al., 2010; Nyika et al., 2009; Sleem et al., 2010); second, a review study of the biomedical RECs in Sub-Saharan Africa (Silaigwana & Wassenaar, 2015). The first draft of the questionnaire was pretested using 10 representatives of the target groups (ethics committee members and researchers) not included in the sample. The questionnaire was then revised based on a field test and expert opinion before distribution. The survey questionnaire includes questions related to demographic information about the respondents (age, gender, and role of respondents), and a question asking the respondents to rank the predetermined challenges of the ethical review process of RECs in the order of importance, with #10 being the most important and #1 the least important (Appendix A). Assessing the level of importance and urgency of the challenges could help the institutions to attain this objective: quality improvement efforts are properly focused on issues that are most important to the REC members and research stakeholders. A total of 267 responses were received with a response rate of 16.02%. Overall, 259 duly completed responses were used for the analysis. Table 1 shows the characteristics of the respondents: slightly more than half of the respondents (57.9%) are scientists, nearly half of the respondents (42.1%) are in the age group of 30–39.

Table 1.

Characteristics of the Online Survey Respondents (n = 259).

Characteristics of respondent	Frequency	Percentage (%)
Role of respondent
REC members	57	22.0
Research coordinator	36	13.9
Scientists	150	57.9
Others	16	6.2
Total	259	100.0
Age (year)
20–29	56	21.6
30–39	109	42.1
40–49	60	23.2
50–59	21	8.1
> 59	11	4.2
Total	257	99.2
Missing value	2	0.8
Total	259	100.0
Gender
Male	118	45.6
Female	141	54.4
Total	259	100.0

Semistructured Interview Instrument and Participants

The face-to-face semistructured interviews and the online survey were conducted concurrently. REC's members, scientists, and research stakeholders were invited using the purposeful sampling method to explore the real-life processes of ethical review of biomedical research and clinical trials (Palinkas et al., 2015). We worked in collaboration with the secretariats of RECs to deliberately selected respondents that are exceptionally knowledgeable about or experienced in the ethical review of biomedical research and clinical trials in Malaysia. The samples were drawn from hospitals, universities, private and government sectors that reflect different socio-legal and research environments, and across the different locations of Malaysia, to achieve a maximum level of heterogeneity. These prospective respondents were contacted, and the interview date, time, and venue were arranged beforehand with an e-mail of introduction that contained the research preamble and interview guide. The interviews were conducted at a time and place of the respondents’ convenience, and after the 11th session, there were no more new themes generated from the interview. At this juncture, it was deemed that the data collection had reached a saturation point (Palinkas et al., 2015). As summarized in Table 2, the 11 respondents include chairpersons, scientific members, legal members, lay members of REC, and clinical trials facilitators.

Table 2.

Characteristics of the Respondents: Semistructured Interview (n = 11).

Respondent code	Gender	Professional role of respondent	Government/private organization
Respondent 1	Male	REC scientific member	Government
Respondent 2	Male	REC Chairperson (scientific member)	Government
Respondent 3	Female	REC member (bioethicists)	Government
Respondent 4	Male	Clinical trial facilitator	Government
Respondent 5	Female	REC scientific member	Private
Respondent 6	Female	REC lay member	Private
Respondent 7	Female	REC lay member	Private
Respondent 8	Female	REC Chairperson (scientific member)	Government
Respondent 9	Female	REC scientific member	Government
Respondent 10	Male	REC lay member	Government
Respondent 11	Female	REC lay member	Government

Note. REC=Research Ethics Committee.

We developed an interview guide based on the literature and findings of previous related empirical studies (Barchi et al., 2015; Chenneville, Menezes, Kosambiya, & Rajendra Baxi, 2016; Matar & Silverman, 2013; Motari, Ota, & Kirigia, 2015; Silverman et al., 2015; Suzuki & Sato, 2016). The instrument was field-tested, and the results of the field test called for minor revisions, and improvements were incorporated into a final interview guide (Appendix B). The interview questions explore three broad topics to understand the deliberations and the ethical review process of the RECs. First, we asked the respondents to give their opinions on the challenges of the ethics review of clinical trials and biomedical research in Malaysia. The intention was to find out the respondents’ views about the current ethics review system. Was there a problem? If the answer was yes, the researcher asked the respondents for the reasons and what kinds of “problems” they had identified. Second, the researcher of this study asked the respondents if they were aware of any “work-around” solutions to the perceived problems (if they identified any problems). Lastly, the interview discussed the development of a central ethics committee or centralized review mechanism; feedback from the respondents regarding the feasibility and support of this suggestion was noted.

Data Analysis

Both quantitative data and qualitative data sets were analyzed separately and independently in the second step of the convergent design. Descriptive summary statistics were used to summarize the responses to the survey question regarding the challenges of the ethical review of clinical and biomedical research in Malaysia. Responses to the question were split into two categories: “the most important” and “not very important.” Responses that were ranked in the top three (tenth, ninth, and eighth) were given equal weight and were coded or grouped into a single dichotomous variable as “the most important” challenges. Responses that were ranked as the first, second, third, fourth, fifth, sixth, and seventh were given equal weight and were coded or grouped into another single dichotomous variable as “not very important” challenges. The dichotomous variables were constructed in this manner to compare respondents who perceived a challenge as very important with those who did not perceive the challenge to be very important. Table 3 shows how the responses to the question were split into two dichotomous variables.

Table 3.

Dichotomous Variables of Online Survey Responses.

Challenges of ethical review of clinical and biomedical research	Online survey responses
	The most important	Not very important
Lack of scientific expertise to review the research protocol	Responses listed as one of the top 3 rankings (#8, #9, #10)	Responses listed other than the top 3 ranking (#1, #2, #3, #4, #5, #6, #7,)
Insufficient resources
Inactive participation of REC members
Lack of ethical reasoning skills to review the research protocol
Lack of recognition of the importance of REC
Lack of ongoing ethics training for REC members
Not completely independent
Inadequate ability to monitor approved protocols
Lack of national accreditation mechanism for the ethics committee
Biased committee members

Note. REC=Research Ethics Committee.

Meanwhile, thematic analysis was used to analyze the interview transcripts, and several themes were identified. We first presented the quantitative descriptive data about the challenges of the ethical review process of RECs in Malaysia, and then followed by qualitative results in the form of quotes about the same topic. The integration of quantitative and qualitative data only takes place in the interpretation and discussion of results (Creswell & Plano Clark, 2018).

Results

The percentages of respondents who ranked the challenges in one of the first three positions of importance are shown in Table 4. “Lack of scientific expertise to review the research protocol” was most often reported as one of the most critical challenges of the ethical review of clinical and biomedical research (48.3%). Almost half (47.1%) of the respondents ranked “insufficiency of resources provided to RECs” as one of the most important barriers, and followed by “inactive participation of members” (35.2%). Another frequently mentioned challenge is “lack of ethical reasoning skills to review the research protocol” (34.3%).

Table 4.

The Most Important Challenges of Ethical Review of Clinical and Biomedical Research in Malaysia (n = 259).

Challenges of ethical review of clinical and biomedical research	The most important (one of the top three rankings)
	Frequency (n)	Percentage (%)
Lack of scientific expertise to review the research protocol	125	48.3
Insufficient resources	122	47.1
Inactive participation of REC members	91	35.2
Lack of ethical reasoning skill to review the research protocol	89	34.3
Lack of recognition of the importance of REC	80	30.9
Lack of ongoing ethics training for REC members	74	28.5
Not completely independent	66	25.5
Inadequate ability to monitor approved protocols	65	25.0
Lack of national accreditation mechanism for the ethics committee	59	22.8
Biased committee members	54	20.8

Note. REC=Research Ethics Committee.

Several themes emerged from a careful review of the interview transcripts. The researcher described four major themes selected, each representing the problems and challenges of ethical review of biomedical research and clinical trials, which lead to the inadequacy of the ethical review system in Malaysia. The themes are: (1) insufficient training for REC members; (2) lack of scientific expertise and ethical reasoning skill; (3) shortage of financial and human resources; (4) lack of communication among RECs and the absence of centralized review mechanism. Table 5 shows the excerpts from the participants’ accounts based on the themes. Every respondent was allocated a number to protect his or her identity in order to safeguard the privacy and confidentiality.

Table 5.

The Challenges of Ethical Review of Clinical and Biomedical Research: Quotes Extracted from the Interview Transcripts.

No.	Major themes	Quotes extracted from the interview transcripts
1.	Challenge 1: Insufficient training for REC members
1.	a. Lack of ethics training for REC members b. New members were only given preliminary training c. Lack of expert in bioethics in Malaysia to provide training d. Mandatory training was not required for the lay member	“I do not think that (training) is sufficient. The training that we have attended was to explain the function of the committee, what they do, what are the issues they are concern with, why review committee is important. They did not go into the details on how to examine the protocol. Members have to pick up those issues concern during the discussion.” (Respondent 10) “The new members were trained by using the SOPs, guidelines and checklists as reference documents for the procedural framework of ethical review.” (Respondent 1) “For new members, they are requested to attend one or two meetings as an observer. They do not go for proper ethics training. Like myself, I do it based on my experiences as a clinical researcher. During the meeting, the new members learn from what is already ongoing. However, things that people practice may not be correct. Perhaps, I think what is lacking is formal ethics training.” (Participant 9) “Nowadays, ethics training is provided by the research institutions to conduct whatever they think is necessary. We do not have experts who are experienced and qualified in ethical review. Most of the training was given by senior members who had experiences in their field but not expert in ethics.” (Respondent 3) “I think training should be given for lay member. If for a doctor, they understand the medical terms and usual practices in research, but for a layperson, it is a little bit difficult to pick up at the initial stage.” (Respondent 7)
2.	Challenge 2: Lack of scientific expertise and ethical reasoning skill
2.	a. Lack of scientific expertise b. Lack of ethical reasoning skill	“We recognised there are the areas that our ethics committees do not have the expertise to conduct the protocol review, especially new areas of research. For example, biosimilar, vaccination, medical devices, herbal and phase one clinical trial.” (Respondent 4) “To conduct an in-depth ethical review is very demanding, especially when the protocol is with too many medical jargons and some of the fields are too specialized. Even the scientific members themselves who are medically trained can sometimes be struggling with the scientific design … ” (Respondent 6). “REC members did not know how to apply the ethical principles listed in the checklist and interpret them for making ethical judgements although they were aware of the ethical principles.” (Respondent 3).
3.	Challenge 3: Limited financial and human resources
3.	a. RECs face a budget constraint b. Insufficient human resources	“We do not receive any budget from the university. As a chairman, I have to plan structure activities for the ethics committee and apply for financial support from university annually. These activities, including sending staff for external training.” (Respondent 2). “REC members are wearing double hats, and even non-scientist members also have other jobs.” (Respondent 4). “We are also facing difficulty in finding new members to join us. When the existing members left, it is not easy to get someone to replace them.” (Respondent 5)
4.	Challenge 4: Lack of communication among RECs and absence of centralized review mechanism
4.	a. Lack of communication among RECs and the absence of centralized review mechanism	“The ethics committees in Malaysia are very scattered. They do not communicate with each other. The same protocol may be approved by one of the committees but rejected by another. We do not have central control.” (Respondent 5) “The problem we face now is that one ethics committee does not have mutual recognition to each other. When all the RECs come together, trust and support each other, and work collaboratively, then a cohesive network is formed, and the resources and expertise can be shared. The network could help to address the problem of the lack of ethical review expertise and resources.” (Respondent 4)
5.	Suggestions to address the challenges of ethical review process
5.	a. Establishment of a central ethics committee b. Mutual Recognition	“I do agree to have a central ethics committee. The central ethics committee standardises the Standard Operating Procedure and establishes the same benchmarks for all the RECs. Every REC follows the same procedures and guidelines. Therefore, once a protocol is approved by one of the ethics committees, it automatically is recognised by other committees. Then, we can save a lot of resources and times.” (Respondent 5) “The central ethics committee can provide ethics training to every REC members and researchers to familiarise them with the standard procedures and guidelines.” (Respondent 8) “By having the same standards and benchmarks among RECs, it can avoid duplication and reduce wastage. RECs can share resources and expertise, and we are mutual recognition.” (Respondent 2)

Note. REC=Research Ethics Committee.

The integrated data of the online survey and semistructured interviews reveal similarities and differences regarding the critical challenges of the ethical review of clinical and biomedical research in Malaysia. Table 6 shows the joint-data display presenting congruent and discrepant results. The similarities include the following: (1) REC members lacked scientific expertise and (2) ethical reasoning skills to conduct the ethical review; (3) RECs do not have sufficient financial and human resources. The three differences identified are: (4) inadequate ethics training for REC members; (5) lack of communication among RECs and the absence of centralized review mechanism; (6) inactive participation of REC members.

Table 6.

Congruent and Discrepant Results of Online Survey and Semistructured Interview.

Challenges of ethical review of clinical and biomedical research	The most important		Mixed methods comparison
	Online survey (n = 259): Frequency, n (percentage, %)	Semistructured interview (n = 11)	Mixed methods comparison
Lack of scientific expertise to review the research protocol	125 (48.3)	Raised by a majority of the respondents	Congruent
Limited financial and human resources	122 (47.1)	Raised by a majority of the respondents	Congruent
Inactive participation of REC members	91 (35.2)	Not mentioned by respondents	Discrepant
Lack of ethical reasoning skill to review the research protocol	89 (34.3)	Raised by a majority of the respondents	Congruent
Lack of recognition of the importance of REC	80 (30.9)	Raised by a minority of the respondents	Congruent
Lack of ongoing training for members in research ethics and bioethics	74 (28.5)	Raised by almost all (10/11) respondents	Discrepant
Lack of communication among RECs and the absence of centralized review mechanism	Not in the predetermined list of alternatives	Raised by a majority of the respondents	Discrepant
Not completely independent	66 (25.5)	Not mentioned by respondents	Congruent
Inadequate ability to monitor approved protocols	65 (25.0)	Not mentioned by respondents	Congruent
Lack of national accreditation mechanism for the ethics committee	59 (22.8)	Not mentioned by respondents	Congruent
Biased committee members	54 (20.8)	Not mentioned by respondents	Congruent

Note. REC=Research Ethics Committee.

‘Inadequate ethics training for REC members’ was raised by almost all of the respondents during the interviews as the most critical challenge. However, it was not given much attention by the respondents of the online survey. The item ‘Lack of communication among RECs and the absence of centralized review mechanism’ was highlighted by the interview respondents as one of the critical problems in ethical review process. This item was not included in the online survey questions. ‘Inactive participation of REC members’ was rated as one of the top three important challenges of RECs in the online survey, but was not mentioned by the interview respondents. There are differences between the results of the online survey and interviews; this shows that a pre-determined list of challenges of ethical review in the online survey questions based on the previous studies may not always be exactly the same as those of our investigation. By merging responses to the interviews with the survey data, we could more easily follow up when new ideas emerged.

Overall, the identified challenges can be categorized into four broad areas: (1) lack of scientific expertise and ethical reasoning skill; (2) inactive participation of REC members; (3) lack of communication among RECs and absence of the centralized review mechanism; and (4) limited financial and human resources.

Discussion

Lack of Scientific Expertise and Ethical Reasoning Skill

A relatively high percentage of respondents from both the online survey and interview rated “lack of scientific expertise and ethical reasoning skill to conduct an ethical review of research protocol” as a common challenge RECs in Malaysia are facing now. However, “inadequate ethics training for REC members” was mentioned by almost all of the respondents during the interviews as a major barrier to the proper functioning of RECs, but was not given the top priority in the online survey. Indeed, these findings are interrelated; insufficient ethics training for REC members is one of the major causes contributing to the incompetency of the members to review research protocols.

Almost all (10/11) of the interview respondents highlighted the problem of inadequate training for REC members in Malaysia. The respondents reported that most of the REC members learned how to conduct the review by observing the senior members during the meeting and from their own research experiences. Most of the respondents described their training experiences as insufficient and explained that the new members were only given preliminary training. Also, mandatory training was not required for the lay members. This has led to REC members lacking in ethical reasoning skills to guide their ethical deliberations. Such a situation is often conducive to poor decision making during the ethical review. Another respondent highlighted one of the factors contributing to the inadequate ethics training is that we do not have an expert in bioethics in Malaysia to train REC members and investigators.

REC members are the backbone of the ethical review system, the ability of REC to conduct ethical review effectively and efficiently depends upon a competent cadre of committee members. Similar to the United States and the United Kingdom, RECs in Malaysia have adopted an integrated approach to assess all the scientific aspects and ethical issues of a particular research. However, the problem of integrating scientific and ethical considerations in the review of research by RECs is challenging. This study found that REC members in Malaysia do not have adequate ethical reasoning skills to deliberate on and be actively involved in the review process, and to make a judgment upon the ethical standing of a research project. This study also reveals that REC members lack scientific expertise to review complex research, especially concerning understanding the issues associated with new technology. For example, data-intensive science, genomic studies, and clinical trials include biosimilar, vaccination, medical devices and herbal as well as phase one clinical trials sponsored by international agencies.

The inadequate scientific expertise and ethics reasoning skills highlighted by the interview respondents are consistent with previously published research. Research conducted in El Salvador showed that the REC members possess basic knowledge of locally relevant ethical principles but highlighted that they need more training to optimize the application of bioethical principles and models to their contexts (Camp et al., 2009). Also, many published studies revealed that REC members in LMICs lack scientific expertise to review research protocol, especially studies with new technologies and multinational clinical trials. Members of RECs in India and other countries in Africa and Asia were reported not having sufficient knowledge to evaluate the risks and potential benefits of clinical trials and vaccine trials as well as to review protocols involving the collection and use of biospecimens (Barchi et al., 2015; Bhowmick et al., 2014; Kochhar, 2013; Nyika et al., 2009). REC members were also found lacking in expertise to understand the ethical issues associated with data-intensive science, such as genomics, including the challenges it presents to traditional notions of informed consent and the need to link data to multiple datasets in other jurisdictions for research discovery (Dove & Garattini, 2018).

REC members lacking in ethical reasoning skills deserve urgent attention as the effectiveness of the deliberation and ethical review process of REC highly depends on how the ethical principles or norms and regulations, are incorporated into the REC review process (Taylor, 2007). The competency of REC members in ethical reasoning and judgment, as well as their familiarity with ethical norms or principles of ethics, affect the ethical quality of the deliberation and ethical review process. Therefore, for each member of the committee to deliberate in a meaningful way and reach an ethical decision, formal ethics training should be given to the REC members (Davies, Wells, & Druml, 2008; Emanuel et al., 2004).

There is widespread consensus among respondents in the semistructured interviews of this study in recognizing the need for more education and training opportunities, particularly the necessary ethical reasoning skills. However, formal ethics training should not only focus on increasing members’ familiarity with principles of ethics or ethical norms and guidelines. It should also increase their competencies in ethical analysis, which includes ethical sensitivity, ethical reasoning, and developing ethical responsibility, as well as improving their knowledge in the scientific method and regulatory framework (Ateudjieu et al., 2010; Cairoli et al., 2012; Camp et al., 2009; Davies et al., 2008).

Inactive Participation of REC Members

The online survey of this study highlighted that REC members do not often participate actively during the deliberation and ethical reviews of the research protocol. We believe the inactive participation of REC members is mainly due to their inadequate ethical review expertise. Literature reported that RECs in LMICs, for example, India, Sri Lanka, and Nepal did not participate actively in the review of industry-sponsored clinical trials, mainly due to the members’ lack of confidence to review the protocols from the large multinational pharma companies. The latter have both the resources and the experiences to draft scientifically sound and ethically plausible protocols (Simpson, Khatri, Ravindran, & Udalagama, 2015). With inadequate ethical review capacity of REC members, they are unconfident and stressed out when they review the research protocols that have already been approved in the funder's home country, and are subject to the review from the sponsored REC to approve the research without further deliberation (McIntosh et al., 2008). As a result, local RECs are highly dependent upon the ethics committees of sponsoring countries in deciding the outcome of an ethics review, even though the research is conducted in their communities (Barchi et al., 2014; Dauda & Dierickx, 2014). Dauda and Dierickx (2014) have commented that double independent ethical reviews of multinational research, which are currently practiced in LMICs, maybe a mere procedural requirement and not sufficient to protect the vulnerable subjects involved in the research.

Evidence suggests that the current independent ethics review system is not working well, especially the reviews of the international research projects that are subjected to multiple ethics reviews of the same research protocol. The literature highlighted that the reviews are redundant and resource-consuming, and the opinions delivered by the ethics committees may be inconsistent both within and across jurisdictions (Dove & Garattini, 2018). The model of mutual recognition is recently proposed by Dove et al. (2016) and Dove and Garattini (2018) as one of the solutions to reforming the independent ethical review system for reviewing and steering data-intensive international research. However, there is a concern that local sensitivities are difficult to acknowledge in mutually recognized ethical reviews of RECs from different jurisdictions, as the local RECs know their local populations better and are more likely to protect and care for them (Dove & Garattini, 2018).

A joint REC review entailing the participation of REC members from the sponsored country and the host country has been proposed by Barchi et al. (2014). This proposal could be one of the international efforts to increase the ethics capacity of RECs in LMICs, which will enhance the protection of human participants involved in international research, as the ethics review systems in much of the world remain understaffed and under-resourced. We believe that a joint REC review is an ideal strategy as the representatives from the local ethics committee are participating in the joint review and would be able to bring the local perspectives to the review process. For example, when there is an international multisite clinical trial going to be conducted at different sites in Malaysia, some members of these committees could be appointed to be part of the joint REC review panels of that project, together with the REC of the sponsoring country. This joint review between the host and sponsoring RECs opens the door for RECs in LMICs to understand how RECs in high-income countries work. The RECs in LMICs can learn about the regulations, rules, and norms, as well as expertise and skills which the RECs of the sponsoring country apply in making decisions. They could also understand the review priority and how potential conflicts are resolved to reach an agreement on common ethical practices and standards (Barchi et al., 2014; Ravinetto et al., 2011).

A joint REC review panel, comprising the leading experts from both the sponsoring and host countries, with joint chairs and members of two or more institutions, is seen as a promising approach to addressing concerns of the local context, both reducing duplicative reviews and building trust amongst RECs, institutions, and countries (Dove & Garattini, 2018). A joint REC review panel provides a unified fair procedure (devoid of paternalistic tendencies) where all views and expertise are adequately considered before decisions are made (Barchi et al., 2014). This joint REC review would provide a platform to strengthen the ethical review expertise of local RECs in LMICs, including Malaysia, and give them the confidence to conduct further reviews.

Lack of Communication among RECs and the Absence of Centralized Review Mechanism

A previous study reveals that there is an absence of a central ethics committee in Malaysia. Each REC in Malaysia has a different standard operating procedure and ethical review process; and urged for the establishment of a central ethics committee (Hooi Yan, Mohd Salim, Siti Nurani, & Wah Yun, 2019). Data in this study underpin these previous arguments, as the respondents of the interviews highlighted that there is a lack of a centralized ethical review mechanism and a lack of communication between RECs in Malaysia. The lack of communication between RECs and the absence of a centralized review mechanism cause variability and inconsistency of ethics reviews (Barchi et al., 2014; Dove & Garattini, 2018), and are the root causes of ineffective and inefficient ethical review system in Malaysia.

Also, some of the interview respondents criticized that different RECs conduct the same kind of review based on the same project, which could lead to time delays in research projects and waste of resources. A REC accepts the completed ethics review of another REC and vice versa through collaborative recognition of similar processes and standards under the mutual recognition agreement (Dove et al., 2016). A national system for mutual recognition could reduce duplication and inconsistency in ethical review and at the same time promotes review efficiency and maintains robust protection of the rights, interests, and welfare of research participants. However, the findings of the interviews show that a lack of communication between RECs could be an obstacle to implementing mutual recognition of ethical review standards in the country, because it is uncertain if the other RECs will make the “right” decision, or some RECs may think their decisions are better than those of others. If RECs fail to communicate with one another adequately, it is difficult for them to trust each other, to recognize the decision rendered by another REC on the same research project (and the reasons thereof). Respondents opined that improved communication between RECs is a way forward in developing mutual recognition of ethics reviews; it is seen as a vital step towards addressing the issues of the lack of ethical review expertise and resources among RECs in Malaysia too.

Interview respondents welcomed the suggestion of establishing a central ethics committee to act as a catalyst to improve communication and harmonize the processes of ethics reviews. The establishment of a central ethics committee and the development of a national standard operating procedure for RECs in Malaysia were suggested as essential steps to harmonize and centralize the ethical review system in Malaysia (Hooi Yan et al., 2019). The central ethics committee is responsible for drafting guidance documents; for example, structure and composition of committees, a standard and common ethics review procedure, forms, and checklists for RECs in Malaysia. In this way, a central ethics committee could eventually pave the path for RECs in Malaysia to mutually recognize one another's decisions of multisites ethical reviews. Therefore, it can be concluded that the establishment of a centralized review mechanism is critical for forging mutual recognition of ethical review decisions among RECs in Malaysia; this concerted effort will gradually improve the efficiency and greatly reduce the variability and duplication of the ethical review process in Malaysia.

Besides, the formation of a national network of RECs is advocated, which will provide a supportive work environment and opportunities to foster more interactions and strengthen cohesion among RECs. RECs could exchange ideas and propose solutions for practical issues in the review process by organizing regular meetings or using the online discussion platform, which will improve the communication between REC members. This national network would serve as a channel for sharing resources, information, expertise, or ideas that could help to increase the operating efficiency of RECs in an environment of resource constraints, with the ultimate goal of providing protection of human subjects (Kennedy et al., 2006). Establishing a cohesive network among RECs in the country is seen by some as a prerequisite to creating a more centralized or coordinated ethics review system (Dove & Garattini, 2018). According to one of the interview respondents (Respondent 4), a network called “The Network of Ethical Review Committees in Malaysia” (NERCIM) has been established for connecting every REC which reviews clinical trials and biomedical research in Malaysia. However, it needs more support and collaborative efforts to operate effectively, and to move towards the goal of mutual recognition of ethics review.

Limited Financial and Human Resources

The efficiency of RECs in Malaysia is also affected by limited financial and human resources. The findings of the online survey and semistructured interviews indicate that most of the RECs which review biomedical research and clinical trials in Malaysia do not have enough financial and human resources for a well-functioning committee. This finding is consistent with a previous study that demonstrated that the funding RECs in Malaysia received is insufficient for them to kick starts their online database management system and online application system (Hooi Yan et al., 2019), which then can reduce their dependency on human capital. Also, RECs in Malaysia do not have enough manpower to conduct protocol reviews and do not have enough funds to provide training for their members. Shortage of financial and human resources is a common problem of RECs in LMICs as reported by previous research. Recent pieces of evidence have shown that most of the ethics committees in LMICs are still not adequately funded (Hyder et al., 2015; Janakiram, Venkitachalam, & Joseph, 2016; Motari et al., 2015; Sleem et al., 2010).

Indeed, every REC needs funds for the operations of the committee, for ethical reviews and research monitoring activities as well as for improving their database management (filing and archiving) systems. Funding is also needed for providing training, for example, sending members to attend international conferences and seminars, conducting public awareness and community sensitization activities. The entities that established the RECs should be aware of the resource requirements of their RECs, and every REC should receive sufficient financial resources and administrative support personnel (Sleem et al., 2010). In fact, inadequate budget and human resources that support REC functioning is a reflection of suboptimal institutional commitment to ethics, and this needs to be rectified (Silverman et al., 2013).

Conclusion

The results of this study reveal that the ethics review of clinical and biomedical research in Malaysia is inefficient and ineffective, and the process operates suboptimally. The inconsistent, duplicate, and inadequate ethical review process causes delays in potentially beneficial research, produces divergent REC opinions and decisions, and might not provide added protection to human subjects. Therefore, we advocate the establishment of a centralized ethical review mechanism, a national system for mutual recognition of ethics reviews, and a cohesive network to collaborate effectively in the review process to reduce the inconsistency and duplication of ethical reviews, promote efficiency in reviews and protect the rights, interests, and welfare of research participants in Malaysia. We also advocate that the best effort should be made to organize and pilot test the model of joint REC review, and determine its effectiveness, among the RECs in Malaysia, and even with RECs from the sponsoring countries. We believe that a joint REC review will strengthen the performance of RECs in Malaysia as this model could directly address the concerns of the lack of expertise and training. Our key point is that a joint REC review mechanism can go a long way in fostering true review partnership and curbing RECs’ review concerns about an inefficient ethics review system in multinational research collaboration.

Best Practices

To address the existing challenges of ethical review system in Malaysia, the NPCB of MOH Malaysia should make a greater effort for the establishment of a central ethics committee, a national system for mutual recognition of ethics reviews, and a cohesive REC network to reduce the inconsistency and duplication of ethical reviews. We also propose setting up a well-coordinated joint REC for a multinational research study to facilitate appropriate deliberation on ethics reviews in order to enhance efficiency while avoiding unnecessary review duplication, as well as to address the concerns of the lack of expertise and training.

Next, research institutions in Malaysia should encourage scholars to conduct empirical research to regularly evaluate the performance of RECs. Evaluation is deemed important to ensure that REC systems are effective, transparent, and accountable. The evaluation of RECs may also be important in helping to improve their performance.

Finally, this research employs mixed methods that involve collecting qualitative and quantitative data by using surveys and semistructured interviews. Qualitative data from semistructured interviews serve mainly as a complement to the quantitative data from the online survey, and increase the validity of our findings. Multiple forms of evidence also enhance the trustworthiness of the analysis by providing a fuller, more rounded account of the challenges of the ethical review of clinical and biomedical research in Malaysia, and reduce bias. The main limitation of our survey is the low response rate of 16%, and since demographic information on the pool of potential respondents was not available, meaningful tests for possible nonresponse bias could not be conducted.

Research Agenda

As biomedical research is increasingly international and complex, the ethical conduct of research involving human subjects is dependent, in part, on the support of the public and researchers. Participant perspectives must be considered when reviewing the research protocol and developing informed consent practices. Future investigations should build from this research to understand the participant perspectives from the public, researchers, and clinical populations regarding their experiences, attitudes, ethical concerns, and motivation for participation in biomedical research and clinical trials, as literature in this area is scarce in Malaysia. These understanding will enable researchers to maximize recruitment and retention and minimize participant burden and participation bias for future biomedical research and clinical trials.

Besides, we have evaluated the challenges of RECs which review biomedical research and clinical trials only; in future, research must involve other RECs which review research in different areas of the social sciences related to human and animal research in Malaysia.

Educational Implications

Overall, we have identified a strong need and demand for a centralized ethical review mechanism and a national system for mutual recognition of ethics review. We, therefore, recommend that the regulatory authority for RECs Malaysia establishes a central ethics committee to move towards a national system for mutual recognition of ethics review, and encourage both researchers and REC members to participate in joint REC reviews. The above-mentioned ethical review strategies are new to ethical review system in Malaysia and have not been thoroughly evaluated. Therefore, there is a clear need for the regulatory authority overseeing the RECs and research institutions to draw up an operational plan before launching a new model of ethics review strategies. It is crucial to pilot test an ethical review strategy beforehand to determine how it might work (or not work) in practice. Ongoing evaluation should also be incorporated. Feedback on effectiveness as well as procedural or technical limitations will help RECs, institutions, and regulators assess how well these strategies address the challenges of the ethics review system (Barchi et al., 2014) in Malaysia.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Postgraduate Research Grant, Centre of Research Grant Management, University of Malaya (grant no. PG051-2015B).

ORCID iDs

Hooi Y. See

Mohd S. Mohamed

Siti N. M. Nor

Supplemental Material

Supplemental material for this article is available online.

Author Biographies

Hooi Y. See is a PhD student in the Department of Science and Technology Studies, Faculty of Science, University of Malaya. Her research focuses on the research ethics and ethical review of biomedical research and clinical trials. She conceptualized and designed the study, developed the research instrument used, conducted a survey and interview, analyzed the data, and wrote the manuscript.

Mohd S. Mohamed is currently a senior lecturer at the Department of Science and Technology Studies, Faculty of Science, University of Malaya, Malaysia. His research is situated in the field of Bioethics, which a special focus on biomedical ethics and policy, ethics in sustainability, and research ethics. He supervised the research, assisted in manuscript editing, and contributed to the discussion.

Siti N. M. Noor is an associate professor at the Faculty of Science, University of Malaya, Kuala Lumpur. She helped to conceptualize the study and edited and approved the final manuscript.

Wah Y. Low is a professor of Psychology at the Faculty of Medicine at the University of Malaya. She is also the Deputy Director of the Asia-Europe Institute and currently the Chair of the University of Malaya Research Ethics Committee. Her research focuses on behavioral medicine, sexual and reproductive health, and social epidemiology. She oversaw all aspects of the study, reviewed and approved the final manuscript for publication.

References

Ali

Kass

N. E.

Sewankambo

N. K.

White

T. D.

Hyder

A. A.

(2014). Evaluating international research ethics capacity development: An empirical approach. Journal of Empirical Research on Human Research Ethics, 9(2), 41–51. https://doi.org/10.1525/jer.2014.9.2.41

Ateudjieu

Williams

Hirtle

Baume

Ikingura

Niaré

Sprumont

(2010). Training needs assessment in research ethics evaluation among research ethics committee members in three African countries: Cameroon, Mali and Tanzania. Developing World Bioethics, 10(2), 88–98. https://doi.org/10.1111/j.1471-8847.2009.00266.x

Barchi

Matlhagela

Jones

Kebaabetswe

P. M.

Merz

J. F.

(2015). “The keeping is the problem”: A qualitative study of IRB-member perspectives in Botswana on the collection, use, and storage of human biological samples for research. BMC Medical Ethics, 16(1), 54. https://doi.org/10.1186/s12910-015-0047-3

Barchi

Singleton

M. K.

Merz

J. F.

(2014). Fostering IRB collaboration for review of international research. The American Journal of Bioethics, 14(5), 3–8. https://doi.org/10.1080/10810730902873927

Bhowmick

Banerjee

Sikdar

Chatterjee

T. K.

(2014). An evaluation of knowledge, attitude, and practice of institutional ethics committee members from eastern India regarding ethics committee functioning and pharmacovigilance activities conducted during clinical trials: A pilot study. Perspectives in Clinical Research, 5(3), 115–120. https://doi.org/10.4103/2229-3485.134310

Cairoli

Davies

H. T.

Helm

Hook

Knupfer

Wells

(2012). A syllabus for research ethics committees: Training needs and resources in different European countries. Journal of Medical Ethics, 38(3), 184–186. https://doi.org/10.1136/medethics-2011-100064

Camp

J. W.

Barfield

R. C.

Rodriguez

Young

A. J.

Finerman

Caniza

M. A.

(2009). Challenges faced by research ethics committees in El Salvador: Results from a focus group study. Developing World Bioethics, 9(1), 11–17. https://doi.org/10.1111/j.1471-8847.2007.00225.x

Chenneville

Menezes

Kosambiya

Baxi

(2016). A case-study of the resources and functioning of two research ethics committees in western India. Journal of Empirical Research on Human Research Ethics, 11(5), 387–396. https://doi.org/10.1017/S0950268814002131

Creswell

J. W.

Plano Clark

V. L.

(2018). Designing and conducting mixed methods research. SAGE Publications Inc.

10.

Dauda

Dierickx

(2014). One size that could fit all: IRBs joint review collaboration as the key to addressing the challenges of multinational research review involving resource-poor countries. The American Journal of Bioethics, 14(5), 15–17. https://doi.org/10.1080/15265161.2014.892170

11.

Davies

Wells

Druml

(2008). How can we provide effective training for research ethics committee members? A European assessment. Journal of Medical Ethics, 34(4), 301–302. https://doi.org/10.1136/jme.2007.021485

12.

Dove

E. S.

Garattini

(2018). Expert perspectives on ethics review of international data-intensive research: Working towards mutual recognition. Research Ethics, 14(1). 1–25. https://doi.org/10.1177/1747016117711972

13.

Dove

E. S.

Townend

Meslin

E. M.

Bobrow

Littler

Nicol

Vries

J. D.

Junker

Garattini

Bovenberg

Shabani

Levesque

Knoppers

B. M.

(2016). Ethics review for international data-intensive research. Science (New York, NY), 351(6280), 1399–1400. https://doi.org/10.1126/science.aad5269

14.

Emanuel

E. J.

Wood

Fleischman

Bowen

Getz

K. A.

Grady

Levine

Hammerschmidt

D. E.

Faden

Eckenwiler

Muse

C. T.

Sugarman

(2004). Oversight of human participants research: Identifying problems to evaluate reform proposals. Annals of Internal Medicine, 141(4), 282–291. https://doi.org/VL - 141.

15.

Hooi Yan

Mohd Salim

Siti Nurani

M. N.

Wah Yun

(2019). Addressing procedural challenges of ethical review system: Towards a better ethical quality of clinical trials review in Malaysia. Accountability in Research: Policies and Quality Assurance, 26(1), 49–64. https://doi.org/10.1080/08989621.2018.1556646

16.

Hyder

A. A.

Ali

Hallez

White

Sewankambo

N. K.

Kass

N. E.

(2015). Exploring institutional research ethics systems: A case study from Uganda. AJOB Empirical Bioethics, 6(3), 1–14. https://doi.org/10.1038/nbt.3121.ChIP-nexus

17.

Hyder

A. A.

Dawson

Bachani

A. M.

Lavery

J. V.

(2009). Moving from research ethics review to research ethics systems in low-income and middle-income countries. Lancet (London, England), 373(9666), 862–865. https://doi.org/10.1016/S0140-6736(09)60488-8

18.

Janakiram

Venkitachalam

Joseph

(2016). Profile of institutional ethics committees in dental teaching institutions in Kerala, India. Accountability in Research: Policies and Quality Assurance, 23(4), 219–229. https://doi.org/10.1080/08989621.2015.1114887

19.

Kaur

(2011). The adequacy of the ethics review process in Malaysia: protection of the interests of mentally incapacitated adults who enrol in clinical trials (Doctoral Theses). Retrieved from https://discovery.ucl.ac.uk/id/eprint/1324539/1/1324539.pdf.

20.

Kennedy

S. B.

Harris

A. O.

Oudemans

Young

Kollie

Nelson

E. S.

Nisbett

R. A.

Morris

Bartee

George-Williams

Jones

(2006). Developing capacity to protect human research subjects in a post-conflict, resource-constrained setting: Procedures and prospects. Journal of Medical Ethics, 32(10), 592–595. https://doi.org/10.1136/jme.2005.014472

21.

Kochhar

(2013). Challenges and impact of conducting vaccine trials in Asia and Africa. Human Vaccines and Immunotherapeutic, 9(4), 924–927. https://doi.org/10.4161/hv.23405

22.

Matar

Silverman

(2013). Perspectives of Egyptian research ethics committees regarding their effective functioning. Journal of Empirical Research on Human Research Ethics, 8(1), 32–44. https://doi.org/10.1016/j.pestbp.2011.02.012

23.

McIntosh

Sierra

Dozier

Diaz

Quiñones

Primack

Chadwick

Ossip-Klein

D. J.

(2008). Ethical review issues in collaborative research between US and low-middle income country partners: A case example. Bioethics, 22(8), 414–422. https://doi.org/10.1111/j.1467-8519.2008.00662.x

24.

Motari

Ota

M. O.

Kirigia

J. M.

(2015). Readiness of ethics review systems for a changing public health landscape in the WHO African region. BMC Medical Ethics, 16(1), 82. https://doi.org/10.1186/s12910-015-0078-9

25.

Ndebele

Wassenaar

Benatar

Fleischer

Kruger

Adebamowo

Kass

Hyder

A. A.

Meslin

E. M.

(2014). Research ethics capacity building in Sub-Saharan Africa: A review of NIH Fogarty-funded programs 2000–2012. Journal of Empirical Research on Human Research Ethics, 9(2), 24–40. https://doi.org/10.1525/jer.2014.9.2.24

26.

Nyika

Kilama

Chilengi

Tangwa

Tindana

Ndebele

Ikingura

(2009). Composition, training needs and independence of ethics review committees across Africa: Are the gate-keepers rising to the emerging challenges? Journal of Medical Ethics, 35(3), 189–193. https://doi.org/10.1136/jme.2008.025189

27.

Palinkas

L. A.

Horwitz

S. M.

Green

C. A.

Wisdom

J. P.

Duan

Hoagwood

(2015). Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Administration and Policy in Mental Health, 42(5), 533–544. https://doi.org/10.1007/s10488-013-0528-y

28.

Ravinetto

Buvé

Halidou

Lutumba

Talisuna

Juffrie

D'Alessandro

Boelaert

(2011). Double ethical review of north–south collaborative clinical research: Hidden paternalism or real partnership? Tropical Medicine and International Health, 16(4), 527–530. https://doi.org/10.1111/j.1365-3156.2011.02732.x

29.

Silaigwana

Wassenaar

(2015). Biomedical research ethics committees in Sub-Saharan Africa: A collective review of their structure, functioning, and outcomes. Journal of Empirical Research on Human Research Ethics, 10(2), 169–184. https://doi.org/10.1177/1556264615575511

30.

Silverman

Edwards

Shamoo

Matar

(2013). Enhancing research ethics capacity in the Middle East: Experience and challenges of a Fogarty-sponsored training program. Journal of Empirical Research on Human Research Ethics, 8(5), 40–51. https://doi.org/10.1525/jer.2013.8.5.40

31.

Silverman

Sleem

Moodley

Kumar

Naidoo

Subramanian

, … M.

Moni

(2015). Results of a self-assessment tool to assess the operational characteristics of research ethics committees in low- and middle-income countries. Journal of Medical Ethics, 41(4), 332–337. https://doi.org/10.1136/medethics-2013-101587

32.

Simpson

Khatri

Ravindran

Udalagama

(2015). Pharmaceuticalisation and ethical review in south Asia: Issues of scope and authority for practitioners and policymakers. Social Science & Medicine, 131, 247–254. https://doi.org/10.1016/j.socscimed.2014.03.016

33.

Sleem

Abdelhai

R. A. A.

Al-Abdallat

Al-Naif

Gabr

H. M.

Kehil

Sadiq

B. B.

Yousri

Elsayed

Sulaiman

Silverman

(2010). Development of an accessible self-assessment tool for research ethics committees in developing countries. Journal of Empirical Research on Human Research Ethics, 5(3), 85–98. https://doi.org/10.1525/jer.2010.5.3.85

34.

Suzuki

Sato

(2016). Description and evaluation of the research ethics review process in Japan: Proposed measures for improvement. Journal of Empirical Research on Human Research Ethics, 11(3), 256–266. https://doi.org/10.1177/1556264616660644

35.

Taylor

H. A.

(2007). Moving beyond compliance: Measuring ethical quality to enhance the oversight of human subjects research. IRB: Ethics & Human Research, 29(5), 9–14. PMID: 17969778.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB