Abstract
In January 1998 the Commonwealth Department of Health and Family Services published the findings and recommendations of a national study of the procedures for the selection of trainees in Australian medical colleges [1]. The report had been sponsored by the Medical Training Review Panel and conducted by Dr Peter J Brennan and Company.
The Brennan report acknowledged that the high standard of postgraduate training conducted by the learned Colleges contributed significantly to the high standard of medical care widely available in Australia. However, it was also acknowledged that the Colleges had come under increasing scrutiny, and that there were concerns about the ways in which doctors were selected for specialist training. The areas of concern included: the perception that the selection processes were in some instances unfair, and lacking in transparency; that the selection processes were inconsistent, between and within Colleges; and that appropriate policies in relation to the training of women in medical specialties were lacking.
The report stressed that selection processes need to be valid and reliable, legal and accountable, consistent and transparent, and well documented; and, that the Colleges have responsibilities and obligations to select trainees in a fashion that is commensurate with Australian fair-trading and anti-discrimination legislation. The report recommended that all Colleges develop a framework for the selection of trainees which incorporated: a clear statement of the principles underpinning selection; a declaration of the number of training positions available; standard written proformas for applications and referee reports; selection criteria which are documented, objective and quantifiable; interviews which are free of bias, documented and quantified by standard proforma; a mechanism of feedback to candidates; a formal appeals process; and, an openness to external review and evaluation.
The unifying principle guiding the selection of medical graduates for vocational training has always been to select the best possible candidates. However, the nexus between training (the domain of the Colleges) and employment (the domain of the health services) was identified as a significant problem area. In circumstances where the hospital or area health service played the dominant role in recruitment there was the risk that workforce issues and service delivery needs may take precedence over the recruitment of only the best candidates. On the other hand, where the Colleges had complete control of the selection process, there was criticism that the process was not fair, and was an exercise in the preservation of the market share and financial security of the College's members. The Brennan report suggested a joint selection process should be developed, or at least a clarification of the roles of the two bodies should be made.
The RANZCP NSW Branch Training Committee (BTC) documented their concerns with the process of selection of doctors for psychiatry training in NSW in 1997 [2]. These can be summarised as follows: applicants for training rarely had an overview of the variety of positions available in NSW; the applicants often attended two, three or more similar interviews to secure one position; staff specialists and human resource personnel were duplicating the interview process across (and sometimes within) area health services; successful applicants were sometimes entering into negotiations for special deals as the basis for determining where they would undertake vocational training; and, the lack of consistency in the process made it difficult for the RANZCP to ensure that minimum standards were being met by all applicants before they were offered RANZCP-accredited training.
With these issues in mind the NSW BTC developed a proposal for the centralised selection of trainees to commence RANZCP-accredited training in NSW in 1999. This paper describes the development, implementation, and outcome of the proposal.
BACKGROUND
The NSW BTC was formed in 1984–85 to oversee psychiatry registrar training in NSW, and is responsible to the (bi-national) Committee for Training, a subcommittee of the Fellowships Board. The NSW BTC is made up of a Chairperson, seven Zone Directors of Training (from Central Sydney Zone, IIIawara Zone, Newcastle and Central Cost Zone, Northern Sydney Zone, North-Western Sydney Zone, South-Eastern Sydney Zone, and South-Western Sydney Zone – corresponding approximately to the existing area health services involved), the Director of Child and Adolescent Psychiatry Training, the Director of the NSW Institute of Psychiatry, an Exam Secretary, a representative of the Association of Psychiatrist-in-Training (APT), and a part-time administrative assistant.
To address the issue of the centralised selection of trainees a sub-committee was formed by calling for interested volunteers from the NSW BTC in March 1998. The sub-committee's proposal was discussed and refined at the NSW BTC over three months and then presented to a meeting of the Area Directors of Mental Health Services at the NSW Health Department Centre for Mental Health in June 1998. The proposal was then further refined and presented to the Areas' Human Resource Managers in July 1998.
CENTRALISED SELECTION PROCESS
The proposal was that the NSW BTC undertake the centralised selection of first year trainees to commence RANZCP-accredited training in NSW in 1999. Newspaper advertisements were timed to appear alongside the NSW area health service advertisements for RMO's and registrars for 1999. Area health services were requested to include psychiatry trainees second year and above in their advertisements, and refer applicants for first year training to the College advertisement. The College advertisement contained the essential and desirable criteria, and directed the applicants to the NSW Branch secretariat for an information package and application form.
The information package included the RANZCP Fellowships Board Training and Examinations By-Laws (Revised 1996), the Fellowship Curriculum, the Code of Ethics, a description of psychiatry training in NSW, and information sheets on the NSW Institute of Psychiatry and the APT. The application kit included a four-page proforma in which the candidate provided resumé details, nominated two referees, and responded to a question on why they wished to train in psychiatry. The candidate also completed a Preference Allocation Form to state his/her preferences for the training zone and/or hospital at which he/she wished to commence training, if selected. Nominated referees were sent a standard written Referees Report proforma to complete.
Eight selection sub-committees were convened and consisted of NSW BTC members, the Area Directors of Mental Health (or delegates), and representatives of the NSW Institute of Psychiatry, the NSW Health Department Centre for Mental Health and the APT. A standard set of questions and a proforma for rating the applicants were developed based on instruments used in Queensland [3] to guide the selection sub-committees. Each selection sub-committee had a chairperson appointed from the NSW BTC who was responsible for completing the interview proforma and signing it as a consensus document of the selection sub-committee. The scores of the selection sub-committee interviews were used to rank the successful candidates and assign them to their preferences, with the highest ranked applicant being the first person to be allocated to their first preference.
All applicants were advised at interview that success at the NSW BTC selection interview did not guarantee employment with any hospital or area health service. Similarly, all applicants were advised that if they were not successful they were free to apply for non-accredited medical officer positions in psychiatry in NSW and the employing body would make a decision on their suitability. Unsuccessful applicants were notified by phone and offered an interview with a sub-committee of the NSW BTC to obtain constructive feedback on their failure to be selected. The appeals mechanism allowed for applicants to express any grievances with the selection process in writing, for review initially by the NSW BTC.
The selection process was formally evaluated by the selection panel members at the end of the interviews, and by the applicants after being advised of the outcome of their application. The interviewers were asked to complete an anonymous 10-item Interview Panel Member Survey and the applicants received an anonymous 11-item Applicant Survey.
APPLICANT SELECTION RESULTS
Fifty-one telephone inquiries were received and of these 42 requested information and application packages. Thirty-eight applications were returned. Thirty-six applicants met the essential criteria and proceeded to interview. The selection sub-committees interviewed all applicants (except four) on the same day. One face-to-face and three telephone interviews were conducted on the previous day, to accommodate prior commitments of those applicants.
Of the 32 applicants selected as suitable, four required review at the allocation sub-committee meeting of the NSW BTC based on the reservations expressed by the selection sub-committees. Each was discussed, their application and interview performance reviewed and referees' reports considered. Ultimately all four were approved for selection.
The first-pass matching of ranking and allocation was conducted manually, and all 32 successful applicants were allocated to their first preference.
Copies of the applications, referees' reports and interview proformas were passed on to the local service directors via the Zone Directors of Training to facilitate the employment process. Two of the three unsuccessful applicants attended a feedback interview with a sub-committee from the NSW BTC. We have not received any verbal or written appeals from any applicant, and we are not aware of any applicant who was successful at the NSW BTC selection interview who was not subsequently offered appropriate employment.
INTERVIEW PANEL MEMBER SURVEY RESULTS
Seven selection sub-committees comprising five panel members each completed the Interview Panel Member Survey (n = 35). Gender balance was not observed on one panel. Twenty-two panel members (63%) had interviewed applicants for selection to RANZCP training previously, between the years 1980 and 1997 inclusive. Twelve of the panel members (34%) had undertaken staff selection/recruitment workshops or training seminars. There was at least one person per panel who had undertaken formal staff selection training.
A number of questions were included to evaluate the interview process. Thirty-one (89%) of the panel members reported receiving the list of questions and interview proforma prior to the interview date. All panelists reported that the material provided was useful, 33 (94%) stated that the interview questions were appropriate, and 31 (89%) found the scoring system easy to complete.
Panel members were asked for further comments on the process. Their responses are broadly categorised into the following areas: the need for the use of clinical vignettes or problem solving exercises as interview tools; the need to include questions relating to multicultural and rural issues in psychiatry; the wish to shorten the length of interviews (each candidate had 40 minutes scheduled for interview and panel discussion); the wish to review the applicants curriculum vitae and referees' reports prior to the day of interview; and, suggested improvements to the wording and scoring of particular questions.
APPLICANT SURVEY RESULTS
The 32 applicants who were successful at interview were sent the Applicant Survey. Twenty (62.5%) completed surveys were returned. The age range of the respondents was 24–48 years (mean = 31 years), with 13 males and 7 females. There were no age or sex differences in the responses to the questions. Respondents reported having more than one method for finding out about the training positions: 13 (65%) responded to newspaper advertisements; 8 (40%) enquired directly through College; 6 (30%) enquired through a particular area health service (AHS) or hospital; and 4 (20%) enquired through a colleague.
The respondents were asked a number of questions about the centralised selection process. Nineteen (95%) respondents stated they received the information package from the RANZCP Branch Office, and indicated that the information was useful. All respondents stated the application form was user friendly. Two commented on the difficulty with complying with the request for undergraduate details and performance assessments from each hospital/practice since graduation. Eighteen (90%) respondents indicated the Preference Allocation Form was user friendly.
In regard to the conduct of the interview, 19 (95%) respondents agreed the panel made every effort to put them at ease, all respondents agreed they were given sufficient time to be interviewed and to ask questions, and all found the interview questions appropriate. Comments suggested the inclusion of clinical or ethical vignettes to bring out the clinical background and experience of applicants.
Fourteen (70%) of respondents were clear at the end of the interview on how the selection process would proceed. The 6 respondents (30%) who indicated the selection process was not clear commented on their unsuccessful attempts to clarify issues at interview. Five different panels interviewed these respondents; hence, the lack of clarity was not due to panel composition.
Applicants were asked whether they felt the centralised selection process was worthwhile: 17 (85%) respondents answered yes and commented positively on the process being fair, standardised, and economical with time. One respondent answered negatively, but did not provide comments.
DISCUSSION
A major criticism of the interview process was that gender balance was not observed on one of the selection sub-committees. Whilst this (all-male) committee ultimately interviewed male applicants only, the need for gender balance per se is acknowledged [4].
A number of other administrative issues were problematic. In the process of advertising, only two area health service advertisements specified that applicants for first year training should refer to the College advertisement. This was identified by one applicant survey respondent as a problem and led to some minor confusion. Secondly, the NSW BTC was comparatively slow in convening the final selection and allocation meeting, which raised the anxiety of some applicants (and some prospective employers). Finally, some applicant survey respondents were unclear about the process despite the available documentation.
The respondents to the Interview Panel Member Survey and the Applicant Survey highlighted the need to include a problem-solving exercise, hypothetical or case vignette, to allow interviewers the opportunity to observe the candidate ‘thinking on their feet’ and allow the candidate to showcase their clinical knowledge and skills.
The low response rate to the applicant survey was striking, and was below response rates reported elsewhere [5]. Strategies to improve this will be considered prior to the next selection round.
In our introduction we highlighted that a lack of appropriate policies in relation to the training of women was an identified area of concern in the Brennan report. While we did not develop any specific response in this area, we were mindful throughout the selection process of the need to actively explore the issues of flexibility of training, part-time training and job-sharing for all potential trainees, particularly women.
CONCLUSIONS
The selection process, as developed and implemented by the RANZCP NSW BTC in 1998, was successful in addressing the Committee's concerns with the previous practices used in recruiting doctors for psychiatry training in NSW. In 1998, all applicants were given the same information and an overview of training in NSW, the applicants' preferences for the location of their training was respected, and all successful applicants met the minimum standards of the RANZCP NSW BTC. The applicants participated in a defensible and equitable selection process, which eliminated the need for multiple applications and multiple interviews. The benefits of a standard, centralised approach clearly outweigh the issues of control and ownership at a local level. The declining pool of applicants for psychiatry training in Australia and New Zealand, and the costs associated with duplicating selection processes argue in favour of the further development of this process to achieve bi-national standardisation and cooperation.
The recommendations of the Brennan report were used to guide all stages of the process. In particular, the eligibility criteria were clearly documented; the selection of applicants was objective, standard, and quantified; and the process was formally evaluated. The goal of the process – to select the best candidates in the fairest manner possible – was achieved.
Acknowledgements
The authors wish to thank Drs Bruce Boman, Murray Mackay, Sam Vadali, Marina Vamos and Fran Orr for their substantial help in developing the proposal; the remaining members of the RANZCP NSW BTC for their assistance; the RANZCP NSW Branch Secretariat for their organisational work; Professor Beverly Raphael and the NSW Area Directors of Mental Health Services for their support; and Ms Ayse Sengoz (AS Consulting) for data and manuscript management.