Safety of Clinical and Non-Clinical Decision Makers in Telephone Triage: A Narrative Review

Abstract

Patient safety is a persistent problem in telephone triage research; however, studies have not differentiated between clinicians’ and non-clinicians’ respective safety. Currently, four groups of decision makers perform aspects of telephone triage: clinicians (physicians, nurses), and non-clinicians (emergency medical dispatchers (EMD) and clerical staff). Using studies published between 2002–2012, we applied Donabedian’s structure-process-outcome model to examine groups’ systems for evidence of system completeness (a minimum measure of structure and quality). We defined system completeness as the presence of a decision maker and four additional components: guidelines, documentation, training, and standards. Defining safety as appropriate referrals (AR) – (right time, right place with the right person), we measured each groups’ corresponding AR rate percentages (outcomes). We analyzed each group’s respective decision-making process as a safe match to the telephone triage task, based on each group’s system structure completeness, process and AR rates (outcome). Studies uniformly noted system component presence: nurses (2-4), physicians (1), EMDs (2), clerical staff (1). Nurses had the highest average appropriate referral (AR) rates (91%), physicians’ AR (82% average). Clerical staff had no system and did not perform telephone triage by standard definitions; EMDs may represent the use of the wrong system. Telephone triage appears least safe after hours when decision makers with the least complete systems (physicians, clerical staff) typically manage calls. At minimum, telephone triage decision makers should be clinicians; however, clinicians’ safety calls for improvement. With improved training, standards and CDSS quality, the 24/7 clinical call center has potential to represent the national standard.

Keywords

telephone triage safety clinical non-clinical decision

Introduction

Telephone triage is a complex process of identifying a patient’s problem, estimating the level of urgency, and rendering advice over the phone¹ while ensuring the safe, timely, and appropriate disposition of patient symptoms.² A disposition is also known as a referral and is defined as a directive from clinician to patient about the time, place, and person by whom the patient’s symptoms are to be further evaluated and/or treated. Safety in telephone triage requires that referrals be appropriate and timely, meaning avoiding delays in -care, -evaluation, -diagnosis and/or -treatment, ensuring that patients are seen before symptoms escalate.

In the United States, telephone triage must shoulder the burden of the competing requirements of improving patient access and safety while containing costs. As of 2001, approximately 100 million people utilized telephone triage.³ That number will be markedly increased as millions of new enrollees to the Affordable Care Act (ACA) seek access to care, beginning with a call about symptoms. ACA makes it increasingly important to evaluate the safety of telephone triage. Even a one percent error rate might adversely affect hundreds of thousands of people.

Although early research focused on physician and nurse practice of telephone triage, current studies describe telephone triage as being delivered by different groups of decision makers, each with wide-ranging levels of education, training, and methods for responding to calls. The four most commonly mentioned groups of telephone triage decision makers consist of clinicians, i.e., doctors and nurses; and non-clinicians, i.e., emergency medical dispatchers, clerical staff, and answering service staff. The variations in definitions, qualifications, tools and strategies for performing telephone triage may lead to negative outcomes as demonstrated by selected case studies (Appendix 1). These malpractice cases were derived from actual cases on which Ms. Wheeler has consulted as an expert witness since 1995.

Telephone Triage Safety

On a Saturday morning, a husband called regarding his wife, four days post partum, age 35, who was complaining of a severe headache. The patient also had a history of migraines. The nurses did not speak directly with the patient, who was “too sick to come to the phone”. They advised that the patient take additional doses of her usual migraine medication.

The next day, the husband again called about worsening headache, neck pain and photophobia. The nurse did not speak with the patient, however s/he made an appointment for Monday afternoon.

The husband called again. Shortly thereafter, the patient suffered a seizure and was taken via paramedic transport to the hospital. Outcome: Cerebral hemorrhage due to hypertensive disorder of pregnancy, death.

This encounter illustrates common errors that lead to a delay in care and treatment:

failure to speak directly to patient

inadequate assessment

inadequate estimation of symptom urgency

failure to consider patient history or post-partum complications

failure to appreciate significance of repeat phone calls

jumping to conclusion that the symptoms were a migraine

Inadequate guidelines, training, and standards (policies) likely lead to these practice errors, which resulted in a delay in care (under referral) and the death of a patient. (See also Appendix 1)

Research examining clinicians’ safety in performing telephone triage began in 1978;⁴ however, research on non-clinicians has been limited in quantity, and of inconsistent quality. Furthermore, previous reviews have combined decision makers into one group,^1,5,6 or ignored decision makers altogether,⁷ making it difficult to ascertain how each group’s performance of telephone triage affects patient safety.

Purpose

Our purpose in performing this review is to examine the safety of telephone triage, as performed by four groups of clinical and non-clinical decision makers. We analyzed recent published studies of telephone triage safety and utilized Donabedian’s structure, process, outcome model⁸ – three categories of information used to infer the quality of care – to examine the safety of each group’s system.

This review of the literature is intended to clarify important differences among decision-maker groups, by first describing each group’s system in detail, and then examining the safety of clinicians and non-clinicians in performing telephone triage. We believe that this may be the first study to examine these categories of information in this way.

Methods

Analytical Method

We performed a narrative review, analyzing research published between 2002 – 2012. We conducted electronic searches in PubMed and CINAHL databases using the following search terms:

Telephone + triage, -medicine, -nurse,

Telephone triage + safety, -malpractice, -error, -risk, -physician, -resident, -nurse practitioner, -receptionist, -EMD, -decision making, - appropriate, and -appropriate referral.

Our search resulted in a total of 50 studies, of which 19 met our inclusion and exclusion criteria. We also examined reviews of the literature on telephone triage safety from 2002 – 2012. We used Huibers' study as a basis, and addressed the issue of clinician vs. non-clinician practice, which Huibers was unable to address. We narrowed our research selection to ten studies already analyzed in Huibers' study, and an additional nine studies that met our criteria. We also modeled our Inclusion and Exclusion criteria on Huibers' study, adding other explicit criteria to attain more specificity to describe the problem and outcomes. The authors feel that 19 were a low final number (also noted by Huibers). We believe safety in telephone triage is an under-researched area.

We excluded studies that focused on telephone consultation for disease management as well as those studies with unclear results or possible bias. We also excluded studies that commingled clinicians with non-clinicians, or included additional groups other than four selected. We found no suitable studies of nurse practitioner practice of telephone triage.

Once the articles were selected, one author identified methods, concepts and outcomes, and with a research assistant independently read the articles, extracted characteristics and outcomes, placing them into a results chart. One co-author served as a legal expert in vetting the legal soundness of the paper. Three authors, with 15–30+ years of telephone triage experience as consultants and legal consultants, discussed, analyzed and compared all extracted data and discussed cases of disagreement until consensus was reached. One researcher checked all information presented in the final tables.

The field of telephone triage and associated research has been plagued by widespread confusion about its purpose and definitions. Thus, our analysis was purposely broad, to provide a general understanding of four typical decision maker groups. We carefully defined inclusion criteria describing elements overlooked by previous researchers (Table 1); we also defined commonly used terms (Table 2). We included studies of physicians and registered nurses, EMDs and clerical staff, in any setting, operating 8 to 24 hours day, and managing symptom-based calls. We excluded studies of triage performed by those outside of the four specified groups. Other excluded studies were those of online communications, mental health crises, and results that combined clinical and non-clinical groups.

Table 1.

Inclusion Criteria.

Theme	Inclusion	Exclusion
Technology	Telephone Encounters via Land line or Cell phone	Email, messaging, texting, tweeting, medical applications, Skyping, telemonitoring, telemedicine, teleconference, other.
Process
Staff	Clinicians: Physicians, residents, licensed nurses. Non-clinicians: emergency medical dispatcher, answering service/clerical staff	Emergency Medical Technician, Paramedic, LVN, LPN, Medical Assistants, Physician Assistants, Nurse Practitioners
Task	Symptom-based encounters: – medical diagnosis of symptoms – identifying or verification of emergent or urgent symptoms – estimating symptom urgency – ruling out urgent symptoms – message taking about symptoms	Clinical Call centers performing disease management Poison control, crisis hotlines, health education, mental health counseling, referrals to specialists Routine follow up appointment calls, medication refills, lab results, other informational calls, or non-clinical advice.
Structure
System Component	Component Presence & Quantity: Practitioner type and level of clinical skill Paper and Electronic Guideline (CDSS) Electronic CDMS Paper and Electronic Documentation (EMR) Training Program type: Clinical or software-based Standards	Component Quality: CDSS or CDMS quality Practitioner years of clinical experience, expertise EMR quality Clinical training quality Standards
Setting	Call Centers: EMD/EMS, clinical call center, answering services ED, Medical office, clinic, ED or home office Physicians taking call from unknown site
Outcome Measures
	Appropriate Referral: Timely ED/Urgent Care visit, Office Appointment. Under referral: Referrals resulting in actual or potential harm, error, mistake, injury, unanticipated hospitalization or ED visit, death. Patient self-referral to ED/Urgent Care	Over referral: unnecessary appointment/ED visit. Satisfaction of Patient, Clinician or Call handler Patient compliance Cost savings, Return on Investment (ROI)
Time Period	Office Hours: 9AM- 5PM, M– F (2080 Hours/Year) After Hours: 5PM-9AM M-F. 24 H/Day: Sat, Sun, Holiday (4296 Hours/Year)
Study Designs	Observational, quantitative and comparative studies 2002 -2012	Unclear results; commingled clinicians and non-clinicians; commingled groups other than four selected. Non-English, no abstract or full text article, editorials, letters.

Table 2.

Common Terminology.

• AMPDS: Advanced Medical Priority Dispatch System is a medically developed electronic software that is used by Emergency Medical Dispatchers.

• ACS: Acute Coronary Syndrome

• Appropriate: suitable or proper in the circumstance.

• Computerized Decision Support Systems (CDSS): Expert software systems that remind experienced decision makers of information to consider that s/he once knew, but may have forgotten.

• Computerized Decision Making Systems (CDMS): Expert software systems that allow an unqualified person to make a decision that is beyond his/her level of clinical training and experience.

• Complete System: A complete telephone triage system is made up of qualified staff, medically approved guidelines, electronic medical record (or audiotape, or paper document), training and standards (policies).

• Disposition: A directive from clinician to patient about the time, place, and person by whom the patient’s symptoms are to be further evaluated and/or treated (also known as referral)

• Error: An umbrella term that includes human error, failures of assessment, failures of communications and under referrals.

• Malpractice: The term “malpractice” is specifically related to professional negligence and is committed by a professional. In effect, professionals are held to a higher standard than non- professionals.

• Negligence: Failure to provide due care to patient.

• Referral: (See disposition)

o Appropriate Referral (AR): A timely, safe disposition: “right place, right time, and right person”. Referrals that avoid delay in -care, -evaluation, -treatment.

o Over-Referral (OR): A referral deemed by some to be unnecessary at the time and place initially recommended. Over Referrals are judged to be safe, but not cost effective.

o Under Referral (UR): A referral to a lower level of care than required, often resulting in a delay in care, and causing, or with potential to cause, patient harm. Under Referral may also be a type of error that can result in a delay in care.

• Root Cause of Error: Establishing the root cause of error is a process in which the initiating cause of error is identified. Root causes may include failures of assessment and communication as well as human error (Joint Commission).

• System: A set of detailed methods, procedures, and routines formulated to carry out a specific activity or solve a problem.

• System Error: The Institute of Medicine defines system error as failures of systems, processes, or conditions, that are intended to prevent errors from occurring, and that might lead people to make mistakes. It may also be defined as the “wrong match of plan” [as system], or “failure to use any plan” [as system] to prevent error. IOM

• Timely: Coming early or at the right time. Timely referrals are Referrals at the “right time, right place, with the right person”.

• Vicarious Liability: Liability on the part of employers, who become accountable for the negligence of an employee.

Occasionally, we found it necessary to include additional studies that were not part of our core group of research. Although it may seem like a digression, we felt the additional studies used as references lent authority, clarified terminology and thereby bolstered our arguments and validated the authors’ view. In a field where research is confusing, additional studies provided good background for the reader.

A Brief History of Telephone Triage

In 1978, the New England Journal of Medicine published a study comparing the telephone triage performances of pediatric nurse practitioners’ (PNP), pediatricians and pediatric house officers.⁴ Researchers reported that, as a group, PNPs performed better than two physician groups in appropriate referral rates, interviewing skills, and other related tasks. PNPs also spent slightly more time on the telephone with patients than physicians. Although researchers concluded that physicians needed more telephone triage training,⁴ specialized training has never been universally implemented for physicians, who still rely on diagnostic expertise.⁹

In the 1980s, telephone triage evolved from a practice performed solely by physicians, to one increasingly delegated to nurses. Health maintenance organizations (HMO) like Kaiser Permanente were among the first institutions to make this change. It is not surprising that pediatricians^10–12 were the first to develop guidelines for nurses; pediatricians’ heightened awareness of risks related to telephone triage of vulnerable children likely contributed to the emergence of telephone triage by nurses.

A 1995 study examined nurses’ decision-making strategies when performing telephone triage in an emergency department (ED) setting.¹³ In the study, researchers reasoned that, without guidelines, physicians made diagnostic hypotheses, whereas, nurses used context and pattern recognition as a decision-making strategy. They hypothesized that medical diagnoses are not necessary in telephone triage, concluding that nurses use heuristics, a technique for quickly solving problems, where estimates are achieved by trading precision for speed, and the focus is on understanding and responding to the urgency of the situation. Two early key studies by Perrin and Lephrohon are notable in that they respectively address nurse practitioner and nurse safety (appropriate decisions), without mention of the use of guidelines. Lephrohon’s research specifically describes the use of pattern recognition, which relates to nurses’ decision making process.

Although initially, nurses used no guidelines, they soon began to use paper guidelines (developed by physicians and nurses), and then electronic guidelines (computerized decision support systems CDSS). Although physicians have never used formal guidelines of their own (electronic or paper-based), it is notable that physicians developed the first electronic guidelines (Physician Referral Times Publication, Richard Cohen, Editor, personal communication, 6/12/13) that nurses were required to use.

Telephone Triage Task

We define the essential task of telephone triage as the telephone assessment and disposition of symptoms, which also requires professional judgment, clinical assessment, and pro-active elicitation of information from the patient.^1,2 Researchers¹³ believe nurses use pattern recognition to estimate and/or rule out symptom urgency to arrive at a disposition. Telephone medicine, performed by physicians, is defined as the “telephonic medical diagnosis of patients’ problems”.¹⁴

Telephone medicine (the practice of medicine by phone)¹⁴ is an informal process. While telephone triage (as practiced by nurses) is a subspecialty, telephone triage still lacks universal standardization, regulation, and the professional recognition of other nursing subspecialties. Currently, the American Academy of Ambulatory Care Nurses (AAACN), and a consensus of professional organizations (e.g., ANA, ENA) consider nurses to be the most qualified clinicians to safely perform telephone triage.

The ability to competently assess a patient without visual cues is essential to telephone triage safety. An important part of the task is the ability to rule out urgent symptoms.¹³ Using lists of questions to passively solicit yes/no responses from patients, without knowing how to interpret patient responses, and thereby asking appropriate follow-up questions, does not qualify as an assessment.

EMDs and clerical staff are believed to perform some aspect of the task (verification of emergencies and message taking respectively), however limited. Although EMDs and clerical staff do not perform decision making tasks integral to telephone triage in the strictest sense, these authors felt compelled to include them in this review for several reasons: 1. Current research treats these groups as legitimate decision makers, 2. clerical staffs are increasingly being used as “preliminary assessors” (using yes/no lists of preliminary clinical questions, or organized chief complaints), which we believe is both an unsafe policy, as well as a growing, unquestioned trend.

Right Match of System

In telephone triage, a complete system includes a decision maker, and a minimum of four additional components: guidelines, documentation, training, and standards (policies and procedures).¹⁵ We defined a complete system as the minimal structure required for safety based on legal tradition and current evidence. Systems in current use for telephone triage are comprised of multiple components and strategies that vary, depending upon who is handling the call.

Using Donabedian’s Model, we broadly examined each groups’ system (structure and process) as a match to the task of telephone triage, using appropriate referral rates (outcomes) as a measure of safety.⁸ We noted and counted system components to measure completeness, and analyzed each groups’ respective process by examining the minimum decision-making qualifications, strategies and objectives believed to be used by each group. We tracked outcomes by measuring available referral rates.

Decision Making System Variations

Clinicians

Physicians

Historically, physicians have had a substantial (and perhaps unacknowledged) presence in telephone triage in that they fill multiple roles: employers of office staff, telephone medicine practitioners (when taking call), and telephone triage guideline developers and reviewers. When acting as employers, physicians are responsible for setting office telephone triage policy, and for developing guidelines for use in their offices.

When performing telephone medicine, physicians typically use a single component (documentation of interactions) and are thought to make diagnoses via phone, based on symptoms described by patients. After hours, physicians practice telephone medicine, taking patient calls from various locations. Whereas some physicians take calls directly from their own patients, typically, answering services relay patient messages to physicians. Although answering service employees have no clinical qualifications, increasingly they are being asked to engage in clinical triage activities.¹⁵

Both telephone medicine,⁹ as well as telephone management, performed by physicians’ office staff, has typically been informal, devoid of standards, training programs or guidelines. Without national standardization, office policies and procedures typically vary dramatically among physicians’ practices. Finally, some physicians serve as developers or reviewers of CDSS (or paper-based guidelines) that nurses are required to use.

Nurses

Nurses base telephone triage decisions on their clinical education and experience, as well as the nursing process.^2,18 Nurses typically take symptom-based calls directly; however some healthcare organizations have begun to utilize clerical staff to essentially perform preliminary assessments of patients’ symptoms, prior to sending the information to nurses.⁴³

Although most nurses typically utilize paper guidelines or CDSS, some use no guidelines at all and one study found that nurses who are provided with CDSS, are not using them as directed.¹⁷ Electronic guidelines, known as computerized decision support systems (CDSS), are defined as expert systems that remind experienced decision makers of information to consider that s/he once knew, but may have forgotten. Nurses who use CDSS typically receive training in how to operate the software. However, software training is not a substitute for clinical training, which stresses eliciting a detailed preliminary assessment of symptoms and patient history as the first step.^2,13,18 Nurses also use the strategies of pattern recognition¹³ to estimate symptom urgency.

Clinical training for telephone triage can be varied; some nurses receive on the job training, take formal classes, or attend seminars at professional conferences. Some nurses have no clinical training. Practice standards, core courses and certification for telephone triage (AAACN), and clinical call center accreditation (URAC) have existed since 1995. However, it is unclear how widespread these programs are.

Non-Clinicians

Emergency Medical Dispatchers

It is safe to assume that those typically calling 911 have already perceived what they believe are emergent or life-threatening symptoms. EMDs must verify these patient-identified emergencies. EMDs use computerized decision making systems (CDMS), defined as expert systems that allow an unqualified person to make a decision that is beyond his/her level of clinical training and experience. EMDs in the US and UK typically use Advanced Medical Priority Dispatch System (AMPDS).

With a minimum education of a high school diploma, and additional specialized training, EMDs must adhere closely to highly deterministic CDMS. EMDs are also responsible for managing resources (level of ambulance dispatch) and coaching callers in pre-arrival instructions. Typically, EMDs receive on the job training.¹⁹ While the National Academy of Emergency Dispatch (NAED) offers specialized training, certification and standards, it is unclear how widely utilized this program is.

Clerical Staff

Clerical staff, working in physician’s offices and answering services, has neither the clinical education nor the qualifications to perform telephone triage. However, some clerical staff may be allowed to perform triage activities.

Conceptual Framework

We devised a Decision Maker Framework (Table 3), which also corresponds to Donabedian’s structure-process-outcome model. The Framework includes common knowledge of, and available research about each group’s structure and process (system). It addresses the limitations of previous reviews by differentiating essential characteristics of each group: minimum education, decision-making strategies and objectives, and system components, and by providing a context within which to analyze each group’s system as a match to task of telephone triage.

Table 3.

Decision Maker Framework.

Decision makers	Minimum qualifications	System components	Decision-making strategies	Task objective
Physician/Resident
Autonomous, licensed clinician	Doctorate level: 15 Yrs Science-based Clinical education & training	1 Component – Documentation – Regulation: State Medical Board	Diagnosis Clinical Judgment Critical Thinking	Medical Diagnosis Identify & verify emergencies & urgencies
Licensed Nurse
Autonomous, licensed clinician supported by medically developed computerized decision support software	AA/BS/MS/DNP Doctor of Nursing level 2-7 Yrs Science-based clinical education & Training	3 + Components – Guidelines: Computerized Decision Support System (CDSS) – Documentation: Electronic Medical Record (EMR) or Audiotaping – Clinical Telephone Triage Training – Practice Standards: (AAACN) – Call Center Standards: (URAC) Regulation: Board of Registered Nursing	Pattern recognition Clinical Judgment Contextual information Nursing Process Critical thinking	Identify & verify emergencies & urgencies Estimate symptom urgency Rule out symptom urgency Interpret patient responses
Emergency Medical Dispatcher
Non-clinician operating medically developed diagnostic decision making software	High school graduate 15-21 Hours of Dispatcher Training¹⁹	2+ Components – Guidelines: Computerized Decision Making System (CDMS) – Documentation (EMR) – Training: Certification – Standards: National Academies of Emergency Medical Dispatch (NAEMD) Regulation: Unknown	Verification of emergent & urgent symptoms in order to: Dispatch correct type/number of paramedic units Coach in CPR, Heimlich, First Aid, etc	Verify emergent & urgent symptoms while operating and adhering closely to medically developed diagnostic Computerized Decision Making System
Clerical Staff, Answering Service Staff
Non- clinician	High School Graduate	1 Component Documentation of messages		Accurately transcribe patient description of symptoms

Error

The Institute of Medicine defines system error as “failures of systems, processes, or conditions, that are intended to prevent errors from occurring, and that might lead people to make mistakes”. System error includes organizational and technical failures.²⁰ Examples of system error include: policy inadequacies for continuity of care (ensuring patients’ safe, coordinated transition between health environments), human factors (inadequacies in staffing -levels, -skill mix, staff -education, - competency assessment, - supervision), and information management (software failures).²¹

The restrictive nature of the telephone encounter likely intensifies uncertainty as well as clinician’s exposure to error more than face-to-face clinical care. Case studies (Appendix 1) illustrate how failures of assessment (inadequate scope of assessment) and failed communications (oral, written, and electronic communication errors, or inadequate communication with physicians, patient or family) and human failures (inadequate performance due to fatigue, bias or rushing) can result in a delay in care. These root causes of error²¹ –failures of assessment and communication – plague telephone triage, possibly leading to delays in care or under referral.

Referral Rates

We define a referral to mean a directive from a decision maker to the patient about the time, place, and person by whom the patient’s symptoms are to be further evaluated and/or treated. We use referral interchangeably with disposition. Safety in telephone triage requires timely, appropriate dispositions, which means avoiding delays in care, diagnosis and/or treatment, and ensuring that patients are seen before symptoms escalate. Messages about patient symptoms from clerical staff to physicians do not qualify as referrals or dispositions.

We divided outcomes into two broad categories: Appropriate Referral (AR) and Under Referral (UR). We define AR as timely, safe dispositions: “right place, right time, and right person”. As appropriate, we synthesized diverse outcome measures and placed them into one of two outcomes: appropriate referrals (safe), and errors or under referrals (unsafe). Our definition of AR purposely includes over referral (OR) which act as a “safety margin”,²² and because nurses are taught to err on the side of caution. Given the limited definitions in the studies we reviewed, it was the most effective way to address patient safety in outcomes.

We defined Under Referral (UR) as referrals resulting in delays in care, causing, or with potential to cause patient harm. Some studies only reported errors in either practice or system, and we included errors with UR. If studies did not report AR, we listed that as NP, “not provided”.

We focused on the outcomes of appropriate-, under- referral and error related to four groups. We addressed over referrals minimally, only because it came up in the results. OR are safe (albeit costly) and are considered wasteful of healthcare resources. Including other papers about OR would pull the paper in the direction of cost effectiveness of telephone triage, rather than safety of telephone triage. We treated OR as a subgroup of AR, explaining our reasoning in the study.

Results

Table 4 describes study designs and settings, quantifies system component numbers and reports errors and referral rate (AR, UR) as percentages. One study examined physicians acting as EMDs, which we placed in the physician category. Two studies compared physicians and nurses’ performance, which we reported separately and placed in their respective groups.

Table 4.

Results by Group.

Author	Study Design, Sample Size, Setting, Time of Day, Patient Populations, Decision Maker	System Component (Structure)	Appropriate Referral (Outcome)	Under Referral & Error (Outcome)
Physicians
Andrews et al. (2002) Australia	Researchers assessed calls using protocols as the standard. N = 25 Hospitals with ED and pediatric wards, Mock Calls, After hours, Pediatric Pediatricians & Generalists	Documentation	76.5% Combined Pediatricians 93% Generalists 50%	Poor or absent documentation. Failure to recognize symptom urgency (32.5%) Lack of training
Fourny et al. (2011) France	Prospective review by attending of initial dispatcher decision for calls from patients with ST-elevation myocardial infarction (STEMI) n = 245 Hospital affiliated with EMS call center, Live calls, 24/7, STEMI patients Physicians with telephone triage experience acting as EMS dispatchers	Guidelines Documentation	70%	Inappropriate initial decision for 30% STEMI patients Ignored repeat calls (2-4) from 78 patients before correct decision reached
Katz et al. (2008) USA	Medical malpractice case review by physician & RN Risk Management specialist. N = 32 Physicians & Multiple Specialty Offices, Live Calls, Office & After hours. Adult & Pediatric Multiple decision makers clinical and non-clinical for which MD is responsible	Documentation	NP	Poor documentation 88% Failure to recognize problem urgency 44% Mismanagement of multiple calls 44% Faulty triage 84% Dysfunctional office systems Lack of policy and procedures 38% Covering MD impact 28%
Killip et al. (2007) USA	Analysis of post call patient interview by 2 RN and 1 MD n = 63 Academic health center, Live Calls, After hours, Adult and Pediatric Residents on call	Documentation	86%	Missing documentation 94% Delay in care and “near miss” 8% Medical Error 14%
Lee et al. (2003) USA	Analyses of post call questionnaire/interview & medical record review by an RN. n = 566 Pediatric faculty practice, Live calls, After Hours, Pediatric Pediatricians on call	Documentation	95.8%	Patients who received unadvised significant care (4.2%)
Registered Nurses
Andrews et al. (2002) Australia	Researchers assessed calls using protocols as the standard. n = 25 Hospitals with ED & Pediatric Units, Mock Calls, After hours, Pediatric, RN	Guidelines Documentation	64%	Poor or absent documentation. Failure to recognize symptom urgency (32.5%)
Ernesäter et al. (2010) Sweden	Retrospective analysis of incident reports n = 452 National call center, Live calls, 24/7, Adult & Pediatric, RN	Guidelines Documentation	NP	Incorrect assessments 25.2% Communication error 6.1% Accessibility problems 40.4% Technical problems 13% Routines/Guidelines 14.8%
Ernesäter et al. (2012) Sweden	Review call documentation from malpractice claims n = 45 National call center, Live calls, 24/7, Adult & Pediatric, RN	Guidelines Documentation Standards	NP	Decision process failure 64.4 % Communication failure 77.7% Organizational deficits 53.3% Understaffing/Nurse workload 13% Ignored repeat phone calls
Geisen et al. (2007) The Netherlands	GPs evaluating simulated calls for level of urgency, compared to the gold standard (devised and agreed upon by 7 GPs) n = 352 GP Cooperative, Mock calls, After hours, Adult & Pediatric, RN	Guidelines Documentation Training: variable	82.5% average for all levels of acuity	7.5% average .
Hirsh et al. (2007) USA	Assessed UR rates using percentage of hospital admissions within 24 hours of receiving a non-urgent disposition n = 126,972 Pediatric hospital call center, Live calls, 24/7, Pediatric, RN	Guidelines Documentation Training Standards	94.8%	5.2% 24-hour under referral rate
Huibers et al. (2012) The Netherlands	Secondary analysis of recorded calls by trained observers with validated measurement tool n = 6739 GP Cooperative, Live Calls, After hours, Adult, & Pediatric, RN	Guidelines Documentation Training	94% (average of 3 appropriateness indicators	6% Inappropriate
Kempe et al (2003) USA	Analysis of post call patient survey and health care utilization data n = 1561 After hours call center, Live calls, After hours, Pediatric, RN	Guidelines Documentation Training Standards	99.7%	Potential under referral with subsequent hospitalization (.03%)
Kempe et al (2006) USA	Analysis of call documentation and health care utilization data for telephone triage calls n = 32,968 After hours call center, Live calls, After hours, Pediatric, RN	Guidelines Documentation Training Standards	99.83%	Potential under referral with subsequent hospitalization (0.2%)
Lee et al. (2003) USA	Analyses of post call patient questionnaire &, medical record review by RN. n = 616 Call center, Live calls, Office & After- hours, Pediatric patients, RN	Guidelines Training Documentation	95.7%	Patients who received unadvised significant care (4.3%)
Marklund et al. (2007) Sweden	Analysis of documentation by 2 generalists & one RN for appropriate referrals n = 362 Call center, Live calls, Office hours, Adult & Pediatric, RN	Guidelines Training Documentation	100 % (2.4 % OR)
North Et al. (2011) USA	Retrospective analysis of documentation comparing nurse referral with what caller states they would have done n = 46 Call center, Live calls, 24/7, Adults & Pediatric with appendicitis symptoms, RN	Guidelines Documentation	91%	9%
Emergency Medical Dispatcher
Deakin et al. (2006) UK	Retrospectively compared accuracy of EMD decisions using AMPDS software with clinical diagnosis: acute coronary syndrome (ACS) n = 263 Ambulance service, Emergency calls, 24/7, Adult found to have ACS, EMD	Guidelines Documentation	Patients with ACS identified by AMPDS 71%	Patients with ACS not identified by AMPDS 29%.
Deakin et al. (2009) UK	Retrospectively compared accuracy of EMD decisions using AMPDS software with clinical diagnosis for stroke n = 126 Ambulance service, Emergency calls, 24/7, Adults found to have had a stroke, EMD	Guidelines Documentation	97% Ambulance arrived within 19 min	Patient allocation to chief complaint other than stroke (52.4%) 3% with stroke ambulance arrival >1 hour
Clerical Personnel
Hildebrandt et al. (2003) USA 2	Physician review of calls not forwarded to them due to non-emergent identification by patient n = 288 Primary care offices, Live calls, After hours, Adult & Pediatric Answering service Operator	Documentation	NP	Policy required patient to self-assess own symptom (system error) 50% of patients failed to assess own symptoms appropriately, leading to self-under referral
Hildebrandt et al. (2006) USA	Using call/record data, physician reviewed calls not forwarded to them, due to non-emergent symptom self-assessment by patient n = 119. Primary care offices, Live calls, After hours, Adult & Pediatric, Answering service Operator	Documentation	NP	Lack of documentation 30% suffered clinical harm, potential for future harm, pain and discomfort
Klasner et al. (2006) USA	Retrospective chart review comparing clerk chief complaint with protocol used by nurse n = 292 Pediatric telephone triage program, Live calls, After hours, Pediatric, Clerical staff	Documentation	NP	4 % of chief complaint was different that protocol used, creating a problem for patient.

Clinicians

Nurse studies were retrospective reviews of calls or analyses of records or live calls and two mock calls. Studies of nurses (11) reported the highest AR rates (99.7%),^23,24 and the most complete systems (2-4 components). AR rates were highest in the two nurse studies reporting four components.^23,24

Physician studies were comparative studies (physician and nurse practice, using live and mock calls), or retrospective reviews or analysis of records of live calls focused on error and medical malpractice. One study of MD/EMD evaluated the accuracy of physician diagnosis using AMPDS. In five studies of physicians, one study found pediatricians had the highest AR rate (95.8%);²⁶ while a second study found generalists (taking calls from pediatric patients) performed poorly (AR 50%).²⁷

All physician studies reported a single system component (documentation); however, three of five reported errors of poor or absent documentation.^11,27,28 In a study of physicians acting as EMDs,²⁹ physicians experienced in telephone triage, and using STEMI-specific AMPDS guidelines to verify Acute Cardiac Syndrome symptoms, performed inadequately (AR 70%); 30% of patients were repeat callers (2-4 additional calls from 78 patients took place before MD/EMDS reached AR.

Non-clinicians

Two EMD studies (and one MD/EMD study) were retrospective analyses of live calls focusing on diagnostic accuracy as an outcome. EMD studies uniformly noted two system components (guidelines and documentation). In both EMD studies,^19,22 EMDs utilized medically developed expert software, specifically designed to identify two different clinical diagnoses (Stroke and ACS). The average AR rate was 92%, where the ambulance was dispatched within an appropriate time frame. However, it is important to note that EMDs’ actual identification of the chief complaint (what AMPDS guidelines were designed to do) was accurate only 47% of the time for stroke,¹⁹ and 71% for ACS,²² which is similar to EMD performance “using their own subjective assessment”.¹⁹

Studies of clerical staff were retrospective analyses of live calls focused on message taking accuracy or operational safety of answering service. Other than documentation, no study of clerical staff (3) mentioned any discernible system components.^30–32 One found that clerical staff took accurate messages;³² two studies of answering services found after hours policies to be unsafe.^30,31 Clerical staff had no referral rates.

Discussion

We examined each group’s system as a match to the task of telephone triage, using numbers of components and referral rate results, beginning with non-clinicians. We made this change in order because we were unable to suggest meaningful recommendations for improvement for non-clinicians, due to their lack of qualifications. We conclude with a discussion of clinicians’ errors, followed by specific recommendations for improvement.

Non-Clinician Processes

The EMD system is predicated upon the belief that it is feasible for EMDs to accurately diagnose specific conditions by telephone, provided AMPDS is operated by highly compliant call handlers. Computerized decision making systems essentially supplant operator’s decision-making skills. This makes it difficult to determine whether the operator or AMPDS is determining the decision, and therefore responsible for outcomes.

It is notable that when presented with two everyday emergent conditions (stroke and MI symptoms), EMDs performed inadequately. Even physicians acting as EMDs performed inadequately²⁹ with 30% UR rate for STEMI patients. Researchers did not provide an explanation for these results. Were physicians non-compliant, or did they attempt to fit patient symptoms with AMPDS? The authors’ best guess was that EMDs’ and MD/EMDs’ poor results might be due to both groups’ strict adherence to AMPDS, possibly pointing to flawed policy (for MD/EMDs) or flawed AMPDS (for EMDs). One expert concluded, “AMPDS cannot perform clinical diagnosis; its extension into EMS does not enable accurate identification of ACS patient”.²²

EMDs – the link between the general public and EMS transport – are required to make decisions about paramedic transport resource allocation. High priority responses are costly and require travel at speeds that put paramedics at risk.²⁹ Such decisions may pit patient safety (avoidance of delay in care) against cost containment and paramedic safety. A second reason for concern about EMD safety is that they may unwittingly be forced to rule out emergencies, a task for which they are unqualified, possibly resulting in high UR rates. Although formalized 35 years ago, EMD systems’ current inadequacies may represent the wrong match of system to task. We question the assumption that close adherence to highly deterministic and diagnostic software by call handlers will produce reliably safe outcomes.

Clerical staff has no discernible system or process; they did not perform telephone triage in the studies we examined (despite misleading titles), but rather transcribed messages. In two studies, Hildebrandt,^30,31 investigated the operational safety of an answering service, whose policy required patients to self-assess their own symptoms (and to decide if their symptoms were emergent). The clerical staff subsequently relayed these patient messages to the on-call physician. Of those calls not forwarded to physicians, 30% of patients were found to have suffered actual or potential harm. The policies of most practices surveyed (93%) required that callers determine if their symptoms were emergent (n = 86).

A third study found that clerical staff accurately transcribed patient-reported symptoms into messages for advice nurses³² who subsequently returned the calls, often selecting guidelines that matched the patients’ description of symptoms. Transcribing accurate messages from patients is not equivalent to making appropriate dispositions. The practice of using clerical staff to relay messages to advice nurses, may delay care, especially for pediatric populations, who may rapidly deteriorate within hours²⁵ while awaiting a callback. Claims by researchers that no delay was incurred³² were unsupported. The authors believe that using clerical staff as intermediaries for symptom-based calls may have unintended consequences. It is redundant and may introduce error into the process. This policy also has the potential to delay care.

Notably, all the studies of clerical staff treated message taking as synonymous with telephone triage.^30–32 Current studies that blur professional boundaries by using titles such as “triagists” with lists of yes/no symptom questions³³ are misleading. Researchers’ misguided approaches may unwittingly contribute to unsafe policies that legitimize the introduction of clerical staff into what is, in fact, a clinical process. We believe the growing trend toward substituting clerical staff for nurses is likely related to cost containment.

Clinicians’ Processes

Because telephone medicine has always been an informal process, one might theorize that physicians’ breadth and depth of clinical expertise compensates for their lack of a system. However, physicians’ high UR rates (average 18%), and a study of malpractice cases reporting that 67.5% of legal allegations against physicians were due to failure to diagnose,¹¹ cast doubt on whether diagnosis by phone is a reliably feasible strategy. Although nurses lack the clinical expertise of physicians, nurses’ average AR rate was nearly 10% higher than physicians, and nurses’ UR rates were 9 % lower than physicians.

Under Referrals, Assessment and Communication Failure

Under referrals were a frequent error of both physicians,^11,26–29 and nurses.^{23–27,34–37} Clinicians had high average UR rates: nurses (9%) ranging from 32%²⁷ to .03%^23,24 and physicians (18%), ranging from 44 %¹¹ to 4.2%.²⁶

Under referrals may be related to assessment failures (not recognizing urgency or inadequately estimating symptom urgency). Studies found that UR were often related to ordinary adult and pediatric symptoms. In one study, (with the exception of pediatricians’ higher AR rates for a febrile 6 week old infant), both nurses and physicians under referred mock calls portraying a toddler with head injury, and gastroenteritis.²⁷ In a second study, nurse UR was related to gastroenteritis, croup, asthma, and bronchiolitis.²⁵ A study found that nurses appropriately referred appendicitis symptoms;³⁴ however another study found that nurses under referred adult chest and abdominal pain when the workload was too high.³⁷

If “ruling out urgency is more difficult than identifying it”¹³ (and thus requires professional qualifications), then these efforts are also likely to be more time consuming. Ruling out requires deeper probing, additional questions and more time. Thus, policies that require a high workload or overly brief talk time may foster error. The authors believe that UR result from inadequate assessments.

Mixed definitions of UR yielded varied results. One study’s definition was broader and more representative,³⁸ while a second study defined UR narrowly,^23,39 excluding patient call backs (2, 3 and 4 times within 24 hours). Several researchers^37,40,41 identified patient callbacks as errors. However, in a study of MD/EMD (physician-dispatchers)⁴² researchers did not identify repeat patient phone calls (2-4 call-backs from 78 patients before appropriate decision was reached) as errors. It is unclear whether narrow definitions of UR are being confused with safe referral standards, especially with regard to pediatric populations. One study identified patients’ self-referrals²⁸ as errors.

Communication failures (documentation inadequacies)^11,27,28 were more frequent physician errors; whereas both physicians and nurses had similar assessment failures: failure to recognize urgent symptoms,^{23,24,26,27,36,37} and ignoring repeat phone calls.^11,29,36 It is notable that current studies continue to report these commonplace errors, despite a decade of research, and despite being previously addressed in a 1993 training manual and in guidelines.^10–12 Finally, these common errors have not been reduced by the use of CDSS, which are intended to enhance safety, and to improve communications and assessment processes.

System Error

Killip blamed scarcity of system components as a contributor to physician error, describing telephone medicine as “pervasive organizational failures constituting system error’ adding, “physician expertise and professionalism alone could not prevent common error²⁸”. Reviews of medical malpractice claims supported these findings. In regard to system error, physicians may be responsible for system error in their offices, whereas, nurses’ practice errors may be the result of an organization’s system errors. For clinicians, the presence of four system components, when appropriately developed, might have mitigated these and other identified clinician errors. System error likely underlies practice errors. “Improved systems improve safety⁴⁰”.

Two studies of nurses reported system errors^36,37 that included accessibility issues, software malfunction, inadequate training, understaffing or high workload and overly brief call-processing time requirements interfering with adequate assessments. Nurses’ systems were not universally complete. Many studies mentioned training without specifying the type of training.^{23–26,40,43–45} A single study reported a clinical program comprised of a two-month orientation, didactic sessions and call observation by preceptor.²⁵ We differentiate between training on how to use software and clinical training.

Research Bias

We found researcher bias in how nurses were evaluated and in researcher’s perception of outcomes (Over Referrals). For example, several studies used physicians as the “gold standard” (an over-used and variously defined term in the field) to evaluate nurse decisions.^23,24,40,45 This approach amounts to hindsight bias. One wonders whether researchers lacked confidence in CDSS guidelines to serve as the gold standard; the CDSS is allegedly an expert system based on medical expert consensus. Using expert-level telephone triage nurses to evaluate other nurses may reduce bias. However, we believe that the only legitimate measures of safe decisions are outcomes – actual referral rates of live calls.

When researchers judge nurses’ dispositions as over referrals (OR),⁴⁵ researchers overlook the fact that these type of referrals represents a “margin of safety”.²⁹ Nurses are trained to err on the side of caution. While admittedly, over referrals are not cost-effective, OR represent a norm of safety, rather than nurse decision-making inadequacies.

After Hours Safety

All studies that we reviewed took place either during after hours^{23,24,27,28,30–32,40,44} or over a 24-hour period,^{11,19,22,25,26,29,34,36–38} with one exception (office hours).⁴⁵ Over the last decade, researchers have observed a lack of safety after hours,^1,6 without offering an explanation for this trend.

One explanation for the lack of safety might be that the after hour period comprises nearly two-thirds of all hours annually (Table 2), and it is a period of extremely limited access to health services (actual on site visits). Lack of accessibility is acknowledged as a key system failure.³⁶ Based on system component results, we believe another explanation for the lack of safety is that after hours represents a lengthy and neglected period of time, during which two groups with the least developed systems – answering services and physicians – manage patient calls. Both the current after hours arrangement as well as inadequate office systems represent archaic and unsafe policies; ongoing safety research has not resulted in improved telephone medicine systems; we believe they are unlikely to do so now.

Organizational Negligence

If a claim of negligent telephone triage is made, the system is often scrutinized.^15,45 In medical malpractice cases, expert witnesses testifying on behalf of patients or their families routinely request guidelines, documentation, training materials, and standards (including job descriptions and qualifications), which often comprise the entire system. If structure and processes are found to be inadequate, physicians and organizations such as Health Maintenance Organizations (HMOs) are vulnerable to claims of organizational negligence.

Our findings indicate that claims of vicarious liability and corporate and physician negligence will continue if office system variability continues and unlicensed, unqualified clerical staff are utilized in the place of clinicians.⁴⁶ A complete, high quality system provides “layers of protection” to institutions that implement them.¹⁶ Developing safe systems demonstrates organizational compliance and accountability, bolstering defendant credibility in malpractice lawsuits.

Recommendations

We have questioned several current assumptions that: 1. clerical staff are qualified to perform the task of telephone triage; 2. telephone medicine requires only minimal system components for safe physician practice; and 3. after hours, there is a safe system in place.

We believe that clinical skill combined with a complete system is the bottom line in telephone triage. Additional research on non-clinicians’ safety will likely show the same results; and further attempts to improve non-clinicians’ safety will likely fail. For example, while nurses’ assessment inadequacies may be addressed by clinical training, additional training will not remedy EMD and clerical staffs inadequacies, due to their lack of basic clinical qualifications.

When compared to nurses’ systems, telephone medicine policy and systems have not significantly evolved in 40–50 years. We agree with researchers who believe that physicians are unlikely to change soon.⁴⁷ Thus, we believe that the best policy is to focus research exclusively on nurses and their current system components. Our recommendations are as follows:

We suspect that substitution of non-clinicians for clinicians may be related to cost-containment, and may produce an unintended consequence of lack of safety. In the interest of safety, we recommend that efforts be made to enable nurses or other clinicians to take clinical or symptom-based calls directly.

Definitions of Under Referrals were varied or narrow; it is unclear whether these are being confused with safe referral standards, especially with regard to pediatric populations. Donabedian noted, “professionals suffer from reluctance or inability to establish valid normative standards for outcomes”.⁸ Organizations such as the American Academy of Pediatrics might begin by defining safe referral standards for pediatric populations.

With the exception of clinical call centers, after hours policies represents a lack of any system. We recommend replacing the current arrangement with 24/7 clinical call centers to improve safety.

Nurse system components, while more complete, still need to be improved; especially clinical training to address assessment failures, and call center standards to address system failures. Research should focus on CDSS reliability, validity and safety.

Telephone triage does not operate in a vacuum. Clinicians’ efforts to provide for timely dispositions are currently being undercut by organizations’ failure to provide commensurate on-site access. In addition to establishing more 24/7 clinical call centers, we recommend concurrent expansion of After Hours access to one or more on-site services (urgent care- clinic- and/or office-visits). For example, expanding access to Urgent Care services from 6A to 10P daily would facilitate patient access to less costly services, and reduce inappropriate and costly ED visits.

Limitations

Our study had several limitations, which include wide variation and inconsistent quality in research designs, definitions and outcomes; only a small number of studies met our inclusion criteria. Two groups (EMDs, clerical) had three or fewer studies cited. We may have overlooked or underestimated the presence of components. We were limited by the difficulties in comparing such dissimilar decision making groups.

Conclusion

In this review, we performed a more orderly analysis of clinical and non-clinical decision makers, while addressing the limitations of previous reviews. We highlighted important differences in essential characteristics of each decision maker group and their respective systems. This narrative review identified persistent problems related to telephone triage safety, and offered some solutions. Several of our recommendations favor patient safety over cost concerns. We believe that patient safety must not be secondary to cost containment, and ways must be found to achieve solutions that are both safe and cost effective. According to one expert, “the best choice (of staff) is the least paid person who can do the job well”.¹² We believe nurses to be the most cost-effective professional who can safely perform the task.

We utilized Donabedians’ model, measured system component(s) completeness (structure), analyzed decision-making strategies (process), and evaluated referral rates (outcome) to examine non-clinician and clinician decision making safety. Our examination found that clinicians are safer than non-clinicians, however clinicians’ UR rates are still unacceptably high. Of all groups, nurses achieved the highest AR rates, and had the most complete systems, whereas physicians used the single system component of documentation and frequently failed to document calls – an error noted in three of the five physician studies. Nurse decision-making safety could be greatly enhanced by improving system quality, especially clinical training as well as practice and call center standards. We also found that non-clinicians are not safe decision makers, even when closely adhering to expert software.

In telephone triage, system error – “the use of wrong plan [as system], or failure to use any plan [as system]”⁴⁸ – threatens patient safety. Based on our review, we believe the first step toward formalizing the “right system” is to work toward developing the 24/7 clinical call center as the national model for telephone triage.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Appendix 1: Error in Telephone Triage: Case Studies

References

Huibers

Smits

Renaud

Giesen

Wensing

. Safety of telephone triage in out-of-hours care: a systematic review. Scandinavian Journal of Primary Health Care 2011 Dec; 29(4): 198–209.

Wheeler

. Telephone triage protocols for adult populations, New York City: McGraw Hill Publishers, 2009.

The Lancet. Nurse-telephone triage. The Lancet 2001; 357(9253): 323–323.

Perrin

Goodman

. Telephone management of acute pediatric illnesses. The New England Journal of Medicine 1978; 298(3): 130–5.

Blank

Coster

O’Cathain

Knowles

Tosh

Turner

. The appropriateness of, and compliance with telephone triage decisions: a systemic review and narrative synthesis. Journal of Advanced Nursing 2012; 68(12): 2610–21.

Bunn

Byrne

Kendall

. Telephone consultation and triage: effects on health care use and patient satisfaction (Review). The Cochrane Library 2009; 1: 1–36.

Carrasqueiro S, Oliveira M, Encarnacao P. Evaluation of Telephone Triage and Advice Services: A Systematic Review on Methods, Metrics and Results. Fairfax, VA: IOS Press, 2011 [cited].

Donabedian A. Quality assurance. Structure, process and outcome. Nursing Standard 1992; 7(11 Suppl QA): 4–5.

Flannery

Moses

Cykert

Ogden

Keyserling

Elnicki

. Telephone management training in internal medicine residencies: a national survey of program directors. Academic Medicine 1995; 70(12): 1138–41.

10.

Brown

. Pediatric telephone medicine: principles, triage, and advice, Philadelphia: Lippincott Williams & Wilkins, 1994.

11.

Katz H. Telephone Manual of Paediatric Care. Indianapolis, IN: Wiley, 1982.

12.

Schmitt

. Pediatric Telephone Advice, Boston, MA: Little Brown & Co, 1980.

13.

Leprohon

Patel

. Decision-making Strategies for Telephone Triage in Emergency Medical Services. Medical Decision Making 1995; 15: 240–53.

14.

Reisman

Brown

. Preventing communication errors in telephone medicine: A case based approach. Journal of general internal medicine 2005 Oct; 20: 959–63.

15.

Mahlmeister L. Affordable Care Act and Telephone Triage. Continuing Education Course. San Anselmo, CA: TeleTriage Systems, 2014.

16.

Smith R. Risk management in Telephone Triage: Two expert views. Continuing Education Course. San Anselmo, CA: TeleTriage Systems, 2005.

17.

Greatbatch

Hanlon

Goode

O’Caithain

Strangleman

Luff

. Telephone triage expert systems and clinical expertise. Sociology of Health & Illness 2005; 27(6): 802–30.

18.

Rutenberg C, Greenberg L. The Art and Science of Telephone Triage. Hot Springs, Arkansas: Telephone Triage Consulting, 2012.

19.

Deakin

Alasaad

King

Thompson

. Is ambulance telephone triage using advanced medical priority dispatch protocols able to identify patients with acute stroke correctly? Emergency Medicine Journal.2009 Jun; 26(6): 442–5.

20.

Norman GR, Eva KW. Diagnostic error and clinical reasoning. Medical Education 2010;44:94–100.

21.

The Joint Commission. Sentinel Event Data: Root Causes by Event Type 2004-2Q 2012. In: Commission TJ, editor. Washington, DC: The Joint Commission, Office of Quality Monitoring; 2012.

22.

Deakin

Sherwood

Smith

Cassidy

. Does telephone triage of emergency (999) calls using Advanced Medical Priority Dispatch (AMPDS) with Department of Health (DH) call prioritisation effectively identify patients with an acute coronary syndrome? An audit of 42,657 emergency calls to Hampshire Ambulance Service NHS Trust. Emergency medicine journal: EMJ 2006 Mar; 23(3): 232–5.

23.

Kempe

Bunik

Ellis

. How safe is triage by an after-hours telephone call center? Pediatrics 2006; 118: 457–63.

24.

Kempe

Luberti

Belman

Hertz

Sherman

Amin

. Outcomes Associated With Pediatric After-Hours Care by Call Centers: A Multicenter Study. Ambulatory Pediatrics 2003; 3: 211–7.

25.

Hirsh

Simon

Massey

Thornton

Simon

. The host hospital 24-hour underreferral rate: an automated measure of call-center safety. Pediatrics 2007; 119: 1139–44.

26.

Lee

Baraff

Guzy

Johnson

Woo

. Does telephone triage delay significant medical treatment? Archives of Pediatrics and Adolescent Medicine 2003; 157: 635–41.

27.

Andrews

Armstrong

Fraser

. Professional telephone advice to parents with sick children: time for quality control!. Journal of Paediatrics and Child Health 2002; 38: 23–6.

28.

Killip

Ireson

Love

Fleming

Katirai

Sandford

. Patient Safety in After-hours Telephone Medicine. Family Medicine 2007; 39: 404–9.

29.

Fourny

Lucas

Belle

Debaty

Casez

Bouvaist

. Inappropriate dispatcher decision for emergency medical service users with acute myocardial infarction. The American journal of emergency medicine 2011 Jan; 29(1): 37–42.

30.

Hildebrandt

Westfall

Fernald

Pace

. Harm resulting from inappropriate telephone triage in primary care. The Journal of the American Board of Family Medicine 2006; 19(5): 437–42.

31.

Hildebrandt

Westfall

Smith

. After hours telephone triage affects patient safety. Journal of Family Practice 2003; 52(3): 222–7.

32.

Klasner

King

Crews

Monroe

. Accuracy and response time when clerks are used for telephone triage. Clinical Pediatrics 2006; 45: 267–9.

33.

Derkx

Rethans

J-JE

Muijtjens

Maiburg

Winkens

van Rooij

. Quality of clinical aspects of call handling at Dutch out of hours centres: cross sectional national study. BMJ 2008; 337: a1264–a1264.

34.

North

Odunukan

Varkey

. The value of telephone triage for patients with appendicitis. Journal of Telemedicine and Telecare 2011; 17: 417–20.

35.

North

Varkey

. How serious are the symptoms of callers to a telephone triage call centre? Journal of Telemedicine and Telecare 2010; 16(7): 383–8.

36.

Ernesäter

Engstrom

Holmstrom

Winblad

. Incident reporting in nurse-led national telephone triage in Sweden: the reported errors reveal a pattern that needs to be broken. Journal of Telemedicine and Telecare 2010 Jul; 16(5): 243–7.

37.

Ernesäter A, Winblad U, Engstrom M, Holmstrom IK. Malpractice claims regarding calls to Swedish telephone advice nursing: what went wrong and why? Journal of Telemedicine and Telecare 2012.

38.

Hirsh

Simon

Massey

Thornton

Simon

. The host hospital 24-hour underreferral rate: an automated measure of call-center safety. Pediatrics 2007; 119: 1139–44.

39.

Kempe

Bunik

Ellis

. How safe is triage by an after-hours telephone call center? Pediatrics 2006; 118: 457–63.

40.

Giesen

Ferwerda

Tijssen

Mokkink

Drijver

van den Bosch

. Safety of telephone triage in general practitioner cooperatives: do triage nurses correctly estimate urgency? Quality and Safety in Health Care 2007; 16: 181–4.

41.

Katz

Kaltsounis

Halloran

Mondor

. Patient safety and telephone medicine: some lessons from closed claim case review. J Gen Intern Med 2008; 23: 517–22.

42.

Fourny

Lucas

Belle

Debaty

Casez

Bouvaist

. Inappropriate dispatcher decision for emergency medical service users with acute myocardial infarction. Am J Emerg Med 2011 Jan; 29(1): 37–42.

43.

Giesen

Smits

Huibers

Grol

Wensing

. Quality of After-Hours Primary Care in the Netherlands: A Narrative Review. Annals of Internal Medicine 2011; 155: 108–13.

44.

Huibers

Giesen

Smits

Mokkink

Grola

Wensing

. Nurse telephone triage in Dutch out-of-hours primary care: the relation between history taking and urgency estimation. European Journal of Emergency Medicine 2012; 19: 309–15.

45.

Marklund

Ström

Månsson

Borgquist

Baigi

Fridlund

. Computer-supported telephone nurse triage: an evaluation of medical quality and costs. Journal of Nursing Management 2007; 15: 180–7.

46.

Dahlberg CP. State probes whether Kaiser call centers endanger patients. The Kaiser Papers, A Public Service Page Sacramento, CA: Sacramento Bee; 2009.

47.

Hannan

Celia

. Are doctors the structural weakness in the e-health building? Internal Medicine Journal 2013; 43: 1155–64.

48.

Institute of Medicine. To Err is Human: Building a Safer Health System. In: Institute of Medicine Committee on the Quality of Health Care in America, editor. Washington DC: Institute of Medicine; 1999.