Frequent major errors in antimicrobial susceptibility testing of bacterial strains distributed under the Deutsches Krebsforschungszentrum Quality Assurance Program

Abstract

The Quality Assurance Program (QAP) of the Deutsches Krebsforschungszentrum (DKFZ) was a proficiency testing system developed to service the laboratory animal discipline. The QAP comprised the distribution of bacterial strains from various species of animals for identification to species level and antibiotic susceptibility testing (AST). Identification capabilities were below acceptable standards. This study evaluated AST results using the DKFZ compilations of test results for all bacterial strains showing the number of participants reporting the strain as resistant (R), sensitive (S) or intermediate susceptible (I) to each antibiotic substance used. Due to lack of information about methods used, it was assumed that what the majority of the participants reported (R or S) was the correct test result and that an opposite result was a major error (ME). MEs occurred in 1375 of 14,258 (9.7%) of test results and ME% ranged from 0% to 23.2% per bacterial group–agent group combination. Considerable variation in MEs was found within groups of bacteria and within groups of agents. In addition to poor performance in proper species classification, the quality of AST in laboratory animal diagnostic laboratories seems far below standards considered acceptable in human diagnostic microbiology.

Keywords

Bacteria antimicrobial susceptibility testing quality diagnostic laboratories proficiency testing

The Quality Assurance Program (QAP) of the Deutsches Krebsforschungszentrum (DKFZ, Dr Werner Nicklas) was a proficiency testing system developed to service the laboratory animal field. The QAP comprised the quarterly distribution of two bacterial strains originating from various species of animal for identification to the species level and antibiotic susceptibility testing (AST).

In 2007 we compared the identification results reported by QAP participants over the years 1996–2004 with those obtained by the Dutch Bacterial Diagnostics reference laboratory, the only participant that used genetically-based methods for identification. We found that the capabilities of laboratory animal diagnostic laboratories to correctly identify bacterial species was considerably below what is considered acceptable for human diagnostic laboratories.¹

The outcome of the AST over the same period is now reported.

Identification results were available for 71 bacterial strains and a fungus. In the present study the outcome of AST of five of the 72 strains was omitted. In QAP distribution no. 24, the same outcome was reported for Staphylococcus cricetus and CDC EF4a, suggesting an error in record keeping; in QAP distribution no. 26 results of European laboratories could not be separated from those from the USA; and the one strain of QAP distribution no. 42 was omitted, being a fungus. As a consequence AST results on 66 bacterial strains remained.

The bacteria comprised of Gram-positive rods (n = 6) and cocci (n = 21) and Gram-negative rods (n = 38) and a coccus (n = 1). Where possible the bacteria were grouped in bacterial families containing at least three strains. According to their Gram stain characteristics and morphology, the remaining bacteria were assigned to the groups ‘other Gram-negative rods’, ‘other Gram-positive cocci’ and ‘other Gram-positive rods’. In this way 11 groups were formed. The number of strains per group ranged from one (the Gram-negative Neisseria) to 21 (Enterobacteriaceae).

The AST results for each strain were compiled by Dr Nicklas in a list showing the number of participants reporting the strain as resistant (R), sensitive (S) or intermediate susceptible (I) to each antibiotic substance used. All data were included in an Excel file which was used for calculations. The 66 strains included in this study were numbered one to 66.

Antimicrobial substances

During the study period 1996–2004, the participating laboratories reported the use of 43 antimicrobial agents.

Piperacillin, naladixic acid, ofloxacin, metronidazole and spectinomycin were only rarely used and results obtained with these five agents were omitted from calculations.

It was possible so far to group the remaining 38 substances according to their mode of action in nine groups comprising aminoglycosides (n = 6: amikacin, gentamycin, kanamycin, neomycin, streptomycin and tobramycin), beta-lactam antibiotics (n = 11: amoxicillin, amoxicilin/clavulanic acid, ampicillin, bacitracin, carbenicillin, imipenem, oxacillin, penicillin, ticarcillin, cefotaxim and cephalotin), glycopeptides (n = 1: vancomycin), inhibitors of folic acid synthesis (n = 3: co-trimoxazole, sulfonamide and trimethoprim-sulfamethoxazole); lincosamides (n = 2: clindamycin and lincomycin), macrolides (n = 3: erytromycin, spiramycin and tylosin), polymixines (n = 2: colistin and polymixin B), quinolones (n = 2 ciprofloxacin and enrofloxacin) and tetracyclines (n = 3: doxycycline, oxytetracycline and tetracycline). The remaining five agents were included in the 10th rest group comprising chloramphenicol, fucidine, nitrofurantoin, novobiocin and rifampicin. The number of antimicrobials per group ranged from 1 (vancomycin) to 11 (beta-lactam antibiotics).

Laboratories did not provide information regarding the methods used and no information was available regarding which strains were studied by specific laboratories. It must be assumed that most laboratories would have used so-called disc methods, and some may have had minimum inhibitory concentrations (MICs). However, this lack of information meant that only overall results could be described and that it was impossible to evaluate the data for possible statistically significant differences, for instance between participants, methods, groups of bacteria and antibiotic substances.

Each test of the susceptibility of a bacterial strain for an antimicrobial substance yields an individual test outcome which obviously may be scored as either R, S or I. It has been assumed that what the majority of the participants reported is the correct test result, as there is no alternative. If the majority reports a bacterium R or S for a given substance, I results are acceptable. Where bacterium–antimicrobial combinations yielded a majority of I results or an equal number of S and R outcomes, a majority was created by addition of the number of I to either the number of R or S (so R + I > S or S + I > R). The opposite outcome (S and R, respectively) reported by the minority was then considered a ‘major error’ (ME). The total number of test results reported from the use of the 38 agents and 66 bacteria was 14,258.

For each bacterium–antimicrobial combination it may be expected that the lower the number of AST test results the more likely it will be to have no MEs. Conversely, the likelihood of finding disagreement in test results between laboratories increases with the number of test results (data not shown; correlation not calculated).

The ME percentages were classified and, although this may seem arbitrarily, 10% MEs means that one in 10 test results deviates from what was reported by the majority; with 20% and 33.3% MEs one in five and one in three test results, respectively, deviates from the correct test result.

I counted 1375 MEs among 14,258 test results (9.7%). Table 1 shows the ME rates for each of the 11 groups of bacteria and the 10 groups of antimicrobial substances. The ME% ranged from 0 to 23.2. Percentages exceeding 20 were found in seven of 11 groups of bacteria (range of averages 6.7–16.6) and with six of 10 groups of antibacterial substances (range of averages 4–15). Figure 1 shows the average ME% calculated for all successively distributed 66 bacterial strains.

Figure 1

Percentage of major errors in antimicrobial susceptibility testing of 66 bacterial strains successively distributed under the Deutsches Krebsforschungszentrum Quality Assurance Program (1996–2004)

Table 1

Percentage major errors (MEs) in testing susceptibility of 66 bacterial strains for antimicrobial agents

		Am	β-L	V	F-I	Linc	Mac	Pol	Quin	Tet	Var	All	ME average	Minimum	Maximum
Neisseriaceae
R		43	56	5	18	9	14	7	14	21	25	212
ME	n =	5	8	0	2	0	2	0	0	1	3	21
	%	11.6	14.3	0	11.1	0	14.3	0	0	4.76	12		10	0	14.3
Alcaligenaceae
R		229	306	16	114	41	71	47	67	116	128	1135
ME	n =	14	43	0	26	0	11	2	4	5	10	115
	%	6.11	14.1	0	22.8	0	15.5	4.26	5.97	4.31	7.81		10.1	0	22.8
Enterobacteriaceae
R		802	1347	90	359	160	293	183	266	379	451	4330
ME	n =	69	79	2	37	6	11	16	12	28	33	293
	%	8.6	5.86	2.22	10.3	3.75	3.75	8.74	4.51	7.39	7.32		6.7	2.22	10.3
Moraxellaceae
R		123	178	11	52	22	41	23	35	49	66	600
ME	n =	2	19	0	12	1	5	0	3	2	3	47
	%	1.63	10.7	0	23.1	4.55	12.2	0	8.57	4.08	4.55		7.8	0	23.1
Other Gram-negative rods
R		368	527	34	152	60	113	81	113	156	176	1780
ME	n =	42	61	2	34	3	23	4	6	23	17	215
	%	11.4	11.6	5.88	22.4	5	20.4	4.94	5.31	14.7	9.66		12.1	4.94	22.4
Enterococcaceae
R		133	198	24	59	34	55	28	37	66	81	715
ME	n =	11	38	3	12	2	1	2	7	10	14	100
	%	8.27	19.2	12.5	20.3	5.88	1.82	7.14	18.9	15.2	17.3		14	1.82	20.3
Staphylococcaceae
R		278	401	39	120	70	105	51	78	129	184	1455
ME	n =	12	31	1	26	3	2	11	3	4	5	98
	%	4.32	7.73	2.56	21.7	4.29	1.9	21.6	3.85	3.1	2.72		6.7	1.9	21.7
Streptococcaceae
R		303	485	44	144	75	122	67	90	142	181	1653
ME	n =	50	43	1	32	8	8	12	12	15	17	198
	%	16.5	8.87	2.27	22.2	10.7	6.56	17.9	13.3	10.6	9.39		12	2.27	22.2
Other Gram-positive cocci
R		189	291	29	89	46	61	40	61	86	113	1005
ME	n =	37	55	2	12	8	8	2	8	13	22	167
	%	19.6	18.9	6.9	13.5	17.4	13.1	5	13.1	15.1	19.5		16.6	5	0
Corynebacteriaceae
R		158	241	27	69	38	69	37	43	77	99	858
ME	n =	13	13	0	16	1	1	6	1	2	6	59
	%	8.23	5.39	0	23.2	2.63	1.45	16.2	2.33	2.6	6.06		6.9	0	23.2
Other Gram-positive rods
R		75	120	9	36	21	29	15	25	41	44	415
ME	n =	15	16	2	5	2	6	1	6	3	6	62
	%	20	13.3	22.2	13.9	9.52	20.7	6.67	24	7.32	13.6		15	6.67	24
All
R		2701	4150	328	1212	576	973	579	829	1262	1548	14158
ME	n =	270	406	13	214	34	78	56	62	106	136	1375
Average	%	10	9.78	3.96	17.7	5.9	8.02	9.67	7.48	8.4	8.79		15	3.96	17.7
Minimum		1.63	5.39	0	10.3	0	1.45	0	0	2.6	2.72
Maximum		20	19.2	22.2	23.2	17.4	20.7	21.6	24	15.2	19.5

Am: aminoglycosides (n = 6); β-L: beta-lactam antibiotics (n = 11); V: vancomycin (n = 1); F-I: inhibitors of folic acid synthesis (n = 3); Linc: lincoamides (n = 2); Mac: macrolides (n = 3); Pol: polymyxines (n = 2); Quin: quinolones (n = 2); Tet: tetracyclines (n = 3); Var: various (n = 5)

Bacteria grouped in families comprising ≥3 strains or according to Gram stain characteristics. Antimicrobials grouped according to mode of activity

Of course grouping of bacteria and grouping of antibacterial substances masks variation in AST results within groups. As an example, Table 2 specifies findings obtained for the 11 bacterial groups with the six individual aminoglycosides. Of the 66 bacterium–antimicrobial combinations possible, 51 of 66 (77%) yielded MEs. Of the 121 bacterium–antimicrobial combinations possible with the 11 beta-lactam antibiotics, 97 of 121 (80%) showed MEs.

Table 2

Percentage major errors (MEs) in testing susceptibility of 66 bacterial strains for six aminoglycosides

	n =			Amikacin	Gentamycin	Kanamycin	Neomycin	Streptomycin	Tobramycin
Neisseriaceae	1	R	n	4	12	7	8	9	3
		ME	%	25	0	0	37.5	11	0
Alcaligenaceae	5	R	n	19	62	36	44	49	19
		ME	%	0	6.5	8.5	7	8	0
Enterobacteriaceae	21	R	n	84	226	124	141	145	82
		ME	%	8.5	7.5	8	8	11	10
Moraxellaceae	3	R	n	13	34	17	22	24	13
		ME	%	0	3	0	0	0	8
Other Gram-negative rods	9	R	n	33	107	54	66	72	36
		ME	%	12	14	4	13.5	14	5.5
Enterococcaceae	3	R	n	10	38	27	21	28	9
		ME	%	0	13	7.5	14.5	3.5	0
Staphylococcaceae	7	R	n	29	74	53	44	51	27
		ME	%	3.5	4	7.5	4.5	4	0
Streptococcaceae	7	R	n	23	87	53	56	63	21
		ME	%	13	21	13	18	11	24
Other Gram-positive cocci	4	R	n	17	48	30	37	38	19
		ME	%	17.5	19	30	16	18.5	16
Corynebacteriaceae	4	R	n	12	46	23	28	36	13
		ME	%	8.5	11	4.5	3.5	14	0
Other Gram-positive rods	2	R	n	5	19	13	17	14	7
		ME	%	0	16	38.5	17.5	28.5	0

Differences in the outcome of AST of bacteria have been reported.^2,3 Discrepancies in findings between laboratories have motivated improvement through the publication of standards and guidelines for AST and interpretation of results.

A series of recommendations has been issued by the Clinical and Laboratory Standards Institute (CLSI),⁴ as a successor to the (US) National Committee for Clinical Laboratory Standards (NCCLS). Other recommendations, however, exist.⁵

The CLSI guidelines provide guidance on the most relevant drugs to test and report on specific organisms, quality control ranges to assure accurate and reproducible results, and interpretative criteria or breakpoints to interpret MICs and disc diffusion zones. The most widely used CLSI recommendations are those for disc diffusion tests (M2)⁶ and for the determination of MICs (M7)⁷ of various drugs. The CLSI issues cover a wide variety of bacterial species including infrequently isolated and fastidious bacteria.⁸ Guideline M37 (version 3)⁹ issued in 2008 deals with antimicrobial substances used in the veterinary field. It is unclear whether the laboratory animal diagnostic laboratories that participated in the QAP followed such recommendations.

The data show that AST of bacteria by laboratory animal diagnostic laboratories may frequently yield conflicting results. If most laboratories report a bacterium S to a given antimicrobial agent, an opposite test result (R) should be considered an ME. The percentage of MEs appeared to be about 10, and in various bacteria–antimicrobial combinations tested ME% were above 20. Based on ME percentages (Figure 1) the capability of participating laboratories in AST did not improve over time. According to CLSI recommendation M23A¹⁰ the ME rate should not exceed 3.5%. AST has several pitfalls and an analysis of recently published articles revealed a number of frequently occurring shortcomings in AST of animal bacterial strains, which may have an impact either directly on the quality of the results obtained or on the conclusions drawn.¹¹

We reported earlier that laboratory animal diagnostic laboratories showed poor performance in proper species classification of bacterial strains.¹ The present evaluation indicates that the quality of AST by laboratory animal diagnostic laboratories participating in the DKFZ-QAP over the period 1996–2004 was also far below standards considered acceptable in human diagnostic microbiology.

In the last 10–15 years, several automated instruments have been developed for identification and AST of bacterial strains considered relevant in human and veterinary infections.^12,13 Their advantages comprise ease of use, reduction of work load and standardization of reading test results. A major disadvantage of all systems is that they require the manual preparation of inocula and inoculation of test panels. Inoculum size has been found to be a primary determinant of the accuracy of AST by the Vitek system.¹⁴ It is unclear to which extent automated instruments are presently used by laboratory animal diagnostic laboratories. Unfortunately, DKFZ-QAP was terminated in 2004. Today the only proficiency testing system to service the laboratory animal discipline is the Performance Evaluation Program (PEP) for Diagnostic Laboratories established in 2006 by ICLAS.¹⁵ The PEP does not however include AST, so no more recent data are available than those summarized here. Until new data are published, one should be concerned by the limited capabilities of laboratory animal diagnostic laboratories in AST of bacteria.

References

Boot

, Reubsaet

FAG

. Identification of bacterial strains by laboratories participating in the DKFZ Quality Assurance Program (QAP). Lab Anim 2007;41:481–91

Tenover

, Mohammed

, Stelling

, Ability of laboratories to detect antimicrobial resistance: proficiency testing and quality control results form the World Health Organization's external quality assurance system for antimicrobial susceptibility testing. J Cin Microbiol 2001;39:241–50

Varaldo

. Antimicrobial resistance and susceptibility testing: an evergreen topic. J Antimicrobial Chemother 2002;50:1–4

Clinical and Laboratory Standards Institute. See www.CLSI.org (last checked 20 May 2012)

Gray

, Shrycock

. Antibiotic susceptibility testing of bacteria isolated from animals. Clin Microbiol Newsl 2005;17:131–5

CLSI. Performance Standards for Antimicrobial Disk Susceptibility Tests. Approved Standard M2-A9. Wayne, PA, 2006

CLSI. Methods for Dilution Antimicrobial Susceptibility Tests for Bacteria that Grow Aerobically.Approved Standard M7-A7 . Wayne, PA, 2006

Jorgensen

, Hindler

. New consensus guidelines for the clinical and laboratory standards institute for antimicrobial susceptibility testing of infrequently isolated or fastidious bacteria. Clin Infect Dis 2007;44:280–6

CLSI. Development of In Vitro Susceptibility Testing Criteria and Quality Control Parameters for Veterinary Antimicrobial Agents. Approved Guideline M37, 3rd edn. Wayne, PA, 2008

10.

NCCLS. National Committee for Clinical Laboratory Standards. Development of In Vitro Susceptibility Criteria and Quality Control Parameters. Approved Guideline M23-A. Villanova, PA, 1994

11.

Schwartz

, Silley

, Simjee

, Woodford

, Assesssing the antimicrobial susceptibility of bacteria obtained from animals. Vet Microbiol 2010;141:1–4

12.

Barth Reller

, Weinstein

, Jorgensen

, Antimicrobial susceptibility testing: a review of general principles and contemporary practices. Clin Infect Dis 2009;49:1749–55

13.

Tenover

. Potential impact of rapid diagnostic testing on improving antimicrobial use. Ann NY Acad Sci 2010;1213:70–80

14.

Doern

, Brueggemann

, Perla

, Multicentre laboratory evaluation of the bioMerieux Vitek antimicrobial susceptibility testing system with 11 antimicrobial agents versus members of the family Enterobacteriaceae and Pseudomonas aeruginosa . J Clin Microbiol 1997;35:2115–19

15.

International Council for Laboratory Animal Science. See www.iclas.org/network.htm (last checked 20 May 2012)