Validity and Reliability of a Systematic Database Search Strategy to Identify Publications Resulting From Pharmacy Residency Research Projects

Abstract

Purpose:

This study aims to develop a systematic search strategy and test its validity and reliability in terms of identifying projects published in peer-reviewed journals as reported by residency graduates through an online survey.

Methods:

This study was a prospective blind comparison to a reference standard. Pharmacy residency projects conducted at the study institution between 2001 and 2012 were included. A step-wise, systematic procedure containing up to 8 search strategies in PubMed and EMBASE for each project was created using the names of authors and abstract keywords. In order to further maximize sensitivity, complex phrases with multiple variations were truncated to the root word. Validity was assessed by obtaining information on publications from an online survey deployed to residency graduates.

Results:

The search strategy identified 13 publications (93% sensitivity, 100% specificity, and 99% accuracy). Both methods identified a similar proportion achieving publication (19.7% search strategy vs 21.2% survey, P = 1.00). Reliability of the search strategy was affirmed by the perfect agreement between 2 investigators (k = 1.00).

Conclusion:

This systematic search strategy demonstrated a high sensitivity, specificity, and accuracy for identifying publications resulting from pharmacy residency projects using information available in residency conference abstracts.

Keywords

pharmacy residency pharmacy education research publications

Background

A pharmacy residency, as defined by American Society of Health-System Pharmacists (ASHP), is a structured, directed, salaried postgraduate training period beyond earning a Doctorate of Pharmacy degree, during which time pharmacy residents will acquire substantial knowledge required for skillful problem solving and growth of clinical judgment.¹ With the continuous evolution of clinical pharmacy, an increasing number of graduating pharmacy students seek to complete a pharmacy residency each year. In 2014, approximately 3600 postgraduate year (PGY) 1 and 2 residency positions were available, and ASHP predicts ongoing growth in the number of pharmacy residents and residency programs.^2,3 As conducting pharmacy practice research is one of the goals designated by ASHP for a pharmacy residency, residents are required to complete at least 1 major research project in addition to one or more minor projects for successful completion of an ASHP-accredited pharmacy residency program.⁴ In general, the major residency project is the primary research focus of the residency year, whereas minor projects are of smaller magnitude. The findings of the residency research projects are commonly shared via scientific poster presentation and/or journal publication.

A substantial amount of human resources are dedicated toward these projects, often with the hope that these research efforts could yield positive outcomes for the resident and health system through external visibility with publication in peer-reviewed scientific journals and by adding to the overall knowledge base of the pharmacy and medical community.^5,6 Despite the growing number of pharmacy residents completing a residency program each year, previous studies have reported that only 4% to 16% of residency research projects achieve publication in a peer-reviewed journal, based on online searches using one or more databases (Table 1).^7
–9 The sensitivity and specificity of the search strategies used in the previous studies, however, are unknown; thus, these studies may have underidentified the proportion of projects that were published. Although direct survey of residency graduates would be ideal to track outcomes of residency research projects, resource limitations to administer such survey may be present, especially for older and/or larger residency programs. Therefore, this study aimed to develop a validated systematic search strategy to accurately identify publications resulting from pharmacy residency research projects. Using the newly developed search strategy, the proportions of residency projects identified as published in peer-reviewed journals were compared to those obtained via online survey of residency graduates.

Table 1.

Comparison of Similar Studies.^a

Characteristics	McKelvey et al⁷	Olson et al⁸	Current study
Data source for abstract booklets	Southeastern Residency Conference	Western States Conference	Southwest Leadership Conference and Midwest Pharmacy Residents Conference
Conference years	1981, 1991, and 2001	1995, 2000, and 2005	2001 through 2012
Search engines used	MEDLINE, then IPA	MEDLINE, then EMBASE	PubMed, then EMBASE
Number of abstracts reviewed	272	270	66
Number of investigators conducting the search	1 (positive results verified by a second investigator)	2	2 (discrepancies settled by a third investigator)
Search time limits	6 years from each conference	5 years from each conference	5 years after graduation (about 5.5 years from each conference)
Search terms	Last name of the first author (with or without a first initial) plus keywords	Last name of the first author plus keywords	See Figure 1
Types of articles considered as publications	Not specified	Any published article	Full-text original research manuscripts
Months to publication, median	23	24	11
Proportion of projects identified as published (95% CI)	15.8%^b (12%-21%)	6.3%^c (4%-10%)	21% (12%-33%)

Abbreviations: CI, confidence interval; IPA, international pharmaceutical abstracts.

^aPublication by O’Dell et al⁹ is not included in this table as the article does not discuss the search strategy in detail.

^bThe proportion of projects identified as published was 20% in 1981, 15.7% in 1991, and 12.5% in 2001. The 95% confidence intervals were calculated by the authors of this article using the data reported in the published manuscript.

^cThe proportion of projects identified as published was 4.2% in 1995, 5.2% in 2000, and 8.2% in 2005. The 95% confidence intervals were calculated by the authors of this article using the data reported in the published manuscript.

Methods

This study was a prospective blind comparison to a reference standard. The reference standard within this study was the information on publication obtained via an online survey of residency graduates. Investigators extracted data from the abstracts and performed the systematic search strategy prior to reviewing the results of the online survey to reduce observer bias. Approval by the institutional review board of the study institution was obtained. Each survey participant provided informed consent.

Residency Project Inclusion

All pharmacy residency research projects completed by PGY1, PGY2, or PGY1/2 health-system administration residents at the study institution, a tertiary-care academic medical center, between 2001 and 2012 were included. Although pharmacy residency training at the institution was established in 1987, the specified time period was chosen to ensure that residency conference abstracts and e-mail addresses for residency graduates could be obtained for all residency graduates.

Data Extraction From Residency Conference Abstracts

Due to their wide availability, previously published studies utilized regional conference abstract booklets as data sources. For instance, McKelvey et al utilized abstract booklets from the Southeastern Residency Conference, and Olson et al used those from the Western States Conference.^7,8 To remain consistent with the designs of these previous studies, this study obtained pharmacy residents’ research abstracts from the conference booklets of the corresponding regional (Midwest Pharmacy Residents Conference) residency conference. The residency program did not send residents to this regional conference in 2001 and 2002, and the abstracts from these 8 residency projects were extracted from the state (Southwest Leadership Conference) residency conference. Except for the word count limit that was higher for the regional conference, the stringency of abstract submission requirements for the conferences was considered to be similar. The projects presented at these conferences were assumed to be the major residency research project for the purpose of this study.

As keywords for each abstract were not available within the abstract booklets, 3 investigators independently reviewed each abstract to identify keywords for use in the systematic database search; keywords were defined as names of pharmacologic agents under investigation (eg, insulin) or nouns that frequently occurred within an abstract and reflected the primary outcome of the study (eg, hypoglycemia). The keywords identified by each investigator were then reconciled by the primary author to remove duplicates. No limit was placed on the number of keywords per abstract. By consensus among all investigators, a single keyword that most accurately reflected the content was selected among multiple keywords for each abstract. Metrics of agreement for keyword extraction among the investigators were not tracked.

Development of the Systematic Search Strategy

A step-wise, systematic procedure containing up to 8 search strategies in PubMed and 8 search strategies in EMBASE was created to identify full-text manuscripts resulting from corresponding residency research projects. These databases were selected due to their wide subject coverage of biomedical, medical, and pharmacy journals.¹⁰ Full-text manuscripts were defined as original research articles only, excluding abstracts, letters to the editor, review articles, and other types of published articles. To minimize transcription errors and inter-rater variability during the systematic database search, the text for each search strategy was electronically generated from a Microsoft Excel 2010 database (Microsoft Corporation, Redmond, Washington) into a single Microsoft Word 2010 document (Microsoft Corporation) using the Microsoft Word 2010 mail-merge feature.

In order to further maximize sensitivity, complex phrases with multiple variations were truncated to the root word (eg, “pharmac*” when searching for word such as “pharmacy” or “pharmacist”). Author names were searched with author search field tags (“[au]” in PubMed and “:au” in EMBASE) to increase specificity. All searches were limited to publications that were printed within a 5-year period that began at the year of graduation from the residency program. This time limit was selected based on the results of a Cochrane review that showed a median time to publication ranging from 9 to 36 months following presentation at a biomedical conference.¹¹

The strategies used in previous studies by McKelvey et al and Olson et al served as the foundation for the systematic search strategy used in this study.^7,8 Through modification of the previous search strategies, a new systematic search strategy was developed (Table 2 and Figure 1) that consisted of 8 search strategies. Multiple pilot searches were conducted to improve the efficiency of the search strategy. Importantly, the results of the pilot searches led to the use of a primary single keyword and root word truncation of keywords to maximize sensitivity while reducing the number of irrelevant citations.

Table 2.

Example Search Using the Search Strategy.

PubMed Search	Search Terms/Results
1A. Last name of the primary author followed by “[au]”	(Swan [au]) AND (2010/1: 2015/12 [dp])
Number of citations found? = 529	⊠ > 25 citations – Go to 1B □ ≤ 25 citations – Review citations
1B. Add keywords separated by “OR” to 1A using “AND”	(Swan [au]) AND (2010/1: 2015/12 [dp]) AND (rituximab OR rapid infusion OR reaction* OR safety)
Number of citations found? = 32	□ > 100 citations – Go to 1C ⊠ ≤ 100 citations – Review the citations
1C. Add the first initial of the primary author to 1B	(Swan J [au]) AND (2010/1: 2015/12 [dp]) AND (rituximab OR rapid infusion OR reaction* OR safety)
Number of citations found? = N/A	□ > 100 citations – Go to 1D □ ≤ 100 citations – Review the citations
1D. Replace the keywords used in 1C with the consensus keyword	(Swan J [au]) AND (2010/1: 2015/12 [dp]) AND (rituximab)
Publication found?	□ Yes – END search here ⊠ No – Go to 1E
1E. Last names of the co-authors followed by “[au]”	(Murillo [au] OR Cox [au] OR Baker [au]) AND (2010/1: 2015/12 [dp])
Number of citations found? = 11242	⊠ > 25 citations – Go to 1F □ ≤ 25 citations – Review the citations
1F. Add keywords separated by “OR” to 1E using “AND”	(Murillo [au] OR Cox [au] OR Baker [au]) AND (rituximab OR rapid infusion OR reaction* OR safety) AND (2010/1: 2015/12 [dp])
Number of citations found? = 1119	⊠ > 100 citations – Go to 1G □ ≤ 100 citations – Review the citations
1G. Add the first initials of the co-authors to 1F	(Murillo J [au] OR Cox J [au] OR Baker K [au]) AND (rituximab OR rapid infusion OR reaction* OR safety) AND (2010/1: 2015/12 [dp])
Number of citations found? = 81	□ > 100 citations – Go to 1H ⊠ ≤ 100 citations – Review the citations
1H. Replace the keywords used in 1G with the consensus keyword	(Murillo J [au] OR Cox J [au] OR Baker K [au]) AND (rituximab) AND (2010/1: 2015/12 [dp])
Publication found?	□ Yes – END search here ⊠ No – Proceed to EMBASE search
	⇩
EMBASE Search	Search Terms/Results
2A. ‘Last name of the primary author’ followed by “:au”	(‘Swan’:au) AND ([2010-2015]/py)
Number of citations found? = 719	⊠ > 25 citations – Go to 2B □ ≤ 25 citations – Review the citations
2B. Add keywords separated by “OR” to 2A using “AND”	(‘Swan’:au) AND ([2010-2015]/py) AND (rituximab OR rapid infusion OR reaction* OR safety)
Number of citations found? = 243	⊠ > 100 citations – Go to 2C □ ≤ 100 citations – Review the citations
2C. Add the first initial of the primary author to 2B	(‘Swan J’:au) AND ([2010-2015]/py) AND (rituximab OR rapid infusion OR reaction* OR safety)
Number of citations found? = 30	□ > 100 citations – Go to 2D ⊠ ≤ 100 citations – Review the citations
2D. Replace the keywords used in 2C with the consensus keyword	(‘Swan J’:au) AND ([2010-2015]/py) AND (rituximab)
Publication found?	⊠ Yes – END search here □ No – Go to 2E
2E. ‘Last names of the co-authors’ followed by “:au”	(‘Murillo’:au OR ‘Cox’:au OR ‘Baker’:au) AND ([2010-2015]/py)
Number of citations found? = N/A	□ > 25 citations – Go to 2F □ ≤ 25 citations – Review the citations
2F. Add keywords separated by “OR” to 2E using “AND”	(‘Murillo’:au OR ‘Cox’:au OR ‘Baker’:au) AND ([2010-2015]/py) AND (rituximab OR rapid infusion OR reaction* OR safety)
Number of citations found? = N/A	□ > 100 citations – Go to 2G □ ≤ 100 citations – Review the citations
2G. Add the first initials of the co-authors to 2F	(‘Murillo J’:au OR ‘Cox J’:au OR ‘Baker K’:au) AND ([2010-2015]/py) AND (rituximab OR rapid infusion OR reaction* OR safety)
Number of citations found? = N/A	□ > 100 citations – Go to 2H □ ≤ 100 citations – Review the citations
2H. Replace the keywords used in 2G with the consensus keyword	(‘Murillo J’:au OR ‘Cox J’:au OR ‘Baker K’:au) AND ([2010-2015]/py) AND (rituximab)
Publication found?	□ Yes – END search □ No – END search
Final Decision
□ NOT published	⊠ Published - Full manuscript

Abbreviations: [au] and “:au', author search field tag; [dp] and “py', publication date search field tag.

Abstract Title: Assessment of safety regarding a rapid rituximab infusion

Primary Author (Last name, First name, Graduation year): Swan, Joshua T, 2010

Co-authors (Last name, First name): Murillo, Jose R; Cox, James E; Baker, Kelty R

Figure 1.

Systematic search strategy.

Since EMBASE is only accessible via a paid subscription, the investigators first queried PubMed to determine what proportion of publications could be identified in this free database. Initially, last name of the primary author alone was searched in PubMed (search 1). If this search yielded more than 25 citations, abstract keywords were added (search 2). If more than 100 citations were returned using the combination of the last name and keywords, the 1-letter initial of the author’s first name was added (search 3). If more than 100 citations were returned, the keywords were replaced with 1 consensus keyword, which was predetermined by investigators during keyword extraction (search 4). If a publication was not identified using the name of the primary author, then the processes used in searches 1 through 4 were repeated by substituting the name of the primary author with the names of all co-authors (searches 5 through 8). If no publication was found in PubMed, then searches 1 through 8 were repeated in EMBASE, as EMBASE has a wider journal and subject coverage.¹⁰ Full-text articles identified through the systematic search were reviewed to confirm a positive match with the content of the abstract. Two investigators blinded to survey results independently conducted database searches in February 2014, and a third investigator settled discrepancies if necessary. This search strategy was completed for all residency projects that were completed during the study period, regardless of whether or not a corresponding survey was completed by the residency graduate.

Survey Development and Deployment

Validity of this search strategy was assessed by obtaining information on publications through an online survey deployed to residency graduates; respondents were asked to answer whether or not their residency projects were published (“Was your residency major project published in a peer-reviewed journal?”). General demographic information was also obtained through the survey for internal records. Results from the online search for projects with no corresponding survey response were excluded. The online survey was developed using Survey Monkey (Survey Monkey Corporation, Palo Alto, California) by the primary author in February 2014. The fee necessary to use the service was paid by the department of pharmacy at the study institution. The investigator team contacted each residency graduate via an e-mail containing a hyperlink to the survey; up to 4 automated reminder e-mails were sent, in addition to contacts made via social media and phone, to maximize participation over the course of 6 weeks. E-mail addresses for residency graduates were obtained through personal contacts and internal records within the institution’s department of pharmacy.

Statistical Analysis

Data analyses were conducted using STATA 13 (StataCorp LP, College Station, Texas) and GraphPad (GraphPad Software, La Jolla, California). A sample size of 80 projects was estimated to provide 80% power to detect an absolute difference of 18% between methods (χ² test with a 2-sided α = .05). It was estimated that the proportion of projects identified as published would be 12% for the search strategy (based on previous estimates^7,8 of 6%-16%) and 30% via direct survey of residency graduates (based on personal experience of the investigators). It is possible that an absolute difference less than 18% could be meaningful, and differences as small as 5% to 10% may be important for preceptors or future residents who are trying to evaluate the proportion of projects from a residency program, or group of residency programs, that achieve publication.

The McNemar’s test was used to compare the proportions of projects that were identified as published by each of the 2 methods. The difference between these proportions will be accompanied by a 95% confidence interval (CI). A 2-sided α = .05 was used to designate statistical significance. Interinvestigator agreement using the search strategy was assessed by calculating Cohen’s κ. Sensitivity, specificity, positive predictive value, negative predictive value, and accuracy of the search strategy were calculated to describe its performance characteristics. Accuracy was calculated by dividing the number of correct assessments by that of all assessments. Sensitivity analyses were conducted to estimate the influence of survey nonresponders, since the projects of survey nonresponders were excluded from the primary analysis. In scenario 1, the count of publications in these excluded projects was estimated to be exactly the same as the proportion of publications identified through the search strategy for that group. In scenario 2, the proportion of publication in these excluded projects was estimated to be 30%.

Results

Over the 12-year period, 74 major research projects were completed by 61 pharmacy residency graduates. Thirteen residents completed both PGY1 and PGY2 residencies at the study institution. Online surveys were sent to 60 residency graduates as contact information for 1 graduate could not be located. Surveys were completed by 53 of the 61 residents that accounted for 66 (89%) of 74 projects, with 14 (21%) of 66 projects reported as published (Table 3 and Figure 2). Of these 66 projects included in the primary analysis, residents self-reported publishing 29.2% (7 of 24) of PGY2 residency projects and 16.7% (7 of 42) of PGY1 projects. The search strategy correctly identified 13 of these 14 publications (93% sensitivity, 100% specificity, and 98% accuracy). Of the 13 published manuscripts that were identified using the systematic search strategy, 12 (92%) were identified in PubMed (3 manuscripts at step 1B and 9 manuscripts at step 1D) and 1 was identified in EMBASE (step 2F). The difference in the proportion of projects reported as published was not statistically different between the methods (19.7% search strategy vs 21.2% survey, difference of −1.5% [95% CI, −5.9% to 2.9%], P = 1.00). The reliability of this search strategy was supported by the perfect agreement between the 2 investigators (k = 1.00). Two articles accepted for publication, but not published at the time of the search, were identified via survey. Both of these 2 projects were completed by PGY2 residents who graduated in July 2012.

Figure 2.

Flow chart of included residency projects.

Table 3.

Characteristics of the Published Residency Projects.

Characteristic of publication	Data (n = 14)
Months to publication^a, median (range)	11 (4-19)
Resident listed as primary author, No. (%)	14 (100)
Number of authors^b, median (range)	5 (2-8)
Projects with nonpharmacy co-authors, No. (%)^c	12 (86)
Journal characteristics of published manuscripts
Impact factor of the journal, median (range)^d	2.65 (0.95-7.92)
Indexed in online database, No. (%)
PubMed	13 (93)
EMBASE	14 (100)
Types of journals, No. (%)
Pharmacy journal^e	1 (7)
Medical journal	13 (93)
Transplant focus	5 (36)
Oncology focus	3 (21)
Cardiovascular/critical care focus	2 (14)
Surgery focus	2 (14)
Pathology focus	1 (7)

^aThe months to publication was calculated starting with the month of graduation from the residency program until the date of publication, either in print or online (whichever was first).

^bNumber of authors includes the pharmacy resident author.

^cAll listed nonpharmacy co-authors were physicians.

^dImpact factors were not available for journals that published 2 of these manuscripts (Community Oncology and Journal of Oncology Pharmacy Practice).

^eAmerican Journal of Health-System Pharmacy was the only pharmacy journal.

Of the 14 publications, 13 (93%) were indexed in both PubMed and EMBASE. Only 1 project was indexed in EMBASE but not in PubMed. All manuscripts, except for one that was published in a pharmacy journal, were published in various medical journals focusing on different disease topics. Although investigators thought that the last name of the primary authors may have changed due to changes in marital status, no changes in the first or last names from time of conference abstract listing were found among the residents whose projects were published.

This study enrolled 66 residency projects, which was less than the estimated sample size of 80. Assuming the original proportions (12% vs 30%), a sample size of 66 had a power of 70%. The sample size was recalculated post hoc where estimated probability of publication of 30% was reduced to 21.2% based on the survey results, and 269 projects would be needed to provide 80% power (12% vs 21.2%, 2-sided α = .05).

Sensitivity Analyses

Of the 60 residency graduates, 8 (13%) accounting for 8 projects did not complete the survey and were excluded from the primary analysis. The search strategy did not identify any publications resulting from these 8 projects. In sensitivity analysis scenario 1, it was estimated that none of the projects were published, since none were found with the search strategy. In scenario 2, it was estimated that 30% (2 of 8) of these projects were published. The difference in the proportion of projects published as identified by either the search strategy or survey was not statistically different in either scenario 1 (17.6% search strategy vs 18.9% survey, difference of −1.4% [95% CI, −5.3% to 2.6%], P = 1.00) or scenario 2 (17.6% search strategy vs 21.6% survey, difference of −4.1% [95% CI, −9.9% to 1.8%], P = .25). Both sensitivity analyses support the primary analysis.

Discussion

To our knowledge, this is the first study to evaluate the performance characteristics of a search strategy that can be used to identify published manuscripts resulting from pharmacy residency research projects. This systematic search strategy appears to be a valid and reliable tool for identifying the proportion of publications resulting from pharmacy residency projects using information that is available in the research abstracts of local and regional residency conference booklets. The utility of this search strategy is not limited to residents or the discipline of pharmacy and can be generalized to identify publications that arise from research abstracts presented at many types of medical conferences.

Strengths of the Present Study

When compared to the search methods used in previous studies, this systematic search strategy has many features that contributed to its favorable performance characteristics. The use of a 5-year time limit from the date of graduation and author search field tag reduced the number of irrelevant citations that were returned with each query. Keywords were carefully extracted and refined from each abstract by 3 investigators; the investigators believe this was critical for the success and reliability of this systematic search strategy. Abstracts often contained complex phrases, and truncation to the root word was a useful technique that maximized sensitivity and reduced the number of irrelevant citations returned. The use of single keyword within the search strategy helped maximize specificity.

The investigators used widely available software to build the data collection tool. By copying and pasting the automatically generated text strings, transcription errors were prevented. Examples of both the Microsoft Word document data collection tool and accompanying Microsoft Excel database are available as online supplementary files. Finally, the survey received a high response rate (89%) from the reference population that allowed for a meaningful comparison between the methods and minimized nonresponder bias. The search strategy validated in this study involves only using information from conference abstract booklets.

Comparison to Previous Studies

The proportion of residency research projects that result in published manuscripts has previously been reported (Table 1). McKelvey et al developed a tool in which 1 investigator searched MEDLINE and International Pharmaceutical Abstracts (IPA) online databases to identify project publications following their presentation at the Southeastern Residency Conference held in 1981, 1991, and 2001. The authors reported that 15.8% of the residency projects presented at the conference led to publications in peer-reviewed journals.⁷ Similarly, Olson et al conducted a cross-sectional study in which 2 investigators evaluated abstracts presented at the Western States Conference in 1995, 2000, and 2005 that identified publications for 6.3% of residency research projects using MEDLINE and EMBASE.⁸ Through a single database search on PubMed, O’Dell et al reported that 4.3% of projects presented at the 25th Western States Conference resulted in publication as full-text articles.⁹

The proportion of publications in the present study is notably higher using the systematic search strategy (20%) compared to what has previously been reported. Given that the previous studies by McKelvey et al and Olson et al included other types of publications, such as letters to the editor and abstracts, this discrepancy could potentially be larger as the current study only reported the proportion of published, full-text original research articles.^7,8 This difference in proportions may also be explained by the high sensitivity and specificity of the search strategy leading to an increased number of identified publications, as compared to the search strategies used in previous studies, which were not validated and may have underidentified publications. Another potential explanation is that this could simply be a characteristic of the residency graduates from the study institution. In addition, the median time to publication of 11 months found in this study was shorter than the median time to publication of 24 months reported in previous studies.^7,8 Authors speculate that this difference could be due to the strong preceptor support available at the institution even after completion of a residency.

Limitations of the Present Study

This study included 66 projects, which was less than the estimated sample size of 80 projects to detect an absolute difference of 18% between the methods. Although the residency was established in the 1980s at the study institution, the investigators were not able to increase the sample size by including residents prior to 2001 due to a loss of residency files during a natural disaster. The publication characteristics (Table 3) may not be generalizable to other residency programs due to spectrum bias, as characteristics of the residency graduates and their projects at this single study institution may differ from residency graduates across the country. Of the 74 projects, 8 conducted at this institution were excluded from the primary analysis; however, results of the sensitivity analyses indicate that these excluded projects did not impact the results of this study.

As major projects are not always presented at regional residency conferences, using the search strategy developed in the present study may underestimate the number of publications resulting from pharmacy residency major research projects. In the current study, 1 publication resulting from a major research project that was identified via survey was not identified by the search strategy because the resident presented the minor project (which was not published) rather than the major project (which was published) at the regional conference. As the median time to publication after presentation at a biomedical conference ranges up to 36 months, research projects conducted by recent graduates that have been accepted for publication but not yet published may not be identified using the search strategy.¹¹ Thus, the risk of underestimating the proportion of projects achieving eventual publication exists when the search strategy is used to identify publications by residents who have graduated in the past 3 years.

An online survey of residency graduates as the reference standard could have carried risks for social desirability (residents over-reporting publications) and information recall biases (residents under-reporting publications). As information obtained regarding publication via survey was compared with the results using the search strategy, these biases were minimized. Although the research protocol was built to ensure blinding, due to the study setting, 2 of the investigators were involved as investigators in 4 residency projects under review.

Impact of the Present Study and Future Directions

Knowledge of the proportion and characteristics of publications resulting from pharmacy residency projects is of interest to residency programs and directors. However, it may be difficult to directly survey previous residency graduates due to limited resources available (eg, time, human, and/or monetary resources) and other hindrances such as loss of contact with residency graduates over time. For instance, the investigators of the present study spent 4 months obtaining the current contact information of the 60 residency graduates and 2 months administering the survey. The systematic search strategy developed and validated in this study may serve as a time- and cost-effective surrogate for direct survey as conference abstract booklets were either readily available within the department or could be easily obtained by contacting the conference organization committee. However, direct survey of residency graduates could be a more effective strategy to identify published projects for a small cohort of recent residency graduates.

Characterization of residency research project outcomes using the search strategy may also lead to funding for new residency programs or expansion of existing programs as residency projects often result in direct benefits (eg, protocol development) for health systems. Findings of such research may allow for comparisons among pharmacy residency programs, and this information would help the health-system pharmacy community identify programs with highly effective research support. These programs may offer advice to the community on improving the quality of residency research projects across the country. Recommendations for future research directions include application of the systematic search strategy to accurately estimate proportion of published projects within a large cohort of pharmacy residents. As a great deal of resources is utilized for completion of such projects, continuous efforts should be made by pharmacy residency programs to encourage their residents to share the findings of their projects with the medical community.⁵

Although the present study utilized EMBASE as a supplementary search engine to identify articles published in journals not indexed in PubMed, other search engines such as IPA and Google Scholar may be used instead in cases where EMBASE subscription is not readily available. The authors included abstract booklets from the local and regional residency conferences that were routinely attended by the residents at the study institution; however, it seems reasonable to assume that our methods could be extended to use with abstracts from other conferences as most abstract booklets provide the information needed (authors, titles, and abstract text) to carry out this systematic search strategy.

Conclusions

This systematic search strategy demonstrated a high sensitivity, specificity, and accuracy for identifying publications that resulted from pharmacy residency research projects using information available in residency conference abstracts.

Footnotes

Acknowledgments

The authors would like to thank Kyle Dawson, PharmD, BCPS, for providing a thoughtful review of the manuscript prior to submission. Thomas Schwartz, PharmD, MBA, and Alex Varkey, PharmD, MS, assisted with obtaining the roster and contact information of the residency graduates. Investigators would like to thank the Department of Pharmacy Services at Houston Methodist Hospital for providing the computer resources and access to Survey Monkey (Survey Monkey Corporation, Palo Alto, California) needed to complete this study.

Author Contributions

NK and JTS developed the research hypothesis and study design. NK, JTS, and NTM developed and refined the survey questions, which were edited by MGL. NK managed data collection and analysis, and JTS assisted with both. NK and NTM conducted the case abstract search strategies, and JTS served as the third investigator to settle discrepancies. NK authored the initial draft of the manuscript. All authors revised and approved the final version of this manuscript.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Supplemental Material

The online data supplements are available at

References

Glossary of Terms. American society of health-system pharmacists Web site. Web site. http://www.ashp.org/menu/Accreditation/ResidencyAccreditation/Glossary.aspx. Accessed December 31, 2014.

Johnson

Teeters

. Pharmacy residency and the medical training model: is pharmacy at a tipping point? Am J Health Syst Pharm. 2011;68(16):1542–1549.

Pharmacy residency match day: numbers continue to increase. American society of health-system pharmacists Web site. Web site. http://www.ashp.org/menu/News/NewsCapsules/Article.aspx?id=531. Updated March 21, 2014. Accessed on May 24, 2014.

American society of health-system pharmacists. Required and elective educational outcomes, goals, objectives, and instructional objectives for postgraduate year one (PGY1) pharmacy residency programs. Web site. http://www.ashp.org/menu/Accreditation/ResidencyAccreditation. Published May 17, 2007. Accessed December 31, 2014.

Irwin

Olson

Joline

. Challenges to publishing pharmacy resident research projects from the perspectives of residency program directors and residents. Pharm Pract (Granada). 2013;11(3):166–172.

Murphy

Downhour

. Perceived value and outcomes of residency projects. Am J Health Syst Pharm. 2001;58(10):889–895.

McKelvey

Hatton

Kimberlin

. Pharmacy resident project publication rates and study designs from 1981, 1991, and 2001. Am J Health Syst Pharm. 2010;67(10):830–836.

Olson

Holmes

Dang

. Publication rates of abstracts presented by pharmacy residents at the western states conference. Am J Health Syst Pharm. 2012;69(1):59–62.

O’Dell

Shah

. Evaluation of pharmacy practice residents’ research abstracts and publication rate. J Am Pharm Assoc. 2012;52(4):524–527.

10.

Wilkins

Gillies

Davies

. EMBASE versus MEDLINE for family medicine searches: can MEDLINE searches find the forest or a tree? Can Fam Physician. 2005;51:848–849.

11.

Scherer

Langenberg

von Elm

. Full publication of results initially presented in abstracts. Cochrane Database Syst Rev. 2007;(2):MR000005.