Abstract
Purpose
We reviewed all 64 articles ever published by The Congenital Heart Surgeons’ Society (CHSS) Data Center to estimate the academic impact of these peer-reviewed articles.
Materials and Methods
The Congenital Heart Surgeons’ Society has performed research based on 12 Diagnostic Inception Cohorts. The first cohort (Transposition) began enrolling patients on January 1, 1985. We queried PubMed to determine the number of publications that referenced each of the 64 journal articles generated by the datasets of the 12 Diagnostic Inception Cohorts that comprise the CHSS Database. Descriptive summaries of the data were tabulated using mean with standard deviation and median with range.
Results
Sixty-four peer-reviewed papers have been published based on the CHSS Database. Fifty-nine peer-reviewed articles have been published based on the 12 Diagnostic Inception Cohorts, and five additional articles have been published based on Data Science. Excluding the recently established Diagnostic Inception Cohort for patients with Ebstein malformation of tricuspid valve, the number of papers published per cohort ranged from 1 for coarctation to 11 for transposition of the great arteries. The 11 articles generated from the CHSS Transposition Cohort were referenced by a total of 111 articles (median number of references per journal article = 9 [range = 0-22, mean = 10.1]). Overall, individual articles were cited by an average of 11 (mean), and a maximum of 41 PubMed-listed publications. Overall, these 64 peer-reviewed articles based on the CHSS Database were cited 692 times in PubMed-listed publications. The first CHSS peer-reviewed article was published in 1987, and during the 35 years from 1987 to 2022, inclusive, the annual number of CHSS publications has ranged from 0 to 7, with a mean of 1.8 publications per year (median = 1, mode = 1).
Conclusion
Congenital Heart Surgeons’ Society studies are widely referenced in the pediatric cardiac surgical literature, with over 10 citations per published article. These cohorts provide unique information unavailable in other sources of data. A tool to access this analysis is available at: [https://data-center.chss.org/multimedia/files/2022/CAI.pdf].
Keywords
Introduction
The Congenital Heart Surgeons’ Society (CHSS) [www.chss.org] is the premier professional society in North America dedicated to pediatric and congenital cardiac surgery. According to the CHSS bylaws [https://chss.org/multimedia/files/CHSS-Bylaws.pdf], the purpose of CHSS is:
To associate persons interested in, and carry on activities related to, the science and practice of congenital heart surgery. To sponsor and oversee multiinstitutional clinical studies evaluating the application of surgical interventions in congenital heart disease. To encourage and stimulate investigation and study that will increase the knowledge of congenital cardiac physiology, pathology, and therapy, and to correlate and disseminate such knowledge. To hold scientific meetings featuring free discussion of problems and developments relating to congenital heart surgery. To succeed to and continue to carry on the activities formerly conducted by The CHSS, an unincorporated association.
The CHSS was formed in 1972. In 1984, Drs John W. Kirklin and Eugene H. Blackstone proposed that the CHSS surgeons pool their experience in managing infants with rare congenital anomalies of the heart, a concept that led to the establishment of the CHSS Data Center. Since then, the CHSS Data Center has conducted research that now includes 12 Diagnostic Inception Cohorts that have been progressively accumulated over a 39-year time frame (1984-2023). Inclusion of the word “inception” is important because “inception” is an important key concept. Enrollment is triggered by a specified event—admission to the hospital—with only one exception (ie, The CHSS Pulmonary conduit [PC] cohort where the specified event is defined as discharge from the hospital following implant of a pulmonary valve). Consequently, CHSS studies report the outcomes of patients as opposed to operations. This concept sets CHSS studies apart from much of the pediatric and congenital cardiac surgical literature. The first CHSS cohort, neonates with transposition of the great arteries (TGA), began enrolling patients on January 1, 1985. The purpose of this current analysis and article is to estimate the academic impact of CHSS research by reviewing all 64 articles ever published by the CHSS Data Center1-64 and assessing the citations generated by these CHSS publications.
Materials and Methods
Congenital Heart Surgeons’ Society Cohorts
The Congenital Heart Surgeons’ Society has performed research based on 12 Diagnostic Inception Cohorts:
Pulmonary atresia with Intact Ventricular Septum5,19,36 & Pulmonary stenosis with Intact Ventricular Septum
6
Coarctation of Aorta
8
Critical Left Ventricular Outflow Tract Obstruction (LVOTO)25,29,45-48,50-52,55 Anomalous Aortic Origin of a Coronary Artery (AAOCA)34,39,42,44,49,56,61 Ebstein Anomaly (Ebstein malformation of tricuspid valve)
A complete listing of the citations for these 64 publications, stratified by Diagnostic Inception Cohorts, is available at the CHSS website at the following hyperlink: [https://data-center.chss.org/research/publications.cgi]. Moreover, a tool to access the detailed information about these 64 publications obtained via the analysis reported in this article is available at: [https://data-center.chss.org/multimedia/files/2022/CAI.pdf].
Table 1 provides the following data about these 12 cohorts:
Cohort Name (Study Name) Cohort Abbreviation (Abbreviation) Number of Institutions That Have Enrolled Patients (Institutions) Years of Enrollment (Accrual) Number of Patients Enrolled (Enrolled) Number of Articles Generated by Cohort (Articles)
Twelve Congenital Heart Surgeons' Society (CHSS) Cohorts.
Abbreviations: AAOCA, anomalous aortic origin of a coronary artery; AVA, aortic valve atresia; AVS, aortic valve stenosis; AVSD, atrioventricular septal defect; CHSS, Congenital Heart Surgeons’ Society; CoA, coarctation of aorta; EBS, Ebstein malformation of tricuspid valve; IAA, interrupted aortic arch; LVOTO, left ventricular outflow tract obstruction; PA-IVS, Pulmonary atresia with intact ventricular septum; PC, pulmonary conduit; PS-IVS, Pulmonary stenosis with intact ventricular septum; TA, tricuspid atresia; TGA, transposition of the great arteries. * = Enrollment in the Critical LVOTO cohort is currently on hold and has been on hold since 2019.
Sources: (1) CHSS Data Center: Clinical Studies: Current Study Protocol [https://data-center.chss.org/research/clinical-studies.cgi]. Accessed September 10, 2022. (2) CHSS Data Center: Cohort Studies [https://data-center.chss.org/cohort-studies/]. Accessed September 10, 2022.
CHSS John W. Kirklin/David Ashburn Research Fellows
The CHSS Data Center offers a 2-to-3-year Fellowship to study data management and the statistical methods of data analysis [https://data-center.chss.org/fellowship/]. The Fellowship was initiated with the appointment of the first CHSS Kirklin Research Fellow, Dr David Ashburn, in 2001.
At the historic First Joint meeting of CHSS and The European Congenital Heart Surgeons Association (ECHSA) in Montreal in 2004, the host, Christo I. Tchervenkov, MD, donated $5,000 from the Montreal Children's Hospital to CHSS to establish an Endowment Fund in the memory of John W. Kirklin, who died earlier that year. The First Joint CHSS-ECHSA meeting was appropriately dedicated to the memory of John Webster Kirklin, MD (April 5, 1917-April 21, 2004) (Supplemental Appendix 1), and the donation from Dr Tchervenkov provided financial support to the previously established Kirklin Research Fellowship, creating the initial endowment for the CHSS Kirklin Research Fellowship. Supplemental Appendix 1 provides the original program from the historic First Joint meeting of CHSS and ECHSA in Montreal in 2004.
The first CHSS Kirklin Research Fellow, David Ashburn, MD, successfully completed his CHSS Research Fellowship and earned a Master of Science postgraduate degree from the University of Toronto. Sadly, on June 4, 2007, as Dr Ashburn was completing his education in congenital heart surgery at the University of Michigan in Ann Arbor, Michigan, he and five colleagues were killed in an airplane crash while on transplant retrieval. In honor of David Ashburn, MD and founding CHSS member John W. Kirklin, MD, the Fellowship of the CHSS Data Center was renamed the John W. Kirklin/David Ashburn Fellowship.
As described in more detail later in this article, a substantial portion of the research generated by the CHSS Data Center has been supported by the leadership and efforts of the CHSS John W. Kirklin/David Ashburn Research Fellows, who are listed below:
Dr David Ashburn (July 2001-June 2003) Dr Tara Karamlou (July 2004-June 2006) Dr Edward Hickey (July 2006-2008) Dr Mohammed Albanna Dr Anusha Jegatheeswaran (July 2008-June 2011) Dr Jeffrey Poynter (July 2011-June 2013) Dr Travis Wilder (July 2013-June 2015) Dr James Meza (July 2015-June 2017) Dr Paul Devlin (July 2017-June 2019) Dr Connor Callahan (July 2019-June 2021) Dr Madison Argo (July 2021-June 2023)
It is notable that CHSS has a unique model of academic productivity, which is currently designed around the inclusion of the Kirklin/Ashburn Research Fellows as the center of publication activity. This strategy and priority of CHSS is important because part of the “academic impact” of CHSS has been the education and training of a cadre of Kirklin/Ashburn Research Fellows. Thus, CHSS not only produces new knowledge from clinical analyses but also invests heavily in the training of future generations of leaders in the field.
Analytic Methods
We queried PubMed from its inception through September 10, 2022, to determine the number of publications that referenced each of the 64 papers generated by the CHSS Database. Descriptive summaries of the data were tabulated using mean with standard deviation (SD) and median with range.
We acknowledge that a variety of metrics can be used to estimate the academic impact of an article, and our use of the number of citations (ie, publications) in PubMed that referenced a given paper is one such estimate. Clearly, citation count represents just one component of “academic impact.” As described in the Discussion section of this current article, clearly other metrics exist to provide complementary estimates of the academic impact of a journal article, and the use of these additional metrics represents an opportunity for future investigation.
Furthermore, Kirklin/Ashburn Research Fellows have also written theses that are available on the University of Toronto website [https://tspace.library.utoronto.ca]. These theses reflect the topics of at least one of the publications from the time of the Kirklin/Ashburn Research Fellows at the CHSS Data Center, but these theses are still academic contributions from the CHSS Data Center and therefore representative of the additional academic impact of CHSS studies. Finally, our current analysis does not quantify the number of presentations at conferences, presentations during plenary sessions of conferences, and other awards associated with each cohort; however, these academic contributions from the CHSS Data Center are also representative of the additional academic impact of CHSS studies.
Results
Table 2 provides the following data about these 12 cohorts:
Number of Patients Enrolled (Patients) Number of Institutions That Have Enrolled Patients (Institutions) Years of Enrollment Number of Articles Generated by Cohort (Articles) Cumulative References Generated Median References Generated Per Article Mean References Generated Per Article Minimum References Generated Per Article Maximum References Generated Per Article
References generated from each Congenital Heart Surgeons’ Society (CHSS) Cohort.
Abbreviations: AAOCA, anomalous aortic origin of a coronary artery; AVA, aortic valve atresia; AVS, aortic valve stenosis; AVSD, atrioventricular septal defect; CHSS, Congenital Heart Surgeons’ Society; LVOTO, left ventricular outflow tract obstruction; PA-IVS, Pulmonary atresia with intact ventricular septum; PS-IVS, Pulmonary stenosis with intact ventricular septum; TGA, transposition of the great arteries.
Source: CHSS Data Center: Publications [https://data-center.chss.org/research/publications.cgi]. Accessed September 10, 2022.
Overall, 64 peer-reviewed papers have been published based on the CHSS Database:
59 peer-reviewed articles have been published based on the 12 Diagnostic Inception Cohorts,1-26,29,31-39,42-64 and 5 additional articles have been published based on Data Science.27,28,30,40,41
The 64 articles generated by the CHSS Data Center are listed in the references list of this current article in chronological order based on their dates of publication.1-64 The first cohort (TGA) began enrolling patients on January 1, 1985. Appendix 1 provides the original 1984 letter to members of CHSS from James W. Kirklin, MD and Eugene H. Blackstone, MD that proposes this very first CHSS study. The relevance of the TGA study is notable even in the current era. At the time of the initiation of the CHSS TGA study (1985), there was considerable angst regarding whether one could justify a new operation with unknown but initially, very high mortality (ie, the arterial switch in 1984) when the low-risk atrial repairs were the standard of care. This notable and remarkable clinical uncertainty was instrumental in initiating this cohort in 1985 and in achieving participation from 100% of the institutions with CHSS members. According to the Senior author of this article (WGW), the CHSS Data Center has “not seen that convergence of interest/uncertainty and enthusiasm for other cohorts until the AAOCA study.”
Between 1985 and 1989, the CHSS TGA cohort enrolled 891 neonates from 24 institutions; this cohort included all neonates born with “complete TGA” (ie, concordant atrioventricular connections and discordant ventriculo-arterial connections) who were admitted to any of the CHSS institutions within the first two weeks of life. This cohort subsequently generated 11 journal articles which resulted in 111 cumulative citations (median number of citations per article = 9 [range = 0-22, mean = 10.1]). The first published paper generated from a CHSS Cohort was titled: “Current results of management in TGA, with special emphasis on patients with associated ventricular septal defect” and was published in the Journal of the American College of Cardiology in November 1987. 1 This initial paper has been cited four times as of September 10, 2022, and was followed four months later, in January 1988, by a publication in The Journal of Thoracic and Cardiovascular Surgery (JTCVS) titled “The early results of treatment of simple transposition in the current era” that has been cited 11 times as of September 10, 2022. 2 The most highly referenced of the 11 publications generated from the CHSS TGA cohort is titled: “Clinical outcomes after the arterial switch operation for transposition. Patient, support, procedural, and institutional risk factors. Congenital Heart Surgeons' Society” 4 and was published in 1992 in Circulation; this paper has been cited 23 times as of January 28, 2023, with the first citation in 2003 and the most recent citation in 2022 (Figure 1).

The landmark publication written by Kirklin and colleagues 4 titled: “Clinical outcomes after the arterial switch operation for transposition. Patient, support, procedural, and institutional risk factors” was published in 1992 in Circulation and has been cited 23 times as of January 28, 2023, with the first citation in 1996 and the most recent citation in 2021. This graph plots the number of citations per year for this paper, which is the most highly cited article of all 11 publications generated from the Congenital Heart Surgeons’ Society (CHSS) transposition of the great arteries (TGA) cohort by the CHSS Data Center. The x-axis is calendar year and the y-axis is citations per year.
The success of data collection in the transposition babies led to 11 subsequent prospective observational studies of neonates, infants, children, and adults with congenital heart disease, and these 12 CHSS cohorts have generated multiple peer-reviewed publications1-26,29,31-39,42-64 (Table 1). Seven of these studies are no longer actively enrolling patients, and five of these studies are still actively enrolling patients (Table 1). However, all but one cohort (Coarctation) continue to have annual cross-sectional follow-up. The five CHSS studies that are still actively enrolling patients are Tricuspid Atresia, Critical Left Ventricular Outflow Tract Obstruction (LVOTO), Anomalous Aortic Origin of a Coronary Artery (AAOCA), Atrioventricular Septal Defect (AVSD), and Ebstein malformation of tricuspid valve, although enrollment in the Critical LVOTO cohort is currently on hold (and has been on hold since 2019).
Overall, individual papers in this list of 64 publications were cited by an average of 11 (mean), and a maximum of 41 PubMed-listed publications. The article written by David Ashburn, MD, titled: “Outcomes after the Norwood operation in neonates with critical aortic stenosis or aortic valve atresia,” 16 was published in May 2003 in JTCVS and has been cited 41 times as of September 10, 2022, with the first citation in 2003 and the most recent citation in 2022 (Figure 2).

The landmark publication written by Ashburn and colleagues 16 titled: “Outcomes after the Norwood operation in neonates with critical aortic stenosis or aortic valve atresia” was published in May 2003 in Journal of Thoracic and Cardiovascular Surgery (JTCVS) and has been cited 41 times as of January 28, 2023, with the first citation in 2005 and the most recent citation in 2022. This graph plots the number of citations per year for this paper, which is the most highly cited article of all 64 publications generated by the Congenital Heart Surgeons’ Society (CHSS) Data Center. The x-axis is calendar year, and the y-axis is citations per year.
Excluding the recently established Diagnostic Inception Cohort for patients with Ebstein malformation of tricuspid valve, the number of papers published per cohort ranged from 1 for coarctation 8 to 11 for TGA1-4,9,11,13,17,18,57,58 (Table 2). Overall, the 64 peer-reviewed articles based on the CHSS Database were cited 692 times in PubMed-listed publications as of September 10, 2022. The first CHSS peer-reviewed article was published in 1987, and during the 35 years from 1987 to 2022, inclusive, the annual number of CHSS publications has ranged from 0 to 7, with a mean of 1.8 publications per year (median = 1, mode = 1) (Figure 3). The mean number of publications during the first 15 years (1987-2001, inclusive) was 1 (median = 1, mode = 1, range = 0-2), increasing to a mean number of publications of 2.3 (median = 2, mode = 0, range = 0-7) during the subsequent 21 years (2002-2022, inclusive), after the initiation of the CHSS John W. Kirklin/David Ashburn Research Fellowship in 2001 (Figure 3).

The number of publications per year generated by the Congenital Heart Surgeons’ Society (CHSS) Database. The x-axis is calendar year and the y-axis is publications per year. The black arrow indicates the time that the first CHSS John W. Kirklin/David Ashburn Research Fellow began his Fellowship in Toronto. The first CHSS peer-reviewed paper was published in 1987, and during the 35 years from 1987 to 2022, inclusive, the annual number of CHSS publications has ranged from 0 to 7, with a mean of 1.8 publications per year (median = 1, mode = 1). The mean number of publications during the first 15 years (1987-2001, inclusive) was 1 (median = 1, mode = 1, range = 0-2), increasing to 2.3 (median = 2, mode = 0, range = 0-7) during the subsequent 21 years (2002-2022, inclusive) after the initiation of the CHSS Kirklin/Ashburn Research Fellowship in 2001.
Discussion
The CHSS was formed in 1972 on the suggestion of Eoin Aberdeen, then at Children's Hospital of Philadelphia, to include surgeons of compatible character with a special interest in pediatric cardiac surgery.40,41,65,66 According to the Senior author of this article (WGW), one of the criteria for selection to join the original group was that the potential CHSS member was not judged to be a poor listener. Thus, the CHSS was to be a small group of surgeons, patterned after the European Cardiac Surgeon's Club, with an initial maximum membership set at 16. The initial letter of invitation was sent by Eoin Aberdeen to Phil Ashmore, Doug Behrendt, Aldo Castaneda, George Daicoff, Anthony Dobell, Henry Edmunds, John W. Kirklin, James Malm, Dwight McGoon, Robert Replogle, Albert Starr, and George Trusler. These surgeons were to meet to relate their pioneering operative experiences with complex congenital cardiac defects in a friendly, open, and uninhibited atmosphere. Of those invited, ten surgeons attended the first meeting, which took place on September 7, 1973, at the Sonesta Beach Hotel, near Miami. The name of this club was discussed, and George Daicoff suggested “Small Heart Club,” while Eoin Aberdeen suggested “Pediatric Cardiovascular Surgery Club.” The group, however, opted for “Congenital Heart Surgeons’ Society.”
From the time of its establishment, a basic premise motivating the establishment of the CHSS Data Center and the associated research cohorts has been the idea that in order to answer fundamental questions about the management of complex and relatively uncommon forms of congenital heart disease, it is necessary to pool information from the experiences of multiple centers and systematically collect patient information over a period of decades. In 1984, Drs John W. Kirklin and Eugene H. Blackstone proposed that the CHSS surgeons pool their experience in managing infants with rare congenital anomalies of the heart (Appendix 1), a concept that led to the establishment of the CHSS Data Center. Drs Kirklin and Blackstone recognized that the occurrence of congenital heart disease is low (ie, 8/1000 live births) and that any single institution requires a great deal of time to learn from their experience and improve the management of patients with congenital heart disease. By pooling the experience of all the CHSS members, Drs Kirklin and Blackstone proposed that CHSS surgeons could improve their ability to determine the best methods of treating patients with pediatric and congenital heart disease.27,28,30
The first group of patients studied included all neonates born with complete TGA who were admitted to any of the CHSS institutions within the first two weeks of life. During the four years of data collection (1985-1989), information on more than 891 babies with TGA was collected. The first study cohort of neonates born with complete TGA established an important component of CHSS research protocols; namely, longitudinal cross-sectional follow-up of the entire cohort each year.57,58 The data analyses of these patients in the initial CHSS cohort have resulted in a wealth of information that is contained in eleven publications to date.1-4,9,11,13,17,18,57,58
The most highly referenced of the 11 publications generated from the CHSS TGA cohort is the landmark publication written by John W. Kirklin, MD, Eugene H. Blackstone, MD, Christo I. Tchervenkov, MD, and Aldo R. Castaneda, MD, titled: “Clinical outcomes after the arterial switch operation for transposition. Patient, support, procedural, and institutional risk factors. Congenital Heart Surgeons' Society” 4 which was published in 1992 and has been cited 23 times as of January 28, 2023, with the first citation in 2003 and the most recent citation in 2022. This CHSS multiinstitutional prospective study with annual detailed follow-up included 513 neonates with simple transposition or transposition and ventricular septal defect entering for diagnosis and treatment at <15 days of age and undergoing an arterial switch repair. 4 The authors reported the following results: “The 1-month and 1- and 5-year survivals were 84%, 82%, and 82%, respectively. The hazard function for death had a rapidly declining single phase that approached zero by 12 months after surgery. Among the eight patients who died ≥ 3 months after the operation, four had severe ventricular dysfunction, probably related to imperfect coronary arterial transfer. Coexisting single ventricular septal defect was not a risk factor for death. Origin of the left main coronary artery or only the left anterior descending or the circumflex artery from the right posterior sinus (sinus 2) was a risk factor that was even stronger when an intramural course was present; multiplicity of ventricular septal defects was a risk factor. Longer global myocardial ischemic time and total circulatory arrest time were risk factors. Certain institutions were shown to be risk factors for death; the results in some improved with increasing experience, in some they did not, and in some they worsened.” This analysis concluded that “Good early and intermediate-term clinical outcomes can be obtained in neonates with simple transposition and transposition and ventricular septal defect by use of the arterial switch operation. Certain coronary artery patterns and certain institutions lessen the goodness of outcome.” Figure 1 is a graph that plots the number of citations per year for this paper. 4
Data collection required the establishment of a Data Center, initially at the University of Alabama at Birmingham in Birmingham, Alabama. In 1997, the Data Center moved from Birmingham, Alabama to the Hospital for Sick Children in Toronto, Canada. In 2022, the Data Center evolved to become The CHSS Center for Research and Quality, hosted at two sites:
Cleveland Clinic, Cleveland, OH, USA Hospital for Sick Children, Toronto, Canada
Patients in the earliest established CHSS cohorts (eg, TGA57,58) continue to be contacted annually, more than 38 years after enrollment for the earliest patients enrolled. Data related to functional status and health-related quality of life are collected, in addition to clinical data pertaining to interventions, medical management, and vital status. In fact, similar to the initial CHSS TGA cohort, patients in all but one CHSS cohort (Coarctation) continue to have annual cross-sectional follow-up.53,57,58
The senior author of this current article (WGW) previously offered the following definitions40,41,67: ‘‘A database is simply a structured collection of information. A clinical database may be a Registry (a limited amount of data for
Based on these definitions, The Society of Thoracic Surgeons Congenital Heart Surgery Database is a Registry that collects “some of the data about all of the patients” undergoing pediatric and congenital cardiac surgery, while the CHSS Database is an Academic Database that collects “all of the data about some of the patients” undergoing pediatric and congenital cardiac surgery.
The CHSS Database has the following strengths:
The CHSS Database is designed to obtain truly long-term lifetime follow-up of enrolled patients, and the CHSS Data Center has the infrastructure to maintain this follow-up.53,57,58 The ability to provide lifetime follow-up of enrolled patients is because the CHSS itself as a professional Society provides the funding for lifelong follow-up. This unique funding arrangement sets CHSS studies apart from short-term research grant funding, as discussed below. A fundamental role of CHSS full-time staff is to maintain contact with patients enrolled in CHSS studies and their health care providers in order to achieve lifetime follow-up.53,57,58 This ability to perform lifetime follow-up makes these studies ideal for establishing temporal relationships for multiple outcomes of interest in these rare diseases. This strength distinguishes the CHSS Database from many other registries that include only short-term follow-up or mid-term follow-up. The CHSS Database and the CHSS Center for Research and Quality have state-of-the-art statistical capability. The CHSS Database has a track record of representative analyses across a wide spectrum of congenital cardiac malformations. The CHSS Database contains specific Diagnostic Inception Cohorts that are generally high-risk and represent small “slices” or subsets of the population of patients with congenital heart disease; however, these cohorts were specifically created to discern the best strategies of management to improve outcomes in these high-risk patients. The longitudinal prospective and systematic collection of data with protocols for standardized annual cross-sectional follow-up53,57,58 facilitates the generation of secondary hypotheses that can answer new research questions that emerge over the life span of these patients.
The major weakness of the CHSS Database is its voluntary enrollment and the knowledge that this method will not capture ALL eligible patients. Furthermore, although the CHSS includes members from the majority of pediatric and congenital cardiac surgical programs in the United States of America and Canada, participation in CHSS research studies is voluntary, and not all member institutions participate in CHSS studies. Therefore, the potential denominator of eligible patients for each CHSS protocol is not known. Furthermore, it is not known whether or not the patients enrolled in CHSS studies are truly representative of the entire population of patients potentially eligible for CHSS studies. The representativeness of data in the CHSS Database has been studied and described in a series of two papers that reported a study funded by The Children's Heart Foundation [https://www.childrensheartfoundation.org/]; this study linked the congenital heart surgery databases of The Society of Thoracic Surgeons and the CHSS [https://www.childrensheartfoundation.org/our-impact/funded-research.html?get_id=G997DTG9qyE20ybRON6havJB1E3J7y97dGgnF3j9IqXjJK2M9K6oW2GiMBjdnMpFJQw%252BY0eiAKvSl8Oq7kCiBWE6NDp7czoyMDoiZGlzcGxheV91c2VyX2RldGFpbHMiO2I6MTtzOjc6InVzZXJfaWQiO2k6MjI4NjE0NztzOjQ6ImRhdGUiO3M6MTQ6IjIwMjMwMTI5MjAyMzI2IjtzOjc6ImV4cGlyZXMiO2k6MDt9].40,41 This analysis reported that “For the CHSS Critical Left Ventricular Outflow Tract Obstruction (LVOTO) study, for the Norwood procedure, completeness of enrollment at centers actively participating in the LVOTO study was 34%. For the Norwood operation, discharge mortality was 15% among 227 enrolled patients and 16% among 1768 nonenrolled potentially eligible patients from the 40 consenting institutions. Median postoperative length of stay was 31 days and 26 days for these enrolled and nonenrolled patients. For the CHSS AAOCA study, for AAOCA repair, completeness of enrollment at centers actively participating in the AAOCA study was 40%.”
41
This study concluded “Determination of the denominator of patients eligible for CHSS studies and comparison of ‘eligible and enrolled patients’ to ‘potentially eligible and not enrolled patients’ provides an estimate of the extent to which patients in CHSS studies are representative of the overall population of eligible patients; however, opportunities exist to improve enrollment.”
41
Moving forward, CHSS is currently exploring collaboration with other databases and registries to address the potential limitations of (1) not knowing the potential denominator of eligible patients for each CHSS protocol, as well as (2) not knowing whether or not the patients enrolled in CHSS studies are truly representative of the entire population of patients potentially eligible for CHSS studies.
An additional important challenge and potential limitation of the CHSS Database is the difficulty in consistently contacting patients during protracted longitudinal “lifelong” follow-up. This challenge is compounded by the evolving landscape of informed consent as patients transition from childhood to adulthood. It is hoped that the transition of the extant CHSS Database and CHSS Data Center to The CHSS Center for Research and Quality, with a clear focus on quality assurance and quality improvement, will facilitate identification of patients and routine longer term follow-up. This transition will include the implementation of the ongoing CHSS strategy to convert many aspects of CHSS data collection to an overall quality assurance and quality improvement program; this strategy should decrease the challenges associated with contacting patients during protracted longitudinal follow-up.
Value of this Analysis
This analysis has cataloged the metrics and “academic impact” of CHSS studies in objective ways, using number of citations in PubMed (Medline). Clearly, multiple other metrics are available to estimate “academic impact” including altmetric score, metrics that quantify “reads,” number of citations in other media, Journal Impact Factor, and Hirsch index (also known as the h-index). 68 Each bibliometric index seems to represent a different phenomenon of journal article consumption. 68 No single bibliometric index in isolation offers ample representation of publication consumption or “academic impact.” 68
Of course, beyond the assessment of the “academic impact” of a given journal article or study, the “clinical impact” of a given paper or study can also be assessed and highlighted by allusion to examples of CHSS work that is now widely disseminated and accepted and employed. This assessment of clinical impact involves describing and evaluating CHSS studies in order to answer the fundamental question: “What impact or influence have CHSS publications had on clinical practice?” Examples of CHSS studies that are now widely disseminated, accepted, employed, and therefore, influence clinical practice across the world include:
TGA: creation of the evidence base for conversion of the field from the atrial switch operations to the arterial switch operation as the “gold standard.” PA-IVS & PS-IVS: shunt and outflow, tricuspid valve z-score, and feasibility of biventricular repair.5,6 LVOTO: CHSS-1 and CHSS-2 Calculators to assess the feasibility of biventricular repair.25,35 Assessment of late functional outcomes of patients in the CHSS cohorts.53,57,58
Many other examples exist as well. These examples demonstrate some of the changes in the clinical practice of pediatric and congenital cardiac surgery that have resulted from CHSS research. Furthermore, these examples demonstrate that the CHSS cohort studies have provided foundational ideas and data, and the field of congenital cardiac surgery predicates many of its assessments and decisions on these ideas and data.
Directions for Future Research
Future research may further explore the “academic impact” of CHSS studies using a variety of other bibliometric indices (ie, bibliometric metrics), including Journal Impact Factor and h-index:
Journal Impact Factor is a citation-based metric which is often used to assess the academic impact of a given journal. For each of the 64 CHSS publications described in this current article, one could tabulate the Journal Impact Factor at the time of publication of the given paper for each of the journals where these articles were published. This calculation would provide an additional metric of the impact or importance of these papers. (Journal Impact Factor considers the citation impact of papers within a given time frame in a given journal. It is calculated by dividing “the number of citations in a given year to papers published in the previous two years in the journal being evaluated” by “the number of citeable items published in the previous two years in the journal being evaluated.” For example, the 2022 Journal Impact Factor is calculated by dividing “the number of citations in 2022 to content published in 2020 and 2021” by “the number of citeable items published in 2020 and 2021.”
68
The Hirsch index is also known as the h-index and is designed to improve upon simpler measures such as the total number of citations or publications. The h-index is defined as the maximum value of h such that the given author/journal has published at least h papers that have each been cited at least h times. In other words, the h-index is the largest number h such that h articles have at least h citations each. For example, if an author has five publications, with 9, 7, 6, 2, and 1 citations, respectively (ordered from greatest to least), then the h-index of the author is 3 because the author has three publications with 3 or more citations. However, the author does not have four publications with 4 or more citations.
68
Each of these bibliometric metrics is complementary and offers a slightly different view of “academic impact.”
68
In fact, impact from research has been variably defined.”
69
Reed and colleagues “define research impact as demonstrable and/or perceptible benefits to individuals, groups, organizations and society (including human and non-human entities in the present and future) that are causally linked (necessarily or sufficiently) to research.”
69
It is a fact that the evaluation of the impact of published research is a complex process. The framework for evaluation may take many forms, just as studies themselves do. Perhaps the ideal framework for demonstrating impact is a randomized, controlled trial, but this design is not often feasible. The outcome measures must be carefully selected. The frequency of citation utilized in this current article is a practical measure, but this metric neither accounts for critical versus supportive citations, nor does it necessarily translate to the impact the paper has on clinical care and outcomes. Even the adoption of a published novel surgical technique may not be suitable as a metric of impact, if improvement in the outcomes of patients has not been thereby demonstrated. Impact may also be time-dependent. And, impact may be detrimental rather than beneficial. Reed and colleagues used Grounded Theory Analysis to examine various approaches for evaluating the impact of research and developed a methodological framework for analyzing this impact.
In our current analysis, we have not attempted to address the impact with this level of rigor. Instead, we limited our analysis to, at most, the “textual, oral, and arts-based” evaluation design described by Reed and colleagues. 69 Indeed, the collective impact of the research published by CHSS may be that at least mid-term outcomes if not long-term outcomes, have been described with statistical rigor for the treatment of a set of relatively rare and complex congenital cardiac malformations. Such analyses cannot often be performed in single-institution studies (due to small sample size), nor can such studies be performed in registries that collect a limited amount of data for every patient undergoing heart surgery and contain limited data about longitudinal follow-up.40,41,67
Furthermore, the long-term follow-up in the CHSS cohorts is comprehensive (ie, tracking not only survival and basic well-being but also detailed medical data and, in selected studies, standardized multidimensional assessment of functional health status):
In the 2010 study of the CHSS interrupted aortic arch cohort,
33
for example, this duration and depth of follow-up allowed the investigators to demonstrate the persistent hazard for mortality and reintervention in patients after apparent “complete repair,” a finding critical to setting the bar for longitudinal clinical follow-up and for informing patients and parents. In the CHSS study of pulmonary conduit replacement, the finding of an “optimal size” for conduit replacement made it clear that “bigger was not better.”
24
Prior to this study,
24
the placement of the largest conduit that would “fit in the chest” was the approach of many surgeons who felt that one could forestall re-replacement for a longer duration with such a strategy. With over 1,000 patients in the CHSS AAOCA cohort, CHSS has become one of the leading research enterprises in the world addressing such issues as the anatomic correlates with the risk of ischemia and outcomes following surgical repair. Finally, CHSS investigators have taken on some of the most difficult problems of surgical decision-making not addressable in single-institution or registry-based designs. These challenges include the initial and subsequent management of patients with “borderline” left ventricle and “moderately unbalanced” complete AVSD. While the corresponding management recommendations are not yet well-defined for this cohort, the growing size of the cohort and the application of newer statistical techniques will enable CHSS to uniquely solve these difficult problems. Furthermore, the impact is created by publishing provisional or interim results, because it sends the message that “we are getting there.”
Given the palliative nature of many of the surgical repairs applicable to CHSS cohorts, it has been important for clinicians, parents, and patients alike to determine functional health status over time. The design of the process of cohort follow-up has allowed CHSS to collect functional health status questionnaires from a few of the longer-standing cohorts (transposition, pulmonary atresia with intact ventricular septum, and interrupted aortic arch). While interpretation of these results is always fraught with survival bias, the findings that, by and large, surviving patients enjoy most categories of functional health comparable to the normal population are fascinating, encouraging, and impactful.
All of this is to say that the “impact” of CHSS studies is multifaceted. CHSS studies, by their design and by the underlying resource of the evolving inception cohorts, can be impactful in multiple domains. Indeed, CHSS studies can:
inform clinicians and patients of the time-related clinical impact of the anomalies, change surgical management, and track and predict the most important patient-centered outcome metrics, namely, survival and functional health status over time.
Conclusion
Since the formation of the CHSS Database, 64 peer-reviewed articles have been published based on the CHSS Database. Fifty-nine peer-reviewed papers have been published based on the 12 Diagnostic Inception Cohorts, and five additional articles have been published based on Data Science.
Excluding the recently established Diagnostic Inception Cohort for patients with Ebstein malformation of tricuspid valve, the number of articles published per cohort ranged from 1 for coarctation 8 to 11 for TGA.1-4,9,11,13,17,18,57,58 The 11 publications generated from the CHSS Transposition Cohort were referenced by a total of 111 articles (median number of references per journal article = 9 [range = 0-22, mean = 10.1]).
Overall, individual articles were cited by an average of 11 (mean), and a maximum of 41 PubMed-listed articles. Overall, these 64 peer-reviewed publications based on the CHSS Database were cited 692 times in PubMed-listed articles.
In the final analysis, CHSS studies are widely referenced in the pediatric cardiac surgical literature, with over ten citations per published article. These cohorts provide unique information unavailable in other sources of data. A tool to access this analysis is available at: [https://data-center.chss.org/multimedia/files/2022/CAI.pdf].
Supplemental Material
sj-pdf-1-pch-10.1177_21501351231190916 - Supplemental material for The Academic Impact of Congenital Heart Surgeons’ Society (CHSS) Studies
Supplemental material, sj-pdf-1-pch-10.1177_21501351231190916 for The Academic Impact of Congenital Heart Surgeons’ Society (CHSS) Studies by Jeffrey Phillip Jacobs, William M. DeCampli, Tara Karamlou, Hani K. Najm, Bradley S. Marino, Eugene H. Blackstone, Brian W. McCrindle, Anusha Jegatheeswaran, James D. St. Louis, Erle H. Austin, Christopher A. Caldarone, Constantine Mavroudis, David M. Overman, Joseph A. Dearani, Marshall L. Jacobs, Christo I. Tchervenkov, Lars G. Svensson, David Barron, James K. Kirklin and William G. Williams in World Journal for Pediatric and Congenital Heart Surgery
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Supplemental Material
Supplemental material for this article is available online.
Abbreviations
Appendix 1
Appendix 1 provides the original 1984 letter to members of CHSS from James W. Kirklin, MD and Eugene H. Blackstone, MD that proposes the very first CHSS study, which was the CHSS TGA cohort.
Supplemental Appendix 1 provides the original program from the historic First Joint meeting of CHSS and ECHSA in Montreal in 2004.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
