Abstract
Scientific discovery increasingly relies on computation through simulations, analytics, and machine and deep learning. Of these, simulations on high-performance computing (HPC) platforms have been the cornerstone of scientific computing for more than two decades. However, the development of simulation software has, in general, occurred through accretion, with a few exceptions. With an increase in scientific understanding, models have become more complex, rendering an accretion mode untenable to the point where software productivity and sustainability have become active concerns in scientific computing. In this survey paper, we examine a modest set of HPC scientific simulation applications that are already using cutting-edge HPC platforms. Several have been in existence for a decade or more. Our objective in this survey is twofold: first, to understand the landscape of scientific computing on HPC platforms in order to distill the currently scattered knowledge about software practices that have helped both developer and software productivity, and second, to understand the kind of tools and methodologies that need attention for continued productivity.
1. Introduction
Simulations performed on high-performance computing (HPC) platforms have been the cornerstone of scientific computing for more than two decades. However, the development of simulation software has, in general, been ad hoc. Most of the available software has grown through accretion, where code addition is driven by the need for producing domain science papers without adequate planning for code robustness and reliability. With an increase in scientific understanding, models have become more complex, rendering an accretion mode untenable to the point where software productivity and sustainability have become active concerns in scientific computing. Here, the terms software productivity and sustainability include aspects of the software lifecycle such as design, verification, maintenance, and reproducibility, where the use of good software practices helps increase the productivity of software developers and the sustainability of software artifacts. For brevity, from hereon we use productivity to imply sustainability as well. Examples of efforts devoted to software productivity and sustainability include the UK-based Software Sustainability Institute (UK SSI) (Software Sustainability Institute, n.d.) and the Interoperable Design of Extreme-scale Application Software (IDEAS) project (IDEAS Productivity Project, n.d.) initiated by the US Department of Energy (DOE). The UK SSI concerns itself with all aspects of research software. The IDEAS project focuses on easing the challenges of software development for extreme-scale computing and ensuring that investment in the exascale software ecosystem is as productive and sustainable as possible, thereby helping improve scientific productivity while ensuring continued scientific success. The National Science Foundation (NSF), under its Cyberinfrastructure for Sustained Scientific Innovation program (CSSI, 2019), supports Software Institutes that either serve specific science domains or provide specific software services to the scientific community. Content related to software productivity is also appearing in education and training. For example, the Argonne Training Program in Extreme Scale Computing (Dubey and Riley, 2017) has included a software engineering and community codes track from the outset. The IDEAS project has conducted several successful tutorials on the topic, including at SIAM-CSE17, SC16-SC18, and ISC 2018.
Common to all the training programs and productivity efforts is the reliance on experience from well-managed publicly available application software and libraries that have pioneered the use of software design and process in computational software. Notable examples include Cactus (Blazewicz et al., 2013), FLASH (Dubey et al., 2009), LAMMPS (Plimpton, 1995), PETSc (Balay et al., 2019), and Trilinos (Heroux et al., 2005). These software efforts have embraced and adapted some effective practices from the mainstream software engineering community and have organically developed some practices derived from their own needs. In this survey paper, we examine a modest set of HPC application codes. Several of them have been in existence for a decade or more and have provided a rich training ground for researchers and developers who have gone on to use their experiences to enhance various productivity efforts. Our objective in this survey is twofold: first, to understand the landscape of scientific computing on HPC platforms in order to distill the currently scattered knowledge about software practices that have helped both developer and software productivity, and second, to understand the kinds of tools and methodologies that need attention for continued productivity. In a study like this, it is important to articulate the parameters used in the selection of software, since a comprehensive list is too large to either study or enumerate. We have focused on software largely from the United States that has significant use at NSF and DOE HPC centers. Our sources for selection include the INCITE program (INCITE Program, n.d.), which allocates computing resources from the DOE HPC facilities to scientific projects; heavy users of NSF HPC facilities; and applications from the US DOE Exascale Computing Project (ECP; The Exascale Computing Project, n.d.). Although several of the codes studied have international users and contributors, the inferences may not be directly applicable to other international efforts, since funding models and scientific priorities differ significantly between countries.
The remainder of this paper is organized as follows. In Section 2, we summarize the path of scientific simulations’ growth over the previous two decades and motivate this study. We also list the categories in which we seek insight. In Section 3, we provide information about the methods used to obtain data for the study and, in Section 4, we describe the scientific codes that we have included in the study. In Section 5, we present a detailed analysis of the collected information and our inferences, and in Section 6, we give our conclusions.
2. Background
In the early days of scientific simulations, models had many simplifying assumptions in order to make computations tractable. Individual researchers and small teams were able to make significant progress through these simple models because they were often the first glimpses into the phenomena of interest. The development process often followed the “heroic programmer” 1 mode, where a huge amount of effort is expended by a few dedicated developers to meet science needs. This mode was not particularly detrimental to scientific productivity in the early days because models and codes were not very complex. Even today in some research efforts the heroic programmer model is adequate, especially if the efforts are exploratory toward eventually building a realistic model for the phenomenon of interest. As understanding grows and more powerful computer resources become available, the heroic programmer model becomes problematic. As an example, we consider the case of Type Ia supernova simulations in astrophysics. As with most science, the simulations started with 1-D and 2-D models (Burrows and Fryxell, 1993; Janka and Müller, 1995; Johnson and McKee, 1971), which gave some insight into their inner workings. Early 3-D simulations assumed symmetry and modeled an octant of the star because of computational resource limitations (Reinecke et al., 2002). While all those simulations advanced the state of scientific knowledge, the real scientific gains from those simulations were the insights about critical physical processes in the model. Those insights led to an elimination of the symmetry imposed by the octant, which in turn produced the gravitationally confined detonation model, the first feasible mechanism for the transition from deflagration to detonation in Type Ia supernovae (Jordan et al., 2008). The 3-D simulations of the full star required multiphysics and multiscales with enough complexity that they could not have been done in the heroic programmer mode. FLASH (Dubey et al., 2009, 2013), the software used for the simulations, had been in development by a team for nearly a decade. During that time it had undergone three revisions to incorporate an extensible and composable software design and had evolved a well-defined software process, both of which were critical to the series of simulations that led to novel scientific insights.
Similar to FLASH’s experience, in the late 1990s, for various reasons including a greater understanding of the underlying phenomena, models began to become more refined and started to acquire a multiphysics aspect (Bryan et al., 2014: 4; Keyes et al., 2013). Investment was made in the development of software frameworks for such multiphysics simulations (e.g. Blazewicz et al., 2013; Center for Applied Scientific Computing (CASC). Not all frameworks survived, but a sizable fraction of the frameworks developed during that time have evolved into ongoing community codes. It is therefore important to understand the factors leading to their adaptation by their respective communities. Also, a study of their methods and processes can lead to a greater understanding of how codes and frameworks may evolve in the future, in order to avoid many of the pitfalls that led to some of the frameworks’ demise in that same timeframe. Additionally, insights about development teams’ concerns can lead to understanding of requirements for productivity in the future.
3. Methodology
Productivity in scientific computing continues to be a loosely defined term that changes its meaning depending on the context. Scientific productivity can be measured in terms of insights gained and ultimately in publications resulting from those insights. Scientific software productivity is much more amorphous and means different things to different people. A few general definitions can be found in the literature (e.g. Heroux and Allen, 2016; Software Productivity for Extreme Scale Science, Workshop Report, 2014), but none of these definitions is comprehensive or authoritative. Therefore, in this survey, instead of focusing on any specific aspect of productivity, we seek answers to several questions that can help us understand the features and processes that contribute to the long-term success of a software project and its acceptance by the user community. These questions can be broadly divided into three categories. Lifetime of the code: In this category, we collect information about how long the codes have been in existence and the way that they have evolved. Code teams and their practices: In this category, we seek insight about the size and nature of the code team. User communities and science domains served: This category can tell us whether a correlation exists between barrier to entry and ease of use and a code’s adoption and also whether synergies exist between domains if a code serves multiple domains.
Note that not all the questions could be answered for all the codes we studied; therefore, some gaps appear in the tables and plots in the results and analysis sections.
We used various resources to obtain data for this study, including information about projects available on the web, general knowledge about the codes in the community, and data from IDEAS-ECP interviews. Methods that resulted in reliable and, therefore, usable data are discussed below.
3.1. Method 1—Literature study
For a survey of scientific work, the available literature is the obvious place to start. Many code projects ask their users to cite specific publications if they obtain results using the code. For every code or project that has a website, we searched for these suggested citations. We collected their citation history over the years from freely available citation analysis tools from Web of Knowledge. 2 Additionally, we searched for any software engineering or experience papers published by the authors of the code for insights about the software practices. We also mined the presentation materials available from a few workshops related to software productivity and community codes (Building Community Codes for Effective Scientific Research on HPC Platforms, 2012; Working Towards Sustainable Software for Science: Practice and Experiences, n.d.)
3.2. Method 2—Application team interviews
A set of interviews were conducted by the IDEAS-ECP team, of which all the authors are members, with several ECP Application teams to gain an understanding of their software development processes and bottlenecks. The interviews took place in person or by teleconferences; interview questions were distributed beforehand if the team requested them. The IDEAS-ECP project uses the information obtained from the interviews to identify high-priority areas of need in the ECP community and to work with teams on overcoming productivity bottlenecks. In the context of this paper, we also use information from the interviews to substantiate and further elucidate data obtained from other sources.
3.3. Method 3—Repository analysis
We considered HPC software repositories and their issue-tracking data on GitHub to follow their growth and development processes. Issue-tracking systems are not widely used in scientific computing yet, although most of the projects we studied have used them to varying extents. Issue trackers enable developers and users to submit and track issues such as bugs, feature requests, enhancements, or any other tag the developers wish to define for their project. We used a combination of the GitHub REST API (2018) and git command-line APIs to retrieve git repository “statistics.” 3 These “statistics” include all the open and closed issues of the projects to get information such as creation date, issue number, issue description, labels attached to the issue, assignee, information about the user who raised the request, and other relevant information headers.
4. Codes
We considered a set of applications codes compiled from three sources. The first source is ECP, whose goal is to ensure that mission-critical applications can efficiently utilize exascale hardware to address key problems in scientific discovery, energy assurance, economic competitiveness, health enhancement, and national security. To facilitate this development, the IDEAS-ECP project is partnering with the ECP community to improve developer productivity, resource management, and software sustainability, while minimizing the costs associated with project maintenance and improvement. Among the key activities of IDEAS-ECP is to conduct software development team interviews in order to understand the current state of their software practices and the challenges that they face. We have mined these interview notes in parts of this study, as explained later.
The second source is the DOE Office of Advanced Scientific Computing Research INCITE program. The bulk of the leadership-class computing resource allocations at the DOE facilities occur under this program. The third source is a similar set of large allocations made to applications by the NSF facilities.
The codes gathered from these three sources are not disjoint. Several of the codes under ECP have broader scope and community, and users in these communities ask for computational resources wherever they can. All the selected codes meet one or more of the following criteria: (1) they are well-recognized community codes in their respective science domains, (2) they are considered mission critical by their funding agencies, and (3) they are among the heaviest users of supercomputing facilities available for open science in the United States. While we attempted to assemble a representative group of applications important to various domains and agencies, some of them had to be dropped because it proved extremely difficult to obtain data about them from the resources available to us. Some communities have suites of codes where it is difficult to attribute available data to specific codes; these codes suites were also not included in the survey.
4.1. Amber
The Assisted Model Building with Energy Refinement (Amber) suite is one of the oldest biomolecular modeling packages, with a focus on biomolecules including proteins, nucleic acids, and carbohydrates (Case et al., 2005; Ponder and Case, 2003; Salomon-Ferrer et al., 2013). Started in 1975, the Amber suite is closed source and is available for industry and academic use through a site license fee. Because the full code is not publicly available, the corresponding column in Table 1 is marked with “NA.” Amber is maintained by an active collaboration of approximately 40 members divided among several institutions, many universities, and one pharmaceuticals company and has had over 1000 downloads.
List of codes used in the survey along with their base information and the corresponding methods used to obtain the information.a
a The blank cell indicates that repository information for XGC could not be obtained.
4.2. Cactus/Einstein Toolkit
The Einstein Toolkit is an open-source subset of software packages from the Cactus code project that is used in computational astrophysics and cosmology where general relativity calculations are needed (Löffler et al., 2012). Begun in 1997, the Einstein Toolkit now has a consortium of 94 members spread over 49 sites in 14 countries. Nine core maintainers from six sites oversee technical developments, while others participate through open development meetings. The user base for the Einstein Toolkit is around 100 and larger for Cactus.
4.3. Enzo
Enzo is an open-source Cartesian code that is capable of simulating a wide variety of problems including hydrodynamics, magnetohydrodynamics, N-body dynamics, cosmology, and star formation using block-structured adaptive mesh refinement (Bryan et al., 2014; Norman et al., 2007; O’Shea et al., 2004). It began as a closed-source project in 1994 and was opened in the early 2000s. Currently, the project maintains a distributed development team with 12 core and 25 outside contributors and no clear leader. In order to facilitate this approach, all discussions take place using mailing lists, Google docs, and archives. As indicated by these mailing lists Enzo has approximately 200 users.
4.4. E3SM
The Energy Exascale Earth System Model (E3SM) is an open-source coupled climate model that combines four separate models used to simultaneously simulate the Earth’s atmosphere, ocean, land, and sea ice (The E3SM Project, n.d.). Begun as the Accelerated Climate Model for Energy, this project brought together multiple components in 2014. A diverse team of 20 people is responsible not only for knitting together the diverse components of this project but also for establishing a software process that meets the needs of the entire DOE climate community. A section of E3SM is part of ECP.
4.5. FLASH
FLASH is a publicly available multiphysics simulation code used to investigate a wide range of problems including radiative and compressible hydrodynamics, thermonuclear flashes, high energy density physics, and fluid structure interactions (Dubey et al., 2013, 2018; Fryxell et al., 2000). First released in 2000, the current team of six developers along with additional contributors will release version 5 as fully open source through GitHub. With such a diversity of capabilities, the project has large community of over 1000 users.
4.6. GENE and XGC
GENE is a gyrokinetic electromagnetic numerical experiment code (The GENE Code, n.d.), and XGC is a particle-in-cell code (The XGC Code, n.d.). Together they are used to model plasma physics associated with tokamaks, that is, magnetically confined fusion plasmas. The two codes have existed separately since around 2000 and are now being coupled under the ECP project WDMApp. The codes are distributed publicly, although most of the code developers are also its users. GENE has around 236 users, and XGC has around 10 users.
4.7. HACC
The Hardware/Hybrid Accelerated Cosmology Code (HACC) is a highly scalable software capable of simulating high-resolution N-body gravitational simulations used to construct distribution statistics for galaxy formation (Habib et al., 2016). HACC is not openly distributed, and its 11 developers also act as its users. HACC is a part of the ECP efforts.
4.8. LAMMPS
The Large-scale Atomic Molecular Massively Parallel Simulator (LAMMPS) is an open-source code well-known in the materials science community. It is capable of modeling the molecular dynamics of materials used in multiple contexts, including quantum mechanics, thermochemical coupling, defect dynamics in solids, granular flow, blood flow, and microfluidics in a large spatial domain (Plimpton, 1995). One of the oldest such codes, a closed first version of the code was released in 1995 and refactored and opened in 2004. The four-member development team is small, but the number of external contributors is large. The project remains popular, with over 1000 users.
4.9. LATTE
LATTE (Bock et al.) is an open-source density functional tight-binding molecular dynamics code that was released in 2016 and is incorporated, along with three other codes including LAMMPS, as part of the ECP project EXAALT focused on modeling materials under harsh conditions. The team consists of five to six colocated developers who use regression testing and version control through GitHub. The user group is small, consisting primarily of the developers themselves.
4.10. MARBL
The MARBL suite combines two codes—one with high-order finite element capabilities and the other with high-order finite difference capabilities—in order to perform multiphysics modeling of high energy density laser-driven, explosive, and/or magnetic energy sources. The suite started in 2014, with roughly 20 people in the team with varying degrees of experience in software engineering. It has a small user base, and the developers have no immediate plan for releasing the code to the public.
4.11. NAMD
NAMD is a molecular dynamics code designed to use platforms at various scales from personal computers to leadership-class supercomputers. NAMD has been developed and licensed since 1995 by the University of Illinois at Urbana-Champaign (Phillips et al., 2005). NAMD is built on top of Charm++ (Kale and Krishnan, 1996) with an asynchronous parallelism that was ahead of other similar codes at the time. The code is currently maintained by a team of four core developers and remains popular with over 7000 users.
4.12. Nek5000
Nek5000 is a highly scalable variant of Nekton (3-D spectral element code) that can model reactor thermal hydraulics, combustion, oceanography, and vascular flow (Lottes et al., n.d.). It was created in 1999 and made open source in 2009. The code is maintained by 10 developers who have implemented a thorough test suite of over 400 tests along with validation when compared against engineering simulation. It supports a community of over 400 users.
4.13. NWChem
NWChem is an open-source code that uses multiples techniques, most popularly density functional theory, for making quantum mechanical calculations of chemical and biological systems (Valiev et al., 2010). The code has quantum mechanical methods, quantum molecular dynamics, molecular mechanics, hybrid models, potential energy surface analysis, and electronic structure. The code began in the mid-1990s and was made open source in 2010. The team has a quality assurance suite on GitHub and is transferring code releases from SVN to GitHub as well in order to support the many external users and 70,000 downloads. The code is currently being adapted for heterogeneous platforms under an ECP activity, NWChemEx.
4.14. QMCPack
The quantum Monte Carlo (QMC) package, QMCPack, is an open-source electronic structure code that is capable of modeling molecular and periodic 2-D and 3-D solid-state systems using a variety of QMC algorithms (Kim et al., 2018). The code has been around for 20+ years and is currently maintained by a team of 5 core developers along with approximately 17 active contributors. Code release and development are supported through GitHub, and strict rules are provided allowing the large user community to contribute as well.
5. Analysis and inferences
We began our analysis 4 by tabulating data and observations. In every table, a row represents a code, while the observations are listed along the columns. We inserted NA where information could not be obtained using any of the methods described above. Where possible we also indicated the reason behind the lack of information.
5.1. Lifetime of the code
For the codes used in this study, Table 1 lists the year when they came into existence, the year they became public if they did, information about the code repository, their source (ECP, INCITE, or NSF), and the methods used to obtain information about them. Table 2 lists the code capabilities, the growth in capabilities over the years, the overall growth in the number of lines of code (the “Size growth” column), and their current version number for codes that have been released. In the second column, where “Multiphysics” is listed, the corresponding code can model multiple physics operators and can be used for more than one science domain. Together with Table 1, this table tells us about the lifetime of the code. We note that although some of the oldest codes still have some versions in SVN, adoption of distributed repositories (git or bitbucket) is almost universal. One could argue that the longevity of the code would have necessitated adopting efficient tools as they become available, and this agreement is reflected in the table. The other interesting observation is that going public is a strong trend. Of the codes that are not open, MARBL is constrained to remain closed because of national security concerns, Amber has found use in industry and has a fee-based licensing model, and HACC is closed because of team preference.
Lifetime, complexity, and growth of the codes.
Another interesting finding from Table 2 is that codes cluster into a few groups. In the list, there are codes for quantum physics and chemistry, for astrophysics and cosmology, for plasma physics, and for computational fluid dynamics. One can also see from Table 2 that the length of a code’s existence is not necessarily indicative of its growth curve. Amber is the longest lived, but it still serves primarily one kind of target research domain, while FLASH has expanded the core physics capabilities it offers and made concerted efforts to diversify its user community (Dubey et al., 2018). Codes that have existed for a while have evolved in terms of their methods, support for architecture, and performance in order to stay relevant, whether or not they have diversified to serve research communities other than their original targets.
To analyze the evolution of code over time for projects with publicly available data, we manually inspected each project’s source code base and labeled directories or individual files with one of seven labels: Science, Math, Infrastructure, Tests, Documentation, External, and Other. This is a high-level categorization that is calculated manually in order to give a rough idea of the projects’ compositions and evolution over time. For most scientific applications, the “Science” category includes many domain-specific mathematics applications as well; we reserve “Math” for generic methods, such as basic linear algebra, which are typically provided by numerical toolkits and do not have any application-specific features. The “Infrastructure” category is used for code that does not directly encode domain-specific solution methods or generic math (e.g. I/O, parallelism management, and visualization). While some repositories are predominantly used for developing domain-specific code (Enzo, Nek5000), we note that most projects’ repositories include a significant fraction of “Infrastructure” code. The “External” category is used for third-party software that is included in the project’s repository but has been developed elsewhere. “Tests” include unit and regression tests, while “Documentation” includes manuals, tutorials, Doxygen specification, and examples. “Other” is used for files that do not fit any of the other categories, such as sample input data files that are included in the repository.
Figure 1 plots the results of this analysis. For some projects, we observe that the code size seems to have reached a steady state (Enzo, GENE), while in other cases it is steadily increasing (E3SM, LAMMPS, NAMD, NWChem, QMCPACK) or, more rarely, decreasing (LATTE). Most of the projects we considered show increased attention to testing and documentation over time. In some cases, the use of revision control systems appears less organized; for example, large components of Nek5000 code are removed from the repository and later added back, and in NAMD sharp spikes in code sizes occur when generated documentation was first included in revision control and then removed. In general, sudden significant changes in code size usually result from generated files being briefly added to a repository or code being migrated to or from other repositories (which are not part of this study).
Code size (in number of lines) over time. We categorized each project’s contents at a high level into several categories: Science, Math, Infrastructure, Tests, Documentation, External, and Other. Line counts are shown for each calendar year for which revision control data is available.
While code size does not necessarily indicate proportional effort (since some projects include generated code in their repositories), it may be useful to consider each project’s composition and growth trends when planning major efforts such as refactoring of some subsystem. With respect to the high-level categories, this is an initial, approximate characterization, which undoubtedly can be continuously improved and refined. Similar to code size, it is not meant to represent the actual time that project members spend on “Science” versus “Documentation,” for example, but only to show the evolution of the specific repositories we considered.
5.2. Code teams and their practices
The data and analysis about code teams and practices are presented in Table 3. The table lists the number of core developers and contributors to the code, whether the team is interdisciplinary, and the kind of testing and verification practices used in code maintenance. The number of core developers is derived mostly from the code websites and interviews. We also used repository mining by considering frequent and long-term committers to be the core developers. This approach may not be completely accurate in terms of being employed directly by the project, but we believe that it is a reasonable reflection of the core team in terms of contributions by individual members.
Size and nature of code teams and their practices.
As expected in academic and research codes, the core teams are small to modest in all projects, although the core size varies depending on the lifecycle and home of the code. The longest existing public codes have a moderate number of contributors who are not part of the core team, although some of the projects have formed formal organizations for code management (e.g. Cactus/Einstein Toolkit and Enzo). Codes are divided roughly evenly between those that use interdisciplinary teams and those that do not. One notable feature of the table is what it tells us about verification. Almost all the codes have a testing program in place, with many using automated testing. The entry “likely automated” indicates that we have good reason to believe that testing might be automated but have not been able to verify that belief. More than half the codes are also using continuous integration in some form.Figure 2 shows “river plots” of the open and closed issue counts for the projects for which these data are publicly available and maintained on GitHub. These plots start from the timestamp when the first issue was created on GitHub. Issues include bugs, feature requests, enhancements, and other types of entries used for tracking progress on various project milestones. Issues tagged as bugs (or similar tags, e.g. “defect”) are shown separately in green for the projects where such tagging was done by the developers. The majority of projects have fairly consistent use of issue tracking and bug reporting, which indicates that software maintenance and process are important to the code teams. Some projects have well-established issue-tracking practices, while others are just beginning to use issue tracking on GitHub. (We have not explored other issue-tracking tools.) In most cases, we observe that while the number of new issues keeps increasing, the number of closed issues is not increasing at a similar rate; that is, the “river” gets wider with time (E3SM, Enzo, LAMMPS, Nek5000, and QMCPACK). This may indicate that additional development resources may be needed for such projects to keep up with the increased demands as represented by the new issue rate. In other cases, the rate of closed issues is mostly keeping up with new issues being opened (FLASH, LATTE, and NWChem), which is an indication of sufficient development resource allocation. Even in cases where the issue-closing rate is lagging, we observe that fixing bugs is prioritized by the teams, as indicated by the plots in green in Figure 2, reflecting the importance of code stability.
Issue (pink) and bug (green) tracking counts versus time for the projects using issue tracking with recognizable tagging of bugs. For LATTE, only issue-tracking data is available.
5.3. User communities
Table 4 gives us information about the user base of the codes. The table lists research domains served by each of the codes and an estimate of the size of their user community. The codes fall roughly into three groups: closed codes where developers are also the users, open codes where users are also most of the developers, and codes that have many more users than developers. HACC, MARBL, and LATTE fall into the first group; GENE and XGC fall into the second group; and the rest of the codes belong in the third group.
Category 3: User community.a
a For some codes we list download statistics. Where available we give an estimate of the number of users. The blank cells indicate that no information was obtained.
Figure 3 plots citations per year of the papers suggested by the code developers to cite if the corresponding code was used to obtain the results. Figure 3 has some anomalous features. For example, although Amber has been in existence since 1975, its citation history appears to start in 2005. The reason is that the website recommends a publication in 2005 to be cited, thus hiding the earlier citation history. LAMMPS and FLASH still suggest citing their original papers, thus presenting a more complete view of their citation history. Despite such anomalies, some useful inferences can be drawn from the figure. LAMMPS’ development and open-sourcing history is reflected in its citation plot. Its number of citations begins to increase every year after the code is made public in 2004. HACC has the fewest citations, as expected because the code is not used by anyone outside the core team. For most of the codes plotted, the trend over the past few years has been a stable and steady rate of citations, which in turn indicates a stable user community.
Number of citations for the publications specified by the application teams as the principal reference when the project is referenced.
More interesting observations about the penetration of code in different communities come from Figure 4, which breaks the citations into percentages of citations for a code in different science and engineering domains. The figure has two sections: the top section covers molecular dynamics codes, while the bottom section covers codes from astrophysics and cosmology. Not surprising, the application domains are nearly mutually exclusive for the two classes of codes. All have citations in physics, math, and computer science, which are likely to be for their methods, algorithms, and performance-related work. The codes in the top section have a broader application domain reach than does the bottom section. Cactus and FLASH have small percentages appearing in more application domains than do the other two codes; both, however, share the design philosophy of having a robust infrastructure layer where physics capabilities can be plugged in for different application configurations (Blazewicz et al., 2013; Dubey et al., 2009).
Number of citations in the most popular domains for the publications specified by the application teams as the principal reference when the project is referenced.
6. Conclusions
In this survey, the objective was to study the state of the art in productivity-related practices of HPC scientific software in the United States. The study is not exhaustive but representative in terms of included software. These codes are among the heavy users of DOE and NSF supercomputing facilities in the United States. The strongest visible trends among the codes studied are that they all make use of distributed repositories for source code management, they all have some degree of verification regime in place, and they all place at least moderate emphasis on code maintenance. The trend toward making codes open source is also encouraging and is taking the community in the right direction. Less obvious observations are that the codes with conscious choices in configurability and extensibility (i.e. FLASH and Cactus) have grown more in their base capabilities and have diversified in order to reach more research domains. These codes enable synergy among scientific domains. Also, the newer codes have started with stronger software engineering practices in place from the beginning; however, the older codes are not far behind, and some are even ahead, consistent with the expectation that long-lived codes have teams with an agile approach and adopt newer tools, methods, and practices as they become available. We could make a stronger statement by comparing with the projects that did not adapt and therefore failed; however, that kind of reliable data is extremely difficult to obtain.
We conclude that the computational science community appears to be headed in the right direction when it comes to software longevity, robustness, and productivity. Increasingly robust software engineering practices and a near-universal move to distributed repositories strongly suggest that software engineering practices in computational science have arrived at an inflection point. Our collective experience from attending workshops and meetings and from communicating with colleagues that produce scientific software and those that use it for their science is that teams are finding that improving software practices is essential to support effective scientific collaboration around software and also to ensure the integrity of scientific results. Greater inclination to allocate resources for improving software practices indicates that the current mode of ad hoc dissemination of knowledge obtained through trial and error may not suffice in the future. Our analysis strongly indicates both the need and the opportunity for research in software engineering for science. Various community efforts are now making strides to address these issues (Katz et al., 2019). Our own future plans in using the tools we have developed and the data we have collected are focused on understanding what might be good productivity metrics for scientific software.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This material is based upon work supported by the US Department of Energy, Office of Science, under contract DE-AC02-06CH11357. This research was partially supported by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of the US Department of Energy Office of Science and the National Nuclear Security Administration.
