Advanced Toxicological Risk Assessment by Implementation of Ontologies Operationalized in Computational Models

Abstract

Given tremendous progress in biology, toxicology, and chemistry knowledge in recent decades, the time is right for serious consideration of options to move away from animal experimentation in chemical hazard and risk assessment. Individual alternative and animal-free assays as a replacement of individual animal studies have met with understandable reluctance in the scientific and regulatory arenas. An integrated conceptual approach based on mechanistic information built from all available information on biology, chemistry, and mechanisms of toxicity, might allow sufficient coverage of the biological system to provide the basis for reliable animal-free chemical hazard and risk assessment for man. We describe construction of an ontology, which can be considered a network of adverse outcome pathways, including feedback loops representing homeostasis. Basic elements in the ontology are subjects (such as enzymes, receptors, and cell types) and their quantitative relationships (response-response relationships), together forming a multidimensional network of biological interactions. This network can be modeled in silico, providing an integrated system toxicology computational model with which toxicity predictions can be made at the level of adverse outcomes in the intact individual. The model will indicate critical rate-determining steps in the network that can be monitored in a battery of dedicated in vitro assays, providing a testing strategy to collect data feeding into the systems model. Connecting this integrated dynamic model with exposure models allows quantitative hazard and risk assessment for man, avoiding animal experiments.

Introduction

Experimental animal studies are routinely used for (chemical) hazard identification and risk assessment. However, the public acceptance of classical toxicological tests using animals is decreasing due to ethical considerations. Meanwhile, the scientific community has realized that classic toxicity testing may offer limited predictivity for human health, as an adverse effect seen in an animal study may not always be relevant for the human situation. For translation of an effect observed in an animal study to man, information is needed on the mechanism of action and the similarities and differences in biology and physiology between humans and animals.

A lack of predictivity of specific effects in animals for human toxicity has been shown for the well-known compound thalidomide, which causes limb malformations when taken by pregnant women early in pregnancy.¹ Limb malformations have not been found in rodents,² possibly due to a difference in the mechanism of action. In pharmaceutical research, species specificity is currently carefully monitored and shows that differences in mechanisms may affect predictivity of animal studies for man.^3,4 Also, the results of studies with different animal species may not be in concordance and thereby raise questions on the interpretation of data.⁵

Our awareness of the differences between animals and humans has led to the design of more mechanistically based in vitro assays, including molecular, cellular, and multicellular assays, and development of models that more closely mimic the human in vivo biology and physiology of various organs, for example the liver.⁶ The development of these assays is directed toward predicting effects for humans and they mostly use mechanism-based readouts as endpoints, which enable consideration of human-relevant findings in the hazard and risk assessment process.

Classical approaches to the application of such alternative methods have often focused on a one-to-one replacement of the animal study with a single alternative animal-free assay. This comparison is hampered by the reductionist nature of alternative methods, which by definition cover only a limited aspect of the entire organism. This reductionist nature of alternative methods unavoidably leads to limitations in predictions for the whole organism. Beside the limited aspects for predictivity of toxicity in the entire organism, there might also be, for example, limitations due to metabolic capacity.⁷

Next to the development of assays for single endpoints, other efforts were directed toward measurements of a wide array of data, like gene expression changes. Measurement of genome-wide gene expression changes after exposure to a compound with known toxicity has been applied to group compounds based on their mode of action and known adverse outcomes.^8,9 Some of these efforts to group compounds seem promising, but also have their limitations, which originate from the models used and the inevitable need for extrapolation of the exposure characteristics. As a consequence, the once standard pursuit of the validation of single assay predictivity has hampered regulatory implementation of alternative methods. From the biological perspective, a single assay does not cover all aspects that need to be considered in a reliable hazard and risk assessment. Therefore, it is understandable that the regulatory arena has been reluctant to move away from animal testing and rely on alternative methods instead.

Despite limited application in the regulatory field, alternative methods have progressively shown their merit in revealing mechanistic characteristics of the interaction of chemicals with biological systems. Such information and the assays providing it are increasingly being employed in the private sector for screening purposes, to guide the development and selection of notable substances for a variety of uses.

Integrated testing strategies, which combine different assays for similar adverse outcomes, were developed to improve predictability of in vitro assays, like for skin sensitization¹⁰ and similar approaches have been discussed for developmental neurotoxicity.¹¹ Computational methods have been used not only to predict toxicity for specific endpoints¹² but also to estimate safe exposure levels.¹³ Overall hazard and risk assessment, however, need an integrated assessment of untoward effects of chemicals on physiology. This requires the combination of alternative assays, which together consider physiology in sufficient detail to make reliable inferences about the toxicological profile of the substance under study.

Ontologies

Definition of an ontology

Coverage of the complexity of biology in combinations of alternative assays can only be reliably captured in a theoretical framework. This framework can guide a standardized and structured description of the physiological network relevant for toxicity testing, for example, in an ontology. This ontology can be used to determine the aspects that need to be covered in a set of alternative assays, which together can provide reliable and sufficient information for chemical hazard and risk assessment in the target species: the intact human. ECETOC's definition of an ontology provides the structure¹⁴: “An ontology is an organized representation of a domain of knowledge consisting of concepts and information, generally referred to as classes, and relationships between classes. Ontologies are useful in organizing information into a structure that makes the information more understandable and facilitates hypothesis generation.”

We define the ontology as a network of subjects and their relationships in a quantitative way, describing the complexity of physiology in a standardized manner. It can be used as a model that can be interrogated as to the consequences of quantitative changes in subjects and relationships for the model as a whole, simulating effects of substance exposure and predicting adverse health effects. Thus, the ontology contains the network of adverse outcome pathways (AOPs) (Fig. 1). It makes use of the wealth of existing knowledge in biology, chemistry, and toxicology collected over a century of toxicological research. This approach, which combines structured biological knowledge with in vitro toxicity assays and chemical information, is currently the most promising tool for future risk assessment.^15,16 The design and composition of the ontology and its application in animal-free human-focused hazard and risk assessment are addressed in the following paragraphs.

FIG. 1.

Applying an ontology-driven computational model for toxicological risk assessment. The ontology can be described as a network of adverse outcome pathways, each leading from a molecular initiating event (hexagon) by key events (circles) to adverse outcome at the organism level (triangle). This network can be described as integrated ontology connecting subjects (shapes) by quantitative relationships (arrows). The ontology forms the basis for the design of an integrated quantitative computational systems toxicology model based on which critical rate-limiting key events (closed circles) can be identified, which need to be monitored for chemical safety assessment. The critical key events (closed circles) are monitored in dedicated in vitro assays (diagrams), with increasing complexity as they represent key events further downstream in the network. The outcome of the test battery, combined with exposure modeling, feeds into the computational systems toxicology model (graphic representation at the bottom), which predicts risk.

Information captured in an ontology

An ontology is built from subjects and their quantitative relationships, which together form a multidimensional structure (Fig. 1). In the physiological context, subjects can be a diversity of molecules, genes, RNAs, proteins, enzymes, metabolites, hormones, receptors, matrix molecules, but also cell types, tissues, organelles, organs, and morphological structures. Relationships describe the way in which a change in one subject affects a change in another subject, for example, an increase in a hormone level will affect receptor occupancy and may trigger receptor transition to the nucleus and subsequent downstream effects at the molecular, cellular, tissue, and organism level.

It is essential to describe relationships between subjects in a quantitative way and regard the consequences of changes in a subject at the level of the ontology. Quantitative description of relationships between subjects allows the distinction between physiological changes within homeostatic control and changes that result in adverse health effects at the level of the organism at large. For example, continuous fluctuations in hormone levels due to external environmental triggers are normal and essential for sustaining integrity of the organism in a continuously changing environment. Taking toxicological testing to the molecular level, the challenge lies in the distinction of physiological adaptation within homeostasis from changes leading to adverse health effects. This requires that, in the ontology model, molecular information is integrated up to the level of the intact organism.

The ontology integrates biological, toxicological, and chemical knowledge. It is based on mechanisms of action, which allow visualizing biological pathways and identification of critical steps in these pathways. Together, these critical steps are indicators of an adverse outcome and give information on the mode of action and adverse outcome. Existing knowledge of human physiology from the molecular to the organism level is extensive and detailed.

Toxicological testing most probably does not need to address all aspects of chemical-induced physiological modifications at the same level as our knowledge from biology. Rather, based on existing toxicological and mechanistic knowledge, a selection of the physiological map covering the toxicity pathways, and specifically the critical rate-determining key events therein, may suffice as the basis for a system that allows for reliable toxicity predictions. These critical key events are assessed with mechanistic assays that contain these events and allow monitoring of their perturbation by chemical exposures. Thus, it can be envisaged that monitoring the perturbation by a substance of a limited number of critical, rate-determining key events in a group of dedicated assays can provide sufficient information for comprehensive prediction of hazard at the level of the intact organism.

Applications of ontologies

The application of ontologies is already being explored in various ways. Gene ontology (GO) was the first ontology to be developed for application in biological research and has since extensively been employed in molecular toxicology to describe gene expression responses to chemical exposures in in vivo and in vitro systems.^17,18 The information contained in GO is also being used in various third-party software applications that help analyze functional enrichment at the three GO domains of biological process, molecular function, or cellular component.

The success of GO led to the development of various other ontologies for different applications in computational life sciences. For example, systems biology markup language (SBML)¹⁹ was developed to describe computational biological models in a machine-readable way, to facilitate the exchange and application in other contexts. Building upon SBML, systems biology ontology (SBO) (www.ebi.ac.uk/sbo) is being developed to address mathematical requirements for modeling, such as enzyme kinetics equations and the quantitative parameters (enzymatic rate constants) that are involved. Biological Pathways Exchange (BioPAX)²⁰ is an ontology for analysis and visualization of biological pathway data in various forms, ranging from metabolic pathways to gene regulations.

From a perspective of describing (adverse) outcomes, several ontologies have been developed to describe phenotypes in humans and model organisms.²¹ Although this forest of ontologies might appear overwhelming, most of them use the same standards for reasons of being machine readable and exchangeable. The Open Biological and Biomedical Ontologies (OBO) foundry aims at creating ontologies based on shared principles of ontology development.²² Thibault et al. have described the development of an informatics infrastructure for data exchange of biomolecular simulations.²³ They conclude that an ontology approach is necessary to make possible complex query of data and review a number of ontologies building on the OBO. Specifically, for the field of toxicology, the OpenTox ontology initiative provides a framework for the support of predictive toxicology data management, algorithms, modeling, validation, and reporting.²⁴

This listing is far from exhaustive, indicating the extent of opportunities and information sources available for building a toxicological ontology. In addition, ontologies can be combined with in vitro high-throughput screening data and AOPs.²⁵

Molecular approaches to toxicological hazard assessment have gained ground in recent years. The US National Academy of Sciences has advocated the use of mechanistic approaches and toxicity pathways in hazard assessment.²⁶ OECD is promoting the use of AOPs to inform about key events that can be monitored to inform about the likelihood of adverse outcomes to occur.²⁷ Moreover, OECD strives toward integration of knowledge into Integrated Approaches for Testing and Assessment (IATA), providing practical approaches for combining mechanistic assays to predict toxic effects.^28,29

The AOP paradigm has its merits and limitations. It rightly focuses attention to mode of action and mechanistic aspects of chemical toxicity, facilitating judgment about the relevance of findings for the human situation.³⁰ AOPs with overlapping key events can be connected into an AOP network, which could be applied in risk assessment.^31,32 The AOP concept provides information on major toxicity pathways that can be instrumental in building an integrated ontology. One important lesson realized from the AOP approach is that not all key events, although essential for the pathway to run from start to end, are rate-limiting in terms of determining adverse outcome. As a consequence, such nonrate limiting events need not be monitored to aid toxicity prediction, as long as the critical rate-limiting events are covered.

The limitations of the AOP idea lie in its strict definition as a linear monodirectional pathway from initiating event to adverse outcome. Neither physiology nor toxicity can be sufficiently described in a linear manner. The example of hormone homeostasis is pertinent here, as it is dependent on feedback loops that neither fit in a linear nor in a monodirectional model. In addition, several different adverse health effects may occur after triggering a single initiating event, for instance, depending on the exposure dose and duration or the life stage exposed.

These notions point at the necessity to combine AOP knowledge at the level of an integrated ontology to be able to provide sufficient basis for reliable overall toxicity predictions. Sturla et al.³³ describe how the molecular events in AOPs can be combined with mathematical models for the various steps, as well as high-dimensional data, such as transcriptomics, to perform risk assessment at a systems toxicology level. Moreover, combining toxicological knowledge as an ontology will capture this knowledge in a standardized and computer-readable form, which greatly facilitates translation of single assays to the level of the whole organism.

Xenobiotic effects are initiated by interaction with a subject on the molecular level, in AOP terms, the molecular initiating event (MIE). Following the perturbation of a downstream network of subjects, this ultimately results in effects on the organism level. The quantitative relationship between two subjects at a certain stage in development can be different from that in another stage during development. As an example, one important process in embryo development is the patterning along the anterior-posterior axis and the dorsal-ventral axis,³⁴ which is established by gradients of different morphogens over time. Concentration gradients of FGF8 and RA initiate the expression of Hox genes dependent on developmental stage and location in the embryo, and are crucial for normal development of the nervous system.³⁵ Disturbances of these gradients may result in malformations depending on the timing of disturbance.³⁶ Therefore, in developmental toxicology, the ontology should be modeled in a time- and space-dependent manner.

Similarly, age dependency of quantitative relationships throughout the life cycle need to be taken into account in building the ontology. A way to achieve this is to capture the functional relationships in formal ontology classes, while allowing adapting quantitative relationships (e.g., receptor binding) to reflect characteristics due to developmental stage, tissue, or species.

From ontology to systems toxicology computer model

An essential requirement for rendering the ontology applicable to chemical hazard assessment is the representation of the toxicological network of subjects and quantitative relationships in a systems toxicological software model to allow an integrated interpretation of the data. This model should enable experimental in silico modification of parameters based on the outcome of testing chemical effects on the critical key events in in vitro assays, leading to an in silico (computer) prediction of the toxicity at the level of the organism.

A wealth of computer models restricted to defined areas of toxicological prediction is already available. Receptor binding can be measured with specific assays or could be determined by in silico modeling.^37,38 Also, statistical tools may be applied to predict adverse outcomes from chemical structure.^39–41 This information gives a rough direction of the information that should be collected to assess the adverse outcome.

For some endpoints, quantitative structure-activity relationship ((Q)SAR) information could be an important determinant of adverse effects.⁴² Parnakonos et al. assessed the relationship between chemical structure and adverse outcome to determine mode of action.⁴³ Based on literature mining, the authors were able to identify important modes of action for carcinogenesis. Shah et al. applied a similar approach, using chemical information for read-across and combined this with data from the ToxCast test battery to assess repeated dose toxicity.⁴⁴ Wu et al. have published a framework for prediction of developmental and reproductive toxicity based on chemical structure.⁴⁵

Computer models in toxicology or efficacy research have also been used to identify new drug targets or assess the effect of a combination of compounds.⁴⁶ Computer simulation models could be used to model as well as visualize the adverse outcome. An example of such visualization is the development of the heart during embryogenesis,⁴⁷ focusing on the process of trabeculation, which is the formation of muscular strands during chamber development. An experimental change represented by changing parameter settings in the model, based on knowledge of compound effects on these parameters in dedicated in vitro assays, could be used to model and visualize an adverse outcome, providing evidence for the toxicity of a compound on a specific endpoint at the organism level.

The Virtual Embryo project at US-EPA has the ambitious goal to develop “a working computer model of a mammalian embryo that can be used to better understand the prenatal risks by environmental chemicals and to eventually predict a chemical's potential developmental toxicity in silico.” As an example, in this project, a model has been developed for secondary plate fusion and disruption.⁴⁸ Perturbation of this model has been shown to predict adverse effects induced by chemical interaction in a dose-related manner. In addition, this model has given insight into molecular mechanisms leading to adverse effects and into the flexibility of the system to restore disturbances. This shows that homeostasis can be built into the computer model and is important in the assessment of adverse outcomes.

Kleinstreuer et al. have designed a computational model for the androgen activity based on a pathway of key events, for each of which one or more in vitro assays were identified. Compound effects in these in vitro assays can be fed into the model to predict the overall outcome.⁴⁹ Other Virtual Embryo project computational models include those for blood vessel development and genital tubercle development.^50,51

These models are built on an ontology defining the cell types contributing to the developmental process and on a description of their interaction based on cell-specific characteristics represented by the specific expression of functionally related genes. Thus, the ontology provides information about genes, molecules, and cells and their quantitative interaction in a structured way, allowing computational modeling of the developmental process. Moreover, parameter settings can be varied in the model, simulating observations in dedicated in vitro assays, allowing the model to predict effects at the apical level of the developmental process at large.

Design and Application of Ontologies

Required information to build an ontology

There are multiple information sources for filling the developmental ontology matrix. An important source is provided by the extensive and rapidly expanding knowledge base in the field of developmental biology. The understanding of embryo development at the molecular level has grown tremendously in the past half century. However, presently this knowledge is only scarcely used for understanding mechanisms of chemical-induced dysmorphogenesis.

The high level of conservation of vertebrate embryogenesis between species allows for the use of existing knowledge from a variety of species. The retinoid pathway provides just one of many examples in which developmental biology has provided crucial information for toxicity modeling.⁵² Human syndromes driven by genetic defects,^53–55 as well as knockout mouse models, like for enzymes involved in maintaining retinoic acid balance,⁵⁶ provide information sources for developmentally relevant genes and their functionality in embryogenesis. Compiling this information in an ontology provides the biological backbone for determining pathways relevant for toxicology, that is, AOPs.

A second information source to be considered is related to chemistry. There is a large and growing database of physicochemical properties and structural characteristics in relationship to the toxicity of chemicals. Wu et al.⁴⁵ have used this information to design a stepwise approach of determining the likelihood of a compound being a developmental toxicant based on structural characteristics. Their model has been applied to over 800 chemicals with promising results. Structure-activity relationships continue to be extensively used in predicting toxicity of chemicals for regulatory purposes. Such information can be used to prioritize the study of toxicity pathways anticipated to be affected by the compound under study.

The third information source is provided by toxicology. The ontology and computational toxicology approach provides a new paradigm for toxicity testing, ultimately avoiding the detour of animal studies for human risk assessment. Nevertheless, the vast amount of existing data available from animal and alternative in vitro testing provides an important data source for filling the ontology. As an example, the ToxCast library contains hundreds of high-throughput assays informing about molecular and cellular effects of chemicals.⁵⁷ Other cell, tissue, organ, and whole embryo cultures also have provided relevant data that can be used to build the ontology.

Thus, the ontology is expanded by combining existing knowledge from biology, chemistry, and toxicology. Combining this information from all sources in practice is complex, but could build upon existing initiatives using a (computational) method to connect systems and combine date from different fields. As the ontology compiles the network of toxicological pathways, it allows the formulation of computational models describing the system in a quantitative way. Interactions of genes, proteins, and cells can be modeled quantitatively. Critical events in this process, defined as rate-limiting as to the occurrence of adverse effects, should be identified from the available knowledge. The challenge will be in the selection of these critical parameters and their representation in a battery of relevant in vitro models. These in vitro models will allow the study of the perturbation of the critical parameters by chemical exposure in a concentration-response manner, allowing identification of possible relevant adverse outcomes for a specific chemical.

Follow-up in vitro assays conducted in a step-wise approach, which are indicative for parameters downstream in the pathway, will allow identification of the possible adverse outcome. Quantitative information is collected using in vitro assays, which can be fed back into the computational model, which will then integrate all compound effects in vitro to calculate toxicity prediction at the level of the intact organism.

Important aspects to consider for implementation

For validation of the model, case studies should investigate the consequences for adverse outcome after experimental modification of parameters in the model, and its similarity to outcomes known from existing experimental or clinical data. This may be assessed, for example, using knockout data from animal studies or polymorphisms in humans. Compounds for which the mechanism of action is known, or for which in vitro assay data and classical toxicological data are available, can be used as cases for validation of the model. Especially, compounds with a different mechanism of action in humans and animals would be interesting for validation, and can be employed to show the added value of the ontology approach. Finally, confidence in sufficient coverage of relevant AOPs for man and their interaction in the ontology and the systems toxicology computer model derived from it, should drive the transition from classical animal experiments to in vitro and in silico animal-free chemical hazard and risk assessment.

Another crucial aspect of in vitro-based toxicology that would need to be covered in ontology-based in silico models is the extrapolation of in vitro effective concentrations to effective doses in the in vivo situation.⁵⁸ This challenge is being addressed in quantitative in vitro to in vivo extrapolation (QIVIVE) models, also referred to as reverse dosimetry, and significant expertise has already been gained in this area.^59,60 It fills the gap between external exposure at the level of the intact organism and the compound concentration at the target site(s) within the organism where the MIE is/are triggered (Fig. 2). The ontology network then tracks the pathway from the MIE at the molecular level up to the adverse effect at the organism level.

FIG. 2.

The road from external exposure to adverse health effect at the organism level. External exposure at the organism level leads by kinetic modeling (Absorption, Metabolism, Distribution, Excretion [ADME]), internal exposure of the target tissue, and triggering of key events at the molecular level and ontology-driven computational modeling via the Adverse Outcome Pathway (AOP) toward adverse health effect (hazard) at the organism level.

In summary, current state of the art in a variety of contributing disciplines do offer opportunities for realizing an integrated ontology-driven systems toxicology approach based on in vitro and in silico testing without animal experimentation. Advances in biology, chemistry, toxicology, ontology, and computational science areas have provided proof of principle for each and every aspect of this innovative approach to chemical hazard and risk assessment. The challenge lies in the integration of knowledge to the level of detail necessary to provide reliable toxicity predictions.

The fundamental advantage of this biology-driven approach is that it is in principle comprehensive as to coverage of toxicity pathways. The classical bottom-up approach of using single or combined in vitro assays to predict toxicity lacks comprehensiveness as it was driven by available assays. The innovative top-down approach uses a biology-driven selection of assays covering all necessary aspects of toxicity testing. This strategy can also be fine-tuned to the human situation. Thus, classical issues of biological and toxicological domain covered and necessary interspecies extrapolation are avoided, facilitating regulatory acceptance.

There is a long way ahead of course in building the integral system, requiring significant investments in time and funding. Current activities such as those toward understanding developmental biology and dysmorphogenesis at the molecular level, expanding the ToxCast library of assays and test data, modeling chemical fate in vitro and in vivo, and collecting AOPs in the OECD-wiki project give a wealth of relevant information for ontology development. These activities are ready to be integrated in a developmental ontology at the basis of a general systems toxicology model for the prediction of chemical-induced human developmental toxicity.

Footnotes

Author Disclosure Statement

No competing financial interests exist

References

Ito

, Ando

, Handa

. Teratogenic effects of thalidomide: Molecular mechanisms. Cell Mol Life Sci, 2011:68; 1569–1579.

Brent

. Drug testing in animals for teratogenic effects. Thalidomide in the pregnant rat. J Pediatr, 1964:64; 762–770.

Wall

, Shani

. Are animal models as good as we think?. Theriogenology, 2008:69; 2–9.

Caldwell

. Problems and opportunities in toxicity testing arising from species differences in xenobiotic metabolism. Toxicol Lett, 1992:64–65 Spec No:651–659.

Roberts

, Schultz

, Api

. Chemical applicability domain of the Local Lymph Node Assay (LLNA) for skin sensitisation potency. Part 3. Apparent discrepancies between LLNA and GPMT sensitisation potential: False positives or differences in sensitivity?. Regul Toxicol Pharmacol, 2016:80; 260–267.

Starokozhko

, Groothuis

. Judging the value of ‘liver-on-a-chip’ devices for prediction of toxicity. Expert Opin Drug Metab Toxicol, 2017:13; 125–128.

Nagahori

, Suzuki

, Le Coz

, et al. Prediction of in vivo developmental toxicity by combination of Hand1-Luc embryonic stem cell test and metabolic stability test with clarification of metabolically inapplicable candidates. Toxicol Lett, 2016:259; 44–51.

Liu

, Yu

, Wallqvist

. Using chemical-induced gene expression in cultured human cells to predict chemical toxicity. Chem Res Toxicol, 2016:29; 1883–1893.

Kanki

, Gi

, Fujioka

, et al. Detection of non-genotoxic hepatocarcinogens and prediction of their mechanism of action in rats using gene marker sets. J Toxicol Sci, 2016:41; 281–292.

10.

Ezendam

, Braakhuis

, Vandebriel

. State of the art in non-animal approaches for skin sensitization testing: From individual test methods towards testing strategies. Arch Toxicol, 2016:90; 2861–2883.

11.

Coecke

, Goldberg

, Allen

, et al. Workgroup report: Incorporating in vitro alternative methods for developmental neurotoxicity into international hazard and risk assessment strategies. Environ Health Perspect, 2007:115; 924–931.

12.

Ford

. Refinement, reduction, and replacement of animal toxicity tests by computational methods. ILAR J, 2016:57; 226–233.

13.

Pizzo

, Benfenati

. In silico models for repeated-dose toxicity (RDT): Prediction of the no observed adverse effect level (NOAEL) and lowest observed adverse effect level (LOAEL) for drugs. Methods Mol Biol, 2016:1425; 163–176.

14.

European Centre for Ecotoxicology and Toxicology of Chemicals. Building a prenatal developmental toxicity ontology. ECETOC AISBL. Special Report 19, August 2016.

15.

National Academies of Sciences Engineering Medicine, Division on Engineering, et al.. Using 21st Century Science to Improve Risk-Related Evaluations, National Academies Press (US) Copyright 2017 by the National Academy of Sciences. All rights reserved. Washington (DC), 2017.

16.

Hartung

, FitzGerald

, Jennings

, et al. Systems toxicology: real world applications and opportunities. Chem Res Toxicol, 2017:30; 870–882.

17.

Ashburner

, Ball

, Blake

, et al. Gene ontology: Tool for the unification of biology. The Gene Ontology Consortium. Nat Genet, 2000:25; 25–29.

18.

Blake

, Dolan

, Drabkin

, et al. Gene ontology annotations and resources. Nucleic Acids Res, 2013:41; D530–D535.

19.

Hucka

, Finney

, Sauro

, et al. The systems biology markup language (SBML): A medium for representation and exchange of biochemical network models. Bioinformatics, 2003:19; 524–531.

20.

Demir

, Cary

, Paley

, et al. The BioPAX community standard for pathway data sharing. Nat Biotechnol, 2010:28; 935–942.

21.

Gkoutos

, Schofield

, Hoehndorf

. The anatomy of phenotype ontologies: Principles, properties and applications. Brief Bioinform, 2017 [Epub ahead of print]; DOI: 10.1093/bib/bbx035.

22.

Smith

, Ashburner

, Rosse

, et al. The OBO Foundry: Coordinated evolution of ontologies to support biomedical data integration. Nat Biotechnol, 2007:25; 1251–1255.

23.

Thibault

, Roe

, Eilbeck

, et al. Development of an informatics infrastructure for data exchange of biomolecular simulations: Architecture, data models and ontology. SAR QSAR Environ Res, 2015:26; 577–593.

24.

Hardy

, Douglas

, Helma

, et al. Collaborative development of predictive toxicology applications. J Cheminform, 2010:2; 7.

25.

Burgoon

, Druwe

, Painter

, et al. Using in vitro high-throughput screening data for predicting benzo[k]fluoranthene human health hazards. Risk Anal, 2017:37; 280–290.

26.

US National Academy of Sciences. Toxicity testing in the twenty-first century: A vision and a strategy. Washington, DC, National Academies Press, 2007.

27.

Tollefsen

, Scholz

, Cronin

, et al. Applying adverse outcome pathways (AOPs) to support integrated approaches to testing and assessment (IATA). Regul Toxicol Pharmacol, 2014:70; 629–640.

28.

OECD Guidance Document on an Integrated Approach on Testing and Assessment (IATA) for Skin Corrosion and Irritation. Paris: OECD Publishing; 2017. http://dx.doi.org/10.1787/9789264274693-en

29.

OECD Guidance Document for the Use of Adverse Outcome Pathways in Developing Integrated Approaches to Testing and Assessment (IATA). OECD Series on Testing and Assessment No. 260 ENV/JM/MONO(2016)6.

30.

Conolly

, Ankley

, Cheng

, et al. Quantitative adverse outcome pathways and their application to predictive toxicology. Environ Sci Technol, 2017:51; 4661–4672.

31.

Knapen

, Vergauwen

, Villeneuve

, et al. The potential of AOP networks for reproductive and developmental toxicity assay development. Reprod Toxicol, 2015:56; 52–55.

32.

Villeneuve

, Crump

, Garcia-Reyero

, et al. Adverse outcome pathway (AOP) development I: Strategies and principles. Toxicol Sci, 2014:142; 312–320.

33.

Sturla

, Boobis

, FitzGerald

, et al. Systems toxicology: From basic research to risk assessment. Chem Res Toxicol, 2014:27; 314–329.

34.

Tuazon

, Mullins

. Temporally coordinated signals progressively pattern the anteroposterior and dorsoventral body axes. Semin Cell Dev Biol, 2015:42; 118–133.

35.

Bel-Vialar

, Itasaki

, Krumlauf

. Initiating Hox gene expression: In the early chick neural tube differential sensitivity to FGF and RA signaling subdivides the HoxB genes in two distinct groups. Development, 2002:129; 5103–5115.

36.

Tibbles

, Wiley

. A comparative study of the effects of retinoic acid given during the critical period for inducing spina bifida in mice and hamsters. Teratology, 1988:37; 113–125.

37.

Banerjee

, Siramshetty

, Drwal

, et al. Computational methods for prediction of in vitro effects of new chemical structures. J Cheminform, 2016:8; 51.

38.

Ruiz

, Sack

, Wampole

, et al. Integration of in silico methods and computational systems biology to explore endocrine-disrupting chemical binding with nuclear hormone receptors. Chemosphere, 2017:178; 99–109.

39.

Carriger

, Martin

, Barron

. A Bayesian network model for predicting aquatic toxicity mode of action using two dimensional theoretical molecular descriptors. Aquat Toxicol, 2016:180; 11–24.

40.

Mulliner

, Schmidt

, Stolte

, et al. Computational models for human and animal hepatotoxicity with a global application scope. Chem Res Toxicol, 2016:29; 757–767.

41.

Zhang

, Cao

, Li

, et al. Novel naive Bayes classification models for predicting the carcinogenicity of chemicals. Food Chem Toxicol, 2016:97; 141–149.

42.

Benigni

, Bossa

, Tcheremenskaia

. A data-based exploration of the adverse outcome pathway for skin sensitization points to the necessary requirements for its prediction with alternative methods. Regul Toxicol Pharmacol, 2016:78; 45–52.

43.

Papamokos

, Silins

. Combining QSAR modeling and text-mining techniques to link chemical structures and carcinogenic modes of action. Front Pharmacol, 2016:7; 284.

44.

Shah

, Liu

, Judson

, et al. Systematically evaluating read-across prediction and performance using a local validity approach characterized by chemical structure and bioactivity information. Regul Toxicol Pharmacol, 2016:79; 12–24.

45.

, Fisher

, Naciff

, et al. Framework for identifying chemicals with structural features associated with the potential to act as developmental or reproductive toxicants. Chem Res Toxicol, 2013:26; 1840–1861.

46.

Xie

, Li

, Chen

, et al. Identification of potential drug targets based on a computational biology algorithm for venous thromboembolism. Int J Mol Med, 2017:39; 463–471.

47.

de Boer

, Le Garrec

, Christoffels

, et al. Integrating multi-scale knowledge on cardiac development into a computational model of ventricular trabeculation. Wiley Interdiscip Rev Syst Biol Med, 2014:6; 389–397.

48.

Hutson

, Leung

, Baker

, et al. Computational model of secondary palate fusion and disruption. Chem Res Toxicol, 2017:30; 965–979.

49.

Kleinstreuer

, Ceger

, Watt

, et al. Development and validation of a computational model for androgen receptor activity. Chem Res Toxicol, 2017:30; 946–964.

50.

Kleinstreuer

, Dix

, Rountree

, et al. A computational model predicting disruption of blood vessel development. PLoS Computat Biol, 2013:9; e1002996.

51.

Leung

MCK

, Hutson

, Seifert

, et al. Computational modeling and simulation of genital tubercle development. Reprod Toxicol, 2016:64; 151–161.

52.

Tonk

ECM

, Pennings

JLA

, Piersma

. An adverse outcome pathway framework for neural tube and axial defects mediated by modulation of retinoic acid homeostasis. Reprod Toxicol, 2015:55; 104–113.

53.

Bertolacini

, Richieri-Costa

, Ribeiro-Bicudo

. Sonic hedgehog (SHH) mutation in patients within the spectrum of holoprosencephaly. Brain Dev, 2010:32; 217–222.

54.

Guo

, Zhao

, Pei

, et al. Single nucleotide polymorphisms of the maternal Msx2 gene and their association with fetal neural tube defects in Han ethnic group in Shanxi Province, China. Chin Med J, 2011:124; 374–379.

55.

Webber

, MacLeod

, Bamshad

, et al. Developments in our understanding of the genetic basis of birth defects. Birth Defects Res A Clin Mol Teratol, 2015:103; 680–691.

56.

Rhinn

, Dolle

. Retinoic acid signalling during development. Development, 2012:139; 843–858.

57.

Kavlock

, Chandler

, Houck

, et al. Update on EPA's ToxCast program: Providing high throughput decision support tools for chemical risk management. Chem Res Toxicol, 2012:25; 1287–1302.

58.

Cohen Hubal

, Richard

, Shah

, et al. Exposure science and the U.S. EPA National Center for Computational Toxicology. J Expo Sci Environ Epidemiol, 2010:20; 231–236.

59.

Yoon

, Blaauboer

, Clewell

. Quantitative in vitro to in vivo extrapolation (QIVIVE): An essential element for in vitro-based risk assessment. Toxicology, 2015:332; 1–3.

60.

Louisse

, Beekmann

, Rietjens

IMCM

. Use of physiologically based kinetic modeling-based reverse dosimetry to predict in vivo toxicity from in vitro data. Chem Res Toxicol, 2017:30; 114–125.