Appropriate Evidence Sources for Populating Decision Analytic Models within Health Technology Assessment (HTA)

Abstract

Background. An increasing number of evidence sources are relevant for populating decision analytic models. What is needed is detailed methodological advice on which type of data is to be used for what type of model parameter. Purpose. We aim to identify standards in health technology assessment manuals and economic (modeling) guidelines on appropriate evidence sources and on the role different types of data play within a model. Methods. Documents were identified via a call among members of the International Network of Agencies for Health Technology Assessment and by hand search. We included documents from Europe, the United States, Canada, Australia, and New Zealand as well as transnational guidelines written in English or German. We systematically summarized in a narrative manner information on appropriate evidence sources for model parameters, their advantages and limitations, data identification methods, and data quality issues. Results. A large variety of evidence sources for populating models are mentioned in the 28 documents included. They comprise research- and non–research-based sources. Valid and less appropriate sources are identified for informing different types of model parameters, such as clinical effect size, natural history of disease, resource use, unit costs, and health state utility values. Guidelines do not provide structured and detailed advice on this issue. Limitations. The article does not include information from guidelines in languages other than English or German, and the information is not tailored to specific modeling techniques. Conclusions. The usability of guidelines and manuals for modeling could be improved by addressing the issue of evidence sources in a more structured and comprehensive format.

Keywords

decision analytic modeling health technology assessment systematic review evidence sources

Introduction

Decision analytic modelers need data for populating model parameters. Yet modeling has become ever more challenging, as the number of potential sources of evidence to be used in a model is constantly rising and goes beyond published literature.

Modeling guidelines and textbooks agree that the data quality of parameters used in a model is a crucial factor for overall model quality.^1,2–5 However, research has demonstrated several limitations relating to model parameters, such as inadequate reporting on the identification of evidence, lack of quality assessment of the evidence used, or lack of justification as to how evidence was selected.⁶

In modeling projects, particularly in countries where decision analytic modeling has not yet been formally established or in (inter)national interdisciplinary modeling networks, participants may have a different level of knowledge about available data sources and about how to use different types of evidence appropriately within a model. Furthermore, country-specific requirements concerning what type of evidence to use may exist.

Decision analytic modeling is often used within health technology assessment (HTA). The aim of this article, therefore, is to review existing HTA manuals and health economic (modeling) guidelines to gain advice on appropriate evidence sources for populating models.

Methods

We analyzed country-specific and general (transnational) HTA manuals and (modeling) guidelines on economic evaluation and systematically extracted 6 types of information that are of practical relevance in the process of populating a model: 1) recommendations on where or how to find evidence, 2) potential evidence sources stratified by appropriateness for different model elements (e.g., clinical effect size, resource use), 3) required data characteristics, 4) data quality issues raised, 5) advantages, and 6) limitations of evidence sources.

We used 3 information sources to retrieve relevant documents: 1) overview on country-specific health economic guidelines by the International Society of Pharmacoeconomics and Outcome Research (searched in January 2013),⁷ 2) responses from a call among the 46 members of the International Network of Agencies for Health Technology Assessment in April 2011, and 3) Web sites of HTA agencies and of systematic review database providers (Cochrane Collaboration, Centre for Review and Dissemination) in April 2011. This was supplemented by a Web search for relevant methodological documents in January 2012 and August 2012, as well as a hand search in the guidelines’ reference lists.

We included documents in English or German that are applicable for health care systems in developed countries. The restriction to developed countries was chosen because we assumed that recommendations on evidence sources would be more transferable across similarly developed health care systems, especially in the case of country-specific sources such as administrative data. We selected country-specific documents (from any European country, the United States, Canada, Australia, and New Zealand) and guidelines that have been developed by transnational expert groups. Another predefined selection criterion was that modeling needed to be addressed in the document per se and that information on sources of evidence for modeling was provided in the modeling chapter directly or elsewhere in the document (e.g., in the costing section). Furthermore, recommendations on evidence sources needed to be relevant for at least 1 of 5 common elements within a model⁵: clinical effect size, natural history of disease, resource use, unit costs, and health state utility values (HSUVs). We excluded documents that did not specify or not even mention modeling. Literature selection and data extraction were carried out independently by 2 researchers.

We systematically summarized the extracted data in the following way. First, we quantitatively and qualitatively described to what extent the 6 types of information are provided in the documents. Second, we summarized the advice on what type of evidence is to be used for the aforementioned 5 core model elements.^6,8 This gave us a list of potential evidence sources for the 5 model elements. Furthermore, we analyzed the origin of these evidence sources and classified them into research- and non–research-based sources. Research-based evidence sources contain data that have actively been generated for research purposes following a systematic process of collecting and analyzing information. Non–research-based evidence sources are data that have been primarily collected for purposes other than research (e.g., administrative data, marketing) or that represent subjective opinions (e.g., expert opinions). This categorization should support modelers in finding the data (e.g., research-based sources will primarily be identified in bibliographical databases; non–research-based sources may have to be obtained from health administrations) and in assessing their validity for different model experiments.

The funding of the study had no role in the method applied nor did it influence the research process.

Results

Identified Literature

Overall, we retrieved 56 references; 28 documents fulfilled our inclusion criteria (Figure 1; Table 1). We excluded the remaining 28 documents because either modeling or data sources were not specified (n = 17), they came from excluded countries (n = 9), they were written in a language other than German or English (n = 1), or they were not manuals (n = 1; see Appendix Table 2). Appendix Table 1 presents the extracted information from the included documents. We summarize this information in Table 2.

Figure 1

PRISMA flow diagram for the selection of literature.

Table 1

Included Documents (Ordered according to Year Published)

Author (Year)	Country	Short Title
Alves de Silva and others (1998)⁹	Portugal	Guidelines for Economic Drug Evaluations Studies
Szende and others (2002)¹⁰	Hungary	Methodological Guidelines for Conducting Economic Evaluation of Healthcare Interventions in Hungary
Weinstein and others (2003)¹	Transnational	Principles of Good Practice for Decision Analytic Modelling in Health-Care Evaluation
World Health Organization (2003)¹¹	Transnational	Making Choices in Health: WHO-Guide to Cost-Effectiveness Analysis
Philips and others (2004)⁴	Transnational	A Review of Guidelines for Good Practice in Decision-Analytical Modelling in Health Technology Assessment
The French Health Economists Association (2004)¹²	France	French Guideline for the Economic Evaluation of Health Care Technologies
Canadian Agency for Drugs and Technologies in Health (2006) ¹³	Canada	Guidelines for the Economic Evaluation of Health Technologies
College voor zorverzekeringen (2006)¹⁴	The Netherlands	Guidelines for Pharmacoeconomic Research
PHARMAC (2007)¹⁵	New Zealand	Prescription for Pharmacoeconomic Analysis
Medical Service Advisory Committee (MSAC) (2008)¹⁶	Australia	Economic Section of the MSAC Guidelines
Pharmaceutical Benefit Advisory Board (PBAC) (2008)¹⁷	Australia	Guidelines for Preparing Submissions to the Pharmaceutical Benefits Advisory Committee
National Institute for Health and Clinical Excellence (2008)¹⁸	England, Wales	Guide to the Methods of Technology Appraisal
Belgian Health Care Knowledge Centre (2008)¹⁹	Belgium	Guideline for Pharmacoeconomic Evaluation in Belgium
Agency for Health Technology Assessment (2009)²⁰	Poland	Guidelines for Conducting Health Technology Assessment
Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen (2009)²¹	Germany	Arbeitspapier Modellierung
Academy of Managed Care (2009)²²	United States	The AMCP Format for Formulary Submissions
Health Information and Quality Authority (2010)²³	Ireland	Guidelines for the Economic Evaluation of Health Technologies in Ireland
Papaioannou and others (2010)²⁴	England, Wales	NICE DSU Technical Support Documents 9: The Identification, Review, and Synthesis of Health State Utility Values from the Literature
Kaltenthaler and others (2011)²⁵	England, Wales	NICE DSU Technical Support Document 13: Identifying and Reviewing Evidence to Inform the Conceptualisation and Population of Cost-Effectiveness Models
Dias and others (2013)²⁶	England, Wales	NICE DSU Technical Support Document 5: Evidence Synthesis in the Baseline Natural History Model
Norwegian Medicines Agency (2012)²⁷	Norway	Guidelines on How to Conduct Pharmacoeconomic Analyses
Gesundheit Österreich GmbH (GÖG) (2012)²⁸	Austria	Methodenhandbuch für Health Technology Assessment
Scottish Medicines Consortium (2012)²⁹	Scotland	Guidance to Manufacturers for Completion of New Product Assessment FORM (NPAF)
Briggs and others (2012)³⁰	Transnational	Model Parameter Estimation and Uncertainty Analysis
Caro and others (2012)³¹	Transnational	Modeling Good Research Practices—Overview
Pitman and others (2012)³²	Transnational	Dynamic Transmission Modeling
Siebert and others (2012)³³	Transnational	State-Transition Modeling
Karnon and others (2012)³⁴	Transnational	Modeling Using Discrete Event Simulation

Table 2

Types of Information Covered in Guidelines and Manuals

Type of Information	n (28)	%^a	References
Evidence identification	16	57	1, 4, 10, 13, 15, 18 –25, 27, 30, 31
Potential evidence sources for model elements
Clinical effect size	12	46^b	9 –11, 15, 18, 20, 22, 23, 27 –29, 32, 33
Baseline clinic/natural history of disease	11	41^c	4, 9, 11, 15, 17, 18, 22, 26, 28, 29, 32, 33
Resource use	18	69^b	4, 9, 10, 12, 13, 15 –20, 22, 23, 27 –29, 34
Unit costs	12	46^b	10 –13, 15 –20, 23, 29
Health state utility value	13	48^d	9, 11, 13 –20, 24, 27, 29
All elements	3	12	15, 18, 29
Required data characteristics/nature of evidence	9	32	1, 4, 9, 13, 18, 19, 21, 24, 30
Quality issues raised	26	93	1, 4, 9 –11, 13, 14, 16 –34
Advantages of evidence sources	12	43	4, 11, 13 –15, 19 –21, 27, 31, 32, 34
Limitations of evidence sources	11	39	4, 11, 12, 15, 18, 21, 24, 25, 32 –34

Documents that focused on specific model elements only (e.g., on health state utility values²⁴) were excluded from total counts for the calculation of percentages.

References 24 and 26 were excluded from total counts for calculation of percentages.

Reference 24 was excluded from total counts for calculation of percentages.

Reference 26 was excluded from total counts for calculation of percentages.

Two of the 28 selected documents^4,21 address all 6 information items in question. Slightly more than half of the documents address the issue of evidence identification and selection. Although all the documents give some information on evidence sources that may be potentially useful for populating models, only 3 documents^15,18,29 address all five 5 of core model elements that we defined as relevant in the method selection. The remaining documents provide information on some but not all core model elements (e.g., on resource use but not on clinical effect size). The majority of the documents (93%) address quality issues to some extent, and about 40% discuss advantages and limitations of different evidence sources.

In qualitative terms, information on the 6 information items is presented in an unsystematic way. For example, quality issues and limitations are discussed for selected evidence sources only (e.g., for expert opinions). In addition, advice is presented only in a brief way. For example, although the German document²¹ addresses all 6 items in question, recommendations on how/where to identify evidence, on what types of model parameters their suggested sources are appropriate for, and on required data characteristics are limited to short general statements (e.g., “The required nature of the evidence depends on the model parameter”).

In the following sections, the results of all 6 information items of interest will be summarized, and guidelines that provide the most detailed and helpful information on single items will be described.

Evidence Identification and Selection

Across all of the documents, the standard method for identifying data for model parameters is a systematic (in contrast to selective) search for evidence, to minimize (selection) bias³⁵ as much as possible. Documents agree that the search for any model data needs to be presented transparently and the sources used need to be stated. The most comprehensive information is provided by Philips and others⁴ and by Kaltenhaler and others.²⁵

Documents vary in the extent to which model parameters need to be based on a systematic search. Several manuals limit the requirement for a systematic search to clinical studies¹⁹ or to key model parameters.^4,20,22 Key parameters are usually efficacy or effectiveness^10,19,23,27 or those that are expected to have the largest influence on the model results.⁴

Some documents offer advice on valid bibliographical sources or provide guidance to define searchable questions.^4,15,25 Yet it has also been acknowledged that there is a lack of methodological support to do a systematic search for evidence other than clinical effect size. For example, difficulties exist when searching for published HSUVs²⁴ or for administrative data.²⁵ Some manuals specifically point out that a variety of model parameters need to be searched for in other sources besides biomedical databases.^24,25

Evidence Sources for Core Model Elements and Their Limitations

Generally, research-based sources have been mentioned more often as being useful for populating a model than non–research-based ones (Table 3). For model parameters related to the model elements “effect size,” “natural history of disease,” and “HSUVs,” the former plays the most important role. For defining parameters related to the model elements “resource use” and “unit costs,” on the other hand, the documents recommend non–research-based sources. Some evidence sources (e.g., disease registries, compiled national statistics) cannot be clearly assigned into 1 of the 2 categories as data could have been collected in a clearly defined research project or for purposes other than research (e.g., administration).

Table 3

Evidence Sources for Different Model Elements according to Manuals and Guidelines

	Evidence Sources
Elements within a Model	Research Based^a	Non–research based^b
Clinical effect size: efficacy, (long-term) effectiveness/health benefit/health outcome, safety, surrogate outcomes (e.g., test accuracy in diagnostics)	Efficacy, safety: (published) clinical trials and their meta-analyses/systematic reviews^{11,18,22,28,29,33}; effectiveness, safety: (published) (large-scale real-life) RCTs,^{9,10,15,18,20,22,27,32,33} observational studies (e.g., cohort studies, case reports),^{9,15,20,22,27,33} nonrandomized trials,¹⁸ cost-effectiveness studies,¹⁰ coherence/adherence studies,²⁸ unpublished data from clinical studies¹⁵; surrogates: test-accuracy studies²⁸	Effectiveness^c: expert opinions,^9,10,15 medical records,¹⁵ surveys of treated prevalence,¹⁷ utilization databases¹⁷
Baseline clinic/natural history of disease	Observational studies (e.g., cohort studies)^4,11,18,29; population-, regional-, or hospital-based epidemiological studies^9,28; control arms of clinical trials^4,11,33; subtypes of observational studies (prognostic studies, risk factor studies)²⁸; serological studies³²; cost-effectiveness studies²⁶; registry studies²⁶; hazard rates from underserved populations with limited access to health care¹¹	Expert opinions^c26
	Compiled national statistics^d (life tables for all-cause mortality),^15,22,28 large population-based surveys,^17,32 public health surveillance data,³² disease registries,²⁸ case registries,¹⁷ mortality registries,¹⁷ registries²¹
Resource use	Clinical trials,^13,19,23 epidemiological studies,⁹ cross-sectional studies,^9,18,29 foreign health economic studies,¹⁰ micro-costing studies¹⁸	Country-specific routine data,^{4,10,12,13,15–20,22,23,27–29} clinical practice guidelines,^13,23 patient records^19,34 expert opinions^{c12,13,15,19,23}
	Compiled statistics,^d10,18,29 primary data collection (e.g., ad hoc survey)^12,18,20,23
Unit costs		National sources on cost data (price lists, routine data/accounting data, reimbursement schemes such as DRG tariffs, benefit schemes, etc.)^{10,12,13,15,18–20,23,29}; WHO database of prices for traded goods and on useful life of capital goods¹¹
HSUVs: HRQL and its valuation	Clinical trials and other published research on HRQL,^{14,16–20,24,27,29} cost-effectiveness studies (registry),^20,24 databases on HRQL data,¹⁵ primary data collection (+ existing national data on health state valuation),^{9,11,13,14,18–20,29} global burden of disease studies (for DALYs)¹⁵

Note: DALY = disability-adjusted life year; DRG = diagnosis-related groups; HRQL= health-related quality of life; HSUV = health state utility value; RCT = randomized controlled trial; WHO = World Health Organization.

Evidence that has actively been generated for research purpose following a systematic process of collecting and analyzing information.

Data that have been primarily collected for purposes other than research (e.g., administrative data, marketing) or that represent subjective opinions (e.g., expert opinions).

Health statistics that have been compiled by governments from census or survey data.

Appropriate only when used in addition to other sources.

Almost all sources mentioned are secondary data sources in the sense that the data had been collected by someone other than the modeler, except for HSUVs, cost data, and for eliciting expert opinions where—in addition to secondary data—primary data collection has been suggested in some references.

Some evidence sources occurred frequently (e.g., observational studies for defining the natural history of diseases, published research on health-related quality of life [HRQL] for defining parameters on HSUV), whereas others appeared only once (e.g., using foreign health economic studies for defining resource use parameters).

Sources for Clinical Effect Size

A number of evidence sources have been suggested for defining clinical effect size in a model. Published experimental clinical studies (randomized controlled trials [RCTs])—belonging to the group of interventional studies (as opposed to observational studies)—and their systematic syntheses are regarded as first-choice data sources. However, manuals also recognize their main limitation, which is the restricted external validity and generalizability of results to the real-world setting.

For parameters in the safety of a technology, observational (clinical) studies are described as a valid source.²² In addition, they provide real-world data (for effectiveness) and data on compliance and may be used to adapt data from RCTs to real-world settings.^15,20,31,33 One manual²⁰ explicitly prefers observational studies to interventional ones for obtaining effectiveness data, although this has been questioned by others.^4,21,28 The stated disadvantage for data obtained from observational studies is that there is little control for confounding and certain types of bias, and they may be of limited use if they lack a control group.¹⁵

Although meta-analyses of RCTs are explicitly mentioned as an important data source for efficacy, effectiveness, and safety issues in those cases where more than 1 study is available,^9,19 the danger of using low-quality meta-analyses (e.g., those that include studies selectively or those that include heterogeneous studies) is also pointed out.¹⁵

Using expert opinion to define effectiveness is mentioned in 3 manuals, but only if no other data are available,¹⁰ to derive real-world estimates on the basis of published data on efficacy⁹ or to clarify unclear, conflicting, or insufficient information.¹⁵

Sources for Baseline Clinic/Natural History of Diseases

For parameters related to the natural history of diseases, several types of observational studies are the sources recommended most often. For example, 4 manuals refer to “epidemiological studies” (without providing a further definition) to extrapolate efficacy data derived from clinical trials or to obtain baseline probabilities.^4,9,28,33 Another manual mentions risk factor studies (to derive data on the incidence of future events).²⁸

Registries, which can also be categorized as observational studies in terms of study design, are found helpful for defining predictive functions that describe the association between clinical event and resource use, quality of care, sick leave, and so forth.^16,17,21,28

Furthermore, the control group of an interventional study may be used to obtain baseline probabilities on the natural history of disease,^4,33 and data from underserved populations with limited access to health care may be used to derive hazard rates.¹¹

Compiled statistics, such as data from the Central Statistical Office,¹⁰ “health care system data,”¹⁷ or national and regional health statistics,²⁸ are recommended as data sources for information on demography, health behavior, or risk factors. Moreover, they are the primary source to define all-cause mortality (from life tables). Data on health behavior can also be derived from large population-based surveys (e.g., on sexual behavior)³² that may not be included in nationally compiled health care statistics.

Three further sources that have been mentioned with respect to dynamic transmission modeling are serological studies, public health surveillance data, and large population-based surveys. For example, seroprevalence curves can be used to estimate the incidence of an infection, and the public health surveillance data or population-based surveys may provide information for contact patterns or on sexual behavior in the case of modeling sexually transmitted diseases.³²

Expert opinions in context with the natural history of disease have been specifically addressed in one manual,²⁶ yet again as a supplementary evidence source rather than a primary one.

Sources for Resource Use and Unit Costs

The most frequent sources stated in context with resource use and unit costs are non–research based. Most often, administrative data in various forms (documents refer to “claims data,” “retrospective data sets,” “national data on healthcare resource groups,” “validated databases,” “routine data and accounting databases,” “tariff lists,” etc.) are cited.

Authors mention it is an advantage that routine data contain information on a large number of patients and may be appropriate if the relevant target population in the model is sufficiently depicted in the data (via diagnostic codes). However, they may be problematic in use if the definition of the model population includes parameters (e.g., clinical results, specific patient characteristics) that are not coded in the database.

Additional data on resource use and/or costs can be derived from compiled statistics (e.g., data on median income to calculate productivity costs),⁴ from medical records,^19,34 or clinical practice guidelines.^13,23 These sources were not commented on further, except for clinical guidelines where authors point to the limitation that they may specify resources that ought to be consumed rather than their actual uptake.³⁴

In terms of research-based sources for obtaining data on resource use and unit costs, manuals mention observational studies,^9,18,29 clinical trials,^13,19,23 and economic studies.^10,18 When using clinical studies for costing information, frequent problems are mentioned. For example, they may not cover all relevant cost categories.²¹ One manual points out that the simple currency conversion of values found in the literature without evaluating their general transferability is not acceptable.¹⁹

Sources for HSUVs

Recommendations on sources for HSUVs are indifferent as to obtaining those data by primary data collection^{9,13,14,18–20,29} or by literature reviews.^{14,16–20,24,27,29} Another manual¹⁵ mentions existing databases on HRQL as a possible source. A number of challenges have been addressed when using values derived from the literature (e.g., when one set of values is better on some criteria but worse on others).²⁴

Quality Issues and Required Data Characteristics

Overall, the methodological guidelines stress that data have to be assessed in terms of relevance and quality before they are used for defining model parameters. For quality assessment, specific quality checklists and tools that exist for various types of data (e.g., data from clinical trials, data from cohort studies) are recommended in some manuals (e.g., Cochrane Risk of Bias tool, “Grading of Recommendations Assessment, Development and Evaluation” [GRADE] system²⁵). It has been acknowledged that the quality of some study types (e.g., on HSUVs) may be difficult to assess, for example, because no agreed reporting standards exist.²⁴ In any case, data limitations and how these were dealt with (e.g., by applying different types of sensitivity analyses) need to be described transparently.

Furthermore, the manuals’ authors point out that formal evidence synthesis techniques (meta-analysis, network meta-analysis) need to be employed when using data across studies^30,31; yet using them with caution when different populations, heterogeneity in geography, and so forth are present is stressed.^19,32 Moreover, an estimation of uncertainty that is consistent with standard statistical approaches should be integral to the process of estimating input parameters.^30,31

The role of expert opinions within a model is viewed differently. Yet there is consensus across all manuals that its use has to be justified and is accepted only as a “last resort” if no better data are available. Furthermore, all documents make clear that the use of expert opinions has to be described transparently (number of experts asked, selection process, etc.), and opinions should ideally be derived via formal methods such as Delphi or nominal group techniques. If parameters are defined on the basis of expert opinions, they must undergo sensitivity analyses to control for uncertainty. Furthermore, the collection of expert opinions may be a time-consuming task if done correctly.^1,4,17

Few documents^1,4,18,24,30 present details on required data characteristics. Weinstein and others¹ point out that ranges (i.e., upper and lower bounds) should accompany base case estimates of all input parameters for which (probabilistic) sensitivity analyses are performed. Similarly, the NICE guideline¹⁸ states that mean values should be presented for continuous variables and that measures of precision should be detailed for all variables. Finally, they highlight that data sources and results should not be rejected solely because they do not reach generally accepted but arbitrary probability cut points defining “statistical significance” (e.g., P < 0.05). For HSUVs, only 1 document provides information on the characteristics in terms of the valuation method required or the tools to describe health states that are to be used.²⁴

Discussion

We identified a large variety of potential evidence sources for modeling in the 28 documents analyzed; however, the topic is addressed in a fragmented way, detailed advice for core model parameters is missing in most of the documents, and recommendations are inconsistent to some degree across documents.

It becomes clear that most parameters in a model will not usually be based on a single data source but may require multiple types of information instead. For example, data from clinical studies may be supported by evidence from administrative medical records, surveys of treated prevalence, or utilization databases, mainly to gain information on the actual uptake or coherence/adherence of a technology that has an influence on the real-world health outcome.

There is consensus across the documents that data in a model should be based as much as possible on systematic searches. There is further consensus that the primary data sources for model parameters related to efficacy, effectiveness, and safety issues are RCTs and meta-analyses of RCTs. Observational studies are the primary data source for modeling the natural history of disease, and HSUVs should either be based on a thorough literature review or on primary data collection. For evidence on resource use and unit costs, researchers are primarily referred to non–research-based sources, which is because this information needs to reflect the jurisdiction of interest. Although the documents acknowledge the value of expert opinions, they also agree on their severe limitations.

There is less consensus on when evidence identification needs to be systematic, on data quality issues, and on the role of single evidence sources in a model (e.g., on the role of observational data for defining effectiveness or on the role of expert opinions).

Although a large number of potentially relevant data sources have been identified, neither the HTA manuals nor the (modeling) guidelines on economic evaluations address the issue of evidence sources systematically. With very few exceptions, the information is rather hidden in different sections of the documents and presented along the way, without paying specific attention to the issue. As the Appendix Table 2 of excluded manuals demonstrates, several manuals do not specify data sources at all. Furthermore, manuals provide only a selected overview of limitations or advantages of the evidence sources they recommend. For example, information is missing that administrative data usually do not reflect true costs or may be misleading due to wrong coding (e.g., upcoding). In several cases, sources are mentioned without critical reflection (e.g., using data from foreign cost-effectiveness studies for defining resource use parameters)¹⁰ and do not necessarily comply with standards in textbooks.³⁶ In transnational guidelines, one reason for these shortcomings may be that a number of model parameters require country-specific data that cannot be sensibly provided in a transnational document.

In previous research, attempts were made to support populating decision analytic modeling by defining hierarchies of data sources.^5,37 None of the guidelines give structured advice on data hierarchies, and only 1 document²⁵ refers to this type of research.

In addition, the terminology used for the different types of data lacks consistency and clear definition. For example, authors in many cases use the term clinical studies when they mean (randomized) controlled clinical trials (that is, experimental clinical studies). In other cases, evidence sources seem to have similar meanings but have been named differently (e.g., patient records and medical records).

The information in the manuals and guidelines does not cover all relevant purposes for which some data sources may be appropriate. As an example, administrative data may also be useful for identifying epidemiological dimensions of the disease in question in addition to retrieving resource use and cost information.

As a result, if modelers are looking for practical advice on how to populate decision-analytic models, they will have to consult several guidelines and additional sources to obtain information on the issues we have defined as relevant.

A limitation of our approach is that we restricted the search for information to documents written in English and German and to “Western” countries. This resulted in the exclusion of 10 (18%) of 56 documents retrieved overall. Some additional information may be available in national-language documents. Furthermore, we limited our search to identify evidence sources used to inform model parameters. We did not address the fact that evidence is also required to inform understanding of the decision problem, of the model structure, and of the analytic modeling approach.²⁵ Finally, the information we summarized in this article is generic and does not take into account specific data characteristics that may be required for certain modeling approaches (e.g., Markov models, agent-based models).

As an implication for guideline development, transnational modeling guidelines may be extended by addressing when to use country-specific data, what type of data these could be, and when to use international data from published literature. In country-specific guidelines, a list of sources where data for different model parameters may be found, contact persons, or institutions that can provide further advice in the form of a separate section would be of help.

In conclusion, HTA manuals and health economic modeling guidelines would benefit from including more specific advice on evidence sources for populating a model, including a hierarchy of evidence.

Footnotes

This article has been funded by the Austrian Research Promotion Agency (FFG; grant 827347). The employers of each author have received a fund by the Austrian Research Promotion Agency (FFG; grant 827347); the funding played no role in the method applied, nor did it influence the research process.

Supplementary material for this article is available on the Medical Decision Making Web site at .

References

Weinstein

O’Brien

Hornberger

. Principles of good practice for decision analytic modeling in health-care evaluation: report of the ISPOR Task Force on Good Research Practices—modelling studies. Value Health. 2003;6(1):9–17.

Briggs

Sculpher

Claxton

. Decision Modelling for Health Economic Evaluation. Oxford: Oxford University Press; 2006.

Hunink

MGM

Glasziou

Siegel

. Decision Making in Health and Medicine. Cambridge: Cambridge University Press; 2001.

Philips

Ginnelly

Sculpher

. A review of guidelines for good practice in decision-analytical modelling in health technology assessment. Health Technol Assess. 2004;8(36):1–158.

Coyle

Lee

Cooper

. Chapter 9: use of evidence in decision models. In: Shemilt

Mugford

Vale

Marsh

Donaldson

eds. Evidence-Based Decisions and Economics: Health Care, Social Welfare, Education and Criminal Justice. Oxford: Wiley-Blackwell; 2010.

Cooper

Sutton

Ades

Pailsley

Jones

; on behalf of the Working Group on the “Use of Evidence in Economic Decision Models.” Use of evidence in economic decision models: practical issues and methodological challenges. Health Econ. 2007;16:1277–86.

International Society for Pharmacoeconomics and Outcome Research (ISPOR). Pharmacoeconomic Guidelines around the World. 2012. Available from: URL: http://www.ispor.org/PEguidelines/index.asp

Paisley

. Classification of evidence in decision-analytic models of cost-effectiveness: a content analysis of published reports. Int J Technol Assess Health Care. 2010;26(4):458–62.

Alves de Silva

Pinto

Sampaio

Pereira

Drummond

Trindade

. Guidelines for Economic Drug Evaluations Studies. Lisbon, Portugal: Infarmed; 1998.

10.

Szende

Mogyorósy

Muszbek

Nagy

Pallos

Dózsa

. Methodological guidelines for conducting economic evaluation of healthcare interventions in Hungary: a Hungarian proposal for methodology standards. Eur J Health Econ. 2002;3:196–206.

11.

Tan-Torres Edejer

Baltussen

Adam

. Making Choices in Health: WHO-Guide to Cost-Effectiveness Analysis. Geneva, Switzerland: World Health Organization; 2003.

12.

The French Health Economists Association (Collège des Économistes de la Santé). French Guideline for the Economic Evaluation of Health Care Technologies. 2004. Available from: URL: http://www.ispor.org/PEguidelines/source/France_Guidelines_HE_Evaluation.pdf

13.

Canadian Agency for Drugs and Technologies in Health. Guidelines for the Economic Evaluation of Health Technologies. Report No. 3rd edition. Ottawa: Canadian Agency for Drugs and Technologies in Health; 2006.

14.

College voor zorverzekeringen. Guidelines for Pharmacoeconomic Research. Updated version. 2006. Available from: URL: http://www.ispor.org/PEguidelines/source/HTAGuidelinesNLupdated2006.pdf

15.

PHARMAC. Prescription for Pharmacoeconomic Analysis. Wellington, New Zealand: PHARMAC; 2007.

16.

Medical Service Advisory Committee. Economic Section of the MSAC Guidelines. 2008. Available from: URL: http://www.msac.gov.au/internet/msac/publishing.nsf/Content/D81BE529B98B3DB6CA2575AD0082FD1B/$File/Economics%20subcommittee%20guidelines%20June%202009.pdf

17.

Pharmaceutical Benefits Advisory Commitee (PBAC). Guidelines for Preparing Submissions to the Pharmaceutical Benefits Advisory Committee (Version 4.3). Canberra, Australia: Pharmaceutical Benefits Advisory Committee; 2008.

18.

National Institute for Health and Clinical Excellence. Guide to the Methods of Technology Appraisal. London: National Institute for Health and Clinical Excellence; 2008.

19.

Belgian Health Care Knowledge Centre. Guideline for Pharmacoeconomic Evaluation in Belgium. Brussels: Belgian Health Care Knowledge Centre (KCE); 2008.

20.

Agency for Health Technology Assessment. Guidelines for Conducting Health Technology Assessment (HTA). Version 2.1. Warsaw, Poland: Agency for Health Technology Assessment; 2009.

21.

Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen. Arbeitspapier Modellierung. Köln, Germany: Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen; 2009.

22.

Academy of Managed Care. The AMCP Format for Formulary Submissions. Version 3.0. Alexandria, VA: Academy of Managed Care; 2009.

23.

Health Information and Quality Authority. Guidelines for the Economic Evaluation of Health Technologies in Ireland. 2010. Available from: URL: http://www.ispor.org/PEguidelines/source/Ireland_Economic_Guidelines_2010.pdf

24.

Papaioannou

Brazier

Pailsley

. NICE DSU Technical Support Documents 9: The Identification, Review, and Synthesis of Health State Utility Values from the Literature. Sheffield, UK: School of Health and Related Research, University of Sheffield; 2010.

25.

Kaltenthaler

Tappenden

Pailsley

Squires

. NICE DSU Technical Support Document 13: Identifying and Reviewing Evidence to Inform the Conceptualisation and Population of Cost-Effectiveness Models. Sheffield, UK: University of Sheffield; 2011.

26.

Dias

Welton

Sutton

Ades

. Evidence Synthesis for Decision Making 5: Evidence Synthesis in the Baseline Natural History Model. Med Decis Making. 2013;33:657–70.

27.

Norwegian Medicines Agency. Guidelines on How to Conduct Pharmacoeconomic Analyses. 2012. Available from: URL: http://www.ispor.org/PEguidelines/source/Norwegian_guidelines2012.pdf

28.

Gesundheit Österreich GmbH. Methodenhandbuch für Health Technology Assessment. Version 1. 2012. Wien, Germany: Gesundheit Österreich GmbH im Auftrag des Bundesministeriums für Gesundheit; 2012.

29.

Scottish Medicines Consortium. Guidance to Manufacturers for Completion of New Product Assessment FORM (NPAF). 2012. Available from: URL: http://www.scottishmedicines.org.uk/files/submissionprocess/Guidance_on_NPAF_V3.1_March_2012.doc

30.

Briggs

Weinstein

Fenwick

Karnon

Sculpher

, ISPOR-SMDM Modeling Good Research Practices Task Force. Model parameter estimation and uncertainty analysis: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group 6. Med Decis Making. 2012;32(5):722–32.

31.

Caro

Briggs

Siebert

Kuntz

, ISPOR-SMDM Modeling Good Research Practices Task Force. Modeling good research practices—overview: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-1. Med Decis Making. 2012;32(5):667–77.

32.

Pitman

Fisman

Zaric

. Dynamic transmission modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group-5. Med Decis Making. 2012;32(5):712–21.

33.

Siebert

Alagoz

Bayoum

. State-transition modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-3. Med Decis Making. 2012;32(5):690–700.

34.

Karnon

Stahl

Brennan

Caro

Mar

Möller

. Modeling using discrete event simulation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-4. Med Decis Making. 2012;32(5):701–11.

35.

Facey

Topfer

L-A

Chan

, on behalf of the International Network of Agencies for Health Technology Assessment. Health Technology Assessment (HTA) Glossary. Sweden: INAHTA Secretariat, c/o SBU; 2006.

36.

Drummond

Sculpher

Torrance

O’Brien

Stoddart

, eds. Methods for the Economic Evaluation of Health Care Programmes. 3rd ed. New York: Oxford University Press; 2005.

37.

Cooper

Coyle

Abrams

Mugford

Sutton

. Use of evidence in decision models: an appraisal of health technology assessments in the UK since 1997. J Health Serv Res. 2005;10(4):245–50.

38.

Alban

Gyldmark

Pedersen

Søgaard

. The Danish Approach to Standards for Economic Evaluation Methodologies. Pharmacoeconomics. 1997;12(6):627–36.

39.

Capri

Ceci

Terranova

Merlo

Mantovani

, The Members of the Italian Group for Pharmacoeconomics. Guidelines for Economic Evaluations in Italy: Recommendations from the Italian Group of Pharmacoeconomic Studies. Drug Inform J. 2001:189–201.

40.

Rovira

Antoñanzas

. Economic Analysis of Health Technologies and Programmes: A Spanish Proposal for Methodological Standardization. Pharmacoeconomics. 1995:8(3):245–52.

41.

Walter

Zehetmayr

. Guidelines zur gesundheitsökonomischen Evaluation. Wien, Germany: Institut für Pharmaökonomische Forschung; 2006.

42.

All Wales Medicines Strategy Group. Untitled. 2012. Available from: URL: http://www.wales.nhs.uk/sites3/page.cfm?orgid=371&pid=2464

43.

Ministry of Social Affairs and Health Finland. Decree of the Ministry of Social Affairs and Health on applications and price notifications made to the Pharmaceuticals Pricing Board—appendix: guidelines for preparing a health economic evaluation 2009. Available from: URL: http://www.ispor.org/PEguidelines/source/GuidelinesinFinland_EnglishVersion.pdf

44.

Deutsche Agentur für HTA des Deutschen Instituts für Medizinische Dokumentation und Information DAHTA@DIMDI. Handbuch für Autoren zur Erstellung von HTA-Berichten. Köln, Germany: Deutsche Agentur für HTA des Deutschen Instituts für Medizinische Dokumentation und Information DAHTA@DIMDI; 2008.

45.

IQIWiG. Allgemeine Methoden. Köln, Germany: IQWIG; 2011.

46.

European Network for Health Technology Assessment. HTA core model for medical and surgical interventions. Available from: URL: http://www.eunethta.eu/activities/JA-WP4/ja-wp4-core-hta

47.

Gartlehner

. Internes Manual. Abläufe und Methoden. 2nd ed. Vienna, Austria: LBI-HTA; 2009.

48.

Behmane

Lambot

Irs

. Baltic Guideline for Economic Evaluation of Pharmaceuticals (Pharmacoeconomic Analysis). 2002. Available from: URL: http://www.ispor.org/PEguidelines/source/Baltic-PE-guideline.pdf

49.

Ara

Wailoo

. NICE DSU Technical Support Document 12: The Use of Health State Utility Values in Decision Models. Sheffield, UK: School of Health and Related Research, University of Sheffield; 2011.

50.

Dias

Welton

Sutton

Ades

A. A

. NICE DSU Technical Support Document 1: Introduction to Evidence Synthesis for Decision Making. Sheffield, UK: Decision Support Unit, University of Sheffield; 2012.

51.

Pharmaceutical Benefit Board. General Guidelines for Economic Evaluation from the Pharmaceutical Benefit Board. 2003. Available from: URL: http://www.ispor.org/PEguidelines/source/Guidelines_in_Sweden.pdf

52.

State of Israel Ministry of Health. Instructions for Performing an Economic Assessment; (Appendix D in the Guidelines for the Submission of a Request to Include a Pharmaceutical Product in the National List of Health Services). 2010. Available from: URL: http://www.ispor.org/PEguidelines/source/Israel-Guidelines-for-submission_2010.pdf

53.

Kristensen

Sigmund

. Health Technology Assessment Handbook. Kopenhagen, Denmark: DACEHTA; 2007.

54.

IQIWiG. Allgemeine Methoden zur Bewertung von Verhältnissen zwischen Nutzen und Kosten. Version 1.0. Köln, Germany: IQWiG; 2009.

55.

Department of Health. Draft Guidelines for Pharmacoeconomic Evaluations of Medicines and Scheduled Substances. 2010. Available from: URL: http://www.ispor.org/PEguidelines/source/guidelines-for-pharmacoeconomic-evaluations-of-medicines-and-scheduled-substances_2010.pdf

56.

China Guidelines for Pharmacoeconomic Evaluations. 2011. Available from: URL: http://www.ispor.org/PEguidelines/source/China-Guidelines-for-Pharmacoeconomic-Evaluations_2011_Chinese.pdf

57.

Research Centre for Clinical and Economic Evaluation and Pharmacoeconomics of the Russian State Medical University. Protocol on the procedure for clinical and economic evaluation of drugs which are submitted for inclusion into reimbursed drug lists. 2010. Available from: URL: http://www.ispor.org/PEguidelines/source/Russia_PE_Recommendations_english_fnal_13_03.pdf

58.

Ministerio da Saude. Methodological Guidelines: Economic Evaluation of Health Technologies. 2009. Available from: URL: http://www.ispor.org/PEguidelines/source/Economic-Evaluation-Guidelines-in-Brazil-Final-Version-2009.pdf

59.

Cuba Ministry of Health. Methodological Guidelines for Health Economic Evaluation. 2003. Available from: URL: http://www.ispor.org/PEguidelines/source/Methodological-Guidelines-for-Health-Economic-Evaluations-in-Cuba.pdf

60.

Mexico National Institute for Public Health. Guidelines for Economic Appraisal Studies for Updating the Health Sector’s National Formulary in Mexico. 2008. Available from: URL: http://www.ispor.org/PEguidelines/source/Mexico-Guidelines-for-Economic-Appraisal-Studies_2008.pdf

61.

Taiwan Society for Pharmacoeconomics and Outcomes Research. Guidelines of Methodological Standards for Pharmacoeconomic Evaluations. 2006. Available from: URL: http://www.ispor.org/PEguidelines/source/2006_PEG_EN_2009.pdf

62.

Health Insurance Review & Assessment Service. Pharmaceutical Economic Evaluation Guidelines. 2006. Available from: URL: http://www.ispor.org/PEguidelines/source/Korean_PE_Guidelines_Korean_Version.pdf

63.

The Council of Medical Association Executive Committee. Health Technology Assessment Guideline. 2008. Available from: URL: http://www.ispor.org/PEguidelines/source/Thailand-Health-Technology-Assessment-Guidelines.pdf

64.

Metodická pomôcka. 2008. Available from: URL: http://www.ispor.org/PEguidelines/source/PEGuidelinesinSlovakRepublic-Metodickapomocka.pdf

65.

Nuijten

. The selection of data sources for use in modelling. Pharmacoeconomics. 1998;13(3):305–16.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB

0.09 MB