Multi-criteria decision making to support waste management: A critical review of current practices and methods

Context of MCDM in SWM

The World Bank reported that about 1.3 billion tonnes of municipal solid waste (MSW) were generated in 2011 and that this amount is expected to increase to 2.2 billion tonnes by 2025 (Hoornweg and Bhada-Tata, 2012). However, this quantity will likely be reached earlier than expected. D-Waste (2014) reported a MSW generation of 1.84 billion tonnes in 2013, and the United Nations Environment Program (2015) estimated that the current MSW generation is approximately 2 billion tonnes per annum. In general, people are discarding increasing quantities of waste, and its composition is more complex than ever before, as plastic and electronic products diffuse (Herva et al., 2014; Vergara and Tchobanoglous, 2012). Therefore, increases in population, urbanisation, and economic development have led to mounting concerns in the field of solid waste management (SWM) (Levis et al., 2013). These elements, as well as lifestyle, income level, and socio-economic and cultural factors, are the main drivers for waste generation patterns and toxicity of waste (Eriksson and Bisaillon, 2011; Herva et al., 2014; International Solid Waste Association, 2002). Thus, SWM has become one of the most challenging service sectors for municipal authorities in the 21st century (Cherubini et al., 2009; Zaman, 2014). Indeed, solid waste issues, including emissions in air, water, and soil, pose serious public health risks, local and global environment hazards, and socio-economic problems (Herva et al., 2014; Ikhlayel et al., 2013).

These complex management requirements are better controlled if supported by tools for assessing the overall performance of the system, including administrative, financial, legal, and planning aspects (Mendes et al., 2013). According to Eriksson et al. (2003), the major benefits of waste management models lie in their ability to deal with complexity and uncertainty; and to handle different goals. For Zurbrügg et al. (2014), formalised assessment tools help to ensure a structured approach and provide a comprehensive method for data collection and analysis. Moreover, these models must be able to complement existing insights and adapt to the increasing complexity of current SWM (Manfredi and Goralczyk, 2013). It is important to note that decision support models in the field of waste management have been developed since the late 1960s (Karmperis et al., 2013). These early works, including those by Anderson and Nigam (1968), Helms and Clark (1971), and Morse and Roth (1970), focused on model assessment of individual elements such as collection itineraries or facilities locations (Eriksson et al., 2003). During the 1980s and 1990s, in turn, some models were designed to provide an assessment of the entire waste management system (Allesch and Brunner, 2014). The diversity and multitude of methods in current usage for supporting waste management decisions cause particular challenges in the choice of assessment methods (Finnveden et al., 2007, Pires et al., 2011).

According to Karmperis et al. (2013), Milutinović et al. (2014), and Morrissey and Browne (2004), the most widely used decision support frameworks in the field of waste management are life-cycle assessment (LCA), cost-benefit analysis, and multi-criteria decision making (MCDM). LCA focuses on environmental aspects, whereas maximisation of economic efficiency is the major goal of cost-benefit analysis. MCDM, however, allows consideration of the three pillars of sustainability: Economic, social, and environmental criteria.

In fact, MCDM can guide decision makers in evaluating existing or potential alternatives by simultaneously applying multiple conflicting criteria (Belton and Stewart, 2002; Kou et al., 2011; Zhou et al., 2010). Because of their ability to handle several criteria, MCDM methods are considered to be some of the most effective and thorough decision support frameworks for decision-making in SWM (Soltani et al., 2015). Indeed, because MCDM methods can help to resolve problems related to the aggregation of incomparable values, which are typical in waste management, such an application is considered pertinent and appropriate (Allesch and Brunner, 2014; Milutinović et al., 2014; Morrissey and Browne, 2004; Xi et al., 2010). Since the 1980s, the use of MCDM models, such as those by Saaty and Gholamnezhad (1982) and Shimizu (1981) in the solid waste context, has increased with each decade. Thus, many MCDM methods have been explored in SWM in previous researches.

However, despite the development of a large number MCDM methods, none can be considered appropriate to all decision making situations (Guitouni and Martel, 1998). For Karmperis et al. (2013), it is difficult to distinguish one decision support framework as the most suitable for all SWM systems. So, the selection of the MCDM method is a challenging issue. Indeed, each MCDM method is applicable for some types of problems (Zardari et al., 2015). For Achillas et al. (2013), it is important to select a suitable method for each case.

The choice of the methods used in each step involved in the MCDM process (indicators selection, weighting, normalisation, sensitivity analysis) is another critical aspect because they could directly affect the results (Çelen, 2014; Dobbie and Dail, 2013; Dodgson et al., 2009; Ebert and Welsch, 2004). In fact, as it is presented in the next section, ‘MCDM fundamentals’, the selection of the procedure used in each step of a MCDM method depends on the available information as well as on the purpose of the study (Zardari et al., 2015).

Therefore, an investigation of the studies on this topic developed during nearly four decades could provide valuable information to deal with the issues aforementioned. In fact, to examine preceding studies could support improvements in the current MCDM applications because it allows determination of their weakness and strengths. So, a review of the previous studies could indicate the most suitable MCDM methods for a specific topic in SWM, supporting researchers and waste managers in the selection of an appropriate method. Furthermore, a review article also allows potential topics in SWM to be highlighted, in which applications of MCDM can be further explored.

So, the objectives of this study are to present a literature review of MCDM applications used in SWM, offer a critical assessment of the current practices, and provide suggestions for future works.

MCDM fundamentals

MCDM methods encompass several techniques developed to support decisions involving conflicting criteria, incommensurable variables, and several possible solutions or alternatives (Doumpos and Zopounidis, 2014; Pohekar and Ramachandran, 2004).

Regarding MCDM approaches, they are normally classified in two main streams: Multi-attribute decision making (MADM) and multi-objective decision making (MODM). MADM comprises selection or ranking problems, while MODM encompasses optimisation problems. In other words, MADM methods aim to compare or rank any set of alternatives based on the criteria adopted, whereas MODM techniques are focused on determining the set of optimal alternatives according to the criteria considered.

So, MADM involves discrete decision spaces with a predetermined and limited number of alternatives that are evaluated against a set of attributes or criteria (Hatami-Marbini et al., 2013). The alternatives are ranked by performing comparisons among alternatives with respect to each attribute or by determining an overall performance based on attribute aggregation (Rousis et al., 2008). A general MADM application involving a number of I alternatives assessed by a total of J criteria can be represented by a comparison matrix of I rows and J columns, where Aij corresponds to the score of the jth criteria for the ith alternative evaluated. Based on this matrix, each MADM method presents a mathematical procedure to enable the comparison and classification of alternatives.

MODM, in turn, comprises a continuous space in which the alternatives are not predetermined; instead, a set of objective functions, subject to constraints, is maximised or minimised to find a set of optimal alternatives. Thus, it is defined a Pareto Front that consists of a frontier that regroups all of the optimal solutions (Pohekar and Ramachandran, 2004). A general MODM model with K objectives to be maximised can be represented by equations (1) to (3):

max f k ( x ) , k = 1 , … , K

(1)

g ( x ) = 0

(2)

h ( x ) > 0

(3)

where f_k(x) is the objective function related to the kth objective, x is a vector that regroups the scores of all criteria considered in the problem, and g(x) and h(x) are constraints that delimitates the problem domain. The solution of this model consists in determining the set of vectors x that gives a satisfactory result to all objective functions f_k(x) subject to the constraints g(x) and h(x).

According to Chang and Pires (2015), MODM methods are normally applied at operational levels, e.g. collection routing optimisation, or at strategic planning, which includes: Allocation of waste among treatment facilities. On the other hand, MADM methods are more applicable at a strategic level, e.g. treatment technologies selection or facilities location (Spengler et al., 1998).

Concerning MCDM steps, as reported by Caramia and Dell’Olmo (2008), Chang and Pires (2015), Dodgson et al. (2009), Riabacke et al. (2012), and Zhou et al. (2010), the following steps are commonly used in the development of MCDM methods:

Morrissey and Browne (2004) reported that formulation of the problem is the most important step in the decision-making process in municipal waste management. In other words, the development of an assessment tool begins by defining the goals and scope. The former is related to the purpose of the appraisal, and the latter is related to the boundary conditions, i.e. the limits of the evaluation. For Allesch and Brunner (2014) and Zurbrügg et al. (2014), the choice of the starting and end points of an assessment in SWM can have a crucial effect on its results; therefore, system boundaries must be defined carefully, because any changes can affect the results. Moreover, according to Karmperis et al. (2013), the scope of the assessment is the underlying factor in the selection of the most appropriate framework. The latter consists of a structure that supports the proposed assessment approach by introducing and describing the theories and concepts behind it (Labaree, 2013). So the framework is highly dependent on the type of the MCDM method adopted.

After defining the theoretical framework, it is necessary to select criteria and indicators. The criteria are the major issues related to a subject and provide sense and operationality to the goals without itself being a direct measure of performance (Prabhu et al., 1998). The criteria in MADM and MODM are expressed by attributes and objective functions, respectively. The status, performance, or condition of a criterion, in turn, is evaluated by quantitative or qualitative indicators; that is, the indicators define the information used to evaluate a criterion (Pokorny and Adams, 2003). Concerning MODM, it is necessary to define constraints that limit the solution to a particular domain in addition to the objective functions and indicators.

Another critical factor of MCDM processing is the definition of criteria weights, which, depending on the method, can reflect the importance of a criterion in an ensemble or express the trade-offs among criteria (Hatefi and Torabi, 2010; Karmperis et al., 2013; Thivierge et al., 2014; Wang, 2015). There are basically two main streams to attribute weights to criteria: Opinion-based and data-based approaches (Gasparatos et al., 2008; Molinos-Senante et al., 2014). In the former, weights are obtained from subjective judgments by means of opinion polls normally involving experts and stakeholders; the latter includes methods derived from statistical or mathematical models. Details about these procedures are presented in Nardo et al. (2005) and Lutero (2010). Other possibilities are the adoption of equal weights to all criteria or simulation of several weight scenarios.

Normalisation, an additional common step in MCDM, provides comparability among indicators having different units or scales (Garcia-Sánchez et al., 2015). Normalisation normally employs linear transformation of the crude data involving the two elementary operations of translation and expansion (Ebert and Welsch, 2004). As discussed in Munda and Nardo (2003) and Podviezko (2014), normalisation procedures need to be used prudently and, if possible avoided, as different methods to normalise raw data can provide different results, which directly affect the decision process. It is important to note that some MCDM methods do not require normalisation procedures, although others require this step. On the one hand, normalisation is necessary if the MCDM method is based on the construction of a single index formed by the linear aggregation of indicators that present different units or scales. On the other hand, normalisation is not mandatory if the indicators present the same unit and scales or the MCDM method adopted is based on pairwise comparisons of individual indicators.

Concerning sensitivity analysis, it is a crucial step to assess the model robustness, which is defined as the characteristic of the model to be insensitive to small changes in the input parameters (Muccillo et al., 2015). Indeed, sensitivity analysis investigates how the variation in the output of a model can be attributed to the input data (Ganji et al., 2016; Pianosi et al., 2016), as well as to the structure of the model (Delgado and Sendra, 2004). So, this step enables evaluation of the degree to which changes in input data and framework elements, such as the MCDM method selected, number of criteria, weight values, and normalisation process, affect the final result of the assessment. It is important to note that applying sensitivity analysis is essential to ensure the consistency of the final decision and can also indicate priority areas for refinement in further versions of the MCDM model (Qureshi et al., 1999; Syamsuddin, 2013). The most common approaches used to carry out a sensitivity analysis are the one factor at a time (OAT) method and variance-based methods (VBM). The former consists in varying the OAT method and measure the variation in the output, whereas the latter is focused on the decomposition of the total output variance into the contributions of each factor or input (Campolongo et al., 2007; Ligmann-Zielinska and Jankowski, 2014).

Focusing on MCDM methodologies, both MADM and MODM include a large variety of methods. The MADM stream can be divided into three groups, including value-based methods, outranking methods, and distance-based methods.

The value-based methods include multi-attribute value theory (MAVT), multi-attribute utility theory (MAUT) (Keeney and Raiffa, 1993), and the analytic hierarchy process (AHP) (Saaty, 1988). MAVT and MAUT, also known as performance aggregation-based approaches, are the most widely used MCDM methods (Riabacke et al., 2012). In MAVT, value functions are used to represent the satisfaction level of an alternative in respect to one criterion, and the overall performance of one alternative is determined by aggregating the value functions of each criterion in a single index. The main difference between MAVT and MAUT is that the latter explicitly considers uncertainty by using utility functions rather than value functions. Although AHP belongs to the same group as that of MAVT/MAUT, it is based on different assumptions; this approach evaluates alternatives using pairwise comparisons in a multilevel hierarchic structure. One of the major advantages of AHP is that it calculates the inconsistency index, which evaluates the level of judgement con-tradictions during the pairwise comparison (Pohekar and Ramachandran, 2004). Moreover, AHP presents a variant called analytic network process (ANP), which is a generalisation of the AHP. The advantage of ANP over AHP is that the former allows consideration of interrelationships among decision levels, criteria, and alternatives.

Outranking methods, also known as preference aggregation-based approaches, involve methods based on comparisons between pairs of options to verify whether an alternative is at least as good as another (Cinelli et al., 2014). Two common methods of this group are Preference Ranking Organization and Method for Enrichment Evaluation (PROMETHEE) (Brans and Vincke, 1985) and Elimination and Choice Expressing Reality (ELECTRE) (Roy, 1991). PROMETHEE includes three major steps: Define preference functions for each criterion; establish multi-criteria preference index and preference flows; and rank alternatives (Ananda and Herath, 2009). PROMETHEE I provides a partial ranking, whereas PROMETHEE II allows a full ranking. Regarding the ELECTRE family of methods, these approaches use pairwise comparisons to determine concordance and discordance indices. The former evaluates whether one alternative is at least as good as another, whereas the latter verifies whether the performance of an alternative in one criterion is worse than an acceptable limit (Doumpos and Zopounidis, 2014). This method has several variants, including ELECTRE I, II, III, IV, and V.

The distance-based methods group includes approaches that compare alternative performance by calculating the distance from the alternative with the worst or the ideal (best) solution. The most commonly used method in this field is the Technique for Order Preference by Similarity (TOPSIS), which was designed by Hwang and Yoon (1981). In this method, which is based on Euclidean distance, the best performance occurs when a system has the shortest (longest) distance to the ideal (worst) solution (Huang et al., 2011).

MODM also includes a large variety of approaches such as multi-objective linear programming (MOLP), multi-objective nonlinear programming (MONLP), and multi-objective mixed integer linear programing (MOMILP). These approaches essentially differ from the nature of variables, such as integer or continuous real values, and type of objective functions and restraints, such as linear or nonlinear. MOLP and MONLP involve only variables defined in a continuous real domain, although the former is related to linear objective elements (functions and constraints) whereas the latter involves nonlinear ones. MOMILP, however, combines integer and continuous variables and assumes linear relations for objective functions and constraints. To solve MODM problems, i.e. to determine the set of optimal solutions, numerous techniques are used, such as goal programming, compromise programming, genetic algorithms, and heuristics methods (Branke et al., 2008; Caramia and Dell’Olmo, 2008).

Other MCDM methods that can be used alone or combined with the aforementioned ones include Decision Making Trial and Evaluation Laboratory (DEMATEL), Novel Approach to Imprecise Assessment and Decision Environments (NAIADE), Thor, Vikor, stochastic-based methods, fuzzy-based methods, dynamic programming, and graphical-based analysis. Further information about MCDM methods are provided in Dodgson (2009), Doumpos and Zopounidis (2014), Huang et al. (2011), Linkov et al. (2004), and Velasquez and Hester (2013).

Previous studies

It is important to note that previous works have already presented reviews related to MCDM in SWM. Table 1 shows a summary of the major aspects of each review evaluated in the present study.

OPEN IN VIEWER

Table 1.

Previous studies related to MCDM reviews in SWM.

Review article	Number of articles analysed	MCDM streams	Type of studies	Waste streams	Results presented in the study
Achillas et al. (2013)	79	MADM	Several SWM topics	Several waste streams	MCDM method Temporal evolution Type of study Criteria considered
Demesouka et al. (2014)	36	MADM	Landfill location	MSW	MCDM method Temporal evolution Criteria considered Normalisation process Weighting method
Soltani et al. (2015)	68	MADM	Several SWM topics	MSW	MCDM method Temporal evolution Type of study Stakeholders involved

MADM: multi-attribute decision making; MCDM: multi-criteria decision making; MSW: municipal solid waste; SWM: solid waste management.

Despite being published after 2013, the three review articles shown in Table 1 include a total of 128 different articles, which corresponds to 49% of the studies involving MCDM applied in SWM that are used in the present research. Moreover, although most of these review studies used the term MCDM, they focus only on MADM methods. Thus, MODM works were not evaluated, which restricts such reviews to MADM applications. Therefore, these previous studies had not considered an important part of articles related to this topic, which could influence the conclusions.

Essentially, all of the review articles aforementioned presented general information related to the number of articles using a type of MCDM method and provided a timeline showing the evolution of articles published on this topic. Achillas et al. (2013) and Demesouka et al. (2014) also provided a detailed list with the main criteria considered according to the nature of the problem studied, whereas Soltani et al. (2015) has a focus on discussing the different stakeholders involved in the studies. However, very little information related to MCDM framework details, i.e. normalisation, weighting, and sensitivity analysis, was explored in these articles. Demesouka et al. (2014) was the only study that listed weighting and normalisation methods adopted, but this article was focused on landfill location.

Concerning scope, Demesouka et al. (2014) examined articles related only to landfill location. On the other hand, Achillas et al. (2013) and Soltani et al. (2015) compiled articles related to other waste management topics, such as strategy, assessment, and monitoring.

Regarding waste streams, Demesouka et al. (2014) and Soltani et al. (2015) focused only on MSW studies and neglect the ones from other waste streams. Achillas et al. (2013), in turn, recorded articles involving other waste types, such as industrial solid waste (ISW), construction and demolition waste (CDW), electronic waste (e-waste), healthcare waste (HCW), hazardous waste, and nuclear waste.

Thus, despite providing useful information, the previous review articles present some limitations related to the number of articles computed, the scope of the analysed studies, the waste streams considered, and mainly the MCDM framework information presented.

Addressing these gaps, the present article aims to provide a more extensive review than those conducted in previous studies.

Articles selection

A total of 260 articles concerning MADM and MODM applications in Waste Management (WM) were compiled for the present study. To guarantee the quality of data and ease of access, the results presented in this article include only studies published in peer-reviewed scientific journals. The articles used in this research were obtained from the following electronic databases: American Society of Civil Engineers, SAGE Journals, Science Direct, Scopus, Springer Link, Taylor Francis, and Wiley Online Library. The articles were selected based on a keyword research in which the following keyword combinations were used:

Information sorting, frequency, and critical analysis

For each article selected, the year of publication and data related to the scope of the study and the MCDM model used were recorded. The information obtained from each study was sorted considering some key aspects that are presented in Table 2. These key aspects were selected so as to enable the evaluation of the articles in terms of the main steps of MCDM described in the section ‘MCDM fundamentals’. In addition, key aspects are also related to characteristics as work objective, criteria considered, and waste stream, which allow the identification of the waste management research topic addressed in the study to be catalogued. It is important to note that Table 2 presents several classes used to characterise the articles according the key aspects considered. In addition, remarks about special features of each article were also recorded. To avoid mistakes during the information sorting process, the definition of the classes used to categorise the analysed articles was carried out independently by two researchers. Then, the responses of both were collated and in case of divergence, the results were discussed until to obtain a consensus.

OPEN IN VIEWER

Table 2.

Key aspects and classes evaluated in each article analysed.

Key aspect	Classes
MCDM method	MODM: MOLP, MONLP, MOMILP MADM: AHP, ANP, ELECTRE, fuzzy methods, MAVT/MAUT, PROMETHEE, TOPSIS, and others
Waste stream	Municipal solid waste Industrial solid waste Construction and demolition waste Health and care waste Electronic waste Nuclear waste Hazardous waste Biodegradable waste (bio-waste) Others (oil waste, recyclables, sludge, etc.)
Work objective	Waste management strategy Waste facility location Service/facility assessment Transport and collection routing
Criteria considered	Type: Economic, environmental, social, technical Characteristic: Qualitative, quantitative
Weighting method	Data-centric, opinion-based, weight scenarios, equal weights
Stakeholders involved in weighting process	Authorities, academics, community, environmentalists and non-governmental organisations, waste professionals, private waste companies
Normalisation method	Linear, scales, fuzzy functions, none
Sensitivity analysis	Criteria, weights, input data, model framework, none
Use of MCDM with other tools	Life cycle assessment, geographic information system

AHP: analytic hierarchy process; ANP: analytic network process; ELECTRE: Elimination and Choice Expressing Reality; MADM: multi-attribute decision making; MAVT/MAUT: multi-attribute value theory/multi-attribute utility theory; MCDM: multi-criteria decision making; MODM: multi-objective decision making; MOLP: multi-objective linear programming; MOMILP: multi-objective mixed integer linear programing; MONLP: multi-objective nonlinear programming; PROMETHEE: Preference Ranking Organization and Method for Enrichment Evaluation.

After the sorting process, to facilitate the overall evaluation of the data and to provide a quantitative measure of the information obtained, a frequency analysis was performed to determine the number of studies categorised in each class of a key aspect. Thus, a critical analysis was carried out based on the information compiled in the sorting process and in the results of the frequency study. This critical analysis focuses on the discussion of the major insights and shortcomings of the approaches presented in the articles analysed, which includes overall tendencies observed as well as findings of specific articles. Finally, according to the strengths and drawbacks identified in the articles analysed, recommendations for future works were provided for each key aspect evaluated.

Current situation

The Appendix presents the articles compiled in this research in order of publication year. For each article, the MCDM method adopted and waste stream analysed is indicated.

It is noted that 78% of the analysed articles adopted the MADM method, 19% used MODM, and 3% used both approaches. These results suggest that the most commonly used practice in SWM is to select the best alternative among a limited number of options previously defined rather finding the optimum solution or ideal alternative.

This predominance of MADM over MODM likely occurred because several SWM problems are related to the comparison of a few options previously selected. For example, landfill location normally includes only a limited quantity of candidate sites with convenient characteristics. The waste treatment facility selection also usually includes a finite number of options represented by the existing facilities in the study area.

Furthermore, problems involving infinite or a high number of options, in which MODM could be applicable, have been typically analysed as a single objective optimisation problem focusing on cost minimisation (Mavrotas et al., 2013). As previously mentioned, collection routing and waste flow allocation are examples of potential applications of MODM. Moreover, the historical preference for MADM in SWM is likely because MODM methods normally require more computational resources (Wallenius et al., 2008). The latter is not a current issue, however. Improvements in computer speed and algorithmic effectiveness allow MODM problems to be solved in seconds (Mavrotas et al., 2015).

Figure 1(a) presents the number of studies for the 10 most often used MCDM methods in SWM. AHP was used in 91 articles, which corresponds to 35% of the analysed studies. This predominance of AHP over other methods is likely related to the convenience of this approach for conducting SWM analysis. AHP allows the researcher to work with criteria divided in several classes and levels. This method fits with SWM assessment requirements because criteria in this sector are often grouped in general classes, such as economy, environment, society, and regulations. Another important characteristic of this method is that it provides an easy way to attribute weights from stakeholders’ opinions by using a pairwise comparison procedure. This method also meets SWM assessment requirements because in the waste management context, criteria weights are frequently defined by the researcher’s judgment, as will be discussed in the section ‘Weighting attribution’ of this article.

OPEN IN VIEWER

Figure 1.

General statistics for MCDM applications in SWM: (a) number of articles in which the 10 MCDM methods most often used were implemented; (b) temporal evolution of publications on this topic.

It was also observed that 26% of articles using the AHP method combined it with the fuzzy approach. Indeed, associate fuzzy logic can improve AHP robustness because allows for consideration of uncertainty in stakeholders’ preferences during criteria weight attribution.

The second most frequent method was MAVT/MAUT, which was identified in 52 studies, or 20%. These results are in contrast to those of Soltani et al. (2015), which did not report any study using this method. It is important to note that the MAVT/MAUT statistics includes articles using simple additive weighted (SAW) and weighted product (WP) methods.

The MOMILP approach was in the third position, with 34 applications. The predominant MODM method in SWM represented 58% of all studies involving multi-objective analysis. This preference of MOMILP over other methods could be attributed to its ability to combine continuous and integer variables. The latter, by means of binary representation, is often used in waste management to model the applicability (value 1) or non-applicability (value 0) of a waste treatment technology according the waste composition and other system characteristics. MOLIP was used in 18 articles, whereas MONLP applications were used only in six studies.

Outranking methods were also implemented often. Indeed, the fourth and the sixth most used methods were ELECTRE and PROMETHEE, respectively. Concerning their variants, ELECTRE III and PROMETHEE II were the preferred approaches, corresponding to the majority of applications using these methods.

Referring to temporal evolution, the earliest recorded articles using MADM and MODM were Saaty and Gholamnezhad (1982) and Shimizu (1981), respectively. From the 1980s to the present, a continuous increase was noted in the number of publications, as observed in Figure 1(b). Concerning articles published after 2010, it was observed that in the middle of the decade, that is in 2015, the number of studies was already 53% greater than those in the previous decade. This shows that MCDM methods applied in SWM has attracted an increasing amount of interest from research groups.

Regarding MCDM stream distribution over time, an increase in the number of publications was noted, although this process was much more intensive for MADM than for MODM. Indeed, MADM articles increased from three in the 1980s to 106 in the 2010s, whereas the number of MODM studies in the same period increased from three to 26.

Concerning the combination of MCDM methods with geographic information system (GIS) tools, for Tavares et al. (2011), which used this approach for waste facility location, the integration of these techniques offers a powerful tool to facilitate and accelerate any siting process. In fact, GIS methods are valuable for applications that involve data dependent on geographic position, because they offer adequate functionalities to capture, store, check, manipulate, and process spatially referenced data (Abastante et al., 2014; Chiueh et al., 2008). Among the articles recorded, it was noted that 44% of the facility location studies combined MCDM and GIS, whereas this combination was observed only in 2% of WM strategy articles. So, in facility-location-related articles, GIS seems to be an already widespread practice, differently to WM strategy, in which these studies are scarce. In spite of the low number of articles adopting GIS plus MCDM for WM strategy purposes, studies such as MacDonald (1996) and Vaillancourt and Waaub (2002) show that the association of these methods is as productive as in location studies. So, considering the aforementioned benefits of GIS, the use of this technique associated with MCDM methods in waste management studies involving geographic data is recommended. Regarding the MCDM methods used in these studies, AHP and MAVT were the predominant approaches representing 60% and 30% of the total number of articles, respectively. Provided that the normal GIS facility siting approach includes the definition of suitability maps based on the overall condition of each location according the criteria considered, single synthesising methods are convenient for representing the general performance. This feature explains the preference of AHP and MAVT.

Criteria considered

The criteria evaluated in each study were grouped according to four groups: Economic, environmental, social, and other. The latter includes criteria not directly related to the other three classes, such as technical, administrative, and regulatory aspects. The most commonly used groups, environmental and economic, were reported in 224 and 191 articles, respectively. Social aspects were reported in 152 articles, and other factors were present in 158 records. Even though environmental aspects were reported most often, only 10% of the articles on this topic incorporated LCA in the MCDM analysis. Su et al. (2010), Pires et al. (2011), Vinyes et al. (2013), Antonopoulos et al. (2014), and Vadenbo et al. (2014b) used the LCA impact categories as criteria in MCDM studies, arguing that it is a consistent and reliable approach to assess environmental dimension. According to Ahluwalia and Nema (2007), industry, government, and other stakeholders agree that environmental impacts must be considered from a life cycle perspective, which justify the use LCA in MCDM studies involving such issues. For Manfredi et al. (2011), the LCA can provide a more environmentally sustainable decision-making science-based support to SWM. In fact, LCA supports a holistic approach to sustainable waste management by enabling a comprehensive view of the impacts involved and providing directions to the development of environmentally sound strategies (del Borghi et al., 2009; Tulokhonova and Ulanova, 2013). So, the association of LCA and MCDM methods is encouraged because they complement one each other, as LCA provides a robust method to calculate environmental criteria, whereas MCDM allows extracting information from the overall performance of a system based on the criteria considered. Therefore, the association of LCA with MCDM approaches could contribute to a more effective evaluation of the environmental aspect.

Another important analysis is related to the number of studies that consider an aspect qualitatively and those that use a simplified set of criteria allowing only a partial assessment of a dimension. The percentage of studies using simplistic or qualitative analysis, only 10%, was not representative of those using economic aspects. Regarding the environmental dimension, the simplistic studies fraction was greater, representing 34%. For social criteria, the proportion was significantly higher; 77% of studies that consider this aspect adopted qualitative or simplistic analysis. The main concern related to qualitative criteria is the fact that their measurement is subjective and complex (Malakooti, 2013). According to Vinyes et al. (2013), who tried to consider social dimension in a quantitative way in a MCDM study, a fundamental question in the definition of quantitative social criteria is how to measure social impacts because the perception of them is very variable. The same author points that the major issue limiting the use of social aspects in studies is likely the absence of a disseminated procedure for selecting social criteria in SWM. Only a few studies, such as those by den Boer et al. (2005) and Osorio (2011), developed a methodology to define quantitative indicators for social analysis in SWM. As social and political aspects can be crucial in the final decision in the waste management sector (Banias et al., 2010; Coronado et al., 2011; De Feo and De Gisi, 2010), the development and investigation of quantitative social criteria are important steps to provide reliability to future research.

Considering only the pillars of sustainability, i.e. economic, environmental, and social criteria, Figure 2 shows the percentage of studies for all possible combinations. Of all studies, 46% considered these three dimensions, which indicates the engagement of researchers with sustainability issues. However, most of these articles conducted only a perfunctory analysis. Indeed, as shown in Figure 2, by eliminating from this group of studies with qualitative data or simplified analysis, only 22% remains.

OPEN IN VIEWER

Figure 2.

Criteria combinations used in the recorded articles.

Work objective

Investigation of the manner in which MCDM has been applied in SWM is very important, because this information identifies areas requiring further research in addition to sectors in which MCDM has already been used as a consolidated tool.

In spite of the increase in the number of MCDM publications studies in a WM context in the last years, the range of application is still concentrated on a few themes. It was observed that 90% of articles included either waste facility location or waste management strategy. Articles dedicated exclusively to management strategy, which includes research involving waste technology selection or waste treatment choice, corresponded to 41% of all studies recorded. Waste facility location was the theme in 43% of all analysed articles. The interest in this type of study is likely owing to decreases in land availability for waste disposal sites and increases in awareness of the environmental risk associated with waste management facilities (Srivastava and Nema, 2012).

Thus, for these types of problems, there are already a large number of articles although only 6% of studies include both WM strategy and facility location. Santibañez-Aguilar et al. (2013) developed a MCDM method considering location and WM strategy, arguing that waste allocation, technology selection, and facility location are decisive in yielding a suitable solution for the entire WM problem. Similarly, Ekmekçioğlu et al. (2010) and Perkoulidis et al. (2010) also performed studies combining location facility and waste allocation. The consideration of both topics in one study could improve the quality and even change the final results, because waste facility siting restrictions could be a limiting factor in the choice of a waste treatment technology, directly impacting WM strategies. Indeed, landfill location is highly dependent on physical characteristics, such as topography, geology, and groundwater depth (Demesouka et al., 2014). Therefore, land availability limitations and adverse site properties can limit the applicability of landfill in certain regions, which consequently could affect the WM strategy adopted. Thus, the assimilation of facility location in articles focusing on waste management strategies is strongly encouraged as it enables a more holistic analysis of the WM system.

In addition, 8% of the articles applied MCDM for assessment and monitoring of SWM services or facilities and only 2% were related to waste transportation and collection routing. Such scarcity of articles related to these two topics indicates their potential for future research. The first group is more closely related to MADM applications because the monitoring of existing services or facilities could include the comparison of a limited set of possible operational conditions.

Collection routing problems, which were historically investigated by single-objective optimisation models based on cost minimisation (Mavrotas et al., 2013), could be further explored by new approaches involving MODM addressing multiple objectives. It is important to note that MODM is largely used in other sectors to solve routing problems involving conflicting criteria and this is often the case in a WM context. For example, for a carrier firm, the preferred solution would be the one with the least cost; while for the public authorities, the desired solution would be the one with the least risk (Alumur and Kara, 2007). Routing problems involving the three dimensions of sustainability are another potential application for MODM method in this topic. Alumur and Kara (2007), Chen et al. (2008), Shih and Cheng (2001), and Shih and Lin (2003) showed that MCDM approaches applied to solve routing problems in WM are a powerful tool to achieve a compromise solution among criteria.

Concerning method applicability, Figure 3(a) shows the manner in which MCDM methods have been used in SWM. The majority of MODM applications were related to SWM strategy, whereas MADM studies were focused mostly on waste facility location. Both results were expected. Indeed, as previously mentioned, MADM meets the requirements for location selection of models because these studies normally involve several conflicting criteria used to compare a limited number of pre-selected sites. On the contrary, in the context of SWM strategy, planning is often necessary to define optimal or ideal conditions considering multiple criteria. Therefore, MODM is a recommended tool for this application, justifying the predominance of system strategy studies using this MCDM stream.

OPEN IN VIEWER

Figure 3.

Distribution of articles by: (a) MCDM stream and type of study; (b) type of study and waste stream.

From the findings presented it is noted that, despite the increase in the number of MODM articles in a WM context presented in the section ‘Current situation’, in absolute numbers the potential of MODM applications are still wasted in comparison with MADM ones. Thus, the MODM approach remains underutilised and could be better explored in applications as routing optimisation and WM strategy in which it could provide valuable contributions to the decision process.

Figure 3(b), in turn, divides waste types into two groups including one formed by MSW-related studies and the other including other waste streams. It could be noted that at 51%, the majority of articles in the MSW context focused on waste facility location, mainly landfill siting, followed by waste management strategies at 35%. For other waste streams, the contrary was observed; 55% of the articles were related to management strategy, whereas location was included in 28%. In addition, the percentage of studies dedicated to service/facility assessment was significantly higher among articles from other waste streams, at 10%, than that for MSW articles, at 6%.

Considering only facility location studies, Figure 4 shows that 78% focused on landfill siting. This emphasis on landfills likely occurs because it is the most largely used method for waste disposal worldwide (Hoornweg and Bhada-Tata, 2012) and it is also a technology with high visual impact and land space requirements. Moreover, as already stated, landfilling is more dependent on the physical characteristics of the disposal area, such as topography, geology, and groundwater depth, than other technologies.

OPEN IN VIEWER

Figure 4.

Types of waste treatment facilities involved in location studies.

Transfer stations and incinerator locations were studied in 7% and 5% of articles, respectively. Since the former is directly related to waste management transportation, the low number of MCDM applications related to this topic was unexpected, because MODM is largely used to solve logistics problems in other sectors, allowing optimisation in number or location of trans-shipment centres. For Massam (1991), transfer station location involves several aspects and complex interactions among planners, analysts, politicians, and the public, which make MCDM methods a suitable approach to assess this problem. In fact, Massam (1991), Önüt and Soner (2008), and Perkoulidis et al. (2010) presented a quantitative way to support decision making in such problems using different MADM methods. Galante et al. (2010) is another remarkable article addressed to this topic, as it presents a MODM model that provides optimal transfer station locations based on economic and environmental aspects and considering different stakeholder preferences. Concerning incinerator location, according to Wey (2005), this topic has become one of the most pressing problems in local government. Indeed, incinerators are generally considered undesirable infrastructures, because they are perceived by population as sources of air pollution problems and other adverse impacts to human health and environment (Chiueh et al., 2008; United Nations Environment Program, 2011).

In addition, for Tavares et al. (2011) the selection of criteria for facility location differs among technologies, as each one presents its own specificities and related concerns. So, the already consolidated and disseminate criteria normally used to assess landfill location are not necessarily the most suitable in studies involving other waste processing plants. It is noted that, owing to the low number of articles dedicated to the location of treatment facilities other than landfill, there is no consensus about criteria to assess them. In fact, among the MCDM-recorded articles linked to incineration location (Abastante et al. 2014; Bottero & Ferretti 2010; Chang et al. 2009; Chiueh et al. 2008; Tavares et al. 2011; Wey 2005), a high diversity of criteria was noted. So, considering the applicability of MCDM in facility location and the lack of a disseminated methodology dedicated to supporting siting waste technologies other than landfilling, studies involving different types of treatment facilities is a recommended topic for future investigation.

Figure 5(a) presents the number of studies that include the six most often used MADM methods divided in three classes according the type of analysis: Management strategy, facility location, and others. AHP was the most often used method independently of the objective of the article, likely owing to the reasons discussed in the section ‘Current situation’. It is also noted that ANP, AHP, MAVT/MAUT, and TOPSIS were used mainly in studies involving waste facility location, whereas outranking methods, such as PROMETHEE and ELECTRE, were applied mostly to support waste management strategies.

OPEN IN VIEWER

Figure 5.

(a) MADM studies classified by type of study, (b) MODM studies classified by type of study. FL: facility location; MS: management strategy.

Figure 5(b), in turn, presents the number of articles using MODM divided by the method and type of analysis. It is noted that MOMILP was the preferred MODM approach for all kinds of studies. Another important point related to MOMILP is the fact that almost 29% of its applications comprise both management strategy and facility location, whereas no register including MOLP and MONLP was recorded for this type of research. Moreover, most articles using MODM approaches focused on management strategy.

Waste streams

The waste stream analysed in each article recorded is presented in the Appendix. The majority of investigated studies were related to MCDM applications in the MSW management context as reported in Achillas et al. (2013). Indeed, 67% of the articles analysed were addressed to MSW, likely because this stream represents 24% of the total mass of waste annually generated (International Solid Waste Association, 2015) and is directly related to population. Articles addressing HCW, HW, e-waste, and NW corresponds to 6%, 5%, 5%, and 4%, respectively. The percentages for other waste streams are inferior to 3%. CDW and ISW were present only in 3% and 2% of articles, despite corresponding to 36% and 21% of the total mass of waste annually generated (United Nations Environment Program, 2015), respectively. The low number of articles dedicated to CDW and ISW is probably related to the difficulty in obtaining data for these waste streams owing to confidentiality issues from the waste-generating industry and/or from the private companies that manage the waste (Laurent et al., 2014).

It is important to note that all articles focusing on HCW, CDW, and e-waste streams were published after 2000, indicating the recent tendency of using MCDM for different waste types; however, the percentages are still relatively low. Therefore, this scarcity of MCDM applications for waste streams other than MSW shows that the waste management sector could be more effectively explored in future research. Indeed, for Gomes et al. (2008) the application of MCDM in CDW assessments can support the definition of strategies for implementing and operating new CDW recycling facilities so that the risk of failure of new projects will be reduced. Banias et al. (2010) and Coronado et al. (2011), in turn, demonstrate the applicability of MCDM methods in the evaluation of sites for the location of CDW management facilities considering economic, social, and environmental impacts. Analogously, Capón-García et al. (2014), Vahidi et al. (2014), and Zhang et al. (2010) showed that the use of MCDM methods in ISW treatment strategies selection could facilitate the decision-making process. So, based on the usefulness of MCDM in a CDW and ISW context, and considering that these waste streams represents an important part of all waste generated worldwide, the development of new MCDM studies dedicated to this topic are encouraged.

Regarding the type of MCDM used according the waste stream. For most waste streams, MODM represents less than 30% of MCDM research. Larger ratios of articles using MODM were found in the classes of ISW, NW, and others. This proportion for ISW was about 40%, whereas the others group was 50%. The more frequent use of MODM in ISW and others classes was expected, because the majority of articles related to these streams included management strategy studies. As discussed in the section ‘Work objective’, this topic represents the preferential application of MODM methods in SWM. On the other hand, studies dedicated to biodegradable residues adopted only MADM approaches.

Weighting attribution

Regarding criteria weighting, Figure 6(a) shows that 66% of the analysed articles adopted opinion-based weights. Mikulić et al. (2015) reported that opinion-based weighting procedures are a reasonable choice when judgments are based on profound knowledge. This is typical in SWM because weights are normally defined according to the opinion of experts from this sector. An additional reason for the preference of opinion-based weight is that the weights provided by this approach are considered more stable than those produced by objective evaluations (Maggino and Ruviglioni, 2009).

OPEN IN VIEWER

Figure 6.

Weighting attribution results. (a) Weighting methods used in the studied articles. (b) Participation statistics for each stakeholder group.

Equal weight and weight scenario approaches were implemented in 18% and 10% of the studies, respectively. These approaches are favoured owing to their simplicity or lack of information required to define weights by opinion-based or data-centric approaches. Obviously, equal weights are also used when criteria are equally important (Hermans et al., 2008).

Concerning data-centric approaches, 5% of the articles used these methods. For example, Caruso et al. (1993) used weights defined based on optimisation techniques, whereas El Hanandeh and El-Zein (2010) and Nakatani and Hirao (2011) defined weights using a Monte Carlo simulation. This low frequency of data-centric methods is likely related to the fact that these methods depend on information availability. Indeed, these methodologies are applicable only if there is a database with sufficient information for performing the mathematical procedures involved. Despite the aforementioned difficulties, as these methods avoid the subjectivity of weights attribution from people opinion, the use of data-centric approaches could be an alternative to assessing systems in an unbiased way in terms of stakeholder preferences. This characteristic could be useful to compare weights proposed by different stakeholder groups and the ones obtained by objective approaches. In addition, to use data-centric approaches does not mean to exclude stakeholder involvement. In Nakatani and Hirao (2011), which defines weights by deterministic methods, stakeholders were involved in the decision process by selecting the acceptance limits to each criterion, instead of defining criteria weights.

Another option is the association of data-centric methods with opinion-based approaches enabling the combination of desired properties of each one (Zheng et al., 2010). Among the studies investigated, only Wang and Zhang (2008) and Liu et al. (2014b) presented a hybrid approach to define weights combining these two weight attribution streams. Wang and Zhang (2008) combined stakeholders opinion with weights defined based on the correlation between criteria scores and global index results, whereas Liu et al. (2014b) associated expert judgments with objective weight defined based on the TOPSIS method. The aforementioned article shows that subjective and objective methods result in different weights, and based on this finding they conclude that the unilateral assessment in only one method may lead to inaccurate and incomprehensive results. Therefore, the association of both weight approaches in future works is recommended, as it achieves a compromise between subjective and objective weight methods.

Regarding the weighting process, different patterns were observed by each MCDM stream, as presented in Figure 6(a). For articles using MADM methods, opinion-based weights were used in 80% of the studies. Concerning MODM applications, equal weights was the predominant method, corresponding to 53%, followed by opinion-based methods and weight scenarios at 22% and 20%, respectively.

For articles using only opinion-based weights, 5% did not include details about the opinions source, 6% let the decision maker define weights, 18% included works based on author’s opinion, and 71% referred to stakeholders’ opinions. By combining these statistics with data shown in Figure 6(a), it was observed that the majority of works analysed used weights defined according to the stakeholders’ opinions. The high adoption of weights depending on stakeholders’ judgment shows a preference of researchers in involving experts, the private sector, and the general public in the decision-making process in the context of SWM. It is important to note that SWM has implications in public health, the environment, and the economy, and it involves multiple stakeholders with conflicting points of view. Therefore, to incorporate these differences in stakeholders’ preferences could be crucial for a correct analysis of the problems and determination of a compromise solution.

Figure 6(b) shows an overview of stakeholders’ participation in these studies. For this analysis, six groups were considered: Local community, including residents and community leaders; authorities; academics and researchers; professionals of the waste sector; private waste companies; environmentalists and non-governmental organisations (NGOs). It should be noted that the authorities group was the most frequently consulted. Considered in 37% of all stakeholders’ opinion works, this group showed a high interest in articles focusing on the government’s perspective. This involvement of authorities is justified, because they are responsible for regulating or controlling the selection and operation of SWM strategies.

Academics and waste professionals, in turn, presented a consulting frequency of 30% and 26%, respectively. The high occurrence of these two groups is likely related to the fact that both include experts with a strong knowledge of SWM-related topics, which provided a technical background to the study. Moreover, since private waste companies maintain operational expertise of SWM systems, this group was also often considered in the analysed studies, showing presence in 24% of the articles.

Local community and NGOs were included only in 14% and 11% of works, respectively. These results indicate that the public opinion sector still lacks representation compared with the other groups despite the fact that local community acceptance is a crucial factor in the implementation of waste management projects (Achillas et al., 2011; Banias et al., 2010; Coronado et al., 2011). Indeed, decisions in the waste management context involve political as well as scientific aspects and depend strongly on public opinion (De Feo and De Gisi, 2010). For these reasons, the inclusion of local community in opinion polls to define criteria weights are highly recommended in future studies.

Normalisation procedure

Regarding normalisation, 14% of the studies used the raw data with no standardisation procedures implemented, whereas 58% used normalised numbers. As presented in the section ‘MCDM fundamentals’, normalisation is directly related to the MCDM method adopted. For methods based on weighted sum criteria aggregation, e.g. AHP and some MAVT/MAUT, normalisation is mandatory; outranking methods do not have this requirement. Since AHP and MAVT/MAUT are the two most frequently used MCDM methods in SWM, it is reasonable that data normalised was used more frequently than raw data. However, 28% of the articles did not provide information on the normalisation procedure.

Concerning only articles using normalised criteria, Figure 7 shows the percentage of studies adopting a specific normalisation procedure. Three groups of methods were observed: Linear functions, in which the data were converted in common scales by using linear functions; fuzzy functions, in which the data were normalised by using a fuzzy approach; and linguistic scales, in which the data were categorised into several ranges according criteria values. In the last method, each category is related to a label, such as excellent, good, regular, or bad, which is then converted to a value in a predefined scale. Figure 7 shows that 34% of the articles that used normalised criteria did not mention the method used. Linear and fuzzy functions were preferred in 26% and 25% of the studies, respectively, whereas linguistic scales were adopted in 13%.

OPEN IN VIEWER

Figure 7.

Distribution procedures used to normalise data in the evaluated articles.

Focusing on linear functions, the most largely used method among the studies investigated, generally they use minimum and maximum values to rescale criteria, as presented in:

V ( x i ) = ( x i − x i m i n ) / ( x i m a x − x i m i n )

(4)

where x_i is the raw value of the ith criteria, V(x_i) is the criteria normalised, x_imin is the minimum value of x_i for the ith criteria, and x_imax is the maximum value of x_i for the ith criteria. These extreme values can be obtained from the dataset in the case of alternatives selection or from benchmarks previously established on the basis of reference systems and best practices. It was noted that 71% of the articles using linear functions adopted extreme criteria values from the dataset, whereas 29% of them adopted the benchmark approach. However, it is important to note that the minimum and maximum criteria values presented by the alternatives analysed can generate rank reversal if the alternatives are eliminated or added. Furthermore, this approach is not applicable if the aim of the appraisal is to evaluate individual performance rather than to compare options. For these reasons, Achillas et al. (2010), Coelho et al. (2012), Galante et al. (2010), and Hagemeister et al. (1996) adopted linear normalisation based on the benchmarks approach in MCDM studies in a WM context. So, if the MCDM method used requires data normalisation, benchmarks approach is recommended.

As previously discussed, 28% of the articles did not use normalisation, whereas 34% of studies that did use normalisation did not indicate the method used. Together, these two classes of articles correspond to 48% of all works analysed, which indicates a lack of information on normalisation in almost half of the studies. Therefore, it is highly recommended that future works should provide this information because it is crucial for better understanding of the research and allows the replicability of the procedure presented.

Sensitivity analysis

Only 40% of articles reported a procedure related to sensitivity analysis. Therefore, despite it being an important step of assessing model robustness (Dobbie and Dail, 2013), MCDM articles in SWM have generally not focused on sensitivity analysis. Actually, instead of an increase over time in the number of works reporting this procedure, which would indicate evolution and maturation of MCDM applications in SWM, the inverse was noted. Indeed, 51% of articles issued before 2000 reported some type of sensitivity analysis, whereas for the studies published after 2000 this proportion corresponds to 38%. It was also noted that sensitivity analysis was most frequently used in articles involving outranking methods. Indeed, 67% of these studies include some type of sensitivity assessment. As already stated in the section ‘MCDM fundamentals’, sensitivity analysis is essential to the consistency of a MCDM study (Qureshi et al., 1999; Syamsuddin, 2013). Therefore, to guarantee the reliability of the study, it is recommended to consider this step as mandatory in future researches.

Referring to the articles that carried out a sensitivity analysis, 72% of them evaluated the effect of criteria weight changes on the final result, 20% examined the sensitivity related to the model framework (aggregation, normalisation, thresholds), 18% assessed the effect of input data (criteria values, waste composition, and waste quantities), and 7% investigated the impact of the criteria adopted.

This predominance of sensitivity assessment based on weight changes is likely related to the fact that this type of analysis is a common tool available in commercial MCDM software. In addition, weight sensitivity analysis is often carried out probably because, as already discussed, weighting attribution normally involves decision-maker judgements. In this case, the impact of the subjectivity in criteria weight definition needs to be assessed, justifying the investigation of the effect of weight changes in the final results. However, the sensitivity of the model in relation to the criteria adopted, input data, and model details, as aggregation, normalisation, and thresholds, can be as important as, or even more important than, the one related to weights. For instance, in Gómes-Delgado and Tarantola (2006), which studied landfill location, the results of a sensitivity analysis using variance-based methods found that the criteria considered were more influent in the results variation than the weights adopted. Similar conclusions were obtained by Chung and Poon (1996), which studied waste treatment alternatives for Hong Kong. The same article also shows that normalised data and the MCDM method used could influence the final ranking of alternatives. In Bottero and Ferreti (2010), in turn, the sensitivity analysis was carried out in relation to the aggregation method, criteria weights, and input data. Concerning Hokkanen et al. (1995) and Hokkanen and Salminen (1997a), which investigated WM strategies, threshold values presented higher effects on results variability than criteria weights. So, to really assess sensitivity of a model and to guarantee its robustness, it is suggested that future works carry out such analysis considering not only weights variability, but also other aspects.