Elicitation of criteria weights for multicriteria models: Bibliometrics, typologies, characteristics and applications

Goal: This study aims to review and classify the main methods of criteria weights definition. Design / Methodology / Approach: From a systematized search were articles selected and analyzed, covering a timeline between 1949 and 2020. Results: Fifty sex methods were identified and discussed. Most (49) are subjective, eleven are objectives and three are hybrids. Considering the aggregation procedure, 33 are compensatory and 23 are non-compensatory. Most of the methods are for a single decision maker and just eleven are multidecision makers. The main contributions are published on European Journal of Operational Research journal. Limitations of the investigation: This work does not make a thorough technical analysis of the presented methods and it is not a notebook to indicate to researchers which method is the best for each type of situation. Some methods may not have been included. Practical implications: The selection and the definition of criteria weight is a fundamental question in the Multicriteria Decision Aid (MDCA) approach. The MCDA has a wide application on operation and production management. The diversity of published papers on this subject, the comprehension of its characteristics and applicability is a complex task for researchers and professionals who demand these techniques. Originality / Value: This work contributes to identify the state of the art of the models applied to the weight of the criteria, as well as to identify its characteristics and functionality regarding MCDA modeling.

The decision making process can be characterized as a sequence of tasks. At first, it specifies the problem and defines the important requirements. Then, the objectives and targets to be met are described. The next step is establish the possible solutions or alternatives, and, after, the criteria to be considered in the modeling of the problem. The following stage, a critical one, determines the multicriteria method to be employed according to the characteristics of the decision problem and its application. Lastly, a sensitivity analysis of the result and performance of the model proposed is conducted (Zardari et al., 2015). Figure 1 shows a hypothetical case of an MCDA problem and how variations in the weighting of the criteria can reflect different results in the model. A real-world decision-making problem example of this type of variation in the ranking of alternatives, in response from changes in the criteria weights, could be seen in Raigar et al. (2020). The relevance of each criterion taken by decision makers is an issue that may influence on the results of the decision process in the multicriteria models (Wang and Luo, 2010). The criteria weighting is a key factor for the accuracy of the MCDA models (Van Ittersum et al., 2007). There are many methods proposed to define criteria weight (Choo et al., 1999;Liu et al., 2012;Larsson et al., 2015) and they are classified in three categories: subjective, objective and hybrid. The criteria weight assessment is probably one of the most important issue in multicriteria modeling (de Almeida et al., 2016).
Decision problems of the MCDA involve many criteria that have to be selected in accordance with their importance to elaborate a well-founded decision. The n criteria weight elucidates, in the model, the meaning each criterion has in the decision process. The distribution weighting process represents an essential step in the multicriteria models, as it can influence on the decision making in a direct way (Wang and Luo, 2010;Alfares and Duffuaa, 2016;Paravidino et al., 2017). However, the definition process of weights is usually complex. Tens of methods that have been categorized according to two lines of thoughts were proposed for this kind of problem (Choo et al., 1999;Liu et al., 2012;Larsson et al., 2015). Weber and Borcherding (1993) and Chou (2013) suggest that the weight elicitation methods are considered to be algebraic or statistical, holistic or decomposed, direct or indirect, compensatory or non-compensatory. Tzeng et al. (1998) and Ma et al. (1999) consider they are divided into two groups: subjective and objective methods. The criteria weight represents the inaccuracies and uncertainties of the decision makers' evaluations in the subjective approach. On the other hand, mathematical models make the assessments in the objective approach, The MCDA has wide application in Operations and Production Management. Is expected that production management requires good decisions from managers, since the success of an organization is the result of choices supported by technical and objective criteria (Gomes et al., 2004). Some decisions are simple and intuitive, but, most of the time, the decision-making process is complex and difficult to elucidate by our usual cognitive skills. The MCDA is an area of Operational Research with wide application to solve problems characterized as complex, involving several criteria, many of which conflict with each other. In this way, this article contributes to support managers, technicians and researchers for the use of MCDA techniques in Operations and Production Management, with a focus on weighting the criteria weights.
The article is structured in five sections as follows. Section 1 presents an introduction and a theoretical foundation of the decision making process and criteria weight elicitation methods. Subsequently, section 2 approaches the methodological procedures. In section 3 the results were presented in three parts: bibliometry, main methods and their classification. Section 4 discusses the results and the conclusions, constraints and perspectives are in section 5.
We consulted the Scopus database for a comprehensive and extensive review to list scientific articles related to the weight elicitation methods in order to map the frontiers of knowledge. The section describes how we made the selection of the articles in the database. The research, performed in July 2020, was limited to the reports of the scientific production in the last 39 years . In this stage, we did not consider theses, reviews, and articles from conferences; we only examined papers from journals. To select the articles, we applied the Boolean characters "AND" and "OR"; in the search field, we used the expressions "Multicriteria decision making", "MCDM", "MCDA", "Multiple criteria analysis" and "Multicriteria" in order to choose the articles related to multicriteria, represented by letter M in Figure 3.
The search for weight elicitation methods was performed employing the following terms: "Determine the relative weights", "Elicitation of criteria weights", "Ranked criterion weights", "Criteria weight", "attribute weight", and "weight" represented by letter W in Figure 3. We researched article titles or abstracts or keywords for a widely analysis. To restrict the target articles the query strings were employed just to title or keywords. This procedure decreased ten times the total number of articles to be investigated. The Figure 3 presents the number of papers found by means of these two researches methods and the intersection between them. Figure. 3. Venn diagrams of papers found in Scopus database when Multicriteria (M) and methods of criteria weights (W) query strings were employed. The research was widely performed using expressions to title, abstract and keywords; and restricted by query strings just to title and keywords.

Source: The authors themselves
The query employed was TITLE-ABS-KEY ((("Multicriteria decision making" OR "MCDM" OR "MCDA" OR "multiple criteria analysis" OR "multicriteria") AND ("determine the relative weights" OR "elicitation of criteria weights" OR "ranked criterion weights" OR "criteria weight" OR "attribute weight" OR "weight"))) AND (LIMIT-TO (DOCTYPE, "ar")). This search resulted in 2.237 articles; after a review of titles, abstract and keywords, we selected a set of 36 articles, which formed the basis for analyzing the theme of interest in details. From this sample, the research was extended to scientific books, journal paper, and theses, which approach the criteria weight methodologies. This way, we found 41 scientific documents, in which we analyzed the methods in detail. The analysis showed there have been documents of the area of interest since the first half of the twentieth century; thus, the final assessment covered the period from 1949 to July 2020.
In order to evaluate the connection between the articles and the theme suggested, and the recurrent terms in these publications, we followed the procedure proposed by de Jesus and  by means of including the abstracts of the articles analyzed in the Wordle application. Subsequently, the works were analyzed according to the frequency of publications, journals, main authors, countries, and continents. After selecting the most important articles, the methods of criteria weight elicitation were analyzed and categorized the methods of criteria weight elicitation were analyzed and categorized into subjective, objective, hybrid, compensatory, non-compensatory, single decision maker and multidecision maker (Zardari et al., 2015).

Statistics
The Figure 4 presents a word cloud generated from the insertion of the research article abstracts into the Wordle application. We observed that the terms "criteria", "method", "preference", "weights" and "decision" have greater emphasis in this cloud, indicating an association within the scope of the research. These terms mean the highest occurrence frequency over the article abstracts analyzed. Source: The authors themselves The Figure 5 shows the quantitative results of the number of scientific texts published regarding the criteria weighting methods per year. Only one scientific texts was indexed per year from 194919902000-2003-2014. Between 1999 scientific texts were indexed. There were peaks during 1986, 2015 and 2016. There was a greater number of documents indexed in 2015.   Table 2 shows the distribution of the criteria weight methods per continents, countries, and scientific texts. The text of Stillwell et al. (1981) presented three methods published in one article, and Edwards and Barron (1994), two methods in one article. The prevalence of one method published in each recorded text was noted. The number of citations is based on data from Scopus (number of articles in journals) and Google Scholar (books and theses). One text is highlighted, presenting the largest number of citations, which is the book of Keeney and Raiffa (1976), with 15,259 citations, as it is classical in the MCDA. This number of citations does not mean that the methods were the most applied, but that they were the most referred by other authors.  In this step, we used the "QGIS" tool to elaborate the map. Figure 6 presents the distribution map of the number of methods per continent, highlighting the territories that indexed articles about the criteria weight methods.

Characteristics of the methods
Because of the diversity of criteria weight methods, they were separated into categories to make the understanding of their characteristics and functionalities easier. This sector was subdivided into three categories: subjective, objective and hybrid methods. The compensation (compensatory and non-compensatory) and number of decision makers (single decision maker or multidecision maker) are discussed in section 3.3.

Subjective Methods
In this item, the methods were subdivided into cluster, among them: fuzzy, use of cards, linear programming, tradeoff, swing, ordinal ranking, (statistical and algebraic), and use of vectors (Figure 7).

Fuzzy
Some subjective methods employ the fuzzy set theory, or fuzzy logic, in their conception. Yeh et al. (1999) developed the Task Oriented Weighting, in which the relation of the criteria with specific task requirements or factors obtains the weights. A set of fuzzy IF-THEN rules can represent the assessment of the influence of each task requirement in the criteria weight. In this set, the linguistic variables are used to evaluate the relation between the task requirements (T1, T2...Tn) and the criteria (C1, C2...Cn). Triangular or trapezoidal fuzzy numbers represent the linguistic variables in order to facilitate the computational process. To define the weights, the basic criteria weights are first determined regardless the task requirements. The decision maker or other criteria methods can directly attribute these basic weights, such as the Analytic Hierarchical Process (AHP) proposed by Saaty (1980). The authors demonstrated the application of the method in a case study at a dredging company in China, analyzing how to better dredge a river. Zang et al. (2004) study a situation where preference information on criteria is given for multiple decisopn makers in different formats. The opinion of the decision maker is evaluated in some different formats and the degrees of the preferences are used to determine the weights of the criteria. They used a multiplicative preference relation to transform different preference formats (pairwise comparison, preference orderings, utility value, vector of linguistic terms, selected subset, fuzzy selected subset and normal preference relation), into a single one by a fuzzy linguistic quantifer (Q). Then, a new function of geometric means aggregation rule is used to obtain the overall values of the criteria weight. This method is Multiple Preference Format (MPF). Liu et al. (2012) developed the Consistent Interval Fuzzy Preference Relation (CIFPR). In that method, the weights are defined by consistent and inconsistent interval fuzzy preference relations, in which the interval fuzzy preference relations are transformed into interval multiplicative preference relation. This method is accomplished in sex steps: 1) consideration of a decision problem with an alternative finite set; the decision makers judge the alternatives based on intervals of fuzzy preference relations; 2) transformation of the interval fuzzy preference relations into interval multiplicative preference relation; 3) consistency check; 4) obtaining weight vectors; 5) definition of the possibility degree matrix; 6) use of a simple method for eliminating row and column to obtain the ranking vector from the possibility degree matrix.
Sotoudeh-Anvari and Sadi-Nezhad (2015) employed the Z-numbers method that uses the fuzzy set theory to generate criteria weights. This method is an adaptation of Zadeh (2011) work, who proposes a new fuzzy concept that approaches, in an efficient way, the subjectivies of information. When compared to other Fuzzy methods, the Z-numbers method has more ability to deal with the uncertainties of information. It consists of two components, according to the model: Z= (N, M). The N component is a restriction of the real value of a variable, and the M component measures the accuracy of the first component. Many fields can apply the method, for example, the economic, decision analysis, risk evaluation, prediction, among other areas. Wang (2014) developed the Fuzzy Quality Function Deployment (FQFD) method, which integrates the Relative Preference Relation method with the fuzzy models to elicit criteria weights. This method was proposed in order to avoid the multiplication of the triangular or trapezoidal fuzzy numbers. An equation that substitutes the original weights according to the relative preference relation determines the adjusted criteria weights. This method can be divided in eight steps: 1) the aim of the problem and its criteria are defined; 2) experts judge the criteria by linguistic terms; 3) the linguist terms are expressed in fuzzy numbers; 4) the means of fuzzy importance is calculated; 5) the degree of preference for the level of significance is determined; 6) the adjusted weight matrix is defined; 7) the means of the adjusted weights are calculated; and 8) the degree of relative preference degree for the means of the adjusted weights is determined.
The Choquet Integral method, propose by Choquet (1954) and after improved by Murofushi and Sugeno (1991), formulate weights by the Principal Components Analysis (PCA), which verifies the interactions among criteria (Rowley et al. 2015). This is an aggregation method for multiple objectives, in which the decision maker defines the preferences for multiple criteria. This procedure is recommended to drive uncertainty and subjectivity-base assessment. The Choquet Integral is an aggregation operator that generalizes the weighted arithmetic average when criteria interact among them, based on fuzzy measures. The fuzzy logic determines the importance of each criteria set instead of considering them independent. A λ-fuzzy measure was proposed by Sugeno to facilitate the fuzzy measure related to the set to be aggregate. However, these methods have deficiencies related to slow convergence, the need for human information and high computational time consumption (Pacheco, 2016). This approach was applied in two case studies in the environmental field. The first study examined the Triple Bottom Line of 135 sectors of the Australian industry considering 11 criteria. The second examination evaluated the environmental life cycle of eight alternatives of treatment for biosolids management, with five criteria. Torfi et al. (2010) applied the Fuzzy Analytic Hierarchy Process (FAHP) for criteria weights. It can lead with uncertainties and subjectivities in judgments. The decision maker judges the criteria by means of linguistic terms, which are converted into triangular fuzzy numbers. These numbers are used for peer comparison to determine weights. The FAHP can be summarized as follows: 1) it normalizes the comparison matrix between [0,1]; 2) it verifies the consistency of the normalized matrix; 3) it employs the pertinence function in the normalized matrix; and 4) it calculates the criteria fuzzy weights. Other uses of the AHP for criteria weight are presented in this paper. Sarker and Biswas (2020) proposed the use of Pythagorian Fuzzy Set (PFS) method to sove the elicitation of weights criteria to multicriteria group decision making. These proposal is a clear trends of how the weights criteria methods should be develop in next years. They employed more than one MCDA methods together to solve more robustly this type of problem. Considering a problem where the uncertainty of information is captured more prominently and leads to provide proper evaluation of decision makers, the PFS was proposed (a entropy weight model) to determine initial criteria weights. Following, these weights are optimized using linear programming model for obtaining the optimal criteria weights. Based on Hamming distance and Hausdorff metric, a new PF distance measure is defined considering membership, non-membership and hesitancy degree of Pythagorian fuzzy number to enhance the proficiency of the modified TOPSIS. The authors suggests that the PFS methodology ensures to take more accurate optimal choice in group decision process.
To solve some shortcomings of the entropy method, Podvezko et al. (2020) propose extension of the earlier introduced criterion impact loss (CILOS) method to fuzzy MCDM. Two methods, the fuzzy CILOS (FCILOS) and fuzzy entropy meyhod (F-entropy) were combined and employed in Fuzzy Integrated Determination of Objective Criteria Weights (FIDOCRIW) as a way to elicitation of the criteria weights. This procedure mitigating shortcomings of the Fentropy method appearing when the method is used in the fuzzy space. Simos (1990a, b) presented the SIMOS Weighting Method that allows the decision makers to think and express their perceptions in a hierarchical structure of different criteria sets in a scenario. This technique informs the experts the data needed to attribute quantitative value to the criteria set. That procedure is described in five steps: 1) each decision maker selects n colored cards, which represent the n criteria of the problem. The name of the criterion and its objective is in each card. In case it is necessary, white cards are used to separate the colored ones; 2) the decision maker organizes the cards from the less essential to the more essential one. When a group of decision makers are challenged to put cards of different colors in a same position, showing that these criteria present similar weights; 3) the decision maker inserts the white cards between the colored ones to express his/her preference. The more the number of white cards, the more the difference between the criteria weights; and 4) the weights are calculated and normalized according to the order of the colored card and the distance of them, defined by the number of white cards between the colored ones.

Cards
From the procedures of SIMOS, Figueira and Roy (2002) developed the Revised SIMOS Weighting Method. The revised procedure of SIMOS differs from the standard method in three aspects: (a) it adds new information of the decision makers; (b) it organizes the information to the normalization of the weights; and (c) it uses a new weight normalization method that reduces the inaccuracy of rounding. Those procedures improve the proportion of weight attributed among the more essential and the less essential criteria and alter certain computational rules of the previous method.
To solve certain gaps of the previous models and give them greater robustness, Siskos and Tsotsolas (2015) developed the Robust Simos Method based on the Simos Weighting Method. They suggested an algorithm to the initiation stage of procedures. This procedure consists of changing the criteria hierarchy, recommended by the decision makers, into a convex n-dimension polyhedron P, where P is defined by all linear restrictions in the model. Then, the holistic procedures are applied in two stages, which gives higher robustness control. The first stage analyzes the stability of criteria weights; the second one evaluates the result of the decision model.
With similar approach (use or cards), Pictet and Bollinger (2008) presented the Extended use of the cards procedure. In the method, the names of the criteria represent the cards. This procedure is divided into three steps: (1) organization of the cards to get an initial order of the criteria; (2) insertion of the white cards among the criteria cards to show the difference of importance; and (3) presentation of a relation of importance f among the more essential and less essential criteria. The method orders the criteria in reverse order; in case of equality, the cards have to be placed in the same position. Subsequently, it calculates the weights by a predetermined formula.

Linear Programming
Some methods apply the linear programming as a principle. Srinivasan and Shocker (1973) developed the LINear programming techniques for Multidimensional Analysis of Preference (LINMAP). It analyzes the individual judgments of each decision maker, estimating the coordinates of ideal points and weights to investigate the judgment of the preferences. The paired comparisons give the preference relationship in an alternative set. The model considers ordinal preference data or interval data. Vansnick (1986) suggested another method that utilizes linear programming. He developed the Treatment of the Alternatives according to the Importance of Criteria (TACTIC). The TACTIC is a non-compensatory method, which employs vectors of additive weights. This method deals with the details of information given by the decision makers in a reasonable way, trying not to exceed the characteristics of the information. An algorithm and a linear programming determine the criteria weights. This method is based on four stages: a) the decision maker provides information about the criteria; b) next, data from the first stage are processed by an algorithm, which considers the weights and defines a value for a parameter from the information produced in the first step; c) the third step consists of processing data in an algorithm that establish the global interactions in the alternative set; and d) finally, the TACTIC method shows graphically the relation of global preference, which helps the understanding of the decision maker.

Swing
Von Winterfeldt and Edwards (1986) presented the Swing Weighting Method (SWING), which consists of including, in the evaluation matrix, a hypothetical alternative that considers the worst performance in all criteria, having, thus, a value of 0. This alternative is used as a parameter for comparison. Then, the criteria are ranked in preference order of the decision maker, which defines the ordering answering the question: Suppose you have to choose this alternative, if you could improve its performance, which criterion would be improved? This procedure has to be applied in all criteria until they are ordered. As a criterion is evaluated attributing weight, it is separated from the process to make the weighting of the other criteria easier. The value 100 is attributed to the most important criterion. The other ones are analyzed giving a value less than 100. Then the weights are normalized. Edwards (1977) developed the Ratio Weighting method. The decision maker classifies the criteria according to their importance; the less important criteria are worth 10 and the others, considered multiples of 10, when the results are normalized on a scale from 0 to 1. The attribute weights used for the representation of all these methods are derived by normalising the sum of given points from the original data. Methods adopting this approach range from quite simple rating procedures (Riabacke et al., 2012). Edwards and Barron (1994) showed the Simple Multi-attribute Rating Technique Swings (SMARTS) e SMART Exploiting Ranks (SMARTER) methods that use linear value functions. The SMARTS method aims to eliminate the complexity of judgement, exploring the ordering of the criteria according to their relative importance; for this, it uses the swing weights. The SMARTER method utilizes the formally justifiable proceeding of criteria weight suggested by Barron (1992), Barron and Barrett (1996).
In SMART, the expert is asked to identify the least important criterion, which receives for example 10 points, and thereafter the user is asked to rate the remaining criteria relative to the least important one by distributing points. The SMARTS had been criticized because the degree of influence of a criterion depends on its spread (the range of the scale of the attribute) and its do not consider the spread specifically (Riabacke et al., 2012). Ahn (2017) proposed a nonlinear aggregation method, called Maximum Entropy Ordered weighted Average approach (MEOWA), where the weights are associated with the objects reordered according to their magnitudes in the aggregation process. The MEOWA presents a result similar to the Rank Order Centroid (ROC) method that will be discussed later. The MEOWA method is reliable in situations where the decision-maker need to judget in a scenario of many alternatives and attributes. This method must be used in decision problems under uncertainty.

Tradeoff
Some methods apply the tradeoff principle, in which the weights show the importance of a criterion compared to another criterion in a compensatory manner (de Almeida-Filho et al., 2017). Keeney and Raiffa (1976) presented the TRADEOFF weighting that consists of a mathematical function, which integrates information on multiple criteria so that the alternatives represent a value function. The decision-maker is asked to choose one of the alternatives, thereby indicating the more important one. The decision maker changes the alternative punctuation regarding each criterion to obtain inequalities that allow the definition of the criteria weights. The expert is asked to state how much he would be willing to give up on the most important criterion in order to change the other to its best consequence, that is, state the trade-off (Riabacke et al., 2012).
The method uses the inequations to determine the criteria weight values based on an initial criterion having pre-set weight. de Almeida et al. (2016) created the Flexible and Interactive Tradeoff method (FITtradeoff) to improve the use of the traditional tradeoff, making the use easier for the decision maker and keeping its axiomatic structure. This method has a number of procedures that can be easily modified and adapted to different conditions and circumstances. The implementation of the elicitation process does not necessarily follow all the standard steps, making the model adjustable to different modeling. Thus, each process step can be used according to different steps during the process.
The way for value obtainment in trade-off method is operationally more complex and more cognitively demanding in practice due to the large number of pairwise comparisons required. Moreover, there is a tendency to give greater weight to the most important attribute in comparison to methods like SWING (Riabacke et al., 2012).

Ordinal Ranking
There are methods that use the substitution of weights (surrogate weights) as principle, in which the decision makers express the importance of the criteria by means of ordinal ranking. A extended review of ranking ordering criteria weighting methods is presented by Roszkowska (2013) and Danielson and Ekenberg (2017). The advantage of those methods is the fact that they rely only on ordinal information about attribute importance. Ordenal ranking methods can be used for instance in situations of time pressure, quality nature of criteria, lack of knowledge, imprecise, incomplete information or partial information, decision maker's limited attention, in a group of decision makers and information processing capability.
In these methods the ratios among weights are determined by the conversion of ranks into ratios. The paper of Roszkowska (2013) and takes a good approach on the following ranking methods that convert such rankings to numerical weights: Equal Weight Method (EW), Ranking Sum Weight Method (RS), Rank Exponent Weight Method (RE), Inverse or Reciprocal Weight (RR), Rank-order Centroid Weight Method (ROC) and, maybe not appropriated to this subject, the AHP method. Danielson and Ekenberg (2017) proposed the Considering Cardinal Rank Sum (CSR) and discuss their advantages in relationship to other rank methods. Stillwell et al. (1981) developed three methods: Rank Sum Weights (RS), Rank Reciprocal Weights and Rank Exponent Weight. The criteria are ordered in the Rank Sum Weight; then, the weights are attributed from the relation of importance of the criteria and the sum of positions that are linearly decreased. The Rank Reciprocal Weights is based on the RS; however, the weights are attributed by reciprocals (inverted numbers). The Rank Exponent Weights is a generalization of the RS. The decision maker judges the most important criteria weights on a scale of 0 and 1, which is formulated by the interactive processes.
Considering the principle of weight substitution, Solymosi and Dombi (1986) proposed the Centralized Weights method, which is an interactive method, in which the decision maker expresses the importance of the criteria by means of ordinal comparison. In this method, the "most important" relationship is assumed as a semi-ordinate. During the process, a viable set of weights and a threshold value α are restricted according to the judgment of the decision maker. Convex combinations obtain the weights and the value α.
Another ordinal ranking method, the Rank-Order Centroid (ROC), proposed by Barron (1992) and formally presented in Barron and Barrett (1996), uses ordinal information for weight attribution, which are obtained from an analysis of the subjective information of the ranks. This method reduces the maximum error of each criterion weight by means of identifying the centroid of all possible weights, considering the order of classification of objective importance. From the vertices of a simplex, the centroid weights are defined for each criterion.
Based on the ROC, Larsson et al. (2015) developed the Cardinal and Rank Ordering of Criteria (CROC) method, which consists of two steps. The first step, called extraction, involves interaction with decision makers, who judge the distance between the criteria to obtain cardinal weights. This step is subdivided into three steps: (i) decision makers obtain ordinal ranks, in which the criteria are ordered from the most essential criterion to the least essential one. The most important criterion is arranged in the upper part and the least important one, in the lower part; (ii) the decision maker evaluates the maximum distance between the criteria; (iii) the criteria are transformed into cardinals according to the rank obtained in step i. The second step is the phase of interpretation, in which the information obtained is processed in cardinal weights. Alfares and Duffuaa (2008) developed the linear-slope variable method (VSL), which transforms an ordinal classification into numerical weights. This approach develops a simple mathematical expression to define the weights of the attribute according to its classification and to the total amount of attributes.
As argued by Riabacke et al. (2012), the CROC method is better than ROC weight method because the handling imprecise and cardinal information, and aims to reduce effects of noisy input on the data extraction step and in the interpretation of the weights. This aspect is a way to lessen the decision-makers' reluctance to reveal their true preferences.
Ahn (2017) developed a method of criteria weight called the minimizing squared deviations of extreme points (MSD), an extension of the ROC weighting method. The search method locates the barycenters of the criteria weights, minimizing the sum of the squares of each vertex. Lolli et al. (2019) employed the PROMETHEE-based ranking approach to elicit the weights of criteria. First, a linear model is introduced to search the most discriminating vector of weights, if the information provided by the user is the ranking of a subset of reference items. At least an iterative quadratic model is proposed to solve data-driven weight eliciting problems. The method Data-driven of the Quadratic Model QM2 -DDQM2 -is proposed and compared with linear model. The DDQM2 were found to be more robust in relationship to linear model but, generally, slightly less precise than linear model. The authors recommended the use of DDQM2 because it avoid constraints violations. Costa and Vansnick (1994) developed the Measuring Attractiveness method by a Categorical based Evaluation Technique (MACBETH). This approach allows the decision maker or groups of decision makers to evaluate alternatives using qualitative judgments about difference in attractiveness. The decision maker expresses his/her judgment based on a semantic scale formed by six categories: (C1) very weak difference of attractiveness; (C2) weak difference of attractiveness; (C3) moderate difference of attractiveness; (C4) strong difference of attractiveness; (C5) very strong difference of attractiveness; and (C6) extreme difference of attractiveness. This method differs from other multicriteria methods because it considers only qualitative judgments about the difference of attractiveness between two elements in order to generate numerical values for options in each criterion. The criteria weights are attributed by means of a matched comparison of the attractiveness of the alternatives, attributing the greatest weight to the alternative that presents greater attractiveness.

Vectors
Based on paired comparison procedures, Saaty (1980Saaty ( , 1990 developed the Analytic Hierarchy Process (AHP), a multicriteria method that has in its procedures an elicitation of criteria weights step. The criteria weights happen by means of a matched comparison matrix of the criteria, where this matrix is converted into a vector of criteria weights. The AHP involves the following steps: 1) definition of the decision problem; 2) construction of the hierarchical structure: structure in hierarchical levels, the problem of decision, criteria, sub-criteria, and alternatives; 3) definition of priorities: it elaborates a matrix of criteria and sub-criteria comparing each element by pair according to preference scale of Saaty, from 1 to 9; 4) eigenvalues calculation: priority ordering; 5) normalization of the eigenvalues for each criterion; and 6) consistency index test and logical consistency ratio.
In AHP the pairwise comparison between criteria is stablished by semantic scales (e.g., "important", "very much more important", etc.) for stating importance of the weights. Nevertheless, some authors have criticized the conversion of real values from Saaty scale (Riabacke et al., 2012). Costa (2017) proposed a hybrid multicriteria ranking method, AHP -De Borda, where the AHP is used as a sharing technique to weighting criteria problem. This proposal is an example of integration between an American school MCDA methods and a classic French MCDA methods, in this case an outracing method. The AHP -De Borda proposal shows the fusion of these two lines of thought in a complementary way by segmenting the problem into two ones and applying each technique for the specific piece of the problem.
Based on the AHP, Srdjevic and Srdjevic (2011) proposed the Weights Estimation by Evolution Strategy Algorithm (WEESA) method, which suggests a bi-criterial optimization evaluation to estimate weights in the AHP comparison matrix. An algorithm is the main part of the search element that simultaneously preserves the estimates of the priority vectors. The coding model and other elements of the search engine are regulated for restrictions related to the normalized values of the weights. Amenta et al. (2020) give a good contribution proposing weights for aggregating judgments in AHP group decision making (AHP Frobenius). The method could be applied in problems with a large dispersion and when the decision makers are unable or unwilling to reconsider their judgments. The novitiate was the measure of congruence between the decision makers and propose a formal method based on the Frobenius norm to compute a suitable set of coefficients for aggregating the individual judgments into a common group preference matrix. The authors highlight the fact that discordant behaviours of the decision makers by means of the introduction of a measure of congruence. Ginevičius (2011), who developed the Factor Relationship (FARE), proposed a method that addresses the relationship between criteria. In the FARE method, the decision maker assigns a minimum amount of information about the relationships between a part of the criteria set as well as the parameters of strength and direction. The direction expresses the positive or negative relation of the criterion, showing the influence or the dependence with other criteria of the system; strength reflects intensity of impact. The criteria weights are determined when the total impact of each criterion is known in relation to the other criteria of the set. Rogers and Bruen (1998) used a technique based on formal psychology. They applied the Hinkle's method or 'resistance to change grid' method proposed by Hinkle (1965) to weigh criteria in non-compensatory models. This method is presented in the following steps: 1) the relevant criteria are listed; 2) two sets of criteria are created: one recommendable and one non-recommendable; 3) each criterion is compared in pairs in its respective sets; 4) the decision maker is asked about the possibility or intention to change some criterion to the nonrecommendable side, and which of them he/she would not be prone to change. After these comparisons and the reassessment of the judgments, the weights are determined.
Statistical and Algebraic Chu et al. (1979) presented the Weighted Least Square (WLS) method to elicit weights using an easy-to-understand concept. The model involves sets of simultaneous linear algebraic equations. Diaby et al. (2016) proposed the ELICIT model, which is based on two steps: ordering and criteria weight. In the first step, the criteria are ordered based on the preference of the decision makers by principal component analysis. In the second step, the criteria weights are estimated applying descriptive statistics using analyses of independent variables and Monte Carlo method. The ELICIT method was employed in a hypothetical case study, involving the criteria weights of five criteria, utilized for the choice of surgical equipment. The criteria were classified from 1 to 5, from the preferences of the decision makers. The weight of each criterion was defined according to the standard deviation and the confidence interval at 95% probability. This method is appropriate in situations where decision makers make ordinal classifications to obtain criteria weights. Another method that uses statistical parameters to define weights is the CCSD proposed by Wang and Luo (2010) and the CRITIC presented by Diakoulaki et al. (1995), discussed in the section on objective methods. Todeschini et al. (2015) suggested The Weighted Power-Weakness Ratio (wPWR), which is a derivation of the Power-Weakness Ratio (PWR), originally developed by Ramanujacharyulu (1964). This method presents a multivariate and comprehensive approach to solve decisionmaking problems. It can simultaneously quantify the interactions between alternatives, offering an understanding of the structure. The key factors of the wPWP are: (1) its multivariate form; (2) the ability to simultaneously analyze the strengths and weaknesses of the criteria; and (3) the possibility of weighing criteria according to the partial knowledge about the decision problem.
The Best-Worst Method (BWM), is a vector method proposed by Rezaei in two seminal articles (2015,2016). The BWM derives the weights based on a pairwise comparison of the best and the worst criteria/alternatives with the other criteria/alter-natives. These method have many similarities to the AHP, considering some principles and applications, and, as the authors, BWM performs better than AHP. Rezaei (2015), argues that BWM requires a lower number of pairwise comparisons and guarantees more reliability in the judgments, as it uses only integer numbers in the matrices. Lately, the BWM was proposed by Raigar et al. (2020) as a methodology to sharing criteria weights in a problem to select the most appropriated additive manufacturing (3D printing) process.

Objective Methods
In this item, the methods were subdivided into three clusters: statistical and algebraic (three methods), probabilistic (one method) and heuristic (one method).

Statistical and Algebraic
Statistical (mean, standard deviation and correlation analysis) and algebraic parameters are used in the methods described in this item. Diakoulaki et al. (1995) suggested the Criteria Importance through Intercriteria Correlation (CRITIC) method, which determines criteria weights by the analysis in the decision matrix to extract all the information included in the criteria investigated. The criteria are weighted according to the expression which Wj = the criteria weights; Sj = standard deviation; rjk = correlation coefficient (Pearson) between the j and k criteria. According to the expression, the higher the value of Wj the greater the amount of information transmitted by the criterion and the higher the relative importance of the criterion in the decision-making process. For application, the criteria weights are normalized between [0,1]. This method was applied in a case study in Greece, where eight pharmaceutical companies were selected and analyzed in relation to the criteria of profitability, market share, and productivity. Wang and Luo (2010), developed the Integration of Correlations with Standard Deviations (CCSD) technique, approach another method that considers statistical and algebraic principles. Like CRITIC, this technique integrates the correlation coefficient (CC) and standard deviation (SD) to determine criteria weights in the multicriteria decision aid, in addition to performing a sensitivity analysis of the weights. The CCSD method determines the weights of criteria by integrating the standard deviation for each criterion and its correlation coefficient with the overall evaluation of decision alternatives. The correlation coefficient is obtained by removing one criterion at a time from the set of criteria and considering the correlation with the overall evaluation of alternatives of decision without inclusion of criteria removed. If the CC for the removed criteria is high, then the removal of this criterion has little influence in the decision making, otherwise it should be given an important weight to the criterion removed. A case study was applied to assess the economic benefits of industrial activities in China. Sixteen municipalities were evaluated in relation to five criteria of economic benefits. Deng et al. (2002) considered the Mean Weight (MW) method, which weights criteria by the following formula: w = 1 / m, in which m is the number of criteria. MW assumes that all criteria are equally important. This method is applied in situations characterized by the lack of information necessary to determine the relative importance of the criteria. Shannon and Weaver (1949), who presented the concept of Entropy, present an approach of probabilities. This approach was later employed by Hwang and Yoon (1981) and Zeleny (1982) to determine criteria weights. This method is applied in information theory, assessing uncertainties and assigning probability to situations of risk of the information. The Entropy Method is used to quantify the uncertainty represented by a discrete probability distribution, p. The criteria weight considers the relative importance of a criterion in a given decision situation that is directly related to the amount of information, the greater the diversity of values, the greater the importance of the criterion in the final decision.

Heuristic
A heuristic method to a single decision maker were proposed by Ciomek et al. (2017) for prioritizing pair-wise elicitation questions. Through of an additive value function this procedure is oriented to minimizing the number of pair-wise questions that should to be answered by de decision maker while he/she not sure about preferred alternative. This procedure is indicated when the decision maker have to emits opinions about several options (when more than a few dozens of alternatives are considered). In this case a heuristic method reduce the number of pair-wised questions, concentrating the effort in the most preferred criteria. Ma et al. (1999) proposed a technique integrating the objective and subjective methods. This technique integrates subjective information provided by decision makers and objective information in order to determine a programming model for assigning weights. This method considers matrices of comparison by pairs of criteria elaborated by a decision maker; all criteria are objective. In the comparison by pairs of the subjective decision matrix, the weights are determined by subjective considerations of the decision maker. In the objective decision matrix, weights are specified by objective information. Thus it integrates objective and subjective factors to establish a model from two matrices, one objective and one subjective. A goal programming model (Gpm) was proposed by Shirland et al. (2003) to generate weight estimates by means of pair-wise comparison between the criteria, using triads to evaluate multiple criteria. The use of triads reduces the apparent number of comparisons by the decision maker during the evaluation of items in the questionnaire. There are two benefits from the triad comparison: the first benefit is the fatigue reduction of decision makers in evaluating the criteria. The second benefit is the reduction in the number of intransitences of the questionnaire results.

Hybrid Method
A new current hybrid method (Yang et al., 2017) combining linear programming (LP) and minimax reference point optimization (LP-Rpm) to determining criteria weights. This method consists in three phases. In the first phase, preliminary weights are determined as initial inputs for the third stage. Then, a LP model is applied to obtain favorite weights for each alternative in order to maximize its utility. In the third phase, it is used the minimax reference point optimization to aggregate the sets of weights of the previous steps in order to find the best weights of criteria. An example of application of this methodology were conducted in two case studies. The first study assesses the performance of 28 European Union countries in relation to nine criteria related to medical essential indicators provided by Thomson Reuters. The second case analyzes the performance of 985 Chinese Project universities.

Classification of the Weight Elicitation Methods
Among the 56 methods found, 49 of them are subjective and 8 are objectives. It means that they the judgment of the decision makers is based on their cognitives preferences and few methods consider mathematical models to obtain the criteria weight. Three methods should be classified as hybrids.
The compensatory methods, with 33 tools, are more common than non compensatoy ones. The compensation is established by the tradeoff, where the change in the weight of a single criterion promotes the change in the weights of the other criteria. Non-compensatory methods comprise 23 of their total and. These methods are recommended for problems where the decision maker has high reliability in some criteria judgments, which should not be altered due the variations in the weighting of criteria that are still poorly understood.
Concerning the number of decision makers involved in the decision making process, the single decision maker methods represent 45 of the total, while the multidecision methods are 11. Table 3 summarizes the methods listed in section 3.3.

DISCUSSION
This study reviewed the criteria weight elicitation methods, which are originally applied to MCDA models. However, their applications extend to other fields of Operational Research, Heuristics, Computational Intelligence, among others. This review organizes the publications scattered in the literature and presents an overview of the characteristics of the proposed methods.
Weights of criteria is an essential part of MCDA methods considerably influencing the result of multiple criteria evaluation. This research matter it is of great practical and theoretical applications (Podvezko et al., 2020). Currently there are many contemporary methods that allow to elicit weights and some good reviews on this subject have been proposed (Weber, Borcherding, 1993;Riabacke et al., 2012;Roszkowska, 2013;Zardari et al., 2015;Danielson and Ekenberg, 2017). Riabacke et al. (2012) addressed a review made an analysis of the most common subjective methods of weight elicitation. Zardari et al. (2015) characterized the criteria weight elicitation methods into subjective and objective ones, highlighting some methods such as Swing Weighting, Simos Weighting, Entropy Method, Critic Weighting, among others.
The scientific production, its periodicity, the main texts, journals, continents and countries that most contributed to this area of knowledge were answered. Regarding the periodicity of publications in this field, it is worth highlighting the year 2015 as having the largest volume of publications ( Figure 5). The main texts that stood out for their scientific contributions are those of Stillwell et al. (1981) and Edwards and Barron (1994), who proposed the Rank sum, Rank reciprocal, Rank exponent, Smarts and Smarter methods ( Table 2). The most popular journals for the dissemination of these studies are the European Journal of Operational Research (29%), the Omega (9%), the International Journal of Information Technology & Decision Making (6%), Applied Soft Computing (6%), and Computers & Operations Research (6%) ( Table 1). This subject could be useful for researchers clearly identify journals to search articles or future submissions, keywords to be used when searching for articles about criteria weighting.
The American continent, represented mainly by the United States, with 15 published methods, stood out as the place that contributed most in this area. Two major schools of thought MCDM influence the criteria weight methods: American School and French School. The American School presents methods of single-criterion of synthesis, which aggregate the criteria into a single utility value. Among others, the multi-attribute utility theory, AHP, MACBETH, SMARTER stand out.
The French School (European) is characterized by outranking methods, based on a relationship of prevalence, in which one alternative may have a degree of dominance over the other; the methods of the ELECTRE family and PROMETHEE family (Gomes et al. 2004) are highlighted. Regarding the criteria weight methods, a polarization for the American continent is observed, with 16 methods proposed. Among these 16 methods, the American School, such as FITradeoff, Roc, Swing, influences most among others. Europe presents 13 methods of criteria weights, in which the thoughts of the French School (European), such as the Simos, It is observed a degree of influence of the French and American Schools, in which the application of fuzzy logic predominates in the criteria weights procedures such as, Fahp, Znumbers, among others. Finally, Oceania presents three methods that do not belong to any of the two schools of thought.
Many methods had been used for weighting criteria in matrices with pairwise comparations, like the eigenvector (frequently used), least (logarithmic) squares method, the spanning tree, etc, and this kind of problem can be seem as a multi-objective optimization problem These authors highlight that the use of eigenvector in weighting pairwise comparison matrix (ex. used in AHP analysis) may be inefficient and suggest the multi-objective optimization problem as a unique solution for consistent pairwise comparison. The linear programs are recommended to construct an efficient dominating weight vector.
Some articles analyzed present methodological similarities. As an example, it can be cited the use of cards to evaluate the judgment of decision makers, which procedure was originally proposed by Simos (1990a, b), and later used by Figueira and Roy (2002), Pictet and Bollinger (2008), and Siskos and Tsotsolas (2015), with variations in the original methodology. Many methods present proposals for association with other conventional methodologies.
The methods discussed in this paper can be observed in Table 3. It presents the main methods of weight elicitation in MCDA and its classification in the following categories: subjective, objective, hybrid, compensatory, non-compensatory, single decision maker and multidecision maker. Regard the number of decision makers required in criteria weights, most of the methods were developed for a single decision maker. Qin et al. (2017) point out that a single decision maker cannot deal with all the variables inherent in decision making, since different points of view are exposed in the course of the decision. Research proposals should be stimulated for multi-decision maker methods.
An important issue in the MCDA models is the compensatory relation of the methods. They can be classified as compensatory and non-compensatory (Guitouni and Martel 1998). In non-compensatory methods, the weights do not suffer tradeoffs, that is, the weight change of one criterion does not influence the other weights of criteria. In many decision problems, non-compensatory models best represent the decision maker's preference (Fishburn 1976). For a more classical definition of non-compensatory structures, see Fishburn (1976), Bouyssou andVansnick (1986), andVansnick (1986). In the compensatory models, the weights interact in the sense that the lower weight in a criterion reflects in a greater weight in another criterion, so that the variation of the weight of a single criterion promotes the change in the weights of the other criteria. This means that the information extracted from the relative importance of the weights determines tradeoffs between the criteria (Ishizaka and Nemery 2013). The use of compensatory or non-compensatory weight elicitation models depends on the characteristic of the decision problem.
Just eight methods are objectives. Objetive methods are characterized by not incorporating the view of the decision makers, since they are based on information obtained from quantitative variables. Although the objective methods are not sufficiently studied, they are more interesting when we have real data, avoiding empirical judgments. Despite being better methods for exact variables and the mathematical rigor of the process, objective methods are criticized for neglect the subjective judgment information of the decision maker (Zardari et al., 2015;Luhrman, 2006).
On the other hand, the subjective methods are more approached in current literature, with 49 proposals. These methods are characterized by involving judgments of the decision makers within an acceptable range of values, in which it is sought to extract their preferences, and this information, often uncertain, are incorporated in the criteria weights (Riabacke et al., 2012, Ahn, 2017Zardari et al., 2015). Despite the increase in researches, the process of weight elicitation subjective is still a challenge for the MCDA due to the following question: how to specify subjectivity? The researchers' effort to reveal the frontier of knowledge about subjectivity can be explained by its applicability to real problems, which, in most cases, present these characteristics. Subjectivity can be understood as the way each person thinks and interprets a subject, constructed by means of acquired knowledge, in the cultural, economic and social contexts (Luhrmann 2006).
Targeting subjectivity is perhaps one of the greatest challenges in the MCDA. In an attempt to minimize the subjectivity of the MCDA models, the connection with Artificial Intelligence is explored. Artificial Intelligence is applied in many fields of research, mainly focused on predictive models and technological systems that imitate human behavior. There is an increase in research involving the integration of these two ideas (Doumpos and Grigoroudis, 2013). As technological knowledge advances, in which it is necessary for algorithms to make autonomous decisions, Artificial Intelligence aims to interpret the subjectivity in a more objective way.

TRENDS
There are many mathematical models developed to describe decision maker behavior and they are used in Operational Research. The MCDA provides powerful approaches to solve complicated problems in many areas. The training of professionals with technical knowledge to operationalize these models is essential. In view of the many possible techniques to be employed, the inappropriate use of a technique can suggest a situation of wrong decision. Many aspects of MCDA science should be better understood to improve the effectiveness of this technique. Themes like different scaling methods, preference relations, aggregation procedures and fuzzy tools, mixing and hybridization of methods are a "hotspot" in future researchs.
MDCA models exert a powerful impact on global economy. Economical decision making is extremely complex due to the intricacy of the systems considered and the competing interests of multiple stakeholders (Zavadskas and Turskis, 2011). Considering the value that MCDA decision techniques have in the economy, this area should be a priority in the economic management of companies (public and private) and organizations. Conducting scientific research on this subject should be encouraged so that we have increasingly realistic and reliable models in the future.
Many authors (Riabacke et al., 2012;Roszkowska, 2013;Danielson and Ekenberg, 2017) agrees on how challenger is eliciting adequate quantitative information from people in decision analysis problems. To evaluate the effectiveness of an elicitation criteria weights method, Riabacke et al. (2012) highlights the importance of three conceptual componentes: extraction (how information is derived through user input), representation (how to capture the retrieved information) and interpretation (ability of the representation used and how to assign meaning to the captured information in the evaluation of the decision model used).
The extraction component is the most error-prone as it concerns the procedural design of the method which could be cognitively demanding during user interaction. One trend in approaches for extracting the required information in a less precise fashion is methods based on visual aids or verbal expressions (Riabacke et al., 2012). This matter, regarding the fidelity of the representability of mathematical models the input data is an aspect that researchers need to consider when proposing new models used in MCDA problems. Moreover, elicitation methods that are more direct are easier and less likely to produce elicitation mistakes.
Many elicitation methods are too cognitively and demanding for a data value manifested by decision makers. There is a clear need for weighting methods that do not require formal decision analysis knowledge (Danielson and Ekenberg, 2017). It is specially important whem the decision makers have a tendency to be overconfident in their judgments, overestimate desirable outcomes, and seek confirmation of preconceptions. Lolli et al. (2019) suggests the development of elicitation of weights to cognitive data. Indirect procedures aimed at eliciting the criteria weights is suggested to be used for inferring the DM's preference structure, while reducing his/her cognitive efforts as much as possible. The importance of reducing the cognitive efforts of the decision maker is a natural matter to be studed. Zeleny (1982) and Kao (2010) argue that there are two ways of eliciting the weights of criterion importance: direct explication and indirect explication. The direct procedure or priori weights consists in obtaining the weights of the criteria (surveys, experts opinion, rulers, etc) before the data of the alternatives are collected and analyzed. In this case, the criteria weights is a guide for future development of a MCDA problem. In indirect explication models or posteriori weight, the weights are determined after the data of each alternative is collected. As opposed to direct explication where the weights. As these two authors the posteriori weights represent the emphases of the alternatives being evaluated and this way is more convincing because the weights are a reflection of the data. This criteria weighting composition by posteriori weights is a way where the MCDA research should be necessary in the coming years.
The paper of Sarker and Biswas (2020), about multicriteria group decision making, is an example how the weights criteria methods should be develop in next years. They employed more than one MCDA methods together to solve more robustly this type of problem. A better understanding of methods of weight elicitation will produce more effective in MCDA problems and contribute to widespread use of decision methods.
Another trend in handling preferential uncertainties and incomplete information in a less precise way is by using intervals as representation where a range of possible values is represented by an interval. The fuzzy theory is designed to quantify the uncertainties and inaccuracies of the information. Thus, the fuzzy approach is a robust tool to deal with the inaccuracies of decision problems (Kahraman et al., 2006). The use of fuzzy logic to treat subjective data should be adopted for this type of variable. Fuzzy weights allow to use the complete fuzzy structure of the fuzzy decision matrix; to use fuzzy MCDM methods; to combine subjective and objective fuzzy weights to hybrid weights; and to evaluate alternatives in the environment of uncertainty.

CONCLUSION
This study provides a systematized review of the criteria weight elicitation models applied in multicriteria modeling. From the Scopus database, a set of initial articles was extracted, which served as a basis for a more detailed analysis of the topic. Based on this sample, the research was expanded to other scientific documents (books, journal papers and theses), which address the criteria weight methodologies. The work presented a synthesis of the functionalities of the methods, classifying them as subjective, objective, hybrid, compensatory, non-compensatory, single decision maker and multidecision maker. The study covered 56 criteria weight methods; most of them, 49, used a subjective approach, eight are objective and there are 3 hybrid methods. Concerning the aggregation procedure 33 were compensatory and 23 non-compensatory. Most methods apply to problems with only one decision maker and just 11 multidecision makers.