Fuzzy logic–based evaluation of visualizations generated by intelligent decision support systems

Abstract

The development of intelligent decision support systems requires much research effort to solve decision-making problems’ complexity. In fact, the combination of both intelligent components and visualization aspects in intelligent decision support system required a lot of efforts in order to develop advanced information visualization schemes for decision-making processes. For this, an efficient evaluation of these systems has become a major concern for applications in multiple fields. The reports of the existing usability evaluation studies are helpful to verify the potential and the limitations of these tools. However, it is important to integrate more relevant metrics for visual analytics tasks in dynamic intelligent decision support system. The proposed method consists of a questionnaire that is given to the users and a subsequent analysis of the resulting data using fuzzy logic. The advantage of the fuzzy model is its ability to transform the input survey scores into linguistic variables, as well as linguistic evaluation of the overall intelligent decision support system visualization tool. With this approach, it is possible to model the vagueness in the ordinal judgments obtained from the users’ evaluation about the visualizations of intelligent decision support system and to support uncertainty in such evaluation.

Keywords

Evaluation intelligent decision support systems visualization fuzzy logic

Introduction

This article proposes a new evaluation method of visualizations generated by decision support systems (DSSs) based on data mining methods, known as intelligent DSS (iDSS). In the literature, evaluating visual representations is based on classical and novel methods. It involves dimensions in usability evaluation such as effectiveness, efficacy, and satisfaction and seeks to determine how well visualizations generate insight to their users in reasonable time and with reasonable satisfaction.^1,2

The existing evaluation methods use the following: (1) quantitative techniques such as surveys/questionnaires, pre/post tests, and heuristic analysis;; (2) qualitative techniques such as observations, interviews, and focus groups;³ or (3) “mixed method” approaches that collect both quantitative and qualitative data.^4,5 Visualization evaluation studies are either based on the user tasks and data characteristics or observational by learning about the users.^6,7 As far as we know, no study captured vagueness in the ordinal judgments obtained from the users’ evaluation on the visualizations of iDSS. For this reason, we propose the use of fuzzy logic technique to analyze responses provided by the users and determine an accurate evaluation based on the combination of criteria characterizing visualizations generated by iDSS.

Fuzzy logic

Fuzzy logic is an approach to computing based on “degrees of truth” rather than the usual “true or false” (1 or 0) Boolean logic.⁸ The interest of fuzzy logic comes from the need to model the human operator behavior with the system rather than modeling the system. This human behavior must be described by qualitative variables rather than specifying numerical values.⁹ A fuzzy set is defined by its “membership function,” which corresponds to the concept of “characteristic function” in classical logic, while considering intuition.⁸ A membership function is defined by linguistic variables (e.g. height) and linguistic terms (e.g. medium, high). These variables and terms can be used directly in fuzzy rules.⁹ The fuzzification allows switching from the real field to a fuzzy one by translating input into truth-values. It consists in determining the degree of membership of a measured value to a fuzzy set.⁸

Fuzzy logic aims to formalize and implement the way of humans’ thinking using the inference mechanism and based on fuzzy rules.¹⁰ It computes output truth-values. Then, these values will be transferred into output (defuzzification).

In this article, we propose to evaluate visualizations in imprecise terms, in much the same way that our brain takes in information. These visualizations are generated by specific systems: iDSS.

iDSS

iDSS is of growing importance in increasingly complex environments. It enables inductive and predictive approaches of data analysis to facilitate decision-making by integrating data mining techniques. It assists organizations to take important decisions faster with a higher confidence level and lowers the uncertainty in the decision process.¹¹ While data mining techniques are developed independently from human–computer interaction and visualization techniques, coupling them showed interesting visual analytic results in many application areas.¹² Generating overviews and exploring data space allow visually extracting potentially useful information. Medicine is one of these areas where data mining methods have been significantly used in combination with visualization methods.¹³

The availability of large amounts of data has attracted the interest of the data mining and visualization communities^14,15 to the opportunities offered by both the raw data and the extracted patterns.¹⁶ Many iDSSs based on data visualization were designed and developed in the literature.^17–20 Visual representations provided by the iDSS include representing data and deducing visually interpreted relationships.¹⁸ We consider the fuzzy logic (defined in section “Fuzzy logic”) more appropriate to evaluate these representations for the reasons presented in the following section.

Why fuzzy logic?

As data mining–based DSS is supposed to allow discovering new trends in data and to generate useful knowledge for decision-making, its evaluation must take into account this specificity. That is, not only to verify the user interface’s usability but also to check their ability to visually create and transmit knowledge for the user (i.e. decision-maker).

To answer this kind of evaluation question, it would not be accurate enough to give or calculate a single number corresponding to a value in some scale. It would be possible to generate natural qualitative evaluation answers such as “very high” and “very low.” These evaluations cannot be described by any single value. For instance, what number stands for very high or very low? The evaluation seems to be even more complicated by answering this question with this vague expression.

Our idea is to express uncertain, vague, and imprecise users’ evaluations in a fuzzy form. Each expression evaluating particular criterion is converted into the form of fuzzy measure inputted in a fuzzy controller and the output is computed using the inference mechanism. Thus, the fuzzy logic–based evaluation allows simultaneously evaluating visualization by handling numerical data and linguistic knowledge.

The scope of our research is to outline a proposed intelligent usability evaluation method to assess the visualizations generated by iDSS. The ultimate aim of our proposed method is not only to integrate new relevant metrics but also to apply the fuzzy logic technique. To verify its applicability, the proposed method is applied to a case study in the medical field. It consists in evaluating an existing iDSS tool for the daily fight against nosocomial infections in the intensive care units (ICUs).

This article is organized as follows: section “Related work” describes the related work used to explain our theoretical framework. Section “The proposed evaluation method” introduces our proposed usability evaluation method. Section “Application and results” describes its application in the medical field and its results. Finally, section “Conclusion” presents our conclusions and open questions.

Related work

Visual iDSS

Decision-making has become a critical part of many information structures. A DSS allows users to have permanent access to updated information that interests them. DSS should not be limited to information retrieval. The challenge is to be able to use the available information in order to gain a better understanding of the past and support the future through better decision-making. Data mining^21,22 technology responds to this challenge.

Data mining–based DSS, commonly named iDSS, is a rapidly expanding activity that represents analytical information technologies’ spectrum. In fact, data mining methods (1) extend the data analysis possibilities by exploiting richer model representations, such as decision rules and decision trees, than traditional statistical methods and (2) make analysis results more comprehensible especially to the non-technology users.²³

This interdisciplinary field of iDSS includes statistical tools, machine learning, and pattern recognition ones to support interesting patterns’ discovery for the sake of facilitating decision-making.²⁴ There are examples of iDSSs in the literature that have already been designed and developed. Bose and Sugumaran²⁵ presented the intelligent data miner (IDM) DSS. It is a web-based application system enabling predefined and ad hoc data mining model creation. It provides data mining phases and necessary parameters for used algorithms. It also allows the connection with an external data mining software tool in order to create a data mining model. Polese et al.²⁶ proposed an iDSS that uses association rules as data mining technique to support decisions of a basketball coach by proposing tactical solutions based on the data of the past games. Lee and Park²⁷ introduced customized sampling DSS (CSDSS) which is a web-based system enabling the autonomous generation of available customized sampling methods using clustering data mining technique. Ltifi et al.²⁸ introduced an iDSS that aims to generate possible solutions for the fight against nosocomial infections in the ICUs. It allows using the dynamic Bayesian networks as data mining technique for the aim of calculating daily nosocomial infections’ occurrence probability.

While iDSSs are used to improve data analysis and decision problem solving, decision-makers could easily understand the relationships of related data. Several visualization techniques in the context of iDSS have been developed to help people gain more value from significant data collections. Related literature in information visualization and iDSS presents different roles that visualization techniques could play in data mining for decision support. In fact, various research works integrate a set of visual data mining techniques. For classification purposes, association rules’ visualization, interactive mosaic plots,²⁹ rule polygons,³⁰ parallel coordinates,^31–33 three-dimensional (3D) graph,³⁴ and CrystalClear with grid view³⁵ are proposed. Ma and Hellerstein’s³⁶ study allows ordering categorical data to improve the visualization clutter. An opportunity map visualization framework was proposed to identify actionable knowledge.³⁷ Other types of visualization show useful results in the knowledge discovery from databases process for decision-making.³⁸

Nowadays, a lot of application domains of iDSS provide huge sets of data such as medicine and science. In this context, several visualization tools have been proposed in the last years due to the need of intelligent decisional applications to analyze these large amount of data.

Recently, various visualization tools have been developed for visual analytics to support decision-making. Ankerst et al.³⁹ introduced a CalendarView as visualization tool integrated into a data mining architecture called DataJewel. Munaga et al.⁴⁰ developed a novel tool called CAST (Clustering And visualizing Spatio-Temporal data) as a solution for studying and analyzing the navigation of moving entities (users, vehicles, etc.). Ramírez-Ramírez et al.⁴¹ discussed a tool called SIMID (SIMulation of Infectious Diseases) providing medical decision-makers with effective spatio-temporal visualization of infectious disease spread. Mittelstädt et al.⁴² introduced a visual iDSS for adverse drug event detection. Fischer et al.⁴³ proposed a visual analytics tool called NVisAware aiming at compressing data streams for visually identifying events to solve the Kronos incident problem. Mittelstädt et al.⁴⁴ presented visual analytics tools to support response in crisis situations. They introduced specific visualizations for mobile devices and control rooms. These iDSS visualization tools are evaluated in controlled case studies to measure their usability with predetermined user tasks or using expert reviews in the framework of evaluation methodologies.⁴⁵

Visualization tools’ evaluation methods

The evaluation of information visualizations is an essential component of the iDSS design process,⁴⁶ allowing researchers to quantify differences across visualizations and iterate as part of the design cycle.⁴⁷ Numerous alternatives to assess visualization tools exist in the literature.⁴⁷ Hartson and Hix⁴⁶ assessed a set of visualization techniques’ usability by interpreting and comparing the results against a predefined usability specification. The study of Kobsa⁴⁸ allowed effectiveness comparison of visualization tools using controlled experiments. Examples of used metrics for heuristic evaluation are design guidelines,³ visual properties effectiveness,⁴ evaluation cognitive models,⁴⁹ and preventive processing and perceptual independence.⁴⁵ Many case studies in realistic settings evaluated the impact of visualization tools on real users’ practices, such as the workplace study in Gonzales and Kobsa,⁵⁰ the field study in Rieman,⁵¹ and the case study in Shneiderman and Plaisant⁵² Mazza and Berre⁵ used the focus groups’ interviews to explore users’ attitudes and beliefs and visualization tools in cognitive tasks. Saraiya et al.⁵³ presented an evaluation method based on the characteristics of data insight that allows identifying and quantifying it in the user’s tests. Kosara⁵⁴ used critical inspection as a form of evaluating visualization tools. Kurzhals et al.⁵⁵ proposed the use of benchmark data to evaluate visualization methods and analysis techniques for gaze processing. Their benchmark data consist of three parts: the dynamic stimuli in the form of video, the eye tracking data, and annotated areas of interest. Ragan and Goodall⁵⁶ discussed an evaluation methodology for comparing memory and communication of analytic processes in visual analytics. Ho et al.⁵⁷ introduced a new approach for evaluating flow visualization methods based on eye tracking analysis.

iDSS generates specific visualizations that aim at visual knowledge discovery and integration for decision-making. Evaluating their usability is challenging because visual data mining tools combine several components (such as visual representations, data transformation, analytical reasoning, and interaction tools and results’ communication) that are integrated into complex interactive DSS used by analysts and decision-makers. Users perform exploratory tasks and might not follow a predefined workflow. To understand these behaviors, an evaluation must target these components and the analytical work environment. Traditional evaluation metrics and methods presented above seem to be insufficient to assess the usability of iDSS visualizations. Within this framework, a new fuzzy logic–based evaluation methodology is proposed. It aims to model the vagueness in the initial judgments obtained from the users’ subjective evaluation of the visualizations of iDSS and to support uncertainty in such an evaluation.

Limitations of other methods

Usability testing and controlled experiments limit user’s thinking and performance¹ that is inappropriate since the evaluation aims at informing the development of a visualization tool. Heuristic evaluation is more difficult to use since it implicates a list of heuristics and cannot identify some unexpected problems.⁵ Focus groups and case studies in realistic settings allow uncovering these problems.⁵ Unfortunately, they consume time to conduct an evaluation and the results could not be generalizable.² We also find more recent methodologies based on insight, inspection, and specific metrics including memory, communication, and eye tracking. These metrics proved to be interesting and generic for visualizations. However, they do not take into account the users’ interview biases and do not capture vagueness in the ordinal judgments obtained from the users’ evaluation. Moreover, they do not determine the precise weights reflecting the importance of criteria to assess the impact of visualization on the decision-making environment.

A fuzzy logic analysis seems to be suitable for its ability to model the uncertainty that characterized iDSS and to use the subjectivity of the users in the evaluation.⁷ In fact, as the human brain thinks in fuzzy reasoning, evaluating iDSS must take into account inaccuracies and uncertainty by allowing an element (i.e. evaluation criterion) to belong to a set with partial membership and not with absolutely belonging. Table 1 summarizes the criteria for judging that fuzzy logic. It is interesting for evaluating iDSS visualizations regarding other existing methods.¹⁰

Table 1.

Intelligent evaluation features.

Criteria	Evaluation methods
	Evaluation using fuzzy logic	Evaluation using existing methods
Handling uncertainty	Yes: thanks to objective evaluation	No: due to subjective evaluation
Handling ambiguity	Yes: by modeling human behavior and human thinking through partial membership	No: due to the use of absolute membership
Handling inaccuracy	Yes: by using linguistic variables and approximate data	Yes: criterion is taken into account by several of these methods
Handling vagueness	Yes: by focusing on the evaluation context	No: absolute evaluation with abstract responses

Using fuzzy logic to interpret initial users’ evaluation results makes it possible to describe vague ideas and represent uncertainty (cf. Table 1).¹⁰ Fuzzy controllers have the feature of interpreting and using the knowledge acquired from the human statement in contrary to classical controllers.⁹ It recourses to the imprecise linguistic description of the evaluation results via a set of control rules that are intrinsic with the knowledge base. Actually, it is very difficult to express knowledge in precise terms. The process of interpreting conclusions from provided data is called inference where new truths are inferred from the old.⁹

In a fuzzy evaluation system, the values may be absolutely true, absolutely false, or anything in between. This causes a frequent encounter of ambiguities and contradictions. Fuzzy logic provides structured ways of handling ambiguities and contradictions in many sources of uncertain data.

In this background, the importance of fuzzy system theory is their ability to deal with these ambiguities allowing to this intelligent technique to be the suitable method, while the other evaluation theories fail. In fact, faced with complex problems, in particular iDSS, the human being must use approximate data and fuzzy logic as a perfect way to describe qualitatively these approximate variables. For this, our aim is to design a new way of modeling human evaluation of iDSS visualization via fuzzy logic.

The proposed evaluation method

The idea of this article is to evaluate visualizations generated by a specific kind of systems (visual data mining–based DSSs). Our contribution is twofold: first, we aim to assess these visualizations using an appropriate questionnaire and taking into account their usability and their visual analytics ability. Second, an intelligent analysis of the resulted users’ judgments, relying on the fuzzy logic technique, which seems to be interesting.

In this context, we propose a procedure that occurs in two steps: (1) the first consists of answering a set of evaluation questions by the users (cf. sections “SET 1: participant background” and “SET 2: evaluation variables”) and (2) the second applies the fuzzy logic technique to analyze the users’ responses (questions of SET 2 in section “SET 2: evaluation variables”). It is an automatic fuzzy application (cf. section “Step 2: fuzzy logic application”).

Step 1: usability evaluation questionnaire

For the first step, we adopt the questionnaire as a quantitative technique to gather evaluation data from a large number of users. It is considered as the most common and popular tool. A good questionnaire can be a powerful tool to inform the evaluation.^47,52,58 The participants have to answer two sets of questions: (1) about participants’ knowledge and experience and (2) summary questions about the visualization tools’ usage after a training session.

SET 1: participant background

We have designed a 5-point Likert scale questionnaire to allow participant provide feedback on their background. Such information allows determining whether they have certain skills and/or background requirements, whether they will be familiar with the evaluation tasks, or have experience with performing certain tasks.

The Likert scale questionnaire measures the participants’ level of expertise and experience with database concepts, management systems, query language, visual data mining, and DSSs (Appendix 1).

Training session

It consists of presenting the following: (1) the motivations behind the visualization tools in the context of dynamic and intelligent decision-making and (2) the detailed instructions of the tools’ usage and their interactivity (zooming, labeling, filtering, etc.) which includes tools’ demonstration. During this session, the participants have the opportunity to explore the tools by themselves and learn how to use them without any exterior intervention.

The training session ends with answering a set of questions relating to the data manipulation and analysis. For each one, participants have to complete the following survey: the difficulty of the questions and the time to answer them.

SET 2: evaluation variables

After the training session, the second set of questions aims to verify how a visualization tool supports the generation of actionable and relevant knowledge for decision-making. In particular, we evaluate the visualization tool’s ability to support visual analysis and reasoning about data.⁵⁹ We introduce a new set of relevant metrics. They are all based on the current usability studies and experts’ recommendations for DSS, data mining, as well as visualization aspects and characteristics.^{1,2,7,47,52,60–62} The participants have to reply to the questions associated with the following criteria (cf. Table 2). Users attribute a value between 1 and 10 for each criterion. A linguistic variable must be emerged using an intersection of numeric value given by participants with the membership function of a particular criterion.

Table 2.

Evaluation criteria.

Criterion	Linguistic variables	Evaluation question
Comprehensibility	Very low, low below medium, medium, above medium, high, very high	To what degree do you consider the information is comprehensible?
Information retrieval	Very low, low, below medium, medium, above medium, very high, high	Evaluate the information retrieval ability
Speed of use	Very low, low, below medium, medium, above medium, high, very high	Qualify the speed of information’s loading
Navigating in time	Very low, low below medium, medium, above medium, high, very high	Specify the degree to which you find the visualization tool’s navigation simple and comprehensible?
Data comparison	Very clear, clear, below medium, medium, above medium, unclear, very unclear	To what degree do you consider the visual object’s comparison is clear in terms of data variables?
Changes over time	Very clear, clear, below medium, medium, above medium, unclear, very unclear	Qualify the identification, tracking, and understanding of the data changes about visualization tool use
Data and knowledge quality	Very low, low below medium, medium, above medium, high, very high	To what degree do you believe the tool provides helpful and important data for mining and knowledge to integrate in the decision-making process?
Causal dependencies	Very low, low below medium, medium, above medium, high, very high	Verify the exploration clarity of the causal relationships between variables
Visual prediction	Very low, low below medium, medium, above medium, high, very high	To what degree the tool is preferment in the prediction task?
Overview	Very low, low, below medium, medium, above medium, high, very high	To what degree the visualization tool is able to provide an overview of the data?
Details	Very clear, clear below medium, medium, above medium, unclear, very unclear	To what degree the visualization tool is able to explore data at different levels of detail?

The evaluation criteria are grouped into three categories:

Category 1: usability

2.1. Comprehensibility (COM): an important factor in visualizations for analytic purpose.

2.2. Ease to use (ETU): the way to use interactive options or execute actions for a visualization task.

2.3. Speed of use (SPU): the speed to use interactive options or execute actions for a visualization task.

2.4. Performance (PRF): concerns the speed of working and the information retrieval.

We propose this set of usability criteria to address, in particular, the performance and users’ subjective satisfaction about the visualization tool.^47,52

Category 2: data manipulation

2.5. Navigating in time (NGT): ensure the navigation in the visualization tool.

2.6. Data comparison (DTC): comparing visual objects in terms of the variables that define the data.

2.7. Changes over time (CHT): assess the identification, tracking, and understanding of data changes.

2.8. Overview (OV): the ability of the visualization tool to provide an overview of the data.

2.9. Details (DET): the ability of the tool to explore data at different levels of details.⁶³

The proposed data manipulation criteria focus on the ability of the tool to incorporate and understand the data. Selected criteria are common in several visualization evaluation methods (cf. section 2.3).

Category 3: patterns discovery

2.10. Data and knowledge quality (DKQ): it is the belief in which the tool provides the user with helpful and important data to mine and the knowledge integrated into the decision-making process.

2.11. Causal dependencies (CD): verify the exploration of the causal relationships between variables.

2.12. Visual prediction (VP): measures the performance of the visualization tool in the prediction task.

We suggest this list of criteria to verify the tool ability to visually provide knowledge about the displayed data.

Table 2 presents possible linguistic variables associated with the evaluation questions. These evaluations do not imply any representative value. The solution for treating uncertainty in the users’ evaluations is to express them in fuzzy numbers’ form corresponding to the linguistic variables of the membership functions. Since a fuzzy number can be considered as a generalization of the membership function interval, it is equal to a quantity whose value is imprecise. In Table 3, we present the fuzzy numbers of our evaluation system. The numeric value representing the meaning of each linguistic variable range is used as the parameter to the Mamdani inference system. In this context, modeling with fuzzy numbers allows the incorporation of uncertainty in evaluation survey results. The final result will reflect more truthfully the opinion of the human evaluator.

Table 3.

Evaluation criteria.

Linguistic terms	Fuzzy numbers		Mean
Linguistic terms	From	To	Mean
Very low	0	1.5	0.75
Very unclear	0	1.5	0.75
Low	1.5	3	2.25
Unclear	1.5	3	2.25
Below medium	3	4.5	3.75
Medium	4.5	5.5	5
Above medium	5.5	7	6.25
High	7	9	8
Clear	7	9	8
Very high	9	10	9.5
Very clear	9	10	9.5

The participants could end Set 2 by presenting suggestions for improvement and/or extensions.

Step 2: fuzzy logic application

The application of fuzzy set theory⁸ on the participants’ answers is a suitable approach for its use of linguistic variables (verbal values) and its ability to deal with limited and vague information. An example to show the impact of using fuzzy logic for evaluation is of Chou’s⁶⁴ work. The author proposed the use of the fuzzy logic to deal with customers’ feeling and demands in order to translate them into the design elements of a product. They claimed that it is difficult to quantitatively evaluate customers’ preferences on attributes of products because such preferences involve the human perceptual interpretation with certain subjectivity, uncertainty, and imprecision. This work presents a fuzzy linguistic preference approach for evaluation. The proposed approach is based on fuzzy linguistic variables associated with the fuzzy weighted average techniques for aggregating preference information.

In this context, we propose a formal tool for representing and analyzing survey results issued from a questionnaire. The fuzzy logic application for visualizations evaluation is expected to handle these survey results by reasoning using linguistic variables. It is based on a set of rules and organized into three steps (cf. Figure 1):

Fuzzification: translating numeric variables in linguistic variables with a membership function defined in the interval [0,1]:

µ_A: X → [0,1]: quantifies the grade of membership of the element in X to the fuzzy set A

where A is the fuzzy set and each element of X is mapped to a value between 0 and 1.

Motor inference: applying the rules of the fuzzy logic method to exploit the tolerance for imprecision and uncertainty. A fuzzy rule is a linguistic IF–THEN construction that has the following form:

If < input variable is A > then < output variable is B >

where A and B are the descriptors of knowledge pieces containing linguistic variables and the fuzzy rule expresses a relationship between inputs and outputs’ variables.

Defuzzification: producing crisp results given the fuzzy sets and their corresponding membership degrees.

Figure 1.

Fuzzy inference system.

In the fuzzy logic method, we assign to each category linguistic variables using the membership function that allows the graphical representation of a fuzzy set. In Figure 2, two examples of fuzzy sets are visible: (1) the first represents an example of input fuzzy set for the changes over time criterion (where associated linguistic values vary between very unclear and very clear) and (2) the second represents our output fuzzy set, which is the evaluation criterion (where associated linguistic values vary between low and very high). The x-axis denotes the universe of discourse and the y-axis denotes the membership degrees in the [0,1] interval.

Figure 2.

(a) Changes over time input fuzzy set and (b) evaluation output fuzzy set.

In our context, we propose to use the Mamdani’s fuzzy inference method,⁶⁵ which is most frequently seen as fuzzy approach. Mamdani’s work was based on Zadeh’s⁶⁶ 1973 publication on fuzzy algorithms for complex systems and decision processes. This method consists of (1) combining the fuzzified inputs according to the fuzzy rules (which will be explained subsequently) in order to establish rule strength, (2) finding the rule consequence by combining the rule strength and the output membership function, and (3) combining the consequences to generate an output distribution.

The implication method (Mamdani’s method) is

{μ'}_{conclusion} (y) = MIN (μ_{premise} (x_{0}), μ_{conclusion} (y))

(1)

It integrates the most common defuzzification method, which is the centroid technique. It allows finding a point that represents the center of gravity (COG) of the output fuzzy set. It is the abscissa of the COG of the area under the curve results

Output = \frac{\int y \cdot μ (y) \cdot dy}{\int μ (y) \cdot dy}

(2)

with universe of discourse (all the considered output values).

Rules’ base

The fuzzy rules are generated from (1) the expert’s reasoning and linguistic expressions and (2) the relationships between the variables. The fuzzy output is concluded by the degrees of realization and the consequent rules’ part. The rules should be aggregated to provide a membership of the output fuzzy variable to a consolidated fuzzy set. We consider that the rules are linked by an OR

μ_{B} (y) = MAX [μ_{Bi} (y)]

(3)

where i belongs to indexes of enabled rules.

Rule antecedent is created following sets of evaluations. However, the rule consequent is determined by the evaluator (human expert) by assigning a linguistic value to the linguistic variable usability of the evaluated criteria. The evaluator chooses the linguistic value of particular criterion according to the intersection of a fuzzy number for the criterion with its membership functions. Then, he or she defines a new fuzzy rule depending on the linguistic value and his or her expert knowledge and experience. With each new evaluation, a new fuzzy rule is added allowing learning our inference module and more accurate results. Generating automatically a fuzzy rule base diminishes the subjectivity of establishing evaluation for the iDSS visualization tools. Figure 3 presents two examples of generated rules by our system.

Figure 3.

Examples of generated rules.

In the following section, we present the application of our method to evaluate of the existing visualization tools in the medical domain.

Application and results

The goal of our work is to introduce an intelligent evaluation methodology assessing the visualizations generated by iDSS. This novel approach allows user’s uncertain evaluation. Since we think that it is not sufficient to apply the existing evaluation methods to support such uncertainty using classical binary logic, this article presents a new evaluation approach, based on artificial intelligence. It interprets and analyzes initial questionnaire evaluation results using fuzzy logic. This new method will be applied for two medical case studies in order to verify its applicability.

Case study: visual iDSS for the fight against nosocomial infections

The proposed evaluation method was applied to an existing medical visual iDSS³⁸ under use in the Teaching Habib Bourguiba Hospital of Sfax, Tunisia. This system was developed to analyze the clinical parameters (antibiotics, medical acts, and infectious examinations) to calculate the daily probability of nosocomial infections’ occurrence in the ICU. It integrates the dynamic Bayesian networks’ technique^28,67 and a set of visualization techniques (such as the perspective wall, the LifeLines, the star and tabular representations). We proposed to evaluate three visualization tools: the perspective wall for the temporal data visualization, the LifeLines, and the interactive bubble chart for the temporal knowledge visualization.³⁸

Perspective wall technique

It is a visualization technique based on a geometric transformation that focuses on the visualization of the linear structure of data. It arranges information from the left to the right on a virtual wall. This technique is well suitable for visualizing temporal large amounts of information. The data are presented chronologically on several panels (three in general). The central panel is in the foreground, and the other panels (right-hand side and of left) are behind the plane, which gives an effect of prospect corresponding to the concept of “focus + context.” The central panel corresponds to a given period, the left to what is anterior to this period and the right to what is posterior.⁶⁸

LifeLines technique

It offers a visualization space that can be applied to medical records, professional histories, and other types of biographical data. This technique allows a one-screen overview that shows multiple facets of the records. Concerning the features, they are displayed as individual time lines.⁶⁹

Interactive bubble chart technique

It is a technique that was issued to visualize a dataset with four dimensions. The first two dimensions are visualized as coordinates, the third as color, and the fourth as size. Such visualization allows comparing and presenting the relationships between labeled circles. It has the ability to (1) automatically generate trend lines and interactive bubbles, (2) adjust them by the decision-makers, and (2) consequently update the associated data.⁷⁰ In the following section, we present our evaluation of these three visualization techniques developed in the system of Ltifi et al.³⁸ and Elouni et al.⁷⁰

Evaluation method application

Step 1

Thirty participants divided into five groups were invited to participate in our user test study: each group includes two ICU resident physicians, two health informatics (HI) professionals, and two volunteer assistants from the university community. We assigned six users (one group) per tool (two per user profile). Based on their profiles, the users belong to one of the following categories:

Domain expert: resident physicians with extensive experience in nosocomial infections surveillance.

Domain novice: HI professionals having some experience with medical data analysis.

Software developers: graduate student research assistants having an M.S. in computer science.

Figure 4 presents the average values describing the participants’ experience. Bars refer to the questions (1.1.–1.5) enumerated in section 3.1. According to their answers, all users have been familiar with the database concepts, the used databases management system, and the query language idea. However, most of them are not familiar with the visual data mining and decision-making technologies, which require a training section (cf. section III.1).

Figure 4.

Background information of participants.

Then, the participants followed a training session to help them understand the tools to be evaluated as described in section III.1. (Session progress and results are presented in Appendix 1.)

Step 2

The usability questionnaire was implemented to allow participants entering, for each item (criteria), a value from 1 to 10 and helping to calculate the corresponding scores. Once all responses have been recorded, evaluation scores will automatically appear on the final user interface. As mentioned previously, three conducted surveys are fulfilled to evaluate the perspective wall (allowing the visualization of antibiotics taking by a patient during his or her hospitalization period), the LifeLines (presenting the decisions taken during the patient hospitalization period), and the interactive bubble chart tools (displaying the calculated probabilities by the dynamic Bayesian networks).

Our user study does not consist in comparing the two existing visualizations, but in verifying whether the proposed method can intelligently analyze the user reviews, and that the calculated results are close to their way of seeing these visualizations.

Perspective wall evaluation

An example of the user interface presenting the answers of a physician on the perspective wall evaluation is visible in Figure 5. According to the participant answers (cf. Figure 5(a)), the system activates two rules (R26 and R27). Applying the Mamdani method, the evaluation value obtained is 7.48 (cf. Figure 5(b)). By projecting this value on the curve, we find that the evaluation is 59% high and 28% very high (cf. Figure 5(b)).

Figure 5.

(a) Questionnaire user interface of the perspective wall tool and (b) evaluation result of the perspective wall for the first physician.

For the other participants, perspective wall evaluation results are presented in Table 4 in the discussion section.

Table 4.

Evaluation results of three groups.

Participant	Results
Perspective wall tool (first group of participants)
Physician 1 G1	COM = 7, ETU = 7, SPU = 7, PRF = 8, NGT = 8, DTC = 6, CHT = 6, OV = 9, DET = 7, DKQ = 8, CD = 6, and VP = 5 Activated rules: R26 and R27 E = 7.48 → projection: 28% very high and 59% high
Physician 2 G1	COM = 9, ETU = 8, SPU = 7, PRF = 8, NGT = 7, DTC = 9, CHT = 7, OV = 7, DET = 7, DKQ = 8, CD = 6, and VP = 4 Activated rules: R26 and R27 E = 7.24 → projection: 10% very high and 60% high
HI professional 1 G1	COM = 6, ETU = 7, SPU = 6, PRF = 7, NGT = 8, DTC = 5, CHT = 6, OV = 7, DET = 8, DKQ = 8, CD = 5, and VP = 6 Activated rules: R14, R15, R17, R18, R23, R24, R26, and R27 E = 6.93 → projection: 79% high
HI professional 2 G1	COM = 7, ETU = 9, SPU = 7, PRF = 6, NGT = 8, DTC = 6, CHT = 7, OV = 8, DET = 7, DKQ = 7, CD = 4, and VP = 6 Activated rules: R26 and R27 E = 7.1 → projection: 5% very high and 70% high
Assistant 1 G1	COM = 4, ETU = 8, SPU = 6, PRF = 6, NGT = 8, DTC = 6, CHT = 7, OV = 6, DET = 6, DKQ = 8, CD = 7, and VP = 7 Activated rules: R15, R18, R24, and R27 E = 5.9 → projection: 5% medium and 65% high
Assistant 2 G1	COM = 7, ETU = 8, SPU = 6, PRF = 6, NGT = 7, DTC = 5, CHT = 6, OV = 5, DET = 8, DKQ = 8, CD = 7, and VP = 8 Activated rules: R15, R18, R24, and R27 E = 5.5 → projection: 25% medium and 50% high
LifeLines (second group of participants)
Physician 1 G2	COM = 8, ETU = 8, SPU = 7, PRF = 6, NGT = 7, DTC = 6, CHT = 6, OV = 7, DET = 5, DKQ = 8, CD = 4, and VP = 6 Activated rules: R23, R24, R26, and R27 E = 7,1 → projection: 3% very high and 68% high
Physician 2 G2	COM = 9, ETU = 7, SPU = 7, PRF = 6, NGT = 6, DTC = 6, CHT = 7, OV = 6, DET = 4, DKQ = 8, CD = 5, and VP = 4 Activated rules: R23, R24, R26, and R27 E = 5.59 → projection: 18% medium and 52% high
HI professional 1 G2	COM = 7, ETU = 7, SPU = 6, PRF = 5, NGT = 7, DTC = 5, CHT = 6, OV = 7, DET = 3, DKQ = 7, CD = 5, and VP = 6 Activated rules: R14, R15, R17, R18, R23, R24, R26, and R27 E = 5.7 → projection: 16% medium and 60% high
HI professional 2 G2	COM = 8, ETU = 8, SPU = 7, PRF = 6, NGT = 7, DTC = 6, CHT = 5, OV = 7, DET = 6, DKQ = 8, CD = 4, and VP = 5 Activated rules: R23, R24, R26, and R27 E = 5.01 → projection: 45% medium and 25% high
Assistant 1 G2	COM = 7, ETU = 8, SPU = 6, PRF = 5, NGT = 8, DTC = 6, CHT = 7, OV = 6, DET = 6, DKQ = 7, CD = 6, and VP = 7 Activated rules: R14, R15, R17, R18, R23, R24, R26, and R27 E = 7.1 → projection: 3% high and 72% very high
Assistant 2 G2	COM = 7, ETU = 8, SPU = 6, PRF = 6, NGT = 7, DTC = 5, CHT = 6, OV = 7, DET = 5, DKQ = 8, CD = 6, and VP = 6 Activated rules: R14, R15, R17, R18, R23, R24, R26, and R27 E = 5.25 → projection: 35% medium and 34% high
Interactive bubble chart (third group of participants)
Physician 1 G3	COM = 7.5, ETU = 5.5, SPU = 6, PRF = 7, NGT = 5, DTC = 7.25, CHT = 4.75, OV = 3, DET = 5.25, DKQ = 3, CD = 3.75, and VP = 4 Activated rules: R15 and R23 E = 5.48 → projection: 10% medium and 70% high
Physician 2 G3	COM = 7, ETU = 5.5, SPU = 6, PRF = 6.25, NGT = 6.6, DTC = 7.25, CHT = 5.52, OV = 3, DET = 4.75, DKQ = 3.25, CD = 5.5, and VP = 6.5 Activated rules: R14, R20, R22, R23, and R25 E = 5.24 → projection: 35% medium and 35% high
HI professional 1 G3	COM = 6.75, ETU = 4.5, SPU = 6.75, PRF = 6.5, NGT = 6.25, DTC = 5.75, CHT = 5.5, OV = 1.5, DET = 5.5, DKQ = 1.5, CD = 5, and VP = 6 Activated rules: R14, R15, R16, R17, R21, R22, R24, and R27 E = 4.93 → projection: 55% medium and 15% high
HI professional 2 G3	COM = 5.5, ETU = 4.5, SPU = 7, PRF = 4.5, NGT = 7.25, DTC = 7, CHT = 6.75, OV = 4.25, DET = 4.30, DKQ = 1.5, CD = 3.75, and VP = 5.5 Activated rules: R21 and R22 E = 6.1 → projection: 5% medium and 70% high
Assistant 1 G3	COM = 7, ETU = 5.5, SPU = 6.75, PRF = 5.52, NGT = 6.75, DTC = 9, CHT = 8.75, OV = 4.75, DET = 5, DKQ = 4.5, CD = 7, and VP = 8.5 Activated rules: R14, R16, R25, and R27 E = 6.9 → projection: 5% very high and 70% high
Assistant 2 G3	COM = 5.5, ETU = 8.25, SPU = 7, PRF = 5.5, NGT = 8.75, DTC = 9.25, CHT = 8.5, OV = 5.25, DET = 5.5, DKQ = 5, CD = 5.75, and VP = 9 Activated rules: R09, R08, R16, and R27 E = 5.5 → projection: 30% medium and 45% high

COM: comprehensibility; ETU: ease to use; SPU: speed of use; PRF: performance; NGT: navigating in time; DTC: data comparison; CHT: changes over time; OV: overview; DET: details; DKQ: data and knowledge quality; CD: causal dependencies; VP: visual prediction.

LifeLines evaluation

Figure 6 shows another example presenting the answers of a HI professional (in the second group of participants) about the LifeLines tool. According to the users’ answers (cf. Figure 6(a)), the system activates eight rules (R14, R15, R17, R18, R23, R24, R26, and R27). The associated evaluation value obtained is equal to 5.7 (Figure 6(b)). By projecting this value on the curve, we find that evaluation is 16% medium and 60% high (cf. Figure 6(b)).

Figure 6.

(a) Questionnaire user interface of the LifeLines tool and (b) evaluation result of the LifeLines for the first HI professional.

The LifeLines evaluation results of the other participants are visible in Table 4 of the discussion section.

Interactive bubble chart visualization

Figure 7 presents an example of the user interface presenting the answers of an assistant on the bubble chart evaluation. According to the participant answers (cf. Figure 7(a)), the system activates four rules (R14, R16, R25, and R27). Applying the Mamdani method, the evaluation value obtained is 6.9 (cf. Figure 7(b)). By projecting this value on the curve, we find that the evaluation is 70% high and 5% very high (cf. Figure 7(b)).

Figure 7.

(a) Questionnaire user interface of the bubble chart tool and (b) evaluation result of the bubble chart for the first assistant.

For the other participants, bubble chart evaluation results are presented in Table 4 in the discussion section.

Results and discussion

Table 4 shows the evaluation results of the three visualization tools that integrated in the visual iDSS for the fight against nosocomial infections in the ICU. These results are given by three groups of participants. In fact, each group of participants is composed of six members and is interested in evaluating visualizations and the results are introduced as inputs to our fuzzy system.

Finally, the output evaluation score is shown. Activated rules are also displayed. As stated previously, these rules are generated and based on new evaluations submitted by the participants. The learnability characteristic of our system allows it to add a new rule with each new evaluation allowing better learning to our inference unit and more accurate results.

As presented in Table 4, the three groups of physicians, HI professional, and assistants evaluated each criterion of the three categories (usability, data manipulation, and patterns’ discovery) by assigning a numeric value. For example, for evaluating LifeLines visualization technique, the first physician of the second group gave the value 8 to COM, 8 to ETU, 7 to SPU, 6 to PRF, 7 to NGT, 6 to DTC, 6 to CHT, 7 to OV, 5 to DET, 8 to DKQ, 4 to CD, and 6 to VP. Then, a linguistic value is specified after projecting each numeric value of particular criterion with the corresponding membership function. Each criterion described by a linguistic value is inputted in the Mamdani inference controller that uses active rules (in which the antecedent part includes evaluated criterion) to generate the output (E) representing the overall evaluation score of visualization technique and taking into account each single evaluation of the criteria set and their weightiness in the overall evaluation procedure. The projection of output variable (E) with output membership function (eval) provides a fuzzy evaluation of visualization that well handles the uncertainty of this procedure (3% very high and 68% high) concerning the evaluation of Physician 1 G2.

Experiment results’ discussion

The evaluation results (visible in Table 4) of the visualization tools about the three user profiles are close and the activated rules are sometimes repeated. This allowed modeling the vagueness in the initial judgments obtained from users’ evaluation. Moreover, these different evaluation values show that users with their different levels of expertise have generally appreciated the use of these visual representations of the iDSS for the fight against nosocomial infections. The use of the fuzzy logic allowed transforming the input evaluation scores into linguistic variables and linguistic evaluation of overall iDSS visualization tools. The projection of the calculated results proves that the evaluation is probabilistic and cannot be simple numeric value, but that the uncertainty is an inherent part of the evaluation and hence is a crucial component in decision-making.

Evaluation procedure discussion

Our method is quite general and could be applied to different kinds of visualization tools of iDSS. We briefly discuss such application in the following points:

The first set of questions aims at collecting the participants’ background information. This step allows judging the entire evaluation results.

The training session is required for all new visualization tools. It is ended by (1) asking participants to find answers to data queries and visually analyze hidden relationships in the dataset and (2) giving information about the questions’ difficulty and the time of their completion.

The second set of questions stores the overall user opinion about the visualization tool usability, data manipulation, and patterns’ discovery capacity. Such questions are necessary for visual analytics tools’ evaluation.

And finally, the second step that enables applying the fuzzy logic to analyze this overall user opinion and eliminate uncertainty by interpreting input variables and generating a final evaluation result.

To show that the proposed evaluation approach contributes to the literature, we classify it relatively into a set of selected existing methods (Table 5). This classification is based on a set of common features’ characterization of iDSS.⁷¹

Table 5.

Classification of our contribution versus previous works.

	Static/dynamic	New data generation	Intelligent system	Rule-based reasoning	Natural language generation	Multiple category classification	Data selection
Chrimes et al.⁶⁰	Dynamic	No	No	No	No	Yes	Yes
Akçay et al.⁷²	Dynamic	No	Yes	Yes	No	Yes	Yes
Mane et al.⁷³	Dynamic	Yes	No	No	Yes	Yes	Yes
Kuru⁷⁴	Dynamic	Yes	Yes	No	No	Yes	Yes
Farri et al.⁷⁵	Static	No	No	No	No	Yes	Yes
Kannampallil et al.⁷⁶	Dynamic	Yes	Yes	No	Yes	Yes	Yes
Our proposed method	Dynamic	Yes	Yes	Yes	Yes (using linguistic variables)	Yes	Yes

Based on Table 5, our method is interesting considering the features’ characterization of iDSS evaluation.⁶⁴ Fuzzy inference process is more dynamic, learnable, and adaptable with evaluation context. In fact, the evaluator chooses the linguistic value of a particular criterion according to the intersection of fuzzy number for the criterion with the membership functions of such criterion. Then, he or she defines a new fuzzy rule depending on the linguistic value and his or her expert knowledge and experience. With each new evaluation, a new fuzzy rule is added allowing learning to our inference module and more accurate results. Furthermore, the proposed evaluation method considers technical specificities of generated visualization. Finally, participants assert that the results achieved following the application of our method are closest to their opinions. However, a limitation presented by the participants of third profile is that the proposed method is specific for the evaluation visualizations generated by iDSS and applied in the medical domain. Its generalizability should concern any visual analytics context, which could provide more evidence that the proposed method will be useful in other visualizations and other scenarios. For this reason, we have planned further work to address such generalization.

Conclusion

Several iDSSs were developed to support the important decision-making processes. Therefore, evaluations are significant to ensure that these systems will live up to the users’ expectations. The research reported in this article provided a new evaluation method of the visualizations of iDSS. It consists of two steps: (1) collecting users’ background information and their overall opinion (in terms of values) about the tool to evaluate and (2) applying the fuzzy logic to analyze these values.

In our study, three criteria are considered for one visualization tool that involves usability, data manipulation, and patterns’ discovery dimensions. Then by applying fuzzy logic, we get a new procedure for evaluating visualization tools. The results of applying this method for the perspective wall and LifeLines tools’ evaluation show that our approach can be a practical method. The proposed evaluation method provides a support for the inexperienced users to fulfill the assessment of iDSS visualization tools. It allowed modeling the vagueness in the initial judgments obtained from this assessment. Our study shows that this evaluation is a probabilistic model that supports uncertainty, a crucial component in decision-making.

As a continuation of our work, we have started the application of the fuzzy logic on the questionnaire variables to automatically generate the usability, the data manipulation, and the patterns’ discovery values where each criterion will be considered as an input fuzzy set. In effect, we assume that participants can be uncertain about their attitudes. We plan to implement the neural approach to natural language processing in the evaluation procedure for learning and adaptation. The method must use natural language expression to interactively communicate with its human environment; it must follow the conversation’s context and understand the meaning of each sentence of user evaluation and then execute it. The generalizability of the method will also be addressed by applying it to other visual analytics contexts.

Footnotes

Appendix 1 Acknowledgements

The authors thank all the ICU staff of Habib Bourguiba Teaching Hospital for their interest in the project and all the time they spent helping us design, use, and evaluate our system. The authors also thank the anonymous reviewers for their relevant remarks.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: The authors would like to acknowledge the financial support for this research by grants from the ARUB program under the jurisdiction of the General Direction of Scientific Research (DGRST), Tunisia.

References

North

Toward measuring visualization insight. IEEE Comput Graph 2005; 11(4): 443–456.

Plaisant

. The challenge of information visualization evaluation. In: AVI ’04 proceedings of the working conference on advanced visual interfaces, Gallipoli, 25–28 May 2004, pp.109–116. New York: ACM.

Shneiderman

. The eyes have it: a task by data type taxonomy for information visualizations. In: VL ‘96 proceedings of the 1996 IEEE symposium on visual languages, Boulder, Colorado, USA, 3–6 September 1996, pp.336–343. Washington, DC: IEEE.

Cleveland

. The elements of graphing data. Summit, NJ: Hobart Press, 1994.

Mazza

Berre

. Focus group methodology for evaluating information visualization techniques and tools. In: IV ’07 11th international conference on information visualization, Zurich, 4–6 July 2007, pp.74–80. New York: IEEE.

Bahloul

Ben Ayed

Alimi

A-M.

Vers une method d’évaluation de Système Interactif d’Aide à la Décision basé sur le processus d’ECD. In: Atelier EvalECD’2010, Hammamet, Tunisia, 26–29 January 2010, pp.1–12. Toulouse, France: Cépaduès-éditions.

Brahmi

Ltifi

Ben Ayed

. Approach for the evaluation of a KDD based DSS visual representations. In: 2014 middle east conference on biomedical engineering (MECBME), Doha, Qatar, 17–20 February 2014.

Zadeh

LA.

Fuzzy sets. Inform Control 1965; 8: 338–353.

William

Buckley

JJ.

Fuzzy expert systems and fuzzy reasoning. 1st ed. Hoboken, NJ: Wiley-Interscience, 2005.

10.

Pham

Streit

Brown

Visualisation of information uncertainty: progress and challenges. In: Liere

Adriaansen

Zudilova-Seinstra

(eds) Trends in interactive visualization. London: Springer, 2009, pp.19–48.

11.

Chen

KC.

Decision support system for tourism development: system dynamic approach. J Comput Inform Syst 2004; 45(1): 104–112.

12.

Nielsen

Usability engineering. San Francisco, CA: Morgan Kaufmann, 1993.

13.

Keim

Andrienko

Fekete

J-D

et al . Visual analytics: definition, process, and challenges. In: Kerren

Stasko

J-T

Fekete

J-D

et al . (eds) Information visualization (Vol. 4950, Lecture Notes in Computer Science). Berlin; Heidelberg: Springer-Verlag, 2008, pp.154–175.

14.

Heer

Perer

A system for modeling, transformation and visualization of multidimensional heterogeneous networks. Infor Vis 2014; 13(2): 111–133.

15.

Kim

Unat

Baden

et al . A new approach to interactive viewpoint selection for volume data sets. Infor Vis 2013; 12(3–4): 240–256.

16.

Jorge

Poças

Azevedo

A post-processing environment for browsing large sets of association rules. In: Proceedings of workshop on integration of decision support and data mining (IDDM 02), associated with ECML/PKDD 2002 (ed. D

Mladenic

Kavsek

Lavrac

et al .), Helsinki, 19–23 August 2002, pp.53–64. Berlin: Springer-Verlag Press.

17.

Aigner

Miksch

Schumann

et al . Visualization of time-oriented data. Berlin: Springer, 2011.

18.

Kriglstein

Pohl

Smuc

. Pep up your time machine: recommendations for the design of information visualizations of time-dependent data. In: Huang

(ed.) Handbook of human centric visualization. Berlin: Springer, 2014, pp.203–225.

19.

Mackinlay

Hanrahan

Stolte

Show me: automatic presentation for visual analysis. IEEE T Vis Comput Gr 2007; 13(6): 1137–1144.

20.

Shneiderman

Interactive information visualization to explore and query electronic health records. Foundations Trends Hum: Comput Interact 2013; 5(3): 207–298.

21.

Fayyad

Djorgovski

Weir

. Automating the analysis and cataloging of sky surveys. In: Fayyad

Piatetsky-Shapiro

Smyth

et al . (eds) Advances in knowledge discovery and data mining. Menlo Park, CA: The MIT Press, 1996, pp.471–494.

22.

Shyur

H-J

Jou

Chang

A data mining approach to discovering reliable sequential patterns. J Syst Software 2013; 86(8): 2196–2203.

23.

Nemati

Barko

CD.

Enhancing enterprise decisions through organizational data mining. J Comput Inform Syst 2002; 42(4): 21–28.

24.

Heinrichs

Lim

JS.

Integrating web-based data mining tools with business models for knowledge management. Decis Support Syst 2003; 35(1): 103–112.

25.

Bose

Sugumaran

Application of intelligent agent technology for managerial data analysis and mining. Data Base Adv Inf Sy 1999; 30(1): 77–94.

26.

Polese

Troiano

Tortora

A data mining based system supporting tactical decisions. In: Proceedings of the 14th international conference on software engineering and knowledge engineering, Ischia, 15–19 July 2002, pp.681–684. New York: ACM.

27.

Lee

Park

Agent and data mining based decision support system and its adaptation to a new customer centric electronic commerce. Expert Syst Appl 2003; 25(4): 619–635.

28.

Ltifi

Kolski

Ben Ayed

et al . Human-centered design approach for developing dynamic decision support system based on knowledge discovery in databases. J Decis Syst 2013; 22(2): 69–96.

29.

Hofmann

Siebes

Wilhelm

. Visualizing association rules with interactive mosaic plots. In: KDD ‘00 proceedings of the sixth ACM SIGKDD international conference on knowledge discovery and data mining, Boston, MA, 20–23 August 2000, pp.227–235. New York: ACM.

30.

Han

Cercone

. RuleViz: a model for visualizing knowledge discovery process. In: KDD ‘00 proceedings of the sixth ACM SIGKDD international conference on knowledge discovery and data mining, Boston, MA, 20–23 August 2000, pp.244–253. New York: ACM.

31.

Bruzzese

Davino

. Visual post-analysis of association rules. In: ECML/PKDD VDM Workshop (LNAI Series), Helsinki, 19–23 August 2002, pp.55–66. Berlin: Springer.

32.

Zhao

Liu

. Visual analysis of the behavior of discovered rules. In: Proceedings of workshop on visual data mining ACM SIGKDD international conference on knowledge discovery & data mining (KDD 2001), San Francisco, CA, 26–29 August 2001, pp.59–64. New York: ACM.

33.

Zhao

Liu

Tirpak

et al . V-Miner: using enhanced parallel coordinates to mine product design and test data. In: Proceedings of the tenth ACM SIGKDD international conference on knowledge discovery and data mining, Seattle, WA, 22–25 August 2004, pp.494–502. New York: ACM.

34.

Blanchard

Guillet

Briand

. Exploratory visualization for association rule rummaging. In: KDD-03 workshop on multimedia data mining, Washington, DC, 24–27 August 2003, pp.107–114. New York: ACM.

35.

Ong

et al . CrystalClear: active visualization of association rules. In: ICDM-02 workshop on active mining (AM-02), Maebashi TERRSA, Maebashi City, Japan, 9 December 2002, pp.120–125. New York: IEEE.

36.

Hellerstein

Ordering categorical data to improve visualization. In: INFOVIS-99, San Francisco airport, Hyatt, San Francisco, CA, 24–29 October 1999, pp.15–18. New York: IEEE.

37.

Zhao

Liu

Tirpak

et al . Opportunity map: a visualization framework for fast identification of actionable knowledge. In: Proceeding of CIKM ‘05 proceedings of the 14th ACM international conference on information and knowledge management, Bremen, 31 October–5 November 2005, pp.60–67. New York: ACM.

38.

Ltifi

Ben Mohamed

Ben Ayed

Interactive visual KDD based temporal decision support system. Infor Vis 2016; 15(1): 31–50.

39.

Ankerst

Anne

Tjoelker

et al . DataJewel: integrating visualization with temporal data mining. In: Simoff

Böhlen

Mazeika

(eds) Visual data mining: theory, techniques and tools for visual analytics. Berlin; Heidelberg: Springer, 2008, pp.312–330.

40.

Munaga

Ieronutti

Chittaro

. CAST: a novel trajectory clustering and visualization tool for spatio-temporal data. In: Proceedings of the first international conference on intelligent human computer interaction (Organized by the Indian Institute of Information Technology), Allahabad, India, 20–23 January 2009, pp.169–175. Berlin: Springer.

41.

Ramírez-Ramírez

Gel

Thompson

et al . A new surveillance and spatio-temporal visualization tool SIMID: SIMulation of Infectious Diseases using random networks and GIS. Comput Methods Programs Biomed 2013; 110(3): 455–470.

42.

Mittelstädt

Hao

Dayal

et al . Advanced visual analytics interfaces for adverse drug event detection. In: Proceedings of the 2014 international working conference on advanced visual interfaces (AVI 2014), Como, 27–29 May 2014, pp.237–244. New York: ACM.

43.

Fischer

Stoffel

Mittelstädt

et al . Using visual analytics to support decision making to solve the Kronos incident. In: 2014 IEEE conference on visual analytics science and technology (VAST), Paris, 25–31 October 2014, pp.301–302. New York: IEEE.

44.

Mittelstädt

Wang

Eaglin

et al . An integrated in-situ approach to impacts from natural disasters on critical infrastructures. In: IEEE 48th annual Hawaii international conference on system sciences, HICSS-48, Kauai, HI, 5–8 January 2015, pp.1118–1127. Los Alamitos, CA: IEEE.

45.

Ware

Information visualization: perception for design. San Francisco, CA: Morgan Kaufmann, 2004.

46.

Hartson

Hix

Developing user interfaces: ensuring usability through product and process. Hoboken, NJ: John Wiley & Sons, 1993.

47.

Isenberg

Chen

et al . A systematic review on the practice of evaluating visualization. IEEE T Vis Comput Gr 2013; 19(12): 2818–2827.

48.

Kobsa

. An empirical comparison of three commercial information visualization systems. In: INFOVIS ‘01 proceedings of the IEEE symposium on information visualization, San Diego, CA, 22–23 October 2001, pp.123–130. New York: IEEE.

49.

Juarez

CAEVA: cognitive architecture to evaluate visualization applications. In: Proceedings seventh international conference on information visualization, 2003, London, 18 July 2003, pp.589–595. New York: IEEE.

50.

Gonzales

Kobsa

. A workplace study of the adoption of information visualization systems. In: Proceedings of I-KNOW’03: 3rd international conference on knowledge management, Graz, 2–4 July 2003, pp.92–102. Verlag: J.UCS.

51.

Rieman

A field study of exploratory learning strategies. ACM T Comput-Hum Int 1996; 3: 189–218.

52.

Shneiderman

Plaisant

. Strategies for evaluating information visualization tools: multi-dimensional in-depth long-term case studies. In: BELIV ‘06 proceedings of the 2006 AVI workshop on beyond time and errors: novel evaluation methods for information visualization, Venice, 23–26 May 2006, pp.81–87. New York: ACM.

53.

Saraiya

North

Duca

An insight-based methodology for evaluating bioinformatics visualizations. IEEE T Vis Comput Gr 2005; 11(4): 443–456.

54.

Kosara

Visualization criticism—the missing link between information visualization and art. In: Proceedings of the 11th international conference on information visualisation (IV), Zurich, Switzerland, 4–6 July 2007, pp.631–636. Los Alamitos, CA: IEEE.

55.

Kurzhals

Bopp

Bässler

et al . Benchmark data for evaluating visualization and analysis techniques for eye tracking for video stimuli. In: Proceedings of the fifth workshop on beyond time and errors: novel evaluation methods for visualization, Paris, 10 November 2014, pp.54–60. New York: ACM.

56.

Ragan

Goodall

. Evaluation methodology for comparing memory and communication of analytic processes in visual analytics. In: Proceedings of the fifth workshop on beyond time and errors: novel evaluation methods for visualization, Paris, 10 November 2014, pp.27–34. New York: ACM.

57.

Yeh

Lai

et al . Evaluating 2D flow visualization using eye tracking. Comput Graph Forum 2015; 34(3): 501–510.

58.

Brooke

SUS: a quick and dirty usability scale in usability evaluation in industry. London: Taylor & Francis, 1996, pp.189–194.

59.

Lam

Bertini

Isenberg

et al . Empirical studies in information visualization: seven scenarios. IEEE T Vis Comput Gr 2012; 18(9): 1520–1536.

60.

Chrimes

Kushniruk

Kitos

et al . Usability testing of Avoiding Diabetes Thru Action Plan Targeting (ADAPT) decision support for integrating care-based counseling of pre-diabetes in an electronic health record. Int J Med Inform 2014; 83(9): 636–647.

61.

Flavián

Guinalíu

Gurrea

The role played by perceived usability, satisfaction and consumer trust on website loyalty. Inform Manage 2006; 43(1): 1–14.

62.

Lewis

Sauro

. The factor structure of the System Usability Scale. In: Kurosu

(ed.) Human centered design. Berlin; Heidelberg: Springer, 2009, pp.94–103.

63.

Keim

Information visualization and visual data mining. IEEE T Vis Comput Gr 2002; 8(1): 1–8.

64.

Chou

J-R.

Applying fuzzy linguistic preferences to Kansei evaluation. In: International conference on kansei engineering and emotion research, KEER2014, Linköping, 11–13 June 2014, pp.339–349. Linköping: Linköping University Electronic Press.

65.

Mamdani

Assilian

An experiment in linguistic synthesis with a fuzzy logic controller. Int J Man Mach Stud 1975; 7: 1–13.

66.

Zadeh

LA.

Outline of a new approach to the analysis of complex systems and decision processes. IEEE T Syst Man Cyb 1973; 3: 28–44.

67.

Ltifi

Trabelsi

Ben Ayed

et al . Dynamic decision support system based on Bayesian networks, application to fight against the nosocomial infections. Int J Adv Res Artif Intell 2012; 1(1): 22–29.

68.

Mackinlay

Robertson

Card

. The perspective wall: detail and context smoothly integrated. In: Proceedings of the SIGCHI conference on human factors in computing systems, New Orleans, LA, 27 April–2 May 1991, pp.173–176. New York: ACM.

69.

Plaisant

Milash

Rose

et al . LifeLines: visualizing personal histories. In: Proceedings of the SIGCHI conference on human factors in computing systems, Vancouver, BC, Canada, April 1996, pp.221–227. New York: ACM.

70.

Elouni

Ltifi

Ben Ayed

. Knowledge visualization model for intelligent dynamic decision-making. In: The 15th international conference on Hybrid Intelligent Systems (HIS 2015), Seoul, South Korea, 16–18 November 2015, pp.223–235. Berlin: Springer.

71.

Sànchez-Marrè

Gibert

Cabello

et al . A methodology for the characterization of intelligent environmental decision support systems. In: International environmental modelling and software society (iEMSs), sixth biennial meeting, Leipzig, 21 September 2012.

72.

Akçay

Ertek

Büyüközkan

Analyzing the solutions of DEA through information visualization and data mining techniques: smartDEA framework. Expert Syst Appl 2011; 39: 7763–7775.

73.

Mane

Bizon

Schmitt

et al . VisualDecisionLinc: a visual analytics approach for comparative effectiveness-based clinical decision support in psychiatry. J Biomed Inform 2012; 45: 101–106.

74.

Kuru

Biomedical visual data analysis to build an intelligent diagnostic decision support system in medical genetics. Artif Intell Med 2014; 62(2): 105–118.

75.

Farri

Pieckiewicz

Rahman

et al . A qualitative analysis of EHR clinical document synthesis by clinicians. AMIA Annu Symp Proc 2012; 2012: 1211–1220.

76.

Kannampallil

Franklin

Mishra

et al . Understanding the nature of information seeking behavior in critical care: implications for the design of health information technology. Artif Intell Med 2013; 57(1): 21–29.