Abstract
Background:
With the aim of evaluating the perfusion simulator at the German Heart Center Berlin, similarity between simulation and clinical operation room (OR) was investigated regarding subjective perception and eye movement.
Methods:
Eight perfusionists performed an operation on the heart-lung machine (HLM) wearing eye tracking glasses, each in real OR and simulator. The three most important phases for perfusionists (going on bypass, cardioplegia administration and coming off bypass) were considered. Additional to eye tracking data as objective measure, questionnaires were completed, and interviews conducted afterwards.
Results:
The structure of simulator and real OR is perceived as basically the same. Yet there are differences in the HLM-models used and the temporal sequence. Different perception of both situations is reported in interviews and reflected in significant differences in the rating scales (NASA-TLX) on three of six subscales. In eye tracking data, certain AOIs could be identified for the individual phases, both in OR and simulator—an indication of fundamental similarity. However, differences regarding the proportions of the individual AOIs, especially in the first and third phase, are leading to the assumption that the simulator, and especially the simulation process, is only valid to a limited extent regarding subjective perception and eye tracking data.
Conclusion:
The use of the simulator for (advanced) training is accepted and explicitly requested by perfusionists. Yet further research is needed to identify the decisive factors (like simulation duration or additional tasks) for a valid execution in the simulator. Furthermore, a larger sample size should be regarded to allow statistical analysis.
Keywords
Introduction
Future perfusionists mostly only gain practical experience in operating a heart-lung machine (HLM) in the real operating room (OR), accompanied by an experienced perfusionist and directly on the living patient. Since 2009, however, the German Heart Center Berlin (DHZB) has a simulator for heart operations in which the use of a heart-lung machine (HLM) can be simulated. The use of simulator training in medical education has several advantages. It strengthens the link between theory and practice before the start of work in a hospital and promotes learning without compromising patient safety. 1 In addition, in a simulator also situations that usually only occur very rarely can be trained. 2 These were also the biggest advantages that students of the DHZB stated in a first evaluation study in which they received specific training at the HLM. Moreover, their handling of a critical situation at the HLM improved significantly through training. 3 Although simulation training was proven to be of high value, it is unclear whether these patterns also show up in the real OR and thus assign a high validity to it.
Though there are several perfusion simulators in various countries,4,5 there are no standards for the simulation or its evaluation. 6 The validity of simulators has often been investigated in driving simulators. Thereby especially the relative validity is investigated, the extent to which the simulator produces the same behavior as shown in reality.7,8 In their study on the validation of a driving simulator, Li et al. 9 also used physiological measures, more precisely heart rate and EEG (electrical activity of the brain) which were collected both in the field and in the reconstructed virtual environment.
In the field of perfusion, eye tracking is an interesting and promising measure, since the targeted perception of visual information, which is presented in a specifically localized way (e.g. when checking values on the monitors), is very important for the work of perfusionists. It is also an indicator of cognitive processing and mental workload.10,11 Furthermore, psychological fidelity, the degree to which users perceive the simulation as authentic or real, is an important prerequisite for simulator validity. 12
The aim of this study is to investigate whether the simulator at the DHZB (with its structure, but also the procedure as it is currently carried out) is comparable to a real OR in terms of subjective perception and eye movement. The focus hereby is on the perfusionist and his/her main task during heart surgery—the operation of the HLM.
Methods
Eye movement data as well as data from questionnaires and interviews on perfusionists’ subjective perception were collected during a real heart operation at the DHZB (in a previous research phase 13 ) as well as during a simulated operation and then compared. A complete operation was performed from the perfusionist’s point of view, in the simulator in condensed time frame. The phases in which the perfusionist is required were of interest for this investigation—accordingly, going on bypass, cardioplegia administration and coming off bypass are considered. The study was carried out as within-design, so each perfusionist participated in both, real OR and simulator. Eye tracking data was only evaluated for the three previously defined phases.
Participants
The participants of the study were a total of 10 perfusionists who are or were active at the German Heart Center Berlin (DHZB). All of them participated voluntarily in the study and received no remuneration or other benefits for their participation. The participants were informed about the aim and design of the study and were free to drop out of the study at any time without consequences. The study was supported by the chief medical officer of the hospital and the supervising cardiothoracic surgeon. Since no patient data were evaluated and the impact on treatment was considered neglectable, a formal ethics committee decision was waived.
Of the 10 recruited participants, all took part in the simulator and eight of them also in the real OR (Table 1). The average age of the participants (N = 10) was 41 years (M = 40.6, SD = 11.32) with an average work experience of 11 years (M = 11.43, SD = 11.40), ranging from 0 to 31 years.
Sample description.
Measures
The eye movement was recorded with the help of Tobii Pro Glasses 2, a mobile eye tracking device for recording eye movements and fixations, using a frequency of 50 Hz. 14 These glasses consist of the head unit with video-camera, microphone and a total of six infrared sensors to measure pupil sizes and three-dimensional orientation of the pupils. Gaze direction can then be mapped onto a previously taken photograph, imported into the Tobii Pro Lab 1.83 software, where so called “areas of interest” (AOI) are defined. Further evaluation of the recorded data is possible, including number of AOI hits, AOI fixation duration and pupil diameters. The method is described in more detail in our previous study. 13 A list of AOI is available at the Supplemental Appendix.
The subjective perception was assessed using the NASA-TLX, a self-description questionnaire for recording the perceived stress. 15 The questionnaire contains six items covering the six subscales of mental demand, physical demand, temporal demand, performance, effort, and frustration and is evaluated on a continuous scale from 0 (low) to 10 (high).
Besides, interviews on the perfusionists perception of the simulator were carried out by a researcher with background in psychology and user experience studies (Interview questions in Supplemental Appendix). Interviews were audio recorded and subsequently transcribed.
Procedure
The study consisted of two parts—data collection in the real OR and the ensuing data collection in the simulator. The data collection period lasted 7 months, the average time between participation in real OR and simulator was 97 days. In the OR as well as in the simulator, perfusionists wore the Tobii Glasses for eye-tracking throughout the operation/simulation and afterwards filled in the questionnaires. At the end, a short interview was carried out as mentioned above.
In the simulator, a short introduction to the Sorin C5 (LivaNova, Munich, Germany) model of the heart-lung machine (HLM) and a simulator test run with all critical phases were performed for each participant, since the HLM-model used in the simulator differed from the one in the OR (Sorin S5). Both HLM have the same functionality and technology, but the C5 model used for simulation has a slightly different orientation of the displays and the values within the displays are also positioned differently. Additionally, not all values and elements of the clinically used HLM are available at the console used for simulation (Figure 1).
Arrangement of main components of the HLM in the OR (left) and in the simulator (right).
The simulation experiment was carried out at a dedicated simulation operation room (Sim OR) at the Academy for Perfusion, the Perfusion Education Program located at DHZB. The Sim OR is equipped with an operation table, anesthesia machine, patient monitoring, surgical instruments, and a video and audio surveillance system. The Orpheus CPB simulator is used to simulate patients’ circulatory responses, and a patient mannequin with cannulated heart and intubated lungs is simulating the patient on bypass. A more detailed description of this setup is available in a recent publication from our group. 16
The team performing the operation in the simulation room consisted of the perfusionist participant and two researchers (perfusionist instructor, human factors specialist) performing the roles of cardiac surgeon and anesthetist.
The simulation scenario was constructed to include the main phases going on bypass, aortic cross clamping and cardioplegia administration, aortic declamping and reperfusion as well as coming off bypass, and lasted on average 9 min. For the purpose of the simulation experiment, only those short phases were simulated in order to compare them to the clinical environment. A full cardiac operation was not scheduled in the simulator. No critical incidents were included.
Results
Preliminary examination
Objectively, the essential components of the simulator corresponded to those of a real OR. Both facilities had all components of clinical operating rooms available, with three main areas of work for surgery, anesthesia, and perfusion.
Eye tracking
For eye tracking analysis, a total of n = 4 complete data records were considered for each phase. Eye movements were recorded for eight participants, both in the simulator and in the real operating room. Of these eight, one file from the OR was damaged and could not be used for evaluation at all. Data of two other participants from the OR were only partially available and could only be considered for evaluation of single phases. Two participants were still in training and had their clinical instructors in the OR. Since those participants received help and the instructors took over handling in situations important for the evaluation (such as cardioplegia application), both data sets were not considered for the eye tracking evaluation but were only included in the general simulator evaluation by questionnaire and interview.
For evaluation of eye-tracking data, manual mapping was undergone since automatic mapping did not provide reliable results due to the detailed environment and the many active movements of the perfusionists.
Phase 1—going on bypass
Among all participants the arterial pump, the reservoir with adjacent elements (such as tubes and pump head) and the patient monitor were identified as important AOIs in this first phase, although proportionally different for the individual perfusionists. In three of the four data sets, differences in terms of the proportions of fixations on the AOIs were also identified between both conditions, especially the patient monitor was looked at much more in the simulator (see Figure 2, visualization of how long certain areas were viewed).
Heatmap of one participant in phase 1—real OR on the left, simulator on the right.
Phase 2—cardioplegia administration
The cardioplegia pump and cardioplegia control were identified as particularly important elements for this phase and were found in all participants. There were no clear differences in the proportion of fixations between the two conditions in any participant.
Phase 3—coming off bypass
In this third phase, the same areas were relevant as in the first one (patient monitor, arterial pump, and reservoir) and were regarded by all participants in the OR as well as in the simulator. It is also noticeable that apart from these three areas, only very few other AOIs were looked at. There were distinct differences among two perfusionists: the three AOIs were important each, but the mainly fixed AOI was different between both conditions.
General overview
The fixed AOIs were quite similar in both conditions for the individual perfusionists, often the same main AOIs were looked at. Yet there were also clear differences between real OR and simulator regarding the frequency of fixations and proportion of AOIs. Especially in the first, but also in the last phase, the differences in this respect were clear. The evaluation is not statistically feasible due to the small sample size and data situation.
Subjective perception
NASA-TLX
The perceived demands on perfusionists differed significantly (p < 0.10) between both conditions on three of the six subscales, namely mental demand, temporal demand, and performance (Table 2). However, for two of those three subscales (mental demands and performance) there were also significant positive correlations, for the subscale temporal demand a significant negative correlation between both conditions.
NASA-TLX.
p ≤ .05; **p ≤ .01.
Interview data
The interview data also showed that the perfusionists’ perception in the simulator was different from the real OR. Almost all participants felt less stress in the simulator, since no human life was endangered, and mistakes would not lead to serious consequences. Especially beginners who are still (somewhat) tense before a real operation felt more comfortable and relaxed in the simulator than in a real OR.
However, many also felt a little tense in the simulator, due to the different HLM-model used. Especially more experienced perfusionists felt more comfortable in the real OR with their usual HLM where they exactly know where which elements and which information are.
Furthermore, the situation in the simulator was unusual for some, and they felt observed or even insecure, as they did not know what to expect. In the simulator some also had a feeling of stress, different from the real OR, that resulted from the speed of the process in the simulator and the resulting temporal pressure.
Participants feedback on the simulator
Overall, the design of the simulator was considered to be very realistic, while the simulation (temporal duration, situation in the simulator) is rather improvable: All participants stated that the simulators’ structure was very realistic and especially when sitting behind the HLM, there were hardly any differences. The machine used was a real heart-lung machine with all the important functions that can also be found in a real OR. One drawback for many, however, was that a different model from the ones at the DHZB was used in the simulator. Since some elements differed in their positioning, it was necessary to change the usual behavior and gaze patterns. Moreover, it was repeatedly noted that coming off bypass in the simulator was somewhat different, since the venous volume and flow behavior reacted much more sensitively than with a patient.
The sequence of the simulation flow was also perceived as realistic. However, almost everyone mentioned the significant temporal differences as a major weakness. For most, the simulation was too fast, and they found it not very close to reality. An entire operation that is done within 10 min could not happen in real life. Also, the individual phases, such as coming off bypass at the end of the operation, would usually take longer.
Apart from the clear temporal differences, another factor that the perfusionists considered not so realistic was that the performance of secondary tasks such as blood gas analysis and the control of heparin concentration were omitted in the simulator. It was also mentioned several times that only three people, two of them not being real surgeon and anesthesiologist, were present during the simulation, and that the atmosphere in the simulator was different (more relaxed and fun) than in a real OR.
Another important point was the use of transparent fluid instead of blood in the simulator. On one hand it is an unusual sight for the perfusionist, on the other hand the color of the blood, and especially the color differences, serve as an important source of information, which is lost thereby. It has also been mentioned several times that the lack of working clothes (sterile surgical gowns, gloves, face mask) reduced the realistic feeling in the simulator. Due to the obvious lack of sterility, perfusionists immediately became aware that they were not in a real OR.
Discussion
Overall, it was shown that the simulator’s construction is comparable to a real OR and is also perceived as very realistic by the perfusionists themselves.
The sequence of the individual phases also corresponded to real surgery. However, the duration of the entire operation, and especially of the individual phases, was much shorter in the simulator than in a real operation. Most of the perfusionists stated that the simulation was not comparable with a real operation in this respect. However, the aim of this experiment was not to duplicate a whole clinical operation in the simulator, but to directly compare single phases of HLM operation. In general, the process in the simulator was much smoother than in a real operation. Compared to real surgeries, there were no incidents or additional tasks during the simulation. Furthermore, the communication and the overall atmosphere in the simulator were different and more relaxed.
Certain AOIs could be identified for the individual phases between the perfusionists, both in real OR and simulator. This is a first indication of a fundamental similarity between the two conditions, as the participants look at similar values and elements in certain situations. However, clear differences were also found across all participants, for example when looking at the patient monitor in the first phase. This is usually not observed at all in the OR, but only in the simulator. There were also further differences between both conditions regarding the proportions of the individual AOIs in some subject, making the eye tracking data only partially comparable in both conditions. One reason why the patient monitor was more important in the simulator could be that the dummy patient reacted differently than a real patient and the perfusionists therefore needed to check certain patient values even more in the beginning. Additionally, there seemed to be more focus on the tubes in the OR, especially in the first phase, probably since the blood begins to flow slowly through the tube at this point. In the simulator, however, the tubes were filled with a transparent liquid (water), which is less recognizable and therefore attracts less attention.
One of the problems with eye-tracking turned out to be the fact that the gaze is also directed according to certain questions or commands. When the surgeon asks for a parameter, the perfusionists looks for the corresponding value. Furthermore, the perfusionists moved around more in the OR than in the simulator, also due to secondary activities.
In addition to the objective eye movement data, the subjective questionnaire data of the NASA-TLX on perceived stress also showed some differences. In the case of mental demand, the participants reported significantly higher stress after the real operation than the simulation. One reason for this could be that the perfusionists must pay much more attention during an operation. In a real operation, the patient values change continuously, but not necessarily so in the simulator. Additionally, in a real patient some parameters are decisive, such as the temperature, which does not necessarily have to be controlled in the simulator. Also, perfusionists carry out secondary activities, such as blood gas analyses, which have been omitted in the simulator and thus also reduce mental demands.
Significant differences were also found in the temporal demands, whereby it is again perceived as greater in the real OR. The negative correlation could come from the fact that real operations can take different lengths of time, between about one and several hours, and therefore the comparison with the simulated operation is different. The participants estimated their performance to be significantly higher in the simulator than in the real operation. This may also be since the task in the simulator was simpler and more clearly defined without secondary tasks. In addition, there were no incidents during the simulation in terms of conspicuous values or critical situations.
Overall, further research on the validity of the simulator is needed. Other objective methods should also be used to supplement the eye movement data. It should also be investigated to what extent certain factors are important for the validity of the simulator (e.g. actors, secondary tasks, work clothing, or representation of blood). When simulating blood, for example, the use of a colored liquid can also be an obstacle, as the color differences in the blood provide the perfusionist with important information such as oxygen content. If these are not mapped accordingly, it can also lead to a deterioration compared to the use of water.
Further research could be used to develop concrete requirements for such simulators to establish certain quality standards and to establish simulators in training and advanced education of perfusionists.
Professional simulators used for training are often built for a specific training purpose, characterized by a high level of detail and neglect aspects that do not correspond to the declared training purpose. 17 This could be adopted in the simulator at the DHZB, so the same HLM-model is used as in the clinic. This seems particularly important for practicing critical situations where a particularly fast and targeted reaction is required. It could also be considered whether it might not be necessary to simulate entire operations at once, but only certain phases/incidents, but in all detail and accuracy.
Limitations
Eye tracking analysis was rather difficult, no individual values could be evaluated on the displays for the eye movement data. By combining different values or elements, however, information may have been lost. Also, only certain phases were considered, this does not allow any conclusion about the similarity of entire operations. Statistical evaluation was not possible due to the small sample size.
Various aspects—for example structure of the simulator (different arrangement of displays), psychological factors (atmosphere and clothing more casual, lack of habit) or functionality (simulator reacted differently to patients, clear fluid rather than blood)—may have led to the fact that simulator and real OR are only comparable to a limited extent. These aspects need to be the focus of future research to identify which are critical and which are negligible.
Conclusion
The simulator has a high face validity, but is only valid to a limited extent regarding the eye tracking data and subjective perception.
In general, the structure of the simulator is comparable to a real OR, but there is still room for improvement in simulation, especially with regards to the duration of the simulation. It could be considered, similar to pilots, to train with the exact model that is eventually used in practice, especially in advanced education, to allow specialization in this model and thus enable perfusionists to act faster and more purposefully in critical situations. More time should be taken for the simulation, especially if things are being practiced. If necessary, only individual situations/phases should be simulated, but as realistic and detailed as possible. The relevance of certain individual factors (e.g. color of the blood, persons present, secondary activities, clothing, etc.) should be investigated in further research. Also, a larger sample size would be useful for statistical evaluation in the future.
By introducing secondary tasks, the requirement in the simulator could be increased and thus the feeling of a lower mental demand in the simulator could be improved.
The simulator has great potential for training and advanced education. In addition, the wish and acceptance on the side of the perfusionists for the use of the simulator also became clear. By using the simulator in training, future perfusionists could learn how to operate the machine, try it out and understand connections between different controllers and changing parameters before applying it to living patients. In the field of advanced training, dealing with critical situations is a particularly large field of application.
Supplemental Material
sj-pdf-1-prf-10.1177_0267659120974617 – Supplemental material for Comparison of a perfusion simulator to a clinical operating room: evaluation of eye tracking data and subjective perception. A pilot study
Supplemental material, sj-pdf-1-prf-10.1177_0267659120974617 for Comparison of a perfusion simulator to a clinical operating room: evaluation of eye tracking data and subjective perception. A pilot study by Cynthia Pawelke, Frank Merkle, Dino Kurtovic, Sina Gierig and Gisela Müller-Plath in Perfusion
Footnotes
Acknowledgements
The authors thank Prof. Christoph Starck, the Senior Consultant Cardiac Surgeon at the German Heart Institute Berlin, who has significantly supported the study and the participants of the study.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Supplemental material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
