Neurosurgical Assessment of Metrics Including Judgment and Dexterity Using the Virtual Reality Simulator NeuroTouch (NAJD Metrics)

Abstract

Advances in computer-based technology has created a significant opportunity for implementing new training paradigms in neurosurgery focused on improving skill acquisition, enhancing procedural outcome, and surgical skills assessment. NeuroTouch is a computer-based virtual reality system that can generate output data known as metrics from operator performance during simulated brain tumor resection. These measures of quantitative assessment are used to track and compare psychomotor performance during simulated operative procedures. Data output from the NeuroTouch system is recorded in a comma-separated values file. Data mining from this file and subsequent metrics development requires the use of sophisticated software and engineering expertise. In this article, we introduce a system to extract a series of new metrics using the same data file using Excel software. Based on the data contained in the NeuroTouch comma-separated values file, 13 novel NeuroTouch metrics were developed and classified. Tier 1 metrics include blood loss, tumor percentage resected, and total simulated normal brain volume removed. Tier 2 metrics include total instrument tip path length, maximum force applied, sum of forces utilized, and average forces utilized by the simulated ultrasonic aspirator and suction instrument along with pedal activation frequency of the ultrasonic aspirator. Advanced tier 2 metrics include instrument tips average separation distance, efficiency index, ultrasonic aspirator path length index, coordination index, and ultrasonic aspirator bimanual forces ratio. This system of data extraction provides researchers expedited access for analyzing the data files available for NeuroTouch platform to assess the multiple psychomotor and cognitive neurosurgical skills involved in complex surgical procedures.

Keywords

NeuroTouch NAJD Metrics neurosurgical simulation performance metrics bimanual dexterity neurosurgical judgment brain tumor resection

Introduction

The National Research Council (Canada) working collaboratively with the Neurosurgical Simulation Research Centre at the Montreal Neurological Institute and Hospital and other research groups have developed a computer-based virtual reality (VR) simulation platform called NeuroTouch.^1-5 This system is based on a finite element method and using real-time computing can assess the multiple features of simulated surgical procedures such as brain tumor resection and normal tissue associated injury. The elimination of patient risks associated with technical skills learning is the ultimate goal of simulation-based training.^6,7 In a safe simulated environment the learner achieves the desired learning outcomes where one can repeat the simulated procedure(s) with appropriate demonstrator and performance feedback.^5,7 The utilization of VR simulators like NeuroTouch and appropriate metric technologies designed to address specific educational, psychomotor, and cognitive issues could improve surgical skill acquisition and assessment, enhance procedural outcomes, and further our understanding of surgical expertise. The NeuroTouch platform generates output metrics data which provides quantitative assessment measures useful to track and compare psychomotor performance during simulated operative procedures.^5,7 A critical component of the collaborative studies of the 15 members of the NeuroTouch Consortium spread across 3 continents is the standardization of validated performance metrics.^1,2,7 The output data file pertaining to a specific task performed on NeuroTouch contains valuable information concerning psychomotor and cognitive performance. Critical data extraction from this file necessitates the use of sophisticated software and engineering expertise.

The purpose of this study is to develop a set of objective metrics using standardized nomenclature to measure neurosurgical psychomotor and cognitive skills using the NeuroTouch platform. Based on the data contained in the NeuroTouch file, we have developed and classified 13 NeuroTouch-derived metrics utilizing an enhanced and expedited methodology that can be easily incorporated into any research simulation program. This should enhance the ability of groups interested in neurosurgical simulation to communicate and facilitate the comparison of data pertaining to resident education and expert performance. The validation of these metrics and their implementation into proficiency-based benchmarks should further enhance the training and evaluation of surgical resident performance.^7,8

Methods

The NeuroTouch platform used in this study has been previously described and is outlined in Figure 1. The simulated physical tools such as the ultrasonic aspirator and suction device to perform a simulated brain tumor resection using NeuroTouch are held by the operator and can be observed and tracked in the form of virtual tools that interact with virtual tissues providing haptic feedback. The physical size, shape, and behavior of these tools are modeled on the similar real surgical tools used for tumor removal. Some tools such as the simulated ultrasonic aspirator are activated by a foot pedal as occurs during a patient operative procedure. A haptic device is connected to each tool providing force feedback corresponding to the interaction between virtual tools and simulated tissues. The force that the operator applies with the tool on the virtual tissue and the tool tip position is obtained from the haptic device.

Figure 1.

(A) The NeuroTouch simulator equipped with stereoscopic viewer, bimanual force feedback handles, and activator pedal. (B) Mannequin head with suction instrument. (C) Ultrasonic aspirator view with operating scene with simulated instruments.

The data output from NeuroTouch system for any task is recorded in a comma-separated values (CSV) file. These data are exported to CSV file at a rate of 50 Hz (50 points of data recorded per second or every 20 ms). The 8 values that are exported are

Name of the instrument: records the name of the instrument and the handle site it was connected to (right vs left).

Left or right switch: records the activity of each instrument. When the foot pedal is activated the value will be 1 and if not activated the value will be 0.

Force feedback: records the amount of force measured (in newtons) applied by each instrument.

Translation (X, Y, Z): records each instrument tip coordinate (X, Y, Z) in 3-dimensional space.

Quaternion (X, Y, Z, W): for a given axis (x, y, z) and angle (w), the quaternion representing instrument rotation of the degree around the axis from the origin (0, 0, 0) for (x, y, z).

Total blood emitted: records the amount of simulated bleeding. This value is cumulative. For each 20 ms, the value will include the entire bleeding amount from the beginning of the task until the current time being recorded.

Brain volume: records the amount of simulated brain tissue available in cubic centimeters. At each 20-ms interval, the value will reflect the amount of simulated brain tissue still available. If the operator resects simulated brain tissue, the amount resects will be subtracted in the next cell.

Tumor volume: records the amount of simulated tumor tissue in cubic centimeters present in the scenario. At each 20 ms, the value will reflect the amount of simulated tumor tissue. If the operator resects simulated tumor tissue, the amount resected will be subtracted in the next cell.

Results and Discussion

The goal of the operator during an intracranial tumor resection is to resect the brain tumor with instruments using cognitive techniques and judgment to apply forces that safely remove the tumor with minimal injury to surrounding normal brain tissue. The major focus of any operation is on safety since neurosurgical procedures resulting in postoperative injury are not only associated with increased patient morbidity but also with decreased patient survival.^9,10 However, the necessary technical and cognitive aspects to accomplish the goal of safe tumor resection by the expert neurosurgeon are incompletely understood and are being investigated using VR technologies such as the NeuroTouch platform.^3,5,7 The authors have classified some NeuroTouch performance metrics into tier 1 and tier 2 groups in a previous pilot study designed to assess whether these metrics would encompass the wide range of technical performance seen during novice/expert resection of simulated brain tumors.⁵ Our results demonstrated that these metrics were useful to assess the safety, quality, and efficiency of simulated brain resection procedures. The tier 1 and tier 2 metrics and their relationships to measures of performance as outlined in this communication along with the primary focus of each metric can be seen in Table 1.

Table 1.

Organization of Tier 1, Tier 2, and Advance Tier 2 Metrics Based on Safety, Quality, Efficiency, Bimanual Dexterity, and Cognitive Performance Assessment.

Metrics	Individual Metric	Metric Focus
Tier 1
Intended to evaluate the safety and quality of operative performance	Amount of blood loss	Safety
	Tumor resection percentage (TRP)	Quality
	Brain volume removed (BVR)	Safety
Tier 2
Intended to study independent hand motor skills to achieve the operative goal	Total tip (ultrasonic aspirator and suction instruments) path length (TTPL)	Efficiency
	Maximum force applied (MFA)	Safety
	Sum of forces utilized (SFU)	Safety
	Average forces utilized (AFU)	Safety
	Pedal activation frequency	Efficiency
Advanced Tier 2
Intended to study complex motor and cognitive bimanual neurosurgical skills interactions	Instrument tips average separation distance (ITASD)	Bimanual Dexterity
	Efficiency index	Efficiency and Decision Making
	Simulated ultrasonic aspirator path-length index (PLI)	Efficiency and Decision Making
	Coordination index	Bimanual Dexterity
	Simulated ultrasonic aspirator bimanual forces ratio (BFR)	Bimanual Dexterity and Cognitive Distraction

Tier 1 Metrics

Tier 1 metrics are provided directly from the NeuroTouch platform after the completion of each simulated tumor resection. These are amount of blood loss, percentage of tumor resected and volume of simulated “normal” brain tissue removed surrounding the tumor.

Amount of Blood Loss (BL)

This metric provides a volume measurement in cubic centimeters (cm³) of blood loss during the tumor resection. The goal for the operator is to remove simulated tumors with minimum blood loss. Increased amount of blood loss is an undesired outcome and thus an inverse measure relating to patient safety.

The value can be obtained from the CSV file under “Total Blood Emitted” column. Since the value is cumulative, total blood loss will be recorded in the last cell of that column.

Tumor Percentage Resected (TPR)

This metric provides a measurement of the percentage of the simulated tumor resected. The goal for the operator is to resect the maximum amount of the simulated tumor without resecting any of the surrounding simulated “normal” brain tissue and is a measure of the quality of the simulated procedure. This value can be calculated by obtaining the value of last cell under “tumor volume” which records the remaining amount of residual tumor. The percentage of resected tumor can be calculated by dividing the amount of residual tumor by the total tumor volume and subtracting this value from 100.

Percentage of tumor resection = 100 - (residual tumor volume / initial tumor volume)

Brain Volume Removed (BVR)

This metric provides a volume measurement in cubic centimeters (cm³) of simulated “normal” brain tissue removed during the tumor resection. The goal for the operator is to remove no or minimal “normal” simulated brain tissue during tumor resection. The brain volume removed is a direct measure of tissue injury during the resection and thus an inverse measure relating to patient safety.

This value can be calculated from the “Brain Volume” column. In the last cell, the amount of final simulated brain tissue volume will be recorded.

Brain volume resected = initial brain volume (first cell) - last recorded value of brain volume .

Tier 2 Metrics

We have classified tier 2 metrics into 2 subgroups: tier 2 metrics and advanced tier 2 metrics. Tier 2 metrics were developed to aid in the understanding of how an operator uses his or her motor skills to achieve the safety, efficiency, and cognitive goals of the simulated operative procedure. These metrics enable researchers to analyze neurosurgical psychomotor skills in an objective manner independently, focusing on the psychomotor skills of each hand. Tier 2 metrics are instruments total tip path length, maximum force applied, sum of forces utilized, average forces utilized, and pedal activation frequency.

Advanced tier 2 metrics are designed to assess the complex psychomotor and cognitive neurosurgical skills focusing on a number of critical areas of expert performance. First, decision-making ability as related to surgical judgment and its execution efficiency is measured by efficiency index and simulated ultrasonic instrument path length index. Second, the assessment of (a) the complex psychomotor skills in 2-hand interaction involves the metrics of coordination index and instrument tips average separation distance, (b) dexterity includes the metrics of instrument tips average separation distance and ultrasonic aspirator bimanual forces ratio, and (c) the economy of hand movements to achieve the operation goal is measured by the efficiency and coordination indices.

Tier 2 metrics are described next.

Total Tip (Simulated Ultrasonic Aspirator and Suction Instrument) Path Length (TTPL)

The length of the path traversed by the tip of the instrument tool measured in millimeters (mm) is used as a metric to measure the efficiency of the tool usage during simulated tumor resection. The goal for the operator is to carry out the resection using the most efficient and safe path trajectory of the ultrasonic aspirator and suction instrument. The specific trajectories and average and maximum velocities related to the tool tip tracking could also be scrutinized to assess the angle and speed of instrument maneuvers used by an operator.

TTPL is calculated based on the change of X, Y, Z coordinates from time 1 (T₁) to time 2 (T₂).

If T₁ coordinate = X₁, Y₁, Z₁ and T₂ coordinate = X₂, Y₂, Z₂, the distance traveled by the instrument (D₁) from T₁ to T₂ can be measured by the equation

D_{1} = \sqrt{{(x_{2} - x_{1})}^{2} + {(y_{2} - y_{1})}^{2} + {(z_{2} - z_{1})}^{2}}

Final TTPL will be the sum of all distances, TTPL = D₁+ D₂+ … + D_n.

Maximum Force Applied (MFA)

This metric provides a measure of the maximum force (in newtons) that the operator applies on the tumor and the “normal” simulated brain tissue by the ultrasonic aspirator and suction instruments during the procedure and is a measure of safe force application. The operator uses cognitive input balancing the continuous haptic input from the instrument-tissue interaction to use force as safely as possible without causing surrounding tissue injury.

MFA is the largest value recorded under the “Force Feedback” column.

Sum of Forces Utilized (SFU)

Sum of all applied force samples (in newtons) during the simulated operation is used as a measure for the overall applied force employed to resect the tumor by each instrument. The goal for the operator is to use the most appropriate safe applied forces during the resection.

SFU is calculated by summing all the value under the “Force Feedback” column in the data file.

Average Forces Utilized (AFU)

Average applied force (in newtons) during the simulated operation was used as a measure for the overall average applied force employed to resect the tumor by each instrument. The goal for the operator is to use the appropriate safe applied forces during the complete tumor resection.

AFU is calculated by taking the average of the value under the “Force Feedback” column in the data file but excluding zero value because the aim is to obtain the value of average forces when there is actually a force being applied.

The AFU metric has some advantages over SFU. First, if the time given to perform the resection is not fixed then the SFU does not control for the length of the procedure. Second, SFU will be influenced by the sampling frequency, which in the current software is at 50 Hz. For the same amount of force applied, the SFU result will increase if the sampling frequency increases and vice versa.

Pedal Activation Frequency

The number of times an operator activates the simulated ultrasonic aspirator or other similar devices with the appropriate foot pedal was assessed by this efficiency metric. To maximize the efficiency of any instrument controlled by a foot pedal such as the ultrasonic aspirator, the operator should employ it using the minimum number of pedal activations.

Pedal activation metric measured by counting the frequency of “Left or Right Switch” cell value when it is changes from 0 to 1.

Advanced Tier 2 Metrics

Instrument Tips Average Separation Distance (ITASD)

The average distance between the instruments tips (in millimeters) is used as a metric to measure how the 2 hands function in the coordinated resection the tumor and relates to operator bimanual dexterity. The goal for the operator is to carry out the resection using the most appropriate distance between the tips of the ultrasonic aspirator and suction instrument while keeping both instruments continuously in the operative field.

ITASD is calculated based on the difference of X, Y, Z coordinates between instrument 1 (ultrasonic aspirator; I₁) and instrument 2 (suction instrument; I₂).

If I₁ coordinate = X₁, Y₁, Z₁ and I₂ coordinate = X₂, Y₂, Z₂, the distance between instruments tips (D₁) can be measured by the equation

D_{1} = \sqrt{{(x_{2} - x_{1})}^{2} + {(y_{2} - y_{1})}^{2} + {(z_{2} - z_{1})}^{2}}

Final ITASD will be the average of all distances (D₁ + D₂ + … + D_n).

Efficiency Index

Efficiency index is defined as the percentage of time an operator spends actively resecting the tumor with the ultrasonic aspirator divided by the total time for the task. The goal for the operator is to use the tool as efficiently as possible during the resection of the simulated tumors without any unnecessary pauses. The efficiency index also measures cognitive–motor skills interaction, concerned with decisions related to the planning of the next step of the operative procedure simultaneously while carrying out the current step with minimum need for extra time off for this forward planning.

The efficiency index can be calculated indirectly based on the force feedback of the ultrasonic aspirator instrument and time.

Efficiency index = amount of time ultrasonic aspirator force feedback > 0 / total time

Simulated Ultrasonic Aspirator Path-Length Index (PLI)

This is defined as the percentage of the ultrasonic aspirator TTPL spent in the simulated intratumoral region divided by the overall ultrasonic aspirator TTPL. The goal for the operator is to carry out the resection while deciding to minimize any unnecessary movement of the ultrasonic aspirator while not present in the defined simulated tumor intraoperative field, which improves the efficiency of the operative procedure.

The PLI value is calculated based on TTPL and force feedback of ultrasonic aspirator.

PLI = ultrasonic aspirator TTPL if its force feedback > 0 / ultrasonic aspirator TTPL

Coordination Index

This is defined as the percentage of time the suction instrument is used simultaneously with the ultrasonic aspirator to control bleeding divided by the time the suction instrument was used overall. The goal for the operator is to introduce the suction instrument into the operative field, only as necessary, without interrupting the function of the ultrasonic aspirator. The coordination index measures the quality of 2-hand interaction during simulated tumor resections. The main difference between this metrics and the ITASD metric is that ITASD measures 2-hand coordination regardless of whether the suction instrument is activated while the coordination index measures both hands coordination only when the suction instrument is active.

The coordination index is calculated based on the time the suction instrument is used and the force feedback from ultrasonic aspirator:

Coordination index = amount of time suction instrument and ultrasonic aspirator force feedback > 0 / amount of time suction force feedback > 0 .

Simulated Ultrasonic Aspirator Bimanual Forces Ratio (BFR)

This metric provides the force ratio between the average forces applied by the ultrasonic aspirator during time intervals when it is being used simultaneously with the suction instrument compared with the average forces applied by the ultrasonic aspirator when it was utilized alone. The goal for the operator is to apply similar and equal forces with the ultrasonic aspirator whether or not the suction instrument is being used. This metrics is a measure of cognitive distraction since the operator must transition from a relatively simple maneuver using only the ultrasonic aspirator in the dominant hand to a more complex bimanual maneuver involving the optimal utilization of a second tool, the suction instrument in the nondominant hand to control continuous bleeding. Junior residents are hypothesized to have higher bimanual force ratio when compared with senior residents while this ratio for neurosurgeons would be closer to 1.

BFR is calculated based on the force feedback from the ultrasonic aspirator and suction instruments:

BFR = average forces applied by ultrasonic aspirator when force feedback of suction instrument and ultrasonic aspirator > 0 / average forces applied by ultrasonic aspirator when the force feedback of the suction instrument = 0

When calculating the average force, it is very crucial to exclude all zero values (when no force is being applied) from the calculation, otherwise the calculated average forces will be an underestimation of the actual average force utilized.

An Excel macro-software to generate the 13 performance metrics outlined directly from the CSV files of NeuroTouch data output has been developed. This program will be available online under a secure network for all investigators having access to the NeuroTouch platform to utilize and further develop.

These metrics as measures of quantitative assessment can be used to track and compare the safety, quality, efficiency, judgment, and dexterity of psychomotor performance during complex simulated operative procedures utilizing the NeuroTouch platform. Face, content and construct validity of the NeuroTouch system has been previously determined in a competitive setting when medical students and residents were compared.³ Valid objective criteria for technical and cognitive skills assessment are presently not included in neurosurgical curriculum since these criteria have not been developed that differentiate novice (resident) from expert (neurosurgeon) performance.⁷ However, the widespread implementation of resident work hour restrictions is influencing resident access to operative experience and may necessitate the development of other forms of training such as VR simulators.^11-13

The metrics outlined in this communication when validated in further studies should facilitate the development of technical and cognitive training curriculum. The availability of Excel macro-software to expedite the generation of all these metrics direct from CSV files of NeuroTouch data output should improve communication among neurosurgical simulation centers further enabling data comparison pertaining to neurosurgical education, training, and evaluation. These standardized metrics would be useful in developing multi-institutional databases from centers utilizing the NeuroTouch platform improving the quality and impact of research in multiple fields of surgical education since this system is also being utilized by other surgical specialties including otolaryngological endoscopic sinus and transphenoidal surgery.^14,15 The NeuroTouch research community using a common simulation output language would have a defined basis for designing cooperative multi-institutional trials and comparing statistical results. The evaluation, validation and implementation of proficiency based benchmarks based on these and other NeuroTouch-derived validated metrics should further the training and evaluation of resident and expert performance.^5,7

Conclusions

Data extraction from NeuroTouch and subsequent metrics development requires the use of sophisticated software and engineering expertise. In this article, we organized, classified, and further developed a set of 13 motor skills objective measures (metrics) with the specific details of their expedited calculation and use in the field of simulation research.

Footnotes

Acknowledgements

The authors would like to thank Dr Robert DiRaddo, Group Leader, Simulation, Life Sciences Division, National Research Council of Canada at Boucherville and all his team for their expertise in the development of the NeuroTouch platform. We would also like to acknowledge the support of Drs Mahmoud Al-Yamany and Lahbib Soualmi, National Neuroscience Institute, Department of Neurosurgery, King Fahad Medical City, Riyadh, Saudi Arabia.

Author Contributions

FEA developed the software to extract NAJD metrics, developed the idea of Tier 2 and Advanced Tier 2 metrics. Wrote the manuscript. GAA reviewed the manuscript and wrote major part of the introduction. AJS reviewed the manuscript and edited the figures. HA interpreted the CSV files. AWS assisted in study conduction, organization, and data collection. RFDM who is a senior author, supervised the whole study. Reviewed the manuscript, Reviewed all metrics; accuracy, validity, Neurosurgical applicability and accuracy.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Di Giovanni Foundation, the Montreal English School Board, the B-Strong Foundation, the Colannino Foundation, and the Montreal Neurological Institute and Hospital. Dr R. F. Del Maestro is the William Feindel Emeritus Professor in Neuro-Oncology at McGill University. Dr H. Azarnoush held the Postdoctoral Neuro-Oncology Fellowship from the Montreal Neurological Institute and Hospital.

References

Delorme

Laroche

DiRaddo

Del Maestro

. NeuroTouch: a physics-based virtual simulator for cranial microneurosurgery training. Neurosurgery. 2012;71(1 suppl operative):32-42.

Choudhury

Gelinas-Phaneuf

Delorme

Del Maestro

. Fundamentals of neurosurgery: virtual reality tasks for training and evaluation of technical skills. World Neurosurg. 2013;80(5):e9-19.

Gelinas-Phaneuf

Choudhury

Al-Habib

. Assessing performance in brain tumor resection using a novel virtual reality simulator. Int J Comput Assist Radiol Surg. 2014;9:1-9.

Rosseau

Bailes

del Maestro

. The development of a virtual simulator for training neurosurgeons to perform and perfect endoscopic endonasal transsphenoidal surgery. Neurosurgery. 2013;73(suppl 1):85-93.

Azarnoush

Alzhrani

Winkler-Schwartz

. Neurosurgical virtual reality simulation, metrics to assess psychomotor skills during brain tumor resection [published online June 27, 2014]. Int J Comput Assist Radiol Surg. doi:10.1007/s11548-014-1091-z.

Miller

. The assessment of clinical skills/competence/performance. Acad Med. 1990;65(9 suppl):S63-S67.

Gelinas-Phaneuf

Del Maestro

. Surgical expertise in neurosurgery: integrating theory into practice. Neurosurgery. 2013;73(suppl 1):30-38.

Gallagher

. Metric-based simulation training to proficiency in medical education: what it is and how to do it. Ulster Med J. 2012;81(3):107-113.

Del Maestro

. Surgical resection and glioblastoma: molecular profiling and safety. Can J Neurol Sci. 2012;39:561-562.

10.

McGirt

Chaichana

Gathinji

. Independent association of extent of resection with survival in patients with malignant brain astrocytoma. J Neurosurg. 2009;110:156-162.

11.

Feanny

Scott

Mattox

Hirshberg

. Impact of the 80-hour work week on resident emergency operative experience. Am J Surg. 2005;190:947-949.

12.

Blanchard

Amini

Frank

. Impact of work hour restrictions on resident case experience in an obstetrics and gynecology residency program. Am J Obstet Gynecol. 2004;191:1746-1751.

13.

Irani

Mello

Ashley

Whang

Zinner

Breen

. Surgical residents’ perceptions of the effects of the ACGME duty hour requirements 1 year after implementation. Surgery. 2005;138:246-253.

14.

Neubauer

Wolfsberger

. Virtual endoscopy in neurosurgery: a review. Neurosurgery. 2013;72(suppl 1):97-106.

15.

Varshney

Frenkiel

Nguyen

LHP

. Development of the McGill simulator for endoscopic sinus surgery: a new high-fidelity virtual reality simulator. Am J Rhinol Allergy. 2014;28:330-334.