Abstract
Functionality and cosmetics are two concerns for future hand prosthesis development and they both can be improved by a combination with artificial soft materials which can mimic human skin. To bridge the gap between the human and artificial side, it is essential to have a comprehensive understanding of the human skin’s biomechanics, especially the fingertip’s haptics-related mechanism. Available studies characterise the mechanical behaviour of human fingertip only by deterministic models based on either statistical data analysis or fingertip structure/viscoelasticity analysis. To take the force uncertainty into consideration, this paper proposes a novel probability-based haptics model, which includes two parts: a force prediction model to obtain the most possible contact force according to the indentation depth, and a probabilistic model based on Gaussian distribution to describe the force uncertainty. Experiments were conducted by pressing subjects’ index fingertips against a cone-shape probe with the measurement of the contact force and the indentation depth under a wide range of 0∼5 mm. Four types of non-linear regression models and the Gaussian distribution model are applied for model training and validation. Experiment results reveal that the contact force varying with the indentation depth presents the characteristics of non-linearity, dispersion, and individual difference. Model testing results confirm the effectiveness of the haptics model on force prediction and force uncertainty description. An example of its application on a virtual hand of a rehabilitation system is demonstrated.
Introduction
Hand manipulation is heavily relied on intuitive human-object interaction and environment exploration. Primary hand-object contacts happen on fingertips and are perceived by mechanoreceptors during hand manipulation [1]. Mechanoreceptors are sensory units which distributed in the human skin to detect mechanical stimulations such as the force, vibration and pressure. When an external contact force deforms the surface of a fingertip, it causes a strain distribution in the underlying soft tissues and subsequently stimulates mechanoreceptors. Then, the mechanoreceptors generate a sequence of voltage pulses which are transmitted through the peripheral nervous system to the central nervous system where the tactile information is processed [2]. It can be indicated that the mechanical changes of the human fingertip, such as deformation, are important for the generation and detection of tactile sensation. Thus, the investigation of the human fingertip’s biomechanics may broaden the research perspectives in the field of biomechanical engineering and robotics. Related studies have gained increasing attention in a broad range of applications, such as tactile sensation rehabilitation, artificial fingertip/skin development [3], finger model establishment in virtual reality [4] and ergonomic design [5]. A comprehensive understanding of human fingertip biomechanics is essential prior to full exploitation of its most potential in both clinical and industrial environment. Most existing studies concerning neural reaction properties under the fingertip skin only conduct experiments under a small indentation depth of 0∼0.2 mm to 0.8∼2.5 mm [6, 7], while large deformations that may occur in the context of hand prostheses were not taken into account. Therefore, this study aims at achieving a quantitative characterisation of human fingertip’s mechanical behaviour for the potential application of tactile rehabilitation in hand prostheses. Experiments in vivo are conducted to measure the force variation under a large range of the indentation depth (0∼5 mm) on human fingertips. A haptics model based on Gaussian distribution is proposed to describe the relation between the contact force and the skin deformation depth.
The remaining parts of this paper are organized as follows. Section 2 introduces related studies on fingertip biomechanics. Section 3 presents the experiment settings, data recording and methods of the haptics modelling. The establishment of the haptics model is demonstrated in Section 4. Section 5 presents a discussion and future work.
Related work
Existing studies of the human fingertip’s biomechanics and the model establishment mainly aimed at the neural reaction properties of the mechanoreceptors under the fingertip’s skin by investigating the physical response of the human fingertip under various load conditions. The final outputs of these studies are mathematical models or simulation models targeting at the relation between the contact force/pressure and the fingertip skin deformation, despite different experimental setups and modelling methods.
Researchers employed various shape of indenters to press the fingertip for the deformation generation and retrieval, such as the point load [8], line load [9–11] and flat load [7, 12]. To test multiple indenters may help to achieve a comprehensive understanding of the fingertip’s biomechanics, because the force-deformation relation may be subject to the conditions of the contact surface. In these studies, experimental data used for modelling can be categorised into 3 classes, including the measurement of force-displacement data [13], force-contact area data [14] and pressure-deformation data [15, 16].
Regarding the methods used for the establishment of the haptics model, some research, from a structural and experiment-based point of view, simplified the human fingertip as one or multiple layers of membranes [9]; while others, in the sight of physical character analysis, built physical-mathematical models according to the viscoelasticity analysis of the fingertip soft tissue. Taking a structure-based model for example, by simplifying the fingertip as an incompressible fluid-filled elastic membrane, Srinivasan proposed a “waterbed” model which predicted the fingertip deflection profile under line loads and the model was validated on humans and monkeys in vivo [9]. The model was created under many simplifications and assumptions, such as neglecting the viscoelastic effects and assuming the membrane to be linearly elastic, so the accuracy was inevitably sacrificed. Then, non-linear models were proposed to describe the relation between the normal force and the radius of the contact area for soft-finger materials [17, 18]. In some research, only soft materials such as rubber and silicone were employed in experiments instead of human fingers. However, to reveal the actual biomechanics of the human fingertip, in vivo tests are necessary. After achieving a deeper understanding of human fingertip characters, more complex models based on the analysis of fingertip viscoelasticity were proposed. Johnsson and Balkenius proposed a LUCS Haptic Hand together with haptic models which employed the self-organizing map (SOM) neural network to identify objects’ size and softness [19]. Jindrich et al. presented a non-linear viscoelastic model to describe the fingertip force-displacement relation and concluded that the non-linear model could predict the fingertip force according to the fingertip pulp compression during dynamic tapping [20]. Duchemin G. et al. conducted in vivo test and proposed a model to reflect the soft finger and the underlying tissue’s behaviour and properties, which took into account the influence of motion velocity and lubrication [21].
Additionally, to investigate the microscopic sensing process of the human finger, 2D/3D finite element (FE) models were established based on the numerical models to simulate human finger sensing characteristics. Wu et al. proposed a two-dimensional (2D) structural fingertip model which could predict the internal force distributions within the soft tissue and simulate the deflection profiles of a fingertip under a line load, a one-point load and a two-point load with a small indentation depth of 1 mm [8]. Three-dimensional (3D) models were also proposed under a flat load, a sharp wedge [11] and a line load [22]. For example, a 3D FE model was developed to predict the temporal force response of fingertip under both a line load and a cylinder load surface deflection [6].
There still exist some problems despite a few achievements in the area of fingertip modelling. Most biomechanical models of human fingertips were established from a microcosmic point of view concerning the mechanoreceptors response to external tactile stimulation with a small indentation depth less than 3 mm. Besides, models based on either the anatomical structure or the physical characterisations of human fingertip inevitably requires numerous assumptions and simplifications, which is quite likely to restrict the accuracy of theoretical models comparing with statistical models. Additionally, critical subject dependency has been observed by various experiments in the recognition of fingertip’s biomechanics and contact force through skin deformation.
So far, no research has been conducted yet to address the aforementioned issues from an external and macroscopic perspective. Thus, this study aims to reveal the fingertip behaviour under a higher indentation depth with a probabilistic haptics model to highlight the subject dependency instead of a deterministic one. Specifically, a Gaussian distribution is employed to fit the relation between contact force and the indentation depth of the human fingertip.
Materials and method
This section presents an experimental paradigm to test the biomechanics of human fingertip, followed by a demonstration of the methods used for the haptics model establishment.
Experimental setup
The experimental setup as shown in Fig. 1(a) comprised a Finger Tactile Pressure Sensing (FingerTPS) system and an indenting test bracket. FingerTPS is a commercial system designed to collect high-quality data of force/pressure exerted on human hand. It consisted of a calibration force sensor, wearable fingertip force sensors, a Chameleon Visualization and Data Acquisition Software which collects and displays the force data through the interface. In this study, we employed this system to measure the contact force varying with the indentation depth on the fingertip in real time. The indenting test bracket with a cone-shape probe was fixed on the edge of a table and was used to press the fingertip to different indentation depths. A vertical view of the indenting bracket with the fingertip is illustrated as Fig. 1(b). The backboard attached to the table side helped to fix the fingertip during the experiment. The probe was adjusted by a screw which has 7 rounds of adjustable whorls and the screw’s vertical length changed 1 mm per round, so the indentation depths of the skin deformation could range from 0 mm to 7 mm.
Experimental setup for the haptics data collection.
Six subjects (23–31 years old, 2 females and 4 males) participated in the experiment. The study was approved by the local ethical committee and the subjects signed the consent before participation. None of them reported any skin or finger injury.
During the experiment, the subject was comfortably seated in an armchair with his/her body-side towards the table where the indenting bracket was fixed. Figure 1(a) presented the indenting and recording scene of the experiment. In preparation, the subject needed to wear a fingertip force sensor on their left index fingertip and press the finger on the reference sensor for calibration according to the instruction of the FingerTPS system. Then the subject was asked to put the tested finger into the indenting test bracket with the middle point of the fingertip pulp directly towards the cone top of the screw as shown in Fig. 1(b). Initially, the cone top of the screw was adjusted to be detached from the fingertip. It should be noted that there would be a tiny but measurable force to the fingertip force sensor and be observed from the Chameleon Visualization and Data Acquisition Software without indentation because of the pressure between the sensor and the fingertip. To eliminate the interference, the force was firstly set to 0.0 N as a measurement baseline and maintained throughout the experiment of the same subject. After then, the subject was reminded to keep the tested finger as stable as possible to make sure that the indentations of all the experiment sessions were within the same region of the finger pulp to avoid measurement error.
At first, the probe was screwed to lightly touch the fingertip but not cause any pressure and deformation, which was considered as the starting point of each test session. Then, the probe was screwed to indent 0.5 mm deeper every time until it reached the max indentation depth of 7 mm or the subject reported discomfort, which was viewed as one session. During a single session, the contact force corresponding to different indentation depths ranging from 0.0 mm to 7.0 mm were recorded, so a maximum of 15 values of force were recorded in each session. At the end of each session, the researcher would screw the probe back to its starting point. To collect enough data to verify the experiment’s repeatability, the indentation tests were repeated 20 times on each subject. The duration of the whole experiment lasted about 40 minutes and slightly varied with individuals.
Data recording
The experiment data collected from subject 1 to 6 are recorded as shown in Fig. 2. According to the experiment results, the maximum indentation depth that all the subjects could endure is 5 mm, so the description and analysis in the following sections will only consider the data ranging from 0 mm to 5 mm.
20 sessions of the measured forces at discrete indentation depth ranging from 0 mm to 5 mm on 6 subjects. At different indentation depths, the mean forces are depicted as points, and the standard deviations are presented by bars.
Taking the data of subject 2 as shown in Fig. 2(b) for example, there are 11 measured indentation depth d (d =0, 0.5,..., 4.5, 5 mm), and each of them corresponds to 20 measured contact forces. The contact force increases monotonically and non-linearly with the indentation depth. The slope becomes bigger during the large indentation than the small indentation and the turning point appears when the indentation depth is around 3 mm, although it is not very obvious. Meanwhile, the data dispersion becomes more obvious with the increase of the indentation depth. The data of the other five subjects also share the same features—monotonically increasing, non-linear and increasing dispersion. Additionally, we can see that the changing range and changing speed of the contact force are different among the subjects, although they show a similar trend. For example, the average contact forces of subject 1, 4 and 5 range between 0∼2 N, while subject 2, 3 and 6 show a larger range of about 0∼2.5 N. The force dispersion of subject 3 and subject 5 is obviously larger than that of the other subjects, especially subject 1 and 6. Thus, the skin conditions of fingertip exactly varies with individuals, although there are also several fundamental features in common for general human skin.
The haptics model aims to characterise the haptics-related biomechanics of the human fingertip, which can be described by the relation between the contact force and the skin deformation. However, a model with certain parameter values may not reflect the specific state and requirement of every subject. Experience indicates that the data, i.e. the biomechanics of the fingertip, varies with individuals, and uncertainty exists during skin deformation. Consequently, an individual-dependent probabilistic model is expected to reflect the individual difference and force variance. Comparing with other probability distributions, Gaussian distribution is finally chosen to describe the force variance due to its several advantages, such as a simple mathematical form, fewer parameters and the association with the Central Limit Theorem, which makes it a good choice in many cases to implement [23]. Thus, the proposed haptics model will include a force prediction part and an uncertainty description part to characterise the fingertip biomechanics under physical contact, as shown in Equation (1).
In Equations (2) and (3), a0, a1, b1, ω, a and b are 6 coefficients which need to be fitted. The deriving process of the haptics model is demonstrated as below.
Firstly, the force prediction model F (d) aims to predict
Then, a continuous model F (d) can be obtained by non-linear regression based on Fourier series model as Equation (2), which is the prediction model of
On the other hand, considering the force variance even under the same indentation depth, it is assumed that the probability density distribution of the contact force f at a certain indentation depth d follow the Gaussian distribution as Equation (5) due to the skin/force uncertainty:
The mean of the measured forces at a certain indentation depth μ|d can be easily estimated according to Equation (4) and Equation (6).
Next, the standard deviation of the measured force can be estimated according to Equation (7), which is also analysed in Section 4.
The exponential model is selected to fit the data, and a continuous model of σ - d can be achieved as Equation (8). The selection of exponential model is explained in Section 4. A typical fitting performance is shown in Fig. 4.
σ - d fitting performance based on exponential regression.
After obtaining the σ|d and μ|d, the force distribution at a certain indentation depth can be obtained by introducing them to Equation (5), and the result is visualized as Fig. 5.
Force probability density distribution at discrete indentation depths.
By introducing Equations (2) and (8) to Equation (5), the probability distribution p (f|d) can be obtained as shown in Equation (3). Thus far, both parts of the whole haptics model are achieved.
This section demonstrates a detailed modelling process based on a subject’s experimental data, including model training, validation and test. The main evaluation measure of the haptics model is the root mean square errors (RMSE) between the actual forces and the model’s fitting outputs to evaluate the accuracy and precision of the haptics model. Data pre-processing was conducted to eliminate outliers. Based on the 20 sessions of the collected data, we calculated the sum of corresponding forces of each, and the sessions with the highest sum and the lowest sum were excluded. The rest 18 sessions of data were used for haptics model establishment, which includes training, validation and testing. Nine-fold cross-validation was applied. Among them, the first 14 sessions of data are used for model training, while the following 2 sessions for validation and the other 2 sessions for testing.
Training
Taking subject 1’s data for example, the 14 sessions of the training data are presented as dots in Fig. 6. To predict the most likely force
Training data of subject 1 and the fitting performance of various regression models.
The fitting performance of different regression models based on the data of subject 1 were compared as curves in Fig. 6 where their respective root mean square errors (RMSE) were also displayed. The polynomial model presented the least RMSE, which indicated that it seemed to be the most suitable fitting model for subject 1 to predict
Likewise, a suitable model to predict the non-linear relation between the standard deviation of force σ and the depths d was also selected among the above regression models (Equation (9), (10), (11), (12), and (13)). Their fitting performance with respective RMSE was presented in Fig. 7. Considering that exponential model had the smallest RMSE and less coefficients than other models, it was chosen to describe the σ - d relation. Thus, the prediction model of σ - d for subject 1 was achieved as Equation (15). It should be noted that when d = 0, the contact force f is always 0 and there is σ = 0. Consequently, the probability of f = 0 at d = 0, is 100% and the probability density tends to positive infinite, so it is not necessary to discuss the probability when d = 0.
Fitting performance of experimental force’s standards deviations.
Then, by introducing Equation (14) and Equations (15) to (5), a continuous probability density distribution of the contact force f and indentation depth d for subject 1 were obtained as Equation (16) and depicted as Fig. 8.
Continuous probability density distribution of f - d.
Above all, the whole haptics model for subject 1 was achieved as Equation (17).
Similarly, the haptics models for other subjects can be obtained in the same way. Tables 1 and 2 list the regression models’ RMSEs based on all the 6 subjects’ experimental data, with the smallest RMSE value for each subject marked in bold. The four non-linear regression models were compared, given the obvious non-linear relation and the largest RMSE of the linear the polynomial model as shown in Figs. 6 and 15. In Table 1, the Fourier series model outperforms the other three models for subject 6, while polynomial model presents the least RMSEs for other subjects. In spite of this, Fourier series model shows the least RMSE on average, and its RMSEs for each subject are comparable with the least RMSEs which are fitted by polynomial model. In Table 2, the exponential model exhibits the least RMSE on most subjects and on average, which indicates it can be a good choice for σ - d ’s fitting.
RMSE of different
RMSE of different σ - d regression models for different subjects based on 14 sessions of experimental data
Considering the consistency of the fitting performance of above regression models on different subjects, it may not be necessary to test various regression models for each subject every time. If Fourier series model/polynomial model and exponential model keep demonstrating good performance on some new data for validation, they can be set as the default models to fit
Another two sessions of data are used for validation. To choose a proper regression model among the Fourier series model, Gaussian model, polynomial model and an exponential model for the fitting of
RMSE of different
regression models for different subjects based on 2 sessions of experimental data
RMSE of different
RMSE of different σ - d regression models for different subjects based on 18 sessions of experimental data
The rest 2 sessions of data are used for testing by comparing the model’s predictive results based on Equation (14) with experimental data. The RMSE of
RMSE of
and error percentage of 2 sessions of testing data
RMSE of
A haptics model based on Gaussian distribution is proposed in this study to reflect the relation of force and the fingertip deformation depth ranging from 0 mm to 5 mm during the physical contact process. Experiments are conducted to collect in vivo data for analysis from multiple subjects. The results verify the non-linearity of the fingertip’s mechanical characteristics. With the increase of indentation depth, the contact force’s dispersion becomes large and its rising speed becomes fast, which may be attributed to the influence of bone. Based on statistical methods and non-linear regression models, a force-predictive model and a force probability distribution model are derived. Both of them form the whole haptics model. Thus, the haptics model can not only predict the most possible force under a certain contact deformation but also cover a reasonable level of force variance, which makes it potential to contribute to the optimization of robot hand control and hand prosthesis manipulation.
Results analysis
The non-linear increasing slope may be attributed to the approach to the bone under the soft tissue of fingertips and the deviation of the contact position when the indentation went deeper. It is evident that the more the skin is pressed, the more force the bone receives. When the skin deformation depth is less than 3 mm, the indentation mostly happens in the soft tissue and it is little influenced by the bone. When the depth is more than 3 mm, the bone will start to undertake the main pressure caused by the external contact. Under the same indentation increment, the reaction influenced by the bone together with the skin becomes much higher than that mostly influenced by the pure skin. Thus, the rising speed of the contact force increases under larger indentation depth when the influence of bone is involved, which also coincides with why the studies on microscopic sensing process of human skin usually experiment under a small indentation depth less than 3 mm.
The force dispersion may result from a slight deviation of the contact position. On one hand, the force dispersion is reasonable in practice, because the contact deviation is inevitable during the experiment. Given the experiment including 20 sessions and lasting for about 30 minutes for each subject, it is not easy to keep their fingers absolutely stable in the same gesture and position, especially under a deep press from the probe. When under a large indentation depth, even a tiny displacement of the pressing point may lead to a noticeable force difference, because the reaction from the bone is much stronger than that of the soft tissue. Besides, this phenomenon also coincides with the practical process of hand manipulation that the contact force may not be the same every time even under same gestures. On the other hand, the capability of body-control varies with individuals. Taking the data of subject 1 and subject 2 for example, the former (Fig. 2(f)) shows small dispersion even under large indentation depth, while the latter (Fig. 2(c)) shows larger force dispersion than others. Despite the individual difference, the stability of the force data may be improved by subject training or by an optimization of the apparatus setup, which may improve the comfort of the experiment process and facilitate body stability.
The force dispersion cannot be characterised by a deterministic model as proposed in previous studies, and this phenomenon will further influence the soft-hand control performance. Thus, it is necessary to introduce an uncertainty element into the characterisation of fingertip’s biomechanical behaviour. To fill the gap, a probability-based haptics model which can cover a reasonable level of force variance is established in the following section.
Future work
This paper is a preliminary exploration of the probability-based haptics model, further research will be done in the future. For example, the experimental setup and design may be improved to obtain a more stable indentation dataset and reduce the number of repeated experiment sessions. In this study, 20 sessions of gradual indentation ranging from 0 mm to 5 mm were conducted on each subject, which easily caused fatigue and made subjects unable to keep fingers strictly stable. Consequently, a big force dispersion under a large indentation depth was observed in this study. It may be resulted from the squeeze on bone and a slight shift of the contact position during different experimental sessions. An optimized design of the experiment apparatus may improve subject’s comfort. Moreover, less consumption of data acquisition is desired upon no compromise of recognition accuracy. Besides, the probability-based haptics model may be further extended to a multi-point model or even a 3D “force-deformation surface” model. They are potential to be applied in a visual monitored environment for force measurement and prediction. Additionally, there are 6 coefficients in the haptics model need to be adjusted individually, although the model’s structure has been unified. It is expected to reduce the number of coefficients for model simplification and generalization.
Footnotes
Acknowledgments
The authors would like to acknowledge all the participants in the experiments and also the support from DREAM project of EU FP7-ICT (Grant No. 611391), Research Project of State Key Lab of Digital Manufacturing Equipment & Technology, China (Grant No. DMETKF2017003), National Natural Science Foundation of China (Grant No. 51575412, 51575338 and 5157540), and China Scholarship Council (Grant No. 201508060299).