Low variable rate stimulation strategy for cochlear implants using temporal cues and electrophysiological factors

Abstract

BACKGROUND:

The performance of a cochlear implant (CI), especially in conveying pitch depends on its electrical stimulation strategy.

OBJECTIVE:

The present study proposes a variable-rate stimulation algorithm which improves speech emotion perception by using temporal fine-structure cues and electrophysiological parameters of the patient.

METHODS:

This method is based on the coding of the phase information at the peak time intervals of the band-passed signals. The stimulation pulse is generated at the time of peak occurrence, which is able to excite the number of fibers with a discharge probability above a threshold. Calculating the discharge probability is based on the excitable fiber model and taking into account the biological characteristics of the patient, such as the fiber threshold and the distribution of remaining intact fibers.

RESULTS:

The results of the emotion detection test on selective reconstructed sentences from the Persian emotional speech database (Persian ESD) indicated that the listeners have been able to detect the emotion by an average of 83.82% using the proposed stimulation algorithm while it was 75% and 48.03% for the zero-crossing and the continuous interleaved sampling (CIS), respectively. Furthermore, the number of pulses compared to the zero-crossing and the CIS has decreased by 76.3% and 75.4%, respectively.

CONCLUSIONS:

In this paper, a stimulation method was proposed for cochlear implants by considering the patient’s biological parameters. It has been successful in transmitting speech emotion despite the reduction of stimulating pulses. This has some advantages such as reducing the interaction of current fields between electrodes during stimulation and reducing battery usage.

Keywords

Cochlear implant electrical stimulation excitable nerve fiber voice pitch frequency emotion detection

1. Introduction

The cochlear implant (CI) is known as the best treatment to restore hearing to people with severe to profound deafness [1, 2]. According to the statistical results released by the end of 2012, 324,000 people have already enjoyed the benefits of this technique worldwide [3]. However, they have difficulties in music perception, speaker recognition and prosody perception [4, 5].

A CI bypasses the hair cells and stimulates the intact auditory nerves directly by electrical impulses [6]. Cochlear auditory fibers stimulation conveys the messages to the brain. The pattern of stimulation determines the type and importance of these messages. Finally, the messages processing by the brain forms the sound perception in the patient. The performance of a CI is greatly dependent on the ability of its speech processor to extract and transmit the speech information. Generally, the speech processors use two categories of speech features in electrical stimulation: coarse features and fine features. Among these methods, those using the coarse features of spectral envelope such as $F_{0}/F_{2}$ [7], $F_{0}/F_{1}/F_{2}$ [8] and multi peak (MPEAK) [9], are inefficient due to errors in detecting these features, especially in noisy condition. On the other hand, methods which use temporal envelope features such as CIS [10] and spectral maxima sound processor (SMSP) [11], are often able to transmit speech content in an acceptable manner in silence, but they cannot usually get information related to voice pitch frequency, such as speaker recognition or prosody and emotion recognition. The efficiency of these algorithms is largely dependent on the stimulation rate. Most studies have found that increased stimulation rates are effective in improving speech intelligibility [12, 13]. It is claimed that this improvement is due to the fact that the higher stimulation rate is able to transmit a better representation of fine temporal modulation [14].

Investigations have shown that CI users need information beyond signal envelope cues for pitch perception, due to the low number of stimulus channels. This information is contained in the phase component of speech or on the other hand in the temporal fine structure [15]. Therefore, in some of the novel methods, usually the stimulation rate is several thousand pulses per second or it is variable and the time interval between pulses is not necessarily equal.

Nie et al. [16] proposed a frequency-amplitude-modulation encoding strategy that transformed the fast-varying fine structure into a slowly-varying FM signal to modulate the center frequency in each band. In another method, they used the interleaved pulses in the peaks of the time envelope after the frequency shift of the signal in each channel [17, 18]. Chen and Zhang [19] applied peak occurrence times in temporal fine structure extracted by a Hilbert transform. In another study [20, 21] they used zero-crossing times to represent the fine structure information. Liu et al. [22] used one-octave wavelet transform zero-crossing stimulation to encode more phase information. Sit et al. [23] proposed a bio-inspired asynchronous interleaved sampling by using a race-to-spike algorithm to encode the phase information. Zhang et al. [24] used a model of the peripheral auditory system to generate stochastic impulses more similar to the auditory neural firing.

The aim of this paper is to introduce a stimulation algorithm based on the phase coding at the time interval between the peaks of signal. The proposed stimulation algorithm tries to avoid the use of pulses with insignificant effect on auditory fibers by considering the patient’s biological conditions and the nonlinear behavior of auditory neurons. The performance of the proposed algorithm in Farsi emotional sentences identification is compared with the CIS and zero-crossing methods.

2. Method

The block diagram of the proposed strategy is shown in Fig. 1. After pre-processing, the audio signal is decomposed into N channels using a bank of band pass filters. The center frequencies of these filters have been distributed based on the Mel scale. Then, the envelope of each channel is calculated by rectifying and low-pass filtering. The obtained signals are compressed in amplitude to be within the narrow electric dynamic range.

The extracted amplitude is used in the amplitude modulation process. A peak detection modulus is involved in a parallel path for determining the peak points in each channel. The output of this modulus will enter the effective stimulation recognizing (ESR) block. The input of the ESR block is accepted as a stimulus pulse if it can produce a discharge probability above a threshold. Then, the generated pulses in each channel are modulated by the extracted envelope and the resulting pulse trains are sent to the stimulating electrodes.

In order to encode the temporal cues of the audio signal the peak times are used. The auditory nerve fibers (ANFs) tend to spike at or near the peak times of the stimulus which is known as the phase-locking phenomenon. Phase-locking is weak at higher frequencies than 5 kHz. The input of the ESR block is considered as a stimulation pulse if it could provide an ANF discharge probability above a threshold.

The discharge probability is calculated using the discharge estimator algorithm which was introduced previously [25] based on the excitable nerve fiber model of Bruce [26]. This algorithm is applicable even when the interval between pulses is not fixed. The model of Bruce was used since in this model, the neuron’s response to two consecutive pulses can be predicted by considering the refractory effect. Also, the membrane noise fluctuations are involved. The threshold determining approach is outlined below.

According to the literature [27], each inner hair cell (IHC) is connected with several nerve fibers. Based on microscopic studies, the number of hair cells and the ANFs in each section of the cochlea could be estimated. It was assumed that if the input of the ESR block could successfully trigger at least one fiber within one millimeter neighborhood, it is accepted as a stimulation pulse. In order to reduce the calculating time, the following criterion was applied. When an ANF discharge probability exceeds $\frac{N_{hk}}{N_{k}}$ , an impulse occurs.

$\displaystyle\textit{Impulse}_{i}(t)=\left\{\begin{array}[]{ll}1&p(t)\geqslant% \frac{N_{hk}}{N_{k}}\\ 0&\textit{otherwise}\\ \end{array}\right.i=1,2,\ldots,N_{hk}$ (1)

$N_{hk}$ and $N_{k}$ are the number of ANFs connected with an IHC in the $k$ th channel and the number of ANFs within one millimeter neighborhood of the $k$ th electrode, respectively.

$\displaystyle\textit{Impulse}(t)=\textit{OR}(\textit{Impulse}_{1}(t).\textit{% Impulse}_{2}(t)\ldots\textit{Impulse}_{N_{hk}}(t))$ (2)

If two impulses occur at the same time in two channels, the priority for pulse generation is applied to the lower one.

3. Experimental protocol

Emotion identification is chosen to evaluate the performance of the proposed algorithm in conveying pitch frequency since the pitch contours are different in various emotional sentences. Synthesized sentences which acoustically simulate the hearing process of implanted patients are presented to normal hearing listeners. The performance of the proposed algorithm is compared with the CIS and zero-crossing methods, which represent fixed rate and variable rate methods, respectively.

3.1 Signal processing

First, the signal is passed through a pre-emphasis filter. Then it is decomposed by 12 Butterworth band-pass filters with central frequencies of 100, 290, 524, 814, 1173, 1617, 2166, 2845, 3684, 4723, 6008 and 7596 Hz. The envelope extraction and compressing processes are the same for the three mentioned algorithms.

3.2 Speech synthesis

The reconstruction method is based on Vocoder synthesis. In this method, the same series of band-pass filters are used for synthesis which have been used in signal decomposition. The pulse train of each channel is convolved with the impulse response of the band-pass filter in its channel. The synthesized signal is formed by summing the reconstructed signal from all channels.

$\displaystyle\textit{re\_sig}(t)=\sum_{k=1}^{12}\ \textit{re\_sig}_{k}(t)$ (3) $\displaystyle\textit{re\_sig}_{k}(t)=\textit{pulse\_tr}_{k}(t)\ast h_{k}(t)$ (4)

where $h_{k}(t)$ is the impulse response of channel $k$ and $\textit{pulse\_tr}_{k}$ refers to the pulse train in $k$ th channel.

3.3 Database

A validated database of Persian emotional speech (Persian ESD) was used in the experiment [28]. This database contains a set of 90 validated novel Persian sentences classified into five basic emotional categories (anger, disgust, fear, happiness, and sadness), as well as a neutral category. The sentences are created using the simple Persian grammatical structure and were recorded in a professional recording studio in Berlin, Germany.

The sentences are spoken by two native speakers (one male and one female) in three categories. (1) congruent: the lexical content of the sentences is in accordance with the expressed emotions, (2) incongruent: neutral sentences are expressed in one of the mentioned emotions and (3) baseline: all sentences with different lexical contents are expressed in neutral.

Seventeen different emotional sentences from three categories were selected randomly for each stimulation algorithm. In this case, it is also possible to examine the effect of the concept of sentences in the emotion recognition. In order to have no response influence, the sentences were not the same for three algorithms. Also, all of the synthesized sentences were randomly located in the electronic questionnaire. The answer options were anger, sadness, happiness, neutral, none of them and lack of recognition. The questionnaire included three control synthesized sentences in a randomized order for detecting inconsistent answers. Subjects who marked different answers to more than one pair of control sentences were excluded from the study. Before questions, subjects listened to some original sentences to be familiar with different emotions. Participants were asked to listen to the voices in a quiet place and with a headset preferably. 44 normal-hearing Farsi speakers from 20 to 38 years old participated in this test.

3.4 Results

The percentages of response distribution were obtained for each sentence and then averaged for sentences with similar conditions in three existing categories. The results are shown in the Table 1. The correct recognitions are indicated in bold.

Table 1
Emotion identification for proposed stimulation strategy

Condition	Lexical	Emotion	Percentage of responses
	content		Neutral	Anger	Happiness	Sadness	None of	Lack of
							them	recognition
Congruent	Anger	Anger	0	100	0	0	0	0
	Happiness	Happiness	0	0	100	0	0	0
	Sadness	Sadness	0	0	0	100	0	0
	Neutral	Neutral	100	0	0	0	0	0
	None	None	8.82	8.82	5.88	20.58	47.05	8.82
Baseline	Anger	Neutral	64.7	17.64	0	11.76	5.88	0
	Happiness	Neutral	82.35	0	17.64	0	0	0
	Sadness	Neutral	88.23	0	0	5.88	5.88	0
	None	Neutral	94.11	0	0	5.88	0	0
Incongruent	Neutral	Anger	5.88	82.35	5.88	5.88	0	0
	Neutral	Happiness	0	0	88.23	0	5.88	5.88
	Neutral	Sadness	5.88	0	0	88.23	5.88	5.88
	Neutral	None	17.64	5.88	0	5.88	52.94	17.64

The phrase “none of them” is an emotion other than neutral, happy, angry and sad. For this section of the test, fear and disgust sentences from the database were used. The results reveal that for the congruent condition all emotions were recognized very accurately (100%). The poor results obtained in “none of them” can be attributed to the participants’ unfamiliarity with such sentences (47.05%). As expected, the obtained results in baseline and incongruent categories are weaker than for the congruent category. The analysis of incorrect answers shows that this could be due to the attention to lexical content of sentences in the baseline category and the absence of lexical cues in the incongruent section. The average of correct recognition for baseline and incongruent were 82.35% and 77.94%, respectively.

In Fig. 2, the correct rates of emotion identification of the proposed stimulation algorithm in congruent, incongruent and baseline categories are compared with two stimulation algorithms: Zero-crossing (stimulation with non-uniform time intervals) and CIS (stimulation at a constant rate of 5,000 pulses per second). Mean values and standard deviations show that, generally, the proposed algorithm and zero-crossing performed better than the CIS. The p-values obtained from t-tests with the Bonferroni correction showed that the differences between each pair of the three strategies in the congruent and baseline categories were not significant (all $p>$ 0.05/2 $=$ 0.025), but in the incongruent section, there was a significant difference between the CIS and two others. The obtained p-values were 1.6 $\times$ 10 ${}^{-2}$ and 9.7 $\times$ 10 ${}^{-4}$ for the zero-crossing and the proposed algorithm compared with the CIS (both of them $<$ 0.025).

Figure 3 shows the results obtained from the emotional sentences classification test. Three stimulation methods have been compared with t-tests. The obtained p-values were 3.9 $\times$ 10 ${}^{-1}$ and 7.2 $\times$ 10 ${}^{-3}$ $(p<$ 0.025) for the proposed algorithm compared with the zero-crossing and the CIS, respectively and 4.4 $\times$ 10 ${}^{-2}$ for the zero-crossing compared with the CIS.

Another important point in comparing these algorithms is the number of stimulation pulses. To investigate this factor, 17 identical sentences composed of 8 to 10 words, were considered for all three algorithms. The average number of pulses used for stimulation of each channel is shown in Fig. 4. The ratio of the average number of pulses used for the zero-crossing and CIS methods compared to the proposed methods is 4.23 and 4.06, respectively.

4. Discussion

The electrical current injected by the stimulated electrode changes the electrical potential in the volumetric conductor. Based on studies [29], the electrical potential around the stimulated electrode is calculated based on the distance of the stimulated electrode as well as the conductivity of the volumetric conductor as follows:

$\psi_{0}=\frac{I_{0}}{4.\pi.\sigma.r}$ (5)

$I_{0}$ , $\sigma$ and $r$ are the electrical current, electrical conductivity coefficient and the distance from stimulating electrode, respectively.

Thus the electrical potential drops around the electrode in bell-shape form [30]. Therefore, the use of band- pass filters in the vocoder method is common in audio signal simulation in many studies.

In order to compare the performance of the proposed stimulation algorithm with two other stimulation algorithms, some observations can be made. As can be seen in Fig. 2, the average of the correct answers in the incongruent category for both CIS and zero-crossing algorithms has decreased compared to the other two categories. This could be due to the fact that the correctness of responses in the congruent category may be affected by the lexical content of the sentences. Thus better results have been achieved in that section. The reduction in the results for the incongruent compared to the congruent category for the CIS and zero-crossing algorithms was 71% and 28.49%, respectively while it was 11.6% for the proposed algorithm. The differences in accuracy of the results in the congruent and incongruent sections are noticeable, especially for the CIS algorithm. In addition, the superiority of the baseline to the other two categories in the CIS algorithm suggests that this algorithm does not have the capability to convey the pitch frequency that is necessary to understand the speech emotion. Also, the results of t-tests with the Bonferroni correction showed a significant difference in the performance of the CIS algorithm compared to the proposed algorithm ( $p<$ 0.025).

In general, the proposed stimulation algorithm and the zero-crossing were more successful in conveying the speech emotion to the listeners. The results of the emotion detection test on selective reconstructed sentences from the Persian ESD database indicated that the listeners have been able to detect the emotion by an average of 83.82% and 75% using the proposed and the zero-crossing algorithms, respectively. While it was 48.03% for the CIS algorithm. In general, the proposed stimulation algorithm has performed better in transmitting the speaker’s feelings, or in other words, in conveying the pitch frequency Fig. 5a shows a part of pitch trajectories of an original sentence. Sections (b), (c) and (d) of this figure show the pitch trajectories of the vocoder-synthesized signal after processing by the proposed algorithm, zero-crossing and CIS, respectively. The extracted pitch trajectory of the processed signal by the proposed algorithmis significantly more consistent with the extracted pitch trajectory of the original sentence than the pitch trajectories obtained by the two other algorithms.

Reducing the number of pulses sent to the electrodes as can be seen in Fig. 4, is another advantage of the proposed stimulation algorithm. The number of pulses compared to the zero-crossing and the CIS has decreased by 76.3% and 75.4%, respectively. Reducing the number of pulses and consequently reducing injection flow to electrodes will cause less damage to neural tissues such as spiral ganglions and inner hair cells. Furthermore, it causes less electrode corrosion [31]. Also, sending fewer pulses to the electrodes has the benefits such as extending the life of the battery. Reducing the number of pulses with preservation of excitation quality can also reduce the interaction of current fields between electrodes during stimulation, which is one of the important issues in the performance of the cochlear implants.

5. Conclusion

In this paper, an algorithm was proposed for electrical stimulating in cochlear implants. This algorithm works based on coding of the phase information of the signal at the time interval between the peaks of signal. Also, the patient’s biological conditions such as the fiber threshold and the distribution of remaining intact fibers are considered in stimulation since the same electrical stimulation in patients with different electrophysiological conditions leads to different firing patterns of neurons. By considering the patient’s condition, at the peak occurrence times, pulses are sent to the electrodes if they are able to excite the fibers with a discharge probability above a threshold. Current reduction will eventually decrease possible neural damage in the long term. The emotion detection test performed by normal hearing listeners on reconstructed sentences from the Persian ESD database demonstrates that the proposed stimulation algorithm obtained better results than the zero-crossing and the CIS. Moreover, the number of pulses and thus the total current flowing through the electrodes were significantly lower than with the other two algorithms.

Using the peak occurrence times improves the transmission of temporal cues and applying a threshold criterion to choose the effective stimulations reduces the total number of electrical pulses.

Footnotes

Conflict of interest

None to report.

References

Loizou

. Introduction to cochlear implants. IEEE engineering in medicine and biology magazine: the quarterly magazine of the Engineering in Medicine & Biology Society. 1999; 18(1): 32-42.

Zeng

F-G

. Trends in cochlear implants. Trends in Amplification. 2004; 8(1): 1-34.

Nih

. Cochlear implants. National institute on deafness and other communication disorders. 2013; No 11-4798.

Vandali

Sucher

Tsang

McKay

Chew

McDermott

. Pitch ranking ability of cochlear implant recipients: A comparison of sound-processing strategies. The Journal of the Acoustical Society of America. 2005; 117(5): 3126-38.

McDermott

. Music perception with cochlear implants: a review. Trends in Amplification. 2004; 8(2): 49-82.

Loizou

. Mimicking the human ear. Signal Processing Magazine, IEEE. 1998; 15(5): 101-30.

Seligman

Patrick

Tong

Clark

Dowell

Crosby

. A signal processor for a multiple-electrode hearing prosthesis. Acta Oto-Laryngologica. 1984; 97(S413): 135-9.

Blamey

Dowell

Clark

Seligman

. Acoustic parameters measured by a formant-estimating speech processor for a multiple-channel cochlear implant. J Acoust Soc Am. 1987; 82(1): 38-47.

Loizou

Dorman

. On the number of channels needed to understand speech. The Journal of the Acoustical Society of America. 1999; 106(4): 2097-103.

10.

Wilson

Finley

Lawson

Wolford

Eddington

Rabinowitz

. Better speech recognition with cochlear implants. Nature. 1991; 352(6332): 236-8.

11.

McDermott

Mckay

Vandali

. A new portable sound processor for the University of Melbourne/Nucleus Limited multielectrode cochlear implant. The Journal of the Acoustical Society of America. 1992; 91(6): 3367-71.

12.

Frijns

Klop

WMC

Bonnet

Briaire

. Optimizing the number of electrodes with high-rate stimulation of the clarion CII cochlear implant. Acta Oto-Laryngologica. 2003; 123(2): 138-42.

13.

Loizou

Poroy

Dorman

. The effect of parametric variations of cochlear implant processors on speech understanding. The Journal of the Acoustical Society of America. 2000; 108(2): 790-802.

14.

Chen

Zhang

Y-T

. An integrate-and-fire-based auditory nerve model and its response to high-rate pulse train 2007 [1051-5].

15.

Luo

Q-J

. Enhancing Chinese tone recognition by manipulating amplitude envelope: implications for cochlear implants. The Journal of the Acoustical Society of America. 2004; 116(6): 3659-67.

16.

Nie

Stickney

Zeng

. Encoding frequency modulation to improve cochlear implant performance in noise. IEEE Transactions on Bio-Medical Engineering. 2005; 52(1): 64-73.

17.

Nie

Atlas

Rubinstein

. Single sideband encoder for music coding in cochlear implants. ICASSP; 2008: IEEE.

18.

Nie

Atlas

Rubinstein

. Harmonic coherent demodulation for improving sound coding in cochlear implants. ICASSP; 2010: IEEE.

19.

Chen

Zhang

. A novel temporal fine structure-based speech synthesis model for cochlear implant. Signal Processing. 2008; 88(11): 2693-9.

20.

Chen

Zhang

. Zerocrossing-based fine structure representation to convey Mandarin tonal information: a study on the noise effect. 30th IEEE EMBS; 2008 August; vancouver,canada.

21.

Chen

Zhang

. Zerocrossing-based nonuniform sampling to deliver low-frequency fine structure cue for cochlear implant. Digital Signal Processing. 2011; 21: 427-32.

22.

Liu

Wang

Liu

Zhang

. An improved speech coding strategy for cochlear implants. BMEI; 2010: IEEE.

23.

Sit

Simonson

Oxenham

Faltys

Sarpeshkar

. A Low-Power Asynchronous Interleaved Sampling Algorithm for Cochlear Implants That Encodes Envelope and Phase Information. IEEE Transactions On Biomedical Engineering. 2007; 54(1): 138-49.

24.

Zhang

X-J

Cao

Sun

W-Y

Zhao

H-M

Tao

. Improved Mandarin Chinese Perception for Cochlear Implants Using an Auditory Model based on Stochastic Stimulation. Engineering Letters. 2016; 24(1).

25.

Zarei

Sadjedi

. A new approach for speech synthesis in cochlear implant systems based on electrophysiological factors. Technology and Health Care. 2017; 25(2): 221-35.

26.

Bruce

Irlicht

White

O’Leary

Dynes

Javel

, et al. A stochastic model of the electrically stimulated auditory nerve: Pulse-train response. Biomedical Engineering, IEEE Transactions on. 1999; 46(6): 630-7.

27.

Spoendlin

Schrott

. Analysis of the human auditory nerve. Hearing Research. 1989; 43(1): 25-38.

28.

Keshtiari

Kuhlmann

Eslami

Klann-Delius

. Recognizing emotional speech in Persian: A validated database of Persian emotional speech (Persian ESD). Behavior Research Methods. 2015; 47(1): 275-94.

29.

Frijns

Mooij

Ten Kate

. A quantitative approach to modeling mammalian myelinated nerve fibers for electrical prosthesis design. IEEE Transactions on Biomedical Engineering. 1994; 41(6): 556-66.

30.

Sadjedi

Motamedi

Firoozabadi

. A new modified multi-electrode stimulation method for ECAP recording in cochlear implant. Engineering in Medicine and Biology Society, 2004 IEMBS’04 26th Annual International Conference of the IEEE; 2004: IEEE.

31.

Clark

. Cochlear implants: fundamentals and applications: Springer Science & Business Media; 2006.

Low variable rate stimulation strategy for cochlear implants using temporal cues and electrophysiological factors

Abstract

BACKGROUND:

OBJECTIVE:

METHODS:

RESULTS:

CONCLUSIONS:

Keywords

1. Introduction

2. Method

3.1 Signal processing

3.2 Speech synthesis

3.4 Results

Table 1 Emotion identification for proposed stimulation strategy

Footnotes

Conflict of interest

References

Table 1
Emotion identification for proposed stimulation strategy