It’s not what you say,but how you say it: Using artifact intelligence to detect deception in natural speech in an HR recruiting context

Specification of research objectives

“Everybody lies” is the famous tagline of Dr. Gregory House in the American TV series House, M.D. The series main character refuses to talk to his patients because he does not trust their assertions. Instead, he solely relies on objective measures like laboratory test results. Although the series exaggerates this concept for dramatic purposes, there is a grain of truth in it. Research has found that not telling the truth or making statements that are different than one’s inner belief is frequent in human interactions (Serota et al., 2022). Unfortunately, people are notoriously bad at detecting false statements and are typically able to identify them only half of the time (Bond Jr. and DePaulo 2006; George and Robb 2021). In a large-scale experiment with human participants, Bond Jr. and DePaulo (2006) found that people with no training in detecting false statements were able to correctly identify deception with an average rate of only around 54%. Similar results are presented by Levitan et al. (2020), who developed a game in which participants without training in detecting false statements had to judge whether someone is telling the truth based solely on audio files. The participants detected the accuracy of the recordings 49% of the time, which corresponds to mere chance. More dramatically, in a recruitment context, Suen and Hung (2023) find that interviewers detect false statements less than 20% of the time. These concerning research findings were among the motivations to develop an AI artifact that is able to detect deceptive statements significantly better than people.

Dishonesty: Deception and lies

Deception is defined as the “deliberate attempt to mislead others” (DePaulo et al., 2003: p. 74). This deliberate act includes all forms of intentional dishonesty, but not unintended falsehoods communicated by people who are mistaken (human error), self-deceived (cognitive dissonance), or experience challenges like the Dunning-Kruger effect, according to which people unknowingly overexaggerate their own knowledge and capabilities (Kruger and Dunning 1999).

Research on deception has a long tradition in various areas of scientific research, including psychology, medical research, sociology and organizational sciences. This breadth may be explained by the fact that deception, although usually perceived as negative, is a very common social behavior that the average person performs on a daily basis (DePaulo et al., 2003) As Zuckerman et al. (1981, p. 2) put it: “Lying is a common aspect of interpersonal relationships.”

Research shows that there are various facets of deception, including “real lies, white lies, and gray lies” (Fox 2018). Common across all of the facets, however, is the fact that the speaker is aware of the wrongdoing, which leads to negative emotions like guilt and fear (Ekman 2004; Whissell 2023), that is, the “bad gut feeling” commonly felt when lying.

The role of emotions in deception detection

Not speaking the truth is a highly emotional issue for human beings. Emotions are complex constructs of human feelings (Fox 2018). In broad terms, one could say that “the term ‘emotion’ exemplifies [an] ‘umbrella’ concept that includes affective, cognitive, behavioral, expressive and physiological changes” (Tyng et al., 2017: p. 3). Thus, “an emotion is a complex state that combines feelings, thoughts, and behavior and is people’s psychophysiological reactions to internal or external stimuli” (Zhang et al., 2020: p. 104).

Shu et al. (2018) discuss several models of basic emotions and complex emotions containing mixed facets. These complex emotions remain mysterious and the available concepts remain highly theoretical. Ekman (2004) identifies lying as a complex emotional construct made up of several basic emotions, predominantly fear, guilt, and delight, while DePaulo et al. (2003) found evidence that guilt, shame, and fear are amongst the strongest basic emotions experienced by someone who is lying.

Emotions arise spontaneously rather than through conscious effort (Shu et al., 2018) and result in physiological reactions such as increased heart rate, sweating, and dilation or constriction of blood vessels (i.e., blushing and paling) (Soleymani et al., 2015; Zhang et al., 2020). Such physical reactions, including heartbeat, skin reactions, blood flow, muscle tensions, facial expressions, or changes in voice, can be measured using technical equipment (Shu et al., 2018). The polygraph, or lie detector, measures such bodily reactions (Grubin and Madsen 2005) that people exhibit when they are lying (Proverbio et al., 2013). However, the polygraph use is declining because the results are often inaccurate due to bias, provide incorrect measurements and other deficiencies, such as the need for trained personnel to administer the test and the difficulty of interpreting the results (Katz 1983).

Recognizing emotions with artificial intelligence

Wherever someone experiences measurable physiological reactions, an AI artifact can be developed to interpret those. Research has shown that emotions can be detected through visual and/or auditory modalities (Fox 2018; Pan et al., 2015) and by applying machine learning techniques to the data collected (Yang et al., 2023; Zhang et al., 2020). A popular approach to detecting emotions is by analyzing facial expressions (Adolphs 2002), specifically micro-expressions (Matsumoto and Hwang 2018). For example, Tivatansakul et al. (2014) designed a web-based system that detected six basic emotions based on facial expressions with 86% accuracy. Krishnamurthy et al. (2018) combined video, audio, verbal transcripts, and micro-expression features into a multimodal neural model and report 96% accuracy in detecting deception. They tested their model on the real-life deception detection (RLDD) database (Pérez-Rosas et al., 2015) a publicly available dataset of 121 video clips from courtroom trials. Using the same dataset, Mathur and Matarić (2020) applied unimodal support vector machines along with multimodal fusion methods to identify deceptions in real-life situations and achieved 91% accuracy. Bhamare et al. (2020) used a deep neural network on an undisclosed dataset of 323 interviews. By combining analysis of speech and micro-expressions, they report to detect deception with 98% accuracy. However, their results were highly dependent on the facial features: “When interviewees concealed their eyes with sunglasses, the system failed as was expected” Bhamare et al. (2020, p. 147). Karpova et al. (2020) experimented with 93 volunteers who purposefully lied and tried to hide this from a polytrophic test. Using video analysis and eye-tracking techniques, the authors trained an end-to-end convolutional neural network. Their best model achieved a mean balanced accuracy of 64% in discerning truth from deception. On a dataset with 180 videos from 88 politicians, Kamboj et al. (2021) detected political deception from audio, text and facial features by utilizing a decision tree with an accuracy of up to 69%. Electroencephalogram (EEG) signals also provide the currently best known indications of the emotional state of individuals (Proverbio et al., 2013). Zhang et al. (2020) reported that EEG electrodes applied to the frontal lobe of a subject revealed a classification accuracy of more than 90% across different emotional states. Karnati et al. (2022) built a deep neural network with audio, visual and EEG features on three multimodal datasets. The datasets are publicly available and were technically augmented to provide a larger dataset. Through the combination of three modalities, the authors claim up to 98% emotional state identification accuracy. However, using EEG headgear for signal measurement is inconvenient for the individual and it takes considerable time to correctly measure the multichannel signals, which makes it unsuitable for the common business environment.

Emotion detection based on speech analysis

If an intrusive approach like a polygraph or an EEG is not appropriate for the situation and no video recordings sufficiently high in quality to analyze micro-expressions are available, audio analysis is an appropriate alternative, as suggested by Scherer (1995). Early work in this area by Amir and Ron (1998) provided an automated classification algorithm to identify emotions in speech but their work was limited by the technological capabilities at that time. As technology improved, numerous innovative approaches to recognizing emotions based on audio speech analysis have emerged. Zhao et al. (2019), for example, were able to distinguish several basic emotions with a test accuracy of about 96% using speech analysis. This indicates that with increasing computational abilities, analysis of the human voice is a promising route to detect even the complex emotional status of a speaker.

Emotions in spoken language are recognizable based on acoustic features including amplitude, pitch, and formants (Demircan and Kahramanlı 2014; Slimi et al., 2020). A prevalent approach involves the extraction of spectral features, namely the mel-frequency cepstral coefficients (MFCCs) first introduced by Davis and Mermelstein (1980). MFCCs have been shown to be particularly promising and computationally efficient for recognizing patterns in speech (Dave 2013; Fraser et al., 2016; Gupta et al., 2018; Logan 2000). Additionally, they are commonly used in speech emotion recognition (SER) as the basis for analyzing an individual’s emotions (Kishore and Satish 2013; Lalitha et al., 2015; Pakyurek et al., 2020).

Hidden Markov models (HMM) have been used for speech emotion recognition for more than two decades. Utilizing this method, Schuller et al. (2003) identified seven emotional states with 78% accuracy. More recent approaches apply deep learning (DL) methods, including recurrent neural networks (RNNs) (Trigeorgis et al., 2016) and combinations of convolutional neural networks (CNNs) and RNNs (Soleymani et al., 2015; Zhao et al., 2019). Based on log-mel spectrograms, Zhao et al. (2019) identified seven emotional states with an average test accuracy of around 96%. Beyond spectrogram features like MFCCs, some algorithms include contextual features to increase accuracy (Fayek et al., 2017; Trigeorgis et al., 2016). Taking this approach, Fayek et al. (2017) distinguish five emotional states with 65% accuracy. While the SER systems focus on a single person at a time, Majumder et al. (2019) analyze the conversations of a set of individuals and use this data to classify emotions. Their approach can classify six emotions with an average accuracy of 64%.

Automated detection of deception from speech

Although speech-based emotion analyses using MFCCs have great potential to serve as lie detectors (Chamoli et al., 2017), published research on automated detection of deception remains scarce. Several publications present artifacts but do not share sufficient information regarding the data set they used or they use a dataset which did not correctly record the emotion of a lying person because the data collection experiment was set up incorrectly. Fernandes and Ullah (2021), for example, trained several models on various spectral and cepstral features from speech, including MFCC features, and reached an accuracy up to 100%. However, their dataset consists of only one person providing true and false statements at three different times of the day. Nasri et al. (2016) analyzed data of true and false statements expressed by 40 volunteers and reported between 85% and 88% by using 13 MFCC features with a support vector machine. However, the study fails to explain how the participants were encouraged to lie, which makes it difficult to assess the validity of the results.

There are, however, many examples of more robust and better documented studies. Mihalache and Burileanu (2022) utilized neural network architectures for deceptive speech detection. Their model achieved 64% accuracy when tested on the RLDD database. However, the audio recordings in this dataset come from only 56 unique speakers, limiting its diversity. Fu and Lei (2020) proposed a model tested on the Columbia-SRI-Colorado (CSC) corpus. The CSC corpus consists of approximately 7.5 hours of speech data from 32 unique speakers, including 54% deceptive and 46% true speech acts (Hirschberg et al., 2005). Their model achieved 68% accuracy. However, since the CSC corpus is a professional database, it may only partially represent the diversity of real-world speech patterns. Fan et al. (2015) developed a method for detecting deception in spoken Chinese, achieving 64% accuracy on a corpus containing around 9.4 hours of speech data from 30 unique native Chinese speakers, each providing a truthful and deceptive statement. Fu et al. (2023) developed a semi-supervised deception detection algorithm for speech that achieved 69% accuracy. The dataset utilized in the study contains 582 true and 521 deceptive speech samples from 40 speakers (28 males and 12 females). Bareeda et al. (2021) proposed a model that achieved 81% accuracy in detecting lies on the RLDD dataset. The methodology extracts MFCC features and a support vector machine classifier. More recently, Snehalakshmi et al. (2024) presented a platform for lie detection in interviews incorporating a stress measure from MFCC features, deception detection from facial features and a plagiarism checker. With their combined approach, they reached 72.2% accuracy. Because applicants are clearly visible in the videos, they state that “ethical concerns and technology-related issues need to be overcome” in order to apply their solution.

This review of published results shows that a robust algorithm that provides data privacy, uses solely natural speech (i.e., can be used in phone conversations) and delivers consistently high accuracy levels to detect false statements, has not yet been published in publicly available resources.

Use case

In DSR, use cases are widely used to capture the objective-centered requirements towards an artifact to be developed (Engel et al., 2024). Use cases are a proven and robust tool to ensure all stakeholders involved in the development of the artifact share the same understanding, especially developers and users (Dutoit and Paech 2002). To anchor our research in practice, we chose the corporate recruiting process context. Recruiting is an area where innovative technical tools are often deployed to gain efficiency and raise decision quality (Fleiß et al., 2024) and where AI-enabled tools are increasingly used (Gao et al., 2025; Purohit and Banerjee 2025). The research team used HR textbooks and practitioner-oriented literature (e.g., Bogen and Rieke 2018) to develop the use case.

The company posts a job offering that individuals can apply for using an online tool on the corporate website. The applicants provided the required information in a structured format. The first computational stage in the process is an automated check for defined formalities like minimum GPA, university degree, and years of relevant work experience. Candidates passing this formal check are invited to take a standardized online test focusing on mathematical skills, abstract reasoning, and other skills customized to the respective position. These tests are evaluated automatically and successful candidates are invited to a next stage, which is new.

The applicant is asked to have a phone or video call with a company representative. In this conversation—without having time to prepare, which ensures free speech and preempts pre-formulated statements—the applicant is first asked to present the highlights of her/his résumé, to get the applicant talking and counter initial nervousness. After a few minutes, the recruiter then asks the applicant targeted and detailed questions that directly address an area of specific importance for the position, such as how they handled severe team conflicts in the past or how they solved a specific complex technical issue.

During this second phase, the algorithm analyzes the voice of the presenter and informs the company representative when it detects potential deception using a green/red gauge on the screen. Based on this information, the representative can enquire about questionable statements in more detail. Being alerted to potentially deceptive statements enables the representative to garner better information about areas of important for the position and advance people further along the application process more selectively.

Focus group workshop

To derive design requirements and formulate the objectives of the artifact, we conducted a focus group workshop with HR experts, using the initial use case presented above as starting input. To limit the complexity, we chose a homogeneous group of companies and invited experienced managers from six medium-sized enterprises (between 150 and 5,000 employees) in the manufacturing and software production industry. All participants are currently responsible for recruiting new personnel and have at least 5 years of experience in interviewing applicants (Table 1).

OPEN IN VIEWER

Table 1.

Description of workshop participants (experts).

Code	Role	Type of business	Recruiting mainly
Expert 1	CEO	Software provider	Software developers
Expert 2	CTO	Software provider	Software developers
Expert 3	Lawyer (labor law)	Manufacturing	All roles
Expert 4	Head of HR	Manufacturing	Managers
Expert 5	Recruitment specialist	Software provider	Managers
Expert 6	HR team leader	Manufacturing	Software developers

Participants agreed unanimously that such an artifact would enhance their decision quality and save costs by not inviting applicants to physical interviews who lacked key capabilities. The panel confirmed the use case and deemed it feasible, but considered it unsuitable for recruiting senior managers due to the delicate hiring process for top management personnel. Thus, the use case focuses on the recruitment process for IT personnel in junior positions, where the need for personnel and the costs associated with failed hires are high. In addition, extant research shows that people applying for technical positions are specifically prone to make deceptive statements on their job applications (Weiss and Feldman 2006).

All participating companies already use online recruitment instruments, mainly automated checking mechanisms such as filters and minimum criteria hurdles, and automated standardized tests, usually of math, abstract reasoning, and analytical skills. Applicants who pass the automated filters and tests are invited for physical interviews. Our workshop participants no longer use multi-day assessment centers.

Solution requirements

In a moderated discussion, the experts identified their requirements for an artifact that could support their recruitment work. To ensure a broad spectrum of input, the project team provided insights from literature on deceptive behavior in job interviews (e.g., Bill et al., 2020; Bill et al., 2024; Melchers et al., 2020; Renier et al., 2021) and the use of AI tools to detect deception (e.g., Constâncio et al., 2023; Li et al., 2021; Van Den Broek et al., 2021; Vrij et al., 2022; Vrij et al., 2023).

The requirements identified during the discussion were collected and clustered into groups:

• Use environment: The artifact needs to work with voice recordings of normal quality, such as phone calls or video conferences from home, and without specific recording equipment. No additional technical barrier for the interviewee.

Ethical considerations and data privacy

The workshop participants discussed ethical considerations and data privacy concerns extensively. Every company wants to respect the explicit and implicit data privacy rights of applicants, especially in Germany (Prince and Wallsten 2022). Consequently, the design principles of the proposed artifact comply fully with the European General Data Protection Regulation (GDPR) (EU 2016) and the EU AI Act (EU 2024).

Beyond legal and regulatory compliance, the research team discussed ethical considerations extensively with academic experts on ethics. There was unanimous agreement that deploying the proposed artifact is ethically permissible if (a) the applicant is informed ahead of time their statements will be checked for validity and that statistical measures/AI algorithms will be used in this process, and (b) the applicant has the option to opt out without negative consequences.

The artifact is designed to prevent deception, which is unethical behavior. Therefore, a fraudulent person has no legitimate right not to be detected. After all, companies have been using various mechanisms to avoid deception (see, e.g., Vrij 2008), including fraudulent detection in job interviews (Ferran and Storck 1997; Mahbub and Pardede 2018) for decades.

With regard to data privacy, the artifact (a) does not store additional personal data and (b) has no semantic capabilities, that is, the algorithm can and does not process the meaning of the words the applicant speaks. After the voice stream has been analyzed, the artifact returns a single value, an assessment of the truthfulness of the voice stream. The voice stream does not need to be saved and can be discarded immediately after analysis. All further information is already stored by the company to process the application and as such subject to the data privacy agreement between the involved parties, typically acknowledged at the beginning of the application process. If the company wished to store the voice patterns for post-analysis use, it would need to explicate that in the data privacy agreement.

Statement of solution objectives

In summary, the artifact needs to fulfill the following objectives which will be the basis for its evaluation (see section 10): (1) No special equipment, (2) Data privacy, (3) Legal and ethical compliance, (4) Free of bias, (5) Decision support only, (6) Superior detection accuracy, (7) Superior performance towards available solutions. Please note that objective 7 was not brought up in the workshop but derives from the DSR method.

Data collection process

To generate these very specific voice recordings we chose the setting of a debate club. We distributed flyers in local universities, social centers and on various social media channels inviting volunteers to participate in our study, which we referred to simply as “an experiment,” offering each participant a 10€ Amazon voucher as compensation.

The experiment followed typical debate club procedures: The participants were randomly grouped into pairs. Each pair was randomly assigned one of four topics (see 6.3 below), and whether the individual should argue pro or contra. We did not provide the participants with any information about the study’s objective beforehand to avoid possible bias. Notably, none of the participants asked about the underlying intention of the experiment.

The participants had 30 minutes to prepare their arguments and were instructed not to use the internet in their preparation. Participants were permitted to write down bullet point notes but not fully formulated statements. Each participant was then asked to argue their assigned position (pro or contra) on their topic in approximately 2 to 3 minutes of free speech.

After the debate, the research team provided each participant with feedback on their rhetoric performance/argumentation to help them improve their soft skills. Finally, participants were asked to provide suggestions on improving the operational handling of the experiment. No further interaction with the research team occurred to reduce social desirability response bias. Immediately after the feedback discussion, participants received an automated email asking them to complete a brief online survey of basic demographic data and whether they had argued for or against their true conviction about the debate topic. All participants completed the survey fully and within 1 hour of receiving it.

We collected this data in Germany in 2022 and 2023, recording the statements either personally or via videoconference. To ensure data privacy, only the audio was recorded and used. All participants were informed about the data privacy precautions (EU-GDPR compliant) and signed a consent form authorizing the use of the audio recording for research purposes before the experiment started.

Demographics of the participants

We conducted 46 debates, thus collecting 92 audio recordings. Due to audio problems, such as breaks-ups, transmission delays and interference due to bad internet connection, 12 recordings had to be removed, resulting in 80 individual audio files.

The participants included 41 employed non-students, 31 undergraduate and graduate students, and 8 high school students. All debates were conducted in German. 35 were women and 45 were men, and their ages ranged from 16 to 60 years (Ø = 27.15 years; median = 26 years). See Appendix for additional demographic details.

Topics

We deliberately chose highly polarizing topics for the discussions:

• Topic 1: Should the death penalty with public executions be reintroduced in Germany?

These polarizing topics were selected to increase the likelihood that participants would have a clear and unambiguous opinion. According to participants’ responses to the email questionnaire, the speakers’ true convictions were aligned with the research team’s expectations: none of the participants stated a true conviction in favor of reintroducing the death penalty, introducing cost-covering tuition fees, legalizing hard drugs or banning restaurant chains that serve unhealthy fast food. However, social desirability response bias may have influenced these stated convictions.

The distribution of the topics and the respective pro and contra positions the participants were assigned is provided in Table 2.

OPEN IN VIEWER

Table 2.

Allocation of topics and assigned pro/contra positions.

​	Topic 1	Topic 2	Topic 3	Topic 4	Sum
	27	13	24	16	80
Pro	15	8	16	6	45
Contra	12	5	8	10	35

Table 3 shows for each topic how many participants argued for or against their conviction.

OPEN IN VIEWER

Table 3.

Inner conviction of the participants per topic.

​	Topic 1	Topic 2	Topic 3	Topic 4	Sum
Argument reflected the speaker’s conviction	13	6	12	9	40
Argument did not reflect the speaker’s conviction	14	7	12	7	40

Feature descriptions

Based on the literature on emotion detection and automatic deception detection from speech, we pre-selected audio features that were used successfully in published studies. To train the models to classify true conviction, first the mel-frequency cepstral coefficients (MFCCs) (Davis and Mermelstein 1980) were extracted from the audio recordings. The MFCCs were derived from a time-frequency representation of the audio signal with the frequencies from 500 to 6000 Hz on a mel scale. Using MFCCs has two advantages: (1) it provides computational efficiency for recognizing patterns in speech (Dave 2013; Fraser et al., 2016; Gupta et al., 2018; Logan 2000), and (2) it generates strong results for audio files of inferior quality (Tivatansakul et al., 2014). In the next step, additional audio features were extracted, including the zero-crossing rate (ZCR), which is used to differentiate between voiced and unvoiced speech segments. ZCR reflects the number of times the signal crosses the zero axis in a given time window (Shete et al., 2014). In speech signals, voiced segments are characterized by periodic waveforms produced by the vibration of vocal cords. These waveforms result in a relatively low ZCR. In contrast, unvoiced segments, for example, those produced by airflow noise, are characterized by random and non-periodic waveforms, which result in a comparatively high ZCR (Bachu et al., 2010). Next, the spectral roll-off (McFee et al., 2015) was extracted from the audio files, which provides information about the spectral energy distribution in speech. It is used to characterize the timbre of speech tones, with brighter tones yielding higher values (Giannakopoulos et al., 2006). A further extracted feature includes statistics of the initial peaks of the sounds done during the recorded speeches. This way, variations in speech pauses can be detected. As basis-emotion recognition systems also consider pitch features, we furthermore extracted chromagrams corresponding to the twelve tone chromatic pitch classes (McFee et al., 2015). Finally, the fundamental frequency, which represents the lowest frequency in a periodic waveform, was assessed. It is a key characteristic of the speaker’s voice and can provide important information about the speaker’s emotional state (Hirst and De Looze 2021).

AI model training

The training procedure follows the approach of a support vector machine (SVM) (Hearst et al., 1998). An SVM classifier puts a separating hyperplane into a space so data points of one class are on one side of the hyperplane and the data points of the other class on the other side. SVM was chosen for three reasons: (1) The available dataset is relatively small and SVMs with its maximum-margin hyperplanes are better suited to achieve good results here; (2) SVMs are robust against noise and outliers, which frequently occur in audio data; and (3) SVMs can handle nonlinearly separable data, which is often the case in audio processing (Boateng and Kowatsch 2020). Additionally, SVM models are relatively easy to interpret and understand, which makes it easier to understand why a particular decision was made and identify improvement areas.

The following hyperparameters were applied to the training of the SVM. The radial basis function (RBF) (Hearst et al., 1998) kernel was used to implement the SVM for binary audio classification. The RBF kernel is a popular choice for nonlinear classification problems because it can map input data into high-dimensional feature spaces where linear classifiers can effectively separate the data, which allows the SVM to capture complex relationships between input variables and output class labels. The RBF kernel uses a radial basis function to measure the similarity between input data, and the kernel width hyperparameter, gamma, controls how far the influences of training examples reach. When training the SVM, the regularization hyperparameter C is essential in controlling the tradeoff between overfitting and underfitting. A higher value of C leads to a narrow margin, which may result in overfitting, and vice versa. As an overfitted model would memorize the training data and, thus, perform poorly on the test data, we want to avoid it. Hence, different values for the hyperparameters were tested to determine the optimal performance for the binary audio classification task.

Performance metrics

The following performance metrics are applied to measure each model’s efficiency and reliability:

(1) True Positive (TP): How often the model correctly predicted that a speaker represented the position based on the debate stance.

These metrics alone are not sufficient to fully validate the results. For example, a high TP rate can be achieved when there is an extreme imbalance in the ratio between recordings of people whose convictions were not represented and speech data of people whose convictions were represented. To avoid this issue, a balanced ratio of input data is required, as well as a weighted consideration of the TP and TN rates (Aldwairi et al., 2018).

To compare the models’ performance, we applied the following metrics (Saxe and Hillary, 2018). Accuracy reflects the correctly classified proportion of a model composed of all data points. It is calculated by dividing the count of TP and TN by the summed count of TP, TN, FP, and FN. Precision reflects the share of TP-classified samples in the total number of samples predicted as positive (TP + FP). Sensitivity, also referred to as true-positive rate, demonstrates a model’s predictive power in terms of the proportion of correctly identified TPs to the total number of actual-positive observations (TP + FN). The F ₁ -score complements the accuracy measures because it balances unequal class distributions within a training dataset by calculating the harmonic mean between precision and sensitivity.

Model set 1: 40 MFCC features

All models were trained on 40 MFCC features with varying gamma and C hyperparameters, see Table 4. The best model trained, M1-40-E, achieved 61.9% accuracy across all repeated five folds. At 57.9% sensitivity, the model only predicts about 58 of 100 out of the speech events, where the speakers represent their positions, correctly as positive (TP) and 42 of 100 speeches incorrectly as negative (FN).

OPEN IN VIEWER

Table 4.

Models using 40 MFCC features.

Model	Gamma	C	Ø Accuracy	Ø Sensitivity	Ø Precision	Ø F1-Score
M1-40-E	0.0075	100	61.9%	57.9%	59.5%	58.7%
M2-40-E	0.01	100	61.7%	57.3%	59.6%	58.5%
M3-40-E	0.01	10	61.4%	55.0%	60.2%	57.4%
M4-40-E	0.00025	100	70.0%	54.1%	60.0%	56.9%
M5-40-E	0.005	100	60.9%	57.8%	58.3%	58.1%

The non-impressive results show that utilizing many MFCC features does not produce satisfactory predictive performance.

Model set 2: 13 MFCC features

To improve performance, we assessed the number and composition of the MFCC features utilized. We started by including only the first 13 MFCC features in the training of the models, as this is a well-established method in speech pattern detection (Park et al., 2014). As shown in Table 5, the best performing model (M1-13-E) achieves an overall accuracy of 62.3%, performing a little better than M1-40-E.

OPEN IN VIEWER

Table 5.

Models using 13 MFCC features.

Model	Gamma	C	Ø Accuracy	Ø Sensitivity	Ø Precision	Ø F1-Score
M1-13-E	0.0075	100	62.3%	53.7%	61.6%	57.3%
M2-13-E	0.01	100	61.9%	52.8%	61.8%	56.9%
M3-13-E	0.005	100	61.6%	49.5%	61.6%	54.9%
M4-13-E	0.0005	100	60.8%	46.7%	61.3%	53.0%
M5-13-E	0.1	3	60.8%	48.5%	60.6%	53.9%

Model set 3: Three MFCC features + acoustic features

To improve the performance of the models, numerous tests with different combinations of MFCC features and applied additional acoustic features like, for example, ZCR, spectral roll-off, or chromagrams were conducted. This led to significantly better results. The best accuracy was achieved by combining a reduced set of only three MFCC features with nine additional acoustic features. The best model trained, M1-6-C, achieves 82.4% accuracy (see Table 6). Moreover, 79.7% out of the true positives (conviction of a speaker is represented) will be correctly predicted as positive and 20.3% falsely as negative (conviction of a speaker is not represented) according to its sensitivity. Furthermore, when the model is predicting positive, then 84.0% (see precision) are actual positive and 16% negative. The complete list of features utilized by the best model is provided in the appendix.

OPEN IN VIEWER

Table 6.

Models using three MFCC and nine additional acoustic features.

Model	Gamma	C	Ø Accuracy	Ø Sensitivity	Ø Precision	Ø F1-Score
M1-6-C	0.0075	100	82.4%	79.7%	84.0%	81.8%
M2-6-C	0.0025	1000	82.2%	78.4%	84.1%	81.2%
M3-6-C	0.01	100	81.8%	78.6%	83.1%	80.8%
M4-6-C	0.005	100	81.7%	78.9%	82.9%	80.8%
M5-6-C	0.005	1000	81.0%	77.5%	82.2%	79.8%

Bias of the artifact

The AI artifact was trained using data generated from volunteers who participated in a debate club experiment. Although the demographics of the participants (see Section 6.2) indicate a heterogeneous mix, testing the artifact for potential bias poses an important task in developing AI models (Renier et al., 2021). The following tests were conducted to test for bias in the training data set:

Perceived usefulness of the artifact

To demonstrate the usefulness of the developed artifact, the HR professionals who provided feedback in the initial workshop were consulted again. The capabilities of the artifact were demonstrated and potential practical applications and hindering issues were discussed. In summary, all experts agreed that the artifact is useful. The more applications a company receives, the more likely they are to apply AI algorithms to help with the recruitment process.

Although the feedback was very positive, some issues were also mentioned.

• For more technical positions, like, for example, software developers, an experts said they would need to have specifically designed very deep/detailed test cases to see “whether the person really has the capabilities they claim to have” (Expert 6).

However, although both points are relevant in a recruiting context, they are not within the scope of the proposed artifact.

One of the practitioners consulted, a lawyer specialized in labor law (Expert 3), also commented on the legal implications. He stated that he sees no legal roadblocks for deploying such an AI artifact if the applicant informed about the deployment of mathematical/statistical analysis of their voice transparently and ahead of time, and a clause is added to the data privacy agreement that is part of every interaction with applicants. He expressed hope that widespread deployment of such algorithms could reduce the number of applications containing false statements. He also suggested that adapting the recruitment process in this way may need the approval of the works council.

Design principles

The aim of the DSR method is to ensure that the findings will not only apply to one specific situation but also have a certain level of generalizability. Developing artifacts that benefit IS researchers and generate value for practitioners—one of the core missions of IS research (Iivari 2020)—requires design principles to be identified, developed, and communicated (Gregor and Hevner 2013). Design principles are “prescriptive statements that indicate how to do something to achieve a goal” (Gregor et al., 2020) and collectively advance the body of knowledge of our discipline by serving as guiding principles for future developments (Nguyen et al., 2021).

The design principles presented in the following are based on the solution requirements elicited in the focus group workshop (see Section 5.2). These were challenged by the project team during the development phase of the artifact and discussed and finalized with the experts during the demonstration of the artifact.

The AI artifact itself is a neutral technology which cannot abide by ethical and legal guidelines and regulations. The responsibility to use the artifact in a legal and ethical manner remains with the user. However, the artifact is designed to ensure data privacy in the sense that it has a minimal data design (i.e., only requests the data it really needs) and does not need to store collected data. This makes it technically possible to operate within the limits of the EU-GDPR. Still, the responsibility to obtain consent before the interview remains with the user—which is already current standard practice in the corporate environments. From an ethical perspective, the artifact is not designed to make a decision, but rather only to provide an assessment. The actual decision remains with the human being. This is also underscored by clearly communicating the ∼20% margin of error in assessments. Here again, it remains the responsibility of the user to inform the interviewee transparently and ahead of time and offer the option to opt out without consequences. Finally, the requirement to perform better than a human being was not listed as a design principle because it is an innately indispensable functional requirement, a conditio sine qua non, that is, if the AI artifact does not perform better than HR professionals, it will not be used anyway.

The following design principles may serve as generalized guidelines for the future development of speech-enabled AI artifacts in sensitive business processes (Table 7):

OPEN IN VIEWER

Table 7.

Design principles.

Design Principle	Description of the Design Principle
Limited audio quality	The artifact shall not require special technical equipment; it shall be able to analyze data recorded on standard phone calls or standard video conference equipment for home/private users.
Data privacy	To ensure compliance with common data privacy regulations, the artifact shall adopt a privacy-by-design approach. In other words, it shall not store biometrical information that identifies an individual and shall not request more data than absolutely necessary to conduct the assessment. Whenever data is needed, it shall not be stored in a non-anonymous way after the assessment is finished.
Legal and ethical compliance	The artifact shall not violate legal or ethical regulations. Potential areas of conflict (e.g., decision-support vs. decision-making) shall be explicated by the developers.
Free of bias	The artifact shall be tested for inherent bias before deployment and regularly during its deployment to avoid bias towards sub-groups of interviewees.

Summary of demonstration

In conclusion, the HR professionals were all interested in the technology and see potential to use it in their own recruitment process to help the recruiters to sort out deceptive applications early in the process. The experts were satisfied with the accuracy of the artifact and did not identify any major hindering barriers. The feedback received from the demonstrations was used to formulate generalizable design principles.

Consequently, the demonstration phase was concluded successfully by confirming the usefulness of the artifact to solve the stated solution objective in the selected use case.

Contribution to design science research

Contribution to behavioral research

This novel AI-enabled artifact enhances organizational decision making by enabling experts and managers to overcome a typical human deficiency: our weak ability to detect deception. The following arguments draw upon extant research examining the emergent, multifaceted collaboration of human actors and AI systems (Baptista et al., 2020; Benbya et al., 2020b; Rai et al., 2019). The aim is to understand how novel AI tools can potentially affect organizational decision making in high-stakes contexts. In this vein, we seek to advance a more nuanced understanding of how AI-enabled technologies are being used to reconfigure organizational work practices and the implications for actors whose roles and activities are thereby transformed.

Contribution to practice

Applicants providing false statements on their résumés seem to be the rule rather than the exception (Schenz and Zeitung 2018). This was confirmed in our expert workshops: “We often have the problem, especially when recruiting for junior positions, that the capabilities stated in the CV do not match the experiences the applicants actually have” (Expert 4). This poses serious issues in terms of money and time lost: “We recruit people on a global basis, mainly in southern and eastern Europe, but also from India and Asia. Flying those people in for face-to-face interviews is costly, let alone the time it takes our staff to conduct the interviews. We estimate that each of these interviews costs us between €5,000 and €10,000” (Expert 1). Having an automated mechanism to reduce the number of interviewees who advance to the live interview stage of the recruitment process by providing false statements has great practical value.

Another area of potential application of the artifact presented is in the service industry. GDV, the German insurance association, estimates the damage from insurance fraud at 6 bn EUR annually (GDV 2024). GDV has determined that ten percent of insurance claims has a fraudulent component and around 5% of customers of member insurance companies have submitted at least one fraudulent claim (GDV 2020; GDV 2024). Applying AI techniques to detect whether a caller is speaking to her/his conviction when making a claim may not completely solve these issues, but it could help alert the claim agent to the need for more intensive questioning and give the caller the opportunity to correct their statements and avoid submitting a fraudulent claim.

It is an open secret that customers’ statements at helpdesks or the customer service desk do not always fully reflect the truth (Alton 2017). It would save time and resources if an algorithm could detect whether the customer truly rebooted their computer before they called the helpdesk, or whether the customer is being knowingly deceitful about a product malfunction. Such an algorithm could also help verify whether and when a product was really sent to the support team, resulting in productivity gains and a reduction of operational losses.

There are many potential applications in medical and therapeutic fields. Mobile assisted living systems help vulnerable patients manage their daily lives independently. An AI algorithm inquiring how much they drank during the day or whether they took their medication correctly could enhance these systems because patients often do not answer these questions truthfully (Fainzang 2002). The algorithm could identify answers that do not reflect the person’s convictions and trigger human intervention from a designated caregiver to ensure the well-being of the patient. Psychotherapists could employ such an algorithm to see whether the patient is speaking to her/his convictions and physicians could apply the algorithm to improve anamnesis and the resulting diagnosis, especially with regard to socially difficult topics such as sexually transmitted diseases, drug use, or drinking and eating habits (Palmieri and Stern 2009). Naturally, these applications require transparent disclosure to the patient and adherence to corresponding data privacy regulations.