Be a detective for a day: How to detect falsified interviews with statistics

Abstract

Interview falsification has been occurring for decades on different levels but is still often insufficiently detected and treated in practice. This is a severe issue as interview fabrication can have major effects on final estimates.

This article aims to draw attention to this important topic, with focus on falsification by telephone (CATI) interviewers, and to provide National Statistical Offices with a base for a data-driven, mostly automatic tool written in the statistical software R to detect interviewers with irregularities in their answers or their time patterns. The further application of traditional methods, e.g. recontact or test calls, is far more effective with such focused samples than with random samples.

As revealing conspicuous interviewers sometimes feels like detective work, this article intentionally has a little detective side story. Together we will look at the means of evidence, investigate it with various methods, filter out the conclusive evidence and conclude how to close the case.

Keywords

Interview fabrication interviewer curbstoning

1. Introduction

“Detective, our data shows serious and unreasonable time series breaks. I count on you!”

1.1 Why would NSOs want to check for fabricated interviews by data-driven means?

Statistical agencies spend a lot of time and caution on preparing data collections, plausibility checks, imputations and weighting of survey data. Final estimates, however, can still be heavily biased by interviewers who provide suspicious and arbitrary answers in interviews.

Interviewing can be very frustrating. Imagine you are conducting interviews all day long and no matter how hard you try, you never reach the threshold of interviews to get a financial benefit – but others do. Or all the participants simply refuse taking the interview with you. Or you always have to ask sensitive questions to which the participants react angrily. In literature [1, 2] these and many more reasons are listed why interviewers potentially falsify interviews.

Therefore, interview falsification is as old as interviews themselves. Hence, according to the US census bureau [3] this topic has already been analyzed for decades. In recent years, new data-driven approaches are discussed as effective supplement to more traditional routine methods as e.g. recontact methods [4]. Still, a globally applicable, data-driven method is missing [2].

1.2 How can other NSOs benefit from this article?

“Ok Capitan, here is a list of observations I made. I think we are already very close.”

This article has two aims:

To once again draw attention to the topic of interview falsification, since it is still discussed too little and data-driven methods are not effectively used in relation to the “seriousness of the issue” [5].

To provide statistical offices with a base of a (mostly) automatic, hardly time-consuming tool for detection of fraudulent telephone (CATI) interviewers.

I proceeded in three steps:

Partly existing, partly novel methods were selected from the literature existing to detect interview falsification. The methods focus on interview falsification in the sense of “curbstoning” – “Whole Interview Fabrication” [2].

R functions for these methods that run with the input of raw survey data and/or survey time stamp metadata were written [6].

The functions were summarized in an Rmd-File for regular outputs in html-document format [6].

R code can be accessed on https://github.com/marle neweinauer/iatools.

Please do not expect the final document to give an error-free classification of “honest” and “fraudulent” interviewers. It does not! But it does provide a list of highly suspicious interviewers due to an accumulation of strongly deviant figures based on interview data and time stamp metadata.

1.3 How is this article structured?

Firstly, I will describe the means of evidence – the data matrices that are used as input data to apply the methods. Then methods to solve the case are presented: For reasons of overall applicability, in this article only methods that do not require a trainings data set are used. Methods applied to raw survey data as well as to time stamp metadata are presented. Then the usage and interpretation of the software implementation are presented.

The last part is a conclusion which provides information on how to, based on your evidence collected, decide which interviewer is a potential falsifier and which one is not. Finally, an outlook is given.

2. Methods

2.1 Means of evidence

To run the full Rmd three matrices are necessary, but the Rmd can also run partly only with the matrices available.

The “Suspect’s statement” of what has happened.

An $n\times(m+1)$ “raw data matrix” $X$ with $n$ interviews in rows, $m$ questions in columns and an $(m+1)^{\rm th}$ column of interviewer IDs. Each cell $x_{ij},i\in\{1,\dots n\};j\in\{1,\dots,m\}$ is filled with the survey result to this specific question. For reasons of simplicity, on survey answer level this article only deals with multiple and single-choice questions. A single-choice question is presented in one column, a multiple-choice question with r items is represented in r columns.

The “background information”: Everything you get to know about the suspect from somewhere else (e.g. information from other witnesses, the crime scene, etc.)

An $n\ \times(m+1)$ “question time stamp data matrix” $Y$ of same structure as the “raw data matrix”. Each cell $y_{ij},i\in\{1,\dots{n}\};j\in\{1,\dots,m\}$ is filled with the amount of time spent on this question. A row sum command applied to this matrix leads to the “whole duration time stamp data matrix”.

The “Suspect’s personal information”: Personal information can be exonerating, e.g. The suspect is right-handed, but the wanted person for sure is left-handed.

An $n\times(b+1)$ “control data matrix” $C$ with n interviewers in rows, b highly correlated socio-economic variables in columns and a $(b+1)^{\rm th}$ column of interviewer IDs. Examples are variables such as sex and age (e.g. employment situation, etc.).

2.2 Hypothesis on falsified interviews

Now the detective’s work begins: You know the evidence, but the puzzle still needs to be solved.

Based on literature research three simple hypotheses to detect falsifying interviewers were chosen:

Hypothesis 1. Falsifying interviewers will in general show different patterns in the answers given in comparison to honest interviewers, see e.g [7, 8]. Additionally, Hood and Bushery [9] concluded that falsifiers “try to keep it simple and fabricate a minimum of falsified data”. Therefore, especially for questions that result in a list of follow-up questions (“gate questions”), deviant shares – leading to shorter questionnaire paths – are common in falsified interviews [4, 9, 10].

Hypothesis 2. Falsifying interviewers will have shorter time patterns – to specific items or the entire questionnaire. Shorter times on the entire questionnaire are often a result of the “keep-it-simple” – principle of hypothesis 1 [1, 4, 7, 9].

Hypothesis 3. Falsifying interviewers will make similar mistakes. An interviewer with similar patterns – in either survey data patterns or time patterns – as a falsifying interviewer will probably also be a falsifier [10].

Methods requiring a training data set, as e.g. logistic regression or random forests, were skipped for reasons of general applicability, despite their proven high accuracy (in some cases up to 96% [1]).

Other suggested hypotheses on behaviors of interviewers as e.g. a low “other ratio” (low ratio of answer options “other” that are followed by an open-text-question to clarify the “other”) [10] were also not included due to general inapplicability. However, an extension of the report, offering optional inclusion of survey-specific measurements is considered.

The methods in this article that are based on the raw data matrix refer to single and multiple-choice questions only. Also, an extension for other question types, e.g. open-text-questions, is considered.

In this article, interviewers who falsified all of their interviews are referred as “complete falsifiers”, interviewers who falsified parts of their interviews as “partly falsifiers”. “Falsifiers” includes members of both previous groups.

In the following part selected methods are explained to check the hypotheses.

2.2.1 Hypothesis 1: Binomial approach

“Firstly, there is the interrogation. There is a good chance that someone will already reveal himself by his own statement”

The approach of analyzing survey data has already existed since the 1980s to provide important insight into what may characterize fabricated data [2].

Single and multiple-choice question items behave equally to a tossed coin. If the desired item is ticked, the coin lands on heads, if the item is not ticked, it lands on tails. If a coin landed on heads most of the time (e.g. 90 out of 100 times), you would wonder whether the coin is rigged as you would expect it to land on heads in average every second time due to the law of large numbers, if the sample size is large enough. The only difference of a survey item is that you do not expect it to be ticked every second time, but another true ratio $p\in[0,1]$ times that is unknown. If, for a specific item, a specific interviewer has a ratio

$\hskip-15.0pt\frac{\text{number of interviews in which this item was ticked}}{% \text{number of interviews in which this item was asked}}$

of about 37/186 $=$ 0.1989, and we assume that the true ratio for this item to be 0.85, this interviewer could be considered conspicuous. If this occurs for several items, this can be considered as very improbable and therefore the interviewer as very suspicious.

To compute whether the results differ significantly from the expected value, a binomial test is carried out for each item and each interviewer. For each test

The number of successes is given by the number of interviews conducted by the specific interviewer in which the specific item was ticked.

The number of trails is given by the number of interviews conducted by the specific interviewer in which the specific item was asked.

And the hypothesized probability of success is the estimated median (assuming that a majority of interviewers do not falsify) of the results of the remaining interviewers, excluding the specific interviewer.

This idea is not completely new – in 2012 Benjamin Birnbaum [7] in his dissertation on detecting fabricated interviews developed the Multinomial Model Algorithm (MMA) to find irregularities. In this article binomial distribution for each single item is applied instead of multinomial distribution to get specific information on each single item. In this article $p$ -value adjustment is used (e.g. “Bonferroni adjustment”, “Holm adjustment”) to regulate the cumulative $\alpha$ -error: If e.g. $\alpha=$ 0.01 and the number of interviewers equals 80, without $p$ -value adjustment the chance of accusing at least one interviewer wrongly is 80%. With the adjustment the chance of accusing at least one of the 80 interviewers wrongly is at 1% only.

Binomial approach 1. To detect a falsifying interviewer who follows the “keep-it-simple” principle, it only takes to test for gate questions (that shorten the path). Every interviewer with a $p$ -value-adjusted significant share for even one of these questions should be observed in further tests.

Binomial approach 2. An interviewer, who does not follow the “keep-it-simple” principle, but still falsifies, can get conspicuous in showing deviant shares on many questions. Birnbaum et al. [1] showed that even if interviewers are alerted to the control for deviant results in all questions, most of them could not reproduce regular shares.

Due to the exact distribution, the binomial test is applicable even for small $n$ . However, the beta error shoots up for small $n$ . But this is only fortunate for falsifiers. A high beta error means that not all fraudulent interviewers are detected and not that a good interviewer is blamed wrongly (which is considered as worse). The usage is therefore recommended only after a minimum of interviews was conducted.

2.2.2 Hypothesis 2: Time stamps approach

“Captain, I think we got them: The video recordings show: Some of them were in the wrong place at the wrong time.”

In 2013 Birnbaum et al. [1] proved supervised classification to detect falsifying interviewers with metadata to have much higher accuracy than analysis with interview data only (accuracy without metadata 61% to 77% in comparison to accuracy with metadata 86% to 96%).

The implementation of hypothesis 2 “Falsifying interviewers are fast” is not as straightforward as the hypothesis itself. Some thoughts on this include: A time stamp ${<}$ 1 sec on a specific question indicates that the question was not read. But is an interviewer with a single time stamp ${<}$ 1 in a question already suspicious and is the cutting line of 1 sec a good cutting line? Literature answers these questions with the median, the mean, and the sum of durations as estimators for the interviewers [1, 4, 7]. But only a combination of these estimators maps the situation well.

It was decided to focus on three estimators of question time stamps which are also applied on whole questionnaire durations:

Time stamp approach 1. The median of times spent on a specific question by interviewers is a robust measure to detect “complete falsifiers” on that question. The “top speeders” regarding this specific question are picked out via one of four selected methods.

A fixed number $t$ of top speeders (e.g. $t=$ 5 gives the 5 top speeders).

A quantile $q$ of the top speeders (e.g. $q=$ 0.05 gives 0.05 quantile of top speeders).

A cutting line of question durations: “Top speeders” are interviewers with a median smaller than the first breakpoint in the sorted vector of medians.

Interviewers with times smaller than the interval median (time) – $c\ \times$ mad (time). A common choice for $c$ is 2.5.

Time stamp approach 2. To detect partly falsifiers, novelly a 0.2-quantile as estimator, assuming that a falsifier would fake between 20% and 50% of his interviews, is used. The rest in analogy with Time Stamp Approach 1.

Time stamp approach 3. In this novel approach the data is not sorted by interviewers beforehand. The 0.05 (variable via function parameter) quantile of all durations on a specific question produced by all interviewers is taken. Then a frequency table of the interviewers producing this 0.05 quantile is created. Only interviewers above the 0.9 quantile (variable via function parameter) in the frequency table and with a minimum absolute number of suspicious cases are considered.

If time stamps are available, analyses of them have shown great accuracy (e.g. for supervised classification needing a training data set) [1]. Analysis of time-stamp metadata is especially insightful for telephone interviews – for face-to-face interviews GPS data is also useful [4]. A disadvantage of the methods a to c in “Time Stamp Approach 1 and 2” is that if no one is cheating, still interviewers will appear as suspicious. A disadvantage of d in the “Time Stamp Approach 1 & 2” is that the distribution of the observations, i.e. the time for the interviews, is assumed to be symmetric. If this is not the case, transformations such as the log, exp or boxcox might be applied with the aim to achieve a symmetric distribution before the methods are applied. This is, for example, recommend by Templ, Gussenbauer and Filzmoser [13].

2.2.3 Hypothesis 3: Cluster approach

“Captain, this suspect’s statement is very similar to the made-up story of the committer. Do you think they are accomplices?”

A not new, but very useful approach in fraud detection, specifically in interview falsification, is to put interviewers with similar patterns in groups via cluster analysis [10]. Cluster analysis is based on similarity measures without any a priori knowledge. It is not to be expected to get two perfectly separated groups (“honest interviewers” & “fraudulent interviewers”), but cluster analysis has proven to put falsifying interviewers in similar groups [10].

In this article cluster analysis for fraud detection has been implemented in interviews in two raw forms:

Cluster approach 1. This approach assumes similar patterns in answers given by fraudulent interviewers. As, according to hypothesis 1 especially for gate questions similar patterns are expected, this approach was implemented for both binomial approaches of H1.

Cluster analysis is applied on an $i\times f$ matrix with i interviewers in rows and f gate questions in columns. Each cell $(k,l)k\in\{1,\dots{i}\};l\ \in\{1,\dots,f\}$ is filled with the shares of questions ticked and questions asked from the binomial approach. Clustering only over gate questions was already proposed 1997 by Hood and Bushery [9].

Cluster analysis is applied on the same $i\times m$ matrix with m being all questions. The rest is equal to a.

Cluster approach 2. This novel approach again assumes similar patterns in time stamps of fraudulent interviewers. Cluster analysis is applied on an $i\times m$ matrix with i interviewers in rows and m questions in columns. Each cell $(k,l)k\in\{1,\dots{i}\};l\in\{1,\dots,m\}$ is filled with an estimator (20%-quantile or median) of the duration of the whole questionnaire.

Cluster approach 3. In this approach it is clustered over the merged matrices of “Cluster Approach 1” and “Cluster Approach 2”.

Interesting extensions as e.g. Bredl’s, Winker’s and Kötschau’s approach [9] who clustered several indicator variables as the “other-ratio” have not been implemented yet, again for the reasons of overall applicability. But they are recommended to use if the questionnaire involves these specific structures.

The main advantage of cluster analysis is that it detects suspicious patterns very quickly, and that it is – if a previous analysis was done – a confirmation of the results obtained so far. Cluster analysis also helps to spotlight previously unsuspicious interviewers in suspicious clusters. The disadvantage is that you never get two perfect groups and it could happen that you get unusable groups from this analysis.

3. Software implementation

“Captain, we have been equipped with some new tools. Let me show you how to use them.”

In the following, the concept of the R Code is explained. At the moment R Code is only implemented in source files, but an R package is being worked on currently. The R code and a vignette with concrete examples are provided on https://github.com/marleneweina uer/iatools).

In all approaches interviewers with a little number of interviews are discarded.

3.1 Hypothesis 1: Binomial approach

The input matrix is the “raw data matrix”. The “control data matrix” is needed to control for identically distributed samples. Otherwise an interviewer who has e.g. interviewed mainly old people can show deviant results compared to an interviewer who has e.g. interviewed mainly young ones without any fabrication. The method of control is implemented in the Rmd.

The output file shows the binomial test applied in two loops:

Binomial implemention 1 Following “Binomial Approach 1” looping is done only over (relevant) gate questions. These gate questions need to be defined in the Rmd. If there are no relevant gate questions in the questionnaire, this part is skipped.

Binomial implemention 2 Following “Binomial Approach 2” looping is done over all questions.

The number of questions for which an interviewer has shown $p$ -value-adjusted significant shares is coun-ted for both approaches separately. All interviewers with any significant results in the first loop and interviewers with the highest number of significant items in the second loop are highlighted.

When interpreting the results, the $\alpha$ -error (some interviewers who did not falsify will show significantly deviant results) must be kept in mind. Significances are also only relevant for interviewers who have no deviant samples according to the “control-data-matrix” regarding relevant factors as e.g. age and sex. An interviewer of this group who shows deviant results in at least one gate question according to the “Binomial Approach 1” or many significances according to “Binomial Approach 2” should be observed further on.

3.2 Hypothesis 2: Time stamps approach

The input matrix is the “question time stamp data matrix”. Before analyzing time data, don’t forget to preprocess the data if necessary (e.g. cut unreasonable high values in the last question because interviewer have not logged out the system).

The three methods are applied in two variants:

Time stamp implementation 1 Each method is applied in a loop over all question durations. For each method a frequency table is constructed that indicates in how many questions an interviewer appeared as conspicuous. The most conspicuous ones (conspicuous in more than a fixed number of questions, e.g. 10) are printed in the final Rmd for each of the three methods separately.

Time stamp implementation 2 Each method is applied on the duration of the whole questionnaire. As this is no loop the most extreme results of the three methods are printed directly in the Rmd.

Results of “Time Stamp Implementation 1” – durations of questions – are interpreted in the following way. Results of “Time Stamp Approach 1” can detect complete falsifiers. Results of “Time Stamp Approach 2” & “Time Stamp Approach 3” in combination can detect partly falsifiers. An interviewer conspicuous in both items is to be regarded as suspicious. An interviewer conspicuous in the second approach, but not in the third one could be a fast reader but still an honest interviewer as he has not produced many (or any) completely extreme results. An interviewer conspicuous in 3, but not in 2 has produced extreme results, but the majority of interviews was not salient. Extreme results could be therefore outcomes of other reasons, e.g. clicking through the questionnaire sessions to get acquainted with the questions.

Results for “Time Stamp Implementation 2” – duration of the whole questionnaire – are interpreted equally: “Time Stamp Approach 1” detects complete falsifiers, “Time Stamp Approach 2” and “Time Stamp Approach 3” in combination partly falsifiers. Results of “Time Stamp Implementation 2” also reveal interviewers following the “keep-it-simple” principle, see “Binomial Implementation 1”.

3.3 Hypothesis 3: Cluster approach

For Cluster Approach 1 the input data matrix is the “raw data matrix”, for cluster approach 2 the input matrix is the “single time stamp data matrix”, which is immediately aggregated to the “whole duration time stamp data matrix”.

For each Cluster method the Rmd produces a dendrogram and cluster centroids. With the cluster centroids, the dendrogram and the results of hypothesis 1 and hypothesis 2 can be checked whether the clusters distinguish between honest and fraudulent interviewers (according to experience they mostly do, but there is no guarantee). If so, it can be checked whether so far unsuspicious interviewers appear in suspicious clusters and vice versa. This additional information helps in making further decisions.

4. Conclusions

Following Mneimneh et al.’s [4] demand of the display of data-driven results “in a manner that is easy for Quality Control personnel to visualize, identify troublesome cases and decide on an action plan”, only highly conspicuous interviewers are listed for each method (accompanied by the number of suspicious cases/the time estimate). Interviewers with following characteristics are suggested to be regarded as suspicious.

Interviewers that are listed in “Binomial Approach 1” or listed in “Binomial Approach 2” with a high number of occurrences.

Interviewers that are listed in “Time Stamp Approach 1” or listed in “Time Stamp Approach 2” and “Time Stamp Approach 3”.

Interviewers that are listed in clusters with suspicious interviewers.

For better insights into a suspicious interviewer’s behavior, the indicators of the three hypotheses for the specific interviewer – in comparison to the expected indicators – are shown in a second report file, an interviewer-tailored summary of results.

The manual selection of suspicious interviewers based on the Rmds is proposed weekly – depending on the survey design.

“Captain, I have now brought together the conclusive evidence. But who will be arrested?”

Like other data-based approaches for detection of falsifying interviewers [4, 12], also the approach presented here is not designed for sole usage. To avoid demoralizing of good interviewers with unjustified accusations, suspicious interviewers must be monitored with “traditional” methods further on, e.g.

Recontact,

Test calls,

Monitoring recorded interviews.

In a literature review Bredl, Storfinger and Men-old [11] concluded clearly that data-driven “focused recontacts are more effective than recontacts based on random samples”. This is supposed to be also valid for other methods like test calls or monitored recalls.

If an interviewer is finally identified as a falsifier, interventions – depending on the severity of falsification – as e.g. retraining, sending persons to observe the interviewer’s field work, evaluating more of their recorded interviews, suspending the interviewer temporarily, or terminating their service on the project are necessary [4].

4.1 Perspectives for future work

“Captain, can we close the case now?” “No, detective. I am afraid there is still a lot of work to do!”

This article has once again drawn attention to an underrepresented part of survey methodology. This article provides a base construct of overall-applicable methods to detect for potential curbstoning on interviewer level especially in telephone interviews based on survey data (single and multiple choice questions) and on time stamp data. The provided software enables an implementation of presented methods in other national statistical offices (see https://github.com/marle neweinauer/iatools).

However, the field of interview fabrication is broad-er. In addition to the metadata used in this article, other metadata, e.g. GPS data or information about how often answers have changed, can also include important information [1, 4]. Other question types, as e.g. open-text questions may give insights as well [1]. For web surveys, which are not conducted by interviewers, methodologies to detect random completing could be developed. Other employees involved directly in the data acquiring process, e.g. supervisors, may also falsify [8]. Also the detection of duplicated interviews gives important hints of interview falsification [12].

Furthermore, connected fields deserve deeper research, e.g. the prevention of interview fabrication, how to give positive feedback without promoting fabrication or the unintentional influence of interviewers [4].

Footnotes

Acknowledgments

I gratefully acknowledge my superiors at Statistics Austria who enabled me to engage in this subject, all my colleagues at Statistics Austria for their inspiration and support and my former professor Matthias Templ for feedback and advice.

References

Birnbaum

Borriello

Flaxman

DeRenzi

Karlin

. Using behavioral data to identify interviewer fabrication in surveys. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems – CHI’13. 2013; doi: 10.1145/2470654.2481404.

Murphy

Biemer

Stringer

Thissen

Day

Hsieh

. Interviewer falsification: Current and best practices for prevention, detection, and mitigation. Statistical Journal of the IAOS. 2016; 32 (3): 313-326. doi: 10.3233/sji-161014.

Brick

Tran

Singer

. Using statistical models for sample design of a reinterview program. Journal of Official Statistics. 2011; 27 (3): 433-450

Mneimeh

Lyberg

Sharma

Vvas

Sathe

Malter

Altwaijri

. Case Studies on Monitoring Interviewer Behavior in International and Multinational Surveys. In: Johnson T, Pennell B, Stoop I, Dorer B, editors. Advances in Comparative Survey Methods: Multinational, Multiregional, and Multicultural Contexts (Wiley Series In Survey Methodology). New York: Wiley; 2018.

Koczela

Scheuren

. Progress in understanding survey data fabrication. Statistical Journal of the IAOS. 2016; 32 (3): 277-282. doi: 10.3233/sji-161026.

R Core Team.

. A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/. 2019.

Birnbaum

. Algorithmic approaches to detecting interviewer fabrication in surveys. Dissertation. University of Washington: Washington. 2012.

Spagat

. Evidence of potential fabrication in iraq public opinion polls from 2005–2008. Project Paper. University of London: London. 2011.

Hood

Bushery

. Getting more bang from the reinterview buck: Identifying “at risk” interviewers. Proceedings of the American Statistical Association (Section on Survey Research Methods). 1997; 820-824.

10.

Bredl

Winker

Kötschau

. A statistical approach to detect cheating interviewers. Discussion Papers 39, Justus Liebig University Giessen, Center for international Development and Environmental Research, 2008.

11.

Bredl

Storfinger

Menold

. A Literature Review of Methods to Detect Fabricated Survey Data. In Winker P, Menold N, Porst R, editors. Interviewers’ Deviations in Surveys (Schriften zur empirischen Wirtschaftsforschung). Bern: Peter Lang; 2011.

12.

Slomczynski, Kazimierz M, Przemek Powalko and Tadeusz Krauze. The Large Number of Duplicate Records in International Survey Projects: The Need for Data Quality Control. CONSIRT Working Papers Series 8 (CONSIRT Labs: Methodology of Survey Data Harmonization).

13.

Templ

Gussenbauer

Filzmoser

. Evaluation of robust outlier detection methods for zero-inflated complex data. Journal of Applied Statistic. 2019. Doi: 10.1080/02664763.2019.1671961.