From COACH to COACH+: Automating output checking with human-in-the-loop

Abstract

This paper presents COACH+ (COmputer-Assisted Output CHecking with Human-in-the-loop), an extension of COACH by (Slokom et al. COACH: computer Assisted output CHecking with human-in-the-loop. U N Econ Comm Eur Conf Eur Stat) that integrates machine learning models with human expertise to automate output-checking processes. Building upon the foundation of COACH, COACH+ improves the collaborative capabilities between automated algorithms and human chequers. The primary objective remains to facilitate the assessment of outputs generated by researchers to determine their suitability for public release. We first provide an overview of our initial iteration of COACH, which revolutionised output checking by combining machine learning models with human expertise. Then, we present the advancements introduced in COACH+, which are divided into two main aspects. In the backend, we have integrated a secondary machine learning algorithm, a convolutional neural network, designed to analyze images such as plots and determine their safety for release. In the front end, we have minimised reliance on human chequers. Unlike the previous COACH version, where human intervention was required to input certain values before making predictions, COACH+ now allows human chequers to upload Excel files and figures directly.

Keywords

output checking human-in-the-loop multi-model machine learning

1. Introduction

Official statistical agencies handle vast amounts of data, referred to as outputs, on a daily basis. These outputs must go through rigorous evaluation to determine their suitability for public release. The evaluation is based on a number of rules called rule of thumb Bond et al.¹ Traditional methods of output checking within statistical agencies heavily depend on manual inspection, consuming substantial time and resources. To address these challenges, we introduce COACH+ (COmputer-Assisted Output CHecking with Human-in-the-loop), an improved approach that semi-automates output checking by integrating human expertise with machine learning algorithms. Building upon the research by Domingo-Ferrer and Blanco-Justicia,² COACH in Slokom et al.³ proposed a novel approach leveraging machine learning in the output checking process. Domingo-Ferrer and Blanco-Justicia² developed simulated data based on a subset of rules, known as the 14 rules-of-thumb (for further details, refer to Bond et al.,¹ and subsequently trained and tested neural network models to learn these rules. By harnessing simulated data and advanced machine learning algorithms, their models achieved high prediction accuracy by discerning complex patterns. Following a similar trajectory,⁴ has recently introduced a new tool utilising machine learning models such as deep neural networks and random forests. This tool was trained using approximately 13,000 previous results of manual output checks. Notably, few existing works have explored the automation of output checking. For instance, the authors in Green et al.⁵ proposed the ACRO toolkit for the automatic checking of research outputs, with subsequent developments such as SACRO and Py-ACRO extending its functionality. Distinguishing itself from earlier projects Cole et al.,⁶’s tool covers a broader range of statistics and aims for general applicability in output-checking processes. COACH+ expands collaborative capabilities between automated algorithms and human chequers. Our primary objective remains to facilitate the assessment of outputs generated by researchers to determine their suitability for public release. We first provide an overview of COACH’s initial iteration, which pioneered the integration of machine learning models with human expertise in output checking. Subsequently, we outline the advancements introduced in COACH+, categorizing them into two primary aspects. We summarize our contributions as follows:

The back-end: COACH+ integrates a secondary machine learning algorithm, specifically a convolutional neural network (CNN), into its backend architecture. This CNN is tailored to analyze visual data, such as plots and images, to determine their safety for public release. By expanding the scope of analysis to include visual elements, COACH+ improves the accuracy and robustness of output-checking processes.

The front-end: COACH+ reduces reliance on human chequers by introducing a streamlined approach to input data. In contrast to the previous version of COACH, which required human intervention to input specific values before predictions could be made, COACH+ now allows human chequers to directly upload Excel files and figures. This improvement simplifies the workflow, saves time, and increases overall efficiency in the output-checking process.

2. Background and related work

In this section, we will give a brief overview of existing work on automating output checking and human-in-the-loop systems.

2.1. Automating output checking

Guidelines for confidentiality serve as a standard template that should be followed by human chequers and researchers. The guideline is based on rule of thumbs to prevent confidentiality errors and as a result to assess if an output is safe to be released or unsafe. For a detailed and elaborate definitions of rules-of-thumb, we point readers to Bond et al.¹ and Domingo-Ferrer and Blanco-Justicia.² In Table 1, we provide a brief description of each rule and corresponding parameters as used in the settings of Domingo-Ferrer and Blanco-Justicia² along with cases in which output should be treated as unsafe.

Table 1.
A summary of rules-of-thumb as used in our experiments and cases in which the output is treated as unsafe.

Type of statistics Type of output Decision is “unsafe” if:

Descriptive Statistics Frequency tables Output is confidential. Some cell contains less than 10 units. A single cell contains more than 90% of the total number of units in a row or column.

Magnitude tables Output is confidential. Single cell contains more than 90% of the units in a row or column In some cell the largest contributor contributes more than 50% of cell total.

Maxima, minima, median, and percentiles Output is confidential

Mode Output is confidential. The frequency of model value is more than 90% of the sample size.

Means, indices, ratios, indicators Output is confidential. Sample size <10. A single cell contributes for more than 50% of the total sample.

Concentration ratios Output is confidential. Sample size <10. A single cell contributes for more than 90% of the total sample.

Variance, skewness, kurtosis Output is confidential. Sample size <10.

Graph Output is confidential

Correlation and Regression Analysis Linear regression coefficients / Non linear regression coefficients Output is confidential Intercept is returned as one of the coefficients

Regression residuals, Regression residual plots Output is confidential

Test statistics_t test statistics_F Output is confidential Degree of freedom <10

Correlation Output is confidential Coefficients that are -1, 0, 1 or <10

Correspondance analysis Output is confidential. Number of variables <2 or sample size <10.

Type of statistics	Type of output	Decision is “unsafe” if:
Descriptive Statistics	Frequency tables	Output is confidential. Some cell contains less than 10 units. A single cell contains more than 90% of the total number of units in a row or column.
	Magnitude tables	Output is confidential. Single cell contains more than 90% of the units in a row or column In some cell the largest contributor contributes more than 50% of cell total.
	Maxima, minima, median, and percentiles	Output is confidential
	Mode	Output is confidential. The frequency of model value is more than 90% of the sample size.
	Means, indices, ratios, indicators	Output is confidential. Sample size <10. A single cell contributes for more than 50% of the total sample.
	Concentration ratios	Output is confidential. Sample size <10. A single cell contributes for more than 90% of the total sample.
	Variance, skewness, kurtosis	Output is confidential. Sample size <10.
	Graph	Output is confidential
Correlation and Regression Analysis	Linear regression coefficients / Non linear regression coefficients	Output is confidential Intercept is returned as one of the coefficients
	Regression residuals, Regression residual plots	Output is confidential
	Test statistics_t test statistics_F	Output is confidential Degree of freedom <10
	Correlation	Output is confidential Coefficients that are -1, 0, 1 or <10
	Correspondance analysis	Output is confidential. Number of variables <2 or sample size <10.

Output checking assesses the disclosure risk of research results based on microdata files, figures, and documents, available in research data centres.¹ It aims to classify outputs as safe for publication, requiring further analysis, or unsafe due to disclosure risk, guided by predefined rules. Despite its importance, research on output checking is still in its infancy. Currently, human chequers, typically skilled staff at statistical agencies, perform this task, leading to time-consuming and costly processes. Recently, researchers have explored methods aiming at facilitating the output-checking process. Domingo-Ferrer and Blanco-Justicia² proposed a novel approach employing machine learning to aid human chequers. They generated simulated output-checking data based on predefined rules, trained neural network models on this data, and evaluated their performance. The primary goal in automating output checking is to minimise false positives, indicating outputs wrongly predicted as safe but deemed unsafe by the agency. Green et al.⁵ introduced the ACRO toolkit, initially developed in STATA and later extended to Python for wider accessibility. Recently, Cole et al.⁶ extended ACRO into SACRO. SACRO stands for semi-automated checking of research outputs. It supports researchers using the major analytical languages (R, Python, and Stata), with minimal changes. SACRO also supports secure environments with different operating models and output-checking workflows. Rigaud et al.⁴ have recently introduced a new tool utilising machine learning models such as deep neural networks and random forests. This tool was trained using approximately 13,000 previous results of manual output checks.

2.2. Human-in-the-loop systems

Human-in-the-loop (HITL) approaches⁷ have applications in a variety of fields, including health care (Wrede and Hellander, 2019), finance, computer-vision,⁸ and natural language processing.⁹ HITL is especially relevant in situations where human expertise is necessary for critical decisions, such as medical diagnosis or financial risk assessment. The central idea of HITL is to combine human judgment and intuition with the computational power of algorithms to achieve improved decision outcomes. Depending on who is in control and in which phase of the learning process, we can identify different approaches to HITL in machine learning.¹⁰ Active learning (AL), in which the system remains in control of the learning process and humans are involved in the annotation of unlabeled data. Interactive machine learning (IML), in which there is a closer interaction between users and learning systems, with people interactively supplying information in a more incremental way compared to traditional machine learning. Machine teaching (MT), where human domain experts have control over the learning process by delimiting the knowledge that they intend to transfer to the machine learning model. In addition to the aforementioned benefits of HITL, we also point to the importance of HITL in that it offers a more Explainable AI (XAI), “Usable AI” and “Useful AI”.¹⁰

3. Methods

In this section, we will describe our recent developments of COACH+. In addition to COACH+, we have another work in the same direction of automating output checking, called AOCH. AOCH stands for assisted output checking. AOCH and COACH+ are complementary.

3.1. Assisted Output Checking (AOCH)

Figure 1 depicts a screenshot of AOCH, a straightforward yet efficient tool designed to aid in the output-checking process. AOCH operates on hardcoded rules, such as threshold values in frequency tables or suppression of individual or group data. Unlike methods utilising machine learning algorithms, AOCH relies solely on these predefined rules. The workflow within AOCH begins with file upload and parameter setup. Subsequently, areas where rules are violated are highlighted using different colours for easy identification. The ”add” and ”reject” buttons categorize files and tables into lists for release or rejection based on the human checker’s decision. Upon export, files in these lists are zipped and sent to the researcher. The ”reset” button restores the classification of a file to unclassified status.

3.2. Computer-Assisted Output Checking (COACH+)

While AOCH offers a straightforward solution to output checking, the challenge of automating this process persists. Human chequers still face the task of configuring parameters and selecting specific rules for analysis, requiring additional effort. COACH+ addresses these limitations by enhancing AOCH with increased flexibility and the capability to analyze not only Excel files but also images. This expansion empowers human chequers with greater control and extends the scope of output checking beyond traditional data formats.

COACH+ represents an evolution from its predecessors, AOCH and COACH, in several key aspects. Firstly, while AOCH relied primarily on predefined rules for output checking, COACH+ embraces the capabilities of machine learning algorithms to make decisions based on previous data. By allowing the machine to learn from past instances and identify exceptions or patterns that may evade human chequers, COACH+ offers a more comprehensive approach to output checking.

Secondly, COACH+ simplifies the input process for human chequers by eliminating the need to manually fill in certain values, as depicted in Figure 2 (left). Instead, in the updated version Figure 2 (right), human chequers simply upload the files, whether Excel files or plot figures. Upon clicking the predict button, each uploaded file is processed individually. Excel files are analyzed by the machine learning model responsible for tabular data, while figures are queried to the machine learning algorithm handling image data. The final output decision is then determined based on the combined results of both ML models. If either model flags the output as unsafe, the final decision is deemed unsafe.

Figure 3 illustrates examples of decisions that can be reported by COACH+, showcasing its ability to provide clear and actionable decisions. For instance, in Figure 3. (Left), COACH+ predicts the output as “safe”, whereas, in Figure 3. (Right), COACH+ predicts another output as unsafe. We note that Slokom et al.³ provided further explanations showing why and how the machine learning algorithm comes to a specific decision (due to space limitation, we omit this part from this paper).

In order to encourage reproducibility and foster collaboration between different statistical offices, we are working on releasing both AOCH and COACH+ to the public.

4. Experimental setup

In this section, we will describe the back-office of COACH+. This includes describing the data that we used in our experiments as well as the machine learning algorithms.

4.1. Data

In our experiments, we used both real data and simulated or synthetic data. Also, we have tabular data, i.e., excel files, and image data, i.,e plots.

4.1.1. Tabular data

Following the methodology of Domingo-Ferrer and Blanco-Justicia,² we generated training and test datasets containing 200,000 and 14,000 records, respectively. Each rule of thumb was represented by approximately 14,700 records in the training set and 1,000 records in the test set. Our real test data consisted of 125 records, predominantly comprising frequency tables, magnitude tables, and regression models. We trained machine learning algorithms on the generated training data and evaluated their performance on both simulated and real test data. Additionally, we conducted experiments involving human-in-the-loop feedback by retraining machine learning algorithms on simulated data combined with human feedback data and testing on the real test file.

4.1.2. Image data

We collected a total of 191 real plots, including histograms, bar plots, line plots, and maps. The dataset was split, with 70 It’s worth noting that both tabular and image data exhibited imbalances, not only in terms of labels (safe or unsafe) but also with respect to the distribution of different rules.

4.2. Machine learning algorithms

Similar to the description of the data sets. We first start by describing the machine learning algorithms that we used for tabular data. Next, we move to describe the machine learning algorithms used on image (plots) data.

4.2.1. Neural Network (NN)

The neural network used by Domingo-Ferrer and Blanco-Justicia² consists of two hidden dense layers, each containing 64 neurons with ReLU activation functions, followed by dropout layers. The networks take a 12-dimensional input and predict a binary output using a sigmoid activation function in the final layer (more details about neural network architecture and hyper-parameters tuning can be found in Domingo-Ferrer and Blanco-Justicia.²

4.2.2. LightGBM (LGBM)

In addition to the neural network algorithm used in Domingo-Ferrer and Blanco-Justicia,², we evaluate the output checking using a lightGBM algorithm.¹¹ LightGBM is a gradient-boosting framework based on decision trees to increase the efficiency of the model and reduce memory usage. LightGBM handles different types of Gradient boosting methods which can be specified with the boosting parameter: GBDT (Gradient Boosted Decision Trees), DART (Dropouts meet Multiple Additive Regression Trees), and GOSS (Gradient-based One-Side Sampling). In our experiments, we used the GBDT boosting method. GBDT is a traditional Gradient Boosting Decision Tree and uses several decision trees that are built sequentially. The first tree learns how to fit into the target variable. The second tree learns how to fit the residual (difference) between the predictions of the first tree and the ground truths’ actual values. The third tree learns how to fit the residuals of the second tree and so on. We used the implementation of LightGBM in the open toolkit PyCaret. We used split-based split, the learning rate is set to 0.1, the minimum number of child samples is set to 20, and $n_e s t i m a t o r$ is equal to 100. We compare the performance of our classifier algorithms to a random classifier. This random classifier serves as a simple baseline to compare against. Our random classifier uses a majority class strategy which returns the most frequent class label.

4.2.3. Convolution neural network (CNN)

is a class of Neural network that has proven its effectiveness in different areas of image recognition, processing, and classification. In our experiments, we focus on binary image classification. The task is similar to the task in tabular data, this time the CNN predicts if a plot is safe to be released or unsafe. There are different layers in the convolutional neural network. First, the input layer takes images, and resizes them for passing onto further layers for feature extraction. Second, the convolution layer acts as a filter for images. This layer aims at finding features from images and is also used for calculating the match feature points during testing. Then, the pooling layer takes large images and shrinks them down while preserving the most important information in them. Next, the rectified linear unit layer or ReLU layer swaps every negative number of the pooling layer with 0. This helps the CNN stay mathematically stable by keeping learned values from getting stuck near 0 or blowing up toward infinity. The final layer is the fully connected layers which take the high-level filtered images and translate them into categories with labels. We used the implementation from Keras. We compare the performance of our CNN classifier to a random classifier.

4.3. Evaluation metrics

In order to assess the quality of the target model predictions, we will calculate: the confusion matrix, F1 macro-average, and Matthews Correlation Coefficient (MCC. Confusion matrix is a table that is used to define the performance of a classification algorithm in terms of True Positive (TP), False Positive (FP), True Negative (TN), and False negative (FN).

The macro-averaged F1 score (F1-Macro) is computed using the arithmetic mean (aka unweighted mean) of all the per-class F1 scores. This method treats all classes equally regardless of their support value.

Matthews Correlation Coefficient (MCC) metric also takes into account all of TP, TN, FP, and FN. MCC is a balanced measure that can be used especially if the classes of the target attribute are of different sizes.¹² It returns a value between -1 and 1.

MCC = \frac{(T P * T N) - (F P * F N)}{\sqrt{(T P + F P) * (T P + F N) * (T N + F P) * (T N + F N)}}

(1)

5. Experimental results

In this section, we provide our results of predictions based on tabular data and image data using machine learning algorithms. We show our results of testing the trained model on real test data for both tabular and image data.

5.0.1. Results of predictions on tabular data

Table 2 presents the results of our predictive analysis on tabular data, comparing the performance of various classifiers. When evaluating predictions without HITL intervention, both the LGBM and neural network (NN) classifiers demonstrate substantially better performance than the random classifier when trained and tested on simulated data. This improved performance persists even when tested on real test data, highlighting the classifiers’ ability to generalize effectively beyond the training set.

Table 2.
Prediction performance is measured in terms of F1 macro-average, MCC (Matthews Correlation Coefficient), TP (true positive), FP (false positive), TN (true negative), FN (false negative).

Data Sets HITL Classifiers F1 Macro TP FP TN FN MCC

Tabular Sim train $+$ Sim test None Random 0.349 0 6500 0 7500 0.00

LGBM 0.849 6500 0 2101 5399 0.738

NN 0.942 6123 377 433 7067 0.884

Real test LGBM 0.614 16 38 1 68 0.405

Sim train $+$ Sim test With Random 0.349 0 6500 0 7500 0.00

LGBM 0.849 6500 0 2101 5399 0.738

Sim train $+$ Sim test 0.909 51 3 8 61 0.823

Data Sets	HITL	Classifiers	F1 Macro	TP	FP	TN	FN	MCC
Tabular	Sim train $+$ Sim test	None	Random	0.349	0	6500	0	7500	0.00
LGBM	0.849	6500	0	2101	5399	0.738
NN	0.942	6123	377	433	7067	0.884
Real test	LGBM	0.614	16	38	1	68	0.405
Sim train $+$ Sim test	With	Random	0.349	0	6500	0	7500	0.00
LGBM	0.849	6500	0	2101	5399	0.738
Sim train $+$ Sim test	0.909	51	3	8	61	0.823

Note: We compare the performance of a random classifier to LGBM (LightGBM) and neural network. Our classifiers are trained on simulated (sim) data. We evaluate the trained models on simulated test data and real test data. The random classifier uses the majority class strategy. HITL stands for human-in-the-loop. Rows sharing the same Sim test are comparable to each other. Rows that share the same real test are comparable.

In scenarios where HITL feedback is incorporated, particularly during real test data evaluations, the retraining of the LGBM model with human insights leads to notable performance improvements. Specifically, the incorporation of human feedback results in a dramatic reduction in false positives (from 38 to 3). This shows the role of human expertise in refining machine learning algorithms, thereby improving the accuracy and reliability of their decisions. We focus specifically on the reduction of false positives (FP) because it is essential to minimise situations where the machine predicts a case as “safe”, but human chequers identify it as “unsafe”. We note that the opposite case, where the machine predicts an output as unsafe when it is actually safe, is not particularly dangerous. Instead, it allows human chequers the opportunity to double-check before making the final decision.

5.0.2. Results of predictions on image data

In Table 3, we present our preliminary results for predicting whether images are safe for release. The table compares the performance of a CNN with a random classifier. We observe that CNN outperforms the random classifier in analyzing visual data, demonstrating the effectiveness of machine learning algorithms in this domain. However, despite the overall performance advantage, CNN shows limitations in accurately identifying true positives (TP)–i.e., images that are correctly predicted as safe. This suggests that the model’s ability to discern safe images needs further refinement and exploration in future research. Additionally, it is worth noting that our current analysis does not incorporate HITL mechanisms. Future work will aim to integrate human feedback into the prediction process to potentially improve the model’s accuracy and reliability.

Table 3.
Prediction performance measured in terms of F1 macro-average, TP (true positive), FP (false positive), TN (true negative), FN (false negative), MCC (Matthews Correlation Coefficient).

Data Sets HITL Classifiers F1 Macro TP FP TN FN MCC

Image Data Real train + Real test None Random 0.344 1 5 26 26 $- 0.204$

CNN 0.949 5 1 0 52 0.904

Data Sets	HITL	Classifiers	F1 Macro	TP	FP	TN	FN	MCC
Image Data	Real train + Real test	None	Random	0.344	1	5	26	26	$- 0.204$
CNN	0.949	5	1	0	52	0.904

Note: We compare the performance of a random classifier to CNN. We aim to get all predictions correct (TP) and to reduce FP.

6. Conclusion and future work

In this study, we explored the effectiveness of machine learning algorithms in automating output-checking processes, focusing on both tabular and image data. Our results highlight the impact of incorporating HITL feedback, particularly evident in the improved performance of the LGBM model when guided by human expertise. Additionally, we demonstrated that machine learning algorithms, such as CNN, show promise in analyzing visual data, outperforming random classifiers in image data predictions. These findings underscore the critical role of integrating human feedback with machine learning algorithms to improve the accuracy and reliability of output decisions.

Furthermore, our study paves the way for exploring novel approaches to incorporating human feedback into machine learning models, such as active learning strategies. Extending our analysis to include a broader range of data types and domains would provide valuable insights into the generalizability and applicability of our findings. Future work could build upon our previous study Slokom et al.³ by incorporating explanations for the predictions made by the models. Providing such explanations would assist human chequers in understanding and adapting to the decision-making process, thereby enhancing their ability to make informed and critical decisions.

Footnotes

Acknowledgement

The views expressed in this paper are those of the authors and do not necessarily reflect the policy of Statistics Netherlands.

ORCID iD

Manel Slokom

Jel Vankan

Peter-Paul de Wolf

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

References

Bond

Brandt

de Wolf

. Data without boundaries: standalone document guidelines for output checking, 2013. [Online; accessed in 10 October 2022].

Domingo-Ferrer

Blanco-Justicia

. Towards machine learning-assisted output checking for statistical disclosure control. In: Proceedings of 18th International conference on modeling decisions for artificial intelligence: MDAI 2021, p.335–345. Berlin, Heidelberg: Springer-Verlag. ISBN 978-3-030-85528-4.

Slokom

de Wolf

Larson

. COACH: computer Assisted output CHecking with human-in-the-loop. U N Econ Comm Euro Conf Euro Statist 2023.

Rigaud

Marquier

Debonnel

, et al. Checking data outputs from research works: a mixed method with ai and human control. U N Econ Comm Euro Conf Euro Statist 2023.

Green

Ritchie

Smith

. Automatic checking of research outputs (acro): a tool for dynamic disclosure checks. ESS Statistical Working Papers 2021. Available at: https://uwe-repository.worktribe.com/output/7449840.

Cole

Smith

Albashir

, et al. Sacro: semi-automated checking of research outputs. U N Econ Comm Euro Conf Euro Statist 2023.

Xiao

Sun

, et al. A survey of human-in-the-loop for machine learning. Future Gener Comp Syst 2022; 135: 364–381.

Madono

Nakano

Kobayashi

. Efficient human-in-the-loop object detection using bi-directional deep sort and annotation-free segment identification. In: 2020 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) , Auckland, New Zealand, 7–10 December 2020, pp.1226–1233. IEEE.

Ristoski

Gentile

Alba

, et al. Large-scale relation extraction from web documents and knowledge graphs with human-in-the-loop. J Web Semantics 2020; 60: 100546.

10.

Mosqueira-Rey

Hernández-Pereira

Alonso-Ríos

, et al. Human-in-the-loop machine learning: a state of the art. Artif Intell Rev 2023; 56(4): 3005–3054.

11.

Meng

Finley

, et al. LightGBM: a highly efficient gradient boosting decision tree. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA, 4–9 December 2017, pp.3149–3157. Curran Associates Inc.

12.

Chicco

Jurman

. The advantages of the matthews correlation coefficient (MCC) over f1 score and accuracy in binary classification evaluation. BMC Genom 2020; 21(1): 1–13.