Integration of ICT survey data and Internet data from enterprises websites at the Italian National Institute of Statistics

Abstract

Since 2013, the Italian National Institute of Statistics (Istat) has been investigating the potential of Big Data sources for Official Statistics. Among such sources, Internet data originated by websites content has been considered as one of the most important to produce information about enterprises. In 2018, Istat started producing experimental statistics on the activities that enterprises carry out through their websites (web ordering, job vacancy advertisement, link to social media, etc.). They are a subset of the statistics currently produced by the “Survey on ICT usage and e-Commerce in Enterprises” and are computed starting from enterprise websites’ contents, acquired by web scraping tools and processed with text mining techniques. A machine learning approach is adopted to estimate models in the subset of enterprises for which the survey and the web sources are both available, with survey data serving as training set for the machine learning task. The content scraped from successfully reached websites is used as input to predict the target values by applying the model fitted in the previous step. The experimental statistics are obtained using two different estimators: (i) a full model based estimator; (ii) an estimator that combines model and survey based estimates. Considering the various domains for which they have been calculated, the three sets of estimates (survey, model and combined) in most cases are not significantly different (i.e. model and combined estimated values lay in the confidence intervals of survey estimates). Simulations have demonstrated that the Mean Square Errors of these new estimates are competitive as compared to those produced in the traditional way.

Keywords

Big Data Internet data web scraping text mining machine learning experimental statistics

1. Introduction

An increasing number of National Statistical Institutes (NSIs) are experimenting with the use of new sources of data in order to produce the same or new statistical information in a multi-source environment, more efficiently and with higher levels of quality [8]. In particular, Big Data (BD) are very appealing for Official Statistics. The general debate on BD is focused on volume, rapidity, variety and IT capabilities to capture, store, process and analyse them for statistical production. Additionally, NSIs may appreciate also other aspects, such as veracity (data quality, including selectivity and trustworthiness of the information) and validity (correct and accurate data for the intended use). Veracity and validity directly affect the accuracy (bias and variance) of the estimates. In order to improve veracity and validity, a multi-source approach (based on a combined use of survey, administrative and BD sources) is expected to overcome the limits of each single source, in particular those affecting BD.

This multi-source approach requires a shift in the paradigm of statistical inference. The traditional paradigm followed by NSIs usually involves design-based survey sampling theory and model-assisted inference. The new paradigm consists of algorithmic-based inference, where the emphasis is on the exploration of all available data, seeking information that has not been extracted yet; models are no longer evaluated based on their interpretability, but rather on their capability to correctly predict values at the unit level, and to use them for estimating the parameters of interest.

Among BD sources, Internet as a data source is gaining more and more importance in Official Statistics. National level projects on the use of Internet data have been carried out by some countries, including Statistics Netherlands and Istat. In particular, in [15] a first domain of experimentation was related to air tickets: the prices of air tickets were collected daily by Internet robots, developed by Statistics Netherlands supported by two external companies, and the results were stored for several months. The experiment showed that there was a common trend between the ticket prices collected by robots and existing manual collection. Two additional domains of experimentation were Dutch property market and clothes prices, the first exhibiting more regularity in the sites structure, the latter more challenging with respect to automatic classification due to lack of a standard naming of the items, and variability in the sites organization. Similarly, in Italy a scraping activity was performed to get on consumer electronics prices and airfares [13].

In 2016, the project ESSnet Big Data Pilots I [10] was funded by Eurostat and included two work-packages dedicated to Web scraping: (i) “Web scraping of job vacancies” and (ii) “Web scraping of enterprise characteristics”. In 2018, a second project was launched as follow-up of the first ESSnet, namely ESSnet Big Data Pilots II [11]. In this second project the two work-packages on Web scraping were upgraded from a pilot state to an implementation one.

Istat is conducting a project using Internet data to obtain a subset of the estimates currently produced by the yearly sampling survey on “Survey on ICT usage and e-Commerce in Enterprises” (in the following ICT survey), carried out by Istat and by NSIs of other member states in the EU. The technical results obtained within the project are reported in [5, 3, 6, 4, 7]. Target estimates of this survey include some characteristics of the websites that the enterprises use to communicate their business, e.g. web ordering facilities, job vacancy management, social networks accounts and activity. Data are collected by means of traditional questionnaires. An alternative way is to use Internet data, i.e. to collect data by accessing directly the websites, processing the collected texts to single out relevant terms, and modelling the relationships between those terms and the characteristics we are interested to estimate. To do that, the sample of surveyed data plays the role of a training set, useful to fit models that can be applied to the generality of enterprises that own a website. Administrative data (mainly contained in the Business Register) are used to cope with representativeness problems related to the BD source. The sequential application of web scraping, text mining and machine learning techniques obtains auxiliary variables, suitable for applying a prediction approach and producing estimates comparable to the survey-based ones.

In this paper, we provide an overall description of the methodological and technological solutions put in place to implement this project. In particular, in Section 2, we describe the generic architecture underlying the project; in Section 3, we detail the so called URL retrieval problem that we had to face to identify enterprises on the Web; in Section 4, we show the different steps performed to estimate the enterprise characteristics at micro-level, while in Section 5, we describe the estimation approach for final aggregates. Finally, Section 6 outlines quality aspects.

2. A generic architecture for Web textual data

Modern organizations have recognized the importance of having a defined and standard architecture that is able to guide the implementation of the organization’s vision and to harness external drivers and changes. However, in the view of considering a reference architecture for BD in Official Statistics, there are few concrete results in terms of available standards, possibly due to the recent attention that the BD phenomenon received by NSIs.

In the following we report the effort we made within the ESSnet Big Data Pilots I [10] to pose some initial bases towards European and International work on standardizing Big Data architectures for Official Statistics. In this respect, it is worth while citing the current work done within the ESSnet Big Data Pilots II [11]. Indeed, the ESSnet includes a dedicated workpackage “Process and Architecture” aimed at defining BREAL (Big data REference Architecture and Layers), which is a Big Data reference architecture for Official Statistics.

Figure 1 depicts a generic logical architecture for Web textual data.

Figure 1.

A logical architecture for Web textual data.

The Collect phase represented in Fig. 1 includes two sub-processes, namely: Internet access and Storage. Each sub-process has some logical blocks. The two ones for Internet access are URL searcher and Scraper. The ones considered for Storage are Retrieved URLs and Scraped content. In order to start a collection of data by scraping websites, firstly a list of URLs identifying the home pages of the sites to be reached is needed. If this is not available, there is the opportunity of setting up a dedicated URL retrieving activity (see Section 3).

The Preparation phase has two specific sub- processes, namely a Generic Preparation and a Task specific Preparation. The introduction of a task specific preparation phase is particularly interesting. As shown in Fig. 1, depending on the specific processing of the text, the preparation can include a Text encoding logical block, e.g. the traditional Bag of Words approach, a Language modelling block, e.g. recent Word Embeddings approaches or a Knowledge representation block, e.g. modelling domains through formal languages for ontologies. The distinction between Generic Preparation and Task specific Preparation is aligned with the “hourglass model” proposed within the reference architecture for Trusted Smart Statistics, which is under definition by Eurostat in collaboration with ESS members [12]. Indeed, the framework does include a D-layer that is task-agnostic and as such similar to Generic Preparation and an S-layer that is task-specific.

Finally, the Process phase for websites is based on the use of a Machine Learning approach, in which either Supervised or Unsupervised Machine Learning logical blocks can be considered (see Fig. 1). Supervised Machine Learning has the usual phases of setting up training and test sets, training and validating the learner and apply it to the input dataset.

For the Unsupervised Machine Learning there is a possible choice among logical blocks that are: Model-based Computation (Fit a model and Compute model predictions), e.g. for classification via mixture models or Algorithmic Computation (Tune, Validate and Run), e.g. for clustering or automated reasoning.

3. The URL retrieval problem and solution

As mentioned in the Introduction, Istat is currently experimenting how to integrate the sampling survey on “Survey on ICT usage and e-Commerce in Enterprises” with a data collection from enterprises’ websites.

A first preliminary problem was related to the retrieval of the URL (Uniform Resource Locator) for those enterprises in the target population that were supposed to have a website (almost three quarters of the total). Actually, for a subset of enterprises, the indication of corresponding websites was already available, resulting from the answers to a specific question that sampled enterprises gave in the ICT survey, and from some other sources integrated with the survey answers. However, this subset was partial and we needed to figure out a partially automated procedure to complete it at least at an acceptable quality level.

Figure 2.

Software solutions for the URL retrieval problem.

The procedure starts by searching the enterprise name through a Web search engine and involves several steps with a mix of techniques, ranging from scraping and crawling techniques to machine learning ones. Our results show the feasibility of addressing this problem with a partially automated solution that gets good results both in terms of quality and efficiency. More in detail, the ICT survey has a population of interest that is composed by enterprises with at least 10 employees and operating in different branches of industry and services (the population size is around 184,000). By the ICT survey estimates, it is known that about 72% of these enterprises own a website, used for different purposes. As said, we did not have the full list of URLs but only a subset. Searching the Web by enterprise names, we first retrieved for each enterprise name a list of possible URLs to be associated to it. Later, we set up a supervised machine learning with the aim to predict the correct association of a URL of the retrieved list with the name of an enterprise. To build the input training set we integrated three editions of the ICT survey (2015, 2016 and 2017) with the Italian National Business Register (ASIA) and with a private data source (Consodata), with enterprises information including websites, email addresses and telephone numbers. The resulting dataset was composed by 81,912 enterprises with URLs.

We fitted a logistic model and chose thresholds to apply to the whole population. In particular, there follow the results obtained by applying the logistic model to the 106,019 enterprises for which the URLs were not available:

26,097 (24.6%) URLs have been evaluated as reliable, and associated to the corresponding enterprises;

68,885 (64.9%) have been considered as erroneous, and excluded;

11,037 (10.4%) have been addressed to interactive controls.

In conclusion, at the end of the process we obtained a total number of identified URLs equal to about 105,000. If we consider a total amount of 133,000 websites pertaining to the enterprises population, we obtain a coverage of almost 80%, which can be deemed a satisfactory one. The reader interested to further details can refer to [6].

The technological solution that we implemented is shown in Fig. 2. As the reader can notice, the figure includes a subset of the blocks shown in Fig. 1. In particular, the Task specific preparation only includes the Text encoding and Dimensionality reduction blocks, being the other blocks unnecessary to the task objective. Let us also notice that the Process phase selects the Supervised Machine Learning among possible logical blocks present in Fig. 1.

We developed a software pipeline mainly made by custom programs in order to accomplish the task; the figure shows how the software solutions instantiate the logical architecture described by Fig. 1.

In particular we used the following software components:

ad-hoc developed software, i.e. URL Searcher, RootJuice and URL Scorer. In addition, some custom R scripts were developed for the machine learning task implementation. The developed software is available at http://github.com/ summaistat.

Apache Solr (http://lucene.apache.org/solr/) as an open NoSQL storing and searching platform.

4. Estimating enterprise characteristics from websites at micro-level

Once retrieved the URLs for the websites of interest, we considered the problem of estimating some characteristics of enterprises starting from such websites.

We envisioned three different use cases, namely:

Web Ordering, estimating if an enterprise performs Web ordering. Web ordering is a sub-type of Web sales that is in turn a sub-type of E-Commerce.

Presence in Social Media, estimating if an enterprise is present or not in a social media.

Job Vacancy, estimating if an enterprise exposes job vacancies on its website or not.

The logical and software pipeline put in place for the three use cases is shown in Fig. 4. This pipeline comes temporally after the pipeline shown in Fig. 2; indeed, this latter produces as output the data of the RetrievedURLs block.

In the following sections, we will describe the methodological and technological solutions adopted for the use cases, by first detailing the solutions for the Collect and Preparation phases (Section 4.1) and then for the Process phase (Section 4.2).

4.1 Collect and Preparation

The Collect and Preparation phases for the three mentioned use cases were the same ones. A relevant task of the Collect phase was Web scraping. Rather than being conducted as a “specific” scraping, where the structure of the website as well as the specific information to be scraped were known in advance, the task was instead conducted as a “massive” and “generic” one. More in detail, the Web scraping task had the following features: (i) the number of sites to be scraped was around one hundred thousand (massive feature), and (ii) the structure of the sites was not known in advance nor reproducible, given the heterogeneity of the sites’ structure (genericity feature).

The pipeline for performing massive web scraping of enterprise websites was revealed as a very complex one: it started from proper scraping activities, then involving data storage and a relevant data preparation before the final processing steps. In particular after that URL Searcher was run to retrieve URLs and store them as text files within the Retrieved URLs block, a Scraper component accessed the Web to scrape the content of the websites. The Scraper was implemented via the RootJuice software, ad-hoc implemented in Java. As illustrated in Fig. 4, in these use cases we scraped the enterprises’ websites and stored the scraped content in Solr. Basically we stored just the textual content of the scraped websites. We applied several data preparation operations on scraped texts by using (i) Solr built-in functionalities, (ii) freely available software (TreeTagger, see https://www.cis. uni-muenchen.de/∼schmid/tools/TreeTagger/) and (iii) custom software modules, thus finally generating a term document matrix, which is the main input for the subsequent Process phase.

4.2 Process

The subset of respondent enterprises to the ICT survey declaring a website, the content of which could be scraped (about 11,900), was considered as the ground truth, i.e. the basis for fitting models to be used as predictors of the target variables. Given the sampling nature of this dataset, it was expanded by using calibrated weights obtained from known totals of the population of enterprises in the ICT usage survey with websites (around 133,000). The resulting dataset was divided into a train dataset and a test dataset, both having equal size. Each record in these datasets contains, together with the values of the target variables reported in the survey questionnaire, also the terms of the term document matrix resulting from the previous steps.

For the three use cases, the Process phase was implemented by using either custom R scripts realizing Supervised Machine Learning or by custom Python scripts realizing Algorithmic Computation.

In the following, the detail of the computation of the estimates for each of the use cases are provided.

For the Web Ordering use case, a full machine learning approach was adopted. A number of different machine learning models (Logistic Model, Neural Networks, Classification Trees, Naïve Bayes, Support Vector Machine, Bagging, Boosting, Random Forest) were applied to predict the target variable Web Ordering (YES/NO). The best resulted to be Random Forest, with an accuracy equal to 0.83 and an F1-measure equal to 0.63.

On the basis of these results, deemed as good but not yet satisfactory, a decision was taken to select all survey cases in which a conflict between survey reported values and machine learning predicted ones was observed, and for each of them to proceed to a manual inspection of the corresponding websites, in order to decide what was the correct value. In about 50% of the cases the correct value was the predicted one. Once eliminated response errors in the training and test sets, the performance of the different learners resulted to be much higher. In particular, the Random Forest predictor resulted to be very good with an accuracy equal to 0.9 and an F1-measure equal to 0.73 (see Table 1), that is, much more than when not considering and treating response errors.

Table 1
Performance indicators related to the different learners applied for Web Ordering (yes/no)

Learner	Accuracy	Recall	Precision	F1-measure
Naïve bayes	0.84	0.56	0.56	0.56
Logistic model	0.84	0.57	0.57	0.57
Classification tree	0.87	0.64	0.64	0.64
Neural networks	0.88	0.65	0.66	0.66
Bagging	0.88	0.66	0.67	0.67
SVM	0.90	0.62	0.76	0.68
Boosting	0.90	0.71	0.71	0.71
Random forest	0.90	0.73	0.73	0.73

For the Presence in Social Media use case, an Information Retrieval based technique was applied. In particular, a direct search of relevant terms with regular expression (i.e. the names of the social media in their different possible formats) in the text was performed: if successful, prediction was positive, otherwise negative. The accuracy of these predictions on survey raw data was 0.72 (with an F1-measure equal to 0.73). Actually, the presence of response errors in these raw data was particularly high: once corrected, accuracy raised to 0.92 (same value for the F1-measure).

For the Job Vacancy use case, the same machine learning approach used for the Web Ordering use case was applied. The performance of the Random Forest learner was: accuracy 0.82 and F1-measure equal to 0.7. In this case, no error correction was carried out, however if it had been done, an analogous increase in performance should have been expected.

Let us remark the importance of having good quality levels at unit level. One possible use of the estimated values of enterprises’ characteristics is to enrich National Business Registers; in such a case the quality of results at micro-level, i.e. accuracy and F1-measure, should be guaranteed at a level good enough to be accepted for the statistical uses of the registers.

5. Estimation

The ICT survey produces on a yearly basis a set of estimates about rates of web ordering, job advertising and presence on social media, declared by enterprises that own or use websites. These estimates are available for the total population, and for different domains of interest, including:

Cross-classification by Size Classes of persons employed (4) and Economic macro sectors (4) (16 different sub-domains);

Administrative Regions (21 different domains);

Detailed economic activities (24 domains);

ICT and non-ICT enterprises.

The production of the current official estimates concerning the three variables of interest (web ordering, job vacancy and social media presence), is of the kind design-based/model-assisted:

$\displaystyle\hat{Y}=\sum_{k=1}^{r}y_{k}w_{k}$

where weights $w_{k}$ are obtained by calibration procedure of basic weights (inverse of inclusion probabilities), making use of known totals in the population in order to reduce the bias due to non-response and the variability due to sampling errors:

$\displaystyle\sum_{k=1}^{r}w_{k}=N_{U}$

Together with the current estimation method, alternative estimates have been calculated by adopting two different estimators: a full model based one and a combined one. In order to fully understand these estimators, it is important to consider the different subsets that can be defined inside the population of interest and that can be visualized in Fig. 3.

Figure 3.

Subsets of the population of interest.

Figure 4.

Software solutions for use cases Web Ordering, Presence in Social Media and Job Vacancy.

Inside the target population we can distinguish:

the subset $U_{1}$ of enterprises that own (or make use of) a website (the value 133,000 is an estimate from the 2017 ICT Survey);

the subset $U_{2}$ of enterprises that own a website, and for which it was possible to identify related URLs and to proceed scraping websites’ content;

the subset $U_{1}$ – $U_{2}$ of enterprises that own a website, but for which it was not possible to identify related URLs and to perform content scraping.

As for the sample, we can consider:

the subset $r_{1}$ of enterprises that answered to the 2017 ICT Survey declaring to own a website, and for which it was possible to identify URLs and to scrape websites’ content;

the subset $r_{2}$ of enterprises that answered to the 2017 ICT Survey declaring to own a website, but for which it was not possible to identify URLs or to proceed scraping related content.

Let us now introduce the estimators alternative to the current one.

The full model based estimator is of the kind:

$\displaystyle\hat{Y}=\sum_{U_{2}}\tilde{y}_{k}w^{\prime}_{k}$

that is, the estimate of the total number of enterprises offering web ordering facilities on their websites is given by the count of the predicted values for all units for which it was possible to reach their websites (population $U_{2}$ ), calibrated in order to make them representative of all the population having websites ( $U_{1}$ ):

$\displaystyle\sum_{U_{2}}w^{\prime}_{k}=N_{U_{1}}$

Table 2

Alternative estimates (% of enterprises offering web-ordering) for economic macro sectors and size classes

Economic sector	Size class	Design based	C.I.	C.I.	Model based	Combined
		estimates	Lower	Upper	estimates	estimates
Manufacturing	From 10 to 49	10.04	8.08	11.99	11.06	9.88
	From 50 to 99	12.09	8.87	15.30	14.80	14.29
	From 100 to 249	15.69	12.60	18.77	15.76	15.38
	From 250 and more	24.18	21.06	27.30	22.65	21.09
Electricity, gas steam, water supply,	From 10 to 49	8.69	6.54	10.84	9.73	11.51
sewerage, etc.	From 50 to 99	10.50	5.98	15.03	11.55	9.73
	From 100 to 249	13.89	8.95	18.84	15.04	11.79
	From 250 and more	18.79	11.86	25.72	16.97	14.55
Construction	From 10 to 49	2.92	2.03	3.81	5.54	5.02
	From 50 to 99	3.10	0.29	5.91	5.32	4.28
	From 100 to 249	2.05	0.30	3.81	5.19	5.19
	From 250 and more	8.12	1.09	15.16	10.00	8.75
Total non financial services	From 10 to 49	20.28	18.26	22.30	20.26	18.40
	From 50 to 99	21.76	18.36	25.16	19.36	17.68
	From 100 to 249	21.76	19.03	24.48	20.89	20.82
	From 250 and more	28.32	25.56	31.07	24.85	31.51

Table 3

Alternative estimates (% of enterprises offering web-ordering) for economic sectors

Economic sector	Design based	C.I.	C.I.	Model based	Combined
	estimates	Lower	Upper	estimates	estimates
Not included in ICT Sector	15.13	13.94	16.31	15.54	14.25
Included in ICT Sector	10.97	8.17	13.77	14.88	13.65
Food, beverages and tobacco	19.40	12.86	25.94	17.04	14.82
Textiles, apparel, leather	16.05	9.20	22.91	13.85	11.93
Wood and paper	2.45	6.55	18.36	13.21	11.30
Petroleum, chemicals and others	10.44	6.85	14.02	11.78	11.73
Metal products	5.94	3.02	8.85	7.65	7.25
Computer, electronic and optical	9.47	4.94	13.99	11.98	9.73
Electrical equipment	5.62	2.86	8.38	10.45	8.88
Transport equipment	–	3.04	30.32	12.49	14.72
Furniture, other manufacturing	8.84	4.57	13.11	11.79	11.27
Electricity, gas, water supply	9.87	8.03	11.71	10.77	11.50
Construction	2.94	2.07	3.81	5.54	5.00
Wholesale of vehicles	20.39	18.98	21.81	20.32	20.28
Transport and storage	14.16	6.57	21.75	11.47	10.90
Postal and courier activities	26.13	16.37	35.89	14.16	18.26
Accommodation	82.57	77.37	87.78	71.77	68.71
Food service activities	23.63	14.59	32.67	22.23	15.48
Publishing activities	62.00	45.62	78.39	49.21	49.44
Motion picture, video production	15.62	2.97	28.27	23.63	17.64
Telecommunications	21.45	13.71	29.19	20.44	20.80
IT and information services	8.82	5.65	11.99	12.89	12.45
Real estate activities	11.08	5.78	16.39	13.68	13.99
Professional and technical act.	5.27	1.57	8.97	10.33	7.50
Administrative and support act.	4.83	2.93	6.73	8.40	7.33
Travel agency, tour operator	44.20	31.80	56.59	44.19	47.71

Table 4

Alternative estimates (% of enterprises offering web-ordering) for NUTS2 regions

Region	Design based	C.I.	C.I.	Model based	Combined
	estimates	Lower	Upper	estimates	estimates
Piemonte	11.96	7.46	16.46	13.77	13.60
Valle d’Aosta	16.80	6.43	27.17	21.77	20.76
Liguria	11.39	5.96	16.83	14.86	12.02
Lombardia	11.76	10.42	13.10	14.38	13.07
Bolzano	31.43	24.93	37.92	29.38	26.64
Trento	19.51	16.87	22.14	22.78	23.21
Veneto	14.72	12.22	17.22	16.67	15.80
Friuli	17.17	5.62	28.73	14.23	14.67
Emilia-Romagna	12.63	9.89	15.36	15.00	14.90
Toscana	14.55	10.30	18.80	15.91	14.35
Umbria	24.23	20.35	28.10	16.34	15.43
Marche	20.37	7.51	33.23	16.58	14.04
Lazio	16.62	12.47	20.77	16.02	13.79
Abruzzo	17.41	9.08	25.74	13.87	14.23
Molise	14.06	4.08	24.03	12.41	15.17
Campania	15.87	10.82	20.91	14.40	14.33
Puglia	20.32	14.46	26.18	14.61	12.21
Basilicata	12.02	4.34	19.70	13.78	8.34
Calabria	20.40	10.93	29.87	17.47	10.05
Sicilia	19.17	6.95	31.40	16.70	12.56
Sardegna	14.00	7.85	20.14	14.93	15.29
TOTAL	14.97	13.81	16.13	15.51	14.22

That is, estimates are produced by summing three components:

The counting of predicted values in the sub-population of units for which it was possible to scrape and process corresponding websites;

An adjustment based on the consideration of the differences between the reported values and the predicted values (expanded to the same sub-population);

The counting of observed values for the respondents that declared a website, that was not found nor scraped, expanded to the whole sub-population.

Calibration is performed by using the administrative data available in the Business Register (number of employees, economic activity, territorial localization).

A more complex estimator is the combined one:

$\displaystyle\hat{Y}=\sum_{U_{2}}\tilde{y}_{k}+\sum_{r_{1}}(\tilde{y}_{k}-{y}_% {k})w^{\prime\prime}_{k}+\sum_{r_{2}}{y}_{k}w^{\prime\prime\prime}_{k}$

where $\sum_{r_{1}}w^{\prime\prime}_{k}=N_{U_{2}}$ and $\sum_{r_{2}}w^{\prime\prime\prime}_{k}=N_{U_{1}-U_{2}}$ .

Point estimates produced by the current ICT survey with respect to the different domains of interest, together with their 95% confidence intervals, are reported in Tables 2–4.

A visual representation of estimates for the four different domains of interest is reported in Fig. 5. In each plot, together with point estimates, 95% confidence intervals are represented by the dotted lines. Most model based and combined estimates are internal to these intervals: considering the total amount of 64 different estimates, in 13 cases model based estimates are external, and again in 13 cases the same happens for the combined ones (8 of these cases refer to the same estimates, other 5 refer to different ones). It is worth while noting that even when model based or combined estimates are external to the intervals, they are not far from their boundaries (this is clearly visible in the plots).

Figure 5.

Estimates in the different domains of interest.

6. Quality evaluation

A complete comparison of all the produced estimates is available on the pages dedicated to the “Experimental statistics” produced with Big Data sources on the official website of Istat, at the link https://www. istat.it/en/archive/216641. These include not only those regarding web ordering, but also job vacancy and presence in social media.

A first analysis of the estimates related to web ordering, job vacancy and presence in social media rates, obtained with the two alternative estimators, compared to the estimates produced by the official survey, allows some preliminary conclusions.

The three different sets are not incoherent, as most of the model based and combined estimates lay internally to the confidence intervals of the design based estimates.

Looking at coherence as one important dimension of quality, both combined estimates and full model based estimates can be considered as equally acceptable. But two considerations can be made, namely:

The second component of the combined estimator is based on an assumption of perfect correctness of reported values, and considers predicted values as errors when they do not coincide with the reported ones. But, as already mentioned, controls have been carried out when fitting models, and in half of the cases in which predicted values were contradictory with reported ones, this was not due to model fault, but to response errors. So, this assumption does not always hold. In any case it would be advisable to deepen this phase also by returning to the respondents to verify if it is an error in response or if, for example, the model has evaluated the content of a site different from that one considered by the respondent.

In order to limit the respondent burden, it is possible that in next rounds of the ICT survey the entire “website section” will become optional, so the use of the combined estimator (produced using also observed values of the survey) would be more problematic and the full model based estimators might remain the only alternative.

The main flaws of the model based estimator are due to:

prediction errors;

undercoverage of the population of enterprises owning websites, part of which has not been reached by web scraping.

As for the first, taking into consideration the presence of response errors in the test set, once eliminating them by manual inspection, the accuracy for the model based estimation increases to acceptable levels (around 90% for web ordering, about the same for the other two variables), in any case comparable with the accuracy for the design based estimation.

Figure 6.

Distributions of quality indicators calculated for the different estimates of web ordering rates by considering the 16 different sub-domains identified by size classes of persons employed (4) and economic macro-sectors (4).

As for the second, pseudo-calibration allows to limit the bias, especially when the difference in the values of the parameters in the two sub-populations is not high, as it is in this case.

6.1 Simulation study

In order to try to quantify the accuracy related to the three different sets of estimates, a simulation study has been carried out (described in detail in [1, 2]).

This simulation study is based on the real data from the 2017 round of the Istat ICT survey, and makes use of a super-population approach: a synthetic enterprise population with websites is generated in each iteration, while the sample (the one corresponding to the design of the ICT survey) remains the same in terms of selected units. Data in the population are generated trying to replicate the joint distributions observed in the survey, in particular those related to the variables that define the domains of interest when calculating the estimates. As target variable, the web ordering (yes/no) has been considered. In the simulation, the different sources of bias for the design based and for the model based have been carefully taken under control.

For the model based estimator, the following two sources of bias have been considered:

the different behaviour of enterprises whose websites have been successfully scraped and those that have not: a difference of about 2 percentage points in favour of the former has been introduced, the same that has been detected in the reality;

the prediction capability of the model, measured in terms of F1-measure, whose value is close to the one actually observed in the reality (about 70%).

Instead, for the design based the bias factors considered are:

the non-ignorable non-response model, where the explanatory variable is the dimension of the enterprise, measured by the number of employees;

the presence of response errors, that in case of web-ordering is determined by a tendency to over-report it (potential positive bias).

Simulation has been carried out distinctly with respect to two domains:

size classes of persons employed (4) and econo- mic macro-sectors (4) (16 different sub-domains);

NACE two digits categories (62 levels).

The reason is in the fact that the first domain is obtained with the same variables that have been used for the ICT survey sample design; while the second domain is not a planned one, with a high number of sub-domains, some of which are with a small number of population units, and a very small number of survey respondent units.

Estimates have been calculated in each iteration by using the three different estimators. Their values have been retained together with the “true” values of the parameters in the generated population. At the end, for each estimate the following quality indicators have been calculated:

the coefficient of variation (CV) measuring the variability component of the MSE;

the relative bias (BIAS) measuring the bias component;

the relative root square mean error (RRMSE) measuring the overall error.

Figure 7.

Distributions of quality indicators calculated for the different estimates of web ordering rates by considering the 62 different sub-domains identified by NACE (two digits).

Figure 8.

Improvement of new calibration on CV’s of estimates by NACE (24 aggregations).

Results are reported in Figs 6 and 7.

Considering Fig. 6, it can be seen that as for the variance component of MSE, the model based estimator performs much better than the other two (as one might expect), while as for the bias the combined one is the best (also in this case it was to be expected, thanks to the third component of this estimator). Looking at the relative root MSE, there is no clear evidence of what estimator is the best.

Considering instead Fig. 7, again the model based estimator is the best in terms of variance, while there is no clear evidence of what is the best estimator with respect to bias and total MSE.

6.2 Using model based estimates in calibration

It is worth noting that there is another (indirect) positive impact on the quality of the ICT survey estimates. It is when we make use of model based estimates as known totals in the calibration step preliminary to the production of these estimates.

For example, if we introduce the model based estimates related to web ordering as known totals in the calibration, we can observe a positive effect in terms of reduction of the variance of the estimates related to variables correlated to web ordering. To evaluate this positive effect, we considered four different variables in the 2017 ICT survey questionnaire:

‘Did the enterprise sell products or services using the website or apps in 2016?’ (e_awsell)

‘If yes, the geographic area of customers was in EU countries (Italy excluded)?’ (e_awseu)

‘Did the enterprise sell products or services using EDI (Electronic Data Interchange) in 2016?’ (e_axsell)

‘Did the enterprise buy products or services by using websites or apps of other enterprises or by using EDI in 2016?’ (e_aebuy)

We calculated the Cramer value indicating the correlation degree with web ordering for each one of the above variables, and considered the values of the coefficients of variation of the estimates obtained for the estimates (by the 24 values of NACE 2 digits) produced without and with the introduction of the estimates of web ordering in the calibration step.

Figure 8 illustrates the improvements (relative reduction) in the CV’s of the four variables estimates after introducing as known totals also the model based estimates of web ordering: for the two variables that are significantly correlated with web ordering (as e_awsell and e_awseu with a Cramer value, respectively, of 0.66 and 0.47), improvements are evident, while for the other two variables (e_axsell and e_aebuy) there are no relevant improvements (though not even worsening).

7. Conclusions and future work

The complex process that Istat developed to harness Internet data in order to improve the estimates currently produced by the Istat ICT survey led to a set of experimental statistics published on Istat official website.

As described in the paper, most of the software used to produce such statistics is publicly available and can be used and tested. Any kind of feedback on the available software is really appreciated.

An important issue that emerged is related to measurement errors in the survey data. There is evidence of a significant incidence of errors, resulting from manual investigation of a set of websites where predictions are contradictory with values reported in the questionnaire. Errors have a relevant impact on both the fitting of models (errors in the training set) and on the evaluation of their performance (errors in the test set). Failure in taking into account those errors leads to an underestimation of variance and bias components of the design based estimators.

Among the estimation procedures considered in the simulation exercise, the alternative estimators (model based and combined) seem to be competitive compared to the design based one, but there are still margins of improvement that can lead to an increase in the quality of the model based procedures. The simulation confirms that one of the most important areas where investing is an increase of the coverage of the population of enterprises with a website. So far, URLs retrieval and web scraping have been based on a variety of sources and techniques, which are constantly being improved. In fact, the number of enterprises whose websites have been successfully scraped in 2018 raised to about 101,000 (from 85,000 in 2017). Actually, a viable solution would be to proceed to a census collection of this information, by asking website URLs in every survey, and offering the opportunity to communicate this information on the Istat Enterprises Portal. A general agreement on this has been already reached. Of course, a higher coverage of the population websites will reduce the bias represented by the websites with unknown or incorrect address. Informing the enterprises that their websites will be accessed and their content collected for statistical purposes will also increase the number of websites successfully accessed for scraping.

The amount of valid information can also be increased by adding to the HTML text also the information derived from images, by using Optical Character Recognition (OCR) techniques: an application to this purpose has already been developed and tested. This will increase the performance of the predictors, and consequently reduce the MSE of the estimators. In addition, more recent techniques to represent textual data are being used to provide more precise representation of the information content of websites. Such techniques are based on the use of Word Embeddings, which is a powerful tool to represent unstructured data (see e.g. [9]).

In general, the methods and the solutions adopted in the different steps of the pipeline should be considered as promising results, though they do need to undergo further steps of evaluation and possible improvement.

The work done so far can go beyond the production of usual statistical information and address new information products. An example in this direction is provided by Statistics Netherlands [14], where scraped data are used to monitor the “Internet economy”. In the short term, Istat has planned to investigate web sales of enterprises by adding to web scraping of enterprise websites also other sources: e-marketplaces, apps, social shopping (for example, Instagram Shopping Cart and Facebook Marketplace).

Finally, activities are already underway to predict characteristics at the unit level for the whole population of enterprises in order to enrich the information in the Business Register. These activities are carried out at European level within the ESSnet Big Data Pilots II.

Footnotes

Acknowledgments

The work presented in this paper is the result of a collaboration with many Istat colleagues, namely: Fabiana Bianchi, Gianpiero Bianchi, Natalia Golini, Alessandra Nurra, Paolo Righi, Sergio Salamone, Francesco Scalfati, Donato Summa, Diego Zardetto. In addition, part of the work has been carried in collaboration with Sapienza University of Roma, namely Renato Bruni, and CINECA, namely Marco Scarnò.

References

Barcaroli,

Golini,

, and Righi,

(2017). Quality issues when using big data in official statistics. Proceedings of the SIS 2017. Statistics and Data Science: new challenges, new generations, Florence 2017.

Barcaroli,

Golini,

, and Righi,

(2018). Quality evaluation of experimental statistics produced by making use of big data. Proceedings of the European Conference on Quality in Official Statistics, Krakow 2018.

Barcaroli,

Nurra,

Salamone,

Scannapieco,

Scarnò,

, and Summa,

(2015). Internet as data source in the istat survey on ict in enterprises. Austrian Journal of Statistics, 44.

Barcaroli,

Nurra,

Salamone,

Scannapieco,

Scarnò,

, and Summa,

(2016a). Machine learning and statistical inference: the case of istat survey on ict. Proceedings of the 48th Scientific Meeting Italian Statistical Society, Salerno 2016.

Barcaroli,

Nurra,

Scarnò,

, and Summa,

(2014). Use of web scraping and text mining techniques in the istat survey on information and communication technology in enterprises. Proceedings of the European Conference on Quality in Official Statistics, Wien 2014.

Barcaroli,

Scannapieco,

, and Summa,

(2016b). On the use of internet as a data source for official statistics: a strategy for identifying enterprises on the web. Rivista italiana di economia, demografia e statistica, LXX, 20: 41.

Bianchi,

Bruni,

Scalfati,

, and Bianchi,

(2017). Text mining and machine learning techniques for text classification, with application to the automatic categorization of websites. Advisory Board Italian National Institute of Statistics.

Citro,

C.F.

(2018). From multiple modes for surveys to multiple data sources for estimates. Survey Methodology, 40, 137–161.

De Fausti,

De Cubellis,

, and Zardetto,

(2018). Word embeddings: a powerful tool for innovative statistics at istat. In JADT-International Conference on Statistical Analysis of Textual Data.

10.

ESSnet Big Data I (2016). https://webgate.ec.europa.eu/fpfis/mwikis/essnetbigdata/index.php/Main_Page.

11.

ESSnet Big Data II (2018). https://webgate.ec.europa.eu/fpfis/mwikis/essnetbigdata/index.php/Main_Page1.

12.

Eurostat (2019). https://www.unece.org/fileadmin/DAM/stats/documents/ece/ces/2019/ECE_CES_2019_32_Eurostat.pdf.

13.

Giannini,

Lo Conte,

Mosca,

Polidoro,

, and Rossetti,

(2014). Web scraping techniques to collect data on consumer electronics and airfares for italian hicp compilation. In Q2014 European Conference on Quality in Official Statistics.

14.

Oostrom,

Walker,

Staats,

Slootbek-Van-Laar,

Ortega-Azurduy,

, and Rooijakkers,

(2017). Measuring the internet economy in The Netherlands: a Big Data analysis. CBS Discussion Paper Netherlands Central Bureau of Statistics.

15.

Ten Bosh,

, and Windmeijer,

(2014). On the use of internet robots for official statistics. In MSIS-Management of Statistical Information Systems.

Integration of ICT survey data and Internet data from enterprises websites at the Italian National Institute of Statistics

Abstract

Keywords

1. Introduction

2. A generic architecture for Web textual data

4.1 Collect and Preparation

4.2 Process

Table 1 Performance indicators related to the different learners applied for Web Ordering (yes/no)

7. Conclusions and future work

Footnotes

Acknowledgments

References

Table 1
Performance indicators related to the different learners applied for Web Ordering (yes/no)