Engendering the measurement of informality – statistical standards and methods

3.1 Design of the pilot studies

Undertaking pilot studies in parallel to (or based on) the revision of statistical standards brings major benefits. Such tests create the opportunity to develop and try different measurement approaches, reflecting proposals emerging from the standard setting discussions, while also allowing reconsideration of those proposals in the light of evidence on the measurement challenges they would create.

ILO accumulated experience in pilot testing to apply latest statistical standards through several rounds of studies, the first of which was initiated in 2015 to assess methods to apply the standards adopted at the 19^th ICLS, initially with a central focus on those elements that would have the greatest impact on key labour market indicators, but also to measure some of the forms of work newly identified in the forms of work framework ^iii. .

This first round of pilot studies was followed up through several additional phases of tests building incrementally on guidance developed from the first phase which included model LFS questionnaires and supporting guidance that were launched in 2018 ^iv. . These new rounds of studies expanded the focus to additional forms of work including volunteer work ^v. and own-use provision of services ^vi. , as well as the implementation of ICSE-18 and a study focussed on the implementation of the 19^th ICLS standards in different types of household surveys ^vii. .

The studies generally used a similar range of methodologies to apply a mix of qualitative and quantitative testing approaches. Each phase included cognitive testing as a primary method of qualitative assessment of questionnaire content, in some cases supplemented by additional qualitative assessments through focus groups or rapid ethnographic assessments.

Qualitative tests were used to refine questionnaires that were subsequently used for quantitative tests of different scale, but generally with a split-sample methodology whereby different sets of content were tested on different samples of households selected with the same sample design.

As proposed by Presser et al., ¹¹ split-sample tests can be used to compare the outcomes of different survey questionnaires if all aspects of the sampling, methodology and implementation, other than the questionnaires, are the same. The analytical approach follows those proposed by Fowler ¹² whereby statistics are compared for the concepts covered by both questionnaires. Where differences in key selected outputs are observed, further analysis is used to attempt to identify the source. This comparative approach has been found to be valuable in several experiments to identify potential improvements to questionnaire content ^viii. .

Under the Engendering Informality Statistics project it was initially envisaged to undertake multiple rounds of qualitative testing in two countries, followed by quantitative testing in one country. The country selected for both qualitative and quantitative testing was Uganda, given a desire to undertake testing in a country with a high prevalence of informal employment and a diverse range of informal activities in both urban and rural areas. The additional country selected for qualitative testing was Peru, a country with a lesser albeit still high prevalence of informality. As the project progressed sufficient funding became available to also enable a round of quantitative testing in Peru meaning that the scope of studies was ultimately expanded with the significant added value of more detailed assessment possibilities across two settings ^ix. .

Before the qualitative testing could take place the ILO developed questionnaires for testing covering the latest standards and available model questionnaires, with the addition of a range of topics selected based on reviews of available literature, consultations and existing surveys on informality such as mixed surveys ^x. . The range of topics selected on this basis were:

i.

Identification of informality in line with emerging proposals from the working group

ii.

Identification of dependent contractors (based on ICSE-18 as established in Resolution I of the 20th ICLS)

iii.

Identification of contributing family workers (ICSE-18) plus measurement of their motivation for working in a family business

iv.

Motivation of independent workers (as defined by ICSE-18) for operating a business

v.

Decision making in family businesses – this is related to ICSE-18 but also relevant to the understanding of agency

vi.

Earnings of “independent workers” as defined by ICSE-18

vii.

Earnings of “dependent workers” as defined by ICSE-18

viii.

Asset ownership in business (types and valuation)

ix.

ICT use in businesses (including digital platforms)

Additionally – later in the project it was decided to add the topic of access to finance among business operators but this was only included for the last round of quantitative testing in Peru.

The qualitative testing work took place between August and December 2021 based on the timeline presented in Table 1 below:

OPEN IN VIEWER

Table 1.

Key milestones and dates for the qualitative testing.

	Uganda	Peru
Round 1 of cognitive interviews
Training	10–16 August 2021	28–30 September 2021
		(training trainers)
		11–16 October 2021
		(training interviewers)
Fieldwork	17–28 August 2021	18–23 October 2021
Focus Group Discussions and Individual Interviews
	10–13 November 2021	Not applicable
Round 2 of cognitive interviews
Training	25–27 November 2021	30 Nov-1 December 2021
		(training trainers)
		6–9 December 2021
		(training interviewers)
Fieldwork	2–11 December 2021	13–19 December 2021

Source: Statistical Methodology Series 9 - Engendering informality statistics - Overview and methodology https://www.ilo.org/media/481621/download

Cognitive interviews were based on purposive samples of 40 participants per round of interviews for a total of 80 interviews in each country (160 overall total). The sample design targeted similar numbers of women and men in both urban and rural areas, covering different types of jobs, but with a primary focus on informal work. Age and education levels were used as secondary criteria. Unlike in the first round of ILO pilot studies only employed individuals were targeted as the testing focussed on questions that would be asked to employed respondents only (e.g., identification of informal employment).

Following the cognitive interviews various updates were made to generate a complete questionnaire for quantitative testing. Three rounds of quantitative testing took place in the second half of 2022, two waves in Uganda and one in Peru. In the case of Uganda the same households were revisited for the second wave in order to enable longitudinal analysis.

Purposive sampling was used for the quantitative phases with districts/regions for testing selected to ensure a mix of urban and rural areas, but with oversampling in urban areas with the intention to cover a wide variety of informal jobs and businesses, while still covering the main agricultural crops in each country. In Uganda, over 1600 households were interviewed in wave 1 and roughly the same number in wave 2. The test in Peru was a smaller scale with just over 800 households interviewed (see Table 2).

OPEN IN VIEWER

Table 2.

Designed and achieved sample of quantitative tests plus fieldwork timing.

	Number of households
	Approach A	Approach B	Total	Average household size	Proportion of households with a female reference persons	Average age of reference persons	Proportion of respondents in working age	Number of respondents of working age	Timing (2022)
Uganda Round 1	824	830	1654	4.6	0.30	45.4	0.61	4680	July
Uganda Round 2	850	786	1636	4.8	0.27	45.2	0.58	4529	October
Peru	407	413	820	3.6	0.55	48.0	0.71	2134	Oct-Nov
Total	2081	2029	4110	4.5	0.34	45.9	0.61	11,343

Source: Statistical Methodology Series 9 - Engendering informality statistics - Overview and methodology https://www.ilo.org/media/481621/download

Two slightly different questionnaires were tested in each round referred to as approach A and approach B. The starting point of the questionnaires were ILO's published model labour force surveys (job-type start ^xi. ) with some adjustments plus the addition of topics considered to be of high relevance to the study of informality and gender. The large majority of questions were identical between approach A and B in order to focus analysis on the key selected topics - consequently content differed for about 10% of the questionnaires or approximately 30 questions.

While the samples of households for the quantitative tests were selected on the same basis it was nonetheless possible that differences in the achieved samples of individuals could impact the comparability of estimates between approach A and approach B in each round of the tests. To minimise the impact of this it was decided to apply a calibration approach whereby the benchmark population was an average between the two samples disaggregated by age group, location (urban/rural in both countries and additionally district in Uganda) and sex. In practice weights applied were close to 1 as the achieved samples were reasonably balanced but nonetheless the weighting reduced potential sources of incomparability and is considered to have been worthwhile.

A further feature of the studies was a dedicated test of the impact of proxy interviewing. This test was implemented in the second round of quantitative collection in Uganda and the round that was implemented in Peru. The objective was to achieve both a direct and proxy interview for at least one household member in each household (for households with at least 2 members). Comparisons of the information collected was expected to provide insight on the impacts proxy reporting had on various estimates. The proxy test was completed with the benefit of updates to the CSPro applications, whereby the software identified relevant household members for which the proxy could be completed. Clear protocols and training for the interviewers also supported implementation with the result that 591 successful test interviews were completed in Uganda and 530 in Peru, covering a range of respondent types.

4. Main findings from the pilot studies

The detailed findings on the different topics listed in the previous section are being incrementally published by the ILO and are extensive. Some key issues of priority have been selected for presentation in this paper.

Links to previous studies and interrelation of topics

Before assessing the results from the pilot studies in Uganda and Peru, a first point to highlight is that, while the different studies undertaken over time have had a different focus, the topics covered are inevitably interrelated. As one example, the first round of ILO pilot studies that commenced in 2015 and the study in Sri Lanka demonstrated that care is required to ensure that employment, as defined in the 19^th ICLS resolution is captured comprehensively and accurately.

The first round of studies demonstrated that, in the countries covered in the studies, multiple questions are needed to identify employment activity in the reference week and that women's employment in particular was at risk of being undercounted if too few questions were used. ²¹ This is a finding that has also been reflected in other studies.^15,22–25

The Sri Lanka study further highlighted this point whereby two different questionnaires yielded a more than twenty percent difference in measured employment among women – despite both questionnaires being designed to apply the 19^th ICLS standards. Detailed analysis showed the issue to relate to differences in the identification of casual and part-time work, or work helping in family businesses and farms – with almost identical estimates for full-time employees. ²¹ In a subsequent round of testing, updates made to the questionnaire previously identifying less employment succeeded in narrowing the gap to approximately six percent. These updates took the form of dedicated recovery questions targeting help in family businesses or farms, or more casual work, already incorporated in the model LFS questionnaire.

Furthermore, the identification of informality, and the questions used to identify informal employment and the informal sector are directly related to the ICSE-18 category, as specific and separate operational definitions of informality are provided for different ICSE-18 categories. As such the work done in Sri Lanka to refine approaches to the application of ICSE-18 is inexorably linked to the measurement of informality, and thus informed the design of the studies in Uganda and Peru.

This emphasises that the different standards adopted in ICLSs since 2013 are linked, as is their proper application in household surveys. For example, if there is a significant misreporting of the level of employment, as shown for one of the questionnaires used in the Sri Lanka study, this can bias substantially estimates based on ICSE-18 or informality where the under-reporting is typically concentrated among those with casual jobs, contributing family workers or those in informal employment – in particular misrepresenting differences between women and men. Furthermore, the application of the 19^th ICLS definition of employment requires the assessment of the main intended destination of production for those engaged in the production of goods on their own-account. Where the main intended destination of production is own-use then the activity is considered own-use production work, not employment – and in terms of informality under the new standards is considered to be part of the household, own-use and community (HOC) sector – not the informal or formal sectors.

Considering this, a holistic approach is needed to questionnaire development and testing, ensuring to apply all sets of standards and good measurement practices across the survey. With this said the next section of this paper focusses on the main findings from the studies in Uganda and Peru.

4.1 Identification of informality

Given that the topic of the 21^st ICLS Resolution is statistics on the informal economy a key initial focus of the studies in Uganda and Peru was the set of questions used to identify informal employment and the informal sector. A review of practices undertaken by the ILO in preparation for the 20^th ICLS had shown a degree of convergence across countries (although nowhere near complete) of practices in criteria that were operationalised in the identification of informality ^{26 xii.} .

On the basis of this review initial discussions in the technical working group for the development of the new statistical standards sought to narrow down the key criteria that would be considered for recommendation. Through these discussions registration of the enterprise, institutional sector and the types of accounts maintained emerged as key proposed criteria for the identification of the formal/informal sector, as well as informal/formal employment among independent workers based on ICSE-18. Similarly, the key proposed criteria to identify informal/formal employment among dependent workers would be payment of job-related social contributions, access to paid annual leave and access to paid sick leave. For dependent contractors something of a hybrid approach was ultimately adopted in the standards, looking at a combination of registration and social contributions as a way to identify informality/formality.

The studies in Uganda and Peru therefore looked at testing approaches to the operationalisation of these criteria – initially through cognitive testing and subsequently through the quantitative tests.

Cognitive testing findings:

By and large the questions related to social contributions, registration, incorporation, book-keeping, paid annual leave and paid sick leave were generally found to be well understood by respondents. However, some findings did emerge that generated changes in the questionnaire content for quantitative testing, including:

It was found that in general relatively greater difficulties were reported by respondents with the understanding of incorporation than registration. ²⁷ In both cases naming specific systems at the national level helped but beyond this some respondents indicated a lack of awareness of incorporation. In existing published ILO model questionnaires the question on incorporation had preceded the question on registration. It was decided to reverse the order of these questions for the quantitative tests, only asking the question on incorporation for those who said that the enterprise was registered – hopefully lowering the instance of don’t know responses with it being relatively more likely that if a respondent knew the enterprise was registered they may also know if it was incorporated. This change was ultimately found to operate well in quantitative tests and has been retained in newly published model questionnaires.

Through cognitive testing a potential difficulty was identified with the understanding of ‘paid sick leave’ with the possibility that respondents may say yes if medical expenses were paid by their employer. On this basis it was decided to test two wordings of this question during quantitative tests, one a traditional wording asking about ability to take paid sick leave (Would you get paid sick leave in case of illness or injury?), the other applying a reverse logic asking whether pay would be reduced if they were on sick leave (Would you lose pay if you could not work due to illness or injury?). ²⁷

During the quantitative testing assessment in most cases focussed on the perceived plausibility of results as well as feedback and observation of operation in the field. Almost universally questions were found to work well but with some notable reflections influencing the implementation of the questionnaire and data use. To give an example of an important finding it was clear that ‘don’t know’ responses were more common for certain questions related to informality and for certain groups. This is demonstrated in Figure 1 below which presents the responses received to questions on business registration by both independent and dependent workers.

OPEN IN VIEWER

Figure 1.

Distribution of responses to questions relating to business registration by country, type of workers in employment and by type of response (direct or proxy). Source ²⁷ : Statistical Methodology Series 11 - Identification of informality through labour force surveys – Frosch et al., 2023: https://www.ilo.org/publications/statistical-methodology-series-11-identification-informality-through-labour.

As shown in Figure 1 don’t know responses were minimal for independent workers who directly responded to the survey (0.6% in Uganda and 0.7% in Peru). However, rates of don’t know responses among independent workers became non-negligible (7.5% in Uganda and 9.2% in Peru) when proxy response was involved. Don’t know responses were more common among dependent workers – for example 12.5% even for direct interviews in Peru – illustrating the unsurprising lesser knowledge of employees of the status of the enterprise they are working in relative to the business operator. Don’t know frequency became particularly notable for dependent workers when proxy response was involved (28.5% in Peru).

This pattern was repeated to different degrees across informality related questions, being even greater for bookkeeping related questions, while somewhat lesser for questions to identify informality among employees (payment of social contributions, paid annual leave, paid sick leave). Further insight on this was drawn from the dedicated test of proxy effects illustrating that proxy response could lead to misreporting in different directions also confirming the higher prevalence of don’t knows in proxy versus direct interviews.

Multiple related conclusions can be drawn from this including but not limited to:

While proxy response remains an important mechanism to manage the cost and resource requirements for household surveys efforts should be made to reduce its prevalence to a reasonable degree, for example a limited number of revisits to increase the number of direct interviews. The impact of proxy response differs across topics, and in the case of informality likely to be of greatest impact in the identification of the formal and informal sector among dependent workers. Impacts on the sector appear to be less among independent workers, which is of note given that for independent workers the sector also determines whether the job is formal or informal.

Related to this it is important to have a strategy in place to allow classification of cases where don’t know responses are received or more likely to be received. As a concrete example the new standards adopted at the 21^st ICLS state that an enterprise will be formal if it has at least one formal employee – allowing an efficiency in questionnaire design whereby employees can be asked about social contributions, paid annual and paid sick leave first – and if identified as a formal employee it becomes unnecessary to ask about registration and bookkeeping as the enterprise will be automatically classified as formal. This reduces the reliance on questions where don’t know responses have been demonstrated to be more common while still allowing classifications to be made in line with the standards.

Impacts on classification of formality appear to be potentially greater in countries where formality is relatively more common. For example, higher risks of misclassification due to proxy reporting were found in Peru than Uganda.

The alternative wording for the question on paid sick leave did not ultimately seem to work well during quantitative testing, with reports from interviewers in the first round of tests in Uganda that the double negative wording of the question “Would you lose pay if you could not work due to illness or injury?” was generating some confusion among respondents. Considering this the original wording “Would you get paid sick leave in case of illness or injury?” was used for the second round of testing in Uganda and the single round in Peru. Ultimately this wording has been found to work well and has been retained in published model questionnaires – albeit with clarifications in related guidance that reference is only being made to payment for the absence, and not payment for medical expenses.

The strategy of combining different criteria and questions to assign a classification needs to be carefully considered. For example, the results showed it to be not uncommon for employers to pay some sick leave, whether or not paid annual leave was available. This led to a conclusion that paid sick leave, on its own may be too weak to indicate formality – thereby leading to a recommendation that both would need to be present, and that these criteria should only be relied on where the response to the questions on social insurance contributions was ‘don’t know’ – i.e., relatively reducing their importance but still allowing for them to recover cases of formal employment among a limited number of respondents.

It has also been found to be important to ensure that, for questions where national systems exist - such as registration, taxation, social contributions – those systems should be named within the questions.

Considering all the results generated it is concluded that all the criteria explicitly identified in the new standards can be operationalised in labour force (or other) surveys, with the expectation being that the formulations now presented in the published model questionnaires are a good reference point for national implementation and should ensure accurate identification of informality as a platform for, among other things, a proper assessment of differences in prevalence of and experience in informality between women and men.

4.4 Dedicated test of proxy effect

As described in the methodology section of this paper, at the request of the technical advisory group for the Engendering Informality Statistics project, a dedicated test of proxy effects on reporting was embedded in the last round of tests in Uganda and Peru, yielding over 1000 double interviews (both proxy and direct interview for the same person). This test will be the subject of a forthcoming report but results indicate that inconsistency between direct and proxy interviews is indeed potentially significant – depending on the topic being covered. Inconsistencies can work in different directions and for some topics/variables aggregate level differences were minor, while for others they were more notable. For informality it was noted that relatively greater inconsistency was found in reporting on bookkeeping with a relatively large increase in don’t know reporting and a potentially significant decrease in ‘yes’ responses – particularly in Peru (see Figure 2).

OPEN IN VIEWER

Figure 2.

Distribution of responses to questions relating to business registration by country, type of workers in employment and by type of response (direct or proxy). Source ²⁷ : Statistical Methodology Series 11 - Identification of informality through labour force surveys – Frosch et al., 2023: https://www.ilo.org/publications/statistical-methodology-series-11-identification-informality-through-labour.

The impact of proxy reporting even extended to labour force status with a general tendency that proxy respondents were likely to report less activity (i.e., less likely to report employment), and that where activity was reported it was less likely to be formal.

These findings support a recommendation to implement a strategy to manage and reduce proxy response to the extent practical, through a limited number of scheduled revisits for example. Low levels of difference in proxy rates is unlikely to generate significant differences at aggregate level in key indicators but higher levels of proxy response could impact results up to and including the assignment of labour force status, with results from the studies indicating that this would be more likely to reduce than increase estimates of employment.

The statistical standards on labour have gone through a transformational phase, commencing in 2013, with the most recent addition being the standards on the informal economy adopted in October 2023. The various standards adopted in this period are inter-related and draw strength from each other, particularly with regard to the gender relevance of the data generated promoting intersectional analysis of the forms of paid and unpaid work people do, the working relationships and conditions involved and the recognition and social protection of those workers. When implemented the range of analytical possibilities are significantly enhanced from those offered by older standards.

Work done by the ILO in recent years has led to the development of an extensive range of guidance to countries as a support to implementation of the standards. Most recent studies undertaken in Uganda and Peru focussed on measurement based on the standards on the informal economy within a labour force survey. This project aimed both to generate good measurement guidance and provide input to the ongoing standard setting on statistics on the informal economy – with a particular aim to ensure the standards and measurement guidance would be sufficiently engendered so as to provide meaningful and comprehensive data to assess differences in the working experiences of women and men.

A wide range of evidence was generated upon which existing published model questionnaires, tools and guidance were updated. The tools provide a strong platform for national measurement, but whether using those tools or not countries should, if they have not already done so, plan for the implementation of the latest standards through all sources of labour data, but in particular in a labour force survey. The strong interrelation between these topics extends to measurement and there are major synergies to be found in implementing all three sets of standards at once.

The cycle of standard setting continues to evolve and the latest ICLS asked the ILO to organise working groups to develop statistical standards on digital platform work and the care economy, two topics with high current interest and gender relevance. Work to advance the development of standards on these topics will take place in parallel with work on measurement, again building on the base provided by existing standards and guidance.