Abstract
Algorithmic outputs are increasingly shaping the employee experience, presenting a host of risks and impacts with far-reaching consequences. This contribution considers how algorithmic impact assessments should complement, as well as inform, an overarching ‘top-down’ framework for the governance of algorithmic management systems. While generalised obligations are crucial, identifying risk mitigations on a case-by-case basis can provide significant added value by (i) identifying and evaluating risks and impacts, and facilitating context-specific responses; (ii) striking a balance between generalised requirements and complete self-regulation; and (iii) ensuring that due regard to anticipated impacts and risk mitigation is built in from the design and development stages, through to deployment in the workplace. The criteria for an effective impact assessment obligation in the algorithmic management context are identified, including the appropriate stages, actors, and procedure. The Good Work Charter, which operates as a synthesis of legal principles, rights, and obligations, as well as ethical principles as they apply to the workplace, is proposed as an assessment framework. Finally, the article compares the proposed model with the existing obligation to carry out data protection impact assessments for high-risk data processing. The shortcomings of the latter obligation are explored, and a legislative approach to avoid duplication is proposed.
Keywords
Introduction
Algorithmic outputs are increasingly shaping the employee experience. 1 The concept of using ‘algorithmic management’ systems to instruct, evaluate, discipline, and reward workers has quickly spread from the gig economy into more traditional workplaces. 2 By 2018, 75% of surveyed HR professionals were already using data to ‘understand…workforce performance and productivity’, with 14% using machine learning to develop ‘people reports’. 3 In July 2021, 66% of firms reported new adoption of digital management practices since March 2020. 4 Some of the most widely deployed operational software packages now provide granular insights into worker activities and behaviour, shaping decision-making at the group and individual level. 5
The use of technology to evaluate and manage workers is not a new phenomenon. Frederick Taylor was already arguing for a scientific management approach to labour by the late 1800s, seeking to maximise efficiency by reducing the time taken for individual tasks. 6 What is new is the scale, ubiquity and impact of these practices. 7 The ‘rapid erosion of technological and economic constraints on employee monitoring’ has facilitated the collection of worker data on a hitherto unimaginable scale, 8 and employers are increasingly able to amalgamate and analyse this data in order to automate or augment managerial decisions. 9 In addition, as performance and capabilities are anticipated, rather than assessed, people analytics tools offer new advisory and predictive managerial functions, alongside traditional ones, to select or nudge behaviours. 10 In these circumstances, algorithmic management is now widely recognised as a socio-technical concept, in the same way that algorithmic systems are a ‘dynamic arrangement of people and code’, 11 reflecting the technological and organisational infrastructures 12 and human choices involved.
It is clear that algorithmic management practices impact workers. 13 Some of these harms are material: in Amazon warehouses, for example, where workers’ movements are tracked down to the second, injury rates are reported to be 80% higher than the average. 14 Research has also begun to document subtler and less visible impacts, such as a reduction in people's autonomy at work as choices are restricted or undermined. 15 Meanwhile, threats to privacy and data protection have grown with the intensity of collection of personal data, 16 alongside impacts on equality of opportunity and outcome: automated systems process huge swathes of data encoding past patterns of behaviour to produce outputs which may compound different types of inequality at a group and individual level without intervention. 17 Some deployments of algorithmic management pose a direct threat to the realisation of rights, such as attempts to combat unionisation by analysing worker interactions. 18 In short, algorithmic management presents a host of risks and impacts with far-reaching consequences.
This contribution will consider how algorithmic impact assessments should complement, as well as inform, an overarching top-down framework for the governance of algorithmic management systems. The model proposed combines different approaches to risk and impact assessment across the dimensions which shape the employment experience. It does so by deploying a framework for the evaluation of impacts on the conditions and quality of work: the Good Work Charter. 19 The Good Work Charter maps onto the broad span of algorithmic management impacts, from access to conditions and quality of work. Our approach enables context-specific evaluation from legal, ethical, and societal perspectives and obligations synthesised in the framework proposed, thus broadening the scope of potential harm mitigations to include those that can only be identified at the more granular level.
We begin in section 2 by defining the scope of the idea by reference to the algorithmic tools which would be subject to assessment. Section 3 considers how case-by-case evaluation of potential harms can complement top-down regulation which applies across the board. A concrete proposal for a new legislative duty is set out in section 4. Section 5 considers the limited extent to which the data protection impact assessment obligation imposed by the General Data Protection Regulation (GDPR) already realises this proposal, and identifies gaps in the existing law. Our model is aimed at new primary legislation, but it is also suitable for adoption in statutory codes or guidance developed by regulators, as proposed by the UK's emerging AI Strategy. 20 It therefore doubles as an assurance model to encourage early intervention and socially responsible behaviours, building understanding and capacity as part of the paradigm shift 21 beyond compliance and technical audits. 22
Algorithmic management tools
The regulatory issue considered in this contribution is the use of technology by employers qua employers, with potential harmful impacts on workers. The purpose of the regulatory response should be to address or minimise those impacts in a proportionate manner. This purposive focus implies a crucial limitation: our concern is with the impact of tools which affect in-work experiences, with a particular focus on work conditions and quality, rather than on technologies which displace labour. 23 This focus maps a recent shift in regulatory discourses on technology and labour, with increasing recognition of algorithmic management's impacts on working conditions and quality now informing the policy response. 24
Clarity of purpose enables clarity of definition: we are concerned with impacts of technologies which affect in-work experiences, and those technologies can be defined by reference to their functions. A function-centric approach is often taken when defining ‘AI’: definitions generally focus on technological capabilities. In their proposal for a summer research project on artificial intelligence in 1955, McCarthy et al expressed an aim to ‘find how to make machines…solve kinds of problems now reserved for humans’. 25 More recently, the OECD has defined ‘AI systems’ by reference to functionalities: a machine-based system is ‘AI’ if it ‘can, for a given set of human-defined objectives, make predictions, recommendations, or decisions influencing real or virtual environments.’ 26
At the sectoral level, function-centric definitions can be even more specific. The EU's proposed AI Act, for example, describes as high risk systems which are ‘intended to be used for making decisions on promotion and termination of work-related contractual relationships, for task allocation and for monitoring and evaluating performance and behaviour of persons in such relationships’. 27 Its proposed Platform Work Directive, meanwhile, would regulate ‘automated monitoring systems which are used to monitor, supervise or evaluate…work performance’, and ‘automated decision-making systems which are used to take or support decisions that significantly affect…workers’ working conditions’. 28 Canada's Directive on Automated Decision-Making, which is subject to consultation at the time of writing with a view to capturing risk and impacts on the employee experience, applies to ‘any system, tool, or statistical models used to recommend or make a decision about a client’. 29
These definitions reflect the fact that, in the employment context, technology impacts on workers’ lives when it exercises, or informs the exercise of, the managerial prerogative. 30 The managerial prerogative can be understood as the employer's power to control and direct the work process. 31 It operates at the individual employee level, as where the employer schedules shifts, assigns tasks, and awards discretionary bonuses; but also at the organisational level, as where the employer decides to relocate a group of workers, change the size of a shift team, review production or planning processes, or otherwise reorganise the business structure. 32
Concerns about algorithmic management tools almost always centre on tools which automate or support the exercise of the managerial prerogative in one dimension or another. Examples can range from fully automated decisions being made on matters such as shift schedules and bonus payments, to intermediate steps such as monitoring or information-gathering being carried out automatically and then informing managerial decisions made by humans. Take, for example, Microsoft's Workplace Analytics tool, which analyses workers’ metadata to provide ‘unprecedented behavioural insights’. 33 Employers have used these algorithmic outputs to inform organisational changes, such as modifying firm-wide reporting structures. 34 In 2020, Microsoft added (and subsequently removed) an additional Productivity Score functionality, which provided managers with individual employees’ scores for attributes like teamwork. 35
Similar tools sold by other vendors not only inform decision-making, but fully automate some aspects of it; for example, by automatically assigning tasks based on individualised data analysis. 36 Institute for the Future of Work (IFOW) research has identified the growing use of ‘connected worker platforms’ which can be readily downloaded from app stores and combine information about work and workers from a range of sources. These platforms, located at the centre of the industrial ‘internet of things’, identify potential transformations of the business model, as well as performing new and traditional management functions. The ease, reach, and speed of connected worker platforms combined with nudges made to encourage experimentation and proactive recommendations to alter organisational decisions and processes can be said to drive the ‘gigification’ of work, including management practice. 37
Although tools which inform the exercise of the managerial prerogative will generally satisfy definitions of ‘AI’ such as those provided by McCarthy et al and the OECD, sector-specific function-centric definitions such as those proposed by the EU and the IFOW avoid the risks of divergent interpretations of the same concept and are therefore more appropriate when defining legal obligations. For that reason, the remainder of this contribution refers to ‘algorithmic risk and impact assessments’ (ARIAs) to refer to assessments of algorithmic systems of all types which form part of an algorithmic management tool. 38 Algorithmic management tools can, in turn, be defined as those tools which exercise or inform the use of the managerial prerogative. In other words, the obligation would be technology-agnostic, but use-specific. 39
The case for algorithmic impact assessments
One approach to mitigating the potential harms of algorithmic management tools is to mandate built-in safeguards. The Platform Work Directive, for example, would, inter alia, require human review of algorithmic decisions in the gig economy on matters such as working time and pay. 40 While generalised obligations are crucial, identifying risk mitigations on a case-by-case basis can provide significant added value, by (i) evaluating impacts and facilitating context-specific responses to risks and impacts identified; (ii) striking a balance between generalised requirements and complete self-regulation; and (iii) ensuring that due regard to anticipated impacts and risk mitigation are built in from the design and development stages through to deployment in the workplace.
Context-sensitive evaluation and response
Legislative obligations are based on a high-level assessment of risk and prevention and are necessarily generalised: an absence of human oversight, for example, might result in arbitrary decisions standing uncorrected. While legislators can seek to identify and mitigate risks arising across a ‘high-risk sector’ or in all ‘high-risk uses’, some impacts and correlative mitigations will vary across contexts. 41 Even where tools fulfil similar functions, specific design and deployment choices can lead to divergent results. 42 Participatory efforts to assess and address impacts on a case-by-case basis mean that mitigations or other responses can be more effective to identify impacts—including unforeseen, invisible, and collective harms—and more responsive to them. Put differently, risk evaluation and mitigation is most effective when those who are exposed to or at risk of suffering the harms are involved. A participatory case-specific approach is also more likely to ascertain potentially positive impacts on job quality, where algorithmic systems can either undermine or improve job quality, depending on the approach taken to design and deployment. 43
Regulatory balance
Secondly, and relatedly, mandatory impact assessments strike a balance between (i) placing unduly burdensome prohibitions or safeguards on potentially useful technologies and (ii) relying on ineffective self-regulation. An employer's use of a system to track and improve supply chain efficiency, for example, may be legitimate in principle but prone to harmful misuse in practice. 44 Mitigating all risks at the macro level could result in disproportionately harsh regulation: blunt legal restrictions can become over-determinative. Meanwhile, unguided self-regulation is likely to result in inadequate and differing standards across industry. 45
By mandating impact assessments, legislators delegate responsibility for contextual risk identification and response to those creating the potential harms: legislators ‘identif[y] the problem, provide[] suggestions of what regulators might consider adequate, and…task[] companies with cooperatively coming up with the solutions’. 46 This is an example of reflexive law, in which legislative action guides and directs participatory self-regulation by dictating the process rather than the outcome. 47 In this context, guiding the process includes defining the approach taken to ensure appropriate stakeholder participation; and specifying the stages at which assessments are to take place. From an organisational management perspective, collaborative governance which relies on sharing information and expertise is more likely to be effective and lead to changes in planning processes or job design by the human system designers. 48
Despite the delegation of responsibility for context-specific assessment, some red lines are plainly appropriate. Automated blacklisting of trade unionists, for example, should never be permitted. 49 Impact assessments thus play a key complementary role, operating alongside ‘command and control’-style regulation. 50 As Kaminski and Malgieri emphasise, algorithmic impact assessments can be a useful part of a ‘much larger system of governance’ but are not a ‘stand-alone mechanism’. 51
A similar collective governance approach can be identified in the GDPR, alongside the prescriptive obligations found therein. Data protection impact assessments (DPIAs), which must be carried out prior to ‘high-risk’ data processing, have been described as a form of ‘meta-regulation’ in which the state makes corporations responsible and accountable for their own self-regulation. 52 Similarly, the GDPR creates a right not to be subject to solely automated significant decisions, 53 but in the exceptional cases where significant automated decision-making is permitted, data controllers are required to proactively identify and provide ‘suitable measures to safeguard the data subject's rights and freedoms and legitimate interests’. 54 Article 35(9) provides a further steer, requiring that the data controller ‘shall seek the views of data subjects or their representatives on the intended processing’. A bright-line distinction between the permitted and the prohibited is thus dissolved into a more nuanced procedural obligation in appropriate cases.
In-built consideration of impacts
Finally, impact assessments embed consideration of potential harm into design and deployment. 55 Traditional ‘command and control’ legislation, which specifies what regulatees can and cannot do, is prone to encourage compliance with the letter, rather than the spirit, of the law. Laws which prohibit unjustified dismissal of employees with a minimum length of service, for example, have limited power to inspire broader shifts beyond those statutory minima. 56 Moreover, where legislation is breached, enforcement or amendment is necessarily reactive. By contrast, ex ante impact assessments can entail holistic consideration of harms from the outset, and therefore encourage a shift away from minimum compliance standards, signalling a move towards design, development, and deployment of algorithmic systems to promote ‘good’ outcomes. By making the obligation procedural, the primary question is whether reasonable efforts have been made, and whether the approach taken is reasonable in the circumstances, rather than whether regulatory minima have been met.
Concretising the proposal
By targeting tools which affect the exercise of the managerial prerogative, it is possible to create a robust definition of the circumstances in which an impact assessment should be required. This is the first step towards legislative clarity. This section proceeds from that first step by describing what an algorithmic impact assessment would look like in practice in the employment context.
It does so by addressing four key questions:
At what stage should the assessment be carried out? By whom should it be conducted? What should it cover substantively? What should the procedural steps entail? This requires consideration of who the assessment should be carried out with, and what transparency requirements should apply.
The IFOW has proposed a detailed approach for algorithmic impact assessments, which can be conceptualised as a systematic framework for accountability based on an overarching new, positive duty in the public interest on employers (and some others) to undertake, disclose, and act appropriately upon assessments once a basic threshold is met. This model, which invites the development of sector and other context-specific protocols, guidance, and standardised techniques over time, provides a touchstone for our discussion throughout the section.
57
Here, we propose use of the Good Work Charter as a framework to underpin an effective, hybrid ARIA in the context of algorithmic management at work.
At what stage and by whom?
The first question is critical for answering the others. One of the key obstacles to ensuring algorithmic accountability is that responsibility for automated decision-making is diffuse, often spanning multiple organisations—as where a model is trained on data provided by a data broker and then sold to an employer for use. 58 In recognition of this difficulty, the ARIA should be undertaken by all actors who are developing or deploying algorithmic systems which may be used in the employment context, as well as other key actors across the design cycle and supply chain. 59 This should be combined with duties for employers to record relevant documentation and to cooperate with vendors where tools have been externally procured. 60 For example, contracting parties must ensure that the contract provides the assessing party (such as the employer) with the documentation required to produce an algorithmic impact assessment.
There are at least three temporal points at which the most critical human decisions are made, and at which an assessment of impacts should take place: design, development, and deployment. 61 Thus far, regulatory instruments and proposals have generally sought to tackle either design or deployment, overlooking both the development process in between and any interaction between the stages, such as making runtime adjustments. The proposed AI Act, for example, primarily places obligations, including for compliance assessments, on the ‘providers’ (viz vendors) of AI systems. By contrast, most obligations under the GDPR, including data protection impact assessments, fall to the data controller: in this case, the employer. 62 The proposed Platform Work Directive also includes an obligation akin to an algorithmic impact assessment, in the shape of a requirement to ‘regularly monitor and evaluate the impact’ of algorithmic management tools—but this obligation would apply only to the employer, with no correlative duty imposed on vendors of such tools. 63
Despite the apparent inconsistency in the regulatory environment, each of these approaches has its merits. Given that one of the key benefits of requiring an impact assessment is that consideration of harm is embedded into design and development, it makes sense that the obligation should be imposed on designers and developers. 64 On the other hand, there is a significant degree of abstraction between the developer and the impacts, which translates into a tension between design-stage consideration and context-sensitivity—the latter being another key benefit of impact assessments. 65 To resolve this tension, tools must also be assessed at the deployment stage by the employer and any agents involved in supporting or monitoring use of the system. For larger employers, when impacts or risks are likely to be significant, it would seem preferable to undertake the assessment at all key human decision-making points: project planning, problem formulation, data collection and procurement, feature engineering and variable selection, model selection, training, validation and testing, implementation, and making runtime adjustments. 66 While there may therefore be multiple impact assessment ‘triggers’, a clear legislative approach would require assessments to build on one another, rather than being carried out in isolation. In all cases, the first assessment must be carried out early on, before the technology is developed or the uses are concretised respectively. 67
Which impacts?
The next question is substantive: what should an algorithmic impact assessment cover? This raises two sub-questions: which categories of impacts should be assessed, and on whom? Views on both points diverge. Kaminski and Malgieri, for example, suggest that ‘a model [algorithmic impact assessment] could take advantage of the fact that it is conducted on a system-wide level to search for, and mitigate, social harms that go beyond impacted individuals’. 68 Charlesworth similarly suggests that focusing impact assessments on ‘legally protected individual or group rights’ can result in a failure to take a ‘holistic view of potential structural inequalities’, meaning that the ‘unfairness of AI outcomes tends to go unchallenged and unaddressed’. 69
A broad approach may also have its downsides. Mantelero and Esposito argue that both legislatures and NGOs consider human rights to be ‘the core of future AI regulation’, and that while ethical considerations are important, they should not form part of the algorithmic impact assessment. 70 The rationale for focusing the assessment on human rights impacts is that they are ‘clearer, better defined, and [more] stable’ than ‘ethical values’. 71 Elsewhere, Mantelero highlights the limitations of a legal approach in isolation, and his model (a ‘Human Rights Ethical and Social Impact Assessment’) aims to combine ‘the universality of human rights with the local dimension of societal values.’ 72 McGregor, Murray and Ng also suggest that (international) human rights law provides a ‘framework capable of capturing the full algorithmic life cycle from conception to deployment’, since it offers a well-defined and comprehensive conception of harm. 73 From an organisational management perspective, a human rights lens can be seen as an important step towards embedding a human-centred approach, developing a culture of decent treatment and preventing and mitigating harm to people, starting with employees but not limited to them. 74
The primary argument against mandating a wide-ranging ethical assessment, then, is the need for clarity: an obligation to consider a defined list of impacts on a cognate group is more obviously concrete than a vague obligation to consider the ‘ethical’ implications of a technology. 75 However, arguments against consideration of ‘ethics’ risk conflating ethics-washing through unenforced ‘codes of conduct’ with recent work on ethics-based assurance as a structured process which allows employers and employees to assess behaviour for consistency against an agreed normative framework. 76 Properly done, and applied in a context-sensitive way, an ethical impact assessment should complement and interlock with legal and social impact assessments in the workplace.
The Good Work Charter sets out fundamental principles that define ‘good’ or decent work for policy orientation and practical application, and was developed by a cross-disciplinary group specifically for the employment context. The Charter operates as a synthesis of both legal principles, rights, and obligations, 77 and ethical principles as they apply to the workplace, with particular regard to the use of artificial intelligence and algorithmic systems. As such, we propose use of the Charter to enable a hybrid ARIA that combines legal, ethical, and social dimensions, evaluated against the Good Work principles. This approach is consistent with, and builds on, the public acknowledgement by the former Chair of the House of Lords Select Committee on Artificial Intelligence and Vice-Chair of the UK's All Party Parliamentary Group (APPG) on the Future of Work that the Good Work Charter can be used as ‘checklist to consider potential impacts [of algorithmic systems] on work and workers’. 78 Moreover, since the Charter synthesises international and European legal principles, it provides an appropriate framework for approaches in a wide range of jurisdictions, including the EU.
In light of the discourse around human rights as a framing for algorithmic impact assessments, it is particularly notable that the Good Work Charter reflects both the European Social Charter and the Charter of Fundamental Rights of the European Union.
79
Three examples are demonstrative:
The Good Worker Charter establishes the principle of ‘access’, specifying that ‘everyone should have access to good work’.
80
The European Social Charter similarly establishes that ‘everyone shall have the opportunity to earn his living in an occupation freely entered upon’,
81
while the Charter of Fundamental Rights establishes that ‘[e]veryone has the right to engage in work and to pursue a freely chosen or accepted occupation’.
82
The Good Work Charter holds that ‘[e]veryone should work on fair conditions set out on fair terms’.
83
The European Social Charter establishes that ‘all workers should have the right to just conditions of work’,
84
and the Charter of Fundamental Rights provides that every worker be granted the ‘right to working conditions which respect his or her health, safety and dignity’.
85
The ethical imperative for fair and decent work is also recognised in the UNESCO recommendation on Ethical AI Regulation,
86
and the imperative to design, develop, and deploy AI systems in line with social and political rights, including the right to just conditions of work, is recognised by the Council of Europe's Ad Hoc Committee on Artificial Intelligence (CAHAI).
87
The Good Worker Charter posits that ‘[e]veryone should be able to take part in determining and improving working conditions’.
88
The European Social Charter establishes that ‘workers have the right to take part in the determination and improvement of the working conditions and working environment in the undertaking.’
89
The Charter of Fundamental Rights provides that ‘[w]orkers or their representatives must . . . be guaranteed information and consultation’,
90
as well as ‘the right to negotiate and conclude collective agreements’.
91
Similarly, the Human Rights, Democracy and the Rule of Law Assurance Framework for AI co-produced by the UK's Turing Institute and the Council of Europe highlights the ethical as well as legal imperatives for active participation throughout all stages of the AI life cycle.
92
This is not to say that the role of the Good Work Charter is equivalent to that of comprehensive international rights instruments: the Charter of Fundamental Rights expressly includes, for example, the right to protection of personal data,
93
and its breadth also means that it can go some way to encouraging consideration of structural issues.
94
The right to family and professional life entails specific considerations of impacts on work-life balance,
95
for example, while rights to information, consultation, and collective bargaining all link into participatory democracy and workers’ voice.
96
Here, the Good Work Charter offers an accessible, high level synthesis under the principles of ‘autonomy,’ ‘dignity’, and ‘participation’. It offers a useful access point, but should not be seen as a substitute for detailed legal analysis in a specific case. In the impact assessment context, the unique significance of the Good Work Charter is that it provides a framework which is particularly well-equipped to deal with the impacts of algorithmic management: ‘dignity’ and ‘autonomy’ provide a counterbalance to loss of agency, for example, while ‘participation’ requires information asymmetries to be addressed.
97
Similarly, ‘autonomy’ demands an ability to make informed choices, thus suggesting a break on the psychological chilling effects on behaviour which have been observed in the context of pervasive monitoring by algorithmic systems in the workplace.
98
Primary legislation to mandate impact assessments should ideally specify the catalogue of impacts to be considered even (or perhaps especially) where the full remit and extent of these impacts cannot be anticipated in advance of undertaking the assessment itself. Given that the Good Work Charter is a synthesis of relevant provisions from various rights and ethics instruments, it provides a helpful starting point for this exercise. 99 Secondary legislation, codes and guidance at a regulator and sectoral level should follow. Legislation could also specify some minimum considerations for each principle—by requiring, for example, that the assessment of impacts on ‘autonomy’ include mandatory consideration of impacts on choice of work assignments and working hours; that any automated calculation of pay be disclosed with variables considered; and that larger employers undertake an equality impact assessment, drawing on the public sector model. 100
Defining the procedure
The fourth and final question is procedural: impact assessments can quickly become a box-ticking exercise if inadequately defined. Different approaches are clearly required for assessments carried out by the developer and the deployer, but some common themes can be identified in the literature—particularly participatory identification of risks and responses (which must be meaningful and imply some level of transparency) and ongoing review of implementation and outcomes (at least periodically). 101 Existing research also includes a variety of model approaches.
The IFOW's proposal, for example, describes an assessment structured around four distinct stages: identification of individuals and communities who might be impacted; an ex ante risk and impact analysis; the taking of appropriate response action; and continuous evaluation to ensure that assessment and appropriate action is ongoing. 102 The content of the assessment, like the steps taken in response to it, should be reasonable and proportionate in the circumstances, respecting and balancing the rights and interests of employee and employer, and considering the size, resources, and capabilities of the business alongside the severity and proximity of the harm or other adverse impacts.
The framework for ethical assurance proposed by Burr and Leslie similarly points to four main procedural ‘buckets’, which broadly equate with the four stages of the APPG and IFOW models and should be seen across the life cycle of the algorithmic system:
103
Stakeholder identification and analysis, which can serve to guide the appropriate level and scope of engagement activities. Evaluation of risks and harms on this group, combining impact assessments required by law (such as data protection impact assessments, discussed below) with best practice (such as equality, human rights, and bias assessments) and exploring impacts against a normative framework (in this case, the Charter). The evaluation should allow for reflection, challenge, and the identification of new and unforeseen risks. Effective co-determination of reasonable and proportionate adjustments or other steps in response to the assessment (such as revisiting the variable selection or training). Finally, the establishment of a process to enable contextually informed, ongoing monitoring of both known and unforeseen impacts.
An empirical analysis of the approaches taken by data protection authorities led Mantelero and Esposito to develop a model methodology for design-stage impact assessments,
104
which assesses impacts by reference to likelihood and severity and builds in stakeholder engagement at the scoping stage. Other proposals have been designed for use in the public sector or for the public-private intersection.
105
In all cases, stakeholder consultation is key. In the work context, it is critical that such consultation has a collective dimension. There are two reasons for this. First, many algorithmic management tools are inherently relational: 106 standards are set by reference to average or best performers. Algorithmic systems work by making predictions about groups based on common features, and workers are evaluated against their colleagues. Focusing on the individual might mean missing group-level harms, such as increasingly strict standards applied across the board. Secondly, collective voice serves to temper the inequality of bargaining power inherent in the employment relationship: the collective is stronger than the individual. 107 This is particularly important in the context of growing information asymmetries at work and concentration of information assets more widely.
Newman, for example, highlights that worker analytics tools can tend to result in ‘least attached workers’ receiving preferential treatment. The tools enable employers to better identify the workers who are most likely to exit the firm, so that retention efforts can be better targeted—to the detriment of ‘older and less marketable’ workers. 108 Newman suggests that a ‘collective-action approach’ is necessary to challenge such problems.
Where design and development take place outside of the firm, consultation could take place via worker representative bodies, such as the European Trade Union Confederation (in the EU), or the Trades Union Congress (in the UK). The feasibility of such consultation is well-established: the ETUC has been involved in the work of standardisation organisations at the Union level since 2017, and their early involvement in standard-setting activities has led to positive results. 109
Meanwhile, procedural requirements for firm-level consultation processes should be drawn from existing approaches in industrial relations. In Europe, these include collective bargaining, works councils, and joint consultative committees. 110 In all instances, the worker voice must have some force behind it. Such force might stem from the possibility of collective action (in the form of strikes, for example) or from co-determination rights (as where works councils can block certain corporate decisions). By contrast, light-touch duties to share information and consider stakeholder views may be ineffective. 111 Mandated algorithmic impact assessments should therefore articulate the information to be documented and shared, and by whom; and should require the establishment of a dedicated process for stakeholder participation and the period for wider consultation. 112
Transparency is a crucial component for effective consultation, starting with the existence, purpose, and remit of the algorithmic system and its anticipated outputs, as well as the ARIA itself: blinkered input will be of little use. Transparency also has independent value, making it more likely that workers are able to understand algorithmic decisions made about them, so long as they have the capacity, time, and space to interrogate and challenge these decisions and the socio-technical context in which they are made. 113
Such transparency is urgently required. In one recent study, 50% of UK-based employees reported that algorithmic management technologies might be in use in their workplaces without their knowledge, and only 21% felt that they would be able to effectively challenge algorithmically-made decisions. 114 In another survey, 52% of workers were ‘not at all confident’ that they knew ‘why and for what purposes’ their employers used data collected about them, and 67% were ‘not at all confident’ that they understood how their data was used to assess their performance. 115 Current requirements and forums for information-sharing, consultation, and participation have been found to lack bite in the context of the use of algorithmic systems at work, and the ARIA provides one example of the new processes which we anticipate will be developed over time in response to this shortfall. 116
Crucially, transparency cannot be mandated only during the assessment itself: access to information about the options for adjustment, the balancing exercise involved as trade-offs are made, and the outcomes of the assessment and likely effectiveness of mitigations, are equally important. 117 Transparency is also required about the nature and process of participation in the substantive assessment. 118 The evidence is that effective impact assessments are iterative processes rather than one-time obligations, and oversight has been identified as a key element in this framework. 119 Publication of impact assessments and follow-up materials, in redacted form where absolutely necessary, should therefore be a non-negotiable procedural obligation. 120 Where such transparency is not realised, the consequences can be severe, as shown below in relation to the GDPR-mandated data protection impact assessments (DPIAs).
By answering the four questions posed at the beginning of this section, a concrete proposal for an ARIA has emerged:
Temporal points: Unless the tool is developed in-house for a designated purpose, a minimum of three impact assessments should be carried out and then updated on an ongoing basis: at the design stage, the development stage, and the deployment stage. For larger employers, assessments at all key seven stages of human decision-making and continual monitoring is best practice. Duty bearers: These assessments should be carried out by the algorithm designer, developer, and employer respectively. Contents: The assessments should consider risks to, and impacts on, workers’ rights, freedoms, and other interests at an individual and group level. The Good Work Charter should be used as a framework for this exercise, alongside concrete and non-exhaustive particularised examples for each of the ten Charter principles. Legal, ethical, and social impacts identified should be recorded in the ARIA. The assessments should also document the proportionate technical and non-technical steps taken as a result of the evaluation to address or mitigate any harms identified, and should enable consideration of potentially ‘good’ impacts on work conditions and quality. For consistency against a baseline, this exercise could similarly be undertaken by reference to the Good Work Charter. Process: The procedure should require mandatory recording of documentation, involvement of stakeholder workers and representatives, publication of outcomes, and published reviews at defined intervals. Procedures for consultation and participation should build on existing work on public sector algorithm impact assessments (in the case of vendors), or traditional labour approaches (in the case of employers).
121
A need for new law
In the UK and EU, the existing DPIA obligation under the GDPR might appear to go a long way towards realising this proposal. Indeed, Kaminski and Malgieri analyse the DPIA as an ‘algorithmic impact assessment’.
122
The relevant provision, Article 35(1) of the GDPR, provides (in part) that:
123
Where a type of processing in particular using new technologies, and taking into account the nature, scope, context and purposes of the processing, is
The obligation is therefore triggered by a risk to rights and freedoms (not just to data protection rights), and Article 35(7) clarifies that the DPIA must include ‘an assessment of the risks to the rights and freedoms of data subjects’ as well as the ‘measures envisaged to address’ these risks. The term ‘rights and freedoms of natural persons’ can be understood to encompass the rights guaranteed by the Charter of Fundamental Rights.
124
Recital 75 further explains:
125
The risk to the rights and freedoms of natural persons . . . may result . . . in particular: where the processing may give rise to discrimination . . . or any other significant economic or social disadvantage; . . . where personal aspects are evaluated,
The material scope of the DPIA is not, however, equivalent to that of the proposed ARIA. Although data protection can operate as a gateway to access other rights and freedoms relevant to algorithmic management, the data protection regime rests on assumed human ability to control and manage information (human ‘sovereignty’ over data 128 ), an idea that has been challenged by the latest wave of algorithmic management tools. Further, there will be some situations in which tools used to inform the managerial prerogative do not require a DPIA simply because they do not process personal data. For example, where non-personal (anonymised) data on supply chain efficiency leads to an entire team being relocated, 129 there may be no DPIA obligation on the employer—even though the team move is an exercise of the managerial prerogative with impacts on workers.
Returning to the four key questions, some further gaps can be identified. First, section 4 argued that to be effective, impact assessments must be carried out during at least three stages. The DPIA obligation will generally only arise at one stage, because it applies to the ‘data controller’, defined as the party who determines the ‘purposes and means’ of data processing. 130 In almost all cases, the employer will decide why and how to use algorithmic management tools. If the developer processes personal data for the employer, it may be obliged to ‘assist’ the employer with the DPIA, 131 but such assistance will only be relevant after the design phase, thus losing the value which comes from the embedding harm mitigation at the design stage. 132
The first and second elements of the proposal—at what stage and by whom—are therefore only partially addressed by the DPIA obligation. Moreover, the proposed AI Act would not fill in this gap: the proposal would mandate ‘conformity assessments’, but these would be by reference to technical standards, not human rights (or other legal, ethical, or social impacts), and would not meet the consultation or transparency standards identified in section 4. 133
The transparency and consultation points both relate to the fourth element of the algorithmic impact assessment proposal: procedure. Even though the harms assessed in the DPIA are broadly similar to those that would be assessed in the ARIA, there is significant procedural divergence. Most strikingly, the DPIA obligation contains no transparency requirements whatsoever. Official EU-level guidance suggests that DPIA publication could help foster trust, adding that publication could just consist of ‘a statement that a DPIA has been carried out’ 134 —but there is no obligation for data controllers to do even this.
Kaminski and Malgieri suggest that the absence of any DPIA transparency obligation is its ‘biggest shortcoming’, 135 and argue that the absence of DPIA transparency has undermined the effectiveness of related obligations. For example, the GDPR requires data controllers to consult relevant national supervisory authorities about proposed processing where the DPIA reveals unmitigable high risks. 136 While this obligation theoretically provides an opportunity to access expert advice on risk mitigation and acceptability, Kaminski and Malgieri suggest that in reality it means that companies can ‘decide…whether [they] should be subject to regulatory oversight’. 137 The authors’ concerns are valid: as of December 2021, the UK data protection authority had been approached only twice by data controllers proposing high-risk processing in an employment context. 138 Research reveals that high-risk data processing was indeed being carried out by various UK-based employers during the same period, 139 but since DPIAs are very rarely published, there is little scope for assessing the mismatch.
A closely related shortcoming of the DPIA is the inadequacy of its consultation element. The GDPR provides that ‘[w]here appropriate, the controller shall seek the views of data subjects or their representatives’ as part of the DPIA. 140 This obligation to ‘seek views’ does not afford the workers’ voice any force: there is no requirement for the employer to address the views it receives. 141 Once again, the shortcoming is compounded by the absence of transparency. The employer need not even inform the workers or their representatives about the outcome of the assessment, so oversight is severely curtailed. We note that this requirement does not form part of the UK's proposed Data Protection and Digital Information Bill.
The DPIA obligation exists alongside other laws, and in some jurisdictions, existing labour consultation obligations may fill in the gaps. In Germany, for example, the use of ‘technical devices designed to monitor the behaviour or performance of the employees’ is subject to co-determination. 142 At the other end of the European spectrum, even employers in the UK are required to disclose some information to recognised unions for the purposes of collective bargaining. 143 This obligation only applies in a narrow set of circumstances, however, and is of limited value given the country's low union density. 144 The challenge is that there is significant heterogeneity in industrial relations across Europe, and the inadequacy of the DPIA is not always mitigated by other laws.
In summary, although the DPIA is similar to the proposed ARIA in terms of aim and scope, the two are not functionally equivalent. On the other hand, a proliferation of impact assessments is clearly undesirable in practical terms. How, then, to marry the two?
Kazim and Koshiyama suggest three options: (i) an AI impact assessment which ‘sits on top of’ the DPIA; (ii) a DPIA which is ‘adapted and modified’ for AI; and (iii) an AI impact assessment which is independent of the DPIA. 145 Despite recognising that a DPIA ‘may be required when using any AI system’, 146 the authors suggest that options (i) and (ii) pose a fundamental challenge: while data protection is ‘mainly an expression of privacy’, ‘AI impact’ is ‘an expression of the value of fairness’, and these can come into ‘direct conflict’ with one another. 147 They therefore argue that integration is possible only if a ‘fundamental value judgment’ can be made about the prioritisation of privacy versus fairness. If such ex ante prioritisation is not possible, for example because value trading is ‘context specific’, then ‘data [processing] and AI [systems] …will each need [their] own impact assessment[s] and how the two relate will have to be worked out through another mechanism’. 148
This argument does not hold, for two reasons. First, it relies on a false equivalence being drawn between the right to privacy and the right to data protection. 149 Although courts have tended to conflate these rights in the past, 150 the right to protection of personal data is protected independently from the right to privacy under the Charter of Fundamental Rights 151 and is inspired by different goals, including the reduction of power and information asymmetries between data subjects and data controllers. 152
Secondly, even if the right to data protection were a subset of the right to privacy, Kazim and Koshiyama's argument seems to suggest that the DPIA process requires privacy to be promoted above all else. This is not so: as above, the DPIA should consider the full set of ‘rights and freedoms of data subjects’. 153 To the extent that contextual value trade-offs need to be made, the DPIA is precisely the forum for such deliberation. Indeed, at a more general level, it makes more sense to consider value trade-offs within a single process than designing separate processes which are ‘fundamentally in tension’, leaving the conflicting outcomes to be ‘worked out through another mechanism’. 154
In practical terms, a single impact assessment should be carried out for each (set of) algorithmic management tool(s). Duplication could be prevented by making two simultaneous legislative changes: introducing the ARIA obligation outlined herein; and adjusting the DPIA obligation such that it is automatically satisfied if the impacts of the high-risk data processing have been considered as part of the ARIA. 155
Framing the obligation in this way makes the DPIA subservient to the ARIA in the labour context. This is sensible when one considers that the ARIA is filling in the gaps of the DPIA: reversing the approach would mean starting from a weaker position. Rolling up assessments should be no anathema to DPIA advocates: Article 35 itself specifies that ‘[a] single assessment may address a set of similar processing operations that present similar high risks’. 156 The proposal would enable the added value of the ARIA to be realised while minimising any additional burden.
Conclusion
This contribution has proposed the introduction of a framework for undertaking algorithmic impact assessments for tools which exercise, or inform the exercise of, the managerial prerogative at work. To be effective, these new impact assessments must be imposed at least at the design, development, and deployment stages; must consider impacts on workers’ rights and freedoms and impacts on work conditions and quality; and must meet certain procedural and substantive minima, including recording and disclosure of relevant documents, effective consultation and genuine transparency. While DPIAs imposed by the GDPR go some way towards realising this proposal, marked gaps remain, both in terms of the scope of application and the procedural standards. The new ARIA should therefore incorporate but build on the DPIA obligation and model, combining it with other dimensions for assessment and expanding its remit and bite to consider all the primary dimensions of good quality work that shape the employee experience from legal, ethical, and social perspectives. We propose the Good Work Charter as a practical framework for undertaking such a hybrid model, and for legislating for it.
While this article has considered the granular details of impact assessments in the labour context, it is possible that eventual legislative action on algorithmic impact assessments will be cross-sectoral, and that sectoral detail will be left to secondary legislation or regulatory instruments. 157 It is therefore useful to identify those points which apply to algorithmic impact assessment proposals more broadly.
The first is that despite the tendency towards omnibus AI regulation, sectoral specificity enables a shift away from ‘risk’ in the abstract and the consequent scope for creative interpretation to curtail scope of application. Regulated technologies, impacts, and procedural steps should be identified as concretely as possible, and this is more feasible at the sectoral level, suggesting that guidance on methods and measurements for the ARIA should follow.
Secondly, experience with the DPIA has demonstrated that transparency and stakeholder power are crucial for technological impact assessments to be effective. An absence of transparency and meaningful stakeholder engagement seriously undermines oversight mechanisms, and ‘consultation’ can be spurious when it takes place without the information, capabilities, or space for active participation in a context of significant power imbalance.
Thirdly, and perhaps more critically, impact assessments are both part of a broader governance regime and will inform its development over time, as ARIAs generate new evidence and understanding about impacts on people, and detailed guidance and case law is built up. 158 ARIAs enable co-regulation where red lines are too blunt, but they do not displace the need for red lines; they provide space for stakeholder voice, but they do not address broader power structures; and they create ex ante consideration of impacts, but do not secure access to justice for those who nonetheless experience harm themselves. While this contribution highlights the significant potential value of impact assessments in the employment context, they cannot tackle the novel concerns posed by algorithmic management in isolation: that will require a rethinking of the entire regulatory toolbox.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Aislinn acknowledges funding from the European Research Council under the European Union's Horizon 2020 research and innovation programme (grant agreement No 947806).