Transparency you can trust: Transparency requirements for artificial intelligence between legal norms and contextual concerns

Transparency in European data protection law

The origins of the transparency requirement in data protection law date to the 31st International Conference of Data Protection and Privacy Commissioners held in Madrid in November 2009, in which the importance of transparency to protect an individuals’ privacy was acknowledged. After being included in the proposal for the GDPR in 2012, the transparency principle made its way into the binding GDPR. Today, transparency is a core principle enshrined in Art. 5(1)(a) of the GDPR which states that personal data must be “processed lawfully, fairly and in a transparent manner in relation to the data subject,” thereby illustrating the close connection between transparency, lawfulness, and fairness. Art. 5(1)(a) of the GDPR, as the first of the core principles of data processing, is a “catch-all” provision, which is going to be typically called upon as a means of last resort if more concrete principles are not applicable in a specific scenario. Failing to adhere to it can be punished with steep fines (cf. Art. 84 of the GDPR).

Prospective and retrospective elements of transparency

Transparency, as understood under Art. 5(1)(a) of the GDPR, includes both, a prospective and retrospective element (Paal and Pauly, 2018). First, prospective transparency means that individuals must be informed about the ongoing data processing before such processing takes place and is therefore linked to the information duties of the GDPR. According to data protection law, prospective transparency requires from data controllers (i.e. the organization processing personal data) to inform data subjects (i.e. the individuals to whom the personal data belong to) in concise, easily accessible, easy-to-understand and clear and plain language (and where appropriate with visualization; see Rec. 39 and 58). Such information must be provided in writing or, where appropriate, by electronic means, and the information must come in an intelligible and easily accessible form (in particular when data controllers target children; see Art. 12 of the GDPR). The new legislation requires data controllers to provide information about themselves (who), the quantity and quality of processed data (how), the time(-frame) of the processing activities (when), the reason (why), and the purpose of processing (what for) (Paal et al., 2018; Plath, 2017).

Second, data protection law includes a retrospective transparency element which refers to the possibility to trace back how and why a particular decision was reached. Recital 71 of the GDPR highlights that the data subject has “the right to obtain human intervention, to express his or her point of view, to obtain an explanation of the decision reached after such assessment and to challenge the decision.” This element has led to lively discussions especially among legal scholars about whether a “right to explanation” exists in the GDPR or not (Casey et al., 2019; Edwards and Veale, 2017, 2018; Goodman and Flaxman, 2017; Kaminski, 2019; Selbst and Powels, 2017; Wachter et al., 2017). These discussions build upon literature and proposals for building “explainable AI,” although more interdisciplinary research between legal scholars and AI developers is needed (Felzmann et al., 2019; Miller, 2019; Santiago and Escrig, 2017).

From a legal perspective, one question has been if a right to explanation can be inferred from the wording of Arts. 13(2)(f) and 15(1)(h) of the GDPR. These articles state that meaningful information about the logic involved as well as the significance and the envisaged consequences of such processing must be provided to the data subjects at least when such decisions produce legal effects on them or significantly affect them (cf. also Rec. 71 and Art. 22 of the GDPR). The wording “meaningful information about the logic involved” and “significance of the consequences” and “envisaged consequences” are, if any, very similar to the concept of an “explanation,” which the data subject has access to via article 15 of the GDPR (Pagallo, 2018; Selbst and Powels, 2017). Yet, it remains unclear what level of detail a “meaningful” explanation has to achieve. It is obvious that an explanation which specifies meticulously the technical processes of automated decision-making processes is unlikely to achieve the aims tied to the transparency requirement (Kuner et al., 2017). An explanation should therefore be evaluated from the perspective of the individual demanding it. Overall, an explanation “should permit an observer to determine the extent to which a particular input was determinative or influential on the output” (Doshi-Velez et al., 2017: 3). Following this definition, the information provided to users should either enable them to determine the main factors in a decision or understand how certain factors alter a decision (Doshi-Velez et al., 2017; cf. also Wachter et al., 2017; Zerilli et al., 2018).

Reasonable inferences

A right to explanation might provide an effective ex-post solution for retrospective transparency because it occurs after a system reaches a decision. However, such an explanation does not per se justify the reason why such a decision has been taken nor does it protect the user from suffering the consequences linked to that decision. Wachter and Mittelstadt (2019) argue that the current legal framework does not accurately protect data subjects from high-risk inferential analytics (i.e. privacy-invasive or reputation-damaging inferences with low verifiability, such as predictive or opinion-based inferences). Therefore, they propose to consider the “right to a reasonable inference,” which follows the idea of prospective transparency; that, before a decision is made, the data subject should have the right to require from the data controller a justification of whether an inference is reasonable. Such a right would demand the disclosure of why certain data is needed to draw an inference, why these inferences are necessary to achieve a specific processing purpose or decision, and “whether the data and methods used to draw the inferences are accurate and statistically reliable” (Wachter and Mittelstadt, 2019: 5). The right to a reasonable inference and a right to an explanation taken together would provide for overall (ex-ante and ex-post) transparency, which in turn can be seen as in line with the aim of Article 5(1)(a) of the GDPR.

Ethical underpinnings of the information- and explanation-based approach to transparency

The importance given to the information requirement, associated with transparency in the GDPR, reflects underlying assumptions about the value of informed consent for technology users. Informed consent is underpinned by an understanding of the technology user as an autonomous individual who makes their decisions independently on the basis of weighing information in light of their convictions and values. Autonomy is a foundational concept in ethics, with a rich history (Schneewind, 1998) and varied meanings (Christman, 1988, 2014; Dworkin, 1988), closely linked to a specific view of the nature of the self as independent, self-contained and internally driven, an “inner citadel” (Christman, 1988). This conception has strong roots in the enlightenment, but its adequacy has been fundamentally questioned for example in postmodern, feminist, and social constructionist thought (e.g. Benhabib, 1992; Foucault, 1979; Taylor, 1989). These positions argue that the self cannot be adequately understood without giving regard to the fundamental impact of historical, relational, and societal aspects (e.g. Marwick and Boyd, 2014).

While such critiques of the enlightenment concept of the self fundamentally question the assumption of autonomy as genuinely independent individual choice, even within a perspective that endorses the autonomous self, achieving a truly autonomy-respecting informed consent would require going beyond the minimalistic requirement of notice and consent that currently characterizes consent in the context of contemporary information technologies. Within data protection law, notice and consent refers to providing information about the envisaged data processing to an individual before the actual data processing takes place (cf. Art. 13 of the GDPR). The individual then has the option to consent to data processing on the basis of this information but must do so freely and state their choice unambiguously (cf. Art. 4(11) of the GDPR). In practice, notice and consent is generally realized through the provision, by the service provider, of statements containing relevant information, such as privacy policies, and the ticking of a box for consent by the service user. The limitations of this use of notice and consent have been widely discussed (Ben-Shahar and Schneider, 2014; Solove, 2013).

In keeping with established criteria of informed consent in ethics (Beauchamp and Childress, 2012; Faden and Beauchamp, 1986), facilitating genuine informed consent would go beyond the mere provision of information, followed by the expression of a choice. Instead, it would require the service provider to adapt such information to user characteristics and needs, to avoid carefully implicitly coercive consent contexts, and to elucidate in a user-friendly, specific, and concrete way what the system was doing. In addition, it also requires to take care to support users in achieving understanding and facilitating users’ informed reflection process, and allowing them to make decisions that reflect their wishes and values. These conditions are quite demanding even in contexts where consent is obtained through personal engagement with trained professionals and may not be met in the more restrictive impersonal settings of notice and consent, even if information on AI is provided transparently in line with legal requirements of the GDPR.

Provision of transparency also encounters further challenges due to the nature of the technologies. For the increasingly popular speech-based AI devices without primary visual interfaces, such as Alexa, even the limited requirements of the notice and consent paradigm are difficult to meet, insofar as the modality of interaction provides challenges regarding how to present relevant information to users (Hoofnagle, 2018). Even more generally, obscurity is a very common, and in some respect unavoidable characteristic of AI explanations (Brauneis and Goodman, 2018; Burrell, 2016). However, beyond these concerns that impact directly on the general question of information provision in transparency, there are significant further challenges to the practical realization of transparency which will be discussed in the following section.

Transparency, stakeholders, and the implementation context

Intended as a technology-neutral piece of legislation, the GDPR’s strength lies in providing general legal requirements across technologies. However, not recognizing specific technologies and associated contexts neglects crucial elements for protecting users’ data-related rights. Transparency for AI systems raises particular challenges beyond the question of how to ensure that information is provided to the user and what information needs to be presented to users.

One challenge is the complexity of stakeholders and the different expectations they have over the same concept. This point is captured, for instance, in Weller’s (2017) investigation on the roles and types of transparency in the context of human intelligibility of AI (Table 1).

OPEN IN VIEWER

Table 1.

Transparency understanding by stakeholder (adapted from Weller, 2017).

Transparency in the context of robotics and AI
For a...	To...
Developer	Understand whether their system is working properly in order to identify and remove errors from the system or improve it
User	Provide a sense for what the system is doing and why, to enable intelligibility of future unpredicted actions circumstances and build a sense of trust in the technology
	Understand why one particular decision was reached
	Allow a check that the system worked appropriately
	Enable meaningful challenge (e.g. credit approval or criminal sentencing)
Society broadly	Understand and become comfortable with the strengths and limitations of the system
	Overcome a reasonable fear of the unknown
Expert/Regulator	Provide the ability to audit a prediction or decision trail in detail, particularly (un)intended harmful actions, e.g. a crash by an autonomous car
Deployer	Make a user feel comfortable with a prediction or decision, so that they keep using the system

The table suggests that the transparency requirement should be tailored to the stakeholder more broadly, including developers, users, regulators, deployers, and society in general. The work from Weller (2017), however, does not include different types of users within each stakeholder group such as secondary or disabled users. For example, with AI systems like Amazon Echo (Crawford and Joler, 2018), bystanders can inadvertently be included in the operation of the AI (Shaban, 2018) who may often have inaccurate, contextually influenced expectations on how information flows within those systems (Nissenbaum, 2011). In such cases, how can informed use on the basis of transparency be ensured for all users?

Making information open and transparent requires the individuals affected to be literate in assessing the risks of AI and automated decision-making systems and puts the onus on them to challenge automated decisions (Edwards and Veale, 2018). For users of AI-based assistive technologies with disabilities or special support needs, the technology is frequently employed in settings where multiple actors across professional and social roles, and with varied knowledge and capacity levels, interact (Kuner at al., 2017). Accordingly, such technologies require that a multiplicity of defined users need to be taken into account by the transparency specifications. Stakeholder groups differ in their ability to make use of information provided, and different types of information pose different barriers to understanding (as identified with regard to clinical populations by Tam et al., 2015; Redelmaier et al., 1993). Even more generally, research on disclosure and informed consent across practice domains has consistently shown that there are significant challenges to the effective use of information provided even for cognitively and clinically unimpaired individuals (Ben-Shahar and Schneider, 2014; Grady, 2015; Solove, 2013), limiting significantly the likely practical benefit of transparency. Therefore, attention to the specificity of the technology, the context, and the different types of users within each stakeholder group is essential for protecting users’ data protection-related rights.

The multiplicity of transparency effects: Lessons from HCI

While a rich body of literature explores transparency in AI systems and its outcomes in computer science, HCI, and HRI (cf. Table 2, see also Biran and Cotton, 2017), so far little research on the transparency expectations or demands of users exists (Berkelaar, 2014), particularly not when it comes to the GDPR. ⁴ In other words, the study of transparency in the sense of explainability and explainable AI (XAI) has been a vivid stream of research in the AI community since the 1990s but it has drawn little from human–human interaction and the social sciences (Miller, 2019). Although there are no firmly established core findings on transparency yet, HCI and HRI research shows that users' perception of and attitudes to transparency differ substantially depending on the technologies and services investigated, tasks given, and the context of use.

OPEN IN VIEWER

Table 2.

HCI and HRI research on the outcomes of transparency on an individual level.

Study	Topic of study	Transparency outcomes investigated	Empirical approach	Key findings
Herlocker et al. (2000)	Movie recommender system	System acceptance; User performance	Experiment	Mixed transparency effects: no effect on performance but positive effect on acceptance
Sinha and Swearingen (2002)	Music recommender system	System satisfaction; Confidence	Experiment	Positive transparency effect on satisfaction and confidence
Kim and Hinds (2006)	Social robots	Blame attribution; Credit attribution	Experiment	Weak transparency effects
Cramer et al. (2008)	Cultural heritage recommender system	System acceptance; Trust; Competence	Mixed-methods study with emphasis on quantitative parts	Weak transparency effects: no effect on trust and competence but partial effect on acceptance
Lim and Dey (2009)	Activity recognition systems	Understanding; Trust; Performance	Experiment	Why- and why-not- explanations increase understanding, trust, and task performance
Kulesza et al. (2013)	Music recommender systems	Mental models; User trust	Mixed-methods experiment with emphasis on qualitative comments	Explanations that are sound and complete are best at fostering understanding and trust
Eslami et al. (2015)	Facebook newsfeed algorithm	Initial surprise, anger and dissatisfaction; Gradual satisfaction	Mixed-methods study with strong qualitative elements	Finding out about existence of (newsfeed) algorithm leads to positive and negative outcomes
Kizilcec (2016)	Peer assessment in an MOOC	Trust	Experiment	The effect of transparency on trust depends on user expectations and violations thereof; When expectations are violated, high levels of transparency result in low trust
Chen and Sundar (2018)	Eco-friendly mobile app	Trust; Perceived Control	Experiment	Positive transparency effect on trust but no effect on perceived control
Eslami et al. (2018)	Online advertising	Trust; “Creepiness”; Satisfaction	Qualitative user study	Too specific and general explanations result in feelings of “creepiness”; middle-ground explanations enhance trust and satisfaction; Algorithmic transparency can lead to disillusionment
Rader et al. (2018)	Facebook newsfeed algorithm	Awareness; Correctness; Interpretability; Accountability	Experiment	Strongest effect on awareness, followed by accountability

In the context of recommender systems, for example, transparency of music recommendations increased participants’ satisfaction with the recommendation and their confidence (Sinha and Swearingen, 2002). By contrast, Cramer et al. (2008) looked at recommender systems in the cultural heritage domain but did not find a positive effect of transparency on trust in the system. However, they could show that transparency increased the acceptance of the recommendations. This is in line with earlier findings from Herlocker et al. (2000). Kim and Hinds (2006) investigated the influence of robot transparency on credit and blame attributions but found no significant effect on the attribution of blame and credit to the robot and the participants.

Following up on these earlier studies, recent research has studied transparency and explanations in algorithms, particularly on social media. Rader et al. (2018), for example, studied the Facebook newsfeed algorithm to examine the effects of what-explanations, how-explanations, why-explanations, and objective-explanations on different outcomes. They found that transparency strengthened awareness and accountability but had a limited effect on the perceived correctness and interpretability of the algorithm. What- and how-explanations worked better than why- and objective-explanations. Interestingly, more than half of the participants did not know that an algorithm curates their Facebook newsfeed. This is in line with Eslami et al. (2015), where 63% of the participants were not aware that a newsfeed algorithm exists on Facebook. In this study, the authors constructed a newsfeed visualization tool, contrasting an unfiltered and filtered version of the algorithm. Transparency about the existence of the algorithm did not only result in positive reactions but also in ambivalent emotions such as surprise and curiosity, and sometimes even in negative emotions such as dissatisfaction. Overall, however, with increased duration of the study, the participants became more satisfied with the way Facebook curates content through its newsfeed algorithm.

In online advertising, transparency refers to explanations why specific personalized ads are shown to a person. Eslami et al. (2018) found that transparency needs to have the right level of specificity to enhance trust and satisfaction. Explanations that are too vague or too specific create feelings of unease and distrust. More algorithmic transparency can lead to algorithmic disillusionment, where algorithms appear less powerful and useful but more fallible and inaccurate than previously thought (see also Kizilcec, 2016). In that sense, enhanced transparency might not always be a blessing but sometimes a burden (Lim and Dey, 2009).

Table 2 presents an overview of HCI and HRI transparency research. These results are mixed, lacking a definite conclusion regarding transparency implications. While the requirement of transparency has strong ethical and rights-based support, the results from HCI and HRI research indicate that, from a pragmatic and user-centered perspective, there is no clear use case for making intelligent systems more transparent. From an industry point of view, this is the same case (Eiband et al., 2018). Investments in transparency by AI developers could be costly, while the effects and benefits are unclear and there is a risk that transparency might backfire, either because it may prioritize seeing over understanding, create false binaries, or because it results in harm (Ananny and Crawford, 2018; see also the following section). Traditional autonomy- and rights-driven demands for transparency need to contend with this.

From Table 2, it also becomes clear that most HCI and HRI studies investigating transparency outcomes were conducted in the US, which might affect their transferability to a European context. For example, it could be that making assistive robotics more transparent would lead to positive outcomes in European countries with a strong trust and transparency culture (e.g. in Northern Europe), but might not have as much of an effect or be even detrimental in societies with less institutional trust and transparency. Furthermore, as Ausloos et al. (2018) note, such research has been mostly unconnected to legal considerations. As discussed, the GDPR comes with new transparency requirements that might clash with established transparency practices and lead to unintended consequences. These concerns are underexplored in HCI and HRI, and the lack of clarity about the implementation of the GDPR transparency requirements calls for more interdisciplinary collaboration between HCI researchers and legal scholars (Ausloos et al., 2018).

The performance of transparency: Organizational and societal aspects

The implications of transparency should be considered not just with regard to human interaction with specific technologies and their contexts of use, but also from a broader theoretical and normative perspective (Miller, 2019) that considers how transparency practices are embedded into wider organizational and cultural contexts. As work in critical algorithm studies has pointed out, transparency practices do not take place in a social vacuum but play particular roles in their specific cultural and organizational settings (Beer, 2017; Kemper and Kolkman, 2018). It has been argued that algorithms should not merely be seen as “objects to be known through observations” (Ziewitz, 2017: 3) but as “only [to] be evaluated in their functioning as components of extended computational assemblages” (Lowrie, 2017: 1). In that sense, algorithms are intimately linked to practices of sense-making, highlighting the trickiness of the “nuts and bolts of how to work with them” (Thomas et al., 2018: 2). As Seaver (2017: 1) argues, algorithms can be understood “as culture,” as “heterogenous and diffuse sociotechnical systems, rather than rigidly constrained and procedural formulas.”

Albu and Flyverbom (2019) in summarizing the literature on organizational transparency differentiated two broad approaches: transparency as verifiability and transparency as performativity. The first approach understands transparency as the disclosure of information. Transparency as outlined in Section 2, with regard to its understanding in the GDPR and in the tradition of informed consent, aligns with this approach. Following this understanding, organizations and institutions are transparent when they release information about their internal practices, for example, their data collection and data analysis. In the context of AI, an example would be a shopping mall that announces at the entrance and on its website whether it uses facial recognition technology to track shoppers, rather than keeping this information hidden (Rieger, 2018). The second approach, however, looks at the tensions, struggles, and discourses inherent in transparency projects, and at unintended consequences and downsides of transparency. Following this approach, transparency should be understood more holistically, including the socio-material and ritualistic practices of organizations when they “perform” transparency. The performativity perspective understands transparency practices as social and organizational phenomena whose meaning goes substantially beyond the information conveyed. Albu and Flyverbom (2019) illustrate the dual nature of transparency with regard to the Snowden disclosures. They highlight that while the disclosed information on the secret US surveillance programs was the focus of attention in public reception, disclosures were taking place embedded in organizational contexts, involved curation by other professionals, and were performed with certain strategic intentions, making it more appropriate to consider them as “complex and dynamic communication processes rather than simple and straightforward transmissions of information” (p. 283). Similarly, technology companies such as Facebook or Google employ strong narratives of openness, connectedness, and sharing on the user side while being highly secretive themselves (Van Dijck, 2013). For instance, a review of Google’s privacy policy shows a combination of an abundance of highly specific and detailed information on types of information collected, partly presented in a very user-friendly manner, alongside extremely vague general (and practically meaningless) statements about the purpose of data usage, presented generically in terms of improvement of user experience. In that case, transparency as disclosure is evident in the detailed insight allowed into some elements of their data collection practices, while at the same time transparency also appears as occluding performativity, where selective disclosure around data use seems designed to occlude their potential scope and problematic nature (Zuboff, 2019). Relevant research also reflects this distinction between verifiability and performativity: studies applying the transparency as verifiability approach tend to find positive outcomes for organizations, for example, positive effects on organizational trust, while some studies within the transparency as performativity approach reveal how transparency can also undermine trust (Albu and Flyverbom, 2019).

In a similar vein, Ananny and Crawford (2018) state that transparency can intentionally occlude, for example, when so much information is strategically disclosed that it is impractical or impossible to sift through by a layperson (needle in the haystack problem). An example is the option that companies such as Google and Facebook provide to download the personal information collected about an individual user. While this potentially enables users to see what is collected about them, the data can be too large and not formatted in a way that they can access and understand it (Curran, 2018). While the GDPR seemingly prevents such practices, as the explanations in recital 58 imply, the formulations still leave ample room for interpretation. The needle in the haystack issue could become an even bigger problem with cloud robotics and Internet of things devices, where the data collected about a user and its interactions are more complex and harder to convey. Thus, it is crucial not only to consider the disclosed information but also the effort, skills, and requirements needed to decode and interpret the information (Kemper and Kolkman, 2018), or in other words the information and privacy literacy demands on the user side (Bartsch and Dienlin, 2016), including the way in which disclosed information is embedded in other practices that may support or hinder its use.

Finally, transparency may be practically inert due to the embeddedness of the technology in a wider network of devices. For large technology companies, such as Google, Apple, or Amazon, which offer increasingly interconnected suites of complex AI services across life spheres, refusing consent to particular elements may not be an option. Even if users disagree with particular elements, once a technology provider has been chosen for the majority of their devices, these users are locked-in. This is the case because refusal on the operation of one part of the system may significantly impair the overall functionalities of the system. Moreover, high switching costs, a lack of functional interoperable alternatives, and the fact that AI systems are increasingly becoming part of our daily infrastructure (West, 2019) mean that users are in a structurally disadvantaged position, with little agency to make demands (Draper and Turow, 2019). Along these lines and based on approaches from glitch studies, Kemper and Kolkman (2018: 3) argue that “transparency of algorithms can only be attained by virtue of an interested critical audience.”