Friction,snake oil,and weird countries: Cybersecurity systems could deepen global inequality through regional blocking

Methods

In this study, we took a qualitative approach employing a combination of participant-observation and in-depth interviews, and explored public social media to identify user complaints about regional blocking. We interviewed 11 experts (See Appendix 1) on topics including the rationales for blocking, its efficacy, the global financial marketplace online, and methods for fraud detection and identity verification in both online and offline settings. We used an unstructured interview approach in which certain topics were planned in advance but follow-up questions were determined in the course of the interview. This allowed for novel insights to be explored. Where possible, we looked for other evidence to further triangulate expert assertions, such as public websites or policy documents. Quotes in this article were chosen either because they represented sentiments repeated by multiple people and invoked themes that arose across data sources, or because they provide unique insight into the blocking phenomenon that can help explain otherwise opaque practices.

The first author attended industry conferences and community meetings and forums in Summer 2017 to establish the broader context in which these blocks occur, cultural patterns in their rationale, and their potential wider implications. These conferences, meetings, and fora include:

The Online Dating and Dating Industry Conference (iDate) in June 2017 in Los Angeles, California, a small (approximately 150 person) two-day event that focused primarily on niche and emerging companies in the online dating field. In addition to attendees who owned or worked for dating websites and apps, many participants worked as vendors for third-party technologies such as payment processing, identity verification, fraud detection and prevention, affiliate marketing, live video streaming, automated emotion detection and sentiment analysis, gift shop plugins, and chatbot creators. We decided to attend this event because of the second author's findings on regional blocking in online dating sites.

At these events, the first author took extensive notes on the context and content of presentations, collected marketing materials, and spoke informally with participants about this project and related themes, with these conversations then incorporated into fieldnotes.

We pursued an inductive analytical process to allow for the discovery of unanticipated themes and insights. Altogether this qualitative and inductive approach was geared toward uncovering the logic and underlying rationale behind these practices from the perspective of people who build and deploy such systems. The primary goal was to understand as fully as possible the scope and range of these practices. Our selection of interviewees and events was likewise guided by a sampling approach that sought breadth and diversity of views. This approach does not, however, allow us to make strong claims about the prevalence or typicality of regional blocking practices.

Relevant literature

In our approach to research on cybersecurity, we explore traditional questions of fair access to network resources and the challenge of distinguishing good and bad actors on the network. However, we do so in a way that unsettles normative ideas of imagined users which appear to be modeled after Western consumers in advanced capitalist economies. We draw insight from critiques of globalization and postcoloniality that decenter this normative “user” pointing to the way marginality is easily cast as aberrant and illegitimate (Burrell, 2012; Dourish and Mainwaring, 2012; Irani et al., 2010). We have worked with research collaborators exploring the question of regional blocking in parallel, but from within methodological traditions in computer science (see Afroz et al., 2018). In their work, they set about validating the existence and prevalence of “server-side” regional blocking utilizing automated techniques. Our approach pursued another set of questions. Defined by Irani et al.'s (2010) framework of postcolonial computing, “a project of understanding how all design research and practice is culturally located and power laden,” we sought to critically examine the cultural assumptions of blocking technologies and how they are influenced by the particular social locations of professionals who develop and deploy such practices.

Previous work has shown the limitations of mainstream cybersecurity approaches in providing protection to users when particular social and geographic locations and practices are not taken into account. Freed et al. (2018) identified how the standard threat models in cybersecurity did not account for network security violations of stalkers in the context of intimate partner violence. Thus there were few appropriate measures developed to effectively handle them. Ben-David et al. (2011) describe the many contextual factors absent from traditional security practices when applied to the Global South. For example, limited bandwidth can make it impossible to download security patches. As a result many machines remain unpatched and vulnerable. Similarly, an overzealous focus on one set of threats and practices of combatting them via regional blocking can leave other vulnerabilities under-examined.

We extend work in this area by analyzing the way that traditional cybersecurity practices overlook or misunderstand the online behaviors of populations that diverge from expected norms and thus entrench differential access along traditional lines of socio-political power. In addition we leverage insights and framing questions from a new literature on machine learning fairness and bias. Despite ongoing debates about how to define and operationalize concepts such as “fairness” in machine learning applications (Hardt et al., 2016; Kleinberg et al., 2016), scholars have consistently established diminished accuracy in algorithmic categorization for minoritized groups and predictions that are biased by historically skewed and prejudicial data patterns (Barocas and Selbst, 2016; Bolukbasi et al., 2016; Buolamwini and Gebru, 2018; Eubanks, 2017).

The literature on fairness in machine learning has especially considered application domains such as criminal justice, lending, and social services where mechanisms of allocation impinge upon civil rights. Cybersecurity has been largely left out of this examination, though it is a key application area for machine learning that is widely applied in spam filtering and fraud detection. A key question is whether such systems work as well for users from one part of the world as they do in another. Are false positives (users identified as ‘bad’ actors who are in fact legitimate users) more prevalent in certain geographies or among certain subpopulations? Furthermore, could the definition of what constitutes legitimate use be reconsidered? These questions resonate with literature in predictive policing and algorithmic identification of child exploitation, where concerns about disparate impact and harm on marginalized communities are crowded out by public safety rationales (Selbst, 2017) and “proximity to innocence” (Thakor, 2018). We take up these questions by examining the framings of those who design, use, and request such systems, exploring how built in expectations about users have the potential to produce unintended consequences.

Regional blocking and its discontents

“What the hell is the threat model that's solved by blocking a region of the world? And the answer is, no one knows.” (William ¹ , IT Director, Transportation Company)

This fragmentation has been aided by a parallel impulse among US-based companies to revoke access to certain sites and services from visitors in some regions due to regulatory requirements, from censorship and hate speech to data protection laws (Drake et al., 2016; Goldsmith and Wu, 2008; Mueller, 2017). This issue drew headlines recently when some EU citizens found themselves blocked from accessing a range of services following the implementation of the latest General Data Protection Regulation (Cimpanu, 2018). However, pre-existing restrictions by other countries had long had similar effects (Cory, 2017).

Meanwhile, even as many companies decried these legal barriers for preventing free market momentum, many quietly continued to put into practice their own systems that filtered out traffic or membership from places that seemed “risky” due to implied or stated security concerns. Regarding one large e-commerce site that she had worked with to develop a security strategy, Samantha, an anti-fraud professional, said that along with places blocked due to economic sanctions, they do not ship to “weird countries” like Ghana, Nigeria, Philippines and Malaysia, and instead would automatically cancel an order from those locales, because “they don't bother, [they] don't do business out there.” Yet despite its ongoing popularity, many were dismissive of this practice as crude and ineffective. William, an IT director for a large transportation related company, dubbed the practice a form of “security snake oil.” He explained,

This is part of both the actual features built in to the security software, as well as the marketing material of like, [sarcastically] ‘Well, you could block China. Now, you'll want to block Thailand. Like, you know, those Eastern European countries that you don't want touching your stuff. You'll want – you can use our product to block them.’ So, whether this is a thing that technology decision makers want to do or not, it is being presented to them as a thing that they want to do, and that sort of self reinforcing thing means that when weird stuff starts happening, then the technology leader says, ‘Well, let's just block China! …we can block China, that won't have any actual negative impact on our business, and I can say, well, we blocked China – and it's still happening – shit! – well, I did something!’

How widespread is regional blocking? There is no straightforward way to measure the exact scale of the problem. However, there are two distinct measures worthy of consideration, (a) potential users blocked from sites and web services given the region where they reside and (b) users actually seeking those services who are blocked. To provide an answer to the first measure, Afroz et al. (2018) crawled the top 500 Alexa websites and found consistent patterns of regional differences. In a comparison between the US and Pakistan into the availability of these sites, the authors found 14 sites they identify as “true server side blocking,” and another 28 where blocking was in place but the source of the block was unverified, for a combined total of 8.4% of the sampled top 500 sites. A comparison between the US and South Africa found 19 sites inaccessible from the latter but available in the former, a total of 3.8%, with 13 of those sites labeled as server-side blocking. However, our observations and interviews, as well as prior work, suggest that regional blocking may be more concentrated in West African and South East Asian countries not represented in that study.

As Afroz et al. point out, their methodology underestimates the prevalence of regional blocking because it does not account for sites that will load but fail to allow people to create an account, list their truthful location in their user profile, make a purchase, ship to the desired country, or process a money transfer. As anti-fraud professional Samantha verified, some users could have their purchases canceled, even when an order initially seemed to reach completion. Leaders of a campaign to advocate for making Paypal accessible in Palestine explained that workarounds, such as claiming to be in Israel or Jordan where Paypal is accessible, might work in the short term, but that Paypal would subsequently shut down those accounts. They had various ways of identifying users who were actually in Palestine based on their IP address or bank information.

The second measure of prevalence, whether users from blocked regions seek to access these sites and consequently encounter blocks, would certainly show a smaller number who are actually impacted by this practice. When functioning as intended, regional blocking often flies under the radar due to the small number of ‘legitimate’ users in a blocked country. Network administrators and other IT security professionals feel comfortable utilizing it for the same reason. As William mentioned, country-level blocking is perceived by firms as something “that won't have any actual negative impact on our business.” However, blocking becomes self-reinforcing—few people in a location use a service and so the company feels justified in blocking that region, meaning that few people in the region attempt to utilize the service, or are even aware of its existence. For this reason, regional blocking is most likely to be noticed by international students, businesspeople, or other cosmopolitan travelers (and those in their extended social networks) who are able to compare experiences of the Internet between two locales. Our preliminary exploration of self-reporting of this phenomenon on social media, including on Twitter, the public Facebook pages of companies that employ regional blocking (i.e. Macy's.com), and specialized sites like isitdownrightnow.com, yielded numerous complaints from prospective site visitors who fit the description of globe-trotting transnational.

Going frictionless and unbiased—The rise of the machines

At the 2017 Money20/20 conference, a consistent tension surfaced, aptly summarized in one panel there titled, “How can secure customer authentication and frictionless customer experience co-exist?” This problem was of particular concern given two frequently discussed trends—the rise of “cross border” transactions—alongside a more general appeal to the cosmopolitan global outlook exemplified by the conference's tri-annual meetings on three continents (Europe, Asia, and North America). There was also the looming double-edged threat of fraud and other security problems on one side, and overzealous regulation on the other. In this context, Uber's Chief Payment Counsel fiercely defended the free market as the solution to any security threats—lamenting, in regards to proposed legislation, that he didn't think government should be setting how fraud is managed, and that “most of what is being proposed will make the experience less magical” for consumers. Over and over again, corporate presenters grappled with strategies for rapidly expanding markets without being weighed down by bureaucrats or criminals. The latter were framed as sophisticated cartels with nothing to lose, “organized crime, not a not guy in his bedroom,” according to more than one panelist, who were “funding nasty stuff including terrorism.” These criminals were seen as more nimble than legitimate companies, and always on the verge of being one step ahead in the battle with multi-billion dollar corporations because they were not hampered by legal compliance. Government regulators were often the second place adversary in this scenario, the accidental handmaiden of fraud and theft by stifling innovation over paternalistic concerns that might be a smokescreen for economic protectionism, or even censorship.

A quieter faction raised another set of hopes, adjacent to frictionlessness and ostensibly more rooted in moral responsibility than profit—inclusion. This faction noted that certain regions differentially face “friction”—or outright refusal—in the current financial system, online and off and in their increasing overlap. Lisa, an interviewee specializing in emerging markets for a payments company, explained that their systems for evaluating costs and benefits were ultimately subject to bias. She framed the problem as stemming from an information deficit that not only discriminated against certain populations, but also failed to be a rigorous measure of market opportunity:

In a perfect world I would have information on risk – I would have a risk score for every country in the world, that's like some objective source. But, instead, we kind of just rely on our own feelings. Which I think is not as robust as we would like to make it out to be.

One broad technique was repeatedly raised as a potential cure for these diverse ailments of friction, fraud, regulation, exclusion, and bias—machine learning and artificial intelligence. Using pattern recognition built on Big Data, such systems could, many presenters claimed, facilitate broader, faster, easier, and safer access to potential customers, while identifying and neutralizing problematic users. Vivek Bajaj of IBM argued at Money20/20 that incorporating AI into the calculation of risk and fraud was “not only about catching bad guys, as much as about letting legitimate clients get on with their lives in easier fashion,” citing “huge problems with false positives”—cases where legitimate consumers had their business refused over security flags. Sarah Kocianski of Business Insider agreed, arguing that “older systems which rely on rules and reputation lists are failing to cope with the sophistication of fraud,” and were facing “growing numbers of false positive declines … [and] failing to cope with growing number of cross border transactions and card not present” sales. Jesse, of the identity verification company, who had also attended Money20/20, explained, in his interview, that AI-based systems would succeed in evaluating the “risk that you actually pose” and support increasing acceptance rates for more sales. AI thus was a solution for multiple problems at once—as one Money20/20 panelist put it, “AI is not the new oil, it's the new electricity, it's going to become invisible, you don't even think about it, it's just there, it works.”

Yet simultaneously, these very professionals highlighted the myriad ways that machine-learning based systems for fraud detection, identity verification, and behavioral recognition are likely to contribute to the continued exclusion of marginalized user populations. These systems are frequently designed in a way that ignores or misunderstands on the ground realities of socio-economically and geographically diverse users. Further, they obscure the monetary incentives undergirding the technical infrastructure of exclusion, which continues to reward the expansion of blocking mechanisms.

Assumptions about behavior, norms, and access

Roughly, there are three approaches to blocking or flagging that can involve regional information:

Block all visitors from a country wholesale.

In addition to blocking website access outright, each means of determining problematic usage can also be implemented to refuse web services (such as purchases or account creation). The latter systems may allow greater access to some potential users who would be blocked under the first set of rules. However, they also expand the potential failure points, by incorporating more information about user behavior and greater dependencies on third party sources to verify a user's identity and reputation. These sources often ignore differentiated global conditions or marginalize those who do not or cannot assimilate to the patterns of the imagined user, facilitating ongoing regional discrepancies in blocking.

Peter, who runs an online fraud detection company, explains that there are cases where anomalies must indicate some sort of problem (though not necessarily a malicious one), and other cases where the flagging of an issue is based on probability, rather than a rigid binary classification of right and wrong. The latter case is prone to capturing “real” traffic alongside manufactured attacks or automated visits attempting to parade as unique visitors. He says,

If a man is two meter high – no, no – ten meter high, there is something wrong, right? Definitely, there can be no person like ten meter high. But these are easy types of fraud – easy to detect, easy to do. The more complex cases, they are always more about machine learning or something like this, trying to cluster the data … to find some kind of discrepancies and abnormalities in traffic distribution. … . there definitely can be false positives, can be mistakes.

Yet depending on the implementation, this too can be hurtful and suggestive of an unwelcoming environment should someone recognize that their very existence in a space sets off alarms, especially if integral aspects of one's identity are framed as security threats. Information Studies scholar Hoffmann (2018) argues this point in relation to incidents of Native American Facebook users being kicked off Facebook under the platform's “real names” policy and transgender passengers being flagged by TSA agents based on body scans. Sociologist Tufekci (2017) similarly explores a case where Western assumptions about user characteristics disproportionately exclude those who fall outside of normative expectations, recounting an instance where Facebook sanctioned a friend for having a name that was equivalent to a vulgar word in English. In order to reinstate her account, this friend had to provide government identification of her legal name. While the bias embedded in Facebook's real name policies has been roundly critiqued, this raises further concerns about the reliance upon standardized government identification in order to sign up for a service or keep from having an account suspended. In many parts of the world, possessing government identification is not required and is uncommon.

Many in network security roles are eager to find “new approaches” using machine learning to address fraud in cross-border payments “without putting up so many hurdles it makes service unusable,” as one panelist described it at a Money20/20 session. To limit these kinds of harmful biases, cautious professionals propose developing profiles either for different types of users or based on each individual's specific patterns of behavior, comparing users to themselves rather than to a larger pool of others who might differ significantly. For example, Jesse, an employee for an identity verification system, stressed the need for any machine-learning based process to be sensitive to regional variations. He argued that what counts as normal should be specific to each locale, so that behaviors that might be particularly common in a context like India or Africa, such as lower value and more frequent transactions, “could generate flags if [occurring] outside of those” locations. A representative from online security firm Kount argued at the iDate conference that more specific and individualized pattern recognition was not only more fair, but more effective: “what can't be broken is you … behavior is hard to mimic.” However, as a Money20/20 panelist explained, the only way to “understand customers better” at such a fine-grained level is to “combine [data on the platform] with other data on the device, accounts, take a whole suite of data points and use machine learning to identify fraud patterns.” In other words, in order for people not to be subject to the tyranny of the norm, they would have to forfeit their privacy. Particularly for those whose data stands out in a crowd—often those most vulnerable to varying forms of violence and discrimination—the price to gain access to these online systems would be their consent to massive amounts of surveillance.

Professionals who advocate for such systems are familiar with the problems of bias in machine learning. At Money20/20, Pedro Fonseca of CrowdProcess questioned, “Should neighborhoods be allowed? Should this data be allowed? People in poor neighborhoods would be inherently behind on your credit score.” He explained that non-discrimination is a difficult but important problem, because otherwise society would “end up with dystopia where you have to consider who you're friends with on Facebook—only friends with rich people on Facebook and keep everybody else on Whatsapp!” This got a hearty laugh from the audience, but can be extended to regionally correlated data and behavioral patterns that can operate in similarly exclusionary ways. Ironically, one set of common flags for machine learning systems that may inadvertently penalize those in certain regions are remnants of the very blocks those systems seek to replace.

Workarounds as flags

Internet users worldwide are innovative and often adapt quickly to barriers put in the way of using web-based services. There are entire markets that have risen up providing the means to navigate dominant commercial blockages or to offer local alternatives. These include proxies, VPNs, freight forwarding, and using someone else's payment or address information when conducting an online transaction. However, these workarounds developed to bypass group-level blocks are, in turn, being picked up as a signal by systems that determine which users to block access to. Such strategies are coming to be viewed as suspicious or as a likely cover for criminal activity.

Removing those who cannot conform

Regional blocking can also be “effective” not only by preventing fraud, but by removing data that could complicate the smooth functioning of algorithmic recognition and sorting practices—taking away “friction” for those users who are considered valuable. At first, the refusal to collect data on people from blocked countries by refusing to provide services to them seems antithetical to the drive described by Fourcade and Healy to capture as much behavioral data as possible to sort people into appropriate categories. But by turning away those who might foreground the limitations of the algorithms, these systems can then boast higher rates of accuracy for those who are allowed in—those who already match pre-determined criteria. As Dourish and Mainwaring (2012) argue, ubiquitous computing has frequently obscured “particular geographical, institutional, commercial, and historical settings,” claiming to develop “global” solutions that actually just re-enforce the dominance of one “global authority” over all other ways of being. Thus websites that want to serve a particular group of people while claiming omnipotence are better positioned to do so if they eliminate the anomalies of people from places less prone to producing profits.

Conclusion

Considering access to online shopping, dating sites, and payment systems, those types of services where regional blocking seems to occur frequently, might be critiqued as a shallow way to address discrimination online. But access to these arenas increasingly has implications for people's social, economic, and political well-being. Dating sites provide romantic and economic opportunity (Burrell, 2012). Online shopping can provide enormous cost savings not only on consumer goods, but also on equipment necessary for one's livelihood—but only for those able to access it. Furthermore, shopping online can help avoid the embarrassment or risk of purchasing stigmatized products in person (Boellstorff et al., 2013). Payment systems are a foundational necessity for those who wish to provide goods and services online to a broader, global customer base. The ability to be paid strikes to the core of economic security. And while some current experiences of regional blocking may simply qualify as a nuisance, increasingly the workarounds users develop are being targeted as aberrant and suspicious behavior compromising their ability to build up a “good” behavioral history online. There are also some signs that these barriers could begin to spillover into other domains directly tied to life chances, such as employment and housing. Third party systems that handle blocking and verification efforts stand to profit by being the brokers of expansion. For example, Jumio's identity verification system is moving from the financial world of know-your-customer policies into the sharing economy and dating ecosystem, offering promises of greater physical security for users, and being adopted to help enforce rental prohibitions. Data from these processes can then in turn be fed back into fraud detection and other security algorithms, so that a person ends up “flagged'” elsewhere, a cascading set of blockages. By paying close attention to the evolving deployment of regional barriers on commercial websites now, we have the opportunity to influence the design of systems that may have more influence over people's life chances in the future. Only by facing these realities as they presently exist can we work towards a more just alternative.