Taking Data Literacy to the Streets: Critical Pedagogy in the Public Sphere

Introduction

The Museum of Random Memory (MoRM) is an ongoing series of performative arts–based public interventions designed to spark deep reflection about the underlying complexities of datafication in everyday digital media usage. The project grew out of a 2016 workshop on Creating Future Memories in Aarhus, Denmark, where I invited 15 people from around the world to think about methods for engaging the public around issues of digitalization, social media, data mining, automation of personal memory, and the future of cultural or civic memory. Their mission was to create an exhibit in 24 hours, combining their experiences in activism, art, computer science, museum curation, architecture, filmmaking, printmaking, university administration, law, photography, and computational art. The results were performed at the Counterplay Festival.

Since this first exhibition, MoRM ¹ has been conducted in Denmark, Italy, Spain, Canada, and the United Kingdom, each time using different formats and scales, depending on location, audience, and purpose (see Figures 1-5).

OPEN IN VIEWER

Figure 1.

Collecting, curating, and displaying objects visitors find useless or would like to forget.

OPEN IN VIEWER

Figure 2.

Giant screen showing gradations of remembering and forgetting memories donated online by visitors/cocurators.

OPEN IN VIEWER

Figure 3.

Using lab coats and multiple stations to show visitors why critical data literacy is needed.

OPEN IN VIEWER

Figure 4.

What is the sound of forgetting? Local DJs remix music with historical audio archives of disenfranchisement and stories from passers by.

OPEN IN VIEWER

Figure 5.

Remixing a single memory to showcase how processes of data degradation and algorithmic infrastructures influence how memory might look and feel in the future?

Participants at each MoRM are engaged in an interactive museum experience, where researchers act as “uncurators,” visitors are memory donors, the collective donations of memories are displayed. The practice is framed as an exercise in developing memory archives for future archeologists. We ² focus on how the stuff ³ of memory is transformed as it is digitalized and what this might mean on micro and macro scales. Through this playful experience, donor/participants are prompted to reflect on digitalization, datafication, and their own production of big data. The goal of such an intervention is to spark curiosity whereby participants will over time and after their experience at MoRM, seek out and gain stronger digital or data literacy.

The reason I’m doing this project, as well as writing about it here, is to address—in both a conceptual and activist way—some long-standing questions about the purpose of academic research: How can researchers shift from studying culture as scholars to impacting culture as activists? What do we mean by ‘applied’ research? How can we make the walls of the academy even more porous to bring our methodological skills and training to bear on a world of complex situations that require everyone’s urgent attention?

In what follows, I first make the case—although it is somewhat self-evident—that globally, improving public literacy around digitalization and datafication is urgently needed. I detail two micro-level instances from the MoRM project that illustrate some of the processes of continuous self-reflexive meta-analysis in this collaborative action research. I then zoom back out to the level of the didactic pedagogy at work as we encounter the public and attempt to empower people to act consciously in a digitally saturated society. What emerges are lessons around methodology for effective public engagement related to data literacy, and the value of experiential events for both social change and in creating robust research environments.

Throughout this project, we operate from an idealistic stance. We attempt to train people to become citizen scientists by giving them basic social science skills like asking good research questions, observing behaviors closely, and conducting thoughtful and systematic analysis of (their own) everyday behaviors and interactions. We’re well aware that because the encounters are brief, this is not the outcome. We can, in actuality, only demonstrate how such research of digital experience might happen, by using their examples and experiences as material to work with in conversation. However, through various visual, verbal, and graphical prompts, we are sparking what Paulo Freire (1968/2000) would call conscientização, or critical consciousness.

This project is practical and informative in learning about methods of engagement, and this is where the skillful—or persistent—use of self-reflexivity as a form of meta-analysis becomes relevant. Among other things, this series of experiments is a continuous study of our own methods. The granular level of detail we collect and reflect on in each instance of MoRM helps us build knowledge about how to engage quickly and effectively with participants as passersby, how to capture an individual’s interest around the critical topics of datafication in their everyday life, and how to give people tools they can work with later.

In labeling this effort as a study of method for the purposes of this particular article, I do not intend to dismiss its artistic, participatory, applied, and activist ambitions. Rather, I seek to illustrate how these goals are achieved in the micro level and iterative decisions we make as researchers. Reflexive self-analysis generates a meta-analytic perspective across multiple intervention points. This project can help us understand reflexivity as a honing and sharpening practice that focuses not only on the researcher’s role, position, and values in the overall project but also on the methodology being applied over time in the field. In the case of MoRM specifically and data literacy generally, this level of reflexive meta-analysis aids our effort to engage effectively, sparking stronger critical consciousness among participants.

What Is Data Literacy?

Data literacy is a type of awareness and curiosity that leads to developing competencies needed to grapple with the complex impacts of digital transformation on individual and cultural wellbeing. A critical data literacy is built on the premise that to be critical in any effective or sustained way requires deep understanding of the contexts within which digitalization or datafication is occurring. Data literacy falls into the same concept pool as multi-literacy, digital literacy, information literacy, digital media literacy, and media literacy.

All of these concepts encompass a broad concern that people have both skills and critical awareness to make sense of, respond to, and be empowered with/in the mediated environments in which they live.

The data literacy definition above helps specify literacy as a particular type and trajectory of critical awareness in otherwise vague definitional terrain. For various reasons, there is a tendency among public stakeholder groups to put everything and nothing into discussions around digital/data literacy. A case in point is the 2017 report from the European Commission to parliament on the digital education action plan. In this report, there is a strong focus on building digital competencies for “effectively tackling the challenges digital transformation creates for online safety and cyber hygiene” (p. 8). “Competencies,” in this report, can be interpreted as sets of skills and the ability to use these skills in smart ways. The issues to be “digitally competent” about include such things as safety, security, privacy, wellbeing, and problem solving (et passim). These are admirable and necessary goals, but they are impossibly broad. They also focus on the outcomes of actions made by literate agents, which obscures both the processes of building critical thinking and the complex knowledge required to apply critical thinking in myriad contexts.

Take, for example, some common questions about social media use, which more than 80% of European Union (EU) youth use for primary social activities (Eurostat, 2015): How do platforms like Google, Facebook, and basic photo management software on our computers or smartphones sort and organize our images? How does Facebook choose what to include when it presents me with my “Year in Review”? Why don’t all people see the same newsfeed? How exactly do companies track and collect information as a user clicks on links or conduct a browser search? What’s the difference between an algorithm and artificial intelligence (AI) and how does that relate to which advertisements I see?

These everyday questions about life in the post-digital age are not easy to answer and require more than basic knowledge about computer programming, infrastructures of software and networks, and automated data analytics. Any question we might raise will be embedded in multiple contexts, which means the sort of literacy we need encompasses more than the knowledge to technically answer these questions, a point Renee Hobbs has made for many years about media literacy (Hobbs, 1998; Hobbs & Jensen, 2009). To augment and specify broad conceptualizations of literacy, then, the definition above seeks to shift attention to what is required to have agency in and across complex data-saturated infrastructure.

X_literacy, then (“X” meaning whatever we want to call it), goes beyond answering these questions; it’s a level of comprehension and critical awareness that, once we have it, keeps us asking new questions. This requires deep curiosity + a critical orientation + enough skills to get started + enough credible background information to know your curiosity is warranted. A certain level of confidence and certainty is required to push back against the forces that yield invasive target marketing, uninteresting Netflix recommendations, biased search results, overly narrow newsfeeds, invasion of privacy, or other problems we encounter on a daily basis.

Applying this curiosity to critically scrutinize our own everyday practices is hardly straightforward. For most of us, we likely don’t notice our own data production until we get low memory warnings on our digital devices. As we snap photos, post updates, send and receive email, conduct regular searches, and use online services, we produce what constitutes big data: massive in quantity; stored in multiple locations across multiple chips, memory cards, and clouds; distributed through multiple platforms; only partially accessible to human perception; and literally too large to comprehend or manage effectively without some level of computational assistance. To deal with this, we must first recognize that the minutiae of everyday interactions and activities actually constitute big data. We can then use a string of competencies and critical analysis tools to help an individual understand what this continual production and accumulation means in their own lives. One can then critically analyze one’s own role and abilities in larger media, digital, and data ecologies and understand why this investigation matters.

These examples point to a crucial second part of the definition of contemporary X_literacy: It facilitates one’s capacity to critically analyze and respond to technological and political hegemonies. There is a long history of efforts to raise consciousness of citizens about how they are part of larger ideological systems of power and control, from pedagogy of the oppressed in the streets of Brazil (Freire, 1968/2000; also see adaptations to “theatre of the oppressed” by Boal, 1974/2008) to feminist consciousness-raising circles in the 1960s (Sarachild, 1978). Sonia Livingstone’s articulation below serves as an anchoring accumulation of many legacies about what this type of critical cultural literacy means as well as why it matters:

Media literacy therefore concerns the relationship among textuality, competence, and power. Indeed, literacy is a concept grounded in a centuries-old struggle between enlightenment and critical scholarship, setting those who see literacy as democratizing, empowering of ordinary people against those who see it as elitist, divisive, a source of inequality. Debates over literacy are, in short, debates about the manner and purposes of public participation in society. Without a democratic and critical approach to media literacy, the public will be positioned merely as selective receivers, consumers of online information and communication. The promise of media literacy, surely, is that it can form part of a strategy to reposition the media user—from passive to active, from recipient to participant, and from consumer to citizen (Livingstone, 2004, p. 20).

A third element of the definition that is implied but should be emphasized is that this critical study must acknowledge and find methods for—not comprehending but—managing to think through and with the complexity of the environments within which literacy must operate. As Bulger and Davison (2018) note, “Media literacy is just one frame in a complex media and information environment” (p. 2). Focusing on fake news, they emphasize,

It is cheap to launch a disinformation campaign, to put a thousand different messages out, because only one needs to work. Yet to counter these campaigns, academics, technologists, and policymakers need to understand multiple dimensions, attempt responses from several different sides of the issue, and multiple efforts need to be successful. (p. 15)

Ellen Mandinach and Edith Gummer (2012) note in their comprehensive review of so-called data literacy efforts,

data literacy is complex and highly systemic . . . . there is an entire landscape around the construct that facilitates or impedes effective data use. It is necessary to examine the knowledge and skills required to understand the landscape to understand data literacy as a complex construct. (p. iii)

In other words, whatever we call data literacy and whether or not this is markedly different from media literacy, multiliteracy, or digital literacy, it is certainly more complex than simply learning to access, use, analyze, or create data through statistics or comprehending data visualizations and data calculations (e.g., Mandinach & Gummer, 2012; McCosker, 2017; Markham, 2017, 2018b).

Experimentation, Remix Methods, and an Iterative Research Design

MoRM aimed to ignite data literacy and critical thinking in the public through participatory experimental interventions. A key element of effective participatory research design in the public environment is that each iteration is designed specifically for that situation. In MoRM, this could be described as a “remix methods” approach (Markham, 2013) to enable more playful experimentation than might be otherwise encouraged in traditional academic research environments. First, we embraced a key feature of remix culture that repetition of an experimental engagement with a phenomenon builds meaning through variation. Second, we reflected on but also tried to dismiss any pressure we might feel individually or as groups about the expected processes or outcomes of the events. This embraces another key aspect of remix culture that each intervention is an experiment, the outcome of which is a temporary contribution to a larger ongoing conversation around a topic.

As we let go of the pressure to know beforehand how we might accomplish the “best,” “correct,” or “ethical” intervention, the team can instead work toward these aims in situ with participants. One outcome of this mind-set was playing more freely with cause and effect, or action and consequence, enabling us to tweak, shift, and otherwise respond in the moment to activate greater consciousness among our public participants. We therefore end up using multiple creative and context-specific methods: Within an overall framework of experimentation and critical pedagogy, we mix perspectives and techniques from situational analysis, ethnographic interviewing, museum curation, speculative fabulation, participatory design and critical making, user experience studies, qualitative social science, performance art, theater, rhetorical criticism, computational art, big data analytics, and pedagogy. Although some modes of interaction are planned, others emerge spontaneously, through either serendipity or necessity, classic characteristics of remix and bricolage (cf, Kincheloe, 2005; Levi-Strauss, 1966; Markham, 2018a, p. 46).

Each version of MoRM represents a different tweak in a longer term series of field experiments, drawing on any number of techniques for interventionist research models including action research, participatory action research, performance ethnnography and theater of the oppressed. It also draws on principles described by Dewey and others, commonly applied in flipped classrooms, feminist pedagogy, and heralded as the basis for the socratic method of teaching and learning. This is what Kincheloe (2004) would emphasize as the “tinkering” aspect of the bricolage:

“This tinkering is a high-level cognitive process involving construction and reconstruction, contextual diagnosis, negotiation, and readjustment. Bricoleurs understand that researchers” interaction with the objects of their inquiries is always complicated, mercurial, unpredictable and, of course, complex” (p. 4).

To target datafication and digitalization, we’re drawing on 20 years of prior research around digital device and media usage and combining this with participants’ own uses and interests to explore, with them, what might constitute a stronger literacy. This changes from individual person to person, obviously, and we have noticed great distinctions among participants. The effort also varies per curator/researcher, as each has distinctive styles of communication and background expertise in digital/social media or arts interventions. We do not seek to standardize a set of “good questions” or experiential exercises for participants. We instead focus on a range of questions that will spark social change by way of increased awareness about the personal and public data of our daily lives. This activist ambition connects directly to the influence of Freire’s critical pedagogy (1968/2000), whose work emphasizes how education and research must deal critically and creatively with reality to engage people in the transformation of their world/realities.

Throughout the project, we systematically, albeit in an emergent and casual fashion, conduct self-reflexive meta-analyses of the detailed methods of our encounters. We do this for pragmatic reasons—we want to make a difference in the world and this requires effective public engagement. We have learned that to do this effectively we need to figure out how to frame the questions. The outcome is a continual tweaking of what is already a good thing, as all our events have been highly successful on a number of measures. This can be frustrating, but has yielded a more nuanced understanding of the micro processes of questioning that are capable of sparking genuine curiosity that will drive further interest and over time, further investigation into one’s own social well being within complex digital contexts and data infrastructures.

A Detailed Look at the Value of Good Questions

The following section excavates the detailed processes we used during the Museum of Random Memory. This detail demonstrate some of the iteration, meta-analysis, and remix involved in fostering robust public engagement. In the second installation of MoRM, our primary prompt was, “Would you like to donate a memory to our museum?” The prompt aimed to stop passersby, at which point we could ask deeper questions around the social impact of digitalization and datafication. For all three full days of this event, I spent most of my time testing different questions and reflecting with the other uncurators, many of which were master’s level students, in order to lure people into more detailed conversation. We would repeatedly try out different questions, come together as a group, and then break apart again to test them again and again. We learned that unless we prompted them with a good question immediately, the participant wouldn’t stick around. By contrast, if we asked a really good question, they would immediately be curious and could easily begin to raise other questions.

In the third installation of MoRM, led by master’s level students, we didn’t have time to ask for memory donations and instead set up a booth wherein we exhibited and discussed the reasons why such a museum should exist (Figure 3). We laid out six “stations” for people to walk through and at each, a MoRM scientist wearing a white lab coat would interact with visitors.

To facilitate the student researcher’s (and my own) ability to anticipate the possible directions people might take the conversation, I had each student research team brainstorm and prepare a series of single-sentence prompts or provocations. We developed and then practiced asking closed-ended and open-ended questions to open up various possibilities for critical thinking. Practicing in advance with each other, we swiftly learned what prompts might work better than others. It also helped us identify question types to avoid: We didn’t want to spark long-winded storytelling from participants because we expected around 400 people of all ages would pass by in only six hours.

We spent so much time and energy crafting the right sentences because we wanted people to really engage, to emerge from each station with new questions they could take away. Our ambition was similar to the classroom teacher’s—to get them started down the path of asking certain questions that would lead them to independent investigation and speculative thinking about the topic.

For example, in Station 1, one prompting question was, “Do you want to see how much you’re being tracked and monitored as you shop or find news online?” We chose the word “online” versus “on your phone” or “on your computer” because it was the most generic, appropriate for anyone regardless of age. We used a yes/no question here deliberately, discarding the more interesting (from a qualitative interviewing perspective) option: “have you ever thought about how you’re being watched?” because we didn’t want them to tell us what they thought (yet). Rather, we wanted to get them to interact with the software on the screen and see how much any click would be tracked.

Once the participant stopped walking and turned their attention to our exhibit, we could point out what was happening on the display screen or, if nobody was using it, we could invite them to search for anything they wanted on the Internet to see how many entities would start to collect their data through tracking software. Ghostery, the software we were using, showed these tracking companies as dots on the screen in real time, so as the person accessed three to four websites, fireworks displays would show anywhere from 10 to 300 companies starting to track them. Viewers would be mesmerized. As they absorbed this display of surveillance, we would use the silence to introduce the project of the MoRM with a phrase like, “we’re doing some public experiments to help citizens understand what’s happening under the surface level of our everyday use of digital devices.” We believe following such statements by questions was also an important step. These would be targeted as narrowly as possible. For example, if the person was visibly elderly, we would ask, “Do you use a smartphone or a computer for shopping or viewing news?” We would not waste time with this question if the person appeared to be in the so-called “digital native” category. In this case, we would move to a more specific question, such as “Do you ever wonder if you’re under surveillance or who might be tracking you?” These yes/no questions would be ill advised for an open-ended interview, but we were not interested in the answers. Rather, we wanted them to start thinking about the questions.

To give an example from a different station, we would say,

We’re playing around with how to visualize how our personal phones produce “big data.” You’ve read about the concerns that a lot of our personal data is being collected and analyzed by giant corporations as big data, right? But how much are we paying attention to the massive amounts of data we’re producing for ourselves? For example, think of all the pictures we take on our phones. How many pictures do you have on your phone right now?

Again, making a string of statements and questions like this without waiting for the person to answer would not be suitable for an ethnographic interview. But we used this string of statements and questions as a lure to get people to bring out their phones and start exploring their own image making and storage practices.

Once they had their phone out, we could observe their interaction with their device, walk through the process of figuring out where their images were stored, and eventually, put a number on the chalkboard where we were keeping a tally. Participants could compare their number to other participants’, or watch the total building. As they worked toward finding the number of photos they’ve taken, we could begin to chat about how algorithms are used by Google or Facebook or phones themselves to auto-curate images, asking, “Have you ever thought about the fact that unless we deliberately manage our photos, they’re organized for us by these large companies?” “What about our future generations? How will they ever find the relevant photos among all the junk we save?” Many of them would begin to ask technical questions such as “Do my images stay on the cloud even if I delete them from my phone?” or “How does an algorithm filter my photos?”

If we started with these last questions about future generations, we didn’t get as much engagement with visitors and some of them would simply nod politely and keep walking. We attributed this to the fact that the questions were too complex to begin a conversation and should be placed later. If we had not been paying attention to our methods of questioning, we might have attributed their lack of response to their disinterest. For me, the ordering of questions and statements became a game, and I would test different ways of getting them to literally stop in their tracks.

This exhibition was also a lesson in the importance of redirecting questions back to participants to incite further thinking. This was challenging on multiple levels; I spent a lot of time floating between the stations to help student researchers field what we perceived as good questions. One station, for example, focused on how big data and multiple data sources make generalizations about people, which was good if you wanted personalized advertisements but bad if you were concerned about predictive policing. If a participant asked “How does an algorithm work anyway?” we would answer the question superficially, but follow up immediately by saying something like, “Right? That’s a good question and shouldn’t we all know the answer to that these days?” When asked “how does facial recognition work?” we could pivot to a similar question like “Does facial recognition work?” to make it more feasible for us to answer, “Not very well! Sociologists have found recently that algorithms are often incorrect in facial recognition and misclassify people.” This could (and did) prompt long conversations about a range of topics, so we had to be willing and prepared to adequately engage in deeper discussions.

Prompting Through the Interfaces

In addition to experimenting with face to face interactions, we tested how the design of the space, whether a physical space or a digital interface might prompt people in useful ways. The first installation of MoRM at Counterplay Festival in 2016 was quite deliberately analog, located in the open floor plan of DOKK1, a new media space/library in Aarhus Denmark. Calling themselves “uncurators,” the researchers asked participants to “donate something useless.” This could be ideas, images, or objects that they wanted to forget. Using professional practices of museum officials, uncurators used a paper intake form to meticulously annotate the artifact and obtain the donor’s story behind the donation. The objects were sealed in plastic bags, hung on strings, and displayed across the space on the second day of the festival, as befitting objects d’art (Figures 1 and 6). It was called a “Museum” to invoke the idea that we would hold, care for, and exhibit memories. At the same time, the team disrupted the permanence and structure of the museum by building it as an imaginary in an open public space, having a grand opening reception and then permanently closing both the exhibit and the museum after 2 hr.

OPEN IN VIEWER

Figure 6.

In 2016, participants were asked to donate something random, useless or forgettable, for example, something in their pockets. In this image, a taxi card.

The second MoRM at Counterplay Festival in 2017 built on this idea but changed the question. We asked participants to donate visual representations of memories. To collect and display these, we created a digital interface to accept memory donations (Figure 7). Rather than focusing only on forgetting, we asked people to “consider the extent to which you would like to remember or forget your memory.” This seemingly simple prompt was developed over several weeks of design meetings and iterative prototyping (for more on this, see Markham & Pereira, 2019).

OPEN IN VIEWER

Figure 7.

In the second installation, participants could choose to donate a memory through a digital interface. In this image, the website visualization of donated memories.

We settled on a design that allowed (forced) memory donors to select on a scale from 1 to 10 how much they want their memory forgotten or remembered. As users moved the slider (Figure 8), the opacity of the image would increase or decrease (Figure 9). Allowing the user to fine-tune how they wanted to remember or forget something led them to pay a lot of attention to their memory, so “forgetting” was never an actual option. Indeed, the goal was not related to the memory itself, at least not for us as researchers, but on getting people to build to a critical threshold of curiosity that would get them asking more questions.

OPEN IN VIEWER

Figure 8.

The donation interface, where the opacity of the image changes depending on how much the donor chooses to forget or to remember this memory along a slider. They can choose along the scale of 1 (totally forget) to 10 (totally remember).

OPEN IN VIEWER

Figure 9.

A collage of three submitted memories (and the position of the slider): completely remembered (left), partially forgotten (middle), and completely forgotten (right). The text is as faded as the images.

In our public exhibition/installation, this web interface became a key interaction point with participants. Aside from their curiosity about the slider, or maybe because of it, most were intrigued by what it meant: “What do the different numbers between 1 and 10 mean?” “What does it mean if I want my memory to be forgotten? What are you going to do with it?” These questions served as a discussion starter for larger questions: “How would you like to forget your memory? What does it mean to remember something in the Internet? Should everything be remembered?”

The point of the gradient, or slider, was not collecting data more efficiently, but to generate an interaction, constrained by the interface, that would perhaps activate some level of critical thinking about their agency (or lack thereof) in the future trajectories of their digital data. Their contribution would be accompanied by viewing the online gallery of all the donated images, some of which might appear maddeningly blank or barely visible (Figure 10). We would also invite them to look at the large-scale view displayed on a 20 × 30 foot screen (Figure 2), asking them what they thought about how other memory donors chose to visually depict both their memory and the degree to which they wanted to have their memories remembered or forgotten. ⁴

OPEN IN VIEWER

Figure 10.

Different shades of forgetting: from slightly transparent to completely invisible.

The Importance of Reflexive Qualitative Analysis for Critical Action Research

Both of these examples involved countless moments of qualitative self-directed analysis of our actions in the field. These occurred at the very immediate level of the next person in line to see the exhibit, and at broader or longer levels, as we built and tested prototype web interfaces, sketched the layout of the physical space, or discussed how people might move, physically and conceptually, through the experience. Although some moments were very obvious, others were microscopic. When designing the web interface, for example, we would test the rhetoric of having 7 versus 10 variations of remembering/forgetting on the slider. We tested whether the stopping points should replicate an analog slider (like old fashioned radio dials in automobiles) or have discrete stopping points (like digital radios stopping on points in the spectrum).

Each version of a conversation or event is part of a larger experiment to make people think differently, which we took as a pedagogical and rhetorical challenge. It’s also about creating models for public engagement that lead to social change on small and large scales. With this ambition, the research practice cannot be haphazard, even though the events themselves seem playful, spontaneous, and even chaotic. Thinking about this through lens of action research design, we have found that the assessment of what works is ongoing, and each future shift in design depends on what happened and if it seemed to accomplish a productive outcome, which may or may not be the same as what was anticipated.

Retrospection is key to analytically strengthening the format of the exhibitions and the micro interactions with participants. We don’t have access to most of the participants after the fact and although we have gathered contact information from them as a part of collecting personal materials from them, we have not found it necessary, yet, to engage in in-depth follow-up with individuals. Rather, we use ourselves and our own reactions to help assess the quality of the prompts. We test it among ourselves as a design and research team, which is generally around 20 people from different countries of origin, ages, ethnicities, level of education, professions, and interests. After each event, we have half or full day meetings to discuss the process, recapture important moments and encounters, and to decompress. Recording these sessions in audio/video, and collecting sketches and notes, we now have layers and layers of data about the process. Whether or not these are ever collated, annotated, or analyzed later is less important than our faith that this documentation is available for us to access later if we need to.

We’re not trying to create best practices for digital or data literacy, but to engage with the public in ways that work. This acknowledges that the nature of one’s audience for such efforts can never be understood in advance. Despite the political pressure to find ways of standardizing literacy education, we understand that the actual situations in which such literacy is prompted are highly contextual. A variety of actions and strategies could work equally well. We’re collaborating with participants on the fly to find the best methods to prompt critical reflection and change. Put differently, we are studying the constantly mutating characteristics of successful interventions, which includes on-the-spot variation of what might be called “control variables” (visual, technical, and conversational prompts), continual iteration with new participants—who count as both collaborators and variables, and random variation of multiple other inputs.

Reflexivity involves a continuous analysis of one’s own actions and choice of tools at each juncture, as well as a critical reflection about the situatedness of one’s own role, position, and ethical stance in the situation. When done well, one can be in the zone, in a state of readiness to simultaneously respond to the specific needs of the persons in the context, reflect on whether or not our response (and theirs) is adequate for the purposes of raising their levels of consciousness, and observe our own actions and responses throughout the experiment.

The goal of this sort of approach, in line with action research design or critical pedagogy, is to evoke a response. We focus on the material element that seems to cause change. What that change looks like is unknown, but we’re not interested to identify patterns of cause–effect but rather to provoke a cause–effect sequence.

Academics and Social Change

One strong outcome of our experiments, experienced at each of the MoRM events in the past 2 years, is consciousness raising, in the classic sense we might associate with the use of this phrase in the late 1960s as a part of the U.S. women’s feminist/liberation movement. The interactions between researchers and participants made possible by the design of the exhibition spark engagement, critical thinking, and curiosity. Participants become more aware of their own digital media use, such as the scope and size of their production of personal “big data.”

They leave the exhibition with more questions than answers and, as many have remarked, more desire to think about the fraught or complicated nature of their own relationships with their digital devices. Many participants leave with a greater worry about how their own or their community’s heritage might be lost or reframed in the future depending on how data are stored or lost, or whose data are stored or considered valuable by entities that write history. Consciousness raising is a powerful interpretive lens, suitable for both inwardly directed introspection and outwardly directed social change.

Whether one draws on Gramsci, Freire, or Sarachilds and countless women in 1960s kitchens, consciousness raising is not about reaching a new point or state of knowledge, but recognizing that there’s a crisis that needs to be addressed. Applied to the possibly less fraught situation of everyday digital life, MoRM is prompting participants to ask themselves what happens (or could happen) to the stuff they produce as they engage in everyday use of their smart devices. When we, and by we I mean those of us who teach or supervise others, those of us with some skill in teaching qualitative methods or research design, are there to guide such exploration, we can help make these explorations stronger. We do this naturally on small scales, when we use our knowledge of research design in everyday conversations with our friends and families about how they might observe, ask questions, and analyze some phenomenon. MoRM functions at larger scales and each of the “uncurators” uses either their pedagogical skills or qualitative research skills to help people dig deeper, and more analytically, into their experiences, practices, and understandings. These are small steps, micro interventions into larger patterns. As I was told by my mentors for many years, we take small steps in each class we teach, to change the world by changing how each student thinks.

The effort is at times frustrating because we’re not sure the intervention works as well as we’d like it to, or in the ways we wish it did. A key challenge is that it is difficult to ascertain if the prompt causes the sort of critical thinking that will get participants to ask further questions of themselves, when they leave the exhibit/event. As an engagement, intervention, and creative experiment, we spend our energy and time on the design and performance of the events. By design, this does not prioritize the sort of social science methodology that would suggest we do pre- and post-test questionnaires or in-depth interviews to help identify some measurable learning impact. ⁵ Rather, the action in the action research is a spark that helps people come to this realization. The methodology is not interested in a long-term evaluation on change, but in the momentary and embodied interaction between researcher and participant, and the moments among researchers. By employing iterative and remix methods the best possible interaction is developed during a MoRM event. A network of experiences build between all involved parties who inevitably carry forward their memory of the event, and who have now practiced data literacy. The value is in this embodied experience–a consciousness raising event–, and its effectiveness measured by the active researchers.

The process speaks to the academic problem of letting go of the perceived right way to do something (whether one is doing something in a lab, a classroom, the ethnographic field, the interview, or some other place/form of inquiry) and instead, placing value on the iterations of participant practice that happen in situ. Reflexivity at both the individual and collective research team level can help reveal some of the elements of the (academic) situation undergirding both the questions and the doubts around this method.

By first reflecting and then critically analyzing my own stance and approach, I recognize what is inhibiting my ability to be fully present and responsive to the specific needs of the situation. I also recognize that my impulse to lay the burden for effectiveness on the measurable outcome of the experiment is the wrong way about it. Because that is not how art, art-based research endeavors, public interventions, or activism work. In all the installations, the initial prompts change, and the outcomes can’t be measured. This is the deliberate design. Strong interest, curiosity, and participation result in excellent conversations between social media experts and lay public and between teachers and people interested in researching their own lived experience. But to reverse engineer these to find any certain causal chain of input and output is impossible. Indeed, the distinctiveness of the intervention might result from the title of the exhibition: “The Museum of Random Memory,” rather than by anything specific we might say as we greet participants. The title—and perhaps more broadly, its ability to generate curiosity—functions as a prompt on a larger scale.

These realizations are not in isolation—indeed, the successful collaboration among researchers from radically different disciplines requires this level of personal reflexivity. A reflexive stance can reveal broader patterns as well as unproductive premises by challenging everyone on an interdisciplinary team to identify their working assumptions and acknowledge their blind spots. When this goes beyond the surface of methods and into the deep structures of analytical and representational processes, it can build better methods. This is not because the methods are more accurate or precise through triangulation, but because the researchers are continually reflecting on their approach, their micro practices, simultaneously defending their own ways, listening to others do the same, and by focusing on the mutual goal or problem together, improving their collaborative approach through each iteration.

Most of the time, we think of qualitative methods as tools or epistemologies to study social life, but they are always also at work at the meta level, where we use qualitative methods to assess other methods, such as the engagement and intervention methods of our research teams at each MoRM event. The case of MoRM emphasizes how effective public engagement, intended to build digital and data literacy, requires dexterity in finding and continually adjusting statements, provocations, questions, material and digital props, and other prompts that best fits the person, the content of the interaction, and the context of the conversation. Qualitative researchers are well equipped to study the micro-level processes involved in such encounters. My own training and experience in interpretive and reflexive methods has enabled me to build critical self-analysis, iterative experimentation, and continuous adjustment into the action research design, with teams of seasoned and junior researchers. Combined with a strong tendency toward didactic models of pedagogy, the process and the outcome together is a robust, but constantly moving target.