Abstract
Keystroke logging has become instrumental in identifying writing strategies and understanding cognitive processes. Recent technological advances have refined logging efficiency and analytical outputs. While keystroke logging allows for ecological data collection, it is often difficult to connect the fine grain of logging data to the underlying cognitive processes. Multiple methodologies are useful to offset these difficulties. In this article we explore the complementarity of the keystroke logging program Inputlog with other observational techniques: thinking aloud protocols and eyetracking data. In addition, we illustrate new graphic and statistical data analysis techniques, mainly adapted from network analysis and data mining. Data extracts are drawn from a study of writing from multiple sources. In conclusion, we consider future developments for keystroke logging, in particular letter- to word-level aggregation and logging standardization.
Due to the emergence and widespread use of digital media such as personal computers and smartphones, an ever increasing number of texts are produced in a range of different contexts and by people of different backgrounds. Although the general characteristics and cognitive activities that underlie such digital writing have not substantially altered, these technological contexts create new challenges and certainly also new opportunities for writing research.
Several research methods have been developed to conduct online writing research (Alamargot, Chesnet, Dansac, & Ros, 2006; Latif, 2009; Sullivan & Lindgren, 2006; Van Waes, Leijten, & Neuwirth, 2006; Van Waes, Leijten, Wengelin, & Lindgren, 2012; Wengelin et al., 2009). These methods enable researchers to study the complexity of writing processes in the workplace, in educational and health settings, and in terms of development, mechanics, and learning disabilities. Keystroke logging is one of those.
Keystroke logging programs are designed to observe writing processes on a computer (for a review, see Latif, 2009; Van Waes, Leijten, Wengelin, et al., 2012). These programs log and time stamp keystroke activity to reconstruct and describe text production processes. Thanks to the evolution in writing research in general and the development of more articulated theoretical frameworks in particular (Berninger, 2012; Kukreja, Stevenson, & Ritter, 2006), measures derived from keystroke logging can now be better interpreted (Baaijen, Galbraith, & de Glopper, 2012; Van Waes & Leijten, 2012). Moreover, recent technological developments have led to an improvement in both the logging procedures and the logging analyses: the logging data and analyses that these programs provide have become very sophisticated. In this article we explore recent evolutions in keystroke logging as a research method, taking Inputlog as a generic example (www.inputlog.net). This program was developed in close collaboration with the developers of two other keystroke logging programs, namely TraceIt/JEdit and Scriptlog, and draws on the same basic concepts. For a comparison between these programs, we refer to Van Waes, Leijten, Wengelin, et al. (2012) and the WritingPro website (www.writingpro.eu).
The article opens with a brief overview of keystroke logging methods, focusing on what kind of research questions can be addressed efficiently with them. Next, we walk through the different components and characteristics of one specific keystroke logging program, Inputlog, and illustrate the program’s use. The following sections focus on an illustration of the complementarity of keystroke logging with other research and observation techniques and on an exploration of recently developed data analysis and visualization techniques, especially in the context of writing from multiple sources. We conclude with avenues for further research and development.
Keystroke Logging in Writing Research
One of the earliest attempts to use keystroke logging in writing research was described by Bridwell and Duin (1985). Their article demonstrates not only how the advent of the personal computer changed the writing context radically but also how it created new opportunities for writing research. The tools that have been developed since are designed to record keystrokes and sometimes also mouse movements and clicks. These logged data are then made available for further analysis and/or enable an exact replay of the emerging text. At the moment, the most widely used keystroke logging programs in writing research are Scriptlog (Andersson et al., 2006; Strömqvist, Holmqvist, Johansson, Karlsson, & Wengelin, 2006), Translog (Jakobsen, 2006), and Inputlog (Leijten & Van Waes, 2006; Van Waes & Leijten, 2006). Each keystroke logging program has its own focus. Scriptlog is mainly developed for experimental research, optionally in combination with eyetracking. Translog is developed for experimental research on translation based on source texts. The focus of Inputlog is on multimodal professional writing environments (cf. section Inputlog). Next to writing research, keystroke logging is also used in other research areas like human–computer interaction—for example, Mousetracker (Freeman & Ambady, 2010), Recording User Input (Kukreja et al., 2006; Morgan, Cheng, Pike, & Ritter, 2013)—and biometrics (e.g., Douhou & Magnus, 2011).
Research applications for keystroke logging in writing include a wide range of areas: studies on cognitive writing processes in general, writing strategies in professional writing or creative writing, the writing development of children—with and without writing difficulties—spelling, first and second language writing, and the writing of expert and novice writers in professional contexts and in specialist skill areas such as translation and subtitling. The research technique can also be used in educational settings: second language learning, computer literacy, spelling, and typing skills. For example, Lindgren and Sullivan (2003) have used keystroke logging to elicit reflection on writing activities through peer-based intervention by replaying the writing session.
The main rationale behind keystroke logging is that writing fluency and flow reveal traces of the underlying cognitive processes. This explains the analytical focus on pause (length, number, distribution, location, etc.) and revision (number, type, operation, embeddedness, location, etc.) characteristics. As in speech, pause times are seen as indexical of cognitive effort. Several studies (cf. Spelman Miller, 2000; Wengelin, 2006) have shown that pause length increases with text unit level. In general, pauses between letters within a word are shorter than those preceding a word; pauses between sentences are shorter than those between paragraphs. Also grammatical, discourse, and morphological boundaries affect pause length (Nottbusch, Grimm, Weingarten, & Will, 2005; Spelman Miller, 2006).
Revisions on the other hand are taken to indicate a discrepancy on a certain level between the writers’ intentions and the text produced so far (Leijten, Van Waes, & Ransdell, 2010; Lindgren, Sullivan, & Spelman Miller, 2008). Revisions relate to grammatical, content-related, or surface-related problems that are observed during text production (Faigley & Witte, 1981; Van Waes & Schellens, 2003).
Toward a Multimethod Approach: Keystroke Logging in Combination With Other Logging and Observation Techniques
In their classification of writing research methods, Janssen, Van Waes, and Van den Bergh (1996) classified observational methods on two dimensions (Table 1):
Elaboration of Classification of Writing Observation Methods.
Source: Based on Janssen, Van Waes, and Van den Bergh (1996).
EVP = evoked potential; fMRI = functional magnetic resonance imaging.
Synchronous and asynchronous (vertical axis)
Direct and indirect (horizontal axis)
Synchronous observation methods gather information about cognitive processes during the writing process. With asynchronous observation methods, cognitive data are gathered after writing. Directness refers to observation methods that claim to provide relatively direct evidence about writing cognition. Indirect research methods are used to make inferences about human cognition from process or product characteristics. Table 1 shows an elaboration of this classification.
This table illustrates that all observation methods have their strengths and weaknesses, and it is important to be aware of these when setting up writing research. In the case of keystroke logging, the method is mainly restricted to synchronously observing “translation” (or “formulation”) and “transcription” processes in order to infer underlying cognitive processes. In terms of Hayes’s latest writing model (Hayes, 2012a), the methodological contribution of keystroke logging is mainly to be situated at the so-called process level. Hayes built his model on three levels: control level (e.g., goal setting), process level (writing processes and task environment), and resource level (e.g., working memory). In his model, the importance of the translation and transcription process (process level) for writing studies is convincingly demonstrated. Moreover, the “transcriber” is added to the new model as one of the most important components and now plays a central role in the model. He mainly grounds this addition on developmental studies (Alvès, Castro, de Sousa, & Strömqvist, 2007; Berninger, 2000; Berninger, Whitaker, Feng, Swanson, & Abbott, 1996), which clearly show the critical role of transcription in writing.
However, keystroke logging is certainly not limited to studies related to the process level. Because logging data also provide information on the pausing and revision behavior, we are able to make well-founded inferences that draw on the resource level (i.e., working memory, long-term memory, reading, and attention; see Hayes, 2012a). A good example is the study of pause bursts (Chenoweth & Hayes, 2001; Hayes & Chenoweth, 2006). Pause bursts are defined as the number of characters produced between two pauses that exceed the given pause threshold (i.e., 2 seconds). The results show that language skills and the available working memory capacity affect burst length, thus linking the process and resource levels.
Although no keystroke logging was used in this original research, it goes without saying that keystroke logging enables researchers to address this kind of research issues more easily and also enables them to conduct these kind of studies on a larger scale. Alvès (2012) remarks, for instance, that “the relation of bursts to writing quality, fluency, writing difficulties, and crucially syntactic units remain largely unexplored” (p. 593). If we want to further explore this kind of research agenda in depth, large-scale observation studies are needed. These studies should, for instance, address the relation between process and product characteristics, taking into account personal preferences (writing profiles, age, expertise, etc.) and genre characteristics. Only by deploying adequate technical instrumentation can these kinds of goals be accomplished.
This article illustrates the research potential of keystroke logging, taking Inputlog as a generic example. Our main focus is on recently added features that illustrate current developments in the use of this research technique that enable researchers to address new research questions in writing studies.
Description of Inputlog
Most writing logging tools were either developed for a specifically—more experimentally—designed word processing environment, or not adequately adapted to Windows environments. Hence they could not be used for research in more naturalistic settings nor for writers using commercial word processors that are used to copy and paste from the Internet or produce text with speech recognition. On the other hand, logging programs that were developed for human-computer interaction studies do not provide writing-oriented analyses. These elements have led to the development of Inputlog. 1
Inputlog features five modules:
Table 2 shows the general output format illustrating the output features. Every row represents one log event. In this example the word “Test” was typed and then “of Inputlog” was copied into the text using the shortcut Ctrl-V. In the third row the focus-change indicates that MS Word was activated (document title: “Wordlog.docx”). Within this Word environment the cursor position and the document length for every event are represented. For instance, the document length increases by one when typing, but when “of Inputlog” is pasted the document length increases by 11 characters. The other columns show the start time (key in) and end time (key out) of every event in milliseconds. These data are used to calculate the action and pause times. An algorithm identifies the pause location and renders a classification in the last column. Mouse clicks are represented by x/y values on the screen.
Example of the Opening Section of a General Output File (Inputlog 5).
It goes without saying that the fine-grained level of event recording has two sides: On one hand it allows for very detailed analyses, on the other hand the huge amount of data is sometimes hard to interpret. Therefore, the analysis component allows for exploring the logged data from different perspectives: product/process, pauses, revisions, and multiple sources. All these analyses are grounded in theoretical and empirical findings, recent research, and discussions with experts in the field. All logging files and analyses are based on algorithmic processing of the raw logging data. The resulting XML files contain specific and/or statistical information about a particular writing session in order to assist researchers in revealing certain process characteristics.
For instance, the summary analysis displays the number of characters, words, sentences, and paragraphs produced, the product/process ratio (total number of characters in the final text/total number of characters produced during the writing process: a ratio of 1 means that no revisions took place), average pause times (based on the defined threshold and the location), different writing modes used (keyboard, mouse, and speech), and length of pause and revision bursts (e.g., Hayes & Chenoweth, 2006).
The pause analysis looks at every nonscribal period. The pause threshold can be set to any user defined level (e.g., 0, 1, 2, or 5 seconds). Pause data are generated on a more general level (number of pauses, mean and standard deviations of pause length) and on a more specific interval level in which the writing session is divided into 10 time slots. Pauses are also classified at each text level, namely within and between words, sentences, and paragraphs. Finally, the number and the length of pause bursts (P-bursts) are reported.
A completely different perspective is presented in the revision analysis. This analysis offers a revision matrix, and a so-called S-notation can be generated. The revision matrix is a linear representation of all insertions and deletions that occurred in the text (revised text fragment together with a time stamp, number of edits, number of characters in the final text before and after the revision, and the location of the revisions in relation to the point of utterance). The S-notation, on the other hand, is a delinearized representation of the evolving text showing normal text production, inserts (both typed and copied), word-level revisions, and deleted fragments (see also Matsuhashi, 1987; Severinson Eklundh & Kollberg, 2003). Fragment 1 describes two revisions in the sentence “Questions of science, science and progress.”
The first operation is a basic one: The writer deletes the last word he has just typed by backspacing the full stop and the word “progress” at the point of utterance and then types “evolution.” At that moment the writer detects a typing error (revision index2) and inserts the letter “e” in the third word of the text (“science”). So, the S-notation is a way of mapping recursivity in a linear representation by indexing the start and the execution of the revision separately (with a subscript and superscript index number, respectively).
The source analyses will be explained and demonstrated in the studies presented below.
Combining Keystroke Logging With Other Research Methods
Another aspect that needs to be highlighted with respect to keystroke logging is that because of its unobtrusiveness, it is easy to combine with other research techniques and tools. In this section we briefly discuss examples from studies in which keystroke logging was combined with other synchronous observation methods (Table 1), namely thinking aloud protocols (synchronous—direct) and eyetracking recordings (synchronous—indirect). Our aim is to demonstrate that the combination of observation methods opens up perspectives to deal with research questions that relate to components in the writing model that are difficult to address via keystroke logging solely (e.g., goal setting or reading). The studies themselves are not presented in depth because our main aim is to illustrate the complementarity of data collection techniques.
Keystroke Logging and Thinking Aloud Protocols
To collect cognitive data during the writing process, many researchers have used “thinking aloud protocols” (TAPs), either concurrent or retrospective. Starting from the early work of Matsuhashi and Hayes (Hayes & Flower, 1980; Matsuhashi, 1982), TAPs have been used to address a variety of research objectives in writing research and related domains, including the relationship between online management and text quality in narrative and argumentative texts (Beauvais, Olive, & Passerault, 2011), the development of academic writing (Llosa, Beck, & Zhao, 2011), the influence of new media on writing (Leijten, Van Waes, & Janssen, 2010), and the comparison of first language and second language writing (e.g., Van Weijen, 2008). TAPs were developed in the field of cognitive psychology by Ericsson and Simon (1984, 1993). By having writers verbalize their thoughts when writing, the researcher collects cognitive data related to (conscious, nonautomated) problem-solving or decision-making strategies (Krings, 2001; Smagorinsky, 1989).
Because keystroke logging is an unobtrusive research instrument, it is possible to combine logging with TAPs. Both research methods complement each other in several ways. Of course, one has to take into account that having writers verbalize their thoughts simultaneously may disturb the problem-solving cognitive process and the fluency of writing as such (for an overview of the validity and reliability issues related to TAP, see Dam-Jensen & Heine, 2009; Fox, Ericsson, & Best, 2011; Göpferich, 2008).
A study by Schrijver, Van Vaerenbergh, and Van Waes (2012) on “transediting strategies” in translation processes is an illustration of how concurrent TAPs and keystroke logging data may complement each other as well as compensating for some of the shortcomings mentioned above. More specifically, TAPs often reveal strategic reflection and considerations in planning and revision that are sometimes difficult to infer from logging data.
To study transediting, the participants were given a translation assignment in which they had to translate an English patient information leaflet (source text) about a medicine not yet commercialized in the European Union. The Dutch target text had to be in accordance with European legislation and guidelines (terminology, norms, standards, structure, and layout). In this study the participants were prompted to produce a simultaneous TAP, and their writing activities were logged with Inputlog.
Table 3 shows a 15-second excerpt taken from the transcribed data illustrating a low-level process. In the linear keystroke logging representation, we see that after about 10 seconds, the writer pauses for about 3 seconds before writing the verb “gebruiken” (use). This is a pause that is hard to interpret. However, in the TAP we see that during that pause the translator hesitates about the word choice (“to use” vs. “to consume”).
Example of a Combined TAP and a Linear Keystroke Logging Representation.
In Table 4 we present an example of a higher-level process. The production of the first item in a bulleted list (“when you suffer from a psychosis”) is interrupted by a long pause of about 1 minute between the first word (“when”) and the rest of the sentence (see linear representation: •wanneer• [56,906 ms]). The TAP transcript reveals the strategic considerations during this interruption of the sentence production. The start of the bulleted list triggers the writer to represent certain items in the source text differently in the target text. She also shortly explains the reason for this reorganization and then continues writing. Again, the TAP and the keystroke logging data complement each other. On the one hand, the keystroke logging transcript shows an exact representation of the writing process, including the exact pause length and minor automated actions like the correction of typing errors. On the other hand, the TAP reveals the contents of reflections that lead to interruptions in the text production (longer pauses).
Example of a Combined TAP and a Linear Keystroke Logging Representation.
Another possibility is to combine keystroke logging with retrospective TAPs. As mentioned above, the use of retrospective TAPs can help overcome certain validity issues that are inherent to concurrent TAPs. To illustrate this combination we refer to a study by Leijten, Van Waes, and Janssen (2010) on revision strategies of professional writers who use speech recognition as their primary tool for text production and revision. More specifically, the researchers were interested in the way in which this new technology affects the cognitive processes that underlie text production. To register the process, they opted for a combination of Inputlog, Dragon Naturally Speaking (Nuance), 3 and another observation tool: Morae (Version 1.3). Morae was mainly developed for usability testing and uses an online screen cam (Morae Recorder) to register every action on the computer screen. A remote viewer makes it possible to “flag” important moments during the writing process, which can later be used as a basis for a stimulated retrospective interview. This type of interviewing offers a valuable complementary perspective to further analyze and interpret the related keystroke logging episodes because in the retrospective TAPs underlying strategies are often explicated.
Keystroke Logging and Eyetracking
The log data of Inputlog, Scriptlog, and Translog can also be merged with log data of eyetracking devices (e.g., Eyelink, Tobii). This enables researchers to characterize reading activities during the writing process. Using eyetracking data in reading research has a relatively long tradition (Rayner, 1975). In writing research, however, only a few projects have been conducted in this field (Andersson et al., 2006; Van Waes, Leijten, & Quinlan, 2010; Wengelin et al., 2009; Wengelin, Leijten, & Van Waes, 2010). Until a few years ago, no adequate research tools were available to study the process of reading during writing. More recently, however, the combination of keystroke logging and eyetracking data has created a better basis to interpret eye fixations in written texts that are constantly changing (insertions and deletions). In this section we show some of the possibilities that reading during writing data provide for writing process research.
The simultaneous logging of eyetracking data and keyboard-and-mouse events enables us to analyze the interaction between reading and writing. The main objective is to get a better insight into the function of the different types of reading that feed and support the distinct subprocesses of writing. The most important obstacle to addressing this issue is the concept of “emerging text.” We should take into account that in eyetracking reading research small units of static text have been the main focus. Eyetracking has been successfully adopted to the study of basic reading processes and to that of syntactic parsing, but there are surprisingly few studies where eyetracking is employed to examine global text processing (Wengelin et al., 2009). A likely reason for this might be the apparent lack of consensus on the measures to be used to tap into global text processing. (Hyönä & Lorch, 2004, p. 131)
In writing process research the most suitable measures for describing reading processes during writing in detail are still being developed (see Wengelin et al., 2009, for a description of Timeline and Eyelink). The Timeline software, for instance, developed by the Scriptlog team, is a tool that synchronizes and visualizes keystroke and eyetracking data for different predefined areas of interest. It generates “a temporally ordered graphical representation of what writers looked at and what keys are pressed during the writing process” (Johansson, Wengelin, Johansson, & Holmqvist, 2010, p. 839).
So far, research has mainly been limited to what writers read during writing and at what moment in the writing process writers look at the screen (or the keyboard). In this section we introduce more fine-grained analyses of the various activities of reading during writing by describing the reading activities of two kinds of writers: monitor gazers (writers who mostly look at the screen and often type with 6 to 10 fingers) and hunt-and-peck typists (writers who mostly look at the keyboard and often type with 2 to 5 fingers) .
The data are based on a short typing test (Leijten, Van Waes, Galbraith, & Torrance, 2011). The writers were typing a nursery rhyme that they knew by heart so as to lower the cognitive load caused by content generation as much as possible.
Figure 1 shows a small excerpt of a hunt-and-peck typist. This is a very straightforward example of “reading during writing.” The figure was created by merging the eyetracking (Eyelink) and keystroke logging (Inputlog) data.
Reading during writing behavior of hunt-and-peck typist.
In this example, the writer is typing the nursery rhyme “Twinkle, twinkle, little star.” When he starts typing (see square dots on x-axis), he apparently almost immediately “feels”—proprioceptive awareness—that he makes a typing mistake in the second word (“twilnlk” instead of “twinkle”). He monitors the text on the screen to identify the error in the text produced so far (see rereading diamonds on top line indicating eye fixations in the text produced so far). After correcting the typing error, he continues to type and produces seven words without looking at the screen, not even while typing. Then he rereads this part of the text and continues. This example is typical of a hunt-and-peck typist who produces new chunks of text in quite large “bursts” without significant pausing and without monitoring the text produced so far on the screen (Chenoweth & Hayes, 2001; Leijten, Van Waes, & Ransdell, 2010).
In Figure 2 a short selection of the writing process of a monitor gazer is shown in which the concept of rereading needs to be subdivided in (at least) two categories.
Reading during writing behavior of monitor gazer.
This writer is constantly monitoring the text produced so far on the screen while typing, more or less following the moving cursor. In this figure, the category “reading” stands for reading during fluent typing. Fluent typing is calculated as a sequence of keystrokes that is produced with an interkey interval of less than 300 ms. The figure shows a very rhythmic pattern of text production and monitoring of the screen, only interrupted by the correction of typos. The first two errors “and w” and “may” are fixated and corrected almost immediately (see black squares on top line indicating rereading activities that indicate “awareness” of the error in the text produced so far). The final error—displayed in a black and white square on the top line—has also been corrected immediately, but it has not been really fixated by the writer. She must have felt the error and corrected it without having to look at it on the screen (parafoveal vision). Apparently, the writer can focus on the text production and does not get distracted by correcting this typing error as such.
These examples show that merging keystroke logging data with eyetracking data sheds new light on the reading-writing interaction that characterizes writing processes. By integrating both types of data, new perspectives are created to analyze writing data, and these allow researchers to make more solid inferences about underlying cognitive processes that relate to “blind spots” in keystroke logging.
Another approach to finding reading-during-writing patterns is presented in the next section and focuses on the interaction with multiple sources and the use of data-mining techniques.
Empirical Illustration: The Composition of Tweeting and Emailing
The previous section illustrated the complementarity of keystroke logging with other research and observation techniques. In this section we explain new data analysis and visualization techniques in the context of writing from multiple sources. Current models of writing capture much of what is important about bookish, singular writing processes, but in our opinion they do no really help us to better understand composition processes that draw on multiple (digital) resources. Nowadays writing seldom starts from a blank screen (especially in professional writing). Contemporary writing takes place in a digital context in which writers have easy access to a wide variety of resources that are only a mouse click away. This interaction has become a fundamental characteristic of writing processes more than ever.
Typical activities related to this kind of writing processes are searching for new information and reading online task-related materials. Although a lot of information might be only a mouse click away, searching relevant information is still a very complex activity. Also, available task materials are nowadays more complex than previously described. Writing process researchers therefore need to describe and analyze these changing writing strategies (i.e., the interaction with multiple sources) in an integrated way. Because these strategies are also an important aspect of writing proficiency (Leijten & Van Waes, 2012; McCarthy, Grabill, Hart-Davidson, & McLeod, 2011; Schriver, 2012; Swarts, 2010), we focus in this section on how keystroke logging can help us to better describe and understand the organization of writing processes that involve searching, reading, and copying from multiple digital (re)sources. 4 In a recent exploratory experiment we studied the writing processes of participants producing professional tweets and emails in response to a closed task instruction. We briefly present this study here in order to illustrate the added value of keystroke logging data to describe the interaction with multiple sources. More specifically, we focus on two approaches to visualize the influence of digital sources on the organization of the writing process, first by means of time-based process graphs and then by means of network graphs and data mining.
Writing Task
The participants had to inform colleagues about a communication conference. The experiment consisted of two small-scale studies. In the first study the participants wrote a tweet to inform their followers about a conference (Flemish Scientific Economic Conference—VWEC). In the second another group of participants also first communicated by Twitter to a large audience (in this case about a conference in the field of communication, Corner-Stone) and then wrote an email about the same conference, directed to colleagues only (internal communication). Participants were able to access information about the conferences on the web (Table 5).
Design of the Multiple Source Study (composition of tweet and email).
Since the second task (email)—which only some of the participants had to perform—was based on the same website as the first task (Twitter), the participants already had some prior knowledge about the topic. We expected them to search for additional information, but in a different way because the task had a different focus.
Participants
In the first part of the study 47 participants wrote a tweet about a communication conference. They were all master’s students in professional communication (aged between 22 and 25, 6 males and 41 females). In the second part, 10 other participants, 5 professionals (mean age 35.2; 4 males and 1 female) and 5 master’s students in communication sciences (mean age 20.2; 2 males and 3 females), performed two writing tasks (tweet and email).
Procedure
For the first part of the study the 47 participants were invited in two groups to a computer classroom lab. They received an oral instruction of the writing task, together with a written description of the tweet task. The URL of the conference was explicitly mentioned. The computers were all connected to the Internet, and the participants were instructed to access whatever content they needed to complete the task. There was a time limit of 10 minutes.
The participants in the second part were invited individually. They received the same tweet instruction, but in this case the task was followed by an email task in which they had to address their direct colleagues and convince them to take part in the conference.
Data Collection and Equipment
Eyetracking data were collected with Tobii (T60) 5 to investigate what the writer is reading/searching in the nonactive writing periods. We used Inputlog to observe their writing and surfing activities because the program registers and analyses these sourcing practices, called “focus events.” Each focus event represents a switch from one Windows application to another: It therefore represents a change in the writer’s focus of attention from text production to reading a source. Inputlog thus not only logs the text development in the current MS Word document but also registers all windows accessed during the writing process (e.g., other Word documents, web page URLs, graphical applications). So, if a writer Googles information when writing a report, Inputlog identifies the web browser used (e.g., Google Chrome), the active URL (e.g., www.google.com), the page title, the keywords used to activate the search operation, and the resulting URL of the web page accessed subsequently (together with an epoch timestamp in milliseconds). Researchers can thus track the writer’s search and consulting behavior. The source analysis output provides basic information about the use of sources, for example, time in source, keystrokes in source. Furthermore, it shows the interaction between sources and their (time-based) impact via an interaction matrix. A comma-separated file is generated, creating a basis for the graphic output (see the Network Analysis and Data Mining section below).
Results
In what follows, we present two data extracts on a case basis.
Time Graph
Figure 3 plots the time-based progression of a student’s writing process (x-axis) against the number of characters produced (y-axis).
Temporal representation of the writing process combined with the use of sources.
The upper black line shows the number of characters produced in the main document during the recorded session (either typed or copied characters). The gray line displays the actual number of characters in the final document at every moment in the process. A difference between both lines means a text deletion (e.g., after about 2:20 minutes the lines do not coincide anymore, which means that a deletion took place). At each deletion the distance between the black and gray line becomes larger. The dotted gray line shows the cursor position. The cursor could be at the end of the text (text production and cursor position line overlap). A lower position indicates changes in the text produced so far, while in the instances where the solid gray and the dotted gray lines meet, new text is produced at the end of the document. Finally, at the bottom of the figure we represent the interaction between the main document (1: tweet) and the sources (2: other resources). If we combine these data with the eyetracking data, we have more information about what the writer is reading/searching during the nonactive periods (see Figure 4).
Eyetracking data of reading/searching activities in other sources.
The top of Figure 4 shows the main page of a conference website. The main page includes three types of structure elements (top banner, left and right trees), a visual in the middle of the page that includes the conference details (what, where, when) and a body text explaining briefly the topic of the conference. In this case, in the beginning of the writing process the writer searches for information, which she mainly finds in the conference program page (accessible via the right tree). After reading information on the conference program page, she writes (translated), “To all students of Communication Science: interesting conference about internal communication.” Consequently, she interrupts her writing process three times. In the middle of her writing process she reverts to the body text of the conference’s main page to refine the wording that she has chosen so far by adding “practice” before “conference” and rewriting “internal communication” into “internal and organizational communication.” The last interruption allows the writer to search for the conference location, which she finds on the map (accessible via the right tree). The final tweet reads as follows (translated): “To all students of Communication Science: interesting practice conference on internal and organizational communication at April 17 in Bussum.” The additional information via eyetracking clearly explains the revisions made in the tweet.
Process data can of course also be used for further analyses in other programs, for example, Excel or SPSS. For instance, the doughnut graph (Figure 5) provides an overview of the use and impact of sources. It shows the two writers composing a persuasive email: on the left, the student writer who often reverts to the resources, mainly in the first part of her writing session; the right-hand picture shows an expert in the field who has a more targeted use of resources immediately resulting in new text production. This visualization also clearly shows the degree of fragmentation of the writing process and the relative weight of the various sources. In this case the professional’s writing process is characterized by less fragmentation and a more targeted use of the available (re)sources as each source consultation leads to a substantial amount of text production.
Fragmentation of email writing process by nonprofessional (left) and professional (right).
Network Analysis and Data Mining
The source analysis also opens up other possibilities for writing process research. For instance by exploring data via network analysis (for a review, see Scott, 2000) and data mining techniques—a common analysis technique in other research areas like engineering, business, or medical sciences (for more information, see, e.g., Baesens, Mues, Martens, & Vanthienen, 2009; Maimon & Rokach, 2010)—new perspectives can be explored. Dynamic network analysis is a visualization technique that focuses on the interaction between sources and shows the relative weight of each source. We illustrate this analysis by drawing on the email writing process of the student writer (Figure 6a) because of the length (almost 20 minutes) and fragmentation of the writing process. The filtered and recoded focus events provided in the general file—and facilitated in the Inputlog pre-process tab—were used as input and a source analyses matrix was imported in Pajek to generate a network graph. Pajek is a large network analysis and visualization tool for MS Windows (freely available for noncommercial use at http://pajek.imfm.si; see also NodeXL by Microsoft). Circle sizes show the percentage of time in the document and sources; the arrows provide information about the (number of) switches between sources. Figure 6b shows the main (re)sources used to write the email. While these visualizations do not represent all the interactions, they do speak to the complexity of the writing process and to the attribution of the different sources.
Network analyses of email writing process by nonprofessional writer, (a) all relations and (b) grouped.
During the whole writing process the student writer switches 68 times between the various sources. In other words, the writer interrupts her writing flow about 4 times per minute to switch from her document to other sources. However, about half of these switches are caused by transitions between the sources (taskbar, Windows explorer, etc.; see middle area of Figure 6a), and have only a technical function. If we ignore these transitional “switches”—as in Figure 6b—then the figure becomes more readable and the interaction is reduced to 35 switches between three sources. The writer spends over 20% of her time in other sources (in the Twitter task this was even 48%). External sources are part and parcel of digital writing and are an important cause for writing task fragmentation. To describe this fragmentation from multiple perspectives, Inputlog 5.2 provides automatic analyses of pause and revision bursts and combines them with the new measure of source-bursts.
Data Mining Analyses
The same network data can be used in process data mining analyses. The basic idea of process discovery is as follows: Starting from a time-based event log, a data miner automatically composes a suitable process model that describes the behavior seen in the log. The purpose of process data mining is “to identify models that correctly summarize the behavior in the event log, striking the right balance between generality (allowing enough [variance in] behavior) and specificity (not allowing too much behavior)” (Goedertier, De Weerdt, Martens, Vanthienen, & Baesens, 2011, p. 1697). This type of process discovery is an innovative means for describing the writing behavior from a complementary process perspective. The fine-grained keystroke logging data easily accommodate such pattern-based analyses, both at the micro and at the macro level. Data mining generates advanced analyses such as the identification of performance bottlenecks (micro) or the localization of paths (macro) in the process model that characterize the behavior of certain subgroups (e.g., experts vs. novices). In other words, data mining unveils implicit processes and tacit knowledge about the organization of (writing) processes, for example, the order in which sources are accessed and the revisiting behavior.
To visualize the process, so-called dependency graphs are generated. In this study we used the ProM Framework (Van Dongen, De Medeiros, Verbeek, Weijters, & van der Aalst, 2005), which consists of a large number of plug-ins for event log analysis. To illustrate this, we analyzed the 47 tweet logs, collected in the first part of the study described above. Figure 7 shows the dependency frequencies between the identified sources (squares) and reveals the results of a heuristic search for detected relations (arrows). For each interaction, the total number of cases is given. To reduce the information overload on the graph, an 80% dependency threshold was used to leave out less frequent interactions. The process graph in Figure 7 is complementary to the network graph in Figure 6 because it emphasizes the process flow from “start” to “end.” Already at a first glance we can identify the composing process that characterizes the tweet design. Having accessed the Twitter page, the writers Google the conference name, “VWEC 2012.” The highlighted arrows indicate an intensive interaction between the tweet and the conference website using other sources while composing the message.
Heuristic visualization of tweet work flow based on process data mining (generated by Prom).
This example is a very limited illustration of process mining. However, the case shows that automated construction of structured process models opens a very promising and innovative perspective for writing process research. Other research domains—like business process management or health studies—already have a long tradition in using data mining, and we can certainly build on this tradition.
Research Perspectives and Further Developments
In this article we have shown some recent developments in keystroke logging to describe and analyze writing processes. Keystroke logging can be used in different settings, either as a stand-alone application or in combination with other observation methods and tools (e.g., eyetracking or think aloud, either simultaneously or retrospectively). Combining research methods provides a more solid basis to align the fine-grained measures derived from keystroke logging with the underlying cognitive processes. Moreover, it allows researchers to address more complex and interdisciplinary research questions. Techniques like merging data files and automated pause and revision analysis reduce manual coding considerably. Research topics that were previously described in pilot studies or case studies can now be replicated on a larger scale to enhance generalization. Of course, data collection and analysis is still a labor-intensive activity and requires careful and cautious interpretation of the output by the researcher.
In this article we also illustrated new graphic and statistical data analysis techniques, mainly adapted from network analysis and data mining. These techniques open new perspectives to explore the complexity of writing process organization in the digital age, mainly focusing on the interaction with multiple (re)sources. The study on tweet and email writing illustrated that drawing on techniques and insights in related research domains offers a valid basis to better understand these complex processes that are an inherent—but currently underexposed—part of digital writing processes.
Moreover, in line with Hayes’s (2012b) keynote at the SIG Writing Conference, these visual representations also clearly illustrate the need to elaborate on the role of searching and consulting digital (re)sources during writing in process models. The interaction with multiple sources—intentionally and unintentionally—has become an inherent part of most writing processes more than ever. Because it requires new writing strategies to deal with the complexity of available digital sources, we think that it is also important to focus on this aspect in writing pedagogy. The visualizations provided by keystroke logging programs (e.g., progress graph and network analyses) might function as an interesting starting point to discuss the interaction with sources in educational settings. They can be helpful for students to better understand their own writing process or their peers’ (peer feedback).
To draw this article to a close, we reflect on some further developments, in particular (a) process data aggregation from the letter level to the word level and (b) the standardization of the XML structure for keystroke log files.
Word-Level Aggregation of Logged Data
We are currently exploring the possibilities of letter to word level aggregation by merging logging data with existing lexica and Natural Language Processing (NLP) tools for Dutch and English (Leijten, Macken, Hoste, Van Horenbeeck, & Van Waes, 2012). It is our intention to create a valuable basis for more linguistically oriented writing process research, establishing a new paradigm in writing process research. By enriching the temporal data with linguistic information on the word and sentence levels, we would like to stimulate interdisciplinary research, especially in domains like literacy, discourse, and media studies. We argue that the combination of linguistic and process analyses allows us to address new and innovative research questions that could not be addressed before because process data can be analyzed on a higher, more complex level. For instance, it will become possible to address questions like these: Are pauses longer before verbs than before nouns? Do professional writers substitute high-frequency words more often by low-frequency words than novice writers? To what extent do P-bursts overlap with part-of-speech boundaries?
To realize this objective, we aggregated the logged process data from the letter level (keystroke) to the word level by merging them with existing lexica and using NLP tools. At this moment we have already succeeded in enriching the Inputlog process data with the following linguistic information: part-of-speech tags, lemmas, chunks, syllable boundaries, and word frequencies.
The main challenge in this new approach is that by definition writing process data do not always represent clean and grammatical text. Therefore, a parser has been developed that extracts four types of data from the S-notation output (see the Description of Inputlog section) generated by the Inputlog revision analysis: normal text production, inserts, word-level revisions, and deleted fragments (Leijten et al., 2012; Van Horenbeeck, Pauwaert, Van Waes, & Leijten, 2012). For the shallow linguistic analysis, we currently use the LT3 tools suite. 6
We believe that this development will further stimulate inter- and cross-disciplinary writing process research, bringing also process and product methods in writing research closer together.
Standardization
At present, each logging program uses its own formats and data analysis modes. To facilitate the exchange of data and expertise between research groups, it is important to standardize the data structure of the output files and agree on a common XML format. 7 In the context of the European COST Framework “Learning to Write Effectively” (Action IS0703), work is in progress to create a “generic structure for logging human computer interaction related to writing” (Van Waes, Leijten, Van Horenbeeck, & Pauwaert, 2012). This proposal is now being further discussed among the main developers of keystroke logging programs used in writing research. A standardized generic XML structure for logging writing data should (a) simplify the interchangeability of research data between the different programs, (b) enable the description of process data in a uniform and unambiguous way, and (c) establish keystroke logging as a mainstream writing research method.
We see standardization to be of use for connecting keystroke logging and related data, and we would also like to accomplish this in the terminology/definitions of variables. In this article we have presented variables like P-burst and R-burst that seems well established in the writing research literature. However, are we sure that basic underlying concepts of “a pause” or “a revision” are analyzed in the same way in the various keystroke logging programs? For example, how should pauses be defined for successive mouse movements (or scrolls)? Should a between-word pause be defined as a combination of the pause after the preceding word and before the word (so surrounding the space)?
To facilitate the exchange of this kind of expertise, a knowledge center for writing process research called WritingPro was launched (see www.writingpro.eu). The main aim of the WritingPro website is to bring together researchers involved in writing research to share, highlight, and further develop their expertise. We are convinced that at this stage in writing process research it is very important to exchange various types of knowledge and expertise on keystroke logging tools and other related techniques of data analysis. Within this context, WritingPro may well be a suitable platform to continue the dialogue launched by this special issue about writing research methods by providing a forum for sharing new analyzing techniques and visualizations.
Footnotes
Acknowledgements
We would like to thank Eric Van Horenbeeck, Tom Pauwaert, Robbe Block, Joris Roovers, Joeri Rammelaere, Alexander Prinsier, Wesley Cabus, Ahmed Essahli, Wim Claesssens, Mathia Van de Poel and Bart Van de Velde for their excellent work in programming Inputlog (BOF 2005-2011). We would also like to thank Nico Verlinden and the KdG-Polytechnic Antwerp for providing internships and a collective project on Inputlog (PWO 2005-2008). We thank Stijn van Even, Guido Gallopyn, Hans Geuns, and Neil Grant of Nuance (previously Scansoft) for all their efforts in making the logging facility at Dragon Naturally Speaking available to us. We acknowledge Kerstin Severinsson Eklundh, Asa Wengelin, Victoria Johansson, Roger Johansson, and Eva Lindgren for the many fruitful discussions on the development of logging techniques and methods. David Martens and Jochen De Weerdt (Katholieke Universiteit Leuven) introduced us into data mining analysis. Nikki Van De Keere and Rutger Brood (students in the master’s in multilingual professional communication program) partly gathered and prepared the analyses of the data for the experiments about digital professional communication. We also thank Professor Haas and the anonymous reviewers for their critical but inspiring comments on a previous version of this article. Finally, we thank Tom Van Hout and Geert Jacobs for proofreading this article. We take responsibility for all remaining errors. The latest version of Inputlog (v 5.2) can be downloaded from 
(free software for researchers). A detailed manual of the program is also available.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Mariëlle Leijten has received a grant for postdoctoral researchers of the Flanders Research Foundation (FWO) to conduct this research project. Because of the FWO funding and additional funding by the University of Antwerp, the research group Writing and Digital Media was able to invest more extensively in the technical development of Inputlog (2011-2013).
Notes
Author Biographies
.
).