Team Innovation Processes

Direct observations

The primary source of data for this study was direct observations of project team meetings by the first author. These included both prescheduled and spontaneous team meetings, playtest sessions, and client calls. While he had some interactions with individual members in these teams, these interactions (aside from informant interviews discussed later) were casual and not directly related to the work of the teams that were being observed. Hence his role as researcher in this setting was one of an observer-as-participant (Adler & Adler, 1994; Gold, 1957) where he had an overt presence in the research setting and interacted with team members but was only passively involved in their work under observation. The site was visited on a weekly basis and extensive field notes were taken based on his observations. Field notes included details that were not only relevant to the research questions but also included details that enhanced our understanding of the situational context (Pratt & Kim, 2011). For example, notes included maps of the meetings to record where people stood or sat; information about who spoke to whom; observations of the general mood; which team members were influential; and team member attitudes toward events, the client, and the project in general. Each visit to the site lasted from 60 min to 4 hr. In total, 25 observations at Alpha and 17 observations at Beta were recorded.

Archival documentation of production processes

Another source of data were project documents of production processes. These included (a) project schedules that contained information about deadlines and milestones, (b) design documents that contained information about output specifications, (c) production documents that tracked the status of game assets, (d) email communication among team members, and (e) task tracking and planning documents called scrumsheets.

Scrumsheets were documents that were updated daily for planning and coordinating team action. These were spreadsheets that contained detailed information (presented for a 3-week period, as described in more detail below) of the task that each individual was responsible for, the status of the task (i.e., whether it was in progress, blocked or completed, the projected total number of hours required to complete the entire task, and the projected number of hours remaining to complete the task), and the capacity of each team member (i.e., number of hours available). These documents were downloaded daily to provide up-to-date snapshots of group activity.

The work done by these teams was highly interdependent in nature, but they used a distributed approach to complete tasks, in which project members often worked on their own for some time on their assigned pieces of the job. Because of this structure, the scrumsheets served as an appropriate and efficient means of capturing the daily activities at the team level. In fact, scrumsheets were used by producers to monitor team progress and inform decisions on staffing allocations and budgets. Furthermore, these documents were shared with clients as a form of accountability, so there was a strong incentive to accurately represent team activities and progress in these documents. The scrumsheets retrieved for this research captured a total of 1,058 unique tasks for Alpha over 76 days and 527 unique tasks for Beta over 37 days.

Interviews

Over the duration of these projects, informal interviews were conducted with team members. These interviews were spontaneous encounters with project team members to obtain updates on project progress and to clarify events that had been observed. Each interview lasted approximately 5 to 10 min and was conducted in the open. Handwritten notes were taken during the interviews, which were summarized and included in the field notes immediately after each interview took place.

One-on-one semistructured interviews with two members from each team were conducted separately after the teams’ projects were completed. The duration of these interviews ranged from 60 to 90 min. The informants for these interviews were core members of each team who were involved with the project from inception till end and who were involved in key decisions, such as those relating to design specifications. Informants were asked about project background, events preceding and subsequent to milestones and critical incidents, obstacles faced, their relationships with their clients, and their overall assessment of the teams’ performance. These interviews were audio-recorded and transcribed verbatim. Although these interviews involved informants’ retrospective account of events, the reliability of their recall was enhanced by allowing them to reference team documents and communications through a computer terminal in the interview room. The semistructured interviews provided an additional source for triangulating on how events unfolded in these teams.

Team artifacts

Team artifacts are outputs produced by the team and include prototypes of the game under development, artwork, video captures of gameplay, and video recordings of playtest sessions. These artifacts were retrieved either from team members or downloaded from a computer folder that was shared among team members.

Planning phase in experimentation cycles

The objective of the planning phase is to establish the tasks required to accomplish goals for the next cycle, the people responsible for performing these tasks, the duration of these activities, and the deadline for completion. At GameCo, it was found that there was a low emphasis on collective planning efforts in the experimentation cycles of both teams because these tasks were likely to be performed independently, albeit at the cost of lower-quality output.

This emphasis on independent work was evident from the Alpha team’s assessment of the workflow that would have been required to allow players to customize the height of their characters in the game. At issue was whether the team could implement this feature by stretching the animations within the game. To explore the feasibility of this option, one of the software engineers wrote code to stretch some preliminary animations in the game that had already been created. This was a one-step process involving only one member on the team. In contrast, the regular procedure for integrating a piece of animation into the game is a three-step process involving the animator, the technical artist, and the software engineer.

The lower interdependence between team members in this phase is also evident from Table 3, which shows that only 6.6% and 5.0% of experimentation-related tasks in teams Alpha and Beta respectively were blocked—that is, obstructed from progressing—by the activities of other team members, compared to validation-related tasks, which formed 14.9% and 35.0% of total blocked tasks in Alpha and Beta, respectively.

OPEN IN VIEWER

Table 3.

Number of Blocked Tasks by Task Type.

	Team Alpha		Team Beta
Project phase	Number of tasks	Percentage of total tasks	Number of tasks	Percentage of total tasks
Administration	10	8.3%	3	5.0%
Experimentation	8	6.6%	3	5.0%
Validation	18	14.9%	21	35.0%
Production	85	70.2%	33	55.0%

When task interdependencies in experimentation cycles are low, as in the GameCo teams, coordination between team members is correspondingly simpler. Likewise, there will be a lower need for frequent, multidirectional communication among team members to coordinate effectively. We observed that communications between group members in the planning phase could be characterized into two forms, depending on whether the communication occurred within or between functional areas. Within functional areas, communications tended to involve collaborative problem-solving discussions to determine how ideas could be quickly implemented. Across functional areas, we observed more directive communication consisting of instructions from programmers to others on the team.

Enacting phase in experimentation cycles

Activities in the enacting phase of experimentation cycles directly relate to ideas decided on in the planning phase. These activities are typically experiments that provide the team with insights into project requirements and technical constraints. These experiments manifest in the form of prototypes that emphasize a particular feature to be tested. In Alpha, for example, YI describes how the team explored the technical capabilities and limitations of the mobile devices on which the game would run:

We just made a quick-and-dirty build on the phone [where we] tried to port it and made sure that we could play an animation of the character and have it all run. So . . . it was just a prototype. Does it run? Just take whatever it is and push it on the phone. And, if it completely failed, we would’ve been in a bad place and if it ran, but it ran slow, we could probably make some adjustments to it.

Outputs of experimentation cycles are quick-and-dirty because these prototypes are built to represent a particular feature of the final product and are rarely intended to be a part of the final deliverable. Since the emphasis during enactment activities is on the speed of completion, rather than on technical quality, enacted outputs in experimentation cycles are usually improvised and less elegant or polished than the final deliverable. In the previous example given for project Alpha, in which the customized height question was explored, the software engineer’s use of animations that had already been created is an example of improvising a quick-and-dirty test to explore the feasibility of implementing this possible customization feature. It was also common for GameCo teams to use basic shapes and crude renderings of final art assets as placeholders in the virtual environment. For example, a prototype from Project Beta that was used to explore the feasibility of replicating the turtle’s swimming motion contained simple two-dimensional renderings of the ocean environment and also lacked the rich texture and detail that would have made the environment look more realistic.

Reviewing phase in experimentation cycles

Activities in the reviewing phase of experimentation cycles involve evaluations of the group’s output and the processes undertaken to produce this output in the enactment phase, followed by forming plausible interpretations of these evaluations.

Performance evaluations of the outputs produced by the team in experimentation cycles are usually exploratory, similar to proofs of concepts. For example, in project Alpha, the evaluation of the stretched animation was to determine whether the end result would satisfy quality requirements. Similarly, in project Beta, initial tests were conducted to verify that the movement of turtles in the ocean could be accurately represented in the game.

The workflow required to implement these ideas is also evaluated in the reviewing phase of experimentation cycles. These evaluations allow team members to better understand project constraints, which facilitates decision making related to design specifications, and to more accurately anticipate resource needs such as staffing and process improvement tools. For example, once the number and type of ocean scenes that the turtles would swim through were confirmed, artists in the project Beta team could then begin producing concept art of various graphics for the game. Concept art served two primary purposes: First, it allowed artists to verify their ideas with team leaders and the client. Second, it gave artists a better understanding of the resource requirements. This process was described by UE, the lead artist for project Beta, as such:

[What] we were doing with the concept art was visually saying “is this what you mean?” . . . That gives a clear picture . . . of how many assets we are really going to need for an area. [The lead designer] can say this area is a kelp forest, but until we really look at a lot of reference, and draw up from that reference what we would want it to look like, it’s not really clear how [many assets are] required for that.

Evaluations of group output and processes in experimentation cycles therefore inform decisions about project scope and specifications—specifically, which features to retain, abandon, or modify as well as the tools and resources that are likely to be required at different points in the project’s lifecycle.

A feature of feedback from experimentation cycles is that it is ambiguous and does not always provide definitive answers to future actions. Feedback may only indicate that changes need to be made, or that an idea is good enough, but it offers limited insights into the correct response, or whether a course of action is the most optimal (Van De Ven & Polley, 1992). Consequently, a future course of action remains equivocal. In project Alpha, for instance, although evaluations of the stretched animations verified that this option would not satisfy quality requirements, the appropriate course of action—whether the team should explore other solutions to implement this feature, or to ask for an increase in the budget and development timeline, or to abandon this feature completely—was still unclear. This decision was made by developing a plausible story of what reality might be (Weick, Sutcliffe, & Obstfeld, 2005) and deciding on a course of action consistent with this interpretation of reality.

The ambiguity of feedback is compounded by evaluations of group output in experimentation cycles that are usually derived from a small number of tests using prototypes that are quick and dirty representations of specific features of the end product. These tests are usually conducted on a small scale because of the unfinished, provisional nature of the output. For instance, playtest sessions at GameCo were usually conducted with no more than 10 participants. Consequently, the generalizability and validity of these tests are potentially limited (March, Sproull, & Tamuz, 1991). The extent of this limitation, however, is not apparent and may only be discovered much later. For example in project Beta, although the team was able to verify that the movement of turtles in the ocean could be accurately replicated in the game environment being tested, it was unclear that the same level of performance could not be maintained in a more graphics-intensive game environment. In fact, later in the project, serious performance issues were discovered and the team had to reduce the quality of graphic art assets to reduce the amount of computational processing resources required for the game to run.

In light of these equivocal options, the team’s interpretation of feedback about their output from experimentation cycles and decisions about future actions also depend on prior information and predictions about the future. Activities in the review phase are thus similar to a sensemaking process involving Weick et al.’s (2005, p. 415) “continued redrafting of the story as it emerges.”

Difference between validation and experimentation cycles

A key distinction between validation cycles and experimentation cycles pertains to the team’s output. In validation cycles, team outputs are components of the team’s final deliverable—not just quick-and-dirty tests as they are in the experimentation cycle. Consequently, the primary emphasis in the enactment phase in validation cycles is the technical quality of the team’s output. Validation cycles can be considered to be correction oriented, while experimentation cycles are discovery oriented. Essentially, team actions in validation cycles involve modifying the value of known parameters, adding new features, or removing existing ones rather than building components from scratch.

The second difference between the cycles is the sense of permanence that encompasses validation cycles in contrast to the playful, provisional nature of experimentation cycles. If experimentation cycles are analogous to dance rehearsals where the choreography has not been established, the analogous comparison for validation cycles will be to rehearsals in a symphonic orchestra with sheet music in hand. In this case, the parts to be performed by each musician are already defined and codified. The goal of rehearsals is to adjust specific parts to achieve a well-blended sound at a rhythm, volume, and style that is consistent with the conductor’s interpretation of the piece. In the subsequent sections, the planning, enacting, and reviewing activities in validation cycles shall be elaborated on by contrasting these activities with those in experimentation cycles.

Planning phase in validation cycles

A feature of the planning phases of validation cycles is the emphasis on coordination, which stems from the fact that team output in validation cycles are components of final deliverables rather than provisional prototypes. Consequently, team members need to actively coordinate their understandings of output specifications and the timing of completion during this phase to minimize errors and delays. The emphasis on coordination during the planning phase of validation cycles was evident in the daily scrums of both Alpha and Beta, where it was common for team members to openly ask what their next tasks were to be after completing a task. Although each member had a list of tasks to accomplish within a 3-week sprint cycle, the priorities of these tasks often shifted during the course of the sprint and tasks were often added to or removed from this list. These priorities were often not determined by a single person when the question was asked but had to be discussed later in short, spontaneous meetings (or huddles) involving the team leads and the respective team member. In these huddles, members would discuss team priorities, the process and resources required to accomplish these priorities, and the team’s capacity to accomplish these priorities. These discussions then yielded a clearer answer about how team members would prioritize their tasks.

The greater emphasis on coordination in validation cycles also necessitated a greater requirement for communication between team members. In team Beta, for example, one of the ways that the team attempted to improve the performance of their game was by reducing the number and size of “collision bubbles” around objects. A collision bubble is a space that two objects in the virtual game environment cannot pass through at the same time, or else they would overlap (or collide) into one another. Reducing the number and size of collision bubbles improves game performance by reducing computational processing. However, this has to be balanced with maintaining the realism of the game so that objects “bounce” off one another in a realistic manner. Decisions about the location of these bubbles and how much to reduce the size of each bubble were observed to involve a detailed discussion between the lead artist (who was responsible for implementing these changes) and the technical director (who was responsible for integrating various components into the main game artifact). This example illustrates the collaborative problem-solving communication between team members across different functional areas that is necessary for effective coordination in the planning phase of a validation cycle.

Enacting phase in validation cycles

As noted above, the main difference between validation and experimentation cycles is related to the teams’ output. That the team had shifted from experimentation to validation, essentially fine-tuning toward the finished product, was evident from the teams’ emphasis on fixing bugs in the days leading up to playtest sessions that involved members of the public and before a milestone deliverable to the client. The emphasis on the technical quality of outputs was also evident in their avoidance of shortcuts that would compromise the quality of their output in spite of time constraints. Instead of relying on short-term fixes like they might have in the experimentation cycle, teams would commit resources to ensuring more robust and elegant solutions that would enable them to accomplish tasks faster without compromising on quality. For instance, YI described a pop-up messaging system in the Alpha team’s product as being “ugly and just in there,” with limited functionality since “it only worked in a battle” when this feature was first built for their game. After learning that the pop-up system would be used in other parts of the game, however, the programmers developed a more robust system that “with like three lines of code, [a software engineer] can add a new pop-up anywhere they want.”

Another strategy to maintain the quality of their output in spite of time constraints was to reduce project scope. In project team Beta, UE described a meeting among the project leads where they “sat down and said we’re not going to get all this done. We need to cut a couple of segments out.” The decision about what aspects of the project to scale back on was based on a determination of “the story and the concepts we knew we needed,” “[art assets] we could reuse,” and whether that feature “was going to take a lot of scope to figure out.”

When more extensive adjustments to the teams’ outputs were needed during the validation cycle, the resulting changes were unlikely to involve modifications to the underlying architecture of the products but were usually extensions of existing features to improve on them overall. In addition to the pop-up tutorial mentioned earlier in the Alpha team’s basketball game, another example from this project is the implementation of leaderboards showing players with the highest point totals, following a suggestion made by the team’s client midway through the project. The team was able to quickly implement the software code to extract scoring information that was already being collected and stored in the database and to present it in a separate part of the game. The addition of this feature thus did not involve deep-level architectural changes to the relationship between components, but rather was a standalone feature that leveraged some existing components within the game’s code.

Reviewing phase in validation cycles

Similar to activities in the reviewing phase of experimentation cycles, activities in the reviewing phase of validation cycles involve evaluations of team output. However, the emphasis in validation cycles is to verify that output performs to specifications, rather than to develop plausible interpretations of project goals from the feedback received during this phase. Because of the emphasis on verification, feedback content is therefore focused more on output quality and less on production processes.

Another feature of reviewing phases in validation cycles is that feedback is less ambiguous, especially in comparison to feedback during experimentation cycles. The greater clarity in feedback can be attributed to the fact that discrepancies between the quality of the team’s output and the more concretely understood project requirements are more easily identified. Moreover, teams are likely to already be familiar with the actions required to address these discrepancies because these actions involve modifying the properties of components that have already been built, rather than building components from scratch.

An illustration of activities in the review phase of validation cycles are the weekly “run-throughs” in team Beta, where a few members of the team would play the game while the rest of the team observed and made notes of the parts of the game that could be improved on. For example, software engineers might look out for art assets, such as rock, hooks, or other ocean creatures, that the turtles would pass through instead of collide with; artists might look out for artwork that needed to be retextured to improve the realism of the ocean environment in the game; and game designers might look out for parts of the game that were too easy or too difficult for players.

In light of this less ambiguous feedback and a greater familiarity with the actions needed to address issues, the communication requirements between team members were also lower in the review phase of validation cycles than the review phase of experimentation cycles. In project team Beta for instance, UE describes how he would review one of his artist’s work at the end of the work day by “[stopping] by and [seeing] how things had gone for the day.” During these informal review sessions, UE reported that his feedback to this artist was usually along the lines of, “you didn’t take this far enough yet” rather than, “that’s not working at all.”

Due to the lower communication requirements in this phase, information may also be transmitted among team members through media that are less rich (Daft & Lengel, 1986). In project team Alpha, for example, it was observed that performance discrepancies in the team’s output, such as technical bugs, could be listed by any team member in a shared database to which all team members had access.