Hidden Surface Problems: On the Digital Image as Material Object

Perspective as a Cultural Technique

The challenge in advancing a theory of computer graphics lies in its hyper-visibility as one of the most emblematic new media technologies of the past 20 years. Computer graphics – particularly its use in virtual reality simulations – was perhaps the most visible manifestation of the futuristic promise of so-called new media technology, garnering widespread attention in both the academy and popular press throughout the 1990s (Pierson, 2002). Yet in the two decades since this rise to visibility, the technology has become altogether diffuse within visual culture, incorporated into the broader machine of visual and material production. Whether captured with a digital camera, constructed and rendered using 3D interactive software, or simply displayed on the pixilated grid of a computer screen, almost all contemporary images are materially structured by the logic of computation. Yet despite this ubiquity there have been few serious treatments of computer graphics beyond their most visible manifestations in popular film and videogames. Perhaps most glaring of all, the history of computer graphics remains largely unwritten. ⁵

Existing scholarship on computer graphics deals largely with its adoption by existing visual media forms and, as such, computer graphics is often framed as the logical progression of photography and film in that it adopts and transforms those formal elements that structure these earlier media. Indeed it is precisely this ability to simulate the formal qualities of other visual media that makes computer graphics so self-effacing and so difficult to describe in any material sense that does not rely upon methods developed for the interpretation of images. As such, even the most useful treatment of computer graphics often relies on visual tropes such as perspective to structure its analysis, a structure that obfuscates much of the material workings of computer generated images. In his seminal work The Language of New Media (2001), Lev Manovich offers a deeply material, ‘bottom up’ approach to the principles of new media technology, with a particular emphasis on the digital image. For Manovich, the image of new media is part of a long and ongoing transformation of vision through technology, from photography and film to radar and virtual reality. ⁶ Manovich’s work in both The Language of New Media and in essays published throughout the 1990s treats computer graphics as a deeply material form, but deals largely with its application in film and other visual media in an effort to stitch together new media with a wide range of historical practices (Manovich, 1993). Likewise Anne Friedberg takes up the digital image in her ambitious work The Virtual Window (2009), exploring the window as a broad cultural technique for mediating vision, centered principally on the development and transformation of the perspectival paradigm from the Renaissance to the modern graphical user interface. While both texts offer distinct and productive accounts of the image writ large, with particular care given to the digital image, their insistence on a narrative of inheritance obscures a detailed historical account of the digital image as computational ab initio. In reading the computer’s simulation of existing visual modes as indicative of its material function, we put the cart before the proverbial horse. That is, the simulation of perspective is better suited to a genealogy of simulation, not one of perspective.

What perspective offers is a structuring system whereby space is mapped and displayed in relation to a viewing subject. As Lacan (1981: 86) notes in ‘On the gaze as objet petit a’, ‘what is an issue in geometric perspective is simply the mapping of space, not sight.’ It simply happens that we tend to privilege sight in the way in which we map space technologically. The most significant development in this history was the introduction of Renaissance perspective and its evolution through a variety of media technologies, including film and photography. As William Ivins notes in On the Rationalization of Sight (1975), perspective has come to serve as ‘a practical means for securing a rigorous two-way, or reciprocal, metrical relationship between the shapes of objects as definitely located in space and their representations’ (p. 9). That is, perspective is one potential solution to the question of object relationality, one particular relational technique with a long cultural history that is adopted by computer graphics in the production of a culturally situated realism.

This is not to suggest that perspective is a monolithic or unified system of representation. As James Elkins (1996: 214) has suggested, there is ‘no coherent history, no connected tradition beneath the word [perspective]’. Nonetheless there is a cultural significance in its deployment across a broad range of visual media. Indeed its prevalence as a structuring logic for visual media suggests that it operates as a kind of deeply embodied cultural technique [Kulturtechnik], that is, a condition whereby ‘signs, instruments, and human practices consolidate into durable symbolic systems capable of articulating distinctions within and between cultures’ (Geoghegan, 2013: 67). It is a form that has been naturalized through its adoption in a variety of media since the Renaissance era, but whose primacy as a means of producing and reflecting the world is historically bound and exists alongside other cultural techniques. ⁷ Thus while numerous art historians have identified the cultural relativism of perspective (Panofsky, 1991), it is perhaps more useful to think through perspective as a set of culturally and historically situated practices that are maintained precisely through their adoption and transformation by emerging media technologies. In other words, it is the malleability of perspective across its multiple cultural and historical media forms that maintains it as a governing structure. As such it is of no surprise that perspective has become the operative relational mode for a great deal of computer graphic visualizations, but that adoption is by no means essential to the way in which graphics produces visualization. Indeed an investigation of the earliest use of perspective projection in a simulated image reveals very little concern for this centuries-old technique. Instead we find a new set of concerns structured not by vision but by a theory of the nature of objects, a computational ontology for which the rendered image is only one of many possible expressions.

Computing Perspective

How then does computer graphics first approach the simulation of perspective? The earliest model of three-dimensional perspective comes from the graduate work of Lawrence Roberts, whose dissertation research at MIT – titled ‘Machine perception of three-dimensional solids’ (1963) – is a seminal text in the history of the field. As William J Mitchell notes in The Reconfigured Eye (1992), Roberts developed the first version of the perspective–construction algorithm that could be executed by computer. For Mitchell this is a critical moment in the history of the algorithmic image, an event ‘as momentous, in its way, as Brunelleschi’s perspective demonstration’ (p. 118). Like Brunelleschi and Alberti some 600 years prior, Roberts’ work would seem to make possible an entirely new form of image making, one that leads directly to today’s computer generated images in film, photography, and digital games. Yet this is a misreading of Roberts’ work. It is true this is one of the first examples of computational perspective, and that the simple graphics produced by Roberts’ program bear a striking resemblance to contemporary computer generated images, but we should not assume a direct narrative of inheritance. If we set aside the rendered image and look to the structure of the program itself, we find a system for producing images that bears little resemblance to contemporary graphical modeling. What’s more, this excavation suggests Roberts’ use of perspective projection is largely incidental to the program’s objectives, creeping in under the guise of an earlier visual form.

The stated goal of Roberts’ (1963) dissertation research was to enable ‘a computer to construct and display a three-dimensional array of solid objects from a single two-dimensional photograph’ (p. 2). That is, rather than construct a virtual object, as is common in contemporary modeling software, the program was intended to digitize objects from two-dimensional photographic representations. To construct a three-dimensional model, a simple object would be photographed and processed by Roberts’ program, which would reproduce the image as a line drawing that could be read by the computer (see Figure 5). The program would then transform the drawing into a three-dimensional representation that could be manipulated interactively from any point of view using a perspective projection. It is unsurprising that the program adopted the perspective model of photography since photographs served as the primary source of visualization for the program. That said, the goal of the project was largely one of computer graphics by means of computer vision, and not the simulation of a particular visual form.

OPEN IN VIEWER

Figure 5

Deriving object edges as feature points from a photograph, MIT 1963, 165 x 73 mm (300 x 300 dpi). Pictures from ‘Machine perception of three-dimensional solids’ by permission of author Lawrence Roberts.

By introducing embodied perspective – or its mechanization through photography – Roberts’ program adopted a set of psychological assumptions about how perception functions and might be procedurally modeled. To this end he drew heavily on the work of psychologist James J Gibson, whose The Perception of the Visual World (1950) he found instrumental in formulating vision as diagrammatic and discrete. Gibson’s theory of vision was developed largely over the course of WWII through his research on airplane pilots and, along with the work of Norbert Weiner (1948), marks a shift toward theorizing vision as explicitly machinic (Patterson, 2007: 45). Contrary to earlier theories, which understood vision as the perception of ‘an object or array of objects in the air’, Gibson (1950: 6) formulates a ‘ground theory’ structured by ‘a continuous surface or an array of adjoining surfaces’. This leads him to a simplified model of perception, in which ‘the elementary impressions of a visual world are those of surface and edge’. For Gibson, vision is not derived from an embodied sense of place or location in space but from the perception of the relational boundaries of objects in a visual array, reduced to a discrete set of primitive forms. Vision in this formulation is no longer concerned with accurate mimesis, but is instead engaged in the capture and replication of an external relationality.

Roberts (1963: 13) uses Gibson’s theory of vision to rationalize a shift in computer vision away from its earlier preoccupation with alphanumeric recognition, and toward a world of objects comprised of lines, edges, and surfaces. From Gibson he derived those qualities that he believed were most important to our perception of the world, and therefore privileged them in constructing his program. These principally included object size, texture gradient, and shape perception, among others. Significantly, Gibson’s philosophy of perception argued strongly in favor of direct perception and direct realism, that is, the belief that we are capable of perceiving objects in the world directly as they are and not as representations or abstractions. Roberts’ program models itself based on these tenants, ‘seeing’ and capturing those essential, perceivable qualities of objects for digitization. When run, the program attempts to identify the edges of objects as a set of feature points. It then attempts to connect those edges and calculate the volume and shape of the object whole. Objects may then be transformed or the perspective projection shifted and manipulated, in effect rendering the two-dimensional photograph as a fully actualized three-dimensional scene.

While the solutions Roberts derived in his dissertation research were significant and pushed the field forward toward the simulation of three-dimensional objects, they were by no means definitive. His system for capturing vision to produce virtual objects bears little resemblance to contemporary 3D graphical simulation, derived as it was from the fields of pattern recognition and computer vision. While Roberts’ program was the first to introduce a theory of perspective into 3D graphics, this is arguably not the project’s most significant contribution. Indeed the introduction of perspective into computer visualization is on the whole largely unremarkable, given that the construction of Renaissance perspective itself is a procedural, mathematical, even algorithmic process, ‘a translation of psychophysiological space into mathematical space; in other words, an objectification of the subjective’ (Panofsky, 1991: 66). It is therefore unsurprising that the field of computer graphics finds a way to reproduce Renaissance perspective very early in its development, as it is the simulation of a largely mathematical technique for constructing vision.

While Roberts’ work at MIT is one of the earliest examples of three-dimensional graphics, he was by no means the only researcher invested in three-dimensional object simulation. In this early period from roughly 1960–1966, computer graphics was a broad field with widely varying approaches to the display of graphical images. As Roberts was developing his computer vision algorithm, similar work was being conducted elsewhere at corporate and academic research centers. Principal among these centers were General Electric, Boeing, and AT&T Bell Laboratories, where in 1963 Edward E Zajac produced one of the first three-dimensional computer animations using a Stromberg Carlson 4020 Microfilm Recorder and Ken Knowlton’s BEFLIX programming language for rudimentary bitmapped computer animation (Zajac, 1964). Significantly, the methods that researchers like Zajac developed for the display and manipulation of three-dimensional objects are entirely distinct from methods developed simultaneously at competing institutions. What’s more, none of these ad hoc systems share a direct lineage with contemporary 3D computer graphical methods, which are based largely on polygonal modeling systems that were not possible given the hardware restrictions of the early 1960s. Much as with early experiments in proto-cinematic visualization at the turn of the 19th century, in which multiple visual forms competed with and functioned alongside what would become the cinematic image, this period is marked by a multiplicity of solutions to the problem of how to simulate a digital object. ⁸ We might therefore think of these early visual forms as experiments that failed to achieve broad standardization or distribution. Yet these diverse forms tell us a great deal about the particular challenges and investments that emerge in the effort to simulate and represent three-dimensional objects. As Erkki Huhtamo (1997) suggests in his essay on an archaeology of the media, ‘registering false starts, seemingly ephemeral phenomena and anecdotes about media can sometimes be more revealing than tracing the fates of machines which were patented, industrially fabricated and widely distributed in the society’ (p. 223). This is particularly true in the case of computer graphics, where the images produced by these early systems bear a remarkable resemblance to modern 3D images, but are structurally unique and highly idiosyncratic. Herein lies the danger of interpreting computer graphics exclusively through rendered image output to the exclusion of the technical and material specificity of the systems that produced such images.

Thus, rather than focus on the successful introduction of perspective in Roberts’ graphical research, we might look instead to its more telling critical failures. In order to extrapolate the dimensions of an object, the program first had to calculate its volume, such that those features not visible in the source photograph might be inferred. As such, Roberts’ program restricts the environment to convex objects whose volume may be calculated in such a way. The algorithm is also limited in the kinds of objects it can display, relying on simple Platonic solids in various combinations to form complex shapes. Most telling of all is the program’s inability to scale with object number or complexity, such that the computation required by Roberts’ algorithm grows roughly as the square of the number of objects in the scene, making it an impractical system for real time interactive graphics or for complex rendered scenes. While these may seem like unrelated problems, in fact they are each exemplary of a single structural challenge. In order for a given program to render a visible object, it must account for what parts of that object should be visible to a viewer, and which should be hidden. The more complex an object, the more difficult it is to calculate object visibility. Roberts’ algorithm is one of the first to find a means of accurately rendering visible surfaces, and in many ways this is its most significant contribution. Nonetheless it would be over 15 years before researchers would settle on a solution to this particularly challenging problem.

It is these challenges that point to the interests and concerns of the field of computer graphics at this early stage. In interrogating the failures of Roberts’ program, we find that one of the primary concerns for computer scientists at this time was not how to reproduce a particular way of seeing, but in the structure of objects themselves. Roberts’ algorithm transforms the photographic image into an array of points in Cartesian space connected by lines to form surfaces that may be extrapolated into three dimensions. In delimiting the world in this way, Roberts’ algorithm makes visual objects legible to computation, a process that includes much more than the reconstruction of something analogous to Renaissance perspective. While a desire for realistic images that simulate the visual appearance of film and photography was a critical concern for researchers from the very beginning (Evans, 1966), and this so-called ‘quest for realism’ would certainly come to dominate the field of computer graphics research by the early 1980s (Blinn, 1999), this realism was predicated on the solution to a number of unique technical challenges that have no basis in earlier visual media forms. Indeed the most glaring challenge for Roberts and other early researchers was not how to render a visible image, but how to restrict that image into displaying only that which is sensible to a viewing subject, a problem that came to be known as the hidden-line or hidden-surface algorithm. Through an analysis of what is arguably the most significant challenge in early graphics research I hope to differentiate computer graphics from the earlier visual modes it simulates, and in doing so derive a theory of the computational image that does not presume a genealogy of the visible.

Making the Present Absent

Cinematic visibility is materially produced through the interaction of light with a camera’s aperture. Only those objects that are directly accessible to that aperture by means of light rays reflecting off its surface may be captured by the apparatus. In this sense, film and photography model visibility on our phenomenological perception of objects in the world, based on the science of optics and the physics of light movement and diffusion. As such, that which is turned away from the eye or the camera lens is radically inaccessible and cannot be seen. Computer graphics do not function in this way. For computer graphics, each object must be described in advance if it is to be rendered visible in a given simulation. As Friedrich Kittler (2009: 228) notes in his lectures on optical media:

… computers must calculate all optical or acoustic data on their own precisely because they are born dimensionless and thus imageless. For this reason, images on computer monitors … do not reproduce any extant things, surfaces, or spaces at all. They emerge on the surface of the monitor through the application of mathematical systems of equations.

Thus, in order to simulate our perception of objects as fixed in a perspective projection, graphics must not only calculate that which is to be seen, but also anticipate and hide that which is known but should not be seen, that which must be made hidden and invisible.

Prior to the 1970s this had not been the case, as graphical objects were produced largely as wireframe models with no surfaces, from which all edges were simultaneously visible. While such images may be suitable for certain tasks, the more complex an image becomes, the more difficult it is to identify and differentiate the object at hand. What’s more, even simple objects can create illusions when viewed from certain angles, in which a given set of lines overlap and collapse the image into an abstract form (see Figures 6 and 7). Outside of these practical concerns, there is the desire for realism. As prominent early graphics scholar Ivan Sutherland (1966: 26) notes in an essay on the critical problems facing early research:

When we look around the world we see opaque objects and we don’t see what is behind the opaque objects. It is hard to make objects displayed by a computer look similarly opaque. It is easy to make a perspective presentation of any individual point in space. It takes a few multiplications and a division or two to implement the coordinate transformation from the three-dimensional space coordinates to the two-dimensional display coordinates. By programming this transformation you can easily display transparent or ‘wire frame’ views of your object … It is much harder to decide whether a point ought to show or not. It is a major task to eliminate hidden lines from the drawing. (Sutherland, 1966: 26)

OPEN IN VIEWER

Figure 6

Computer generated image with no hidden lines removed, obscuring the object’s shape, 1970, 95 x 99 mm (300 x 300 dpi). © Gary Watkins.

OPEN IN VIEWER

Figure 7

Previous image with hidden surfaces removed and shading added using Warnock’s algorithm, 1970, 99 x 100 mm (300 x 300 dpi). © Gary Watkins.

While the elimination of hidden lines may seem trivial, in fact it was one of the most significant challenges for the field of computer graphics well into the 1970s. This is due not only to the complexity and variety of solutions to the problem, but also to the processing limitations of the computers of the period. Indeed it is conceptually easy to eliminate hidden lines by brute force – point-by-point, line-by-line – but, as with Roberts’ algorithm, this task becomes exponentially more difficult as objects increase in number and complexity. As such, the field required a scalable solution for the elimination of invisible data, a means by which it might preemptively omit the calculation of that which should not be seen.

Beginning in 1963 with Roberts’ algorithm, dozens of researchers developed independent solutions to the hidden surface problem, each with their own methods and limitations. Their solutions are so remarkably varied, in fact, that in 1974 a team of researchers headed by Ivan Sutherland produced a co-authored essay with the explicit intention of constructing a taxonomy of hidden surface algorithms in the hopes of identifying the fundamental root of the problem (Sutherland et al., 1974). Taking into account over 10 years of work on hidden surfaces, they created a schema for categorizing these algorithms as a function of the way in which they sort a given scene. Ultimately they found that the primary difference between each algorithm lay in the way it conceived of and handled the thing undergoing simulation: as image, as object, or as something in between. In categorizing the algorithms in this way, their taxonomy draws out the dual function of computer graphics as both structured object and rendered image, irreducible to either one or the other. It is here we might begin to derive a theory of the simulated image in its dual function as both structure and simulation, database and display, image and object.

Image Objects

The earliest solutions to the hidden surface problem generally fall under the schema of ‘object–space’ algorithms, in that they perform computations ‘to arbitrary precision, usually the precision available in the computer executing the algorithm. The aim of the solution is to compute “exactly” what the image should be; it will be correct even if enlarged many times’ (Sutherland et al., 1974: 19). Object–space algorithms ask whether each potentially visible item in the environment is visible, treating each object component as a potentially significant aspect of the object or environment as a whole. Algorithms that function under this schema include Roberts’ algorithm for ‘Machine perception of three-dimensional solids’ (1963), discussed in detail above. Object–space algorithms have built into them a set of material limitations, tied primarily to the processing power of the hardware at hand, but also to the potential complexity of the object to be rendered. As such they do not scale well and cannot efficiently render complex scenes.

In an effort to increase the efficiency of this computationally intensive task, later solutions to the hidden surface problem utilized ‘image–space’ algorithms, which only calculate that which is visible to the raster of a given screen. ⁹ The goal of these algorithms is to simply calculate an intensity for each of the resolvable dots or pixels on the display screen, and as such they do not scale beyond the hardware at hand. Perhaps the most significant example of an image–space algorithm is the one developed by John Warnock in 1968 for his doctoral dissertation at the University of Utah, commonly referred to as the ‘Warnock algorithm’ (Warnock, 1969). ¹⁰ Warnock’s algorithm functions by breaking a given screen into subregions and applying a standard procedure to each one. For each region the algorithm identifies all possible surfaces and attempts to determine whether they are entirely outside of, surrounded by, or intersect a given subregion. The algorithm then eliminates any surfaces that it finds to be behind another surface. The most significant contribution of Warnock’s algorithm is its response to a subsection that is too complex for it to calculate hidden surfaces. In such a case it simply subdivides the region into smaller regions and begins again with the new divisions, working its way back out to the larger section. If at its smallest division a section is found to be too complex, the algorithm simply picks a value from an adjacent subregion and moves on. The result of the Warnock algorithm is a fractal effect that scales with object complexity, limited of course by the resolution of a given screen (see Figure 8). It is representative of the ‘image–space’ sub-class of hidden surface algorithms in that it only treats an object and its complexity given the limitations of a particular viewing position and the screen technology used in its display.

OPEN IN VIEWER

Figure 8

A visualization of Warnock’s solution to the hidden surface problem, 1970, 86 x 86 mm (300 x 300 dpi), Scientific American (1970).

OPEN IN VIEWER

Figure 9

An early hidden surface test and the earliest color image produced at the University of Utah, 1968, 160 x 154 mm (300 x 300 dpi). Image courtesy of John E Warnock.

The most contemporary solutions to the hidden surface problem fall under a third category of algorithm, and function somewhere between the object– and image–space. These ‘list-priority’ algorithms were designed for high quality interactive simulation, an application that required real-time speed and visual realism. As such they split the task of determining visibility based on a process of object categorization. In a given interactive scene, some aspects of the world – such as the sky or a distant vista – do not change very much, and as such can be calculated using static image–space algorithms. Other objects in the scene may move and change, and therefore it is to the advantage of the algorithm to calculate them with greater precision and to be aware of any changes in position that might obscure an object from view – concerns which require an object–space calculation. Thus, in this last instance, the question of hidden surface calculation becomes a matter of the ordering of algorithms and the proper categorization of a given scene.

While Sutherland’s taxonomy is concerned primarily with uncovering the technical challenge at the heart of the hidden surface problem, it is perhaps more useful as an unintentional reflection on the first 10 years of research into three-dimensional computer graphics. In the paper’s description of the ways in which these algorithms transform and come to replace one another over time we can trace a broader transformation in the use and application of three-dimensional computer graphics in these early years of experimentation. Far from a clear genealogy, whereby Roberts’ development of perspective projection ushers in an era of realistic simulated images, what we find is a liminal period in which researchers struggle to grasp at the challenges of a new kind of technical image. By examining their solutions, we find a set of concerns that reflect the disciplinary biases of the field: object–space algorithms function best when applied to simple objects and line drawings with a high degree of accuracy and scalability, well suited to the engineering and technical diagrams of early research institutions like GE or IBM; image–space algorithms were designed to produce visually stunning images with less regard for their technical accuracy, and as such were often used to produce pioneering films and rendered images at sites like AT&T Bell Laboratories; list-priority algorithms were best suited for interactive environments that required degrees of both utility and mimesis, and were therefore developed for research into interactive flight simulation at institutes such as the University of Utah.

Computer graphics is each of these things. Not only do the different uses for graphics help structure the types of images they produce, they also structure the means by which they produce visual absence through hidden surface removal. While the final image produced by each of these algorithms may appear to fit into the broad category of computer generated images, in fact each is materially distinct from the other, the product of a unique set of interests and concerns. In 1978, some four years after Sutherland’s taxonomy, a Utah graduate named Edwin Catmull developed a technique for hidden surface removal known as Z-Buffering, which utilizes a custom physical memory storage – known as a ‘buffer’ – to store depth information for the purpose of hidden surface removal (Catmull, 1978). Catmull’s technique remains the current standard solution to the visibility problem, though there are many others, and marks a shift in the materiality of computer graphics toward hardware solutions to particularly challenging technical problems. ¹¹ This trend is made manifest in contemporary systems through the use of graphics cards that accelerate, decode, connect, and transform graphical data in a given system. ¹²

Interfacing Vision

This focus on the hidden surface problem may seem counterintuitive. It is, after all, that which is meant to go unnoticed, those parts of a world that should remain unseen. And yet, I would argue, it is in the solution to this problem that computer graphics reveals the specificity of its construction: it produces vision by constructing absence. Rather than capturing the world through the indexical trace of light on a surface, computer graphics simulates our knowledge of the world by constructing objects for visual interaction. It is a simplification that necessitates the removal of that which is irrelevant or unknown, a making absent. Much as with Gibson’s theory of vision, the world is reduced to a set of legible primitives – point, line, vertex, surface – that can be made meaningful in the last instance as a rendered image. In this sense, the hidden surface problem functions as an analogy for computational materiality itself, or more accurately the computer’s disavowal of its own materiality through the black boxing effect of the interface. Here the rendered image serves effectively as an interface for vision, configured to conform to the limitations of that vision, though not reducible to it, structured as it is by an excess of data that must be removed and restricted in order to be rendered legible. This world of graphical objects thus exists prior to the rendered output of the screen – as patch definitions, object databases, and graphical algorithms – and the image is only one of many meaningful forms this data might take. By focusing exclusively on the visual in our analysis of computer graphics, we limit ourselves to this restricted image, this black-boxed object. Through an examination of the material history of the digital image – particularly one that refuses a simple genealogy of inheritance from earlier visual forms – we can begin to identify those broad technical structures that shape, limit, and transform our contemporary digital visual culture.