What is Interactivity?

Interactivity is a relatively recent term that remains obscured by common usage. To put this into perspective, a quick search of interactive in the Google Books Ngram Viewer yields the following graph:

This graph is interesting because it parallels the increased alliance between art and technology from the 1950s onward. This alliance ushered in a new age of interactivity—one that becomes apparent when considering the emergence of television and video art in the 60s, for instance. Closed circuits made spectators integral to works of art in arguably unforeseen ways, and in doing so they required new descriptive words such as interactivity.

I believe we are in need of somethingmore than input in order to qualify what happens in an encounter with art as interactivity.

Given that my background in art history consists of knowledge of art post-World War II, this graph—while both interesting and revealing—is for the most part predictable. I say this because analog computing began to rise to prominence and be adopted by artists in the 50s, overwhelming support was given to the technological arts by backers such as Bell Labs in the 60s, and then—jumping ahead—interest in virtual reality experienced a rush in the 90s. Yet, for all its predictability and for all my familiarity with practices that have been loosely called interactive, I cannot give you a straightforward definition for the word. My hunch is that many others have stumbled over defining interactivity too and have opted for other words or descriptive phrases to use in its place as of late. 

So, what do we mean when we say that something is interactive? And, in the realm of art, what may that something consist of? There are a number of examples and strata to be considered.

The spectrum of action & ‘live’ art

Unless we’re prepared to go down a philosophical rabbit hole, we typically do not regard the experience of viewing a painting in a museum as an inter-active one.

The painting is, by contradiction or negation, a categorical case of non-interactivity.

In our encounter with a painting we can say we are, at most, active as interpreters. When the object of visual art is a sculpture, we can describe our experience of it as being slightly more active than our experience of a painting, as we are enticed to walk around it. Put it outside, and we might even be drawn to touch it or climb on top of it. But, traditionally, in the gallery, what we see, we don’t touch; there is no inter to our action.

There is by contrast a significant amount of contemporary art that does invite our touch or, alternatively, our gesture. Much of it even goes one step further and responds to that physical input. Contemporary art of this kind is typically (but not necessarily) characterized as interactive by its use of software, or technology more generally. Therefore, addressing an example that involves both corporeality (or corporeal literacy) and human-computer interaction is essential.

Opposing the painting on the spectrum of interactivity is Ruairi Glynn’s Fearful Symmetry, an installation featuring a glowing tetrahedron that some would say “dances” with members of the audience. Glynn used an array of Kinect cameras—among other devices and materials—to create Fearful Symmetry, which was commissioned by the Tate Modern to inaugurate the London museum’s new ‘live’ art space, The Tanks, during August of 2012. FearfulSymmetry brings together robotics, interaction design, and gesturerecognition algorithms into an event of puppetry in which both the tetrahedron and its visitors take turns being the metaphorical puppets and puppeteers.

Participation as a governing factor

In between these endpoints — that of the painting and the robot puppet—there is a spectrum of other aesthetic works or practices that can be described as interactive with varying tenability. This spectrum has been theorized by computer art pioneers Stroud Cornock and Ernest Edmonds (1973) as consisting of the following categorizations or strata. Given the extensiveness and precariousness of what can be called interactive art, I take these to be neither mutually exclusive nor all-encompassing.

(a) The static system—the art object is unchanging, as is any traditional painting.

(b) The dynamic-passive system—the art object changes with time by the artist’s program or by factors in the environment; Dutch artist Theo Jansen’s wind-powered sculptures are great illustrations of this kind of system.

(c) The dynamic-interactive system—the art object is participatory: that is, it accommodates input from the participant. Many of Jeffrey Shaw’s installations, which are part kinesthetic and part digital or virtual, make an obvious case for such a system, but technology isn’t a necessity, as is demonstrated by Surasi Kusolwong’s participatory work which emphasizes social engagement.

(d) The dynamic-interactive (varying) system—a network in which the conditions of (b) and (c) apply, with the additional condition that the art object’s behavior can be changed through the modification of system specifications by a human or software agent (Schraffenberger & van der Heide 2012). This last strata is the most complex. It helps to imagine a dynamic-interactive (varying) system as capable improvising via learning & adaptation rather than just following a script.

With all the above strata in mind, it is possible to qualify the painting as “static” and Fearful Symmetry as “dynamic-interactive (varying),” with the varying element being Glynn’s behind-the-scenes team of software puppeteers who programmed the robot’s responses in real-time when they saw appropriate. The case of Fearful Symmetry is, however, a curious one. When deprived of human participants, we would expect the glowing tetrahedron to revert to being a static system rather than one which responds to factors in its environment.

Interactive public art, even when it uses strikingly similar mechanisms, functions differently. 

A dog that runs in front of NAPA’s Blue Hour, like an unaware passerby, would technically elicit a response by activating motion sensors. It would, nonetheless, evoke the timeless thought experiment: "If a tree falls in a forest and no one is around to hear it, does it make a sound?" In other words, just because something is responsive doesn’t mean it’s interactive. Without establishing a feedback loop, an art object that affords participation but does not embrace that participation as a governing factor is not interactive. Carsten Höller’s functional playgrounds nicely expose this lack.

The supposed interactivity of augmented reality (AR) should hypothetically fit into the spectrum outlined above. However, because the categorization of AR is complicated by the possibility of considering either or both the augmenting interface and the objects in its real space of operation as the art object(s) or system(s), AR warrants a discussion of its own.

(Not) placing a finger on the concept of interactivity

While Cornock’s and Edmonds’ array of interactivity may help us plot contemporary art on a graph of interactivity based on the degree of human agency afforded, I believe we are in need of somethingmore than input in order to qualify what happens in an encounter with art as interactivity. And what that interactivity consists of is in need of conceptualization, not definition or distribution.

To put it another way, the spectrum explained above is in need of a conceptualization of interactivity that’s not wholly invested in a demonstration of technology’s mediating capabilities, but interested in the technology’s tweaking of lived experience and the reciprocal effect of that experience on technology—a back and forth of action and reaction. Thus, following Brian Massumi, I believe:

What is central to interactive art is not so much the aesthetic form in which a work presents itself to an audience—as in more traditional arts like painting, sculpture or video installation art—but the behaviour the work triggers in the viewer. The viewer then becomes a participant in the work, which behaves in response to the participant’s actions. Interactive art needs behaviour on both sides of the classical dichotomy of object and viewer. (2008)

One would not be wrong to interpret Massumi’s use of the term behavior, as having human(e) connotations. What Massumi is describing suggests that we forgo speaking of paintings, wind sculptures, and kinesthetic design when speaking of interactivity, and instead embrace works, objects, or systems that surprise us with properties that surpass mimicry and formulaic response with unexpected but relational feedback, as in Fearful Symmetry. In other words, the experiencer and the art object or system together should co-govern the interaction.

The something left to be desired

All in all, I think that the something more we need in a contemporary conceptualization of interactivity may be immediacy of presence (as in, let’s get rid of mediating interfaces that dilute our access to the object). Or some kind of communication—even coordination—which necessarily supposes a possible feedback loop that is predicated on a perpetual state of transitory open-endedness. This would make possible the emergence of otherwise unattainable properties that would make the happening a complete and iterative, but non-repeatable whole. And perhaps what that something entails, then, are the qualities of an event—a singular but passing occurrence that “has an arc that carries it through its phases to a culmination all its own: a dynamic unity no other event can have in just this way” (Massumi 2011). In this sense, I see installation art as advantaged, because it demarcates a special place in time and space.

Moreover, in any event of interactivity, I would expect not only social interaction among human participants, but also a mediating object—imaginably, with a liveness of some kind—that makes the event or the experience possible in the first place. Drawing from the artistic impulses of the 60s, this is a conceptualization of arelational aesthetics, but with technology at its center. It is a conceptualization that shines a light on the engineering of experience where engineering retains its mechanical connotations without manipulating human agency in such a way as to reinforce the subject-object binary. In this aesthetics, gestural and physical modalities are preferred over representational or linguistic ones, solely on the basis of their dynamism. For public art, an interactivity like this is key, because:

For art to switch its role from the private, exclusive arena of a rarefied elite to the public, open field of general consciousness, the artist [has] to create more flexible structures and images offering a greater variety of readings than were needed in art formerly. (Packer & Jordan 2002)

With a greater variety of readings, there may be greater chance for meaningful improvisation. And with this, we cannot go without reconceptualizing our notion of the artist in this approach to interactivity: it’s no longer necessary to assume that he or she has a particular craftsmanship, rather he or she is “a catalyst of creative activity” (Cornock & Edmonds 1973). As Bruno Latour (1996), speaking of puppetry, puts it:

If you talk with a puppeteer, then you will find that he is perpetually surprised by his puppets. He makes the puppet do things that cannot be reduced to his action, and which he does not have the skill to do, even potentially. Is this fetishism? No, it is simply a recognition of the fact that we are exceeded by what we create.

Looking back on The Universal Set as an experiment in play

The Universal Set was an installation created by New American Public Art in 2015 for the Lawn on D, an outdoor event space in Boston’s Waterfront District, replete with beer, art, and activities. Unlike most of NAPA’s other works, The Universal Set was not technically responsive, and in many ways, it was a complete failure.

what people are doing when they create art is ‘creating’ deviance.

In place of a summary of technical details, though, a word on the geometric construction of The Universal Set is needed to adequately tell the story of this work. The Universal Set was a dodecahedron—a regular solid of twelve pentagonal faces—made of whiteboards rather than snow. Legend has it that when Plato associated the four elements—earth, air, fire, and water—with the four regular solids, he assigned the sphere of the universe to the dodecahedron, the fifth. Perhaps this was because the volume of a regular dodecahedron is closer to that of a sphere than the volume of any other Platonic Solid (Phillips 1965).

With this in mind, nearly two and a half millennia later, it becomes clear why NAPA’s dodecahedron was aptly named The Universal Set. In its brief life on the Lawn on D, it was at once a shape representing the universe and a work of art presenting to its public an unlimited set of allowable actions...that is, anything that could be done—and undone—to a whiteboard with a marker. 

Art & play are two sides of the same coin

According to game designer Raph Koster, games, like machines, are meant to wiggle. “You poke and prod them to see what comes out the other end.” In spaces of explorable areas, where agency comes in the form of different verbs, this ‘wiggle room’ can be better understood as play. Rather than pointing simply to foolishness or childishness, the play-concept signifies a temporary sphere of activity with a disposition separate from that of ordinary life. Therefore, describing the interactivity of public art in terms of play makes a lot of sense.

In 1955, a Dutch cultural theorist named Johan Huizinga wrote a book on the significance of play in culture and society. In it, he explains that play—like art or creativity more broadly—instantly assumes form as a cultural phenomenon. Similarly, Huizinga observes:

The words we use to denote the elements of play belong for the most part to aesthetics, terms with which we try to describe the effects of beauty: tension, poise, balance, contrast, variation, solution, resolution, etc. (1955)

These terms are more likely in the purview of those interested in game studies than in our common anecdotes about children on the playground. But consider this: it’s important for games to be neither too simple nor too complicated; bound up in this sentiment are all the effects listed above. Moreover, we can draw a parallel between the work of game development and the design of interactive public art. Both increasingly include predicting and governing human behavior through technical and social means. In other words, the mechanics and signs in games, as in interactive art, are configured to provide compelling problems and afford creative opportunities. In the case of The Universal Set, the compelling problem and creative opportunity were one and the same: 12 blank spaces.

Even art needs some certainty

Maybe we should think of The Universal Set as a possibility space. In abstract terms, a possibility space is a decision-space or a conceptual space of possible meaning. Of course, when talking about blank canvases, this is not so abstract at all. The Universal Set was materially a 3D space holding nearly limitless potential for playful illustration.

Great games engage players with rule sets and constraints which make only a subset of all possible interactions available. Choice of input, or of agency in such games is therefore limited but consequential. Constraints as well as affordances—what the properties of an object offer to an interacting agent—are two tools for defining a player’s agency. When there’s a single but exhaustive affordance (i.e. unlimited, over-writable space) without any constraints (i.e., prompts or repercussions), the poise and the balance mentioned above are forfeited for the production of otherwise unanticipated results.

Thus, it’s fair to say that the intentional physical and conceptual structure of The Universal Set did not go so far as to structure its reception. The forms of expression that NAPA hoped forwere not imparted onto the piece by the public. This begs the question of what kinds of affordances and constraints interactive public art needs to incite creative participation.

To design for dynamic expression, one should design for counterplay

A quick and easy detour through statistics may help us pinpoint what kind of participation aesthetic interaction should aspire to. When we’re talking about the expressivity of the public we can think of the variance that may exist among individuals in terms of standard deviation. Behaviors that differ from those of the majority will fall on either side of the standard bell curve.

At the far ends of this figure, we have individuals who either want to offend or delight the majority of the population. On the left, there are “sinister deviants” who will use an entire pentagon of The Universal Set to draw obscenities. And, on the opposite end, there are the “righteous deviants” who have the capacity to pick up a marker and draw the next Mona Lisa amidst the basic autographs and Twitter handles. Sometimes, the righteous and sinister creations may even be mistaken for each other. This is because what people are doing when they create art is ‘creating’ deviance.

To design for dynamic expression, then, I would argue that one should design for counterplay. In game studies, counterplay refers to a constructive form of deviance in which players creatively toy with the rules and their boundaries. Counterplay opens the possibility of an antagonistic relationship between the game and its players. To encourage counterplay, what’s needed are limited but flexible models of interaction—possibility spaces—that afford not only rule-following behaviors, but also opportunities for reconfiguration or modification. Most people enjoy a challenge.

The next element to consider when designing for dynamic expression is behavioral mimicry. While partly left to chance, mimicry has a lot to do with anticipating the potentials and pitfalls of bringing together people and creative objects or technologies in particular places.

An article published in the Journal of Experimental Psychology in 2009 sheds some greater light on this. Its authors argue that

[B]eing mimicked by an interaction partner cues convergent thinking by signalling a social opportunity for collaboration, while not being mimicked cues divergent thinking by signalling a social demand for improvisation and innovation. (Ashton-James & Chartrand 2009)

Another way to look at this would be to say that both being mimicked and not being mimicked, or not having an interaction partner to influence or be persuaded by, signals an opportunity for deviance. In addition, we can gather from this that interactive public art can be a means of social adaptation. The right object or technology put in the right space affords us the possibility of working with others or following their traces to connect dots and also think outside of the box. This is the stuff that “generative expression” is made of.

As Darwin (1859/1999) noted, ‘‘in the long history of humankind, those who learned to collaborate and improvise most effectively have prevailed.” (cited in Ashton-James & Chartrand 2009)

What can potentially emerge from mimicry but also from the right folding together of people, place, and medium—whether anticipated or not—is a kind of curation that can perhaps be best described as a cultural thermometer, with each ephemeral or enduring trace imparted a signifier of the zeitgeist.

Undirected consensus can make for bad public art

All this in mind and the photos to boot, The Universal Set is perhaps best described as an experiment in random noise whose outcome would make Darwin quite concerned. It failed because it did not provide its audience with enough structure to meaningfully interact or with enough rules to playfully counter. It would be unfair to explain typical participants as compelled by the bad kind of deviants or deviance though. Rather, they were more likely overwhelmed by all the undirected possibility put before them in the environment.