Performance is so many things: the synchronized sounds of a symphony; actions with words in a play; steps and turns in a dance; words from a pulpit. Performance art, too, is variable, perhaps too multifarious to define, even with semicolons. At traditional performances with traditional support materials, from symphonies with program notes to theatre productions with playbills, performance acts as replay, a repeat of an event, a memorization of a string of notes or a set of lines, a reformulation of a tested formula. Then there are those performances that vary, that respond to the moment, that unfold through the implementation of chance or improvisation or, more and more, digitization. With the insertion of new technologies into performance, the question arises – do actions result from numbers? What indeed is the connection between the physical and the digital? Does the digital component determine the performance, or do actions generate a numeric pattern, which then underlies the piece's structure?

The aesthetic and conceptual import of digital performance pieces is linked to the ordering of a piece's technological components. Random sequencing is one form of structuring immersive environments or data-triggered scenes. Fixed sequencing of scenes, with a predetermined index of performed actions and triggered events, follows a preset score. Alternately, sensory responsive improvisation is flexible and often produces variations in structure. In each case, the piece's content is the result of a digital system: programming or computer responses to external stimuli determine how the performance plays out. Even interactive improvisation, in which a human action triggers a computerized event, is a digital system, albeit one that emphasizes the human element, or input, in that system. The content of an interactive piece is closely related to its structure – the interaction between trigger, whether generated by the viewer or performer, and event. Interesting variations in content emerge when the structure becomes the art.

Below, by electronic interview, four new media artists describe their modes of working with interactive technologies and probe the relationship between order and content in their work. Johannes Birringer makes telematic connections in performance, installation, and video. Mark Coniglio co-directs the interactive dance company, Troika Ranch, with Dawn Stoppiello; he both designed and implements the interactive software, Isadora. Cat Jones' alternate persona catgURL interacts with viewers while performing live on and off the screen. Steve Shoffner instigates interactions with viewers while performing simultaneously on video and within his installations.

Telematics, the exchange of video between remote locations, is the favored medium – or media – of Johannes Birringer. Because of and in spite of the spatial distance between participants, making a connection either within a remote space or with another individual is often the creative objective. Again, the structure determines the content: the personal connection embodies the technological link…

Johannes Birringer:

I understand "telepresence" to be a form of connection through the use of the internet (as a medium), allowing performers to act upon each other, perform with each other, together or in a shared (virtual) space, and thus be “physically" present in a remote site, so to speak. One is not really physically present in the other/connected location, but the effect is as if one were present, and for me, as someone coming from dance or theatre, performing together is based on physical action and creation, while the paradoxical challenge, in this case, is the use of digital media. The internet/streaming media, as well as various kinds of interactivity (using software to mix data objects in real time), is of course to imagine touching someone in a place where you are not. It is more than touching or speaking of course (as one would in a videoconference, or using mobile telecommunications) – what interests us is how we create a multimedia performance situation online, with partners, with other human, creative people, and how such a situation of love, or collaborative creation in the network creates a third space where what is being created is subject to a rather radical (or complex) contingency; the network is the space, the telecommunications the medium, but the performance is human and generated through our embodied actions and consciousness, how we experience such a shared/distributed space – especially if it is subject to real time mixing, manipulations, unexpected happenings/distortions, delays, degeneration, and so on.

For me the most human element is this fragility of physical communication across a huge distance, but of course there are also digital processes involved (camera based, sonic, software, telecommunications protocols and mediations), and what was once possible, in multimedia theatre or dance or music, to create in one space, can now be created in connected spaces, multiple sites.

I have created a lab in Germany which meets every summer (since 2003), and although many of us work with interactive designs, and sophisticated technologies, the basis for our work is not only the human actor or performer, but the basic philosophy of the lab (called Interaktions-Labor: http://interaktionslabor.de) is literally an investigation of interaction in a concrete location, in this case a historical site (an abandoned coal mine). The site has many empty rooms, and so we need to fill them again through our actions, and our instruments (the digital tools), but data alone, or data exchange alone, for me does not constitute interaction. My assumptions about art as relational form, or inter-action, are based on a social understanding of all art as human communication or as creativity involving intelligence. Perhaps we can include machine intelligence, and we of course do so, but I am not a robotics engineer or AI engineer, and artificial intelligence is not my field. Therefore in telepresence someone is present whom I consider an actor, or someone who moves and is moved by me (I hope).

Isadora, the software by Mark Coniglio, accommodates such interactions. Its design lends itself to both fixed and flexible indexing of interactive pieces.

Mark Coniglio:

Isadora is a graphic programming environment that provides interactive control over digital media, with special emphasis on the real-time manipulation of digital video. Because every performance or installation is unique, Isadora was designed not to be a "plug and play" program, but instead offers building blocks that can be linked together in an almost unlimited number of ways, allowing the user to create a custom interactive system that follows his or her artistic impulse. A friendly user interface helps novice users quickly create rich interactive relationships, but also offers the expert a fine degree of control.

Isadora's modular approach is the most important aspect in defining Isadora's structure. Data flows from one module to the next, being filtered and massaged as it moves through the scene (the Isadora parlance for a group of related modules.) This modular approach suggests a sort of complex "plumbing"; one could imagine water flowing through a set of pipes, with the modules mixing and controlling the flow, or perhaps adding food coloring along the way to change the way the water looks.

Mark Coniglio implements his software in multimedia dance works performed by Troika Ranch. 16 [R]evolutions is the company's latest project.

In our systems, the movements of the performer are analyzed, and the qualities of those movements used to generate the media response. In one way, and quite the literal sense of the word "index,” one could think of the movement as an index into a database of visual and aural results. This is not 100% accurate, because in fact each movement and each response exists in a continuum. But, still, a given set of movements, if repeated exactly and preceded by exactly the same sequence of movement would produce exactly the same media response (This is unlikely if not impossible in an improvisation.). Still, the movement is a means to accessing the imagery and sound – and that is a function of indexing.

16 [R]evolutions is a multimedia dance theater piece that explores the tension between our animal and intellectual selves. The work portrays the disconnection and confusion faced by four characters who have lost touch with the animal drives of their pre-human ancestors. Theatrical sections show the characters becoming increasingly numb to their surroundings, while dance sections suggest a pure, powerful and sometimes brutal, animal nature.

The piece features three-dimensional, interactive visuals that warp and morph in response to the dancers' movement. This real-time interaction is accomplished by linking camera-tracking systems to Isadora. At one moment the performers dance with delicate, living strands of DNA, at another, a gigantic digital rib cage ripples and breathes as it envelops their movement. By giving the dancers interactive control over the visuals and sound, the media becomes an extension of their body and ensures that each performance is unique.

Deploying various technological tools, from pressure sensors to cell phones, Cat Jones also uses Isadora in her interactive performances. A naturally adaptive performer, Cat Jones infuses a fixed index with spontaneity. The range of content in her performances depends upon viewer input and her reactivity. Cat describes the digital persona and performance catgURL :

Cat Jones:

catgURL is a live, cross-art form, performance event with integrated media and audience interaction. As catgURL, I connect with the eclectic nature of cyberspace, fads, fetish and cults in popular culture such as super heroes, comics, cyborgs, science fiction, electronic toys, b grade films, the virtual, the visceral and the voyeuristic. †I become a web persona: †catgURL hosts cyber sites, presenting feline fictions of the feminine kind. †The persona is kitsch, queer, carnal and just a little bit cyber, patrolling a world of many genders, fetishising jargon and exploring the universal desire for definitive sexuality.

In building the performance site, I hijack jargon from archaeology, anthropology, engineering, medicine, animal husbandry, programming languages and domestic software. I write the work in non-linear chapters reflective of the Internet. The structure of the performance is modular. The sites can be rearranged, removed or rewritten depending where and when the performance happens. The work can be augmented by performance modes or other platforms outside of the main venue. Audiences might see a few seconds of catgURL on a video screen at another venue, hear a radio broadcast in the car, pick up a zine, connect with web streaming at a cafe, receive an SMS on their mobile phone, or catch a surprise performance at a club. This multiplicity reflects the notions of time and place in cyberspace.

Indexing plays a huge role in the writing of this work – as I said I do a lot language hijacking and rewriting – so my research is a lot in reference books and, funnily enough, I get the most out of the indexes – both condensed uses of language and the order that the field requires the information to be in to allow greatest understanding and progression. On top of this some of my writing processes involves writing lists of phrases, which I search for meaning in and then go through a process of editing and reordering the lists.

The performance of catgURL is a cross between performance art, poetry, and [digital] cabaret. The audience engages with the work in various ways – through participation and contributing vocal samples and writing. The set is reconfigurable, and the audience is guided through and around it by the performance's action and media. The media, in turn, is often triggered or manipulated by the performer and/or the audience. Viewers alternate during the piece from interactive to passive – they start with interactive though – it's much harder for people to change modes the other way.

In the foyer viewers log on to a computer and create an identity; they give vocal samples, etc. When they enter the space, they get scanned – have a bit of a portal-like experience. the performance area is a bit like an installation at first, and they have to activate an object in order for the show to start. So one might move into another area whilst the rest stay together. During the show sometimes an audience member has to do something – there's a chat booth…

My chat with them has a structure but the lines also have a certain ambiguity and open-endedness. The chattee responses are dependant on their individual and unique interpretation of what I might be talking about. It's also structured so that if they are struggling my text is such that it can help them along or fill in for them altogether. I also have a selection of drop-in lines ready for certain types of responses – and I can disengage them verbally and technically without the audience realizing that its not part of the show if its not working or they become disruptive.

While Cat Jones responds to an audience as a performer onstage and onscreen, Steve Shoffner responds to viewers through the guise of video installation, in the context of a visual and spatial illusion. A character in his own art, Steve Shoffner manipulates video images of himself, either by controlling existing clips or responding live to viewers via closed-circuit transmission of a live camera feed. His sets include video extensions of the architecture, with the video component bleeding into the real (and real time) environment.

Steve Shoffner :

My video installations explore the interactions between people and things in a space where expectations and perceptions are askew. I play with the surrounding environment, sound and imagery, all of which are both virtual and actual (due to the performative element in the work, as my real-time recorded image is inserted into the live piece). With the use of closed-circuit video cameras, false walls, and my own physical interaction, I set up illusions for the viewer to negotiate the validity of real time and real space. I want there to be an elongated fuse between the time that the spectator approaches the work, and the time that they deduce what is actually happening. Once the viewer becomes aware that the installation might be a live broadcast, and is not pre-recorded, or in some cases, is pre-recorded but the footage is controlled and manipulated live, I interact with the viewers to further bewilder them. The increasingly startled viewer can then reflect on how awkward it is to interact with another human being through technology.

I want my projects to be humorous without being confrontational. All of my installations aim to reenact scenarios where technology leaves us confused and disconnected. The intention of my installations is to provide an experience of discomfort, to provoke an odd interaction, and to emphasize the strange things surrounding us.

All of the artists exploring the possibilities of interactivity depend upon their audience's active interest in the relationship between an event and its stimulus. Art that plays out uncertainly, letting viewers discover its unknowns, is decidedly braver than art that follows a prescribed order, since the unknown is not always successful. Audience response corresponds with cognitive awareness – the ah-ha – that occurs when the viewer makes the connection between an interactive digital system and the performance. Without cognizance of this interactivity, the audience is wary, deeming the piece illogical, a sequence of disconnected actions when in truth the actions are interconnected by a technological interface, one that proposes a digitally infused reality. While those who recognize the interactivity within the digitally indexed performance may respond with an ah-ha, those who are unaware remain outside the interaction – disconnected.

Regarding CatgURL,

Cat Jones:

The structure is interactive and non-linear, but it's not random. It's structured within an inch of its life – this is a key component for catgURL to exist as a ‘successful' live show. It's quite punchy so the timing and energy is quite important. There is a relationship between the viewers' interaction, though, because they contribute content – in the chat booth they can say anything to me – and that will determine my responses and how the rest of that piece turns out and then what its meaning is. Their interaction with media is perceivable to them and to each other (and fun). The audience members are guided to the interaction – some space is allowed for the discovery – because in this show that creates a whole other fascinating performance. Audience members watch each other experience the interactions – they share each other's excitement, discomfort, surprise – and either want to do it themselves or are so relieved it wasn't them.

In chat noir, I have made a structure that supports the audience's interaction. My spoken text is ordered/indexed to allow both for audience spontaneity as well as control from my end. That same structure can limit the audience – I want them to be involved with the media. When I see shows that have some really fun-looking contraptions, I get really excited but so bummed that I can't play with them. It's a rare opportunity as an adult to be immersed imaginatively and get to play physically at the same time. So I am creating an environment of fantasy/fiction that the audience can get their hands on too. They get to become someone else – the show is about exploring identity. When they log on at the beginning of the show they create a logon name and answer questions about their logon ID. I call them from the audience for chat noir by their logon name – so they come to chat noir as a created persona – one that they created. They enter a booth so that physically they are protected from the rest of the audience. The audience can see them on a screen, but the image is mixed so that it's also a fictional version of them.

By integrating the viewers' personal chats into the performance, the private and public converge. The contributions and responses from the audience contribute both to the work and to how the rest of the audience sees the work. In this sense, form influences content – in the same way that identity is formed by the influence of environment.

Steve Shoffner relies upon the same discomfort that catgURL provokes. That moment when the unknown becomes known is crucial to the piece, as is the relationship between performer and viewer.

Steve Shoffner:

My video installations are built to appear pre-recorded so that the viewer feels free to gaze at the subject (me). Viewers feel comfortable gazing at the subject, as they have become familiar with examining people on TV and in movies without fear of the subject matter looking back. When they realize that the video is actually interacting with them, the role of the voyeur reverses. The viewer now realizes that he or she is the subject – the concept behind the work. This feeling of awkwardness arises from the sensation of being put on display without notice.

The boundaries of public and private space become blurred when the viewer becomes aware that the assumed pre-recorded video is actually live. Viewers feel as if they have been placed on stage, therefore tossing all comfort zones out the window. The relationship of the performer/audience contrasts with traditional interactive performances, primarily due to the fact that the audience is almost always initially unaware that what they are viewing is a live broadcast, and because of course there is no traditional stage. That relationship is again inverted when, conceptually, the viewer becomes the performer.

What, beyond the compelling parallel to the mathematical and the digital, makes the computer-enhanced experience enhanced? Is it the live aspect of the work that creates the discomfort, the surprise and, optimally, the eventual ah-ha? Is it the novelty of multiple modes of viewing – live action and recorded phenomena, or perhaps the remarkable clash between the physical and the ethereal – human presence vs. technology? The parallel between life and numbers acted out by artists and computers, though certainly taking new forms, may not be novel. Consider the musician, converting numerals into notes, or the dramatist, matching players with words, actors with stories. The interplay between performers and their art is perpetually hyphenated, the physical and ethereal connected by time. Story unfolds and action falls steadily forward.

The technological, when introduced to live performance, acts as an applied index, an ordering of sorts. This ordering may be linear, as when one toggles through scenes, though more often computers allow for the non-linear: programming a random control or creating a reaction to external phenomena recorded as sensory data, like movement in a room or the moisture in the air. Add a random repeat, and time hiccups. That hiccup is what makes the piece computer-enhanced. Digital input/output – the overlay of data over time-based performance, enhances: program random into art and press play .

Johannes Birringer:

When we work with computing, and data processes through specialized software that analyzes and does something with the data, we are not doing video anymore – i.e. there is no direct realist/representational or mimetic relationship between dance and image. Even experimental digital video based on shooting a dancer, and then manipulating the image, is not the same as what I am talking about here (although final cut pro, or whatever you use for editing, and Adobe Aftereffects, are of course allowing you to work with digital manipulation of the data, but it is recorded data). So I tend to distinguish between postproduction (editing) and real time synthesis, and in our workshops we explore what we mean by telepresence and live synthesis. So when you have a dancer perform with sensors applied to her body, or tracked by infrared cameras or by a camera linked to an interactive software system (Isadora, Max/Msp/Jitter, soft VNS, Eyecon, BigEye, Eyesweb, Soundbeam… The former are standards in our community and, like Lifeforms and BIGEYE, which have been around since the late 80s and early 90s, some since the mid 90s, so we are talking about a 10-year history of interactive dance), then you are designing a work or a landscape (for improvisation) that is unstable and depending on constant, continuous input. The movement is tracked and processed in real time and the software activates various outputs (sonic, light, video, graphics, text, 3D animation, etc.). That relationship defines for me a new era in the development of composition and of working with computing tools, concepts of the digital as medium, and interaction design. And this is what the workshops and the collaborative process are about, and one important dimension of working in a digital team (i.e. multi-user, multimedia, and multimodal platform) is that in such live performances, everyone is able to access the media and transform each other's live data so that there is no more solo, no more duet, not more dialogue: it is always more of a complex system process.

The hiccup introduced by technology may not be random at all. It may be a coordinated insertion of a pre-recorded motion into the video backdrop of an elaborate theatre piece, or perhaps a storm of static produced by the computer in response to the level of audience applause. The hiccup enhances, or it annoys, depending on the viewer, because of the uncertainty of the relationship between random and play, between technology and life.

Mark Coniglio:

The structure of 16 [R]evolutions is linear but quite flexible, as several sections of the piece are improvised. It has always been our feeling that, if one is going to offer the performers interactive control over the media, then it is important to allow some level of improvisation. All live performance is a negotiation of a work's fixed and performative elements: the ideas behind the piece itself, the choreography, the interactive systems versus moment-to-moment awareness between the performers, the audience, and the space. So, while the overarching structure of the piece is the same each night, each improvisational section can be fairly different, depending on how the performers interpret these tensions.

The work is presented quite traditionally, in that the audience views the work in a proscenium or black box theater, watching the action from their seats. But, in terms of their how they are called to interpret what they see, I would say that I am happy to offer clues, but not to provide explanations. My hope is that by providing a rich and compelling series of images, and by placing those images in contrast to each other, the audience will form their own personal narrative, that they will engage with the symbols and metaphors provided by the work and integrate them into their own experience of the world. I have always found that the artworks that affect me most profoundly operate in this way.

Well, as I said, the structure is not random. That being said, the images presented do not build up to a climax in any sort of traditional way. I do think that some viewers are puzzled as they search for the "message" of the piece, because the symbols and metaphors don't add up for them. Others have found the work deeply moving. The way an audience responds depends completely on the individual. A recent performance of 16 [R]evolutions in England generated two emails, one very, very negative ("It amounted to digital graphic shapes and patterns and nothing more; loud, brash, and unsophisticated") and another extremely positive ("the dream logic of the piece resisted instant gratification but was deeply affecting & rewarding"). If we were to make a piece that everyone could understand, then I reckon we'd be making television. The fact that we receive disparate responses to what we present is, to me, a good indication that we are neither being too obtuse nor catering to the obvious.

I don't think that the interactive structures/systems have much to do with their being able to "get" the piece. But, if they are aware that these images are being created by the performer interactively within an improvisatory structure (which is made fairly clear in the opening section of 16 [R]evolutions) it will only enrich their appreciation of being "in the moment", which is one of the core reasons to see live performance in the first place. That being said, the fact that a performer takes control of the media is actually an important metaphor in this particular work. In several sections, the audience sees characters who have become totally numb and disconnected from their environment. The moment at which they control the media is a moment of empowerment and engagement for these characters.

If anything, interactive indexing emphasizes the chaotic nature of contemporary life. As I said above, the piece does not move through any kind of narrative arc. Instead, different scenes are presented, moving from a dance section to a very theatrical section, a section that emphasizes the powerful side of our animal nature versus disengaged intellectual beings. We mix these things up, because, for Dawn and me, the context of the materials is key. By placing these dialectics in opposition to each other, we feel deeper opportunities for interpretation by the audience are created.

The complexity of a digitally layered performance reflects the unruliness of life – the chaos that Mark Coniglio relates to nonlinear, interactive indexing. The initial query, “Do actions result from numbers?” retreats as questions of reality and illusion, reaction and interaction, and physicality and digitality emerge. Performances integrated with new media almost always relay an ambiguous relationship between the physical and digital. That ambiguity and the resulting disconnect and then connect between the performance and technology, or action and numbers, requires an active audience. With the blurring by the digital – the unknown and variable – the mapping is indistinct, the viewer left searching, the experience now a quest. Of course not all audiences search deeply; not everyone really experiences. Yet there is something compelling – that which is unexplained, the ghost in the machine.