Two worlds confront each other, the world of culture and the world of life, the only world in which we create, cognize, contemplate, live our lives and die or- the world in which the acts of our activity are objectified and the world in which these acts actually proceed and are actually accomplished once and only once.  (M.M. Bakhtin, 1993)

What Bakhtin, a Russian philosopher and literary scholar, chiefly concerned with the novel, ironically calls to our attention in this quote draws a fuzzy boundary between the actual body-based process in momentary experience or “real-time” actions, improvisations, interactions, generative processes or simulations, and how we utilized media work to create representations or reflections of this experience we refer to in narrative forms. Bakhtin was expressing the limitations of the medium of literature to really get to the core of our being. We know that technology is closely tied to creative production, as computational speeds increase and technologies become more inexpensive and pervasive, our media world changes from a focus on functions of representation or “objects” of expression to those that include components of ineffable experience- momentary acts, performances, conversation-like interactions, improvisational play and participatory culture. VJing and live A/V practices are an expression of this change and represent a continued desire for the intersection of the senses, such as sight and hearing, as well as the performance and interface of the body with technology and human expressivity. VJing and live A/V practices support a convergence or mixing of the elements of rhythm and movement in sensorial and motorial engagement, from the extended interfaces of sensors used in performance, to mass scale gaming systems like the Nintendo Wii. We also see the process in the making of media work through techniques of software and hardware exposed as part of the work itself. This process becomes performance.

What happens in the movement from forms of media work where we are identified through representational practices, to a media-scape based on actions, preferences, improvisations and performances that we create and share in an increasingly socio-technological context? New media, such as games, socially interactive websites, simulations, live A/V performances or VJ practices are situated in a context that foregrounds the body and environment as part of the cognitive act. These momentary, time pressured relational processes engage persons and environments in complex ways quite different from more traditional, representational and mass scale media and arts practices. What rises to the surface is that these occurrences build further upon representational structures, as they are more than what can be represented, and require a thinking from and an analysis related to content that is generative, participatory, interactive, communicative, simulated, performed, body-environment based and experiential in nature. Fundamental practices in these forms emerge, such as the use of rhythm and movement, dramatic intensity and improvisational performance. VJ practices are often composed and improvised in a collaborative environment, as well as physically manifested and interfaced with the body. These cultural practices require us to rethink and expand our toolbox of inquiry beyond representation and into presentation, the experiential and convergences of the psycho-social-physiological.

Early Abstract A/V Practices- Rhythm and Movement

All of a sudden it hit me---if there was such a thing as composing music, there could be such a thing as composing motion. After all, there are melodic figures, why can't there be figures of motion? (Lye)

We perceive rhythm in three different ways.  There’s rhythm we can hear, rhythm we can see, and rhythm we can feel. (Block, 2001)

In Western art, there is a historical intersection of musical practices in visual work that synthesize rhythm and movement dating back to the early 1900s with the film experimentation of a number of artists that use abstract, non-narrative content as part of the work. Some of this work finds itself contributing to VJing and similar practices like audio-visual or improvised performances. The work of artists like Hans Richter, Walter Ruttman, Viking Eggeling, Mary Ellen Bute or Oscar Fischinger, for example, create convergences and remediations of the audio-visual and performed.  Although not technically performances, the work was created via a performance process. They also worked to synthesize disparate sense modalities and take advantage of the complex cognitive processing of the perceiver. They intuited ideas that are now explored in science through cross-sensory or multi-modal perception and media psychology, such as how vision influences sound, sound influences vision, sound influences touch, etc. The early work of such artists was also influenced by the nuanced, abstract and time-pressured expression of movement in audio and visual, time-based forms and a kind of delight in the sensorial and performed moment. Contemporary live A/V work has similarities to this earlier work beginning with “light” or “color” organs, instruments designed for the live performance of light and visual media beginning in the mid 1700s, as well as early filmic and animation experimentation. However one could also consider even earlier works using light and shadow, such as shadow puppetry.

Sound and light both share wave-like properties and we perceive them in both time and space. Coupled with performance, there arises a time-pressured practice that engages the process of being human. The marriage of waveforms of sound and light established in experimental cinema and animation embody an art of movement. With the invention of film, creative work reintroduced time. Artists remediated performance and process in these new forms. This can be seen in changes from captured or fixed time, as in painting and photography, to time-based, process oriented work, organized around a “frame rate” similar to the “pulse” in music. However, the work still lacked the presentation of human performance.  A number of artist’s practices were built up around the desire to achieve this momentary, shared process of performance.

Experimental film and animation has shared its history with VJ practices via a context for working, based on movement- rhythm: pattern, repetition and tempo and measures of dramatic intensity: the expression of power or energy in a system, which arose from early artist’s frustrations with certain media, such as spatial limitations and the fixity of painting and photography. Much of this work in sound and image is extended or augmented through varying relational performance strategies of the body. With the introduction of interactive software and fast personal computing, this experimentation becomes creative, improvisational performance practice in real-time, which is often collaborative work between artists/VJs and musician/composers/DJs. This work demonstrates a further desire for tools and practices that engage in the choices in the moment, time-pressured challenges of mastery the responsiveness of an instrument or system, as well as how media can support social practices such as collaboration and participation in an environment.

There is a whole minority history of visual work that comes out of a musical, movement and experimental sound tradition and experiments in animation and cinema. This is a history of work that is challenging and difficult to commodify, has been appropriated into other forms, such as narrative film or music video, or fallen into the gaps in the institutionalization or the disciplinization of creative work. The Hirshhorn Museum and Sculpture Garden at the Smithsonian Institution in Washington, D.C., for example, organized Visual Music, a white-cube based show in 2005 that, “surveys the rich and resonant relationship between abstraction, color, and sound over the past century.” However, the lack of mention of rhythm and movement indicates a neglect of consideration for such time-based work, other than extending and expanding painting collections and “reframing” these via the appropriation of other artist’s practices. 

Two key components in this time-based work, rhythm and movement, appeared in early works by those such as Man Ray, Marcel Duchamp, Ferdinand Leger, as well as aforementioned filmmakers like Hans Richter and Oscar Fischinger, who utilized the sequential, time-frame structure of film as a material, as well as the performance process in the making of the work. For example Oscar Fischinger’s quick paced, jump cut laced in camera editing in his Walking From Munich to Berlin film from 1927, which predates the music video form. This project documents his walk in as a performance. In this work, he also uses the camera and camera technique as a performer. Other artists, such as Mary Ellen Bute, created experimental abstract animations from about 1934 until the late 1950s in the US. She had studied stage lighting in order to create a “color organ” for live performance, as well as collaborating with musicians in her films, (Moritz, 1996). She was interested in the time element of light and incorporating a performance aspect in her work. 

Rhythm and movement are important commonalities throughout audio-visual performed practices. Movement in film and video is simulated by presenting samples, painted or manipulated stills or captured images, at a fast enough rate that we perceive a smooth, connective movement of shapes and forms in the frame. Rhythm is created by the regular (or irregular) alternation of content and the negation of content over time. For example, sonic rhythm is created by the repetition of a drum hit followed by silence, repeated over time. One drum hit (1), does not create rhythm, however by repeating sound and silence, we create rhythm- 010101010101010101010101010101, etc. Similarly, the space in the frame can be divided up and rhythms can be created visually by repeating shapes in areas of the frame, as well as repeating pictures, described as pictorial rhythm, and through editing, known as editorial rhythm, (Block, 2001). Rhythm and movement are endemic to time-based media at the level of the frame or sample, where images and sounds or samples and stills begin and collect over time or frame rate to create movement and rhythm. Movement also becomes rhythm or rhythm can become movement. For example as we speed up a regular drumbeat, soon our ears cannot separate the sounds and we perceive a changing tone.  Rhythm and movement are also expressive and relational to the body, both in practice through walking, dance, etc., as well as psycho-physiologically, such as with heart rate, blinking, muscular activity and the cognitive processing predisposed to movement, etc. Artists would naturally explore these sensorial and bodily occurrences in expressive forms as different media technologies encourage or inspire, often playing with multi-modal perception, such as seeing sound, creating visual music or seeing and feeling an image, etc.

The 1921 project, Rhythmus 21, created by Hans Richter, uses rectangles and squares in negative space with chance procedures. Chance procedures have often been utilized in time-pressured contexts, such as performance or improvisation, and accidents are as important to the creative and inventive process as intention. Richter’s film attempts to show the transformation of content in time, similar to dramatic features in other media work, but without characters, plotlines and the grammar of film storytelling. Recto-linear shapes appear, grow and shrink, taking advantage of both visual rhythms in negative space, as well as the editorial rhythm of how the piece is cut. This process builds intensity in tempo and in density of forms. Richter said of his piece, ”The simple square of the movie screen could easily be divided and ‘orchestrated.’ …In doing so, I found a new sensation:  rhythm- which is, I still think, the chief sensation of any expression of movement.” Rhythm, as discussed above, the repeated alternation of sound and no sound, is created visually through the repetition of visual elements in sectors of the visual frame amidst negative space, (Block, 2001). By varying the tempo or speed, different intensities are created, charging the work with a kind of dramatic shape known in traditional narrative form as rising and falling action and in sound design as an “envelope.” These formal structures are phrases or gestures as well, where movement begins, moves through a shape and is finally completed, just like the movement of the human form. These varying intensities and rhythms are translated cognitively through contrasts that attract our attention and influence how the work is received emotionally through this tension and release.  The synchronies between the audio-visual content and the changing rhythms and intensities collect over time to create larger structural forms or movements.

Viking Eggeling’s Symphonie Diagonale produced in 1924 is another piece that is interesting in contrast to VJ and live A/V practices. Symphonie Diagonale uses basic forms like curves and line segments to create basic geometric shapes that form and ebb rhythmically in the black space of the frame into ever increasingly complex shapes. As the title states, the piece is a remediation of the musical term ‘symphony,’ which refers to a piece with three or more movements for a symphonic orchestra. The title may also refer to a cohesive piece with many distinct elements or instruments that function together harmoniously as in an orchestra.  The word symphony also refers to a creative production that requires human performance for its articulation. 

What this early work helps us understand about new audio-visual practices is that when put together, the visuals and the sounds are extensions or amplifications of each other, like instruments in an orchestra or dancers on a stage. This work predates recent research that explores the plasticity, activation and interaction of our various sense modalities for the processing of experience. For example, the moving image helps us “feel” the music. The mixing of sensorial forms and synesthetic metaphors becomes a complex, pleasurable tableau of forms that play between the areas of sensation-perception, physiology-body, emotion and memory. Larger structures are built up from simple components, which work together through varying intensities and mimic or require human performance.

For work that revolves around a particular sense, that sense may not even need to be present to articulate itself clearly. Sound is not always required for instance, even if implied in the creation of work. Jules Engel, an experimental animator, speaks of his work,

Conductors, composers and musicians have described my work as musical through the composition, timing and direction that they sense. They are moved by the rhythm and by the ‘complete, fulfilling process’. This is so interesting to me, as I do not rely on music as a starting point. Since 1969, I have animated more than thirty abstract films, adding the scores to my films at their completion; I prefer to do the graphic choreography from my own sense of timing instead of a predetermined sound or musical score. In my films, ‘sound score’ is often far more appropriate, since a formal musical composition in not always necessary to provide enhancement, nor is it the basis of stimulus. (Engel, Tobey Moss Gallery, 2008)

Pattern and repetition are two important elements in the remediation of music in experimental film and animation and both are subcomponents of rhythm. Pattern in visual form can be created by repeating information in the space of the frame and over time. Pattern in sound, described as meter in music, is created by accenting certain beats of the pulse. For example, accenting every first beat in a division of three beats is a waltz. Patterns are very expressive forms in many cultures as well, seen for example in Navajo rugs or African textiles, however these cultures do not apprehend these as abstractions, but as culturally expressive, iconic forms that have deep meaning. Pattern in Western culture seems to come in and out of vogue. In interactive media, for instance, the simple, clean look of vector graphics seems to have created a surge in abstraction or simple, non-representational pattern in design. There is also an influence as from pattern and abstraction in early video gaming systems and 8-bit computer graphics, where content was created from simple color grids of pixels, due to limitations with the technology (Wolf, 2003).

Rhythm, movement, and the subcomponents pattern and repetition are carried throughout human experience, society and culture. Early film/animation experiments segue into VJ and live A/V practices, such as complex, symmetrical moving images, which often incorporate kaleidoscopic repetitive patterns, strobing content, as well as visual, pictorial and editorial rhythm. These are added to a time-based practice that communicate expressive forms in performance, both with scored, semi-scored or improvised work. This also includes work that relates to the body of the performer, the performance environment or space, as well as an interaction between collaborators and participants.

Differences exist between the senses and it may very well be that these differences are as important in these creative works as the convergences, synergies or commonalities between the senses. For example, we cannot separate different wavelengths of light when mixed, unlike our ability to separate out frequencies in hearing, such as following the different melodic lines of instruments. Research has also shown how one sense modality can take precedence and even influence another sense, such as the McGurk effect (McGurk and McDonald, 1976), where seeing the shape of the mouth influences the perception of speech sounds. One could theorize that such differences may be important to understanding such convergent forms in new media, as well as how we gain more information about the world through comparing information from multiple senses.

“Saturated Sweets and Crunchy Beats”

The dream of creating a visual music comparable to auditory music found its fulfillment in animated abstract films by artists such as Oskar Fischinger, Len Lye and Norman McLaren; but long before them, many people built instruments, usually called ‘color organs,’ that would display modulated colored light in some kind of fluid fashion comparable to music, (Moritz, 1997).

Musicians, composers, experimental animators and filmmakers have been encouraged to negotiate with the abstract, time-based process of sound and light.  Sound can easily become abstract, what semiotics attempts to describe as “floating signifiers.” VJ practices, as well as music, eating a great meal or forms of amusement, actions, sensations, etc. are experiential in that they relate directly to sensation and perception in the relationships and differences between the senses over time. These practices are much more than representational and require further orders of cognitive processing related to perception, the body, time and the context or setting. They perform meaning in a manner outside of a linguistic sense in their experience. They not only transfer information, but also elicit and inspire a praxis of pleasure (and sometimes pain) in the direct experience of forms, as well as socio-cultural and historical negotiations. Unlike wholly representational forms of work, these process-practices require momentary experience, improvisation or play.  How does one experience riding a roller coaster through representational forms, without ever riding one?  How does one articulate the experience, without creating, as Bakhtin notes, two worlds, “the world of culture and the world of life, the only world in which we create, cognize, contemplate, live our lives and die or- the world in which the acts of our activity are objectified and the world in which these acts actually proceed and are actually accomplished once and only once.” (M.M. Bakhtin, 1993). 

Live A/V practices, like games and other interactive media, require experience and participation. Artists understand this process, especially those who are influenced by the performing arts. The experimental animator Jules Engel, for example, states, “My work is abstract, but it contains an organic element that brings people close to their inner feelings.  It doesn’t ‘explain’; within feeling, one can discover answers,” as well as, “movement is the content. Don’t merely look at a movement, FEEL it." (Engel, Tobey Moss Gallery, 2008).  Here Engel highlights the importance of the experience of the work using a cross-sensory means of perception. He describes how we can actually feel the movement we are seeing in our bodies. Perhaps this freedom of expression and interpretation in abstraction is what draws people into creating this kind of work, as much as it is the sensual interplay of qualities of vision, hearing and bodily movement. We should also note that there is something fundamental in sense experience that is concrete he is expressing.  We can see sounds, something those who created visual music understood. If we think beyond the hegemonic construct of narrative, realism and representation, it becomes clear that what we experience with the mixing of sense forms is quite concrete and as Jules Engels notes, even visceral.

Composers have long been partial to the abstract, constructed and subjective experience in their creative work. Musical instruments produce specific sounds that offer little in signification outside their potential reference to the register of the human voice or the gestural/bodily qualities of playing and their interface with the body. Such instruments “sing,” but they are also generators of abstract timbres or tonalities that the listener construes as pleasurable. Timbre is what makes the same note across different instruments sound different, just as the configuration of the vocal chords, register and structural cavities of the head make each person’s voice different. Music has traditionally been much more abstract or nonrepresentational than the traditional visual arts, such as sculpture and painting, for example, and it is difficult to think of a historical equivalent to these in music, aside from choral work, where you actually see and hear human bodies in space. Extremes of form have never been a problem for artists who come from a musical background. Forms that are “traditional” in music are still abstract and non-representational; we have to be told what they represent. 

Much of the history of musical expression has also included an exploration of an emancipation of sound from traditional practices. Composers like Edgar Varese experimented with noisy and percussive timbres, as did John Cage, as well as with the acousmatic or recorded samples form real world sounds used by Pierre Schaeffer. Interestingly enough, Schaeffer helped create the compositional form of “musique concrete,” which like the work of Varese, Cage and others challenged traditional notions of music, such as melody, harmony, rhythm and meter. Composers were the first to embrace new technologies of electronics and the computer, opportunities that came later and were slowly absorbed in visual work, partially due to the intense computational resources required for computer graphics. Electronic and computer music gave composers the freedom to expand their sonic palette and the creative potential for creating and programming sound. They became focused on the timbral qualities of sound. Perhaps these freedoms and associations of abstraction in sound have a relationship to simple forms like curves, stripes, etc., that seemed to suggest a symphony to Viking Eggeling in “Symphonie Diagnale,” or the visual music of artists like Mary Ellen Bute.  The production of sound, outside of narration, has a more difficult time with concrete figures and often only evokes character and story, unlike visual representation, which has a difficult history of trying to represent or capture the world as opposed to only experiencing the work through process and performance. 

What is retained from this history in VJ culture is the experience of the moving, performed and improvised image in all its abstract magnitude, even as it calls on viewers or participants to mix the concreteness of their seeing, hearing, feeling of movement, etc. Both audio and visual abstraction has suffered from criticism and this misunderstanding. The image’s movement- the rhythms, patterns, repetitions and the negotiation of both experience and meaning in the frame are also often negatively described as “decorative,” “entertaining,” “formal,” or “indulgent”. VJing has often been derided as “eye candy”. However, such A/V practices retain a liminal negotiation of sound and image, which is relevant to the human condition. 

The term “eye candy” refers to a simulated or multimodal synesthetic-like mixing of the saturation of color and pattern and the intensity of sweetness of candy and the kinds of people who would indulge in bright colors and “sweets”. Some of the earliest experiments converging the audio-visual were by people who have been described as syneasthetes, those who have involuntary mixing of the senses, such as the composers Franz Liszt, Michael Torke or Duke Ellington, to name a few, who saw sound as color. The physicist Richard Feynman saw letters and numbers and color, “When I see equations, I see the letters in colors – I don't know why. As I'm talking, I see vague pictures of Bessel functions from Jahnke and Emde's book, with light-tan j's, slightly violet-bluish n's, and dark brown x's flying around. And I wonder what the hell it must look like to the students," (Feynman, 1988).  However, all humans have a certain level of plasticity and interaction between the senses, (Stein and Meredith, 1993). Interactive media and technologies offer increasing opportunities for different sensorial forms to be converged in real-time, as well as performance practices that engage in the mixing of the senses, as well as bringing media work out into the environment and engage with the body. What could be more concrete than that?

Intensity in Audio-Visual Structure

The power to pull you in, hold your attention, make you “feel”, and create and build dramatic structure?  Intensity. Intensity is directly related to the level of energy expressed or modeled through a system. For example, we can think of the amount of energy required to produce different volumes from a stereo speaker, the energy consumed by a light or the amount of energy required with different kinds of physiological activity, such as sleeping or dancing.  Varying intensities create dramatic shape and are cognitive attractors. In vision, psychological studies show that it is not color, but contrasts in light intensities that are important in the perception of movement, depth, perspective, the relative movement of objects, shading and gradations of texture. Intensities in audio-visual and movement based practices can create dramatic tension and hold that attention though varying contrasts or through the building of intensity. Examples of this include speed and movement, changes in direction or quality movement, shapes and changes of shape, hue, brightness and saturation, timbre and tonal qualities, rhythm and meter and other compositional devices. 

VJing performances share a potential for dramatic structure. Dramatic structure in narrative is forged along plotlines of conflict, rising action and resolution that build intensity to a climax. Drama is created through intensity. The simplest description of this dramatic shape relies on a major climax and resolution. In sound design, the articulation of time in sound is called the “envelope.” The envelope is comprised of the attack, sustain, decay and release. Envelopes are familiar means for shaping and manipulating sounds in samplers and software over time.   The envelope is also used to create different timbres of or tonalities of sound, for example, a percussive attack of a piano versus a smooth attack of a flute initiating a note. In terms of perception, some of the neurons involved in hearing are only sensitive to the beginning of a sound and others to the end of a sound. This means that we process sound in terms of temporal shape or envelope, which influences the perception of timbre. Other general methods of producing dramatic intensity in music are frequency or pitch, sound intensity or amplitude, space and timbre. VJ practices should take into consideration these varying intensities and VJ artists should not be afraid of using the empty frame, just as a composer uses silence or “rests” in notated music.

Intensity is an important element of the art of movement or similar expressive forms. To understand that intensity does not require conventions of story is important to creating and exploring new practices that converge audio-visual and bodily-physical and socially interactive work. Understanding how intensity works is also important in terms of shaping longer expressive forms. This is especially true in terms of musical forms, which contain varying intensities and contrasts delineated often by a number of sections. Using intensity expressively in multimedia means that we understand how to express intensity in sound, with light, in the moving images and with body-based movement, as well as how these sense forms can by mixed and shaped in order to form more expressive, engaging work.

Playing the Audio-Visual: Interactive Media as Process and Performance

Recently, with the advent of computer technology and multimedia software, have we gained a new widespread ability to play or perform the image, something composer-performers have had a longer history of in the practice improvisation. Qualitative differences in live creative works are their improvised, generative, played and performed processes. These works often have pre-composed elements, which are mixed with work that is improvised or generated from the context of the performance, such as using and manipulating software or code or using a live video feed, etc. Scores can also be created in order to create a basis for the process of performance. These instructions can form an algorithmic basis or recipe-like scaffolding for what is to occur in a performance. Often, these scores leave open the opportunity for improvisation or choices made in the moment for what to do and include the possibility of the unknown. These improvised practices are playful in the sense that they engage with the moment in an experimental, fun and even humorous way, in their balance between intent and accident. Often a performer will intentionally use material from errors, accidents or problems in a performance. This is especially prevalent in noisy practices. The “glitch” or “glitching” is a descriptive term that applies to digital errors, as well as a kind of stylistic of work. Notions of glitching point to experimentation and play, as well as the importance of accidents in digital practices.  Some process-based artists leave aspects of the work unfinished, or continue to rework the piece over time.  Practices that include improvisation, intention and accident or unfinished elements underscore the importance of process and play in these media arts.

VJ performances are also embodied, in other words, they include interfaces and controllers that require the movement and gesture of the performer or participant for control, as well as the use of expressive movements of the body in space that are translated into audio-visual material. They can also be participatory, co-authored and collaborative, which brings another dynamic into improvised work. The unexpected outcomes from the system, as well as the process are layered against the unknown complexities of socially improvised work.  Improvisation makes use of momentary processes that engage liminal experiences with others or within groups of people. The practice of a VJ is meant to initiate the participation or interaction of other performers, as well as engaging the audience, material and system. The VJ is bounded in a kind of flow of his or her perception, audience reception and the interaction of the content and system over time. The VJ will often accompany the audio work of a musician, DJ, composer or sound artist. These real-time group improvisations create a sense challenge, even of danger and excitement for the performers, as they are required to respond to each other in the moment and embody various political forms and practices. Forms ebb and flow in the interplay of the audio-visual-performed and improvised in the momentary negotiations of the participants and audience using pre-composed, scored and free improvisational practices. These practices are embodied. They rely on the physical attributes of the human body and how these performers interface with the work, their bodies, the space and other participants.

Real time and Representation

I do not look for any kind of narrative that would lend itself to graphic expression. I must convey ideas/feelings through movements that could not be put into words. Lines, Squares, Spots, Circles, varieties of Color -- sometimes difficult to comprehend -- provide the keys to our pictures. (Engel, Tobey Moss Gallery, 2008).

Scholars of new media bring theories of narrative into the realm of the digital, but narrative can only account for a portion of what occurs in the momentary process of digital work like VJing and other live A/V practices. These moment-to-moment body-sense experiences are overshadowed by theories of narrative, such as canonical ideas of story, where characters traverse a developmental arc from beginning to climax, and finally to resolution. One can argue for using notions of narrative in order to conceptualize the complex process for how the brain processes and remembers information, however this limited perspective of narrative denies liminal acts such as performances, playful practices, improvisation and participation, as well as communicative and generative processes in the exchanges of those creating live A/V work, such as artists and VJs. Narrative does not factor in other means of constructing meaning, based on the experiential. Situated cognition is one theory that helps to explain time-pressured, real-time choices, as well as the importance of actions and context in these creative practices. How we offload cognitive activity onto our environment using tools and techniques, as well as movement and space in such performed occurrences seems almost more vital than how meaning is constructed or expressed in retrospect in this work. This seems even more evident when we expand creative practices into socially performed and improvised realms, where participants play and engage in momentary creative acts. Meaning is leveled in the experience, whereas narrative begs for some overarching, connective construction of knowledge formed in retrospect.

As the momentary performance and exploratory process in socially interactive media continues to build cultural pervasiveness, new theories will be needed to differentiate fundamental structural means in media work that move beyond limitations of older methods of representation, such as narrative. The structural mythos of narrative does not articulate the improvised, performed and shared momentary acts of individuals engaged in creating meaning in the digital medium within a specific context, space and with body-physiological engagement. The practice of a VJ, an interactive website experience, dance improvisation, the improvised actions of a digital media performer, someone involved in game play, a participant in a multi-user virtual world, and even amusement park rides, meditation, sports, even car crashes and other traumatic events, as well as communicative exchanges in a social networks are formed in a time-pressure moment, comprised of systems of continuous sensorial inputs, psycho-physiological processing, actions and motorial tasks and bodily experiences and reactions. Narrative is only a sampling construction of meaning in the analog complexity of the moment. This is a moment that narrative desires to fix or document in its framework of linear continuity. Narrative desires to account or recount as means of structuring, in the reflexive, retelling and representation of experience, but narrative is only part of the experience, and some would argue it is something altogether different. The reciprocal, dialogic and dynamic exchange process in interactive media is a moment-to-moment sense experience, an embodied act and an engaged and interpenetrating process that implies notions from cybernetics and the biological, in the feedback system of communication and control.

Interactive media has a history and functionality built up from momentary processes related to systemic bodily processes- a continuous measurement, comparison and exploration that provides for a certain kind of pleasure and meaning. Interactive media, such as live A/V practices, offer affordances for what can potentially occur in any given moment. These real-time, immediate and dynamic processes and exchanges form intensities described in the tradition of the dramatic structure, where characters are developed over an arc, and where actions and plotlines coalesce into a climax, however the protagonist is often the user or participant, who controls the outcome. However, intensities and exposition don’t always work hand-in-hand. This has been a problem for game designers, where storytelling slows down or completely stops game play. There is a compromise, a kind of “explorytelling” or “storyplay,” which seeks to weld a mixture of play, exploration, game-play and story-like structures of beginning, middle and ending, as well with dramatic shape. This seems to be the kind of practice developing in virtual worlds, but also arises in other practices of new media work that offer meta design contexts where participants are also designers of the system. Here we often have the dissolution of the author and spectator, as well as dissolution of narrative and the experiential. Perhaps it is these moments that are truly free, where bodies and minds combine in the flow-like process of experience, the improvised and played.

Narrative is considered the retelling or recounting of (past) events (Genette, 1980), yet much of live A/V work is based on real-time processes. The practice of interactive media, as argued above, is not purely representational; it is experiential, participatory and generative.  The narrative or the retelling of the experience of interactive media is what happens in the reflection or the sharing of the experience, like a play-by-play. Live A/V practices are based upon real-time interactions and performances, and only partially on the kind of representation structured in the form of fixed sequences of signs, as in traditional forms of narrative. The immediate, participatory and improvised experience of the players in a VJ environment and the construed events: images and sounds, movement, patterns and rhythms, be they programmed or generative, form a problem for thinking of live A/V work in narrative terms.  This is because it is not necessarily the narrative quality, the retelling or the “re-presentation” of information that matters, but what the person does with the content, the performance, their play, their experience and their agency- the relations and behaviors which occur randomly and/or generatively as the work is experienced and produced. What a DJ or musician performs and what a video artist produces, plays and manipulates in a project may arise from intent, but the signification of the audio and visual material together creates accidents and exciting dangers or challenges, which spin forth in the process of play and improvisation.

Narrative has been critiqued in interactive media, such as gaming, as only a portion of what is occurring and that interactive media extends into simulation, “to simulate is to model (a source) system through a different system which maintains (for somebody) some of the behaviors of the original system,” (Frasca, 2003). A simulation is a conceptual mechanism or a real-time machine with some algorithmic form or intelligence. The VJ process is closer to a situated activity, like improvised play and experimentational process, which often relies on simulation. As Frasca defines simulation above, narrative is a limited means for thinking about work that relies on a system that models and interfaces with human experience.

Why not take other literary forms as a starting point for a discussion of such practices, such as VJing?  A discussion of poetry would be as apt a means as notions of conventional narrative and dramatic structure for discussing VJing. Poetry extends practices beyond Aristotelian notions of story into convergences of sonic, semantic and performance practices in a very playful manner. Poetry also has an extended history in relation to other forms of creative production, such as music and other performing arts. Rhythm exists in poetry in both rhyme (assonance), as well as the accenting of syllables, also called meter and in sound quality, via the assonance and alliteration of speech sounds. Layers are composed via metaphor and simile, creating different and complex interpretations. Overall characteristics, such as euphony, defined as the general flow and sound of speech, can be created by both repetition, as well as the pitched and percussive qualities of vowels and consonants and the repeating of lines and phrase. Why not consider new media practices in terms of poetry, rather than narrative?

End Note

There is a history of the art of movement that forms relational streams and tributaries into the practice of live audio-visual-performed work, such as VJing. This history of experimental film and animation shares and overlaps with sonic practices in levels of abstraction in elements of form like rhythm and movement, the lack of conventional narrative, dramatic intensity, as well as improvisational, time-based and performance practices. Practices, which are shaped by momentary actions and reactions in space or context and time, as well as new means of social interaction and participation. These practices continue to evolve in the live audio-visual work of VJs, as well as in the work of contemporary musicians, composers, artists, designers and performers. Technology and digital practices have fed the flood of audio-visual and movement based work, from early filmic and experimental animation practices. The prevalence of technology and media now enables the live performance and interaction of sound and image in ways that were not possible fifteen years ago. We will continue to see art and media created and influenced by such practices as VJing continue to expand in the areas of interaction and the participation of the subject, as well as work and systems with more complex and meaningful interfacing with the body.

Bibliography

M.M. Bakhtin, Toward A Philosophy of the Act (Austin: University of Texas Press, 1993).

Bruce Block, The Visual Story:  Seeing the Structure of Film, TV, and New Media (Boston:  Focal Press, 2001).

R. Feynman, What Do You Care What Other People Think? (New York: Norton, 1988).

Gonzalo Frasca, J.P. Wolf and Bernard Perron, Editors, The Video Game Theory Reader (New York and London:  Routledge, 2003).

Gerard Genette, Narrative Discourse (New York:  Cornell University Press, 1980).

J. Huizinga, Homo Ludens, (Beacon Press, Boston, MA, 1950).

V. Jousmaki and R. Hari, Parchment-Skin Illusion:  Sound-Biased Touch, Current Biology, Vol. 8, No. 6.

H. McGurk and J. MacDonald, Hearing Lips and Seeing Voices, Science, 1976 pg. 264: 746-289.

William Moritz, Animation World Magazine, Issue 1.2, 1996.

William Moritz, Animation World Magazine, Issue 2.1, 1997.

Len Lye, http://www.iotacenter.org/visualmusic, [accessed November 2008].

Jules Engel, Tobey Moss Gallery, http://www.tobeycmossgallery.com/still.html, [accessed November, 2008].

J.P. Wolf, J.P. Wolf and Bernard Perron, Editors, The Video Game Theory Reader (New York and London:  Routledge, 2003).

B. Stein and M Meredith, The Merging of the Senses, MIT Press, Cambridge, 1993.