MusicFace Working Page (in progress)

MusicFace is a program (in development) written in Python that analyses multi-tracked music (in midi format) using a palette of rule-based "face choreography" creative commands ( i.e. blink on beat, raise eyebrows mapped to pitch, get funky on this passage) and creates a face animation synced and generated by the music. Basically dancing with faces.

Early Experiments: (still jerky and not life-like or flowing enough): music and edit by

         realistic test:          quicktime          (4mb .mov file)
         artistic test:            quicktime          (4mb .mov file)

This first piece was created for, and shown at, the New Forms Festival '03 in Vancouver.
     concept, programming & animation by Steve DiPaola
     original score and special edit by Scot Gresham-Lancaster

More process details:

The program takes in midi of the music and outputs synced 3ds max script that animates very realistic face animation. To "face choreograph", I listened to Scott's piece via a midi editor where I could put in range points ( punch in/out) in the midi to section up the music into conceptual groupings. I could then assign from my growing palette of rules-based commands, multiple gestures and emotional behaviors (gestural painting if you like) to these sections of the music. Once I setup up the midi with sections and created a rule mapping, I ran the python program which first read-in the sections / notes / pitch ... all in standard midi and processes it for conceptual re-mapping. Letting the musical notes of the song create the face movements via the creative behaviors rules). . So if I cut up a simple song into the following conceptual groupings:

|  .. .. . ..  | . .. .  |  .. . . ... |  . . ... |             - the music
     A.A      A.B      B.A    B.B               - my sub regions I map
             A         |            B

Now I can script:

For: A
     Slow to happy
     Eyebrow raise on pitch info
     Rotate face on beat 5 degrees in x
For: B
     Slow to sad
     Eyebrow raise on pitch info
     Rotate face on beat 5 degrees in x
For: A.A
     Blink on first note
For: A.B
     Blink on all notes
For: B.A
     Blink on first note
For: B.B
     Blink on all notes


This would give me an animated face getting happier over the long A section, blinking on the first note of section A.A and B.A and for the later sections on all notes for emphasis. While the head rocks up and down to the beat and the eyebrows move up and down in unison to the pitch of the notes. See the test animation of this here. (400K .mov file)

Now, I'm writing more sophisticated rules both on the musical side and on the face side. Some rules are structural (i.e. blink on beat) and others are more emotional (do it in a crazed but heroic way here).

I can eventually re-map to other time based output other than faces, for instance I could animate a conceptual painting being painted by the music - starting with a blank canvas, the music and rules create paint strokes on it, when the music ends - a painting is born from it. How do you re map one form of art to another? More ramblings on that later ...

------------------------------------------------------------------------------------

Here is the original New Forms Festival proposal that might explain more:

------------------------------------------------------------------------------------

New Forms Festival Proposal (except)


Concerto for Virtual Strings and Faces ( working title)
- a presentation and screening

By Steve DiPaola & Scot Gresham-Lancaster

Overview:
This is a collaborative piece between an electronic composer and an interactive 3D artist where an original electronic music composition will drive 3d virtual emotive faces in a collaborative remapping of the emotional and gestural state of both the music and the lifelike facial animation. It is based on a system which explores “face space” which has been used in several multidisciplinary collaborations (i.e. art, internet communication, electronic games, social science) but this New Forms proposal will mark the first collaborative experiment in the relationship of “choreography in human emotion face space” via artistically remapping/interpreting/converting the emotive stream of the musical instruments line of an original composition.

Background and Motivation:
For many years, Steve DiPaola has been creating and using a visual development system for exploring face space, both in terms of facial types and animated emotional expressions. See the following paper. Within this multi-dimensional space of faces, it would be possible to traverse a path from any face to any other face, morphing through faces along that path. Predecessors of this system have been used (via commissioned work) for best selling electronic game “The Sims” and in a collaboration with the video artist Nam June Paik and in music videos for Kraftwerk. This system has been extended into emotion and gesture space making it possible to build a space language of facial emotion and behavior.
The system allows us to traverse morphing face paths through face space, (both facial types and emotion types) with the ability to affect the trajectory of path (straight, curved) and the key points (faces) to target along the way. Music can be described as paths through tonal space with significant cultural biases described by certain paths or key points (i.e. notes, intervals, scales) determining whether certain motifs sound appealing. Can we similarly write a composition in face space? Imagine a representation of two intertwining paths through face space that use their relationships, both harmonic and discordant, to create a composition. Can we dance with faces, both moving through facial types as well as ebb and flowing through emotional gestures?

It is this idea – dancing or composing in face space that is the spark for this collaboration. We will be both be exploring our artistic domains (Scot’s electronic music composition and Steve’s interactive face-based emotion composition) to combine them into one synergistic movement, much as a dance choreographer and a composer would do. Except with our interactive knowledge based tools, we can choreograph an “living” emotive faces, creating a process that interprets and translates the emotion state of the composed music, where each instrument would drive an aspect of emotive faces.

Steve's CV: http://dipaola.org/berkeley/res_dipaola.pdf
website: http://www.dipaola.org

Scott'sCV: http://isis.csuhayward.edu/dbsw/music/sglbio.html
website: http://www.o-art.org/Scot/

Sims work link : http://www.dipaola.org/steve/facelift.htm


Contact Info: Steve DiPaola – www.dipaola.org
steve@dipaola.org
sdipaola@sfu.ca
604.268.7479 work