Published on November 23, 2007
Slide1: Expressive gesture in interaction: the role of movement and gesture in emotion Ginevra Castellano, Antonio Camurri, Gualtiero Volpe WP6 HUMAINE Workshop, Paris, March, 10-11, 2005 Infomus Lab http://infomus.dist.unige.it Department of computer science, systems and telematic, University of Genoa What is expressive gesture : What is expressive gesture Gesture: support to verbal communication, movement of the body containing information Expressive gesture: high-level non-verbal expressive and emotional communication (Camurri et al., 2004) Artistic context: gesture as conveyor of information related to the emotional domain (dance or music performances) Expressive gesture in HCI : Expressive gesture in HCI Aims: to communicate emotions to users to recognize users’ emotional engagement Expressive gesture: movement as conveyor of emotional information component of an emotional process Expressive gesture analysis: a layered approach (1) (Camurri et al., 2004, 2005): Expressive gesture analysis: a layered approach (1) (Camurri et al., 2004, 2005) Modelling techniques: prediction of emotions (e.g., multiple regression, neural networks, decision trees) Techniques for gesture segmentation, representation of gestures as trajectories in virtual, expressive spaces Video and audio processing techniques (computer vision techniques on the incoming images, signal processing on audio signals) Physical signals, video and audio pre-processing techniques (e.g. motion detection, audio filtering, etc.) ↑ ↑ ↑ Expressive gesture analysis: a layered approach (2): Expressive gesture analysis: a layered approach (2) Not only analysis but also synthesis of expressive gesture Experiments on expressive gesture are carried out with Eyesweb open software platform (Camurri et al., 2000) Perspective: mapping info about users’ behaviour onto real time generation of expressive behaviour of virtual agents such ECAs Our activity in HUMAINE: Our activity in HUMAINE Expressive gesture as a component of an emotional process Component-process model of emotion provided by Klaus Scherer (GERG) has been investigated (Scherer, 1984, 2000; Scherer and Zentner, 2001) We used motor activation component to evaluate the emotional engagement of users exposed to emotional stimuli Music, emotion and movement: Music, emotion and movement Research in collaboration with Professor Klaus Scherer’s group (GERG, Geneva Emotion Research Group) Aim: to investigate the relationship between emotions induced by musical stimuli and movement Pilot experiment: are there correlations between the emotional characterizations of music excerpts and human movement ? Continuous measures of emotions: Continuous measures of emotions Music as induction technique Music and emotion: time-varying relationship Several indicators Problem: conscious Vs unconscious measurements Verbal report Physiological measures Coding of non-verbal behavior, subject interfaces: from sliders to multimodal interfaces Idea: laser pointer as semi-conscious interface through which movement to communicate an emotional experience generated by music A pilot experiment (1): A pilot experiment (1) 20 subjects equipped with a laser pointer and asked to move it on a white wall in front of them while listening to music excerpts Stimuli: a set of classical music excerpts provided by GERG Grouped in four characterizations defined on the basis of valence and energy: slow positive, slow negative, fast positive and fast negative A pilot experiment (2): A pilot experiment (2) Method Each subject listened to four music excerpts, one for each emotional characterization Trajectories performed by the subjects moving the laser pointer on the wall have been recorded Questionnaire to indicate emotions felt by subjects during listening to music Preliminary analysis: Preliminary analysis Aim: to look for correlations among features of the trajectories performed by subjects with the laser pointer and emotional characterization of the music excerpt a subject was listening to Global and static analysis: integration of the laser trajectories over time To obtain for each video file with the movement of the laser a bitmap summarizing the trajectory followed during the whole listening Each bitmap represents a graphical subject response (GSR) from the listening of a single music excerpt Is it possible to separate the GSRs in classes and to verify if these classes can be correlated with the characterization of the music excerpts? CLUSTERING ANALYSIS Extraction of global trajectories: Extraction of global trajectories Patch summarizing the path followed by the laser pointer (Eyesweb platform) ..\Presentazione\Presentazione.eyw Identification and measure of trajectories features: Identification and measure of trajectories features To identify a collection of descriptors Related to specific features of the trajectory patterns Angularity, rarefaction, spatial occupation, vertical symmetry, horizontal symmetry, central symmetry, compactness, lateral location,vertical location, angular tendency, spatial extension Providing measures for relevant trajectory features Manual annotation Unambiguous criteria Patterns are evaluated with a value from 0 to 4 with respect to each specific feature Five evaluators An example: angularity: An example: angularity The trajectories drawn by the laser can be smooth (0) or angular (4) Smooth trajectory: wavy, soft lines Angular trajectory: direct, sharp, nervous lines Smooth trajectory Angular trajectory An example: rarefaction: An example: rarefaction The pattern can be thick and intense (0) or rarefied (4) White pixels / total pixels in the boundary rectangle Thick trajectory: high degree of filling of the occupied space Rarefied trajectory: low degree of filling of the occupied space Thick trajectory Rarefied trajectory Statistical analysis: mean of all the ratings of all the features for the four emotions : Statistical analysis: mean of all the ratings of all the features for the four emotions Useful for deciding how to realize a clustering analysis Hypotheses to be verified during the clustering analysis: Hypotheses to be verified during the clustering analysis Angularity, rarefaction and compactness seem to explain the motor activation analyzed with this static and global analysis: critical features Slow patterns: low angularity, high rarefaction, low compactness Fast patterns: high angularity, low rarefaction, high compactness Clustering global trajectories: Clustering global trajectories Eyesweb patch with a block implementing the K-Means algorithm Aim: to verify if the grouping create clusters that are consistent with the emotional characterizations of the music excerpts used to induce the emotions in the subjects Choice of the best type of clustering: two clusters, three features Two different classifications: fast/slow and positive/negative Results: Results Fast/slow patterns explained by angularity only Positive and the negative patterns don’t distinguish from each others Subjects, moving the laser pointer, synchronize with the rhythm of the excerpts If the velocity of the music increases, consequently the velocity of the arm movement increases as well as the direction changes frequency There could be a sort of correlation between characteristics of the music listened to and movement performed Resonance between music and motor activation Future developments : Future developments Dynamic analysis of laser pointer trajectories: how can be correlated with the musical structure at different time scales Aim: to discover how rules can be established to recognize emotions of users Possible perspective: to contribute to define the role of attention in emotion-oriented systems such ECAs Applications: Applications Motor rehabilitation Multimedia content analysis through novel affective interfaces (e.g. mobiles, embedded systems, new media) Music industry: music information retrieval from huge databases based on emotional responses Artistic and musical applications Cultural applications, museums, and science centers References: References Camurri A., Hashimoto, S., Ricchetti, M., Trocca, R., Suzuki, K., and Volpe, G., (2000), “Eyesweb – Toward Gesture and Affect Recognition in Interactive Dance and Music Systems”, Computer Music Journal, 24:1, pp. 57-69, MIT Press, Spring 2000. Camurri, A., Mazzarino, B., Ricchetti, M., Timmers, R., and Volpe, G., (2004), “Multimodal Analysis of Expressive Gesture in Music and Dance Performances”, in A.Camurri, G. Volpe, (Eds.), “Gesture-based Communication in Human-Computer Interaction”, LNAI 2915, Springer Verlag, 2004. Camurri, A., De Poli, G., Leman, M., and Volpe, G., (2005), “Communicating Expressiveness and Affect in Multimodal Interactive Systems”, IEEE MultiMedia, January-March 2005, pp.43-53. Scherer, K.R., (1984), “On the nature and function of emotion: a component process approach”, in K.R. Scherer & P. Ekman (Eds.), Approaches to emotion (pp.293-317). Hillsdale, NJ: Erlbaum. Scherer, K.R., (2000), “Emotions as episodes of subsystem synchronization driven by nonlinear appraisal processes”, in Lewis, M. & Granic, I. (Eds.) Emotion, Development, and Self-Organization (pp. 70-99). New York/Cambridge: Cambridge University Press. Scherer K.R., Zentner M.R., (2001), “Emotional effects of music: production rules”, In P.N. Juslin & J.A.Sloboda (Eds). Music and emotion: Theory and research (pp. 361-392). Oxford: Oxford University Press.