Non-rigid registration-based data-driven 3D facial action unit detection

Abstract

Automated analysis of facial expressions has been an active area of study due to its potential applications not only for intelligent human-computer interfaces but also for human facial behavior research. To advance automatic expression analysis, this thesis proposes and empirically proves two hypotheses: (i) 3D face data is a better data modality than conventional 2D camera images, not only for being much less disturbed by illumination and head pose e ects but also for capturing true facial surface information. (ii) It is possible to perform detailed face registration without resorting to any face modeling. This means that data-driven methods in automatic expression analysis can compensate for the confounding effects like pose and physiognomy differences, and can process facial features more effectively, without ering the drawbacks of modeldriven analysis. Our study is based upon Facial Action Coding System (FACS) as this paradigm is widely accepted to be capable of describing practically all types of human facial expressions and enables their systematic evaluations. Coding with FACS is done with Action Units (AUs) that are closely related with muscular activations. To validate the first hypothesis we develop person-independent detectors and intensity estimators of AUs, which use 2D maps of 3D facial surfaces. This approach enables us to compare 2D luminance modality with the 3D surface geometry data modality under the same set of algorithms. In addition, our detectors and estimators are free from biases of model-driven techniques to guarantee a fair assessment of the two modalities. For the second hypothesis, we first investigate non-rigid registration on 2D facial surface curvature maps. Our non-rigid registration algorithm is capable of handling large deformations and yet it is computationally efficient. To realize our second hypothesis we explore and develop AU detectors using this algorithm. Our work is the first example of detailed registration in data-driven expression analysis and surpasses the performance of state-of-the-art AU detectors.