FaceTracker is a C/C++ API for real time generic non-rigid face alignment and tracking.


Goal:

Non-rigid face alignment and tracking is a common problem in computer vision. It is the front-end to many algorithms that require registration, for example face and expression recognition. However, those working on algorithms for these higher level tasks are often unfamiliar with the tools and peculiarities regarding non-rigid registration (i.e. pure machine learning scientists, psychologists, etc.). Even those directly working on face alignment and tracking often find implementing an algorithm from published work to be a daunting task, not least because baseline code against which performance claims can be assessed does not exist. As such, the goal of FaceTracker is to provide source code and pre-trained models that can be used out-of-the-box, for the dual purpose of:

  1. 1. Promoting the advancement of higher level inference algorithms that require registration.

  2. 2. Providing baseline code to promote quantitative improvements in face registration.


Features:

  1. • Real time: ranging from 20-30 fps (depending on processor, compiler and use of OpenMP)

  2. • Generic: designed to work for most people under most conditions

  3. • No training required: a pre-trained model is provided

  4. • Detection based initialisation: no user intervention required

  5. • Automatic failure detection: requires no user re-initialisation

  6. • Camera or video input


Download:

FaceTracker is available for download (for research purposes only). The library includes the C/C++ API, example code for interfacing with the API, a pre-trained model and documentation. To download it, please send an email to Jason Saragih (Jason.saragih@csiro.au) from your official university/work email address.


The Tracker:

The code requires OpenCV 2.x and the provided model was trained using the MultiPIE database. The tracker is based on a modified version of the constrained local model described in:

J. Saragih, S. Lucey and J. Cohn, ``Deformable Model Fitting by Regularized Landmark Mean-Shifts", International Journal of Computer Vision (IJCV), 2010


People Using FaceTracker:

Some people are using FaceTracker to do some really cool stuff:

  1. • FaceOSC

  2. • Perfect Creatures

  3. • Face Cloning

  4. • A music Video!

  5. • Face Projection

  6. • Being John Malkovich

  7. • pkmFace


Videos:

Below are some examples of difficult YouTube videos that were processed using FaceTracker with the pre-trained model. There was absolutely no user intervention in any of the videos.

Unmodeled Expressions


The subject in this video exhibits expressions not present in the training set. As a result, the shape prior (i.e. a linear point distribution model) lacks the capacity to accurately track some of the expressions. Nonetheless, reasonable estimates are made for these expressions and total failure is avoided.

Unusual Appearance


The subject has dark skin (a notoriously difficult case) and significant facial hair as well as glasses. Significant reflections of the glasses can be observed throughout the sequence.

Face Paint and Extreme Pose


The subject has paint on parts that are unnatural. Specifically, the tear like black paint under the eyes is a source of confusion for the tracker and poor tracking around the eyes can be observed in parts of the video. Significant pose is also exhibited in the video, with facial pose changes ranging from left to right profile.

Unmodeled Shape


The training set for the tracker consists entirely of adults. Since the facial shape of babies differ significantly from adults, the tracker fails to accurately track parts of the subjects face. Furthermore, the appearance of facial features on the subject are poorly pronounced, such as the eyebrows.