2016-076 – ZFace (3D Facial Landmark Tracking SDK)

Face alignment is the problem of automatically locating detailed facial landmarks across different subjects, illuminations, and viewpoints. Previous methods can be divided into two broad categories. 2D-based methods locate a relatively small number of 2D fiducial points in real time while 3D-based methods fit a high-resolution 3D model offline at a much higher computational cost. Our approach exploits dense 3D cascade regression, where the facial landmarks are consistent across all poses. In our system the face is annotated completely in 3D by selecting a dense set of 3D points (shape). Binary feature descriptors (appearance) associated with a sparse subset of the landmarks are used to regress projections of 3D points. The method first estimates the location of a dense set of markers and their visibility, then reconstructs face shapes by fitting a part-based 3D model. The method was made possible in part by training on the BU-4D Facial Expression and BP-4D-Spontaneous datasets that contain over 300,000 high-resolution 3D face scans. Because the algorithm makes no assumptions about illumination or surface properties, it can be applied to a wide range of imaging conditions. The method was validated in a series of tests. We found that 3D registration from 2D video effectively handles previously unseen faces with a variety of poses and illuminations. Our implementation runs at 50 fps using a single core of an i7 processor, enabling real-time performance.The software provides facial expression transfer, that enables real-time and person independent avatar animation. Video: https://www.youtube.com/watch?v=Xb61CmsPq_IPaper: http://www.laszlojeni.com/pub/articles/Jeni15FG_ZFace.pdf Scott McEvoy smcevoy@andrew.cmu.edu 412-268-6053