HD Face

Microsoft Kinect is an amazing device with state-of-the-art body tracking capabilities. However, the newest SDK version includes a touch of some great new magic: Face tracking.

Utilizing the infrared and color streams, Kinect sensor can accurately track thousands of facial points. Oddly, Microsoft has implemented two APIs for tracking a face: Face Basics and Face HD. The first one only provides limited capabilities in the 2D space. Face HD, though, includes a ton of hidden goodies for tracking a face in the 3D space.

{ Vitruvius is featured in the official Microsoft Kinect Website and Channel9 }

After reading this article, you’ll be able to understand how face tracking works, how you can access the points in the 3D space and how to display them in the 2D space.

Introducing the Face class

Even though the native Kinect API is powerful, it’s quite messy. Microsoft only exposes a huge array of face vertices, along with a big enumeration. So, while building Vitruvius, we re-imagined the whole face tracking experience, from a purely developer perspective.

Please, meet the all-new Face class. The Face class contains all of the information you’d ever wish to know. Using the Face class, accessing properties like the nose, eyes, jaw, forehead, chin, cheeks, or chin is now a matter of one line of code.

Virtuous extends the native APIs and exposes a single, unified, powerful interface.

Let me show you how.

Accessing an HD Face object using C#

Just like every Kinect stream, HD Face has its own frame type. To properly use a Face frame, you need to include the following namespace:

using LightBuzz.Vitruvius;

After that, you need to subscribe to the FrameArrived event, just like you’d do for the Color, Depth, Infrared, or Body streams. If you are using Unity, you do not need to subscribe to the event, but simply check the frame readers in your Update method.

Since you’ve done those trivial tasks, you can simply call the Face() extension method.

// Private members

private KinectSensor _sensor = null;
private BodyFrameSource _bodySource = null;
private BodyFrameReader _bodyReader = null;
private HighDefinitionFaceFrameSource _faceSource = null;
private HighDefinitionFaceFrameReader _faceReader = null;

// Initialization

_sensor = KinectSensor.GetDefault();

if (_sensor != null)
{
    _bodySource = _sensor.BodyFrameSource;
    _bodyReader = _bodySource.OpenReader();
    _bodyReader.FrameArrived += BodyReader_FrameArrived; ;

    _faceSource = new HighDefinitionFaceFrameSource(_sensor);
    _faceReader = _faceSource.OpenReader();
    _faceReader.FrameArrived += FaceReader_FrameArrived; ;

    _sensor.Open();
}

// Event handlers

private void BodyReader_FrameArrived(object sender,
BodyFrameArrivedEventArgs args)
{
    using (var frame = args.FrameReference.AcquireFrame())
    {
        if (frame != null)
        {
            Body body = frame.Bodies().Closest();

            if (!_faceSource.IsTrackingIdValid)
            {
                if (body != null)
                {
                    _faceSource.TrackingId = body.TrackingId;
                }
            }
        }
    }
}

private void FaceReader_FrameArrived(object sender,
HighDefinitionFaceFrameArrivedEventArgs args)
{
    using (var frame = args.FrameReference.AcquireFrame())
    {
        if (frame != null && frame.IsFaceTracked)
        {
            Face face = frame.Face();
        }
     }
}

Accessing the HD Face properties

Let’s now get to the funny part. Every facial characteristic is a property of the Face class. Every facial point is expressed as a CameraSpacePoint (X, Y, and Z values).

var eyeLeft = face.EyeLeft;
var eyeRight = face.EyeRight;
var cheekLeft = face.CheekLeft;
var cheekRight = face.CheekRight;
var nose = face.Nose;
var mouth = face.Mouth;
var chin = face.Chin;
var forehead = face.Forehead;

Insanely easy right? Here’s the result:

Even if you need to access the entire collection of vertices (more than 1000 points), Vitruvius has got you covered:

var vertices = face.Vertices;

Yeah, the result is really creepy! Don’t try it home alone!

Accessing the HD Face methods

What if you need to know more about the points that form each facial feature? For example, what if you need to get the outline of the eyes or the mouth? For such purposes, the Face class includes the following methods:

var eyeLeftOutline = face.EyeLeftPoints();
var eyeRightOutline = face.EyeRightPoints();

This is the result of drawing the contour of each eye, after calling the EyeLeftPoints() and EyeRightPoints() methods:

Converting from 3D to 2D coordinates

Accessing the HD Face features as a list of CameraSpacePoints gives you all the information about the 3D coordinates. To display those points in the 2D screen space, though, you’ll need to convert the 3D coordinates into 2D coordinates.

Kinect uses the CoordinateMapper class to convert between coordinates from different spaces. The Color space is an array of 1920×1080 pixels. The Depth & Infrared space is an array of 512×424 pixels. Vitruvius simplifies the coordinate mapping process with the handy ToPoint() method.

Converting from the 3D space to the Color space:

var nosePoint = face.Nose.ToPoint(Visualization.Color);

Converting from the 3D space to the Depth or Infrared space:

var nosePoint = face.Nose.ToPoint(Visualization.Depth);

Using the coordinate mapping process, you can now display the points on screen using Unity or XAML.

Supported platforms

Vitruvius HD Face supports the following platforms and frameworks:

• Unity3D
• WPF / .NET 4.5+
• Windows Store

Frequently Asked Questions

Finally, let me shed some light to a few topics almost every Kinect developer needs to know about.

1) What is the optimal distance from the sensor?

The sensor can accurately track a face between 40 centimeters and 2 meters. For best results, the optimal distance is 60-90cm.

2) What is the optimal rotation angle?

Kinect Face tracking works best when you are facing the sensor directly (enface). However, the tracking algorithm is pretty decent even if you rotate your head up to 50 degrees to the left or right. Face tracking won’t work if your head is rotated e.g. 90 degrees to one side.

3) What about the lighting?

As mentioned above, Kinect face tracking is strongly relying on the Color stream. As a result, the room should have a decent amount of lighting. Also, you need to avoid pointing laser beams directly to the sensor.

4) Can I find the documentation online?

Definitely. You can check the documentation online.

So, this is it! How are you planning to use the HD Face capabilities to your Kinect apps? Let me know in the comments below.

‘Till the next time, keep Kinecting!

Get Vitruvius

As you see, Vitruvius is helping innovative companies create Kinect apps fast. Vitruvius simplifies Kinect development, so you can now focus on what’s really important: your app, your research, and your customers. Why not give it a try?