Avateering (Kinect + Unity)

Avateering is the most exciting part of Kinect development and one of the most complex features to implement. Vitruvius encapsulates all of the hard work and avateering algorithms and exposes an easy-to-use API.

You only need a few rigid 3D models. Vitruvius comes with 9 models, which you can use for free in your personal or commercial applications. Also, Vitruvius includes numerous textures to choose from, so each model can appear with different clothing!

{ Reasonably, Vitruvius is featured in the official Microsoft Kinect Website and Channel9 }

If you need a different model and have a 3D artist, just provide them with the rigid skeleton and let them create a new 3D model on top of it. If you need different models and do not have a 3D artist, give us a shot and we can create a 3D model tailored to your needs.

In this tutorial, I am going to show you how you can integrate Vitruvius within your Unity project and animate the bundled male and female 3D models. Vitruvius allows a single player control a single avatar or multiple avatars. The videos below demonstrate the difference:

Prerequisites

• Kinect for XBOX v2 sensor with an adapter (or Kinect for Windows v2 sensor)
• Kinect for Windows v2 SDK
• Windows 8.1 or higher
• Visual Studio 2013 or higher
• A dedicated USB 3 port

Source Code

The source code, along with the required assemblies, is included in the following versions of Vitruvius:

• Academic
• Premium
• Platinum

Step 1: Create a new Unity project

Launch Unity and create a new project, or open an existing Kinect project. To get started, simply double-click the .unitypackage file from the fodler you downloaded.

You’ll also need to import Windows.Kinect.dll and LightBuzz.Vitruvius.dll. These are the required assemblies to use the complete power of Vitruvius.

Step 2: Import the 3D models

Open your scene file in the Editor and import the 3D models. Place them whenever you think it’s necessary for your game. I have imported the male and female 3D body models. They are named “male” and “female”, respectively.

This is how my editor looks like:

Step 3: Action!

It’s now time to write your Kinect script. You’ll need to include the following namespaces:

using Windows.Kinect;
using LightBuzz.Vitruvius;

After importing the assemblies, we’ll create the required members to open the sensor, acquire the Body data and update the avatars accordingly.

The separatedPlayers variable indicates whether a single player will control one avatar or multiple avatars.

Members

// Kinect
BodyFrameReader bodyReader;
Body[] users;

// 3D models
Model[] models;

public FBX female;
public FBX male;

// Determines whether
// one player will control all avatars, or
// whether each player will control one avatar
public bool seperatedPlayers;

Start

The Start method initializes Kinect and Vitruvius. It’s important to open the sensor and enable avateering functionality to your 3D models. The code is really simple and self-explanatory:

void Start()
{
    // 1. Initialize Kinect
    sensor = KinectSensor.GetDefault();

    if (sensor != null)
    {
        bodyReader = sensor.BodyFrameSource.OpenReader();

        sensor.Open();
    }

    // 2. Enable Avateering
    Avateering.Enable();

    // 3. Specify the 3D models to animate.
    models = new Model[]
    {
        female,
        male
    };

    // 4. Initialize each 3D model.
    for (int i = 0; i < models.Length; i++)
    {
        models[i].Initialize();
    }
}

Update

Now, it’s time for the cool part! To see your avatars animated, you simply have to loop through the tracked bodies and update the corresponding avatar. Vitruvius does this with just one line of code.

void Update()
{
    if (bodyReader != null)
    {
        using (var frame = bodyReader.AcquireLatestFrame())
        {
            if (frame != null)
            {
                users = frame.Bodies().
                        Where(b => b.IsTracked).ToArray();

                if (seperatedPlayers)
                {
                    // Each user controls a different avatar.
                    for (int index = 0; index < users.Length; index++)
                    {
                       Model model = models[index];
                       Body user = users[index];

                       // Yes, this line does ALL of the hard work for you.
                       Avateering.Update(model, user);
                   }
                }
                else if (users.Length > 0)
                {
                    // A single user controls all of the avatars.
                    for (int index = 0; index < models.Length; index++)
                    {
                        Model model = models[index];
                        Body user = users[0];

                        // Yes, this line does ALL of the hard work for you.
                        Avateering.Update(model, user);
                    }
                }
            }
        }
    }
}

Dispose

Finally, do not forget to dispose Kinect and clean any resources.

public void Dispose()
{
    // 1. Dispose Kinect
    if (bodyReader != null)
    {
        bodyReader.Dispose();
        bodyReader = null;
    }

    if (sensor != null)
    {
        if (sensor.IsOpen)
        {
            sensor.Close();
        }

        sensor = null;
    }

    // 2. Dispose 3D models
    for (int i = 0; i < models.Length; i++)
    {
        models[i].Dispose();
    }

    // 3. Disable avateering
    Avateering.Disable();
}

This is it! You have just developed an avateering app using Kinect v2!

BONUS – New 3D models

Vitruvius is bundled with nine 3D models you can use in your commercial projects. We are adding new 3D models all the time. Our models are perfectly rigged and work with Kinect just fine. Each model is coming in multiple textures. to modify a texture, simply open the Samples/Models/FBX folder and pick the 3D models and textures of your choice.

The following models are included:

• Female Casual – a beautiful female model with various hair colors and clothes.
• Male Casual – a male model with various clothes.
• Female Athletic – a female model with sports clothing.
• Male Athletic – a male model with sports clothing.
• Kid – a 3D model of a kid.
• Skeleton – a fantastic model of a human skeleton with bones and skull!
• Longsleeve – a shirt with long sleeves and various colors.
• Tshirt – a simple t-shirt model with various colors.
• Pants – a pair of trousers with various styles (jeans, black, formal).

Bodies

Clothes

This is a video of the skeleton model in action:

So, what will you do with Vitruvius? Let me know in the comments below!