Qualisys Mocap Management

Summary

A large amount of my work currently revolves around using the Qualisys Motion Capture systems. As evidenced through the various other projects that can be seen on this blog, most of our Unity projects and templates revolve around the primary use of either Meta's Oculus Quests, or the use of our motion capture system, Qualisys, or both.

One of my major responsibilities is to manage and run our 12 infrared camera mocap system, created by Qualisys.

Here we would show off our mocap capabilities to faculty, external partners, and students who were interested. In this, we would do recordings, as well as live demonstrations due to Qualisys' ability to stream the skeletal data in real-time over our local network, which we could then input into Unity or Unreal Engine.

We are also able to track rigid objects with 6 degrees of freedom (6DOF - X,Y,Z, tracking for both position and rotation), which could be use for a variety of purposes, most often to act as a virtual camera, allowing us to use a generic object to control the a camera in Unity or Unreal Engine in real-time. We have also done demonstrations similar to virtual production, tracking an actual camera which would record our actors, that is also motion tracked and controls the camera in Unity, and we project that view onto a screen behind the actors.

Overview

This is the primary UI you'll be staring at when using QTM, with a saved capture of a dance student we had come in for a collaboration with School of Dance's instructor, Jesse Zaritt, you can learn more about this here.

I'll briefly describe all the sections:

  1. This is an AIM Model, which is QTM's way of describing a skeleton. This data can be used to drive 3D character's in either playback or real-time, streaming or exported to external DCC's or game engines.

  2. These are motion trails, which can be extended to show the full time line or just a small section, trails can also be shown after, sort of like an onion-skinning. These are only a visualizer, and that data cannot be used or exported in anyway, but a similar effect can obviously be re-created in external software.

  3. This list shows the labeled tracked trajectories, in order to capture something, Qualisys needs to track markers, and in the case of a AIM Model, it specifics each marker so it knows how to configure a skeleton. To the right of the name of each trajectory, is a Fill Level, which is represented as a percentage of 100. This how much the system has the marker for the duration of the capture, 100% is great, and the worse this percentage is, the worst the data exported is going to be.

  4. This is similar to above, except these are trajectories that are unlabeled, they could be a marker that just fails to identify, or external markers or otherwise temporary bugs, lighting artifacts (QTM tracks the markers based on infrared cameras). Generally speaking, I rarely pay attention to this window, as long as the skeleton looks good, and the labeled trajectories are decently filled.

5. A small viewport optios menu, for panning, zooming and rotating. Some more useful features are the 2 buttons at the bottom, one show's the camera's viewcones, which can be displayed individually or all at once. The one below that is calibrated volume, which as it sounds shows the bounds of the calibrated area, more on this later. Another useful button is the 2D button at the top, which switches to 3D when clicked. This allows you to swap back and forth between 3D view, and the 2D camera view, where you can inspect what each individual camera sees.

2D Camera View

6. This is the project path data tree, it shows all files and folders in the project's directory. ROM's, AIM models, and captured performances are saved and selected here. It's essentially just a file explorer.

7. The toolbar, which contains all the essential buttons for using the software that you'll need. Above that is the menu bar, which also has contains essential buttons and menus for working within the program.

Calibration

To start the process and setup the stage for capture, we must first do a Calibration of the space, this is uses 2 tools provided by Qualisys in order to, well, calibrate the space.

It sets the world origin, and gets an understanding of the size of the space, and of objects and markers in it, as well as their distance between each other.

The L-Frame, which helps define the X and Y axis of the space, is placed in the center of the stage, and oriented in whatever way you want the axis to align.

Note: This is useful for transferring data, captured or real-time, over to another DCC, which may have a different axis. For example, whenever we brought the data into Unity, the object would always be rotated 180 degrees the wrong way, this is a pretty simple fix to make in-engine, but it has to be done every time for every object, and when we start doing more complicated stuff, it's nice to not have to worry about those arbitrary rotations screwing things up. In this instance, we simple inverted the way our L-shape was facing to invert the world.

The wand is then held at one corner of the space, and when we start calibrating, the user will wave the wand around in a figure 8, or infinity symbol, across their body, attempting to get as close to the floor and high in the air as possible to fully capture the space.

They will then walk back and forth along the space in a "mowing the lawn" type motion, waving the wand, at a relatively slow pace. The duration of the capture is up to the individual, we typically go for 3 minutes.

Also worth taking note is making sure the correct settings and calibration type are setup in the Calibration options menu. See below:

After the calibration is finished, a new menu will pop up showing the results of the calibration. It will either fail, pass, or pass but recommend another run as the results aren't great.

The results will list some numbers captured by every camera, though I can't really say I know what any of them really mean. The only important number we pay attention to is the Avg. residual (mm), which we want to be as low as possible. If it starts to get up to 3 or higher, that's bad.

You can see our current calibration results below, all the numbers hover around the 0-1 range, with only one camera having a high residual of 2.1

Once this is done, the space is calibrated and ready to go, a volume of the calibrated space can also be visualized in the viewport. With that, we can start tracking markers and making recordings.

There are 2 primary ways to track something with markers:

  • 6DOF Rigid-body - a rigid or static object that doesn't bend, a variety of things can fall into this category, in the past we've used it to track an empty picture frame to act as a virtual camera, and we've tracked wireless headphones.

  • AIM Model - the bread and butter, this AIM Model tracks an actor with a variety of proportions and creates a useable skeleton rig that can be transferred over in real time or as recorded motions for use in game engines or other DCC apps.

In order to do either of these, we must first capture and save a range of motion (ROM), this is what the program will use to identify and save the placement of markers on either an object or actor.

Creating a Range of Motion (ROM)

Creating a ROM is simple and doesn't actually require a specific button or options menu to tweak. Simple have either the actor or the rigid-body object placed at or around world origin, capture using the default record button, end it, and save it. That's it, but the process slightly differs based on what you're capturing.

6DOF Rigid-body: Here we'll place the object at center space/origin, and start capturing. After a second or so passes of the object being still, facing forward, we'll move the object around the space, rotating and twisting it around, attempting to move it around the space in a similar way that we'll be using it for. Once we want to finish, we'll place the object back at it's starting position. It can either be a continuous capture that we'll stop whenever we feel enough data's been captured, or use a set timer. For rigid-bodies, we typically set a timer for a 20 second capture, which is typically enough.

After you have the movement captured, you simply drag select all the recorded markers, and in the right-click context menu, a Define Rigid-body option will be there, this will actually create and save the rigid-body. It will ask if you want to base the rigid-body on the current frame, or from a series of frames, and finally you'll name it.

AIM Model: This process is a bit different, we'll start with the actor in T-pose somewhere close to world origin at the beginning of capture, they'll hold the pose for a second or two. They'll then start doing arm circles, rotating just their elbows around for a few full rotations, then start using the whole shoulder for a few full rotations. They'll do the same, but with the legs, first just the knee rotating around, then the whole leg/hip. After that, they'll roll their neck around a bit, and finally swing their hips in a circular motion. When finished, they go back to standing in T-pose.

We'll save the capture, and if we'll be free to move on. Of course, I should probably go into a bit more detail of how setting up a Rigid-body or AIM Model.

6-DOF Rigidbody

Starting with Rigid-body, it's as simple as placing markers on an object, specifically at least 4 markers need to be placed, and they should be setup in a way that creates a convex kind of shape. Meaning, the markers shouldn't form a line or a flat plane, as the system will have trouble recognizing the object's orientation.

This image on the left demonstrates the basic idea, if you just had those 4 markers equidistant on an object, any nothing else, the system would have trouble identifying which way the object faces, since it would look the same on either side. In this instance, adding another marker that juts out from it will fix this.

It's really about asymmetry, adjusting the planar markers so they're not all equidistant from each other would also help.

After the markers are placed, and a ROM is captured for the rigid-body, the process is essentially done. All that's left is to check the settings for that rigid-body in the 6DOF Tracking menu

The 6DOF Tracking menu allows you to save out and load in rigid-bodies, change the identification color and label, as well as other small tweaks, like changing it's origin point of transform and rotation.

AIM Models

Setting up the AIM Model is a bit more involved, and differs depending on what it is you're trying to accomplish. A guide and specific layout of markers on a suit is recommended based on the data your capturing, for us, that's animation, so we follow an "Animation Marker Set", but Qualisys also supports capturing data to be used in medical and biomechanical research industries, measuring things like force, trajectories and so on.

The adjacent images demonstrate one of our AIM models loaded up from a project we worked on with Jesse Zaritt, a dance instructor from the School of Dance. All markers are visible, and create a sort of cage to act as the placement for the actual skeleton itself (visualized in orange), which is created after solving for the skeleton.

All 42 markers are set, the system also shows symmetry across the X-axis by coloring the left and right side markers.

4 orange markers exist as the sit on the plane of symmetry.

You can also see a little gray marker on the right side of the image by the hand. This is an "unlabeled trajectory", which is to say the system doesn't recognize it as being part of the model. This is just a bug, as there shouldn't be any marker there at all, something that is probably related to the camera's vision, one of the camera's may have detected some stray reflected light that tricked it into thinking something was there.

Fortunately, the system is relatively flexible and smart enough that such issues don't have much of an overall impact on the performance or data captured. This is good since these types of errors are pretty commonplace in this system, and I imagine most motion capture solutions as well that aren't setup in a perfectly sterilized environment.

Occasionally markers will dance around, it'll identify the markers wrongly as others, and of course, due to occulsion and other things, sometimes tracked markers disappear entirely. Thankfully, the system has a sort of prediction system that will try and guess where certain solved bones will be if markers disappear, so the skeleton doesn't break entirely and data is lost.

After markers are placed, following the marker set guide, we have the performer walk into center stage. We also load up an existing animation AIM Model for the system to base off of in the AIM project settings window.

This essentially acts as a sort of starting base or template for one to build their own custom AIM models for animation.

With this preset animation model applied, and a performer with all markers placed, the system should automatically begin to start identifying the full model, with all the correct connections between the markers. It won't be perfect obviously, but it should be pretty close, if it doesn't identify at all, or entire sections of the body are unidentified or missing, then you know something is wrong.

This just usually means the markers aren't in the exact location they should be, and some adjustments need to be made, or the model isn't being applied correctly as seen in the image below.

Once the system beings recognizing the performer in space using the animation model, a ROM is recorded and saved.

We then take all the labeled trajectories, add a prefix name to them, run the "Generate Model" command under the AIM sub-menu in the menu bar.

If all the previous steps went well, then typically the rest of this process is a breeze. In this new window pop-up for generating the AIM model, we tell it to create a new model.

A new window pops up giving a connection hierarchy of sorts that chain links all the markers together. To be honest, I don't really understand this step, and typically we just look at it to make sure nothing looks insane, and we just click next, editing these connections is possible but I'm not sure it's ever fixed any issues. If we do have issues, we just start back at the beginning and try again by adjusting marker placement and/or restarting QTM.

After that, we get a final entry prompt for giving the AIM model a name, which we copy from the prefix we added, and we finished. The AIM model is generated and ready, all we do is click on the "Calibrate Skeletons" button (shortcut: F8), and this will do the skeleton solving that will place the animation ready rig inside the cage of the AIM model.

It'll take a few seconds but after that it should be done. A new real-time scene can be started, and your actor should be able to move around and be recognized in the space, with the solved orange skeleton, and their name above their head. This means any captures can exported to your game engine or DCC of choice, and used as is or for retargeting. 

(Note: the .fbx that QTM exports by default doesn't actually work very well at all when imported into Blender, I go into this a bit more in the Jesse Zaritt Performance project page, but it works fine in Maya, and strangely enough, exporting it straight out of Maya and then into Blender does work, go figure.)

Even more exciting, this data can also be streamed in real-time locally or through a network into a game engine, which I'm sure I've mentioned a couple of times already. It is a really neat aspect of the system, and frankly, almost every major project I've worked on during my team here involved streaming mocap data real-time to Unity or Unreal Engine. And the system does a decent job at working and solving the skeleton in real-time, 6DOF rigid-bodies work even better, which is also important as a lot of our VR projects are tracked by our QTM system and work in a hybrid system with the Quest's Inside-Out Tracking capabilities. More on this can be seen on the VRQTMMP project page.