About | Support | Discussion Forum | Index | Service Status | ros @ Robotics Stack Exchange

Only released in EOL distros:   electric fuerte groovy

Depreciated

Please note that this package is no longer being maintained. The same functionality can be found in the rbx1_vision package in the ROS By Example repository which is actively maintained and currently available through ROS Indigo.

Overview

The pi_face_tracker ROS package uses OpenCV's Haar face detector together with Good Features to Track and the Lucas-Kanade Optical Flow tracker to perform face tracking in a live video stream or recorded video file. Works best with an RGB-D camera such as the Kinect so that depth information can be used to reduce the number of false positive face detections.

In the video above from the Honda/UCSD Video Database, the left image uses only the Haar face detector while the right image uses the detector only on the first frame to get the initial face region, then relies on Good Features to Track and Optical Flow to follow the face through the rest of the video. Notice that the right video never loses the face and runs about three times faster (CPS = "cycles per second").

How It Works

When running in automatic face detection mode, the video is scanned using three of the OpenCV Haar face detectors, two frontal cascades and one profile cascade (see HaarDetectObjects). Once a face is detected, the GoodFeaturesToTrack filter is run over the face region to extract a collection of points to track. These points are then fed to the Lucas-Kanade Optical Flow Tracker CalcOpticalFlowPyrLK which follows the points from one frame to the next.

Over time, feature points will be lost due to occlusions or picked up from the background. Therefore, two additional methods are run to maintain the integrity of the tracked feature cluster. First, feature points are pruned (dropped) if they lie too far outside the current cluster in the x-y camera plane (see parameter ~std_err_xy) or in depth (see ~use_depth_for_tracking). At the same time, new feature points are added by expanding the current cluster ROI by a small amount (10%) and running GoodFeaturesToTrack over the expanded region. If the feature cluster shrinks below a specified minimum number of points (see ~abs_min_features), the algorithm returns to the Haar face detector and scans the video to relocate the face.

When not using automatic face detection, the only difference from above is that the original region to track is defined by the user by dragging a rectangle over a part of the image.

Installation Instructions

First install Eric Perko's uvc_cam package if you don't have it already:

$ git clone https://github.com/ericperko/uvc_cam.git
$ rosdep install uvc_cam
$ rosmake uvc_cam

For rosdep and rosmake to work, new packages need to be on your ROS_PACKAGE_PATH. So clone them into your sandbox, or use rosws (do not forget to source setup.sh), or change your ROS_PACKAGE_PATH manually.

Next, install the pi_vision stack as follows:

$ svn co http://pi-robot-ros-pkg.googlecode.com/svn/trunk/pi_vision
$ rosmake pi_vision

How To Use

Note: Best results are obtained at video resolutions of 640x480 or higher. Lower resolutions will work OK if the face is not too far away from the camera.

Keyboard Commands Once Tracking Has Started

First make sure the video window is in the foreground (click on the title bar). Then type one of the following characters:

• 'q' - quit
• 'c' - clear all current features and re-detect face
• 't' - toggle text markers off/on
• 'f' - toggle features markers off/on
• 'n' - toggle night mode on/off (displays markers only)
• 'a' - toggle auto face tracking on/off

Tracking with the Kinect RGB-D Camera

$ roslaunch ros2opencv openni_node.launch

Then launch the Kinect face tracker launch file:

$ roslaunch pi_face_tracker face_tracker_kinect.launch 

Tracking with a UVC-Compatible Webcam

If you have only one webcam attached to your computer and it is device /dev/video0, then simply launch:

$ roslaunch ros2opencv uvc_cam.launch

To connect to a different device, e.g. /dev/video1, use the following form:

$ roslaunch ros2opencv uvc_cam.launch device:=/dev/video1

Then launch the UVC face tracker launch file:

$ roslaunch pi_face_tracker face_tracker_uvc_cam.launch 

Tracking with an AVI File

$ roslaunch pi_face_tracker face_tracker_uvc_cam.launch 

Then start the video file:

$ roslaunch ros2opencv avi2ros.launch input:=/path/to/avi/file

Special Keyboard Commands for AVI Files

First make sure the video window is in the foreground (click on the title bar):

• space - pause/resume video
• r - restart video from beginning

Tracking a Non-Face Video Patch

To track objects other than faces, first turn off face tracking either by typing the letter 'a' in the image window or by setting the auto_face_tracking parameter to False in the appropriate launch file and re-running the launch file. Then use your mouse to draw a box around the object you want to track. You can select a different patch at any time.

Nodes

pi_face_tracker.py

A ROS node for face tracking using the OpenCV face detector together with Good Features to Track and the Lucas-Kanade Optical Flow tracker.

Subscribed Topics

~input_rgb_image (sensor_msgs/Image)

• The ROS RGB image topic on which to perform face detection and tracking

~input_depth_image (sensor_msgs/Image)

• The ROS image topic containing depth information

Published Topics

/roi (sensor_msgs/RegionOfInterest)

• The region of interest around the tracked face

/target_point (geometry_msgs/PointStamped)

• The PointStamped centroid of the tracked cluster (when using a 3D camera)

Services

~key_command (pi_face_tracker/KeyCommand)

• Send a std_msgs/String command such as 'c' to clear the current features

~set_roi (pi_face_tracker/SetROI)

• Set the ROI to track using message type sensor_msgs/RegionOfInterest.

Parameters

~auto_face_tracking (boolean, default: True)

• Run face tracking automatically at startup

~use_haar_only (boolean, default: False)

• Use Haar face detection only. (Does not use Good Features or Optical Flow)

~use_depth_for_detection (boolean, default: True)

• Use depth info to reduce false positive face detections.

~use_depth_for_tracking (boolean, default: False)

• Use depth to prune features that are outside the main cluster.

~auto_min_features (boolean, default: True)

• Use the number of features from the first detection to monitor addition or pruning of additional features.

~min_features (int, default: 50)

• If auto_min_features is False, use this number.

~abs_min_features (int, default: 6)

• If auto_min_features is False, and number of features falls below this number, rescan the image with the Haar detector to re-detect the face region.

~good_feature_distance (int, default: 5)

• minimum pixel distance between features for the Good Features to Track algorithm

~add_feature_distance (int, default: 10)

• minimum pixel distance between features when adding new features after initial face detection

~std_err_xy (float, default: 2.5)

• How far (standard error) must a feature deviate from the cluster mean in the x-y camera plane before we drop it.

~pct_err_z (float, default: 0.42)

• How far (percent) must a feature deviate from the cluster mean in the depth (z) direction before we drop it.

~max_mse (float, default: 100000)

• Maximum mean squared error of the feature cluster before we start over again with a new Haar detection.

~show_text (str, default: True)

• Show the text messages on the image by default. (Can be toggled later with the "t" command.)

~show_features (str, default: True)

• Show the feature points on the image by default. (Can be toggled later with the "f" command.)

~max_face_size (float, default: 0.28)

• Maximum size of face in meters we will accept during initial detection.

~expand_roi (float, default: 1.02)

• Amount to multiply the ROI around the current tracked cluster when adding new points.

~fov_width (float, default: 1.094)

• Width in radians of camera's field of view. (Default is for the Kinect.)

~fov_height (float, default: 1.094)

• Height in radians of camera's field of view. (Default is for the Kinect.)

~flip_image (boolean, default: False)

• Flip the image 180 vertically (some laptop cameras are upside down)

~feature_type (int, default: 0)

• What type of feature to track: 0 = Good Features, 1 = SURF