Intelligent Systems Research Group

Human keypoint estimation is a fundamental task in computer vision, with applications ranging from motion capture to human-computer interaction. However, current models often struggle with images that feature incomplete poses, where parts of the body are occluded or out of frame. Recent research [1] highlights this challenge, showing the limitations of existing models in accurately estimating keypoints from partial visual data.

The ISRG operates a 3D scanner that often captures images with incomplete poses due to the camera setup. While these incomplete captures make human keypoint estimation difficult, they also present an interesting problem.

In this project, the student focused on developing and evaluating a novel method to address this challenge by incorporating generative outpainting into the human keypoint estimation pipeline. Outpainting is a technique that uses generative models to extend the boundaries of an image, effectively adding context that was not originally present. By using Stable Diffusion [2], the student systematically outpainted images to provide additional contextual information.

The outpainted images were then processed using an off-the-shelf pose estimator to assess the performance improvements. The evaluation was conducted on a previously captured and annotated dataset from the 3D scanner, ensuring that the results were both relevant and grounded in real-world data.