MultiCam: On-the-fly Multi-Camera Pose Estimation Using Spatiotemporal Overlaps of Known Objects

Abstract

Multi-camera dynamic augmented Reality (AR) applications require a camera pose estimation to leverage individual information from each camera in one common system. This can be achieved by combining contextual information, such as markers or objects, across multiple views. While commonly cameras are calibrated in an initial step or updated through the constant use of markers, another option is to leverage information already present in the scene, like known objects. Another downside of marker-based tracking is that markers have to be tracked inside the field-of-view (FoV) of the cameras. To overcome these limitations, we propose a constant dynamic camera pose estimation leveraging spatiotemporal FoV overlaps of known objects on the fly. To achieve that, we enhance the state-of-the-art object pose estimator to update our spatiotemporal scene graph, enabling a relation even among non-overlapping FoV cameras. To evaluate our approach, we introduce a multi-camera, multi-object pose estimation dataset with temporal FoV overlap, including static and dynamic cameras. Furthermore, in FoV overlapping scenarios, we outperform the state-of-the-art on the widely used YCB-V and T-LESS dataset in camera pose accuracy. Our performance on both previous and our proposed datasets validates the effectiveness of our marker-less approach for AR applications.

Shiyu Li

Doctoral Candidate

My research interests include 3D vision, AR application.

Hannah Schieber

Doctoral Candidate

I am interested in computer vision and extended reality. I research 3D scene content creation using neural rendering and guidance of people in 3D.

Kristoffer Waldow

Doctoral Candidate

Kristoffer is a Doctoral candidate at the Technical University of Munich, Human-Centered Computing and Extended Reality Lab and a research associate at the TH Köln, Computer Graphics research group. His field of interest is Mixed Reality (MR) technologies, especially human-computer interaction in MR environments to improve interpersonal communication and accessibility with avatars.

Julian Kreimeier

Senior Researcher

I research human-computer interaction, accesibility and extended reality.

Daniel Roth

Director

Assistant professor at TU Munich and Director of the HEX Lab