Tracking Beyond Detection

Through object detection and tracking, autonomous systems become aware of their whereabouts and determine their future motion. The leading vision-based paradigms for multi-object tracking [8, 3] heavily relies on robust object detectors [35, 10, 34], which i) can currently detect only object classes that are observed frequently and ii) are trained using huge quantities of carefully labeled samples.

On the other hand, mobile robots experience a continuous stream of sensory data and need to operate in a physical 3D world. Furthermore, our world is inherently open-set [9, 5] and populated with unknown dynamic objects (Fig. 1). It is therefore crucial for future mobile robots to continuously model and learn from previously unseen objects – especially if they can move and pose a safety hazard. Motivated by these challenges, my main research drive is to enable future mobile robots to adapt to novel environments and to learn about objects and their behavioral patterns automatically, without exhaustive human supervision.

These goals are closely related to tasks such as representation learning from video [39], weakly-supervised detector learning [18], zero-shot object recognition and detection [40, 2], object class discovery [23, 17, 17, 11, 11, 12] and learning object detectors from dominant video tubes [33]. The main difference to the aforementioned is that we are learning to segment, track, and recognize objects from a raw, continuous stream of sensory data, without any form of manual pre-processing. In our scenarios, objects of interest are not pre-localized, and we do not assume that videos contain a dominant, salient moving region – objects are interacting with each other and are continuously entering and leaving the sensing area.

I. Current and Past Research

At the core of robot perception is the ability to track and analyze the moving objects online. Via object tracking, mobile robots have the ability to foresee potential collisions and react to possibly harmful situations in time.

Multi-Object Tracking.  In [25], we proposed a method that lifts the detection-based multi-object tracking paradigm to 3D using an inexpensive stereo setup and that can detect, track, and localize surrounding objects in 3D space. Our system combines 2D object detections and stereo-based depth measurements to improve image-based tracking and, importantly, 3D localization. Our experiments show that the proposed method was on-par with state-of-the-art image-based tracking approaches and can localize objects in 3D robustly.

Fig. 2.   CAMOT [26] and 4D-GVT [29] localize tracked objects in 3D space and time.

While the focus of that method was on online multi-object tracking and 3D object trajectory reconstruction, the recent trends in vision-based multi-object tracking are heading towards leveraging the representational power of deep learning. One of the major challenges in end-to-end learning of multi-object tracking is the lack of differentiable loss functions that directly correlate with established MOT evaluation measures [4]. This stems from the fact that tracking evaluation necessitates establishing a matching between ground truth objects and track prediction – typically performed using the Hungarian algorithm, which contains non-differentiable operations. To this end, we proposed a trainable matching layer, inspired by Hungarian algorithm [15], that allows to back-propagate gradients to the tracking network and novel loss functions that directly correlate to established tracking evaluation measures [4].

Category-Agnostic Multi-Object Tracking and Video Object Mining.  The aforementioned methods follow the common paradigm for vision-based multi-object tracking, that extract object evidence from images using a pre-trained object detector, e.g., [35, 10, 34]. It is important to study multi-object tracking in a well-controlled, closed-set environment and dis-entangle object tracking performance from object recognition accuracy. However, obtaining reliable detectors for every possible object class will clearly not be feasible, as we can expect the frequency of object category observations to follow a power-law distribution with some object categories occurring very frequently and the vast majority being increasingly rare. To this end, we investigated vision-based generic multi-object tracking in the open-set world, where the set of object classes that need to be tracked and detected is unbounded, and we proposed CAMOT, a vision-based, Category-Agnostic Multi-Object Tracker [26]. This method leverages recent developments in learning-based object proposal generation and estimates trajectories of arbitrary objects. This has been largely inspired by the success of the early tracking-before-detection paradigm in the context of LiDAR-based multi-object tracking for autonomous driving [37, 24]; however, vision-based object instance segmentation of arbitrary objects is a very challenging and open research problem [31, 30], as we cannot rely on the reliable spatial proximity cues for grouping.

At the core of our approach is an efficient mask-based representation of tracked objects that can be simply lifted to 3D space in the presence of depth estimates (see Fig. 2). This allows for a robust data association based on an estimated 3D motion vector and pixel-precise representation of the object tracks. CAMOT achieves comparable performance to detection-based methods for the known object classes in the camera near-range and can track a large variety of other objects. One of the critical components of this method is precise, segmentation-mask based data association – we investigated the impact of such representation for tracking in the follow-up work [38] and successfully applied this paradigm to the task of (unsupervised) video-object segmentation [20, 21]. In [1], we further demonstrate that by tracking objects in a category-agnostic manner by grouping spatio-temporal volumes, we can generalize across different tasks and datasets related to pixel-precise object tracking [38, 6, 41, 32].

Beyond online tracking, we can use such an approach to fuse information across time from different views in an offline fashion and build a 3D semantic map of the world as a composition of dynamic objects. Such a fused map can be used to mine and discover unknown objects and their respective trajectories. In this line of work, we have proposed 4D Generic Video Object Proposals (4D-GVT) [29] for offline video object proposal generation, designed to mine objects from a large corpus of video data. 4D-GVT unifies two separate networks used in CAMOT for proposal generation and track classification. We have demonstrated that we can compensate for the absence of training data for generating video object proposals by combining learning-based methods and prior knowledge about parallax, motion, and appearance consistency in a probabilistic framework. 4D-GVT achieves remarkable generalization to unseen object classes. In particular, we obtain a better recall by training our method on the COCO dataset [19] using information about 80 object classes compared to the large-scale object instance segmentation method by [13], which trains jointly on COCO and VisualGenome [14] datasets, containing labels for over 3,000 object classes. Additionally, in contrast to [13], our approach precisely tracks each candidate object in 3D space.

Video Object Discovery.  In [28], we present a large-scale study for object mining and category discovery and show that our CAMOT [26] can be used for large-scale video object mining and discovery in automotive scenarios. In total, we mined roughly 10 hours of video data (from the Oxford RobotCar dataset [22]) consisting of more than 560,000 frames. From this data, we extracted more than 360,000 object tracks. We used these tracks to evaluate the suitability of different feature representations and clustering strategies for object discovery. We published preliminary results on using these discovered clusters for self-supervised object detection in [27].

II. Future Work

Simultaneous Multi-Object Tracking, Localization, and Mapping.  I am very thrilled by the future potential of my research area. In the near term, I plan to combine our trackers with methods for simultaneous localization and mapping (SLAM). Such fusion can benefit both fields of research. Multi-object tracking (MOT) could benefit from temporal integration of depth measurements and improve robustness to sensory failures (e.g., extreme lighting and weather conditions). Existing vision-based SLAM methods could, via tracking, explicitly account for moving objects and estimate precisely the full 3D shape of tracked objects to improve 3D tracking precision. In addition to moving objects, our methods can track and localize static landmarks such as traffic signs, trees, road signs, markers, cones, etc., which could be used to improve the robustness of existing SLAM pipelines by incorporating object-level cues.

Self-Supervised Object Detector Learning.  I am planning to continue my research in the area of vision-based object discovery and self-supervised detector learning. By mining large video collections, we demonstrated [28] that we can group semantically similar objects and discover novel object classes via clustering. These clusters could then be used as a basis for learning new detectors without human supervision.

Motion Prediction and Shape Completion.  Our 4D video-object proposals [29] could be used for several tasks. These proposals do not only localize possible objects but also provide 3D localization of object trajectories and capture the evolution of 3D shape over time. I plan to investigate whether the estimated trajectories can be as a supervisory signal for predicting future motion [16] and 3D shape-completion [36].

Cross-modality Video Object Mining.  Thanks to new automotive datasets [7], we believe there is significant potential in extending our methods towards leveraging different sensor modalities (cameras, LiDAR, RADAR) to mine novel object tracks from recently introduced large-scale automotive datasets.

Acknowledgements:  I would like to thank all my collaborators, especially my PhD advisor Bastian Leibe and PostDoc advisor Laura Leal-Taixé.

References