Aljosa Osep, Ph.D.

In Colombia, my fave

Hi, I'm Aljosa! I am a Senior Research Scientist at NVIDIA, working on learning to understand the dynamic world from raw, unlabeled streams of sensory data.

I come from the Alpine side of Slovenia. I earned my Ph.D. from RWTH Aachen University under the supervision of Prof. Bastian Leibe. I was a postdoctoral fellow at the Technical University of Munich and the Robotics Institute, Carnegie Mellon University.


Research Statement (2019) Twitter Scholar

Research

My research focuses on enabling AI systems to robustly understand the dynamic, 3D world from raw sensor streams, such as video and LiDAR. Key areas include learning directly from raw data, tracking and segmenting objects, understanding complex spatiotemporal scenes, and predicting future events in open-world environments. Hover over each topic below to explore related publications.

Learning From Raw Sensory Data image Learning From Raw Sensory Data

Relevant Papers: Learning From Raw Sensory Data

4D Scene Understanding image 4D Scene Understanding

Relevant Papers: 4D Scene Understanding

Tracking and Panoptic Perception image Tracking and Panoptic Perception

Relevant Papers: Tracking and Panoptic Perception

Forecasting & Behavioral AI image Forecasting & Behavioral AI

Relevant Papers: Forecasting & Behavioral AI

News

Students Supervised

Talks

Teaching

Service

Publications

A. Takmaz, C. Saltori, N. Peri, T. Meinhardt, R. de Lutio, L. Leal-Taixé, A. Ošep: Towards Learning to Complete Anything in Lidar, International Conference on Machine Learning (ICML), 2025.
paper page


Y. Zhang, A. Ošep, L. Leal-Taixé, T. Meinhardt: Zero-Shot 4D Lidar Panoptic Segmentation, Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
paper page


A. Ošep, T. Meinhardt, F. Ferroni, N. Peri, D. Ramanan, L. Leal-Taixe: Better Call SAL: Towards Learning to Segment Anything in Lidar, European Conference on Computer Vision (ECCV), 2024.
paper video page


A. Chakravarthy, M. Ganesina, P. Hu, L. Leal-Taixe, S. Kong, D. Ramanan, A. Ošep: Lidar Panoptic Segmentation in an Open World, International Journal of Computer Vision (IJCV), 2024.
paper code


J. Seidenschwarz, A. Ošep, F. Ferroni, S. Lucey, L. Leal-Taixe: What Moves Together Belongs Together, Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
paper page code


C. Saltori, A. Ošep, E. Ricci, L. Leal-Taixé: Walking Your LiDOG: A Journey Through Multiple Domains for LiDAR Semantic Segmentation, International Conference on Computer Vision (ICCV), 2023.
paper video code


A. Agarwalla, X. Huang, J. Ziglar, F. Ferroni, L. Leal-Taixé, J. Hays, A. Ošep, D. Ramanan: Lidar Panoptic Segmentation and Tracking without Bells and Whistles, International Conference on Intelligent Robots and Systems (IROS), 2023.
paper page


X. Wu, K. Lau, F. Ferroni, A. Ošep, D. Ramanan: Pix2Map: Cross-modal Retrieval for Inferring Street Maps from Images, Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
paper video poster page


V. Fomenko, I. Elezi, D. Ramanan, L. Leal-Taixé, A. Ošep: Learning to Discover and Detect Objects, Neural Information Processing Systems (NeurIPS), 2022.
paper video poster page code


P. Dendorfer, V. Yugay, A. Ošep, L. Leal-Taixé: Quo Vadis: Is Trajectory Forecasting the Key Towards Long-Term Multi-Object Tracking?, Neural Information Processing Systems (NeurIPS), 2022.
paper video code


A. Kim, G. Brasó, A. Ošep, L. Leal-Taixé: PolarMOT: How far can geometric relations take us in 3D multi-object tracking?, European Conference on Computer Vision (ECCV), 2022.
paper video poster code


Q. Zhou, S. Agostinho, A. Ošep, L. Leal-Taixé: Is Geometry Enough for Matching in Visual Localization?, European Conference on Computer Vision (ECCV), 2022.
paper code


L. Nunes, X. Chen, R. Marcuzzi, A. Ošep, L. Leal-Taixé, C. Stachniss, J. Behley: Unsupervised Class-Agnostic Instance Segmentation of 3D LiDAR Data for Autonomous Vehicles, IEEE Robotics and Automation Letters (RA-L), 2022.
paper code


M. Gladkova, N. Korobov, N. Demmel, A. Ošep, L. Leal-Taixé, D. Cremers: DirectTracker: 3D Multi-Object Tracking Using Direct Image Alignment and Photometric Bundle Adjustment, International Conference on Intelligent Robots and Systems (IROS), 2022.
paper video page


N. Peri, J. Luiten, M. Li, A. Ošep, L. Leal-Taixé, D. Ramanan: Forecasting from LiDAR via Future Object Detection, Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
paper code


M. Kolmet, Q. Zhou, A. Ošep, L. Leal-Taixé: Text2Pos: Text-to-point-cloud cross-modal localization, Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
paper code


Y. Liu, I. Zulfikar, J. Luiten, A. Dave, D. Ramanan, B. Leibe, A. Ošep, L. Leal-Taixé: Opening up Open-World Tracking, Conference on Computer Vision and Pattern Recognition (CVPR), oral, 2022.
paper code


S. Agostinho, A. Ošep, A. Del Bue, L. Leal-Taixé: (Just) A Spoonful of Refinements Helps the Registration Error Go Down, International Conference on Computer Vision (ICCV) (oral), 2021.
paper code


M. Fabbri, G. Brasó, G. Maugeri, A. Ošep, R. Gasparini, O. Cetintas, S. Calderara, L. Leal-Taixé, R. Cucchiara: MOTSynth: How Can Synthetic Data Help Pedestrian Detection and Tracking?, International Conference on Computer Vision (ICCV), 2021.
paper video


M. Aygün, A. Ošep, M. Weber, M. Maximov, C. Stachniss, J. Behley, L. Leal-Taixé: 4D Panoptic LiDAR Segmentation, Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
paper video poster code


A. Kim, A. Ošep, L. Leal-Taixé: EagerMOT: 3D Multi-Object Tracking via Sensor Fusion, IEEE International Conference on Robotics and Automation (ICRA), 2021.
paper video code


M. Weber, J. Xie, M. Collins, Y. Zhu, P. Voigtlaender, H. Adam, B. Green, A. Geiger, B. Leibe, D. Cremers, A. Os̆ep, L. Leal-Taixé, L. Chen: STEP: Segmenting and Tracking Every Pixel, NeurIPS Benchmarks and Datasets, 2022.
paper code


P. Dendorfer, A. Ošep, L. Leal-Taixé: Goal-GAN: Multimodal Trajectory Prediction Based on Goal Position Estimation, Asian Conference on Computer Vision (ACCV), 2020.
paper video page code


P. Dendorfer, A. Ošep, A. Milan, K. Schindler, D. Cremers, I. Reid, S. Leal-Taixé: MOTChallenge: A Benchmark for Single-camera Multiple Target Tracking, International Journal of Computer Vision (IJCV), 2020.
paper


J. Luiten, A. Ošep, P. Dendorfer, P. Torr, A. Geiger, L. Leal-Taixé, B. Leibe: HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking, International Journal of Computer Vision (IJCV), 2020.
paper code blog


S. Mahadevan*, A. Athar*, A. Ošep, S. Hennen, L. Leal-Taixé, B. Leibe: Making a Case for 3D Convolutions for Object Segmentation in Videos, British Machine Vision Conference (BMVC), 2020.
paper video code


A. Athar*, S. Mahadevan*, A. Ošep, L. Leal-Taixé, B. Leibe: STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos, European Conference on Computer Vision (ECCV), 2020.
paper video code


Y. Xu, A. Ošep, Y. Ban, R. Horaud, L. Leal-Taixé, X. Alameda-Pineda: How To Train Your Deep Multi-Object Tracker, Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
paper video code


J. Gross, A. Ošep, B. Leibe: AlignNet-3D for Fast Point Cloud Registration of Partially Observed Objects, International Conference on 3D Vision (3DV), 2019.
paper video poster code


P. Voigtlaender, M. Krause, A. Ošep, J. Luiten, B. Sekar, A. Geiger, B. Leibe: {MOTS}: Multi-Object Tracking and Segmentation, Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
paper video code


A. Ošep, P. Voigtlaender, M. Weber, J. Luiten, B. Leibe: 4D Generic Video Object Proposals, International Conference on Robotics and Automation (ICRA), 2020.
paper video code teaser


A. Ošep, P. Voigtlaender, J. Luiten, S. Breuers, B. Leibe: Large-Scale Object Mining for Object Discovery from Unlabeled Video, International Conference on Robotics and Automation (ICRA), 2019.
paper video


A. Ošep, W. Mehner, P. Voigtlaender, B. Leibe: Track, then Decide: Category-Agnostic Vision-based Multi-Object Tracking, International Conference on Robotics and Automation (ICRA), 2018.
paper video code


A. Ošep, P. Voigtlaender, J. Luiten, S. Breuers, B. Leibe: Towards Large-Scale Video Object Mining, ECCV 2018 Workshop on Interactive and Adaptive Learning in an Open World, 2018.
paper


A. Ošep, W. Mehner, M. Mathias, B. Leibe: Combined Image- and World-Space Tracking in Traffic Scenes, International Conference on Robotics and Automation (ICRA), 2017.
paper video code teaser


D. Klostermann, A. Ošep, J. Stueckler, B. Leibe: Unsupervised Learning of Shape-Motion Patterns for Objects in Urban Street Scenes, British Machine Vision Conference (BMVC), 2016.
paper video


D. Kochanov, A. Ošep, J. Stueckler, B. Leibe: Scene Flow Propagation for Semantic Mapping and Object Discovery in Dynamic Street Scenes, International Conference on Intelligent Robots and Systems (IROS), 2016.
paper video


A. Ošep, A. Hermans, F. Engelmann, D. Klostermann, M. Mathias, B. Leibe: Multi-Scale Object Candidates for Generic Object Tracking in Street Scenes, International Conference on Robotics and Automation (ICRA), 2016.
paper


D. Mitzel, J. Diesel, A. Ošep, U. Rafi, B. Leibe: A Fixed-Dimensional 3D Shape Representation for Matching Partially Observed Objects in Street Scenes, International Conference on Robotics and Automation (ICRA), 2015.
paper


M. Weinmann, A. Ošep, R. Ruiters, R. Klein: Multi-View Normal Field Integration for 3D Reconstruction of Mirroring Objects, International Conference on Computer Vision (ICCV), 2013.
paper


M. Weinmann, R. Ruiters, A. Ošep, C. Schwartz, R. Klein: Fusing Structured Light Consistency and Helmholtz Normals for 3D Reconstruction, British Machine Vision Conference (BMVC), 2012.
paper