I am an assistant professor in the CS department at UT Dallas. I lead the Intelligent Robotics and Vision Lab (IRVL)

Biography

Yu Xiang is an Assistant Professor in the Department of Computer Science at the University of Texas at Dallas. Before joining UT Dallas, he was a Senior Research Scientist in Robotics at NVIDIA Research from 2018 to 2021. He received his Ph.D. in Electrical and Computer Engineering from the University of Michigan at Ann Arbor in 2016 advised by Prof. Silvio Savarese. He was a postdoctoral researcher with Prof. Dieter Fox in Computer Science & Engineering at the University of Washington from 2016 to 2017, and was a visiting student researcher in the Artificial Intelligence Lab at Stanford University from 2013 to 2016. He received an M.S. degree in Computer Science from Fudan University in 2010 advised by Prof. Xiangdong Zhou, and a B.S. degree in Computer Science from Fudan University in 2007. (CV, Google Scholar)

Research Interests

My research interests primarily focus on robotics and computer vision. I am interested in studying how can an intelligent system or a robot understand its 3D environment from sensing and accomplish tasks in the real world, which is a very challenging and unsolved problem. Perception serves as an interface between an intelligent system and the 3D world, which provides useful information for planning and control of the system in conducting different tasks. I am interested in integrating perception, planning and control in a systematic way and deploying robots in the real world which are capable of accomplishing tasks for humans. I apply machine learning, especially deep learning, to tackle the challenges in robot perception. I explore how to introduce domain knowledge such as geometric constraints into a deep neural network architecture to learn a useful representation of the 3D environment for perception. I am also interested in how to learn a joint representation for perception, planning and control with deep neural networks, and how to enable robots to learn skills in a self-supervision way by interacting with the 3D environment.

Joining My Group

For perspective students, if you are interested in coming to UT Dallas to join my group as a Ph.D. student, please apply to the Ph.D. program in Computer Science and mention my name in your research statement. For current master and undergraduate students at UT Dallas, if you are interested in doing research with me, please feel free to send me an email.

Recent Research Highlights

A robot doing tasks in a kitchen with 6D object pose estimation and manipulation trajectory optimization 6D grapsing of unseen objects with unseen object instance segmentation and a learned RL policy for closed-loop control.

News

  • 9/13/2021 Two papers accepted to CoRL 2021: 6D grasping (GA-DDPG) and unseen object segmentation (RICE).
  • 8/1/2021 I start as an assistant professor in CS at UT Dallas.
  • 5/25/2021 Co-organizing the Visual Learning and Reasoning for Robotics workshop at RSS 2021.
  • 4/12/2021 I will join the Department of Computer Science at UT Dallas in August!
  • 2/28/2021 Two papers accepted to CVPR 2021: Hand-Object Interaction and Transparent Object Depth Completion.
  • 2/8/2021 Our work on unseen object instance segmentation is accepted to the IEEE Transactions on Robotics (T-RO).
  • 1/28/2021 Our work on long-horizon visual navigation is accepted to the IEEE Robotics and Automation Letters (RA-L).
  • 1/17/2021 Co-organizing the 3D Vision and Robotics workshop at CVPR 2021.
  • 12/28/2020 Our work on 6D Object Pose Tracking is accepted to the IEEE Transactions on Robotics (T-RO).
  • 12/13/2020 We release our work on unseen object instance segmentation by learning RGB-D feature embeddings UnseenObjectClustering.
  • 12/13/2020 We release our implementation of PoseCNN and DeepIM in PyTorch.
  • 10/14/2020 Our work on learning feature embedings for unseen object segmentation is accepted to CoRL 2020.
  • 10/6/2020 We release our joint motion and grasp planner OMG-Planner.
  • 9/3/2020 We release our 6D Object Pose Estimation System based on PoseRBPF.
  • 6/19/2020 Co-organizing the Visual Learning and Reasoning for Robotic Manipulation workshop at RSS 2020.
  • 5/5/2020 Our work on manipulation trajectory optimization is accepted to RSS 2020.
  • 2/23/2020 Our work on Unseen Object Pose Estimation is accepted to CVPR 2020.
  • 1/21/2020 Our works on self-supervised 6D pose and topological navigation are accepted to ICRA 2020.
More news

Publications

2021

RICE: Refining Instance Masks in Cluttered Environments with Graph Neural Networks
Christopher Xie, Arsalan Mousavian, Yu Xiang and Dieter Fox
In Conference on Robot Learning (CoRL), 2021.
arXiv, Bibtex, Code, OpenReview
Goal-Auxiliary Actor-Critic for 6D Robotic Grasping with Point Clouds
Lirui Wang, Yu Xiang, Wei Yang, Arsalan Mousavian and Dieter Fox
In Conference on Robot Learning (CoRL), 2021.
arXiv, Bibtex, Project, Code, OpenReview
DexYCB: A Benchmark for Capturing Hand Grasping of Objects
Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj Narang, Karl Van Wyk, Umar Iqbal, Stan Birchfield, Jan Kautz and Dieter Fox
In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
arXiv, PDF, Supplementary, Bibtex, Project
RGB-D Local Implicit Function for Depth Completion of Transparent Objects
Luyang Zhu, Arsalan Mousavian, Yu Xiang, Hammad Mazhar, Jozef van Eenbergen, Shoubhik Debnath and Dieter Fox
In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
arXiv, PDF, Supplementary, Bibtex, Project
Learning Composable Behavior Embeddings for Long-horizon Visual Navigation
Xiangyun Meng, Yu Xiang and Dieter Fox
In IEEE Robotics and Automation Letters (RA-L), 2021.
arXiv, PDF, Bibtex, Project
Unseen Object Instance Segmentation for Robotic Environments
Christopher Xie, Yu Xiang, Arsalan Mousavian and Dieter Fox
In IEEE Transactions on Robotics (T-RO), 2021.
arXiv, Bibtex, Project, Code
PoseRBPF: A Rao-Blackwellized Particle Filter for 6D Object Pose Tracking
Xinke Deng, Arsalan Mousavian, Yu Xiang, Fei Xia, Timothy Bretl and Dieter Fox
In IEEE Transactions on Robotics (T-RO), 2021.
PDF, Bibtex, Video, Code

2020

Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentation
Yu Xiang, Christopher Xie, Arsalan Mousavian and Dieter Fox
In Conference on Robot Learning (CoRL), 2020.
arXiv, PDF, Bibtex, Video, Code
Manipulation Trajectory Optimization with Online Grasp Synthesis and Selection
Lirui Wang, Yu Xiang and Dieter Fox
In Robotics: Science and Systems (RSS), 2020.
arXiv, PDF, Bibtex, Video, Project, Code
LatentFusion: End-to-End Differentiable Reconstruction and Rendering for Unseen Object Pose Estimation
Keunhong Park, Arsalan Mousavian, Yu Xiang and Dieter Fox
In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
arXiv, PDF, Bibtex, Project, Code
Scaling Local Control to Large-Scale Topological Navigation
Xiangyun Meng, Nathan Ratliff, Yu Xiang and Dieter Fox
In International Conference on Robotics and Automation (ICRA), 2020.
arXiv, PDF, Bibtex, Video, Project
Self-supervised 6D Object Pose Estimation for Robot Manipulation
Xinke Deng, Yu Xiang, Arsalan Mousavian, Clemens Eppner, Timothy Bretl and Dieter Fox
In International Conference on Robotics and Automation (ICRA), 2020.
arXiv, PDF, Bibtex, Video

2019

The Best of Both Modes: Separately Leveraging RGB and Depth for Unseen Object Instance Segmentation
Christopher Xie, Yu Xiang, Arsalan Mousavian and Dieter Fox
In Conference on Robot Learning (CoRL), 2019.
arXiv, PDF, Bibtex, Video, Project, Code
PoseRBPF: A Rao-Blackwellized Particle Filter for 6D Object Pose Tracking
Xinke Deng, Arsalan Mousavian, Yu Xiang, Fei Xia, Timothy Bretl and Dieter Fox
In Robotics: Science and Systems (RSS), 2019.
arXiv, PDF, Bibtex, Video, Code
Object Discovery in Videos as Foreground Motion Clustering
Christopher Xie, Yu Xiang, Zaid Harchaoui and Dieter Fox
In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
arXiv, PDF, Bibtex, Video
Neural Autonomous Navigation with Riemannian Motion Policy
Xiangyun Meng, Nathan Ratliff, Yu Xiang and Dieter Fox
In International Conference on Robotics and Automation (ICRA), 2019.
arXiv, PDF, Bibtex, Poster, Project

2018

Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects
Jonathan Tremblay, Thang To, Balakumar Sundaralingam, Yu Xiang, Dieter Fox and Stan Birchfield
In Conference on Robot Learning (CoRL), 2018.
arXiv, Bibtex, Project, Code
DeepIM: Deep Iterative Matching for 6D Pose Estimation
Yi Li, Gu Wang, Xiangyang Ji, Yu Xiang and Dieter Fox
In European Conference on Computer Vision (ECCV), 2018.
arXiv, PDF, Bibtex, Technical_Report, Project, IJCV_Version, Code MXNet, Code PyTorch (Oral)
PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes
Yu Xiang, Tanner Schmidt, Venkatraman Narayanan and Dieter Fox
In Robotics: Science and Systems (RSS), 2018.
arXiv, PDF, Bibtex, Code TensorFlow, Code PyTorch, Project
Recurrent Autoregressive Networks for Online Multi-Object Tracking
Kuan Fang, Yu Xiang, Xiaocheng Li and Silvio Savarese
In IEEE Winter Conference on Applications of Computer Vision (WACV), 2018.
arXiv, PDF, Bibtex, Poster, Slides

2017

DA-RNN: Semantic Mapping with Data Associated Recurrent Neural Networks
Yu Xiang and Dieter Fox
In Robotics: Science and Systems (RSS), 2017.
arXiv, PDF, Bibtex, Poster, Slides, Code, Project
Subcategory-aware Convolutional Neural Networks for Object Proposals and Detection
Yu Xiang, Wongun Choi, Yuanqing Lin and Silvio Savarese
In IEEE Winter Conference on Applications of Computer Vision (WACV), 2017.
arXiv, PDF, Bibtex, Technical_Report, Poster, Slides, KITTI_Results

2016

Anticipating Accidents in Dashcam Videos
Fu-Hsiang Chan, Yu-Ting Chen, Yu Xiang and Min Sun
In Asian Conference on Computer Vision (ACCV), 2016.
PDF, Bibtex, Project (Oral)
ObjectNet3D: A Large Scale Database for 3D Object Recognition
Yu Xiang, Wonhui Kim, Wei Chen, Jingwei Ji, Christopher Choy, Hao Su, Roozbeh Mottaghi, Leonidas Guibas and Silvio Savarese
In European Conference on Computer Vision (ECCV), pp. 160-176, 2016.
PDF, Bibtex, Technical_Report, Poster, Slides, ObjectNet3D (Spotlight Oral)
Pose Estimation Errors, the Ultimate Diagnosis
Carolina Redondo-Cabrera, Roberto López-Sastre, Yu Xiang, Tinne Tuytelaars and Silvio Savarese
In European Conference on Computer Vision (ECCV), pp. 118-134, 2016.
PDF, Bibtex, Code
Deep Metric Learning via Lifted Structured Feature Embedding
Hyun Oh Song, Yu Xiang, Stefanie Jegelka and Silvio Savarese
In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4004-4012, 2016.
arXiv, PDF, Bibtex, Technical_Report, Code, Project (Spotlight Oral)

2015

Learning to Track: Online Multi-Object Tracking by Decision Making
Yu Xiang, Alexandre Alahi and Silvio Savarese
In International Conference on Computer Vision (ICCV), pp. 4705-4713, 2015.
PDF, Bibtex, Technical_Report, Poster, Slides, MOT_Results, KITTI_Results, Code, Project (Oral)
Data-Driven 3D Voxel Patterns for Object Category Recognition
Yu Xiang, Wongun Choi, Yuanqing Lin and Silvio Savarese
In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1903-1911, 2015.
PDF, Bibtex, Technical_Report, Poster, Slides, KITTI_Results, Code, Project (Oral)
A Coarse-to-Fine Model for 3D Pose Estimation and Sub-category Recognition
Roozbeh Mottaghi, Yu Xiang and Silvio Savarese
In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 418-426, 2015.
PDF, Bibtex, Technical_Report, Poster, Project

2014

Monocular Multiview Object Tracking with 3D Aspect Parts
Yu Xiang*, Changkyu Song*, Roozbeh Mottaghi and Silvio Savarese (*equal contribution)
In European Conference on Computer Vision (ECCV), pp. 220-235, 2014.
PDF, Bibtex, Technical_Report, Poster, Slides, Code, Project
Beyond PASCAL: A Benchmark for 3D Object Detection in the Wild
Yu Xiang, Roozbeh Mottaghi and Silvio Savarese
In IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 75-82, 2014.
PDF, Bibtex, Poster, Slides, PASCAL3D+

2013

Object Detection by 3D Aspectlets and Occlusion Reasoning
Yu Xiang and Silvio Savarese
In the 4th International IEEE Workshop on 3D Representation and Recognition in ICCV (3dRR), pp. 530-537, 2013.
PDF, Bibtex, Technical_Report, Slides, Code, Project

2012

Object Co-detection
Sid Yingze Bao, Yu Xiang and Silvio Savarese
In European Conference on Computer Vision (ECCV), vol. 7572, pp. 86-101, 2012.
PDF, Bibtex, Poster, Slides, Project
Estimating the Aspect Layout of Object Categories
Yu Xiang and Silvio Savarese
In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3410-3417, 2012.
PDF, Bibtex, Technical Report, Poster, Slides, Code, Project

2010

Semantic Context Modeling with Maximal Margin Conditional Random Fields for Automatic Image Annotation
Yu Xiang, Xiangdong Zhou, Zuotao Liu, Tat-Seng Chua and Chong-Wah Ngo
In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3368-3375, 2010.
PDF, Bibtex, Technical Report
Learning Contextual Metrics for Automatic Image Annotation
Zuotao Liu, Xiangdong Zhou, Yu Xiang and Yan-Tao Zheng
In Advances in Multimedia Information Processing - PCM, vol. 6297, pp. 124-135, 2010.
PDF, Bibtex

2009

A Revisit of Generative Model for Automatic Image Annotation using Markov Random Fields
Yu Xiang, Xiangdong Zhou, Tat-Seng Chua and Chong-Wah Ngo
In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1153-1160, 2009.
PDF, Bibtex
Adaptive Model for Web Image Semantic Automatic Image Annotation
Hongtao Xu, Xiangdong Zhou, Yu Xiang and Baile Shi
In Journal of Software (in Chinese), vol. 21, no. 9, pp. 2183-2195, 2009.
PDF, Bibtex
Exploiting Flickr's Related Tags for Semantic Annotation of Web Images
Hongtao Xu, Xiangdong Zhou, Mei Wang, Yu Xiang and Baile Shi
In Proceedings of the ACM International Conference on Image and Video Retrieval (CIVR), no. 46, 2009.
PDF, Bibtex
Automatic Web Image Annotation via Web-Scale Image Semantic Space Learning
Hongtao Xu, Xiangdong Zhou, Lan Lin, Yu Xiang and Baile Shi
In Advances in Data and Web Management, vol. 5446, pp. 211-222, 2009.
PDF, Bibtex

PhD Thesis

Master Thesis

Talks

  • Perceive, Plan, Act and Learn: Towards Intelligent Robots in Human Environments (PDF)
    UNC, 2/24/2021; UT Dallas, 3/16/2021.

  • Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentation (PDF)
    In NVIDIA Research, Seattle, Washington, 10/12/2020.

  • PoseRBPF: A Rao-Blackwellized Particle Filter for 6D Object Pose Tracking (PDF)
    In University of Washington, Seattle, Washington, 9/27/2019.

  • Object Perception for Robot Manipulation (PDF)
    In Toyota Research Institute, Cambridge, Massachusetts, 7/12/2019.

  • PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes (PDF)
    In Robotics: Science and Systems (RSS), CMU, Pittsburgh, Pennsylvania, 6/26/2018.

  • Perceiving the 3D World from Images and Videos (PDF)
    Nvidia Research, Redmond, Washington, 11/07/2017; University of Michigan, 3/15/2018.

  • 3D Object Recognition and Scene Understanding from RGB-D Videos (PDF)
    GRASP Lab at Penn, 10/11/2017; Microsoft Research, 10/17/2017; Vision Lab at Stanford, 10/23/2017.

  • 3D Object Recognition and Scene Understanding (PDF)
    In Mitsubishi Electric Research Laboratories, Boston, Massachusetts, 7/14/2017.

  • DA-RNN: Semantic Mapping with Data Associated Recurrent Neural Networks (PDF)
    In Robotics: Science and Systems (RSS), MIT, Massachusetts, 7/13/2017.

  • Subcategory-aware Convolutional Neural Networks for Object Proposals and Detection (PDF)
    In IEEE Winter Conference on Applications of Computer Vision, Santa Rosa, California, 3/29/2017.

  • 3D Object Recognition (PDF)
    In the International Conference on 3D Vision, Stanford University, 10/28/2016.

  • 3D Object Representations for Recognition (PDF)
    VASC Seminar, CMU, 3/28/2016; University of Toronto, 4/4/2016; MIT, 4/12/2016; UC Berkeley, 4/21/2016; UIUC, 5/5/2016; University of Washington, 5/31/2016.

  • 3D Object Detection and Pose Estimation (PDF)
    In the 1st International Workshop on Recovering 6D Object Pose in conjunction with ICCV, Santiago, Chile, 12/17/2015.

  • Learning to Track: Online Multi-Object Tracking by Decision Making (PDF)
    In International Conference on Computer Vision, Santiago, Chile, 12/16/2015.

  • Data-Driven 3D Voxel Patterns for Object Category Recognition (PDF)
    In IEEE Conference on Computer Vision and Pattern Recognition, Boston, Massachusetts, 06/08/2015.

  • Monocular Multiview Object Tracking with 3D Aspect Parts (PDF)
    In the 1st Stanford-SNU Workshop on Automated Driving, Stanford University, 02/24/2015.

  • Beyond PASCAL: A Benchmark for 3D Object Detection in the Wild (PDF)
    In IEEE Winter Conference on Applications of Computer Vision, Steamboat Springs, Colorado, 03/24/2014.

  • Object Detection by 3D Aspectlets and Occlusion Reasoning (PDF)
    In the 4th International IEEE Workshop on 3D Representation and Recognition in conjunction with ICCV, Sydney, Australia, 12/08/2013.

  • Estimating the Aspect Layout of Object Categories (PDF)
    In Midwest Vision Workshop, University of Illinois at Urbana-Champaign, 09/21/2012.

Links