Panoptic Video Scene Graph Generation

Jingkang Yang1,*, Wenxuan Peng1,*, Xiangtai Li1,
Zujin Guo1, Liangyu Chen1, Bo Li1, Zheng Ma2,
Kaiyang Zhou1, Wayne Zhang2, Chen Change Loy1, Ziwei Liu1,✉
1 S-Lab, Nanyang Technological University    2 SenseTime Research
*Main Contributors   ✉Corresponding Author
Accepted to CVPR 2023
arXiv

🤗

 Dataset Code Annotation Tool

🎞

 Demo Video

The PVSG Dataset comprises 400 videos characterized by their length (averaging 76.5 seconds), perspective diversity (combining first and third-person views across different scenarios), and dynamism (featuring significant camera and object motion), with rich annotation includes Video Panoptic Segmentation and Temporal Scene Graph on 150K frames, and video-level dense captions and QA pairs. Here we show some examples. All 400 visual examples are [here].

Abstract

Towards building comprehensive real-world visual perception systems, we propose and study a new problem called panoptic scene graph generation (PVSG). PVSG is related to the existing video scene graph generation (VidSGG) problem, which focuses on temporal interactions between humans and objects localized with bounding boxes in videos. However, the limitation of bounding boxes in detecting non-rigid objects and backgrounds often causes VidSGG systems to miss key details that are crucial for comprehensive video understanding. In contrast, PVSG requires nodes in scene graphs to be grounded by more precise, pixel-level segmentation masks, which facilitate holistic scene understanding. To advance research in this new area, we contribute a high-quality PVSG dataset, which consists of 400 videos (289 third-person + 111 egocentric videos) with totally 150K frames labeled with panoptic segmentation masks as well as fine, temporal scene graphs. We also provide a variety of baseline methods and share useful design practices for future work.

Dataset Overview


Methods and Experimental Results

Paper (Latest)

BibTeX

@article{yang2023pvsg,
  title={Panoptic video scene graph generation},
  author={Yang, Jingkang and Peng, Wenxuan and Li, Xiangtai and Guo, Zujin and Chen, Liangyu and Li, Bo and Ma, Zheng and Zhou, Kaiyang and Zhang, Wayne and Loy, Chen Change and others},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={18675--18685},
  year={2023}
}