UC San Diego

Augmented reality (AR), virtual reality (VR) and immersive video, as emerging types of multimedia, have recently gained more and more attention. With the development of devices that can capture 3D content and a rapid increase of related applications, the delivery and storage of 3D content have become an important research area. MPEG hosted a Call for Proposals to collect ideas to efficiently compress 3D content in three categories: static point clouds, dynamic point cloud sequences and dynamic acquisition. Among the proposals, Video-based Point Cloud Compression (V-PCC) achieves the highest quality for the second category, dynamic point cloud sequences, under a bit rate constraint.

However, the V-PCC framework is not spatially scalable. In this research, interpolation components are proposed for the V-PCC framework to make it suitable for flexible spatial resolution. As outliers might be brought in by the interpolation, a patch-aware averaging filter is applied to eliminate most outliers. Experimental results show that the interpolation component performs well both on objective evaluation and subjective visual quality.

Point cloud and mesh are the two main representations for 3D content. While compression methods for them are actively studied, there are few studies of their perceptual compression quality and none that consider observation distance. We studied the perceptual quality of compressed 3D sequences, for both a point cloud compression method (V-PCC) and a mesh-based compression method (Triangle FAN (TFAN)). Two main factors that could impact perceptual quality are considered, bit rate and observation distance. Evaluation of perceptual quality is carried out both by collecting viewer opinion scores of the compressed sequences separately, and with a side-by-side comparison. A functional model for mesh and point cloud compression quality is estimated to predict Mean Opinion Score (MOS) which yields high Pearson correlation and rank correlation scores with measured MOS.

Although V-PCC achieves the highest quality among methods proposed to MPEG, outliers and various other artifacts can degrade the V-PCC quality especially when high quantization parameter (QP) is set. After examining the causes and types of V-PCC artifacts that occur, we propose a framework to remove the highly noticeable outlier and crack artifacts caused by V-PCC so as to improve compressed point cloud visual quality. A subjective experiment showed that our approach significantly improves visual quality, and the improvement becomes more obvious with increasing QP values.