Many XR applications require the delivery of volumetric video to users with
six degrees of freedom (6-DoF) movements. Point Cloud has become a popular
volumetric video format. A dense point cloud consumes much higher bandwidth
than a 2D/360 degree video frame. User Field of View (FoV) is more dynamic with
6-DoF movement than 3-DoF movement. To save bandwidth, FoV-adaptive streaming
predicts a user’s FoV and only downloads point cloud data falling in the
predicted FoV. However, it is vulnerable to FoV prediction errors, which can be
significant when a long buffer is utilized for smoothed streaming. In this
work, we propose a multi-round progressive refinement framework for point cloud
video streaming. Instead of sequentially downloading point cloud frames, our
solution simultaneously downloads/patches multiple frames falling into a
sliding time-window, leveraging the inherent scalability of octree-based
point-cloud coding. The optimal rate allocation among all tiles of active
frames are solved analytically using the heterogeneous tile rate-quality
functions calibrated by the predicted user FoV. Multi-frame
downloading/patching simultaneously takes advantage of the streaming smoothness
resulting from long buffer and the FoV prediction accuracy at short buffer
length. We evaluate our streaming solution using simulations driven by real
point cloud videos, real bandwidth traces, and 6-DoF FoV traces of real users.
Our solution is robust against the bandwidth/FoV prediction errors, and can
deliver high and smooth view quality in the face of bandwidth variations and
dynamic user and point cloud movements.
Expert Commentary: The Multi-Disciplinary Nature of Point Cloud Video Streaming
Point cloud video streaming is an important aspect of multimedia information systems, as it enables the delivery of volumetric video with six degrees of freedom (6-DoF) movements to users. This technology is a multi-disciplinary field that combines concepts from animations, artificial reality, augmented reality, and virtual realities.
The article discusses the challenges of delivering point cloud videos, which consume higher bandwidth compared to traditional 2D or 360-degree videos. Additionally, the user’s field of view (FoV) is more dynamic with 6-DoF movement, making it necessary to optimize the streaming process to save bandwidth and provide a high-quality viewing experience.
To address these challenges, the proposed multi-round progressive refinement framework for point cloud video streaming is introduced. This framework simultaneously downloads and patches multiple frames falling into a sliding time-window, leveraging the scalability of octree-based point-cloud coding. By allocating the optimal rate among all tiles of active frames, the solution ensures high-quality viewability based on predicted user FoV.
The multi-disciplinary nature of this framework becomes evident when considering its various components. The use of point cloud videos brings in concepts from animations and 3D modeling, as it requires the representation of objects as a collection of points in 3D space. The integration of artificial reality, augmented reality, and virtual realities is crucial in understanding the user’s dynamic field of view and predicting their FoV accurately for optimized streaming.
From a multimedia information systems perspective, this framework addresses the challenge of delivering volumetric video effectively. Bandwidth efficiency is essential in multimedia systems, especially when dealing with resource-intensive formats like point clouds. By optimizing the rate allocation and leveraging the scalability of octree-based coding, the proposed solution tackles the bandwidth consumption issue and ensures a high-quality viewing experience.
The evaluation of the streaming solution using simulations driven by real point cloud videos, bandwidth traces, and 6-DoF FoV traces of real users demonstrates its robustness against bandwidth and FoV prediction errors. This is significant in the context of multimedia information systems, as it validates the effectiveness of the framework in delivering high and smooth view quality despite variations in bandwidth and dynamic user and point cloud movements.
In conclusion, point cloud video streaming is an area that intersects various disciplines within the field of multimedia information systems. The proposed multi-round progressive refinement framework addresses the challenges of delivering volumetric video with 6-DoF movements by optimizing rate allocation and leveraging octree-based coding. This solution demonstrates the multi-disciplinary nature of point cloud video streaming and its relevance to animations, artificial reality, augmented reality, and virtual realities.