Creating and sharing knowledge for telecommunications

Project: PLenoptics for Immersive Visual Experience

Acronym: PLIVE
Main Objective:
This project will research new coding techniques for plenoptic data and associated quality evaluation methodologies. It will consider the three plenoptic representations: light fields, point clouds and digital holography. Plenoptic data is typically represented by huge amounts of data, making efficient compression of utmost importance.
Hence this project aims to research compression models for plenoptic data:
- In the case of digital holography new compression models are required for the hologram compression on the hologram plane. The distribution of the frequencies in the reconstruction plane (or visual domain) do not correspond to the ones on the holographic plane. Hence, the typical quantization step of the lossy codecs should not be used in the holographic plane because the quantized frequencies will not be the highest ones on the visual domain. New quantization models need to be investigated for lossy compression in the hologram plane.
- In the case of light field compression, new models that allow features such as: 1) improved random access to different views and 2) progressive decoding. This will avoid having to decode the complete bit stream that might cause a huge overhead on the system memory. Moreover, the profiles of JPEG Pleno will be studied in order to incorporate these important features.
- In the case of point cloud compression, new models that allow the access to a part of the point cloud or that allow the extraction of a point cloud with a lower quality or resolution will be researched. The current compression solutions will be studied to understand how to create specific access to a part of the point cloud and to provide progressive decoding with different layers of quality or resolution.
- Although more immature, scalability and random access will also be considered for holographic data. The partial decoding of selected parts of the holographic data allow the selection of specific views with different resolutions.
Dealing with the development and evaluation of compressing tools, quality evaluation is required when lossy compression tools are involved. Although a strong research effort for the study of effective subjective quality analysis models of volumetric data has recently taken place, these methodologies are mostly limited to 2D representations. Moreover, no effective quality analysis mechanisms have been defined, which consider scalability and random access functionalities. Hence, the project aims to research suitable quality evaluation protocols that allow:
- The development of subjective quality models for immersive environments simulated using head mounted displays, under different compression methods and bit rates. This requires appropriate rendering of the plenoptic data:
o without causing additional distortions, while keeping and displaying the distortions caused by the compression mechanisms;
o creating a natural visualization environment, using appropriate knowledge of the human visual system, and avoiding the visualization problems caused by long exposition to virtual reality environments that lead to nausea or disorientation.
- The development of quality models (subjective or objective) that allow to quantify the scalable and random access solutions’ quality of experience performance. These studies will be considered in the three plenoptic modalities.
It is important to mention that, although the objectives of this project are ambitious, the research team has already gained a strong experience working in the development of both coding tools and quality evaluation models of plenoptic data, inside the EmergIMG project and within the collaboration with the JPEG Pleno development. This project will result in a follow up that will use and reinforce the synergies already created.
Reference: UIDB/EEA/50008/2020
Funding: IT/LA
Start Date: 01-09-2020
End Date: 01-09-2022
Team: António Manuel Gonçalves Pinheiro, Maria Manuela Areias da Costa Pereira de Sousa, Luis Alberto da Silva Cruz, Marco André Vieira Andrade Bernardo, Fernando Jose Pimentel Lopes
Groups: Multimedia Signal Processing – Cv, Multimedia Signal Processing – Co
Partners: FibEnTech
Local Coordinator: António Manuel Gonçalves Pinheiro
Associated Publications