Nowadays neural-network-based image- and video-quality metrics show better
performance compared to traditional methods. However, they also became more
vulnerable to adversarial attacks that increase metrics’ scores without
improving visual quality. The existing benchmarks of quality metrics compare
their performance in terms of correlation with subjective quality and
calculation time. However, the adversarial robustness of image-quality metrics
is also an area worth researching. In this paper, we analyse modern metrics’
robustness to different adversarial attacks. We adopted adversarial attacks
from computer vision tasks and compared attacks’ efficiency against 15
no-reference image/video-quality metrics. Some metrics showed high resistance
to adversarial attacks which makes their usage in benchmarks safer than
vulnerable metrics. The benchmark accepts new metrics submissions for
researchers who want to make their metrics more robust to attacks or to find
such metrics for their needs. Try our benchmark using pip install
robustness-benchmark.
Deep Analysis of Neural-Network-Based Image- and Video-Quality Metrics
In recent years, neural-network-based image- and video-quality metrics have shown remarkable advancements in terms of performance compared to traditional methods. However, with this progress comes an increased vulnerability to adversarial attacks that can manipulate the scores of these metrics without actually improving the visual quality. In this multidisciplinary study, we investigate the robustness of modern metrics against various adversarial attacks.
The field of multimedia information systems encompasses various domains such as computer vision, machine learning, and human-computer interaction. Understanding the performance and vulnerabilities of image- and video-quality metrics is crucial for developing reliable multimedia systems that can accurately assess the visual quality of images and videos.
Animations, artificial reality, augmented reality, and virtual realities are all interconnected with multimedia information systems. These technologies heavily rely on accurate assessment and manipulation of visual content. Therefore, it is essential to evaluate the robustness of quality metrics in these areas to ensure a seamless user experience.
In our comprehensive analysis, we compared the efficiency and resilience of 15 state-of-the-art no-reference image/video-quality metrics against adversarial attacks derived from computer vision tasks. By subjecting these metrics to various attacks, we gained valuable insights into their susceptibility and possible vulnerabilities.
Interestingly, some metrics exhibited high resistance to adversarial attacks, making them safer choices for benchmarking purposes. These robust metrics can provide reliable and consistent assessments of image and video quality even in the presence of adversarial manipulation.
Our benchmark framework offers researchers a platform to submit their own metrics, allowing them to enhance the robustness of their models against adversarial attacks or identify suitable metrics for their specific requirements. Using pip install robustness-benchmark, researchers can easily access and utilize this benchmark for their experiments and studies.
In conclusion, this study highlights the importance of examining the adversarial robustness of neural-network-based image- and video-quality metrics. By analyzing their vulnerabilities, we can improve the reliability and accuracy of multimedia systems and ensure a seamless user experience across various domains such as animations, artificial reality, augmented reality, and virtual realities.