arXiv:2310.06958v4 Announce Type: replace-cross
Abstract: Nowadays, neural-network-based image- and video-quality metrics perform better than traditional methods. However, they also became more vulnerable to adversarial attacks that increase metrics’ scores without improving visual quality. The existing benchmarks of quality metrics compare their performance in terms of correlation with subjective quality and calculation time. Nonetheless, the adversarial robustness of image-quality metrics is also an area worth researching. This paper analyses modern metrics’ robustness to different adversarial attacks. We adapted adversarial attacks from computer vision tasks and compared attacks’ efficiency against 15 no-reference image- and video-quality metrics. Some metrics showed high resistance to adversarial attacks, which makes their usage in benchmarks safer than vulnerable metrics. The benchmark accepts submissions of new metrics for researchers who want to make their metrics more robust to attacks or to find such metrics for their needs. The latest results can be found online: https://videoprocessing.ai/benchmarks/metrics-robustness.html.
Analysis of Modern Image- and Video-Quality Metrics’ Robustness to Adversarial Attacks
Image- and video-quality metrics play a crucial role in assessing the visual quality of multimedia content. With the advancements in neural-network-based metrics, the performance of these metrics has significantly improved. However, these advancements have also introduced a new vulnerability – adversarial attacks.
Adversarial attacks manipulate certain features of an image or video in a way that increases the quality metric scores without actually improving the visual quality. This poses a significant threat to the integrity of quality assessment systems and calls for research into adversarial robustness.
This paper focuses on analyzing the robustness of 15 prominent no-reference image- and video-quality metrics to different adversarial attacks. By adapting adversarial attacks commonly used in computer vision tasks, the authors were able to evaluate the efficiency of these attacks against the metrics under consideration.
The results of the analysis showcased varying degrees of resistance to adversarial attacks among the different metrics. Some metrics demonstrated a high level of robustness, indicating their reliability in real-world scenarios and making them safer options for benchmarking purposes. On the other hand, certain metrics showed vulnerabilities to the attacks, raising concerns about their suitability for quality assessment.
This multi-disciplinary study bridges the fields of multimedia information systems, animations, artificial reality, augmented reality, and virtual realities. It highlights the importance of considering the robustness of image- and video-quality metrics in these domains, where accurate quality assessment is crucial for user experience and content optimization.
The research also addresses the need for a benchmark that includes adversarial robustness as a criterion to evaluate and compare different metrics. By providing a platform for researchers to submit their metrics, this benchmark fosters the development of more robust quality metrics and aids in finding suitable metrics for specific needs.
The topic of adversarial attacks and robustness has gained significant attention in recent years, and this paper adds valuable insights to the ongoing discourse. Researchers and practitioners can refer to the online platform mentioned in the article to access the latest benchmark results and stay updated with the advancements in this field.
Conclusion
As the reliance on neural-network-based image- and video-quality metrics continues to grow, understanding their vulnerabilities to adversarial attacks is crucial. This paper’s analysis of modern metrics’ robustness provides valuable insights into the effectiveness of various attacks on different metrics. It emphasizes the importance of considering robustness in benchmarking and highlights the need for more research in this area.
Furthermore, the integration of multiple disciplines such as multimedia information systems, animations, artificial reality, augmented reality, and virtual realities demonstrates the wide applicability and impact of this research. It encourages collaboration across these fields to develop more robust quality assessment techniques that can enhance user experience and optimize multimedia content.
Overall, this study contributes to the ongoing efforts in ensuring the reliability and security of image- and video-quality assessment systems, paving the way for advancements in the field and fostering innovation in research and development.
Reference: Announce Type: replace-cross (arXiv:2310.06958v4)