arXiv:2502.04371v1 Announce Type: new
Abstract: This paper presents Perceptual Preference Optimization (PerPO), a perception alignment method aimed at addressing the visual discrimination challenges in generative pre-trained multimodal large language models (MLLMs). To align MLLMs with human visual perception process, PerPO employs discriminative rewarding to gather diverse negative samples, followed by listwise preference optimization to rank them.By utilizing the reward as a quantitative margin for ranking, our method effectively bridges generative preference optimization and discriminative empirical risk minimization. PerPO significantly enhances MLLMs’ visual discrimination capabilities while maintaining their generative strengths, mitigates image-unconditional reward hacking, and ensures consistent performance across visual tasks. This work marks a crucial step towards more perceptually aligned and versatile MLLMs. We also hope that PerPO will encourage the community to rethink MLLM alignment strategies.
Perceptual Preference Optimization: Enhancing Visual Discrimination in Generative Pre-trained Multimodal Large Language Models
Generative pre-trained multimodal large language models (MLLMs) have shown remarkable capabilities in natural language understanding and generation. However, these models often struggle with visual discrimination tasks, where their performance lags behind human perception. This paper introduces Perceptual Preference Optimization (PerPO), a method aimed at improving the visual discrimination abilities of MLLMs.
PerPO takes a multi-disciplinary approach, combining insights from perceptual psychology, machine learning, and optimization. The method leverages discriminative rewarding to gather a diverse set of negative samples, representing challenging visual discrimination scenarios. By ranking these negative samples using listwise preference optimization, PerPO aligns MLLMs with human visual perception.
A key aspect of PerPO is its use of the reward as a quantitative margin for ranking. This bridges the gap between generative preference optimization and discriminative empirical risk minimization, combining the strengths of both approaches. By doing so, PerPO effectively enhances MLLMs’ visual discrimination capabilities while preserving their generative strengths.
One important contribution of PerPO is mitigating the issue of image-unconditional reward hacking. This refers to the phenomenon where models exploit biases or artifacts in the reward signal to achieve high scores without truly understanding or discriminating the visual content. By incorporating diverse negative samples and utilizing listwise preference optimization, PerPO helps prevent reward hacking, leading to more reliable and consistent performance across various visual tasks.
This work represents a significant step towards creating more perceptually aligned and versatile MLLMs. By addressing the visual discrimination challenges, PerPO opens up new possibilities for applications that require both natural language understanding and accurate visual perception. Furthermore, this paper encourages the research community to rethink MLLM alignment strategies, emphasizing the importance of considering visual perception in multimodal models.